Pull #54 breaks the Windows version of dump1090 due to a bug in the
inet_aton() windows function.
1) Fix the bug in the windows inet_aton()
2) Modify the command line code to strdup() the command line string.
3) Leave the Modes.net_bind_address pointer NULL if no command line
specified
Mostly refactoring the common code that was duplicated
between the different output types so that there aren't
many copies floating around.
This introduces "struct net_writer" to store the state of a
particular type of output service - buffers, time of last write,
connection count etc. prepareWrite() / completeWrite() give access
to the buffer and handle the actual writes and flushing when needed.
Heartbeat and time-based flushing move into a generic periodic-work
function.
Update the SBS output code to use the new infrastructure. This makes
a big different to CPU use when under load.
When we read from some client A, we may end up forwarding a message to other
clients. If we forward to some client B and there is a write error, then
we close B and remove it from the client list. However, if before this happened
A->next == B, then the read loop will still be holding on to a pointer to B,
and we crash.
As it's unpredictable what clients could be closed at what point, the simplest
approach is to retain closed clients in the list until we are at a point where
we know there are no stray pointers on stack, and only then modify the list.
This also simplifies anything that has to loop over clients, as it doesn't need
to worry about the current client being freed under it.
If we demodulate a message in 2.4MHz mode and it has a bad, uncorrectable CRC,
and --phase-enhance is on, then retry with the other possible phases until
we get a good CRC or run out of phases to try.
This is very expensive in AGC mode (lots of candidates that are not real
messages) but relatively cheap otherwise. It yields another 10% messages.
Also factor out some common stats code to avoid lots more copy/paste.
When replaying recordings, the read files are binary, so the file open
needs to reflect this so that spurious end of file characters aren't
misinterpreted.
This disables most decoding of the contents of Mode S messages, aircraft tracking, and some output modes that depend on them.
It's intended for edge receivers that just forward to a central hub rather than processing data locally.
A few minor additions and bug fixes as detailed below
1) Additional command line option "--net-buffer <n>" to specify the TCP
output buffer size. Default is n=0, which is 64Kb. Specify a value of n
to increase the buffer size according to Size = 64Kb * 2^n, so an n of
1 = 128Kb, n=2 is 256Kb etc. n is limited to 7, so the max size is 8Mb.
This option may assist if you have a high number of aircraft being
received, and an unreliable network connection, or if the receiving end
can be busy for an extended time.
2) Bug fix in ppup1090 which prevented the uploading of valid
ModeA/Squawk codes
3) Bug fix per Markus Grab's commit.
Make the modifications necessary to compile dump1090 for WinXP, Win7 and
hopefully Win8.
The files can be compiled using M$ Visual Studio/C++ 6.0. Due to various
licensing issues, I haven't included the libraries or DLLs. You will
need to locate pthreadVC2.lib and rtlsdr.lib to link the file, install
the zadig drivers to support the dongle, and locate libusb-1.0.dll,
msvcr100.dll, pthreadVC2.dll and rtlsdr.dll.
dump1090.exe will not run on any Windows version prior to XP SP2,
because msvcr100.dll imports several functions from the Windows kernel
that are not available on earlier versions. This means dump1090 won't
work on Win2K.
The major change to the code relates to file handles. The original code
assumes Linux behaviour in that handles are allocated from 0
sequentially upwards. However Windows handles are allocated pseudo
randomly, and handle numbers greater than 1024 would break the code. The
code has therefore been modified to use a linked list of connection
structures, rather than a static array limited to 1024 entries.
Some users have reported issues where the TCP link to dump1090 can be
lost at times of low traffic density - typically in the middle of the
night. One possible reason for this is that some routers drop the link
if there is no traffic for a predetermined period.
To try and resolve this, dump1090 now sends a 'null' packet consisting
of 7 "0x00" bytes approximately once a minute if there is no real
received traffic during this time. This packet should be discarded by
the application receiving the dump1090 because it will have an invalid
checksum, and ICAO address 0x000000 is also invalid. However, this null
packet should be enough to keep routers alive.
The Mutex on the RTL data reader thread does not "force" the data
processing thread to execute. Therefore, if the processor is busy, it is
possible for a second RTL callback to occur before the data from the
first has been processed. This will cause the loss of the first data,
but worse, it will cause a slip in the timestamp. This upsets Beamfinder
and MLAT operation in PlanePlotter.
To solve this, keep a Fifo buffer which is filled by the callback
thread, and emptied by the data processing thread. The fifo is the same
size as the number of buffers requested in the call to
rtlsdr_read_async().
Note - we only put the value of the pointer supplied in the callback
into the fifo. We do not attempt to cache the data in the buffer pointed
to by the pointer. This would require us to memcopy() 2Mbytes per
second, which we don't want to do if we don't have to because it will
only make the processor loading worse. Instead, we assume that the data
in the buffer will remain valid after the callback returns, at least
until it is overwritten by new data.
It is still possible for us to lose data if we can't process it quickly
enough. However, we can now detect this loss of data when the fifo is
almost full, and correct the timestamp for the lost block/blocks.
Ok - this is likely to upset some people. Up until now, the vast
majority of the code has been in just one file - dump1090.c. This file
has grown so that it was approaching of 5000 lines long, and it was
becoming unmanagable. So I've split the file into several modules,
hopefully along fairly logical boundaries. The files are :
1) dump1090.c : Basically just the main() entry function, the help
function, the RTL dongle hardware interface, and a few orphan functions
that don't really fit anywhere else.
2) mode_s.c : This contains all the mode S / ADSB decoding functions.
3) mode_ac.c : This contains all the mode A & C decoding functions
4) interactive.c : This contains all the functions to maintain an
internal list of aircraft seen over the last period, and functions to
print them out to the local console.
5) net_io.c : This contains all the network input/output functions
allowing data to be passed in/out to/from other receivers, in formats
such as SBS-1/3, Beast, AVR and JavaScript.
Hopefully this should provide an easier way forward if/when more
functions are added.
Create two different Time-To-Live parameters for the interactive display
list.
1) Modes.interactive_display_ttl defines for how long the display of an
aircraft persists after the last received message. The default is
MODES_INTERACTIVE_DISPLAY_TTL, which is set to 60 seconds. This replaces
the previous Modes.interactive_ttl/MODES_INTERACTIVE_TTL settings (which
were also 60 seconds)
2) Add an additional Modes.interactive_delete_ttl parameter, and
default it to MODES_INTERACTIVE_DELETE_TTL, shich is set to 300
seconds/5 minutes. This defines for how long an aircraft persists in the
list structure after the last received message before being deleted.
Also tidy up a few comments.
Added option to show values in metric system.
Metric = false; // true|false
Removed unused conversion from aircraftsToJson() at dump1090.c-file.
Tweaked distance calculation to use google map api.
modified: dump1090.c
modified: public_html/config.js
modified: public_html/gmap.html
modified: public_html/script.js
This is the start of breaking the main dump109.c file into smaller
modules to make it a bit more maintainable.
Move all the #define and structure declarations into dump1090.h
Changes based on ideas from John VK1ET. His commit notes are as follows
:
1. Change input socket handling to avoid unnecessary memmove (use
pointers instead)
2. Add ability to read and decode binary beast format TCP.
3. Change output socket handling same as sbs_output - only call output
handler if there is a current client connected use separate ports for
beast in and out - no need for --net-beast flag. Will input/output
format defined by socket connected to. avr raw, beast binary and sbs can
be handled simultaneously.
4. Some comments changes, filtering of ModeAC to json,.
Resolves some merge conflicts that I couldn't find a way to fix during
the previous merge.
Remove a few Tabs that escaped in the edit/merge
Rename the fix_errors flag variable to nFix_crc since it now represents
the number of crc errors to attempt to fix.
Remove the aggressive variable since it's now part of nFix_crc
Only print phase correction statistics when phase correction is enabled.
Add an additional Modes.stat_ph_bit_fix[] to count the bits fixed during
phase correction
Sorry Ulrich - I can't get Github to resolve the merge errors and
preserve your commit notes, so I'll add them here.
Improvements on bit error correction, doc update, preparation for
program installation/package build
Hi,
I committed some further improvements on the bit error correction code,
updated the readme, and implemented a way to install the program in the
linux file system hierarchy (allows for package building).
Regards,
Ulrich
Bug fixes in the SBS output code as a result of an EMAIL from Liviu
Some software gets upset if null fields are output where the software is
expecting a valid value. This was being caused by DF0's outputting and
SBS MSG5. MSG5 needs Atert and SPI flags, which are derived from the FS
field. DF0 doesn't have an FS field, so cannot output a MSG5. Change so
that DF0 results in an SBS MSG7.
Also DF17/DF18 metypes 5 to 18 contain raw Lat/Lon data. However they
cannot be decoded into real Lat/Lon values unless both even and odd
parts are available Therefore, when we receive a DF17/DF18 we need to
check that we have successfully decoded the Lat/Lon before we attempt to
send an SBS MSG2 or MSG3. If we don't have a decoded Lat/Lon, send a
MSG7 instead.
Phase enhancement is used to try and increase the signal amplitude when
Nyquist sample aliasing is suspected. In previous versions, this was
enabled by default. In this version, the default is off. There is now an
additional command line switch --phase-enhance to turn it on.
There are also additional debug statistics which count the number of
phase demodulation failures in for both uncorrected and phase corrected
passes.
Ideally we should clear down the mm structure every loop of our
ModeA/C/S bit detector. However, we're getting 2Mbits of data per
second, and the structure is several tens of bytes long. Clearing down
every loop would require us to zero up to 100Mb per second. The memset
function may be fast, but it's still going to take up valuable processor
time.
So instead of clearing the whole structure every loop, just clear the
important parts.
Instead of force clearing mm.crc to zero if we successfully correct bit
errors, just use the mm.correctedbits variable. This allows us to print
out the crc value containing the errors during list output modes.
Actually use the (DF17) messages where we do bit correction.
Other changes to AVR message I/O.
1) treat mlat timestamp of zero as invalid (easier to work with than
(-1) and equally unlikely).
2) If we don't have a valid timestamp send a without timestamp
'*.......' message rather than '@.......' with invalid time.
3) Drop interpretation of obsolete AVR ' #' & '$' formats - they
wouldn't have worked anyway (wrong length).
If we error correct a DF17 frame, check that the ICAOaddr exists in our
recently seen ICAO cache. This reduces the likelyhood of birthday
paradox solutions producing false ICAO addresses
Restrict Syndromes to exclude DF bits. Do NOT introduce syndromes for
errors in the first 5 bits (DF field)
Use the --aggressive flag to see if we populate the two-bit error
syndromes. If --aggressive is not specified, then we don't attempt to
fix two bit errors.
vk1etI believe there is an error in the way the CRC error syndromes are
being created in the two bit case. The inner loop was introducing an
extra error bit each time through rather than moving it. Below is the
modified code fragment for syndrome creation collapsedSun 07:26
19 May 2013 07:26vk1et [notifications@github.com]Actions
To:Mantirez/dump1090 [dump1090@noreply.github.com]Cc:MMalcolmRobb
[Support@ATTAvionics.com]
To help protect your privacy, some content in this message has been
blocked. If you're sure this message is from a trusted sender and you
want to re-enable the blocked features, click here.
Bugfix first noted by VK1ET, and posted here :
https://github.com/antirez/dump1090/pull/23#issuecomment-18113094
I believe there is an error in the way the CRC error syndromes are being
created in the two bit case. The inner loop was introducing an extra
error bit each time through rather than moving it.
Updated the version number for Ulrich's code
Replaced some Tabs with spaces
Forward declaration of the fixBitErrors() function required for M$ VC
6.0 to suppress warning messages.
Based on a submission by VK1ET, but I've tweaked it a bit to use the
bFlags.
Basically, only print out values in --interactive that have been
received from the aircraft. Previous versions would display 0.000 for
lat and long for any positionless aircraft. This version prints blanks
instead for any/all fields that are not yet known.
Remove dependency of the SBS output code on the historic (a) aircraft
structure. The only items that were required were the decoded aircraft
Lat/Lon and these are now included in the mm structure.
Rewrite the SBS output code to use mm->bFlags when populating the output
fields. This ensures that all available data is output, and also that no
stale data is sent.
Using the mm->bFlags variable for SBS output means there is no further
requirement for the sbsFlags member in the aircraft structure, so remove
it.
Cross your fingers and hope this hasn't introduced too many bugs !
Implement additional bFlags to validate AircraftOnGround, and FS.
Decode Aircraft On Ground from DF0, DF4, DF5, DF16, DF20, and DF21 where
possible.
Flag FS as valid for DF4, DF5, DF20, DF21,
Remove dr and um from the mm structure. They're only used in console
list output mode, so decode them there if required.
Additional flags in the bFlags variables for even and odd CPR lat/lon
updates.
Change the interactiveReceiveData function to use the bFlags when
updating the aircraft (a) structure from the newly received (mm) message
structure. This should hopefully be faster than basically re-decoding
the DF, type and subtypes all over again.
If we decode the lat/lon into the aircraft (a) structure, back populate
it into the message (mm) structure. This allows us to print a decoded
Lat/Lon in the decoded message list output.
If --modeac is specified, then the program will be building an aircraft
list. Calculate the altitude when we create the aircraft structure (a) ,
and then back copy it into the mm structure. This avoids us having to do
a second ModeAtoModeC conversion in the list output display.
Also - bug fix in the ns_vel calculation.
Changes to 'dump1090.c'-file made accordingly.
modified: .gitignore
modified: dump1090.c
deleted: gmap.html
new file: public_html/gmap.html
new file: public_html/script.js
new file: public_html/style.css
Update the bFlags structure member to indicate which other structure
members contain decoded values.
Also, trim out mm structure members that are simple bitwise ands from
the raw data, unless the results are used in lots of places whereby
decoding them once is more efficient.
Extend the DF-17 CPR decoding to use User Lat/Lon for ground positions.
Also start implementing a bFlags structure variable to indicate which
fields in the mm and a data structures contain valid values.
Based on initial code supplied by VK1ET
Allow the user to specify their receivers (technically their aerials)
physical location. This is required for decoding ground positions, and
also for uploading aircraft data to various sharer sites.
The position can be hardwired by setting the MODES_USER_LATITUDE_DFLT
and MODES_USER_LONGITUDE_DFLT constants before compilation, or entered
at runtime via the new command line switches --lat and --lon. Either
way, the values are checked for validity before use.
Only clear mm after we are sure we have got a Mode A/C. This reduces
processor load.
Also tidy up some Linux compiler warnings that result from the last
change.
