This is the start of breaking the main dump109.c file into smaller
modules to make it a bit more maintainable.
Move all the #define and structure declarations into dump1090.h
Changes based on ideas from John VK1ET. His commit notes are as follows
:
1. Change input socket handling to avoid unnecessary memmove (use
pointers instead)
2. Add ability to read and decode binary beast format TCP.
3. Change output socket handling same as sbs_output - only call output
handler if there is a current client connected use separate ports for
beast in and out - no need for --net-beast flag. Will input/output
format defined by socket connected to. avr raw, beast binary and sbs can
be handled simultaneously.
4. Some comments changes, filtering of ModeAC to json,.
Resolves some merge conflicts that I couldn't find a way to fix during
the previous merge.
Remove a few Tabs that escaped in the edit/merge
Rename the fix_errors flag variable to nFix_crc since it now represents
the number of crc errors to attempt to fix.
Remove the aggressive variable since it's now part of nFix_crc
Only print phase correction statistics when phase correction is enabled.
Add an additional Modes.stat_ph_bit_fix[] to count the bits fixed during
phase correction
Sorry Ulrich - I can't get Github to resolve the merge errors and
preserve your commit notes, so I'll add them here.
Improvements on bit error correction, doc update, preparation for
program installation/package build
Hi,
I committed some further improvements on the bit error correction code,
updated the readme, and implemented a way to install the program in the
linux file system hierarchy (allows for package building).
Regards,
Ulrich
Bug fixes in the SBS output code as a result of an EMAIL from Liviu
Some software gets upset if null fields are output where the software is
expecting a valid value. This was being caused by DF0's outputting and
SBS MSG5. MSG5 needs Atert and SPI flags, which are derived from the FS
field. DF0 doesn't have an FS field, so cannot output a MSG5. Change so
that DF0 results in an SBS MSG7.
Also DF17/DF18 metypes 5 to 18 contain raw Lat/Lon data. However they
cannot be decoded into real Lat/Lon values unless both even and odd
parts are available Therefore, when we receive a DF17/DF18 we need to
check that we have successfully decoded the Lat/Lon before we attempt to
send an SBS MSG2 or MSG3. If we don't have a decoded Lat/Lon, send a
MSG7 instead.
Phase enhancement is used to try and increase the signal amplitude when
Nyquist sample aliasing is suspected. In previous versions, this was
enabled by default. In this version, the default is off. There is now an
additional command line switch --phase-enhance to turn it on.
There are also additional debug statistics which count the number of
phase demodulation failures in for both uncorrected and phase corrected
passes.
Ideally we should clear down the mm structure every loop of our
ModeA/C/S bit detector. However, we're getting 2Mbits of data per
second, and the structure is several tens of bytes long. Clearing down
every loop would require us to zero up to 100Mb per second. The memset
function may be fast, but it's still going to take up valuable processor
time.
So instead of clearing the whole structure every loop, just clear the
important parts.
Instead of force clearing mm.crc to zero if we successfully correct bit
errors, just use the mm.correctedbits variable. This allows us to print
out the crc value containing the errors during list output modes.
Actually use the (DF17) messages where we do bit correction.
Other changes to AVR message I/O.
1) treat mlat timestamp of zero as invalid (easier to work with than
(-1) and equally unlikely).
2) If we don't have a valid timestamp send a without timestamp
'*.......' message rather than '@.......' with invalid time.
3) Drop interpretation of obsolete AVR ' #' & '$' formats - they
wouldn't have worked anyway (wrong length).
If we error correct a DF17 frame, check that the ICAOaddr exists in our
recently seen ICAO cache. This reduces the likelyhood of birthday
paradox solutions producing false ICAO addresses
Restrict Syndromes to exclude DF bits. Do NOT introduce syndromes for
errors in the first 5 bits (DF field)
Use the --aggressive flag to see if we populate the two-bit error
syndromes. If --aggressive is not specified, then we don't attempt to
fix two bit errors.
vk1etI believe there is an error in the way the CRC error syndromes are
being created in the two bit case. The inner loop was introducing an
extra error bit each time through rather than moving it. Below is the
modified code fragment for syndrome creation collapsedSun 07:26
19 May 2013 07:26vk1et [notifications@github.com]Actions
To:Mantirez/dump1090 [dump1090@noreply.github.com]Cc:MMalcolmRobb
[Support@ATTAvionics.com]
To help protect your privacy, some content in this message has been
blocked. If you're sure this message is from a trusted sender and you
want to re-enable the blocked features, click here.
Bugfix first noted by VK1ET, and posted here :
https://github.com/antirez/dump1090/pull/23#issuecomment-18113094
I believe there is an error in the way the CRC error syndromes are being
created in the two bit case. The inner loop was introducing an extra
error bit each time through rather than moving it.
Updated the version number for Ulrich's code
Replaced some Tabs with spaces
Forward declaration of the fixBitErrors() function required for M$ VC
6.0 to suppress warning messages.
Based on a submission by VK1ET, but I've tweaked it a bit to use the
bFlags.
Basically, only print out values in --interactive that have been
received from the aircraft. Previous versions would display 0.000 for
lat and long for any positionless aircraft. This version prints blanks
instead for any/all fields that are not yet known.
Remove dependency of the SBS output code on the historic (a) aircraft
structure. The only items that were required were the decoded aircraft
Lat/Lon and these are now included in the mm structure.
Rewrite the SBS output code to use mm->bFlags when populating the output
fields. This ensures that all available data is output, and also that no
stale data is sent.
Using the mm->bFlags variable for SBS output means there is no further
requirement for the sbsFlags member in the aircraft structure, so remove
it.
Cross your fingers and hope this hasn't introduced too many bugs !
Implement additional bFlags to validate AircraftOnGround, and FS.
Decode Aircraft On Ground from DF0, DF4, DF5, DF16, DF20, and DF21 where
possible.
Flag FS as valid for DF4, DF5, DF20, DF21,
Remove dr and um from the mm structure. They're only used in console
list output mode, so decode them there if required.
Additional flags in the bFlags variables for even and odd CPR lat/lon
updates.
Change the interactiveReceiveData function to use the bFlags when
updating the aircraft (a) structure from the newly received (mm) message
structure. This should hopefully be faster than basically re-decoding
the DF, type and subtypes all over again.
If we decode the lat/lon into the aircraft (a) structure, back populate
it into the message (mm) structure. This allows us to print a decoded
Lat/Lon in the decoded message list output.
If --modeac is specified, then the program will be building an aircraft
list. Calculate the altitude when we create the aircraft structure (a) ,
and then back copy it into the mm structure. This avoids us having to do
a second ModeAtoModeC conversion in the list output display.
Also - bug fix in the ns_vel calculation.
Changes to 'dump1090.c'-file made accordingly.
modified: .gitignore
modified: dump1090.c
deleted: gmap.html
new file: public_html/gmap.html
new file: public_html/script.js
new file: public_html/style.css
Update the bFlags structure member to indicate which other structure
members contain decoded values.
Also, trim out mm structure members that are simple bitwise ands from
the raw data, unless the results are used in lots of places whereby
decoding them once is more efficient.
Extend the DF-17 CPR decoding to use User Lat/Lon for ground positions.
Also start implementing a bFlags structure variable to indicate which
fields in the mm and a data structures contain valid values.
Based on initial code supplied by VK1ET
Allow the user to specify their receivers (technically their aerials)
physical location. This is required for decoding ground positions, and
also for uploading aircraft data to various sharer sites.
The position can be hardwired by setting the MODES_USER_LATITUDE_DFLT
and MODES_USER_LONGITUDE_DFLT constants before compilation, or entered
at runtime via the new command line switches --lat and --lon. Either
way, the values are checked for validity before use.
Only clear mm after we are sure we have got a Mode A/C. This reduces
processor load.
Also tidy up some Linux compiler warnings that result from the last
change.
Based on a submission by VK1ET. Fully decode DF-17, metype 19. Take care
to only update fields that are valid in the data.
Also, initialise (to zero) the mm structures before use
1) Implement additional timestamps to remember when a new Lat/Lon value
is generated.
2) Increase callsign[] character array to 16 chars to make it long word
aligned.
3) Change some more comments from /* to //
Use memset to clear entire structures at initial creation to zero, and
then populate values that should not be zero. Do this rather than
individually clear each structure member to zero.
This should be faster, plus it's safer when new structure members are
added because you don't have to remember to initialise them.
Use 32 bit registers to hold multiple characters and shift them. This
removes the need for multiple memory reads.
Also tidy up the DF-17 extraction routine to make it more readable.
ICAO documents refer to AC13 for altitude encoding and ID13 for squawk
wncoding. Rename decodeGillhamField to decodeID13Field to better
represent this.
There is a quite complex if/else construct for gathering statistics
inside the main decodeModeS loop. To speed the loop up when not
gathering statistics, make the whole construct conditional on Modes.stat
being set.
Also tidy up a few comments
Allow the decodeModeMessage function to re-calculate msgtype and
msgbits. This makes the code a bit more portable by not requiring the
calling function to fill in these fields before the call.
Changes supplied by John VK1ET.
This tidies up the thread termination, which then allows for use of [a
trapped] Ctrl/C to cleanly terminate the program and dump stats if flag
set.
Lots of changes to the decodeModesMessages() function, and associated
helper functions, with the aim of speeding the code up by reducing the
number of calls to the CRC functions. Also, only update the mm variables
which are part of the message we're decoding.
Change the way CRC's are generated and checked.
Change the way single and double bit error is implemented.
General tidy up of the function to make it more readable.
Modify all the Mode=S Gillham altitude decoding to use a new function
decodeGillhamField()
Change the Mode-S signal strength so that it rounds to the nearest
integer
Assume that DF0-DF15 are all short 56 bit messages
Assume that DF16-DF31 are all long 112 bit messages
This helps us error detect and error recover for bits in the first byte
The DF's for altitude include a Q bit to distinguish either 100ft or
25ft altitude encoding. This version implements the 100ft increments
used by older aircraft. These codes are basically ModeC altitudes
transmitted over ModeS.
Reformat the --interactive display to show more information
Add a "Mode" column. This displays the The primary message type "S" or
"A" plus any matching modes "a" and/or "c". Requires --modeac switch.
Add a "Sig" column to show the signal strength.
Add a --fix command line switch to enable error correction using CRC.
Change the default to OFF/Disabled.
Error correction can very occasionally result in an incorrect solution
to the crc equation. It also takes significant CPU work for very few
corrected results. Therefore, the default has been changed to OFF, but
you can re-enable it if you wish by using the --fix command line switch.
Implement changes suggested by John/VK1ET
Changes to cprNFunction to ensure fflag {was 'isodd'} is used as a flag.
Fix to decodeCPRrelative to pass 'surface' flag.
The original code stores the squawk as a decimal (base 10) value. Change
this to a hex value to reduce the amount of maths required to convert
the input binary value..
Rename all the variables used to store the squawk to "modeA", rather
than variously identity, squawk and modeA.
Several variables within structures contain the same information in
different forms.
The aircraft structure contains addr and hexaddr. Hexaddr is a printable
string version of addr. However, hexaddr is only ever used in printf
statments, so we can use printf(%06x) to print addr. This saves a printf
for every received command.
The modesMessage structure contains addr, and aa1,aa2,aa3 as separate
bytes. aa1,aa2 and aa3 are only ever used to construct addr, and to
print out. Therefore, we can use addr instead of them..
The original code made an attempt to reconcile a newly arrived ModeA/C
message with every known Mode-S report at the time of detection.
However, the results of matching up Modes A/C and S are only used in the
interactive display routine, and that is only called periodically from
within the BackgroundTasks loop.
Doing the matching on every ModeA/C arrival incurs quite a large CPU
processing load. Moving the matching up routine to the backgroundTasks
loop means that the task is performed muck less frequently and therefore
uses less CPU time.
Allow a greater range and negative values for Mode C (down to -1200
feet)
Stop attempting to feed ModeA/C data to SBS Output stream.
Allow Mode A only matches to Mode-S squawks when the Mode A code does
not conflict with any possible (legal) Mode C code.
Allow Mode C matches to track aircraft climbing and descending
relatively slowly. This also helps when trying to match Mode-S altitudes
which are 25 foot increments, with Mode C altitudes which are in 100
foot increments.
First attempt at decoding legacy SSR Modes A and C.
If the command line switch --modeac is used, the program will now
attempt to recover Mode A/C signals contained in the raw I/Q data
stream. The current recovery mechanism is quite strict and does not cope
well with overlapping and corrupt SSR replies. I estimate that less than
20% of possible returns are decoded correctly. Hopefully over the next
few iterations this can be improved.
If outputting raw data it is recommended to use the --net-ro-size and
--net-ro-rate command line options to reduce the number of very small
ethernet packets that will be generated by mode A/C replies.
Thanks to vk1et for these.
1) Correct for additional timestamp langth in raw output buffer when
using mlat mode
2) Don't output a timestamp when the message has been received from a
remote site (the internet). This is to avoid upsetting MLAT because
there is an indeterminate delay between reception at the remote site and
subsequent message arrival in the local dump1090 instance.
3) Allow @ character for raw data input with timestamp, and correctly
calculate the length.
The original code calculates Lat/Long only if it receives two DF-17
(subtype 9 or 18) within one second of each other. I have no idea why.
It then caches the results in the aircrafts data structure for use in
the --interactive display.
When SBS-1 style ASCII output is selected (port 30003) the code does not
attempt to calculate Lat/Long from the data just received - instead it
picks it up from the cached information in the aircraft's data
structure.
However, if the data isn't being updated this results in stale Lat/Long
being sent out. This is most likely to occur when the aircraft is at the
extreme edge of the receivers range when it may be getting some DF-17s
containing Lat/Long, but not 2 per second. The program will continue
sending the stale data until the aircraft finally times out (default 60
seconds)
I have modified the code to include a sbsflags variable in the aircraft
data structure. When a new Lat/Long is decoded and put into the
structure, a bit is set to indicate SBS_LAT_LONG_FRESH. Then, once the
Lat/Long is output the first time, the bit is cleared. Thereafter the
code will not populate the Lat/Long fields in the output message until
SBS_LAT_LONG_FRESH is set again.
The default is 0. This works in conjunction with --net-ro-size.
The program will attempt to gather up "net-ro-size" raw bytes before
sending them out over Ethernet. However, to avoid a long wait if the
traffic density is very low, the program will only wait for
"net-ro-rate" 64mS periods since the last send. before sending any data
added to the output buffer since the last send. This allows the user to
tailor the network load to suit their requirements.
Move memcopy to outside the main bit loop, and just flip the modified
bit back at the end of each loop if it didn;t work
Pass in a pointer to the mm structure being corrected, and fix-up the
crc with the value inside the function, rather than re-calculate on
return
1) Populate Field 3 witn "111"
2) Populate field 4 with "11111"
3 Populate Field 6 with "111111"
4) End the record with <CRLF>, rather than just <LF>
5) Increase the ctrCommon buffer size to cope with additional field data
Apparently, this makes the output more compatible with Plane Plotter and
RTL1090.
Allow the user to specify the minimum size of raw data to be sent to the
TCP port. Dump1090 will buffer up raw data until it has at least this
many bytes to send to the TCP socket.
The default is 0, which means every frame is sent to the TCP socket as
it is decoded. The maximum value is limited to 1300 bytes.
Note the buffer will be flushed every 65 ms regardless of the amount of
data in it so that excessive in transmission do not occur.
The original code wrote every individual received frame to the TCP port.
Some O/S's buffer smaller writes into larger packets. It appears that
some versions of Linux don't. The result is that the (Ethernet) network
gets bombarded with lots of small Ethernet packets.
Therefore, I've added a 1500 byte output buffer to the raw output
functions. Data is written into this buffer by the raw output routines.
Data is flushed out to the TCP port when either.
1) The latest write to the output buffer takes the contents to more than
1300 bytes
2) At the end of every processed block of data supplied by rtl-sdr. This
will be every 56mS or so,
The end result should be that on systems detecting a lot of traffic, you
should see lots of > 1300-byte Ethernet packets. On systems receiving
less traffic, you should see one network packet every 56 mS or so.
The total number of network packets should be much reduced, and their
average size much bigger. The worst case delay in transmission will be
56 mS.
As requested by mlino
Note : I haven't been able to validate that the format is correct. I
think it should be Ok, but it needs someone with an SBS setup to check
it. Any offers?
M$ VC 6.0 does not like long long
UNIX compilers don't like printing int64_t or uint64_t as %lld
Raspberry Pi Linux doesn;t like PRId64
So I give up.
I've changed the affected variables to bog standard unsigned ints.
Assuming these compile as 32 bit unsigned's, its unlikely youll have the
program running long enough for these to overflow.
If noist/sampling/Nyquist errors cause bit detection errors in the DF
pare of the frame, then we may not be able to work out the correct
length of the message. We have to guess whether the bit should be a 0 or
a 1.
In such circumstances we assume the message length is long(112 bits)
However, if we start to get encoding errors after bit 56 the we attempt
to change our original guess at the bit, and invert it. If this change
of guess would have resulted in a short message, and if the short
message would have been error free then we can recover.
--mlat option introduced: to display raw data in Beast ascii format with
counter (@...;), does not affect Beast binary format ;
--interactive-rtl1090 option introduced: order of flight table Iin
interactive mode) and some formats adopted to RTL1090 format, so
comparison is easier
Change (and hopefully improve) the Message bit decoder.
When decoding bits, a "0" is encoded as (m[j]<m[j+1]), and a "1" is
(m[j]>m[j+1]) . However, there is a problem if mpj[ == m[j+1] because we
can't decide what it's supposed to be.
Antirez's original code defaults to a '1', and then lets the bit error
detection code sort it out. I *think* it's better to default to '0'
because it's more likely that noise added to the signal will produce a
spurious '1' rathar than anything subtracting from the signal to produce
a spurious '0'
Also, Antirez''s code only looks for errors in the first bit of the
message. I don't know why this is.
There is a potential problem in deciding the message length if there are
any errors in the first 5 bits decoded, because this defines the message
type (the DF-??), and it's the DF that determines how many bits the
message shall contain.
If there is an error in the first 5 bits, then we could ignore the DF,
and continue decoding a long format (112 bits). However, for weak
signals, if the message is a short one (56 bits) this results in the
sigStrength decaying to the point where it's level drops below squelch,
so we discard a possibly decodeable 56bit
However, if we assume it's a short message, and only decode 56 bits, and
it's really a long message we won't have decoded all the bits.
Not sure what to do about this
Three changes in this one
1) Change the checksum testing for DF-11
2) Recode the Checksum generation routine to use pointers.
3) Tidy up the appearance of some print debug statments
Change the I/Q lookup table for better detection. Changes fully
described in the source dump1090.c at line 347 onwards. This change
results in about 30% more frames being detected at weak signal input
levels.
Also a bug fix from the last commit - C doesn't support the min()
function.
Original code loops through the analogue array m[] detecting data bits
and putting them into the bits[] array. It then loops through all the
bits[] creating the msg[] byte array. It then loops through the analogue
array m[] again calculating the signal strength.
Change this so that everything is done in one loop so we can go straight
from analogue samples to bytes, calculating the signal strength on the
fly.
Also use the results of the signal strength calculation to populate the
message records mm.signalLevel variable.
Create a pointer, pPayload, which points to the start of the data bits
in the analogue sample buffer m[]. So pPayload =
&m[j+MODES_PREAMBLE_SAMPLES] Then use this
pointer to perform the data bit detection tests. It should save a few
cpu cycles per test because accessing pPayload[2] should be quicker than
m[2+j+MODES_PREAMBLE_SAMPLES].
Also change the way phase correction works. the original code saves the
original data (from m[pPayload] to aux[], and then phase corrects m[],
and then restores aux[] back to m[] afterwards. Change this so that m[]
is copied to aux[] and then phase correction is carried out in aux[],
and the pPayload pointer points to aux[]. This then avoids the
requirement to copy aux[] back to m[] afterwards, which saves a fair few
CPU cycles.
Create a pointer, pPreamble, which points to the start of the preamble
in the analogue sample buffer m[]. So pPreamble = &m[p] Then use this
pointer to perform the preamble detection tests. It should save a few
cpu cycles per test because accessing pPointer[2] should be quicker than
m[p+2].
Also move the decision on whether to try OutOfPhase correction to the
end of the first pass, rather than automatically going into phase
correction if the first pass fails. This saves two memcpy's if the
decision in the second pass is to not do phase correction.
Change the following so that M$ compilers and debuggers complain less
1) change all long long data types to uint64_t.
2) Typecast all malloc function returns to the correct pointer types.
3) Explicitly typecast all float to int conversions.
4) Remove inline variable declaration. Allowed in C++, but not in C.
Apparently, the Beast output timestamp has a frequency of 12 Mhz.
Therefore, I've updated the timestamp counter to simulate a 12 Mhz
frequency.
Also incorporate terribls latest updates
Increase the speed of the I/Q to magnitude calculation lookup by
expanding the table to 65536 entries (256*256*2 bytes). At runtime, this
allows us to pick up raw I/Q bytes as a 16 bit value and index into the
magnitude table to get a 16 bit result. This removes the need for
subtracting 127, and then correcting for -ve numbers, so should be
faster, at the expense of a larger data table.
Change the maglut lookup table from 129*129 to 256*256
Initialise the maglut buffer accordingly
Change the data->maglut lookup to use the new maglut buffer
Change the I/Q data buffer pointet to a uint16_t *
I messed up merging the Squawk display in interactive mode into my
master.
However, the original source posted by terribl causes a print line
length greater than 80 characters. This in turn causes the lines to
spill over on a terminal display. I have therefore re-formatted some of
the output so that it fits within 80 characters.
To try to decode messages with a fundamentally flawed preamble is mostly
a useless waste of CPU time.
The new aggressive mode still detects a sensible percentage of
additional messages because of the error tolerance and two-bits fixes
but does not waste your CPU time.
The phase correction was applied only to a subset of bits! Because of an
offset error.
The detection code layout was modified a bit to make it simpler to
implement more corrections in the future. However only phase correction
is performed currently. Slope correction, or to compensate for the cycloid
effect are two possible improvements.
* Better preamble detection to skip most of the messages we'll likely
not be able to decode.
* A Phase correction algorithm that improves the recognition compared
to the previous algorithm used.
* Javascript output in debug mode, and a debug.html file that can be
used in order to see graphically undecoded samples.
* Ability to detect cross-read messages, that are, messages that happen
to start and end across two different reads from the device or file.
* A few bugx fixed.
* README improved.
This commit address two issues with the implementation of CPR decoding:
Fix#1: The two functions N() and DLon() used to have the latitude
parameter as an integer (!), basically truncating the fractional part
before calling the NL() function to perform the lookup.
This resulted into random strange movements of aircrafts, especially
jumps or shifted positions.
Fix#2: Use milliseconds for CPR odd/even timestamps. Dump1090 already
tried to use the most recent packet received among the odd and even
packets currently available, however to do this correctly millisecond
resolution should be used, otherwise many times the odd and even packet
appear to have the same time in seconds and we don't always use the
latest received.
The old approach was to loop over a small circular buffer of recently
seen addresses recomputing the CRC every time: this prevented the use of
a big cache, and 20 entries was too small in case of high traffic and
big range antennas.
@prog on ##rtlsdr suggested to use an alternative algorithm where
instead we compute the CRC of the message, and xor it to obtain the
address, that is later checked in our list of recently seen addresses.
This is a lot better and allows for bigger tables in O(1) lookup time.
I used this idea to implement a larger 1024 elements table. Instead of
writing a proper hash table I used the fact we are just dealing with a
cache, so I just hash the ICAO addess and overwrite the old entry at
that idex with the address-timestamp pair, not caring about collisions.
The bigger table makes a huge difference:
In a test vector of 113 seconds recording with 76 simultaneous aircrafts
the new algorithm detected around 10k more messages (!).
As @keenerd noted on ##rtlsdr, using an 8 bit magnitude vector is not
enough in order to distinguish every different I/Q pair.
With this commit a few more messages with good CRC are detected.
The demodulation algorithm now only skips the current message if the
CRC is valid, this improves the amount of messages detected with good
CRC by ~ 4% since sometimes we are just a few samples out of sync.
This also improves the amount of messages that can be fixed.
The --stats option was added in order to better evaluate how the
algorithm performs on the same input after some change. So if you run
the program with both --ifile and --stats no Mode S message is logged
at all, but at the end of the processing the program shows you the
amount of messages decoded with good, bad crc, bit corrections, valid
preambles and so forth.
