Thanks to Blort on the PP list.
The Beast binary message stream uses the 0x1a character a and escape to
mark the beginning of a new message. However, the 0x1a character could
occur in the body of a message since the message is binary. Therefore,
the 0x1a is repeated -as Blort put it :
This 56-bit Mode S Frame (containing a 1a) is supposed to look like
this:
1a 32 00 00 48 7b a6 1a 1a 0c 20 28 17 b0 c0 c3 b0
What is actually looks (looked) like (from wireshark) is this:
1a 32 00 00 48 7b a6 1a 0c 20 28 17 b0 c0 c3 b0
There are supposed to be two 1A’s (Gunter’s Escape Character) in a row
(1A 1A) whenever a data byte contains 1A, after the initial two 1A 3x
characters.
The Mutex on the RTL data reader thread does not "force" the data
processing thread to execute. Therefore, if the processor is busy, it is
possible for a second RTL callback to occur before the data from the
first has been processed. This will cause the loss of the first data,
but worse, it will cause a slip in the timestamp. This upsets Beamfinder
and MLAT operation in PlanePlotter.
To solve this, keep a Fifo buffer which is filled by the callback
thread, and emptied by the data processing thread. The fifo is the same
size as the number of buffers requested in the call to
rtlsdr_read_async().
Note - we only put the value of the pointer supplied in the callback
into the fifo. We do not attempt to cache the data in the buffer pointed
to by the pointer. This would require us to memcopy() 2Mbytes per
second, which we don't want to do if we don't have to because it will
only make the processor loading worse. Instead, we assume that the data
in the buffer will remain valid after the callback returns, at least
until it is overwritten by new data.
It is still possible for us to lose data if we can't process it quickly
enough. However, we can now detect this loss of data when the fifo is
almost full, and correct the timestamp for the lost block/blocks.
The modesReadFromClient() funtion is called from modesReadFromClients(),
which in turn is called from backgroundTasks(). backgroundTasks() is
called from within the main processing loop.
However, modesReadFromClient() can and does block. It attempts to read
characters from the input stream, and loops whilst there was no error.
This stalls the main RTL processing loop until an error occurs. In order
to support simultaneous local reception (via our RTL dongle) and remote
forwarding (data received from the interweb) we cannot allow this
internet read to stall.
To fix this, in modesReadFromClient() attempt to read a buffer of data
(currently 0x400 bytes). If we get a full buffer of bytes, then process
them, and attempt to read another full buffer. Keep doing thios untill
we read only a partial buffer (less than 0x400 bytes). Process the
partial buffer bytes and return.
This allows us to occasionally process data that is arriving from the
internet (which is buffered anyway in the TCP stack), without blocking
local RTL dongle decoding.
Ok - this is likely to upset some people. Up until now, the vast
majority of the code has been in just one file - dump1090.c. This file
has grown so that it was approaching of 5000 lines long, and it was
becoming unmanagable. So I've split the file into several modules,
hopefully along fairly logical boundaries. The files are :
1) dump1090.c : Basically just the main() entry function, the help
function, the RTL dongle hardware interface, and a few orphan functions
that don't really fit anywhere else.
2) mode_s.c : This contains all the mode S / ADSB decoding functions.
3) mode_ac.c : This contains all the mode A & C decoding functions
4) interactive.c : This contains all the functions to maintain an
internal list of aircraft seen over the last period, and functions to
print them out to the local console.
5) net_io.c : This contains all the network input/output functions
allowing data to be passed in/out to/from other receivers, in formats
such as SBS-1/3, Beast, AVR and JavaScript.
Hopefully this should provide an easier way forward if/when more
functions are added.
