This directory contains various files and scripts associated with analysing 
historical data from the Reverse Beacon Network ("RBN").

These scripts should work on any modern operating system with the necessary 
packages installed. In particular, the scripts require one or more of the 
following be present:

1. bash, or a compatible shell (I use bash 5.0);
2. python (I use python 3.7.3)
3. R (I use 3.5.2)

Note that if you maintain a copy of the entire RBN dataset locally using 
the download-rbn and convert-rbn scripts you need sufficient space for two 
copies of the file during the process of updating the dataset. Also, 
obviously, the files need to reside on a filesystem that supports files 
of sufficient size. As of this writing, the RBN dataset is around 50GB. 
Files of this size should not present a problem for any modern filesystem, 
but it does mean that you must be sure not to use an old filesystem with 
limits that do not allow for files of this size. 

If stored on a filesystem that supports compression, the actual amount of 
space used on the disk by the RBN dataset is much smaller because of the 
inherent redundancy in the dataset. On my desktop system the compression 
ratio is 3.95, meaning that the entire dataset requires only about 12GB of 
actual disk space; reading the entire dataset using, for example, 
extract-rbn.py, takes about 150 seconds.

If you need the RBN data for only a particular date range, rather than 
downloading the entire dataset with the download-rbn and convert-rbn 
scripts, you might find it more convenient to download one of the files at:
  https://www.adrive.com/public/cQwkEB/rbn
That directory contains compressed year-by-year files of the RBN data for 
the years from 2009 to 2016. These files are in exactly the same format as 
that produced by the convert-rbn script. The directory also contains a 
compressed summary file covering the same period; the format of this file 
is described in the blog post: 
  http://drevans.blog.enginehousebooks.com/2017/01/
    summary-file-for-rbn-data-2009-to-2016.html  

----

compare-multiple-sigs
  Compare the signals from several stations against a single station, over 
  a given time period, on a particular band and to a particular continent. 
  Provides a 99% confidence interval for the difference between the means 
  of each underlying signal distribution as compared to the signal 
  distribution of the single target station.
  
compare-multiple-sigs.R
  Takes output generated by call to compare-sigs.py within the bash script 
  compare-multiple-sigs, and generates and displays the final plot.

compare-sigs:
  Compare the signals between two stations over a given time period, on a 
  particular band and to a particular continent. Provides an estimate of 
  the difference between the means of the underlying signal distributions, 
  the 95% and 99% confidence limits of that estimate, and the probability 
  that one signal really is stronger than the other. 
  
compare-sigs.py: 
  Used by compare-sigs. Scans the local RBN database and extracts the 
  details of comparison reports, then outputs a list of the SNR differences 
  for use by compare-sigs.R.

compare-sigs.R
  Takes output from compare-sigs.py and plots it as a simple histogram. 
  Also includes a calculation of the best estimate of the true difference 
  in signal between the two compared stations given the input sample, and 
  the 95% and 99% confidence limits of the estimate, if there are sufficient 
  data.

convert-rbn:
  Convert RBN data from the as-is format in the RBN zip files to the 
  single-file CSV format used by these scripts.
  
diurnal-posts-from-summary.R
  Generates the plot presented as an example of the use of the RBN summary
  file in the blog post: 
    http://drevans.blog.enginehousebooks.com/2017/01/
      summary-file-for-rbn-data-2009-to-2016.html. 

download-rbn:
  Download daily historical files from the RBN as-is, putting the data into 
  files with friendly names.

extract-rbn.py
  Extract data for a range of dates from a local RBN database, sending the 
  result to standard output.

get-rbn:
  Bring local database (rbndata.csv) up to date.
  
plot-sig.R:
  Show reported signal strength over a given time period for a single 
  station on all continents, for a particular band.

plot-sig-5.R:
  Show reported signal strength over a given time period for a single 
  station on all continents, for 80/40/20/15/10.

plot-sig-9:
  Create and display plot that shows reported signal strength over a given 
  time period for a single station on all continents, for 
  160/80/40/30/20/17/15/12/10
  
plot-sig-9.R:
  Used by plot-sig-9. Generate plot of reported signal strength over a 
  given time period for a single station on all continents, for 
  160/80/40/30/20/17/15/12/10.

plot-sig-zones-9:
  Create and display plot that shows reported signal strength over a given 
  time period for a single station on the six CQ zones with the largest 
  number of RBN posts of that station, for 160/80/40/30/20/17/15/12/10.  

plot-sig-zones-5.R:
  Show reported signal strength over a given time period for a single 
  station on the six CQ zones with the largest number of RBN posts of that
  station, for 80/40/20/15/10.

plot-sig-zones-9.R:
  Used by plot-sig-zones-9. Generate plot of reported signal strength over 
  a given time period for a single station on the six CQ zones with the 
  largest number of RBN posts of that station, for 
  160/80/40/30/20/17/15/12/10.  
  
posting-statistics.py:
  Generates a summary file of annual, mensal and diurnal data that is useful
  for various kinds of summary analyses.
  
posting-statistics-with-scatter-and-grid-metric.py:
  Generates a summary file of annual, mensal and diurnal data that is useful
  for various kinds of summary analyses. This is essentially the same as 
  posting-statistics.py, but with some additional columns in the output.
  
rank-sig:
  Compare the signal of a station to the signals of others on the same 
  continent over a given time period, on a particular band and to a 
  particular continent.

rank-sig.py:
  Used by rank-sig. Scans the local RBN database and provides as output a 
  file that contains the following space-separated information over a given 
  time period in each line:    
    Epoch  target-stn-SNR  all-other-SNRs
  all-other-SNRs are the SNRs for all stations on the same continent as the 
  target, as heard near to the epoch time by stations that have heard the 
  target station.

rank-sig.R:
  Takes output from rank-sig.py and plots it, showing both absolute SNR and 
  percentile of SNR compared to other stations on the same continent.

----

In addition, the directory "cw-activity" contains a series of scripts that generate
a plot of the annual CW metric as defined in the blog post
  http://drevans.blog.enginehousebooks.com/2020/03/
    rbn-based-proxy-metric-for-relative-cw.html

These scripts are:

----

doit.sh:
  The driver script that executes the other scripts in the correct order.

job1.py:
  Reads the RBN data and performs initial analysis

job2.R:
  Confirmatory plots to ensure that data look reasonable

job3.R:
  Plot the CW activity metric 

On my desktop system, simply executing the doit.sh script is sufficient to, eventually, create
the plot of CW activity from 2009 to 2020.

----

All the scripts in this project are by N7DR, except that:
  (1) download-rbn is based on a script by HA5FTL;
  (2) convert-rbn is based on a script by HA5FTL.
  
If you find these scripts useful, please take a moment to look at my web site, 
and consider reading one of my books:
  http://enginehousebooks.com/drevans/

If you have patches to improve these scripts, please let me know.

In addition to these scripts, there is a brief note in the file "What the 
RBN reports" that describes what the RBN documentation means when the term 
"SNR" appears.

Also, the directory "examples" contains some example output plots:

examples/FT5ZM-20140125-20140211-by-zone-9.png
  Output from plot-sig-zones-9 command

examples/K1N-20150201-20150215-by-continent-9.png
  Output from plot-sig-9 command

examples/N7DR-10m-EU-20140215-20140216.png
  Output from the compare-multiple-sigs command

examples/N7DR-K0EU-10m-EU-20141129-20141130.png
  Output from the compare-sigs command

examples//N7DR-10m-EU-20141129-20141129.png
  Output from rank-sig command
  
  73 -- Doc  N7DR  ([email protected])