Merge pull request #164 from semuconsulting/RC-1.4.27

RC 1.4.27

.vscode/settings.json

@@ -4,6 +4,6 @@
     "editor.formatOnSave": true,
     "modulename": "pygpsclient",
     "distname": "pygpsclient",
-    "moduleversion": "1.4.26",
+    "moduleversion": "1.4.27",
     "python.defaultInterpreterPath": "python3",
 }

README.md

@@ -87,7 +87,7 @@ Normally installs without any additional steps.
 sudo apt install python3-pip python3-tk python3-pil python3-pil.imagetk libjpeg-dev zlib1g-dev
 ```
 
-*⁵* If you're compiling the latest version of Python 3 from source, you may also need to install tk-dev (or a similarly named package e.g. tk-devel) first. Refer to http://wiki.python.org/moin/TkInter for further details:
+*⁵* If you're [compiling the latest version of Python 3 from source](https://github.com/semuconsulting/PyGPSClient/blob/master/examples/python_compile.sh), you may also need to install tk-dev (or a similarly named package e.g. tk-devel) first. Refer to http://wiki.python.org/moin/TkInter for further details:
 
 ```shell
 sudo apt install tk-dev
@@ -244,7 +244,7 @@ You will need to logout and login for the launcher to take effect.
 | User-selectable 'widgets' | To show or hide the various widgets, go to Menu..View and click on the relevant hide/show option. |
 |---------------------------|---------------------------------------------------------------------------------------------------|
 |![banner widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/banner_widget.png?raw=true)| Expandable banner showing key navigation status information based on messages received from receiver. To expand or collapse the banner or serial port configuration widgets, click the ![expand icon](https://github.com/semuconsulting/PyGPSClient/blob/master/src/pygpsclient/resources/iconmonstr-arrow-80-16.png?raw=true)/![expand icon](https://github.com/semuconsulting/PyGPSClient/blob/master/src/pygpsclient/resources/iconmonstr-triangle-1-16.png?raw=true) buttons. **NB**: some fields (e.g. hdop/vdop, hacc/vacc) are only available from proprietary NMEA or UBX messages and may not be output by default. The minimum messages required to populate all available fields are: NMEA: GGA, GSA, GSV, RMC, UBX00 (proprietary); UBX: NAV-DOP, NAV-PVT, NAV_SAT |
-|![console widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/console_widget.png?raw=true)| Configurable serial console widget showing incoming GNSS, NTRIP and SPARTN data streams in either parsed, binary or tabular hexadecimal formats. Supports user-configurable color tagging of selected strings for easy identification. Color tags are loaded from the `"colortag_b":` value (`0` = disable, `1` = enable) and `"colortags_l":` list (`[string, color]` pairs) in your json configuration file (see example provided). NB: color tagging does impose a small performance overhead - turning it off will improve console response times at very high transaction rates.|
+|![console widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/console_widget.png?raw=true)| Configurable serial console widget showing incoming GNSS, NTRIP and SPARTN data streams in either parsed, binary or tabular hexadecimal formats. Double-click to copy contents of console to clipboard. Supports user-configurable color tagging of selected strings for easy identification. Color tags are loaded from the `"colortag_b":` value (`0` = disable, `1` = enable) and `"colortags_l":` list (`[string, color]` pairs) in your json configuration file (see example provided). NB: color tagging does impose a small performance overhead - turning it off will improve console response times at very high transaction rates.|
 |![skyview widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/skyview_widget.png?raw=true)| Skyview widget showing current satellite visibility and position (elevation / azimuth). Satellite icon borders are colour-coded to distinguish between different GNSS constellations. For consistency between NMEA and UBX data sources, will display GLONASS NMEA SVID (65-96) rather than slot (1-24). |
 |![graphview widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/graphview_widget.png?raw=true)| Graphview widget showing current satellite reception (carrier-to-noise ratio or cnr). Double-click to toggle legend. |
 |![static map](https://github.com/semuconsulting/PyGPSClient/blob/master/images/staticmap.png?raw=true)| Map widget with various modes of display. Map Type = 'world': a static offline Mercator world map showing current global location.
@@ -253,7 +253,7 @@ You will need to logout and login for the launcher to take effect.
 |![import custom map](https://github.com/semuconsulting/PyGPSClient/blob/master/images/importcustommap.png?raw=true)| <a name="custommap">Import Custom Map dialog</a>. Click ![load icon](https://github.com/semuconsulting/PyGPSClient/blob/master/src/pygpsclient/resources/iconmonstr-folder-18-24.png?raw=true) to open the custom map image location (*the default file suffix is `*.gif` - select Show Options to select any file suffix `*.*`*). If the `rasterio` library is installed and the image is georeferenced (e.g. using [QGIS](https://qgis.org/)), the bounding box will be automatically extracted - otherwise it must be entered manually. Import the custom map path and bounding box settings by clicking ![play icon](https://github.com/semuconsulting/PyGPSClient/blob/master/src/pygpsclient/resources/iconmonstr-arrow-12-24.png?raw=true).|
 |![spectrum widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/spectrum_widget.png?raw=true)| Spectrum widget showing a spectrum analysis chart (*GNSS receiver must be capable of outputting UBX MON-SPAN messages*). Clicking anywhere in the spectrum chart will display the frequency and decibel reading at that point. Double-clicking anywhere in the chart will toggle the GNSS frequency band markers (L1, G2, etc.) on or off. **NB:** Some receivers (e.g. NEO-F10N) will not output the requisite MON-SPAN messages unless the port baud rate is at least 57,600. |
 |![sysmon widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/sysmon_widget.png?raw=true)| System Monitor widget showing device cpu, memory and I/O utilisation (*GNSS receiver must be capable of outputting UBX MON-SYS and/or MON-COMMS messages*). Tick checkbox to toggle between actual (cumulative) I/O stats and pending I/O. |
-|![scatterplot widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/scatterplot_widget.png?raw=true)| Scatterplot widget showing variability in position reporting over time. Select from either Planar or Great Circle (haversine) distance approximation mode. Select from either Dynamic or Fixed to center plot on dynamic average position (*displayed at top left*) or fixed reference position (*if entered*). Use the scale slider or mousewheel to adjust scale. Double-click to clear existing plot. The default mode, center, scale index and fixed reference position can be stored in the json configuration file as `scattermode_s`/`scattercenter_s`/`scatterscale_n`/`scatterlat_f`/`scatterlon_f`. |
+|![scatterplot widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/scatterplot_widget.png?raw=true)| Scatterplot widget showing variability in position reporting over time. (Optional) Enter fixed reference position. Select Average to center plot on dynamic average position (*displayed at top left*), or Fixed to center on fixed reference position (*if entered*). Check Autorange to set plot range automatically. Set the update interval (e.g. 4 = every 4th navigation solution). Use the range slider or mouse wheel to adjust plot range. Right-click to set fixed reference point to the current mouse cursor position. Double-click clear existing plot. The default center mode, autorange, update increment, scale index and fixed reference position settings can be stored in the json configuration file as `scattercenter_s`/`scatterautorange_b`/`scatterinterval_n`/`scatterscale_n`/`scatterlat_f`/`scatterlon_f`. |
 |![rover widget](https://github.com/semuconsulting/PyGPSClient/blob/master/images/rover_widget.png?raw=true) | Rover widget plots the relative 2D position, track and status information for the roving receiver in a fixed or moving base / rover RTK configuration. Can also display relative position of NTRIP mountpoint and receiver in a static RTK configuration. Double-click to clear existing plot. (*GNSS rover receiver must be capable of outputting UBX NAV-RELPOSNED messages.*) |
 ---
 ## <a name="ubxconfig">UBX Configuration Facilities</a>

RELEASE_NOTES.md

@@ -1,5 +1,13 @@
 # PyGPSClient Release Notes
 
+### RELEASE 1.4.27
+
+ENHANCEMENTS:
+
+1. Further enhancements to scatterplot widget; add autorange, update interval, right-click to set fixed ref position, standard deviation in addition to average lat/lon, improved font scaling
+1. Add Height Above Ellipsoid (HAE) to banner. 'sep' = HAE - hMSL.
+1. Add double-click to copy contents of console to clipboard.
+
 ### RELEASE 1.4.26
 
 ENHANCEMENTS:

RTK-TIPS.md

@@ -61,7 +61,32 @@ If you're relatively new to GNSS and RTK techniques and terminology, you may wan
 - Ideally, an NTRIP source ('mountpoint') should be no further than 30km from your antenna location, such that its correction data is [representative of local atmospheric conditions](https://www.semuconsulting.com/gnsswiki/#ION).
 - **NB:** While free public domain NTRIP resources exist and can often give perfectly satisfactory results, there is no guarantee as to the quality of the correction data available. Some public domain CORS are essentially amateurs working out of their back yards.  The best results *for your location* may be obtained from proprietary or subscription services which use specialised, carefully calibrated equipment in optimal locations with ITRF or local plate-fixed datums - but it's always worth trying public domain resources first.
 
-### 7. Check your receiver is successfully receiving and processing RTK data
+### 7. Use the highest precision message type(s) available
+
+- If using a UBX receiver like the ZED-F9P, different NMEA or UBX navigation message types will yield different levels of precision.
+
+  | lat/lon decimal places | approx precision | example message type |
+  |----------------| -------------------------|----------|
+  |      3         |        110  m            |          |
+  |      4         |        11  m             |          |
+  |      5         |        1.1  m            |          |
+  |      6         |        11 cm             | NMEA GGA, GNS in standard precision mode (5dp minutes) |
+  |      7         |        1.1 cm            | UBX NAV-PVT, NAV-POSLLH |
+  |      8         |        1.1 mm            | UBX NAV-HPPOSLLH, NMEA GGA, GNS in high precision mode (7dp minutes) |
+  |                |                          |  |
+
+- 8dp (± 1.1 mm) represents the practical limit for RTK and PPK techniques - further decimal places are likely to be spurious (*though internal computations may retain them e.g. to avoid cumulative rounding errors*).
+- **NB** The highest precision message types/modes are generally **NOT** enabled by default - they will need to be enabled via the [UBX Configuration panel](https://github.com/semuconsulting/PyGPSClient?tab=readme-ov-file#ubxconfig). Always use the highest precision available for your receiver. For the ZED-F9P and other 9th generation u-blox receivers...
+  - NMEA high precision mode can be enabled by setting CFG_NMEA_HIGHPREC = 1
+  - UBX NAV-HPPOSLLH can be enabled by setting CFG_UBX_MSGOUT_HPPOSLLH_xxxxx = 1, where xxxxx represents the port ('USB', 'UART1', 'I2C', etc.)
+  - A broad indication of available precision can be obtained from the scatterplot widget set to a range of 10 cm or less. If the points appear to be aligned to a coarse grid, this suggests relatively low precision. If the points appear to be scattered more or less randomly, this suggests higher precision.
+
+    | **lower precision** | **higher precision** |
+    |:-----------------:|:------------------:|
+    | ![low precision](https://github.com/semuconsulting/PyGPSClient/blob/master/images/low_precision.png?raw=true) | ![high precision](https://github.com/semuconsulting/PyGPSClient/blob/master/images/high_precision.png?raw=true) |
+    | | |
+
+### 8. Check your receiver is successfully receiving and processing RTK data
 
 - If you're using a UBX receiver like the F9P, you may find it easier to configure it to output UBX messages rather than the default NMEA, as the former are more concise and offer more comprehensive and consistent RTK status information.
 - Successful receipt and processing of RTK correction data is generally indicated as follows:
@@ -81,13 +106,13 @@ If you're relatively new to GNSS and RTK techniques and terminology, you may wan
   - If using an internet RTK service (NTRIP or MQTT), ensure you have active internet connectivity and that the service's IP port (typically 2101, 2102, 443 or 8883) is not being blocked by a firewall.
   - Ensure that your F9P is configured to allow incoming RTCM3 data on the designated port - this is the default setting, but it may have been overwritten.
 
-### 8. "FLOAT" vs "FIXED"
+### 9. "FLOAT" vs "FIXED"
 
 - It is important to understand that these RTK status values are based, in essence, on a statistical analysis (using the principles of [Kalman Filtering](https://en.wikipedia.org/wiki/Kalman_filter)) of the [various errors](https://www.semuconsulting.com/gnsswiki/#Errors) ('residuals') which affect the calculation of carrier phase pseudorange, and hence the accuracy of the navigation solution.
 - The receiver's firmware applies a complex proprietary algorithm to determine if this statistical analysis is within a certain tolerance. If it is, the receiver will report an 'RTK-FIXED' status. If it is outside the designated tolerance, the receiver may report an 'RTK-FLOAT' status. This doesn't *necessarily* mean that the solution is less accurate - it just means that the *confidence* level is lower than 'FIXED'.
 - An 'RTK-FLOAT' status will therefore generally yield a lower accuracy estimation (`hAcc`, `vAcc`) than 'RTK-FIXED'.
 
-### <a name="caveats">9. Some caveats on "hAcc" and "vAcc"</a>
+### <a name="caveats">19. Some caveats on "hAcc" and "vAcc"</a>
 
 - **NB:** `hAcc` and `vAcc` values are *not* available from the default cohort of NMEA messages output by most GNSS receivers (including the F9P). In order to receive these values, you will need to enable either proprietary NMEA sentences like u-blox's UBX00, or UBX messages like NAV-PVT.
 - Much is made of reported horizontal (hAcc) and vertical (vAcc) accuracy figures, but they are **merely estimates** based on a statistical analysis of:
@@ -97,7 +122,6 @@ If you're relatively new to GNSS and RTK techniques and terminology, you may wan
   - Pseudorange residuals ([UERE](https://en.wikipedia.org/wiki/Error_analysis_for_the_Global_Positioning_System))
 - In the case of the F9P receiver, the analysis itself is proprietary and is performed within the F9P's firmware, rather than in client software.
 - Ultimately, the only *definitive* way of establishing the accuracy of an RTK (or PPK) navigation solution is to compare it with a known fixed reference point whose 3D coordinates have been independently verified via high precision surveying techniques.
-- Note that the 7th and 8th decimal places of a decimal lat/lon coordinate are equivalent to about 1.1cm and 1.1mm respectively, which represents the practical limit of any RTK or PPK GNSS application - anything more than this would be spurious.
 
 ---
 ## <a name="example">Illustrated Example</a>

pygpsclient.json

@@ -106,8 +106,9 @@
     "lbandclientoutport_s": "Passthrough",
     "lbandclientdebug_b": 0,
     "spartndecode_b": 0,
-    "scattermode_s": "Planar",
-    "scattercenter_s": "Dynamic",
+    "scatterautorange_b": 0,
+    "scattercenter_s": "Average",
+    "scatterinterval_n": 1,
     "scatterscale_n": 6,
     "scatterlat_f": 37.23345133,
     "scatterlon_f": -115.81513477,
@@ -175,16 +176,8 @@
     ],
     "colortags_l": [
         [
-            "NMEA",
-            "green"
-        ],
-        [
-            "UBX",
-            "orange"
-        ],
-        [
-            "RTCM",
-            "lightblue1"
+            "txbuf alloc",
+            "red"
         ],
         [
             "UNKNOWN PROTOCOL",
@@ -199,24 +192,12 @@
             "red"
         ],
         [
-            "NTRIP>>",
-            "#9f30c1"
-        ],
-        [
-            "NTRIP<<",
-            "#9f30c1"
-        ],
-        [
-            "MQTT>>",
-            "#a53570"
-        ],
-        [
-            "L-BAND>>",
-            "#a53570"
+            "carrSoln=2",
+            "green"
         ],
         [
-            "SPARTN>>",
-            "#a53570"
+            "carrSoln=1",
+            "deepskyblue"
         ]
     ]
 }

pyproject.toml

@@ -7,7 +7,7 @@ name = "pygpsclient"
 authors = [{ name = "semuadmin", email = "[email protected]" }]
 maintainers = [{ name = "semuadmin", email = "[email protected]" }]
 description = "GNSS Diagnostic and UBX Configuration GUI Application"
-version = "1.4.26"
+version = "1.4.27"
 license = { file = "LICENSE" }
 keywords = [
     "PyGPSClient",
@@ -52,7 +52,7 @@ classifiers = [
 dependencies = [
     "requests>=2.28.0",
     "Pillow>=9.0.0",
-    "pygnssutils>=1.1.6",
+    "pygnssutils>=1.1.7",
     "pyserial>=3.5",
 ]
 
@@ -90,11 +90,12 @@ profile = "black"
 exclude_dirs = ["docs", "examples", "tests"]
 skips = [
     "B104",
+    "B311",
     "B318",
     "B404",
     "B408",
     "B603",
-] # bind 0.0.0.0; minidom; subroutine
+] # bind 0.0.0.0; randrange; minidom; subroutine
 
 [tool.pylint]
 jobs = 0

src/pygpsclient/_version.py

@@ -8,4 +8,4 @@
 :license: BSD 3-Clause
 """
 
-__version__ = "1.4.26"
+__version__ = "1.4.27"

src/pygpsclient/app.py

@@ -201,6 +201,7 @@ def __init__(self, master, *args, **kwargs):  # pylint: disable=too-many-stateme
         self._socket_server = None
         self._colcount = 0
         self._rowcount = 0
+        self._consoledata = []
 
         # Load configuration from file if it exists
         self._colortags = []
@@ -956,15 +957,17 @@ def process_data(self, raw_data: bytes, parsed_data: object, marker: str = ""):
         elif msgprot == MQTT_PROTOCOL:
             pass
 
-        # update console if visible
-        if widget_state["Console"][VISIBLE]:
-            if msgprot == 0 or msgprot & protfilter:
-                self.frm_console.update_console(raw_data, parsed_data, marker)
+        # update consoledata if console is visible and protocol not filtered
+        if widget_state["Console"][VISIBLE] and (msgprot == 0 or msgprot & protfilter):
+            self._consoledata.append((raw_data, parsed_data, marker))
 
         # periodically update widgets if visible
         if datetime.now() > self._last_gui_update + timedelta(
             seconds=GUI_UPDATE_INTERVAL
         ):
+            if widget_state["Console"][VISIBLE]:
+                self.frm_console.update_console(self._consoledata)
+                self._consoledata = []
             self.frm_banner.update_frame()
             for _, widget in widget_state.items():
                 frm = getattr(self, widget["frm"])