Merge pull request #21 from nutechsoftware/dev

Release version 1.0.7 with new HTML5 web user interface and new upgrade tool that allows for build features.

.astylerc

@@ -1 +1,4 @@
-style=otbs
+style=otbs
+recursive
+formatted
+exclude=st-device-sdk-c

.gitignore

@@ -0,0 +1,4 @@
+.pio
+build
+.vscode
+sdkconfig

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "st-device-sdk-c"]
+	path = st-device-sdk-c
+	url = https://github.com/SmartThingsCommunity/st-device-sdk-c.git
+	branch = release/v1.4

.travis.yml

@@ -24,10 +24,11 @@ before_script:
 script:
   - cd $TRAVIS_BUILD_DIR
   - export ARTISTIC_STYLE_OPTIONS=".astylerc"
-  - astyle -Q `find . -name *.cpp -o -name *.h` > astyle.out
+  - astyle *.cpp,*.h > astyle.out
   - TEST=$(cat astyle.out | wc -l)
+  - ASTYLE_HEADER_LINES=4
   - |
-      if [[ $TEST -ne 0 ]]; then
+      if [[ $(($TEST - $ASTYLE_HEADER_LINES)) -ne 0 ]]; then
         cat astyle.out
         git --no-pager diff
         echo "Please fix style issues as shown above"

CHANGELOG.md

@@ -4,7 +4,10 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 ## [Unreleased] Open issues
-- [ ] CORE: Certificate store or some way to avoid having to store fixed static public keys for twilio, pushover and other REST api plugins.
+
+### SM - Sean Mathews coder at f34r.com
+- [ ] CORE: Audit Espressif v3.2 api usage look for more that are soon to be deprecated.
+- [ ] STSDK: TODO. FIX Ability to build stsdk component inside of pio build environment. Currently only possible to build with STSDK build.py script.
 - [ ] CORE: FIXME: Setting HOST NAME when using static IP over ethernet not working.
 - [ ] CORE: FIXME: reboot of esp32 causes connected ser2sock clients to hang. So far various attempts to fix have been unsuccessful. Will need to do some network captures to determine the problem.
 - [ ] CORE: HUP/RESTART needs to be centralized so cleanup ex(_fifo_destroy) can happen first. How to connect with STSDK having its own restart calls.
@@ -20,6 +23,47 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - [ ] CORE: TODO: better hardware abstraction. Need to remove _esp_ specific code to make it easier to port to other hardware. Trying to keep the code as POSIX as possible with the limited resources I have.
 - [ ] CORE: TODO: ```'ping'``` command could come in handy. Again today needed this with ST MQTT servers seeming to be down.
 - [ ] STSDK: TODO: Add SmartThings Zone devices.
+
+### AJ
+- [ ] Add a GitHub Action to run a `pio` build on every PR
+- [ ] Migrate `astyle` to GitHub Action
+- [ ] Update README.md to reflect `pio` build changes
+
+## [1.0.7] - 2021-06-26 Sean Mathews - coder @f34rdotcom
+ - [X] AJ - CORE: Add [PlatformIO](https://platformio.org/) (SM: AMAZING how much this has helped! Thanks!)
+ - [X] SM - CORE: Improve network state management. Remove g_ad2_network_state. Tested by disconnecting network etc. Still need to improve storing of events for Twilio, Pushover when network is down for just just dropping new events if network is down. Put webUI inside of a thread to monitor for disconnects and restart if needed.
+ - [X] SM - WEBUI: Connect panic buttons with 3 second resetting timers.
+ - [X] SM - WEBUI: Fix default index.html page. Continued improvements.
+ - [X] SM - CORE: Remove some deprecated flags from sdkconfig.defaults.
+ - [X] SM - CORE: change command vpaddr to vpart as it is less confusing. Updated README commands for upgrade, fixed some spelling, rename vpaddr command to vpart.
+ - [X] SM - WEBUI: Wired up the refresh button. Change arguments for webUI to be vpartID and codeID to match api calls such as ad2_disarm.
+ - [X] SM - WEBUI: Found and fixed some issues found during testing multi partition and fire alarm conditions.
+ - [X] SM - STSDK: Testing with latest release of st-device-sdk-c-ref v1.7 on July 26, 2021. It has been 7 months since the last release. No issues noted so far. Still using and older ESP-IDF v4.0.1-319-g79493f083 with no websocket support. With platformio build we are using v4.3.
+ - [X] SM - OTA: Add ability to fetch firmware by build flags. The default is 'stsdk' but now I can publish a 'webui' build with ST disabled and websockets enabled.
+ - [X] SM - WEBUI: Continued work. Disable WebSocket code if not available such as with STSDK building. UI now supports multiple partitions by specifying 'vpaddr' and 'code' on the query string and the UI will use the AD2IoT configuration settings for the 'vpaddr' and 'code'. Connected buttons to ARM AWAY/STAY, DISARM, EXIT.
+ - [X] SM - API: Fixed a few bugs in the parser from I think when I switched from Arduino Strings to std::strings where the use of substr is different.
+ - [X] SM - WEBUI: New module ported from ArduinoAlarmDecoder with websocket, templates and IPv4 ACL control for access. 24hr 86,400 web request load test PASS! Wiring up html and js code. Status works and screen updates in real time to show alarm state. Buttons roughed in but not sending anything yet.
+ - [X] SM - CORE: Did some work on the GPIO/HAL. This still needs work and hardware testing with ethernet, uSD and AD2pHAT.
+ - [X] SM - CORE: Fix cert bundle build issue if MBEDTLS_CUSTOM_CERTIFICATE_BUNDLE_PATH is enabled.
+ - [X] SM - CORE: valid public certs are no longer needed but keep these old certs around for building in older ESP-IDF versions by moving to component folder.
+ - [X] SM - CORE: Cleanup menuconfig removing MQTT references that need to be in component yet to be made. Moved ser2sock settings to ser2sock component.
+ - [X] SM - OTA: Cleanup a little and get it working with ESP-IDF v4.3 and older v3.2.
+ - [X] SM - PIO: Update to latest release v3.3.0 of platform-espressif32 now with ESP-IDF v4.3!
+ - [X] SM - API: Improve tracking quirk on Ademco with system messages.
+ - [X] SM - CORE: rename partition.2MB.csv to partition.4MB.csv was already a 4MB partition just wrong filename.
+ - [X] SM - CORE: add "build" to .gitignore for stsdk build process.
+ - [X] SM - TWILIO: Increase thread stack by 1k.
+ - [X] SM - CORE: Centralize certificates. Enable MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_CMN to use a common certificate bundle and allow to add custom certificates in the new 'certs' folder. When building with older Espressif revert back to older individual certificate management api calls.
+ - [X] SM - OTA: Set 10 second timeout on update socket. Change first version test from 30s after boot to 60. Fix a memory leak after failed firmware version connection.
+ - [X] SM - CORE: Tune tasks for UARTS.
+ - [X] SM - CORE: Change to require break sequence '...' to interrupt startup to prevent spurious data down the uart from interrupting normal boot operation.
+ - [X] SM - CORE: Improve break detection from ad2term and fix some some uart driver issues I was seeing between Espressif v4.2 and v3.2.
+ - [X] SM - CORE: README.MD add placeholder for building with platformio and seperate STSDK build.
+ - [X] SM - CORE: Fix platformio build upload problem. The address of 0x10000 is hard coded and it needs to be 0x20000 to match the standard OTA partition schema. Refactor platformio.ini to allow building for different ESP32 boards.
+ - [X] SM - CORE: Improve building newer PIO Espressif32 releases that support newer stable Espressif v4.2 or older stable v3.2. Refactor code to better keep the versions of Espressif defined inside of compiler switches for ESP_IDF_VERSION_VAL.
+ - [X] SM - CORE: Running menuconfig inside of VSCODE using pio does not work. Flashes the menu screen then goes POOF!. For now test in a shell. ( PEBCAC screen size. )
+ - [X] SM - CORE: Certificate store or some way to avoid having to store fixed static public keys for twilio, pushover and other REST api plugins.
+
 ## [1.0.6 P6] - 2021-04-05 Sean Mathews @f34rdotcom
 - [X] CORE: Move log mode init.
 - [X] API: Event format string for no match case to show event ID.

README.md

@@ -5,21 +5,31 @@
 
 ## Overview
 
-This project provides a framework for building an IoT network appliance to monitor and control one or many alarm systems.
+This project provides an example framework for building an IoT network appliance to monitor and control one or many alarm systems.
 
-By integrating the AD2pHAT from AlarmDecoder.com and a compatible ESP32 SoC based board the resulting device connects a compatible alarm system to a public or private MQTT server for monitoring. The device can also be configured to initiate SIP voice calls, SMS messages, or e-mail when a user defined alarm state is triggered. The typical time from the device firmware start to being delivery of a state event is less than 10 seconds. Typical latency between an alarm event and message delivery is 20ms on a local network.
+The [AD2IoT network appliance](https://www.alarmdecoder.com/catalog/product_info.php/products_id/50) integrates the AD2pHAT from AlarmDecoder.com and a ESP32-POE-ISO. The result is a device that easily connects to a compatible alarm system and publishes the alarm state to a public or private MQTT server for monitoring. The device can also be configured to initiate SIP voice calls, SMS messages, or e-mail when a user defined alarm state is triggered. The typical time from the device firmware start to being delivery of a state event is less than 10 seconds. Typical latency between an alarm event and message delivery is 20ms on a local network.
 
 The device firmware is stored in the onboard non-volatile flash making the device resistant to corruption. With OTA update support the flash can be securely loaded with the latest version in just a few seconds over HTTPS.
 
 ## Tested and recommended hardware
-* ESP32-DevKitC-32E. WiFi only applications.
 * ESP-POE-ISO. Ethernet+WiFi applications.
+* ESP32-DevKitC-32E. WiFi only applications.
 
 ## Firmware
-TODO: HOWTO flash the latest available firmware or compile this project from source. [Building firmware](#Building-firmware)
+The firmware is already compiled for the ESP32-POE-ISO board and is available in the release page or via OTA(over-the-air) update. Currently the firmware is built with the following build flags 'stsdk' and 'webui'.
+
+To switch to a specific build over the internet using OTA include the buildflag in the upgrade command.
+
+    upgrade webui
+
+
+See the README-FLASH.MD inside the release file for instructions on flash the firmware over the ESP32-POE-ISO USB.
+
+### SmartThings build (stsdk) - alarmdecoder_stsdk_esp32.bin
+- Components: Pushover, Twilio, Sendgrid, ser2sock, SmartThings
 
+This is the default buildflag. This build is compiled using the [st-device-sdk-c-ref](https://github.com/SmartThingsCommunity/st-device-sdk-c-ref) from the SmartThings github repo and has the webUI component disabled.
 
-## SmartThings app integration
 A few options are available as the AD2IoT device moves toward being certified and directly available in the SmartThings app. In order to Discover and adopt the AD2IoT device it needs to be visible in the "My Testing Devices" under the "Adding device" panel.
 
 First you will need to [enable Developer Mode in the app
@@ -32,6 +42,14 @@ https://smartthings.developer.samsung.com/partner/enroll
 
 - Use the SmartThings/Samsung developer workspace to create custom profiles and onboarding as described in this howto guide [How to build Direct Connected devices with the SmartThings Platform](https://community.smartthings.com/t/how-to-build-direct-connected-devices/204055)
 
+### webUI build (webui) - alarmdecoder_webui_esp32.bin
+- Components: Pushover, Twilio, Sendgrid, ser2sock, webUI
+<img style="border:5px double black;"  src="contrib/webUI/EXAMPLE-PANEL-READY.jpg" width="200">
+
+
+This build uses the latest ESP-IDF v4.3 that supports WebSockets. The SmartThings driver is disabled and the webui component is enabled.
+
+Copy the contents of flash-drive folder into the root directory of a uSD flash drive formatted with a fat32 filesystem on a single MSDOS partition. Place this uSD flash drive into the ESP32-POE-ISO board and reboot.
 
 ## Configure the AD2IoT device.
  - Connect the ESP32 development board USB to a host computer and run a terminal program to connect to the virtual com port using 115200 baud.
@@ -49,8 +67,12 @@ Choose one of the following configurations.
 
 Configure the notifications using the notification components CLI commands.
 
-## AD2Iot CLI command reference
-### Base commands
+## AD2Iot CLI - command line interface
+Currently all configuration is done over the the ESP32 usb serial port. For drivers see the [ESP32-POE-ISO product page](https://www.olimex.com/Products/IoT/ESP32/ESP32-POE-ISO/open-source-hardware). The USB port also provides power so disconnect the device from the alarm using the quick connect terminal block and and connect the device to a computer using a USB 2.0 A-Male to Micro B Cable for configuration using the command line interface.
+
+　<font color='red'>Do not connect the [AD2IoT network appliance](https://www.alarmdecoder.com/catalog/product_info.php/products_id/50) to alarm panel power when connected to a computer.</font> If connection from both PC and alarm at the same time is needed be sure to connect to the alarm panel power first then connect to the PC last. The power from the panel when first connected to the AD2IoT will provide an unstable 5v output for a few microseconds. This unstable voltage can be sent back through the USB to the host computer causing the host to detect the voltage fault and halt.
+
+### Standard commands
   - Show the list of commands or give more detail on a specific command.
 
     ```help [command]```
@@ -74,7 +96,11 @@ Configure the notifications using the notification components CLI commands.
 
   - Preform an OTA upgrade now download and install new flash.
 
-    ```upgrade```
+    ```upgrade [buildflag]```
+
+    - [buildflag]: Specify build for the release. default to 'stsdk' if omitted.
+
+      See release page for details on available builds.
 
   - Report the current and available version.
 
@@ -89,7 +115,7 @@ Configure the notifications using the notification components CLI commands.
       - [W]iFi: Enable WiFi network driver.
       - [E]thernet: Enable ethernet network driver.
     - [arg]
-      - Argument string name value pairs sepearted by &.
+      - Argument string name value pairs separated by &.
         - Keys: MODE,IP,MASK,GW,DNS1,DNS2,SID,PASSWORD
 
     Examples
@@ -142,7 +168,7 @@ Configure the notifications using the notification components CLI commands.
 
   - Manage virtual partitions.
 
-    ```vpaddr {id} {value}```
+    ```vpart {id} {value}```
 
     - {id}
       - The virtual partition ID. 0 is the default.
@@ -151,13 +177,13 @@ Configure the notifications using the notification components CLI commands.
 
     Examples
       - Set default address mask to 18 for an Ademco system.
-        - vpaddr 0 18
+        - vpart 0 18
       - Set default send partition to 1 for a DSC system.
-        - vpaddr 0 1
+        - vpart 0 1
       - Show address for partition 2.
-        - vpaddr 2
+        - vpart 2
       - Remove virtual partition in slot 2.
-        - vpaddr 2 -1
+        - vpart 2 -1
 
       Note: address -1 will remove an entry.
 
@@ -190,7 +216,7 @@ Configure the notifications using the notification components CLI commands.
           - Example: ser2sockd enable Y
       - [acl] Set / Get ACL list
         - {arg1}: ACL LIST
-          -  String of CIDR values seperated by commas.
+          -  String of CIDR values separated by commas.
           - Default: Empty string disables ACL list
           - Example: ser2sockd acl 192.168.0.123/32,192.168.1.0/24
 
@@ -262,7 +288,7 @@ Configure the notifications using the notification components CLI commands.
       - [R] AUTO Reset.
         - {arg1}:  time in ms 0 to disable
       - [T] Message type filter.
-        - {arg1}: Message type list seperated by ',' or empty to disables filter.
+        - {arg1}: Message type list separated by ',' or empty to disables filter.
           - Message Types: [ALPHA,LRR,REL,EXP,RFX,AUI,KPM,KPE,CRC,VER,ERR,EVENT]
             - For EVENT type the message will be generated by the API and not the AD2
       - [P] Pre filter REGEX or empty to disable.
@@ -337,7 +363,7 @@ Configure the notifications using the notification components CLI commands.
       - [R] AUTO Reset.
         - {arg1}:  time in ms 0 to disable
       - [T] Message type filter.
-        - {arg1}: Message type list seperated by ',' or empty to disables filter.
+        - {arg1}: Message type list separated by ',' or empty to disables filter.
           - Message Types: [ALPHA,LRR,REL,EXP,RFX,AUI,KPM,KPE,CRC,VER,ERR,EVENT]
             - For EVENT type the message will be generated by the API and not the AD2
       - [P] Pre filter REGEX or empty to disable.
@@ -473,7 +499,10 @@ Configure the notifications using the notification components CLI commands.
         ```
 
 ## Building firmware
-### Setup build environment
+### PlatformIO
+#### TODO: Setup notes.
+### SmartThings device SDK build environment
+#### Setup build environment
 - Follow the instructions in the [SmartThings SDK for Direct connected devices for C](https://github.com/SmartThingsCommunity/st-device-sdk-c-ref) project for setting up a build environment. Confirm you can build the switch_example before continuing.
 - Select the esp32 build environment. Branch v1.4 seems to be the current active branch and uses espidf v4.0.1-317-g50b3e2c81.
 ```
@@ -485,13 +514,13 @@ cd ~/esp
 
 - Place the contents of this his project in ```st-device-sdk-c-ref/apps/esp32/```
 
-### Configure the project
+#### Configure the project
 
 ```
 ./build.sh esp32 AlarmDecoder-STSDK menuconfig
 ```
 
-### Build, Flash, and Run
+#### Build, Flash, and Run
 
 Build the project and flash it to the board, then run monitor tool to view serial output:
 

certs/README.MD

@@ -0,0 +1,13 @@
+mbedTLS certificates
+***
+Newer Espressif now has a certificate bundle api with common certificates that can be automatically linked into the firmware and loaded by the mbedTLS api. https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/esp_crt_bundle.html
+
+  FIXME: 2021-07-19: Must have at least one certificate in this folder if MBEDTLS_CUSTOM_CERTIFICATE_BUNDLE_PATH is
+  enabled. For now keeping this dummy cert here.
+
+Place user specific certificates here to be included in the certificate bundle by adding the PEM file here.
+
+   Example using openssl command on Linux.
+```console
+openssl s_client -showcerts -connect www.howsmyssl.com:443 </dev/null
+```