Merge pull request #57 from nutechsoftware/dev

Release v1.1.1
nutechsoftware · Sep 18, 2022 · 51f2e99 · 51f2e99
2 parents 019cfc5 + 69d6609
commit 51f2e99
Show file tree

Hide file tree

Showing 47 changed files with 842 additions and 610 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -33,6 +33,40 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ---
 ## Releases
+## [1.1.1] - 2022-09-18 Sean Mathews - coder @f34rdotcom
+Changes:
+  - Partition change add new coredump partition.
+    - This change will require manual flash OTA will not work. Hopefully this is the last partition change.
+  - Improved top command from 1.1.0 fixed bugs and rollover issues.
+  - Twilio
+    - New sub command to disable or enable notification slots.
+    - Expanding total notification slots from 8 to 999.
+    - Memory optimization.
+  - General
+    - Continued work on docs.
+    - Continued work on memory and CPU optimization.
+### Change log
+- [X] SM API: Crashes during long term testing have happened during regex parsing in the API as a result of low memory. Adding a TRY/CATCH to fail more gracefully but a better fix needs to be looked at in pushing the issue upstream to the application layer via a async call that the device is low on memory or a notification failed due to low memory. The low memory is usually been the result of some leak in one of the modules when everything is turned on all at once so some decisions need to be made on what services to use. Every service that uses SSL has a big effect on overall memory consumption when initialized.
+- [X] SM STSDK: Remove stsdk json files that are not used.
+- [X] SM CORE: Misc cleanup of docs. Remove redundant code in factory_reset. Bump AD2 sendQ stack from 4k to 8k to be safe but it needs more study.
+- [X] SM TWILIO: Add 'disable' verb to twilio notification slots to make it easy to turn on/off. Log the info when an event is sent out to the sendQ. Max sub account change from 8 to 999. Fix missing pre 'filter' verb value not being saved into search object. Add INFO log report for each notification to help debug REGEX strings. Reduce memory remove saved notification values and instead lookup as needed.
+- [X] SM CORE: Create default ad2iot.ini with settings for easy configuration over ftp.
+- [X] SM CORE: Tweak partitions again. Added 64k coredump partition and expanded spiffs by 64k by reducing the two firmware image partition size by 64k. Max program size is 1,835,008 bytes.
+- [X] SM CONTRIB: Add new schematics for AD2IoT main board and carrier with component values.
+- [X] SM CORE: Improve top command and make optional with menuconfig.
+- [X] SM FTPD: Added more sleep time to task it was using too much cpu when idle.
+- [X] SM HAL: Rename esp_uart_* to uart_*, Remove esp_timer replace with hal_uptime_us().
+- [X] SM CORE: Improve top to deal with overflow. TODO IDLE task will overflow so make a spacial case keep a private IDLE run tiem counter.
+- [X] SM SER2SOCK: High CPU fix old bug found with new top command.
+- [X] SM CORE: Refactor top and add docs. The FreeRTOS in stats reports were glitchy so it was replaced with uxTaskGetSystemState().
+- [X] SM STSDK: Small fixes. Still a big mess :(
+- [X] SM *: Nice task names.
+- [X] SM CORE: Fix from prior change const cast.
+- [X] SM CORE: Change platform.ini to be more clear on IDF version and remove old comment that is no longer relevant.
+- [X] SM CORE: Add new command ```top``` to show task list and some stats. Needed something for sanity checking and tuning.
+- [X] SM OTA: Remove private key and use global ca store and real world certs. For now point to www.alarmdecoder.com site and use it's cert for encryption.
+- [X] SM CORE: Remove remaining cdecl that is no longer needed except for the one entrypoint from FreeRTOS.
+- [X] SM CORE: Misc cleanup no code changes. Updated address_info.txt missed on last release.
 ## [1.1.0] - 2022-05-23 Sean Mathews - coder @f34rdotcom
 Big changes:
   - Help file and docs refactor to make more user friendly.

diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -7,5 +7,5 @@ include($ENV{IDF_PATH}/tools/cmake/project.cmake)
 # Problem with IPv6 being disabled and these components.
 set(EXCLUDE_COMPONENTS "coap" "asio")
 
-project(alarmdecoder_stsdk_esp32)
+project(alarmdecoder_ad2iot_esp32)
 
diff --git a/Makefile b/Makefile
@@ -3,7 +3,7 @@
 # project subdirectory.
 #
 
-PROJECT_NAME := alarmdecoder_stsdk_esp32
+PROJECT_NAME := alarmdecoder_ad2iot_esp32
 
 EXTRA_COMPONENT_DIRS := ${IDF_PATH}/../../iot-core
 

diff --git a/README.md b/README.md
@@ -6,7 +6,7 @@
   * 3.2. [SmartThings build (stsdk) - alarmdecoder_stsdk_esp32.bin](#smartthings-build-(stsdk)---alarmdecoder_stsdk_esp32.bin)
 * 4. [Configuring the AD2IoT device](#configuring-the-ad2iot-device)
 * 5. [AD2Iot CLI - command line interface](#ad2iot-cli---command-line-interface)
-  * 5.1. [Main commands](#main-commands)
+  * 5.1. [Basic commands](#basic-commands)
   * 5.2. [Ser2sock server component](#ser2sock-server-component)
     * 5.2.1. [Configuration for Ser2sock server](#configuration-for-ser2sock-server)
   * 5.3. [Web User Interface webUI component](#web-user-interface-webui-component)
@@ -110,8 +110,9 @@ Configuration of the AD2IoT is done directly over the USB serial port using a co
       - ```ad2source COM 4:36```
     - Network shared AD2* device over ser2sock
       - ```ad2source SOCK 192.168.0.121:10000```
-  - Configure the AlarmDecoder firmware settings for the the attached alarm system. For Ademco mode a free keypad address needs to be assigned to each partition to control. DSC mode is ZeroConf and only requires the partition # from 1-8.
-    - ```ad2config mode=A&address=18```
+  - Configure the AlarmDecoder firmware settings for the the attached alarm system. For Ademco mode a free keypad address needs to be assigned to each partition to control. DSC mode is ZeroConf and only requires the mode 'D' and the partition # from 1-8.
+    - Typical Ademco Vista setting: ```ad2config mode=A&address=18```
+    - Typical DSC Power Series setting: ```ad2config mode=D&address=1```
   - Configure the default partition address and optional zones in partition 1.
     - ```partition 1 18 2,3,4,5```
   - Define any additional partitions and optional zones.
@@ -134,7 +135,7 @@ Configuration of the AD2IoT is done directly over the USB serial port using a co
       - ```restart```
     - Configure notifications
 
-  - SmartThings Direct-connected device mode.
+  - SmartThings Direct-connected device mode.```*stsdk firmware build only```
     - Disable networking to allow the SmartThings driver to manage the network hardware and allow adopting over 802.11 b/g/n 2.4ghz Wi-Fi.
       - ```netmode N```
     - Configure the default partition in slot 1 for the partition to connect to the SmartThings app.
@@ -151,7 +152,7 @@ Configuration of the AD2IoT is done directly over the USB serial port using a co
       - ```restart```
     - Additional notification components will only work after the device is adopted and connected to the local Wi-Fi network.
 
-###  5.1. <a name='main-commands'></a>Main commands
+###  5.1. <a name='basic-commands'></a>Main commands
 - help
 ```console
 Usage: help [command]
@@ -182,6 +183,39 @@ Usage: restart
 Usage: factory-reset
     Erase config storage and reboot to factory defaults
 ```
+- top
+```console
+Usage: top
+    Provides a dynamic real-time view of the running system
+    Press any key to exit
+
+Example:
+
+top - 15:40:23.477 up 31 days TS: 2734823413319 Tasks: 14
+Mem: 298328 total, 95508 free, 37876 min free
+
+Name            ID  State Priority Stack CPU# Time                 %TBusy %Busy 
+sys_evt           8 B           20  1048    0                 4646   0.00   0.00
+AD2 ota check    15 B            0  1368    0             50118679   0.00   0.00
+Tmr Svc           4 B            1  1480    0            343028056   0.01   0.03
+IDLE              3 R            0  1820    0        2692433658498  98.45  96.25
+AD2 cli           6 R            2  1932    0          10170914876   0.37   1.25
+tiT               7 B           18  2364    0           1134852575   0.04   0.05
+emac_rx           9 S           15  3236    0           1149697995   0.04   0.02
+httpd            13 B            5  3340    0                  769   0.00   0.00
+AD2 main         14 B            1  3468    0          11172565723   0.41   0.41
+esp_timer         1 S           22  3672    0                29148   0.00   0.00
+AD2 webUI        11 B            1  4192    0             48403756   0.00   0.00
+AD2 GPIO COM RX   5 B            2  4288    0          18014970412   0.66   2.00
+AD2 sendQ        10 B            1  5460    0            298731230   0.01   0.00
+ftp daemon       12 B            1  7376    0                  615   0.00   0.00
+
+   State legend
+    'B'locked 'R'eady 'D'eleted 'S'uspended
+   Column legend
+    Stack: Minimum stack free bytes, CPU#: CPU affinity
+    TBusy: % busy total, Busy: % busy now
+```
 - upgrade
 ```console
 Usage: upgrade [buildflag]
@@ -390,10 +424,11 @@ Examples:
 ```
 
 ###  5.4. <a name='smartthings-direct-connected-device.'></a>SmartThings Direct Connected device.
+###### ```Only available in stsdk firmware build```
 Direct-connected devices connect directly to the SmartThings cloud. The SDK for Direct Connected Devices is equipped to manage all MQTT topics and onboarding requirements, freeing you to focus on the actions and attributes of your device. To facilitate the development of device application in an original chipset SDK, the core device library and the examples were separated into two git repositories. That is, if you want to use the core device library in your original chipset SDK that installed before, you may simply link it to develop a device application in your existing development environment. For more info see https://github.com/SmartThingsCommunity/st-device-sdk-c-ref.
 
 ####  5.4.1. <a name='-configuration-for-smartthings-iot-client'></a> Configuration tool for SmartThings IoT client
-- Enable SmartThings component
+- Enable SmartThings component *stsdk
   - ```stenable {bool}```
     - {bool}: [Y]es/[N]o
 - Sets the SmartThings device_info serialNumber.
@@ -481,26 +516,27 @@ Twilio (/ˈtwɪlioʊ/) is an American cloud communications platform as a service
 
 ####  5.6.1. <a name='configuration-for-twilio-notifications'></a>Configuration tool for Twilio notifications
 ```console
-Usage: twilio (sid|token|from|to|type|format) <acid> [<arg>]
+Usage: twilio (disable|sid|token|from|to|type|format) <acid> [<arg>]
 Usage: twilio switch <swid> [delete|-|notify|open|close|trouble] [<arg>]
 
     Configuration tool for Twilio + SendGrid notifications
 Commands:
-    sid acid [hash]         Twilio String Identifider(SID)
+    disable acid [Y|N]      Disable notification account(acid)
+    sid acid [hash]         Twilio String Identifier(SID)
     token acid [hash]       Twilio Auth Token
     from acid [address]     Validated Email or Phone #
     to acid [address]       Email or Phone #
     type acid [M|C|E]       Notification type Mail, Call, EMail
     format acid [format]    Output format string
     switch swid SCMD [ARG]  Configure switches
-Sub-Commands:
+Sub-Commands: switch
     delete | -              Clear switch notification settings
-    notify <acid>,...       List of accounts [1-8] to use for notification
+    notify <acid>,...       List of accounts [1-999] to use for notification
     open <message>          Send <message> for OPEN events
     close <message>         Send <message> for CLOSE events
     trouble <message>       Send <message> for TROUBLE events
 Options:
-    acid                    Account storage location 1-8
+    acid                    Account storage location 1-999
     swid                    ad2iot virtual switch ID 1-255.
                             See ```switch``` command
     message                 Message to send for this notification
@@ -805,8 +841,8 @@ AD2IOT # help
 Available AD2IoT terminal commands
   [ser2sockd, twilio, pushover, webui, mqtt, ftpd,
    restart, netmode, switch, zone, code, partition,
-   ad2source, ad2config, ad2term, logmode, factory-reset, help,
-   upgrade, version]
+   ad2source, ad2config, ad2term, logmode, factory-reset, top,
+   help, upgrade, version]
 Type help <command> for details on each command.
 
 ```

diff --git a/address_info.txt b/address_info.txt
@@ -1,4 +1,5 @@
-ota_data_initial.bin : 0x11000
 bootloader.bin : 0x1000
-alarmdecoder_stsdk_esp32.bin : 0x20000
-partitions.2MB.bin : 0x8000
+partitions.bin : 0x8000
+ota_data_initial.bin : 0xd000
+spiffs.bin : 0x11000
+alarmdecoder_ad2iot_esp32.bin : 0x60000
diff --git a/components/alarmdecoder-api/alarmdecoder_api.cpp b/components/alarmdecoder-api/alarmdecoder_api.cpp
@@ -452,10 +452,10 @@ void AlarmDecoderParser::notifySearchSubscribers(ad2_message_t mt, std::string &
             /// only test if supplied.
             if (pfregex.length()) {
                 std::smatch m;
-                std::regex e(pfregex);
+                std::regex re(pfregex);
                 const auto& tmsg = msg;
                 std::match_results< std::string::const_iterator > mr;
-                bool res = std::regex_search(tmsg, m, e);
+                bool res = std::regex_search(tmsg, m, re);
                 if (!res) {
                     // no match next subscriber.
                     continue;
@@ -466,21 +466,29 @@ void AlarmDecoderParser::notifySearchSubscribers(ad2_message_t mt, std::string &
             /// only test if a list is supplied.
             if (!done && eSearch->CLOSE_REGEX_LIST.size()) {
                 for(auto &regexstr : eSearch->CLOSE_REGEX_LIST) {
-                    std::smatch m;
-                    std::regex e(regexstr);
-                    const auto& tmsg = msg;
-                    std::match_results< std::string::const_iterator > mr;
-                    bool res = std::regex_search(tmsg, m, e);
-                    if (res) {
-                        // no match next subscriber.
-                        eSearch->setState(AD2_STATE_CLOSED);
-                        outformat = eSearch->CLOSE_OUTPUT_FORMAT;
-                        // Clear last output results before we collect new.
-                        eSearch->RESULT_GROUPS.clear();
-                        // save the regex group results if any.
-                        for(auto idx : m) {
-                            eSearch->RESULT_GROUPS.push_back(idx);
+                    try {
+                        std::smatch m;
+                        std::regex re(regexstr);
+                        const auto& tmsg = msg;
+                        std::match_results< std::string::const_iterator > mr;
+                        bool res = std::regex_search(tmsg, m, re);
+                        if (res) {
+                            // regex match
+                            eSearch->setState(AD2_STATE_CLOSED);
+                            outformat = eSearch->CLOSE_OUTPUT_FORMAT;
+                            // Clear last output results before we collect new.
+                            eSearch->RESULT_GROUPS.clear();
+                            // save the regex group results if any.
+                            for(auto idx : m) {
+                                eSearch->RESULT_GROUPS.push_back(idx);
+                            }
+                            done = true;
+                            break;
                         }
+                    } catch (std::exception const& e) { // catch (std::regex_error& e) {
+#if defined(IDF_VER)
+                        ESP_LOGE(TAG, "!ERR: regex error: '%s' '%s' '%s'", e.what(), regexstr.c_str(), msg.c_str());
+#endif
                         done = true;
                         break;
                     }
@@ -496,7 +504,7 @@ void AlarmDecoderParser::notifySearchSubscribers(ad2_message_t mt, std::string &
                     const auto& tmsg = msg;
                     bool res = std::regex_search(tmsg, m, e);
                     if (res) {
-                        // no match next subscriber.
+                        // regex match
                         eSearch->setState(AD2_STATE_OPEN);
                         outformat = eSearch->OPEN_OUTPUT_FORMAT;
                         // Clear last output results before we collect new.
@@ -521,7 +529,7 @@ void AlarmDecoderParser::notifySearchSubscribers(ad2_message_t mt, std::string &
                     std::match_results< std::string::const_iterator > mr;
                     bool res = std::regex_search(tmsg, m, e);
                     if (res) {
-                        // no match next subscriber.
+                        // regex match
                         outformat = eSearch->TROUBLE_OUTPUT_FORMAT;
                         eSearch->setState(AD2_STATE_TROUBLE);
                         // Clear last output results before we collect new.

diff --git a/components/ftpd/ftpd.cpp b/components/ftpd/ftpd.cpp
@@ -1424,7 +1424,7 @@ void FTPD::processCommand()
 
             }
         }
-        vTaskDelay(10 / portTICK_PERIOD_MS);
+        vTaskDelay(100 / portTICK_PERIOD_MS);
     } // End loop processing commands.
 
 #if defined(FTPD_DEBUG)
@@ -1851,7 +1851,7 @@ void ftpd_init()
     }
 
     ad2_printf_host(true, "%s: Init done. Daemon starting.", TAG);
-    xTaskCreate(&ftp_daemon_task, "ftp_daemon_task", 1024*8, NULL, tskIDLE_PRIORITY+1, NULL);
+    xTaskCreate(&ftp_daemon_task, "ftp daemon", 1024*8, NULL, tskIDLE_PRIORITY+1, NULL);
 
 }
 #endif /* CONFIG_AD2IOT_FTP_DAEMON */
diff --git a/components/otaupdate/CMakeLists.txt b/components/otaupdate/CMakeLists.txt
@@ -1,4 +1,3 @@
-set(EXTRA_EMBED "ota_update_server_root.pem")
 idf_component_register(SRCS "ota_util.cpp"
                     REQUIRES idf::nvs_flash
                     REQUIRES idf::alarmdecoder-api

diff --git a/components/otaupdate/ota_update_server_root.pem b/components/otaupdate/ota_update_server_root.pem