feat(examples): websocket autobahn test suit integration (IDF-14864) #960

gabsuren · 2025-11-28T11:41:58Z

Note

Adds an Autobahn testsuite example and Linux CI to build/run it and upload reports, plus safer websocket transport error logging and minor linux_compat timer deps.

Examples:
- Autobahn testsuite: New components/esp_websocket_client/examples/autobahn-testsuite with Docker configs, quick/full configs, scripts (run_tests.sh, monitor_serial.py, generate_summary.py, etc.), and a testee app to run all cases and generate reports.
CI:
- Linux workflow (.github/workflows/autobahn__linux-test.yml): starts fuzzing server, builds Linux testee, runs tests, generates summary, uploads reports.
- Target build workflow (.github/workflows/autobahn__target-test.yml): builds and merges ESP32 binaries for the testee (run-on-target job stubbed/commented).
WebSocket client (components/esp_websocket_client/esp_websocket_client.c): safer transport error logging using guarded esp_err_to_name() across write/read/connect/poll paths.
Linux compat timer:
- Add esp_common requirement and include esp_err.h in esp_timer linux shim.

^{Written by Cursor Bugbot for commit fa1ef1a. This will update automatically on new commits. Configure here.}

CLAassistant · 2025-11-28T11:42:06Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

cursor

This is the final PR Bugbot will review for you during this billing cycle

Your free Bugbot reviews will reset on December 20

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

components/esp_websocket_client/examples/autobahn-testsuite/testee/main/autobahn_testee.c

components/esp_websocket_client/examples/autobahn-testsuite/run_tests.sh

components/esp_websocket_client/examples/autobahn-testsuite/testee/main/autobahn_testee.c

components/esp_websocket_client/examples/autobahn-testsuite/scripts/quick_test.sh

components/esp_websocket_client/examples/autobahn-testsuite/scripts/monitor_serial.py

.github/workflows/autobahn__linux-test.yml

components/esp_websocket_client/examples/autobahn-testsuite/testee/main/autobahn_testee.c

components/esp_websocket_client/examples/autobahn-testsuite/scripts/analyze_results.py

components/esp_websocket_client/examples/autobahn-testsuite/testee/main/autobahn_testee.c

euripedesrocha · 2025-12-08T15:30:03Z

.github/workflows/autobahn__linux-test.yml

+    if: github.event_name == 'push' || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'autobahn'))
+
+    env:
+      TEST_DIR: components/esp_websocket_client/examples/autobahn-testsuite


We should move this to a different place, as this isn't an example. Better to create a test folder.

euripedesrocha · 2025-12-09T11:51:05Z

.github/workflows/autobahn__linux-test.yml

+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          submodules: recursive


Websocket client doesn't require the submodules, right?

euripedesrocha · 2025-12-09T11:53:39Z

.github/workflows/autobahn__linux-test.yml

+          docker logs fuzzing-server
+          exit 1
+
+      - name: Build testee (Linux target)


Suggested change

- name: Build testee (Linux target)

- name: Build test (Linux target)

euripedesrocha · 2025-12-16T11:17:16Z

.github/workflows/autobahn__target-test.yml

+jobs:
+  build_autobahn:
+    # Run on push to master or if PR has 'websocket' label
+    if: contains(github.event.pull_request.labels.*.name, 'autobahn') || github.event_name == 'push'


Suggested change

if: contains(github.event.pull_request.labels.*.name, 'autobahn') || github.event_name == 'push'

if: contains(github.event.pull_request.labels.*.name, 'websocket-autobahn') || github.event_name == 'push'

euripedesrocha · 2025-12-16T11:21:06Z

.github/workflows/autobahn__target-test.yml

+    name: Build
+    strategy:
+      matrix:
+        #idf_ver: ["release-v5.0", "release-v5.1", "release-v5.2", "release-v5.3", "latest"]


We should cover IDF versions that are currently in service period[5.5, latest], and we should at least have a manual run on older idf versions to identify potential fix to be added.

euripedesrocha · 2025-12-16T11:39:16Z

components/esp_websocket_client/examples/autobahn-testsuite/testee/main/autobahn_testee.c

+#define BUFFER_SIZE         16384  // Reduced from 32768 to free memory for accumulator buffer
+#define START_CASE          1
+#define END_CASE            300
+// Configure test range here:


This comment is misplaced as it refers to the range above. We could have esp_console for this and have this as runtime option.

euripedesrocha · 2025-12-16T13:32:47Z

components/esp_websocket_client/examples/autobahn-testsuite/testee/main/CMakeLists.txt

@@ -0,0 +1,10 @@
+# Conditionally require components based on target
+if(${IDF_TARGET} STREQUAL "linux")


This could be simplified and more maintenable if we create a base list with common components and extend it with the specific and add that list to the requirements.

euripedesrocha · 2025-12-16T13:35:23Z

components/esp_websocket_client/examples/autobahn-testsuite/testee/CMakeLists.txt

+# COMPONENTS must be set before include() to limit component scanning
+if(${IDF_TARGET} STREQUAL "linux")
+    set(common_component_dir ../../../../../common_components)
+    set(EXTRA_COMPONENT_DIRS


We should move this to be handled by component manager.

euripedesrocha · 2025-12-16T13:44:15Z

components/esp_websocket_client/esp_websocket_client.c

                    esp_websocket_client_error(client, "esp_transport_connect() failed with %d, "
                                               "transport_error=%s, tls_error_code=%i, tls_flags=%i, esp_ws_handshake_status_code=%d, errno=%d",
-                                               result, esp_err_to_name(error_handle->last_error), error_handle->esp_tls_error_code,
+                                               result, error_name ? error_name : "UNKNOWN", error_handle->esp_tls_error_code,


Is this necessary? esp_err_to_name handles inexistent code and from quick code inspection always returns a valid pointer.

euripedesrocha · 2025-12-16T13:50:35Z

components/esp_websocket_client/examples/autobahn-testsuite/README.md

+```
+This generates a detailed console summary and an HTML summary at `reports/summary.html`.
+
+**Option 3: Direct File Access**


This are unnecessary instructions.

gabsuren added the autobahn label Nov 28, 2025

gabsuren force-pushed the test/autobah-testsuite-2 branch 10 times, most recently from 338e6c6 to 0b8ebbf Compare December 3, 2025 10:54

gabsuren force-pushed the test/autobah-testsuite-2 branch 2 times, most recently from c6f9902 to 20bbe35 Compare December 8, 2025 11:09

gabsuren marked this pull request as ready for review December 8, 2025 11:11

cursor bot reviewed Dec 8, 2025

View reviewed changes

gabsuren changed the title ~~feat(examples): websocket autobahn test suit integration~~ feat(examples): websocket autobahn test suit integration (IDF-14864) Dec 8, 2025

gabsuren force-pushed the test/autobah-testsuite-2 branch from 20bbe35 to 138574c Compare December 8, 2025 11:54

cursor bot reviewed Dec 8, 2025

View reviewed changes

gabsuren self-assigned this Dec 8, 2025

gabsuren force-pushed the test/autobah-testsuite-2 branch from 138574c to 473ca9f Compare December 8, 2025 14:12

cursor bot reviewed Dec 8, 2025

View reviewed changes

components/esp_websocket_client/examples/autobahn-testsuite/scripts/analyze_results.py Show resolved Hide resolved

components/esp_websocket_client/examples/autobahn-testsuite/testee/main/autobahn_testee.c Show resolved Hide resolved

gabsuren requested review from bogdankolendovskyy, david-cermak and euripedesrocha December 8, 2025 14:22

gabsuren force-pushed the test/autobah-testsuite-2 branch 3 times, most recently from 875d3b8 to 048b0c5 Compare December 9, 2025 16:00

euripedesrocha reviewed Dec 16, 2025

View reviewed changes

fix(websocket): prevent NULL pointer crash in error handling

fb37ea6

feat(examples): websocket autobahn test suit integration

fa1ef1a

gabsuren force-pushed the test/autobah-testsuite-2 branch from 048b0c5 to fa1ef1a Compare December 18, 2025 13:11

	- name: Build testee (Linux target)
	- name: Build test (Linux target)

	if: contains(github.event.pull_request.labels.*.name, 'autobahn') \|\| github.event_name == 'push'
	if: contains(github.event.pull_request.labels.*.name, 'websocket-autobahn') \|\| github.event_name == 'push'

		@@ -0,0 +1,10 @@
		# Conditionally require components based on target
		if(${IDF_TARGET} STREQUAL "linux")

feat(examples): websocket autobahn test suit integration (IDF-14864) #960

Are you sure you want to change the base?

feat(examples): websocket autobahn test suit integration (IDF-14864) #960

Uh oh!

Conversation

gabsuren commented Nov 28, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Nov 28, 2025

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

This is the final PR Bugbot will review for you during this billing cycle

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gabsuren commented Nov 28, 2025 •

edited by cursor bot

Loading