feat(esptool): allow filtering ports by VID/PID (ESPTOOL-878)

[bryghtlabs-richard]

Enables user to specify --usbVid and/or --usbPid, which filter the com port list before checking for ESP chips. With many serial ports on the computer, this greatly speeds up the process of esptool identifying the correct com port by allowing esptool to skip all com ports without a matching VID/PID. Implements #987

I have tested this change with the following hardware & software combinations:

NO TESTING ON MAC/LINUX. Tested on Windows64 with ESP32-S3-DevKitC and custom ESP32-S3 board.

$ time esptool read_flash_status
esptool.py v4.7.0
Found 4 serial ports
Serial port COM30
Connecting......................................
COM30 failed to connect: Failed to connect to Espressif device: No serial data received.
For troubleshooting steps visit: https://docs.espressif.com/projects/esptool/en/latest/troubleshooting.html
Serial port COM29
Connecting......................................
COM29 failed to connect: Failed to connect to Espressif device: No serial data received.
For troubleshooting steps visit: https://docs.espressif.com/projects/esptool/en/latest/troubleshooting.html
Serial port COM18
Connecting...
Detecting chip type... ESP32-S3
...long text omitted...
real    0m18.193s
user    0m0.000s
sys     0m0.015s

$ time esptool --usbVid 0x303a read_flash_status
esptool.py v4.7.0
Found 1 serial ports
Serial port COM18
Connecting...
Detecting chip type... ESP32-S3
...long text omitted...
real    0m1.067s
user    0m0.000s
sys     0m0.000s

I have run the esptool.py automated integration tests with this change and the above hardware:

$ pytest test_esptool.py --port COM49 --chip esp32s3 --baud 230400
============================= test session starts =============================
platform win32 -- Python 3.11.5, pytest-8.2.2, pluggy-1.5.0
rootdir: C:\Projects\esptool
configfile: pyproject.toml
collected 88 items

test_esptool.py sss.......ss.................s.s.........s....sss.s..... [ 63%]
......s...sssssss..ssss..s..ssss [100%]

============================== warnings summary ===============================
test_esptool.py:120
C:\Projects\esptool\test\test_esptool.py:120: PytestUnknownMarkWarning: Unknown pytest.mark.flaky - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.flaky(reruns=1, condition=arg_preload_port is not False)

test_esptool.py:1204
C:\Projects\esptool\test\test_esptool.py:1204: PytestUnknownMarkWarning: Unknown pytest.mark.flaky - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.flaky(reruns=5)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========== 59 passed, 29 skipped, 2 warnings in 415.91s (0:06:55) ============

[dobairoland]

Thank you for the contribution. I think this is a good idea but we might think about a little what new CLI arguments we add. I'm worried that others might need other type of filtering (e.g. serial_number, location) for which we would have to add also new arguments.

• This is one solution to merge the arguments: --port-filter <vid>:<pid> where <vid> and <pid> could be * as well.
• An even more general solution: --port-filter vid=<number> --port-filter pid=<number> where --port-filter can be repeated several times and can be easily extended.

What do you think @radimkarnis, @peterdragun?

[peterdragun]

Overall I like the idea, but as mentioned I would go with one argument for both options.
Maybe we can extend the filter to also cover the port name? For example on macOS ports can have different names based on whether they are using a USB to UART bridge (/dev/cu.usbserial-XX) or USB Serial/JTAG (/dev/cu.usbmodemXX). So in case I know which peripheral I want to use but don't know the number it may be helpful. I mean we can do that based on VID as well, but it would be easier to remember the port name than VID.

[dobairoland]

Overall I like the idea, but as mentioned I would go with one argument for both options. Maybe we can extend the filter to also cover the port name? For example on macOS ports can have different names based on whether they are using a USB to UART bridge (/dev/cu.usbserial-XX) or USB Serial/JTAG (/dev/cu.usbmodemXX). So in case I know which peripheral I want to use but don't know the number it may be helpful. I mean we can do that based on VID as well, but it would be easier to remember the port name than VID.

I think this could be covered by the following if we go with my second proposal:

--port-filter name=usbserial

[bryghtlabs-richard]

An even more general solution: --port-filter vid= --port-filter pid= where --port-filter can be repeated several times and can be easily extended.

I like this.

I think this could be covered by the following if we go with my second proposal: --port-filter name=usbserial

Are you thinking a substring search, or should we check against a prefix from the pyserial port name, like name=/dev/cu.usbserial?

[dobairoland]

Are you thinking a substring search, or should we check against a prefix from the pyserial port name, like name=/dev/cu.usbserial?

Substring search is what I had in my mind. ttyUSB and ttyACM could be easily used on Linux, or even simply USB or ACM. But this was just a suggestion so we don't end up needing several similar arguments. You don't have to address this in this PR (if you don't want).

On the other hand, what would be great to update in this PR is the documentation and mention this new feature somewhere. Perhaps in a separate section in https://docs.espressif.com/projects/esptool/en/latest/esp32/esptool/advanced-options.html.

[dobairoland]

Hi @bryghtlabs-richard. Could you please get back to this pull request and finish it as we've discussed?

[github-actions]

	Messages
📖	🎉 Good Job! All checks are passing!

👋 Hello bryghtlabs-richard, we appreciate your contribution to this project!

---

📘 Please review the project's Contributions Guide for key guidelines on code, documentation, testing, and more.

---

Click to see more instructions ...

This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...

We do welcome contributions in the form of bug reports, feature requests and pull requests via this public GitHub repository.

This GitHub project is public mirror of our internal git repository

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved, we synchronize it into our internal git repository.
4. In the internal git repository we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
5. If the change is approved and passes the tests it is merged into the default branch.
5. On next sync from the internal git repository merged change will appear in this public GitHub repository.

Generated by 🚫 dangerJS against 5dd3dcc

[bryghtlabs-richard]

Hi @dobairoland , I'm back from a vacation and I think I've got everything we want, and updated the documentation.

[dobairoland]

Thanks. This looks pretty good. I've left a couple of last suggestions.

[bryghtlabs-richard]

Thanks Roland, I've updated the commit with your suggestions.

[radimkarnis]

Suggestion: A nice touch would be to add a mention about the Espressif USB customer-allocated PID repository
https://github.com/espressif/usb-pids. Essentially we allow users to register a PID under the Espressif VID for a specific piece of hardware. So one of the examples could maybe mention this?

Feel free to ignore, though!

[radimkarnis]

Nice feature, thanks for finishing this! LGTM.

[Jason2866]

Agree, nice feature!

[dobairoland]

@bryghtlabs-richard Thank you for the contribution. It was merged.