Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: Thermocycler fails performing basic PCR program after lid closing/shifting temperatures #9175

Open
peradastra opened this issue Jan 1, 2022 · 5 comments
Labels

Comments

@peradastra
Copy link

peradastra commented Jan 1, 2022

Overview

When running a simple PCR program the thermocycler (Tc) fails with a uniformity error after lid closing. Interestingly, this error only happens when the Tc has been turned off long enough for the lid and plate to cool down to the ambient temperature.

Steps to reproduce

  1. Turn off all hardware (OT2* & Tc**) and allow Tc plate and lid to return to ambient (~23°C) temperature.
  2. Turn on OT2.
  3. Turn on Tc.
  4. Upload this run protocol.
  5. After simulation is complete, initiate protocol.

*OT2 server version is 4.7.0.
*Tc firmware version is v1.1.0.

Documentation of error

I was able to reproduce this error 3 separate times using 2 different OT2s (A & B) and 3 different Tc's (A, B, & C). Below are 3 videos simultaneously showing the OT2 and app running in real-time up until they encounter the error. Below is also one API log which I downloaded and parsed only for the error below.

Videos

Error on OT2 A with Tc A

Error on OT2 A with Tc A

Error on OT2 A with Tc A

API Logs

Dec 31 22:08:33 e9eb27 opentrons-api[208]: rpc built call run on <opentrons.api.session.Session object at 0x70c47690>
Dec 31 22:08:33 e9eb27 opentrons-api[208]: Updating instrument model cache
Dec 31 22:08:34 e9eb27 opentrons-api[208]: loaded: p300_multi_v2.0, pipette offset: [-1.5000000000000089, -1.3000000000000185, 1.5987211554602254e-14]
Dec 31 22:08:34 e9eb27 opentrons-api[208]: Doing full configuration on LEFT
Dec 31 22:08:34 e9eb27 opentrons-api[208]: Updated move split config with {'B': MoveSplit(split_distance=1, split_current=1.75, split_speed=1, after_time=1800, fullstep=True)}
Dec 31 22:08:34 e9eb27 opentrons-api[208]: loaded: p20_multi_v2.1, pipette offset: [1.6000000000000014, -2.3000000000000043, 0.7999999999999972]
Dec 31 22:08:34 e9eb27 opentrons-api[208]: Doing full configuration on RIGHT
Dec 31 22:08:34 e9eb27 opentrons-api[208]: Instruments found: {<Mount.LEFT: 1>: <Pipette: P300 8-Channel GEN2 1892634800>, <Mount.RIGHT: 2>: <Pipette: P20 8-Channel GEN2 1873690832>}
Dec 31 22:08:34 e9eb27 opentrons-api[208]: Starting hardware event watcher
Dec 31 22:08:34 e9eb27 opentrons-api[208]: Session.set_state state=running user_message=None duration=None reason=None
Dec 31 22:08:34 e9eb27 opentrons-api[208]: Session.set_state state=running user_message=None duration=None reason=None
Dec 31 22:08:34 e9eb27 opentrons-api[208]: Homing axes ZA in sequence ['ZA']
Dec 31 22:08:36 e9eb27 opentrons-api[208]: Updating instrument model cache
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Skipping configuration on LEFT
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Skipping configuration on RIGHT
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Instruments found: {<Mount.LEFT: 1>: <Pipette: P300 8-Channel GEN2 1892634800>, <Mount.RIGHT: 2>: <Pipette: P20 8-Channel GEN2 1873690832>}
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Resetting configuration for LEFT
Dec 31 22:08:38 e9eb27 opentrons-api[208]: loaded: p300_multi_v2.0, pipette offset: [-1.5000000000000089, -1.3000000000000185, 1.5987211554602254e-14]
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Resetting configuration for RIGHT
Dec 31 22:08:38 e9eb27 opentrons-api[208]: loaded: p20_multi_v2.1, pipette offset: [1.6000000000000014, -2.3000000000000043, 0.7999999999999972]
Dec 31 22:08:38 e9eb27 opentrons-api[208]: /data/robot_settings.json not found. Loading defaults
Dec 31 22:08:38 e9eb27 opentrons-api[208]: /data/deck_calibration.json not found. Loading defaults
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Updating instrument model cache
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Skipping configuration on LEFT
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Skipping configuration on RIGHT
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Instruments found: {<Mount.LEFT: 1>: None, <Mount.RIGHT: 2>: None}
Dec 31 22:08:38 e9eb27 opentrons-api[208]: Updating instrument model cache
Dec 31 22:08:39 e9eb27 opentrons-api[208]: Skipping configuration on LEFT
Dec 31 22:08:39 e9eb27 opentrons-api[208]: Skipping configuration on RIGHT
Dec 31 22:08:39 e9eb27 opentrons-api[208]: Instruments found: {<Mount.LEFT: 1>: <Pipette: P300 8-Channel GEN2 1873692752>, <Mount.RIGHT: 2>: <Pipette: P20 8-Channel GEN2 1892636080>}
Dec 31 22:08:39 e9eb27 opentrons-api[208]: Homing axes XYZAXZA in sequence ['ZA', 'X', 'Y']
Dec 31 22:08:45 e9eb27 opentrons-api[208]: Homing axes B in sequence ['B']
Dec 31 22:08:52 e9eb27 opentrons-api[208]: Homing axes C in sequence ['C']
Dec 31 22:08:58 e9eb27 opentrons-api[208]: command.THERMOCYCLER_OPEN: 
Dec 31 22:08:58 e9eb27 opentrons-api[208]: Cannot assure a safe gantry position to avoid colliding with the lid of the Thermocycler Module.
Dec 31 22:08:58 e9eb27 opentrons-api[208]: command.THERMOCYCLER_SET_BLOCK_TEMP: temperature: 4, hold_time: None
Dec 31 22:09:39 e9eb27 opentrons-api[208]: command.THERMOCYCLER_SET_LID_TEMP: 
Dec 31 22:12:17 e9eb27 opentrons-api[208]: command.THERMOCYCLER_CLOSE: 
Dec 31 22:12:17 e9eb27 opentrons-api[208]: Cannot assure a safe gantry position to avoid colliding with the lid of the Thermocycler Module.
Dec 31 22:12:43 e9eb27 opentrons-api[208]: command.THERMOCYCLER_OPEN: 
Dec 31 22:12:43 e9eb27 opentrons-api[208]: Cannot assure a safe gantry position to avoid colliding with the lid of the Thermocycler Module.
Dec 31 22:13:12 e9eb27 opentrons-api[208]: command.THERMOCYCLER_CLOSE: 
Dec 31 22:13:12 e9eb27 opentrons-api[208]: Cannot assure a safe gantry position to avoid colliding with the lid of the Thermocycler Module.
Dec 31 22:13:36 e9eb27 opentrons-api[208]: Thermocycler has encountered an unrecoverable error: /dev/ot_module_thermocycler0: 'Received error response 'ERROR:Plate temperature not uniform. Deactivating.
                                           Error:Plate temperature is not uniform
                                           Lid:closed'. Please refer to support article at https://support.opentrons.com/en/articles/3469797-thermocycler-module for troubleshooting.
Dec 31 22:13:36 e9eb27 opentrons-api[208]: Exception during run:
                                           Traceback (most recent call last):
                                             File "usr/lib/python3.7/site-packages/opentrons/protocols/execution/execute_python.py", line 60, in run_python
                                             File "<string>", line 1, in <module>
                                             File "thermocycler_testing.py", line 20, in run
                                             File "usr/lib/python3.7/site-packages/opentrons/commands/publisher.py", line 94, in _decorated
                                             File "usr/lib/python3.7/site-packages/opentrons/protocols/api_support/util.py", line 420, in _check_version_wrapper
                                             File "usr/lib/python3.7/site-packages/opentrons/protocol_api/module_contexts.py", line 495, in close_lid
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/adapters.py", line 56, in call_coroutine_sync
                                             File "usr/lib/python3.7/concurrent/futures/_base.py", line 432, in result
                                             File "usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/modules/thermocycler.py", line 181, in close
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/modules/thermocycler.py", line 345, in _wait_for_lid_status
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/modules/thermocycler.py", line 349, in wait_next_poll
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/poller.py", line 75, in wait_next_poll
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/poller.py", line 141, in _poller
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/modules/thermocycler.py", line 549, in read
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/thermocycler/driver.py", line 119, in get_lid_status
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/asyncio/communication/serial_connection.py", line 119, in send_command
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/asyncio/communication/serial_connection.py", line 138, in send_data
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/asyncio/communication/serial_connection.py", line 170, in _send_data
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/asyncio/communication/serial_connection.py", line 212, in raise_on_error
                                           opentrons.drivers.asyncio.communication.errors.ErrorResponse: /dev/ot_module_thermocycler0: 'Received error response 'ERROR:Plate temperature not uniform. Deactivating.
                                           Error:Plate temperature is not uniform
                                           Lid:closed'
                                           
                                           The above exception was the direct cause of the following exception:
                                           
                                           Traceback (most recent call last):
                                             File "usr/lib/python3.7/site-packages/opentrons/api/session.py", line 627, in _run
                                             File "usr/lib/python3.7/site-packages/opentrons/protocols/execution/execute.py", line 27, in run_protocol
                                             File "usr/lib/python3.7/site-packages/opentrons/protocols/execution/execute_python.py", line 71, in run_python
                                           opentrons.protocols.execution.errors.ExceptionInProtocolError: ErrorResponse [line 20]: /dev/ot_module_thermocycler0: 'Received error response 'ERROR:Plate temperature not uniform. Deactivating.
                                           Error:Plate temperature is not uniform
                                           Lid:closed'
Dec 31 22:13:36 e9eb27 opentrons-api[208]: Session.set_state state=error user_message=None duration=None reason=None
Dec 31 22:13:36 e9eb27 opentrons-api[208]: Smart exception in protocol
                                           Traceback (most recent call last):
                                             File "usr/lib/python3.7/site-packages/opentrons/protocols/execution/execute_python.py", line 60, in run_python
                                             File "<string>", line 1, in <module>
                                             File "thermocycler_testing.py", line 20, in run
                                             File "usr/lib/python3.7/site-packages/opentrons/commands/publisher.py", line 94, in _decorated
                                             File "usr/lib/python3.7/site-packages/opentrons/protocols/api_support/util.py", line 420, in _check_version_wrapper
                                             File "usr/lib/python3.7/site-packages/opentrons/protocol_api/module_contexts.py", line 495, in close_lid
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/adapters.py", line 56, in call_coroutine_sync
                                             File "usr/lib/python3.7/concurrent/futures/_base.py", line 432, in result
                                             File "usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/modules/thermocycler.py", line 181, in close
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/modules/thermocycler.py", line 345, in _wait_for_lid_status
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/modules/thermocycler.py", line 349, in wait_next_poll
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/poller.py", line 75, in wait_next_poll
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/poller.py", line 141, in _poller
                                             File "usr/lib/python3.7/site-packages/opentrons/hardware_control/modules/thermocycler.py", line 549, in read
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/thermocycler/driver.py", line 119, in get_lid_status
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/asyncio/communication/serial_connection.py", line 119, in send_command
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/asyncio/communication/serial_connection.py", line 138, in send_data
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/asyncio/communication/serial_connection.py", line 170, in _send_data
                                             File "usr/lib/python3.7/site-packages/opentrons/drivers/asyncio/communication/serial_connection.py", line 212, in raise_on_error
                                           opentrons.drivers.asyncio.communication.errors.ErrorResponse: /dev/ot_module_thermocycler0: 'Received error response 'ERROR:Plate temperature not uniform. Deactivating.
                                           Error:Plate temperature is not uniform
                                           Lid:closed'
                                           
                                           The above exception was the direct cause of the following exception:
                                           
                                           Traceback (most recent call last):
                                             File "usr/lib/python3.7/site-packages/robot_server/service/legacy/rpc/rpc.py", line 267, in make_call
                                             File "usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
                                             File "usr/lib/python3.7/site-packages/robot_server/service/legacy/rpc/rpc.py", line 258, in call_and_serialize
                                             File "usr/lib/python3.7/site-packages/opentrons/api/session.py", line 657, in run
                                             File "usr/lib/python3.7/site-packages/opentrons/api/util.py", line 33, in decorated
                                             File "usr/lib/python3.7/site-packages/opentrons/api/session.py", line 649, in _run
                                             File "usr/lib/python3.7/site-packages/opentrons/api/session.py", line 627, in _run
                                             File "usr/lib/python3.7/site-packages/opentrons/protocols/execution/execute.py", line 27, in run_protocol
                                             File "usr/lib/python3.7/site-packages/opentrons/protocols/execution/execute_python.py", line 71, in run_python
                                           opentrons.protocols.execution.errors.ExceptionInProtocolError: ErrorResponse [line 20]: /dev/ot_module_thermocycler0: 'Received error response 'ERROR:Plate temperature not uniform. Deactivating.
                                           Error:Plate temperature is not uniform
                                           Lid:closed'

Current behavior

mostly speculation on a brief review of a few tickets
I suspect that this behavior is the result of one, or both, already notes issues. Ticket #5602 documents this exact error where a plate uniformity error arises if the lid is opened or closed immediately after set_block_temperature().
However, the resolution to this bug is documented in ticket #5807 through the use of a TEMP_THRESHOLD value calculated by averaging the last 10 polled Tc temperatures to determine a "fluctuation threshold". Given that this error only happens after the Tc has been turned off for several hours which is necessary for both the lid and plate to return to an ambient temperature that there's not enough history for it to poll such that the calculated "fluctuation threshold" is presumably to low. While not documented in this ticket, I have a similar issue where the Tc returns a plate uniformity error after holding at a temperature for 15 minutes and then shifts to another temperate afterward. For example, an RT-PCR reaction is first done by incubation at a specific temperature for several minutes followed by cycling through several different temperatures quickly for PCR. I would suspect that the acceptable TEMP_THRESHOLD` calculated from extended at a specific temperature would lead to such a small value that cycling out of that temperature would return a plate uniformity error only unless the Tc did not get to the next temperature quickly and without overshooting the target temperature.

Expected behavior

For the thermocycler to be able to not fail when doing basic PCR protocols and not when shifting from 1 long incubation to another temperature.

@peradastra peradastra added the bug label Jan 1, 2022
@peradastra
Copy link
Author

peradastra commented Jan 2, 2022

More Data

I built a new version of the Tc firmware locally from the opentrons-module repo which includes commits that allow the print out the thermistors values and uploaded it to the Tc in the event of failure. Here they are...

ErrorResponse [line 43]: /dev/ot_module_thermocycler2: 'Received error response 'Error:Plate temperature is not uniform. T1: 2.0172 T2: 8.0703 T3: 4.2907 T4: 3.7157 T5: 8.7126 T6: 4.4932 T.Lid: 105.6111 T.sink: 61.8712 T_error: 0.0000

@sanni-t
Copy link
Member

sanni-t commented Feb 4, 2022

Thank you for the detailed recording and troubleshooting steps of this issue!

The behavior you are seeing is not due to having the thermocycler turned off for a long time but due to the consecutive, multiple lid open & close commands (although, having the module start off from an ambient temperature could be aggravating the issue slightly). The issue lies in the way the lid's movement affects internal temperature reading. This behavior is documented in Opentrons/opentrons-modules#104.

tl;dr of the issue is that every time the lid is moving, the thermocycler is unable to intelligently correct its block temperature. So each time the lid is opened or closed, the block/plate's temperature deviates slightly. It gets corrected immediately after the motion is completed. But if you have consecutive open and close commands, the thermocycler doesn't get any time to correct its plate's temperature; each consecutive lid command adds to the temperature deviation, ultimately making it large enough to result in a uniformity error.

We don't have a fix planned for this behavior immediately as having the lid open and close multiple times consecutively is not something a protocol would typically need to do. That said, improving the lid functionality is something on our radar.

@sanni-t
Copy link
Member

sanni-t commented Feb 4, 2022

While not documented in this ticket, I have a similar issue where the Tc returns a plate uniformity error after holding at a temperature for 15 minutes and then shifts to another temperate afterward.

This seems to be a different and more concerning issue. Would you mind creating a different ticket for it with the observed behavior and logs? Did this protocol also have consecutive lid open and close commands?

For example, an RT-PCR reaction is first done by incubation at a specific temperature for several minutes followed by cycling through several different temperatures quickly for PCR. I would suspect that the acceptable TEMP_THRESHOLD` calculated from extended at a specific temperature would lead to such a small value that cycling out of that temperature would return a plate uniformity error only unless the Tc did not get to the next temperature quickly and without overshooting the target temperature.

The TEMP_THRESHOLD value you are referring to does not detect the uniformity of the plate. This value is only monitored for a short amount of time in the python API to check whether the temperature set after set_temperature has stabilized, before moving on to the next protocol step. The uniformity detection is done by the thermocycler firmware which continuously monitors the temperatures of different spots on the plate/block. It raises the error only when the plate is supposed to have reached stable temperatures and two spots on the plate are at very different temperatures, like those shown in your logs T1: 2.0172 and T2: 8.0703. Merely cycling through different temperatures shouldn't result in this error. So it'll be helpful for us to see the protocol you are using and where this error is happening.

@peradastra
Copy link
Author

Thanks for the thorough reply and all your hard work in particular addressing the Tc issues! I've been following your commits as I tried to debug. I'll try to adjust my protocols accordingly. I've been automating longer protocols which require multiple opening and closings of the Tc but will break them down. Though once this initially error happens and the Tc is "warmed up," I don't see this error happen again.

Thank you for the detailed recording and troubleshooting steps of this issue!

The behavior you are seeing is not due to having the thermocycler turned off for a long time but due to the consecutive, multiple lid open & close commands (although, having the module start off from an ambient temperature could be aggravating the issue slightly). The issue lies in the way the lid's movement affects internal temperature reading. This behavior is documented in Opentrons/opentrons-modules#104.

tl;dr of the issue is that every time the lid is moving, the thermocycler is unable to intelligently correct its block temperature. So each time the lid is opened or closed, the block/plate's temperature deviates slightly. It gets corrected immediately after the motion is completed. But if you have consecutive open and close commands, the thermocycler doesn't get any time to correct its plate's temperature; each consecutive lid command adds to the temperature deviation, ultimately making it large enough to result in a uniformity error.

We don't have a fix planned for this behavior immediately as having the lid open and close multiple times consecutively is not something a protocol would typically need to do. That said, improving the lid functionality is something on our radar.

@peradastra
Copy link
Author

While not documented in this ticket, I have a similar issue where the Tc returns a plate uniformity error after holding at a temperature for 15 minutes and then shifts to another temperate afterward.

This seems to be a different and more concerning issue. Would you mind creating a different ticket for it with the observed behavior and logs? Did this protocol also have consecutive lid open and close commands?

For example, an RT-PCR reaction is first done by incubation at a specific temperature for several minutes followed by cycling through several different temperatures quickly for PCR. I would suspect that the acceptable TEMP_THRESHOLD` calculated from extended at a specific temperature would lead to such a small value that cycling out of that temperature would return a plate uniformity error only unless the Tc did not get to the next temperature quickly and without overshooting the target temperature.

The TEMP_THRESHOLD value you are referring to does not detect the uniformity of the plate. This value is only monitored for a short amount of time in the python API to check whether the temperature set after set_temperature has stabilized, before moving on to the next protocol step. The uniformity detection is done by the thermocycler firmware which continuously monitors the temperatures of different spots on the plate/block. It raises the error only when the plate is supposed to have reached stable temperatures and two spots on the plate are at very different temperatures, like those shown in your logs T1: 2.0172 and T2: 8.0703. Merely cycling through different temperatures shouldn't result in this error. So it'll be helpful for us to see the protocol you are using and where this error is happening.

Yeah, I'll open a new ticket this week documenting the above mentioned error modality. Thanks again for you feedback on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants