Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dynamic-sidecar times out and raises occasionally error when calling stop_log_fetching #7288

Open
GitHK opened this issue Feb 27, 2025 · 2 comments · May be fixed by #7302
Open

dynamic-sidecar times out and raises occasionally error when calling stop_log_fetching #7288

GitHK opened this issue Feb 27, 2025 · 2 comments · May be fixed by #7302
Assignees
@GitHK
Labels
bug buggy, it does not work as expected

Comments

@GitHK
Copy link
Contributor

GitHK commented Feb 27, 2025 •
edited
Loading

dy-sidecar logs

"2025-02-25T18:18:17.563Z","ip-10-0-2-79","dy-sidecar_c36f3364-0706-5d99-ace4-198dafb4f4b6.1.mp3rwl323vzy9ndyzuy6jh634","log_level=WARNING | log_timestamp=2025-02-25 18:18:17,563 | log_source=servicelib.long_running_tasks._task:get_task_result_old(265) | log_uid=None | log_oec=None| log_msg=Task simcore_service_dynamic_sidecar.modules.long_running_tasks.task_runs_docker_compose_down.0029945b-6834-4f43-844b-08e399056b63 finished with error: Task simcore_service_dynamic_sidecar.modules.long_running_tasks.task_runs_docker_compose_down.0029945b-6834-4f43-844b-08e399056b63 finished with exception: ''
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/long_running_tasks/_task.py"", line 424, in _progress_task
    return await handler(progress, **task_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_sidecar/modules/long_running_tasks.py"", line 304, in task_runs_docker_compose_down
    await stop_log_fetching(app, container_name)

  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_sidecar/core/docker_logs.py"", line 115, in stop_log_fetching
    await background_log_fetcher.stop_log_fetching(container_name)

  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_sidecar/core/docker_logs.py"", line 89, in stop_log_fetching
    await task

  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_sidecar/core/docker_logs.py"", line 40, in _logs_fetcher_worker
    async for line in cast(

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/containers.py"", line 210, in _logs_stream
    async for item in multiplexed_result_stream(response, is_tty=is_tty):

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/multiplexed.py"", line 73, in multiplexed_result_stream
    async for item in _DecodeHelper(log_stream, encoding=encoding):

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/utils.py"", line 163, in __anext__
    stream = await self._gen.__anext__()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/multiplexed.py"", line 26, in __anext__
    response = await self._gen()
               ^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/multiplexed.py"", line 39, in fetch
    header = yield from self._response.content.readexactly(hdrlen)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/streams.py"", line 481, in readexactly
    block = await self.read(n)
            ^^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/streams.py"", line 425, in read
    await self._wait(""read"")

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/streams.py"", line 343, in _wait
    with self._timer:

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/helpers.py"", line 671, in __exit__
    raise asyncio.TimeoutError from exc_val",
"2025-02-25T18:18:17.563Z","ip-10-0-2-79","dy-sidecar_c36f3364-0706-5d99-ace4-198dafb4f4b6.1.mp3rwl323vzy9ndyzuy6jh634","log_level=WARNING | log_timestamp=2025-02-25 18:18:17,563 | log_source=servicelib.long_running_tasks._task:_cancel_asyncio_task(296) | log_uid=None | log_oec=None| log_msg=Timed out while awaiting for cancellation of 'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_runs_docker_compose_down.0029945b-6834-4f43-844b-08e399056b63'",

director-v2 logs

"2025-02-25T18:18:17.565Z","s4lManager3","production-simcore_production_director-v2.1.so2trafo3l72ix16v6tb7qajt","log_level=INFO | log_timestamp=2025-02-25 18:18:17,565 | log_source=simcore_service_director_v2.modules.dynamic_sidecar.scheduler._core._events_utils:service_remove_containers(138) | log_uid=None | log_oec=None| log_msg=Could not remove service containers for dy-sidecar_c36f3364-0706-5d99-ace4-198dafb4f4b6. Will continue to save the data from the service! Error: <class 'servicelib.long_running_tasks._errors.TaskClientResultError'>: Task simcore_service_dynamic_sidecar.modules.long_running_tasks.task_runs_docker_compose_down.0029945b-6834-4f43-844b-08e399056b63 finished with exception: ''
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/long_running_tasks/_task.py"", line 424, in _progress_task
    return await handler(progress, **task_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_sidecar/modules/long_running_tasks.py"", line 304, in task_runs_docker_compose_down
    await stop_log_fetching(app, container_name)

  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_sidecar/core/docker_logs.py"", line 115, in stop_log_fetching
    await background_log_fetcher.stop_log_fetching(container_name)

  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_sidecar/core/docker_logs.py"", line 89, in stop_log_fetching
    await task

  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_sidecar/core/docker_logs.py"", line 40, in _logs_fetcher_worker
    async for line in cast(

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/containers.py"", line 210, in _logs_stream
    async for item in multiplexed_result_stream(response, is_tty=is_tty):

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/multiplexed.py"", line 73, in multiplexed_result_stream
    async for item in _DecodeHelper(log_stream, encoding=encoding):

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/utils.py"", line 163, in __anext__
    stream = await self._gen.__anext__()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/multiplexed.py"", line 26, in __anext__
    response = await self._gen()
               ^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiodocker/multiplexed.py"", line 39, in fetch
    header = yield from self._response.content.readexactly(hdrlen)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/streams.py"", line 481, in readexactly
    block = await self.read(n)
            ^^^^^^^^^^^^^^^^^^

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/streams.py"", line 425, in read
    await self._wait(""read"")

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/streams.py"", line 343, in _wait
    with self._timer:

  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/helpers.py"", line 671, in __exit__
    raise asyncio.TimeoutError from exc_val",
@GitHK GitHK self-assigned this Feb 27, 2025
@GitHK GitHK added the bug buggy, it does not work as expected label Feb 27, 2025
@matusdrobuliak66
Copy link
Collaborator

@GitHK can you please explain a bit more what is the issue about? Thanks!

@GitHK
Copy link
Contributor Author

GitHK commented Mar 4, 2025

@matusdrobuliak66 I'm unsure why, but sometimes get a timeout when stopping the log fetcher in the dynamic-sidecar. It's OK to ignore it. I will put out a fix for it.

@GitHK GitHK linked a pull request Mar 4, 2025 that will close this issue
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@GitHK GitHK

Labels
bug buggy, it does not work as expected
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

🐛 Fixes raises when stopping log fetching in dynamic-sidecar GitHK/osparc-simcore-forked
2 participants
@GitHK @matusdrobuliak66