[BUG]: Try to append batches web console lines for record #4922
Comments
|
Hi @ovaar, |
|
Hi @ovaar! I tried to reproduce this issue using a Python script that executes a long-running task, but did not have success both with the agent version v3.236.1 and the latest one. Could you please provide more details on what your build.py script looks like? Also, it would be great if you could provide the Worker file's log for a pipeline run with |
|
I am closing this issue for now. @ovaar if this issue is still relevant feel free to re-open |
|
Thanks @aleksandrlevochkin for investigating. Since I have a workaround the issue, for me, is not that pressing. #!/usr/bin/env python3
import argparse
from contextlib import nullcontext
from enum import Enum
import platform
import shlex
import subprocess
from pathlib import Path
from typing import Final, List, Optional
try:
from enum import StrEnum
except:
raise Exception("Install python >= 3.11")
BUILD_PATH: Final[Path] = Path(__file__).parent / "Build"
WIN_PROJECT_FILE: Final[Path] = BUILD_PATH / "app.sln"
class BuildType(StrEnum):
Debug = "Debug"
Release = "Release"
def to_build_type(arg: str) -> BuildType:
try:
return BuildType[arg]
except:
raise argparse.ArgumentTypeError("Invalid argument build_type!")
class LoggingMode(Enum):
NO_LOGGING = 0
LOG_TO_TERMINAL = 1
LOG_TO_FILE = 2
def run_command(
command: str,
logging_mode: LoggingMode = LoggingMode.LOG_TO_TERMINAL,
throw_on_error=True,
logfile: Optional[str] = None,
encoding: str = "utf-8",
):
"""Invoke a command on a subprocess and redirect the outputs"""
if logfile:
log_dir: Path = Path(__file__).parent
if not log_dir.exists():
log_dir.mkdir(parents=True, exist_ok=True)
print()
print(f"$ {command}")
stdout_lines: List[str] = []
rc: int = 0
try:
process = subprocess.Popen(
shlex.split(command),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
encoding=encoding,
)
with Path(logfile).open("a") if logfile else nullcontext() as f:
while True:
output = process.stdout.readline()
if output == "" and process.poll() is not None:
break
if output:
stdout = output.strip()
if logging_mode is LoggingMode.LOG_TO_FILE:
f.write(stdout + "\n")
if logging_mode is LoggingMode.LOG_TO_TERMINAL:
print(stdout)
stdout_lines.append(stdout)
rc = process.poll()
if rc and throw_on_error:
raise subprocess.CalledProcessError(rc, cmd=command)
except Exception as e:
if throw_on_error:
raise e
rc = 1
print()
return stdout_lines, rc
def main(rebuild: bool, build_type: BuildType):
if platform.system() == "Windows":
from python_modules.pyvswhere import get_msbuild_path
"""Build using MSBuild"""
msbuild_path = get_msbuild_path(
version="17", product="Microsoft.VisualStudio.Product.BuildTools"
)
cmd: str = (
f'"{msbuild_path}" "{WIN_PROJECT_FILE}" /property:Configuration={str(build_type)} /property:Platform=x64'
)
if rebuild:
cmd += " -t:rebuild"
output, rc = run_command(cmd)
parser = argparse.ArgumentParser()
parser.add_argument(
"-b",
"--build_type",
type=to_build_type,
help=f'Set the build type "{"|".join(list(BuildType._member_names_))}"',
required=True,
)
parser.add_argument(
"--rebuild",
dest="rebuild",
action="store_true",
help="When set a rebuild is started.",
)
args = parser.parse_args()
build_type: BuildType = args.build_type
main(args.rebuild, build_type) |
What happened?
While executing long-running python tasks tasks using PythonScript@0 or just bash, which invokes python, the agent seems to lose connection to the host, maybe because it didn't receive enough messages in the stdout/stderr to feed the watchdog, resulting in a broken connection that makes the build freeze?
Running the (long running) build scripts as shown above, results in Python using buffered streams for stdout and stderr.
Setting the PYTHONUNBUFFERED resolved the issue.
Versions
Azure Devops version 3.236.1 / Windows 10
Environment type (Please select at least one enviroment where you face this issue)
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
Windows 10
Version controll system
No response
Relevant log output
*** Edited ***
Set the PYTHONUNBUFFERED as a workaround.
The text was updated successfully, but these errors were encountered: