Skip to content

Logging & troubleshooting

Jump to bottom
Yury V. Malovichko edited this page Oct 27, 2025 · 2 revisions

This is a concise guide on using TOGA2 logging system for debugging crashes and inspecting produced results. For more information on specific TOGA2 pipeline steps, output files, and potential errors, see the relevant pages on this Wiki.

Logging

TOGA2 logs are formatted with Python logging package. Generally three logging levels are used, depending on urgency of the reported message:

  • INFO: general runtime logging channel. This includes announcement of each pipeline step and its successful execution, passed runtime sanity checks (see below), notification on parallel steps’ progress, etc.
  • WARNING level: scenarios that might require manual inspection but do not affect the TOGA2 performance. Reasons triggering WARNING messages include edge cases in input data or interim results (e.g., using paralogous projections instead of orthologs for transcript annotation) and failed runtime sanity checks.
  • CRITICAL: technical errors and illegitimate runtime scenarios. CRITICAL level messages are thrown by TOGA2 before error-exiting.

Most of the TOGA2 progress is logged to main log file logs/TOGA2_${project_name}, where ${project_name} is the name of the respective run (set with --project_name option, run start date and time by default). Parallel steps performed by Nextflow/Parasol get their own output logs stored under respective names in the nextflow/ directory. Finally, individual batches from parallel steps have their own execution logs.

Debugging crashed runs

Standard output log and TOGA2.log

Main TOGA2 log is saved at logs/TOGA2_${project_name}.log. If -v/--verbose flag is set, the same log is printed to standard output channel.

As stated above, execution errors are reported at CRITICAL logging level. Error messages contain a brief description of the error cause; if error occurred during subprocess execution, the original standard error output follows on the next line:

[YYYY-MM-DD H:M:S][toga_main.py] - CRITICAL: Genome compression check failed:
/bin/sh: 1: set: Illegal option -o pipefail

Note

Certain interruption/failure scenarios do not trigger controlled shutdown, thus leaving no CRITICAL level message in the main log. These include:

  • Illegitimate argument values (nonexistent file paths, etc.) - those are reported before log initialising
  • External interruption scenarios (power outage, keyboard interruption, etc.)
  • Unforeseen runtime scenarios (missing necessary provisional results)

While TOGA2 error handling is constantly improving, some of the listed triggers cannot be handled within the code by default. In this case, you still can get the idea of crash circumstances and potential reasons from the last logged messages.

Nextflow logs

Debugging runs crashed at either of the parallel steps requires additional log digging since progress for individual batches is not captured by the main log. Standard output logs for Nextflow/Parasol are stored at nextflow directory, each stored in a separate directory:

# ls nextflow/
cesar_preprocess_${project_name}
cesar_align_${project_name}_${bin1}
cesar_align_${project_name}_${bin2}
cesar_align_${project_name}_${bin3}
tree_resolution_${project_name}

, where ${project_name} is the name of the current TOGA2 project (see above), and ${binX} is alignment memory bucket. In its turn, each directory contains a step-specific log file:

# ls nextflow/cesar_preprocess_${project_name}
cesar_preprocess_${project_name}.log

Note

One of the frequent reasons for parallel step crashes is improper Nextflow/cluster job executor configuration. If a problem exists, you will most likely encounter it at the first parallel step, which by default is data preprocessing (nextflow/cesar_preprocess...). Consider inspecting this log for potential errors and modify Nextflow-related TOGA2 settings and/or cluster configuration accordingly.

failed_batches.tsv

Unless --ignore_crashed_parallel_batches flag is set, TOGA2 halts execution after the respective parallel step if any of the individual batches failed. TOGA2 identifies failed batches by checking for a completion stamp file (/.ok) at every individual batch output directory (see tmp directory content description). If any crashed batches were found, TOGA2 exits with the following error message:

[YYYY-MM-DD H:M:S][toga_main.py] - CRITICAL: Some of the monitored parallel processes at ${step} step died

, where ${step} is ‘preprocessing’, ‘alignment’, or ‘fine orthology resolution’, depending on the pipeline step the crash occurred at.

Notably, the message does not indicate the exact identifiers for batch identifiers. To look those up, consult the logs/failed_batches_${project_name}.tsv, where failed batch IDs are listed with respect to the parallel step separated with comma:

ALIGNMENT    batch1,batch42

Note

The default failed batch handling strategy described above suggests that a single failed_batches.tsv instance normally contains failed batches for a single parallel step: the one that TOGA2 crashed at. On the contrary, if --ignore_crashed_parallel_batches flag is set, the file will list failed batches for each step they were recorded at:

PREPROCESSING   batch1,batch2
ALIGNMENT    batch3,batch4
ORTHOLOGY   batch9,batch10

Batchwise execution logs

Each batchwise output directory contains a log file called log.txt. The logging format used here is the same as for the main log.

As in the case of the main log, batchwise logs might not report all the potential technical errors. However, they might still help with locating the exact gene/projection/chain at which the code crashed:

# tail tmp/cesar_preprocessing_res/batch0/log.txt
...
[YYYY-MM-DD H:M:S][shared.py] - INFO: Input chain is: 439807
[YYYY-MM-DD H:M:S][shared.py] - INFO: Single chain ID was provided; expecting all projections to be contained within the chain
[YYYY-MM-DD H:M:S][shared.py] - INFO: Input transcript is: ENST00000402739.9#CTNNA2
[YYYY-MM-DD H:M:S][shared.py] - INFO: Parsing reference annotation BED file
## log file ends here

While the exact error has not been reported here, you can still infer that execution crashed while parsing reference annotation for command preprocessing data for chain 439807 and reference transcript ENST00000402739.9#CTNNA2. Checking for such logging milestones helps debugging and reporting errors on GitHub (see below).

Inspecting TOGA2 results

Runtime sanity checks

summary.txt

rejected_items.tsv

Restarting TOGA2 runs

Setting e-mail notifications

Setting e-mail notifications is a convenient way of controlling multiple parallel TOGA2 runs without any manual progress checkup involved. To enable notifications, provide your e-mail address with `--email ${your_email_address} argument. General reasons for e-mail notifications include:

Note

E-mail notifications are currently available only for run mode

TOGA2 sends e-mail notifications via mailx utility. Due to mailx requiring proper mail server setup on the user’s side, its installation is left for user. Its availability in the run environment is still checked at the initial pipeline step. If you enable e-mail notification but mailx is missing from your $PATH, TOGA2 dies soon after launching with the following error message:

[YYYY-MM-DD H:M:S][toga_main.py] - CRITICAL: E-mail notification was requested by the user but executable "mailx" was not provided, with no defaults in $PATH

If your mailx server is improperly configured, TOGA2 will die with the respective technical error reported to the log upon its first attempt to send you an e-mail notification.

Reporting an error

TOGA2 strives to maintain lucid, user-friendly execution logging and documentation. If you encounter a technical error or any inconsistency in the produced results, we encourage you to inspect the output logs and go through the pages of this Wiki first. Most of the time, following the execution logs in top-down fashion (TOGA2 main log -> Nextflow logs -> batchwise logs) helps correctly identify the reason for the encountered problem.

If you further want to report your case, please consult with the following checklist:

  • Check standard output log and the log files for error messages; attach all logs you consider necessary to your issue.
  • If certain parallel step batches died, batch log can help trace the exact command. If you manage to do so, you might want to run the failed command manually with -v/--verbose flag added to provide additional context to the reported error.
  • If necessary, provide the affected TOGA2 output files and/or your input data. If these files contain unpublished or otherwise sensitive data or are too large for attachment, consider removing/obfuscating sensitive information, reducing input files to relevant minumum, or devising a mock example that emulates the encountered error.

Contents

Clone this wiki locally