Add structured logs for SplitRunner and rest of CLI

[erbesharat]

Detailed Description

Currently, the SplitRunner worker logs are minimal and not very informative, making it difficult to diagnose issues when tasks fail or behave unexpectedly. We propose introducing enhanced, structured logging capabilities within the SplitRunner worker to provide richer context around execution steps, timing, and error conditions. This will involve adding configurable log levels (e.g., debug, info, warn, error) and potentially leveraging a structured logging framework (such as pino or winston) for consistent, JSON-formatted logs. The improved logs should capture key data points like the worker's lifecycle events, the splitting strategies used, the status of tasks in the queue, performance metrics, and detailed error stacks.

Context

By improving the logging, developers and operators can more easily pinpoint the root causes of failures, optimize task processing, and ensure that the SplitRunner worker behaves reliably under load. For example, if a particular task type repeatedly fails at the same processing step, the enhanced logs will help identify patterns, allowing for quicker debugging and faster fixes. This change can benefit all users who rely on SplitRunner for distributed or parallelized processing, particularly in production environments where visibility into complex worker behaviors is essential.

Possible Implementation

1. Integrate a robust logging library (e.g., winston) into the SplitRunner module.
2. Introduce structured logging with JSON output, enabling easier log parsing by third-party tools.
3. Add configurable log levels through napi configuration files, allowing users to set the verbosity of logs (e.g., debug vs. info) without code changes.
4. Include contextual metadata in logs, such as task identifiers, timestamps, worker IDs, and environment details.
5. Write logs at critical execution points: task start/end, error handling, worker startup/shutdown, and performance checkpoints.