Skip to content

Expand IDataQueueHandler for DataSource consumers #9272

@Romazes

Description

@Romazes

Problem

IDataQueueHandler is currently too rigid for long-running DataSource consumers. When a breaking change is needed, we risk interface churn and downstream breakage.

Additionally, unhandled exceptions inside LongRunning tasks are swallowed by the task infrastructure and do not propagate to the main thread, which leaves the DataSource running in an undefined or stalled state.

At the moment, there is no reliable mechanism for a DataSource to notify Lean that a fatal error has occurred and that it should be terminated.

Expected Behavior

  • A DataSource should be able to:
    • Surface fatal exceptions to Lean
    • Terminate cleanly when an unrecoverable error occurs
  • Lean should receive a clear error signal (e.g. via BrokerageMessageEvent) when a DataSource fails
  • Future extensions to IDataQueueHandler should not require breaking interface changes

Actual Behavior

  • Exceptions thrown inside LongRunning tasks are swallowed
  • The DataSource continues running (or appears to) without notifying Lean
  • No XXXMessageEvent is emitted to indicate failure
  • Consumers cannot safely extend behavior without modifying the core interface
// Exception thrown here is swallowed
_dataReceiverTask = new Task(
    () => MonitorDataReceiverConnection(token),
    token,
    TaskCreationOptions.LongRunning);

_dataReceiverTask.Start();

Potential Solution

  1. Introduce an extensible abstraction
    1. Add a base class (e.g. DataQueueHandlerBase) that implements IDataQueueHandler
    2. Allow new virtual hooks without breaking consumers
  2. Create an extended interface
    1. e.g. IExtendedDataQueueHandler : IDataQueueHandler
    2. Optional adoption by advanced DataSource implementations
  3. Add a fatal error signaling mechanism
    1. Allow IDataQueueHandler / base class to raise a Lean event on failure:
OnMessage(new BrokerageMessageEvent(
    BrokerageMessageType.Error,
    "DataSource",
    "Fatal error occurred. DataSource is terminating."));
  1. Lean should react by stopping the DataSource deterministically
  2. Centralize exception handling
    1. Catch exceptions inside LongRunning tasks
    2. Route them through the error event instead of letting them disappear

Benefits

  • Prevents silent failures in live data sources
  • Improves observability and debuggability
  • Enables safer evolution of IDataQueueHandler
  • Reduces breaking changes for downstream consumers

Checklist

  • I have completely filled out this template
  • I have confirmed that this issue exists on the current master branch
  • I have confirmed that this is not a duplicate issue by searching issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions