chore(observability): adds OpenTelemetry metrics for wallet balance reload check job

[ygrishajev]

Introduce instrumentation service to track job health and user issues:

• Job execution metrics (duration, success/failure)
• Reload actions (triggered, skipped, failures)
• Error tracking (validation, scheduling)
• Cost and amount distributions (histograms)

Metrics are Prometheus-compatible for dashboard visualization and alerting.

refs #1779

Summary by CodeRabbit

• New Features

  • Enhanced monitoring and observability for wallet balance reload operations, including tracking of reload triggers, skips with reasons, and failures.
  • Added performance metrics for job execution duration, reload amounts, and balance coverage.
  • Improved error tracking for validation and scheduling issues.

• Tests

  • Updated test coverage for instrumentation-based wallet balance monitoring.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

This pull request introduces a new OpenTelemetry-based instrumentation service for the wallet balance reload check flow. It establishes a centralized MetricsService for metric creation, then uses it to define counters, histograms, and logging for tracking job execution, reload events, validation errors, and scheduling failures. The instrumentation service is integrated into the WalletBalanceReloadCheckHandler, replacing direct logging calls.

Changes

Cohort / File(s)	Summary
Metrics Infrastructure apps/api/src/core/services/metrics/metrics.service.ts, apps/api/src/core/services/index.ts	Introduces new MetricsService singleton with factory methods for creating OpenTelemetry counters, histograms, and up-down counters. Exposes getMeter() for component-level meter retrieval and re-exports the service from core module index.
Instrumentation Service apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check-instrumentation.service.ts	Introduces new WalletBalanceReloadCheckInstrumentationService that creates and manages metrics (job executions, reloads triggered/skipped, failures, validation/scheduling errors) and provides methods to record these events with contextual data.
Handler & Tests apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.ts, apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.spec.ts	Updates WalletBalanceReloadCheckHandler to accept WalletBalanceReloadCheckInstrumentationService instead of LoggerService. Replaces direct logging with instrumentation calls throughout the handler lifecycle. Updates test suite with instrumentation mocks and assertions.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

• Areas requiring extra attention:
  • Metric naming consistency and OpenTelemetry conventions in MetricsService and WalletBalanceReloadCheckInstrumentationService
  • Conditional metric recording logic (balance-to-cost ratio, projected cost) in instrumentation methods
  • Test mock setup and assertion accuracy for all instrumentation calls in handler spec
  • Error type classification and propagation in recordValidationError and recordSchedulingError

Suggested reviewers

• stalniy
• baktun14

Poem

🐰 Metrics hop and counters leap,
Telemetry's promises we keep,
Wallets balanced, durations tracked,
Every reload, every act!
With histograms and tales so bright,
Observability shines its light! ✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: adding OpenTelemetry metrics for wallet balance reload check job instrumentation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feature/jobs

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (3)

apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.spec.ts (2)

358-365: Consider adding coverage for recordReloadFailed on payment intent errors

You already mock recordReloadFailed but no test exercises the branch where stripeService.createPaymentIntent rejects; adding a test that forces this failure and asserts the corresponding instrumentation call would close the last major instrumentation gap for the reload flow.

---

10-13: Optional: Align JobPayload import with JobMeta for consistency

JobMeta is imported from @src/core while JobPayload comes from a relative "../../../core" path; switching JobPayload to also use @src/core would make the imports more consistent and easier to maintain.

apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.ts (1)

60-78: Optional: Include job id in job execution instrumentation for easier correlation

If you find yourself debugging individual jobs, consider extending recordJobExecution (and its metric/log payload) to also receive job.id in addition to userId, so you can more easily correlate metrics/logs back to the underlying queue job.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1f55f3f and a84058a.

📒 Files selected for processing (5)

• apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check-instrumentation.service.ts (1 hunks)
• apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.spec.ts (12 hunks)
• apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.ts (5 hunks)
• apps/api/src/core/services/index.ts (1 hunks)
• apps/api/src/core/services/metrics/metrics.service.ts (1 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

**/*.{ts,tsx,js}

📄 CodeRabbit inference engine (.cursor/rules/general.mdc)

**/*.{ts,tsx,js}: Never use type any or cast to type any. Always define the proper TypeScript types.
Never use deprecated methods from libraries.
Don't add unnecessary comments to the code.

Files:

• apps/api/src/core/services/index.ts
• apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.spec.ts
• apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.ts
• apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check-instrumentation.service.ts
• apps/api/src/core/services/metrics/metrics.service.ts

**/*.spec.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/no-jest-mock.mdc)

Don't use jest.mock() in test files. Instead, use jest-mock-extended to create mocks and pass mocks as dependencies to the service under test

Use setup function instead of beforeEach in test files. The setup function must be at the bottom of the root describe block, should create an object under test and return it, accept a single parameter with inline type definition, avoid shared state, and not have a specified return type.

**/*.spec.{ts,tsx}: Use <Subject>.name in the root describe suite description instead of hardcoded class/service name strings to enable automated refactoring tools to find all references
Use either a method name or a condition starting with 'when' for nested suite descriptions in tests
Use present simple, 3rd person singular for test descriptions without prepending 'should'

Files:

• apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.spec.ts

🧬 Code graph analysis (3)

apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.spec.ts (2)

packages/logging/src/services/logger/logger.service.ts (1)

• error (141-143)

apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.ts (1)

• job (81-99)

apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.ts (3)

apps/api/src/core/services/job-queue/job-queue.service.ts (2)

• JobPayload (241-241)
• JobMeta (250-250)

apps/api/src/billing/events/wallet-balance-reload-check.ts (1)

• WalletBalanceReloadCheck (5-15)

packages/logging/src/services/logger/logger.service.ts (1)

• error (141-143)

apps/api/src/core/services/metrics/metrics.service.ts (1)

apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check-instrumentation.service.ts (1)

• singleton (7-154)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: validate / validate-app
• GitHub Check: test-build
• GitHub Check: Analyze (javascript-typescript)

🔇 Additional comments (8)

apps/api/src/core/services/index.ts (1)

3-3: Re-exporting metrics.service via core services index looks correct

This cleanly exposes MetricsService and related types through the core barrel without altering existing exports.

apps/api/src/core/services/metrics/metrics.service.ts (1)

4-61: MetricsService abstraction is simple, typed, and cohesive

MetricOptions and the three factory methods provide a clear, minimal facade over the meter API, and getMeter keeps call sites consistent; no issues spotted.

apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.spec.ts (1)

34-85: Instrumentation expectations in handler tests are aligned and realistic

The updated tests correctly verify calls to recordReloadTriggered, recordReloadSkipped, recordSchedulingError, and recordValidationError with arguments that mirror the handler’s logic (including walletAddress, balance, and costUntilTargetDateInFiat), giving good coverage of the new observability behavior.

Also applies to: 121-171, 197-213, 215-307

apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check.handler.ts (3)

60-78: Job execution instrumentation in handle is robust

Wrapping the core logic in try/finally and updating a success flag ensures recordJobExecution always fires, distinguishing successful runs from both validation failures and thrown errors, which is exactly what you want for job health metrics.

---

167-205: Reload decision logic and instrumentation are consistent and side‑effect safe

The zero‑cost and sufficient‑balance branches (Lines 179–187) now exit early after calling recordReloadSkipped, and the reload amount is computed only when needed using Math.max(...) with the configured minimum; success and failure of the payment intent are cleanly distinguished via recordReloadTriggered and recordReloadFailed, preserving behavior while giving richer telemetry.

---

207-217: Scheduling error handling is correctly instrumented while preserving failure semantics

#scheduleNextCheck now records scheduling failures via recordSchedulingError before rethrowing, so metrics/logs capture the issue without changing the fact that the handler still fails the job on scheduling errors.

apps/api/src/billing/services/wallet-balance-reload-check/wallet-balance-reload-check-instrumentation.service.ts (2)

7-68: Metric setup in the instrumentation service is coherent and well‑scoped

Initializing a dedicated meter plus clearly named counters/histograms for executions, reload behavior, validation, and scheduling gives a good, self‑contained observability surface for this job, and keeps metric concerns out of the handler.

---

70-153: Record methods correctly capture key dimensions and avoid edge‑case pitfalls

The record methods apply sensible attributes (status, reason, error_type), reuse shared histograms (reloadAmounts, balanceCoverageRatio, projectedCost), and defensively skip balance‑coverage calculations when cost is zero, while emitting structured logs with the provided context—this all looks solid.

[codecov]

Codecov Report

❌ Patch coverage is 35.29412% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.52%. Comparing base (1cf3a36) to head (90a0d67).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...et-balance-reload-check-instrumentation.service.ts	8.16%	45 Missing ⚠️
...s/api/src/core/services/metrics/metrics.service.ts	27.27%	8 Missing ⚠️
...eload-check/wallet-balance-reload-check.handler.ts	92.00%	2 Missing ⚠️

❌ Your patch status has failed because the patch coverage (35.29%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2323      +/-   ##
==========================================
- Coverage   50.86%   50.52%   -0.35%     
==========================================
  Files        1050     1037      -13     
  Lines       28727    28195     -532     
  Branches     6349     6255      -94     
==========================================
- Hits        14612    14245     -367     
+ Misses      13854    13698     -156     
+ Partials      261      252       -9     

Flag	Coverage Δ		*Carryforward flag
api	80.80% <35.29%> (-0.48%)	⬇️
deploy-web	30.06% <ø> (-0.24%)	⬇️	Carriedforward from 349effc
log-collector	?
notifications	87.94% <ø> (ø)		Carriedforward from 349effc
provider-console	81.48% <ø> (ø)		Carriedforward from 349effc
provider-proxy	84.35% <ø> (ø)		Carriedforward from 349effc

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines	Coverage Δ
...eload-check/wallet-balance-reload-check.handler.ts	95.45% <92.00%> (-2.05%)	⬇️
...s/api/src/core/services/metrics/metrics.service.ts	27.27% <27.27%> (ø)
...et-balance-reload-check-instrumentation.service.ts	8.16% <8.16%> (ø)

... and 34 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.