Skip to content

fix(task): enforce group concurrency at dequeue time#105

Merged
moshloop merged 5 commits into
mainfrom
fix/task-group-concurrency-dequeue
May 31, 2026
Merged

fix(task): enforce group concurrency at dequeue time#105
moshloop merged 5 commits into
mainfrom
fix/task-group-concurrency-dequeue

Conversation

@moshloop

@moshloop moshloop commented May 31, 2026

Copy link
Copy Markdown
Member

What

  • Move semaphore acquisition from task function wrapper to worker dequeue time
  • Set parent group via withParent option before task enqueue
  • Add PrettyShort interface for compact cell rendering
  • Add ClickyDocument and ClickyText for direct Textable serialization
  • Export clicky* types to support external consumption
  • Add JSON field to Link struct with structured serialization

Why

  • Prevents worker slots from being blocked on group semaphores
  • Allows independent tasks to run while concurrency-limited group is busy
  • Fixes deadlock risk when concurrency=1 and prevents worker starvation
  • Preserves rich formatting (code blocks, collapsibles, tables) without flattening
  • Enables structured-JSON consumers to render links as anchors with metadata

Notes

  • Adds TestGroupConcurrencySlots to verify independent task concurrency
  • Comprehensive tests for Link.MarshalJSON, PrettyShort precedence, and JSON round-trip fidelity
  • Updates TryTypedValue to check Pretty before Textable for proper method precedence

Summary by CodeRabbit

  • New Features

    • Added admonition callouts with severity levels (note, tip, warning, danger) supporting formatted headers and bodies.
    • Compact field rendering support for tables via PrettyShort.
    • Optional positional ID support for action commands.
    • Structured Clicky JSON document output preserving rich formatting.
    • OpenAPI parameter role detection for enhanced UI rendering.
  • Bug Fixes

    • Improved error rendering to preserve rich formatting through the standard output pipeline.
    • Fixed Markdown table pipe character escaping to prevent layout issues.
    • Enhanced task group concurrency handling.
  • Documentation

    • Substantially rewrote README with usage examples, installation instructions, and API patterns.

@coderabbitai

coderabbitai Bot commented May 31, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@moshloop, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 46 minutes and 40 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 431739ca-797a-420f-9bb5-665c1fd16b3f

📥 Commits

Reviewing files that changed from the base of the PR and between b21bc8a and 5c5c408.

📒 Files selected for processing (32)
  • api/admonition.go
  • api/admonition_test.go
  • api/column.go
  • api/keyed.go
  • api/keyed_test.go
  • api/link.go
  • api/link_test.go
  • api/meta.go
  • api/meta_test.go
  • api/parse_tags.go
  • api/parser.go
  • api/table.go
  • api/table_test.go
  • api/types.go
  • cobra_command.go
  • entity.go
  • flags/request.go
  • flags/request_test.go
  • format.go
  • formatters/clicky_document.go
  • formatters/clicky_document_test.go
  • formatters/html_react_formatter.go
  • formatters/html_react_formatter_test.go
  • rpc/openapi.go
  • rpc/openapi_test.go
  • sub_command.go
  • sub_command_test.go
  • task/group.go
  • task/group_concurrency_test.go
  • task/options.go
  • task/task.go
  • task/worker.go

Walkthrough

This PR introduces substantial enhancements to clicky's rendering infrastructure, task execution model, and RPC command handling. New callout, keyed wrapper, and document types enrich the rendering system. A PrettyShort interface enables compact field rendering, while improved error handling formats rich errors through the standard rendering pipeline. Task concurrency is refactored to defer semaphore acquisition to worker dequeue time, preventing slot starvation. A new request-based flag population function enables clean RPC HTTP execution without mutating Cobra state. Entity actions gain optional ID support, filter definitions can be lifted from nested options, and subcommands support slash-delimited paths. OpenAPI generation now includes UI routing metadata for parameter roles, and the Swagger server becomes more composable with configurable path prefixes and exported handlers.

Changes

Rendering, Concurrency, and RPC Infrastructure

Layer / File(s) Summary
Admonition callout type and renderers
api/admonition.go, api/admonition_test.go
Introduces Severity enum with icon mappings and Admonition struct with String, ANSI, HTML, Markdown renderers for styled callout blocks with optional title and body.
Keyed wrapper and Link JSON serialization
api/keyed.go, api/keyed_test.go, api/link.go, api/link_test.go, format.go
Adds Keyed type wrapping Textable with a stable key for JSON object field serialization; extends Link with optional JSON payload and MarshalJSON for structured output; exports WithKey helper.
Clicky document structuring and type export
formatters/clicky_document.go, formatters/clicky_document_test.go, formatters/html_react_formatter.go, formatters/html_react_formatter_test.go
Introduces NewClickyDocument and ClickyText wrapper for versioned JSON document generation; exports previously unexported Clicky* types (ClickyDocument, ClickyNode, ClickyField, ClickyColumn, ClickyRow, ClickyTreeItem, ClickyStackFrame, ClickyStyle) for public API access.
PrettyShort interface and short-form rendering
api/types.go, api/meta.go, api/parse_tags.go, api/parser.go, api/column.go, api/meta_test.go, entity.go
Adds PrettyShort interface for compact single-line rendering; extends FieldMeta.Short and PrettyField.Short flags; updates tag parser to recognize short; refines TryTypedValue to prioritize Pretty over Textable; integrates short rendering into schema parsing and pretty-row rendering.
Markdown table pipe escaping
api/table.go, api/table_test.go
Escapes literal pipe characters in Markdown table cells with escapeMarkdownPipes helper to prevent unintended GFM table layout disruption.
Renderable error detection and formatting
cobra_command.go, cobra_command_test.go
Adds renderableError helper that walks the Unwrap chain to detect errors implementing clicky rendering interfaces (api.TryTypedValue); formats matching errors through PrintAndWriteSinks instead of plain logging.
Worker-level task concurrency gating
task/group.go, task/worker.go, task/options.go, task/task.go, task/group_concurrency_test.go, task/task_status_test.go
Refactors task concurrency enforcement from in-method semaphore wrapping to dequeue-time TryAcquire(1) in the worker loop; adds withParent option to link tasks before enqueueing; defers permit release to ensure cleanup on all exit paths; prevents worker-slot starvation.
Request-based flag population for RPC
flags/request.go, flags/request_test.go
Implements PopulateFromRequest to parse HTTP request flag maps and args into struct fields via reflection, supporting scalars, slices (with CSV expansion and file/URL loading), duration.Duration, and time.Time without mutating shared Cobra/pflag state; includes comprehensive concurrency and state-leak tests.
Filter lifting and optional ID support for actions
entity.go, entity_filters_test.go
Introduces Filterable[T] interface and LiftFilters function to project nested filter definitions; adds OptionalID field to ActionInfo with WithOptionalID() builder; updates generateIDCommand to support optional [id] syntax and skip missing-id validation when enabled.
Cobra command wiring for filters and RPC execution
cobra_command.go
Updates AddNamedCommand to detect Filterable[T] opts, derive subcommand filters, register lookup functions, and wire shell completions; refactors dataFuncRegistry RPC handler to capture fieldInfos and use PopulateFromRequest instead of mutating Cobra flags and accessing pre-bound flagValues.
Slash-delimited subcommand path resolution
sub_command.go, sub_command_test.go
Extends RegisterSubCommand to support slash-delimited parent paths (e.g., xero/accounts, billing/policy), auto-creating intermediate grouping nodes and enabling hierarchical subcommand organization.
OpenAPI parameter metadata and UI roles
rpc/openapi.go, rpc/openapi_test.go
Adds ClickyParameterMeta with role field to OpenAPI parameters; introduces paramRole classifier assigning pagination (limit, offset), time-range (time-from, time-to), and filter roles to query parameters for UI-driven widget selection; adds GenerateFromCobraWithConfig for configurable spec generation.
RPC converter and server configuration updates
rpc/converter.go, rpc/serve.go
Updates converter to suppress duplicate entity list commands when parent is runnable; configures SwaggerServer to respect executor PathPrefix during converter initialization and OpenAPI generation; adds exported handler wrappers (HandleOpenAPIJSON, HandleOpenAPIYAML, HandleEntities, HandleHealth) and route registration helpers (ConverterConfig, RegisterExecutionRoutes).
README refactor and dependency updates
README.md, go.mod, examples/go.mod
Rewrites README with concise usage-oriented sections covering library formatting/tasks, CLI binding, OpenAPI/extensions, middleware, and schema formatting; updates Go toolchain to 1.26.1 and advances direct and indirect dependencies for Flanksource libraries, OpenTelemetry, Kubernetes, and ecosystem tools.

Possibly related PRs

  • flanksource/clicky#102: Adds renderableError helper walking Unwrap chain with api.TryTypedValue to format rich errors through the --format pipeline, matching the error handling enhancement in this PR.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/task-group-concurrency-dequeue
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch fix/task-group-concurrency-dequeue

moshloop added 5 commits May 31, 2026 13:35
…and UI parameter roles

Introduce three major capabilities:

1. Filter lifting (entity.go): Add Filterable interface and LiftFilters function to allow subcommand options structs to expose typed filter lookups that diverge from parent entity filters. This enables fine-grained filter control when positional arguments pin certain identifier filters.

2. HTTP request population (flags/request.go): Add PopulateFromRequest function to populate struct fields from HTTP request data without shared pflag state. This fixes state leakage across concurrent requests where slice values would accumulate across calls. Includes support for CSV expansion, file loading, and type parsing (int, bool, duration, time).

3. OpenAPI parameter roles (rpc/openapi.go): Add ClickyParameterMeta and paramRole function to classify query parameters for UI routing (pagination, time-range, filters) instead of generic text inputs. Roles are assigned based on parameter name conventions and operation capabilities.

4. Hierarchical subcommand paths (sub_command.go): Extend RegisterSubCommand and RegisterSubCommandFn to accept slash-delimited paths (e.g., "billing/policy") for nested command registration, creating missing intermediates and reusing existing nodes.

Includes comprehensive test coverage for concurrent request isolation, CSV parsing, default values, file loading, parameter role classification, and path-aware subcommand registration.
…ering

Introduce Admonition type to support callout blocks with severity levels (note, info, tip, warning, danger) and multi-format rendering (text, ANSI, HTML, Markdown). Add Keyed wrapper type to associate stable identifiers with Textable values for structured JSON output without affecting other formats.

Also fix Markdown table rendering to escape pipe characters in cell content, preventing GFM table layout corruption. Add shell completion support for Filterable subcommands and refactor RPC data function to use PopulateFromRequest for proper concurrent request handling.

Add WithKey helper function to format.go for convenient Keyed wrapper creation.
…body

Move semaphore acquisition from task function wrapper to worker dequeue time. This prevents worker slots from being blocked on group semaphores, allowing independent tasks to run while a concurrency-limited group is busy.

The parent group is now set via withParent option before task enqueue, ensuring the worker can access the group's semaphore at dequeue time. This fixes a deadlock risk when concurrency=1 and prevents worker starvation.

Adds TestGroupConcurrencySlots to verify independent tasks can run concurrently with a saturated serial group.
…le serialization

Introduce NewClickyDocument() to build clicky documents directly from Textable objects, bypassing PrettyData conversion. This preserves rich formatting (code blocks, collapsibles, tables) verbatim without flattening through PrettyData.

Add ClickyText wrapper type that implements json.Marshaler to emit structured clicky documents instead of plain strings, enabling use in struct fields with `json:"field"` tags.

Export previously private clicky* types (ClickyDocument, ClickyNode, ClickyStyle, ClickyField, ClickyColumn, ClickyRow, ClickyTreeItem, ClickyStackFrame) to support external consumption and round-tripping through encoding/json.

Add comprehensive tests verifying code blocks, collapsed sections, and JSON round-trip fidelity.
…ink JSON payload support

Introduce PrettyShort interface to enable objects to provide compact, single-line representations for table cells (typically self-links), distinct from Pretty()'s fuller detail view. Cell renderers now prefer PrettyShort() over Pretty() when available.

Add JSON field to Link struct with WithJSON() builder method and MarshalJSON() implementation to serialize links as structured objects with href, text, tooltip, and optional payload. This allows structured-JSON consumers to render links as anchors while preserving navigation metadata.

Add `short` pretty-tag flag to FieldMeta to control field rendering via PrettyShort() in structs, tables, and clicky+json printers.

Update TryTypedValue to check Pretty before Textable, ensuring types implementing both (e.g. EntityLink) honor their Pretty() method rather than serializing as bare structs.

Includes comprehensive tests for Link.MarshalJSON, PrettyShort precedence, and Pretty/Textable interaction.
@moshloop moshloop force-pushed the fix/task-group-concurrency-dequeue branch from b21bc8a to 5c5c408 Compare May 31, 2026 10:36
@moshloop moshloop merged commit 7702184 into main May 31, 2026
13 of 16 checks passed
@moshloop moshloop deleted the fix/task-group-concurrency-dequeue branch May 31, 2026 10:41
@flankbot

Copy link
Copy Markdown

🎉 This PR is included in version 1.21.11

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants