Support natural language search in explore in CLI

[BYK]

Plan: sentry search — Natural Language Search via Seer AI

Summary

Add a sentry search command that translates natural language queries into structured Sentry queries using Seer AI, executes the query, and displays results. Also add --ask flags to existing list commands (issue list, log list, trace list) that delegate to the same NL translation.

User Experience

# Top-level NL search
sentry search "show me slow database queries from the last hour"
sentry search "error rate by endpoint in production" --strategy logs
sentry search "p95 latency for /api/users" --web
sentry search "which transactions have the most errors" --json
sentry ask "unresolved issues affecting 100+ users"   # alias

# NL on existing commands (future phase)
sentry issue list --ask "unresolved errors affecting users"
sentry log list --ask "error logs from payment service"

Architecture

User NL query → Search Agent API → Translated structured query
                                      ├─ Execute via Events API → Render table
                                      ├─ --web → Build Explore URL → Open browser
                                      └─ --json → Output structured results

Strategy → Dataset Mapping

Strategy	Dataset	Explore Tab	Default Fields
traces (default)	spans	/explore/traces/	span.op, span.description, span.duration, timestamp
logs	logs	/explore/logs/	timestamp, severity, message, trace
issues	N/A (issues endpoint)	/issues/	title, events, users, lastSeen
errors	spans	/explore/traces/	span.op, error type, timestamp

API Flow

1. Translate: POST /organizations/{org}/search-agent/translate/ (synchronous)

   • Input: { natural_language_query, project_ids[], strategy }
   • Output: { responses: [{ query, group_by, sort, stats_period, mode, visualization }] }
   • Fallback: If 403 → try async start/ + poll state/{run_id}/

2. Execute: GET /organizations/{org}/events/?dataset=X&field[]=...&query=...

   • Map translated query fields to events API parameters
   • Dynamic field discovery from meta.fields in response

3. Render: Dynamic table built from response metadata

   • Column types (string, number, date, duration) → automatic alignment + formatting
   • JSON mode: raw response data with --fields filtering

Files to Create

src/types/search-agent.ts — Zod schemas + types

// Schemas for:
// - SearchStrategy enum: "Traces" | "Logs" | "Issues" | "Errors"
// - NLQueryResponse: { query, group_by[], sort, stats_period, mode, visualization[] }
// - NLTranslateResult: { responses[], unsupported_reason? }
// - SearchAgentSession: { run_id, status, current_step, completed_steps, final_response }
// - ExploreResult: { data: Record<string, unknown>[], meta: { fields: Record<string, string> } }

src/lib/api/search-agent.ts — Search Agent API functions

// translateNLQuery(orgSlug, projectIds, query, strategy) → NLTranslateResult
//   POST /organizations/{org}/search-agent/translate/
//   Uses apiRequestToRegion (not in @sentry/api SDK)

// startSearchAgent(orgSlug, projectIds, query, strategy) → { run_id }
//   POST /organizations/{org}/search-agent/start/

// getSearchAgentState(orgSlug, runId) → SearchAgentSession
//   GET /organizations/{org}/search-agent/state/{runId}/

// queryExplore(orgSlug, params) → ExploreResult
//   GET /organizations/{org}/events/?dataset=X&field[]=...
//   Generic events query execution (any dataset)

src/lib/formatters/explore.ts — Dynamic table formatter

// formatExploreTable(result: ExploreResult) → string
//   Builds Column<> definitions dynamically from meta.fields
//   Uses existing formatTable/renderTextTable infrastructure
//   Type-aware alignment (numbers right, strings left)
//   Type-aware value formatting (dates, durations, percentages)

// formatTranslatedQuery(query: NLQueryResponse) → string
//   Human-readable display of the translated query parameters

src/commands/search/index.ts — Route map

// Route with 'search' as default command (single-command route)
// Subcommands can be added later if needed

src/commands/search/search.ts — Main command

// Positional: natural language query string (required)
// Flags:
//   --strategy / -s: "traces" | "logs" | "issues" | "errors" (default: "traces")
//   --web / -w: Open results in browser instead of displaying
//   --limit / -l: Max results (default: 25, max: 100)
//   --json: Machine-readable output
//   --fields: Filter JSON output fields
//   --translate-only: Show translated query without executing (for learning Sentry syntax)
//
// Flow:
// 1. Resolve org + project (DSN auto-detect, explicit org/, env vars)
// 2. Fetch project ID(s) via fetchProjectId()
// 3. Call translateNLQuery() wrapped in withProgress() spinner
// 4. If unsupported_reason → show error with suggestion
// 5. If --translate-only → show translated query + Explore URL
// 6. If --web → buildExploreUrl() + open browser
// 7. Otherwise → execute query via queryExplore() → render dynamic table
// 8. Show Explore URL as hint for all modes

src/lib/sentry-urls.ts — Add buildExploreUrl()

// buildExploreUrl(orgSlug, options: {
//   strategy: string,
//   query?: string,
//   groupBy?: string[],
//   sort?: string,
//   statsPeriod?: string,
//   mode?: string,
//   visualize?: object[],
// }) → string
//
// Maps strategy to path: traces→/explore/traces/, logs→/explore/logs/, issues→/issues/
// Encodes all query parameters matching the Explore page URL format

Files to Modify

src/app.ts

• Import searchRoute from ./commands/search/index.js
• Import searchCommand from ./commands/search/search.js
• Add to routes: search: searchRoute, ask: searchCommand
• Add to PLURAL_TO_SINGULAR: (none needed — ask is already a leaf command)

src/lib/api-client.ts (barrel)

• Re-export from ./api/search-agent.js

src/types/index.ts

• Re-export search-agent types

NL Query → Events API Translation Logic

The search-agent returns:

{
  "query": "span.op:db span.duration:>1s",
  "group_by": ["transaction"],
  "sort": "-count()",
  "stats_period": "24h",
  "mode": "aggregates",
  "visualization": [{"chart_type": 1, "y_axes": ["count()"]}]
}

This maps to events API as:

dataset=spans
field[]=transaction         (from group_by)
field[]=count()             (from visualization y_axes + sort)
query=span.op:db span.duration:>1s
sort=-count()
statsPeriod=24h
per_page=25

Field derivation rules:

• Aggregate mode: field[] = group_by + unique aggregate functions from visualization[].y_axes and sort
• Samples mode: field[] = strategy-specific defaults (span.op, span.description, span.duration, timestamp, etc.) since the NL response doesn't explicitly list sample columns
• For Issues strategy: Don't use events API — use the existing listIssues functions with the translated query

Test Files to Create

test/lib/api/search-agent.test.ts

• Mock API responses for translate, start, state endpoints
• Test field derivation from NL response → events API params
• Test error handling (403 feature flag, 500 Seer unavailable)

test/commands/search/search.test.ts

• Test full command flow with mocked API
• Test --translate-only output formatting
• Test --web opens browser with correct URL
• Test --json output structure
• Test strategy → dataset mapping
• Test dynamic table rendering from explore results

test/lib/formatters/explore.test.ts

• Test dynamic column generation from meta.fields
• Test value formatting for each field type
• Test translated query display formatting

Phase 2 (Future): --ask on Existing Commands

Add --ask flag to issue list, log list, trace list:

• When --ask is provided with a value, translate the NL query first
• Replace --query with the translated query string
• Keep the rest of the command's normal flow
• This is additive and can be done after the main search command ships

Verification

Manual Testing

# Requires auth + Seer access
bun run --env-file=.env.local src/bin.ts search "show me slow database queries"
bun run --env-file=.env.local src/bin.ts search "errors in production last 24h" --strategy errors
bun run --env-file=.env.local src/bin.ts search "p95 latency by endpoint" --json
bun run --env-file=.env.local src/bin.ts search "log errors from auth service" --strategy logs --web
bun run --env-file=.env.local src/bin.ts ask "which issues affect the most users"
bun run --env-file=.env.local src/bin.ts search "slow queries" --translate-only

Automated Testing

bun test test/commands/search/
bun test test/lib/api/search-agent.test.ts
bun test test/lib/formatters/explore.test.ts
bun run typecheck
bun run lint:fix