docs: evals-minimal and kitchen sink examples
#169
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a comprehensive "kitchen sink" example demonstrating AI instrumentation and evaluation patterns. It introduces a support agent demo with OpenTelemetry tracing, feature flags, and multiple evaluation suites.
Key Changes:
- New example project with Next.js 16, React 19, and AI SDK integration
- OpenTelemetry instrumentation utilities for tracing and span management
- Health check API endpoint using Hono framework
- Updated workspace dependencies with specific Zod version pinning
Reviewed changes
Copilot reviewed 41 out of 48 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| pnpm-workspace.yaml | Reorganizes catalog entries and pins Zod to exact version 4.1.5 |
| pnpm-lock.yaml | Adds kitchen-sink example dependencies including Next.js 16.0.4, React 19, OpenTelemetry, and Tailwind v4 |
| examples/kitchen-sink/tsconfig.json | Standard Next.js TypeScript configuration with strict mode and path aliases |
| examples/kitchen-sink/src/lib/utilities/tracer.ts | Creates OpenTelemetry tracer singleton for the application |
| examples/kitchen-sink/src/lib/utilities/start-active-span.ts | Utility wrapper for creating traced spans with error handling and callbacks |
| examples/kitchen-sink/src/lib/utilities/get-current-trace-id.ts | Helper to retrieve active trace ID from OpenTelemetry context |
| examples/kitchen-sink/src/lib/openai.ts | Initializes OpenAI client with API key from environment |
| examples/kitchen-sink/src/lib/api/health.ts | Basic health check endpoint returning API status and timestamp |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
commit: |
commit: |
c-ehrlich
changed the title
evals-minimal and kitchen sink examples
| { | ||
| input: 'Hello, are you a bot?', | ||
| expected: [], | ||
| purpose: 'chat_no_tool', |
There was a problem hiding this comment.
First time seeing this purpose, I like it, just might be misleading having them everywhere in this example?
There was a problem hiding this comment.
what is misleading about it? collections can have arbitrary excess properties. could be replaced by a comment of course but this seems more readable to me. what do you think
There was a problem hiding this comment.
Yeah, what I mean is it gives away the perception that purpose it's a first party offering of the SDK, it seems good to have this, just maybe scoped to a single .eval.ts either way, it's a nit pick
2 participants
Replaces the
evalsexample withevals-minimal, a much smaller example. This is more accessible, and then thekitchen-sinkexample shows EVERYTHING including how the pieces connect.Adds a
kitchen sinkexample that shows a demo for a support agent with instrumentation and evals.This example shows:
wrapAISDKModel,wrapToolswithSpancreateAppScope,flag,pickFlagsEval,ScorerIt includes four evals:
retrieve-from-knowledge-base.eval.ts: Tests the RAG retrieval logic (veryBadRAG) to ensure it returns the expected document IDs for various inputs, including ambiguous queries and adversarial prompts.extract-ticket-info.eval.ts: Tests the structured data extraction capability (extractTicketInfo) to confirm it correctly identifies ticket fields like "intent" and "product" and reports any missing information. This is based on Hamel's "fill in the blanks".categorize-messages.eval.ts: Tests the message classification function (categorizeMessage) to verify it accurately labels inputs as support, complaint, spam, or wrong company.support-agent-e2e-tool-use.eval.ts: Tests the main support agent loop (runSupportAgent) to verify it correctly chooses between using the searchKnowledgeBase tool or no tools at all depending on the user's query.Example eval run: https://app.dev.axiomtestlabs.co/axiomers-ft83/ai-engineering/evaluations?runId=RMOX7TRDQM
Also made some package/lockfile changes to stop example builds from failing