Skip to content

docs: fix 30 documentation inaccuracies found by source code audit#697

Open
aidandaly24 wants to merge 11 commits intoaws:mainfrom
aidandaly24:docs/fix-documentation-inaccuracies
Open

docs: fix 30 documentation inaccuracies found by source code audit#697
aidandaly24 wants to merge 11 commits intoaws:mainfrom
aidandaly24:docs/fix-documentation-inaccuracies

Conversation

@aidandaly24
Copy link
Copy Markdown
Contributor

@aidandaly24 aidandaly24 commented Mar 27, 2026

Description

Comprehensive audit of all .md documentation files against actual source code (Zod schemas, CLI commands, CDK constructs). Fixes 50+ inaccuracies across 14 files where docs had drifted from the implementation, including alignment with several upstream refactors that landed during this effort.

Upstream refactors accounted for

PR Change Doc Impact
#709 Remove type fields from resource schemas, rename credential typeauthorizerType Removed stale type fields from all JSON examples; renamed credential discriminator
#705 add identityadd credential, --force-y/--yes, --plan--dry-run Updated all command references and flag tables
#706 agentsruntimes array, --agent--runtime flags Updated all flag tables, examples, and JSON config across 7 doc files
#707 Dev command --invoke → positional [prompt] arg Already reflected in docs
#713 Revert CUSTOM strategy and semanticOverride No doc changes needed (CUSTOM was never documented)

New features documented (merged after initial audit)

PR Feature Doc Files
#763, #780 Import subcommands (import runtime, import memory, import evaluator, import online-eval) commands.md
#750 Exec mode for invoke and dev (--exec, --timeout) commands.md, local-development.md
#739 Code-based evaluator support (--type, --lambda-arn, --timeout) commands.md, evals.md
#732 Ground truth reference inputs (--assertion, --expected-trajectory, --expected-response) commands.md, evals.md
#531, #534 Memory record streaming (--data-stream-arn, --stream-content-level, --delivery-type) commands.md, memory.md
#782 --skip-install flag on create commands.md
#774 --identity-name option on fetch access commands.md

Key fixes by category

Schema/config drift (configuration.md, gateway.md, container-builds.md):

  • Added missing EPISODIC memory strategy and reflectionNamespaces field
  • Added lifecycleConfiguration (idleRuntimeSessionTimeout, maxLifetime) to agent spec
  • Fixed CUSTOM_JWT fields: allowedAudience/allowedClients/allowedScopes/customClaims are conditionally required
  • Added missing customClaims field to CUSTOM_JWT config
  • Fixed MCP Server outbound auth: removed api-key (only oauth/none valid per schema)
  • Added enableSemanticSearch, exceptionLevel, policyEngineConfiguration gateway fields
  • Fixed memory strategies default: [] not @min 1 (empty array valid for short-term memory)
  • Removed stale "type": "AgentCoreRuntime" / "AgentCoreMemory" / "CustomEvaluator" / "OnlineEvaluationConfig" from all JSON examples

Flag renames (commands.md, evals.md, local-development.md):

  • --agent--runtime, -a-r across all commands (status, dev, invoke, logs, traces, run eval, evals history, package, add online-eval, logs evals)
  • --agent-arn--runtime-arn, --agent-runtime-id--runtime-id
  • --plan--dry-run, --force-y/--yes
  • add identityadd credential, remove identityremove credential
  • --agent-client-id--client-id, --agent-client-secret--client-secret (gateway CUSTOM_JWT)
  • --exception-level valid values: NONE/DEBUG (not ALL)
  • --lines--limit in logs evals
  • Note: --agent-id and --agent-alias-id are unchanged (Bedrock Agent import flags)

Missing flags added (commands.md):

  • add agent: --authorizer-type, --discovery-url, --allowed-audience, --allowed-clients, --allowed-scopes, --custom-claims, --client-id, --client-secret, --idle-timeout, --max-lifetime
  • create: --idle-timeout, --max-lifetime, --skip-install
  • add gateway: --runtimes, --custom-claims, --policy-engine, --policy-engine-mode
  • dev and invoke: -H, --header, --exec
  • invoke: --bearer-token, --exec, --timeout
  • add evaluator: --type, --lambda-arn, --timeout (code-based evaluator support)
  • run eval: --assertion, --expected-trajectory, --expected-response (ground truth inputs)
  • add memory: --data-stream-arn, --stream-content-level, --delivery-type, --stream-delivery-resources
  • fetch access: --identity-name

New sections added:

  • commands.md: Full import command section with 4 subcommands (runtime, memory, evaluator, online-eval)
  • evals.md: Evaluator Types section (LLM-as-a-Judge vs code-based), Ground Truth Reference Inputs subsection
  • memory.md: Memory Record Streaming section with CLI flags and JSON config example
  • local-development.md: Exec Mode section covering dev exec, invoke exec, and TUI shortcuts

Stale values (commands.md, evals.md, frameworks.md):

  • Fixed model ID claude-3-5-sonnet-20250929claude-3-5-sonnet-20250514
  • Added policy-engine/policy to status --type enum
  • Removed ghost --tool-filter-description flag (not in schema or CLI)
  • Fixed evaluator level values: added TOOL_CALL, removed SPAN

Structural/naming (AGENTS.md, README.md, assets):

  • Removed AutoGen from supported frameworks list (5 frameworks, not 6)
  • Fixed "agents""runtimes" in gateway JSON example
  • Renamed identitycredential in all command references
  • Fixed file paths and directory references
  • Fixed regex constraints and enum values to match Zod schemas

Files modified

File Changes
docs/configuration.md Schema field tables, JSON examples, type removals, credential rename
docs/commands.md All flag tables and examples, import subcommands, exec mode, streaming, evaluator types, ground truth
docs/evals.md Flag tables, examples, evaluator types section, ground truth inputs, --runtime rename
docs/gateway.md identity → credential, agents → runtimes in JSON example
docs/local-development.md --runtime flag in dev examples, exec mode section
docs/memory.md Removed stale types, added memory record streaming section
docs/container-builds.md Removed stale "type": "AgentCoreRuntime" from JSON example
docs/frameworks.md Model ID fix, framework list correction
AGENTS.md identity → credential in primitives
README.md identity → credential
src/assets/README.md identity → credential in vended template
src/assets/__tests__/__snapshots__/assets.snapshot.test.ts.snap Updated snapshot

Related Issue

N/A — discovered via systematic source code audit

Documentation PR

N/A — this PR is the documentation fix

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Other (please describe):

Testing

  • I ran npm run test:unit and npm run test:integ
  • I ran npm run typecheck
  • I ran npm run lint
  • If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

All changes are documentation-only (.md files + snapshot update). Each fix was verified against the corresponding Zod schema definition or CLI source code (src/cli/commands/*/command.tsx, src/cli/primitives/*.ts).

Note: 19 pre-existing test failures exist on upstream/main from the schema refactor (#709) — not introduced by this PR. The 2 e2e failures are expected on fork PRs (no AWS credentials).

Checklist

  • I have read the CONTRIBUTING document
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

@aidandaly24 aidandaly24 requested a review from a team March 27, 2026 04:40
@github-actions github-actions bot added the size/m PR size: M label Mar 27, 2026
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 27, 2026
@aidandaly24 aidandaly24 force-pushed the docs/fix-documentation-inaccuracies branch from f2e7d91 to f957db1 Compare March 27, 2026 17:36
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 27, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 27, 2026

Package Tarball

aws-agentcore-0.7.1.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-697-tarball/aws-agentcore-0.7.1.tgz

@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 27, 2026
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 27, 2026
@aidandaly24 aidandaly24 force-pushed the docs/fix-documentation-inaccuracies branch from bd9373b to c4a8c0a Compare March 27, 2026 18:36
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 27, 2026
@aidandaly24 aidandaly24 force-pushed the docs/fix-documentation-inaccuracies branch from c4a8c0a to 727e1cf Compare March 27, 2026 21:39
@github-actions github-actions bot removed the size/m PR size: M label Mar 27, 2026
@github-actions github-actions bot added size/l PR size: L and removed size/m PR size: M labels Mar 27, 2026
Hweinstock
Hweinstock previously approved these changes Mar 28, 2026
Copy link
Copy Markdown
Contributor

@Hweinstock Hweinstock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! Its possible we'll need another update after the nits are addressed.

@aidandaly24 aidandaly24 force-pushed the docs/fix-documentation-inaccuracies branch from 8b133b3 to 1b43c59 Compare March 28, 2026 00:51
@github-actions github-actions bot added size/l PR size: L and removed size/l PR size: L labels Mar 28, 2026
@aidandaly24 aidandaly24 force-pushed the docs/fix-documentation-inaccuracies branch from 1b43c59 to c8a57f4 Compare March 28, 2026 01:22
@github-actions github-actions bot added size/l PR size: L and removed size/l PR size: L labels Mar 28, 2026
@github-actions github-actions bot added size/l PR size: L and removed size/l PR size: L labels Mar 28, 2026
@github-actions github-actions bot removed the size/l PR size: L label Mar 28, 2026
- configuration.md: fix networkMode PRIVATE→VPC, remove stale mcp.json
  section, add 7 missing AgentEnvSpec fields, add Node.js runtimes,
  fix gateway name max 63→100, list all 6 targetType values, fix memory
  strategies wording, fix deployed-state.json path, add missing project
  fields, fix CUSTOM_JWT field requirements, add customClaims field
- README.md: add CrewAI framework, update all 4 default model IDs,
  add missing command sections
- frameworks.md: add CrewAI with model providers, add protocol
  compatibility matrix (HTTP/MCP/A2A)
- commands.md: fix agent name max 64→48 chars, add fetch access section,
  add policy-engine/policy to status --type, remove ghost
  --tool-filter-description flag, fix model ID date
- gateway.md: add enableSemanticSearch/exceptionLevel config options,
  fix mcp.json→agentcore.json, remove api-key from MCP Server auth
- transaction_search.md: fix traces command syntax
- container-builds.md: remove nonexistent --progress flag
- evals.md: fix model date 20250929→20250514
- local-development.md: fix mcp.json→agentcore.json
- AGENTS.md: remove AutoGen, remove stale mcp.json reference
- src/assets/README.md: add CrewAI to template list
- src/assets/agents/AGENTS.md: remove AutoGen

Constraint: All fixes verified against Zod schemas and CLI runtime
Constraint: CLI --help text bugs left for separate agent
Confidence: high
Scope-risk: narrow
- Remove `type` fields from agent, memory, evaluator, online-eval-config
  examples and field tables (removed in aws#709)
- Remove `modelProvider` from agent spec (removed in aws#709)
- Rename credential `type` to `authorizerType` (renamed in aws#709)
- Rename `add identity` to `add credential` everywhere (aws#705)
- Replace `--force` with `-y/--yes` on remove commands (aws#705)
- Replace `--plan` with `--dry-run` on deploy command (aws#705)
- Update snapshot for src/assets/README.md change
Update `"agents": [...]` to `"runtimes": [...]` in the gateway
configuration example to match the schema rename in aws#706.
Update all documentation to reflect upstream renames:
- --agent → --runtime, -a → -r across all commands
- --agent-arn → --runtime-arn, --agent-runtime-id → --runtime-id
- Remove stale "type" fields from JSON examples in evals.md,
  memory.md, and container-builds.md

Constraint: --agent-id, --agent-alias-id, --agent-client-id,
  --agent-client-secret are unchanged (different semantics)
Constraint: --type agent filter value in status/fetch stays as-is
Confidence: high
Scope-risk: narrow
- Add -H, --header flag to dev and invoke commands
- Add --bearer-token flag to invoke command
- Fix --lines → --limit in evals.md logs evals section

Confidence: high
Scope-risk: narrow
- Rename --agent-client-id → --client-id, --agent-client-secret →
  --client-secret in gateway docs (naming unification aws#705)
- Add missing flags to add agent: --authorizer-type, --discovery-url,
  --allowed-audience, --allowed-clients, --allowed-scopes,
  --custom-claims, --client-id, --client-secret, --idle-timeout,
  --max-lifetime
- Add missing flags to create: --idle-timeout, --max-lifetime
- Add missing flags to add gateway: --runtimes, --custom-claims,
  --policy-engine, --policy-engine-mode
- Fix --exception-level values: NONE/ALL (was undocumented)

Confidence: high
Scope-risk: narrow
The CLI help text says ALL but the schema (GatewayExceptionLevelSchema)
defines the valid values as NONE and DEBUG. Aligns commands.md with
gateway.md and the Zod schema.
- Remove stale "type": "AgentCoreMemory" from memory streaming example
- Add memory streaming flags: --delivery-type, --data-stream-arn,
  --stream-content-level, --stream-delivery-resources
- Add --exec to dev command flag table (Container agents)
- Add --skip-install to create command flag table
- Add --identity-name to fetch access flag table
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/l PR size: L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants