Skip to content

Commit 4a637e3

Browse files
committed
docs(skills): encode integration-hardening learnings into the authoring skills
Eight traps found across a 15-PR integration-hardening sweep, each written into the skill that owns it and cross-referenced rather than duplicated. add-tools - Reserved param names. The shared transport reads `timeout`, `proxyUrl`, and `method` off `params` before `request` sees them; `timeout` is its own HTTP deadline in milliseconds, so Daytona's documented 10-second sandbox timeout aborts the call after 10ms. - Path traversal. `encodeURIComponent` does not stop `.`/`..` — they are unreserved and the URL parser removes dot segments after decoding. Documents when each of the three `tools/url-path.ts` helpers applies, and why `params.x?.trim()` guards `undefined` rather than the type. add-block - Omitting a key from `tools.config.params` does not drop it; the executor merges the patch over the raw inputs, so clearing a key needs an explicit `undefined`. - Renaming a subBlock id orphans saved workflow state. Rename the tool param and map it; `_removed_` migrations cover genuine removals. - Declared `outputs` do not drive variable resolution — the resolver walks the runtime object, so changing an output's shape breaks references that were never declared. validate-integration - Path-safety harness design: enumerate (tool, param) pairs, fuzz one at a time, assert named rejection rather than path shape, probe conditional and presence branches, and assert the skip ledger is empty. - Test files are type-checked by nothing — tsconfig excludes them and Vitest transpiles without checking. - Replaces the two checklist lines that taught the now-known-defective `${params.id.trim()}` path pattern. add-integration gets pointers only.
1 parent 2cda264 commit 4a637e3

4 files changed

Lines changed: 216 additions & 5 deletions

File tree

.agents/skills/add-block/SKILL.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -550,6 +550,30 @@ Maps multiple UI fields to a single serialized parameter:
550550
- ONLY use `canonicalParamId` to link basic/advanced mode alternatives for the same logical parameter
551551
- Do NOT use it for any other purpose
552552

553+
## Renaming a SubBlock Id Orphans Saved Workflow State
554+
555+
A subBlock `id` is the storage key for every value users have already saved in deployed workflows.
556+
Renaming it silently orphans that state — the field renders empty and the workflow runs without it.
557+
558+
**Rename the *tool* param and map it; keep the subBlock id.** `tools.config.params` is where the two
559+
names meet:
560+
561+
```typescript
562+
// subBlock id stays `timeout` — saved state keeps resolving
563+
params: (params) => ({ timeoutSeconds: params.timeout, timeout: undefined }),
564+
```
565+
566+
`apps/sim/scripts/check-block-registry.ts` enforces this: it diffs the block definitions and fails
567+
when a subblock id disappears (`check-block-registry.ts:181`), pointing you at the migration table.
568+
It separately fails when a required `user-only` tool param has no subBlock whose `id` **or**
569+
`canonicalParamId` equals it (`check-block-registry.ts:225`), which is the other half of the same
570+
contract.
571+
572+
For a field that is genuinely gone — not renamed — record it in `SUBBLOCK_ID_MIGRATIONS` in
573+
`apps/sim/lib/workflows/migrations/subblock-migrations.ts`. A `to` value prefixed with `_removed_`
574+
(`subblock-migrations.ts:20`, `:109`) means "deleted outright", and is what lets the check pass
575+
without pretending the value moved somewhere.
576+
553577
## WandConfig Pattern
554578

555579
Enables AI-assisted field generation.
@@ -607,6 +631,29 @@ tools: {
607631
}
608632
```
609633

634+
### Omitting a key from `tools.config.params` does NOT drop it
635+
636+
`tools.config.params` returns a **patch**, not a replacement. The executor merges it over the raw
637+
inputs — `apps/sim/executor/handlers/generic/generic-handler.ts:191`:
638+
639+
```typescript
640+
const transformedParams = blockConfig.tools.config.params(inputs)
641+
finalInputs = { ...inputs, ...transformedParams }
642+
```
643+
644+
So a subBlock value reaches the tool even when `params` never mentions its key. This matters when a
645+
subBlock id collides with a name the shared transport reserves (`timeout`, `proxyUrl`, `method` — see
646+
**Reserved Parameter Names** in the `add-tools` skill): renaming only the tool-side param leaves the
647+
old key merging straight back in. Clearing it requires an **explicit** `undefined`:
648+
649+
```typescript
650+
params: (params) => ({
651+
// ✓ Removes the reserved key from the outbound params
652+
timeout: undefined,
653+
timeoutSeconds: params.timeout,
654+
}),
655+
```
656+
610657
### V2 Versioned Tool Selector
611658
```typescript
612659
import { createVersionedToolSelector } from '@/blocks/utils'
@@ -681,6 +728,24 @@ Nested object outputs (`plan: { id: { type: 'string' }, ... }`) are a **tool-out
681728

682729
If the output shape is unknown because the underlying tool response is undocumented, you MUST tell the user and stop. Unknown is not the same as variable. Never guess block outputs.
683730

731+
### Declared `outputs` do not drive variable resolution
732+
733+
`outputs` is documentation and editor autocomplete — it is **not** the contract `<block.path>`
734+
references resolve against. `apps/sim/executor/utils/block-reference.ts:239` navigates the **runtime**
735+
output object, and the declared schema is only consulted at `:242``if (value === undefined && schema)`
736+
— to produce a better error for a path that resolved to nothing.
737+
738+
Two consequences:
739+
740+
- `<block.some.path>` resolves fine even when `some.path` was never declared. An undeclared field is
741+
still a live reference someone may have wired up.
742+
- Changing an output's **shape** therefore breaks saved references that never appeared in `outputs`
743+
at all, and no check will tell you. Adding a field is safe; re-nesting, renaming, or wrapping the
744+
existing keys is not.
745+
746+
When restructuring a `transformResponse`, spread the raw provider keys **last** so every previously
747+
reachable path survives alongside the new shape.
748+
684749
## V2 Block Pattern
685750

686751
When creating V2 blocks (alongside legacy V1):
@@ -1028,6 +1093,9 @@ changes.
10281093
- [ ] Tools.access lists all tool IDs (snake_case)
10291094
- [ ] Tools.config.tool returns correct tool ID (snake_case)
10301095
- [ ] Outputs match tool outputs
1096+
- [ ] No existing subBlock `id` was renamed or removed without a `SUBBLOCK_ID_MIGRATIONS` entry
1097+
- [ ] Restructured outputs still expose every previously reachable runtime key (raw keys spread last)
1098+
- [ ] Any reserved transport key a subBlock still sends is explicitly cleared with `undefined` in `tools.config.params`
10311099
- [ ] Block + meta registered in registry-maps.ts (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`)
10321100
- [ ] If any tool was added, changed or removed alongside the block: ran `bun run tool-metadata:generate` and committed the artifacts
10331101
- [ ] Ran `bun run scripts/generate-docs.ts`, reviewed the generated diff, and committed the integration catalog changes

.agents/skills/add-integration/SKILL.md

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -930,4 +930,15 @@ requiredScopes: getScopesForService('{service}'),
930930
11. **Never hardcode scopes** - Use `getScopesForService()` in blocks and `getCanonicalScopesForProvider()` in auth.ts
931931
12. **Always add scope descriptions** - New scopes must have entries in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
932932
13. **OAuth service IDs need deployment capabilities** - Every visible OAuth integration must resolve through `OAUTH_CLIENT_CAPABILITIES`; shared Google/Microsoft aliases map to their provider capability
933-
14. **Keep runtime and presentation separate** - Runtime OAuth fields live in `packages/deployment-config/src/env-capabilities.ts`; CLI input modes live in the exhaustively checked `packages/sim-setup/src/capability-config.ts` mapping
933+
14. **`timeout`, `proxyUrl`, and `method` are reserved** - `apps/sim/tools/request-transport.ts`
934+
reads all three off `params` (`:191`, `:198`, `:167`); `timeout` is its own HTTP deadline in
935+
**milliseconds**. See **Reserved Parameter Names** in `.agents/skills/add-tools/SKILL.md`
936+
15. **Never interpolate a param into a URL path raw** - `encodeURIComponent` does not stop `.`/`..`
937+
traversal. Use the helpers in `apps/sim/tools/url-path.ts`; see **Path Parameters: Reject
938+
Traversal, Never Just Encode** in `.agents/skills/add-tools/SKILL.md`, and add the
939+
`path_safety.test.ts` shape from `.agents/skills/validate-integration/SKILL.md`
940+
16. **Never rename or drop a subBlock `id`** - it is the storage key for deployed workflows. Rename
941+
the tool param and map it in `tools.config.params`; see the `add-block` skill
942+
17. **Omitting a key from `tools.config.params` does not drop it** - the executor merges the patch
943+
over the raw inputs, so clearing a key needs an explicit `undefined`; see the `add-block` skill
944+
18. **Keep runtime and presentation separate** - Runtime OAuth fields live in `packages/deployment-config/src/env-capabilities.ts`; CLI input modes live in the exhaustively checked `packages/sim-setup/src/capability-config.ts` mapping

.agents/skills/add-tools/SKILL.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,27 @@ fallback, or caller-controlled `_context` authority.
199199
- Always explicitly set `required: true` or `required: false`
200200
- Optional params should have `required: false`
201201

202+
### Reserved Parameter Names
203+
204+
The shared transport reads three names off `params` for its own purposes, before your `request`
205+
config ever sees them (`apps/sim/tools/request-transport.ts`):
206+
207+
| Param | Read at | What the transport does with it |
208+
|---|---|---|
209+
| `timeout` | `request-transport.ts:191` | Outbound HTTP deadline **in milliseconds**, clamped to `getMaxExecutionTimeout()` |
210+
| `proxyUrl` | `request-transport.ts:198` | Egress proxy URL for the request |
211+
| `method` | `request-transport.ts:167` | Overrides a *static* `request.method` string (a `method` **function** wins over it) |
212+
213+
Never declare a user-facing param with one of these names unless it means exactly what the transport
214+
means. The collision is silent and unit-blind: `apps/sim/tools/daytona/execute_command.ts:49`
215+
declares `timeout` as *"Timeout in seconds (defaults to 10 seconds)"*, so a documented 10-second
216+
sandbox timeout aborts the HTTP call after **10 milliseconds**.
217+
218+
Give the param a distinct Sim-side name (`timeoutSeconds`, `executionTimeout`, `httpMethod`) and emit
219+
the provider's spelling from `request.body` or `request.url`. Renaming the tool param is not enough
220+
on its own if a block already sends the reserved key — see **Omitting a key from `tools.config.params`
221+
does not drop it** in the `add-block` skill.
222+
202223
## Resolved Secrets and Provenance Boundaries
203224

204225
- Leave ordinary external API inputs and third-party results unchanged. Add provenance handling only
@@ -219,6 +240,48 @@ fallback, or caller-controlled `_context` authority.
219240
- Add focused tests for named projection, identical unproven public text, malformed/incomplete
220241
metadata, metadata stripping, scope isolation, and legacy compatibility where applicable.
221242

243+
## Path Parameters: Reject Traversal, Never Just Encode
244+
245+
`encodeURIComponent` does **not** stop path traversal. `.` and `..` are *unreserved* characters, so
246+
they survive encoding verbatim, and the WHATWG URL parser that `fetch` uses removes dot segments
247+
**after** decoding — the percent-encoded spellings included:
248+
249+
```
250+
new URL('https://x/v1/a/b/..').pathname // => '/v1/a/'
251+
new URL('https://x/v1/a/b/%2e%2e').pathname // => '/v1/a/' (still removed)
252+
```
253+
254+
A removed segment pops a path segment on a fixed host with the workspace's bearer token still
255+
attached, including on DELETE routes. Path IDs are typically `visibility: 'user-or-llm'`, so prompt
256+
injection controls them. Rejection is the only mechanism that closes this; the module note at the top
257+
of `apps/sim/tools/url-path.ts` states the rule in full.
258+
259+
Never interpolate a param into a request path yourself. Use the helpers in `apps/sim/tools/url-path.ts`:
260+
261+
| Helper | Use when the parameter is | Trims? |
262+
|---|---|---|
263+
| `safeUrlPathSegment` (`url-path.ts:172`) | **One opaque id**`user_abc`, `12345`, a repo name. Rejects any `/` or `\`: a separator means the caller passed something other than what the parameter addresses. | Yes — surrounding whitespace on a copy-pasted id is transport noise. |
264+
| `safeUrlPath` | **A real slash-delimited path** the provider documents as such (GitHub `path`, `branch`, `ref`). Splits on `/`, rejects any `.`/`..` segment, percent-encodes each segment, keeps the separators. | **No** — a leading or trailing space is a legal git filename, so trimming would read, update, or *delete* a different file. |
265+
| `safeEncodedUrlPathSegment` | **One value that may itself contain `/`** but the provider reads as a single parameter (a GitHub label `area/api` in `DELETE .../labels/{name}`). Preserves the separator as `%2F`. | Yes. |
266+
267+
Prefer `safeUrlPathSegment`. Reach for the other two only when the provider documents the parameter
268+
as slash-bearing — never to make a separator stop erroring on a single-segment id. (`safeUrlPath` and
269+
`safeEncodedUrlPathSegment` arrive with the path-safety sweep; if your checkout only exports
270+
`safeUrlPathSegment`, add them there rather than hand-rolling a local encoder.)
271+
272+
### `params.x?.trim()` guards `undefined`, not the type
273+
274+
A param's declared `type: 'string'` is enforced nowhere between the LLM tool call — or a
275+
`<Block.output>` reference resolved out of stored workflow state — and your URL builder. A
276+
numeric-looking id (a Vercel `deploymentId`, a Daytona `sandboxId`) arrives as a JSON **number** and
277+
stays one, so `params.id?.trim()` throws a bare `TypeError: params.id.trim is not a function` naming
278+
neither the tool nor the parameter.
279+
280+
`toGuardedString` (`apps/sim/tools/url-path.ts:98`) is why the helpers do not have this problem: it
281+
accepts `string`, `number`, and `bigint`, rejects everything else **by parameter name**, and refuses
282+
number spellings whose decimal text is not the id the caller meant. Route path params through the
283+
helpers instead of hand-rolling an optional-chained `trim()`.
284+
222285
## Critical Rules for Outputs
223286

224287
### Output Types
@@ -527,6 +590,8 @@ All tool IDs MUST use `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`
527590
- [ ] No tool declares `directExecution`; in-process work uses a registered operation
528591
- [ ] All params have explicit `required: true` or `required: false`
529592
- [ ] All params have appropriate `visibility`
593+
- [ ] No param is named `timeout`, `proxyUrl`, or `method` unless it means what the transport means
594+
- [ ] Every param interpolated into a request path goes through a `tools/url-path.ts` helper
530595
- [ ] All nullable response fields use `?? null`
531596
- [ ] All optional outputs have `optional: true`
532597
- [ ] No raw JSON dumps in outputs

.agents/skills/validate-integration/SKILL.md

Lines changed: 71 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,10 @@ For **every** tool file, check:
8282
- `'user-only'` — for API keys, credentials, and account-specific IDs the user must provide
8383
- `'user-or-llm'` — for everything else (search queries, content, filters, IDs that could come from other blocks)
8484
- [ ] Every param has a `description` that explains what it does
85+
- [ ] No param is named `timeout`, `proxyUrl`, or `method` unless it means exactly what the shared
86+
transport means — `apps/sim/tools/request-transport.ts` reads all three off `params`
87+
(`:191`, `:198`, `:167`), and `timeout` is milliseconds. See **Reserved Parameter Names** in
88+
`.agents/skills/add-tools/SKILL.md`
8589

8690
### Request
8791
- [ ] URL matches the API endpoint exactly (correct base URL, path segments, path params)
@@ -92,8 +96,12 @@ For **every** tool file, check:
9296
- [ ] `Content-Type` header is set for POST/PUT/PATCH requests
9397
- [ ] Body sends all required fields and only includes optional fields when provided
9498
- [ ] For GET requests with query params: URL is constructed correctly with query string
95-
- [ ] ID fields in URL paths are `.trim()`-ed to prevent copy-paste whitespace errors
96-
- [ ] Path params use template literals correctly: `` `https://api.service.com/v1/${params.id.trim()}` ``
99+
- [ ] Every param interpolated into a URL path goes through a helper from `apps/sim/tools/url-path.ts`
100+
(`safeUrlPathSegment` for an opaque id, `safeUrlPath` for a documented slash-delimited path,
101+
`safeEncodedUrlPathSegment` for a single value that may contain `/`) — never a bare
102+
`` `${params.id.trim()}` `` and never a bare `encodeURIComponent`. See **Path Parameters:
103+
Reject Traversal, Never Just Encode** in `.agents/skills/add-tools/SKILL.md` for why encoding
104+
is insufficient and why `params.id?.trim()` throws a raw `TypeError` on a numeric id
97105

98106
### Response / transformResponse
99107
- [ ] Correctly parses the API response (`await response.json()`)
@@ -329,13 +337,68 @@ If any tool lists, searches, exports, imports, downloads, uploads, paginates, ba
329337
- [ ] Large result payloads are summarized, paginated, referenced, or capped rather than raw-dumped
330338
- [ ] Pagination and download tests cover caps, early stop behavior, or partial-result preservation when relevant
331339

332-
## Step 9: Validate Error Handling
340+
## Step 8: Validate Path-Traversal Safety
341+
342+
If any tool interpolates a param into a URL path, the integration needs a path-safety suite. Design it
343+
as follows — the naive shape does not work, and looks like it does.
344+
345+
**Why the naive harness is worthless.** A suite that fuzzes every param at once and wraps the build in
346+
`try { ... } catch { return }` reports green on completely unguarded parameters: the first *guarded*
347+
sibling throws, the case is skipped, and every unguarded sibling is never exercised. Whole tools drop
348+
out of coverage the same way, and an aggregate count cannot detect it, because a case that never
349+
existed cannot fail. `x_manage_block.targetUserId` and `okta_remove_user_from_group.userId` were both
350+
fully unguarded while passing every vector a suite of that shape threw at them.
351+
352+
A sound suite has five properties:
353+
354+
1. **Enumerate (tool, param) pairs, fuzz one at a time.** Discover pairs by probing — build the URL
355+
with a sentinel in one param and check whether it lands in `pathname` — and hold every sibling at a
356+
safe value. Give *other* number params a real number so a sibling's own validation cannot abort the
357+
build. Reference: `apps/sim/tools/rootly/path_safety.test.ts:151`,
358+
`apps/sim/tools/github/path_safety.test.ts:171`.
359+
2. **Assert rejection, not path shape.** A shape assertion (origin + segment count + unchanged
360+
segments) catches only a minority of the vectors: `%2F` is never decoded back into a separator, and
361+
a trailing bare `.` collapses to the parent with the segment count intact. Add an explicit
362+
`expect(() => build(param, '..')).toThrow(new RegExp(paramName))` — the error must *name the
363+
parameter*. Reference: `apps/sim/tools/clerk/path_safety.test.ts:206`,
364+
`apps/sim/tools/rootly/path_safety.test.ts:193`.
365+
3. **Probe conditional branches.** A param that only appears on one branch of a conditional URL builder
366+
is invisible to a single all-params probe. Harvest the string literals the builder compares against
367+
and re-probe under each (`apps/sim/tools/spotify/path_safety.test.ts:185`), **and** probe the
368+
presence branches — the shape taken when an optional param is *absent*
369+
(`apps/sim/tools/discord/path_safety.test.ts:203`).
370+
4. **Assert the skipped and unbuildable sets are empty.** Record every failed baseline build with its
371+
reason and assert the ledger against an explicit, justified allowlist
372+
(`apps/sim/tools/github/path_safety.test.ts:229`, `:260`, `:293`), plus a second assertion that no
373+
URL-building tool sits outside the suite unaccounted for (`:301`). This is the check that surfaced a
374+
tool missing from an entire suite.
375+
5. **Pin the legitimate values too.** `..foo`, `foo..`, `v1.2.3`, and a UUID must pass through
376+
unaltered — a guard that over-rejects is its own bug.
377+
378+
- [ ] A `path_safety.test.ts` exists for the service and enumerates (tool, param) pairs
379+
- [ ] Each pair asserts a *named* rejection of the bare `.` and `..` segments
380+
- [ ] Conditional and presence branches of every conditional URL builder are probed
381+
- [ ] The skipped/unbuildable ledger is asserted empty against a justified allowlist
382+
- [ ] Legitimate dot-bearing values pass through unchanged
383+
384+
### These tests are type-checked by nothing
385+
386+
`apps/sim/tsconfig.json` excludes `**/*.test.ts` and `**/*.test.tsx` from `include`, and
387+
`apps/sim/vitest.config.ts` declares no `typecheck` block — Vitest transpiles with esbuild and never
388+
type-checks. So `bun run type-check` will pass over a harness full of type errors, and a harness that
389+
silently narrows to `any` will still run.
390+
391+
To type-check one, write a temporary tsconfig that extends `apps/sim/tsconfig.json`, drops the
392+
`**/*.test.ts` exclusion, and includes only the harness — then delete it. Do not commit it; the
393+
exclusion exists deliberately.
394+
395+
## Step 10: Validate Error Handling
333396

334397
- [ ] `transformResponse` checks for error conditions before accessing data
335398
- [ ] Error responses include meaningful messages (not just generic "failed")
336399
- [ ] HTTP error status codes are handled (check `response.ok` or status codes)
337400

338-
## Step 10: Report and Fix
401+
## Step 11: Report and Fix
339402

340403
### Report Format
341404

@@ -446,6 +509,10 @@ After fixing, confirm:
446509
- [ ] Regenerated deployment config when block/OAuth metadata changed and ran both catalog checks
447510
- [ ] Validated pagination consistency across tools and block
448511
- [ ] Validated memory load safety using `.agents/skills/memory-load-check/SKILL.md` when tools list/search/download/import/export/batch data
512+
- [ ] Validated path-traversal safety: url-path helpers at every path interpolation, and a
513+
`path_safety.test.ts` that enumerates (tool, param) pairs, asserts named rejection, probes
514+
conditional and presence branches, and asserts an empty skip ledger
515+
- [ ] Validated no param collides with a transport-reserved name (`timeout`, `proxyUrl`, `method`)
449516
- [ ] Validated error handling (error checks, meaningful messages)
450517
- [ ] Validated registry entries (tools and block, alphabetical, correct imports)
451518
- [ ] Validated model-visible/opaque inputs and Sim-durable/internal-execution provenance at their

0 commit comments

Comments
 (0)