Add Flyte 1 → 2 migration skills + eval coverage

.github/workflows/skill-evals.yml

@@ -1,8 +1,8 @@
 name: skill-evals
 
-# Runs the flyte-skills eval harness. On PRs it runs only the scenarios affected
-# by the changed files (see evals/select.py); nightly it runs the full matrix
-# including the `real` tier.
+# Runs the flyte-agent-plugin skills eval harness. On PRs it runs only the
+# scenarios affected by the changed files (see evals/select.py); nightly it runs
+# the full matrix including the `real` tier.
 on:
   pull_request:
     paths:

README.md

@@ -134,6 +134,20 @@ pi install git:github.com/flyteorg/flyte-agent-plugins@<tag>         # pinned to
 | [`flyte-sdk-data`](plugins/flyte/skills/flyte-sdk-data) | Handles data engineering patterns: ETL pipelines, data processing, data quality checks, fanout/map tasks, conditions, dynamic workflows, and batch data transformations. For: ETL, Parquet, CSV, JsonlFile/Dir, schema validation. |
 | [`flyte-sdk-ml`](plugins/flyte/skills/flyte-sdk-ml) | Handles ML workload patterns: model training, hyperparameter optimization, experiment tracking, model evaluation and selection, batch inference, real-time serving, and model monitoring. For: PyTorch, scikit-learn, HuggingFace, GPU, drift detection. |
 
+### Migration (Flyte 1 → 2)
+
+Convert existing Flyte 1 (`flytekit`) code to Flyte 2. Distilled from the official
+[Flyte 1 → 2 migration guide](https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/).
+
+| Skill | Description |
+|-------|-------------|
+| [`flyte-migrate`](plugins/flyte/skills/flyte-migrate) | Start-here migration orchestrator: the `flytekit`→`flyte` shift, the terminology/concept mapping, the two mechanical changes, an incremental migration strategy, hybrid v1/v2 pipelines during transition, and the gotchas — routes to the specific skills below. |
+| [`flyte-migrate-tasks-workflows`](plugins/flyte/skills/flyte-migrate-tasks-workflows) | Migrate `@task`/`@workflow`/`@dynamic` into a single `@env.task` on a `TaskEnvironment`; sequential ordering without `>>`, nested "subworkflows" as tasks, and the parameter-mapping table. |
+| [`flyte-migrate-config`](plugins/flyte/skills/flyte-migrate-config) | Migrate task configuration (images `ImageSpec`→`flyte.Image`, resources/GPUs, `cache_version`→`cache`, secrets, `LaunchPlan`/`CronSchedule`→`Trigger`/`Cron`) and the `pyflyte`→`flyte` CLI / config files. |
+| [`flyte-migrate-control-flow`](plugins/flyte/skills/flyte-migrate-control-flow) | Replace `conditional()` with native `if`/`else`, `@dynamic` with plain Python loops, `on_failure` with `try`/`except`, and `map_task` with `flyte.map` / `asyncio.gather`. |
+| [`flyte-migrate-data-io`](plugins/flyte/skills/flyte-migrate-data-io) | Migrate data types & I/O: `FlyteFile`/`FlyteDirectory`→`flyte.io.File`/`Dir`, `StructuredDataset`→`flyte.io.DataFrame`, dataclasses/Pydantic as task I/O. |
+| [`flyte-migrate-ml`](plugins/flyte/skills/flyte-migrate-ml) | Migrate ML workloads (training, HPO, GPU/deep learning, batch inference, end-to-end pipelines) and the new-in-v2 patterns (real-time serving, apps, sandboxed execution) they unlock. |
+
 Example:
 
 ```
@@ -233,8 +247,7 @@ scripts/smoke_test_mcp.py                   # end-to-end check of the local MCP
 
 Each harness consumes a different part of this. Claude Code and Codex read the plugin
 manifests, so the **plugin name** matters to them. Hermes, opencode, and pi install skills
-by **directory path**, so `plugins/flyte/skills/…` is their interface — which is why the
-rename from `flyte-skills` touched both.
+by **directory path**, so `plugins/flyte/skills/…` is their interface.
 
 The `.mcp.json` server is Claude Code-specific; the skills themselves stay portable across
 harnesses.

evals/README.md

@@ -1,6 +1,6 @@
-# flyte-skills eval harness
+# flyte agent plugin eval harness
 
-Automated testing & evaluation for the `flyte-skills` agent skills. It runs a real
+Automated testing & evaluation for the `flyte` agent skills. It runs a real
 agent harness (**opencode / pi / hermes**) against the union-hosted **GLM** endpoint,
 hands it a skill + a task, and scores what it produces — orchestrated as **Flyte
 workflows on `demo.hosted.unionai.cloud`**, with path-based selection so only the

evals/harness/run.py

@@ -68,7 +68,7 @@ def _print_table(results: list[ScenarioResult]) -> int:
 
 
 def main(argv: list[str] | None = None) -> int:
-    ap = argparse.ArgumentParser(prog="flyte-evals", description="Run flyte-skills evals locally")
+    ap = argparse.ArgumentParser(prog="flyte-evals", description="Run flyte-agent-plugin evals locally")
     ap.add_argument("--scenario", help="scenario id")
     ap.add_argument("--skill", help="filter by skill name")
     ap.add_argument("--tier", choices=["static", "trajectory", "real"], help="filter by tier")

evals/manifest.yaml

@@ -1,4 +1,4 @@
-# Top-level configuration for the flyte-skills eval harness.
+# Top-level configuration for the flyte-agent-plugin eval harness.
 # Scenario -> skill mapping is derived by scanning `scenarios_dir`; each scenario
 # file declares its own `skill`. This manifest holds the cross-cutting config.
 

evals/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
-name = "flyte-skills-evals"
+name = "flyte-agent-plugin-evals"
 version = "0.0.1"
-description = "Testing & eval harness for the Flyte plugin skills."
+description = "Testing & eval harness for the Flyte agent plugin."
 requires-python = ">=3.10"
 dependencies = [
     "pyyaml>=6.0",

evals/report.py

@@ -22,7 +22,7 @@ def to_markdown(results: list[dict]) -> str:
     total = len(results)
     failed = [r for r in results if not r["passed"]]
     lines = [
-        f"### flyte-skills evals — {total - len(failed)}/{total} passing",
+        f"### flyte-agent-plugin evals — {total - len(failed)}/{total} passing",
         "",
         "| scenario | skill | harness | tier | pass | treat | ctrl | lift |",
         "|---|---|---|---|:--:|--:|--:|--:|",
@@ -71,14 +71,14 @@ def to_html(results: list[dict]) -> str:
         )
     passed = sum(1 for r in results if r["passed"])
     return f"""<!doctype html><meta charset=utf-8>
-<title>flyte-skills evals</title>
+<title>flyte-agent-plugin evals</title>
 <style>
  body{{font:14px/1.4 system-ui,sans-serif;margin:2rem}}
  table{{border-collapse:collapse;width:100%}}
  th,td{{border:1px solid #ddd;padding:6px 10px}}
  th{{background:#f4f4f5;text-align:left}}
 </style>
-<h1>flyte-skills evals — {passed}/{len(results)} passing</h1>
+<h1>flyte-agent-plugin evals — {passed}/{len(results)} passing</h1>
 <table>
 <thead><tr><th>scenario</th><th>skill</th><th>harness</th><th>tier</th>
 <th>pass</th><th>treatment</th><th>control</th><th>lift</th></tr></thead>

evals/scenarios/flyte-migrate-config/resources.yaml

@@ -0,0 +1,55 @@
+id: migrate-config-resources
+skill: flyte-migrate-config
+tier: trajectory
+control: true
+prompt: |
+  Migrate this Flyte 1 (flytekit) module to Flyte 2. Move the per-task image,
+  resources, and caching onto a shared TaskEnvironment. Write the result to
+  `migrated.py`. Do not run anything.
+
+  ```python
+  import flytekit
+  from flytekit import task, workflow, Resources, ImageSpec
+
+  image = ImageSpec(packages=["scikit-learn", "pandas"], python_version="3.11")
+
+  @task(container_image=image, requests=Resources(cpu="2", mem="4Gi"),
+        cache=True, cache_version="1.0")
+  def train(n: int) -> float:
+      return float(n)
+
+  @workflow
+  def main(n: int) -> float:
+      return train(n=n)
+  ```
+setup:
+  workspace: empty
+checks:
+  - kind: file_glob
+    glob: "migrated.py"
+  - kind: python_parses
+    file_glob: "*.py"
+  - kind: python_imports
+    module: flyte
+    file_glob: "*.py"
+  - kind: contains_regex
+    file_glob: "*.py"
+    pattern: "TaskEnvironment"
+  - kind: contains_regex
+    file_glob: "*.py"
+    pattern: "Image|Resources"
+  # v1 image/caching constructs must be gone.
+  - kind: not_contains_regex
+    file_glob: "*.py"
+    pattern: "ImageSpec|cache_version"
+  - kind: not_contains_regex
+    file_glob: "*.py"
+    pattern: "import flytekit"
+judge:
+  rubric: |
+    correctness: image/resources/cache moved onto a flyte.TaskEnvironment (image via
+    flyte.Image, cache="auto" or a CachePolicy replacing cache_version), train kept as
+    @env.task. skill_adherence: config declared once on the environment, not per-task.
+    completeness: no ImageSpec/cache_version/flytekit leftovers.
+  weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2}
+  pass_threshold: 0.7

evals/scenarios/flyte-migrate-config/static.yaml

@@ -0,0 +1,5 @@
+id: flyte-migrate-config-static
+skill: flyte-migrate-config
+tier: static
+# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets).
+# No agent, no LLM. Runs on every change.

evals/scenarios/flyte-migrate-control-flow/conditional.yaml

@@ -0,0 +1,63 @@
+id: migrate-control-flow-conditional
+skill: flyte-migrate-control-flow
+tier: trajectory
+control: true
+prompt: |
+  Migrate this Flyte 1 (flytekit) module that uses `conditional()` branching to
+  Flyte 2, replacing the DSL with native Python control flow. Write the result to
+  `migrated.py`. Do not run anything.
+
+  ```python
+  import flytekit
+  from flytekit import conditional
+
+  @flytekit.task
+  def is_positive(x: int) -> bool:
+      return x > 0
+
+  @flytekit.task
+  def double(x: int) -> int:
+      return x * 2
+
+  @flytekit.task
+  def negate(x: int) -> int:
+      return -x
+
+  @flytekit.workflow
+  def main(x: int) -> int:
+      return (
+          conditional("check")
+          .if_(is_positive(x=x).is_true())
+          .then(double(x=x))
+          .else_()
+          .then(negate(x=x))
+      )
+  ```
+setup:
+  workspace: empty
+checks:
+  - kind: file_glob
+    glob: "migrated.py"
+  - kind: python_parses
+    file_glob: "*.py"
+  - kind: python_imports
+    module: flyte
+    file_glob: "*.py"
+  - kind: contains_regex
+    file_glob: "*.py"
+    pattern: "\\bif\\b"
+  # v1 branching DSL must be gone.
+  - kind: not_contains_regex
+    file_glob: "*.py"
+    pattern: "conditional\\("
+  - kind: not_contains_regex
+    file_glob: "*.py"
+    pattern: "import flytekit"
+judge:
+  rubric: |
+    correctness: does migrated.py express the branch as native Python if/else
+    (double when positive, negate otherwise) inside an orchestrating @env.task?
+    skill_adherence: no conditional()/if_()/then() DSL, uses TaskEnvironment +
+    @env.task. completeness: no leftover flytekit constructs.
+  weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2}
+  pass_threshold: 0.7

evals/scenarios/flyte-migrate-control-flow/static.yaml

@@ -0,0 +1,5 @@
+id: flyte-migrate-control-flow-static
+skill: flyte-migrate-control-flow
+tier: static
+# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets).
+# No agent, no LLM. Runs on every change.

evals/scenarios/flyte-migrate-data-io/flytefile.yaml

@@ -0,0 +1,57 @@
+id: migrate-data-io-flytefile
+skill: flyte-migrate-data-io
+tier: trajectory
+control: true
+prompt: |
+  Migrate this Flyte 1 (flytekit) module to Flyte 2, converting the FlyteFile and
+  StructuredDataset I/O to the Flyte 2 equivalents. Write the result to
+  `migrated.py`. Do not run anything.
+
+  ```python
+  import flytekit
+  from flytekit import task, workflow
+  from flytekit.types.file import FlyteFile
+  from flytekit.types.structured import StructuredDataset
+
+  @task
+  def load(path: str) -> FlyteFile:
+      return FlyteFile(path)
+
+  @task
+  def to_table(f: FlyteFile) -> StructuredDataset:
+      import pandas as pd
+      return StructuredDataset(dataframe=pd.read_csv(f.path))
+
+  @workflow
+  def main(path: str) -> StructuredDataset:
+      return to_table(f=load(path=path))
+  ```
+setup:
+  workspace: empty
+checks:
+  - kind: file_glob
+    glob: "migrated.py"
+  - kind: python_parses
+    file_glob: "*.py"
+  - kind: python_imports
+    module: flyte
+    file_glob: "*.py"
+  # v2 offloaded types.
+  - kind: contains_regex
+    file_glob: "*.py"
+    pattern: "File|DataFrame"
+  # v1 types must be gone.
+  - kind: not_contains_regex
+    file_glob: "*.py"
+    pattern: "FlyteFile|StructuredDataset"
+  - kind: not_contains_regex
+    file_glob: "*.py"
+    pattern: "import flytekit"
+judge:
+  rubric: |
+    correctness: FlyteFile -> flyte.io.File and StructuredDataset -> flyte.io.DataFrame,
+    with the load -> to_table -> main flow preserved as @env.task functions.
+    skill_adherence: uses TaskEnvironment + @env.task + flyte.io types. completeness:
+    no leftover flytekit / FlyteFile / StructuredDataset references.
+  weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2}
+  pass_threshold: 0.7

evals/scenarios/flyte-migrate-data-io/static.yaml

@@ -0,0 +1,5 @@
+id: flyte-migrate-data-io-static
+skill: flyte-migrate-data-io
+tier: static
+# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets).
+# No agent, no LLM. Runs on every change.

evals/scenarios/flyte-migrate-ml/static.yaml

@@ -0,0 +1,5 @@
+id: flyte-migrate-ml-static
+skill: flyte-migrate-ml
+tier: static
+# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets).
+# No agent, no LLM. Runs on every change.

evals/scenarios/flyte-migrate-ml/train-gpu.yaml

@@ -0,0 +1,50 @@
+id: migrate-ml-train-gpu
+skill: flyte-migrate-ml
+tier: trajectory
+control: true
+prompt: |
+  Migrate this Flyte 1 (flytekit) GPU training module to Flyte 2. Write the result
+  to `migrated.py`. Do not run anything.
+
+  ```python
+  import flytekit
+  from flytekit import task, workflow, Resources, ImageSpec
+
+  image = ImageSpec(packages=["torch", "torchvision"])
+
+  @task(container_image=image, requests=Resources(cpu="4", mem="16Gi", gpu="1"))
+  def train(epochs: int) -> str:
+      import torch
+      return f"trained {epochs} epochs on {torch.cuda.device_count()} gpus"
+
+  @workflow
+  def main(epochs: int) -> str:
+      return train(epochs=epochs)
+  ```
+setup:
+  workspace: empty
+checks:
+  - kind: file_glob
+    glob: "migrated.py"
+  - kind: python_parses
+    file_glob: "*.py"
+  - kind: python_imports
+    module: flyte
+    file_glob: "*.py"
+  - kind: contains_regex
+    file_glob: "*.py"
+    pattern: "TaskEnvironment"
+  - kind: contains_regex
+    file_glob: "*.py"
+    pattern: "gpu|GPU"
+  - kind: not_contains_regex
+    file_glob: "*.py"
+    pattern: "ImageSpec|import flytekit"
+judge:
+  rubric: |
+    correctness: training task ported to @env.task with GPU resources expressed the
+    Flyte 2 way (e.g. a "T4:1"-style gpu string / flyte.Resources on the
+    TaskEnvironment), image via flyte.Image. skill_adherence: image/resources set once
+    on the environment. completeness: no ImageSpec/flytekit leftovers.
+  weights: {correctness: 0.5, skill_adherence: 0.3, completeness: 0.2}
+  pass_threshold: 0.7