Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ touch lightx2v/models/schedulers/$MODEL/__init__.py
| `video_encoders/hf/{model}/vae.py` | 包装 diffusers AutoencoderKL* |
| `input_encoders/hf/{model}/text_encoder.py` | 包装 T5 / CLIP / Qwen 等 |
| `runners/{model}/{model}_runner.py` | Runner 注册(见第七步) |
| `runners/{model}/cli.py` | 任务及请求参数声明(见第七步) |
| `schedulers/{model}/{model}_scheduler.py` | Flow Matching / DDIM 等 |

---
Expand Down Expand Up @@ -227,25 +228,33 @@ def infer_with_blocks_offload(self, blocks, x, pre_infer_out):
```python
from lightx2v.models.runners.default_runner import DefaultRunner
from lightx2v.utils.registry_factory import RUNNER_REGISTER
from lightx2v.models.runners.request_fields import VIDEO_REQUEST_FIELDS

@RUNNER_REGISTER("{model_cls}")
class MyModelRunner(DefaultRunner):
def load_transformer(self):
return MyModel(self.config["model_path"], self.config, self.init_device)
def load_text_encoder(self):
return [MyTextEncoder(self.config)]
supported_request_fields_by_task = {"t2v": VIDEO_REQUEST_FIELDS}

def load_vae(self):
return None, MyVAE(self.config) # T2V 示例;有图像输入时还需 encoder
def init_scheduler(self):
self.scheduler = MyScheduler(self.config)
```

在 `lightx2v/models/runners/runner_factory.py:RUNNER_MODULES` 中添加 `model_cls → runner 模块路径`;`build_runner()` 延迟导入,CLI 的 choices 直接使用该表,不要在 `infer.py` 再添加 eager import 或重复 choices。
在 `runners/{model}/cli.py` 中声明任务和请求参数:

```python
from lightx2v.models.runners.request_args import VIDEO
from lightx2v.utils.request_spec import RequestSpec

SPECS = {"{model_cls}": RequestSpec({"t2v": VIDEO})}
```

模型专属参数直接在这个文件中通过 `Argument` 声明。`BaseRunner` 从 Spec 校验任务和字段,CLI 也从同一份声明生成参数和帮助。

在 `lightx2v/models/runners/registry.py:RUNNER_MODULES` 中添加 `model_cls → runner 模块路径`;`build_runner()` 延迟导入,CLI 的 choices 直接使用该表,不要在 `infer.py` 再添加 eager import 或重复 choices。

同时核对 `supported_request_fields_by_task`、`InputInfo` 与实际任务;上面的类仅展示关键 hook,不是可独立运行的完整模型。`configs/model_pipeline.json` 不参与当前 `build_runner()` 的原生 CLI 注册。
同时核对模型 Spec、`InputInfo` 与实际任务;上面的类仅展示关键 hook,不是可独立运行的完整模型。`configs/model_pipeline.json` 不参与当前 `build_runner()` 的原生 CLI 注册。

---

Expand Down
4 changes: 2 additions & 2 deletions .agents/skills/model-enable-intel-xpu/xpu-exec/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ XPU 示例配置在 `configs/platforms/intel_xpu/`,多卡配置在其 `dist_in
dir configs\platforms\intel_xpu\
```

当前示例 JSON 通常不含 `model_cls` / `task`。CLI 的 `--model_cls` 来自 `runner_factory.RUNNER_MODULES`,`--task` 还须通过 runner 的支持范围与请求字段校验。
当前示例 JSON 通常不含 `model_cls` / `task`。CLI 的 `--model_cls` 来自 `registry.RUNNER_MODULES`,任务和请求字段由模型 `cli.py` 中的 Spec 声明并校验。

常见任务包括 `t2v/i2v/t2i/i2i/t2av/i2av/fl2av/ref2av/i23d`;完整 CLI choices 见 `lightx2v/infer.py`,每个模型支持的子集以 runner 和匹配脚本为准。

Expand Down Expand Up @@ -114,7 +114,7 @@ pipe = LightX2VPipeline(
pipe.create_generator(config_json="configs/platforms/intel_xpu/wan_t2v_1_3.json")

pipe.generate(seed=42, prompt="a cat", save_result_path="output.mp4")
# size/num_frames 等请求覆盖仍受 runner 的 supported_request_fields_by_task 限制。
# size/num_frames 等请求覆盖须通过模型 Spec 的字段校验。
```

不使用 JSON 时,可在这一次 `create_generator()` 调用中传入 `attn_mode`、`infer_steps`、`size=(height, width)` 等实际参数。传入 `config_json` 后,除 `fps` 外的这些推理参数以文件内容为准;不要同时传两套设置并期待合并。
Expand Down
21 changes: 13 additions & 8 deletions .agents/skills/support-new-model/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Always map the upstream pipeline onto LightX2V's existing architecture:

```text
CLI/config/script
-> build_cli_inputs / build_startup_config
-> parse_cli_inputs / build_startup_config
-> runner_factory.build_runner (init_modules, optional warmup)
-> prepare_request -> InputInfo -> run_request
-> runner.run_pipeline
Expand Down Expand Up @@ -69,7 +69,10 @@ Before editing, inspect the current repo and the upstream model:
- `lightx2v/utils/input_info.py`
- `lightx2v/utils/registry_factory.py`
- `lightx2v/models/runners/runner_factory.py`
- `lightx2v/models/runners/request_fields.py`
- `lightx2v/models/runners/registry.py`
- `lightx2v/models/runners/request_args.py`
- `lightx2v/utils/request_spec.py`
- nearest family's `lightx2v/models/runners/<family>/cli.py`
- `lightx2v/models/runners/base_runner.py`
- `lightx2v/models/runners/default_runner.py`
- nearest family runner, for example `lightx2v/models/runners/wan/wan_runner.py`
Expand Down Expand Up @@ -155,8 +158,9 @@ Runner should follow the closest existing family style.

Responsibilities:

- register with `@RUNNER_REGISTER("<model_cls>")` and add its import path to `runner_factory.RUNNER_MODULES`
- declare `supported_request_fields_by_task` using `request_fields.py` sets; add `input_info_cls_by_task` or `INPUT_INFO_TYPES` entries only when needed
- register with `@RUNNER_REGISTER("<model_cls>")` and add its import path to `registry.RUNNER_MODULES`
- declare tasks and arguments in the family's `cli.py` through `SPECS`, `RequestSpec` and `Argument`; reuse common groups from `request_args.py`, and keep model-specific parameters local
- add `input_info_cls_by_task` or `INPUT_INFO_TYPES` entries only when needed
- initialize/load modules
- initialize scheduler(s)
- bind `run_input_encoder`
Expand Down Expand Up @@ -195,14 +199,15 @@ Derive cache keys, reset boundaries and token ordering from the actual autoregre
When adding `model_cls`:

- register runner with `RUNNER_REGISTER`
- add the runner module path to `lightx2v/models/runners/runner_factory.py:RUNNER_MODULES`; `build_runner()` imports that module lazily
- `infer.py` uses `choices=RUNNER_MODULES`, so do not add eager runner imports or a second model choice list there
- update task choices only if a new `--task` string is truly needed; wire its request fields and `InputInfo` as described above
- add the runner module path to `lightx2v/models/runners/registry.py:RUNNER_MODULES`; `build_runner()` imports that module lazily
- `utils/cli.py` uses `choices=RUNNER_MODULES`, so do not add eager runner imports or a second model choice list in `infer.py`
- declare new tasks in the model's Spec and wire their `InputInfo` as described above; CLI task validation and request options come from that Spec
- keep `cli.py` and its package initializer free of inference runtime imports so `--help` can run without initializing a device
- `configs/model_pipeline.json` is not consumed by the current `build_runner()` registration path; do not add entries as a native CLI registration step

If the upstream config name says `i2av` but the actual infer mode is image-to-video-action, choose one LightX2V task name and keep it consistent. Prefer semantic names in code (`i2va`) and use config filenames only for compatibility if needed.

When an error says `argument --model_cls: invalid choice`, check the model's entry in `runner_factory.RUNNER_MODULES` first. Do not rename the model to an unrelated existing `model_cls` just to pass argument parsing.
When an error says `argument --model_cls: invalid choice`, check the model's entry in `registry.RUNNER_MODULES` first. Do not rename the model to an unrelated existing `model_cls` just to pass argument parsing.

## Parity And Porting Checks

Expand Down
202 changes: 0 additions & 202 deletions configs/model_pipeline.json

This file was deleted.

File renamed without changes.
Loading
Loading