-
Notifications
You must be signed in to change notification settings - Fork 6.1k
[modular diffusers] more refactor #11235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
* cfg; slg; pag; sdxl without controlnet * support sdxl controlnet * support controlnet union * update * update * cfg zero* * use unwrap_module for torch compiled modules * remove guider kwargs * remove commented code * remove old guider * fix slg bug * remove debug print * autoguidance * smoothed energy guidance * add note about seg * tangential cfg * cfg plus plus * support cfgpp in ddim * apply review suggestions * refactor * rename enable/disable * remove cfg++ for now * rename do_classifier_free_guidance->prepare_unconditional_embeds * remove unused
* cfg; slg; pag; sdxl without controlnet --------- Co-authored-by: Aryan <[email protected]>
…oisee step to accept aditional contrlnet kwargs
…g *_controlnet_kwargs
…peline, and update so that intermediates_outputs are only new variables
…_from_config to just create and load method
…y if not specific names) 2. update acceept create spec 3. move the updte _componeent_spec logic outside register_components to each method that create/update the component: __init__/update/load
* update * update
…odel_name_or_path
* save_pretrained for serializing config. * remove pushtohub * diffusers-cli rough --------- Co-authored-by: YiYi Xu <[email protected]>
|
||
return block_cls(**block_kwargs) | ||
|
||
def init_pipeline(self, pretrained_model_name_or_path: Optional[Union[str, os.PathLike]] = None, component_manager: Optional[ComponentsManager] = None, collection: Optional[str] = None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe to_pipeline
is a better name? in terms of indicating this returns a new object @DN6
) | ||
|
||
|
||
class ModularPipeline: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did not completely reverting to the initial design, for now
we now have a ModularPipeline
that contains loader and pipeline blocks; at run time it prepares inputs and call pipeline blocks
the initial design, we have a pipeline
that handles both loading and runtime (consistent with our DiffusionPipeine
). I had felt that, with that design, the relationship between the Pipeline and pipeline blocks is a little bit circular: pipeline contains pipeline blocks and is also an input to the pipeline blocks. i.e. at runtime, you would have to pass self
down to blocks so that each pipeline block would have access to the global components
class ModularPipeline
def __call__(self, prompt, ...):
...
pipeline_componenets, pipeline_state = self.blocks(self, pipeline_state)
it is not a huge deal, just a slightly struggle to wrap my head around. so I ended up with the current design for now. It feels more intuitive to me, but we have one more abstraction. From here we can very easily to switch back to the initial design as well, just some renaming and move around some methods so I'd keep the discussion open :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ohhm actually I think i can just pass a dictionary contains all the components to the pipeline blocks, so that's not a problem at all. I think it is easier to just have ModularPipeline also handle loading this way then, will refactor
continue #9672 here
Getting started with Modular diffusers
Guides on the basics
PipelineBlock
: notebookSequentialPipelineBlocks
: notebookLoopSequentialPipelineBlocks
: notebookComponentSpec
(coming soon, you can read here for now [modular diffusers] introducing ModularLoader #11462)ModularLoader
(coming soon, [modular diffusers] introducing ModularLoader #11462)Components Manager
AutoPipeline
Modular Repo
see an example of modular repo here https://huggingface.co/YiYiXu/modular-diffdiff/tree/main
it includes
remote pipeline blocks
you can load a pipeline block from hub with one line of code and mix-and-match it into your workflow
You can also load it directly into a
ModularNode
- diffusers abstraction to create UI nodesit comes with the node definition too
you can use it to make a mellon node like this
ModularNode is platform-agnostic, it will work with both comfy and mellon or whatever UI platform we support in the future. It will save a default node config, allow you to make changes and automatically convert it to usable module map in different platform (this is TO-DO but very doable!)
see an example of what we curreently have here
modular_model_index.json
You can pack all the relevant checkpoint for your workflow in
modular_model_index.json
, you can reference subfolders in other repos; it also work beyound diffusers and transformers, as long as it has afrom_pretrained
method that accept the standard loading arguments, subfolder, variance etc, for example, we used image_gen_aux here https://huggingface.co/YiYiXu/modular-depth-block/blob/main/block.pyTODO
__init__
of pipeline block, no longer do theself.component["vae"] = None
thingModularPipeline
, replaced it withModularPielineMixin
:- it is the base class for
PipelineBlock
,SequentialPipelines
andAutoPipelines
- it provides a
set_loader
method that creates a ModularLoader based on itsexpected_componeents
- it provides
run()
method to invoke the pipeline call (you can use it likeimages = node.run(prompt=...)
)LoopSequentialBlocks
: explore ways to make the denoise loop itself modular, i.e you can add/remove blocks inside the loop ([Modular diffusers] more refactors #11484)ModularLoader
] support partial loading/save for model componentsModularLoader
] should be able work with model_index.json tooDocumentation
_repr_
of various pipeline blocks and will further simplify, the goal is to make_repr_
concise and simple while move more detailed information onto@doc
propertyTesting
Custom code sharing
testing under this PR https://huggingface.co/YiYiXu/modular-diffdiff
UI Nodes
faster test script
(Click to expand)
slower testing script
(Click to expand)