[feat]Add strength in flux_fill pipeline (denoising strength for fluxfill) #10603

Suprhimp · 2025-01-17T10:09:23Z

What does this PR do?

allows the fluxfill pipeline to have a denoising strength (denoising strength).

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

I use fluxfill pipeline to edit image or outpatin. Yes it works very well but I don't agree Flux Fill pipeline does not require strength as an input like regular inpainting pipelines this line in the docs.

When remove object in the image, We need denoising strength I guess. without denoising strength I can't get clean image that remove object that I want.
let me gives you an example I tested.

this is mask and original image that I want to edit.

And this is the result with denoising strength (0.6 with revised pipeline) and none(default pipeline)

I did lama inpaint to remove more clearly in both case but output is different.

As you can see with denoising strength we can have more control with the image quality. So I think there is no reason to not use denoising strength in fluxfill pipeline

I changed the fluxfill pipeline code with reference sdxl inpaint pipeline.
So, I think there would be things that need to be fixed.

Thanks for reading :)

hlky

I've left some comments. However, we should note that Flux Fill is intended to fill areas based on a text description and not for object removal.

hlky · 2025-01-17T10:35:20Z

src/diffusers/pipelines/flux/pipeline_flux_fill.py

        self.mask_processor = VaeImageProcessor(
            vae_scale_factor=self.vae_scale_factor * 2,
-            vae_latent_channels=latent_channels,
+            vae_latent_channels=self.vae.config.latent_channels,


We need to use latent_channels here not the config directly. This allows pipelines to be used without the component e.g. FluxFillPipeline(vae=None, ...).

I didn't know that, Thanks, I changed. :)

hlky · 2025-01-17T10:36:05Z

src/diffusers/pipelines/flux/pipeline_flux_fill.py

@@ -627,6 +659,8 @@ def disable_vae_tiling(self):
    # Copied from diffusers.pipelines.flux.pipeline_flux.FluxPipeline.prepare_latents


Can we copy from FluxImg2ImgPipeline.prepare_latents or FluxInpaintPipeline.prepare_latents?

Sure , That would be more clean. Thanks for the review :)

hlky · 2025-01-17T10:39:30Z

src/diffusers/pipelines/flux/pipeline_flux_fill.py

@@ -809,6 +866,10 @@ def __call__(
        self._joint_attention_kwargs = joint_attention_kwargs
        self._interrupt = False

+        original_image = image


yep :) Maby that came from sdxl inpaint pipeline, but it is not used in this pipeline

src/diffusers/pipelines/flux/pipeline_flux_fill.py

hlky · 2025-01-17T10:41:46Z

src/diffusers/pipelines/flux/pipeline_flux_fill.py

@@ -855,13 +952,13 @@ def __call__(
        if masked_image_latents is not None:


Above # 6. Prepare mask and masked image latents.

Suprhimp · 2025-01-17T10:48:12Z

Thankyou for the fast and kind review, I'll revise it and test :)

Co-authored-by: hlky <[email protected]>

Suprhimp · 2025-01-17T11:06:36Z

I've left some comments. However, we should note that Flux Fill is intended to fill areas based on a text description and not for object removal.

Hmm, Maby I missed the intention than ;)
Although I think we can use fluxfill with more controllable, with denosing strength:)

@hlky Thanks for the review again, I changed the code that you've reviewed.

asomoza · 2025-01-17T13:08:12Z

Hi, you're saying you used lama to remove the object before doing this? If that's what you did, you're just essentially using this pipeline as an img2img one and the model as just a refiner.

The original fill model was trained to do this without the need of changing the strength and this is from the original implementation, I'm not saying that what you're doing is wrong though, people used redux in a more different and creative way and it works alright but I want to understand your use case since I haven't encountered the need to lower the strength to make it work.

Did you try what you're doing here with just the regular Flux model and the img2img pipeline?

Suprhimp · 2025-01-17T13:27:02Z

Did you try what you're doing here with just the regular Flux model and the img2img pipeline?

Thanks for your response :)

I tried with sdxl model with img2img model. But not flux image2image.
And yep I understand what you said, Maby I'm just using the fluxfill pipeline as an refiner.

As I use this pipeline an image edit pipeline, I just want to use the single pipeline as an object adder or outpainter but also object remover.

If I have to use my case with flux img2img pipeline I can do that, but as you know flux model is quite large.
So I'd like to use it as refiner but also object adder or outpainter etc.

If I can use fluxfill pipeline with strength parameter, I don't need any other pipeline :)

@asomoza

hlky · 2025-01-17T13:34:01Z

@Suprhimp To confirm, you're using lama cleaner to remove the object first?

Suprhimp · 2025-01-17T13:38:06Z

To confirm, you're using lama cleaner to remove the object first?

@hlky Yep I used lamacleaner before both case (my pipeline, and default pipeline)

ukaprch · 2025-01-18T13:35:24Z

lama with the refiner is very good at removing objects. I thought the whole point of Flux fill was to replace or insert an object. Big difference there.

Suprhimp · 2025-01-20T01:53:44Z

Hmm I think even we use fluxfill without lamainpainter to replace or insert an object, There will be certian point to need stength. Maby, keep the original color masked but change object, or change object but referenced based image in masked area.

You guys think it is unecessary to use strength referer to the orignal masked area in replace or insert an object?

asomoza · 2025-01-22T03:36:27Z

There will be certian point to need stength. Maby, keep the original color masked but change object, or change object but referenced based image in masked area.

This are valid use cases but that it's again just a nasked img2img, in fact, if you want to preserve as much of the original image intact, you should always do it with img2img instead of this pipeline, and just replace the new part in the original image.

Still I think they're valid points and also the one that you don't want to switch models because they're really big, but this will make the pipeline more complex, @yiyixuxu WDYT?

github-actions · 2025-02-16T15:02:24Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

jonluca · 2025-03-05T03:31:26Z

This would be a welcome addition, would be good to be able to use strength with flux fill

asomoza · 2025-03-06T14:05:16Z

This would be a welcome addition, would be good to be able to use strength with flux fill

@jonluca we're still waiting for @yiyixuxu feedback here, but in the meantime, do you have some demo images of using the strength with the vanilla flux fill pipeline? I mean without using it as a refiner only or with another inpainting model like lama cleaner.

yiyixuxu · 2025-03-06T20:11:33Z

@asomoza I think it's ok to support if you think the use case is meaningful :)

lvwilson · 2025-04-01T03:48:28Z

I +1 this use case.

asomoza · 2025-04-02T18:12:53Z

sure, I think it's worth it because of the VRAM usage of Flux, I'll do some quick tests as soon as possible. In the meantime @Suprhimp would you mind resolving the conflicts or do you prefer if I take it over?

Suprhimp · 2025-04-03T02:04:00Z

I'll try right away:) @asomoza

Suprhimp · 2025-04-04T00:19:22Z

@asomoza I fixed conflict and I did test with my code, it seems work well to me

HuggingFaceDocBuilderDev · 2025-04-04T08:54:36Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

asomoza · 2025-04-04T08:55:15Z

@bot /style

github-actions · 2025-04-04T08:56:04Z

Style fixes have been applied. View the workflow run here.

asomoza

I took the liberty to fix the bad name in the prepare_latents copy, hope you don't mind, but now we have an issue with it.

asomoza · 2025-04-04T09:23:18Z

src/diffusers/pipelines/flux/pipeline_flux_fill.py

+        image = image.to(device=device, dtype=dtype)
+        image_latents = self._encode_vae_image(image=image, generator=generator)
+        if batch_size > image_latents.shape[0] and batch_size % image_latents.shape[0] == 0:
+            # expand init_latents for batch_size
+            additional_image_per_prompt = batch_size // image_latents.shape[0]
+            image_latents = torch.cat([image_latents] * additional_image_per_prompt, dim=0)
+        elif batch_size > image_latents.shape[0] and batch_size % image_latents.shape[0] != 0:
            raise ValueError(
-                f"You have passed a list of generators of length {len(generator)}, but requested an effective batch"
-                f" size of {batch_size}. Make sure the batch size matches the length of the generators."
+                f"Cannot duplicate `image` of batch size {image_latents.shape[0]} to {batch_size} text prompts."


this part it's not the same as the prepare_latents in FluxImg2ImgPipeline but you're commenting that this function was copied from it.

Thanks for the detail review, and yes, since FluxImg2ImgPipeline was updated 2 weeks ago, It has to be changed.
So I updated prepare_latents function as same with in FluxImg2ImgPipeline.

And also I tested with my code, it works well. (honestly, it works better than before i think, lol)

src/diffusers/pipelines/flux/pipeline_flux_fill.py

asomoza · 2025-04-04T14:21:15Z

thanks a lot! Failing tests are not related to this PR

Suprhimp · 2025-04-05T00:17:54Z

Thank you! I wanted to revise the last part as well, but I couldn't respond because I couldn't use the computer at that time. But I wanted to merge this PR quickly, so I asked for help.

[feat]add strength in flux_fill pipeline

14c452a

hlky reviewed Jan 17, 2025

View reviewed changes

Suprhimp and others added 4 commits January 17, 2025 19:55

Update src/diffusers/pipelines/flux/pipeline_flux_fill.py

a7e1501

Co-authored-by: hlky <[email protected]>

Update src/diffusers/pipelines/flux/pipeline_flux_fill.py

cf60e52

Co-authored-by: hlky <[email protected]>

Update src/diffusers/pipelines/flux/pipeline_flux_fill.py

25fa97c

Co-authored-by: hlky <[email protected]>

[refactor] refactor after review

5d6b78c

[fix] change comment

3a1ea2e

github-actions bot added the stale Issues that haven't received updates label Feb 16, 2025

github-actions bot removed the stale Issues that haven't received updates label Mar 5, 2025

Merge branch 'main' into main

a3acf58

Merge branch 'main' into main

f42f90f

Apply style fixes

a6737f1

asomoza approved these changes Apr 4, 2025

View reviewed changes

asomoza added 2 commits April 4, 2025 11:06

empty

c489b57

fix

e87c9eb

asomoza requested changes Apr 4, 2025

View reviewed changes

asomoza added the close-to-merge label Apr 4, 2025

Suprhimp and others added 2 commits April 4, 2025 20:03

update prepare_latents from flux.img2img pipeline

cb43412

style

f0fac62

asomoza approved these changes Apr 4, 2025

View reviewed changes

hlky approved these changes Apr 4, 2025

View reviewed changes

src/diffusers/pipelines/flux/pipeline_flux_fill.py Outdated Show resolved Hide resolved

hlky and others added 2 commits April 4, 2025 14:18

Update src/diffusers/pipelines/flux/pipeline_flux_fill.py

a0ffed1

Merge branch 'main' into main

b448707

asomoza merged commit 94f2c48 into huggingface:main Apr 4, 2025
10 of 12 checks passed

asomoza removed the close-to-merge label Apr 4, 2025

		@@ -627,6 +659,8 @@ def disable_vae_tiling(self):
		# Copied from diffusers.pipelines.flux.pipeline_flux.FluxPipeline.prepare_latents

		@@ -855,13 +952,13 @@ def __call__(
		if masked_image_latents is not None:

[feat]Add strength in flux_fill pipeline (denoising strength for fluxfill) #10603

[feat]Add strength in flux_fill pipeline (denoising strength for fluxfill) #10603

Uh oh!

Conversation

Suprhimp commented Jan 17, 2025

What does this PR do?

Before submitting

Who can review?

Uh oh!

hlky left a comment

Choose a reason for hiding this comment

Uh oh!

hlky Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

Suprhimp Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

hlky Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

Suprhimp Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

hlky Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

Suprhimp Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hlky Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

Suprhimp commented Jan 17, 2025

Uh oh!

Suprhimp commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asomoza commented Jan 17, 2025

Uh oh!

Suprhimp commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hlky commented Jan 17, 2025

Uh oh!

Suprhimp commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ukaprch commented Jan 18, 2025

Uh oh!

Suprhimp commented Jan 20, 2025

Uh oh!

asomoza commented Jan 22, 2025

Uh oh!

github-actions bot commented Feb 16, 2025

Uh oh!

jonluca commented Mar 5, 2025

Uh oh!

asomoza commented Mar 6, 2025

Uh oh!

yiyixuxu commented Mar 6, 2025

Uh oh!

lvwilson commented Apr 1, 2025

Uh oh!

asomoza commented Apr 2, 2025

Uh oh!

Suprhimp commented Apr 3, 2025

Uh oh!

Suprhimp commented Apr 4, 2025

Uh oh!

HuggingFaceDocBuilderDev commented Apr 4, 2025

Uh oh!

asomoza commented Apr 4, 2025

Uh oh!

github-actions bot commented Apr 4, 2025

Uh oh!

asomoza left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Suprhimp commented Jan 17, 2025 •

edited

Loading

Suprhimp commented Jan 17, 2025 •

edited

Loading

Suprhimp commented Jan 17, 2025 •

edited

Loading

asomoza left a comment •

edited

Loading

asomoza Apr 4, 2025 •

edited

Loading

Suprhimp Apr 4, 2025 •

edited

Loading

asomoza commented Apr 4, 2025 •

edited

Loading