FluxImg2ImgPipeline on Flux-1 Schnell image to image output troubles

[Dannyvannnn]

Hi im trying to use the image to image pipeline for Flux-1 Schnell to do corrections on images (e.g changing clothes color, hair color, eye color etc.) However it seems like Flux-1 schnell cannot properly do such a thing no matter what prompts i tried. I have tried other models such as SD2 or SDXL and they manage to make the corrections on the image, even if the quality of the output may not be very good. I want to ask how can i edit the code/parameters/prompting style/etc. in order to achieve my requirements.

First image of each row is the input image, difference between first screenshot and second screenshot is increasing strength from 0.7 to 0.8, which appears to be the threshold for the change. Code is copied from the FluxPipeline page on HuggingFace

[ukaprch]

Have you considered using the inpaint pipeline masking the areas you wish to change or fill? Then your prompt would be more successful in achieving your goals.

[Dannyvannnn]

Hi! yes i've considered using the inpaint pipeline for this project if i want more accurate results, However i would like to test the Img2Img pipeline and its limits and possible use cases as inpainting takes more work to integrate into a webapp project. So far I am unsure on how to properly use the Img2Img pipeline as from my testing, it either follows the input image too strictly, or it does not follow the image at all.

Here is a clearer example of what i mean.

My Expectation: Even without specifying the desired art style (anime style), the pipeline should generate an anime art style image with the due to the input image. At lower strength levels, the model should follow the base image (e.g changing the orange sweater of the image to a green sweater). At higher strength levels, there can be more variety in for e.g poses or accessories, but there should still be an anime style girl in a green sweater.

Reality: At lower strength levels, my prompts appear to not do anything, the model only appears to make minor edits to the base image. At higher strength levels, it does not take my input image into consideration at all and outputs what it understands from the text prompt

Am i misunderstanding how the Img2Img pipeline works? is there a set of examples where i can properly see how the Flux pipeline's Img2Img is intended to work?

In comparison with SDXL-Turbo (also a timestep-distilled model similar to Flux-1 Schnell), the results of which can be seen below, even though the quality of the edit may not be good, I can still see attempts by the model to follow the prompt.

Other closed-source models also are able to follow similar prompts with outputs closer to what i am imagining Flux's Img2Img should be able to output