When height and width change, the inference speed will significantly slow down.

[serend1p1ty]

Whenever height / width change, pipe.prepare_run needs to be re-executed, which is very time-consuming. Is there a better approach?

[feifeibear]

Which script are you using？

[serend1p1ty]

It seems that xdit requires the resolution used in prepare_run to be the same as the resolution in the actual call. If prepare_run uses 1152x1152, but the actual call uses 1056x1056, it will be very slow.

[serend1p1ty]

@feifeibear
example/run.sh

...
TASK_ARGS="--height 1152 --width 1152 --no_use_resolution_binning"

N_GPUS=6
PARALLEL_ARGS="--ulysses_degree 6"

COMPILE_FLAG="--use_torch_compile"
...

flux_example.py

  output = pipe(
      height=1152,
      width=1152,
      prompt=input_config.prompt,
      num_inference_steps=input_config.num_inference_steps,
      output_type=input_config.output_type,
      max_sequence_length=256,
      guidance_scale=0.0,
      generator=torch.Generator(device="cuda").manual_seed(input_config.seed),
  )

It taken 5s. If we modify flux_example.py as following:

  output = pipe(
      height=1056,
      width=1056,
      prompt=input_config.prompt,
      num_inference_steps=input_config.num_inference_steps,
      output_type=input_config.output_type,
      max_sequence_length=256,
      guidance_scale=0.0,
      generator=torch.Generator(device="cuda").manual_seed(input_config.seed),
  )

It taken 20s.

[feifeibear]

I see, you did not prepare run the correct image used for inference. You can run multiple inference and see if it still slow after the first run.

[serend1p1ty]

@feifeibear Thanks for your response.
Multiple inference can solve this problem. But we facing a new problem, if users submit tasks at different resolutions every time, how should we handle it

[feifeibear]

Have you used the torch compile option?

[serend1p1ty]

Have you used the torch compile option?

yes

[serend1p1ty]

@feifeibear How to deal with the problem of frequent resolution changes? Not use torch compile option?

[feifeibear]

dynamic shape of torch.compile is a well-known challenge. We will investigate the problem and see if we can find some good solutions. Tell us if you find some good ideas.