[FLUX] Add FLUX inference test in CI

[wwwjn]

Model test time breakdown:

Further time breakdown load the "flux validation test" is:

• model initialization takes 2-3min (Flux model + T5-xxl encoder, CLIPs encoder)
• validation run (validation steps=default, each step validates the model, generate image at step 1,5,10): 5min30s

In this PR, following changes are being made to save time:

1. merge FLUX related test into one time to save FLUX model setup time
2. Reduce validation step to be 5, using smaller validation set for integration test

[tianyu-l]

Sorry I somehow feel this PR is doing some overkill stuff. Would it be simpler if we just provide a truncated version of prompts.txt in test assets, say 1/2 prompts on each rank? Our goal is to make CI job lighter. Users should be fine if they always have to specify prompts in a .txt file? I just feel dealing with two paths doesn't seem to be elegant.

[wwwjn]

Thanks! That would be easier. I think specify prompts in a .txt file is easier, I will remove the single prompt path. However, there are a minor bug to fix: If the number of prompts < number of ranks, some rank will get 0 prompts. During forward path of T5/clips encoder, the program will hang because FSDP is applied on encoder, and some ranks didn't run forward so all_gather will hang.

I will modify the PR to fix the bug and always using prompts.txt file.

[tianyu-l]

Sounds good! I think we can just error out early in that case, for simplicity.

[tianyu-l]

Instead of removing all of them, we can create a test asset with only small portion of it.

[tianyu-l]

using smaller validation set for integration test

where are you doing this?

also you mentioned some difficulties when combining CP with validation, is it no longer a blocker?

[wwwjn]

using smaller validation set for integration test

where are you doing this?

Here I updated the validation.steps = 5 (default is 48 , which is calculated from the validation dataset size)

also you mentioned some difficulties when combining CP with validation, is it no longer a blocker?

The error message is NCCL timeout during generating images, then I figure out it happens occasionally in both CI docker and local dev machine. So I thought it could because each rank takes different inputs thus different generation speed.

[tianyu-l]

sgtm