The inpainting step - keyframes as storyboard for prompting corrections

[steinhaug]

I have been struggling with this potentially very fun idea for some hours now, however the inpainting does not really work atleast not to my confort. So I started looking and searching abit, modifying and testing. Wee... arn't we all :)

I noticed people mentioning controlnet, which would be a great option - that is - given there was a logical and easy way of implementing the whole thing. So what am I wondering about? Well, the cell called "generate frames/video", the one that renders the images for the video.

My question is this:

Looking at the cell at "generate video" there is a pipe() that creates the image with prompt amd such, you then crop and create the frames thereafter. Does the system work in a way here that say I use 15 outpainting steps an 10 interpol frames (150 extra frames'ish). If I inject myself into the creation of those 15 frames (the creation of images[0]), will the rest of the video be generated correctly or are there more I need to take into consideration.

If so - think of this. In my opinion this would be a something.

An animation will have certain keyframes, that is the executions that triggers the pipe() function. An extra cell should be created before the existsing one, this one will ONLY render the keyframes or in this example from above 15 frames. The 15 frames could be displayed as a strip with mention of what prompt and neg used. For each frame it should be possible to edit the negative prompt (an extra array of text values, which corresponds to the iterated keyframe. When the cell is run again theese values could if present be appended as negative prompts. This way one would have direct control for each keyframe. Same would be for the promt, an array for each keyframe here being appended next time.

Also this "strip" should have a box superimposed on the image so that the outpainted area is clearly visible - I am under the impression that with a 128x128 mask this area would always be unabled for correction anyways - only the area outside.

Functions: clear append prompts, clear negative append prompts, adding append prompts on execution if exist

What I havnt considered is if the prompts are mutating, in that case what do I need to know ?

I notice that while writing this I probably would be more helpful for myself by doing some coding and just trying this out instead, after all not that much code to fiddle around with here...