![]() When prompted to generate “a newspaper and a cup of vanilla latte on a table in a coffee shop”, the model never actually generated any newspapers in my experiments.Some funny weird behaviors I observed with regards to the prompts:.Num_inference_steps = 50, guidance_scale = 10).images Mask_image=mask, negative_prompt=negative_prompt, Result = pipe(prompt=prompt, image=inp_img, inner_image=inner_image, Inp_img = my_input_image # loaded with PIL.Image Generating selected image areas in an image-to-image fashionĪlternatively, the generated image can be created in an image-to-image fashion.įor this, I adapted an example from the huggingface/diffusers repository, along the lines of: from diffusers import DiffusionPipeline Num_inference_steps = 50, guidance_scale = 11).images Result = pipe(prompt, image = inp_img, mask_image = mask, negative_prompt=negative_prompt, Inp_img = square_padding(init_image) # my own function, init_image is loaded with PIL.Image Using the above model, I was able to generate images with code like: Pipe.enable_attention_slicing() # to save some gpu memory in exchange for a small speed decreaseīefore applying the models, I resized and square-padded all images to 512×512 pixels (I saw the recommendation for square-padding in someone else’s stable diffusion inpainting code, I don’t remember where exactly, and didn’t do any experiments without square-padding). "stabilityai/stable-diffusion-2-inpainting", Here is a corresponding code snippet to download and initiate the pre-trained models and other components of the diffusion pipeline: The chosen image areas can be generated from scratch, in which case I used the stable diffusion v2 inpainting model.Generating the selected image areas based on a text prompt “from scratch” Mask = omarray(np.uint8(mask)).convert('RGB') Because I didn’t want to stress about it, I simply guessed by eye rectangular image areas to be masked, for instance as follows (note that I used somewhat different masks for different text prompts):.However, first, one has to define an image mask: Either replace/transform the boy, or replace/transform the drawing that he is holding. There are two clear ways in which inpainting could be applied to the image I started with (top left in the collage above). So, I’m providing some relevant code snippets here. I learned a lot from great tutorials about stable diffusion such as the FastAI notebook “Stable Diffusion Deep Dive”, but I haven’t specifically seen examples of inpainting so far (though I haven’t looked :stuck_out_tongue:). Immediately I was (and still am) positively surprised by how easy and pleasant the developers made it to use stable diffusion via the Huggingface diffusers library in Python.īut I haven’t looked at inpainting techniques until today. I already had played around a few times with image generation with stable diffusion in Python, and with textual inversion for representation of a specific artistic style. That’s a photo I took of my son during breakfast at a restaurant this morning, and he found it absolutely hilarious how we can drastically modify it with the computer – the text prompts we used were based on his suggestions to a large part. ![]() The top left panel shows the original (real) image. You can see some of my results in the collage above. Taking advantage of the driving free days, I and my 4-year old son had some great fun with the open-source stable diffusion models in particular, the Text-Guided Image Inpainting techniques.īasically, inpainting allows you to replace or transform image areas of your choice with something else AI-generated based on a text prompt. Happy New Year!!! :fireworks: I am currently driving with my family coast-to-coast on a road trip through the United States, but for New Year’s Eve and New Year’s Day we stayed in one place.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |