Creating illustrations using NovelAI Diffusion Vibe Transfer (@kuroito_tsumugi) (EN)
Using their own drawn art with vibe transfer, (@kuroito_tsumugi) shows off their process of creating new images. Don’t forget to follow them! Thank you, Kuroito Tsumugi, for allowing us to use your original article on our Medium blog. You may find the original article here https://note.com/kuroito_tsumugi/n/n6eb6ecb0e2e6
This is a illustration test using the “Vibe Transfer” function newly implemented in NovelAI. The device I used in this test is an iPad Air.
Making Illustrations
Illustration Generation (NAI)
Here are the images I used in this project.
Since the underlying theme of this experiment was “Can I incorporate elements from my own pictures?” I prepared digital illustrations that I personally drew myself.
If my self-made picture on the left is set into the Vibe Transfer in NovelAI (hereafter referred to as NAI), the following image can be generated.
The character’s appearance reflects Vibe Transfer’s reference image characteristics. The coloring is closer to an AI rendering than to a hand-made drawing (reference strength values range from 0.4 to 0.57, with a range of 0.57 to 0.4). The default value is lower than 0.6, so the percentages may be closer to AI.
Vibe Transfer also allows prompt input, so for this image I gave prompt instructions to turn the body sideways and to generate a pattern background. It seems that the prompts can be used to fine-tune the generated content while incorporating elements from the reference image.
However, depending on the image and settings Vibe Transfer can have difficulty generating images that deviate significantly from the reference image — for example, generating the whole body from a bust-up image. In order to generate a whole body, it is necessary to give strong instructions (for example, enter {{{full body}}} in the prompt & set the reference strength of vibe transfer to less than 0.5).
So now I experimented using Vibe Transfer and image2image (i2i) together. Here are the results generated by setting my own drawing as the reference image for Vibe Transfer and an AI illustration (cowboy shot with background) for i2i.
The characteristics are
Character Head: Vibe Transfer reference image
Body and background: i2i original image
Vibe Transfer and i2i can be used together to overwrite the i2i original illustration with features (appearance, texture, etc.) of the Vibe Transfer reference image.
I found the coloring was a little flat, presumably because it was pulled by Vibe Transfer’s own picture, so I set the generated illustration as the reference image for Vibe Transfer and then re-generated it.
The layering of Vibe Transfer has blended the picture overall into a rich AI rendering. Conversely, the elements of the self-made picture have faded considerably, and while the character appearance is retained, the original look of the coloring and paleness is barely perceptible.
If you want to preserve the atmosphere of the original picture, it may be better to avoid overlapping Vibe Transfer images and just put higher values in the extraction information and reference strength of Vibe Transfer.
As a result, my own illustration elements were diminished, but I personally liked the illustration used above, so I decided to use it as it is. The NAI’s Inpaint function is used to correct legs/feet, etc. and then manual corrections are made to finalize the piece.
Additions, Corrections, and Finishing Touches（ibisPaint X)
I used the application ibisPaint X to edit the image.
Since a full description of the modifications and alterations would be overwhelming in volume, I will only describe excerpts of points that I was more keenly aware of and that I wanted to document.
◆Facial Modifications
I felt that the face was not sufficiently impressive in the image before modifications, so I fine-tuned it so that the entire character stands out while focusing on the face area.
For hair and skin, I added reddish color by multiplication and overlay. For the eye color, I put yellow and pink in it to make it stand out by using different colors from the original, all while keeping a similar impression close to my own artwork.
◆Hand Modifications
The parts that overlap were selected from the image before editing, then copied to a layer, and then modified. To avoid interference when adding shadows or making corrections to each part, I often manage my work by separating the layers into separate layers for each individual part, instead of working on just a single layer.
◆Other modifications
・Added a white line to the character outline.
・Adjustment of background brightness (Screen and Lighten layer)
・Minor corrections to skirt, legs, and background (Special Pen > Liquify Pen, Perspective Form, etc.)
I wanted to retain the atmosphere of the generated AI illustrations as much as possible, so I did not try to make them look like my own artwork, and only corrected or modified noticeable defects.
I also added a white line around the character to make it look as if the character and background are separate. AI illustrations rarely show white lines around a character unless a prompt is intentionally included to emphasize the outline.
If you could sense a unique hand-drawn look in this AI illustration, I am happy to say that is exactly the look I was aiming for.
Impressions and Optional Illustration Creation:
I think the Vibe Transfer function has made it easier to explore expressions and methods of generation that were previously out of reach.
Below I listed the ones that I felt would be most beneficial to someone like me.
◆Efficiency
・Character creation by synthesizing parts
・Character creation with complex shapes that are normally difficult to reproduce.
・Mass production of characters with the same, or similar appearance
I think this particular test run falls under the first category. The character head is made using Vibe Transfer, and the other parts are created using prompts and i2i illustrations.
Personally, I find the second point to be a great benefit as well, and I appreciate the fact that Vibe Transfer makes it easier to generate characters that are difficult to reproduce with prompts and i2i alone.
Some illustrations that had been originally generated on a text2image basis for NAI v1-v2 had become difficult to generate through NAI v3 text2image, so we were glad to be able to generate them alternatively with Vibe Transfer.
When trying to intentionally synthesize different objects on a prompt basis (e.g., dress, wings, liquefied hair, etc.), the input prompts would tend to grow convoluted and overly long, and the generation success rate was also low. Vibe Transfer made generation easier by extracting features from reference images, as well as greatly reducing the amount of prompt input, so I had more free energy to devote to directing and improving the quality of the picture.
◆Merits
・Generation of illustrations by incorporating elements of your own work
・Generation of new picture styles by combining patterned materials and different types of pictures
Personally, I see a lot of potential in the first point, that it has become easier to achieve the creation of illustrations that incorporate my own pictures.
This image is an example of an illustration I made when I was experimenting with the AI to see if I could reproduce my original character. On the left is a text2image illustration I produced last year, and on the right is an illustration I generated by referencing my own digital drawings using Vibe Transfer. If you are familiar with AI illustrations, the right side of the illustration looks quite unconventional.
The AI illustration created using Vibe Transfer has the aspect of the blonde, sparkling texture that I liked when I was drawing this character. I’ve felt I’ve had difficulties with expressions or fixed generations that deviate from the latest and popular style of high quality illustrations or photorealism when using prompt-based techniques alone. So I am glad to see the emergence of a technology like this that brings us one step closer to what we want to depict and what we are striving to create.
I was impressed by the Vibe Transfer function and see it as a technology that expands the range of creative expression exponentially. I would like to continue to research what kind of illustrations I can create using this tool.
