[1.1.443] IP-Adapter CLIP mask and ip-adapter-auto preprocessor

[huchenlei]

CLIP mask

The CLIP mask allows you specify part of image that will be ignored by CLIP preprocessor. The feature idea comes from https://github.com/cubiq/ComfyUI_IPAdapter_plus.

Here is an example:

No Mask

With Mask

Infotext

skull sheild, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1714864127, Size: 512x512, Model hash: b42b09ff12, Model: cetusMix_v4, Clip skip: 2, ControlNet 0: "Module: ip-adapter_clip_h, Model: ip-adapter-plus_sd15 [836b5c2e], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", Version: v1.8.0-273-g8687163f

ip-adapter-auto preprocessor

Previously there are many ip-adapter preprocessors. User often struggle to pick the correct one. Now we have ip-adapter-auto preprocessor that automatically pick the correct preprocessor for you.

You should see following log line in your console:
2024-03-29 23:09:19,001 - ControlNet - INFO - ip-adapter-auto => ip-adapter_clip_g that indicates ip-adapter-auto is getting mapped to the actual preprocessor.

[dermesut]

auto preprocessor:
although i can see it in control_types module_list (http://127.0.0.1:7860/docs#/default/control_types_controlnet_control_types_get):
{
"control_types": {
"All": {
"module_list": [
"none",
"invert (from white bg & black line)",
"ip-adapter-auto",
...

it doesn't pop up in module_list (http://127.0.0.1:7860/docs#/default/module_list_controlnet_module_list_get):

...
{
"module_list": [
"revision_clipvision",
"revision_ignore_prompt",
"ip-adapter_clip_sd15",
"ip-adapter_clip_sdxl_plus_vith",
"ip-adapter_clip_sdxl",
"ip-adapter_face_id",
"ip-adapter_face_id_plus",
"instant_id_face_keypoints",
"instant_id_face_embedding",
"color",
"pidinet",
...

[huchenlei]

Fixed now. See

sd-webui-controlnet/tests/web_api/modules_test.py

Line 31 in e90491a

"ip-adapter-auto",

[gohan2091]

How do I get ip-adapter-auto preprocessor in Forge?

[huchenlei]

Currently it is not supported in forge. 😞