Train a LoRA for a Consistent AI Character in ComfyUI

Short answer: Train the LoRA with ComfyUI’s own Train LoRA node on FLUX.2 [klein] base 4B, using about 20 checked pictures, a learning rate of 0.00005 and 1,000 steps in one run. Then use it at a low strength — 0.35. At 1.0 the first LoRAs came out as texture with no face; the LoRA wasn’t broken, it was too strong. Everything here ran locally in Promptus on a laptop.

When a LoRA is worth it

There are three ways to keep a character the same: describe her in words, give the model one reference photo, or train a LoRA — a small add-on that teaches the model her face so you don’t need a photo every time. A LoRA is for anything that must look the same again and again: a character across a series, a brand mascot, a product, or a visual style. Only ever train on people who’ve said yes.

Step 1: the pictures

You need about 20 pictures. If you only have one, use it as a reference to make more. Here, 20 new scenes of Juno were checked one by one — is it clearly her, is the bob right, is the scar in her eyebrow visible? Nineteen passed. Adding her original photo and four character-sheet shots gave 24 in total.

One rule: don’t train on the pictures you’re going to test with, or it memorises them instead of learning her.

Step 2: train the LoRA in Promptus

Click ComfyUI in Promptus and open the training workflow. Only a few nodes matter: one loads the model, one loads your folder of pictures, one resizes them, and the Train LoRA node does the work. The settings that worked:

On an RTX 5090 Laptop GPU with 24 GB, the filmed run took 26 min 55 s and the card peaked at about 14 GB. It hasn’t been tested on a 12 GB card.

Step 3: use it — the strength box

Open a picture workflow, add a Load LoRA node right after the model, pick your file, and start the prompt with your trigger word. Then look at the strength box. It starts at 1.0, and at 1.0 there was no face. Same seed at 0.35, and she was back. The strength was chosen on one training scene before the test shots were ever run.

One side effect: the stronger the LoRA, the more she turns to face the camera — because almost all the training pictures faced the camera, and the LoRA learned that too.

The results: a fair test

The same 20 shots, the same seeds, one try each, scored with a face-match model against her reference photo (OpenCV SFace; 0.363 or above counts as the same person):

The LoRA kept her hair and jacket shot after shot, but the scar was in the right place in only about 7 of 19 shots, and she always got two earrings instead of one. One good photo won easily. The full three-way comparison is in same face, every shot: three methods tested.

Step 4: make her move with Wan 2.2

A LoRA trained for FLUX.2 — a picture model — can’t be loaded into Wan. Video LoRAs have to be trained on the video model itself. The way round it: make the shot with your LoRA, then use it as the first frame of the Wan 2.2 14B Image to Video CosyTemplate.

In the prompt, describe only the movement, not her face — she’s already in the picture. And check every clip: she looked down when asked to look up.

Should you train one?

For a character in just a few scenes, use one photo. For a long film or a campaign that runs for months, a LoRA saves attaching a photo every time. Train picture LoRAs on picture models and video LoRAs on video models.

Models and licences

• FLUX.2 [klein] base 4B (Apache 2.0)
• Wan 2.2 (Apache 2.0)
• OpenCV YuNet (MIT) and SFace (Apache 2.0)

Check each model’s licence before you sell anything you make with it. Download Promptus to train locally with ComfyUI built in.