Lora in your pocket: Part 1

TLDR: I was thinking about how everyone has already a crap ton of data of themselves, in the form of our photo rolls/selfies etc. I wondered if this alone could already provide us with a good starting off point to generate a character likeness lora. I used myself as the first testing dummy/subject.

Tools: ComfyUI, Musubi-Tuner, Ai-Toolkit, Nano Banana

So I first, pretty lazily scrolled through my camera roll and picked out a couple of photos that could work, you will notice too that above there’s actually a couple of images that has multi-views. In fact i’d reckon most people’s selfies on their phones are probably all gonna be from the same general angel so the idea for now is that AI would probably work as long as our dataset is “good-enough” so what if we use the photos from my camera roll as a starting point and use nano banana to fill in the gaps?

I used pretty much only nano banana during this synthetic gen data creation step.

Anyways, I then moved on to doing the captioning and lora training of this dataset using Gemini Pro 3.1 which provided good physical description of me.

The next step would then to be generating a 3d rigged representation of me.

The face is all messed up but not to worry, we can iterate and try to fix using the crop in inpaint workflow.

And to continue this experiment of integrating these new gen-ai models in traditional vfx pipelines I also wanted to play around with a 3D mesh creation model, I used Tripo in this case

I had to do some general mesh cleanup and decimation in houdini to get the mesh to be workable. And just to be extra lazy I also re-rigged this mesh using mixamo…because the tripo autorigger is terrible.

But at this point I now have some animations and a 3d thing that generally will have my silhouette. The whole point of the above process is to generate something that looks kinda good enough and kinda like me enough. Cause out from unreal currently I just want camera movement, basic animations and control maps.

Goal here is to also eventually test more unreal-y things, like groom/cloth/vfx sims

Here are then some results of using a gen-AI generated mesh, to create a “good-enough” digital double of me that I will then using the lora made with synthetic data of me render these control passes

Overall this was just part 1 in what I think is an experiment that has a lot of fun potential, the intersection between traditional vfx, game engine techniques and gen-ai is boundless!