Hugging Face releases LoRA fine-tuning scripts for TRELLIS.2 3D model

Hugging Face releases LoRA fine-tuning scripts for TRELLIS.2 3D model

Hugging Face has published a guide to fine-tuning TRELLIS.2, a popular 3D generation model, on your own 3D assets. It comes with fine-tuning scripts and two Docker containers, one for preprocessing and one for training. They run either on your own GPU or on Hugging Face Jobs. If your dataset is already on the Hugging Face Hub, the scripts need only a few arguments; you can also tinker with the code in the linked repository.

The guide opens with background. 3D generation is used in robotics, simulation, media and entertainment, but it is hard to make generated objects obey strict rules, and there are many file formats (GLB, PLY) and representations (NeRFs, 3D Gaussian splatting). TRELLIS introduced a common internal representation, a unified latent representation, whose decoders convert it into the required output format. The same team then released TRELLIS.2, which improves the model's representation of 3D structure. Neither model, the authors say, gives an easy way to adapt it to a specific task.

The worked example uses the Google Scanned Objects (GSO) dataset. The authors take a subset of 188 footwear objects out of the 1,030 objects in GSO. In their initial tests the pretrained model usually produced shoes that looked correct but had incorrect soles, so the experiment tests whether a LoRA can make the soles more accurate.

To decide what to train, the guide walks through the TRELLIS.2 pipeline. It has five main stages: remove the image background, encode the image, predict the coarse structure, generate detailed geometry, and generate materials, then export a textured mesh. BiRefNet removes the background and DINOv3-L encodes the image into feature tokens. The sparse-structure flow DiT, a dense diffusion transformer, makes a small 16³ × 8 latent, which a VAE decoder turns into a 64 × 64 × 64 occupancy grid. The shape flow DiT then uses that grid and the DINOv3 tokens to make a latent, which another VAE decoder turns into O-Voxel geometry at 512 × 512 × 512. O-Voxel is the TRELLIS.2 3D format, a sparse voxel grid that keeps only surface voxels, so a CPU converts a mesh to O-Voxel in seconds and a GPU converts it back in seconds. The model uses only about 10% of the voxels in the grid. Finally the texture flow DiT defines materials, and a decoder outputs base color, metallic, roughness and alpha values per voxel.

LoRA adapters go only on the sparse-structure flow DiT and the shape flow DiT, which control shape and geometry. The texture flow DiT is frozen. The authors give two reasons: the dataset has diffuse color maps with baked lighting and lacks the separate material maps that stage needs, so training on them could teach the model to treat lighting as a surface property and could cause catastrophic forgetting. Also, their problem is the shoe's shape, and the base model already makes good colors and textures. Their line: "Choosing what to freeze matters as much as choosing what to fine-tune."

They pick LoRA over full fine-tuning because the dataset is small; full fine-tuning costs much more and can overwrite learned priors. The configuration uses rank 8 adapters in the attention and feed-forward (MLP) layers of the sparse-structure DiT and rank 16 adapters in the attention layers of the shape DiT, about 9.3 million trainable adapter parameters in total for both. Adapters are added and trained with Hugging Face's PEFT library.

The workflow has four steps: give the containers access to your data, set parameters, run preprocessing, run fine-tuning. Preprocessing converts GLB or OBJ meshes into the latent representations TRELLIS.2 trains on and renders 16 views of each object with Blender (run headless through xvfb-run); LoRA training uses those views as conditioning images. For the 188 objects the full run took about 1 hour on an a100-large job. The example command sets a 6h timeout, a 0.05 holdout, 16 views and 8 workers. The preprocessed dataset is saved so all training runs can reuse it.

Training runs as one job per DiT. The 'ss' mode fine-tunes the sparse-structure flow DiT and the 'shape' mode fine-tunes the shape flow DiT; they simply point to different config.json files. The example job commands use an a100-large flavor with a 12h timeout, push adapter checkpoints to the Hub when --push-to is set, and log metrics with Trackio. The shape-mode example passes max_steps=3000 and i_save=500. For readers who want to skip preprocessing, the guide points to a ready-made dataset, suvadityamuk/gso-footwear-trellis2-latents, with commands for both modes. Prerequisites are an HF_TOKEN, a Hugging Face account with access to the gated DINOv3 model, and, for your own data, meshes uploaded to the Hub. The guide then goes on to explain the fine-tuning script and its settings.

Key facts

  • Hugging Face published fine-tuning scripts and two Docker containers (preprocessing and training) for TRELLIS.2 that run locally or on Hugging Face Jobs.
  • The example uses 188 footwear objects from the 1,030-object Google Scanned Objects dataset, aiming to fix incorrect shoe soles the pretrained model produced in the authors' initial tests.
  • LoRA adapters go only on the sparse-structure flow DiT (rank 8, attention and MLP layers) and the shape flow DiT (rank 16, attention layers), about 9.3 million trainable parameters in total; the texture flow DiT stays frozen.
  • Preprocessing renders 16 Blender views per object and took about 1 hour on an a100-large job for the 188 objects; training uses one job per DiT with a 12h timeout.
  • Access to the gated DINOv3 model and an HF_TOKEN are required.

Why it matters

The authors say neither TRELLIS nor TRELLIS.2 offers an easy way to adapt the model to a specific task, and that a general-purpose model can get details wrong that matter for an application, such as accurate geometry for furniture. This guide supplies ready containers and scripts for that adaptation, so a team does not have to assemble the many dependencies of a 3D model setup itself. The design lesson it stresses is selective training: tune the stages that govern shape, freeze the one whose training data is unsuitable.

Who it affects

Teams working with 3D generation in robotics, simulation, media and entertainment, the fields the guide names, who want TRELLIS.2 to handle a particular object category better. It also suits anyone with a set of meshes (GLB or OBJ) who can upload them to the Hugging Face Hub and has either a local GPU or access to Hugging Face Jobs.

How to use it

Get an HF_TOKEN, make sure your Hugging Face account has access to the gated DINOv3 model, and upload your meshes to the Hub. Run the preprocessing container (the example uses hf jobs run on an a100-large flavor with a 6h timeout) to render 16 views per object and build the latents, then run the training container once with --config ss and once with --config shape, pushing adapters to your own Hub repository and monitoring with Trackio. Settings can be overridden with --set, as in max_steps=3000 i_save=500. To skip preprocessing, the guide offers a prepared footwear dataset, suvadityamuk/gso-footwear-trellis2-latents. The same containers run on your own GPU through local Docker commands.

How solid is it

This is a first-party Hugging Face tutorial. The pipeline description, adapter ranks, parameter count and commands are stated directly by the authors. The portion of the guide covered here reports no training results, metrics or before/after comparison of sole accuracy, so there is no evidence yet that the LoRA fixes the soles. The preprocessing time (about 1 hour for 188 objects) is the only measured duration given; the 6h and 12h figures are job timeouts.

Risks and caveats

The experiment is narrow: one object category (footwear) from one dataset of 188 shoes, and the stated goal is only to test whether a LoRA can make soles more accurate. The color maps in that dataset have baked lighting, which is why texture is left untouched, so this recipe does not improve materials. Access to DINOv3 is gated, which can block a first run. No cost for the LoRA runs is given. The guide itself warns that training the texture DiT on lit color maps can teach lighting as a surface property and cause catastrophic forgetting.

“Choosing what to freeze matters as much as choosing what to fine-tune.”

— Hugging Face blog guide on fine-tuning TRELLIS.2