Luce generates relightable 3D assets with PBR materials from a single image

High-fidelity single-image-to-3D generation needs a representation that captures both geometry and appearance, but for a 3D asset to be relit convincingly or dropped into a standard rendering pipeline, it also needs physically based rendering (PBR) modalities: albedo, metallic-roughness, and surface normals. A new paper introduces Luce, a 3D representation built specifically around that requirement. It unifies geometry and PBR materials inside a single voxelized, multimodal Gaussian cloud, using a dedicated set of Gaussian primitives for each modality rather than treating material properties as an add-on.

To generate that representation from a single photo, the authors first train a variational autoencoder that compresses the multimodal Gaussian cloud into one unified, material-aware latent space. A rectified-flow transformer then generates this latent directly from an input image, conditioned on multi-layer features pulled from a pretrained image encoder, features chosen to preserve both the image's overall semantic context and its fine spatial detail. Decoding the resulting latent produces relightable PBR Gaussians, plus an optional textured mesh complete with a tangent-space normal map, so the output can go straight into a renderer that expects standard material maps.

On Toys4K, an existing single-image-to-3D benchmark, Luce achieves state-of-the-art results, improving the FID score by 28% over what the paper calls the strongest baseline (the baseline itself is not named). The authors also built a second, new benchmark composed of AI-generated images rather than photographs. On it, Luce raises the CLIP image-alignment score to 0.8519, compared with 0.8299 for the best baseline on that benchmark. The paper states that Luce's output assets are relightable, geometrically accurate, and materially faithful, and specifically preserve fine details such as text, logos, and inscriptions.

The abstract does not name any authors, institution, or affiliation behind the work, nor does it mention a publication venue or submission date. It also does not identify the baseline methods used for either comparison, describe the Toys4K benchmark itself, or say anything about releasing code, model weights, or the training dataset, so the results cannot currently be independently reproduced or checked outside the paper.

Key facts

Luce packs geometry and PBR materials (albedo, metallic-roughness, surface normals) into one voxelized, multimodal Gaussian cloud, using separate Gaussian primitives for each modality.
A rectified-flow transformer generates Luce's compressed latent from a single input image, conditioned on multi-layer features from a pretrained image encoder that keep both semantic context and fine spatial detail.
On the Toys4K single-image-to-3D benchmark, Luce improves FID by 28% over the strongest baseline, reaching state-of-the-art results.
On a new benchmark of AI-generated images introduced in the same paper, Luce raises the CLIP image-alignment score to 0.8519, versus 0.8299 for the best baseline.
Output assets are relightable and can be decoded into an optional textured mesh with a tangent-space normal map, preserving fine details such as text, logos, and inscriptions.

Why it matters

Most single-image-to-3D methods reconstruct shape and a flat color, which looks fine in a static render but breaks the moment you change the lighting or bring the asset into a renderer that expects real material properties. The paper argues that a representation aimed at relighting and integration into standard rendering pipelines should carry physically based rendering modalities: albedo, metallic-roughness, and surface normals, not shape and color alone. Luce is built around that argument. It folds those modalities directly into a voxelized, multimodal Gaussian cloud, with a dedicated set of Gaussian primitives for each one, so material and geometry live in the same representation from the start rather than being bolted on afterward.

Who it affects

The immediate audience is the single-image-to-3D research community, who now have a new representation and a new evaluation benchmark to measure against. The paper's second benchmark, built from AI-generated images rather than photographs, points to a more specific use case: turning an AI-generated 2D image into a relightable 3D asset for use in a rendering pipeline. The abstract names no authors, institution, or affiliation, so who built Luce and who, if anyone, plans to release or maintain it is not stated.

How to use it

The abstract gives no indication that code, trained model weights, or the training dataset will be released, so there is currently no way to run Luce on your own images. What it does describe is the pipeline shape: a single photo goes through a pretrained image encoder, a rectified-flow transformer turns the resulting features into a compressed latent, and a decoder converts that latent into relightable PBR Gaussians plus an optional textured mesh with a tangent-space normal map, output meant to plug into a renderer that already expects standard material maps.

How solid is it

The evidence rests on two comparisons. On Toys4K, an established single-image-to-3D benchmark, Luce improves FID by 28% over what the paper calls the strongest baseline, though that baseline is not named and Toys4K itself is not described. On the CLIP image-alignment benchmark, Luce scores 0.8519 against 0.8299 for the best baseline, but this second benchmark is one the same paper introduces, built from AI-generated images, so it has no track record as an independent, external test. With no named baselines, no released code or weights, and no third-party replication mentioned, the numbers currently stand on the authors' own reporting alone.

Risks and caveats

The abstract does not discuss limitations, failure cases, or compute and runtime cost, so it is not possible to say from the source how Luce behaves outside its two benchmarks, how expensive it is to run, or where it might fail. Both comparison baselines go unnamed, which makes the 28% FID improvement and the CLIP score gain hard to independently sanity check. And because one of the two benchmarks is new and authored by the same team, the strongest external validation currently available is limited to Toys4K.