Generative Neural Materials

Abstract

Advancements in neural rendering techniques have sparked renewed interest in neural materials, which are capable of representing bidirectional texture functions (BTFs) cheaply and with high quality. However, content creation in the neural material format is not straightforward. To address this limitation, we present the first image-conditioned diffusion model for neural materials, and show an extension to text conditioning. To achieve this, we make two main contributions:

First, we introduce a universal MLP variant of the NeuMIP architecture, defining a universal basis for neural materials as 16-channel feature textures.

Second, we train a conditional diffusion model for generating neural materials in this basis from flash images, natural images and text prompts. To achieve this, we also construct a new dataset of 150k neural materials.

We demonstrate real-time decoding performance for our universal materials at 1024 x 1024.

Videos: Universal Neural BTF Basis

Leather Chesterfield

Fabric Camouflage

Material 3

Eroded Mud Wall

Ancient Chinese Armor

Concrete Wall

Melted Metal

Venice Mosaic Tile

Ceramic Tiles

Train Tracks

Cable Knit Wool

Dry Rocky Mud

Generated Materials: Flash Images

IMG_0370

IMG_5994

IMG_7630

IMG_0360

IMG_0821

IMG_1056

IMG_6869

IMG_1109

IMG_7088

IMG_5151

IMG_9066

IMG_1019

IMG_0299

IMG_0231

IMG_0420

IMG_0253

IMG_0815

Shiny Metal

Home Wood

Home Leather

Generated Materials: Casual Captures

Cat

Birds

Floor

Lamp

Studded Stone

Pillar

Plaid

Dress

Generated Materials: Text-to-Image

Bumpy uneven texture of an orange peel

Soft cloud-like texture of a floating island

Texture of dragon etched glass

Texture of dragon-embossed printed circuit board

Flaky crumbly texture of a fresh croissant

Velvety luminescent petals of a night-blooming flower

Polished concrete

Old worn out hardwood floor

Dirty concrete

Red rock cliff

Brown matte leather

Linen fabric

Green moss

Translucent jellyfish-like surface of a mystical portal

Wavy holographic surface of an interdimensional mirror

Pipeline

Overview of our pipeline. (a) Shows the data generation and training of the universal basis network. For each of 512 selected materials, 400 slices of data are rendered, where each pixel contains a different camera and light direction. The 512 materials are jointly used to train (Offset and RGB) neural textures, as well as corresponding MLPs. (b) Shows the training process for the full dataset of approximately 150,000 materials. For each material, training data is rendered similarly as in (a), however, the universal MLPs are used with frozen weights to fit each material's neural texture. (c) The resulting 150,000 neural materials are used as training data for training the diffusion model

Network Architecture

Network Architecture (a) Shows our diffusion model's architecture, which can conditionally (image, or optionally text) generate a neural texture pair (offset and RGB). (b) Shows how the neural textures are used for inference, i.e., render under a given (novel) camera and light(s)

BibTeX

@article{raghavanmullia2025genneumat, author = {Raghavan, Nithin and Mullia, Krishna and Trevithick, Alexander and Luan, Fujun and Ha\v{s}an, Milo\v{s} and Ramamoorthi, Ravi}, title = {Generative Neural Materials}, year = {2025}, isbn = {9798400715402}, series = {SIGGRAPH Conference Papers '25}, volume = {43}, articleno = {162}, pages = {11}, doi = {10.1145/3721238.3730746} }