3D Product Viewer

{"title":"8.7 3D Generation","description":"","content":"# 8.7 3D Generation\n\n## The Problem\n\nCreating a single 3D model by hand takes hours, sometimes days. A game studio might need thousands of assets for a new title. An e-commerce company wants to show every product from every angle, but hiring 3D artists for a catalog of 50,000 items is not realistic. An architect wants to quickly prototype building designs, but going from sketch to 3D model in traditional CAD software is a slow, manual process.\n\nThis is not just about cost. It is about speed and iteration. When a designer needs to explore 20 variations of a chair, manually modeling each one kills the creative process. When a small game studio cannot afford a full 3D art team, they are stuck with asset store models that look like everyone else's game.\n\nThe promise of AI-powered 3D generation is simple: describe what you want in text, or hand the model a photo, and get a usable 3D object back. In practice, this is one of the hardest problems in generative AI. Unlike images, which live on a flat 2D grid of pixels, 3D objects have geometry, texture, and structure that must be consistent from every viewing angle. A generated chair that looks great from the front but has a melted leg when you rotate it is useless.\n\nThe field has made remarkable progress in the last two years, but it is still far from the reliability of image generation. Some approaches produce rough shapes suitable for prototyping, others generate surprisingly detailed meshes but take minutes to run. Understanding what works, what does not, and where the gaps are will help you decide when 3D generation is ready for your use case and when it is not.\n\n**What you will learn:**\n\n- The current state of 3D generation: what works, what does not\n- Text-to-3D approaches: Point-E, Shap-E, and DreamFusion\n- Image-to-3D pipelines: turning photos into 3D models with TripoSR and InstantMesh\n- Neural scene representations: NeRFs and Gaussian Splatting\n- Practical use cases: product visualization, game assets, architectural prototyping\n- Limitations, quality gaps, and where the field is heading\n\n---\n\n## How 3D Generation Works: The Big Picture\n\nBefore diving into specific tools, it helps to understand the landscape. There are fundamentally different strategies for generating 3D content from AI, and they each make different trade-offs between speed, quality, and flexibility.\n\n\n```mermaid\nflowchart TD\n A[3D Generation Approaches]:::primary\n\n A --> B[Text-to-3D]:::orange\n A --> C[Image-to-3D]:::teal\n A --> D[Neural Scene
Representations]:::purple\n\n B --> B1[Point-E / Shap-E
Direct prediction]:::orange\n B --> B2[DreamFusion
Optimization-based]:::orange\n\n C --> C1[TripoSR
Feed-forward]:::teal\n C --> C2[InstantMesh
Multi-view + reconstruction]:::teal\n\n D --> D1[NeRFs
Neural Radiance Fields]:::purple\n D --> D2[Gaussian Splatting
Point-based rendering]:::purple\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\nThis diagram lays out the three main families. Text-to-3D takes a text description and produces a 3D object. Image-to-3D takes one or more photos and reconstructs a 3D model from them. Neural scene representations are a different paradigm entirely: instead of outputting a traditional mesh, they learn a continuous function that represents the 3D scene and can render it from any viewpoint.\n\nThe key distinction is between **feed-forward** models (one pass through a neural network, fast) and **optimization-based** methods (iteratively refine a 3D representation, slow but often higher quality). We will cover both.\n\n---\n\n## Text-to-3D: From Words to Objects\n\nThe idea is compelling: type \"a red ceramic coffee mug\" and get a 3D model. In practice, this is the hardest variant of 3D generation, because text is ambiguous and 3D space is complex. A mug could have a hundred different shapes, and the model has to pick one that looks coherent from all 360 degrees.\n\n### Point-E and Shap-E (OpenAI)\n\nOpenAI released Point-E in late 2022 and Shap-E in mid-2023. Both are feed-forward models, meaning they generate a 3D output in a single pass without iterative optimization.\n\n**Point-E** works in two stages. First, it uses a text-to-image model to generate a synthetic view of the object. Then, a second model takes that image and predicts a 3D point cloud (a set of points in 3D space that approximate the object's surface). The output is rough, more like a 3D sketch than a finished model.\n\n**Shap-E** is the successor. Instead of producing point clouds, it generates implicit functions (neural representations that define which points are inside or outside the object). These implicit functions can be converted to meshes, which are much more useful for downstream applications. Shap-E is also faster and produces cleaner results than Point-E.\n\n\n```mermaid\nflowchart LR\n A[Text Prompt]:::primary --> B[Text-to-Image
Model]:::orange\n B --> C[Synthetic
Image]:::teal\n C --> D[Image-to-3D
Model]:::orange\n D --> E[Point Cloud
or Implicit Function]:::green\n E --> F[Mesh
Extraction]:::purple\n F --> G[3D Model
.obj / .glb]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\nThis pipeline shows the Point-E approach. Notice that it routes through an image as an intermediate step. This is a common pattern: it is easier to generate a good 2D view first and then \"lift\" it into 3D, rather than going directly from text to 3D geometry.\n\nHere is how to use Shap-E with the open-source library:\n\n\n```python\nimport torch\nfrom shap_e.diffusion.sample import sample_latents\nfrom shap_e.diffusion.gaussian_diffusion import diffusion_from_config\nfrom shap_e.models.download import load_model, load_config\nfrom shap_e.util.notebooks import decode_latent_mesh\n\n# Load the pretrained models\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nxm = load_model(\"transmitter\", device=device)\nmodel = load_model(\"text300M\", device=device)\ndiffusion = diffusion_from_config(load_config(\"diffusion\"))\n\n# Generate a 3D latent from text\nprompt = \"a red ceramic coffee mug\"\n\nlatents = sample_latents(\n batch_size=1,\n model=model,\n diffusion=diffusion,\n guidance_scale=15.0,\n model_kwargs=dict(texts=[prompt]),\n progress=True,\n clip_denoised=True,\n use_fp16=True,\n use_karras=True,\n karras_steps=64,\n sigma_min=1e-3,\n sigma_max=160,\n s_churn=0,\n)\n\n# Decode the latent into a mesh and save it\nmesh = decode_latent_mesh(xm, latents[0]).tri_mesh()\nwith open(\"mug.obj\", \"w\") as f:\n mesh.write_obj(f)\n\nprint(f\"Generated mesh with {len(mesh.verts)} vertices and {len(mesh.faces)} faces\")\n```\n\n\nThe generation takes roughly 15-30 seconds on a GPU. The result is a mesh with geometry that approximates the described object. The quality is acceptable for prototyping, but you would not ship this directly in a polished game or product page.\n\n### DreamFusion: The Optimization Approach\n\nDreamFusion, introduced by Google Research, takes a completely different strategy. Instead of predicting 3D geometry in a single pass, it optimizes a 3D representation by using a pretrained 2D image generation model as a critic.\n\nThe core idea is called **Score Distillation Sampling (SDS)**. Here is how it works:\n\n1. Start with a random 3D representation (like a NeRF, which we will cover later).\n2. Render the 3D object from a random camera angle to get a 2D image.\n3. Ask a pretrained text-to-image model: \"How well does this rendered image match the text prompt?\"\n4. Use the model's feedback to compute gradients that update the 3D representation.\n5. Repeat from step 2, thousands of times, from many different camera angles.\n\nOver many iterations, the 3D object converges to something that looks like the text description from every angle, because the 2D model has been judging it from every angle.\n\n\n```mermaid\nflowchart TD\n A[Random 3D
Representation]:::primary\n B[Render from
Random Angle]:::orange\n C[2D Image]:::teal\n D[Pretrained Diffusion
Model evaluates match
to text prompt]:::purple\n E[Compute
Gradients]:::orange\n F[Update 3D
Representation]:::green\n G{Converged?}\n\n A --> B --> C --> D --> E --> F --> G\n G -->|No| B\n G -->|Yes| H[Final 3D
Model]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nDreamFusion produces higher quality results than Point-E or Shap-E, but it is much slower. A single generation can take 30 minutes to several hours. It also suffers from the \"Janus problem,\" where the model generates an object with the face or front features repeated on every side (because the 2D model thinks every angle should look like the \"default\" view). Subsequent work like Magic3D and ProlificDreamer have improved on these issues, but the optimization-based approach remains fundamentally slow.\n\n\n| Approach | Speed | Quality | Output Format | Best For |\n|----------|-------|---------|---------------|----------|\n| Point-E | ~20 seconds | Low | Point cloud | Quick previews |\n| Shap-E | ~20 seconds | Medium-low | Mesh (via implicit) | Prototyping |\n| DreamFusion | 30-90 minutes | Medium-high | NeRF / mesh | High-quality single assets |\n| Magic3D | 40-120 minutes | High | Mesh with texture | Production-quality assets |\n\n\n**Exercise:** Install the `shap-e` library from GitHub and generate 3D meshes for five different text prompts: \"a wooden chair,\" \"a running shoe,\" \"a medieval sword,\" \"a potted plant,\" and \"a smartphone.\" Save each as an OBJ file. Open them in a 3D viewer (Blender, MeshLab, or the online viewer at https://3dviewer.net) and rate each on a 1-5 scale for geometric accuracy and visual quality.\n\n---\n\n## Image-to-3D: From Photos to Models\n\nText-to-3D has an inherent ambiguity problem: \"a chair\" could mean anything. Image-to-3D sidesteps this by starting from a concrete visual reference. You give the model a photo, and it reconstructs the 3D object depicted in that photo. This is a more constrained problem, and the results are noticeably better.\n\n### TripoSR\n\nTripoSR, developed by Stability AI and Tripo, is a feed-forward transformer model that converts a single image into a 3D mesh in under one second. That speed is not a typo. It processes the image through a transformer architecture and directly predicts a 3D representation using a technique called triplane representation.\n\nA triplane represents a 3D volume using three 2D feature planes (one for each axis: XY, XZ, YZ). Any 3D point can be looked up by projecting it onto each plane and combining the features. This is much more memory-efficient than a full 3D voxel grid.\n\n\n```python\nimport torch\nfrom tsr.system import TSR\nfrom tsr.utils import save_video, to_gradio_3d_orientation\nfrom PIL import Image\n\n# Load the pretrained TripoSR model\nmodel = TSR.from_pretrained(\n \"stabilityai/TripoSR\",\n config_name=\"config.yaml\",\n weight_name=\"model.ckpt\",\n)\nmodel.to(\"cuda\")\n\n# Load an input image (should have the object centered, clean background)\nimage = Image.open(\"product_photo.png\")\n\n# Run the model: image in, 3D representation out\nwith torch.no_grad():\n scene_codes = model([image], device=\"cuda\")\n\n# Extract the mesh from the triplane representation\nmeshes = model.extract_mesh(scene_codes, resolution=256)\n\n# Save as GLB (a common 3D format for web viewers)\nmeshes[0].export(\"product_3d.glb\")\nprint(\"Mesh exported successfully\")\n```\n\n\nThe quality depends heavily on the input image. A product photo with a clean white background, good lighting, and the object centered will produce a much better mesh than a cluttered photo taken from a weird angle. This is why e-commerce is one of the strongest use cases: product photos are already shot in controlled conditions.\n\n### InstantMesh\n\nInstantMesh takes a different approach. Instead of predicting 3D geometry from a single image directly, it first generates multiple synthetic views of the object (front, back, left, right, top) using a multi-view diffusion model, and then reconstructs a 3D mesh from those views using a Large Reconstruction Model (LRM).\n\n\n```mermaid\nflowchart LR\n A[Single Input
Image]:::primary\n B[Multi-view
Diffusion Model]:::orange\n C[6 Synthetic
Views]:::teal\n D[Large Reconstruction
Model]:::purple\n E[3D Mesh
with Texture]:::green\n\n A --> B --> C --> D --> E\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThis two-stage approach tends to produce more geometrically consistent results than single-view methods, because the reconstruction model has multiple viewpoints to work from. The multi-view generation step takes about 10-15 seconds, and the reconstruction step takes another 10-20 seconds. So the total is still under a minute, much faster than optimization-based methods.\n\n\n```python\n# InstantMesh example using the Hugging Face pipeline\n# Note: this is a simplified example; the full pipeline requires\n# the InstantMesh repository and its dependencies\n\nfrom PIL import Image\nimport subprocess\n\n# Step 1: Preprocess the input image (remove background, center object)\ndef preprocess_image(image_path: str, output_path: str):\n \"\"\"Remove background and center the object for better 3D reconstruction.\"\"\"\n from rembg import remove\n\n input_image = Image.open(image_path)\n output_image = remove(input_image)\n\n # Center and pad the image to a square\n width, height = output_image.size\n max_dim = max(width, height)\n padded = Image.new(\"RGBA\", (max_dim, max_dim), (255, 255, 255, 0))\n padded.paste(output_image, ((max_dim - width) // 2, (max_dim - height) // 2))\n padded = padded.resize((256, 256))\n padded.save(output_path)\n return output_path\n\n# Step 2: Run InstantMesh inference\n# Using the command-line interface from the InstantMesh repo\ndef generate_3d_from_image(image_path: str, output_dir: str):\n \"\"\"Generate a 3D mesh from a single image using InstantMesh.\"\"\"\n preprocessed = preprocess_image(image_path, \"preprocessed.png\")\n\n # Run the InstantMesh pipeline\n subprocess.run([\n \"python\", \"run.py\",\n \"--input\", preprocessed,\n \"--output_dir\", output_dir,\n \"--export_mesh\",\n ], check=True)\n\n print(f\"3D model saved to {output_dir}\")\n\n# Usage\ngenerate_3d_from_image(\"shoe_photo.jpg\", \"./output_meshes/\")\n```\n\n\n### Using 3D Generation APIs\n\nIf you do not want to run models locally, several cloud APIs offer image-to-3D generation. These are easier to integrate and do not require a GPU.\n\n\n```python\nimport requests\nimport time\n\n# Example: Using the Tripo3D API (one of several available services)\nAPI_KEY = \"your-tripo3d-api-key\"\nBASE_URL = \"https://api.tripo3d.ai/v2/openapi\"\n\ndef image_to_3d(image_path: str) -> str:\n \"\"\"Convert an image to a 3D model using an API.\"\"\"\n\n # Upload image and create task\n with open(image_path, \"rb\") as f:\n response = requests.post(\n f\"{BASE_URL}/upload\",\n headers={\"Authorization\": f\"Bearer {API_KEY}\"},\n files={\"file\": f},\n )\n image_token = response.json()[\"data\"][\"image_token\"]\n\n # Start the 3D generation task\n task_response = requests.post(\n f\"{BASE_URL}/task\",\n headers={\n \"Authorization\": f\"Bearer {API_KEY}\",\n \"Content-Type\": \"application/json\",\n },\n json={\n \"type\": \"image_to_model\",\n \"file\": {\"type\": \"jpg\", \"file_token\": image_token},\n },\n )\n task_id = task_response.json()[\"data\"][\"task_id\"]\n\n # Poll until the task completes\n while True:\n status_response = requests.get(\n f\"{BASE_URL}/task/{task_id}\",\n headers={\"Authorization\": f\"Bearer {API_KEY}\"},\n )\n status = status_response.json()[\"data\"][\"status\"]\n\n if status == \"success\":\n model_url = status_response.json()[\"data\"][\"output\"][\"model\"]\n return model_url\n elif status == \"failed\":\n raise Exception(\"3D generation failed\")\n\n time.sleep(2)\n\n# Generate a 3D model from a product photo\nmodel_url = image_to_3d(\"product_photo.jpg\")\nprint(f\"3D model available at: {model_url}\")\n```\n\n\n**Exercise:** Pick three product images (a shoe, a mug, and a toy). Run each through either TripoSR locally or a cloud API. For each output, document: (1) generation time, (2) vertex count, (3) visual quality on a 1-5 scale, and (4) any artifacts you notice (missing geometry, floating fragments, texture issues).\n\n---\n\n## Neural Scene Representations: NeRFs and Gaussian Splatting\n\nThe approaches we have covered so far output traditional 3D formats like meshes and point clouds. Neural scene representations take a fundamentally different approach: they encode a 3D scene inside a neural network or a set of learned parameters. You never get a mesh. Instead, you can render the scene from any viewpoint by querying the representation.\n\n### NeRFs (Neural Radiance Fields)\n\nA NeRF represents a scene as a continuous function: given a 3D coordinate (x, y, z) and a viewing direction, it outputs the color and density at that point. To render an image from a specific camera position, you shoot rays through each pixel, sample points along each ray, query the NeRF for color and density at each point, and composite them together.\n\nThe training process works by taking a set of photos of a scene from known camera positions, then optimizing the neural network so that rendering from those positions produces images that match the actual photos. Once trained, you can render from any position, including positions that were never photographed.\n\n\n```mermaid\nflowchart LR\n subgraph Training\n A[Photos from
known positions]:::primary\n B[Optimize
NeRF network]:::orange\n end\n\n subgraph Rendering\n C[New camera
position]:::teal\n D[Cast rays
through pixels]:::orange\n E[Query NeRF at
points along ray]:::purple\n F[Composite colors
and densities]:::orange\n G[Rendered
image]:::green\n end\n\n A --> B\n B --> E\n C --> D --> E --> F --> G\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nNeRFs produce photorealistic novel views of scenes. They excel at capturing complex materials like glass, reflections, and translucent objects that are difficult to represent with traditional meshes. The catch is that training a NeRF from scratch requires dozens of input photos and takes minutes to hours, and rendering is relatively slow because you need to evaluate the neural network thousands of times per pixel.\n\nHere is a minimal example using the `nerfstudio` framework, which is the most popular NeRF toolkit:\n\n\n```python\n# NeRF training with nerfstudio (command-line workflow)\n# Step 1: Capture photos of a scene (or use a provided dataset)\n# Step 2: Process the images to estimate camera positions\n\n# Install nerfstudio\n# pip install nerfstudio\n\n# Process a set of images to get camera poses\n# ns-process-data images --data ./my_photos/ --output-dir ./processed_data/\n\n# Train a NeRF model\n# ns-train nerfacto --data ./processed_data/\n\n# Render a video from the trained model\n# ns-render camera-path --load-config outputs/my_photos/nerfacto/.../config.yml \\\n# --camera-path-filename camera_path.json \\\n# --output-path renders/output.mp4\n```\n\n\n\n```python\n# Programmatic NeRF rendering example using nerfstudio's API\nfrom nerfstudio.utils.eval_utils import eval_setup\nfrom nerfstudio.cameras.cameras import Cameras\nfrom pathlib import Path\nimport torch\n\n# Load a trained NeRF model\nconfig_path = Path(\"outputs/my_scene/nerfacto/2024-01-01_120000/config.yml\")\nconfig, pipeline, _, _ = eval_setup(config_path)\n\n# Define a new camera position to render from\ncamera = Cameras(\n camera_to_worlds=torch.tensor([[\n [1.0, 0.0, 0.0, 0.5],\n [0.0, 1.0, 0.0, 0.5],\n [0.0, 0.0, 1.0, 2.0],\n ]]),\n fx=500.0, fy=500.0,\n cx=400.0, cy=300.0,\n width=800, height=600,\n)\n\n# Render the scene from this new viewpoint\noutputs = pipeline.model.get_outputs_for_camera(camera)\nrendered_image = outputs[\"rgb\"] # Tensor of shape [H, W, 3]\n\nprint(f\"Rendered image shape: {rendered_image.shape}\")\n```\n\n\n### Gaussian Splatting\n\n3D Gaussian Splatting (3DGS), introduced in 2023, has quickly become the dominant alternative to NeRFs. Instead of representing the scene as a neural network, it represents it as a collection of 3D Gaussian ellipsoids. Each Gaussian has a position, size, orientation, color, and opacity. To render an image, you project (or \"splat\") these Gaussians onto the image plane and blend them together.\n\nThe advantages over NeRFs are significant:\n\n- **Rendering is 100-1000x faster.** Splatting is a simple rasterization operation, not a neural network evaluation. You can render at 100+ FPS in real time.\n- **Training is faster.** A scene that takes a NeRF hours to learn can be trained with Gaussian Splatting in 5-15 minutes.\n- **Quality is comparable or better.** The visual fidelity of Gaussian Splatting matches or exceeds NeRFs on most benchmarks.\n\nThe downside is that you are storing millions of Gaussians (the model for a single scene can be 100-500 MB), and editing the scene is less intuitive than working with a mesh.\n\n\n```mermaid\nflowchart TD\n A[Input Photos +
Camera Poses]:::primary\n B[Initialize
Sparse Point Cloud]:::orange\n C[Optimize Gaussian
Parameters]:::teal\n D[Split / Clone / Prune
Gaussians adaptively]:::orange\n E[Trained Scene:
Millions of Gaussians]:::green\n F[Real-time
Rendering via Splatting]:::purple\n\n A --> B --> C --> D\n D -->|Repeat| C\n D -->|Converged| E --> F\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\nThe training loop starts by initializing a sparse point cloud (often from COLMAP, a structure-from-motion tool). Each point becomes a Gaussian. The optimizer adjusts each Gaussian's position, shape, color, and opacity to minimize the difference between rendered images and the ground truth photos. An adaptive density control step periodically splits large Gaussians into smaller ones (to capture fine detail) and removes transparent or redundant ones.\n\n\n```python\n# 3D Gaussian Splatting training (using gsplat library)\n# pip install gsplat\n\n# The most common workflow uses the original 3DGS codebase:\n# 1. Prepare your images and camera poses (using COLMAP)\n# 2. Train the Gaussian Splatting model\n# 3. Render from novel viewpoints\n\n# Command-line training with the original repo:\n# python train.py -s ./data/my_scene/ -m ./output/my_scene/\n\n# Programmatic rendering with gsplat\nimport torch\nfrom gsplat.rendering import rasterization\n\n# Gaussian parameters (loaded from a trained model)\n# Each Gaussian has: mean (3D position), covariance, color, opacity\nmeans = torch.randn(100000, 3, device=\"cuda\") # 100K Gaussians\nquats = torch.randn(100000, 4, device=\"cuda\") # Rotation quaternions\nscales = torch.rand(100000, 3, device=\"cuda\") * 0.1 # Scale\nopacities = torch.sigmoid(torch.randn(100000, 1, device=\"cuda\"))\ncolors = torch.sigmoid(torch.randn(100000, 3, device=\"cuda\"))\n\n# Camera parameters\nviewmat = torch.eye(4, device=\"cuda\").unsqueeze(0) # Camera-to-world matrix\nK = torch.tensor([[500, 0, 400],\n [0, 500, 300],\n [0, 0, 1]], device=\"cuda\", dtype=torch.float32).unsqueeze(0)\n\n# Render the scene\nrendered_image, alphas, meta = rasterization(\n means=means,\n quats=quats,\n scales=scales,\n opacities=opacities,\n colors=colors,\n viewmats=viewmat,\n Ks=K,\n width=800,\n height=600,\n)\n\nprint(f\"Rendered image shape: {rendered_image.shape}\") # [1, 600, 800, 3]\n```\n\n\n### NeRFs vs Gaussian Splatting\n\n\n| Feature | NeRFs | Gaussian Splatting |\n|---------|-------|--------------------|\n| **Representation** | Neural network | Set of 3D Gaussians |\n| **Training time** | 30 min to hours | 5-15 minutes |\n| **Rendering speed** | Slow (seconds/frame) | Real-time (100+ FPS) |\n| **Visual quality** | High | High (often better) |\n| **Model size** | Small (few MB) | Large (100-500 MB) |\n| **Editing** | Difficult | Possible (move/delete Gaussians) |\n| **Mesh export** | Requires post-processing | Requires post-processing |\n| **Best use case** | Compact scene storage | Interactive viewing, VR/AR |\n\n\nFor most new projects in 2025/2026, Gaussian Splatting is the default choice. NeRFs are still relevant for research and for cases where compact model size matters, but the speed advantage of splatting has made it the practical winner.\n\n**Exercise:** Use nerfstudio to train a NeRF from the \"poster\" sample dataset (included in nerfstudio). Then, using the same dataset, train a Gaussian Splatting model. Compare the two on: training time, rendering speed (frames per second), and subjective visual quality. Document which one you would choose for a real-time product viewer.\n\n---\n\n## Practical Use Cases\n\nNow that you understand the technical landscape, here is where 3D generation is actually being used today, and where it is not ready yet.\n\n### E-commerce Product Visualization\n\nThis is the strongest current use case. Online retailers want 3D product views so customers can rotate and inspect items before buying. Image-to-3D models like TripoSR can convert existing product photography into 3D models at scale. The quality is good enough for web-based 3D viewers, especially for simple objects like shoes, furniture, and accessories. Complex objects with fine details (watches with small text, jewelry with intricate patterns) still need manual cleanup.\n\n### Game and AR/VR Asset Generation\n\nGame studios are using AI-generated 3D models as starting points. The workflow is: generate a rough model with AI, import it into Blender or Maya, and refine it manually. This is faster than modeling from scratch, especially for background props and filler objects that do not need to be perfect. For hero assets (the main character, key items), manual modeling is still necessary.\n\nGaussian Splatting is particularly exciting for VR/AR because it renders in real time and captures photorealistic detail. Several companies are building Gaussian Splatting viewers for VR headsets, allowing you to \"scan\" a real room with your phone and then walk through it in VR.\n\n### Architectural Prototyping\n\nArchitects use NeRFs and Gaussian Splatting to create digital twins of existing buildings and sites. You photograph a building from many angles, train a scene representation, and then you can view the building from any angle on screen. For new designs, text-to-3D is not yet reliable enough for architectural work (the shapes are too complex and precision matters too much), but image-to-3D can convert renderings into interactive 3D previews.\n\n### What Is Not Ready Yet\n\n- **Text-to-3D for production assets:** The quality gap between AI-generated and artist-made 3D models is still large for anything that needs fine detail or precise proportions.\n- **Animated 3D generation:** Generating 3D models that move or have articulated joints is an active research area but not production-ready.\n- **Consistent style across assets:** If you need 20 chairs that all look like they belong in the same room, current models cannot guarantee stylistic consistency.\n- **Texturing and materials:** Most AI-generated meshes have baked-in vertex colors or basic textures. PBR (Physically Based Rendering) materials with roughness, metallic, and normal maps are not yet generated reliably.\n\n---\n\n## Putting It All Together\n\nBuild a pipeline that converts product photos into 3D models and displays them in a web viewer. This exercise combines image-to-3D generation with a simple Three.js viewer.\n\n### Step 1: Generate 3D Models from Product Photos\n\n\n```python\nimport os\nimport json\nfrom pathlib import Path\n\n# Use TripoSR or an API to convert images to 3D models\n# This example uses a generic API approach\n\ndef batch_generate_3d(image_dir: str, output_dir: str) -> list[dict]:\n \"\"\"Convert all images in a directory to 3D models.\"\"\"\n results = []\n os.makedirs(output_dir, exist_ok=True)\n\n image_extensions = {\".jpg\", \".jpeg\", \".png\", \".webp\"}\n image_files = [\n f for f in Path(image_dir).iterdir()\n if f.suffix.lower() in image_extensions\n ]\n\n for image_path in image_files:\n print(f\"Processing: {image_path.name}\")\n\n output_path = Path(output_dir) / f\"{image_path.stem}.glb\"\n\n try:\n # Replace this with your actual 3D generation call\n # Option A: TripoSR locally\n # Option B: API call (see the API example earlier)\n model_url = image_to_3d(str(image_path))\n\n # Download the model\n import requests\n response = requests.get(model_url)\n with open(output_path, \"wb\") as f:\n f.write(response.content)\n\n results.append({\n \"name\": image_path.stem,\n \"image\": str(image_path.name),\n \"model\": str(output_path.name),\n \"status\": \"success\",\n })\n except Exception as e:\n results.append({\n \"name\": image_path.stem,\n \"image\": str(image_path.name),\n \"model\": None,\n \"status\": f\"failed: {str(e)}\",\n })\n\n # Save a manifest file\n manifest_path = Path(output_dir) / \"manifest.json\"\n with open(manifest_path, \"w\") as f:\n json.dump(results, f, indent=2)\n\n return results\n\n# Run the batch generation\nresults = batch_generate_3d(\"./product_photos/\", \"./3d_models/\")\n\nfor r in results:\n print(f\" {r['name']}: {r['status']}\")\n```\n\n\n### Step 2: Build a Simple Web Viewer\n\nCreate an HTML file that loads and displays the generated GLB files using Three.js.\n\n\n```python\ndef generate_viewer_html(manifest_path: str, output_html: str):\n \"\"\"Generate an HTML viewer for the 3D models.\"\"\"\n with open(manifest_path) as f:\n manifest = json.load(f)\n\n successful = [r for r in manifest if r[\"status\"] == \"success\"]\n\n # Build the model selector options\n options_html = \"\\n\".join(\n f' '\n for m in successful\n )\n\n html_content = f\"\"\"\n\n\n \n 3D Product Viewer\n \n\n\n

\n \n

\n\n \n\n \n\n\"\"\"\n\n with open(output_html, \"w\") as f:\n f.write(html_content)\n\n print(f\"Viewer written to {output_html}\")\n\n# Generate the viewer\ngenerate_viewer_html(\"./3d_models/manifest.json\", \"./3d_models/viewer.html\")\n```\n\n\n### Step 3: Evaluate Quality\n\nAfter generating and viewing the models, fill out an evaluation table like this:\n\n\n| Product | Generation Time | Vertices | Geometry (1-5) | Texture (1-5) | Issues |\n|---------|----------------|----------|----------------|---------------|--------|\n| Shoe | | | | | |\n| Mug | | | | | |\n| Toy | | | | | |\n| Lamp | | | | | |\n| Watch | | | | | |\n\n\nPay attention to: missing backs or bottoms (a very common artifact), floating geometry fragments, incorrect symmetry, and texture blurring. These are the most frequent failure modes of current image-to-3D models.\n\n---\n\n## Summary\n\nHere is what we covered and why it matters:\n\n- **3D generation from AI is real but immature.** Image-to-3D works well enough for prototyping and e-commerce previews. Text-to-3D is improving fast but not production-ready for detailed assets.\n- **Feed-forward models (Shap-E, TripoSR)** are fast (seconds) but produce lower-quality output. **Optimization-based methods (DreamFusion)** are slow (minutes to hours) but produce better results. The trade-off is always speed vs quality.\n- **Image-to-3D is more practical than text-to-3D** because starting from a real photo eliminates the ambiguity of text descriptions. TripoSR and InstantMesh are the strongest options in this space.\n- **NeRFs** encode scenes as neural networks and render photorealistic novel views, but they are slow to train and slow to render. **Gaussian Splatting** uses millions of 3D Gaussian ellipsoids and renders in real time at 100+ FPS. For most practical applications, Gaussian Splatting is the better choice.\n- **The strongest use cases today** are e-commerce product visualization, game asset prototyping, and digital twin creation via Gaussian Splatting.\n- **Key limitations** include inconsistent quality, poor fine detail, no reliable animation support, and difficulty maintaining stylistic consistency across multiple generated assets.\n\nThe field is moving fast. Models that were state-of-the-art six months ago are already being replaced. The best strategy is to understand the fundamental approaches (feed-forward vs optimization, mesh vs neural representation) so you can evaluate new tools as they appear.\n\n---\n\n## References\n\n- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts (OpenAI, 2022)](undefined)\n- [Shap-E: Generating Conditional 3D Implicit Functions (OpenAI, 2023)](undefined)\n- [DreamFusion: Text-to-3D using 2D Diffusion (Poole et al., 2022)](undefined)\n- [3D Gaussian Splatting for Real-Time Radiance Field Rendering (Kerbl et al., 2023)](undefined)\n- [TripoSR: Fast 3D Object Reconstruction from a Single Image (Stability AI, 2024)](undefined)\n- [InstantMesh: Efficient 3D Mesh Generation from a Single Image (Xu et al., 2024)](undefined)","pageType":"ai-engineering"}

8.7 3D Generation

Ashish Pratap Singh

Get Premium