SceneConductor: 3D Scene Generation from a Single Image with Multi-Agent Orchestration

Jeonghwan Kim¹, Yushi Lan², Yongwei Chen¹, Hieu Trung Nguyen³, Chuanyu Pan³, Xingang Pan¹

¹Nanyang Technological University  ·  ²University of Oxford  ·  ³Meshy AI

🔭 Pipeline

The pipeline runs in three stages, shown above.

• (a) Stage 1 — Initialize Scene. GroundedSAM produces masks. An Opus mask-evaluator merges them. SAM 3D turns each object into a textured GLB. GALP predicts the layout (pointmap, floor polygon, coarse placements).
• (b) Stage 2 — Environment Construction. An Opus vision director designs a rectilinear floor plan. It builds a separable Floor/Wall/Ceiling stage. A look-dev pass matches the photo. Finally it renders 5 reference views.
• (c) Stage 3 — Scene Refinement. A relation graph drives a heuristic + Opus planner pass (attach-to-floor/wall, align, remove). An Opus validator flags problem groups. A dedicated island-refiner agent fixes each group. Then it renders the final 5 views.

🗺️ Roadmap

• Code release
• Checkpoint release
• Codex version — an OpenAI Codex / codex-cli compatible variant of the pipeline

🚀 Quickstart

💻 Terminal = your normal shell · 💬 Claude Code prompt = inside the Claude Code CLI (after claude). Full details: INSTALLATION.md.

✅ Step 1 (recommended) — one-skill setup

# 💻 Terminal
git clone --recursive https://github.com/jhkim0759/SceneConductor.git SceneConductor
cd SceneConductor
claude

# 💬 Claude Code prompt
/sceneconductor-setup

💻 Step 1 (manual alternative)

Set up each piece yourself: INSTALLATION.md.

💬 Step 2 — Run the pipeline

# 💬 Claude Code prompt — recommended: stage by stage
/stage1-initialize-scene <scene_dir>

/stage2-environment-construction <scene_dir>

/stage3-scene-refinement <scene_dir>

# or all three at once
/scene-orchestration <scene_dir>

# 💻 Terminal — or non-interactive
SCENE_DIR=/path/to/scene FORCE=1 bash scripts/build_one_scene_seq.sh

📋 Prerequisites

• 🐧 OS: Linux x86_64. The vendored Blender path targets Linux. macOS/Windows are untested for the Stage 1 SAM3D/GALP GPU paths.
• 🎮 GPU: NVIDIA, CUDA 11.8+, ~30 GiB VRAM peak. The SAM3D Stage 1 post-process is the bottleneck.
• 💾 Disk: ~50 GB free (Blender ~4 GB + checkpoints ~21 GB + per-scene outputs).
• 💬 Claude Code CLI — the pipeline runs on slash commands. Install it from https://github.com/anthropics/claude-code.
• 🐍 conda / miniconda — ./setup.sh builds the five envs. Each is invoked via conda run -n <name>.
• 🚫 Git LFS is not required.

📁 Repository Layout

SceneConductor/
├── .claude/
│   ├── agents/                    # subagent definitions (Haiku/Opus per agent)
│   ├── skills/                    # per-stage skill folders (SKILL.md + src/)
│   ├── rules/                     # shared norm files
│   └── settings.json
├── submodules/
│   ├── GALP/                      # git submodule (jhkim0759/GALP)
│   ├── Grounded-SAM/              # git submodule
│   ├── SAM3D/                     # git submodule
│   └── Qwen3.6/                   # git submodule
├── scripts/                       # batch runners (build_all_scenes.sh, etc.)
├── setup.sh                       # one-shot conda env provisioner (all 5 envs)
├── DIRECTORYS.yaml                # machine-specific paths
├── INSTALLATION.md                # full install guide
├── CLAUDE.md / AGENTS.md          # project rules
├── README.md
├── checkpoints/                   # gitignored — user downloads
└── blender-4.2.1-linux-x64/       # gitignored — user downloads

📂 Outputs per scene_dir

<scene_dir>/
├── image.png                          # INPUT — the only required file
├── inputs/                            # Stage 1 outputs (masks, GLBs, layout)
├── json/                              # Stage 2 + 3 JSON state
├── blend/                             # Stage 2 + 3 .blend files
├── render/blender_scene_view_*.png    # Stage 2 5-view renders
├── render/final/                      # Stage 3 final 5-view renders
├── relation_groups/                   # Stage 3 per-group islands
└── logs/                              # Stage 1 logs

inputs/relation_graph.json is produced by Stage 3's auto pre-step (stage3-sub-scene-analyze-prepare), not by Stage 1.

😊 Acknowledgements

We thank all the authors who made their code public. It tremendously accelerated this project.

• Grounded-Segment-Anything
• SAM 3D Objects
• Qwen3.5-VL
• Blender
• Claude Code

📚 Citation

If you find our work helpful, please consider citing:

@misc{kim2026sceneconductor3dscenegeneration,
      title={SceneConductor: 3D Scene Generation from a Single Image with Multi-Agent Orchestration},
      author={Jeonghwan Kim and Yushi Lan and Yongwei Chen and Hieu Trung Nguyen and Chuanyu Pan and Xingang Pan},
      year={2026},
      eprint={2606.08402},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.08402},
}