Scaling 3D Gaussian Splatting to Billions: A Cache-Centric Approach to Out-of-Core Training

Introduction

The transition from neural radiance fields to 3D Gaussian Splatting marked a significant architectural shift in neural rendering. While NeRFs compress scene geometry into the weights of multilayer perceptrons, 3DGS represents scenes explicitly as millions of anisotropic Gaussian primitives. This explicitness enables real-time rasterization and superior rendering fidelity, yet it introduces a severe memory bottleneck that has constrained practical scalability. Each Gaussian primitive carries 59 floating-point parameters encompassing geometric attributes and spherical harmonic coefficients. When combined with gradients and optimizer states during training, a scene containing merely 100 million Gaussians demands approximately 90 GB of memory, far exceeding the capacity of commodity GPUs.

Prior to recent developments, practitioners faced a stark choice: constrain scenes to roughly 11 million Gaussians for single GPU training, scale to approximately 100 million via sophisticated host-offloading techniques, or invest in complex multi-GPU infrastructure. The paper "TideGS: Scalable Training of Over One Billion 3D Gaussian Splatting Primitives via Out-of-Core Optimization" by Zhong et al. fundamentally restructures this tradeoff by recognizing that 3DGS training exhibits the same access patterns that database and operating systems designers have exploited for decades. By treating GPU memory not as persistent storage but as a high-bandwidth cache for a working set determined by camera visibility, TideGS enables training with over one billion Gaussians on a single 24 GB GPU.

The Memory Architecture of Explicit Representations

To understand why TideGS represents a qualitative shift in scalability, one must examine the memory semantics of 3D Gaussian Splatting. Unlike implicit representations where capacity scales with network depth and width, 3DGS stores scene information in a dense parameter table of size N by D, where N is the primitive count and D equals 59 under standard degree-3 spherical harmonic parameterization. Training requires maintaining not only these parameters but also their gradients and Adam optimizer moments, creating a memory footprint that grows linearly with N but with a substantial constant factor of approximately 12 bytes per parameter per state component.

This explicit storage architecture would seem to mandate either massive GPU memory or distributed training across multiple devices. However, Zhong et al. identify a crucial property: visibility-induced sparsity. During any training iteration, only Gaussians visible from the current camera batch receive non-zero gradients and require parameter updates. In large-scale scenes, this active set remains a small fraction of the total primitive count. This pattern mirrors the sparse access patterns observed in recommendation system embedding tables, where only a subset of embeddings are touched per batch despite billions of total parameters.

The insight that 3DGS training is inherently sparse and trajectory-conditioned suggests an alternative to persistent GPU residence. If the active set per iteration is small, GPU memory can function as a working-set cache backed by a slower but larger storage hierarchy. Yet extending this hierarchy to SSD storage introduces severe challenges. SSD bandwidth is orders of magnitude lower than GPU memory bandwidth, and latency is substantially higher. Naive parameter offloading would stall the GPU repeatedly, rendering training impractical. TideGS addresses this through three synergistic techniques that transform SSD-backed training from a theoretical possibility into a practical reality.

The TideGS System Design

The TideGS architecture implements a hierarchical storage system spanning SSD, CPU memory, and GPU VRAM, unified by a virtualized geometry representation. The first component, block-virtualized geometry, addresses the spatial locality requirements of efficient SSD access. Rather than storing Gaussians individually, TideGS packs spatially coherent primitives into blocks aligned with SSD page boundaries. This organization ensures that when the system fetches geometry for a particular view frustum, it retrieves contiguous storage blocks, maximizing throughput and minimizing random access penalties.

The second component, a hierarchical asynchronous pipeline, overlaps computation with data movement across all three storage tiers. While the GPU processes the current working set, the system prefetches blocks anticipated for future camera positions and writes back updated parameters from previous iterations. This pipelining hides latency by ensuring the GPU rarely waits for storage operations, maintaining high compute utilization even when parameters reside on external storage.

The third and most sophisticated component is trajectory-adaptive differential streaming. This technique exploits temporal coherence between consecutive training views. Rather than transferring entire working sets between iterations, TideGS retains overlapping Gaussians in GPU memory and transfers only incremental block deltas. Since camera trajectories in reconstruction tasks typically exhibit smooth motion, the working set changes gradually, meaning most required data is already resident from previous iterations. This differential approach bounds communication costs to the rate of working set change rather than working set size.

The technical results validate this design. TideGS trains scenes with over one billion Gaussians on a single 24 GB GPU while maintaining reconstruction quality competitive with or superior to prior single-GPU baselines. At scales where standard training fits in memory, TideGS incurs less than 15% overhead compared to fully GPU-resident training. In the out-of-core regime, it scales an order of magnitude beyond prior methods, enabling city-scale reconstruction previously requiring multi-GPU clusters.

Analysis and Implications

Beyond the immediate technical achievement, TideGS suggests a broader reconceptualization of how we approach explicit neural representations. The field has spent years optimizing implicit networks for memory efficiency through techniques like pruning, quantization, and distillation. TideGS demonstrates that explicit representations can instead leverage classical computer systems techniques: caching, prefetching, and virtual memory management. This represents a convergence between graphics and systems research that may prove fruitful for other memory-intensive neural architectures.

Several limitations warrant consideration. First, the effectiveness of trajectory-adaptive streaming depends heavily on camera path coherence. For scenarios involving discontinuous view sampling or random access patterns during training, the working set overlap between iterations diminishes, potentially degrading performance toward the bandwidth limits of the SSD tier. Second, the write-back traffic to SSD raises endurance concerns for consumer hardware. Training involves millions of iterations with frequent parameter updates, which could accelerate wear on consumer-grade solid-state drives not designed for such write-heavy workloads.

Third, the current implementation focuses on scaling primitive count while holding the per-primitive parameterization constant at 59 floats. There remains substantial opportunity for compression within the parameter table itself through quantization or codebook methods, which would further extend scalability. The paper does not address whether block virtualization could accommodate such compression without disrupting spatial coherence assumptions.

Looking forward, TideGS opens several research directions. The framework naturally extends to multi-GPU settings, where the SSD tier could serve as a shared backing store for multiple devices, potentially enabling training with tens of billions of primitives. Additionally, the working-set concept might apply to dynamic scenes, where temporal coherence in object motion could drive differential updates rather than purely camera trajectory. Finally, the integration of level-of-detail techniques with out-of-core streaming could allow even larger scenes by adaptively loading detail based on view distance.

Conclusion

TideGS resolves a fundamental tension in 3D Gaussian Splatting between representation capacity and hardware accessibility. By recognizing that training accesses only a trajectory-conditioned subset of primitives at any moment, the system virtualizes the parameter table across a storage hierarchy, enabling billion-scale reconstruction on consumer GPUs. This approach shifts the scaling bottleneck from GPU memory capacity to storage bandwidth, a dimension where hardware trends favor continued expansion through NVMe and future storage class memories.

The work also poses questions about the future of neural scene representations. As explicit methods grow to encompass city-scale environments, we may see increasing adoption of database-inspired systems techniques for managing neural primitives. The distinction between training a neural representation and querying a spatial database begins to blur, suggesting that the next generation of 3D reconstruction tools may emerge from the intersection of computer graphics and large-scale data management systems. Whether implicit or explicit representations ultimately dominate may depend less on rendering quality than on which paradigm better accommodates the memory hierarchies of future hardware.