Study notes I'm keeping as I learn to program. The ideas here come from Imran Ahmad's 50 Algorithms Every Programmer Should Know — I'm sharing them in my own words. The plain-language explanations, the examples, and any mistakes are mine.
The concept in plain language
Punchline first: the technology behind the eerily photorealistic 3-D captures showing up in phones and film is built from two ideas you already half-know — the bell curve, and “how fast things change here.” None of this sits in a silo. Let’s build the ladder one rung at a time.
Rung one: a Gaussian is just the bell curve. Every physician reads one daily without ceremony: lab reference ranges are bell curves. A normal sodium isn’t a single number — it’s a hump of likely values, densest in the middle, fading smoothly at the tails, described completely by a center (the mean) and a width (the standard deviation). Now free the bell curve from the lab report. In two dimensions it becomes a soft hill; in three, a fuzzy blob — dense at its center, translucent toward its edges, with a center, a size, and (if you let the widths differ by direction) an orientation. Hold that image: a soft, stretchable, tiltable blob of density. That is all a 3-D Gaussian is.
Rung two: a derivative is “how fast things change right here.” Speed is how fast position changes; a rising creatinine’s slope is how fast kidney function is worsening. Zoom in far enough on any smooth curve and it looks like a straight line — the derivative is that line’s steepness, a local linear stand-in for a complicated curve. Now widen the idea to many dimensions at once. A camera projection takes a point in 3-D space (three input numbers) to a pixel on your screen (two output numbers). Ask “if I nudge the point a hair in each 3-D direction, how does its screen position shift?” and the answer is a small grid of slopes — every output’s sensitivity to every input. That grid is the Jacobian. It is the derivative’s grown-up form: the local linear window that tells you how a tiny patch of space gets stretched, squeezed, and rotated by a transformation. Complicated mapping, honest simple summary — valid right here, in this neighborhood.
Rung three: put them together and you get Gaussian splatting. 3D Gaussian splatting (Kerbl, Kopanas, Leimkühler & Drettakis, 2023) represents a scene as millions of those soft 3-D blobs, each with a position, shape, color, and transparency. To draw the scene from your viewpoint, each blob must be projected — “splatted” — onto the screen. Projection is a warping, nonlinear map, but each blob is tiny, and inside a tiny neighborhood the Jacobian’s linear window is an excellent stand-in. So the renderer uses exactly it: the Jacobian of the camera projection turns each 3-D Gaussian into the right stretched, tilted 2-D ellipse on screen — the local linear approximation doing real work, a trick worked out in the EWA splatting literature (Zwicker and colleagues, 2002) two decades before. Layer millions of soft ellipses, blend them front to back, and a photograph-like image emerges.
Where you’ve seen it in the real world
This is the technology behind the current wave of photorealistic 3-D capture — walk around an object or a room with an ordinary camera, and software reconstructs a scene you can fly through from any angle. Earlier “radiance field” methods (NeRFs) produced stunning stills but rendered slowly, because answering “what color is this pixel?” meant querying a neural network over and over. The Gaussian-splatting paper changed the trade: represent the scene as explicit blobs instead of a network, splat them with the Jacobian trick, and the authors reported real-time rendering (≥ 30 frames per second at 1080p) with state-of-the-art visual quality. Real-time is the threshold that moves a technique from “research demo” to “product” — which is why splat-style captures have spread quickly into phone scanning apps, VR walkthroughs, visual effects, and game tooling. When you see a drone-scanned house you can tour in a browser, or a film set digitized for reshoots, you are very likely looking at millions of bell curves.
The healthcare angle
Two bridges, one honest and one already under your feet.
The honest one: 3-D reconstruction is an active research direction in medicine, not deployed practice. The appeal is easy to state. Surgical planning and simulation want faithful, explorable 3-D models built from ordinary imaging; endoscopy and laparoscopy produce video from a moving camera — exactly the input these reconstruction methods eat — and research groups are exploring splat-style techniques for reconstructing anatomy from such footage, where real-time rendering matters because a simulator must react at interactive speeds. Whether these approaches become clinical tools depends on the same gauntlet every clinical technology faces: validation on real cases, failure-mode analysis, and regulatory review. Treat this paragraph as a map of potential, not a claim that your hospital does this today. (Illustrative and educational — not medical advice, and not a product claim.)
The bridge already under your feet: you have trusted Gaussians your whole career. A reference range is a statement that a population’s values form a bell curve and that we act differently in its tails. Splatting simply promotes the same object from describing a lab value’s uncertainty to describing where matter sits in space — density highest at the blob’s center, fading at its edges. The math didn’t change when it left the lab report; it just got a graphics card.
Why it matters when you work with AI
Here is why this deep-dive ends the ladder rather than decorating it: differentiability is the engine of modern AI, and the Jacobian is the engine part. A neural network learns by asking, millions of times: “if I nudge this internal knob slightly, does my error go up or down, and how fast?” That question is answered by derivatives, and pushing the answer backward through a network’s many layers — backpropagation, the algorithm popularized by Rumelhart, Hinton & Williams in 1986 — is nothing more than chaining local linear windows together: each layer’s Jacobian says how small changes pass through it, and multiplying the windows layer by layer tells every knob its share of the blame. Gaussian splatting trains the same way — the renderer itself is differentiable, so each blob’s position, shape, and color get nudged by gradients until the rendered images match the photographs.
So the ladder closes into a loop. The bell curve from your lab reports gives us a soft, adjustable building block. The derivative — “how fast things change here” — grows up into the Jacobian, the local linear window on any complicated transformation. Rendering uses that window to draw; learning uses that same window, chained, to improve. One piece of calculus, pointed in two directions. When someone tells you a system “learned,” you now know the mechanism underneath: not magic, not understanding — millions of tiny, local, honest measurements of how fast things change here, followed accumulatively downhill. That intuition travels to every AI system you will ever evaluate.
Sources
- Kerbl, B., Kopanas, G., Leimkühler, T. & Drettakis, G., 3D Gaussian Splatting for Real-Time Radiance Field Rendering (2023) — ACM Transactions on Graphics / SIGGRAPH (preprint, arXiv)
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering — project page, Inria / GraphDeco
- Zwicker, M., Pfister, H., van Baar, J. & Gross, M., EWA Splatting (2002) — IEEE Transactions on Visualization and Computer Graphics
- Rumelhart, D.E., Hinton, G.E. & Williams, R.J., Learning representations by back-propagating errors (1986) — Nature
— Jeremy Tabernero, MD · More in this series · Get in touch