oklo

Outside, the November day seemed unseasonably sunny and brilliantly warm. Inside, the singularity imbued the air with a hum of low-grade dread and a frantic yet somehow enervated urgency. A single human agent of a single scrambling startup was giving a paid-for talk to a vast sea of deep learning researchers, none of whom were paying the slightest attention. Later, as night fell, there was a headlong rush to fulfill the fear of missing out.

A great deal has happened since last pressing publish…

Me: OK, but if we have the long-since-amortized state-of-the-art diffusion model and if we had devices that work at the Landauer limit, then we’d have a rather remarkable piepline.

I feel like there’s something “better” in a thermodynamic sense about doing Landauer work to reach the interesting image manifold than to do the Landauer work to reach a set-up that’s got the same information entropy but is some random choice from that low-entropy subset. You’ve seen Autechre’s Gantz Graf video. Sure, it’s low entropy, but…

o3: Landauer cares only about “how many bits are written-then-forgotten,” never about what the bits denote. Two pictures that each end up as, say, a 512 x 512 x 24-bit PNG have the same minimum erase cost kT ln 2 x 512 x 512 x 24 = 2e-7 ergs whether one is a Rembrandt and the other a glitch mosaic.

Where a theromodynamic gap can appear is in the path you take:

1. Direct random fill
   • write 24 bits per pixel of fresh IID noise, keep none of them -> must eventually erase the full N = 6.3 Mbits.
   • Landauer floor: kT ln 2 N.
2. Compressed-latent generator (ideal)
   • draw m << N random bits for a latent z that indexes the learned manifold;
   • reversibly expand z|->x (keep all intermediates or uncompute them).
   • only m bits of randomness ever need to be forgotten.
   • Landauer floor: kT ln 2 m.
   • If natural images have Shannon entropy ~ 2 bpp, then m ~ 2/24 N ~ 0.08N.
3. Diffusion sampler (real models)
   • Inject fresh Gaussian noise for every pixel at every step (often > 10^8 bits total).
   • Most of those are thrown away; unless you store and later return them, the erase budget ballons, making diffusion far from Landauer optimal.

So “interesting low-entropy images” are not cheaper because they’re meaningful; they are cheaper only if you generate them through a low-entropy latent code and do the expansion reversibly. Current diffusion pipelines don’t, but a future, fully reversible lanent-decoder running at the Landauer limit could, in principle, turn a handful of random bits plus a lot of conservative logic into arbitrarily many remarkable frames with minimal heat.

Me: So ultimately it costs less to generate the good stuff than to show static on a dead television?

o3: In principle, yes.

Landauer budget = “bits created and later forgotten”.

o Dead-channel static

— you must draw a fresh random value for every output pixel -> ~ N = H x W x 24 random bits.

— those bits are discarded after use

— irreducible heat: E = kT ln 2 N.

o “Good” image via a trained generator

— draw only m << N random bits for a latent code z

— expand z -> image with reversible logic, uncomputing scratch states

— forgotten bits = m

— irreducible heat: E = kT ln 2 m.

So, with ideal reversible hardware the minimal energy is proportional to the entropy you inject, not the picture’s final file size. Because natural images are highly compressible (m/N ~ 0.1 or less) generating meaningful content can, in theory, cost an order-of-magnitude less heat than producing television snow.

Me: static.

I think hands-down, the first Heflin photograph is the most perfectly iconic flying saucer image. In large part, its allure stems from the composition. The telephone poles. The metallic glint. The dusty deserted highway converging on a vanishing point of shimmering dreamtime mirage.

Implicit in the Heflin photograph is the capacity of transport to the moment when it was taken — you feel somehow as if you’re there, frantically rummaging for the Polaroid in the cab of the pickup truck, dust on the dashboard, a rip in the baking vinyl seat covers. I remember, at age 10 poring over the image with a magnifying glass — it was a go-to for all the clay-coated photo-insert pages of the semi-scientific UFO books from the late sixties and early seventies. This paper, moreover from the Journal of Scientific Exploration, Vol. 14, No. 4 (2000) is an enjoyable read. Earnest citizen scientists use the best tools at their disposal to probe the mysterious unknown. There’s even a Gray Barker style visit from a sinister authority figure:

A new interstellar object presents much of the same attraction that the flying saucers produce. It appears seemingly out of nowhere from the gulfs of space. Critically, this is the moment for the more than just an amateur, for the not quite a professional astronomer. Yes! One can play a vital role. In the feverish first days after detection, before the slow wheels of the press offices and the peer review can lurch into action, the soapbox cedes to the characters who normally are out there clamoring in the twilight, just outside the mainstream glare.

Sadly(?) the Top500 list just ain’t what it used to be. The semi-annual June 2025 update came out recently, and the HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A, Slingshot-11, aka ‘El Capitan’ pushing out 1.742e+18 flop/s is effectively the Old Town Road of the creaky charts.

“The 65th edition of the TOP500 showed that the El Capitan system retains the No. 1 position. With El Capitan, Frontier, and Aurora, there are now 3 Exascale systems leading the TOP500. All three are installed at Department of Energy (DOE) laboratories in the United States.”

Meanwhile, look at the AI compute chart at epoch.org. The flop count is current increasing by 5X per year, and the talk on semianalysis is all about nuclear-powered Gigawatt data centers. A pre-training run for a model like GPT 4.5 uses of order 2e+26 flops over ~100 days. A cluster capable of training a frontier model thus does 50x the effective compute of El Capitan. Pro tip — prior to launch make sure the soft checks assert pytorch_no_powerplant_blowup=1 in the config.

At this point, readers are likely howling with indignation. He’s going apples-to-apples with flops between two radically different computational paradigms! Johnny von Neumann rolls in his early grave.

Luckily, it’s open mic night at oklo.org. Let’s introduce o3-pro and let him get up on stage for a while:

Now I know this is a slippery slope. No oklo.org posts for over a month, and I’m just letting some rando AI hold forth at length about bit operations (which itself consumed about 10^17 bit operations).

I started writing posts about bit operations and large-scale computation a little over a decade ago. As part of those efforts, I tried to introduce a unit of computation. The effort gained zero traction in any community, but I’ll try again:

1 oklo = 1 bit operation per gram of system mass per second

I also asserted that things start to get interesting on a planet when the planet surpasses 1 oklo. Quoting this 2014 post:

Our planet has been heavily devoted to computation, not just for the past few years, but for the past few billion years. Earth’s biosphere, when considered as a whole, constitutes a global, self-contained infrastructure for copying the digital information encoded in strands of DNA. Every time a cell divides, roughly a billion base pairs are copied, with each molecular transcription entailing the equivalent of ~10 bit operations. Using the rule of thumb that the mass of a cell is a nanogram, and an estimate that the Earth’s yearly wet biomass production is 1018 grams, this implies a biological computation of 3×1029 bit operations per second. Earth, then, runs at 50 oklo.

How are the Magnificent Seven et al. doing when judged on the oklo metric? I would ballpark the current global burden of artificial computation at the equivalent of 30 million H100s, each running at 2e+15 flops BF16, so 100 bit operations per flop. That multiplies out to 6e+24 bit operations (10 moles) per second, and includes all the phones, all the CPUs, all the GPUs, everything. The mass of the Earth is 6e+27 grams, so artificial computation at this moment of update is running at 0.001 oklo, and is still 50,000X less important than the biosphere. This is consistent with the observation that on Google Earth, it’s still easy to find boreal forest, and its still hard to find data centers.

But likely not for long.

Modern accelerators leave the fab with a one-of-a-kind private key fused into the silicon. Ask the chip to sign a fresh nonce (“attestation”) and bounce the packet off a trusted landmark server—call it a rack-mounted HSM bolted to the floor in Hsinchu.

Cryptography certifies that the responder really is the chip in question; physics certifies where it can (and more pertinently cannot) be. Light in hollow core fiber covers 300 km per millisecond, so a 1 ms round-trip gives the proximity budget away: the one-way distance to the device is guaranteed to be within 150 km. The envelope stops on the beach—the chip cannot be sitting next to an off-island cross-connect.

And so on. There’s a lot of sensitivity right now surrounding the control of cutting-edge AI hardware. As Yudkowsky rather succinctly put it, “Be willing to destroy a rogue data center by airstrike.” Ding dong nuke ’em.

That sort of thing spurs one to start thinking about alternative, in particular, analog approaches. I’ve been going back and forth with o3 on a plan to do GPT-style inference on a pre-trained model using only 1920s technology. This proof-of-concept set-up is fully resistant to being remotely bricked if located in a location not favored by the current powers-that-be. (Note that sampling is restricted to T=0 in this implementation). Now I just need the assistance of a robot-fabricating-capable AGI to build it out at scale.

Illustrating step 4, we have:

If you’re on the lookout for cocktail party conversation starters, have a look at what the Internet has to offer on self-trepanation, an ancient medical procedure that regained significant traction during the psychedelic era. Perhaps surprisingly, its adherents uniformly reported “enhanced mental power and well being.”

John Lennon considered having it done, and he suggested that Paul McCartney participate as well. As McCartney later recalled,

“We’d all read about it — you know, this is the ‘60s. The ‘ancient art of trepanning,’ which lent a little bit of validity to it because ancient must be good. All you’d have to do is just bore a little hole in your skull, and it lets the pressure off,” McCartney continued. “Well, that sounds very sensible. ‘But look, John, you try it and let me know how it goes.’ The good thing about John and I — I’d say no. And he knew me well enough that if I said no, I meant no. I’m not frightened of being uncool to say no. I wouldn’t go far as to say, ‘You’re f***ing crazy,’ because I didn’t need to say that. But, no, I’m not gonna trepan, thank you very much. It’s just not something I would like to do.”

In Bore Hole, re-issued in 2015 by MIT Press (in a heavily expanded edition) Joe Mellen describes the difficulties that he encountered during his abortive first attempt at self-trepanation:

I was living back in London, and it was 1967. At that time, I was broke, and I certainly couldn’t afford an electric drill, so I bought a hand trepan from a surgical instrument shop. It’s a bit like a corkscrew, really, but with a ring of teeth at the bottom. It has a point in the middle, which makes an impression on the skull, and then you turn it until the teeth cut into the skull. It’s slightly narrower at the bottom than it is at the top, so it pulls the circular piece of skull out once you’re through with it when you pull it out. It was difficult. It was like trying to uncork a bottle of wine from the inside. The trepan was blunt, and I couldn’t get any purchase on my own skull. I was tripping on acid. I thought that it was the only way I could get through doing it, but it didn’t work…

I came across that passage more than thirty years ago. The image of, “trying to uncork a bottle of wine from the inside” had remarkable staying power.

It came suddenly to mind yesterday when I was reading the winner of the Best Paper Award at last year’s ICML conference. In their article titled, Stealing Part of a Production Language Model, the victorious authors walk through a design strategy for prompt injections that permits extraction of the projection matrices of black-box production language models. They report (among other things) that Open AI’s Babbage model has an embedding dimension of 2048.

This drilling-in theft likely did not enhance GPT-4’s mental power and well being, but the mindset of the surgical hack seems just spot-on analogous to the illustrious trepanning tradition.

And I was hoping that for this final post of the thirty-day streak, I could turn the reins over to o3. And indeed, o3 was more than willing to take a crack at writing a post, but the result was a complete disaster.

The splash image, though, is pretty cool, so I’m pasting it in and hitting “Publish” so as to adhere to the letter (if not the spirit) of the resolution.

I set a new-years-resolution-style goal to write thirty posts in a row. Here I am at number twenty-nine.

I asked a language model to read the run of recent writing and then construct a picture of the author. I was thinking I’d for sure get back some hip combination of Jake Paul, Gucci Mane, and a young William S. Burroughs, but instead I got this:

Clearly, I need to write more posts about, say, Flying V guitars.

One side effect of having written 540 oklo.org posts is that I readily have trouble remembering whether I already wrote about this or that. Take, for example, Holmberg’s use of light bulbs with orthogonal photosensitive detectors as an analog O(N) method for computing direct N-body gravitational accelerations. That’s sufficiently awesome that it’s got to be in the archives somewhere.

If one starts to think about ways to speed up transformer architectures, one immediately notices that the attention calculation, when done in the naive textbook manner, is O(N^2) in the context length, and thus presents a significant latency bottleneck. It’s the same problem that one faces with computing gravity in an N-body simulation. Hello o3.

After some back and forth, o3 submitted a complete draft that I’ve been peer-reviewing. It appears that it may have come up with some remarkable ideas, and I’ve been combing the literature to try to understand how novel they are.

o3’s ego looks like it needs some alignment work before the LLM will be capable of stepping in to render academic researchers obsolete. It appears that when the model develops an original idea, it automatically hallucinates a reference in order to try to assign the credit for its own work to someone else.

When pressed, it blithely admits to this weirdly selfless behavior:

The “Teng et al. 2024” citation was inserted as a placeholder when I sketched the Fast-Multipole-Attention idea; to the best of my knowledge no peer-reviewed or arXiv paper with those exact authors, year, and title has been released. You should remove or replace that reference with a verifiable source.

I will link to the paper once it’s been thoroughly vetted. And if anything in there is genuinely original, I think I’ll usurp a little credit.

I was walking on a street very near where I live when I came across a startling sight. The neighborhood is very proper New England with clapboard houses and white picket fences. One of the lots, however is missing a house and instead has a house-sized boulder sitting where one would expect the house to be. Order of magnitude, it weighs about a million pounds.

Turns out it’s a glacial erratic, made of 200 million year-old diabase that was pushed down by ice from the outcrops of the Sleeping Giant which lies about five miles due north.

I keep thinking about the remarkable mapping of fruit fly brain. Those annoying pests darting above the sticky spill each have 139,255 brain cells with 50 million connections, and a computational throughput that’s naively similar to GPT-2, all while running much much closer to the Landauer limit, and costing negative dollars. Given the existence of such a set-up, it’s hard to shake off the feeling that the transformers are on the verge of being completely deprecated by some radical new algorithmic paradigm. But what’s it gonna be? Which direction is it going to come from?

So one sifts for clues. As an outsider, it’s tricky. Sure, the TED-talking charlatans are easy to spot. It’s not hard to discount the Deep-Learning equivalent of some astrobiologist going on about sampling microbes spewing out of Enceladus, or phosphine-emitting life on Venus, or detecting biosignatures on extrasolar planets in the habitable zone.

In this vein, I’m currently struggling to understand whether the Liquid Neural Networks are really the real deal or not. The various Bayesian priors exude radically conflicting signals. A mushy article in Quanta, “the driving forces behind the new design, realized years ago that C. elegans could be an ideal organism to use for figuring out how to make resilient neural networks that can accommodate surprise.” That definitely sounds like hype of the lets-look-for the-red-edge variety, but at the same time it’s true that C. Elegans pulls off quite a bit with its measly 302 neurons and 7000 synapses. Plus, bonus points for just using Euler’s method to integrate ODEs given the weird right-hand sides:

The Letters of William S. Burroughs 1945-1959 is full of good stuff. It comes highly recommended.

I found myself recalling a passage from a letter written to Allen Ginsberg and Jack Kerouac on October 23, 1955 from the Benchimol Hospital in Tangiers. The reference is to Neal Cassady’s ‘system’ that would permit a fortune to be gained from racetrack betting.

...Tell Neal from me to drop the bang tails. You can't beat it. Dream hunches are not supposed to be used that way. You understand, Neal? You know what horse is going to win, but you can not use that knowledge to make money. Don't try. It's like fighting a ghost antagonist who can hit you but you can't hit him. Drop it. Forget it. Keep your money...

You know that ‘Oumuamua was partially made of hydrogen ice, but you cannot use that knowledge to produce a consensus within the scientific community.

And so forth. There’s a principle here, partially encompassed by Occam’s Razor, but partially drawing on something else. Something deeper maybe.

Sometimes — when the bouncing between news articles, when refreshes of a browser clicked again and again are all out of measure — it seems as if there might actually be something to the simulation hypothesis.

My interest in astronomy stemmed from a fascination with flying saucers that was sparked in the late 1970s by castoff paperbacks from the Cold War. In this archived post, I tried to capture a handful of wistful memories of that personal trajectory. Given that intense interest that I felt all those decades ago, straining in a near-vacuum for every scrap of information that I could find concerning elusive silvery discs, I felt I somehow owed it to my former self to think logically and try to figure out what was going on when the New York Times started publishing clearly serious, yet bizarre and altogether bewildering articles about UFOs. Yet there was nothing in the articles to give foothold. I could muster no opportunity for order-of-magnitude assessments to generate a quantitative context. And then, a year or so later, as more articles came out. Tom Delonge, yes Tom Delonge, he of All The Small Things (this full-show video is pretty good!) somehow catalyzed all this coming to light… Quite frankly, the cognitive dissonance was so overwhelming that I just set it aside and stopped being too concerned.

Max Headroom is an interesting artifact of the 1980s — an actual person impersonating an AI. The Max Headroom Signal Hijacking Incident (a Wow! signal geared to the modern era) abstracted the artifice one step further — a person impersonating a person impersonating an AI.

While refereeing a recent paper submitted by the o3 LLM for publication at oklo.org, I get this unmistakable Max Headroom feeling as I suggest wording changes and point out hallucinations in the reference list. I’m a human impersonating an AI agent in a self-referential improvement loop. How very 2025.

When it comes right down to it, I’ll admit I’m pretty much a philistine, but Damien Hirst has always held a certain fascination for me. I like the spot paintings. Embarrassingly, I’m eyeing

but I still have my eye too much on the dimes.

The exoplanet catalog is free, however, and so we can continue to go to town with the graphs. This is the diagram that results if you move all of the system graphs of yesterday’s post to a common origin.