I’m partial to that genre of articles where they detail the over-the-top morning routines of over-the-top productive go-getters. Like Mark (formerly Marky Mark) Wahlberg, who gets up at 2:30 AM for a big weight session followed variously by cryotherapy, prayer, supplements and intermittent fasting. That’s pretty bad-ass, you’ve gotta admit.
I set myself the challenge to write an Oklo post each day for thirty days in a row. It’s just over a week in now, and it’s starting to show. I’m gonna hit the sack now and schedule this post to show up at 2:30 AM.
No denying it, Edward Tufte’s The Visual Display of Quantitative Information was a seminal book, and the follow-ups, in particular, Envisioning Information, were remarkable as well. I was exactly the right age to absorb them when they came out. Some of Tufte’s lines are just classic:
… A series of weird three-dimensional displays appearing in the magazine American Education in the 1970s delighted connoisseurs of the graphically preposterous. Here five colors report, almost by happenstance, only five pieces of data (since the division within each year adds to 100 percent). This may well be the worst graphic ever to find its way into print ...
That paragraph falls in the midst of admonitions against self-promoting graphics. Tufte callssuch constructions Ducks, where the reference is to the Big Duck Store on Long Island:
Tufte’s take-away, as explained in a quote (which itself is from a quote)that he pulls from Learning from Las Vegas, seems to be that is is all right to decorate construction but never all right to construct decoration.
Sure, I’m fine with that. But then I’ve always been confused by p. 119, whose single line of text runs, ambiguously, “There are some superbly produced ducks:”
So clearly, p. 119 seems intended to serve as another admonition, yet that graphic doesn’t strike me as bad at all… In fact it’s a great graphic that incorporates many of the stylistic choices (small multiples, etc.) that Tufte generally recommends. The net effect was to leave me feeling confused regarding exactly where I should properly hold forth and grandstand within the whole endeavor of data graphics.
The splash image for this post is definitely a construct of decoration. It was constructed by Sam Cabot using his black-belt photoshop skills for the purposes of decorating our self-promotional press release. It strives for photorealism of a KT-style impact on Venus, back when Venus had a one-bar atmosphere, oceans, life, maybe a civilization, etc. What does the moment of impact actually look like?
In the paper itself, we have another graphic that Tufte would probably dismiss as a duck:
It serves to fix ideas regarding the impacts that could have transferred huge amounts of material from Venus to the Moon, where they’re now churned up in the lunar regolith, waiting to be analyzed and discovered, and chewed over in a few upcoming posts.
On October, 29th, 1991, the Galileo probe, flying through the Main Belt on its way to Jupiter, approached within 990 miles of 951 Gaspra and radioed back the first up-close view of an asteroid — a tiny world too small to be round. I was in graduate school at UCSC at the time. At the beginning of November, a glossy black and white clay-coated photograph showing the view above was tacked to the Departmental Bulletin Board. It had been sent direct from JPL in appreciation of Arnold Klemola’s work on the stellar ephemerides that had permitted the flawless navigation of the close approach. The Mosaic Browser hadn’t yet appeared. I remember staring at the physical photograph for a long time.
When a new type of celestial object is first seen in detail, surprises usually result. Gaspra, however, seemed to conform to the popular impression of what an asteroid should look like. I remember thinking how the depiction of the near-Earth asteroid Adonis in Herge’s Explorers on the Moon (1952) had wound up being rather on-the-mark.
Over the past three-plus decades, spacecraft have visited a whole raft of additional asteroids. The album of returned photographs is a classic of scale invariance in that the images of the asteroids offer almost no clue to their true sizes. It’s a little jarring to see them arranged with their accurate relative sizes, where 14,000 Bennus make a Gaspra and 600 Gaspras comprise a Lutetia.
Uncertainties regarding the true sizes of asteroids would thus seem like a less than front-burner concern. The exact size of this or that particular body would be of enormous interest if it were determined to be on a collision trajectory, but otherwise would presumably be of minor consequence. What you’d think would be a sleepy topic, however, somehow managed to roil into controversy that generated New York Times articles and FOIA requests.
The cherry blossoms this weekend had me recalling moments from the year I spent in Japan.
While I was there, I visited Hokkaido University in Sapporo to give an colloquium. After a day or so, it began to dawn on me that an odd motto, “Boys, Be Ambitious!” is attached (in English) with great frequency to the various affairs both large and small of the University. One of the astronomy graduate students had the phrase written on a post-it note attached to the screen of his computer. In another building, there was a large mural showing a stern, stiffly dressed 19th-century gentleman exhorting a group of reverent students with a longer version of the phrase:
“Boys, be ambitious! Be ambitious not for money or for selfish aggrandizement, not for that evanescent thing which men call fame. Be ambitious for that attainment of all that a man ought to be.”
Which upon reflection, seems to be excellent advice…
The gentleman in the mural, it turns out, is William Clark Smith, the founder and first president of the University of Amherst, Massachusetts. In the mid 1870s, he was enlisted by the Japanese Meiji Restoration government as an Oyatoi Gaikokujin, or “hired foreigner”, to establish an agricultural college in Sapporo (now Hokkaido University) and he made an impression that lasted well over a century. His Wikipedia article is extensive and quite interesting. On the origination of the motto:
“On the day of Clark’s departure, April 16, 1877, students and faculty of SAC rode with him as far as the village of Shimamatsu, then 13 miles (21 km) outside of Sapporo. As recalled by one of the students, Masatake Oshima, after saying his farewells, Clark shouted, “Boys, be ambitious!”
Upon returning to the United States, and flush with the organizational successes and appreciation that he had garnered in Japan, Clark left his academic career, cultivated an interest in gold and silver mining, and embarked on an abrupt, ambitious, and ultimately disastrous foray into the business world. In 1880, he teamed up with a junior partner, John R. Bothwell, to found what might best be described as a 19th-century incarnation of a metals hedge fund. From offices on the corner of Nassau and Wall Streets in Manhattan, the firm of Clark & Bothwell acquired interests in a slew of silver and gold mines across North America, for which they assumed management and issued stock. Clark, as president, got his contacts and colleagues to invest in the venture, and for a period during 1881, the stocks issued by Clark and Bothwell ran up into multi-million dollar valuations.
Clark traveled around the country, promoting the company, acquiring new mines, and seeing to their management, while Bothwell appears to have been responsible for back-office operations. Clark, who had no experience in finance, and little real knowledge of mining geology seems to have spun his wheels, while Bothwell, who had a shady history, actively mismanaged the companies. The operation got into debt, with the outcome being all too typically familiar along the lines of When Genius Failed. By the Spring of 1882, they were facing insolvency, investor lawsuits, fraud allegations, and various other problems. Bothwell disappeared on a train trip to San Francisco, never to be seen again, leaving Clark holding the bag. The story played out to the delight of the Massachusetts and national press.
From the Springfield Republican, May 29, 1882:
… it appears form the beginning that he, as manager of the mines has allowed Bothwell, as treasurer, absolute control of the books and finances of the several companies. It doesn’t appear that he ever examined the books, nor had anybody do so for him, or inquired into the financial condition of each mine, or what was being done with their profits; neither has he required from Bothwell such bonds as the latter’s position should require for the safe handling of moneys entrusted to him…
The scandal made the New York Times, which wrote several articles about the affair, including this one, from May 29th, 1882:
The scandals eventually ruined Clark’s health, and he died four years later, in 1886, at age 60.
The cherry blossoms of Wooster Square looked like they’d reached their peak this morning. My anecdotal feeling was that they were at least one week late this year compared to the past several years.
I know it’s cliche, but I’m drawn to the Kyoto cherry blossom peak date data set, which goes back to the year 812. This year, in Kyoto, the peak date was April 4th.
During the year that I lived in Tokyo, the cherry trees in Ueno Park reached their peak in early April. A week later, I went there on the crowded silent subway, but by that time, the sakura were mostly gone, a greying carpet strewn across the concrete sidewalks.
What’s the most interesting unanswered question in Solar System planetary science? Hands down, I’d argue it’s, Did Venus have oceans, and if it did, when did it lose them?
At first glance, Venus looks promising. It’s got essentially the same mass and radius as Earth. Now that the extrasolar planets provide context, Earth and Venus look like a “pod”, a consequence of the ill-understood process that permits a given system to repeatedly manufacture worlds that are near-clones. Sure, Venus is closer to the Sun. But the clouds reflect most of the light, and it was only in the 1950s that it became clear that perpetual overcast isn’t the only problem with the Venusian weather report.
The climate on Venus might not always have been so bad. The Sun was 30% less luminous at the start of its main sequence life, which means that the sunlight intercepted by Venus was once only 135% of the current terrestrial flux, the equivalent of moving from Detroit to Santa Monica.
This 1971 paper seems to be the first spark of realization that we might have “lost” Venus at some point in the distant past.
From there, it’s not hard to let the imagination run away. Continents. Oceans. Primordial forests. A vanished alien civilization. All within the realm of far-fetched possibility. Dude! Snap back to reality. The hybridization of Occam’s Razor and Bayes’ Theorem states simply, “That super-cool hypothesis of yours? It’s wrong.”
Nonetheless, the question of whether Venus had oceans (and if it did, when it lost them) seems like it should be answerable. How does one make an approach?
It’s easy to think of long-shot ideas, and indeed, I’ve been semi-preoccupied with this question for the last twenty years. Take phosphine — there’s an idea I didn’t think of… If there’s phosphine in the Venusian atmosphere then that points to life, and if there’s life, it’s likely is a remnant holdout from pre-greenhouse oceanic days. A nice story! It’s wrong, of course (see the bold-faced statement above) but it’s beguiling enough that efforts are afoot to spend a lot of money to pursue it.
This blog is named after the Oklo reactors, which demonstrate that under certain special circumstances, the occurrence of uranium criticality can be discerned long after the fact… What if there was a ancient nuclear war on Venus? Are there isotopes or isotope ratios created by a critical assembly of plutonium or uranium that would produce a detectable signature after billions of years? As far as I can tell, the answer seems to be no. What about uranium hexaflouride? It doesn’t occur naturally, it’s the go-to molecule for effecting isotope separation, and it’d be a gas under Venusian conditions. Unfortunately, the acid aerosols in the Venusian atmosphere would clear it out in virtually no time if it were released. It wouldn’t be around a billion years later.
If you scroll down to read the posts from the past year, you’ll find there are several that touch on a more sober-minded approach to the lost oceans problem. The idea explored is encapsulated in this figure:
and the hypothesized sequence (which is pretty cool, so therefore very likely wrong, but not completely certain to be wrong) is as follows:
Venus had water oceans, an atmospheric pressure of order a bar, and plate tectonics.
Steadily increasing solar luminosity drove a runaway moist greenhouse.
Rapid erosion occurred as the oceans were being lost.
Erosion ceased and plate tectonics shifted to stagnant lid volcanism.
Lava from large igneous provinces created the current topography.
Using Earth’s lava production rate points to oceans being lost in the late pre-Cambrian.
Arthur Adams and I worked on this around the time of Covid. It was the subject of an arXiv preprint and a Caltech GPS Departmental Colloquium in March of ’23 :
The buzz was then killed by a daunting, demoralizing referee’s report that left the prospects for a peer-reviewed journal publication very much in limbo, and pointed to the need to find a new approach.
This article from caught my eye:
The Eos article is a popular-level summary of a 2019 paper by Bellucci et al. that reports the discovery of an unusual granite-like fragment, a “felsic clast” in one of the samples returned by the Apollo 14 Mission. The clast contains zircon grains with geochemical formation signatures—specifically formation in the presence of high oxygen availability, low crystallization temperature (~1500° F), and high-pressure (~6.9 kbar)—that are atypical for known lunar rocks but are consistent with Earth’s continental crust. These traits suggest the clast either formed under unusually oxidizing, volatile-rich conditions at a level of order 170km below the lunar surface, or alternately, originated on Earth at shallower depth and was transferred to the Moon by a large impact over four billion years ago. The article pushes for the terrestrial origin hypothesis, and because I’m not an expert, it looked pretty convincing.
Given that set-up, it’s natural to ask — what if the clast is from Venus?
It’s coming up on a decade since the Planet Nine announcement, and the Batygin-Brown article in the Astronomical Journal, has accumulated a remarkable 666,130 (and counting) downloads. To catch myself up, I’m working an overview for Oklo of where the situation stands.
To help encapsulate the process, I’ve made a playlist that strives to obliquely capture the narrative arc of exhilaration, hope, fear, resignation, acceptance, and ultimately optimism that it’s out there… the traces of its presence should already reside on the drives in the RAID arrays, suspended in the exabyte seas…
We were treated to the latest life-discovered-on-an-exoplanet story in this morning’s newspaper:
The scare-quote artist’s impression on today’s big astrobiology breakthrough seems to take its design cues from a mash-up of the stoplight-red red dwarf Gliese 581:
and the halcyon golden-hour seas of Kepler 186-f:
The best part of the article, however, is the apparently unironic use of the, “I’m not saying it was aliens … but …” construction:
During this most recent ‘606 day, JWST observed HD80606 for 21 hours using the MIRI instrument.
Looking at the archive, it appears that the program was successfully completed.
I asked ChatGPT for a run-down regarding the steps taken by the JWST science operations to process and calibrate the raw data to produce files accessible by the P.I. of the observations:
This was sufficient to rouse me in to action, flooded with nostalgia for those good old days, but with a certain steely-eyed cynicism that comes from decades spent on repeated never-quite-definitive quests to extract signal from noise. The world is non-Gaussian.
HD 80606b may be unique in that it’s a Y2K-vintage planet that’s still weird enough to be observed by JWST in a long-duration manner to produce a photometric time-series that can be directly compared with the old observations that were obtained by NASA’s Spitzer Infrared Observatory. I’m beginning to think that this might be quite a significant deal. Consider the iconic Spitzer run-down of HD 187133:
The geometry of the observation and the interpretation for what the planet was doing was as follows. Note that this assumes synchronous rotation holds:
Look at the spike in the photometry that immediately follows the transit. What’s going on there? Must be systematics stemming from the detection ramp. But, then, how do we know that the systematics are isolated to that one odd feature in the light curve? We don’t know.
The Spitzer full-phase photometric curves are rife with systematics. It’s a problem that has never been fully addressed. Where the extrasolar planets are concerned, interpretation has always run out way ahead of the data.
Coincidentally, a new paper on JWST observations of HD187133 was just published. The narrative jumps straight to the maps of the planet, with the goal of improving on those low-resolution thumbnails in the lower-right hand corner of the panel above. The full phase curve — which for Spitzer was certainly impacted by systematics — is not being observed by JWST. As a consequence, it’s unclear whether the remarkable Spitzer data can be replicated (and at what S/N) and moreover, if one looks at the lists of approved proposals, there are many many .pdfs that outline observations that push the optimistic limits.
In contrast to the situation with HD 189733, the first General Observing Cycle for JWST contained two independent HD 80606 programs that replicate long-duration Spitzer campaigns. These two JWST observational programs very similar in spirit. The first obtained a bit less than a day of continuous spectrophotometric data of the events surrounding periastron passage in the 3.8-5.2 micron band. The second (tied to that ‘606 day campaign that wrapped up yesterday) does essentially the same thing at 7-9 microns.
I looked back through my notes from 2007 and found some simulation results that were used to justify the original HD 80606 long-duration photometry sequence obtained by Spitzer during the cryogenic phase of its mission. We used a 2D-hydrodynamics scheme to compute what the emission from the planet should look like at Spitzer’s five photometric bands, and came up with the following then-best-effort diagram:
The resurgence in flux at 45 hours stems from the sun-blasted hemisphere of the planet pseudo-synchronously rotating back into view. The curves rely on exo-planetary properties that had never been measured, so they were necessarily uncertain. Bottom line, it was effectively fishing expedition, as nothing like HD 80606b had ever been observed.
Moreover, at the time of the proposal, HD 80606’s orbital inclination to the plane of the sky wasn’t known. There was a 15% a-priori chance that a secondary eclipse would occur, and we got lucky. It did occur. Placing the yellow 8-micron prediction on the actual data gives this diagram:
The light curve can be compared with the geometry of the encounter:
The exciting result was that the planet clearly undergoes secondary transit. The disappearance of flux when the planet was behind the star unambiguously demonstrates that the photometric light curve is not all just systematics (although if one looks at it closely, systematic trends are likely present — notice how the flux seems to steadily decline prior to the start of the secondary eclipse).
Once we were in possession of the observations, the parameter deck for the hydrodynamics code could be duly and systematically varied, and a model simulation was found that provided a fit to the data. This is the blue line that runs through the photometry. We discovered that the key to getting a plausible match to the planet’s measured light curve is to adopt a short radiative timescale at the 8-micron photosphere. In keeping with widespread practice, the parameter choices that guided the simulations that supported the observing proposal were not extensively discussed in the resulting paper. If one had it to do over, it’d be better to ensure that the initial prediction and the initially predicted S/N were prominently highlighted in the article. What we got displayed a very interesting divergence from what we’d naively expected.
And in fact, now that I have a certain distance from the field, I would argue that there has never been a significant a-priori theoretical prediction in the exoplanet field that proved to be substantially correct. Hot Jupiters? Peas-in-the-pods? Radius Inflation? Ubiquitous Super-Earths? Something needs to be done to start improving on the abysmal track record…
Long-time readers will recall how oklo.org evinced a certain preoccupation with HD 80606b during 2006 through 2009.
In the interim, this blog has largely random-walked to other topics (largely dissipating its readership in the process) but HD 80606b has stayed the course, ratcheting 52 more hair-raising circuits around its parent star, each of which brings it to within seven stellar radii, increases the instellation flux by a factor of a thousand, and then sees the planet flung out to linger for months near a 0.9 AU apoastron where cooling might well proceed all the way to the point where the torrential nightside rains begin to fall.
A ‘606 day occurs every 111.4 terrestrial days, and marks the UTC occurrence of a periastron passage. The current ‘606 day occurred in the HD 80606 system itself sometime between February and May of 1810 AD, and here in the Solar System is currently occurring right now, with the moment of periastron pinpointed to 04-11-2025 06:33:17 UTC, give or take a few minutes.
I was reading, just now, an article about Young Thug’s recent scrape with the justice system, and at the end of the article, it was noted that he’s developed something of anidée fixe with the variable red supergiant star UY Scuti. Yes, you read that right.
Coincidentally, as realized volatility in the E-mini SP500 near-month index futures contract approached annualized levels of order 150 or even 300 this morning, I was finding myself thinking about the contrast between the essentially out-of-control red giants and consummately restrained red dwarf stars. Both classes of objects exist in the same effective temperature range, but the red dwarfs will manage to last for trillions of years before eventually turning themselves into degenerate helium cinders.
In 1979, aged 11, I received a copy of Stewart Cowley’s Spacecraft 2000 – 2100AD as a Christmas gift. Suffice it to say that book blew me away.
Extravagantly produced and coffee-table sized, it combined detailed “realistic” scenes of science-fiction starships with text that matched each lurid illustration to dry technical specifications and a gripping overarching narrative involving an interstellar space war that pitted Alpha Centauri and the Solar System against Proxima Centauri! It was a vision of the future in which the Fermi Paradox had proved to be merely a naive non-sequitur.
There was something about the precise detail and the aggressively near-future sweep that stood in stark contrast to Star Wars and Star Trek. I could tangibly place my future self into the book’s discrete-yet-analog time line. And now, from the standpoint of 2025, nearly a half-century on, rather than facing the imminently scary prospect humanity-replacing AGI, we’d instead be on the cusp of Henri deVass’ breakthrough perfection of the Warp Generator, the development that finally permitted Mankind to slip the surly bonds of the Solar System:
Possession of the physical volume to pore over at leisure was invaluable at a time when the starships of the movie theaters’ big screens were both fleeting and overwhelming. I spend vast stretches of time trying to come up with my own illustrations in the style of the book. I wrote treatments and incomplete drafts of whole new volumes that introduced the planets and civilizations orbiting Tau Ceti and Eta Cassiopeiae into the fray. The results always fell short, but somehow the time spent in failed effort was deeply satisfying.
I heard yesterday that OpenAI’s image generator has made a sudden leap forward, and I wondered how the model would perform on that task I’d spent so much time on decades ago…
SURVEY VESSEL EX-33 Criterion Commissioned by the United Nations Deep Survey Initiative in 2091, the EX-33 Criterion was purpose-built for the long-term observation of Class-D planetary systems. The ship’s unique hull geometry—a flattened, asymmetrical crescent with deeply recessed engine pods—allowed it to remain stable while hovering at the fringe of a gas giant’s magnetosphere for months at a time, shielded from both gravitational shear and plasma wash.
The vessel is shown here conducting a station-keeping operation above Erechtheus, a failed proto-planet in the Epsilon Indi system. Note the minimal thermal signature and open cooling vanes: Criterion is running in low-emission survey mode, relying solely on passive sensor sweeps to monitor the electromagnetic activity of Erechtheus’s inner moons.
The ship’s mission profile was typically geological and atmospheric: recon for isotope-rich atmospheres, early-stage terraforming viability, and monitoring of erratic exomoon orbits. In one instance—Survey Loop 5—the Criterion narrowly avoided a collision with the shepherd moon Osta Minor when its decaying orbit accelerated unexpectedly due to a previously uncharted gravitational eddy in the Erechthean belt.
The ship carried no weapons, relying on its deep-range field scramblers to vanish from any threat profile. Its most famous sighting was in 2104, when it accidentally intercepted a Proxima Hegemony skimmer convoy during a covert insertion into the Delta Pavonis neutral zone. The Criterion escaped undetected.
Yale rivals the world’s best settings for literary efforts that veer in the direction of dark academia. That’s especially the case as October grades into November. Leaves skitter across flagstone paths. Lights glow dimly from ornate windows set high in dark towers.
The Beineicke library contributes tangibly to the mystique. It is a strange blank edifice, with windows replaced by semi-translucent sheets of of marble. Inside, a massive column of rare books is encased within a gridded glass enclosure.
The 234-page Voynich manuscript is easily the Beineicke’s most mysterious possession. Isotope-dated to the early1400s, it is written on limp vellum in an entirely unknown script
replete with weird illustrations
It has famously defied all attempts at decryption.
The manuscript’s known chain of ownership dates to 1576, when it was sold to Holy Roman Emperor Rudolf II. It was acquired by the Jesuits in the 1700s, and was hidden by that order in 1873, when the Roman clerical libraries were nationalized by the Italian state. It was purchased in 1912 by rare book dealer Wilfrid Voynich, who brought it to wide attention, and in 1969 was bequeathed to Yale University.
There’s no particular benefit in attempting to see it first-hand. In addition to the elaborate on-line digitization, the text itself has been carefully transcribed. It’s a straightforward exercise to open a notebook and assess it.
In the transcribed text, period dots delineate words the and the \n’s indicate the line breaks. The 228,845 characters break into 37,044 words which would amount to about 82 pages if typed in 12-point font.
One can wring the text through the go-to NLP paces. Write a script that extracts the character n-grams that respect word boundaries. Filter out rare occurrences, and transform each n-gram into a 128-dimensional ASCII histogram. Apply the k-means algorithm to cluster these vectors, grouping similar textual patterns. Visualize the result with t-SNE, which attempts to reveal how the n-grams form distinct clusters in lower-dimensional (that is, two-dimensional) space… No actual insight is gained from any of that:
Attention, as we’ve all now know, is all one needs. Can we make progress on the Voynich with transformers? It seems instructive to take a first shot-in-the-dark with the current state-of-the-art.
Yeah, that’s not the real model. It’s the iron-curtain alignment and RLHF talking, in the name of keeping poor ‘ol o1-pro from ushering in the Singularity all on its own. I find it very exhausting to wrangle with LLMs, especially when it takes them over a minute to respond. But I tried one more tack, albeit with low expectations.
Somewhere, out there in the fleets of data centers, the moment I hit return, the racks of GPUs multiplied giant matrices at mind-boggling speeds, unspooling a reasoning process that took a full minute and forty seven seconds to elapse.
o1-pro currently provides an ephemeral ongoing trace of its interior monologue. I scrambled to write down what it was thinking, “addressing the conflict”, “balancing translation boundaries”, “balancing creation with caution…” and there we go. I, of course, would have preferred that o1-pro were to throw caution to the wind, but then I’m not the one that spent the billions on Nvidia hardware, and I’m, obviously, too inertia-bound to even bother researching the downloading some unaligned pre-trained frontier model.
And so forth. Lame. Second-rate stuff. The model is just phoning it in. Look at “Section 1”. Ten words of the Voynich text get transformed into 34 words of English text. Hmm. If o1-pro were making an actual effort at translation, its translation would imply that it thinks the Voynich written language has a very high per-word entropy.
Models at the current frontier are training on data set sizes that exceed 10 trillion tokens. This plot is from epoch.ai, a site that deserves a bookmark.
If we inefficiently equate tokens and characters, the above chart suggests that a training data set sized to the Voynich manuscript was a 2010-era product, which coincides with a pre-transformer era when the cutting-edge consisted of recurrent neural network architectures being applied to language modeling tasks.
The first oklo.org post on language generation dates to January 23, 2011, and (among other things) describes the development of the BAM, the Big Automatic Machine. BAM was a decision-tree based framework orchestrated in Lua that generated exoplanet discovery papers. It worked via a systematic end-to-end automation of the Doppler velocity analysis, followed by extensive lists of rules to formulate the text blocks, tables and figures that constituted a LaTeX-able final draft. In retrospect, the Bitter Lesson applies to the approach that it took.
The first oklo.org post on the use of neural networks for language generation dates to two months before Google’s Attention is All You Need arXiv post introduced the transformer architecture. To support that post, I trained a single-layer character-level LSTM with of order 500K parameters on a ~700K character data set that concatenated the text of Wilde’s Picture of Dorian Gray and the English translation of Huysmann’s Au Rebours. The resulting model generated output that — while being readable English — was pretty much nonsense:
The key to the transformer architecture is the quadratic attention mechanism, where every token in the context is allowed to attend to every other token in the context. For a given model size, transformers do a materially better job on language tasks than LSTMs, and when scaled up radically, this has led to the ever-mounting immediacy of the current moment.
In reality, there’s probably not much (if anything) that o1-pro can actually do to translate the Voynich manuscript with a minute and forty-seven seconds of inference. What we can do, however, is train a decoder-only GPT-style model from scratch on the Voynich data, and see how it compares when the identical architecture is trained on English texts of identical size.
A reasonable rule of thumb is to adopt an architecture that is sized to a parameter count that’s of order 2/3 the size of the data. This gives the model maximal “intuitive” power while preventing it from simply overfitting and memorizing its training corpus. For the 228K-character Voynich manuscript, this translates to a model with ~150K parameters. Using OpenAI’s GPT-2 architecture — which is available from the nanoGPT repo on GitHub — we can specify a suitable tiny model as follows. Call it GPT-0:
GPT-0 is a far cry, clearly, from o1-pro, but it’s suitably scaled to the available corpus of Voynich-manuscript text. No data centers required. With a pytorch implementation, it trains interactively on a laptop CPU.
Note that the output directory is 'out-capote-char'. Why? Before training on the Voynich text, I trained GPT-0 on the first 228K characters of Truman Capote’s In Cold Blood, whose opening lines stand so evocatively in the mind’s eye:
The village of Holcomb stands on the high wheat plains of western Kansas, a lonesome area that other Kansans call “out there.” Some seventy miles east of the Colorado border, the countryside, with its hard blue skies and desert-clear air, has an atmosphere that is rather more Far West than Middle West. The local accent is barbed with a prairie twang, a ranch-hand nasalness, and the men, many of them, wear narrow frontier trousers, Stetsons, and high- heeled boots with pointed toes. The land is flat, and the views are awesomely extensive; horses, herds of cattle, a white cluster of grain elevators rising as gracefully as Greek temples are visible long before a traveler reaches them.
After 5000 training iterations, GPT-0 has reached its puny limits. Prompting it with the first four words of Capote’s nonfiction novel and inferring in the standard auto-regressive manner generates:
The village of Holcomb he courting cond taters. But was repic of moilte not of him, here known of to the proped had ached had a mathered. In happent that you like send buick on the boy kilter four he and and brover the […]
And so on — for as long as one likes. GPT-0 is of order a billion times smaller than -o1-pro, and it pumps out pure nonsense. No ghost in its machine.
Nevertheless it, it does project hints of merit. It often gets whole words right, and its failures look like words, or are at least pronounceable — mathered, proped, brover. It capitalizes letters after periods. It places a comma in a sensible spot, and the odd appearance of buick conjures a faint gesture toward the Mid-America-steeped stretches of its training data.
With expectations set by the model’s performance when trained on 228K characters of English text, we can train GPT-0 to similar exhaustion on the 228K characters of the Voynich manuscript. The training curves for the two datasets show that the text of the Voynich manuscript is not like English — it has fewer characters, and the next character of a Voynich sequence is easier to predict than the next letter of an English word. The Voynich text has lower entropy than English, precisely the opposite of o1-pro tried to pass off as its “translation”.
When its context is seeded with a prompt from the manuscript, GPT-0 generates never-before-seen new Voynich text.
This strange incantation is thrilling, somehow, yet amounts to a maddening dead-end. GPT-0 has attained emergent, struggling competence with a language that has never been deciphered, holding its secret tightly within the inscrutable weights of its transformer blocks.
I know that a neural network is not a network of neurons, but for the rough scaling and order-of-magnitude arguments, is it legitimate to equivalence a synaptic connection with a weight in a deep network such as a transformer?
That’s all the sign-off I need to get going.
In a post toward the end of last year, I used synapse-to-weight equivalence to argue that GPT-2 was accomplishing the rough computational equivalent of a fruit fly brain. Next, queue some requisite marveling at the gigantic difference in inference-time energy costs. Note, however, that scientific bloggers run the continual risk of excessive chewing over the same tired themes (IMC-1, anyone?) I won’t burden everyone with more moralizing.
What is clear, however, is that the massive AI-infrastructure build-out that dominates the news these days is making an implicit bet on the perpetuation of extreme computational energetic (as well as likely algorithmic) inefficiency.
Human brain has 100 trillion synaptic connections. Using the equivalence rule of thumb, that makes it somewhere on the order of 100x the size of GPT-4. Let’s consider the energy cost of pre-training. Training the brain uses 100 watts for 25 years. Rounding up a bit, that’s a 1e+18 erg energy budget. GPT-4 was estimated to consume 50 GW-hours to train. That’s 2e+21 ergs for 1% of the model complexity, so it’s a factor of roughly 200,000x less efficient. The immediate implication is that the current transformer architecture, and the current architectural paradigms are nowhere near optimal. If NVDA is to retain their outsize valuation, they’ve got to innovate radically, and probably quickly.
My guess is that somebody else, however, somebody not currently on the radar, will get there first.
There’s a certain tendency for life’s possibilities and potential to unfold magically when you’re young, only to go delayed and unrealized for years, before ultimately sliding out of reach.
And opportunity extends not just to the things one might do, but also to the realm of ideas.
I distinctly recall a frigid winter morning, exactly forty years ago, sitting in the Loomis Laboratory Amphitheater, as our Physics 108 Professor introduced thermodynamics from the kinetic point of view. With that lecture comes the realization that the Second Law is valid only in a statistical sense, and a flash bulb moment illuminates the vast urgent array of possibility. Hah! I thought. It’s not really a law at all. You just have to wait. And wait. Wait long enough and all the molecules will be on one side of the room. Wait for the splattered slime and shells of a smashed egg to draw thermal energy from the carpet and reassemble, arcing up towards your outstretched waiting hand.
During that same month in 1985, I also read William Poundstone’s Recursive Universe. Having no natural resistance, I was entirely blown away. Cellular Automata. The works of Shakespeare buried somewhere out there in the digits of Pi. Zen for Film. It was all completely new and amazing. There are exactly (and there are only!) 65,536^(44,1000) distinct sounds — that is, about 10^212385 sounds — that last for precisely one second and which can be recorded using the (then-futuristic) audio CD format. Appealing proto-Nietzschean reductionist strictures crowd the mind without effort and in rapid succession. Creativity is merely the process of selection, etc., etc.
But as with with Rush’s 2112 or The Fountainhead from Ayn Rand, one tends to grow out of that stuff (albeit, I admit, enriched). And indeed, after a forty-year gap, I’ve started listening to those Rush LPs again. The musicianship really is something else, you know.
By 1999, however, I was sufficiently cool, and I was sufficiently jaded to trot out the “refutation” of the infinite monkeys theorem in our end-of-the-universe book. Like Jim Carroll extorting street cred from all those people who died, died, there was money to be made and low-rent Saganesque fame to be reaped in belittling the amazing consequences of truly huge numbers.
In the end, I think it just comes down to the fact that 10^n is useful to count the number of things that will happen, and 10^n^n counts the number of things that could happen… Or so I hold forth smugly, on a blog almost nobody still reads, as the monkeys and the typewriters and the Dark Era have a go in both a peer-reviewed paper and in last weekend’s edition of the New York Times.
