12/15/24
Reasoning Machine --ar 16:9 --sref 2214001679 3194121359 --v 6.1
Every month, I host a bookclub where we read fiction and nonfiction to provoke thought on how human society will shape and be shaped by AGI. We’ve read books like Frankenstein, The Unabomber Manifesto, Klara & the Sun, and more.
In December, we read the ur-text of AGI & Safety/Alignment, Nick Bostrom’s Superintelligence. The book (written pre-LLM) outlines a few arguments for how AGI will emerge, what shape it may take, and what consequences of misalignment might look like. Modern rationalist, accelerationist, and doomer philosophy is all some kind of response to this text.
It’s hard to get people to agree on the shape, mean, and deviation of AGI’s outcomes (to say nothing of takeoff, polarity, and alignment) - I’d contend most of this disagreement stems for different definitions of what actually compromises “AGI”. Even amongst the 8-odd people who came to the discussion, there was a fair amount of divergence in definition. I thought I’d share a few of the interesting ones - I thought it would be cool to memorialize a handful of PoVs as an artifact when we decide in the future which is accurate. I’ve anonymized the participants, but included some approximation of what I think their philosophy can be summarized as.
Nick Bostrom (cautious pessimism): Any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest. (Pulled directly from the book.) No clear ex-ante measure.
Neel & Attendee 1 (cautious optimism): AGI is literally artificial, (not organic material), general (can function across different domains), and intelligent (can extrapolate from incomplete information). I’ve previously written about this PoV. Ex-ante measure has already been passed around GPT-2/3 level.
Attendee 2 (rational pessimism/Yann LeCun school): AGI is something that can “reason”. This is defined as being able to understand some abstract principles, and then deterministically apply them when useful. LLMs do not meet this definition, largely because they are stochastic output machines - most models will say that 9.11 > 9.9 for this reason. The ex-ante measure for this could be the ARC AGI prize - one of the only few benchmarks that has withstood incursion from different model classes.
Attendee 3 (rational skeptic): AGI is some combination of both cognitive capability and embodied intelligence. Unclear how to measure either of these ex-ante.