For the past couple of weeks, I've been working on a series of memos where I'm trying to explore and ultimately make up my mind on the risks, and likelihood of those risks, that AI poses in relation to:

Employment
The existential basis of humanity

While doing research for these, I came across an interview where Dwarkesh Patel and Grant Sanderson explore AI's evolving role in mathematics. That has given me a basis for some of the ideas and thoughts that I will now share.
Is AI bound to recycle humanity's existing beliefs or can it produce knowledge that is genuinely new? And is this capability domain bound?

This question became of special importance given that, 16 hours ago, on July 19, 2026, mathematician and Anthropic researcher Levent Alpoge posted on X that Claude Fable 5 had produced a counterexample disproving the Jacobian Conjecture.

The Jacobian Conjecture was posed by Ott-Heinrich Keller in 1939, so it had been unresolved for 87 years. Keller made a precise claim about polynomial functions.

The Jacobian Conjecture was one of the most worked-on problems in algebra. It was legendary for many reasons, multiple incorrect proofs were published in peer-reviewed journals over the decades and among algebraists it had the reputation of being a trap that could deceive you into believing your proof was correct. Generations of mathematicians proved special cases and ran computational searches hunting for counterexamples.

If you are like me, you had no idea what a conjecture even is in the first place but based on the scale and strength of the reaction that this announcement received, I knew it must be important.

Math has a strict division of labor between four kinds of objects:

Definitions: Assert the meaning of a word, can't be right or wrong, just agreed.
Functions: A set of allowed inputs and a rule giving each input exactly one output.
Statements: Assert something about functions. A statement must be true or false.
Conjectures: Precise statement, necessarily either true or false, that seems plausible and important, but that nobody has settled either way

While a proof, which is needed to prove a statement, is very difficult to check, verified over the course of months if not years by mathematicians a counterexample can be verified by anyone plugging the numbers in.

AI and Math

AI's role in math isn't a secret, and the track record predates this weekend. In July 2025, experimental models from both OpenAI and DeepMind scored gold-medal level on the International Math Olympiad. In August 2025, researcher Sébastien Bubeck showed GPT-5 producing a new proof that improved a known bound in a convex-optimization problem

All of this collided with a prior belief of mine: since the models don't run traditional deductive reasoning, its "truth" could just be what is generally accepted. This loss of entropy could keep us, humanity, in a loop of darkness. Establishing the validity of my belief, in the context of Fable's ability to disprove the Jacobian conjecture is what I wanted to explore further.

This worry is not just hypothetical, several elements of the underlying architecture of LLMs further compound this fear:

Pretraining models rewards models for prediction. Default answer gravitates toward what humans have most often written, the consensus, not toward what is true..

Reinforcement learning biases the model towards answers that human reviewers approve.
A growing share of the internet's text is now model-generated, and it flows back into future training data. Researchers have looked at where this recursion leads and formalized it as model collapse: each generation trains partly on the previous generation's output, rare knowledge thins out first, and the distribution narrows toward its own average.
And even the model's visible reasoning can't fully reassure us: Anthropic's faithfulness research shows models sometimes reach an answer first and then write a chain of thought that rationalizes it. We humans have a similar tendency.

So on questions where humanity is confidently wrong and there's no feedback from reality, the model inherits the error.

How LLMs Answer W/Facts and How They Do Math

A model's ability to answer factually is not because it has saved every fact of the world. During training, the learning algorithm (such as gradient descent) adjusts the model's weights so that it minimizes its "error" or the difference between the correct answer and what it generated. By minimizing this error, it gets good at correctly generating the likely next word and in the process it seemingly absorbs the facts. I say absorb, as the model's weights (literally just billions of numbers) are an abstraction and no single weight means anything on its own.
The absorbed facts only surface when context, assembled from your question, activates the representations saved in the weights.

How does this apply to math?

The model can do math following two pathways:

"Mental math." Learned computation encoded in its weights, but it doesn't make use of an arithmetic processor such as a calculator. Anthropic's circuit-tracing work (the 2025 "Biology of a Large Language Model" interpretability papers) actually reverse-engineered how Claude adds two-digit numbers: it runs several pathways in parallel. One estimates the answer's rough magnitude, another computes the ones digit precisely using something like "things ending in 6 plus things ending in 9 end in 5".

The model isn't necessarily making use of or apply the rule we learned in school (work right to left; when a column overflows, carry the one aka the carry algorithm). Training just found some computation in the weights that produces answers consistent with arithmetic.

If you asked the model how it did it, it would due to its lack of self-understanding or misinterpreting your intention, tell you it applied the arithmetic rules.

Just because the model can correctly tell you what the gross profit formula is doesn't mean it applies that formula when asked to calculate a company's gross profit. It's generating both from its weights. But, it can increase the likelihood of a correct answer if it first generates the formula into its context, which is then an input into its next generation. This is why chain of thought, either prompted or trained into the model, can increase its accuracy.

Tools. The model can use an agent harness (when we send a message in the Claude or ChatGPT app, we are interacting with a harness that routes the message to the underlying LLM) to recognize that a prompt needs exact arithmetic, and instructs the model write Python or to call an external calculator tool, and read back the result. A lot of the perceived math improvement in recent models is really improved judgment about when to delegate.

How do Humans do Math?

Heuristics. That pile looks bigger than the other; it therefore has a higher number of pebbles. (This one runs so deep it's pre-verbal — infants and animals estimate magnitudes before any schooling.)
Memorized fact retrieval. 7×8=56 stops being computed at all.
Explicit algorithm execution. For anything beyond the memorized table, you consciously run the procedure you were taught. Some have enough working memory to do this in their head; the rest of us write it down and the paper is doing exactly what chain of thought does for the AI model.

So dismissing AI models by claiming "it's just pattern-matching, not real reasoning" turns out to implicate humans at the same time. In reality, when given a problem, and given the appropriate context, the AI model searches, hits dead ends, backtracks, and verifies, even though underneath it is generating a distribution of likely next tokens, not necessarily directly applying the laws of math.

What Drives Progress?

Grant Sanderson has an interesting way of positioning how general mathematical progress is driven:

Raw hustle (just computation, no new theorems)
Connections between fields
New ways of thinking about a problem

Expanding on this, there is also an elegance component: progress is not just about solving problems, but compressing knowledge so that it can be applied and understood.

Scoring AI against that taxonomy is, I think, the honest answer to my question.

So Can AI Drive Progress?

Despite the four structural points previously mentioned (pre-training, RL, model collapse, and post-hoc rationalization) it seems that modern AI still has significant advantages over humans when it comes to raw hustle, ability to form connections between fields, and new ways of approaching a problem.

Raw hustle: AI now dominates. It has near-infinite raw hustle (the ability to parallelize), and the only constraints, still very real, are energy and the medium intelligence runs on: the chips inference runs on and the physical infrastructure making up networks.

Connections between fields: Currently weak but can be directed. While the model is an expert in many fields, many have noted its difficulty in naturally drawing from expertise in one field to think about problems in another. I almost see it as the model switching hats depending on a given question, it changes its context and in the process its identity and thethe lens through which it approaches problems. This will be another memo in itself, but the short explanation of why this occurs is because the training corpus is itself siloed, and knowledge retrieval depends on the right words to activate the context it has "stored" in its weights.
This is where humans still play a major part, providing in prompts the correct words that will lead to cross domain activation of model's knowledge.

New ways of thinking: This compartmentalized knowledge, is also an advantage in of it self. This allows it to avoid traps human mathematicians face, including the trap of not asking whether we should be trying to prove a thing impossible rather than hunting for a solution. A human mathematician cannot unread what they know. They carry the field's folklore, their teacher's framings, their own sunk years. The model's context can be restricted and widened.

AI's application of these elements can drive progress and lead to Kuhnian accumulation of anomalies that increase the likelyhood that humanity creates completely new paradigm's of understanding.

Conclusion

I don't have the final answer to the question I opened with. But it is becoming increasingly clear that, at least in some domains, AI will not keep us in a loop of darkness.

The boundary seems to run along domains that have verifiability: in math the proof checks or it doesn't, in programming the code passes the tests or it doesn't. That should shape what we do next in two ways: keep pushing AI's limits in mathematics, where every gain is checkable and self-certifying, and keep working on the architectures and the ways we apply these models so that this capability extends across domains that lack math's built-in verifiability.

In 1843, Ada Lovelace observed that a machine "has no pretensions to originate anything; it can do whatever we know how to order it to perform." She was describing the machine of her day. Alan Turing challenged the claim directly in Computing Machinery and Intelligence (1950) proposed that the way for computers to escape this boundary would be to build a "child machine" and train it rather than program it.

We are now fine tuning and building these "child machines" that are redefining the boundaries of computation.

While I am sure we can find many pitfalls, and argue why this in itself poses threats, my natural techno-optimism leads me to the belief that AIwill fundamentally expand our understanding of ourselves and the universe.

SINGULARITY