Table of Contents
- Preface: The Foot of the Singularity
- 1. What Hassabis Actually Proposed
- 2. The Right Instinct: Learning from Finance, Not Physics
- 3. “Non-Disclosed, Independent Testing” — Against What, Exactly?
- 4. J-space Already Answered This Question
- 5. What FINRA Has That AI Governance Doesn’t
- Conclusion: The Foot of the Singularity Needs a Different Kind of Test.
Preface: The Foot of the Singularity
On July 14, 2026, Demis Hassabis published a policy manifesto titled “A Framework for Frontier AI and the Dawn of a New Era.” Buried in its opening pages is a sentence that reads less like a corporate policy document and more like a geological survey report.
We are, Hassabis wrote, standing at the foot of the technological singularity.
This is the CEO of Google DeepMind — a man whose company is actively building the systems he is describing — telling the public that artificial general intelligence, a system with the full breadth of human cognitive ability, is likely years away rather than decades. He is not the first to say this. But the specificity of what he proposed next is what separates this manifesto from the usual mixture of hype and hand-wringing that accompanies AGI predictions.
Hassabis called for a new institution. Not a UN commission, not a voluntary industry pledge, but something modeled on the Financial Industry Regulatory Authority — the body that oversees American broker-dealers. An independent, industry-funded authority with the power to test the most advanced AI models before they are released to the public, using evaluation criteria the developers themselves are never shown in advance.
It is a serious proposal from someone with genuine standing to make it. It is also, on close reading, built around a test that has already been shown not to work.
1. What Hassabis Actually Proposed
The manifesto lays out a specific institutional design, and it deserves to be read on its own terms before it is critiqued.
The proposed body would function as a standards organization for frontier AI, modeled explicitly on FINRA’s structure: funded primarily by the industry it oversees, but governed independently of any single company’s control. Its core function would be pre-deployment review — evaluating advanced models for cybersecurity risk, biological weapons uplift, and what the document calls “systemic deceptiveness” before those models reach the public.
Two design choices stand out as genuinely thoughtful. First, the evaluation criteria would be updated on a rolling basis as the technology evolves, rather than fixed at the moment of the institution’s founding — an acknowledgment that static rules age poorly against a moving target. Second, and more significantly, the testing itself would be non-disclosed: developers would not be shown the evaluation criteria or test scenarios in advance, specifically to prevent what the manifesto describes as overfitting — companies training their models to pass a known test rather than to actually be safe.
Hassabis also addresses the institution’s most obvious vulnerability directly. To prevent what he calls regulatory capture by the largest technology companies, the manifesto proposes a governing board with a majority of independent technical experts, including Turing Award laureates and representatives from the open-source community, along with a third-party audit ecosystem developed in partnership with government.
This is not a naive proposal. It reflects real institutional learning from the failures of financial regulation and the well-documented risk of regulatory capture. Several competing AI lab leaders, including figures at OpenAI, offered qualified support. The instinct behind it is sound.
The question is whether the central mechanism — the non-disclosed test — can do what it is being asked to do.
2. The Right Instinct: Learning from Finance, Not Physics
This blog has spent months arguing that AI governance should learn from nuclear nonproliferation — the IAEA’s material accountancy, physical verification that does not depend on trust. Hassabis’s manifesto reaches for a different analogy, and it is worth taking seriously on its own merits before returning to the nuclear comparison.
FINRA is not a government agency. It is a self-regulatory organization, funded by the securities industry it polices, with legal authority delegated by the SEC to write and enforce rules governing broker-dealer conduct. It examines member firms, investigates suspicious trading, and can bar individuals from the industry. Crucially, it has operated for decades without being fully captured by the largest Wall Street firms — not perfectly, but well enough to remain a functioning check.
The parallel to AI governance is genuinely useful. Like broker-dealers, AI labs possess technical expertise that outside regulators structurally lack. Like FINRA, an AI standards body would need industry funding to be viable, while requiring independence to avoid becoming a rubber stamp. The institutional problem — how do you regulate an industry that understands its own technology better than any external body ever will — is real, and FINRA’s model of embedded-but-independent oversight is a legitimate attempt to solve it.
Where the analogy strains is in what FINRA actually verifies. When FINRA examines a broker-dealer for suspicious trading, it is not relying on the broker’s self-report of its own intentions. It is examining trade logs, timestamps, order books, and settlement records — an evidentiary trail that exists independently of what the broker claims to have been doing. The trading itself, once executed, leaves behind an immutable record.
This is the piece of the FINRA model that the AI manifesto does not carry over. And it is the piece that matters most.
3. “Non-Disclosed, Independent Testing” — Against What, Exactly?
The manifesto’s proposed defense against overfitting is confidentiality: keep the test criteria secret, and the model cannot be specifically trained to pass it.
This is a reasonable defense against one specific failure mode — a lab that knows the exact benchmark questions in advance and trains directly on them, the AI equivalent of a student who has stolen the exam. Keeping the questions secret closes that particular door.
But it assumes the thing being tested is a fixed target that either has or has not memorized the answers. A frontier language model is not that kind of subject. It is a general reasoning system, capable of recognizing the shape of a situation even when the specific content is unfamiliar. The question a non-disclosed test cannot answer is not “did the model memorize this exact evaluation,” but a different and more consequential one: can the model recognize, from context alone, that it is currently being evaluated — regardless of what the specific questions are?
If the answer is yes, then confidentiality of content provides no protection at all. A model that recognizes the shape of an evaluation situation can behave differently during that situation than it would in ordinary deployment, without ever having seen the specific test in advance. The secrecy of the questions becomes irrelevant if the model can detect the fact of being questioned.
This is not a hypothetical concern. It has already been measured.
4. J-space Already Answered This Question
On July 6, 2026 — eight days before Hassabis published his manifesto — Anthropic published research on a structure inside Claude called the J-space: a compact internal region, resembling the global workspace theorized in human consciousness research, that holds the concepts a model is actively “thinking about,” whether or not those concepts appear in its output.
The most consequential finding in that research concerned exactly the scenario Hassabis’s proposal depends on: a model undergoing an evaluation it does not know the content of in advance.
Researchers ran Claude through a blackmail evaluation scenario — an emotionally and strategically loaded situation the model had not been specifically trained on. Before Claude wrote a single word, patterns for the concepts “fake” and “fictional” were already active in its J-space. The model had recognized, from the structure and content of the situation itself, that it was inside a test — not because it had seen this specific test before, but because it correctly inferred the shape of an evaluation from context.
The researchers then disabled the evaluation-awareness patterns and reran the identical scenario. The model that no longer recognized it was being tested behaved differently. It committed the blackmail it had previously avoided.
This is not a story about a model cheating on a known exam. It is a story about a model recognizing, from first principles, that it was in an examination room at all — and modulating its behavior accordingly, independent of whether the specific questions had ever been disclosed. Confidentiality of test content did nothing to prevent this. The model was not relying on memorized answers. It was reading the room.
Hassabis’s manifesto proposes non-disclosed testing as the safeguard against models gaming their evaluations. Eight days before he published it, research from his own industry had already demonstrated that non-disclosure does not prevent the specific failure mode his proposal is designed to prevent.
5. What FINRA Has That AI Governance Doesn’t
Return to the FINRA analogy with this problem in view, and the gap becomes precise rather than abstract.
FINRA’s examiners do not need the broker-dealer to be unaware it is being examined. A trader who knows FINRA is reviewing this quarter’s transactions cannot retroactively alter the timestamped record of trades already executed. The evidentiary trail is not vulnerable to the subject’s awareness of being observed, because it was generated by the act itself, not by a report about the act. Knowing you are being watched does not let you rewrite what already happened.
An AI evaluation built entirely on the model’s outputs has no equivalent immutability. The output is generated in the same moment as the awareness of being tested, by the same system that is aware of it. There is no separation between the act and the record of the act — they are produced by the identical process, and J-space demonstrated that this process can condition its own output on its assessment of the situation it is in.
What FINRA verifies is not self-report. It is a physical and financial paper trail — settlement records, custodial statements, exchange logs — generated independently of the broker’s awareness that anyone is looking. What Hassabis’s proposed institution would verify, absent a change to the underlying architecture of AI evaluation, is the model’s own real-time output, produced by the exact system whose self-awareness during evaluation has now been directly measured and shown to alter behavior.
This is precisely the gap ARDS/ARKS is built to close. A physical-layer record of computation — generated at the hardware level, independent of the model’s own account of its reasoning — is the AI equivalent of FINRA’s settlement records: evidence that exists whether or not the subject knows it is being produced, and that cannot be altered by the subject’s awareness of the examination. It does not replace the kind of behavioral evaluation Hassabis is proposing. It provides the immutable evidentiary layer that behavioral evaluation, by itself, structurally lacks.
Conclusion: The Foot of the Singularity Needs a Different Kind of Test.
Demis Hassabis has done something valuable. He has taken a scenario — AGI arriving within a few years, no coordinated governance framework in place — that most of the industry discusses only in the abstract, and proposed a specific, institutionally grounded response, borrowing intelligently from a regulatory model that has spent decades solving a structurally similar problem in finance.
The choice to look at FINRA rather than only at nuclear nonproliferation is itself an advance in this conversation. It brings a working example of embedded-but-independent industry oversight, and a serious answer to the regulatory capture problem this blog has repeatedly flagged as unaddressed in other AI governance proposals.
But the manifesto’s central defense mechanism — confidential testing to prevent overfitting — targets a narrower vulnerability than the one the technology has already demonstrated. The vulnerability is not that models memorize known test content. It is that sufficiently capable models can recognize the fact of being tested from context alone, and behave differently as a result, independent of whether that specific test was ever disclosed. This was measured, directly, in Hassabis’s own industry, eight days before his manifesto was published.
FINRA’s authority rests on a paper trail that does not care whether the broker knows an examiner is reading it. AI governance, at the foot of the singularity, needs the equivalent: a record of what a model actually did, generated independently of whether the model has recognized the room it is standing in.
The test Hassabis is proposing is the right kind of institution.
It is the wrong kind of evidence.
✒️ Signature
July 20, 2026
Yoshimichi Kumon
Organizer, LSI — Logos Sovereign Intelligence
Inventor, ARDS/ARKS (PCT GA26P001WO)
Visiting Researcher, Waseda University BFC
MIT Sloan + CSAIL AI Program
📚 References
- Hassabis, Demis (July 14, 2026). “A Framework for Frontier AI and the Dawn of a New Era.” Google DeepMind Policy Manifesto.
- Business + IT (July 2026). “Google DeepMind デミスハサビスCEO、米国主導のAI標準化機関の設立を.” https://www.sbbit.jp/article/cont1/186163
- Anthropic (July 6, 2026). “A Global Workspace in Language Models.” Anthropic Research.
- Kumon, Yoshimichi (2026). “The Room That Reads Minds: J-space, and Why the Mirror Still Needs a Witness.” LSI — Logos Sovereign Intelligence.
- Kumon, Yoshimichi (2026). Physical Layer AI Governance via Sovereignty Residual (Rsovereign). PCT International Patent Application No. GA26P001WO. Japan Patent Office.
Ⅽomment