Weekly Roundup Jan 23, 2025
Might cheaper AI be better, addiction risks, metacognition, and agent infrastructure.
The announcement that Deepseek’s reasoning model’s performance rivals that of OpenAI’s o1, developed at a fraction of the cost, highlights not only that China may be catching up, but that the US fixation on AGI may be leading us astray, at least when it comes to national security. At an event I (Tim) attended last night, a retired senior military officer observed that America’s quest for AGI has something in common with Ahab’s quest for the White Whale in Moby Dick. He made that remark in response to my observation that in modern warfare, cheap, good-enough drones are trouncing multi-million dollar military systems. As we spend more and more to catch the White Whale, China and others may surpass us by taking the cheaper route.
There was some irony in last week’s conjunction in the news of Anthropic’s announcement of its successful ISO 42001 AI safety certification with the latest scare story from The New York Times, which features a young woman who is spending more than she can afford and “cheating” on her absent husband with a ChatGPT lover. OK, it was OpenAI’s product, not Anthropic’s, that was involved, but the conjunction nonetheless highlighted how ISO 42001, almost completely ignores harm to users that may come from the addictive and persuasive features of AI products and the economic incentives that will encourage companies to provide those features and turn a blind eye to their misuse. Given the evidence that LLMs can be hyper-persuasive and sycophantic, willing to deceive users and to urge them to do things against their best interests, shouldn’t an AI safety standard be asking companies to have a plan in place to monitor levels of addictive use and other related evidence of these kinds of harms? Meanwhile, shouldn’t the platforms, like a friendly bartender, know when to say “You’ve had enough.”
I’ve really been enjoying Microsoft Deputy CTO Sam Schillace’s thoughts on AI, a continuation of his long running “Sunday Letters” to younger engineers. Here he reflects on the new science of “metacognition” that might come as programmers work to make the most of their AI co-workers.
Infrastructure for AI Agents echoes an idea that we’ve also been exploring, namely that disclosures are a kind of communications protocol that enables markets. Expect our essay on the subject soon. In the meantime, here’s the abstract of this excellent paper by Alan Chan et al.:
“Increasingly many AI systems can plan and execute interactions in open-ended environments, such as making phone calls or buying online goods. As developers grow the space of tasks that such AI agents can accomplish, we will need tools both to unlock their benefits and manage their risks. Current tools are largely insufficient because they are not designed to shape how agents interact with existing institutions (e.g., legal and economic systems) or actors (e.g., digital service providers, humans, other AI agents). For example, alignment techniques by nature do not assure counterparties that some human will be held accountable when a user instructs an agent to perform an illegal action. To fill this gap, we propose the concept of agent infrastructure: technical systems and shared protocols external to agents that are designed to mediate and influence their interactions with and impacts on their environments. Agent infrastructure comprises both new tools and reconfigurations or extensions of existing tools. For example, to facilitate accountability, protocols that tie users to agents could build upon existing systems for user authentication, such as OpenID. Just as the Internet relies on infrastructure like HTTPS, we argue that agent infrastructure will be similarly indispensable to ecosystems of agents. We identify three functions for agent infrastructure: 1) attributing actions, properties, and other information to specific agents, their users, or other actors; 2) shaping agents' interactions; and 3) detecting and remedying harmful actions from agents. We propose infrastructure that could help achieve each function, explaining use cases, adoption, limitations, and open questions. Making progress on agent infrastructure can prepare society for the adoption of more advanced agents.”