Weekly Roundup

From the world of AI safety, commercialization, and open communication standards

Terms of engagement. What’s in a protocol? As covered by the Batch this week: “Stripe announced Stripe Agent Toolkit, a library for Python and Typescript that supports agentic workflows that use API calls to execute monetary transactions. You can download it here.” A number of mutually supportive efforts are underway to make the decentralized growth and integration of AI agents into the internet economy a reality. Anthropic released a new open-source Model Context Protocol (MCP), which allows developers to connect their AI agents to any information and tools supported by this protocol. This could also be an important way to constrain model behaviour and isolate model risk. An open standard protocol could help keep AI companies’ models honest. Perhaps this is what we should be talking about when we discuss AI ‘‘disclosures’’?

AI Safety Institutes take a peek into Anthropic’s upgrades. In a test of (international) cooperation between governments and industry, the US AI Safety Institute in collaboration with the UK AI Safety Institute and Anthropic conducted a pre-deployment safety evaluation of Anthropic’s newest model, Upgraded Sonnet 3.5. The model was evaluated for biological risks, cybersecurity risks, software development capabilities & agentic risks, and model safeguards. Although it scored record highs for pure performance, both AI safety institutes were able to jailbreak the model, getting close to 100% model compliance in providing the researchers with harmful information for criminal activity and on HarmBench (pg.35). But it’s unclear if Anthropic made any adjustments to its model based on this testing, prior to its release. And the updated system card leaves us none the wiser!

Distributed LLM training. Prime Intellect and Nous Research are hard at work on distributed training, with the goal of training new smaller foundation models efficiently on GPUs spread throughout the world. This opens the door for smaller companies – and even users – to collaborate on training foundation models of their own. Prime Intellect describes their work as “democratizing AI development and preventing the consolidation of AI capabilities within a few organizations”. Prime Intellect released INTELLECT-1, a 10B foundation model trained on 1 trillion tokens, while Nous Research is training a 15B foundation model for 100 billion tokens. (By comparison, LLaMa 3 was trained on 15 trillion tokens and comes in sizes ranging from 1b to 405b parameters.)

When advertising comes to town. OpenAI wants to incorporate ads into its products as it looks to increase revenue outside of its consumer subscriptions and low-margin API. Has society sufficiently considered what guardrails are needed to limit deception or excessive paid content in AI output? Based on the model system cards for the major AI model developers, we think not. Bloomberg reports that Black Friday holiday deals in the U.S. were increasingly sought out via AI sites, highlighting what will be its growing influence in redirecting web traffic. One AI site widely used for Black Friday shopping was Weever AI, where the influence of Reddit in its rankings is very evident.

We can steer you..right? Speaking of guardrails for monetization (‘commercialization’) risks, a new study by researchers at the Max Planck Institute highlights that in the course of a conversation, AI can “steer” users to specific products, shifting their preferences, even when they are aware that AI is actively attempting to do so. Still, no routine evaluations (‘evals’) currently exist for assessing AI’s persuasiveness and potential for misinformation in commercial contexts. Most evals focus on outputs not conversations; and assume an isolated AI technology rather than one integrated into troves of user data, as would be the case in commercial contexts.

Watch that (vertically integrated) space. Amazon introduced Nova, its foundation model, as it prepares its cloud infrastructure, AWS, for the deployment of AI products across the AI stack. In doing so Amazon’s vertical integration in the AI space is becoming a bit more potent – a key risk the FTC wants us to investigate. Amazon’s vertical integration is on top of its deepening partnership with Anthropic’s Claude model, becoming the primary source for its model’s training, and using Amazon’s own chips for it. AWS was falling behind though, argues Semianalysis, “due to weak hardware specs and poor software integration”. Amazon is still in crisis mode, they note, due to their internal models Titan and Olympus having failed, but maybe this will serve as a major course correction.

Music to your ears? If you want to be blown away, listen to this demo by Nvidia’s text-to-audio model called “Fugatto” (short for Foundational Generative Audio Transformer Opus 1). It generates sound from text. It can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice – even let people produce sounds never heard before. They describe it in more detail: here. But it’s a must listen to.

A new type of scholar. A collaboration between Allen Institute for AI and the University of Washington has released OpenScholar: an all-in-one research assistant designed to answer academics’ research questions in science, medicine, physics, and computer science using academic and scientific sources. Operating without the need for an internet connection, it searches through 45 million published research papers and answers a user’s questions using a response grounded in this literature and related detailed citations. It reportedly has “superior generation quality and citation accuracy compared to existing proprietary systems”, performing well on ScholarQABench.