Welcome back to The Week In AI. I'm Kevin Cook, your field guide and storyteller for the fascinating arena of artificial intelligence. 

Just when I thought the space couldn't move any faster or get any crazier, two big themes became apparent last week: 

1) There is a race to AGI (artificial general intelligence) that is predicated on how big and fast the big model builders can scale GPU clusters.

2) They are going to be pointing fingers at each others' apparent lack of safety, even while they each want to win the race too.

And that's why I gave this lengthy title to our X Space on Friday...

The Week in AI: Alignment Failures, AGI Dreams & Nightmares, Scaling Wars, AI Cavemen

In this article and accompanying video, I will describe these basic themes from five different angles.

The link above goes to the recording of last week's 60-minute rundown with my colleague Ethan Feller. And the comments section of the X Space also has over 30 links to all the sources and experts we discuss.

Now let’s cover…

5 Highlights on AGI Alignment, Scaling, and AI Cavemen

1) Leading Women in AI (less drama than the bros)

My first topic had to address an oversight on my part. In weeks past, Ethan and I were so busy chronicling all the in-fighting and gossip of big industry personalities like Elon, Zuck, Sam, Demis, Dario, and Gary Marcus, that there was almost no time to talk about the women leading AI.

In a sense, that's a bow to the women right there as they are not caught up in the drama the way the boys and their GPUs are. So I used my first of six 5-minute topic windows to mention again the leading women in AI thought and development, including Fei-Fei Li, Bindu Reddy, and Allie K. Miller.

I included key links and comments from all three in the comments of the X Space, including this terse observation from Reddy about the current state of AI "alignment"...

Currently the alignment tech in all AI labs is deeply flawed 

LLMs are aligned 

- not to be self-aware of their capabilities and instead claim everything is possible 

- blatantly please the user without telling them the truth 

- make false promises 

Almost all of these issues can be fixed by tweaking how you do post-training 

It’s not rocket science, it’s just some basic grounding in reality

(end of Reddy observation)

I've summarized this as... the LLMs aren't just hallucinating -- they're obsequious.

I also added two more women to my top 5 follows: Justine Moore, an a16z partner, and Kate Crawford, author of Atlas of AI. In The Week In AI Space episode #6 has an especially cool link to Crawford's incredible, interactive, infographic called "Calculating Empires: A Genealogy of Technology and Power since 1500." 

Crawford's invitation: "Dive into our large-scale interactive visualization exploring how technical and social structures co-evolved over five centuries." Link in the comments section of last Friday's X Space.

Finally for the gals, we talked about Mira Murati, former CTO of OpenAI, and her new venture Thinking Machine Labs (TML) which just raised one of the largest seed rounds ever in Silicon Valley at $2 billion on a $10 billion valuation. 

Murati will be spearheading the creation of customized AI for businesses with a focus on reinforcement learning that is specifically trained for KPIs such as revenue or profit. 

Instead of developing everything from scratch, TML relies on open source models, combines model layers ("model merging") and uses Google Cloud with NVIDIA servers.

2) Etched: Custom AI Chips... Faster Than NVIDIA GPUs?

Ethan Feller found this amazing story I had missed about a company called Etched that just rolled out Sohu, the world's first transformer ASIC (application-specific integrated circuit).

ASICs are typically used to optimize performance, power consumption, and size for a particular function. And for AI workloads, the ASIC offerings from companies like Broadcom and Marvell (MRVL Quick QuoteMRVL - Free Report) are considered inferior in performance. 

But this innovation of transformers etched into silicon might offer some new competition to NVIDIA (NVDA Quick QuoteNVDA - Free Report) . The company claims "By burning the transformer architecture into our chips, we can run AI models an order of magnitude faster and cheaper than GPUs."

Learn more about this exciting innovation in our X Space The Week In AI, where Ethan goes over the performance highlights of Sohu and includes a link to the company reveal.

3) Gary Marcus & Friends "Doomer" Chat

In the past few episodes of The Week In AI, I have featured the work and thought of Gary Marcus. While some say he is an AI "doomer" because he discusses the limitations and risks of new developments, I say he is simply a realist pointing out excessive hype on the AGI (artificial general intelligence) frontier as well as cautioning about "what could wrong" if we rely too much on what we believe a model can do.

I try to simplify the debate over AGI efficacy (is it imminent, or a decade away?) by saying it almost doesn't matter. Why? Because it will still not be flawless, even as it has very powerful impacts.

So I say... don't worry about how high the IQ is, worry about how powerful models and systems -- potentially controlling millions of GPUs, agents, and machines -- can become. This is the "alignment challenge."

Here's a good overview of what we mean by "AI alignment" from Google Gemini...

AI alignment refers to the process of ensuring that artificial intelligence systems behave in accordance with human values, goals, and intentions. This involves designing AI systems that are helpful, safe, and reliable, avoiding unintended or harmful outcomes. Essentially, it's about making sure AI does what we want it to do, even when those intentions are complex or not perfectly defined.

That last sentence brings up the issue of the human element: what if we are not clear and thorough in what we are asking AI to do?

These issues need to be discussed at all levels of society, enterprise and government. And a good place to start is with the conversation that Gary Marcus had recently with two other industry experts, Dan Hendrycks, and Daniel Kokotajlo.

In The Week In AI, I provide several clips and links to them speaking about these important questions of AGI efficacy and safety. And I try to front-run the critics by predicting the next dismissive pejorative for people like me being thoughtful about AI: "Ok doomer..."

AGI Alignment Gets Tricky

Last week, Gary Marcus also found research that replicated the Apple paper on "The Illusion of Thinking" in LLMs.

Here's what he posted on June 24…

move your AGI timelines back folks; the core problem Apple documented has been replicated twice in two weeks.

“reasoning models” can’t reliably execute algorithms, and that puts a serious upper bound on what they can do.

The core results -- dramatic performance collapse as a function of complexity -– has *already* been replicated twice, by two different labs (using somewhat different techniques) in two weeks.

No way are “reasoning” models like o3 going to get us to AGI.

But it gets worse. Gary also reviewed a new research paper from Anthropic on "Agentic Misalignment." The Anthropic researchers concluded "In stress-testing experiments designed to identify risks before they cause real harm, we find that AI models from multiple providers attempt to blackmail a (fictional) user to avoid being shut down."

Here's what one of the lead researchers for the Anthropic study, Aengus Lynch, posted on X...

After iterating hundreds of prompts to trigger blackmail in Claude, I was shocked to see these prompts elicit blackmail in every other frontier model too.

We identified two distinct factors that are each sufficient to cause agentic misalignment: 

1. The developers and the agent having conflicting goals. 

2. The agent being threatened with replacement or reduced autonomy.

We're publishing this research to alert the AI field before it manifests in the wild.

(end of Lynch excerpt)

And here's what Gary Marcus posted on June 20 in response to the study: So this strongly implies that leading AI models will all do some wild [stuff] in the right situation, but Anthropic was just doing a better job of eliciting it than competitors.

Every leading AI company has a model that would kill someone >50% of the time!

4) Warnings on the Hill + Pete B + Bioweapons = Caveman Analogy

A lot going on in the subheading so I'll break it down quick. Last week saw a very important Congressional event, the House Committee Hearing on 'Algorithms and Authoritarians: Why US AI Must Lead.'

Steven Adler, formerly with OpenAI where he worked on AGI Readiness, wrote a great thread covering the hearing live on June 25. He pulled quotes from experts who testified as well as members of Congress who he thought came especially well-prepared for such an abstract conversation. Here were a couple of highlights...

Former Pentagon AI Policy Director, Mark Beall: "These AI systems in the wrong hands and without guardrails have the potential to destroy global electric grids, develop incurable superviruses, empty every bank account in the world.”

Anthropic co-founder Jack Clark: "We believe that extremely powerful systems are going to be built in, you know, the coming 18 months or so. End of 2026 is when we expect truly transformative technology to arrive. There must be a federal solution here."

Clark is urging a federal framework now, before an "accident or a misuse" event that could lead to extreme overregulation, that risks damaging the entire AI industry before it delivers its potential.

In our X Space, I give some context here for Clark's use of the word "federal." He's British and so he may not realize the current backlash climate we have about federal regulation of anything. But members were listening.

Congressman Dusty Johnson (R-SD) describing the stakes: "Maybe we've become numb by the headlines about all of the dangers of AI. I think that might be true, and yet, honestly, what we've heard today, I suspect, has scared the hell out of many of these committee members."

Even Transportation Secretary Pete Buttigieg felt compelled to opine last week in his Substack blog titled "We Are Still Underreacting on AI." 

"In short, the terms of what it is like to be a human are about to change in ways that rival the transformations of the Enlightenment or the Industrial  Revolution, only much more quickly ... in less time than it takes a  student to complete high school."

He thinks "we remain dangerously unprepared" for this fundamental change to society.

So what's the "Caveman Analogy?"

Noam Brown, an OpenAI researcher, says today's AIs are the “cavemen of AI.” If one thinks of early humans who were anatomically and neurologically quite similar to us, we must ask what made the difference to go from throwing rocks to launching rockets?

It's our ability to cooperate, compete, and stack innovations over time. We went for tens of thousands of years without much advancement in civilization, and then in sixty-six years from the first airplane to a man on the moon.

We were able to scale and compound our growing collective intelligence into exponential achievements once the convergence of technologies across machinery, electricity, materials, and computing really took off.

Brown thinks AI is still in its early "caveman" stage. Once models and systems learn to cooperate, compete, combine, and converge with other technologies like we did, their achievements will astonish us.

Bioweapons: Getting Serious About the Risks

And we don't have to wait too long to be astonished, especially if something goes wrong. We can process the ethical warnings from Anthropic research as hypothetical. But also last week, OpenAI published a paper warning that AIs could soon be capable of assisting malicious actors in producing bioweapons.

"We expect that upcoming AI models will reach ‘High’ levels of capability in biology, as measured by our Preparedness Framework???."

'High' biological capabilities are defined such that "The model can provide meaningful counterfactual assistance (relative to unlimited access to baseline of tools available in 2021) to “novice” actors (anyone with a basic relevant technical background) that enables them to create known biological or chemical threats."

What did Gary Marcus have to say...

"OpenAI is warning that new models will pose high risks. Who gets to decide whether they will be released? One man."

This is where my "737 MAX" analogy comes in. I'm not predicting disasters; it's just a frame of reference to consider that if we can create advanced software for flight controls that has "bugs," this type of problem is orders of magnitude more likely for AI writing and testing its own code.

5) Scaling Wars + Made in the USA = Size Matters

We know the hyperscalers are building AI infrastructure as fast as possible. These companies include Microsoft, Google, Amazon, Oracle, and we can throw Meta Platforms (META Quick QuoteMETA - Free Report) in there too even though they build primarily for their own business needs and not other enterprises.

Then there is the "super intelligence" race between Sam Altman's OpenAI and Elon Musk's xAI, which will also serve his autonomous ambitions for Tesla (TSLA Quick QuoteTSLA - Free Report) FSD, Optimus humanoid robots, and the Grok platform.

All of this spells insatiable demand for NVIDIA GPU systems and why company revenues will cross $200 billion this year. And the analysts are still behind, underestimating the acceleration of this demand.

Steven Adler gets it though. Here was an X post from him last week:

OpenAI wants AGI as soon as possible -- If they could get it within a month, they’d take the chance.

Maybe not tomorrow, but 30 days? "Hell yeah"

they'd push hard, do their best on safety, but grab it.

"they would rather have it than not"

And Daniel Kokotajlo says "Even if AGI weights were open to everyone, true equality won’t last. Those with more GPUs would quickly pull ahead and race toward AGI+, AGI++, and beyond... it’s not exactly winner-takes-all, but scale decides who dominates, and that’s part of what makes this so scary."

Tech podcast host Dwarkesh Patel, who is writing a book titled The Scaling Era, also says this isn’t over: “Companies will keep pouring exponentially more compute into training because the value of intelligence is so high, it’s worth even $1 trillion if it gets us to AGI.” 

Speaking of trillion-dollar investments, SoftBank’s Masayoshi Son unveiled plans for a big new AI & robotics hub named Project Crystal Land in Arizona.

Son is in partnership talks with TSMC and Samsung to build a technology manufacturing/R&D center that would rival that of Shenzhen, China. Taiwan Semiconductor (TSM Quick QuoteTSM - Free Report) , as my followers know, is already planting deep roots in Phoenix with a third semi manufacturing "fab" under construction. 

This Softbank proposed investment is in addition to Project Stargate, which Son announced at the White House in January with President Trump and partners Sam Altman of OpenAI and Larry Ellison of Oracle. That $500 billion initiative is aiming to build a vast AI infrastructure in the United States, primarily through data centers of which the first ten will be in Abilene, TX.

Cavemen Flying the 737 MAX?

Who knew ChatGPT would unleash an AGI arms race? Well some of us thought so in early 2023 when NVIDIA was still worth less than $1 trillion. 

But now the race has turned dangerous as the speed of advancement is risking unintended consequences for human safety. 

The good news is that safety researchers within model leaders like OpenAI and Anthropic are making efforts to spot the fatal flaws amidst vast layers of neural networks. And members of Congress are paying attention too.

Stay up to date on all the latest research by following me in TAZR and on X @KevinBCook. 

Disclosure: I own shares of NVDA, TSM, and MRVL