Sayash Kapoor - How seriously should we take AI X-risk? (ICML 1/13)

Machine Learning Street Talk (MLST)

How seriously should governments take the threat of existential risk from AI, given the lack of consensus among researchers? On the one hand, existential risks (x-risks) are necessarily somewhat speculative: by the time there is concrete evidence, it may be too late. On the other hand, governments must prioritize — after all, they don’t worry too much about x-risk from alien invasions.

MLST is sponsored by Brave:

The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at brave.com/api.

Sayash Kapoor is a computer science Ph.D. candidate at Princeton University's Center for Information Technology Policy. His research focuses on the societal impact of AI. Kapoor has previously worked on AI in both industry and academia, with experience at Facebook, Columbia University, and EPFL Switzerland. He is a recipient of a best paper award at ACM FAccT and an impact recognition award at ACM CSCW. Notably, Kapoor was included in TIME's inaugural list of the 100 most influential people in AI.

Sayash Kapoor

https://x.com/sayashk

https://www.cs.princeton.edu/~sayashk/

Arvind Narayanan (other half of the AI Snake Oil duo)

https://x.com/random_walker

AI existential risk probabilities are too unreliable to inform policy

https://www.aisnakeoil.com/p/ai-existential-risk-probabilities

Pre-order AI Snake Oil Book

https://amzn.to/4fq2HGb

AI Snake Oil blog

https://www.aisnakeoil.com/

AI Agents That Matter

https://arxiv.org/abs/2407.01502

Shortcut learning in deep neural networks

https://www.semanticscholar.org/paper/Shortcut-learning-in-deep-neural-networks-Geirhos-Jacobsen/1b04936c2599e59b120f743fbb30df2eed3fd782

77% Of Employees Report AI Has Increased Workloads And Hampered Productivity, Study Finds

https://www.forbes.com/sites/bryanrobinson/2024/07/23/employees-report-ai-increased-workload/

TOC:

00:00:00 Intro

00:01:57 How seriously should we take Xrisk threat?

00:02:55 Risk too unrealiable to inform policy

00:10:20 Overinflated risks

00:12:05 Perils of utility maximisation

00:13:55 Scaling vs airplane speeds

00:17:31 Shift to smaller models?

00:19:08 Commercial LLM ecosystem

00:22:10 Synthetic data

00:24:09 Is AI complexifying our jobs?

00:25:50 Does ChatGPT make us dumber or smarter?

00:26:55 Are AI Agents overhyped?

00:28:12 Simple vs complex baselines

00:30:00 Cost tradeoff in agent design

00:32:30 Model eval vs downastream perf

00:36:49 Shortcuts in metrics

00:40:09 Standardisation of agent evals

00:41:21 Humans in the loop

00:43:54 Levels of agent generality

00:47:25 ARC challenge