Pretraining data, not verifiability, is why LLMs are especially good at math (and coding) — LessWrong

kept by eddie sep 21

LLMs' strength in math and coding comes from the abundance of relevant pretraining data they see, not from the ease of verifying answers.

What is neuralese and why is it bad? — LessWrong
Neuralese refers to models' internal non‑linguistic thought vectors, which threaten interpretability by hiding intent from chain‑of‑thought monitoring and complicating safety oversight.
also on LessWrong, #machine-learning, #ai
Futurism Is Always Extreme
Any formal model of the world’s evolution, whether qualitative or quantitative, when run to its normal form, yields only two stable extremes: superintelligent utopia or total collapse.
also on LessWrong, #ai
AI-Written Code Is Still Your Code. Are You OK With That?
AI coding agents lower writing costs but shift the burden to understanding generated code, arguing that human comprehension must remain a core design goal.
also on LLMs, #ai-coding
A Severe Misalignment of AI in Mathematics
Twenty‑five Fields Medalists warn that AI companies’ focus on solving math problems undermines deep understanding and risks misaligning AI progress with the goals of mathematics.
also on LLMs, #ai