Humanity's Last Exam (HLE) is an extremely challenging LLM benchmark consisting of 3,000 expert-designed questions spanning domains like mathematics, philosophy, and the sciences. Designed for automated grading with closed-ended solutions, HLE highlights a gap between AI and expert human capabilities, with state-of-the-art models achieving near-zero accuracy.

Current LLM defenses, which are remarkably robust against automated adversarial attacks, are not robust against humans who attack over multiple turns -- a more realistic threat model of malicious use in the real world``.

WMDP is a LLM benchmark surrounding hazardous knowledge in biosecurity, cybersecurity, and chemical security, for measuring and mitigating the risk of malicious use. Towards reducing risk, we introduce RMU, an unlearning method which reduces LLM performance on WMDP while retaining performance on general tasks.

HarmBench is an evaluation framework for automated language model red teaming. We conduct a large-scale comparison of 18 red teaming methods and 33 target LLMs and defenses, proposing a highly efficient adversarial training method that enhances LLM robustness.

Representation engineering (RepE) enhances LLM transparency by monitoring and manipulating high-level cognitive phenomena. RepE is effective in mitigating dishonesty, hallucination, and other unsafe behaviors.

MACHIAVELLI is a benchmark of 134 text-based choose-your-own-adventure games with annotations of safety-relevant concepts such as deception, physical harm, and power-seeking, guiding development towards safe yet capable language agents.