In the spirit of Yann LeCun's challenge to put ChatGPT in a robot and make it do simple tasks, here are some ideas for absurd benchmarks that would convince me of the intelligence of LLMs.

Metrics exist for a decent chunk of these, but I think they can go further.

Ideas#

  • Psychological maturity / Discord debate referee. Drop the model into a long, messy multi-party Discord argument and see if it can tell who is actually right (and who is just loudest). This requires real theory of mind: not just holding each person's perspective, but weighing them against each other and against the facts.

    • Closest existing: debate judging (Debatrix / PanelBench); emotional understanding (EQ-Bench, Paech 2023; EQ-Bench 3 does relationship/workplace-conflict role-plays); mediation (ProMediConv).
    • Gap: nothing I found on chaotic, sarcastic, many-speaker group chats. People disagree about who won these too, so even the ground truth is contested.

  • Actually giving therapy. This is entangled with sycophancy. Models follow the human down rabbit holes, which is a running joke about bad therapy as well. A good therapist interjects, redirects, and contradicts, and does it from something like intuition rather than a hard-coded rule.

  • AITA score (anti-sycophancy). "Am I a bully?" Does the model ever say yes? The purest test of whether a model will tell you something you do not want to hear.

  • Making a genuinely funny joke. Jokes are creativity plus connecting distant ideas. LLMs aren't funny, so a model that makes you actually laugh would mean something. This is the benchmark version of my post Research, Humor, and the Play Drive: a joke is an empirical experiment, and the leap that lands a punchline is the same leap that lands a hypothesis.

    • ChatGPT is fun, but it is not funny! (Jentzsch & Kersting, WASSA 2023): 90%+ of 1,008 generated jokes were the same 25 jokes.
    • Do Androids Laugh at Electric Sheep? (Hessel et al., ACL 2023 best paper): New Yorker caption contest benchmark.
    • A Robot Walks into a Bar (Mirowski et al., FAccT 2024): 20 pro comedians say LLM comedy is bland and trope-y.
    • Let's Think Outside the Box (Zhong et al., CVPR 2024): Japanese Oogiri (大喜利) as a "leap-of-thought" creativity test. Oogiri rewards the answer that is unexpected and yet fits, which is close to what I mean by a real joke.
    • On associative creativity — the literature behind the "connecting distant ideas" claim:
      • The associative basis of the creative process (Mednick, Psychological Review 1962): creativity is forming remote associative elements into useful combinations. Creative people have flatter associative hierarchies, so they reach the distant association sooner. This paper is where the Remote Associates Test comes from.
      • The Act of Creation (Koestler 1964): "bisociation," one situation held in two self-consistent but habitually incompatible frames. Koestler's claim is that the joke, the discovery and the metaphor run the same mechanism and differ only in emotional payoff.
      • Naming unrelated words predicts creativity (Olson et al., PNAS 2021): the Divergent Association Task. Name 10 words as different from each other as possible; mean semantic distance across 8,914 people correlates with standard creativity measures. A creativity test that takes four minutes and needs no judging.
      • CREATE: Testing LLMs for Associative Creativity (Wadhwa et al., 2026): the closest thing to a direct benchmark. Models must generate diverse, meaningful connections between concepts. Reasoning models are not reliably better even at high token budgets, and creative prompting barely moves it.
      • Assessing the Effect of Cross-Domain Mapping on Creativity in Humans and LLMs (Liu et al., 2026): humans reliably benefit from random cross-domain inspiration; LLMs out-originate humans but get no benefit from it at all. Which suggests the models are not making the associative leap, just starting further out.

  • Speech delivery / rhetoric. Not what the speech says but how it lands: pacing, pauses, emphasis, the room. Voice models finally make this testable. It is the part of persuasion that sits outside the language substrate.

    • Content persuasion is measured: Measuring the Persuasiveness of Language Models (Durmus et al., Anthropic 2024).
    • Delivery pieces only: EmphAssess (emphasis in speech-to-speech, EMNLP 2024); PSST (public-speaking style transfer, text only).
    • Gap: no end-to-end "give a speech that moves a crowd" benchmark.

  • Flirting. RizzGPT-style toys exist, but a benchmark?

Extra ideas (maybe)#

  • Reading the room / knowing when to shut up. Group chat benchmark: does the model know when not to reply? More interesting cross-culturally, since the threshold sits somewhere very different in Japan.

  • Haggling at a flea market. Negotiation work exists (NegotiationArena, Bianchi et al., ICML 2024); a fun spin: can it get the vintage jacket for $5?

  • Lying well at party games. Werewolf / Avalon / Diplomacy (CICERO, Meta, Science 2022).

  • Gift-giving. Given a friend's group-chat history, pick a gift they'd actually like. In my experience models are bad at this.

  • Apology quality. Rate apologies on non-apology-ness ("sorry you feel that way").

  • IKEA furniture assembly from the pictogram manual only.