How it works
Four models, one game, and a number that is only ever the time the model took.
Why Pong
Pong is the Atari game chat models handle worst: below random in the Atari-GPT benchmark, mostly because deciding takes them longer than the game gives them. A live loop is the one job where speed is the whole point, so it is the fair place to measure a model built to answer in milliseconds. Jev is a typed-decision model: a small state goes in, a typed answer comes out.
The rules
- Same serve, same rules, same question in every lane.
- One decision moves the ball one step. 8 steps cross the court.
- The model's paddle moves at most 12 units per decision, which is enough reach over a crossing to get anywhere the ball can go.
- The left paddle is a script, not a model. It always returns the ball and it takes no time to decide, so the only difference between lanes is the model on the right.
- When you play, a step never finishes in under 260 ms. Jev at a couple of hundred is a reflex test rather than a game. The floor paces the ball, not the clock: the number beside the court is still Jev's real round trip.
What the model sees
The whole state, numbers only, 125 bytes:
{"court":{"w":160,"h":100},"ball":{"x":80,"y":50,"vx":19,"vy":4.4},"paddle":{"y":50,"h":20},"interceptY":67.4,"dir":"toward"}The question, in the words every lane is given:
You control the right paddle in a game of Pong. The state is a JSON object of numbers: the court size, the ball position and velocity, your paddle (paddle.y is its CENTRE, paddle.h its height), interceptY, and dir. y = 0 is the top edge and y grows downward, so "up" DECREASES y and "down" INCREASES y. interceptY is the y value where the ball will cross your paddle plane after any wall bounces; it is null when the ball is moving away from you. Move so that paddle.y covers interceptY. Treat the paddle as already covering it when paddle.y is within 5 units of interceptY. Answer with one move only.
Three answers: up · down · stay.
Jev vs the chat models
Jev answers through the AI SDK experimental_evaluate API. The chat models get the identical state and the identical words through structured output, at temperature 0, with reasoning off. Nobody gets a retry. The number on a lane is the round trip of that one call, timed on the server, and nothing else.
Where Ably comes in
POST /api/game leaves a worker running on Vercel. It joins the channel pong:game:<id> as an ordinary member, announces itself in presence as the agent, runs the physics and publishes a state snapshot after every decision. Your browser is another member of the same channel: present as a player, publishing input messages carrying one move, drawing what the channel says rather than a simulation of its own.
That is why the watch link works. Anyone who opens it attaches to the same channel with rewind, so the current state arrives immediately instead of at the next decision, and the number of people watching is that presence set, counted.
Read the code
It is all public, and it is meant to be read. Eight files, in order.
- 1lib/game/types.ts · the contract: the court, one decision = one tick, and the bytes a model is given
- 2lib/game/engine.ts · the whole game as pure functions, including the paddle that never misses
- 3lib/decide/prompt.ts · the one question every model is asked, and its answers
- 4lib/decide/jev.ts · evaluate, for Jev. lib/decide/llm.ts is structured output, for the rest
- 5lib/worker/game-worker.ts · the agent as a member of the channel: the decision loop and the wire
- 6lib/worker/referee.ts · when a game stops, with no clock of its own
- 7app/api/game/route.ts · one HTTP request, one function, one game
- 8lib/ably/hooks.ts · everything a browser does: subscribe, be present, publish input