Generative AI aid raised novice support agents' output most

A staggered rollout inside a real company

The NBER working paper “Generative AI at Work”, circulated in April 2023 by Erik Brynjolfsson, Danielle Li and Lindsey Raymond, studies something different from a lab experiment: the actual, staggered introduction of a GPT-based conversational assistant to customer support agents at a Fortune 500 business-software firm. The full paper covers 5,179 agents handling live customer chats. The tool monitored conversations in real time and suggested responses; agents stayed responsible for what they sent and could ignore it. Because the rollout happened in waves rather than all at once, the authors could compare agents with and without access while controlling for time and tenure.

Who gained, and the mechanism proposed

Access to the assistant raised the number of chats an agent resolved per hour by 14% on average. The gain was not evenly spread: novice and lower-skilled agents improved by 34%, while the most experienced, highest-skilled agents saw almost no change, and some produced marginally lower-rated conversations. The authors' proposed mechanism is that the model had learned patterns from the best-performing agents' past chats and disseminated them to newer staff, letting an agent with two months of tenure perform like an untreated agent with six months. They also report higher customer sentiment, fewer requests to escalate to a supervisor, and lower attrition among newer hires, alongside evidence of durable skill gains that persisted during software outages when suggestions were unavailable.

What the study does not settle

The paper is explicit that it was not designed to measure aggregate employment or wage effects, and the authors say plainly that their data cannot show whether firms respond to novice gains by hiring more junior staff, deskilling roles, or building tools to replace them entirely. A separate announcement from the Stanford Digital Economy Lab, one of the study's funders, describes an earlier draft with a slightly different sample count and a 15% headline figure rather than the 14%/34% split in the paper's revised November 2023 version, a reminder that a widely cited statistic can shift between working-paper drafts. The setting is also a single company's chat-support software; the specific 14% and 34% figures describe that company's contact-centre workflow, not customer support in general or knowledge work more broadly.

Questions to ask before you adopt it

• Is the tool trained on your own top performers' patterns, or on generic data unrelated to your workflow?
• Are the expected gains concentrated in newer staff, and does your rollout plan account for uneven benefit by experience?
• What does the vendor's or paper's stated figure actually measure, and has it changed between drafts or press coverage?

Evidence from one company's contact centre is unusually rich because it is measured, not surveyed, and staggered rollout gives it real comparison groups. It is still one company, one tool, and one job. That is worth remembering whenever the 14% or 34% figures are quoted as a general AI productivity rate rather than what they are: the measured effect of one assistant on one workflow.