What happens when AI doesn't know the answer
Amazon / AWS
In a time before Claude and ChatGPT, I redesigned how Amazon recruiters access institutional knowledge by layering community-sourced expertise into an AI chatbot to close coverage gaps, restore trust, and help recruiters move forward when answers were uncertain.
Some details have been modified to protect confidentiality.
↓
Did you know? AWS started as internal infrastructure Amazon built for itself, then they realized others would pay for it.
Background
Amazon Talent Acquisition (TA) was struggling with operational efficiency and talent outcomes as fragmented tools weighed teams down, and the AI meant to unify them could only answer half the questions it received.
Thousands of recruiters across teams and time zones relied on a patchwork of email, Slack, internal wikis, and Quip to find and share critical information. This fragmentation slowed decisions, duplicated effort, and made knowledge harder to access, especially during handoffs and onboarding. Amazon's answer was the Talent Intelligence Learning Tool, affectionately known as TILT, an AI chatbot built to answer recruiter questions and surface talent data faster and more accessible. But TILT was only as good as its data. When information was missing, siloed, or undocumented, it failed exactly when recruiters needed it most, creating an opportunity for me to help shape how knowledge was structured and surfaced across the product.
Role
1 of 2 designers on a lean 11-person team
Focus
Interaction design, UX research and testing, stakeholder management
Partners
TPM, engineering, TA leadership
Impact
81 SUS, 4.3 / 5 CSAT, 20% fewer duplicate posts
Behind every hiring decision at Amazon, 5,000+ recruiters navigated a maze of disconnected tools, workflows, and institutional knowledge.
Challenge
When TILT couldn't answer, recruiters had no safe fallback, creating privacy risks, duplicating effort, and leaving valuable knowledge outside approved systems.
Slack became the default backup: fast, familiar, and full of people who could help in real time. But those informal exchanges routinely involved resumes, candidate details, and hiring decisions happening outside approved systems creating privacy risk at scale.
The deeper issue was structural, not behavioral. Recruiters weren't being careless — they were compensating for system gaps. Each Slack thread solved an immediate need but kept valuable knowledge outside TILT, while incomplete AI responses further eroded trust and pushed users away. The challenge wasn't just closing knowledge gaps. It was bringing informal knowledge exchange back into the system, where answers could be verified, reused, and improved.
Early versions of TILT could handle straightforward questions, but when decisions required nuance or policy interpretation, it hit a wall and left recruiters with not a lot of places to go.
Insights
Speed and reach mattered more than formality.
Recruiters went to Slack because it was fast and likely to produce an answer. Any internal solution had to match that immediacy, not just improve accuracy.
"I'll use the channel with the most members just to up my chance of getting my question answered quicker."
— Tech sourcing recruiter
Recruiters needed accessible human judgment, not more indexed content.
When interviewed, participants said they already relied on coworkers or internal channels when documentation failed. Many questions required policy nuance, team judgment, or situational context that indexed content alone couldn't resolve.
75% of participants already relied on coworkers or internal channels when documentation failed.Trust broke when the system sounded confident without being right.
Recruiters tolerated missing answers more than misleading ones. TILT answered only about 50% of questions, but the bigger failure was presenting incomplete answers as sufficient and letting institutional knowledge quietly disappear every time a recruiter left or a thread went cold. TILT needed a visible way to admit limits while still helping users move forward.
Design
The solution wasn't a better AI. It was giving recruiters a trusted path forward when the AI inevitably fell short.
I embedded expert escalation directly into TILT so that when the assistant couldn't answer, recruiters could surface the question to a peer without leaving the workflow. Unresolved questions stayed inside the system, where answers could be verified, reused, and fed back into TILT over time.
Designed to become what recruiters reach for first for on-demand, trusted answers.
Introduced an escalation path that balanced AI efficiency with human expertise for nuanced or contextual questions.
Escalated questions remain visible and trackable while awaiting expert review. Shared feeds increase discoverability and reduce repeated policy questions across teams. Inbox notifications surface expert replies and unanswered threads requiring attention. Verified expert responses reinforce trust and preserve organizational knowledge within TILT.
Accepted answers train the system and improve future AI-assisted responses. Operational analytics surface resolution rates, response efficiency, and support trends. Knowledge-gap insights help leadership identify recurring employee friction points and policy blind spots.
Outcomes
81 SUS
4.3 / 5 CSAT
20% fewer duplicate posts with autocomplete
Improved task success when unanswered questions could escalate to peers
Although broader organizational changes paused full rollout, testing validated strong potential to reduce Slack dependency, retain institutional knowledge, and improve onboarding by keeping answers inside the product.
The strongest version of AI isn't one that knows everything. It's one that knows when to hand off to someone who does. AI perform better when they acknowledge uncertainty, and become more useful when they support human judgment rather than try to replace it.
