Problem
Unit21 helps banks and fintechs monitor fraudulent transactions through an alerting and rule-based system. We launched Alert Scores — ML-based probabilistic scores showing the likelihood an alert is fraudulent. Since end users were non-technical, there was real confusion about how a score came about and which factors drove it. Agents needed to be empowered to use it correctly.
How do you build trust in ML-based scores well enough for people to act on them? What level of explainability is right for a non-technical audience?
Approach
I worked with the ML and product team to define a visualization experience for explainability, starting with 5 design partners across banks and credit unions. Within a few months of launch, that grew to 47 design partners and paying customers.
Solution
Designing for AI requires its own set of design principles. We optimized for user trust through model explainability, using SHAP visualization to show how the model actually works.
For the first release, scores had to read as one more piece of evidence — not the deciding factor, and deliberately not something rules could auto-action on. Trust had to be earned before automation could be expanded.
2. In-context explanation
Customers were trained on alert scores ahead of time, but we still made sure they had explanation available to reference in the moment they were using the feature.
3. Building trust through model confidence
Agents needed to see which factors led to a given score. I used SHAP — a game-theoretic approach to explaining any ML model — to visually show how each feature increased or decreased it, helping agents avoid over-trusting the number itself.
4. Optimizing for understanding
Using partial explanations, we translated each feature into human-readable language with context-appropriate framing, rather than raw model terms.
