Case Study — Noibu
Product owners sorted errors by revenue lost. But the costliest ones often couldn't be fixed. Three dead-end handoffs to a developer, and there was a 62% chance that customer was gone.
Product owners triaging errors in Noibu defaulted to one metric: Annual Revenue Loss. High ARL meant high priority. The problem was that many of those high-ARL errors were difficult or impossible to fix, and assigning them to developers anyway was quietly eroding trust in the product. Research showed that if a developer got burned by an unresolvable error three times, there was a 62% chance the customer churned.
The real gap wasn't prioritization. It was that product owners had no visibility into whether an error was actually fixable. As lead designer, I partnered with data science to introduce a new ML-powered metric called resolvability: a model that predicted whether a given error could be resolved, trained on historical fix data.
The version we shipped wasn't the version I recommended. My research surfaced a broken user workflow underneath the feature, and the case I made to leadership — pause and fix the foundation first — ended up changing how the team weighed speed against user impact. This case study is as much about that call as it is about the screens.
The model was early. It could correctly identify resolvable errors 81% of the time, and non-resolvable errors 75% of the time. That meant two buckets: errors we were confident were fixable, and a second bucket that was genuinely ambiguous.
That asymmetry created a design problem. Left uncommunicated, product owners would drain the first bucket — all the safe assignments — while avoiding the second entirely. The model would never learn from the errors it hadn't seen resolved. We'd be feeding it a skewed diet, and its predictions would degrade over time.
Written language wasn't going to solve this. Labeling something "unknown resolvability" would read as a soft no. We needed iconography that communicated confidence without triggering avoidance.
Before touching the design, I ran two rounds of research. The first was seven product owners and four developers. What I found reframed the project: this wasn't primarily a UI problem. Product owners were flying blind on effort estimation, and developers were losing confidence in Noibu every time they were handed something unresolvable.
What I found reframed the project: this wasn't primarily a UI problem.
That pushed me toward a data-first framing. The design would serve the model, not the other way around. A feedback loop was essential: when a user closed an error as fixed, a prompt asked how difficult the resolution was. When they closed it as ignored, we asked whether the error was resolvable at all. Clean signal back into the model.
That pushed me toward a data-first framing. The design would serve the model, not the other way around.
But a second round of research — eleven product owners this time — revealed a more fundamental problem. Users weren't closing errors properly in the first place. Seven left resolved errors in "in progress" to monitor recurrence. Three forgot to update state at all. One flagged a Jira sync issue that turned out to be real.
But a second round of research, eleven product owners this time, revealed a more fundamental problem. Users weren't closing errors properly in the first place.
The feedback loop depended on state changes that weren't reliably happening. Before escalating, I ran a series of brainstorm sessions with my group to pressure-test the findings and shape a set of options rather than a single opinion. Then I brought a clear recommendation to the PM, our tech lead, the VP of Product, the CPO, and the CTO: pause resolvability and fix the workflow first.
Then I brought a clear recommendation to the PM, our tech lead, the VP of Product, the CPO, and the CTO: pause resolvability and fix the workflow first.
The business decision was to ship anyway. A startup needs to show consistent value to customers, and leadership made the call to launch resolvability without the feedback loop, with a commitment to tackle user workflow the following quarter.
I disagreed, and I said so clearly. But I also understood it. The easier path would have been to ship the feedback loop UI, close the ticket, and move on. The design was done, the model was integrated, the feature was ready. Choosing to surface the workflow problem anyway — knowing it would complicate the roadmap — is the call I'd make again. What I'd do differently is make it earlier, before the resolvability work was scoped. By the time I had the data, the roadmap commitment was already in place, and that kind of call only lands if you make it early enough to matter.
Choosing to surface the workflow problem anyway, knowing it would complicate the roadmap, is the call I'd make again.
I left Noibu not long after resolvability shipped, so I don't have adoption metrics to point to. What I did see before leaving was a shift in how the team made decisions. The default of moving fast and shipping regardless gave way to a more honest question: is this actually helping the user, and if we can't say yes, do we pull back and evaluate? Leadership started acknowledging that the pace they'd been pushing might have been too fast. I can't take sole credit for that, but this project was the case that forced the conversation.
I can't take sole credit for that, but this project was the case that forced the conversation.
The lesson I carry forward: features with ML dependencies have a data quality floor. You can design around a lot of things, but you can't design around a model that isn't learning. Make that dependency legible to the business before a single screen gets designed.