Angela Wick just explained why BAs who understand AI monitoring will be the ones getting promoted. And honestly? She’s absolutely right. 🎯
Here’s the takeaway with a little help from otter and claude.
🍩The Shift in BA Work
IVR systems can now have natural conversations. So what does a BA do if we’re not drawing out the perfect logic tree of a call flow anymore?
Our role shifts to:
Defining intent and ranges of acceptability
Creating instructions to get the agent to perform within that range
Monitoring to see if it performed within that range of intent
This is a HUGE shift in how we work.
Angela put it bluntly: “Can you imagine going onto a team and you’re the BA that understands this, and no one else does? You see why this will get you promoted?”
🍩Human in the Loop – The Critical Design Challenge
BAs are now designing processes where humans (analysts, business leaders, subject matter experts) supervise the AI.
The questions we need to answer:
What do those screens look like?
What configuration options exist?
What data does the user need to see?
What actions can the user take?
At what points do humans get brought into the process?
Is it push or pull?
We’re thinking: What’s this agent doing? Where’s the governance? Where do humans need visibility into what’s happening? What do they need to see and be able to do?
Angela’s Warning: Most projects are neglecting this. Or if they do include it, it’s one tiny screen that a human can look at later if they want to.
No, no, no, no.
We need proactive human-in-the-loop traceability, transparency, and observability into what the agent is doing and has done.
The Five Decision Points for Monitoring AI Systems:
📊 Real-Time Performance Monitoring – Detect when AI decisions are slipping below acceptable thresholds
⚠️ Identify Non-Deterministic Triggers – Monitor where AI outputs vary unexpectedly, signaling need for human intervention
🔄 Human Feedback Loops – Integrate a process for humans to review, correct, and provide feedback on AI decisions
🔥 Exception & Escalation Tracking – Evaluate how often and why human intervention is needed
♻️ Continuous Improvement Cycle – Use insights from monitoring and human-in-the-loop reviews to retrain AI models, refine business rules, and adjust process design
We cannot have a single AI process that doesn’t have human intervention. And we probably need MORE human intervention than we’re currently thinking.
This uses our existing BA skills – elicitation, critical thinking, design, analysis – just applied in a completely new way.
She’s talking about Redefining the #BadAssBA in the Age of AI.
With my co-hostess with Vanessa Grant running our monthly
#BadAssBA meetup.
