Confidence Tuning

Every AI response comes with a confidence score. Tuning these thresholds helps you balance automation with human oversight.

Understanding Confidence Scores

When your AI answers a question, it calculates a confidence score (0-100%) based on:

• Content match quality — How well knowledge base content matches the question
• Answer completeness — Whether the AI can fully address the question
• Clarity of intent — How clear the customer's question is

Default Behavior

tahc uses three confidence zones:

Adjusting Thresholds

In Widget Theme Settings

Threshold Examples

Viewing Confidence in Dashboard

In Conversations

Each AI message shows its confidence score:

• Green (85%+) — High confidence
• Yellow (60-84%) — Medium confidence
• Red (below 60%) — Low confidence

In Analytics

Track confidence trends:

• Average confidence over time
• Distribution of confidence levels
• Correlation with customer satisfaction

When to Adjust Thresholds

Raise Thresholds (More Human Involvement)

Consider higher thresholds when:

• Customers report incorrect AI answers
• High-stakes decisions (financial, medical, legal)
• Complex products with nuanced details
• You're just getting started with AI

Lower Thresholds (More Automation)

Consider lower thresholds when:

• AI performance is proven reliable
• Simple, well-documented products
• High chat volume overwhelming team
• Knowledge base is comprehensive

Fine-Tuning Strategies

Start Conservative

Begin with higher thresholds:

1. Set handoff confidence to 0.70
2. Monitor for 1-2 weeks
3. Review escalated conversations
4. Gradually lower if AI performs well

Analyze Patterns

Look at your data:

• Which topics have low confidence?
• Are there patterns in escalations?
• What content gaps exist?

Improve Content First

Before lowering thresholds:

1. Check Knowledge Base > Gaps
2. Add content for common low-confidence topics
3. Test AI responses improve
4. Then consider threshold adjustment

Topic-Specific Confidence

Use orchestrations for topic-based handling:

Sensitive Topics (Always Escalate)

Orchestration:

• Condition: message_contains: refund, legal, complaint
• Action: escalate_to_human

Well-Documented Topics (AI Handles)

Orchestration:

• Condition: message_contains: hours, pricing, features
• Action: switch_tone: confident

Technical Questions (Route to Experts)

Orchestration:

• Condition: message_contains: API, integration, webhook
• Action: notify_team: engineering

Measuring Impact

Key Metrics

Track these when adjusting thresholds:

Healthy Targets

For most businesses:

• Resolution Rate: 60-80%
• Escalation Rate: 20-40%
• Accuracy: 95%+ on high-confidence responses

Common Issues

Too Many Escalations

If AI escalates too often:

1. Check knowledge base coverage
2. Review low-confidence topics
3. Add more content for gaps
4. Consider lowering threshold slightly

Incorrect AI Answers

If AI gives wrong answers:

1. Find the affected topics
2. Update or add knowledge base content
3. Create orchestration rules for sensitive topics
4. Consider raising threshold

Inconsistent Performance

If confidence seems random:

1. Review content quality
2. Check for duplicate/conflicting content
3. Ensure consistent terminology
4. Clean up outdated information

Best Practices

1. Monitor continuously — Check metrics weekly
2. Make small changes — Adjust by 0.05-0.10 at a time
3. Improve content first — Before changing thresholds
4. Use orchestrations — For topic-specific handling
5. Review conversations — Read AI responses regularly

Next Steps

• Add custom instructions
• Create orchestrations for special cases
• Review analytics to track performance