Use AI to find the support conversations worth reviewing

What shipped

Customer support tools are spreading out. An Irish SME might use Intercom, Zendesk, HubSpot, Freshdesk, Help Scout, Gmail, WhatsApp, or a mix of two or three depending on the team. Recent coverage of Intercom alternatives is a reminder that the tool choice is no longer the main issue. The harder question is whether managers can see what is happening across all customer conversations.

For many SMEs, support quality assurance still means a manager reading a handful of tickets when they get time. That can catch obvious training issues, but it misses a lot. Poor tone, incomplete answers, weak escalation, or unresolved problems are often found only when a customer complains.

The practical AI opportunity is not to replace the support manager. It is to score conversations against an agreed checklist, summarise what happened, and flag the conversations that most deserve human attention. This is decision support, reviewed by a person before operational use.

The workflow is simple. Each closed conversation is checked against a 10-point rubric covering tone, completeness, policy compliance, escalation handling and unresolved issues. The AI produces a short summary, a score, and a reason for any flags. The named human review step is Support manager review. The manager confirms whether the flag is fair, records any coaching point, and updates the checklist where the AI has misunderstood the standard.

This is the kind of customer service workflow Artellis can help design in days or a few weeks through AI workflow design, using the tools and data an SME already has where possible.

Who should care

This is for owner-managers, operations leads and support managers in Irish SMEs with a steady volume of customer conversations. It is most useful where one manager is responsible for quality, complaints and coaching, but does not have time to read every ticket or chat.

It suits teams of about 5 to 40 support, sales support or customer operations staff. The conversations might be email threads, live chat transcripts, helpdesk tickets, booking queries or account support messages. The important point is that the business already has written conversations that can be exported, copied by integration, or reviewed in batches.

The pain is familiar. A customer receives a technically correct but cold reply. A refund policy is explained differently by three different agents. A query that should have been escalated stays in the queue. A repeated product issue appears in tickets for two weeks before anyone spots the pattern.

Traditional QA often relies on random sampling, such as five conversations per agent per month. That gives some visibility, but it is blunt. If the sample is too small, serious problems are missed. If the sample is large, the manager spends too much time reading good conversations that need no action.

AI changes the order of work. Instead of asking a manager to search for issues, it brings likely issues to the manager. The manager still decides. The aim is not an automatic pass or fail. The aim is a better queue for review.

It is also useful where the business wants more consistent coaching. Rather than saying someone needs to be warmer or more complete, the manager can point to recurring themes: missed next steps, no acknowledgement of frustration, poor handover to accounts, or failure to check order history before replying.

A worked example

Take a Cork-based e-commerce wholesaler with 28 staff and six people handling customer queries. They receive about 900 customer conversations a month across email, live chat and their helpdesk. The support manager currently reviews 30 conversations a month. Most are fine. The real problems usually surface through complaints, bad reviews, or calls from key accounts.

The first step is to agree a 10-point quality checklist. For example:

1. Did the reply answer the customer’s main question?
2. Was the tone polite and suitable for the situation?
3. Was the customer’s frustration acknowledged where relevant?
4. Were order, delivery or account details checked before advice was given?
5. Was the company policy applied correctly?
6. Was any exception or goodwill gesture handled within authority?
7. Was the next step clear?
8. Was escalation needed, and if so, was it done?
9. Was the issue fully resolved by the end of the conversation?
10. Is there any coaching point for the agent?

A first run is done on 50 recent conversations. The AI summarises each conversation in three or four lines, scores it against the checklist, and flags exceptions. An exception could be low confidence, possible policy mismatch, unresolved issue, negative sentiment, missing escalation, or a customer asking the same question more than once.

The support manager then carries out Support manager review. They read the flagged conversations, compare the AI score with their own judgement, and mark each flag as useful, wrong, or unclear. This matters. If the AI unfairly marks brief answers as rude, the checklist needs tightening. If it misses cases where the customer was passed between teams, the rubric needs a clearer escalation test.

After that, the workflow can be run weekly. The manager might review 40 flagged conversations instead of 40 random ones. They can still keep a small random sample to make sure the AI is not creating blind spots. This gives a balanced approach: exception-led review plus a small control sample.

The value signal is measurable. Track the percentage of conversations reviewed, the number of unresolved issues found before complaints, recurring coaching themes, and how often the AI flags are confirmed by the manager. If 70 percent of flagged items are useful after two or three rounds, the workflow is probably helping. If only 20 percent are useful, the checklist or source data needs work.

This is close to the pattern described in another Artellis piece on moving from AI ideas to practical workflow change in SMEs, where the useful unit is not the model but the changed process. See more AI workflow thinking in the Artellis insights library.

Where I'd start

Start smaller than you think. Do not connect every channel, score every agent, or build a dashboard on day one. Pick one channel, one team and 50 recent conversations.

Write the 10-point checklist in plain English. Avoid vague standards such as be excellent or delight the customer. Use observable tests: answered the question, gave a clear next step, applied policy correctly, escalated where needed.

Choose three flag types for the first pass. I would usually pick unresolved issue, possible policy problem and escalation missed. Tone is useful, but it can be subjective, so it should be reviewed carefully before being used in coaching.

Run the AI review and ask the support manager to judge the output. The key question is not whether the AI is perfect. It is whether it helps the manager find conversations they would otherwise have missed.

Then decide the operating rhythm. For example, every Monday morning the system reviews the previous week’s closed conversations. By lunchtime, the support manager has a shortlist for human review. By Wednesday, coaching points are shared with the team. Once a month, the manager reviews the checklist itself, removes unfair tests and adds any new policy or service issue.

Keep the governance practical. Tell staff what is being reviewed and why. Use the output for coaching before performance management. Do not let an AI score become an automatic judgement on an employee or a customer issue. The reviewed manager decision is the operational record.

For an Irish SME, the likely first benefit is not a dramatic cost saving. It is earlier visibility. You find weak replies before they become complaints, spot policy confusion before it spreads, and give managers a fairer way to spend their limited review time.