Projects

2024.06.01  /  Internal build, later adapted for client environments

Ticket Automation & Routing System

A ConnectWise ticket classification pipeline using Python and the OpenAI API, with routing and a manual review queue.

Problem

Incoming tickets needed a category, priority, and routing board before the right technician could pick them up.

Stack

Python

ConnectWise API

OpenAI API

Polars

SQLite

Cron

Outcome

About 80% of tickets classified and routed automatically after two months. Other tickets sent for review; technician overrides logged for follow-up.

Background

Incoming tickets needed someone to read the description, assign a category and priority, and choose a board. Straightforward requests went through the same manual triage as ambiguous ones.

I built a pipeline to handle the categories it could classify consistently and send the rest for review.

What I built

A classification and routing pipeline that runs on ticket creation via a ConnectWise webhook. When a ticket arrives:

  1. The payload is received and normalised
  2. The ticket summary and initial description are passed to a classification model
  3. The model returns a category, suggested priority, and routing board
  4. If confidence is above threshold, the ticket is automatically routed and assigned
  5. Low-confidence tickets are flagged for manual review with the classification attached

02

Pipeline

Ticket classification and routing pipeline A ConnectWise webhook sends each new ticket to a Python service, which normalises the subject and body and passes them to the OpenAI API for classification. A confidence gate at 0.75 decides what happens next: tickets at or above the threshold are routed to a board automatically, and the rest go to a review queue where a technician confirms or overrides the suggestion. Both outcomes are written to SQLite, and a weekly Polars job reports accuracy and override rates back against the classifier. 01 Intake ConnectWise webhook 02 Normalise Python · subject + body 03 Classify OpenAI · structured JSON 04 Gate confidence threshold ≥ 0.75 < 0.75 05a Routed board · priority set 05b Review queue tech confirms or overrides 06 SQLite: classifications and overrides weekly Polars report
The gate is the whole design: everything above 0.75 confidence is assigned without a person reading it, and everything below is handed to one, with the suggestion already attached.

Why LLM classification?

I tried rule-based keyword matching first. It worked for obvious cases but fell apart on anything ambiguous. A ticket titled “can’t get in” could be a physical lock, a VPN, or an application. A model could use more of the description, though the result still needed checking.

03

Live demo: classification & routing

Or start from an example

Pick an example or write a ticket, then classify it. The pipeline scores every category, commits when it is confident enough, and hands the rest to a review queue.

Classifying sends your text to a language model through this site's own endpoint. This site doesn't store it.

Classification model

I used OpenAI’s API with a structured output prompt that returns JSON. The prompt includes:

  • The ticket subject and description
  • A list of valid categories with descriptions
  • Examples of each category (few-shot)
  • Instructions to return a confidence score

The model returns something like:

Example response

{
  "category": "network_connectivity",
  "priority": "high",
  "board": "infrastructure",
  "confidence": 0.91,
  "reasoning": "Ticket describes inability to access company resources remotely, consistent with VPN or firewall issue."
}

Classify a ticket above and this becomes the response for it.

Routing logic

ConnectWise’s API handles the actual ticket update. The routing step is a simple API call that sets the board, status, and priority based on the classification result. If confidence is below 0.75, the ticket goes to a review queue with the classification pre-populated. The tech can confirm or override in one click.

Data pipeline

All classifications and outcomes are logged to SQLite. A weekly Polars script computes:

  • Classification accuracy (based on technician overrides)
  • Distribution by category
  • False positive rate by category
  • Trends over time

This feedback loop is used to refine the few-shot examples in the prompt.

Key decisions

Structured output over free text. A defined JSON structure gave the routing code fields it could read directly: category, priority, board, and confidence.

A review queue. The system doesn’t try to be 100% automated. Anything below 0.75 confidence gets flagged. This was important for buy-in because the team needed to trust the system before they’d accept it routing tickets without review.

SQLite for the logs. The pipeline runs on a single machine with low write volume, so SQLite kept the storage setup small.

Prompt iteration over model fine-tuning. The few-shot examples in the prompt are stored in a config file and updated based on classification logs. This let me revise the examples without running a training job.

Outcome

After two months in production:

  • ~80% of incoming tickets are classified and routed automatically
  • Technician override rate stabilised at about 12%
  • Two categories, “licensing issues” and “hardware failure”, were identified as too ambiguous and removed from automatic routing pending better prompt examples

The remaining tickets went through manual review.

The classification log also gave us a record of the types of requests coming in. We used that to review recurring problems and inform resource planning.

All projects