An issue-tracking agent that lives on every teammate’s page.
I built it from backbone to front, as a designer.
Dify, an open-source, n8n-ish agent platform, let me build the whole environment myself at no extra cost. It saved the time I’d have spent figuring that out, so my effort went to the design.
This case study covers every inch I touched, and it’s written in English because I want to focus on the content, not honorifics or any Korean grammar.
Redmine was a daily friction for me. I want Linear-level comfort every day, and Redmine isn’t that. (I even tried to get my teammates onto Linear instead of Redmine. That didn’t work.)
Thinking about the team, an agent would also make my fellow engineers’ daily tech-support work easier.
The agent reads more than five years of our issue-handling history, so it can tell an engineer whether anything similar has come up before. When they file a new issue, it helps them report it more specifically. (Just like grill-me.)
The user-facing chat interface might seem like the important part, but it’s not. The most important part is the backbone behind it.
I had to build the MCP server to be the agent’s hand for touching Redmine. Luckily there was an API available, so it wasn’t too hard.
Gemma 4 is small.
A 31-billion-parameter open model, not one of the frontier models.
Ask it something broad, like “what kind of issues do we have,” and almost every time it got stuck in a loop.
When someone asked broadly, the agent reached for list_issues, and it returned everything on every issue: description, dates, custom fields down to the OS version, with the system prompt stacked on top. The same fat payload get_issue gives for one issue, now multiplied across the whole list. For a model that little, that was never a reasonable amount of tokens to handle.
So I built a second engine. When the intent is broad, about issues in general and not one specific issue, it triggers list_issues_summary instead. That one runs server-side, dropping the heavy fields (attachments, dates, custom fields) before they ever leave the backend, so it controls how many tokens reach the model on every call. Give it less to hold, and it stops losing its place.
Eval mattered more than I expected. I came in knowing nothing about agents, so at first I’d just send a prompt, read the reply, and rate it by gut. That’s worthless: my own bias is in every score, and I’ve already decided what I want to see.
So Claude and I built a sandbox and scored every version against one harness: a single prompt, twelve missions, each demanding a big, structured response, with Sonnet 4.6’s 39/40 as the ceiling to chase.
V2 leaned on completeness rules and hit 30/40, but it misfiled an issue. V3 forced a candidate and squeezed the output budget to save tokens; that backfired into two false positives. The version live now runs on the summary tools, and it’s the cleanest yet: 32/40, zero false positives, nothing out of place. Seven points off Sonnet 4.6, from a model we run on our own A100s.
But twelve missions in one prompt is a stress test, not a workday. Scored the way it actually gets used, one case at a time, the live version lands 9.2/10 against the frontier model’s 9.6/10. That’s about 96% of a model many times its size.
After all the backstage work, I started on the interface, the face users actually meet. I could have left it as the default Dify look, but the problems ran deeper than aesthetics. Default Dify was a bare, generic chat box and nothing more.
For a simpler assistant, the default would have been fine. This isn’t that.
My own daily use of ChatGPT, Claude, and Gemini had a huge influence on this interface. It let me settle on a few principles that became the spine of the design.
An agent that just sits there thinking to itself feels unnatural. People want to know what’s going on, and they want some control over it.
So I used the agent’s SSE stream as the source and built a thinking stream that shows up the moment you send a prompt. Now you can see it’s working, not just sitting there.
I used to think great automation was the kind that never asks for your attention. The research says the opposite. One well-known study found people will lean on an imperfect algorithm, and feel more satisfied with it, as long as they keep some control over the final call, even a little. So the question was never how to remove the human, it was when to pull them in.
There’s a harder reason too. The agent acts on Redmine under your own API key, so if it hallucinated and wiped every issue in five seconds, the log would say you did it. You can’t blame the machine. That makes keeping a human in the loop not a nicety, but the only honest shape for it.
Our engineers have a daily routine.
I want to help with it, because that’s the most certain way to save their time.
So I changed the widget to open on a dashboard. The first screen shows them what’s going on and what kind of action to take, before anyone types a thing.
Four engineers lean on it every day. Each one runs around 49 questions and 2.76 million tokens through it, so between them they push past 11 million tokens a day, all on a 31-billion-parameter model living on eight A100s the company already runs.
We were already paying that GPU bill, so the agent never showed up as a cost. It showed up as help, the kind that quietly gets the work done.
And me? I don’t open Redmine anymore. I ask.
The work isn’t finished. The direction I’m pushing now is collective intelligence: instead of leaning on one model to carry a whole task, fire a handful of agents and give each one a smaller chunk.
It’s the summary-tool lesson again. A small model does its best work on a small, well-shaped piece, so the move is to break a task down, run the agents in parallel, and let them reconcile, the way a team splits a problem and comes back together.
The other direction is letting it speak first. Right now it waits to be opened; the next step is for it to reach out on its own, a quiet “a new issue just got assigned to you” the moment it happens, instead of waiting to be asked. It wouldn’t fling the whole panel open, though. A gentle bubble floating up, like a toast, is enough.
This was the first project at my company to point our internal AI infra at a real workflow instead of a demo, and it started from a designer, not the platform team.
I didn’t do it alone. An AI engineer handed me the pieces I couldn’t reach myself, the server access keys and the like, and I picked up how agents actually work along the way.
Mostly, it left me with a feeling I hadn’t really had before: ownership. I built this from the backbone to the front, so the whole thing is mine, end to end. Turns out that changes how much you care about every part of it.