Enterprise AI Evaluation Infrastructure

Overview

I led the architecture and implementation of the AI Evaluation platform for internal AI and robotics teams. The system handled several different inputs and produced single run evaluation results and rollups. As this was a PoC that turned into a full-scale deployment, I ended up not only designing the architecture, but also building the entire project myself (with support from my AI companions...). The result was a working platform which ingested a variety of AI outputs, processed and scored them, and surfaced aggregated results and alerts to the relevant stakeholders.

Once the platform was live, it started ingesting AI outputs immediately. These included OCR parsed outputs, customer support chat logs, form parsing results, and other automation and AI outputs. Results never needed to be instant, so the entire ingestion pipeline was asynchronous -- even results which theoretically were instant (such as regression). The volume of data started as a trickle, but grew to a steady state over time as more and more existing automation got onboarded with the API.

This project's purpose was to "check the audit box", but also to catch problems in AI's judgement and prevent disasters, especially in customer-facing applications. I think we all know how things can go awry when LLMs are let loose inside company systems. Once a dashboard was available and rollups started to flow, stakeholders were happy to see even rudimentary tables and charts.

Challenges

The project started out with literally no requirements other than "we want to see how our automation is performing over time". The first step was to clear up this rather extreme ambiguity; Collect information from stakeholders, define metrics and evaluation criteria, and settle on an ingestion strategy.

Ingestion

There were several different types of inputs: filled forms on the web, OCR of paper forms, customer support chat logs, training data AI outputs, summaries of client interactions, customer feedback, etc. The quality of data was also questionable in some scenarios -- especially when dealing with OCR! Azure Document Intelligence is pretty good, but not perfect. To top it all off, when ingesting customer data, sometimes the process would fail due to PII/PHI being present.

Processing

Processing the inputs was also hit and miss. While some inputs like algorithmic regression, completeness and accuracy were easy to process, others like agentic regression, fairness, and bias were definitely a lot more work. Fairness and bias specifically required splitting the pipeline into multiple different isolated stages to prevent the evaluator LLMs from being biased during the evaluation. Using strategies like adding counterfactual examples helped in making the evaluation more robust.

Rollups

It's relatively simple to create rollups and pretty charts for regression, completeness, and accuracy. More subjective results required a bit more thought on how to display them to stakeholders. For example a fairness and bias result could result in Pass, NeedsReview or Fail. Did we want to surface the failure rate? Or how many minor issues were found? These questions needed to be answered, and stakeholders didn't all agree.

Choices & Tradeoffs

While the evaluation run results definitely had the shape of a JSON document that would fit nicely inside a NoSQL store, one of the constraints for this project was that we had to keep all data inside Azure SQL. This was a hard requirement, no way around it. I chose to create a table to store general evaluation run results, with each result having a type colum and a field housing the serialized JSON of the actual result.

Before you balk at this solution thinking "but how did you get the rollup data!!!??" -- you're not wrong. I ended up creating the rollups in a two ways. First, as an evaluation run finished, I would update the existing rollup data (stored in a different table) with its results. Second, I created a scheduled job to aggregate the rollup data just in case. I knew the rollups had to be "eventually correct" so there wasn't as much pressure to make them real-time.

The front-end was written in Blazor and used MudBlazor. This is the company's default choice for front-end development. An excellent choice for this type of project!

Since each ingestion request was asynchronous, we returned a 202 Accepted response and tossed the request into a queue. While the initial PoC used the Channel interface we later transitioned to Azure Service Bus for message queuing. Channels let us get something out the door quickly while a follow-on improvement waited in the wings post-MVP.

Lastly, the agentic framework. I ended up going with Microsoft Agent Framework and using the workflow pattern over manually orchestrating AI Foundry requests, because Microsoft finally shipped a 1.0 version of it right as this project was starting up. I had previously used their Semantic Kernel library, and if you know anything about Microsoft's SDKs over the past 10 years, it's that they tend to deprecate features and shuffle APIs around pretty often. With MAF, I felt like they finally settled on something, but... time will tell! MAF is awesome though, and it improved my personal productivity working with AI by reducing boilerplate and abstracting a lot of necessary fluff. Though, of course, that is a tradeoff -- abstractions can be risky if you don't know what is actually being abstracted under the covers! That said, having previously worked with Semantic Kernel and pouring over Microsoft's code on Github, I determined everything was legit and pressed forward.

Considerations

If I were rebuilding this today, I would move long-running evaluations into a durable workflow model (via Durable Tasks). I ended up writing custom code to handle logging and checkpointing, despite MAF's workflow having built-in support for these features. That's because MAF's support wasn't quite flexible enough (though it might be by the time you read this) to meet all our needs.

There are still a few pieces of the system which I'm not satisfied with. Specifically the way the agents are set up -- they have hardcoded system prompts. These could be pulled from the database and potentially versioned. The way the deadletter queue is handled is also a bit sus. We do some retrying, but eventually just log a failure to Application Insights and have an alert set there. Improvements can be made...

All things considered, in order to get this out the door and meet deadlines, had to cut some corners somewhere. Given the constraints and the fact this platform is helping the company pass audit, I think we struck a good balance between functionality and polish.