Specimen Visualizer

Overview

Upon joining the team as a Staff Software Engineer, I was responsible for the UI and later the deployment pipeline. The project started out as a ASP.NET Core 8.0 monolith and was slowly broken up into microservices as more features were added. This project, which I'll call 'Specimen Visualizer' for the purposes of this deep dive, was meant to revolutionize the way biologists store and analyze tissue samples.

Microscopes would scan series of TIFF images which would then be ingested by our system over gRPC, processed through a robust pipeline which split them into different layers, and then stored and indexed. Users would then be able to open the image in a 'viewer' window and do analysis on it (think QuPath). This project was also slated to be "self-hosted" with a valid per-seat license... whether locally on-prem or on AWS, the choice was up to the user.

Although the project was shelved at ~85-90% completion, we were able to achieve some pretty amazing results running test scenarios on real data. The entire pipeline, web application and visualization client code was in place. The product aimed to combine disparate systems scientists use now, reduce data duplication and improve scan-to-analysis speed. It achieved these goals even as the project got canned. Sad.

Challenges

There were multiple areas of the project which needed serious thought; ingestion, cutting up the images into smaller bits, storage, organization within the browse UI and the client-side performance in the visualizer itself.

Ingestion and Processing

Images coming in from the microscopes were enormous -- often larger than 100mb per image. Each scan would consist of multiple images, sometimes dozens, depending on the zoom level. Sending the images over from the microscope wasn't a big deal, but once the image was fully ingested into the system, we had a constraint of around 20 minutes to fully process it and surface it in the UI.

We started processing the images as soon as they arrived. Bits themselves were stored on the disk, then jobs kicked off to slice the image up for display in a map view. As the images came in and were sliced up, the math had to make sense and we had to ensure the alignment of the slices was correct all around. This part, right here, was the most important and intense part of the entire application.

Frontend - Browsing

The application itself, written in a custom flavor of Blazor (with WPF influences -- observables, etc), was written on top of an API hosted in the same project. Conceptually the architecture was pretty simple - front end makes a request, it goes to an API which looks in the database (via EF Core) and returns data, which gets bound to the UI. The implementation itself was a rat's nest of abstractions which slowed down development substantially. Adding a dropdown required changes to 14 files across 5-6 different projects.

Frontend - Visualizer

Once you're in the UI and click on a completely ingested scan, the visualizer pops up. The basis for loading the image was OpenSeadragon, and the annotation engine was Annotorious. These libraries were doing the heavy lifting, but in order to achieve what the stakeholders were aiming for we had to apply some modifications to the source code -- especially Annotorious.

As you may imagine loading a giant amount of image data into the browser memory was asking for trouble. With a 2GB limit per browser tab, we had to implement efficient memory management strategies like tile coordinate detection and unloading and caching via service workers. We wanted to achieve a snappy "time to first paint" on the visualizer if the user has loaded the scan before.

Choices & Tradeoffs

In order to successfully process such a significant amount of image data, we settled on using RabbitMQ and spun up/down transient queues as the demand changed. Since the project was self-hosted, we couldn't rely on any existing cloud services like SQS, Sevice Bus, etc.

For authentication, we went with Keycloak. One of the requirements was to support SSO and ASP.NET Identity did not support this. We looked at Duende Identity Server, but the licensing structure didn't work with what we wanted to do. Keycloak was chosen as the auth platform for the project, though nobody had any prior experience to it so there was definitely some ramp-up time.

Data storage was Postgres and Redis. We cached some cellular analysis outputs in Redis to improve performance on the front-end while being respectful of the user's hard disk storage (we were already caching some images there...). Again, these choices were pretty standard given the ability to run Postgres and Redis on-prem and not requiring a license.

On the front-end we went with Blazor, because majority of the team had .NET experience. It was also augmented with WPF-specific elements like observables to aid the WPF developers on the team in transitioning to Blazor. As mentioned earlier, this was supposed to improve productivity, but the team was still light on HTML, CSS and JavaScript skills, so in my opinion this backfired and slowed down development, potentially resulting in the project getting canned.

Going with OpenSeadragon and Annotorious was the right choice, because writing these libraries from scratch was going to be too heavy of a lift. The tradeoff was of course rigidity which was had to work around. Upon joining the team, most of the code was spread across disparate JS files and injected into Blazor components. I refactored all the client code into TypeScript and a single Vite build. Also moved from writing custom CSS all over the place to Tailwind. This also helped devs on the team to get accustomed to CSS without needing to worry about the syntax and CSS' various quirks.

Considerations

When I joined the project, many decisions were already made and code was far enough along so that it wouldn't make sense to refactor the core systems. That said, if I were rebuilding this today, I would definitely go with a more unified architecture on the frontend, such as Nuxt, Next or SvelteKit. DX with these frameworks is superior to Blazor, especially with the custom flavor in this project. This is doubly true given the amount of JavaScript we were loading in throughout the application -- especially in the visualizer.

The backend was very robust and well thought out. I was party to many design sessions where we discussed how to structure data for fastest access (binary trees were used, yes, outside of an interview!). If there was one thing I'd improve here it would be observability. When something blew up, we had to dig deep into logs to see the exceptions. Plan was to bring Elastic in later for log storage, but we never got there.

The deployment was Kubernetes from the start, but it started out on Minikube as a proof of concept. The Helm charts were copy-paste with a lot of extra unnecessary configuration options. I ended up refactoring them all to be more maintainable. The deployment itself was a shell script. This made some sense, because the deployment had to be self-contained since it could be run on airgapped on-prem servers. However, I would have liked to look into some pre-baked deployment strategies instead of custom rolling a giant shell script.