The Problem#

The billing team gets an e-mail about an overcharge, a declined payment, or a dispute claim. Currently, they have to go across multiple legacy systems to collect data. Between PDFs, Salesforce, legacy on-prem software and Excel, it’s a lot to keep track of and an arcane process living on people’s heads that honestly shouldn’t.

The billing pipeline’s aim is to grab PDFs, process them for data, and deposit them inside a Snowflake data warehouse for further analysis via Cortex AI. The Cortex AI bit is actually another project that was exclusively built inside Snowflake. The billing pipeline is just the AI Workflow piece for PDF processing.

So, the target for this project is the billing team, their process is arcane and slow, and the goal is to speed it up drastically. The billing pipeline is part of the planned improvements.

Workflow#

As with every pipeline, we start with ingestion. The project is a simple ASP.NET Core REST API with several endpoints. The main two are UploadDocument and GetDocumentProcessingStatus. Uploading a document does some initial validations in the endpoint, then tosses the file into a queue backed by Redis. With the PDF, we also require some metadata to be sent in the form request. This gets included in the queue message. We’re leveraging the Foundatio library to handle the queueing.

A background task using Foundatio’s Job abstract class picks up the message and kicks off the processing. This is a 5 step process. We’re using Microsoft Agent Framework and the Workflow class, with just 4 sequential nodes.

1. First, we parse all the pages using Azure Document Intelligence.
2. Next, we classify each page of the document.
3. Once we have classifications for each page, we pick out specific pages for metadata extraction. For both classificaiton and metadata extraction, we use an LLM via Azure AI Foundry.
4. Once this flow is complete, we drop the resulting JSON into another queue.
5. A second worker picks up this message and attempts to store the data in Snowflake.

Decisions and Tradeoffs#

Explain the constraints, alternatives you considered, why you chose your approach, and what you gave up.

One of the first constraints we had to deal with was a legacy PDF storage system. Another constraint was having to deposit into Snowflake. A throughput requirement was to handle between 500-1000 documents per day. I guess some billing periods are more busy than others…

We opted to go with a Python script for picking up the PDFs and depositing them into the pipeline. We didn’t want to build this into the pipeline itself, because of networking restrictions with the hosting the pipeline (in AKS, and inside a VPS). There was too much red tape around opening ports and such, so we opted for a workaround script.

Depositing into Snowflake was easy to handle, but there were some requirements around batching that we needed to work with. We ended up storing up to 20 documents in Redis, then batching them into Snwoflake in one request. Once successfully deposited (and verified with a separate call) we deleted the local copies in Redis.

The throughtput requirement was an interesting case. We originally wanted to do all the processing in Snowflake using their PDF_EXTRACT, AI_CLASSIFY and AI_COMPLETE functions, but these did not scale at all. Each document took upwards 3 minutes to process which, if you do the math, 3000 minutes is 50 hours, so we would get backed up during busy times. Going with Azure Document Intelligence and AI Foundry we got the processing time down to under 10 seconds for even the largest documents. We classify the pages in parallel, 20 at a time so as to not explode our AI Foundry token limits.

Challenges#

As mentioned above, when we initially started on this project, we wanted to just use Snowflake for the entire pipeline. Since performance in Snowflake was unacceptable, we did waste a bit of time there trying to optimize our queries. To be frank, trying to write an entire pipeline in just SQL is not great. Do not recommend. Going back to .NET and building out using MAF was the right call. We already used MAF before in other projects.

The reason we tried Snowflake-only is because the team at Snowflake who was onboarding us told us this was totally possible. Trust but verify. Well, verification failed. It was a surprise honestly, because … I mean, it’s Snowflake. Famous, big tech. But yeah, lesson learned.

Another challenge we came across was tracing. Fortunately MAF comes with extensions to make this easier. Since the process itself is so fast, we opted to skip checkpointing to keep the architecture simple, but in retrospect this is an area for improvement. If the document fails during metadata extraction (for example PII trips the content filters), we end up having to reprocess the entire document including the PDF extraction. This is obviously inefficient.

Lastly, error handling. We currently retry PDF processing 3 times then fail it into DLQ. There’s no mechanism for DLQ audit. We have endpoints that let us view the DLQ and re-submit failed messages, but the error handling process could be more robust for sure.

Results#

The pipeline has been running for a few weeks and documents are getting parsed, deposited and absorbed by Snowflake’s Cortex AI. We had an early access program running for a few select billing team members and they came back with some excellent feedback. Improvements are slated in the Snowflake area, but the billing pipeline does what it’s supposed to without fail. Hard to estimate overall time improvement as the entire experience is still being built out, but early signs are promising!

We are processing around 300-400 documents daily without pause, with very few failures. Usually these are related to content filters, because some of the documents will have critical PII in them. We have a Copilot agent checking the DLQ for failures every few hours, getting the details, and sending e-mails to people. There’s no action plan right now on what to do with these failures.

Overview#

ClaimReady is a web app which helps home owners inventory their belongings specifically for insurance purposes. It spawned out of a need I found I had, so I decided to build it. It’s a solo project, built entirely client-side with NuxtJS. How we got here, though, is a story… read on!

Challenges#

One of the main challenges with this project was not actually technical, but psychological. My initial approach was to have users take photos with their phones, these photos would get uploaded to an S3 bucket, a signal would be sent to the web app over a websocket, and the app would update itself. I had a prototype of this working and everything was peachy, but then realized people may not want to upload photos of their private belongings to some random website on the internet. Even though we were deleting all images after the PDF was successfully generated, that wasn’t good enough.

After this realization, I had to change the app flow. Users would just take pictures of their stuff, then sit down at the app and upload all images as they were adding the items. It was a bit more manual, but it allowed me to go fully client-side.

Another bit of trouble I ran into was the payment process. Since the app was fully client-side, I had to ensure users wouldn’t be able to just skip the payment process and get the generated PDF for free. While no client-only solution is fully unhackable, after applying some hashing and obfuscation, I think I arrived at a pretty good end result.

Choices & Tradeoffs#

Initially, when the app was server side, I actually had a .NET backend hosted on Railway and a volume with S3 bindings enabled to storage images. In that design, Nuxt acted just as the front-end and the .NET app as the worker. I also had setup Resend to e-mail the PDF to the customer.

Then it all changed when the Fire Nation att… I mean, when the app flow changed to client-only, I went all in on Nuxt/Vue. I leveraged IndexedDB to store the JSON structure of the inventory as well as compressed images in base64 format.

PDF generation also happened on the client. The tradeoff here was that JavaScript libraries are not as good as .NET ones for PDF generation, but I wasn’t creating a masterpiece, just something passable with pictures of people’s stuff. I had to dumb down the design a bit, but mitigated that with a shift to “minimalist” design using shades of gray.

For payment, I reached for Stripe initially… but ultimately went with Polar.sh. I was okay with the slightly higher fees as long as taxes were taken care of. Who wants to deal with the tax man anyway…?

Considerations#

The main lesson from this project was to do thorough market research before jumping into implementation. Since the idea was spawned from my own need, I figured I’d have no problem uploading images to a webserver… because it’s my webserver. A mistake.

An other lesson I learned was that shipping an app is a lot easier than getting users to use it and pay for it. The marketing part is something I’m still working on to this day. If you need to inventory your home, go ahead and visit ClaimReady and start inventorying today!

Overview#

I led the architecture and implementation of the AI Evaluation platform for internal AI and robotics teams. The system handled several different inputs and produced single run evaluation results and rollups. As this was a PoC that turned into a full-scale deployment, I ended up not only designing the architecture, but also building the entire project myself (with support from my AI companions…). The result was a working platform which ingested a variety of AI outputs, processed and scored them, and surfaced aggregated results and alerts to the relevant stakeholders.

Once the platform was live, it started ingesting AI outputs immediately. These included OCR parsed outputs, customer support chat logs, form parsing results, and other automation and AI outputs. Results never needed to be instant, so the entire ingestion pipeline was asynchronous – even results which theoretically were instant (such as regression). The volume of data started as a trickle, but grew to a steady state over time as more and more existing automation got onboarded with the API.

This project’s purpose was to “check the audit box”, but also to catch problems in AI’s judgement and prevent disasters, especially in customer-facing applications. I think we all know how things can go awry when LLMs are let loose inside company systems. Once a dashboard was available and rollups started to flow, stakeholders were happy to see even rudimentary tables and charts.

Challenges#

The project started out with literally no requirements other than “we want to see how our automation is performing over time”. The first step was to clear up this rather extreme ambiguity; Collect information from stakeholders, define metrics and evaluation criteria, and settle on an ingestion strategy.

Ingestion#

There were several different types of inputs: filled forms on the web, OCR of paper forms, customer support chat logs, training data AI outputs, summaries of client interactions, customer feedback, etc. The quality of data was also questionable in some scenarios – especially when dealing with OCR! Azure Document Intelligence is pretty good, but not perfect. To top it all off, when ingesting customer data, sometimes the process would fail due to PII/PHI being present.

Processing#

Processing the inputs was also hit and miss. While some inputs like algorithmic regression, completeness and accuracy were easy to process, others like agentic regression, fairness, and bias were definitely a lot more work. Fairness and bias specifically required splitting the pipeline into multiple different isolated stages to prevent the evaluator LLMs from being biased during the evaluation. Using strategies like adding counterfactual examples helped in making the evaluation more robust.

Rollups#

It’s relatively simple to create rollups and pretty charts for regression, completeness, and accuracy. More subjective results required a bit more thought on how to display them to stakeholders. For example a fairness and bias result could result in Pass, NeedsReview or Fail. Did we want to surface the failure rate? Or how many minor issues were found? These questions needed to be answered, and stakeholders didn’t all agree.

Choices & Tradeoffs#

While the evaluation run results definitely had the shape of a JSON document that would fit nicely inside a NoSQL store, one of the constraints for this project was that we had to keep all data inside Azure SQL. This was a hard requirement, no way around it. I chose to create a table to store general evaluation run results, with each result having a type colum and a field housing the serialized JSON of the actual result.

Before you balk at this solution thinking “but how did you get the rollup data???” – you’re not wrong. I ended up creating the rollups in a two ways. First, as an evaluation run finished, I would update the existing rollup data (stored in a different table) with its results. Second, I created a scheduled job to aggregate the rollup data just in case. I knew the rollups had to be “eventually correct” so there wasn’t as much pressure to make them real-time.

The front-end was written in Blazor and used MudBlazor. This is the company’s default choice for front-end development. An excellent choice for this type of project!

Since each ingestion request was asynchronous, we returned a 202 Accepted response and tossed the request into a queue. While the initial PoC used the Channel interface we later transitioned to Azure Service Bus for message queuing. Channels let us get something out the door quickly while a follow-on improvement waited in the wings post-MVP.

Lastly, the agentic framework. I ended up going with Microsoft Agent Framework and using the workflow pattern over manually orchestrating AI Foundry requests, because Microsoft finally shipped a 1.0 version of it right as this project was starting up. I had previously used their Semantic Kernel library, and if you know anything about Microsoft’s SDKs over the past 10 years, it’s that they tend to deprecate features and shuffle APIs around pretty often. With MAF, I felt like they finally settled on something, but… time will tell! MAF is awesome though, and it improved my personal productivity working with AI by reducing boilerplate and abstracting a lot of necessary fluff. Though, of course, that is a tradeoff – abstractions can be risky if you don’t know what is actually being abstracted under the covers! That said, having previously worked with Semantic Kernel and pouring over Microsoft’s code on Github, I determined everything was legit and pressed forward.

Considerations#

If I were rebuilding this today, I would move long-running evaluations into a durable workflow model (via Durable Tasks). I ended up writing custom code to handle logging and checkpointing, despite MAF’s workflow having built-in support for these features. That’s because MAF’s support wasn’t quite flexible enough (though it might be by the time you read this) to meet all our needs.

There are still a few pieces of the system which I’m not satisfied with. Specifically the way the agents are set up – they have hardcoded system prompts. These could be pulled from the database and potentially versioned. The way the deadletter queue is handled is also a bit sus. We do some retrying, but eventually just log a failure to Application Insights and have an alert set there. Improvements can be made…

All things considered, in order to get this out the door and meet deadlines, had to cut some corners somewhere. Given the constraints and the fact this platform is helping the company pass audit, I think we struck a good balance between functionality and polish.

Overview#

Prism is a project spawned out of the need to have comprehensive insight into my personal finances. At first I used my credit union’s web app, but it wasn’t doing quite what I wanted, so I moved to Monarch. I used Monarch for a bit, but again, it didn’t quite have the features I wanted. With the advent of AI, it’s a lot easier to build these kinds of personalized solutions, and thus, Project Prism was born!

Challenges#

One of the first challenges I ran into was that copying Monarch or CU web app was not going work. It’s easy to just take screenshots and tell AI “go build this”, but since neither of those apps did what I wanted, I had to first figure out what it is that I actually wanted. Yes, some product design! Turns out I’m not a budget person, so I didn’t bother implementing that at all. I’m not a product manager, so I had to take a bit of time with /grill-me to figure things out.

Another challenge was ensuring that the app could handle a variety of financial data sources and formats. I had to design a flexible data ingestion pipeline that could accommodate different types of transactions and accounts. I researched about different ways to accomplish this (ie Plaid), but honestly, for a home-spun non-commercial app, that kind of ingestion was not in the cards. I ended up using my CU’s Plaid integration to merge in all the transactions, then export 1 transaction CSV out of there. I built an importer tailored to that CSV format.

Another challenge was classification of transactions. Usually, these don’t come out very well classified in the CSV. For example something from Home Depot can just be “Shopping” and something out of Costco can be “Groceries”. Theoretically that’s not wrong, but I wanted it to be more granular. I knew this would take a bit of work up front, so I created a Rules engine for both categorizing the transaction into a spending bucket as well as the attaching a specific retailer to the transaction.

Choices & Tradeoffs#

I built the entire application in Nuxt using basic-auth for authentication. Data is stored in Postgres which is also hosted in my homelab. The choice of hosting data locally made the most sense. I don’t want my financial data out in the cloud. Even though I’m encrypting it at rest, I’m still not comfortable having it be out there. The tradeoff is obviously that I have to maintain the database, its backups, uptime, etc. Fortuantely, I’m really into homelabbing so this is not a problem at all!

Transactions and spending were just one part of the financial picture. I also wanted to have insight into my assets, both real estate and the market. Again, integrating with these platforms was not in the cards, so I’d have to do the input manually. I did integrate with yahoo-finance2 so I can get latest ticker values, but the amount of stock is something I need to manually update every now and then. Since I rebalance just once a year, this is not a big deal, but it is a tradeoff nonetheless.

Lastly, I wanted to integrate an AI advisor into the application. This was going to be the main differentiating factor between my app and Monarch or the CU’s app. I tested out Mastra first and it was pretty good, but I wanted to see what Google’s ADK was all about. It was lighter than Mastra, so I opted to go with that. Maybe a mistake though, since they already came out with ADK 2.0. That’s the tradeoff when going with a Google product… you do an integration, then it gets deprecated within weeks.

I also chose to go with OpenRouter for my AI provider. While I have experience with Azure AI Foundry, OpenAI and Google Gemini APIs, OpenRouter is model agnostic and super lightweight (AI Foundry is not). It’s also easy to see spending there. I also use OpenRouter for my OpenCode / Copilot. There’s no tradeoff when using the API, and I’m using relatively cheap models so the overall cost is fractions of pennies.

Considerations#

When building this app, I wanted it to be a comprehensive “House Management Control Panel”. I quickly learned that as the codebase grew, things started to get out of control. That is, as AI was writing most of the code, I started to be like “hmm where is the code for this”, which made me feel anxious a bit, given that this app deals with my money! I scaled it down to just focus on personal finance and wealth management and trimmed some features I didn’t need or use.

I also learned that I like the graph-like ‘node/edge’ approach to AI workflows better than Mastra’s more object oriented approach. I don’t really know if that’s the correct way to describe it… but hey, it’s one thing to read the docs, and another to actually use the library. Some things become apparent only when you’re deep in the weeds.

Overview#

Upon joining the team as a Staff Software Engineer, I was responsible for the UI and later the deployment pipeline. The project started out as a ASP.NET Core 8.0 monolith and was slowly broken up into microservices as more features were added. This project, which I’ll call ‘Specimen Visualizer’ for the purposes of this deep dive, was meant to revolutionize the way biologists store and analyze tissue samples.

Microscopes would scan series of TIFF images which would then be ingested by our system over gRPC, processed through a robust pipeline which split them into different layers, and then stored and indexed. Users would then be able to open the image in a ‘viewer’ window and do analysis on it (think QuPath). This project was also slated to be “self-hosted” with a valid per-seat license… whether locally on-prem or on AWS, the choice was up to the user.

Although the project was shelved at ~85-90% completion, we were able to achieve some pretty amazing results running test scenarios on real data. The entire pipeline, web application and visualization client code was in place. The product aimed to combine disparate systems scientists use now, reduce data duplication and improve scan-to-analysis speed. It achieved these goals even as the project got canned. Sad.

Challenges#

There were multiple areas of the project which needed serious thought; ingestion, cutting up the images into smaller bits, storage, organization within the browse UI and the client-side performance in the visualizer itself.

Ingestion and Processing#

Images coming in from the microscopes were enormous – often larger than 100mb per image. Each scan would consist of multiple images, sometimes dozens, depending on the zoom level. Sending the images over from the microscope wasn’t a big deal, but once the image was fully ingested into the system, we had a constraint of around 20 minutes to fully process it and surface it in the UI.

We started processing the images as soon as they arrived. Bits themselves were stored on the disk, then jobs kicked off to slice the image up for display in a map view. As the images came in and were sliced up, the math had to make sense and we had to ensure the alignment of the slices was correct all around. This part, right here, was the most important and intense part of the entire application.

Frontend - Browsing#

The application itself, written in a custom flavor of Blazor (with WPF influences – observables, etc), was written on top of an API hosted in the same project. Conceptually the architecture was pretty simple - front end makes a request, it goes to an API which looks in the database (via EF Core) and returns data, which gets bound to the UI. The implementation itself was a rat’s nest of abstractions which slowed down development substantially. Adding a dropdown required changes to 14 files across 5-6 different projects.

Frontend - Visualizer#

Once you’re in the UI and click on a completely ingested scan, the visualizer pops up. The basis for loading the image was OpenSeadragon, and the annotation engine was Annotorious. These libraries were doing the heavy lifting, but in order to achieve what the stakeholders were aiming for we had to apply some modifications to the source code – especially Annotorious.

As you may imagine loading a giant amount of image data into the browser memory was asking for trouble. With a 2GB limit per browser tab, we had to implement efficient memory management strategies like tile coordinate detection and unloading and caching via service workers. We wanted to achieve a snappy “time to first paint” on the visualizer if the user has loaded the scan before.

Choices & Tradeoffs#

In order to successfully process such a significant amount of image data, we settled on using RabbitMQ and spun up/down transient queues as the demand changed. Since the project was self-hosted, we couldn’t rely on any existing cloud services like SQS, Sevice Bus, etc.

For authentication, we went with Keycloak. One of the requirements was to support SSO and ASP.NET Identity did not support this. We looked at Duende Identity Server, but the licensing structure didn’t work with what we wanted to do. Keycloak was chosen as the auth platform for the project, though nobody had any prior experience to it so there was definitely some ramp-up time.

Data storage was Postgres and Redis. We cached some cellular analysis outputs in Redis to improve performance on the front-end while being respectful of the user’s hard disk storage (we were already caching some images there…). Again, these choices were pretty standard given the ability to run Postgres and Redis on-prem and not requiring a license.

On the front-end we went with Blazor, because majority of the team had .NET experience. It was also augmented with WPF-specific elements like observables to aid the WPF developers on the team in transitioning to Blazor. As mentioned earlier, this was supposed to improve productivity, but the team was still light on HTML, CSS and JavaScript skills, so in my opinion this backfired and slowed down development, potentially resulting in the project getting canned.

Going with OpenSeadragon and Annotorious was the right choice, because writing these libraries from scratch was going to be too heavy of a lift. The tradeoff was of course rigidity which was had to work around. Upon joining the team, most of the code was spread across disparate JS files and injected into Blazor components. I refactored all the client code into TypeScript and a single Vite build. Also moved from writing custom CSS all over the place to Tailwind. This also helped devs on the team to get accustomed to CSS without needing to worry about the syntax and CSS’ various quirks.

Considerations#

When I joined the project, many decisions were already made and code was far enough along so that it wouldn’t make sense to refactor the core systems. That said, if I were rebuilding this today, I would definitely go with a more unified architecture on the frontend, such as Nuxt, Next or SvelteKit. DX with these frameworks is superior to Blazor, especially with the custom flavor in this project. This is doubly true given the amount of JavaScript we were loading in throughout the application – especially in the visualizer.

The backend was very robust and well thought out. I was party to many design sessions where we discussed how to structure data for fastest access (binary trees were used, yes, outside of an interview!). If there was one thing I’d improve here it would be observability. When something blew up, we had to dig deep into logs to see the exceptions. Plan was to bring Elastic in later for log storage, but we never got there.

The deployment was Kubernetes from the start, but it started out on Minikube as a proof of concept. The Helm charts were copy-paste with a lot of extra unnecessary configuration options. I ended up refactoring them all to be more maintainable. The deployment itself was a shell script. This made some sense, because the deployment had to be self-contained since it could be run on airgapped on-prem servers. However, I would have liked to look into some pre-baked deployment strategies instead of custom rolling a giant shell script.

Overview#

The Antenna Control Center (ACC) was an all-in-one antenna control and monitoring system for managing flat panel satellite antennas. Users used the portal to buy and register antennas, purchase an internet subscription, monitor their traffic and location, and manage their hardware.

I was a founding engineer on the team and helped pitch the PoC to management, which then opened the gate to investment and further team growth and product development. From 2016 until 2023, I was the lead engineer on the project, mentored junior engineers, and drove the technical vision for the entire system from arcane 3rd party integrations to the CSS in users’ browsers.

The ACC was a turnkey solution meant to promote the hardware product, reduce friction for the customer when interacting with the hardware, and cut support costs for the company by giving users the ability to self-debug issues. The word “meant” does a lot of work there. This was a sprawling microservices project that took shape over many years. Strap in for a lengthy deep dive!

Challenges#

The PoC pitched the overall concept of being able to add your antenna to an inventory and then see where the antenna is in the world. We had placeholder buttons and copy for features we thought we could implement down the line if given the green light. While the PoC (written in Angular2) won the hearts and minds of the executive team, that was the easy part.

Antenna telemetry ingestion#

There were multiple different types of telemetry. The antenna had its own, but not real time. It just threw a 500mb zip of logs at the servers every hour. Connectivity providers had their own telemetry too – Intelsat, Viasat, etc – and their telemetry came over in real time or with a very short delay. In all cases, the telemetry overlapped in many respects (for example location, connection strength). The challenge was merging different telemetry sources into a single coherent record we could then analyze. For example, one of the queries could be “I see that in this location we always get poor connectivity on antennas with model A, but model B is fine.” These are the kinds of things we wanted to surface to the NOC for analysis.

3rd Party Integrations#

On topic of integrating with 3rd party providers, this was also a challenge. For example one provider wanted us to upload satellite configuration files to an FTP, then wait 5 minutes for it to be applied. Another provider had a SOAP API, that’s the best they could do. Oh yeah speaking of SOAP, Oracle. These integrations were very tedious to implement and brittle just by nature of the technology being used.

Microservices#

The entire system started out as microservices from the get-go. We used Azure App Services to host the executables and Azure Service Bus as well as REST calls to communicate. This design made sense at the time, but late 2010s were also microservices hype. Adding and event hub to the architecture seemed like the way to go, but in retrospect, it was overkill and unnecessary overhead. The entire application could have just been a modular monolith hosted in a single process, potentially across multiple instances.

Front-end#

On the front-end started off on Angular 2, moved to Aurelia and then to VueJS. The front-end was a pure SPA with custom CSS using oAuth2 with Identity Server 4 for authentication. The challenge of working with VueJS (2.6 at the time) was that there wasn’t a very solid way of scaling the app code out. Nowadays you’d reach for a meta framework which bakes in a lot of the ‘required’ functionality like routing, SSR, backend server, storage and real time abstractions, and so on. We had to write a lot of that stuff by hand. Jumping from framework to framework early on cost a bit of development time, and in retrospect we should probably have stayed with Angular 2 and followed the version updates.

Product Direction#

The vision for the product shapeshifted throughout the years multiple times, resulting in wasted developer time. We had a single portal to begin with, then two portals – one for the NOC and one for customers, then back to one portal and a REST API. All the while we had to support multitenancy, RBAC, SignalR hubs, Salesforce integration, and an in-app shopping wizard. I am actually quite proud of the shopping wizard (or ‘configurator’ as it was called). I modeled it on the Tesla shopping experience, and it made it really easy for customers to purchase accessories for their antennas right in the app.

Choices & Tradeoffs#

Let’s address the elephant in the room – why Vue over React? I chose Vue over React because it had a gentler learning curve and the React ecosystem, at the time of decision making, was in a bit of a hot mess. React has come a long way since its early days and would totally be a great choice now, but yeah – time and place. JSX was also not exactly the prettiest to look at and Vue’s separation of concerns inside the components looked a lot cleaner. I also considered going back to Angular, but the community around the second iteration of Angular was far smaller and there was also drama about the Angular versioning process. All things considered, Vue seemed like the right direction at the time. I believe, to this day, that direction was correct, as the Vue is healthier and more robust than ever.

As mentioned earlier, we went with a microservices approach right off the bat. Seemed to make sense initially, but we ended up using the same database and Redis cache for all the services, so… ya, you see how this could have just been a modular monolith and it would have been fine. There was never a single service that needed its own CPU cycles and memory, because we never had anything truly enormous to process. Telemetry logs were a fair bit of data, but it was just text. Additionally, our team was small - just 4 developers - and we had dozens of microservices and often deployed dozens at the same time, and did schema migrations from within Visual Studio since we were using Dapper.

In retrospect, there were some decisions that were made early on which planned for a grand future that never materialized. The thought was thousands of users a couple years in, but the reality was dozens of users after 5 years. This was a good lesson which I took to heart – do not overengineer early.

Later on in the project we opted to switch to JSON document storage instead of SQL since the data mapped better. After some evaluation, we went with RedisJSON on RediStack. This let us keep data super accessible and easy to modify without needing to write complex queries. It did introduce some challenges with denormalization of various properties, but nothing we couldn’t handle. At the same time we switched to JSON, we introduced a serious implementation of eventing, so services which needed to know when some other services updated an entity were notified with the relevant payloads.

We eventually moved off of Azure App Services and to Azure Kubernetes. That reduced cost substantially and improved performance, but introduced additional training requirements for the team who have never used Kubernetes (myself included). I took the challenge head on and ended up organizing some brown bags to share the knowledge around the team.

Considerations#

If I were rebuilding this entire system today, I would go with a meta framework – probably Nuxt. The original design was aligned with the dream of the future, but the reality played a different tune. I would bake a lot of the Redis transactions into Nitro endpoints instead of reaching out to disparate services. With the benefit of hindsight, the entire architecture can be vastly simplified and not only reduce maintenance burden but also increase feature delivery speed. That’s what these meta frameworks are made for.

We were pretty bright eyed about the future so we planned ahead. Nothing wrong with that… if the market is verified. It wasn’t. We were trailblazing.

Building with microservices introduced reliability and observability challenges. Event driven systems are notoriously difficult to debug with stock tools. Given the small team we were running with, maintenance also ended up being a problem. Once stakeholders got wind of how much cash we’re burning on cloud services, we had to abandon expansion plans for various customer and NOC facing features. I learned a lot about what costs the most in the cloud (it’s not compute).

Whenever approaching a greenfield project now, I take the initial planning super seriously. Given all the experience from this project, the (now) obvious pitfalls to avoid (ex: overengineering for an unproven future) and the choices which stood the test of time (ex: Vue) all come into play when drafting the initial design.