Satellite Antenna Control Center

Overview

The Antenna Control Center (ACC) was an all-in-one antenna control and monitoring system for managing flat panel satellite antennas. Users used the portal to buy and register antennas, purchase an internet subscription, monitor their traffic and location, and manage their hardware.

I was a founding engineer on the team and helped pitch the PoC to management, which then opened the gate to investment and further team growth and product development. From 2016 until 2023, I was the lead engineer on the project, mentored junior engineers, and drove the technical vision for the entire system from arcane 3rd party integrations to the CSS in users' browsers.

The ACC was a turnkey solution meant to promote the hardware product, reduce friction for the customer when interacting with the hardware, and cut support costs for the company by giving users the ability to self-debug issues. The word "meant" does a lot of work there. This was a sprawling microservices project that took shape over many years. Strap in for a lengthy deep dive!

Challenges

The PoC pitched the overall concept of being able to add your antenna to an inventory and then see where the antenna is in the world. We had placeholder buttons and copy for features we thought we could implement down the line if given the green light. While the PoC (written in Angular2) won the hearts and minds of the executive team, that was the easy part.

Antenna telemetry ingestion

There were multiple different types of telemetry. The antenna had its own, but not real time. It just threw a 500mb zip of logs at the servers every hour. Connectivity providers had their own telemetry too -- Intelsat, Viasat, etc -- and their telemetry came over in real time or with a very short delay. In all cases, the telemetry overlapped in many respects (for example location, connection strength). The challenge was merging different telemetry sources into a single coherent record we could then analyze. For example, one of the queries could be "I see that in this location we always get poor connectivity on antennas with model A, but model B is fine." These are the kinds of things we wanted to surface to the NOC for analysis.

3rd Party Integrations

On topic of integrating with 3rd party providers, this was also a challenge. For example one provider wanted us to upload satellite configuration files to an FTP, then wait 5 minutes for it to be applied. Another provider had a SOAP API, that's the best they could do. Oh yeah speaking of SOAP, Oracle. These integrations were very tedious to implement and brittle just by nature of the technology being used.

Microservices

The entire system started out as microservices from the get-go. We used Azure App Services to host the executables and Azure Service Bus as well as REST calls to communicate. This design made sense at the time, but late 2010s were also microservices hype. Adding and event hub to the architecture seemed like the way to go, but in retrospect, it was overkill and unnecessary overhead. The entire application could have just been a modular monolith hosted in a single process, potentially across multiple instances.

Front-end

On the front-end started off on Angular 2, moved to Aurelia and then to VueJS. The front-end was a pure SPA with custom CSS using oAuth2 with Identity Server 4 for authentication. The challenge of working with VueJS (2.6 at the time) was that there wasn't a very solid way of scaling the app code out. Nowadays you'd reach for a meta framework which bakes in a lot of the 'required' functionality like routing, SSR, backend server, storage and real time abstractions, and so on. We had to write a lot of that stuff by hand. Jumping from framework to framework early on cost a bit of development time, and in retrospect we should probably have stayed with Angular 2 and followed the version updates.

Product Direction

The vision for the product shapeshifted throughout the years multiple times, resulting in wasted developer time. We had a single portal to begin with, then two portals -- one for the NOC and one for customers, then back to one portal and a REST API. All the while we had to support multitenancy, RBAC, SignalR hubs, Salesforce integration, and an in-app shopping wizard. I am actually quite proud of the shopping wizard (or 'configurator' as it was called). I modeled it on the Tesla shopping experience, and it made it really easy for customers to purchase accessories for their antennas right in the app.

Choices & Tradeoffs

Let's address the elephant in the room -- why Vue over React? I chose Vue over React because it had a gentler learning curve and the React ecosystem, at the time of decision making, was in a bit of a hot mess. React has come a long way since its early days and would totally be a great choice now, but yeah -- time and place. JSX was also not exactly the prettiest to look at and Vue's separation of concerns inside the components looked a lot cleaner. I also considered going back to Angular, but the community around the second iteration of Angular was far smaller and there was also drama about the Angular versioning process. All things considered, Vue seemed like the right direction at the time. I believe, to this day, that direction was correct, as the Vue is healthier and more robust than ever.

As mentioned earlier, we went with a microservices approach right off the bat. Seemed to make sense initially, but we ended up using the same database and Redis cache for all the services, so... ya, you see how this could have just been a modular monolith and it would have been fine. There was never a single service that needed its own CPU cycles and memory, because we never had anything truly enormous to process. Telemetry logs were a fair bit of data, but it was just text. Additionally, our team was small - just 4 developers - and we had dozens of microservices and often deployed dozens at the same time, and did schema migrations from within Visual Studio since we were using Dapper.

In retrospect, there were some decisions that were made early on which planned for a grand future that never materialized. The thought was thousands of users a couple years in, but the reality was dozens of users after 5 years. This was a good lesson which I took to heart -- do not overengineer early.

Later on in the project we opted to switch to JSON document storage instead of SQL since the data mapped better. After some evaluation, we went with RedisJSON on RediStack. This let us keep data super accessible and easy to modify without needing to write complex queries. It did introduce some challenges with denormalization of various properties, but nothing we couldn't handle. At the same time we switched to JSON, we introduced a serious implementation of eventing, so services which needed to know when some other services updated an entity were notified with the relevant payloads.

We eventually moved off of Azure App Services and to Azure Kubernetes. That reduced cost substantially and improved performance, but introduced additional training requirements for the team who have never used Kubernetes (myself included). I took the challenge head on and ended up organizing some brown bags to share the knowledge around the team.

Considerations

If I were rebuilding this entire system today, I would go with a meta framework -- probably Nuxt. The original design was aligned with the dream of the future, but the reality played a different tune. I would bake a lot of the Redis transactions into Nitro endpoints instead of reaching out to disparate services. With the benefit of hindsight, the entire architecture can be vastly simplified and not only reduce maintenance burden but also increase feature delivery speed. That's what these meta frameworks are made for.

We were pretty bright eyed about the future so we planned ahead. Nothing wrong with that... if the market is verified. It wasn't. We were trailblazing.

Building with microservices introduced reliability and observability challenges. Event driven systems are notoriously difficult to debug with stock tools. Given the small team we were running with, maintenance also ended up being a problem. Once stakeholders got wind of how much cash we're burning on cloud services, we had to abandon expansion plans for various customer and NOC facing features. I learned a lot about what costs the most in the cloud (it's not compute).

Whenever approaching a greenfield project now, I take the initial planning super seriously. Given all the experience from this project, the (now) obvious pitfalls to avoid (ex: overengineering for an unproven future) and the choices which stood the test of time (ex: Vue) all come into play when drafting the initial design.