Introduction
The company I worked at recently shut down. I worked there for over four and a half years, and I was involved in many of the decisions that influenced the engineering organization’s technology choices. Some of those decisions paid great dividends down the line, and some did not. I found myself thinking: If I were to join a new organization, would I make the same bets, or what would I change?
Disclaimer
This is purely my subjective opinion. Every organization’s context is different. Understanding that context and the long term vision is crucial to making sound decisions.
Decision Log
| Choice | Decision | Notes |
|---|---|---|
| AWS | Good | To name a few benefits: AWS allowed us to scale dynamically to the business growth and demands. It was a cornerstone of our DevOps culture and kept our infrastructure teams lean. AWS was responsible for 2-3 incidents that crippled the business which is impressive for the duration we used them. I would stay out of us-east-1, however. |
| Fargate | Good | We run most of the production systems on Fargate. Fargate enabled us to delegate most of maintenance work to AWS (ec2 batching, upgrades,..etc) and have the SREs focus on other value-enabling streams. |
| Kubernetes | Bad | There are legitimate use-cases for k8s. However I believe that most of the organizations that think they need k8s, don’t need it. Supporting k8s platform would have required massive bandwidth from the infrastructure teams for the ongoing maintenance and KTLO. Public Cloud providers have solutions that work for most organizations unless there is a business need that demands k8s. |
| Postgres | Good | We run one of the biggest transactional DBs I’ve ever worked with in my career (~10TB-14TB). It was a solid choice, and I’d definitely use it again |
| Aurora | Bad | Expensive!! We were able to scale well with vanilla Postgres high throughput (read and write) systems. |
| Redis | Good | Great In-memory database and cache. Never use it as your main db however (well…never say never) |
| Elasticcache | Bad | Expensive! Most of the usecases we had didn’t really need ElasticSearch. If I ever needed a full-text-search or realtime metrics analytics then ES is a great tool for that. |
| SNS, SQS, Eventbus | Good | They are reliable and reasonably easy to setup. |
| Kinesis | Bad | Too expensive if all you need is a regular message bus, try SNS, SQS, Eventbus first. There are legitimate usecases for Kinesis and Firehose and those are strong products in their respective areas |
| Dynamodb | Bad | It is ok for a simple key-value storage. However, for the love of god, don’t use it to hammer every nail. It struggles to adapt to changing business requirements and can get expensive real quick if you aren’t careful. |
| Cloudfront | Good | While it doesn’t have as many features as Akami and Cloudflare,it does integrate well with AWS and can do some powerful routing when combined with Lambdas. |
| AWS WAF | Good | After launching WAF, we started blocking one third of production traffic outside ofbusiness hours. Overall easy to use with AWS Firewall Manager to ensure WAF protection is enabled for every endpoint in your environment. The AWS-managed rules worked well.. |
| ACM | Good | Setting up SSL certificates has never been easier. |
| VPC Lattice | Good | For connecting applications across multiple VPCs, I highly recommmend VPC Lattice for most usecases. |
| VPC Private Endpoint | Good | Perfect for connecting to datastores across multiple VPCs |
| VPC Peering | Bad | It’s been offset by a few product launches over the past few years (Transit Gateway, Lattice, Private Endpoints) |
| VPC Transit Gateway | Bad | Try Lattice and Private Endpoints first. |
| Terraform | Good | This was one of the best bets we made. We started using Terraform to manage AWS infrastructure, but eventually expanded to Github, Launchdarkly, Datadog and configuration management in many providers. |
| Cloudformation | Bad | Unreliable, unreadable and limited to AWS |
| Serverless Lambda Apps | Bad | Lambda is excellent for building custom integrations between AWS services and small one-off scripts. However, as with Dynamodb, things get ugly when you hammer every nail with a lambda function. Building complex business application workflows on Lambda will give you a handicap. It gets expensive, complicated, and challenging to maintain really quickly. |
| CDK | Bad | Infrastucture and configuration management should be easy and simple. CDK stacks can sometime get complicated and allow engineers to shoot themselves in the foot by building infrastructure configuration that are as complicated as the business logic itself |
| Terraform Cloud (TFC) | Bad | It’s a glorified and expensive Terraform CICD. Overtime, it became too expensive. Their new pricing model that charges you by the number of resources is absolutely insane, and they know it. It also leads to having two cicd platforms (infra cicd and services cicd) which isn’t ideal and too much cognitive load. You could replace it with Terraform Cli + S3 backend + Your regular CICD platform |
| Terraform Sentinel | Bad | I’m all for compliance as policy. However, Sentinel is coupled with TFC. Nowadays, You can replace it with OpenPolicyAgent |
| Python | Good | Given Python’s popularity, hiring Python engineers is easy. Many of the vendors on this list treat it as a first-class citizen with support at first launch. |
| Ruby | Undecided | Many of the engineers we hired had to learn Ruby on Day 1. Given how opinionated the Rails ecosystem is, this meant that engineers would often run into some friction for the first few weeks. In addition, many of the vendors we worked with didn’t support Ruby on the same level as Python or Java. That being said, in the hands of capable engineers familiar with how Ruby (and especially Rails) operates, it is a performant solution that can accelerate development velocity/. |
| Github | Good | While GH is great, Gitlab is a really good alternative. It offers full devops package inlcuding cicd, sast, test coverage,…etc and would be a better choice. However, Github Copilot is uncontested |
| Buildkite | Bad | At the time (2020), it was the best solution in the market for hybrid CICD platforms. You host the compute and Buildkite hosts the control plane. BK is a great vendor and I greatly respect what they built. As of 2024, GH Action added support for self-hosted agents and their library of GH Actions is uncontested. |
| Github Actions | Good | Great community support and massive library of ready-to-use actions. They have added self-hosted agents in recent years. Their great integration with Github platform is a huge plus |
| Artifactory | Good | There are good alternatives depending on which languages you need to support (GIthub Packages, AWS CodeArtifact). Unfortunately, neither supports both Ruby and Python. |
| Opsgenie | Bad | I can’t remember the last time Opsgenie released anything new that I was excited about. I’d go for PagerDuty instead or DD Incident Management tool. I wouldn’t be surprised if Datadog expanded their catalogue to provide both incident managenet (currently supported) and oncall (not supported yet) functionality |
| Datadog | Good | It is expensive but it is the best in the market. It is no longer just a monitoring tool but rather a platform that ingest most of the engineering operations telemetries with best in the market integrations. Datadog was one of the most innovative vendors we worked with |
| Honeybadger | Bad | Good errors tracker, but in the spirit of consolidating vendors, DD can replace it |
| Splunk | Bad | I have nothing nice to say about them: bad support, bad product, bad company and rotten sales department. |
| Trivy | Good | Great tool for enforcing terraform and infrastructure best practices |
| Snyk | Bad | it does the job, but it lacks some core functionality like native github integration (work inprogress). DD can replace it and there are free open source tools that would work well for small companies |
| LogRocket | Bad | Not a bad product; but DD has replaced it |
| Swagger Hub | Bad | Fewer features than Postman |
| Postman | Good | It can be expensive but it is a good product overall. |
| OpenVPN | Undecided | It gets the job done and is reliable. The free version however is “not enterprisey” enough. Another alternative that i never explored myself but was recommended to me is https://tailscale.com/pricing |
| LaunchDarkly | Good | We introduced it too late but I had positive experience overall. Great Terraform support |
| Sophos EDR | Bad | Having seen what Defender offers, it is difficult to consider Sophos ever again |
| Slack | Good | I love the integrations and level of customization Slack provides. |
| Jira | Good | It is bulky, it is slow but it is customizable. We used it to track incidents postmortem analysis, services, teams and all sorts of metadata. |
| Confluence | Good | I love what Confluence provides. However, knowledge management at scale is challenging for every product and organization. As years go by, the knowledge keeps growing, and it takes hard and deliberate effort to maintain it (remove irrelevant pages, organize content, simplify knowledge discovery, etc.). I’m not aware of a product in the market that has solved these problems yet but I am hopeful that someone will finally crack them. |
| Zoom | Good | Overall, it is stable and reliable, and Zoom keeps innovating. The meeting summary feature has so much potential. |
| EntraID | Good | I would not want to manage on-prem Active Directory. It is an enterprise-grade identity provider. It can be complicated to manage, but overall, the positives outweigh the negatives. |
| Microsoft Defender | Good | One of the best products out there. Defender can ingest signals from many systems with minimal configuration and provide full end-to-end security monitoring. It is expensive, but I think it is worth every penny. |
| Exchange Online | Good | Reasonably reliable (With the exception of when your email gets quarantined and it is not very clear why it got flagged). When combined with the Microsoft entire eco-system you get an enterprise grade package (DLPs, Compliance, Security,..etc). |
| Auth0 | Good | Identity management is challenging, and Auth0 provides an initial boost to help you get started. However, as the company’s requirements evolved, we had to build additional internal abstractions on top of Auth0. |
| Meraki | Good | Reliable hardware, cloudbased and easy to setup and maintain. |
| Arbua | Bad | Given a redo chance, sticking with one vendor such as Meraki would be prime |
| Rapid7 | Bad | Their product is more geared towards on-prem enterprises. We had few bad experiences with their CSOC. I’d like to give CrowdStrike a shot. |
| New Relic | Bad | My experience is based on an older pricing model they changed four years ago. That being said, Datadog’s platform offers more features beyond observability platform (Khajiit has wares if you got the coins) |