Functionality

AI functionality has been criminally overcooked and oversold. When you wince and wipe your eyes for a second, you see the same primitives that build great web applications: a browser window, a text box, and a submit button. The key difference is that a lot of these web elements now have a sparkle emoji (✨) next to them to indicate “this is powered by AI.”

AI is not a new engineering architecture in many ways. Engineers will have to write HTTP handlers, push messages to queues, and respond back to requests from users. The main difference is that they aren’t writing the “if/else” statements anymore. This is where “AI powered” comes in most of the time.

“AI powered” is effectively a magical box that can take text (or base64-encoded bits) and figure out a stream of if and else statements for you. Engineers pre-2023 would have to write billions of conditional logic statements (by hand no less) to achieve the same functionality that inference models achieve for us now. Trained models + situational awareness + context engineering is the ultimate if/else statement in an engineer’s toolbox now.

From an engineer’s point of view, they are replacing if statements with “agentic loops” that can call a chunk of code (or tool, in LLM parlance). These agentic loops have a single point of failure with HTTP requests to OpenRouter, Claude, OpenAI, etc. But I digress.

The point I’m driving home is that “AI Powered,” while exciting, isn’t that much different than the applications of old. In many ways, AI-powered features are trading code complexity for accuracy complexity.

LLM Interfaces matter more than model intelligence

AI is magical. Despite my tone in the above section, I do believe we’re entering a new era of software. From my perspective, AI will only take off if the user interfaces to access it enter their own new era as well.

For example, OpenAI (and ChatGPT) caught fire because someone had the idea to create a chat interface for their model. They created a simple HTTP API, a submit button, and a simple 2008-esque web page and BAM! Lightning struck. OpenAI was several versions (and years) deep into their model development, but the barrier to value was lowered to anyone that had an embarrassing AIM screen name in 2003. (Like me: Gsur6). LLMs then became: type a few words, hit submit, and wait for the page to “load” the response... seems familiar. The thing that is missing is the hissing and squawking of a 56k modem.

For AI to reach the heights of possibility the world is anticipating, I’m assuming we will need to reimagine the interfaces that allow humans to interact with AI in the first place. Will we even need keyboards in 10 years, for example? Or will tools like Wispr Flow become the predominant way to control our computers? Neuralink, a reimagined “user interface” to computers, might seem bananas now, but what if that is the way AI can achieve its full potential? Would users prefer that?

The interfaces humans use will have to change for the advancements of AI to provide more value to said humans. It’s unlikely that smarter models will do us much good until we improve the interfaces to them (again).

Selling AI

AI is exciting to build, market, and sell. But the proclivity to buy AI does not match yet. After three years, the osmosis of excitement has not happened between sellers and buyers. Skepticism is rampant with AI, and not only from software engineers. Even my girlfriend, who works in PR and Communications, has her own doubts about AI.

There’s a certain level of dissonance surrounding AI to me. If you drive up the 101 towards San Francisco from the airport, you’d be right to assume that AI is the only thing that matters right now. Yet your Lyft driver is still manually controlling their car, using the gasoline they pumped themselves, using the cash they received as a tip. To me, the biggest thing that has changed due to AI (up to this point) is the shift in marketing energy to it, rather than the sheer value I receive from it. Even Dell has admitted that AI computers are not selling because of AI itself.

When it comes to selling AI, I’ve seen one selling strategy that works best: building pragmatic AI systems. Customers, by and large, are inundated with AI in every corner of their lives right now. If they are seeking out AI features, they will tell you. I promise. It’s more important to sell the value of your platform like the sales gods intended, not the AI that maybe(?) improves the core functionality a prospect is seeking in the first place.

Even the AI model providers themselves follow this principle. Simple APIs. Simple user interfaces.

What AI features you can build now

Reading comprehension scores are high for LLMs, and you are better off leveraging AI to read than to write in the immediate term. What that means in the real world is that using AI to transform information is better than using AI to create information. You need to find creative solutions when applying this principle to feature development. But one that I’m actively exploring is using AI to migrate between systems.

For example, an agentic system that can read an API response from, say, PagerDuty can quite easily transform that information into, oh I don’t know... FireHydrant. AI might be the best solution to a robust ETL system you could build today.

By using AI to read and transform information, you’re providing a known “system” that humans would have to go through anyway. This proves out the value of an agentic system with less “selling” than an innovative one that may not work in the first place.

Again: Selling practical AI features will triumph over innovative but misunderstood functionality. Bring people along with AI. Preferably not by force.

Where Is This All Going?

I don’t know. Neither do you. My best guess is that AI will find a way to improve most aspects of life by an unknown percentage, with some areas more than others. That’s why there’s so much money being poured into AI startups: even a marginal improvement of humanity will have an enormous return.

Said another way, I wouldn’t mind a 10% increase in a better life for everyone on Earth if it means a few years of annoying billboards, AI in my refrigerator for some fucking reason, or an AI music “artist.” I don’t need to open my wallet for those things. I’m confident they’ll die off on their own. Me not caring is the best assist I can give to ensure that. Ironic.

I believe AI will unlock innovations in medicine, transportation, food production, energy, and law (to name a few) far faster than it ever improves our sales funnels. I want a better sales funnel. But I also want food production to be stable for centuries to come. AI will be a part of the solution for both. Hopefully more for the food problem, though.

AI will change the world. Slower than businesses want, and always at the tolerance that we, the buyers, want AI in our lives. Swinging between impatient overestimation and practical underestimation is what creates lasting progress.

Now, to get to my dinner party. I’m late. If only AI could’ve called me a car.

We’ve achieved a fascinating milestone together. As of now, we will always have had more time apart than with each other. Before your death, we had almost 17 years together, and now, we’ve had 18 apart. For most, the amount of time they have with their parents far exceeds the amount of time they do not. Many of my friends still have grandparents alive for that matter. Yet here we are, you and I, with our distance expanding like space itself.

In the years following your death, I spent several of them feeling as if my age and my reality were out of alignment. While friends were at college, I was progressing in my career already. When people were at spring break, I was at conferences. When people were drinking... I was drinking with them. Ok, so maybe not everything was out of alignment.

Alas, your early death and my late teenage years were at an interesting cross-section. On one hand, several components of my personality were already well formed by your upbringing. On the other, a multitude of gaps were left that I had to stumble through life to find answers to. Such as: “What the fuck is a W2?” and “What does a healthy relationship look like?” Your absence meant that my life’s oddities and crossroads had no hotline for me to call.

After your death, I felt like I was launched off of an aircraft carrier. A commercial takeoff is gentle, with a restricted climb to make it comfortable for the souls aboard. In our situation, I had to grab the cockpit handle, give a salute, and have my organs rearranged as I was catapulted out over an open ocean with no time to correct if something went wrong.

They may not know it, but I adore spectating the people closest to me interacting with their parents. It never takes long before a parent gives some type of advice in a conversation. I don’t think they can help it. As far as I can tell, parents never stop being parents. They’re keen to advise, protect, and foster. It’s a default setting that has no off switch. They are always there to provide guidance, similar to air traffic control. I’ve also come to realize that the sudden departure of a parent such as yourself is also a (bleak) way of tutelage itself.

Which is why I think this milestone we have together now, more time apart than together, is so fascinating. Because in a way, half of what makes me is your masterpiece and the other is a strange amalgamation of my own style of growing up. I had the dumb luck of having an incredible group of friends around me. Pairing that with drum corps, relationships, career progression, founding FireHydrant, travel, sobriety, and a healthy dose of therapy has led to a mix that no one could have predicted the outcome of. I’m the human equivalent of a mutt. I think you’d love to see it in action.

It’s unnerving knowing that each day I wake up is one day more without a letter in return. This cold fact is amplified knowing that your infinite sabbatical will only add to the counter, never to reset it. But I’m here to say that at this juncture of time with and apart, I am thankful for your departure.

It’s macabre to say you’re “thankful” a parent has passed away. And I hope you know it’s not that I wished it, but more that there’s nothing more I can do but to thank you for it. There’s no sense in being resentful for your departure. Nor is it logical to carry on as if nothing happened. The only sensible thing I can do is mark December 23rd, 2007, as the day I lurched towards adulthood faster than I could have imagined.

It’s the day I took your unfinished map that you handed to me as I said goodbye at your hospital bedside, and had to begin filling in the directions myself. For 18 years now, that’s what I’ve been doing every single day.

And it was the gaps you left that had the most positive influences on my life.

So, to you, Ellen: The unfinished map you gave me is beautiful, and my adventures with it are forever ongoing. And I love you for it.

— Robert

This post is a follow up to a piece I wrote in 2017: Dear mom.

I don’t care about being sober anymore. It’s not that I’ve reverted to drinking (I still live an alcohol-free life) but I literally don’t care much anymore. I cared quite a bit seven months ago when I wrote Six Months Sober. And right after I published that piece, the outreach was palpable. Friends, family, and colleagues reached out also sharing how it had affected them. Many now reach for booze less.

My friends even stopped asking if I’m drinking at dinner. I was officially relegated to being the sober friend, huzzah..? One of my closest friends sent me a 6-month sobriety chip as a joke. It sits prominently on my desk. (Commit to the bit, Alejandro, and send me a one-year.)

I’ve been quite annoyed the last month, though. My plan was to write the “One Year Sober” follow-up. But it has been disturbed by, well, a lack of new sober benefits. The best piece of material I’ve posted in two decades of (sporadically) posting musings on my websites didn’t have an obvious sequel. My Marvel sobermatic universe was seemingly over before it even began. Yet here we are.

7 months later

I hate instant contradictions, but I now have reverse veganism with sobriety: I’m telling fewer and fewer people I have it. Newly discovered benefits of removing alcohol from my diet leveled off shortly after I wrote SSS. But I still sleep well, think clearer, and a slew of other positives. Although, I must admit, my reintroduction of soda did rebound my waistline a skosh. Whatever. Nicole still seems to find me attractive.

Alcohol’s absence in my life is still felt on a daily basis in many boring ways, but not nearly like the first six months of sobriety. I could argue this is a wonderful discovery, but given so many people still ask me what it’s like, I figured I’d play a last song about it.

Turbulence

I’m writing this late encore from a plane. And as a frequent flier, I experience a lot of turbulence – like even at this exact moment these keystrokes are happening. Despite logging dozens of flights each year, I’m terrified during moderate turbulence. I pay $8 to United to get WiFi so I can see when the turbulence will end when it starts.

I, like everyone, hate uncertainty and not having control of my situation. Turbulence embodies both. The sporadic air surrounding our plane creates uncontrollable conditions we passengers experience, but when we hit “smooth air” – it’s like (what I presume) smoking a cigarette is like.

It’s also exactly what the 7-month milestone of being sober felt like for me.

Sweet Sweet Satisfaction

After a plane leaves behind uncertain air, the captain turns the seatbelt sign off, and our collective butts unclench, it’s a pure injection of satisfaction. We feel the bliss of stability and the fictitious obituary I’m writing for myself gets crinkled up and tossed into the wastebasket.

But a flight without any turbulence? A Grade-A Snooze Fest.

The omnipresence of alcohol in my body was, at best, self-inflicted turbulence. The symptoms were exactly the same. My body was clenched, my thoughts unclear, and yes, the occasional intrusive thought of death would meander through my head.

Turning the corner after six months of being sober has felt like driving through Nebraska on a cross-country road trip. I haven’t discovered anything I didn’t know about. It’s flat. There’s nothing new to write about and I can only write about how many cows there are so much.

But long boring drives are when I do my thinking.

Decisions... decisions...

I’ve thought a lot about the negativity that surrounds alcohol the last year. The recent US Surgeon General’s scathing review of alcohol likely didn’t help, and the terms we’ve adopted to describe alcohol almost feel like Big Water™ wrote them.

For example, being “under the influence” has an especially negative tone in our society now, but one way or another, we are all under the influence of many things, at many times, in many forms. For example: The news is an influence. Your job is an influence. Age is an influence. And where you physically are at any given moment influences you.

I stand by what I wrote earlier this year: Alcohol was a net positive for me. It helped influence the bond of friendships, hundreds of wonderful nights in beautiful places, and in many ways, the founding of FireHydrant. Alcohol is fungible, and its purpose is one you may have the luxury to decide (for alcoholics, this isn’t an option unfortunately.)

As a recipe

Alcohol is an ingredient. But in my next metaphor, it’s a figurative ingredient.

The dish of life only gets more complex. A chef will tell you crowding the pan with too many ingredients means all of them cook less effectively. And for me, the brightness of other things in my life’s skillet weren’t coming through enough for my taste. Things felt, as Paul Hollywood would say, stodgy.

Something had to give. I chose alcohol. You may choose something else. You should choose crack if you do crack. Or meth. Or Fox News.

No matter what you choose to take out of the skillet of your life, though, you’ll definitely continue to think about it. I certainly haven’t forgotten about my love of cocktails.

My Martini

I still want a martini every once in a while. Back in the day, I’d bring a book, notepad, or even a laptop to the bar to unwind. I’d banter with the bartender. I’d meet new people. I chose hotels based on how close a great bar was.

But do I miss my martini or the situation I’d drink my martinis in?

Most of our total time in a bar isn’t literally drinking. In fact, by the numbers, I’d guess less than 95% of our time we’re even touching our drinks. People go to bars to hangout, get a little silly, and maybe meet someone new. I wanted to be in a place where the unexpected could happen. What better place for that than a bar?

On my Nebraska road trip the last 6 months I’ve realized that my beloved martinis were nothing more than a ticket for admission to sit at my favorite bars. Hell, even as I edit this very post (after surviving my turbulent-laden flight earlier) I’m sitting at a bar at my San Francisco hotel. Now I’ll order a bitters and soda, some food, and tip the staff out accordingly. It’s their livelihood, after all.

Early mornings without consequences

When I think about the early mornings I had to battle with a hangover, my head hurts all over again. The money spent, the damage done, and the lost Sunday. Those days are gone for me, but recently I had a fun reminder of what we ourselves used to do after having a few too many.

During the summer, my partner Nicole (who is my sober co-founder) and I had one of those early morning flights that makes you wonder what ghoul possessed you when you booked it. Our 4am alarm went off, then our 4:05am alarm went off, and we left our apartment en route to Newark Liberty.

But as the apartment elevator door opened to our lobby two neighbors were coming home from a night out. Their surprise at seeing our luggage prompted an unbridled and slurry “Are you going to the airport right now?!” – Witnessing their DUI (disbelief under the influence) was a poetic reminder of my past. After we replied with a simple “yes” not expecting a conversation at 4:30am in our lobby, they yelled back “Ok! Have a great flight!” as the elevator doors closed. But knowing we would not be upset with our dastardly drunkenness keeping us up so late felt familiar.

Flying into smooth air, and the seatbelt sign turning off.

The AI SRE category is red hot. The promise is compelling: What if machines could go on-call for you? Several startups have emerged to hunt this white whale of a product, so much so that our customers keep asking me which ones they should integrate with FireHydrant.

But WTF Even Is AI Site Reliability Engineering?

The SRE name has been bruised and battered for years. DevOps teams were renamed to SRE overnight, hoping they'd somehow "do SRE things." Now "AI" has latched onto the sacred SRE name, confusing it even more. I'm here to help the overwhelmed citizens.

Let's get this straight: SRE is a huge role that encompasses dozens of tasks. There's a reason SREs at large tech companies get paid the big bucks: It's a very demanding job with a very wide blast radius (of both success and failure).

In its current 2025 form, "AI SRE" encompasses two things:

1. Autonomously investigating incidents just like an engineer would open a dashboard and attempt to find the smoking gun in their logs.

2. Mitigating incidents by autonomously fixing the underlying cause(s) either with code fixes or rollbacks.

These AI SRE Agent Startups do not include features to notify on-call engineers, perform retrospectives, or create workflows for their incidents. This is the distinction between “Incident Management” and “AI SRE.”

The List

I've been tracking AI SRE startups for months, and I think this list could be valuable to businesses looking to try this new technology. I'm neutral on who's the best player—and I believe everyone should assume this category will change dramatically in the next 2 years.

Alright, let's get down to business!

Dedicated AI SRE Startups

These AI SRE companies focus on the investigation and remediation layer of the "R" in SRE. They're not building on-call scheduling, service catalogs, incident management, status pages, or retrospectives—just good ol' laser-focused products on helping engineers resolve incidents faster.

In alphabetical order:

Causely

Website: https://www.causely.ai/

Causely's causal reasoning engine automatically infers the single root cause when a storm of alerts begins cascading through your environment. The platform auto-discovers your environment and starts delivering insights in seconds from your existing telemetry—no setup or tuning required.

Cleric

Website: https://cleric.ai/

Cleric is the first AI for application teams that investigates like a senior SRE, autonomously investigating production issues and delivering findings directly to Slack. Backed by Zetta Venture Partners in a $4.3M seed round, Cleric reasons through problems it's never seen before by forming hypotheses and running real queries with your tools.

Neubird

Website: https://neubird.ai/

NeuBird's Hawkeye is an AI-powered SRE co-pilot that brings the reasoning power of LLMs to telemetry data with reliable, secure Agentic AI built for enterprise IT. The company recently raised $22.5M in funding led by M12 (Microsoft's venture fund), with participation from Mayfield, StepStone Group, and Prosperity7 Ventures.

Phoebe

Website: https://phoebe.ai/

Troubleshoot faster. Agentic search for your tech stack. Investigate errors, incidents, and more.

Resolve

Website: https://resolve.ai

Created by the co-creators of OpenTelemetry, Resolve AI handles all alerts, performs root cause analysis, and troubleshoots incidents within minutes. The platform operates autonomously to handle common alerts and actions, reducing escalations and saving up to 20 hours per on-call engineer per week.

SRE.ai

Website: https://www.sre.ai/

“Meet your new AI teammates that learn fast, act responsibly, and always deliver.”

Tierzero

Website: https://www.tierzero.ai/

TierZero AI automatically investigates, triages, and resolves infrastructure issues, believing that infrastructure should be self-driving by surfacing the right insights and anticipating issues. The company is SOC 2 Type II certified and hosts its production services on Amazon AWS with enterprise-grade security measures.

Traversal

Website: https://traversal.com

Traversal's agent parses logs, metrics, traces, and your codebase to narrow down root causes of errors or latency, replacing the flood of alerts and logs with easy natural language. The team consists of CS PhDs from MIT and UC Berkeley, with experience at industry leaders like Uber, Amazon, Citadel, and Mistral AI.

Vibranium Labs

Website: https://vibraniumlabs.ai/

Vibranium AI acts as your 24/7 on-call teammate, eliminating alert fatigue, pinpointing root causes, and providing actionable insights for faster incident resolution. The platform can slash Mean Time to Resolution (MTTR) by up to 82% and includes a real-time AI assistant that can join calls and transcribe discussions.

Wildmoose

Website: https://www.wildmoose.ai/

Wild Moose provides fast, efficient root cause analysis that improves with every incident, converting tribal knowledge into smart automations to navigate complex environments. The platform constantly improves performance with a system model that learns from each incident and integrates within minutes via APIs.

🕵️‍♂️ Did I miss your company? Email me! robert@firehydrant.com

Observability Tools Dipping Their Toes

Most O11Y players are dipping their toes—or cannonballing—into this space. It makes perfect sense: They have the data that investigations would be performed on anyway, so they're building agentic workflows on top of that goldmine.

None of these should come as a surprise:

• Honeycomb*

• Datadog

• Observe, Inc (observeinc.com)

• NewRelic

• Coralogix

• Dynatrace

• ...basically everyone

*Honeycomb is a genuine thought leader in this space. I recommend reading Austin Park's post. Their blog is a treasure trove of thoughtful pieces on AI in software development.

Incident Management Startups

For the past five years, several incident management companies have emerged to fill the void that PagerDuty left wide open—and all of us started taking advantage of it. As the CEO of an incident management and on-call tool (FireHydrant.com, if you didn't know), I have just a few… million… perspectives on this space.

Here's a list of incident management startups I'm watching that are also including AI/SRE capabilities in their offerings:

Incident.io

With a name like "Incident," of course they're going to build "AI SRE." Incident.io threw their hat into the AI SRE ring in the past month with the promise of "AI SRE resolves incidents like your best engineer" featured prominently on their product page header.

Of all FireHydrant's competitors, I have the most respect for incident.io. Their team and founders have always been graceful with me and FireHydrant, and I'm curious to see how their AI SRE + everything else platform plays out.

P.S. I will destroy you Stephen Whitworth 😈

PagerDuty

Obviously, the oldest of the bunch. Definitely not a startup anymore. They’re building an AI SRE too. They appear to be using their Rundeck acquisition from several years ago to bolster their position as an automated investigation and remediation tool. They've been integrating Slack with investigations, albeit poorly from what we hear from the field. I have yet to hear of a single company using it successfully—but maybe that's because I only speak with companies that switch to FireHydrant? Who knows.

Rootly

Rootly won't list FireHydrant on their comparison page (probably because of how much I’ve called them out publicly for the past several years).

Rootly is building an AI SRE, but it's really hard to know what it does because their screenshots are just copies (literally) of other vendors in the space.

To each their own.

Other Businesses

Many businesses have an "AI SRE" SKU of sorts in their product. Several DevOps platform and CI/CD companies and even code writing editors have begun to dabble with AI investigation and remediation. They're worth noting for completeness:

• Harness: https://www.harness.io/products/ai-sre

• Sentry: https://sentry.io/product/seer/

• GitHub: https://github.com/features/ai

• Microsoft Azure: https://learn.microsoft.com/en-us/azure/sre-agent/overview

• Cursor Bugbot: https://docs.cursor.com/en/bugbot

• Starsling (waitlist): https://www.starsling.dev/

I'm sure there are dozens more. If I missed yours, email me: robert@firehydrant.com.

And now FireHydrant!?

Nope. Not quite.

Alas, it’s me saying something I’m more excited about: Partnering with the next generation of AI SRE and Observability tools. All of them.

I’ve spoken with countless technology leaders recently, and they’re all saying the same thing: “We’re looking at all of the AI SRE tools on the market – and they need to integrate with FireHydrant.”

Businesses are looking for the right AI Agent(s) that fit their specific needs. Because tech stacks vary so widely in their complexity and design – no AI SRE will be a “one size fits all” tool. Businesses are even likely to purchase several AI tools for their needs.

And all of those AI tools will need a place to retrieve incident context, read retrospectives, and page the humans when they come up short. That’s the platform we’re building.

AI SRE is a genuine opportunity for FireHydrant. By letting teams connect their AI agents with our incident management platform, businesses actually get something useful. FireHydrant becomes the connective tissue between AI SREs and the real world—teams can bring whatever AI SRE agent works best for them and plug it into us as their management and on-call layer. It just makes more sense to us that way.

We at FireHydrant have our sights on something else that we’re more excited to solve 👀

Every day during the summer, a group of 165 young performers wake up in a high school gym on air mattresses. After they get up, their breakfast (the first of four meals a day) is served by dozens of volunteers out of a catering semi-truck parked outside as they prepare for rehearsal. It's a non-stop schedule for nearly three months with a typical day that looks like:

• 8am: breakfast

• 9am: visual rehearsal

• 12pm: lunch

• 1pm: music rehearsal

• 5pm: dinner

• 6pm: ensemble rehearsal

• 9:40pm: show run-through

• 10pm: snack

• 11pm: lights out

The group only had 4 hours of floor time last night – that is, sleeping horizontally. 6 hours of their night was spent on one of the 6 tour buses they've traveled over 5,000 miles on at this point in the season.

A hefty percentage of the summer season is spent in transit on a bus. Members spend so much time on their busses that they will make them into a studio apartment. Square footage: 2 feet by 2 feet. They'll even stick a bathroom caddy to the window to give them shelf space for the long-hauls members undergo. They get comfy, because a season consists of at least 15 contests in different towns across the United States. And the busses smell like nothing you have ever smelled before in your life.

Today is a routine morning for a group competing in the Drum Corps International (DCI) season. DCI is a summer activity that concludes with finals at Lucas Oil Stadium in Indianapolis, Indiana. The tag line of the activity is "Marching music's major league" – and it has been ongoing since the 1970s. It's a physical and creative activity that members from the age of 16-22 audition for to earn a spot.

Each of their shows are judged, scored, and expected to be better than the last time they performed it. Parents, friends, and enthusiasts cram into stadium stands on humid nights waiting for their favorite group to throw down a show. They are in the stands asking for mild tinnitus – and the drum corps' members happily give it to them. After their scores are announced, the busses and equipment trucks are once again loaded up to head to their next destination in the middle of the night.

But for a rehearsal day like today the only goal is improving their score. Today this group needs to fix an unmissable "tear" that keeps happening in their closer, make some adjustments to music, and get a few more reps in of the entire performance.

As members finish breakfast, they'll start the trek for rehearsal. By 8:50am members will have grabbed their equipment (instruments, flags, drums, etc), water jug, sunscreen, and hunt for the football field in the unknown school. Public schools are a cost-effective way for a drum corps to house the members for the summer season – and they need at least one football field. Operating a drum corps requires being extremely cost aware as it is a capital intense activity, with few ways to generate revenue.

At 8:59am the young performers within the hornline will set up, elbow to elbow in an arc shape, instruments pointed at the horn sergeant, shoulders back, standing perfectly straight, feet angled at 45 degrees, fingers curled over the valves, looking towards the center to make sure their spacing is the same between the nearly 70 horn members.

The horn sergeant, usually a member in at least their 3rd season shouts a rhythmic "Ready... front!" and the members snap their heads forward, and then their horns down in front of their chest. It's 9am, and their rehearsal has officially started.

Why did I write this?

The cost-benefit analysis of sleeping on a gym floor, traveling by bus, showering with your peers, and rehearsing during the hottest months of the year is admittedly insane. To the members, though, it's worth every single penny. These young members are paying to get sunburned, play music, toss flags, and make shapes on a football field. And I myself would not be where I am if I hadn't done the activity.

The last 7 years I’ve been building and running FireHydrant and the more I realize how many lessons could be extracted from DCI. Because DCI is a masterclass in organizational alignment and execution.

On Band: Pre-Tour

Before the DCI season even begins the show music and drill (shapes on the field) has been designed and written. There will be edits as the season progresses and feedback emerges, but for the most part the music, the visual, and the show design will stay the same for the entirety of the summer and receiving a new score each performance.

For a corps to increase their score – they need to increase their risk and execution. A 120 beats per minute tempo while playing simple notes and moving a few feet? Piece of cake. Running and playing Metropolis 1927 at 184 tempo as a closer perfectly? That's how you win.. But to increase execution, the corps needs to immerse themselves for several weeks to get started. Enter: Pre-tour.

Pre-tour is when the corps goes all-in on learning and rehearsing their show. Corps members spend every day learning their show for upwards of three weeks with no public performance. Members say goodbye to their loved ones and lock themselves away for the hardest part of their season. They can expect a few sunburns, sore knees, and harsh but fair feedback from the staff. The show memorization process starts on day 1 with every member getting their first "dots" on the football field.

Dots are what make a band a marching band. A dot, put simply, is a coordinate on the field. Dots are the "longitude and latitude" of marching band and DCI shows. It details to every performer where they need to be, at what count of the show, and how many steps they must take to get there. By stringing together sets of dots, you get the well-known shapes and effects that marching bands create called “drill.” If you can imagine a huge sheet of graph paper over a football field – that's how dots work.

It's not possible to memorize a dot on the first try. Members will write down their dots, any notes, and reset multiple times during rehearsal to remember two measly sets before learning the next one. A single rehearsal might allow the corps to learn 10-15 dots. The most competitive shows will have over 200 sets members must learn.

Once learning drill has built momentum within the corps – they start cooking with gas very quickly.

On Business: Routes and Roadmaps

In business there are three terms that all vaguely resemble each other. Businesses typically have a mission statement, a vision, and a product roadmap. In my eyes, they're all different phrases to help explain the same thing: Where we are right now, and what are we doing next to get to where we want to go.

For example, FireHydrant's vision is to help businesses build more reliable software. We have a roadmap that helps us know the route we are taking to achieve that, and several fiscal and quarterly goals that define the boundaries of the current mission. You can ask any member of the FireHydrant team what our fiscal year goals are right now, and they'd be able to tell you without hesitation.

But getting there was not an overnight achievement. Before we started this fiscal year (FY26 for us) we planned, and planned some more. We had a pre-tour for ourselves.

In essence, the leadership team was responsible for drawing the sets of drill that the rest of the organization will ultimately "perform" with. We know that for us to hit a large release effectively, we need to maneuver our engineering team into a position where they can deliver results with exceptional execution. We focused on what we wanted to achieve, and who we wanted to achieve it with.

A significant portion of my job as CEO is repeating our goals and current mission in various ways – the drill of our company. Because, as with band, it's impossible for anyone on our team to recite it after hearing it for the first time. I need to visualize what dots need to be drawn on the imaginary field our business is playing on – and make sure everyone else knows them, really knows them, too.

Because it's not enough to give a Google Slides presentation about what is up ahead while ending with “Ready, set, go!” Humans can't retain complex information on the first try – and need to hear it repeated over and over again for it to truly stick. Eventually, when everyone knows their drill, only then you can start cranking up the tempo.

On Band: Tempo

When performers take the field for competition they've already rehearsed for hundreds of hours and have earned their spot to be on the field. When a show is about to start, the most important thing every member needs to internalize is their tempo. If members start their show with the wrong tempo, they will (quite literally) be off on the wrong foot.

During rehearsals, educational staff will use mild torture by blasting a beat into the ears of every member of the corps during rehearsal. The machine is called (and this isn't a joke) "Dr. Beat." I can still hear it to this day over 13 years later. But there's good reason for it.

Tempo is the life force of a show. Performers are taught to link their foot steps to the conducting hands of the person on the podium: the Drum Major. The tempo on the field at any moment is set by the podium's hands, but it is also amplified and aligned further to the steps of the center snare (the snare that is always in the middle of the 8 other snare players). Without a drum major, there isn't a tempo.

For all 165 performers to know the tempo, there's a staring competition between the center snare player and the drum major. The visual tempo of the drum major's hands is amplified by the percussion section of the corps – a group that every member can hear at all times. The drum line’s drill is commonly located behind the corps on the field to achieve this as well. This means the hornline and color guard see the drum majors hands in the front, and hear the drum line behind them. This enables "vertical alignment" to the music and visual movement: perfectly synchronized harmonies and foot steps. However, when the drum major's hands are not aligned with the snare players or worse – the entire corps – terrible things can happen, like a tear.

A "tear" is a nasty moment in a drum corps performance, and it's hard to miss in the judges box. If the horns are suddenly 1 beat behind during a tempo change from the drum line, the flags, pit orchestra, rifle line, and drum line will hit a big show moment a ahead of time. It's a jarring experience for the audience, and it impacts the score the corps will receive from the judges. Most corps can self-correct a tear mid-show by realigning to the drum major's conducting, and a fast correction can mitigate a large impact.

But a really bad tear – one that goes on for a few bars of music – can drag the entire energy of the performance down. The moment a corps finishes their last set – every member knows what kind of a show they just had. They can feel it as the reverberations of their last note fade along with the applause of the audience.

In a business, this is no different. Every team member knows when the business has its own version of a tear.

On Business: Operating Rhythm

As the CEO of FireHydrant, I'm the drum major of the organization. My job is to make sure everyone knows what we are doing, when we are doing it, and how fast we are going to do it. The tempo I set needs to be matched by every team member, or we'll have the same score-killing tear.

The tempo I give to my team also needs weeks of preparation in tandem with the organization's leaders. If my leadership team has no idea what their next tempo is – or they make one up, our whole company rips apart. A tear in drum corps might mean less points, but a tear in a startup means you go out of business.

I feel that it's necessary to clarify one thing: Operating rhythm is not our goals. Tempo is a function of taking how long we have, and how many things we need to accomplish. For example, If we need 1000 new logos in a fiscal year, our tempo is roughly (using a linear scale) 19 logos a week. The importance of maintaining that tempo is also important. It's why we have an all-hands every month – it's the downbeat that ensures everyone is marching on the same foot.

If people aren't using the same tempo in their work as their counterparts in other departments – it gets messy. Because when engineering is building a feature but it falls behind and sales has been preparing to sell that new feature for its release – that's a tear. If marketing is using a new messaging framework but sales is using the old one in demos, that's a tear. If we tell the entire business we have 8 weeks to ship a feature, but then change the expectation to 6 weeks – you guessed it: another tear.

To create an aligned business, I need to have the same staring competition a drum major has with their center snare – but with my executive teammates. If our collective tempos are not locked into each other – the entire company feels it.

On Band: One show

Sometimes I'll accidentally pin someone down into a conversation about Drum Corps. People are polite about it, but a question that comes up consistently is "how many different shows do they put together in a season?" – One.

It's a fair question in the world of viral videos of college bands performing a different show at every home game. 400 Texas college band members on the field making a Mario show is entertaining and perfect for the setting, but the 10 hours it took to learn that show wouldn't make it by drum corps standards.

In a single season, a corps competes with one show. One book of music. And one book of drill. They rehearse the same parts over and over again throughout the summer to increase their execution, and therefore, their score. Earning a higher score eventually becomes less about the edits they make, and instead about how much easier the show becomes because of how well the members know it.

On day one of pre-tour, you are fighting for your life to learn a show. The 200 sets of drill per member is an incredible amount of memorization. Then layer music memorization on top of that, and it's downright super human.

But something amazing happens in the back half of the season: members can perform their show with near muscle memory alone. Over the course of 2.5 months, they become so accustomed to the show that a staff member can make an edit to the music the day of a contest and they'll be able to perform it later that night.

Members also create a unique camaraderie each season for themselves. This one-of-a-kind social fabric of a corps creates an excellence standard that few people ever experience. No one wants to be the member that holds the corps back from earning a medal that season. Everyone is expected to pull their own weight.

And because the show, the theme, and the members stay the same – a corps can execute (and edit) their performance at a higher level through sheer repetition. It's the manifestation of "A river cuts through rock, not because of its power, but because of its persistence." - James N. Watkins.

On Business: Consistency conquers

At FireHydrant – we've had inconsistent seasons in the past (that is my fault, and another post for another time). Roadmaps changed mid quarter, features fell out of thin air, and engineers were building a totally different set of products than what sales is currently selling. When we are locked into one "show" for a season, the momentum builds on itself.

Take Signals – our on-call product we launched last year. We dedicated the entire organization to building Signals over the course of 4 months (not much longer than a DCI season), and built a kick ass on-call product replacement that is taking on incumbents easily. That level of alignment allowed us to close hundreds of thousands in revenue before general availability. As the months went on, minor features that we wanted to add were completed in a single day because we knew the "show" we were working with so well. We had muscle memory of which files contained which features, we had the same tabs open in our browsers, and we knew exactly what each person was responsible for.

When the team is focused on one thing – one single show – it's far easier to build momentum on top of it, instead of switching all of the time. A social fabric also forms, creating a team that can take edits far faster than if we are constantly switching.

The Prize

Throughout the season, a drum corps show takes on a life of its own. We'd give sections nicknames only we knew. We'd carry out odd rituals like running and touching a water tower in the distance when we weren't rehearsing well. We'd stand in the rain rehearsing music that we hadn't nailed down yet. We'd miss weddings, funerals, and graduations while we listened to judges tapes on the bus ride headed to our next gym to sleep in.

There was a common saying we heard in our drum corps that always stuck with me. "The effort is the prize" – my trumpet tech in my last season even had it tattooed on his arm. We knew that our best show would most likely be on a rehearsal day with no audience or judges. It would be just us, some fireflies, and our loyal staff on a warm night in the middle of Pennsylvania. Our final hit would have no crowd’s applause, no scores, and no ceremonies.

It's important to remember that no matter what team you are on – a drum corps or startup or otherwise – that the moments that matter are rarely the ones that have a score. The best features a business builds won't be met with applause, and the hardest won sales deals won't have a medal ceremony. But the team you worked so damn hard with to earn these moments is, in fact, the greatest prize. But it’s also true that the more effort that is put in, the more likely it is you’ll have the receipts in the form of a medal – or an IPO.

Eventually a drum corps show has its last audience. When finals concludes and scores are announced, that show will never be performed ever again. It's the one parallel I could not draw between band and business – and I'm grateful for it, because I don't want to play a last note with FireHydrant.

Not yet.

When my relationship with alcohol ended nearly six months ago I knew it was the right outcome despite the heartache it imposed on me at the time. I’m annoyed I have to admit this, but it was a bit of a heartbreak suddenly not knowing what a life without her would be like. We had so many planned trips together!

Before you ask – I’m not labeling myself as an alcoholic. I simply had a moment where I went “What’s the point of this?” And here we are six months later and I'm getting the hang of being sober. To mark the rather mundane occasion, I’ve written six things (one to represent each month) that I've had the pleasure of discovering.

October

My first rule of being sober: I have to tell you immediately or it will be awkward lickety-split. And believe me, it's more for me than it is for you. I'm scared you'll offer me a drink, or (worse) very generously, hand me one without asking. Then I have to turn you down, you polite purveyor of provisions. I hate turning people down. It's better if I blurt out "I don't drink" or if I don't want to explain myself I'll ask for a Coca-Cola with a sheepish "Just a coke please." It's how I cope.

November

The second thing I learned was that... no one cares. When you self-impose booze as a dietary restriction, you almost feel like you're also imposing it on others at the table. I found that, after several dinners and shindigs in New York City, people actually enjoy having a sober cat in the room. As it turns out, some people don't want to drink much at all, and the sober person provides that "out" per-se. Many have even thanked me. Several more have said "I wish I could do that." One of my favorites is when the bill comes and people go "Wow this is cheap without alcohol!" Yes, Shane, it is.

December

I also learned that sobriety benefits are not as punctual as you'd like them to be. Don't get me wrong, you'll immediately have more energy from sleeping better. But when it comes to wearing those old jeans from 10 years ago – give it time. I didn't begin seeing any weight loss until about a month in, and when it started, it fell like a rock. I've lost 25 pounds in six months. I didn't need to lose those pounds, but I did and I'm not one to complain about a fun thing.

Other health benefits I can't explain with medical terms were notable too. I seem to remember more mundane details of things. Certain foods also taste better – that one I was not expecting. I think I've become even wittier and faster to make a good joke. Maybe my friends would have something to say about that. I'm also more self-aware.

January

The fourth thing I learned is that booze provided a particular entertainment that is, even six months later, hard to find a replacement for. For example, I'm a big fan of pop-punk and went to well over 30 concerts last year. But, I must admit, the Fall/Winter shows without drinking were fairly lackluster. As it turns out, some bands I liked were actually not for "sober-me" live and in-person. The wet truth of the matter is that alcohol made things easier to enjoy. There's even a bar I used to frequent in Brooklyn that now when I walk by it's open door I can't help but think: "Did it always smell like shit?"

February

The fifth thing I have the pleasure of enjoying is money. And by money, I mean more of it. Booze is not friendly to one's wallet. The last few years there were times where a bar figuratively mugged me. I had exchanged hundreds of dollars for alcohol and all I got was a bruise (somehow), a biblical hangover as I wake up to the sound of Futurama playing on my television with an (empty) Joe's pizza box on the floor.

I'm now using my new wealth to buy legos, skis, and camera equipment I do not need.

March

I maybe should have written this at the beginning, but I actually deeply appreciate what alcohol has given me – and not drinking made that realization more of an epiphany than I would have expected.

Many of my cherished fuzzed-edged memories include a friend doing something profoundly dumb because we had a few too many together. Many days of many trips were topped off by a wonderful glass of wine with a spectacular view (that I may never see ever again.) Many of my career milestones were paired with pint glasses clinking at The Whiskey Tavern. My (only) perfect 900 in Skee-Ball was met with cheers from strangers and shots. I've imbibed at some of the greatest restaurants in the world and made friends in far-away places like Happiness Forgets in Shoreditch, London. I wrote the original diagrams and ideas for FireHydrant with several martinis at Dante – one of the best bars I had the pleasure of becoming a regular.

Alcohol has netted out as a positive. The breakup with it six months ago was merely a natural phenomena. It became clear that the benefits of alcohol were no longer exceeding the cost to obtain them in the first place. The impact on health, finances, and (in some unsettling circumstances) relationships with friends and family hit my self-defined tipping point. I didn't want to see my alcohol-influenced-experiences portfolio go from being in the black all these years, to going into the red.

As with every other relationship I've had, be it with human beings or hyper-personified substances, it set up the next relationship to be a far cry better. You learn what you like, what you don't like, and then you refactor your life and move forward with new adjustments. Someone told me to never reply "I'm so sorry" to someone who is going through a divorce. Instead, say "Congratulations." because it means they achieved a high-point of clarity – despite the pain or challenges.

I've also discussed ditching alcohol at length with my peers, friends, and family for the last several months. Many of them have cut back on their own accord with others joining the sober ranks, too. It feels like a new trend to me. Contrary to popular belief, It's still fun to go a hole in the wall with friends who drink, and each time, I always see my “ex” sitting at the bar. We always acknowledge each others existence with a gentle nod before returning to our groups.

And when the bartender finally comes my way, I'll reach into my new bag of replies that I've developed for the last six months.

"I'll just have a coke for now."

1. There’s large scale fraud occurring and nobody has noticed

2. Microsoft is incompetent and can’t find my transactions despite me giving them everything they’d need.

I’m leaning towards incompetence, but I can’t be sure. Come along to see why.

This year alone I’ve been charged almost $300 dollars from “Microsoft.” The random credit makes no sense either. The charges are at random times, and the descriptor on the transaction is convincing enough that I thought it was Microsoft.

MICROSOFT*14 DAY TRIMSBILL.INFO

I’ve had these charges in the past and had no idea what they were – so I cancelled my Amex and had a new one sent to me. But the charges persisted. So I reached out to Microsoft directly to ask what exactly these were.

I provided all versions of my Amex (I have the history of my cards in 1Password) to the support agent, and, to both of our surprises – no transaction for my name, cards, and amounts is in Microsoft’s system. At least that’s what support told me.

This is when I started reaching for my Reynolds Wrap. For a few reasons:

1. “14 day free trial subscription” is an oxymoron.

2. It happened every few days in October

Digging Deeper

The transaction descriptor has an interesting domain on it: msbill.info – So a quick visit to it does a 301 Redirect to:

https://support.microsoft.com/en-us/account-billing/how-to-investigate-a-billing-charge-from-microsoft-398c5328-364c-d5e4-ea8f-f5ad60562a93

I followed the steps here… and nothing. No closer to figuring out this charge. So I decided to Google it with exact search.

Ok good so I’m not the only one confused by this charge. But all of these are community forums and might be dominating the actual answer. What if I remove them and search for only “MSBILL.INFO” on Microsoft’s website.

Well that’s odd:

You’d think that Microsoft would have a page explaining their domain for transactions somewhere right? Well now I am very curious about this domain. Let’s do a quick cURL of it and see what it’s doing:

$ curl -vvvv msbill.info
* Host msbill.info:80 was resolved.
* IPv6: (none)
* IPv4: 20.112.250.133, 20.236.44.162, 20.231.239.246, 20.70.246.20, 20.76.201.171
*   Trying 20.112.250.133:80...
* Connected to msbill.info (20.112.250.133) port 80
> GET / HTTP/1.1
> Host: msbill.info
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Sat, 16 Nov 2024 21:23:16 GMT
< Server: Kestrel
< Location: https://support.microsoft.com/help/10623/microsoft-account-unknown-charges
< Strict-Transport-Security: max-age=31536000
<
* Connection #0 to host msbill.info left intact

Server: Kestrel is a .NET web server, so that would hint at this being a real Microsoft domain. But, to make things interesting, let’s try it with SSL:

curl -vvvv https://msbill.info
* Host msbill.info:443 was resolved.
* IPv6: (none)
* IPv4: 20.236.44.162, 20.231.239.246, 20.70.246.20, 20.76.201.171, 20.112.250.133
*   Trying 20.236.44.162:443...
* Connected to msbill.info (20.236.44.162) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=*.oneroute.microsoft.com
*  start date: Oct 25 02:40:26 2024 GMT
*  expire date: Apr 23 02:40:26 2025 GMT
*  subjectAltName does not match host name msbill.info
* SSL: no alternative certificate subject name matches target host name 'msbill.info'
* Closing connection
curl: (60) SSL: no alternative certificate subject name matches target host name 'msbill.info'

I have a hard time believing Microsoft would host any domain they own without valid SSL on it. So let’s see who owns it:

$ whois msbill.info
Domain Name: msbill.info
Registry Domain ID: 851c4c135e6742d7a0311f7c380cad9a-DONUTS
Registrar WHOIS Server: www.whois.corporatedomains.com
Registrar URL: http://www.cscglobal.com
Updated Date: 2024-09-25T05:04:49Z
Creation Date: 2016-09-29T22:33:19Z
Registry Expiry Date: 2025-09-29T22:33:19Z
Registrar: CSC Corporate Domains, Inc.
Registrar IANA ID: 299
Registrar Abuse Contact Email: domainabuse@cscglobal.com
Registrar Abuse Contact Phone: +1.3026365400
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Registry Registrant ID: REDACTED FOR PRIVACY
Registrant Name: REDACTED FOR PRIVACY
Registrant Organization: Microsoft Corporation

Ok this makes me think my charges are actually from Microsoft. CSC is a major domain registrar and it does indeed say “Microsoft Corporation” – but that doesn’t mean someone doesn’t have a CSC account and registered msbill.info

A lot of the information is redacted – and a whois for microsoft.com is not. Why redact msbill.info and not microsoft.com ? Maybe Megacorp™ problems, maybe fraud. So let’s see if crt.sh has anything about this smelly domain: https://crt.sh/?q=msbill.info

I don’t know about you, but I feel like this list a little shady. “Catposters.net” – cmon. It’s not the the exact certificate common name, but why oh why is it in here in the first place. Why does Microsoft have a domain that has NO certificates ever registered?

Let’s look at something else, the “digital receipt” that Amex has a link to.

Again – “14 Day Trial Recurs Monthly” is sketchy enough as it is.

I’ve never seen this product “Ethoca” before but it’s legit and there’s plenty of resources online saying that. So this is where I lean more towards “this is a real charge and just no one can tell me why” – IE: Microsoft is incompetent.

But that’s no fun. Let’s keep going.

I shared this screenshot along with my bank statement screenshots with Microsoft support and they claim they cannot find these charges.

So how easy is it to charge myself this amount with this smelly transaction descriptor? Extremely easy it turns out.

I happened to have a Stripe account and I thought “Fraud against myself isn’t fraud, right?” And if it is – it was totally the other guy that made me do.

So if it took me a whopping 5 minutes to fire an “MSBILL.INFO” charge against myself … is it possible that someone has been sneaking this by for years? I honestly don’t know. Again, it’s Microsoft – they are so big it’s almost impossible to tell.

Conclusion

I called Amex and even the lovely support person I was connected was beyond baffled, too. None of it made sense. The descriptor, the “14 day free trial” yet I’m getting charged constantly, the timing of it all, the pages upon pages of real people complaining on Microsoft’s website.

So, for now, I’ve added a Merchant Lock to my cards preventing Microsoft from charging me. But if it’s NOT Microsoft and I’m still charged – well that’s the smoking gun.

And if you’re someone that found this blog post after searching for the same thing – welcome, we’ve got jackets. 🧥

Just like incidents, there are multiple contributing factors – and I wanted to take some time to write why this market is shifting so unbelievably fast from my perspective as the CEO of FireHydrant.

The rapid rise of commercial incident management tooling

Incident management is not a new idea, but it is a relatively new market. Just over five years ago the Google SRE book made its debut, Jon Allspaw spoke at every conference imaginable, debates about root cause rattled Twitter timelines, and all of a sudden, there was a groundswell of interest in more efficiently responding to incidents, running effective retrospectives (postmortems), and adopting best SRE practices. That interest was quickly met with solutions: commercial tools and structured advice for how to actually implement change. In a rapid clip, SRE became one of the most in-demand jobs. 

I merged the -m “initial commit” of FireHydrant in September 2017 and I announced it on HackerNews six months later (cue the Dropbox comment vibes). Blameless, Kintaba, RigD, Squadcast, and Transposit all entered the scene around the same time. Monitoring and observability providers took notice, as did the alerting and on-call incumbents. There was a frenzy of acquisitions that resulted in Atlassian’s status pages, PagerDuty’s rundeck automation and Grafana on-call. By 2021, focused incident management providers emerged to compete with FireHydrant (hello Rootly and Incident.io). More recently BetterStack joined the club.  

It’s easy to understand why this explosion of standalone reliability tools has occurred: all software will break at some point. And there’s money to be saved (and made) by the way teams respond.  

A cosmic shift toward tool consolidation

For more than a decade, the budget for software development (and its accompanying tooling) seemed to grow on trees. And then, shortly after the solidification of the incident management category, we saw the world press pause. For the last few years engineering teams have been under increasing pressure to get by with less expensive tools, limit new purchases, and consolidate the jobs they need done into fewer vendors. 

Since PagerDuty is often a massive line item on an engineering budget, we felt the squeeze from some of our customers and prospects who needed to justify a tool for both alerting and on-call and incident management. Certainly they couldn’t wipe out their alerting provider (who takes the batteries out of their smoke detector?). But perhaps this incident management stuff could go. “Why can’t PagerDuty just do both?”, our customers would report being asked by their CFOs. “Their website says they do incident response.”  

But by 2023 incident response tooling had proven itself as more than a nice-to-have. Teams could leverage in-product analytics to demonstrate critical improvements in their reliability (and the $$ associated with less downtime). Last year FireHydrant closed some of our largest logos yet – including one featured front and center in a recent PagerDuty earnings report. 

The writing was on the wall: modern engineering teams don’t want to live without commercial incident management tooling. The pressure is on to find a budget and consolidate tools. But not every team can withstand this pressure. And why should they have to? For savvy incident management providers, consolidation of alerting and on-call with incident management tooling is a natural progression. 

But aren’t you scared PagerDuty will just build this? 

I’m openly critical of PagerDuty. I mean no ill-will to the hard working employees of PagerDuty. You’re generating over $400M in revenue yearly – way more than anyone in the incident management category has (for now 😏). But, let’s be honest here: The PagerDuty product has not had any material innovation in its core focus area (alerting and on-call) in years. You can’t add multiple services to an incident, you can’t page a team, you can’t alert anyone without also opening an incident – it’s genuinely baffling. From the outside, it has felt like the solution to lack of innovation was to simply build more things. The Billboards listing a bunch of random tech terms don’t help with that sentiment.

Naturally, the leadership at PagerDuty sees the same trends we do – both the market opportunity and customer demand for a consolidated incident workflow. Otherwise, they wouldn’t have acquired Rundeck and more recently Jeli. The trouble is, engineers aren’t demanding more tools. They’re demanding more efficiency. And you can’t Frankenstein your way to cleaner and more efficient incidents. 

What I expect in the next year

PagerDuty isn’t going to cease to exist tomorrow, but at the current rate, I’d expect a significant change in their business operations in the next year or so. There’s already been one PE buyout rumor this year. The stock itself has not moved. And multiple financial institutions are downgrading the stock entirely.

No matter what happens, though, one thing is certain: Alert + Incident Management belong together, and those markets are going to merge, and the companies that moved the fastest with excellent execution will be the long-term winners. The ones that didn’t, will struggle. A classic example of innovators dilemma. 

PagerDuty left the gate down over their moat, and FireHydrant’s Signals (general availability) led the charge into the castle grounds. Days later, Incident.io (invite only) and Rootly (coming soon) followed suit. It’s become clear that there’s plenty of excitement for a consolidated alerting and incident management tool and that there are lots of smart people building their own, innovative takes. For now, our eyes are laser focused on our own approach: deeply integrated, staunchly reliable, rich with data, meaningfully bolstered by AI, and charting toward a place where the best incidents are the ones that never happen. ✨Watch this space ✨. 

I wouldn’t be doing my job if I didn’t also say: Check out Signals – our modern, fairly priced, and kickass on-call and alerting tool.

"Why hasn't PagerDuty done this?"

The first few years of FireHydrant were a constant barrage of this question. Investors, friends, and prospective customers asked it in their own way, and I responded the same way each time: "Why haven't they done anything?"

My response, admittedly, is cheeky. But universally, it garnered the same reaction: "That's a fair point."

As the CEO of FireHydrant, an incident management tool, it's my job to have answers to these types of questions about other businesses in the same vicinity as ours. But that one has remained the most elusive to me. We've been building an incredible incident management tool for almost five years, and 95% of our customers have already integrated an alerting tool. Which meant the question turned towards us:

And then I started replying with the same thing: "That's a fair point." So today, we're opening up the waitlist for a modern replacement for PagerDuty or any other alerting tool you can think of.

It's called Signals.

The problem with alerting tools today

I see four main problems with the alerting ecosystem as it stands today, and I've spent time with customers and industry experts validating them all. Here's what I've learned:

1. Alerting as a standalone tool is "crazy fucking expensive," and no one can tell you why. People have reached their boiling point, and current macroeconomic conditions are drawing even more scrutiny to the price of alerting tools. CFOs are not happy campers right now.

2. Scheduling and substitutions are painful, and it hasn't materially changed for years. It took several years even to get simple round-robin logic on schedules. Moreover, you can't do apparent tasks, such as temporary coverage to walk the dog or pick up the kids inside Slack, either.

3. The service directory is a sham, and teams mean nothing. Every business I spoke to inevitably griped about their configuration: "We just use the service directory to represent teams because there's no other way to page them otherwise." – This is asinine, and means service ownership isn't genuinely possible without ugly hacks.

4. Alerts and incidents are the same, meaning people get screwed regarding noisy monitors. This also prevents getting valuable insights such as alert noise ratio and accurate mean time to detect metrics.

This list is far from complete, and several issues with current providers are unnamed for brevity. I know I have many of my own as a recovering on-call engineer, too.

But the feedback was too consistent to ignore, and we decided to do something about it this year.

FireHydrant: Signals

Alerting and incident response belong together.

First, alerting is a must-have for most businesses building software in 2023. But what isn't a must-have is paying for hundreds (or thousands) of seats that are never utilized for sometimes several months each year.

So we're resetting the pricing standard: Active user invoicing. You'll only pay for the notified users in any given month when you use Signals for alerting. Signals is also an add-on to our core incident management platform, meaning you get all of the power of modern incident management, alerting, status pages, and retrospectives for less than the average per seat on PagerDuty. This also assumes you're notifying 90%+ of your entire on-call rotation – a high percentage for most businesses.

Next up: Configuration. FireHydrant Signals leverages our existing service catalog to scope escalation policies, signal rules, and schedules to a team. You need to notify teams, not services, about incidents. It also means you can quickly ask at any given point, "Who is on call for Team X?" and page them immediately. Additionally, since services have ownership in FireHydrant already, you get the same backward compatibility with whichever tool you use. And, yes, our Terraform provider will be updated on day one.

Finally, In Signals, there's a clear separation between an incoming Signal, an outgoing alert notification, and a declared incident. The benefit? Clear-as-day analytics to determine which teams receive the most alerts and which ones have the best signal-to-noise ratio. This allows teams to have data-backed discussions about which alerts they need and which they can drop entirely. The result means more sleep, happier on-call teams, and faster assembly time.

Signals is also built from the ground up by FireHydrant. We're not acquiring or merging with a company to be able to claim that we now do alerting overnight. The reality is that no company has solved it the way we think it should be, necessitating a greenfield build. Spoiler: It's badass.

Coming soon

Signals is being released this Winter to everyone. Still, if you want it earlier, head to www.firehydrant.com/signals to sign up for our waitlist and get access to the beta when it launches. We'll also release technical details for several months about how Signals is built, considerations we've made, and why it's kickass software solving a real problem.

I'm excited to show everyone what we've built soon.

- Robert

Join the waitlist

I did this every time production broke and was responsible for responding to the incident. I've seen incidents varying from dropped production databases because rake db:drop was ran in the wrong terminal window to a Spinnaker instance redeploying three-month-old code because Redis restarted. 

Our changes are the common catalyst for production incidents, but why?

The Life of a Codebase

Code, and the infrastructure that runs that code, are constantly changing. Nothing can stop us when we're bushy-tailed and bright-eyed building our first features in a wide open field. And that's (sort of) right. Short of writing just lousy code, new codebases that have very few components are inherently more reliable—small projects Just Work™.

Just look at rubular.com – a long-standing website I've used countless times to write regex statements through trial and error (I hate regex). In the 13 years I've written Ruby, this website has not changed (other than the underlying Ruby version).

But only some codebases can be like Rubular. The codebases that pay the bills are under constant scrutiny from those who rely on what they provide. Customers, those damn paying customers, are the most prominent reason why codebases change. We write new code to compete, but that new code often takes on a life of its own.

Writing code is like playing Jenga, kinda.

Jenga, the excellent game of wood blocks and gravity, is how I think of an evolving codebase. The tower starts as a perfectly stable grid of wood girders and excited players. The first player to go rarely has rhyme or reason for their initial move. The tower is so stable that any piece can go nearly anywhere and won't fall. But as the game progresses, the tower becomes taller and more unstable. Each player gets increasingly nervous, hoping they're not the one to make it ultimately collapse, ending the game. 

Each turn in Jenga is the same as when you merge a pull request on GitHub.

There's one minor (oh so minor) difference between my Jenga analogy and writing code, though. When we write code, we're also minting new wood blocks and attempting to slot them into place. As engineering and product teams, you're given a choice on where we think we should place our next block. Do you want more features or more stability? Should you fortify any load-bearing tech debt you have?

If we're making a new feature, the block is likely getting added to the top. If we're refactoring an old feature, we could slot our girder into the middle section. If tackling overall tower reliability, we could slot a tiny steel brick at the bottom, bringing the center of gravity down and creating more stability.

Jenga, but with a twist.

As a Jenga master code carpenter, you can play different blocks, and they change the dynamic of the game you're playing. I reckon you have three types of main blocks to play each turn:

1. Wood block: Create and deploy a new feature. They can be moved or added to add height to the tower each turn.

2. Toil block: These blocks must touch at least three other blocks, and can't be moved for at least 25 turns after being placed.

3. Process block: These blocks can be added anywhere on the tower but can't be moved for at least 50 turns. Also, at least 2/3rds of players must agree to the location of the process block.

New features and code aren't the only things that stabilize your Jenga tower. Your decisions to go down a few floors and refactor girders of choices past make a considerable difference in which towers can continue to grow and which crumble and fall. Similarly, your process is a part of your change rate failure and should accommodate the height of your tower and then some, but not too far beyond the desired height you're trying to reach. Process blocks should be "long-term ephemeral" (is that a thing? I think it can be) because they can't move quickly.

Fighting gravity

That production change that took down service wasn't singularly the cause, but it was the catalyst. The mixture of changes that preceded an outage was already fighting gravity; the latest change made is the one that just took the blame. The order of the changes we made, where we put our blocks (and where we didn't), and maybe someone barely hit the table as they got up to grab another beer gave gravity its inevitable win.

But guess what? The tumbling tower of production is precisely what needed to happen to highlight where your weak spots were. Eventually, you get good enough at playing Jenga through constant trial and error to play like this.

Assembly time

On-Call pages, customer support emails, and manual discovery are all unique ways we discover fires. In real life, we call emergency services (ie: 911) when we know something is wrong. But even 911 itself was an improvement on a convoluted system for getting help quickly. Residents had to know the local number for police and fire emergency services up until 1968, which is not very helpful when you’re in a different city visiting and suddenly need help.

A standardized way of dispatching emergency services was a critical step to have full-cycle incident management for cities, especially ones operating at enormous scale like New York City. Since 2013, NYC has been publishing their response times to 911 calls. This data is a treat in so many ways, but it has a notable metric missing: resolution.

Instead, NYC focuses on the part that is within the realm of their control: How fast they can route a 911 call to the right team suited for the job, and how long it takes for that team to travel to where the incident is occurring.

Assigning neighborhoods

New York City, and every city for the matter, can quickly respond to incidents because of the careful organization and planning that has gone into which departments respond to which incidents, and which of those stations get dispatched to emergencies when a 911 call is received. When it comes to software, it’s all too common to see teams haphazardly assemble before the first hypothesis on what is going wrong is formed.

Software teams can mimic the organization of cities by assigning which teams own which “neighborhoods” of their stack. We can think of a software neighborhood as a functionality, a service, or even an entire environment. The important thing is to remember that these neighborhoods have assigned teams when an incident is being dispatched, much like a real fire department in Manhattan.

The best people to assign neighborhoods for incident management are the locals themselves. People who build the software that will inevitably have an incident (big or small), are the sane default to assign when you smell smoke. Service ownership expands the entire software development lifecycle, including incidents.

Travel time

When it comes to incidents, reaction and travel time matters. Smokejumpers, for example, are trained to be airborne and enroute to a wildfire in a matter of minutes. When it comes to aircraft safety, getting out of the plane in less than 45 seconds best guarantees survival during an accident. Airplane safety cards, as it turns out, actually work.

So when it comes to software, what is the “safety card” procedure your teams can use? Oftentimes we call these runbooks or playbooks, but the goal is the same: getting to the point where you can mitigate an incident faster.

Greasing the mental wheels of an incident response team can dramatically decrease the amount of time an incident takes to be acted on, which overall decreases the amount of time that individual incident is impacting others.

Response time matters

Every phase of an incident has things that can be improved, and response time has one of the highest returns on investment when reducing the overall impact of an incident. So assign your neighrborhoods, standardize your declaration, and get to the fire faster.

TL;DR: I'm asking people to go from calling me Bobby to Robert.

For the last 10 or so years, I've maintained a nickname for myself that I didn't quite choose. Xkcd, a popular nerd comic, released a 4-panel cartoon depicting a mother who receives a call from her son's school. The school mentions that her child's name, Robert'); DROP TABLE STUDENTS; --, has caused a database to be erased. Knowing full well what her son's name does to databases, the mom snidely responds, "And I hope you've learned to sanitize your database inputs."

Being a nerd named Robert, this comic naturally found itself in my life. Folks started calling me Bobby Tables, and being the person I am, I accepted it. I even introduced myself as Bobby for several years. At this point, nearly my entire professional network knows me as Bobby. I had matched my social media usernames to the new nickname of Bobby Tables and even my personal domain that you're reading this on (however, I'm looking for an alternative).

However, “Bobby” never really resonated with me. I never felt like a Bobby. I enjoyed the joke; what nerd wouldn't? But I always felt a lot different, even relieved, when people referred to me as Robert, especially the last year. I even began to believe that my late mother would be absolutely livid if she knew I went by Bobby. After all, she named me after her best friend and professional ice skater, Robert Wagenhoffer. After visiting my family for this last Christmas (a rarity) and was only called Robert, I knew I needed to make an oddly difficult decision public: I'd prefer to be named Robert by everyone.

Believe me, I don't relish asking everyone to suddenly change how I'd like to be addressed. I feel as if I'm asking hundreds of people to change a habit that only benefits me. I recognize those of you who have only known me as Bobby will almost inevitably call me that accidentally for possibly the rest of our lives. My only ask is that you try, and I promise to never get upset. I'm well aware it will be a challenge, especially for those of you who are particularly close to me.

Thanks for reading,

Robert Ross

P.S. I'll likely keep the online moniker of Bobby Tables; it's too fun to have that username for everything (GitHub, Twitter, LinkedIn, Instagram, etc.) at this point.

Around three years ago, I was a Staff Software Engineer at Namely, a New York SaaS for HR, Payroll, and Benefits. Today, I sit as the CEO for a 70+ person SaaS company called FireHydrant, a Series B reliability platform. I transitioned from head bobbing to music while writing code to writing emails and enthusiastically communicating our vision. I have a consistent desire to teach others all I can, and with this blog post, I want to give some key lessons I've learned in what has felt like a blink of an eye. There are three, and I've broken them up by year.

Year 1 - If the shoe fits, take it off

Our first purchase was a pull-out couch for my friend and co-founder Dylan to sleep on in my tiny one-bedroom Greenwich Village, NYC apartment in our first year. For the next several months, our life was to wake up, maybe go to the gym together, and write code. Daniel, the third co-founder, worked hard to build our infrastructure on Google Cloud, CI pipelines, and monitoring. All three of us have engineering minds and souls. A large chunk of FireHydrant that exists today was built in those long days between December 2018 and April 2019.

We had a rather substantial surface area in our product by April 2019. Our product enabled excellent incident management in Slack, simple retrospectives, and even integrated with Jira. In addition, it came with single sign on, credit card payments, a service catalog, change event tracking, and a few other nice integrations. We built a lot of great features in short order.

I'm sure you may assume that the lesson I will give here is "follow the lean startup method and ship tiny things fast!" Ship small and fast is undoubtedly sound generic advice. Instead, the lesson I learned is that it is wildly uncomfortable to put on another hat and change lanes in the business when you're very good at writing software. It's the same reason kids love watching the same movie over and over again, why we become regulars at local pubs, and why a Big Mac still hits the spot. We may not fear change, but all of us almost certainly love normalcy. I had been writing software for over 15 years before starting FireHydrant, so naturally, it was my norm. In many ways, it was also my identity.

I learned that you need to shake off your specialized skills the moment you've delivered what could be considered valuable. I've come to believe that people aren't avoiding shipping small things and iterating but more that they're just great at a particular skill. They forgo attempting to sell the great something they've painstakingly built because it's too damn fun to have carte blanche as a founder on your product.

Shifting your identity is not easy. FireHydrant was fortunate in that we had a great group of friends, investors, and former colleagues encouraging us to sell our damn product. I'm incredibly thankful for those who helped me shed my skin as a developer (although not entirely) and emerge as a young CEO with a lot to learn.

Year 2 - If you build it, they probably won't come

I'm a firm believer that developers are different when it comes to purchasing, frankly, anything. We viciously debate the best coffee, whiskey, keyboards, editors, languages, and then back to whiskey again (for good measure). In developer culture, it's common to research products and services in excruciating detail before concluding to purchase. Minor flaws, such as a dead link on the website, can convince a developer that the tool is not worth opening their wallet. I don't intend these observations to come across as a dig on developer culture; I'm guilty of many of them myself. I intend to articulate a simple truth: developers are picky.

I learned a harsh lesson that I need to remind myself daily in our second year: don't measure others with your ruler. I was angling FireHydrant to be the tool I wanted to buy from day one. It's the reason we supported credit card billing right at launch and had no sales team. I had a firm belief that if we built a fantastic product, developers would come and be as excited to use it as I was. I anticipated that others bought software the same way I did: I'll swiftly purchase it and implement it myself if I like it.

Indeed, not having a sales team earlier was a mistake. Since then, we have assembled an absolute world-class sales organization, and they impress me every day. There are several growth models that any startup can employ. Still, the reality is that I was measuring the market with my ruler, which prevented me from seeing a better alternative to growing our business: kick-ass sales. We needed a team of people whose sole job was to think about obtaining currency.

I encourage you to think about what assumptions you're making about your market and product based on your ruler. You might believe that everyone knows they even have a problem, to begin with, simply because you struggled with something so much. Maybe you assume people won't need customer success because you've always been able to figure out your problems. The subtly's of our assumptions can and will creep into our business, and I've learned to be mindful of my own. But be sure they're always an advantage and not a detriment, as our experiences are the crux behind us starting companies in the first place.

Year 3 - There's no such thing as a high performing family

I dislike when companies try to build a foundation for their teams on the idea of "family." As a CEO, my role is to chase our company vision with a great product and business and do it all with a great team. I've rarely encountered a family that I'd consider "high performing." On the other hand, teams should be aiming for high performance anytime they come together. You're a member of your family day in and day out, and there's no selection in most cases.

On the other hand, teams are selected and cultivated by both sides. Team members want to join, and you want them to join, and it's a mutually agreed-upon arrangement.

FireHydrant has, by definition, employees (73 at the time of writing). However, I seldom refer to people that work here as employees. Instead, I prefer to call everyone teammates because that's who they are. We wear the same jersey, play the same game together, and are all in this together no matter what department you are in at FireHydrant. We win together, and at times, we lose together.

The lesson here was an accidental discovery. I've worked for companies that attempted to create a family, and in almost every scenario, the "family" was dysfunctional. How do you give your family feedback? How do you hold a family member accountable? Societal norms prevent us from these necessary tactics when we operate as a family.

It would be best to build an organization that operates as a team. Building the company with the strict concept of "we are a team" will also be a forcing function for work/life balance. I have yet to see disadvantages to operating as a team at FireHydrant or other organizations in the past.

However, one other thought I'd like to offer is that sometimes members on that team become someone you'd consider family. FireHydrant has several members on its team that I believe are my closest friends and family. I've been to their weddings, traveled the world with them, and been there for challenging moments.

It's extremely fun to be a team member of a high-performing team that is continuously improving; I encourage you to build your organization with this in mind.

Recap

• Pay special attention to when you need to change lanes and hats because your previous identity will get in the way.

• Your ruler doesn't measure the same way when compared to others. Figure out others rulers and measure with those instead.

• Build a team, not a family. Families can't be high-performing. You all wear the same jersey.

What's next?

I've learned countless lessons over the last few years as the CEO of FireHydrant. But these three stand out as timeless ones that I can take anywhere and refer to anytime.

As a CEO, I'm interviewing for my job every day, and I expect the following chapters will be no different.

1. Amazon Web Services / Google Cloud / DigitalOcean

2. Kubernetes / Nomad

3. Prometheus / DataDog / SignalFX

4. ElasticSearch + Kibana / LogDNA / Splunk

5. FireHydrant (right?)

You might look at this list and go, "that's a pretty typical stack nowadays," and you'd be right. But all of these technologies also support labels.

What is a label?

A label is a method of organizing something. Labels are used to attach small pieces of information to an asset. They're effective ways of denoting what something is, its expiration date or manufactured it. Labels are everywhere, from a banana in the grocery store to your Kubernetes deployment running in AWS.

Software is badly labeled.

We've all collectively (whether we realize it) have agreed on one label: name—the name of our repositories, Kubernetes deployments, AWS account names, domains. We're really good at giving things a name consistently. Names are an easy way to identify something; they always will be.

However, the name label is no longer adequate for the complexities of our jobs as engineers. We need to know what revision a process is currently running. We need to discover every memory metric for all of our Go applications. We need to know which processes are a part of our Kafka pipeline. Sadly, the lowly name label cannot deliver on these needs.

The power of keys and values

It is the absolute wild west out there when it comes to labeling software systems. For example, no one has agreed on the same term in the first place. AWS and DigitalOcean use "tags" on their assets. Kubernetes and Prometheus have settled on "labels." Fundamentally, labels and tags aren't very different. A label is a key/value; a tag is the application of that label to something. 

A name sticker at a speed dating event is the label, and placing it on your nicely pressed shirt makes it a tag. You look great.

Having a key with a value is simple but powerful. When you introduce the concept of a key that has a value, it means you can search on the value's intent, not just "hey do you have this value?". It aligns the value of the label to something tangible. It's why we see "Hi, my name is" on those stickers for awkward social events.

Keywords, on the other hand, are unstructured words attached to an asset. When read by a person, keywords have implied keys to humans, but to computers, they're utterly meaningless. For example, if I keyword a DigitalOcean droplet "rails@5.2.1", I can read that on a screen and go, "oh, this is probably a rails application." But ask a database to give you a list of rails, and now it has to search every keyword and effectively guess what you mean.

Standard labels will set us free.

We have far too many things operating in our software stacks to not use standard labels on them anymore. Kubernetes has recommended labels, but that's Kubernetes. We need to follow the FDA's example and create a labeling standard that everything uses. Using that list above, we'd be empowered to find all related assets in our system using a standardized labeling scheme. We need to define standard labels for assets that can be applied to all layers of the stack. My proposal: A standard called OpenAsset.io

The labels

This standard does not need 100 label definitions to be successful. It needs to cover the most extensive use case while also not tiring out hands. If a labeling standard has too many options, it becomes less and less valuable. Automation tools become harder to maintain, and engineers will simply omit labels because it becomes redundant typing them (admit it, you've done this).

Format

All open asset label keys start with "openasset.io" and must be a valid URL. A valid URL ensures other parties (vendors, internal tools) can parse keys quickly. Every language has a URL parser. By using a domain/path format, it makes it easy to create a statement that recognizes the host "openasset.io"

Specified keys and their purpose

openasset.io/name: "laddertruck"

The most straightforward label of all, what is the name of this asset?

openasset.io/language: "ruby"

What programming language is this asset/application written in?

openasset.io/language-version: "3.0.0"

What is the language of the version of the programming language used?

openasset.io/framework: "rails"

If this asset uses a framework, which framework is it?

openasset.io/framework-version: "5.2.1"

What is the version of the framework this asset is using?

openasset.io/component: "web"

What component of your stack is this asset a part of?

openasset.io/deployed-by: "weave"

What deploys this asset to an environment?

openasset.io/revision: "3c12d41301a7eca481c8eda0564d79a935bafd27"

What is the revision for this asset? This can be a git commit, semver, etc.

openasset.io/tier: “4”

What service tier does this asset have?

An example with Kubernetes

Kubernetes is probably the best example of how this standard could be applied. The API allows filtering by key presence and value set.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pubsub
  namespace: laddertruck
  labels:
    openasset.io/name: "laddertruck"
    openasset.io/language: "ruby"
    openasset.io/language-version: "3.0.0"
    openasset.io/framework: "rails"
    openasset.io/framework-version: "5.2.1"
    openasset.io/component: "laddertruck"
    openasset.io/deployed-by: "weave"
    openasset.io/tier: "4"
    openasset.io/revision: "3c12d41301a7eca481c8eda0564d79a935bafd27"

My vision for this labeling standard is knowing I can take the labels above and quickly search in my logs for openasset.io/revision: "3c12d41301a7eca481c8eda0564d79a935bafd27" and instantly see all records for that revision. This becomes especially powerful when combined with your infrastructure provider. If I have a database hosted on AWS RDS powering Laddertruck web, I can tag that database in AWS with openasset.io/component: "laddertruck".

What does this unlock?

Discoverability

The most apparent advancement teams gain by utilizing a standardized labeling scheme is discoverability. Logs, metrics, deploys, etc, are all easily found since there’s no variance in the labels used. Knowing what assets are running and what they do helps breaks down village knowledge barriers.

Billing insights

When you tag your infrastructure with standard labels, it makes billing insights and management easier to understand and maintain. Most infrastructure providers (such as GCP and AWS) allow getting insights on asset spend filtered by labels. 

Access management

Using an open labeling standard allows building tools that enforce rules about access such as SSH, resource creation, etc.

Incident management

By labeling assets with the keys/values proposed, the service's incident management process becomes even more flexible. Defining tiers, components, and revisions empowers responding engineers to have more context about the degraded service.

Why do I care

I've built several internal tools in my career. I worked on the internal inventory management at DigitalOcean (named Atlantis) for a bit. I helped build service deployment and discovery at Namely, and now I'm making FireHydrant which offers service catalogs. After years of watching CNCF technologies explode, everyone moving to the cloud, and orchestration take over like Kubernetes, it has become painfully evident that our industry needs this standard. We currently live in an unorganized hell of infrastructure and service. It's time we get organized.

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!

Read The Room

Over the years one thing that has always helped me inch forward in my success and life was the ability to “read the room”. Looking someone in the eye as you shake their hand is customary, but reading their cheek bones, clothes, tone of “nice to meet you”, I attribute that skill to you.

Time and time again I’ve had people tell me how much of a joy you were to have at parties, events, work, hell I even thought my friends liked you more than me at times. But it was the little things you did that made that happen. In the last 10 years I’ve realized a few bullet points that have never failed me based on observing how you interacted with others.

• Inflect upward when first meeting someone. A monotone “hello” as you exchange a handshake can give a bad first impression, whereas “Hi nice to meet you Matt” with an ending on a tone higher than where you started will often times leave a lasting impression.

• A name is one of the most important things a person has, and remembering it and using it early and often will create a stronger relationship. Don’t pretend you heard their name in that crowded bar, ask for it again and repeat it back.

• Introduce your company. If you’ve brought a guest to an event and you’re saying hello to your previous acquaintances, before diving into any conversation make sure your guests have been introduced. There’s nothing quite worst sitting idly aside as your friend talks to people you don’t know and in a place you’ve never been.

• Just be yourself. Overtime, people tend to interact with the same types of people after many trial and errors. If being yourself means you’re incompatible with that type of person, it’s merely a data point and that you can move on to others. This can extend as far as a mismatch in values or even their jokes just aren’t that funny. Being yourself is the only way to really find your clique yet people love to go the other way and try to change who they are inevitably shackling their personality to peoples expectations.

These bullet points came from several situations where I saw you meet new people. Whether it was for work, meeting my friends parents, or just a random conversation at the grocery store, you had a way of making people feel better after they met you.

Do Things For Yourself by Yourself

One thing that has actually taken a long time for me to truly realize it’s value has been doing things for yourself, by yourself. Having friends that you can spend countless hours with and not hate each other is great, however, the person you spend your whole life with is yourself. It stands to reason you should do something with yourself too. I realize in hindsight that you did this fairly constantly. While I didn’t approve of how much sometimes as your son, they’re valuable to recognize and practice.

One of the most prominent memories in my mind of “you doing you” was camping in Kennedy Meadows. Every summer you’d drive us to the Sierra Nevada’s for at least a week to fish, camp, and sit by the fire playing Uno with grandma. Without fail, you’d leave us and go fish the Stanislaus river alone, beers in stored in your fishing vest (which I still have and wear when I fish now). I think you just enjoyed the catharsis brought out by the sound of the river, a tight line, and your beer. It’s taken me years to understand the value of doing something by yourself, but thinking back to you walking back with a smile on your face and trout dangling from your fathers stringer, that memory alone makes me remember to do it more often.

I used to think doing something for yourself had to be big and expensive. A trip to Europe, getting a massage at a spa, or even buying a new television. What I have only recently realized though, is reading on the couch with a glass of water and a candle that crackles as it burns is just as great.

Practice Being Creative

You loved Christmas. Every year we’d drive Garrison street in San Diego to look at the other people’s homes that also get very into Christmas decorations. The Christmas spirit also kicked you into your creativity gear for a month.

I remember complaining about having to go to Michael’s (the arts & crafts store) to shop for the tiny little ceramic buildings that were plain white. We’d traverse the aisles looking for paints, hot glue gun supplies, and homes for ants. After the dreadful trip you’d take hours to paint the ceramic homes with whatever you felt made the most sense. You did this for a dozen of pieces at least every year. We’d attach Christmas tree holders and drape them from our Noble fir we had gotten for the year or arrange them on fake snow in the living room.

I only bring this up because it ties into something that makes me who I am. I love being creative. Whether its playing my Trumpet, making a website, or finding a unique way out in a tight situation in Halo, I’ve always loved being creative. The simple acts of creativity that you involved me in during your life led me to be this way. If you didn’t push me to be a musician, I wouldn’t have done Drum Corps, which is, by and large, why I am successful.

Mental Health Can Ruin Your Life

I wish you were alive for me to tell you this, but its one of the reasons passed away in the first place in my mind. But stress and anxiety will ruin your life. It will debilitate you more than anyone can properly explain with words. But, at the same time, I learned this from you.

Our last few years together were anything but golden arches mom. I think we can both agree on that. We fought, yelled, slammed doors, ran away, called relatives complaining… you name it, we probably did it.

A lot of this stemmed from a lack of money, which stemmed from not working. A lot of it also stemmed from me having bad grades. A lot of it might be because of booze. While speculating what exactly caused it 10 years later benefits no one, the truth of the matter was, I saw the affects of you being stressed, depressed, and anxious until your last day.

It became obvious I knew what was going on with you years ago when I started to feel the same way. I wanted to sleep anytime of the day for hours. I drank a lot. I would have outbursts at games I used to love to play. I’d get frustrated at petty things. My head would hurt for no reason. I’d cry out of nowhere by myself.

Having seen the affects of these in the longer term first hand, I’ve made it a first priority to always fix them first. It took a rough start to 2017 to make this a real priority. However, I can confidently say I have the tools and resources necessary to combat my own brain when it decides to throw a tantrum. This is only because I unfortunately saw you struggle with it but not know how to fix it, or the willingness sometimes. I wish I could tell you what I know now, when you were going through it too.

Farewell

A decade ago, I said my final farewell to you. It was the most painful thing I’ve ever endured. But I’m constantly thanking you to this day for the things you taught me without knowing it. You’ll never know the lasting effects of me playing an instrument, searching “how to make a website”, or reading a room as I walk in. All of which stemmed from the wonderful times we had together.

Thank you, Mom
Sincerely, Robert