Posts straight from FireHydrant's Engineering Team
February 4th, 2021 Incident Retrospective
Between 2021-02-05 00:20 and 2021-02-05 02:44, FireHydrant experienced an incident resulting in delayed runbook execution steps (Slack channel creation, etc) and intermittent availability issues on app.firehydrant.io. This is our incident retrospective.
It's Time We Throw Out the Usage of 'Postmortem'
Why are we using the term 'postmortem' when no one died? In any other job, conducting a postmortem means someone perished, so we need to switch to another phrase to lessen the gruesomeness of software incidents. I wanted to provide some ideas that your organization could possibly run with as a replacement to “Postmortem.”
The Final Episode - Episode 10 of Throughput Thursdays
We made it to our final episode! Thank you to everyone that tuned in and watched Bobby get a Terraform provider up and running. We hope you enjoyed watching me through the good, bad, and ugly these past 20 or so hours.
Configuring a Runbook - Episode 9 of Throughput Thursdays
In episode 9 of Throughput Thursdays, we work to configure a Runbook and get it to work! Watch part 1 of our two-part finale below to see what happens.
Breaking down the interface - Episode 8 of Throughput Thursdays
In episode 8 of Throughput Thursdays, we break down all the logically grouped pieces into their own interfaces and create an interface on our client that can return.
More New Terraform Resources - Episode 7 of Throughput Thursdays
In episode 7, we create resources for managing teams and severities through the Terraform provider, which means we now can now manage more of users’ FireHydrant configurations with code.
Creating a Data Source - Episode 6 of Throughput Thursdays
In Episode 6, we update our Terraform resource for FireHydrant functionalities and create a data source for FireHydrant services. This allows us to pull services from a list and link them to functionalities. Linking resources like this lets us do a lot of cool things with Terraform.
Moving from Redux Thunk to Redux-Saga: A walk-through
At FireHydrant, we recently began to replace our usage of thunks with Sagas to handle our data fetching. Read how we moved from Redux Thunk to Redux-Saga.
Incident Ready: How to Chaos Engineer Your Incident Response Process
We’re pretty sure using a real incident to test a new response process is not the best idea. So, how do you test your process ahead of time? Learn how to use chaos engineering principles to stress test your incident management process.
Testing Our Terraform Resources - Episode 5 of Throughput Thursdays
In this episode of Throughput Thursdays, we test our Terraform resources. If you missed it, you can watch it here.
How to: Automatically Archive Incident Slack Channels using conditions in FireHydrant Runbooks
FireHydrant’s Slack integration is a great way to speed up your incident response, especially if FireHydrant Runbooks is automatically creating channels in your Slack workspace for each incident.
Adding Two Terraform Resources - Episode 4 of Throughput Thursdays
In episode 4, we were able to achieve creating two full-blown Terraform resources for FireHydrant environments and functionalities. While simple resources, they unlock a lot of power that did not exist previously for teams that want to document their infrastructure using Terraform.
Fixing Some Code Sins - Episode 3 of Throughput Thursdays
In episode 3, we built a flexible API client for our Terraform provider that implements a really simple interface. We also wrote some simple but effective tests and replaced the original cruft in the provider code with our new API client.
Live from Cape Cod - Episode 2 of Throughput Thursdays
In Episode 2, Bobby is live in Cape Cod, sitting on a dock about 4 inches from the edge of a lake. Last week we built a skeleton of a Terraform provider. Now we’ll get the provider to create and delete resources, like services in FireHydrant.
7 Ways to Get Acquainted With a New Codebase
Tori Crawford, one of our engineers, walks through some ways that you can get immersed in unfamiliar code. She gathered input and insights from the rest of the FireHydrant team to create this quick playbook on best practices that will make tackling any new codebase easier.
We’re Building a Terraform Provider! - Episode 1 of Throughput Thursdays
In Episode 1, we started out the Terraform provider with a simple data resource against the FireHydrant API. We were able to successfully retrieve information about a single service and display its name in our terminal!
How FireHydrant's CI/CD Infrastructure Fixes Bugs Faster
Almost everyone knows that working with third-party APIs can be challenging. Sometimes the errors happen unexpectedly. Sometimes the error information that you receive is inaccurate. While most people feel these pains acutely, I’d like to share how we answer these challenges at FireHydrant and how it’s helped us avoid headaches and stress.
Sticking to Your SLAs with FireHydrant Runbooks
In today’s world, systems are increasingly becoming more and more complex. Due to this complexity, it’s no longer a matter of “if” our systems will fail but “when”. To manage expectations for when our systems do fail, we can look no further than our Service Level Agreement.
Grow your Blame-Free Culture with These Postmortem Best Practices
Here are 3 postmortem practices that embrace a blame-free culture.
A Single Person On-Call “Rotation” is a Critical Vulnerability
Why distributing your on-call workload is critical.
NFS with Docker on macOS Catalina
You like living on the edge, life is fun on the edge until the edge is a macOS major update. Then you use vibrantly colorful words, some that your dead ancestors heard, all because your development environment now doesn’t work in spectacular fashion.
Open Source can be a Silver Bullet, but your Application Might be a Werewolf
A story about open source.
3 Defensive Programming Techniques for Rails
Incidents happen all the time because of bad code deploys. Defensive programming is great for codifying how a bug could be introduced, and raising an error right before it would happen, or choosing an alternative path. Here are some simple ideas to defend yourself against simple mistakes.
Instrumenting Ruby on Rails with Prometheus
If you’re running a production application, you need metrics. In the Rails community, this is commonly achieved by using NewRelic and Skylight. But for some of us, we achieve visibility by using Prometheus and Grafana that we build and host ourselves. Check out this guide on how to use Rails with Prometheus.
Developing a Ruby on Rails app with Docker Compose
Learn how to set up a new Rails app in Docker compose.
Flexible Ruby on Rails Reader Objects
Rails and ActiveRecord are great at providing a simple interface for retrieving information from a database. With a few simple characters, I can retrieve all of my users with User.all. While this simplicity is great, it breaks down when you want to start doing more advanced queries such as paginated results, filtered records, etc.