Why We Love (Forward) Failure

Yes, failure is inevitable. But ask yourself this: would you rather have your sleep interrupted by a 3 a.m. call saying your application isn’t working properly, or be proactive about weaknesses as a team? If you’d prefer your issues to surface during daylight hours, it’s time to embrace failure. In this fast-moving, devops world, we knew we’d need to think out-of-the-box to find solutions fast. We came up with an approach to performance hacking that involves applying devops concepts to organizational development, encouraging all departments to dive head-first into potential weak links to not only find failures fast, but to fail forward. And it applies to every level of the company, from management down to the individual employee.

We live for applying the “Chaos Monkey” approach to devops. Inspired by software built by Netflix to run on Amazon Web Services, chaos Monkeys are programmed to run through applications, randomly breaking things when they’re least likely to interrupt user experience to reveal what’s working and what isn’t. At ruxit, we inject our own form of intentional chaos, figuring out the trickiest or riskiest piece of the organizational puzzle and seeing how far we can take it before it breaks. But before any monkeys are let loose on the hunt for problem spots, we ask ourselves three things:

  1. Is the feature in question crucial to operations? If yes, it probably makes sense to release a monkey and test the feature’s strength. Proceed to number two.
  2. Do we have the resources in place to fix this feature if it breaks? If not, find a work-around before releasing a monkey. If yes, proceed to number three.
  3. The first two questions say “go ahead,” but will this feature be more critical in two weeks? If yes, put it on the backburner and repeat steps one through three.

By asking these questions, we’ve been able to make key changes to our process that should improve the ruxit experience for all parties involved. Just last week, a monkey showed us that onboarding of early signups wasn’t happening fast enough. Then, we tested how our platform handles an increasingly large load. Turned out, some emails were lacking a tracking code. Searching for failure can be nerve-wracking for sure, but it ultimately leads to a better product. And isn’t that why we’ve put in so much work in the first place?

Making the Chaos Less Chaotic

You’ve decided to release a monkey, but chasing failure—even for the sake of moving forward—is a frightening proposition. Our solution? Start small.

Getting comfortable with this process requires conducting everything on a negligible scale so issues are only visible to a small percentage of users. There’s no need to sacrifice experience for everyone when you can get a clear picture from a few. This approach allows teams to pinpoint a problem’s optimal solution instead of implementing a quick, temporary patch and going on with business as usual.

And starting small doesn’t just apply to technical work. A/B testing and partial software deployments aren’t the only way to fail safer and smarter. You can use the fail-forward approach for marketing strategy and idea generation, too. Here are some other ways starting small can lead to big progress (and a lot less shaking in your boots when it is time for the big stuff).

  1. Not yet ready for a major trade show presence? Scope out the local scene and treat it like a dress rehearsal for future industry events. If we notice our messaging isn’t reaching our target audience, we might use trade shows to test potential tweaks. Practice talking to anyone and gather as much feedback as possible.
  2. Tackle software problem spots one process at a time. Order from most- to least-likely to break, dive in, and only take on a new project when the first is completed.
  3. When the idea of weekly site updates sounds daunting, release one change at a time. That way, it’s easier to spot the source of any problems. Order updates from most to least critical before making any decisions. It’s a simple step that’ll save you endless amounts of time in the long run.
  4. Introduce big production changes to your most loyal customers first. The people you’ve already established a relationship with are most likely to give you honest feedback—without jumping ship.

Whether you’re acting on an individual or company level, embracing the potential for failure isn’t easy, but it’s one of the most beneficial things you can do. We’ve learned that, by starting small and building up your courage and industry know-how, it’s easier to track progress closely and eliminate any hiccups before they have a greater impact.

What are your best tips for getting over the fear of failing? Leave us a comment below, or send us a tweet.

Share this post:Share on FacebookTweet about this on TwitterGoogle+Share on LinkedIn

Stay in touch with us

New Here? Get Started with Our Best Features


Performance Hacking Posts of the Week

Alois Reitbauer

Is Your Distributed Team Making These Four Mistakes?

Alois Reitbauer

Five Steps to Giving Better Feedback