How to keep passion from hugging your systems to death
(Note: When talking about 'Ops engineers' here, I'm referring to anyone skilled and responsible for doing the ops/systems/devops/whatever regardless of their actual title.)
There are 2 types of Ops engineers:
- Those that want to do MORE Ops
- Those that want to do LESS Ops
The Passionate: Give me MORE Ops!
Passion is great. It's why many of us got into Ops in the first place.
It's what pushes the bleeding edge forward and is a large part of why we can enjoy so many wonderful things right now in Ops.
Think of all your most favorite magical tools you use in Ops life — nearly all of them were started and driven forward with passion that came from some mystical galactic dolphin-nebula.
At research labs, universities, and large infrastructure companies that need to push the bleeding edge of Ops, these are exactly the type of engineers you want.
But at almost every other typical company, they can be a huge liability if you don't give them an outlet for their passion.
The Dispassionate: We're just passionate about something else
Engineers that aren't passionate about Ops don't lack passion. They're just passionate about something else. They want to get Ops over with so they can get back to their main goal.
The Ops needed to get done and the burden fell on them. Ops are their chore, so they just do what is needed and no more so that they can get back to app development, growing their company, or whatever thing they're actually passionate about.
At the vast majority of normal web/mobile business these are the Ops engineers you want.
At research labs, universities, large infrastructure companies that need to push the bleeding edge of Ops, these folks don't do so good.
Big Imbalance in the Market
There seems to be way more of the passionate-Ops-engineers than the dispassionate-Ops-engineers.
It's unfortunate because I'd guess there are 100 times more jobs available that would be better served by the dispassionate-Ops-engineers.
NEWSFLASH: Basic Web Ops are a solved problem!
This is great news for businesses. But bad news for many passionate Ops engineers that want to push the envelope.
There just isn't much need for bleeding-edge innovation at the vast majority of companies.
The fundamentals of Web Ops are largely unchanged from 20 years ago. In fact things have only gotten much easier and simpler.
(Sidenote: You would get the opposite impression if you listen to the marketing hype coming from those who benefit from keeping your Ops dependent on their solutions — more on this in a following post).
Is passion-in-the-wrong-place really a problem?
I've reviewed hundreds of systems over the years. Most of the worst messes I've seen were caused by the passionate Ops engineers.
They would get fascinated with bleeding edge toys they desperately wanted to learn. So they added them to their production systems. After a while, their production systems become a complex nightmare of abandoned toys.
Each new toy held an obsessive place in their heart, well... until it didn't. Then they moved onto the next toy.
You can tell a passionate Ops engineer has had their way with your systems when you can look at the system pieces and see the history of Ops fads over the last few years. "There's that one fad from 2012." It's like rings on a tree —you can even figure out the age of the system based on the once-trendy-but-totally-unnecessary tools buried in there.
It may seem harmless to have an over-engineered systems mess, but it's an order of magnitude (or 2!) more expensive to manage and many times slower than a simple system built on fundamentals.
It may not kill a company, but it's certainly a huge drag on it.
Solution 1
Hire dispassionate Ops engineers that base decisions on fundamentals, not fads.
This is the ideal solution, but it's not foolproof. Many of us fluctuate between being passionate and being dispassionate about Ops. Even the most level-headed engineer can sometimes be seduced by a new toy.
Solution 2:
Protect production with a playground and playtime.
It may seem counterintuitive, but to achieve this, the key is to provide a safe outlet for the passion of your engineers to run free. Essentially, you must provide a dedicated playground and playtime for your engineers.
What?! A playground for my engineers?! Are you insane?!
Boys will be boys. Girls will be girls. Ops engineers will be Ops engineers.
If you don't give passionate engineers a safe outlet to learn, grow, experiment, and be challenged, trust me —they will find a way to do these things on your production systems.
Remember, if they are using your production systems as a playground, your production systems will turn into a collection of abandoned toys over time. And some of the most fun toys are the most experimental and dangerous. These abandoned toys require constant maintenance (security patches, upgrades, backups, recovery, etc.) and are all potential points of failure.
These poor, abused systems are a real pain to manage and just get worse and worse over time. Systems development gets slow. There are frequent emergencies. Embarrassing security breaches happen. Top engineers leave. Hiring replacement engineers who are willing to inherit the mess gets more and more difficult.
A bit of dedicated play time is WAAAAAAY cheaper
An Ops playground might seem like an unnecessary waste if you're not looking at the whole picture. But it is many times cheaper than the alternative. With a dedicated completely-separate-from-production playground, your engineers can "get their wiggles out" after they've ensured that the production systems are simple, secure, and reliable.
I regularly see companies overspending on their systems by 5-50 times. That's a crazy amount of "room for improvement" ;-)
You can avoid that fate with something as simple as 20% time for your engineers. By giving them 20% of their work time to play in a safe environment, you are investing a very small 20% in order to avoid a 5000% wasteful nightmare later on.
(Note: Firmness required)
Even with a playground, passionate engineers will often fight to put the toys into production —you must resist this! Unless there is a really compelling reason and the fundamentals back it up strongly, don't do it! Just let them play with it and nearly always they'll eventually run out of steam and move on to a new toy.
Conclusion:
Aligning interests = Win Win Winning
- Your business cares about low-maintenance, low-cost, high-reliability, high-security systems.
- Your passionate engineers care about learning, growing, and pushing-the-envelope.
With a dedicated-completely-separate-playground-from-production, you're investing in the interests of both your company and your passionate engineers. Win win! Mom will be so proud :D
Last Updated: 10 May 2017