The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win
Really interesting concept to structure this as a novel. The authors paint a very persuasive vision of the future and now have me thinking about how my colleagues and I can get to where the characters in the novel did.
-
Damien Ryan
I’m late, so I’m driving ten miles per hour over the speed limit, instead of my usual five.
Each year, it gets harder. We have to do more with less, to simultaneously maintain competitiveness and reduce costs.
CIO stands for “Career Is Over.”
As the saying goes, if your colleague tells you they’ve decided to quit, it was voluntary. But when someone else tells you they’ve decided to quit, it was mandatory.
Andrew Doran
This is so true.
“If I fail, I’ll try to make sure it’s in a new and novel way.”
“You know the saying, right? The way you can tell a vendor is lying is when their lips are moving.”
We need to establish an accurate timeline of relevant events. And so far, we’re basing everything on hearsay. That doesn’t work for solving crimes, and it definitely doesn’t work for solving outages.
Information Security is always flashing their badges at people and making urgent demands, regardless of the consequences to the rest of the organization, which is why we don’t invite them to many meetings.
They’re always coming up with a million reasons why anything we do will create a security hole that alien space-hackers will exploit to pillage our entire organization and steal all our code, intellectual property, credit card numbers, and pictures of our loved ones. These are potentially valid risks, but I often can’t connect the dots between their shrill, hysterical, and self-righteous demands and actually improving the defensibility of our environment.
In the twenty-two hours since I was promoted, 526 new e-mails have arrived in my inbox. Holy crap.
Something seems wrong in a world where half the e-mail messages sent are urgent. Can everything really be that important?
In the past, they’ve just pointed to a network folder and said, ‘Deploy that.’ There are newborn babies dropped off at church doorsteps with more operating instructions than what they’re giving us.
Brent needs to work with developers to fix issues at the source so we can stop fighting fires. But Brent can’t attend, because he’s too busy fighting fires.
knowing is always better than not knowing.
Andrew Doran
Is it?
JS Ng
Not knowing is better than knowing wrong.
Damien Ryan
As far as DevOps is concerned, yes, yes, and thrice yes. Nothing is worse than gearing up for a deployment and having issues that could have easily have been resolved by moving the switchover date.
“a ‘change’ is any activity that is physical, logical, or virtual to applications, databases, operating systems, networks, or hardware that could impact services being delivered.”
you can’t achieve the strategic until you’ve mastered the tactical,
WIP is the silent killer.
you don’t release work based on the availability of the first station. Instead, it should be based on the tempo of how quickly the bottleneck resource can consume the work.
Eliyahu M. Goldratt, who created the Theory of Constraints, showed us how any improvements made anywhere besides the bottleneck are an illusion.
Your job as VP of IT Operations is to ensure the fast, predictable, and uninterrupted flow of planned work that delivers value to the business while minimizing the impact and disruption of unplanned work, so you can provide stable, predictable, and secure IT service.
ensure that your most constrained resources are doing only the work that serves the goal of the entire system, not just one silo.
The First Way helps us understand how to create fast flow of work as it moves from Development into IT Operations, because that’s what’s between the business and the customer. The Second Way shows us how to shorten and amplify feedback loops, so we can fix quality at the source and avoid rework. And the Third Way shows us how to create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery.
The First Way helps us understand how to create fast flow of work as it moves from Development into IT Operations, because that’s what’s between the business and the customer. The Second Way shows us how to shorten and amplify feedback loops, so we can fix quality at the source and avoid rework. And the Third Way shows us how to create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery.
For these types of changes, I’ll have my staff generate some reports on the changes’ success rates and any associated downtime. This will help the business make more informed decisions around the changes.
Holy crap.
Andrew Doran
Seems to be the favourite phrase of the book!
I’m starting to associate the smell of pizza with the futility of a death march.
I smirk at the reference to smoke tests, a term circuit designers use. The saying goes, “If you turn the circuit board on and no smoke comes out, it’ll probably work.”
Andrew Doran
I’ve also heard the term ‘sniff test’, although I have no idea where that comes from.
Nothing worries auditors more than direct edits of data without audit trails and proper controls.
But problems, like dog poop left in the rain, rarely get better just by ignoring them.
any improvement not made at the constraint is just an illusion
Remember, unplanned work kills your ability to do planned work, so you must always do whatever it takes to eradicate it.
reliance on Brent
spending all his cycles on features, instead of stability, security, scalability, manageability, operability, continuity, and all those other beautiful ’itties.
Being able to take needless work out of the system is more important than being able to put more work into the system.
I’m sorry you feel that way, Bill, but the drawers open on my side of the desk.
Erik described the relationship between a CEO and a CIO as a dysfunctional marriage. That both sides feel powerless and held hostage by the other.
A great team doesn’t mean that they had the smartest people. What made those teams great is that everyone trusted one another.
in order to have mutual trust, you need to be vulnerable.
Oh, shit. That’s another part of “management off-sites” I forgot about. Touchy-feely crap.
You’ve just described ‘technical debt’ that is not being paid down. It comes from taking shortcuts, which may make sense in the short-term. But like financial debt, the compounding interest costs grow over time. If an organization doesn’t pay down its technical debt, every calorie in the organization can be spent just paying interest, in the form of unplanned work.
“Priority 1 is whoever is yelling the loudest, with the tie-breaker being who can escalate to the most senior executive. Except when they’re more subtle. I’ve seen a bunch of my staff always prioritizing a certain manager’s requests, because he takes them out to lunch once a month.”
every work center is made up of four things: the machine, the man, the method, and the measures.
“What you’re building is the bill of materials for all the work that you do in IT Operations. But instead of a list of parts and subassemblies, like moldings, screws, and casters, you’re cataloging all the prerequisites of what you need before you can complete the work—
“Properly elevating preventive work is at the heart of programs like Total Productive Maintenance, which has been embraced by the Lean Community. TPM insists that we do whatever it takes to assure machine availability by elevating maintenance. As one of my senseis would say, ‘Improving daily work is even more important than doing daily work.’ The Third Way is all about ensuring that we’re continually putting tension into the system, so that we’re continually reinforcing habits and improving something. Resilience engineering tells us that we should routinely inject faults into the system, doing them frequently, to make them less painful. “Mike Rother says that it almost doesn’t matter what you improve, as long as you’re improving something. Why? Because if you are not improving, entropy guarantees that you are actually getting worse, which ensures that there is no path to zero errors, zero work-related accidents, and zero loss.”
Opposite them are the external auditors who are legally liable for finding financial reporting errors and fraud, and yet they still want to keep us as clients.
This should be your guiding principle: you win when you protect the organization without putting meaningless work into the IT system. And you win even more when you can take meaningless work out of the IT system.
You’d be amazed at how fast work is getting completed, because we’re limiting the work in process.
How does one objectively decide whether “consolidating and upgrading e-mail server” is more or less important than “upgrading thirty-five instances of SQL databases”?
Improving something anywhere not at the constraint is an illusion.
Creating and prioritizing work inside a department is hard. Managing work among departments must be at least ten times more difficult.
We have met the enemy. And he is us.
To tell the truth is an act of love. To withhold the truth is an act of hate. Or worse, apathy.
Usually company managers like me only see our departmental objectives. This slide shows the bigger picture.
I believe with utter certainty that whatever we’re trying to figure out is critical to the First Way. He talked about the need to understand the true business context that IT resides in. I’m pretty sure no one has linked Dick’s top measurements to the prerequisite IT objectives.
Some of the wisest auditors say that there are only three internal control objectives: to gain assurance for reliability of financial reporting, compliance with laws and regulations, and efficiency and effectiveness of operations. That’s it.
‘COSO Cube.’
Our organizational key performance indicator (KPI) is on-time delivery. So to achieve it, you would create a new forward-looking KPI of, say, the percentage of vehicles that have had their required oil changes performed.
Preventive oil changes and vehicle maintenance policies are like preventive vendor patches and change management policies. By showing how IT risks jeopardize business performance measures, you can start making better business decisions.
on average, our R&D investments return more than ten percent. That’s the internal hurdle rate. If we don’t beat the hurdle rate, the company capital would have been better spent being invested in the stock market or gambled on racehorses.
The operational risks posed by IT need to be managed just like any other business risk. In other words, they’re not IT risks. They’re business risks.
It seems to me that if anyone is managing IT without talking about the Three Ways, they are managing IT on dangerously faulty assumptions.
He saw a presentation given by John Allspaw and his colleague Paul Hammond that flipped the world on its head. Allspaw and Hammond ran the IT Operations and Engineering groups at Flickr. Instead of fighting like cats and dogs, they talked about how they were working together to routinely do ten deploys a day!
“you need to create what Humble and Farley called a deployment pipeline. That’s your entire value stream from code check-in to production. That’s not an art. That’s production. You need to get everything in version control. Everything. Not just the code, but everything required to build the environment. Then you need to automate the entire environment creation process. You need a deployment pipeline where you can create test and production environments, and then deploy code into them, entirely on-demand. That’s how you reduce your setup times and eliminate errors, so you can finally match whatever rate of change Development sets the tempo at.”
Even one small mistake could take everything down. Surely this meant that we needed even more rigor and discipline and planning than in manufacturing.
For each of the boxes, she asks how long each of these operations typically takes then jots the number on top of the box. Next, she asks whether this step is typically where work has to wait then draws a triangle before the box, indicating work in process.
now all we have to do is check out the virtual machine that Brent and team built, and they’re all ready to go.
I need each and every one of my business managers to take calculated risks, without jeopardizing the entire enterprise.
“In ten years, I’m certain every COO worth their salt will have come from IT. Any COO who doesn’t intimately understand the IT systems that actually run the business is just an empty suit, relying on someone else to do their job.”
Lars systemic.de
i am not sure that would be a good idea... IT usually takes the systems approach, and sometimes resist thinking outside the box...
Story of my working life.