Change is not science, its anthropology

Several years ago, I was working with a company that had a global network that was driving us crazy.  Similar to some of the excuses we’ve seen  for companies that have had technical issues during the pandemic, we blamed external culprits for our delays and our outages:  volumes were spiking, oil was especially volatile, our new network partners in Asia were letting us down. Mostly, it was our own changes to the infrastructure that created problems;  we might make a data line change or add an exchange or a group of instruments, or add a new execution gateway or change the direction we were sending traffic.  It all added up to huge numbers of changes every week, more delays, failovers, and outages than we liked, and lots of shouting from the customers and at each other!

What was clear was that it needed to change.  Ops pretty much knew over time they could handle most of the external changes and had an architecture that could scale and handle massive increases of volume and volatility; but it was internal changes that were creating the majority of the problems in the meantime.    So, we set an aggressive target; zero company caused outages.  But here we were located in multiple geographies around the world where technically and as a global business, we needed the changes to be around the clock and by different teams and individuals from all parts of the organization; some planned over time and some nearly spontaneous. 

So, network ops developed a “process” that created cues around the globe that triggered teams across functions to pause and pay attention as a group briefly to be sure they had each other’s backs.  We needed a script, and identified the stakeholders, both business and technical, for the changes.  Each change communication included: the change intended, the date and/or time, the risk, and the business reason.   It was then communicated by the person making the change. It was an adequate process but not as complete or detailed as it might be. It was not the process however that reduced the outage rate from several a month to less than one a year. It was the fact that “CHANGE” coming  from or through on your screen created habits. The person accountable for the change communicated to those impacted and had the accountability to think through the impact. Those impacted looked up from their own accountabilities to be sure what was proposed did not conflict with another change. It created a form and habit of communication spontaneous around the process and decisions in which management rarely needed to become involved because the team supported one another. 

Implementing change is rarely about the solution alone but about the altering of behavior by the group seeing themselves as in it together, and engaging their pride in striving together to eliminate that customer fear of  wondering when their system might go down next.