Beyond Risk Management: handling the unexpected

Ideas, thoughts, and resources from the permanently curious.

Living

World

Politics

Beyond Risk Management: handling the unexpected (8/1/05)

We all concern ourselves with finding the best way to conduct the daily aspects of our lives. We read Consumer Reports and watch the news and talk to friends to determine what the "best" is. I rarely purchase anything over $5 now without first checking Amazon.com reviews or a Google search as they are just too useful to ignore. This is probably the first time in history anyone can behave as such, not just as an informed customer but as an informed person. It is just as easy to do a google search to find another, better, way to do something than the way we are already using. With a large number of people doing this we get a problem though: with more time devoted to reviewing material less time is available for creating new material, so we'll see a drop in original thinking. I worry about how we'll compensate for that. I wonder if this is the reason we're seeing the drop in Math and Science scores for US students.

Anyway, after having reached a certain degree of efficiency at this, some of the more mundane problems disappear. Things tend to run smoothly. There are less headaches because of that. But this is where things get very interesting, because the headaches I encounter now all seem to come out of left field. Addressing them is a chore, let alone anticipating them and addressing them before anyone is confronted by them. So I've been putting some thought into this, because I'm being hit by it on a number of fronts at the moment. I've identified a couple of lifecycles:

Time-Based. Here we see problems develop on a consistent basis, but the consistency is over a period of months and years rather than days or weeks. In the past it was easy to identify problems because they were more "in your face". Now the few that do occur rarely make it high enough on my personal radar to identify an over-arching one that needs to be addressed. This is exacerbated by the amount of projects I am juggling, because now that everything is operating so efficiently I am managing more of them and am thus less likely to notice a pattern of problems specific to one of them.
Procedure-Based. This one is tremendously annoying, and is akin to the "I was just following orders" mindset. Troubleshooting a complex issue in a political environment can delay the process immensely. It is easy to get caught in a loop of just taking issues from one person to the next, checking off possibilities and then moving to the next one. To check of one I may need to wait for the person to simple get back to me, let alone get back from vacation or a sick day. In the mean time the problem can worsen immensely. I deal with this primarily within a computer environment, where an issue might be odd behavior on the part of an application. So I move between the application vendor, networking, and the server folks trying to find out what is wrong. In the meantime it turns out to be a new hack or something alone those lines.
Perspective-Based. These are the real zingers. Everything is working fine until someone speaks up about something and then we're discover a lot of folks are experiencing the issue and just didn't think to speak up about it. It sounds crazy but it happens far more often that I ever thought possible. We're rewarded for adapting to difficulty so people do so.

I mentioned that I wanted to be able to deal with these types of issues before the user even encounters them, and it is hard, at the moment, to see how others are doing so. I got a glimpse one day from the fine folks at SAS. I was using their website to get a license key when I encountered an error. I tried once again and got the error again, so I figured they were having issues and decided to come back in an hour or two and see if they'd solved them or not. What happened next astounded me. Five minutes later my phone rang. It was a web developer from SAS. Not only had they noticed I was having a problem, but they'd fixed it and were now calling me to apologize and to see if I could now use the site. In 5 minutes. Let's have a look at what they were doing to enable this:

Errors on the site were immediately being sent to a person for analysis.
That person had to be available to work on the issue immediately.
The error data needed to be cross-referenced with their customer database so they could call the customer encountering the error.
The Developer then also had to have the initiative to make a personal phone call rather than posting an "It's fixed" message back up to the server.

So there are a variety of procedural, programming, and customer service mechanisms in place to make this happen. This is clearly a company that is good at what they do. With that as inspiration here's how I'm handling the challenges I outlined above:

Time-Based. To deal with the spacing of events and other problems pushing them out of my mind, I've got a project folder for each of my projects. In each folder are two more, one for the regular stuff one might find, entitled "Information", and the other titled "Events". For every event, from site crashes to simply overhearing someone talking about the project, it is recorded in this folder. This provides something for me to look back at on a consistent basis, to look for patterns and recall past issues I've forgotten about.
Procedure-Based. Taking care of these is hard. At the moment I know now that when I start feeling like a hamster on a track wheel it means I need to elevate the problem to a group of people who can attack it jointly. It is one thing for a company to say their software is fine so go talk with the hardware folks. It is another entirely to bring those groups together and let them duke it out themselves. That is where results happen. So I'm working to make this a reality on an as needed basis. I.e. I'll bring the people together when needed.
Perspective-Based. Here we're dealing with something that isn't known to me, so the solution is to expand my radar essentially. I'm trying to organize regular surveys from the users and also make the incident reporting abilities easier, and to organize them better to handle the volume. This can be done automatically as well, through triggers and alerts on the servers. I also want to get the users talking to each other so I can observe those discussions and react to them if necessary. This should also help the users use the products better and really inspire the original thinking I was concerned about above, driven by multiple perspectives rather than a singular one.

It'll be interesting to see how this goes. More updates will be forthcoming.