I just read a post on Marcus Ranum’s Tenable Network Security blog, titled “Anatomy of a Security Disaster“. One of his main points appears to be that corporate risk management, by attempting to boil down risks to something easily quantifiable (“trying to balance an unjustified estimate of cost of failure against a wild-ass guess multiplied by a fudge factor”), contributes to bad things happening. According to Ranum, it does this by giving uninformed managers a potential issue that they should not be allowed to sign off on without fully understanding the underlying problem, and without someone in, say, a development process being forced, through increased accountability, to be responsible for making sure things are done right straight off the bat.
There is a very strong thread in the article that risk management, while maybe not directly at fault for either acute or systemic failures, is at least partially to blame for it:
“Unfortunately for us all, the Wall St crash of Dec 2008 serves as a complete debunking of the value of risk management. All the big firms that lost billions or went out of business had risk management departments and practices and felt they were taking acceptable risks. Perhaps the risk management departments were wrong, or perhaps management was living with a reality gap.”
This, bluntly, is nonsense.
First, it is not the fault of a risk management department if management fails to accept a demonstrated risk. Risk analysis will, by its very nature, always be an approximate art (no, not a science, you need to hire people with a lot of experience, and maybe a bit of gut instinct, pay them a bunch of money, and be prepared to trust their analysis.) Otherwise, why would anyone ever address a potential security flaw?
Risk analysis is nothing more than the prioritization of potential problems with a system, process, tool, what have you. It is at the core of any bug fix, response to security failures, in short, any security event. And security events do happen, like it or not. It’s not a perfect world, nobody will ever have all the information, and it is a tired cliché that “hindsight is 20/20″, it’s still true.
Second, a proper risk management process does not just entail handing an incompetent manager a set of gift-wrapped risks on a silver platter and saying, “here you go, enjoy.” That’s called covering your ass. Correctly done risk assessment and management consists of accompanying, for example, a development process from a to z and being the additional pair of eyes to spot non-obvious flaws.
Ranum’s ideas about how to help prevent or reduce the probability of something going pear-shaped are all valid. However, he doesn’t seem to understand that these all constitute part of a correctly designed risk management process.
Hi John,
I think you’ve missed the point Marcus Ranum was trying to make. As I understand it, he’s not saying risk management as a concept is bad, but that it is typically poorly implemented in information security, because the data is not there to do a solid quantitative risk analysis.
I have also observed this. Because of poor or nonexistent historical data, the risk-benefit analysis leading to information security decisions often boils down to one person’s gut feeling vs. an other’s. Furthermore, given that the person making the final decision might not have any experiential basis for even a good gut feeling, the decision often comes down to who makes the best presentation of their argument, or who has the best relationship with the decision maker, rather than the merits of the argument itself.
The only way I know of to improve the situation is to implement comprehensive quality improvement systems based on solid scientific research. One place we can look for examples as to how to do this is medicine, where it is now commonplace. What we also see in medicine, unfortunately, is that such systems are very expensive, and require cooperation between disparate entities, such as governments, higher education and private enterprise, on a scale that I don’t think the private sector will be willing to fund.
That means we will not see substantial gains until governments step in to fund and coordinate the necessary research, and/or we see the formation of large nonprofits to do it.
Thanks — I re-read the article in response to this, and I am going to have to respectfully disagree; if MR’s point is that risk management can be done well, he sure hides it well.
Of course it is often (I wouldn’t say “typically”, having worked in three organizations in a row with pretty good procedures) badly done, but that’s not a fundamental failure of the concept.
Yes, on the historical data thing — but as for the gut feeling, it may be a strong factor for the risk analyst, but I expect any organization to hire guys with enough experience to be able to make a fairly reliable educated guess. As I said, there is always an element of witchcraft involved.
Regarding the guy making the final decision — any risk management organization worth its salt has a reasonably well-developed quantifiable set of classifications that will let a risk analyst present a manager with an easy decision.
Because, rant as much as one may about “it should always be done right”, security is _always_ an economic decision, and when the choice is spending a million dollars to rectify a potential problem that might bring down your company but has in a one-in-a-million chance of occurrence, even if that one-in-a-million is not based on any methodical statistical analysis but on basic common sense, nobody has the right to fault a manager’s acceptance of that.
I’ve been doing a lot of medical compliance work, and while the idea of over-quantifying the scientific aspect of risk management, I’ve seen it very rapidly degrade into a paper storm that exists for its own sake, without anyone really paying attention to the original reason for the paperwork’s existence — form becomes more important than substance. The people I worked with in the medical industry were generally very conscientious, but at some point, paper and process fatigue will always set in.