A planned Black Hat talk on High-Frequency Trading (HFT) vulnerabilities was recently pulled from the 2010 Black Hat conference, ostensibly at the request of one of the authors’ clients who probably felt that the planned disclosures hit a little close to home.

HFT is a hot topic in circles ranging from regulatory and compliance discussion forums, over smaller traders, to conspiracy wingnuts.  Despite the fact that it’s just one technological tool among many to give exchange participants an edge over competitors, I tend to side with the conspiracy theorists, especially insofar as it’s an approach to transactions that by its very definition gives an edge to larger market actors — thus skewing the idea of a “fair market”.  Various individuals have claimed that this goes so far as to allow participants to manipulate prices using fake orders, but I don’t know enough about trading technology to comment on this.

Due to the very technologically intricate and detailed nature of HFT platforms, very few people understand how they work — and overtaxed regulators and security & compliance organizations thus are left in the dust when it comes to ensuring that such solutions do not present a security and operational risk, not just to the companies who run them, but to overall market stability.  Remember, complexity is almost always bad if you can’t reliably understand it with a reasonable grasp of the subject matter.

The article has one particular paragraph that rings very true:

While applications are combed for typical application vulnerabilities like SQL injection or cross-site scripting, they’re not examined for operational vulnerabilities: A rogue trader could, for instance, change a single variable to allow far more risky trades than a bank or its clients intend–the sort of trick that Société Générale trader Jerome Kerviel may have used to make unauthorized trades in 2008 that cost the firm $7 billion.

Yeah, basically.  Many of the people who use these toys are very bright autodidacts, creating customer tools for exotic, structured products.  Even off-the-shelf software is frequently written using what I like to call “functional programming” — i.e. a very smart person with a Visual Basic book coding a solution to an operational requirement without paying attention to best practices that may, in any case, be outside of the scope they care about.  Investment firm management is likely to turn a blind eye to even obvious flaws in such software, due to the fact that traders (a) bring in massive amounts of revenue and (b) are increasingly the only people who understand certain market types.

The rise in black pools and other off-exchange trading will only increase the latter phenomenon; I’ve seen trading floors where it was pretty common for one person to be responsible for a particular exotic product; frequently, this person might even have been the one who helped create the actual market.  Remember, you can trade pretty much anything, as long as you have a willing counterparty…

How do you deal with such issues as a security professional?  To many who are not trained in the black arts of financial transactions, much technological innovation in modern markets is the driving force behind an increased complexity that regulators cannot hope to oversee effectively.  Analyzing security issues in such tools also becomes difficult, even from the inside, even when a company is willing to implement controls — the line between legitimate exploitation of a weaker player’s market position and an actual security intrusion blurs to the point where traditional technical vulnerability analysis is no longer an option.

I don’t have a solution, beyond an innate distrust of anything I don’t understand in detail, but I believe that companies’ willingness to reduce their exposure to security issues stemming from overly complex trading software is  more a philosophical than a policy or technical question — how far are we willing to go to exploit loopholes in the spirit of market regulation?  Are we willing to sacrifice potential high risk + high profit combinations in order to remain in more staid trading areas?

And most importantly, if management won’t listen, as a security guy, can you sufficiently CYA to avoid being caught up in a potential technical or regulatory failure of one of your employer’s systems that’s just too intricate to reliably review for risk?

An increasing amount of security work these days is very academic and theoretical, rather than hands-on.  The development of risk management (a.k.a. “covering your ass”) as a serious factor driving security awareness over the past ten years has led to a massive amount of paper-generation and abstraction, going hand-in-hand with ever-more-complex attempts to needlessly quantify security realities.

Partially this stems from the desire to apply financial metrics to risk and security, partially from the need of managers lacking a strong technical background or the ability to acquire this to have easy-to-understand, purely numerical descriptions and graphs of where their money is going.

One of my colleagues recently came up with the concept of an abstract measure of security investment (time, effort, grief, etc.) — the securitron.  As in (hold hands zombie-like out front and repeat in a robot manager voice), “we’re going to need more securitrons!  Throw two hundred additional securitrons at this project!”

For some reason, this came up again during a meeting to discuss increased efficiency of risk management paperwork.  The upshot of this meeting, as with pretty much any business meeting consisting of over three people (and not taking place by the coffee machines, where most productive work gets done), was (a) use your own damn judgment, that’ s what you’re paid for, these are only suggestions, (b) if you still have questions, ask your colleagues, that’s why we sit together, and (c) when will this damn meeting end?

However, an interesting point did arise — the idea that security, risk management, and compliance work tends to end up with reams of paper that may or may not ever be read.  This was the case with all medical regulatory and compliance work I’ve ever done, as well as with pretty much every experience I’ve had in financial services since 2000.

The conclusion?  My personal view is that documentation and paperwork does not make much sense unless it’s in an easily accessible format (e.g. a well-organized database schema accessible by very flexible views to whomever wants to use and evaluate the data.)

Another very important point that my colleague raised, though, was the concept of the “thunk test” — i.e. if a security policy is dropped on a desk and makes a “thunk”, it’s too big to be of any use.  Nothing that heavy can possibly have the elegance and simplicity to be easily enough understood for practical everyday application.  At the risk of advocating the development of tired business doublespeak, I could see a use for a plainly worded non-mealy-mouthed type of goals and rules statement (“don’t leave passwords lying around.  Never talk to strangers.  Don’t pet strange dogs”) for security organizations that actually fits on a single piece of A4 paper (or at least in a document that makes no more than a soft whisper as it’s dropped.)

Now, we are working on finding a good name for a measure of the quantitative relationship between securitrons and reams of paper.

Wired Magazine recently published a set of articles titled 12 Shocking Ideas That Could Change the World. Among them was a suggestion by a Danish gentleman named Thorkil Sonne, of Specialisterne, to hire autists.  His reasoning is that people with autism or asperger’s are good at routine tasks;

“As a general view, they have excellent memory and strong attention to detail. They are persistent and good at following structures and routines”

So, I got to thinking; one of the discussions I had with colleagues following a recent meeting with a provider of managed security services (see previous post) dealt with the idea that building and maintaining a decent SOC and log/event management procedure is a near-impossible task, due to the difficulty of creating a sustainable operations process; one of my co-workers made the point that most people would tend to get bored with the repetitive nature of looking through logfiles, or even dealing with things like security advisories and updates.

I countered with the assertion that your basic event management and intelligence correlation should, to a large degree, be automated anyway (whether non-dedicated companies can do this as well as, or better than, dedicated outsourcers is a separate discussion that’s probably best investigated on a case-by-case basis.)  However, the routine nature of any IT operations-type job aside, he did have a point that (a) even when you do automate everything, you’ll still want a degree of human second-guessing, and (b) that gets mighty dull.

It seems like a logical conclusion that if Thorkil Sonne is right, and autists (is that a word?) can really be excellent at work that requires focus, consistent attention to detail despite the drudgery involved, and clear rules, why not use these guys for exactly the tasks described above?  Someone who is excellent at pattern detection, and is willing to follow the same process day after day after day would seem to be ideal for an SOC monitoring / log & event analysis position.

I don’t want to come across as somehow insensitive, since I know next to nothing about autism, but purely going by the value proposition put forward by Specialisterne, it seems like a rational conclusion to hire the best people for any given job — whether or not the fact that they are qualified at what they do stems from extraordinary dedication, experience, or a mental condition should be completely irrelevant.  And in the process, maybe do some good for someone who might be difficult to employ otherwise.

Between the two of us, Arjo and I have helped design, set up or run on the order of a dozen public key infrastructures (PKIs). Looking back, not a single one of these was not somehow embroiled in administrative or organizational infighting, bogged down by uncooperative management, or less-than-optimally functional due to design decisions taken for some sort of political reason. To use a colleague’s phrase, these things seem to attract managers like FOS — flies on you-know-what.

We were recently asked to put together a requirements paper for a PKI that was being re-engineered from the ground up. It occurred to me that this might actually be a stellar opportunity to help a client get things right from the start. So, why do we always get bogged down in PKI-related organizational or management issues?

Obviously, the PKI’s main purpose is management of digital certificates for persons, organizations and technical entities. Certificate keys fall into three general categories, which I’ll use Entrust nomenclature to describe: encryption (obvious enough), signing (you can be sure that what you got from me is what I intended to give you) and non-repudiation (you can be sure that I cannot disavow what I gave you.) So, let’s look at what you might actually use these for:

  • User authentication (such as web sites or to workstations via smart cards)
  • Email and file encryption
  • Secure key exchange (e.g. for phase II IPSEC negotiation) between devices
  • Source code, binary and other data signing (e.g. financial transaction information)

Why is this important? Two reasons: first, your certificate templates, which govern the formats of the certificates and their usage, depend on what you intend to do with certificates. In most certificates, for example, there is a key usage flag (a hex value) that is used for this.

Second, and more importantly, your LDAP structure must reflect you organizational structure, certificate usage and a number of other factors from the outset. This is usually the largest bone of contention in PKI design; the classic example is the Jane Doe who marries Bob Smith and changes her last name. So a distinguished name (DN) like C=Albania,O=Bob’s Widgets Inc,OU=Research,CN=Jane Doe is now kind of messy. This can be dealt with by using a friendly name, but you get the idea. Changes in requirements for LDAP attributes and DN structure affect other areas, like host naming, validity of signing and encryption keys, and and and. It gets even more fun when you start doing things like cross-certification, setting up a metadirectory for multiple directories, or integrating existing directory structures.

The upshot of all this is that a PKI affects all aspects of an enterprise that are in some way involved in authentication, security, encryption, verification of data integrity, etc. — and it has to pretty much be done right from the outset, or you run into problems of application integration, user acceptance and overall cost. We know of several directory projects that had their scope drastically cut down after they were built, when it was discovered that they simply wouldn’t work for some of what they were intended for; someone obviously hadn’t done their homework.

Most tragically, sometimes design decisions are taken or changed late in the development phase. I will recount one infamous experience of a client who, based on a few casual observations our team made, called a stakeholder meeting consisting of a room full of 35 people shouting at each other, and reversed his entire enterprise directory strategy _twice_ in the course of two hours. We’d done the underlying design and recommended a certain set of project technology several months previously, but were ignored due to political reasons. In this meeting, our group just tried to remain rational and calm, and to give information when prompted, but in the end we realized there was just nothing to be done and snuck out. It was telling that nobody noticed.

Probably, everyone who’s affected will want a piece of the discussion, and we all know how well design by committee works. Many managers also seem naturally threatened by (or nervous about) any technology that effects a lot of blanket organizational and procedural change — a perfect categorization of an enterprise PKI. It doesn’t help that such products sometimes carry the stigma of taking away autonomy from other groups; at some level, any infrastructure component using PKI certificate must follow certain standards and procedures, there’s just no way around that. This is often seen as a loss of self-determination, or it may require technical effort, and who’s going to pay for that?

So, how does one deal effectively with this? A lot of good PKI design practice involves some pretty common-sense elements:

  • A knowledgeable, strong project manager who can organize management buy-in and support
  • A small design team who’ve done this before and have a clue what they’re doing (good luck!)
  • Individual discussions with potential stakeholders about what they want and how their concerns can be addressed
  • A _lot_ of time spent on research and design — use the “hire a 6-year-old to spot any obvious flaws in your plan” model liberally here
  • A clear idea of budget responsibilities in PKI deployment — but not to the end that every affected team will try to pawn off even minor changes on the PKI budget

And last but defnitely not least, a really good LDAP guy, you won’t regret it.

p.s.:  If you’re interested in reading more about non-repudiation, including how it differs from the usual technical usage of “signing”, have a look at Non-Repudiation in the Digital Environment.

I created a basic computer security incident response presentation a while ago. It outlines the benefits and concerns involved in the creation of teams and mechanisms tasked with action after computer-related security issues.

Note that a CSIRT doesn’t replace the regular R&D and operations groups, but is a supplementary body, coordinating both preparation for and response to things like information breaches and attacks.

By its very nature, a CSIRT has to have a strong mandate and a lot of authority in an organization; this requires management support, and a lot of it, in addition to a really close and positive relationship with all other teams involved in these activities. Examples of these are the network security organization, various helpdesks, human resources, email administrators and others who may be attached as auxiliaries to a CSIRT.

In order to get this sort of mandate, you will have to coherently demonstrate an overall risks-rewards scenario. So:

  • What are the threats to my organization and what will I lose if I do nothing about them?
  • What does my organization stand to gain (productivity enhancements, etc.) by dealing with these in an organized manner?

In my experience, the most positive elements of a CSIRT is that it allows coordinated vulnerability management and a coherent, organization-wide process for things like patching systems and updating security software, in addition to being a locus for discussion and cooperation by various technical groups (the Windows server administrators, UNIX, email, network, security, etc.) This sort of collaboration tends to extend beyond the direct focus of the CSIRT’s day-to-day activities.

A few examples of the logic I use:

csirt.png

Simply put, if you don’t do anything, and something goes wrong, the results can vary from your looking stupid to someone going to jail. There’s a fine line between FUD and realistically outlining possible consequences, and IMHO Sarbanes-Oxley has been a bit overused by enthusiastic security people, but I would clearly outline things like data protection laws or past Bad Things happening when organizations suffered attacks (i.e. British Airways losing several thousand workstations to Blaster.)

csirt11.png

I think this is a pretty good outline of what it is I’d propose as a CSIRT’s roles and competence. In your case, you might want to play up the final business value a bit more, maybe put numbers on the risk reduction.

If you contact me and ask nicely, I’ll be glad to share the whole thing.

Or: when not to swat flies with a Buick. So hi ho, hi ho, onto my soap-box I go.

At one point, someone at one of my early employers decided that I would be “the security expert.”  OK, fine.  I attended a meeting with a manager in Germany. At the time, a group of people within the company was attempting to leverage a combination of their excellent and pioneering security and cryptography products, as well a relevant selection of Internet solutions from external suppliers. Among these was the then-brand-new Netscape proxy server. My meeting partner immediately began to enthuse about the possibilities of monitoring the activities of all employees, monitoring this, logging that, etc. etc. etc. I had to bring him down from his tree and remind the guy that if your people are spending all day looking at porn, that’s a leadership and motivation issue that you can’t solve technically.

Shortly thereafter, when I installed the first of these things at a client site, I left their brand-new leased line up and running, and had the proxy log connections as a test. The next morning I noticed a huge amount of logs; as it turned out, one of the IT guys had stayed late and downloaded inordinate amounts of fairly nasty porn. My client’s IT manager had made it clear that he was of the same persuasion as the German colleague described above; the staffer would doubtlessly have been fired, so I asked a colleague of his to quietly inform him that all connections were being logged explicitly as of now, and deleted the logs (this was before we handed over the infrastructure so I wasn’t technically outside my authority.) I don’t suppose he did it again, and I still think this approach was better than costing the man his job over some naked girls and farm animals and whatnot.

In the meantime, I have gotten into numerous discussions about this and similar topics. What I call the “powerpoint mentality” often seems to take hold of many managers of all backgrounds, ranging from freshly promoted former engineeers to MBAs from top schools with years of experience. By this, I mean that frequently things look good in a theoretical presentation, but in real life they break down. The main culprit among these is the habit of small minds and bad leaders to try to quantify what they do not understand, such as human nature. When you quantify something, you can start to restrict it so it fits your world view. Unfortunately, it often leads you to neglect responsibility for intangible issues that you just have to play by ear based on skill and experience–in German, this is called “Fingerspitzengefuehl”, or “finger-tip feeling.”

I see parallels to the discussion about limiting sysadmin privileges. To my discredit, I am prone to lumping this argument in with the category of “control-freak managers.” I shouldn’t do that, as there are good reasons for proper privilege separation and audit trails. However, my bugaboo with this is that it very often (a) goes to extremes, seeking to artificially restrict and control what a sysadmin does (in direct contravention of one of my rules of Getting Stuff Done: “hire good people, pay them a lot of money, and trust them.” Within reason, of course.) And (b) lets crappy management off the hook by focusing on technical staff as the primary threat to the enterprise. I tend towards inexcusable knee-jerk reactions, not being very informed about statistics of privilege abuse by technicians, and having seen far too many incidents of managerial abuse of privilege not to be cynical whenever the topic comes up.

At risk of ranting and rambling, I’d like to draw a parallel to security professionals uncovering incriminating evidence on other employees, and being informed by HR either that no action be taken due to the “suspect’s” standing (i.e. a profitable trader), or that employment and privacy laws be knowingly broken in the course of the investigation, as the potential fine, should this violation be discovered, would be far outweighed by the likely financial loss to the company if the incident were not investigated with all available means. All this in all likelihood (no paper trails please) with the tacit knowledge of C-level executives. I’m aware of both of numerous occurrences of both of these.
I leave it as an exercise to you, dear reader, to decide where the line lies between restricting and auditing employee and sysadmin activity, and doing the same for managers who seem to have trouble with basic right and wrong.

© 2010 Chakraborty Software Suffusion WordPress theme by Sayontan Sinha