Apr 30 2014

Free Can Make You Bleed

By now anyone concernedwith internet security has heard about the Heartbleed security vulnerability in OpenSSL.  What you may not be aware of is how much money and personal information is riding on this “free” security program and others like it (OpenSSH).  Free is not usually a bad thing, but it can be when it causes the software your business depends on to be under resourced.  

What you might not be aware of is just how under staffed and underfunded some of these “free” open source programs like OpenSSL and OpenSSH (OpenBSD) are. OpenSSL for example is largely staffed by one fulltime developer and a number of part-time volunteer developers.  The total labor pool for OpenSSL maybe adds up to two fulltime developers.  Think about it, OpenSSL only has two people to write, maintain, test, and review 500,000 lines of business critical code.  Half of these developers have other things to do. 

bsd
OpenSSH, part of the OpenBSD project, isn’t any better off.  OpenBSD includes a number of projects other than OpenSSH.  As with OpenSSL, OpenBSD gets relatively little in funding from industry that depends on it the most and relies on volunteers.  The OpenBSD project leader, Theo De Raadt, is concerned about the lack of funding from the industry.  In his own words:  "I think that contributions should have come first from the vendors, secondly from the corporate users, and thirdly from individual users. But the response has been almost entirely the opposite, with almost a 15 to 1 dollar ratio in favor of the little people.”  OpenBSD is so underfunded that it almost shutdown because it had difficulty covering just the electrical expenses for the project.  At the beginning of 2014 a request for funding was issued just to cover the electrical costs for the project.  OpenBSD developer Bob Beck suggests that the project will shut down completely if a more sustainable source of funding is not found.

It is ridiculous when you think about all of the business capital that depends on such grossly underfunded applications.  OpenSSL has never received more than a million dollar yearly budget and OpenSSH can’t pay its electric bill.   The OpenSSL foundation’s president, Steve Marquess, said “The mystery is not that a few overworked volunteers missed this bug; the mystery is why it hasn't happened more often." 

Marquess has noted an increase in donations and support for OpenSSL.  Even with this increase in donations OpenSSL is still underfunded.  Marquess says, the OpenSSL developers are not to blame for this security vulnerability; they did what they could with the resources they had.  The true culprits are those who take from the open source community without giving enough back.  It seems like people now realize they should pay more than nothing for the software that keeps their private information secure, but how long will the donations keep flowing and will they be enough?  Will open source users donate now that this issue is in the news and forget to donate later?

There is an ongoing debate on whether open source software is more or less secure than commercial software.  Those who believe that open source is more secure argue that you have more eyes on the code and more people will review it.  At face value this makes sense and most people would agree.  However is this really true?  It could be argued that with so many smart people available to review the code that few will actually do so.  This psychological phenomenon is called the bystander effect and it states the probability of help is inversely related to the number of bystanders.

The error in question was one of the most basic and common errors in programming.  Many people might make the assumption that this was some noob making their first open source update and nothing could be farther from the truth.  The person responsible was an expert programmer and reliable contributor to the open source community.  This just demonstrates that mistakes happen and anyone can make them.  The programmer should not be singled out or maligned for this mistake. 

One of the community’s top programmers wrote the code and the world’s smartest people were available to scour the code for any errors or security breaches.  So why didn’t anyone catch this for two years?  I think the bystander effect and the brilliance of the developer can answer this.  Simply put, no one expects Michael Jordan to miss a free-throw and no one expects a crack programmer to make a simple mistake.  Chances are if a freshly minted programmer was making the update, it would have received more scrutiny and the mistake would have been caught.

With so few resources available to open source projects, they don’t have time to check the work of experienced programmers for basic errors.  People like this almost never make these kinds of mistakes.  They instead focus their limited review resources on more complicated and technical areas where it is far more likely a mistake will be made.  After the code has been reviewed by top IETF programmers few people will donate their precious time to check for basic mistakes.  Are you going to go out right now and review two year old open source updates for uninitialized variables or memory overlays?  No, almost no one will because somebody else will do it and you are too busy.  

I have worked as a commercial software developer for more than ten years and when I ask one person to review something it gets more scrutiny than if I send a mass email to everyone in the department.  When you have just a few people responsible for quality control and this is their job, they know they can’t point the finger at someone else or slip in to obscurity if something gets by them.  When it comes to code reviewers less is more.  

The people who are more likely to start reviewing open source security programs like OpenSSL and OpenSSH for flaws like this are those with the most to gain, cyber criminals and hackers.  Maybe someone really did spot this vulnerability sooner and decided to exploit it rather than fix it.  It is sort of like leaving a bag of money out in the open and hoping someone will do the right thing.  We don’t know if this happened and we might never know because of the nature of the Heartbleed vulnerability.  Critical information could have been available to cybercriminals for two years.  A good cyber criminal can use your stolen information for years before you discover that you bought someone a house in the Bahamas.   

When your business critical information is stolen the price of open source products might be free, but the cost could be much higher.  There are places to cut costs, but your business security is not one of them.

John Walsh by John Walsh Software Engineer, Core Development
John Walsh is a Software Engineer and a member of R&D at SSH Communications Security where he has focused on core product development and technical support. John has over 10 years of experience in software design in the IT security industry. Prior to joining the company, he worked at IBM where he designed and developed a number of key software features for security products such as LDAP, Firewall, and Java Cryptography. John holds a BS in Computer Science from Binghamton University as well as an MS in Management Information Systems from Marist College.