Aug 18 2014

Black Blob of Death Threatens Data Center Security

Researchers at SSH Communications Security recently uncovered a serious security vulnerability that impacts data centers in the vast majority of banks and financial institutions. Okay, so admittedly, it almost sounds like a story from The Onion or some made up news by a bored blogger. But this really isn’t a satirical post about the over-hyping of security issues or fake news. The Black Blob of Death is real.

BlobThe image above shows the topological view of trust-relationships using Secure Shell key authentication inside of a regional bank’s data center. The smaller gray circles indicate hosts while the lines connect private keys with public keys. The red lines indicate rogue or unknown keys with access to the data center - an obvious security risk. Let’s dig a little deeper into why a lack of proper access controls in Secure Shell environments can create a real security risk for banks and other financial institutions – and on a scale that most CXO’s can even imagine.

Here are some interesting facts we have uncovered in our work with a dozen or so major corporations:

  • Secure Shell key-based authentications easily number in the hundreds of millions (yes that's right, 100,000,000+ annually) and 90% of the time the credentials used to authenticate are not properly managed
  • Secure Shell is often second or third in the number of authentications, usually only trailing customer logins via the web and employee logins to various internal applications
  • In every enterprise we have worked with, we have found on average, 1 unknown root key for every 10 servers. That means in 10,000 host environments you can expect to have 1,000 servers with a serious issue
  • In some of the largest environments we have tested, 90% of keys are inactive or redundant. This is because virtually no one removes keys once they are deployed and over time they just keep piling up.
  • IP’s were accessing the production environment with unknown keys and often from places where access to those servers is a policy violation

So who cares, right? The Black Blob of Death has been out there for a long time and the world hasn’t come to an end. So why should we be worried now?

Here are a few reasons: Careto, Fokirtor, Snowden and Host Gator just to get started. So the security-minded should take note. For those in the audit community PCIv3, NIST/FISMA, SOX, Monetary Authority of Singapore and various US banking regulators have all issued new or revised compliance directives around Secure Shell access controls. Why? They’ve seen the Big Blob of Death and it scares the bejesus out of them.

With so many potential hack points and security issues coupled with limited budgets and manpower, what is a CXO to do? Let’s look at the Black Blob of Death from a risk management perspective.

Assuming you agree you have a similar problem (and if not please contact us so we can collect our missing key remediation technician - he has a family), see if you agree with the following:

  • If you managed your customer’s user names and passwords in the same manner as you manage your Secure Shell keys, would your customer’s trust you? Would you trust the security of your data?
  • If 10% of your hosts had unknown keys granting root level access, is it likely that at-or-near 100% of your data passes through one of those systems making it highly likely that you could suffer a major security breach?
  • Have you been properly deploying, rotating and removing Secure Shell keys? Do you know what all of your keys do? If you remove them do you break a business process?
  • Let’s assume the worst; you have a breach and the attack vector is Secure Shell via public key authentication. Say you have a 10,000 host environment - how long would it take for you to remediate the situation and stop the attack?
  • Are you certain that no one has placed back doors, rogue keys or test/dev keys into your production environment? Can you produce a report to show that?

Using my psychic powers, I already know how you answered - and from a risk management point of view, the situation isn't good. The good news is we’ve been working on a solution to this challenge, and we’ve already helped some major companies get a handle on the problem. If you want a topological view of your trust-relationships, and begin the process of getting your organization back in the clear, give us a shout.

Interested in learning more? Register for our upcoming webinar "The Black Blob of Death Threatens Your Data Center: How to Find It and How to Stop It" on September 24, 2014. You can register by clicking here.

Jason Thompson by Jason Thompson Former Vice President of Worldwide Marketing
Jason served as Vice President of Worldwide Marketing until October 2014.