How to Kill a Fortune 500?
A Fortune 500 enterprise is hard to kill in the physical world. In cyberspace, it can be surprisingly vulnerable, and the loss in shareholder value could exceed $30bn. Just look at the Equifax data breach, which cost the company over $5bn in shareholder value over a week, and didn't even involve any disruption of operations.
The most critical and vulnerable part of an enterprise is its information systems. If you can control, corrupt, and destroy the servers and the data in them, the enterprise stops.
Millions of Credentials
Most enterprises have not established or enforced policies on who can create and configure new SSH keys that grant access to servers. They lack a process to remove these credentials when employees leave or systems go away.
We have found over 3,000,000 keys in several Fortune 500 enterprises. Most large enterprises have hundreds of thousands to several million SSH keys that have been unaudited and unaccounted for. They grant access to servers, storage, and network devices.
90% unused, 10% grant root access
In the typical enterprise, about 90% of SSH keys are no longer used. They are access that was not properly terminated when people left. They grant access into disaster recovery systems, into backup systems, from test/development into production, and from personal user accounts to system accounts without a password.
What's worse, typically 10% of the keys grant root access (highest-level administrative access). Often, the keys grant access that bypasses privileged access management systems.
We have surveyed lartge enterprises that have over 5,000,000 recurring automated logins using SSH (over 2 billion logins per year). There is nothing wrong with that, but just illustrates that the keys also fill critical business functions.
Spreading Throughout the Enterprise
SSH keys can be used to spread an attack server-to-server, data center to data center. Penetrate a server, read private keys from the server, use the keys to penetrate other servers. Repeat. Using vulnerabilities to gain local root helps read all keys from a server.
To be stealthy, the attack can monitor the server for days or weeks to see which keys are used with what servers, and then piggyback on legitimate connections.
Once the attacker has root access, it is simple to corrupt data, wipe disks and tapes, and reprogram firmware to render the machine unbootable. Possibly even installing viruses in firmware so that the problem repeats if the old hardware is reused.
How long can a Fortune 500 be down before the reputation damage alone is so bad it never recovers? The damage to shareholders could be most of its market cap - perhaps $30bn. There have been hundreds of articles published on the topic, NIST has published guidelines, and multiple analysts have written about it. It is time to act!
SSH keys are industry best practice for server management automation, but the they must be managed and the legacy must be sorted out. Our best practice for addressing the problem includes:
- Defining a controlled provisioning and termination process for SSH keys
- Discovering existing legacy keys
- Monitoring key usage and eliminating the 90% of keys that are never used (a major quick win and cost saving!)
- Eliminating policy-violating keys, such as access from DEV/TEST to production or access from personal accounts to service accounts (another major quick win).
- Empowering application owners to sign off on all access and data transfers to/from their applications.
See Universal SSH Key Manager for more detailed information on our advanced key management solution for large enterprises.
NIST (US National Institute of Standards and Technology) has published guidance on how to manage SSH keys. See NIST IR 7966 for more information.
ISACA has published on guidance on how to audit SSH, including SSH keys. See ISACA SSH Audit Practitioner Guidance.