SSH Key Management – What, Why, How
The SSH protocol is the global gold standard for remote system administration and secure file transfers. SSH is everywhere. One of the features behind the immense popularity of the protocol is the strong authentication using SSH keys.
An Access Management Problem
SSH keys provide the same access as user names and passwords. Furthermore, they often grant access to privileged accounts on the operating system level, giving a command line. Yet, in many cases, SSH keys have been completely overlooked in identity and access management planning, implementation, and audits. Users have been able to create and install keys without oversight and controls. This has led to violations of corporate access policies and dangerous backdoors.
Over the last few years, it has turned out that most large organizations have massive numbers of SSH keys in their environment. These keys are like passwords. They grant access to resources - production servers, databases, routers, firewalls, disaster recovery systems, financial data, payment systems, intellectual property, and patient information.
Information security starts from controlling who is given access to systems and data. The rest is mostly just enforcing that access and making sure it cannot be circumvented.
If there is no control over access, there is no security, no confidentiality, no integrity, and no guarantees of continued operation.
Insights from Real Key Management Cases
We have worked with many companies, including several global top-10 banks, leading retailers, and other large Fortune 500 companies. Based on our findings, most organizations:
- Have extremely large numbers of SSH keys - even several million - and their use is grossly underestimated
- Have no provisioning and termination processes in place for key based access
- Have no records of who provisioned each key and for what purpose
- Allow their system administrators to self-provision permanent key-based access - without policies, processes, or oversight.
In the case of one representative customer, we went through a quarter of their IT environment as part of a major SSH key management project. They had five million daily logins using SSH, most of them using SSH keys for automation. We analyzed 500 business applications, 15000 servers, and found three million SSH keys that granted access to live production servers. Of those, 90% were no longer used. Root access was granted by 10% of the keys.
A compromise of a root account allows the attacker to modify the operating system, steal or subvert any data, or install malicious software on the system.
While the customer had well-established identity and access management (IAM) practice for user names, passwords, and privileged access, they had not addressed SSH keys, even though they grant the same kind of access. As a result, their IAM solutions missed over 90% of all their access credentials. They had also failed to establish proper provisioning, termination, and audit processes for key-based credentials. SSH keys were also used to bypass their privileged access management solutions.
Regulatory Compliance Requires SSH Key Management
Typical requirements for compliance include:
- Managing identities and credentials - SSH keys are access credentials
- Provisioning and termination process for access - including access based on SSH keys
- Segregation of duties - elimination of key-based access from test and development systems into production
- Principle of least privilege - keys should be restricted to only allow running commands they are needed for
- Disaster recovery - limiting attack spread from primary systems to disaster recovery sites and backup systems
- Privileged access controls - SSH keys are often used to bypass jump servers
- Boundary definition and documentation of connections for payment systems, financial data environments, patient data environments, or between government information systems
- Regular audit of security management practices and how they relate to SSH keys
- Incident response and recovery - changing compromised SSH keys.
Besides regulatory compliance, unmanaged SSH access poses significant practical security risks:
- Reputation risk - major extended outage or large loss of sensitive data
- Financial risk - cost of outages, liability to business partners and customers
- Fines or even personal criminal liability for violations (e.g., with SOX)
- Spread of malware and viruses
Uncontrolled Access Provisioning with OpenSSH
OpenSSH is a free, open source implementation of the Secure Shell protocol. It ships with every Unix, Linux, and Mac operating system. Users use the ssh-keygen program to create new keypairs. They use the ssh-copy-id tool to install those keys as authorized keys that grant access to any server they have (even temporary) access to. System administrators have been doing this for the last 20 years, typically without policies, control, or oversight. They have deployed new keys whenever they need to integrate applications, automate file transfers, have systems management or incident response tools perform operations on many servers. They have also installed keys for personal convenience, for example to avoid going through jump servers for privileged access management.
OpenSSH also provides a proprietary certificate mechanism. With OpenSSH certificates, it is not even possible to audit a server to determine which keys grant access to that server. Revoking certificates requires reconfiguring every server, and is just as complex as SSH key management to begin with. We generally caution against using OpenSSH's proprietary certificates for granting access because of the auditing difficulty. Using them for host certificates is fine.
Tectia SSH supports standards-based X.509 certificates for both host and user authentication. This enables normal hardened certificate authorities to be used. Such certificate authorities are designed to allow auditing every issued certificate.
Solving the Key Management Problem
NIST IR 7966 is a good starting point for understanding how to manage access using SSH. We at SSH Communications Security also have our own guidelines that build on the NIST guidance. We wrote most of the NIST guidelines, together with the NIST staff. Our understanding of the space is based on working with many customers having tens of thousands of servers in some of the world's most demanding and complex IT environments.
Our Project-Driven Approach to SSH Key Management
Bringing a corporate SSH environment under management control is a complex undertaking. With a growing number of real life deployments underway, we understand the intricacies of such projects. We provide turnkey solutions that leverage our mature software platform and the expertise of us and our consulting partners. We are also open and happy to work with the consulting partners our customers, including any of the big four. When needed, we can bring in integration partners with a track record of successful implementations.
Proper planning and execution of an SSH key management project is important. Improperly executed, removal or rotation of SSH keys could render critical information systems inoperable. We have seen multimillion dollar transactions blocked when an application team improperly signed off on removal of an important key.
The role of SSH Communications Security in these projects is typically to provide the software and help structure and manage the project, define SSH-related policies, and typically we provide 1-2 subject matter experts for larger projects to work with the consulting partners.
A project often starts with an assessment of SSH and SSH key usage. We provide a low-cost, quick SSH Risk Assessment service for this.
Technology companies and companies with good internal expertise are typically able to deploy our software themselves and execute key management projects using their internal resources, using only our support services or a relatively small amount of expert consulting. We provide 24x7 support and are committed to ensuring the success of your key management project.
Together with consulting services, Universal SSH Key Manager makes it easy for customers to solve their key management issues. We can run the whole project for you, working with your application teams and identity & access management, cryptography, security engineering, operations, and/or IT transformation groups as needed, globally. We've run SSH key management projects in the US, UK, Germany, and Singapore, among others.
Managing SSH keys in legacy environments and through transition to cloud and DevOps is a complex undertaking. We make it simple for you.
SSH Risk Assessment
SSH Risk Assessment is a high-impact and low-cost service to evaluate and assess the scope of an organization's SSH key management situation. It is conducted in the space of a few days without disruptions to the existing environment. We also provide the software free of charge to selected customers for self-evaluation, and to partners and auditors for evaluating SSH key management in their customers' environments.
Universal SSH Key Manager
Universal SSH Key Manager is our flagship product for managing SSH keys. It has been used by numerous large and mid-sized organizations for solving their key management problems. It is at least two years ahead of the competition in real-world deployments in large environments. It handles the entire life cycle for key-based access, and integrates to leading identity and access management systems, privileged access and privilege elevation systems, as well as SIEM and configuration management.
CryptoAuditor is a product that monitors, controls, and audits actions within encrypted communications protocols, such as SSH, RDP, and SSL. CryptoAuditor complements any SSH key management project by enabling visibility into encrypted communications to and from the corporate core resources. It also enforces tunneling policy, adds analytics and data loss prevention capabilities, and monitors access to internal systems by third parties such as consultants, outsourcing partners, and vendors maintenance personnel.
Risks of Unmanaged SSH Key Based Access
Unmanaged access exposes organizations to significant risks that could in the worst case bring down critical information systems for months. Unmanaged keys risk systemic failure of critical infrastructure, especially in a cyberwarefare scenarios.
Millions of Keys...
There are way more SSH keys around than anyone seems to believe. In one financial sector customer case we encountered as 3 million keys (750,000 distinct keypairs) from 15,000 servers. In another case the customer found 4.5 million authorized keys from their 100,000 servers. Typical numbers for Fortune 500 companies range from hundreds of thousands to millions - many times more than they have employees or system administrators with command line operating access.
What If One Key Is Lost?
A single key is enough to gain access. We have found that in several customer cases about 10% of the discovered keys grant root level access. A compromise of a root access is of course the worst case scenario.
What Can Happen?
Failure to protect financial systems in a public company may lead to misrepresentations and expose CEO and CFO to personal criminal and civil liability under the Sarbanes-Oxley Act (Public Company Accounting Reform and Investor Protection Act).
Failure to protect credit card processing environments may subject the enterprise to massive penalties and loss of reputation due to theft of credit card data. For example, the famous Target breach was estimated to cost the company hundreds of millions of dollars.
In some jurisdictions, companies are required to publicly disclose cybersecurity breaches and loss of customer's private information. For example, the EU NIS directive requires timely disclosure and has potential penalties up to 4% of revenues.
Lack of proper security controls, especially basic controls on who can access and modify what, may lead to higher insurance premiums, loss of banking license, or other penalties.
Without effective access control and management of keys, companies are unable to change the keys - even in the event of a breach. Withouth SSH key management, it is impossible to change keys when they have been stolen or when threat intelligence that indicates that some keys have been compromised.
Hackers know how to utilize SSH keys to spread an attack and configure long-term backdoors. There have been several observed instances of malware programs that have been collecting SSH keys. The attackers will then use the collected keys to break into additional systems. Combined with SSH tunneling, access is also possible from the public Internet.
If keys are used for file transfers between business partners, they may be used to spread the attack between organizations. Improperly configured SFTP file transfer connections may be used to log into other organizations using stolen keys. The keys may be used for stealing other authentication credentials or installing data collection software for furthering the attack in other ways.
Given the vast number of keys, it is likely that once an attackers have penetrated to one server, they will be able to use SSH keys to log into one or more other servers, and then onwards using keys found there. With lateral attack movement like this manner, the attack can quickly spread throughout the entire server infrastruture. The spread can easily be automated in attack tools and malware.
Many companies use SSH keys for copying data into disaster recovery sites, backup systems, or for updating configurations when switching to using the disaster recovery systems. The same keys may be used for spreading the attack into disaster recovery sites and backup systems.
Given that 10% of keys seem to grant root access, and other attacks to escalate privileges locally may be combined with key-based spread, it is likely that attackers will eventually gain root access. Once they have root access, they can modify the operating system, install new kernel modules, modify firmware (e.g., to install firmware viruses or virtualized rootkits), subvert encryption systems, or outright overwrite all data on disks, backup tapes, and finally reprogram hard drive, network adapter, and BIOS firmware to brick the server - potentially tens of thousands of servers.
Vulnerabilities in hypervisors are publicized several times an year. An attack may also combine these vulnerabilities into key-based attack to spread from virtual machines to real hardware and to other virtual machines.
For the modern society, cyberwarfare is a relevant risk, and a coordinated attack across critical infrastructure with the intention to destroy and confuse is a real possibility. Governments must take this scenario into account. We've already seen government-sponsored malware and attacks in Ukraine utilizing SSH keys offensively.
Real World Complications That Can Make or Break a Key Management Project
While we make key management easy for our customers, we also wish to point out certain complications that are important to take into account in RFPs, project planning, and vendor selection. Addressing these issues can easily make or break the project.
Obsolete Software and Platforms
Legacy environments tend to contain all operating systems and incredibly old versions. For example, in 2016 we saw hundreds of servers running Sun Solaris versions from 1990's at a customer. Larger companies tend to have every Unix/Linux flawor, old Windows versions, IBM mainframes (z/OS or zLinux), and sometimes also IBM i series. Many large companies have many commercial SSH implementations from several vendors running in their Unix, Linux, and mainframe environments. On Windows alone, there are at least 15 different products actively used.
Complex User Identity Provisioning
User accounts may come from multiple sources, such as Active Directory, NIS (Network Information Service), and local accounts. Many companies have several AD trees and numerous NIS domains. Many companies use products like Centrify or Quest Vintela for AD bridging.
Key management must integrate into provisioning systems, ticketing, privilege elevation, analytics, and SIEM systems used in the enterprise.
Keys in NFS Home Directories
Key locations are commonly configured in configuration files. Keys may reside on NFS (Network File System) volumes that do not allow root access (root_squash option). Number of hosts times number of users on them may become prohibitively large, leading to combinatorial explosion or scans that seem to run forever. Updating the account on one host may affect thousands of other hosts, confusing the key management system. Such problems may not be noticed until late in the project - possibly after millions of dollars have been spent.
Understanding the authorized_keys File
The authorized_keys file ont an SSH server lists the known public keys that can be used for logging in as a particular user.
Authorized_keys files contain options that are important to interpret for properly understanding the keys. The options have changed between versions. For example, from-stanzas are important for reducing risks damage due to stolen/copied keys, and some organizations mandate their use for most keys.
Installing and repeatedly upgrading agents on tens of thousands of servers can be very expensive and time-consuming. Agentless approach is preferable in many cases.
Multiple SSH Implementations, Custom Builds, and Unusual Key Locations
Many organizations use multiple SSH implementations, such as Tectia SSH, OpenSSH, SunSSH, F-Secure SSH, Attachmate Reflection for Secure IT, from Windows to z/OS. Configuration files must be understood for each implementation to automatically discover key locations and prosess their key formats. Surprisingly many organizations also use custom-compiled versions that have non-standard paths compiled in for SSH keys. SELinux on RedHat brings its own complications.
Keys may be stored in LDAP, especially in elastic cloud and DevOps environments. SELinux is commonly enabled, and it may not be possible to directly read authorized_keys files.
Most enterprise IT organizations are structured around application teams, and associating keys to application teams and maximizing the efficiency of working with application teams impact the cost of a key remediation project by millions of dollars.
Large environments have all possible misconfigurations. We have seen authorized keys files containing garbage, corrupt password files, hung machines (that must not stop scans), faulty hardware, and sudo to service accounts requesting ticket numbers interactively before executing provided commands.
Cloud and DevOps Transition
As organizations transition their information systems into cloud environments and adopt agile DevOps processes, the role of SSH keys changes. Many containers will ideally not have SSH keys, through in reality they are often required to provide developer access or diagnostics capabilities.
For short-lived instances, particularly in elastic services, it does not make sense to store SSH keys on the instances. The ability to manage keys using LDAP becomes important.
Proper control of software deployment in DevOps requires auditability of who deploys what in production, how the continuous integration workflow is maintained, and how developers access production instances for debugging purposes. We have implemented many such projects using CryptoAuditor and Universal SSH Key Manager.
Users Circumventing Privileged Access Management
It is common for system administrators to use SSH keys for circumventing privileged access management systems. For example, we have seen Oracle DBAs installing their personal SSH key on various Oracle service accounts to go around jump servers.
CryptoAuditor works in conjunction with the Universal SSH Key Manager to enable full session recording and privileged access management of SSH and Windows Remote Desktop connections transparently, on the network level, without forcing users to go through jump servers. It enables transparent audit, analytics, and provides session recordings as videos for internal audit and forensics investigations.
Preventing SSH Tunneling
SSH tunneling can be used for opening tunnels from the public Internet to the internal network. This can enable using SSH keys from the outside.
It is difficult to prevent SSH tunneling when the organization has servers in a public cloud and those servers must be accessed from the inside. CryptoAuditor provides an elegant solution for preventing SSH tunneling.