OpenSSH: Key Management Needs Attention

What is OpenSSH?

SSH (Secure Shell) is a tool for secure system administration, file transfers, and other communication across the Internet or other untrusted network. It encrypts identities, passwords, and transmitted data so that they cannot be eavesdropped and stolen.

OpenSSH is an open source implementation of the SSH protocol. It is based on the free version by Tatu Ylonen and further developed by the OpenBSD team and the user community.

Tatu Ylonen founded SSH Communications Security to provide commercial support for enterprises, and the original version evolved into Tectia SSH. The commercial version also supports Windows and IBM mainframe (z/OS) platforms and includes full support for X.509 certificates and smartcard authentication (for example the CAC and PIV cards used by US government).

The open source version is delivered as source code or precompiled binaries under a BSD-style license. The project team provides no support services for end-users, but community-based support is available (on a voluntary basis) from various security web forums.

What is an SSH Key?

SSH provides an authentication mechanism based on cryptographic keys, called public key authentication. One or more public keys may be configured as authorized keys; the private key corresponding to an authorized key serves as authentication to the server. Typically both authorized keys and private keys are stored in the .ssh directory in a user's home directory. Fundamentally, such keys are like fancy passwords, only the password cannot be stolen from the network and it is possible to encrypt the private key locally (so that using it requires both a file and a passphrase only known to a user). However, in practice most keys are used for automation and do not have a passphrase.

The keys used for user authentication are called user keys. SSH also uses host keys for authenticating hosts. Together these are called SSH keys.

SSH keys replaced the insecure .rhosts authentication that was vulnerable to active network-level attacks. Replacing .rhosts improved the security of the Internet and enterprise information systems tremendously. However, the vast success of the SSH protocol combined with the fact that OpenSSH shipped free with most operating systems led to a lack of policies and oversight in relation to SSH keys. Consequently, auditors and IT security managers did not pay enough attention to SSH keys.

SSH keys provide the same level of access as user names and passwords, and typically to privileged accounts with operating system access. It is common that there are 10 times as many SSH keys as there are passwords. Yet SSH keys have been ignored in most identity and access management projects. They require the same kind of provisioning and termination processes and audit attention as passwords or any other authentication method.

What Risks Are Associated with SSH Keys?

SSH risks expose organizations to major security risks for a number of reasons:

  • Lack of provisioning, termination, and oversight processes and policies for SSH keys.
  • The number of existing legacy SSH keys seems to surprise everybody. In one customer case we found 3 million keys (750,000 distinct keypairs). In another case the customer found 4.5 million authorized keys. Most large enterprises seem to have at least hundreds of thousands of keys and don't know how many. Compromise of just one key is enough to gain illegitimate access.
  • Many keys are unused and represent access that was never properly terminated. In one typical major enterprise we have found that about 90% of all the SSH keys were not used during a six-month monitoring period.
  • Nobody knows who has the corresponding private keys. The keys can be kept offline.
  • SSH keys have typically never been changed. Like passwords, keys should be changed periodically (a key rotation period of six months is common).
  • SSH keys grant privileged access. In one typical major enterprise we found that 10% of all SSH keys granted root access.
  • Many recent malware packages have been collecting SSH keys.
  • SSH keys can be used to hide backdoors. Such backdoors are not likely to be ever discovered without SSH key management, and no-one can tell if it was a backdoor or a key that had been forgotten.
  • Given the vast number of keys, uncontrolled access graphs formed by the keys, and the fact that keys are used to copy data to disaster recovery sites and backup systems, they may be used to spread an attack throughout the server environment, including backup sites, in minutes in an automated fashion. This represents a major cyberwarfare and cyberterrorism threat.
  • Many publicized cybercrime incidents have involved rogue system administrators utilizing SSH keys and criminals outright buying SSH keys from insiders.
  • Improperly managed server keys expose hosts to network-level man-in-the-middle attacks.

NIST Guidelines for SSH Key Management

NIST published guidelines in late 2015 for managing SSH keys for both automated and interactive access as NIST IR 7966. The NIST guidelines explain the problem and establish a framework for addressing SSH key management.

SSH Key Management

OpenSSH presents several special challenges for SSH key management:

  • Common lack of policies and provisioning/termination processes. Users are commonly allowed to self-provision keys for themselves (and anyone they like).
  • The proprietary certificates supported by OpenSSH make auditing SSH key based access impossible. With them, you cannot tell which keys a server will accept by analyzing the server. Typically there are no reliable records or audits of which keys a certificate authority key has signed. We do not recommend the use of OpenSSH certificates for user authentication.
  • Key locations are commonly configured in configuration files.
  • Some organizations use custom-compiled versions that have non-standard paths compiled in for specifying the location of SSH keys.
  • SSH keys may be stored in NFS (Network File System) volumes and a change may affect many hosts.
  • User accounts may come from Active Directory, NIS (Network Information System), or LDAP (Lightweight Directory Access Protocol) servers.
  • Keys may be stored in LDAP.
  • SELinux is enabled in many common Linux distributions, and it may not be possible to directly read authorized_keys files.
  • Configuration file syntax and the meaning of certain options has changed many times between versions.
  • Options in authorized_keys files are important for understanding what access is provided and for restricting the use of stolen/copied keys using from-stanzas.

The only solution that is currently able to practically address the SSH key management problem with the full complexity of enterprise environments is the Universal SSH Key Manager from SSH Communications Security.

For more information on SSH key management and what it involves, please see the generic SSH key management page.

Information security starts by controlling who is given access to systems. The rest is just enforcing the access. If there is no knowledge of who has been given access using SSH keys, there is no security. No confidentiality, no integrity, no guarantee of continued operation.

What to Do with OpenSSH Proprietary Certificates?

Because OpenSSH's proprietary certificates for user authentication cannot be audited by inspecting the server and no trustworthy, production-quality OpenSSH certificate authorities are known to exist, we do not recommend using OpenSSH's proprietary certificates for user authentication. We do not see how they could pass a proper audit. NIST SP 800-53, PCI, Sarbanes-Oxley, HIPAA, and NERC CIP all require controlling who has access to what systems and data, and require access to be auditable. It is also fundamental to several of the 20 SANS critical security controls.

However, certificates for server authentication can be quite useful for ensuring all servers are properly authenticated, while avoiding the need to manage huge known hosts files.

Comparison to Tectia SSH

While the open source OpenSSH and commercial Tectia SSH implementations address the same underlying need for securing of logins, communications, and file transfers to hosts, there are some differences in their approaches.

Organizations requiring commercial support, 24x7 support, or smartcard authentication may wish to check out Tectia SSH.

The table below presents a concise comparison of these two leading SSH implementations.

Comparison - SSH and OpenSSH

User Interface Considerations - Command Line or Graphical

Due to its history and heritage in the open source operating system community and its availability on primarily Linux/BSD the default user interface to the OpenSSH functionality is the command line. The figure below present a typical view of an OpenSSH client started from a Linux command line.

OpenSSH command line user interface

The users of the open source version are familiar with working in this environment, and the project team has not seen the need for graphical user interfaces that are the norm outside the system administration and certain software development market segments.

SSH client users that require graphical user interfaces have the option to use one of the commercial or non-commercial SSH client software packages, many of which provide a platform native GUI.

OpenSSH Download

OpenSSH is available for source code download at the project website as well as on various operating systems and distribution-specific package management systems. Downloading and compiling the source code suits certain audiences (software developers and linux distribution developers). End users are advised to acquire OpenSSH via the operating system specific package managers, or to consider a commercially supported product such as Tectia SSH.

The project accepts contributions to the project via the OpenBSD Foundation.

OpenSSH Client - ssh

The OpenSSH client program is called ssh. The SSH client generally uses information in the .ssh directory in the user's home directory. It also reads /etc/ssh/ssh_config, which contains system-wide configuration for it.

OpenSSH Server - sshd

The OpenSSH server program is called sshd. The server is typically started during boot, and reads its configuration from /etc/ssh directory. Its main configuration file is usually /etc/ssh/sshd_config.