So you have bought your expensive and extensive Privileged Access Management (PAM) solution. Controlling the access of users who deal with the most valuable information in your organization is generally a good idea. Now you are convinced that the access controls of your system administrators, database administrators, M2M connections and DevOps teams are securely in place. Unfortunately, we have bad news for you. PAM can be bypassed.
Here are the top 5 ways privileged users are bypassing PAM systems today:
1. System admin fast-track PAM flanking
Your system administrators are in a highly privileged position for a reason: they know what they are doing. Because of their exceptionally powerful access, a PAM solution is introduced to monitor and log sys admin access. And then they do this:
- Your system administrator logs in to the target server though PAM as she is supposed to.
- Once logged in, she places a root SSH Key directly on the target server.
- The next time the sys admin can make the connection directly from her laptop to the critical database, and your PAM is blissfully ignorant of this. Why? The traffic in the SSH tunnel is encrypted and therefore invisible to your PAM. Sounds a lot like a backdoor access, doesn’t it?
Let me take this opportunity to shortly introduce the SSH protocol, since it is the primary circumvention method:
“The SSH protocol (also referred to as Secure Shell) is a method for secure remote login from one computer to another. It provides several alternative options for strong authentication, and it protects the communications security and integrity with strong encryption.”
Tatu Ylönen, founder of SSH.COM and inventor of the SSH protocol
System administrators can self-provision permanent SSH key-based access - without proper policies, processes, or oversight. It doesn’t mean they are evil or arrogant. It often just means that they want to get the job done fast and choose the path of least resistance. In addition to being an insider risk, self-provisioned and unmanaged keys are also non-compliant and an audit failure point.
2. Legacy keys that existed prior to PAM deployment
Many PAM solutions actually handle SSH Keys. They have password vaults where SSH Key secrets are stored, and when someone needs privileged access, they get the credentials (and a tagged SSH Key) from the vault to establish the secure connection. These keys (and their associated credentials) are rotated so that the same key cannot be used twice in two different sessions.
Here’s the but: someone from your IT team would need to register all your SSH Keys to the password rotation process, and we are talking perhaps about tens of thousands of servers with hundreds of thousands of SSH Keys in big enterprises!
At the same time there is pressure on IT staff to push out new and shiny stuff and perhaps not to worry too much about the legacy environment. This often leads to a situation where legacy SSH Keys are simply not incorporated into the vault at all and simply continue to exist and provide access outside the PAM solution.
3. M2M connections that are not vaulted
So far, we have only discussed interactive uses of SSH Keys, which actually covers the vast minority of all SSH-based access. 80% of all SSH connections are actually machine-to-machine (M2M) and mostly automated. Automation poses a big problem for vault-based systems: if the idea of a vault is that you assign a password to an SSH Key that opens the secure connection, who is going to enter that password in connections between two servers? In practice, this means that all these keys are not registered in a vault. It is possible to configure a script that automates fetching the credentials, but this defeats the entire purpose of a vault, since the secrets would exist outside it!
4. 3rd party access with SSH Keys
What can happen with your own sys admins, can happen with 3rd parties, meaning that privileged access to your mission-critical data can move entirely outside your company.
Let’s say that you have hired an outsourced team that is located in different countries. They need to upgrade your payment systems and some of those databases contain credit card information. The fastest and easiest way to grant access is by using self-provisioned SSH Keys.
Based on our findings, this happens all the time and system administrators even share their own keys to accelerate the process. In addition to this being another case of PAM bypass, crucial information that should be always be logged and auditable, including key strength, validity, ownership and purpose, is often missing. It means that you might have 3rd party employees accessing a database with credit card information without a record of who was accessing what!
You think this is far-fetched? Think again!
5. DevOps: from development to production using SSH
In the frantic world of DevOps, speed is the key. Any extra step that adds friction often meets ideological opposition. That is why the same SSH Key might sometimes be used both in the test environment and the production environment. This setup increases the risk of accidents and basically invalidates the idea that the test environment is your safe zone where you can go crazy.
There is a reason the concept of segregation of duties (SoD) was developed. It states that one person should not have access to both the development and production environments. At least the account should be different in dev and production. But with SSH Keys, a developer might have the same level of access even if she has a different role in two environments (like "admin" in the development environment and "user" in production). More food for thought here.
Why is PAM bypassed?
Based on our experiences, the following five reasons can be highlighted:
- Time constraints. Key persons in organizations are often under tremendous pressure to be efficient and they cut corners to meet their targets.
- Complexity. Connections based on SSH Keys often build a complex web of dependencies. Deleting just one key might mean a major disruption in numerous critical transactions.
- Not understanding the scope and prevalence of unmanaged SSH Keys.
In the case of one representative customer, we analyzed 500 business applications, 15000 servers, and found three million SSH Keys that granted access to live production servers. Of those, 90% were no longer used. Root access was granted by 10% of the keys
- No tools to detect SSH Keys that bypass PAM. Many companies might not even know that they have this issue, let alone have the tools or expertise to address the problem.
- Convenience. We all like things that are convenient. And that’s exactly what PAM bypass often is.
Talk to us about PAM bypass
Do you want to address PAM bypass in your organization? Talk to us. Our expertise in understanding how big enterprises can prevent PAM bypass, solve their SSH Key management issues and reduce the complexity of daily routines in access administration is simply unrivalled. We are the company that invented the SSH protocol.
To get started on your journey towards reducing insider risk, gaining compliance and taking better control of your own business, we recommend taking a look at KuppingerCole's Executive Report on PrivX®. PrivX is a lean and agile PAM solution for multi-cloud and hybrid environments, and it solves many privileged access issues in smarter way than many incumbent PAM solutions.
Miikka guides the software architecture and development at SSH. He has over 20 years of experience in IT industry, building teams and developing products in startups and large enterprises.