Machine Identities and Secrets

Keeping Secrets Is Hard

Tom McNamara

May 14, 2021

When it comes to security and privacy in our digital economy, people and businesses rely on many different techniques and tools. The most prolific of these for people is the password. It's a secret that we create, and it’s supposed to keep our digital information secure. But sometimes that doesn’t work out too well. Once a password loses its secrecy we may not even know it, and then the security that it provided vanishes and leaves us vulnerable. A loss of secrecy makes our digital information seem secure when it's really not.

For many years, the mainstay of secrets of all types has been entropy and mathematics. Security professionals know all about entropy; many believe that it is the most important quality of a strong secret. And there is merit to this view, the theory and the math suggest that the more random and long the secret the greater it’s security strength. 

Another type of secret that is becoming more prevalent in our digital economy are encryption keys, and keys that control access to the machines and workloads that run in the cloud. These secrets are easy to identify. They are usually a long string of random characters. The growth of the cloud is creating many more of these types of secrets than there are human passwords.

But, whether the secret is a password or a key, its promise of security is built on the assumption of secrecy. In fact, a secret’s security strength is only as good as their secrecy. And both passwords and machine secrets suffer from the same problem: leakage.


Leakage occurs when a secret is not protected from disclosure and its secrecy is lost. In some cases it is lost slowly like water erodes the earth, and in other cases it is a dramatic instantaneous loss. Leakage can occur through many ways, including theft, sharing, improper storage, exposure during transport, and human error. Any of these causes for leakage are likely to occur at some point in time, and it doesn’t matter how strong the digital secret may be, its secrecy will eventually be lost. It's guaranteed. 

And it doesn’t matter if the digital secret is for a human or for a machine. Many of the avenues to leakage occur for secrets in the cloud as well as for human passwords. For example, “ghost” secure shell (SSH) accounts may exist without a business’ knowledge or control and they put its critical infrastructure at significant risk because an intruder can scan file for them and then misuse the elevated privileges they provide. And for APIs, developers often embed API keys in their code to speed development. But if they are left in production code, configuration files, or environment variables, then attackers can easily identify them there and misuse them. 

Vaults Reduce Leakage

One method to block several paths for leakage is to remove machine secrets from workloads, configuration files, and environment variables and store them in an encrypted digital vault. If you use a Password Manager or digital wallet, then this concept is very familiar to you. Secrets Managers exist for storing secrets for machines and workloads that operate in the cloud or within large enterprises with their own data centers. They limit access and provide secure interfaces and code tools (such as APIs) to let authorized workloads access the vault only when they need the secret (e.g. and API key). Secrets Managers typically include a secure digital vault that must be unsealed for use (to add a secret or change a secret) and resealed by business employees with appropriate authorizations. Secrets are often stored as as “key-value pairs” (i.e., keyName=keyValue). And with the proper SM credentials a workload can programmatically gain access to the vault and the stored secret that it needs using an API. While vaulting improves secrecy because it reduces the potential for leakage with stronger storage protection, it also creates a new problem. There are now new secrets to protect: one for unsealing the vault and one for API that the workload must use to retrieve the vaulted secret. This adds two new possibilities for leakage, and those new secrets must also be securely stored in yet another vault. (You can guess where this is headed.) Some Secrets Managers approach this problem by “sharding” the vault secret and distributing the pieces among several privileged users. While this seems secure it actually increases the attack surface area because it increases the number of humans involved (often the weakest security element) and it increases the number of separate vaults needed to store a sharded key.

But there is one particular characteristic of secrets that makes them most vulnerable to leakage: they’re static. Even vaulted secrets have a severe security problem because they seldom, if ever, change. If leakage occurs from human error or vault misconfiguration, or unauthorized vault access, or when the secret is used outside the vault, then secrecy is lost. Static secrets are a main reason why attackers are so successful in breaking through security. 

Rotation to the Rescue

The easy fix for static machine secrets is to automatically rotate them so they change frequently and become invulnerable to leakage. Secrets Managers have begun to implement ways to replace secrets in their vaults on a time schedule so they have a short useful life. One way they do this is to treat static secrets as a ‘lease’ that must be renewed before the lease expires. Secrets with leases are still static - they have to be configured to rotate, but they have additional parameters such as Time-To-Live (TTL) that lets developers control their rotation. Secrets rotation prevents potential damage from leakage, but rotating a secret in a vault must also consider how the workload endpoints that need the secret actually get the right secret at the right time to successfully authenticate the workloads. Within a single cloud, such as AWS, synchronization is easier because the AWS Secrets Manager has access to internal AWS cloud endpoints. But it is very difficult to synchronize secrets at workload endpoints at operational cloud scale and across clouds (e.g Azure to AWS). Most Secrets Managers that rotate secrets across cloud boundaries have to schedule the rotation for maintenance periods when machine services would be offline. 

Better Secrets Rotation 

What Secrets Managers really need is a way to rotate a secret naturally without a lot of developer configuration each time one is needed (i.e., avoid the DevOps burden to configure lease times and TTLs). And even better, the workload endpoints that needed the rotated secret would get it instantly and at the same time so their work would not be interrupted. And still better, the solution would work across cloud boundaries; so a workload endpoint in AWS could interoperate with one in Azure just as if they were in the same cloud.

To realize this vision of better secrets rotation and management, we innovated a different approach to secrets management. Our patent pending technology uses dynamic ephemeral secrets that self-rotate and self-synchronize. The secrets are built from dynamic Internet elements that change frequently and naturally. Our solution also uses  “sidecar” code (installed with each workload) to enable and manage secrets synchronization with the hopr secrets manager. This improves usability and speed for DevOps and improves security, too. 

Keeping secrets is hard because they have to stay secret to deliver security. And for machines and workloads the consequences to lost secrecy can ripple through the entire business and bring down many digital operations in an instant. As a tech innovator hopr works with leading cloud security practitioners to ensure digital secrets and machine identities remain secure and operational.