[remark] Passwords in deployment or development scripts?

by Ciprian Dorin Craciun (https://volution.ro/ciprian) on 

Questions (without definitive answers) about how to securely manage secrets in scripts and development environments?

// permanent-link // Lobsters // HackerNews // index // RSS

Also on Lobste.rs

This is a transcript of the question with the same title I've posted on Lobste.rs.

I republish it here mainly for archival (and discovery) purposes, but also to highlight any interesting ideas that emerged from that discussion thread.

If you have a different take than what was discussed (especially a non-mainstream one) email me as described in the contact section below.

This topic is also in relation with my latest open-source tool: z-tokens.


Before laying out my question, let me put it into context. It seems that lately attackers are more successful by targeting administrators or developers than attacking the actual deployments and servers. Meanwhile some attacks social engineer employees into credential theft or reset, other attacks go after operational credentials that are used in semi-automated operations or development scripts.

To be clear, I don't want to focus on the issue of secure online credential management (I hope we all use some form of password manager), or the security of online authentication protocols (even OAuth seems to be lacking), or the security of MFAs, not even the security of cloud API keys that are actively used by unattended running services (if the attacker has compromised the actual server he most likely already has access to all the sensitive data he wants).

What I want to focus on is those credentials that are used mainly for semi-automated operational or development scripts. Like for example AWS secret keys used to deploy or to gain access to the underlying buckets or backups; or passwords used to access the SQL database for maintenance; or other such secrets that are used outside the browser or specialized applications like SSH.

Also, let's assume we are speaking in the context of a small company with at most 10 employees. Large companies probably have 10 people on the security team tasked only with this topic, thus perhaps this problem is solved in more complex ways.

So, how do the administrators and developers manage such secrets when writing semi-automatic scripts?

I'm interested especially on the following aspects:

Sorry for the lengthy question, but I do wonder about this for some time...


There have been more answers, but the following are the ones I considered the most interesting or insightful ones.

[See this Lobste.rs comment thread started by github.com/mac-chaffee.]

I have a bone to pick with secret management strategies.

IMO there are two options for secret management: either use a hardware key as your root of trust for everything, or lie to yourself. No one does the former perfectly, so everyone's doing the latter.

Developers are constantly executing arbitrary un-sandboxed code on their computers due to dependencies, so you can just assume any file and any block of memory on your computer can be stolen without you knowing. As a result, we can conclude:

Any combination of those tools still depends on some root of trust that an attacker can exfil. The only solution is a key stored in hardware that physically can't be exfil'd. Use something like PKCS#11 to send challenges to your hardware key to access everything.

This assumes a perfect world where you have no legacy systems of course, but I'm shocked at the amount of engineering time thrown at alternate solutions that just add complexity.

[This was one of my replies.]

Let's take their example from https://developer.1password.com/docs/cli/secrets-config-files/:

## config.yml.tpl
    host: http://localhost
    port: 5432
    username: op://prod/mysql/username
    password: op://prod/mysql/password
op inject -i config.yml.tpl -o config.yml

Now your secrets are in plain on the disk inside config.yml. It might be excluded from Git, or you might delete it afterwards, but if the attacker gets into your laptop, he can just dd if=/dev/sda | tr -c ' -~' '\n' | grep -C 100 -F -e 'username:' and now he can start looking at older versions of these files.

The only protection against this type of attack is to make sure that the config.yml is stored on a ramfs (which in Linux doesn't leak in swap) or in tmpfs with encrypted swap.

The safer alternative with this tool is to use the environments approach:

op run --env-file="./prod.env" -- aws