[remark] Passwords in deployment or development scripts? -- Volution Notes

Also on Lobste.rs

This is a transcript of the question with the same title I've posted on Lobste.rs.

I republish it here mainly for archival (and discovery) purposes, but also to highlight any interesting ideas that emerged from that discussion thread.

If you have a different take than what was discussed (especially a non-mainstream one) email me as described in the contact section below.

This topic is also in relation with my latest open-source tool: z-tokens.

Question

Before laying out my question, let me put it into context. It seems that lately attackers are more successful by targeting administrators or developers than attacking the actual deployments and servers. Meanwhile some attacks social engineer employees into credential theft or reset, other attacks go after operational credentials that are used in semi-automated operations or development scripts.

To be clear, I don't want to focus on the issue of secure online credential management (I hope we all use some form of password manager), or the security of online authentication protocols (even OAuth seems to be lacking), or the security of MFAs, not even the security of cloud API keys that are actively used by unattended running services (if the attacker has compromised the actual server he most likely already has access to all the sensitive data he wants).

What I want to focus on is those credentials that are used mainly for semi-automated operational or development scripts. Like for example AWS secret keys used to deploy or to gain access to the underlying buckets or backups; or passwords used to access the SQL database for maintenance; or other such secrets that are used outside the browser or specialized applications like SSH.

Also, let's assume we are speaking in the context of a small company with at most 10 employees. Large companies probably have 10 people on the security team tasked only with this topic, thus perhaps this problem is solved in more complex ways.

So, how do the administrators and developers manage such secrets when writing semi-automatic scripts?

I'm interested especially on the following aspects:

where do you store the secrets? in plain-text files, protected with GPG or Age, or even Google Drive?
how do you share the secrets between employees? just throw the sensitive material in Slack or email? (can one actually safely remove something from Slack or GMail?)
how do you dispose of the secrets once they aren't needed (or have been exposed)? (because if they are in Git, good-luck with that...)
how do you keep track of what credentials are still required and in use? grep-ing the scripts code for random looking tokens?
how do you know who has access to what?
how often, and how do you actually rotate such secrets? (I admit I don't want to answer this question...)
how do you actually provide it to the scripts? via environment variables? via command line arguments? via world-readable files in /tmp?
is anyone actually using any of the PKCS#11-related tools out there?
what if you need to pass such a secret to a tool that runs on a remote system which you access via SSH? do you just scp it to /tmp?
do you use any existing tool for this purpose? or is everything built in house?

Sorry for the lengthy question, but I do wonder about this for some time...

Answers

There have been more answers, but the following are the ones I considered the most interesting or insightful ones.

[See this Lobste.rs comment thread started by github.com/mac-chaffee.]
I have a bone to pick with secret management strategies.
IMO there are two options for secret management: either use a hardware key as your root of trust for everything, or lie to yourself. No one does the former perfectly, so everyone's doing the latter.
Developers are constantly executing arbitrary un-sandboxed code on their computers due to dependencies, so you can just assume any file and any block of memory on your computer can be stolen without you knowing. As a result, we can conclude:
putting keys in files that decrypt other files is pointless;
using a local password manager that decrypts a local vault into memory is pointless;
using a cloud-based password manager that uses browser cookies (unencrypted files stored on disk) to authenticate you is pointless;
using short-lived tokens that refresh (e.g. OIDC/AWS STS) is pointless; (think an attacker can only strike outside of 9-5 when you're logged out?)
Any combination of those tools still depends on some root of trust that an attacker can exfil. The only solution is a key stored in hardware that physically can't be exfil'd. Use something like PKCS#11 to send challenges to your hardware key to access everything.
This assumes a perfect world where you have no legacy systems of course, but I'm shocked at the amount of engineering time thrown at alternate solutions that just add complexity.

[This was one of my replies.]
Let's take their example from https://developer.1password.com/docs/cli/secrets-config-files/:
## config.yml.tpl
database:
    host: http://localhost
    port: 5432
    username: op://prod/mysql/username
    password: op://prod/mysql/password
op inject -i config.yml.tpl -o config.yml
Now your secrets are in plain on the disk inside config.yml. It might be excluded from Git, or you might delete it afterwards, but if the attacker gets into your laptop, he can just dd if=/dev/sda | tr -c ' -~' '\n' | grep -C 100 -F -e 'username:' and now he can start looking at older versions of these files.
The only protection against this type of attack is to make sure that the config.yml is stored on a ramfs (which in Linux doesn't leak in swap) or in tmpfs with encrypted swap.
The safer alternative with this tool is to use the environments approach:
op run --env-file="./prod.env" -- aws

Also on Lobste.rs

Question

Links

Answers