Skip to content

LearnCloud

AWS secrets in Lambda config and user data

Brief

A secret written into a function's environment, an instance's user data or a task definition is readable by every principal allowed to describe that resource, which on most accounts means anything holding the ReadOnlyAccess policy. The review sweeps those places in every region, records the secret's name and location and never its value, and ranks functions by how many secrets each one exposes.

Track
Cloud
Stage
Escalation
Reviewed
On this page
  1. Check first
  2. AWS
  3. The defender sees
  4. Done when
  5. Not covered
  6. Tools
  7. Related research
  8. Where this page fits
  9. Reviewed

All commands

AWS book Chapter 7 of AWS Identity and Secrets, "Secrets in configuration: functions, user data, parameters", goes further than this page.

Situation
A read-only review role in each account in scope with ReadOnlyAccess or SecurityAudit plus Lambda, EC2, ECS, SSM and CloudFormation read, or the identity another finding gave us, with rules of engagement that say which accounts and regions we may sweep. Every command here describes a resource; nothing is invoked, started, updated or decrypted. Where a command would return a value, the query or the filter cuts it to names, and only the accounts and regions the scope letter names are read.
Goal
An exposure table: each secret by name, the function, instance, template, task definition, document or stack it sits in, the region, and the count per resource that orders the rotation list.
Outputs
Captured on 8 Nov 2026 (UTC) in the Atarus lab AWS Organization from the Atarus Bench edition 2026.11-cloud-0.3, AWS CLI 2.39.3 and jq 1.7.1, with account ids and other identifiers replaced by stable placeholders. Every step on this page is captured, in a member account through the lab's read-only review role atarus-review. The lab seeds one region, so the regional calls name it, and every planted secret is a canary that never prints.

Check first

  • The regions enabled in each account, from aws ec2 describe-regions; functions, instances, templates and documents are regional.
  • The client's secret store standard, Secrets Manager or Parameter Store SecureString, which a plain value is measured against.
  • Which of lambda:GetFunctionConfiguration, ec2:DescribeInstanceAttribute, ecs:DescribeTaskDefinition and ssm:GetDocument our identity holds, from the first hour.
  • The rule, agreed in writing: names and locations go in the notes, values go nowhere, and anything seen by accident is reported and rotated.
  • Where CloudTrail is delivered, to match the sweep's reads afterwards.

Steps

AWS

Which functions hold secrets in their environment, and how many each?

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  aws lambda list-functions --region "$r" --output json 2>/dev/null \
    | jq -r --arg r "$r" '.Functions[] | [$r, .FunctionName,
        ((.Environment.Variables // {}) | keys | length),
        ((.Environment.Variables // {}) | keys | map(select(test("pass|secret|token|key|cred|pwd"; "i"))) | length)] | @tsv'
done | sort -t$'\t' -k4,4nr
aws lambda get-function-configuration --region us-east-1 --function-name orders-sync \
  --query '[KMSKeyArn, keys(Environment.Variables || `{}`)]'

Expected output

us-east-1	orders-sync	6	3   # <- three secret-named variables in one function: first on the rotation list
us-east-1	report-mailer	4	1
us-east-1	image-resize	2	0
us-east-1	order-webhook	0	0
[
    null,   # <- no customer managed key: encrypted at rest with the Lambda service key, returned decrypted to any reader
    [
        "SMTP_PASSWORD",
        "PAYMENTS_API_KEY",
        "QUEUE_URL",
        "DB_HOST",
        "LOG_LEVEL",
        "DB_PASSWORD"
    ]
]

The sweep found four functions, all in the one region the lab seeds, and order-webhook has no environment at all. 2>/dev/null keeps out the refusals from regions the lab's service control policy (SCP) denies; a region that refuses is unswept, not clean, so we run the loop once without it and note those regions. This loop and the launch template loop below use $'\t', which is bash syntax that some sh shells lack; the lab ran both under bash -o pipefail. Both calls return the environment with its values, so neither runs here without a query or jq filter that keeps only the keys. The count of secret-named variables per function is the ranking, and a payment key outranks a log token whatever the count. KMSKeyArn of null means the service key; a customer managed key narrows who can read the values through the API to principals its key policy lets decrypt, which the service key does not. A value the console's encryption helpers wrapped shows as ciphertext, the one case where a value in a variable is not a finding. The name match is a heuristic: a function whose names look clean but whose purpose is a database sync gets read by hand, keys only.

Negative result. Zero in the fourth column for every function in every region, and KMSKeyArn set on functions that handle regulated data.

Fix. Store the value in Secrets Manager or as a SecureString parameter and fetch it at initialization, through the SDK or the Parameters and Secrets Lambda extension, leaving only the secret's name in the environment. Encrypt what remains with a customer managed key whose policy names the function's role, and grant lambda:GetFunctionConfiguration, lambda:GetFunction and lambda:ListFunctions as you would secretsmanager:GetSecretValue: all three return the same values.

Which instances, launch templates and launch configurations carry secrets in user data?

aws ec2 describe-instances --region us-east-1 --query 'Reservations[].Instances[].InstanceId' --output text
aws ec2 describe-instance-attribute --region us-east-1 --instance-id i-0abc1234def567890 --attribute userData \
  --query 'UserData.Value' --output text | base64 -d \
  | grep -Eion '[A-Za-z_]*(password|passwd|secret|token|api[_-]?key|access[_-]?key)[A-Za-z_]*='
aws ec2 describe-launch-template-versions --region us-east-1 --launch-template-name web-tier \
  --query 'LaunchTemplateVersions[].[VersionNumber,DefaultVersion,LaunchTemplateData.UserData]' --output text \
  | while IFS=$'\t' read -r v d ud; do printf '%s\t%s\t%s\n' "$v" "$d" \
      "$(printf '%s' "$ud" | base64 -d 2>/dev/null | grep -Eioc '(password|secret|token|api[_-]?key)[A-Za-z_]*=')"; done
aws autoscaling describe-launch-configurations --region us-east-1 \
  --query 'LaunchConfigurations[].LaunchConfigurationName' --output text

Expected output

i-0abc1234def567890	i-0aaaa000000000001
14:DB_PASSWORD=   # <- line 14 of the boot script sets the database password in the clear
31:PAYMENTS_API_KEY=
3	True	0
2	False	2   # <- the current version is clean; version 2 still holds two secrets and is one launch away
1	False	2

The user data read runs once for each instance id the first call prints; the block shows it for i-0abc1234def567890. The template loop prints each version, whether it is the default, and how many secret-named variables its user data sets. The launch configuration call prints nothing: AWS stopped letting new accounts create launch configurations, so the lab has none. User data is a boot script stored unprotected: DescribeInstanceAttribute returns it to any principal allowed the call, and the instance reads it from the metadata service without a credential. The pattern prints the line number and variable name up to the equals sign; -o drops the value. Launch templates keep every version until one is deleted, so a secret removed from the latest version lives on in older ones, and an Auto Scaling group pinned to $Default or a number launches whatever that version says. On an older account, launch configurations carry UserData in the describe call and are read with the same filter.

Negative result. No match in any instance's user data, zero on every template version, and no launch configuration still in use.

Fix. Have the boot script fetch its secrets from Parameter Store or Secrets Manager with the instance role, scoped to those names alone. Create a new template version without the values, set it as default, delete the old versions, and replace launch configurations with templates. Scope ec2:DescribeInstanceAttribute and ec2:DescribeLaunchTemplateVersions to the operators who need them, and require IMDSv2 so a request forgery on the host cannot read the script.

Which ECS task definitions pass secrets as plain environment?

aws ecs list-task-definitions --region us-east-1 --status ACTIVE --query 'taskDefinitionArns[]' --output text
aws ecs describe-task-definition --region us-east-1 --task-definition orders-api:1 \
  --query 'taskDefinition.containerDefinitions[].{container:name,plain:environment[].name,referenced:secrets[].name}'

Expected output

arn:aws:ecs:us-east-1:111122223333:task-definition/orders-api:1
[
    {
        "container": "api",
        "plain": [
            "DB_HOST",
            "DB_PASSWORD",   # <- a value in the task definition, readable by anyone with ecs:DescribeTaskDefinition
            "SESSION_SIGNING_KEY"
        ],
        "referenced": [
            "PAYMENTS_API_KEY"   # <- the right shape: a reference, resolved by the agent at start
        ]
    }
]

environment holds literal values and is returned in full by describe-task-definition, so the query keeps names only. secrets holds references whose valueFrom is a Secrets Manager or Parameter Store ARN, resolved by the ECS agent at start, and a name there is the fix already applied. Every revision of a family stays readable, so a secret moved to secrets in a new revision is still plain text in the older ones until they are deregistered and then deleted, on Fargate and EC2 alike.

Negative result. Every environment name is a host, port, flag or queue URL, and every credential is under secrets.

Fix. Move each credential to secrets with a valueFrom ARN, grant the task execution role read on those ARNs and the key that protects them, deregister and delete the revisions that still carry values, and rotate what they carried.

Which SSM documents and CloudFormation stacks spell a secret out?

aws ssm list-documents --region us-east-1 --filters Key=Owner,Values=Self \
  --query 'DocumentIdentifiers[].[Name,DocumentType]' --output text
aws ssm get-document --region us-east-1 --name patch-and-join --query Content --output text \
  | grep -Eion '[A-Za-z_]*(password|secret|token|api[_-]?key)[A-Za-z_]*(=|":)'
aws cloudformation describe-stacks --region us-east-1 \
  --query 'Stacks[].{stack:StackName,params:Parameters[].ParameterKey,masked:Parameters[?ParameterValue==`****`].ParameterKey,outputs:Outputs[].OutputKey}'

Expected output

patch-and-join	Command
1:DOMAIN_JOIN_PASSWORD=   # <- a domain credential in a document any ssm:GetDocument holder can read
[
    {
        "stack": "orders-db",
        "params": [
            "DbPassword",
            "InstanceClass",
            "DbUser"
        ],
        "masked": [],   # <- DbPassword has no NoEcho: its value is in every describe-stacks response and in the console
        "outputs": [
            "Endpoint",
            "AdminConnectionString"   # <- an output named like a credential: read the template, not the value
        ]
    }
]

An owned SSM document is a script the client wrote, and a join password inside it is as readable as user data. The lab's document is JSON on a single line, so the match reports line 1; in a YAML document the number points at the line itself. A CloudFormation parameter without NoEcho comes back as its literal value from DescribeStacks and sits in the stack's history, and an output can carry anything a template author chose to export, so a name such as AdminConnectionString sends us to the template body, never to the output value. Credentials held in plain String parameters belong to the secrets and KMS review. Both calls return values beside the names, which is why each query drops the value column.

Negative result. No match in any owned document, and every credential parameter listed under masked.

Fix. Pass credentials to documents and templates through SecureString parameters and Secrets Manager dynamic references rather than literals, mark every credential parameter NoEcho, remove outputs that carry secrets, and rotate every value found. Treat ssm:GetDocument and cloudformation:DescribeStacks as reads of the secrets they return when scoping roles.

The defender sees

Every call on this page is a CloudTrail management event under the review identity, written in the region it was made: ListFunctions20150331, GetFunctionConfiguration20150331v2, DescribeInstances, DescribeInstanceAttribute with userData in its request parameters, DescribeLaunchTemplateVersions, DescribeLaunchConfigurations, ListTaskDefinitions, DescribeTaskDefinition, ListDocuments, GetDocument and DescribeStacks, plus one DescribeRegions. CloudTrail records the reader, not the response. The shape to alert on is one identity describing every function or every instance's user data in every region inside a few minutes, which no deployment does. GuardDuty's Discovery:IAMUser/AnomalousBehavior may follow such a sweep when the identity has no history of these calls; CredentialAccess:IAMUser/AnomalousBehavior watches GetSecretValue and GetPasswordData, which this review never calls, so a sweep that reads secrets through configuration passes under it. If the client's monitoring saw a review role read every function configuration and raised nothing, that is a finding beside the exposure: the same reads from a stolen key would have looked the same.

Done when

Every region in scope is swept, every function, launch source, task definition, owned document and stack is classified as clean or listed with its secret names, the per-resource counts are written, and the identities able to read each location are named for the fix.

Not covered

Secrets Manager, Parameter Store and KMS policies (see AWS secrets and KMS key policy review), secrets in deployed code or container images, and CI variables (see CI pipeline secrets and OIDC trust review).