Secrets Management for Go Services on AWS ECS: Getting Off .env in Production

Every team starts with .env files. Then someone rotates a database password in production, updates the ECS task definition manually, forgets to update the staging environment, deploys the wrong version, and triggers a 2am incident. Secrets management is the infrastructure practice that prevents that class of failure, makes audit trails possible, and removes plaintext credentials from your deployment pipeline. This is how we manage secrets for Go services running on AWS ECS, including runtime injection, rotation without restarts, and the access control patterns that make it maintainable.

Why do .env files fail in production environments?

The problems with .env files in production are not theoretical. They accumulate over time and create a specific failure mode: credentials drift.

When a credential is rotated, every .env file that contains it must be updated simultaneously. In practice, someone rotates the database password, updates production, deploys, and forgets staging. Three weeks later, a staging deployment pulls the old password and fails in a way that takes an hour to diagnose.

The second problem is auditability. .env files scattered across servers or checked into a private repository provide no audit trail. You cannot answer who changed which credential and when. In a regulated environment or a SaaS handling payment data, this is a compliance gap.

The third problem is accidental exposure. A debug log that prints environment variables, or a crash dump that includes process memory, exposes every credential in the .env. AWS Secrets Manager solves all three problems by centralizing secret storage, providing an audit trail through CloudTrail, enabling automatic rotation, and delivering secrets to ECS tasks at startup without the secrets ever touching your deployment pipeline.

AWS Secrets Manager versus Parameter Store: which one to use?

AWS provides two secret storage services with overlapping capabilities.

AWS Secrets Manager costs $0.40 per secret per month plus $0.05 per 10,000 API calls. It supports automatic rotation via Lambda functions, integrates directly with ECS task definitions as native secret references, has a dedicated SDK for programmatic access, and keeps up to 20 previous versions of each secret for rollback.

AWS Systems Manager Parameter Store is free for standard parameters up to 10,000 entries. SecureString parameters encrypted via KMS cost $0.05 per parameter per month. It also integrates with ECS task definitions but does not support automatic rotation natively.

For a SaaS backend handling database credentials, API keys, and payment provider secrets, Secrets Manager is the right choice. The automatic rotation support alone justifies the cost difference. For non-rotatable configuration like feature flags, service URLs, and environment labels, Parameter Store free tier is sufficient.

How do you inject secrets into ECS task definitions?

ECS supports native secrets injection from both Secrets Manager and Parameter Store. The secret value is retrieved when the task starts and injected as an environment variable inside the container. The secret never appears in the task definition JSON or in CloudFormation templates in plaintext.

Task definition JSON:

{
  "containerDefinitions": [
    {
      "name": "api",
      "image": "your-account.dkr.ecr.eu-west-1.amazonaws.com/api:latest",
      "environment": [
        { "name": "APP_ENV", "value": "production" }
      ],
      "secrets": [
        {
          "name": "DATABASE_URL",
          "valueFrom": "arn:aws:secretsmanager:eu-west-1:123456789:secret:prod/api/database-url-abc123"
        },
        {
          "name": "STRIPE_SECRET_KEY",
          "valueFrom": "arn:aws:secretsmanager:eu-west-1:123456789:secret:prod/api/stripe-secret-key-def456"
        }
      ]
    }
  ]
}

The ECS execution role needs secretsmanager:GetSecretValue permission scoped to those specific ARNs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": [
        "arn:aws:secretsmanager:eu-west-1:123456789:secret:prod/api/*"
      ]
    }
  ]
}

Scoping the permission to prod/api/* rather than * means a compromised execution role can only access secrets in that path prefix, not every secret in the account.

How do you consume secrets in Go at startup?

When secrets are injected as environment variables, consumption in Go is identical to reading any other environment variable:

type Config struct {
    DatabaseURL string
    StripeKey   string
    JWTSecret   string
}

func LoadConfig() (*Config, error) {
    cfg := &Config{
        DatabaseURL: os.Getenv("DATABASE_URL"),
        StripeKey:   os.Getenv("STRIPE_SECRET_KEY"),
        JWTSecret:   os.Getenv("JWT_SECRET"),
    }

    if cfg.DatabaseURL == "" {
        return nil, fmt.Errorf("DATABASE_URL is required")
    }
    if cfg.StripeKey == "" {
        return nil, fmt.Errorf("STRIPE_SECRET_KEY is required")
    }
    if cfg.JWTSecret == "" {
        return nil, fmt.Errorf("JWT_SECRET is required")
    }

    return cfg, nil
}

Validating required secrets at startup and exiting early prevents the application from starting in a broken state. A service that starts successfully but fails on the first database query is much harder to diagnose than a service that refuses to start with a clear error message.

How do you rotate credentials without restarting services?

The standard ECS deployment approach injects secrets at task startup, which means rotating a credential requires deploying new tasks. For a small ECS service, this is acceptable. For a large fleet or a service with strict uptime requirements, you need in-process secret refreshing.

AWS Secrets Manager supports automatic rotation via Lambda. When rotation occurs, Secrets Manager calls a Lambda function that updates the credential at the external system (generates a new PostgreSQL password, updates pg_hba.conf), writes the new value to Secrets Manager, and the next task deployment picks it up automatically.

For Go services that need to pick up rotated credentials without a restart:

type RotatingSecret struct {
    secretARN string
    client    *secretsmanager.Client
    mu        sync.RWMutex
    value     string
    lastFetch time.Time
    ttl       time.Duration
}

func (r *RotatingSecret) Get(ctx context.Context) (string, error) {
    r.mu.RLock()
    if time.Since(r.lastFetch) < r.ttl {
        val := r.value
        r.mu.RUnlock()
        return val, nil
    }
    r.mu.RUnlock()

    r.mu.Lock()
    defer r.mu.Unlock()

    // Double-check after acquiring write lock
    if time.Since(r.lastFetch) < r.ttl {
        return r.value, nil
    }

    result, err := r.client.GetSecretValue(ctx, &secretsmanager.GetSecretValueInput{
        SecretId: &r.secretARN,
    })
    if err != nil {
        return r.value, err // Return cached value on fetch error
    }

    r.value = aws.ToString(result.SecretString)
    r.lastFetch = time.Now()
    return r.value, nil
}

A TTL of 5 to 15 minutes balances freshness against API call costs. During rotation, the service picks up the new credential within one TTL window.

How do you structure secrets paths across environments and services?

A consistent naming convention prevents confusion when your ECS cluster runs multiple services across staging and production:

/{env}/{service}/{secret-name}

prod/api/database-url
prod/api/stripe-secret-key
prod/api/jwt-secret
prod/worker/database-url
prod/worker/redis-url
staging/api/database-url
staging/api/stripe-secret-key

This structure makes IAM policies clean (scope by environment and service prefix), makes secret discovery consistent when debugging, and makes rotation automation straightforward because the rotation Lambda can discover all secrets for a service by prefix.

For SaaS platforms serving multiple tenants, add a tenant namespace only for secrets that genuinely differ per tenant (API keys that the tenant provides). Shared infrastructure secrets like your own database URL should not be per-tenant.

Key lessons from production

Rotate secrets before you are forced to. The first time you rotate a credential on a SaaS platform under pressure (a suspected breach, a contractor offboarding, a compromised API key) is not the time to discover your rotation process is untested and manual. Run a rotation drill quarterly on non-critical credentials to validate the full path end to end.

Tag every secret with service, environment, and owner. Secrets Manager supports tags and those tags appear in cost allocation reports. A SaaS with 30 services and 150 secrets needs this tagging discipline to understand what costs what and who is responsible for rotating what.

Never log configuration at startup. The pattern log.Info("loaded config", "config", cfg) prints every field in the struct including credentials. Use a custom MarshalLog method that masks secrets fields.