Skills Operations

Last updated: 2026-05-17

Use this runbook to deploy Skills, confirm data collection readiness, and avoid leaking secret material. Current operating scope is development only.

Safety Rules

• Never print PATs, passwords, Cognito tokens, .env values, or raw deployment logs that could contain credentials.
• Treat AWS Secrets Manager values as secret-bearing even when checking whether a secret is ready.
• Enable collector schedules only when the target environment has an approved Azure DevOps PAT secret.
• Production is parked for now. Do not deploy, smoke, or troubleshoot prod unless the team explicitly reactivates it.

Safe Secret Readiness Check

This pattern checks whether a secret still has the scaffold replacement marker without printing the secret:

aws secretsmanager get-secret-value \
  --secret-id "$SKILLS_PAT_SECRET_ID" \
  --query SecretString \
  --output text |
node -e 'let input="";process.stdin.on("data",(c)=>input+=c);process.stdin.on("end",()=>{process.stdout.write(input.includes("REPLACE_WITH_APPROVED_AZURE_DEVOPS_PAT")?"placeholder\n":"non-placeholder\n")})'

Use AWS_PROFILE and AWS_REGION for the intended account before running the check.

Deploy

Development deploy with collector schedules enabled requires the existing approved dev secret ARN:

AWS_PROFILE=iDPCC-DEV-New \
AWS_REGION=us-east-1 \
ENVIRONMENT_NAME=skills-idpcc \
COGNITO_DOMAIN_PREFIX=skills-idpcc-dev-852507783007 \
AZURE_DEVOPS_PAT_SECRET_ARN="$DEV_SKILLS_PAT_SECRET_ARN" \
DASHBOARD_DOMAIN_NAME=skills.idpcc.ceirr-network-dev.org \
DASHBOARD_HOSTED_ZONE_ID=Z05416731M9AYVKD2JHC \
scripts/deploy-skills-aws.sh

The deploy script defaults to the existing development CloudFormation stack. Override STACK_NAME only in private operator notes or local shell history when targeting a different stack.

Production is intentionally omitted from the normal deploy path. The existing production stack is parked with schedules disabled; do not spend time deploying or validating it during dev-only operation.

Manual Collector Run

After a development deploy with an approved secret, invoke the collector once before trusting the schedule:

aws lambda invoke \
  --function-name "$SKILLS_COLLECTOR_FUNCTION_NAME" \
  --payload '{}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/skills-collector-result.json

Then run the recommendation function:

aws lambda invoke \
  --function-name "$SKILLS_RECOMMENDATION_FUNCTION_NAME" \
  --payload '{}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/skills-recommendation-result.json

The Lambda responses should contain counts and status fields only. If an invocation fails because of source access or secret readiness, disable schedules before finishing the ship.

Smoke Checks

Run these checks after each deploy:

curl -fsSI https://skills.idpcc.ceirr-network-dev.org/
curl -fsS https://skills.idpcc.ceirr-network-dev.org/data/skills.json

When the CloudFront default distribution domain is known, confirm it redirects to the meaningful hostname:

curl -sI "https://$SKILLS_CLOUDFRONT_DOMAIN/index.html?host-check=1" | sed -n '1,8p'

Expected result: 301 with a Location header on the skills.idpcc.ceirr-network-dev.org hostname.

Rollback

• Static-site rollback: redeploy from the previous merge commit and invalidate CloudFront.
• Collector safety rollback: redeploy without AZURE_DEVOPS_PAT_SECRET_ARN, or set COLLECTOR_SCHEDULE_STATE=DISABLED and RECOMMENDATION_SCHEDULE_STATE=DISABLED.
• CloudFront hostname issue: remove the custom-domain parameters only for an emergency test stack; the active dev hostname should normally stay on the meaningful URL.

Record every rollback or schedule-state change in Skills Status.

Note: the CloudFormation stack names still carry the original pre-rename spelling so existing Cognito users, DynamoDB data, and CloudFront resources are updated in place instead of recreated. Keep those names in operator commands, not product-facing copy.