← Documentation index

Production-readiness Soak Plan

Goal: validate the deployed v1.0 stack against real audio for 2 hours, with deliberately injected failures, before declaring the system event-ready.

Run this: AFTER Phase 1 + Phase 2 are deployed, BEFORE the deploy-frozen final week (2026-06-29 → 2026-07-01).

Total wall-clock: ~2.5 hours including setup and post-mortem.

Pre-flight (15 min before starting)

[ ] Confirm the deployed engine is the latest revision: aws cloudformation describe-stacks --region ap-northeast-1 --stack-name HibikiTranslate-Split --query 'Stacks[0].LastUpdatedTime'
[ ] Confirm hibiki-stage is the latest revision: aws cloudformation describe-stacks --region ap-northeast-1 --stack-name HibikiStage-Tokyo --query 'Stacks[0].LastUpdatedTime'
[ ] Subscribe an active email to the OpsAlarmsTopicArn (CFN output of HibikiTranslate-Split): aws sns subscribe --region ap-northeast-1 --topic-arn <ARN> --protocol email --notification-endpoint <YOUR_EMAIL> then confirm via the email link
[ ] Open CloudWatch console in a tab — alarms view filtered to HibikiTranslate-Split
[ ] Open the operator UI in a tab; verify login works, settings load
[ ] Open the audience URL in a phone browser; verify it loads
[ ] Have a long source ready: 2-hour recorded talk (a re:Invent keynote is ideal — long enough, real-world audio quality)

Soak timeline

T+0:00 — start session

[ ] Click "Start GPU" in the operator UI; wait for green status
[ ] Click "Record" in the operator UI; start the recorded talk audio
[ ] Confirm rows appearing on operator screen
[ ] Confirm rows appearing on audience client(s)
[ ] Note the start timestamp: _______________

T+0:05 — synthetic Cognito test

[ ] In the operator browser console, run: localStorage.setItem('hibiki_translate_token', 'expired-fake-token-string')
[ ] Trigger any operator action (e.g. click any settings UI element that fires a fetch)
[ ] Expected: silent recovery — fetch wrapper sees 401, refreshToken() runs, retry succeeds, no Cognito form
[ ] Failure mode to catch: if you see the login form, the silent-refresh path failed (check browser console for [auth] warnings)
[ ] Note result: pass / fail / details: _______________

T+0:15 — network blip on bridge

[ ] In the AWS console, find the hibiki-stage Fargate task
[ ] Briefly stop the task (force a ECS replacement): aws ecs stop-task --cluster HibikiStage-Tokyo-... --task <TASK_ARN> --reason "soak test"
[ ] Expected: With Phase 1 drain — the operator's session continues against the new task within ~30s. Operator may see a brief "reconnecting..." status but rows resume.
[ ] Failure mode to catch: session ends and operator must re-click record
[ ] Note result: pass / fail / details: _______________

[ ] Pause the YouTube tab / mute the audio source
[ ] Wait 4+ minutes
[ ] Expected: silence banner appears on operator UI saying "No audio detected for 4 min"
[ ] Resume the audio
[ ] Expected: banner clears within 30 seconds (next sentence_committed)
[ ] Note result: pass / fail / details: _______________

T+0:45 — Bedrock error injection (SIMULATION)

This is the trickiest one because we can't actually break Bedrock. Two options:

[ ] Option A (preferred): temporarily set chain.primary.timeout_s = 0.05 in the settings UI. This guarantees Sonnet always times out; chain hedges to Nova. Watch chain-metrics: errors/timeouts on Sonnet should climb. Restore the setting after.
[ ] Option B: skip — accept that this case is unverified until a real Bedrock issue happens.
[ ] Expected: translation continues uninterrupted (Nova hedges and wins). No marker rows in output.
[ ] Failure mode to catch: if marker rows appear during this window, the chain isn't hedging properly.
[ ] Note result: pass / fail / details: _______________

T+1:00 — engine deploy mid-session (Phase 1 drain validation)

This is the critical validation for Phase 1.

[ ] Open a second terminal; have the deploy command ready (do NOT execute yet): bash cd /home/robinwllms/git/kizuna-translate/cdk CDK_DOCKER=docker CDK_DEFAULT_ACCOUNT=108782074531 cdk deploy HibikiTranslate-Split --require-approval never \ -c deployMode=split -c instanceType=g6e.xlarge -c region=ap-northeast-1 \ -c account=108782074531 -c vpcId=vpc-0274625e1a6eda10b \ -c cognitoUserPoolId=ap-northeast-1_DF0shsAHl -c cognitoClientId=a524o4kd9u2v2ujglulnhgo7d \ -c hostedZoneId=Z05584763IOXJBZJEIZL1 -c hostedZoneName=liveprod.cloud \ -c acmCertArn=arn:aws:acm:us-east-1:108782074531:certificate/3d3915d3-cb49-4203-a5f7-926aff42ccaa \ -c cloudMapNamespaceId=ns-6yfyu72qov56dats -c cloudMapNamespaceName=hibiki.local
[ ] With operator session active and producing rows, run the deploy
[ ] Watch the operator UI throughout the ~7 min deploy
[ ] Expected: rows continue arriving without interruption. The old engine task drains; new one comes up; bridge migrates without operator action.
[ ] Failure mode to catch: session ends mid-deploy (the old failure mode). If this happens, Phase 1 hasn't worked and we need to investigate before the event.
[ ] Note result: pass / fail / details: _______________

T+1:30 — alarm injection: silent-output

[ ] Mute the audio source for 5 minutes (longer than T+0:30's window — we want the alarm to fire)
[ ] Expected: within 5-7 minutes, OperatorSilentAlarm fires; SNS email arrives
[ ] Resume audio
[ ] Expected: alarm clears within 5 minutes
[ ] Note result: pass / fail / details: _______________

T+1:45 — long-tail soak

[ ] Let the rest of the audio play through (~15 minutes)
[ ] Watch for any unexpected behavior
[ ] At T+2:00, stop the recording cleanly via the operator UI
[ ] Expected: session ends cleanly; no errors in logs

T+2:00 — measurement pass

Pull these metrics for the soak window:

[ ] Memory growth on each task: bash for service in <Grpc> <Web> <Wlk>; do aws cloudwatch get-metric-statistics --region ap-northeast-1 \ --namespace AWS/ECS --metric-name MemoryUtilization \ --dimensions Name=ClusterName,Value=HibikiTranslate-Split-... \ Name=ServiceName,Value=$service \ --start-time <T+0> --end-time <T+2> --period 300 --statistics Average,Maximum done
[ ] gRPC memory: start ___ end ___ max ___ (should be flat or near-flat)
[ ] Web memory: start ___ end ___ max ___
[ ] WLK memory: start ___ end ___ max ___ (will be high; should not grow)
[ ] GPU utilization (if available in CloudWatch container insights for the WLK task)
[ ] Total alarms fired during the window: bash aws cloudwatch describe-alarm-history --region ap-northeast-1 \ --history-item-type StateUpdate \ --start-date <T+0 ISO8601> --end-date <T+2 ISO8601> Expected: 1 silent-output alarm (from T+1:30 injection). Anything else needs explanation.
[ ] Re-anchor / drift count over the window: bash aws logs filter-log-events --region ap-northeast-1 \ --log-group-name HibikiTranslate-Split-SharedLogsBEDC897C-Ax81o4KVjOcP \ --start-time <T+0 ms> --end-time <T+2 ms> \ --filter-pattern '?"re-anchored after shrink" ?"drift detected"' \ --query 'events | length(@)' Note the count: ___
[ ] Bedrock chain failure count (markers): bash aws logs filter-log-events --region ap-northeast-1 \ --log-group-name HibikiTranslate-Split-SharedLogsBEDC897C-Ax81o4KVjOcP \ --start-time <T+0 ms> --end-time <T+2 ms> \ --filter-pattern '"all hedges failed"' \ --query 'events | length(@)' Note the count: ___ (expect 0 outside the T+0:45 simulation window)
[ ] Row distribution (sanity-check min_chars floor): bash aws logs filter-log-events --region ap-northeast-1 \ --log-group-name HibikiTranslate-Split-SharedLogsBEDC897C-Ax81o4KVjOcP \ --start-time <T+0 ms> --end-time <T+2 ms> \ --filter-pattern '"POST-STITCH" "Fire"' \ --query 'events[*].message' --output text \ | grep -oE '\([0-9]+ chars' | awk '{print $1}' | tr -d '(' | sort -n | uniq -c
[ ] Confirm the distribution: min row size should be ≥ configured min_chars (modulo timeout-fires which the post-stitcher's Timeout fire log line marks)

Pass / fail

The soak passes if ALL of these are true:

[ ] No unplanned alarms fired (only the 1 silent-output injection)
[ ] Mid-session deploy did not drop the operator session (T+1:00)
[ ] Cognito silent refresh worked (T+0:05)
[ ] Silence banner showed when expected and cleared when expected (T+0:30)
[ ] Memory on each service is flat or near-flat over the 2-hour window
[ ] No Bedrock-related markers appeared outside the T+0:45 simulation window
[ ] At least one synthetic alarm fired and was received by email (T+1:30)

If ANY of these fails, file it as a regression to fix before the event. Do not ship to production with known soak failures.

Post-soak review (with the team)

Within 24 hours of the soak:

[ ] Walk through this checklist with the team; confirm pass/fail for each
[ ] Update runbook.md if any new failure mode was discovered
[ ] Update alarm thresholds in cdk/stacks/split_stack.py if the soak data suggests they're too sensitive or too loose
[ ] Tag the deployed revision: git tag v1.0-soak-passed-YYYY-MM-DD
[ ] Decide: ship to event, or do another iteration?