TailSpend Docs

Core API: Two Functions, Two REST APIs

Issue #632 consolidates TailSpend's request/response API from one Lambda per route into two core API functions that run the same router and application code:

App core API Developer core API
Gateway App REST API (api.<stage> after cutover) Developer REST API (developer-api.<stage>…/v1)
Authentication Cognito user-pool authorizer (ID tokens) + accepted-client check in the router Personal-token Lambda authorizer (no caching)
Function tailspend-<stage>-core-api-app tailspend-<stage>-core-api-developer
Routes Every app route except the exceptional ones DEVELOPER_ROUTE_SCOPES only
Concurrency Reserved (guaranteed and capped) Independent ceiling
Limits 120 weighted units/min per user 60/min per user across tokens, 30/min per token, 500 creates/day

The personal-token authorizer (tailspend-<stage>-developer-token-authorizer) is a third, separate function. The exceptional routes keep their own functions, because they need materially different privileges or dependencies: admin, receipt scanning, account deletion, and the AdMob SSV callback. The scan worker, the reconciler and the recurring-expense tick are unchanged.

API request routing through the app REST API, the developer REST API and the web distribution's AdMob callback

Where Things Live

Concern Location
Route manifest (exceptional, HTTP-only, developer scopes, starting limits) packages/api-schema/src/core-api.ts
Shared router apps/services/src/core/router.ts
Route tables apps/services/src/core/app-routes.ts, developer-routes.ts
Entry points apps/services/src/core/app-handler.ts, developer-handler.ts
REST ↔ HTTP event adapter apps/services/src/runtime/api-event.ts
Principal model apps/services/src/runtime/principal.ts
Tokens, idempotency, usage apps/services/src/developer/
Stacks apps/infra/lib/stacks/core-api-stack.ts, service-stack.ts (cutover)
Per-stage policy (cutover flag, concurrency, throttles) apps/infra/lib/config/core-api.ts

Every operation is still written as a per-route withHandler(...) handler. Those handlers expose their validated operation, and the router dispatches to it. Validation, membership and role checks, ownership rules, persisted creator attribution, optimistic version checks, pagination and error envelopes are therefore byte-for-byte the code that ran before. The route tables are checked against the OpenAPI document in app-routes.test.ts, and CDK synth cross-checks the manifest (validateCoreApiRoutes).

Request Path

For each request the router:

  1. Checks gateway identity. It serves only REST events whose apiId equals the function's configured CORE_API_GATEWAY_ID.
  2. Checks the route allowlist. The gateway's matched route template must be in the deployment's table; unknown routes return 404. The developer router refuses to start if any route lacks a scope.
  3. Builds the principal. It reads only verified authorizer output, never a header or body field:
    • App: a Cognito ID token whose aud is a TailSpend client (web, mobile, integration tests), or anonymous on the explicitly public invite reads.
    • Developer: the token authorizer's context.
    • Token context at the app entry point, or Cognito claims at the developer one, is a 401.
  4. Applies token restrictions (developer only): the route's scope, the path's group in the token's groups, and a strongly consistent membership check. The check confirms the base-table participant row, and denies whenever the membership index and that row disagree. Removing someone from a group, or changing their role, therefore takes effect on their next request.
  5. Applies limits (see below).
  6. Runs the shared operation.
  7. Records one usage aggregate, bounded to 250 ms (see below).

Request-scoped state (event, principal, logger) never leaves the invocation. Only clients and immutable configuration are reused across warm invocations.

Route Registries

REST_ROUTE_AUTH and REST_ROUTE_GRANTS contain every OpenAPI route. Synthesis rejects missing or extra entries. EXCEPTIONAL_ROUTE_HANDLERS in apps/infra/lib/rest-route-handlers.ts contains only the eight EXCEPTIONAL_ROUTES, with exact subset validation and a check that each still exists in OpenAPI. Core operations import their handlers directly in services/src/core/app-routes.ts. The stale documentation-only handlers/index.ts is removed.

The canonical internal event is payload v2. REST proxy payload v1 is adapted once at entry by runtime/api-event.ts; the AdMob Function URL supplies v2 natively so its signed raw query string survives unchanged.

Authentication Notes

Limits and Capacity

All allowances are initial policies to tune from measurements. They are set in CORE_API_RATE_LIMITS (@repo/api-schema), and can be overridden per function through APP_RATE_LIMIT_UNITS_PER_MINUTE, DEVELOPER_USER_RATE_LIMIT_UNITS_PER_MINUTE and DEVELOPER_TOKEN_RATE_LIMIT_UNITS_PER_MINUTE.

Budget Key in tailspend-<stage>-rate-limit Default
App, per user RL#<cognito sub>#<minute> (unchanged) 120 units/min
Developer, per user across all tokens RL#DEV#USER#<userId>#<minute> 60 units/min
Developer, per token RL#DEV#TOKEN#<tokenId>#<minute> 30 units/min
Developer expense creates QUOTA#DEV_EXPENSE_CREATE#…#<day> 500/user/day, 20,000/day overall

The daily create allowance is charged inside the create operation, only after the body, the Idempotency-Key and the expense itself (payments fund items, splits) validate, and only once per key: the charge is recorded on the idempotency claim, so matching replays, retries after a crash and rejected requests are never charged twice or skipped. If the allowance refuses, nothing is recorded and the same key can be retried once capacity returns.

Gateway Throttles and Concurrency

These values live in apps/infra/lib/config/core-api.ts.

dev beta prod
App stage throttle (rps/burst) 30/60 30/60 30/60
Developer stage throttle 10/20 10/20 10/20
App core reserved concurrency 5 5 20
Developer core concurrency ceiling 5 5 5
Token authorizer concurrency 5 5 5

The app API keeps the HTTP API's per-route throttles: scans 2/5, public invite reads 5/10.

Derive concurrency from measurements: concurrency ≈ peak RPS × p99 duration (s), with 2× headroom. All stages share one AWS account, which must keep at least 100 unreserved executions. Check aws lambda get-account-settings before raising any reservation.

Separate concurrency does not isolate shared resources:

Watch these metrics:

Usage Aggregates

Usage history reuses the Users table, with one row per day and sort-key namespace:

Record UserId SK
Daily developer-token usage <ownerUserId> USAGE#DEVELOPER#DAY#YYYY-MM-DD#TOKEN#<tokenId>
Daily app usage <userId> USAGE#APP#DAY#YYYY-MM-DD

Each row carries these counters: RequestCount, SuccessCount, RedirectCount, ClientErrorCount (including 429s), ServerErrorCount, RateLimitedCount (the 429 subset) and AdmittedUnits. It also carries ExpiresAt, set to the end of the UTC day plus 90 days.

Cost Estimate

The estimate assumes US East (N. Virginia) on-demand pricing of $0.625 per million standard write units, items under 1 KB, and one aggregate write per request.

Requests per user per day (1,000 users) Monthly tracked requests Added write cost
10 300,000 ~$0.19
100 3 million ~$1.88
1,000 30 million ~$18.75

Plan on $5–$10/month for 1,000 moderately active users as a conservative incremental estimate. This covers writes, the additional on-demand dashboard reads, and slightly longer Lambda execution. It is not an upper bound. Validate it against measured item sizes and latency, as described under Performance Baseline.

Observability

AdMob SSV Callback

The AdMob server-side-verification callback verifies Google's signature over the raw query string, which REST proxy events don't carry. To retire the HTTP API it is served from the web distribution instead:

The AdMob console's SSV callback URL is the Web stack's AdmobSsvCallbackUrl output (https://<web host>/ads/admob/ssv).

Migration History

The HTTP API (one Lambda per route) was replaced in stages, all through the normal pipeline:

  1. Both APIs ran side by side while the integration suite ran against each target; promotion waited on both.
  2. A per-stage REST_CUTOVER flag repointed each stage's root api.<stage> mapping at the REST API in place (same CloudFormation resource, so the host was never unmapped) and removed the superseded per-route functions. Dev and beta first, then prod.
  3. The AdMob SSV callback moved to the web distribution (see above), and the HTTP API, its JWT authorizer, the ads mapping and the cutover flag were removed. The root mapping keeps the HTTP API's former construct path, and so its logical id, so the removal didn't replace it.

Rollback is now an ordinary revert-and-redeploy: there is no second API to switch back to.

Performance Baseline

Do not assume consolidation reduces latency: one larger bundle can cold-start slower, while fewer, warmer functions cold-start less often. Before and after each stage's cutover, record:

Measured: 11–12 October 2026

The core functions' log groups were recreated during the cutover, so the window is about 13.5 hours: 2026-10-11 10:00 to 23:42 UTC. Nearly all traffic is the pipeline's integration and E2E suites. New Relic was on in beta and prod for the whole window. Method: Logs Insights over the REPORT lines (count, ispresent(@initDuration), avg/pct of @initDuration, @duration, @billedDuration, max(@maxMemoryUsed)), ConcurrentExecutions and Throttles (hourly maxima over 7 days) and API Gateway Latency/IntegrationLatency percentiles.

Function (1024 MB; authorizer 256 MB) Invocations Cold Init avg / p95 / p99 (ms) Duration avg / p95 / p99 (ms) Billed p99 (ms) Max memory Peak concurrency / reserved
prod app core 3,408 44 (1.3%) 1,529 / 1,720 / 1,723 61 / 162 / 278 1,612 240 MB 11 / 20
prod developer core 1,125 14 (1.2%) 1,505 / 1,750 / 1,750 60 / 149 / 317 1,640 255 MB 2 / 5
prod token authorizer 1,149 14 (1.2%) 1,482 / 1,648 / 1,648 59 / 131 / 715 2,017 165 MB 2 / 5
beta app core 3,566 26 (0.7%) 1,442 / 1,708 / 1,721 57 / 149 / 247 289 231 MB 5 / 5
beta developer core 1,200 12 (1.0%) 1,519 / 1,708 / 1,708 58 / 142 / 343 477 257 MB 3 / 5
beta token authorizer 1,224 14 (1.1%) 1,441 / 1,621 / 1,621 55 / 101 / 730 2,007 168 MB 3 / 5

No throttles on any core function in the 7 days.

API Gateway, 7 days (p50 / p95 / p99, ms):

API Latency IntegrationLatency
prod App 73 / 234 / 2,525 55 / 226 / 2,534
prod Developer 96 / 220 / 1,985 53 / 149 / 1,735
beta App 68 / 211 / 2,369 52 / 197 / 2,388

The p99s are cold starts: about 1% of invocations pay a ~1.5 s init, which the warm p95 (~150–230 ms) hides.

Usage recording. Its time can't be separated from the request: durationMs covers the whole request, and requests without tracking (anonymous, 169 in prod) are different work, so comparing the two groups says nothing. Recording is bounded by the 250 ms timeout. There were no api_usage.record_failed events in prod. Prod core.request, tracked requests: cognito 3,239 (avg 49 ms, p95 121, p99 229), token 1,125 (avg 46, p95 117, p99 286).

Tracing overhead. The core functions have had New Relic since they were created, so there's no before/after for them. Like-for-like on beta's other functions, New Relic roughly doubles init (~650 ms to ~1,580 ms) and adds about 60 ms to warm invocations (avg billed 145 to 208 ms). Details are on #633.

Tuning:

Re-measure once real user traffic dominates the suites.

Cold-start reduction (#690, 21 October 2026)

Separating the remaining startup costs

API Startup Investigation creates probe resources only on manual dispatch. Maintainer-authored same-repository PRs changing its code run only aggregate read-only New Relic and CloudWatch observations. It creates three fresh beta cold starts per cell: core app/developer at 1024/1769 MB and the token authorizer at 256/512/1024 MB. Every probe retains the deployed ARM64 runtime, New Relic layer, dedicated ESM wrapper and SSM license-key lookup. An added preload observes the agent's existing CommonJS entry compilation without loading the agent early; the synthetic handler imports the deployed bundle but never calls its handler. Three synthetic invocations allow telemetry to flush. One reports a fixed, source-mapped synthetic error on the temporary function only.

Isolated beta startup investigation

The report includes Lambda InitDuration, agent initialization wall/CPU time, application import wall/CPU time, extension initialization from its own logs, and the extension's startup-to-license-obtained interval. The latter is an upper bound on the license phase, including Rust/AWS-client setup; it does not isolate the SSM network request. Agent/extension intervals are not a proven disjoint partition, so their times must not be mechanically added. These small, interleaved samples compare allocations; they do not establish production end-to-end p99. The workflow also collects a 24-hour pre-update and up-to-six-hour post-update production REPORT window, because the original one-hour pre-update window was empty. Traffic composition and cold-start counts must accompany any comparison. Source hashes and layer versions make the comparison reviewable.

The workflow also queries aggregate production invocation counts through the existing New Relic query credential, if configured, and looks for the synthetic mapped probe errors. A successful configuration check alone does not demonstrate ingestion. No production request is induced for this check.

Source inspection of the pinned Rust extension v2.7.1 (commit e785bad) shows one SSM license-key fetch, in parallel with extension registration. It does not use the older Go extension's startup checks; NEW_RELIC_IGNORE_EXTENSION_CHECKS is therefore not an applicable optimization. Node agent v14.7.0 initializes before the real ESM bundle. Memory/CPU contention and runtime SDK loading remain hypotheses to measure, not reasons to disable monitoring or change production memory blindly.

Probe functions use a run-specific tailspend-beta-startup-690-<run-id>- prefix, have no API integration, alias or provisioned concurrency, and are deployed through the existing CDK bootstrap roles and deleted with their log groups in finally plus an unconditional authenticated workflow cleanup. Reports expire after two days. Signed download URLs, credentials, raw logs and environment dumps are never GitHub artifacts. Code/configuration use private temporary files and the normal private CDK bootstrap assets/templates; these assets follow the existing bootstrap retention policy. This workflow is an isolated diagnostic, not part of normal promotion or integration tests.

Controlled results (21 October 2026)

Avoidable environment churn was found and fixed. Previously every deployment used the moving GITHUB_SHA as TAILSPEND_RELEASE, changing every monitored Lambda's environment even when only web, docs or CI changed. Configuration updates can replace warm execution environments; the probe rounds also observed fresh initialization after a round-only environment change. This increases cold frequency independently of how long each initialization takes.

The stage workflow now resolves NEW_RELIC_SERVICE_RELEASE from the last commit affecting service/schema/OpenAPI sources, Lambda bundling/monitoring inputs or shared build/lock inputs. It checks out full Git history and fails rather than using an unreliable shallow-history result. The history guard checks ancestors of the deployed commit, so unrelated cached shallow PR refs do not block deployment on persistent runners. CI runs the actual resolver after the same full-history checkout as deploy. Unrelated source/docs merges retain the same backend telemetry tag. Lock/build changes are conservatively treated as backend-affecting. Frontend release IDs still use the full deployment commit; local callers without the new variable retain the existing Git-SHA fallback. The first adoption updates the release tag once. This avoids metadata-only reconfiguration afterwards; it does not remove cold starts from real deployments or otherwise guarantee warm environment retention.

Two completed experiments collected 42 verified cold starts, three per cell in each run, with the same deployed application bundles and New Relic layer. The second run additionally measured SDK evaluation. Init medians in milliseconds:

Entry Memory (MB) Run 1 init Run 2 init
App 1024 1485 1873
App 1769 1884 1808
Developer 1024 1836 1446
Developer 1769 1809 1814
Authorizer 256 1519 1233
Authorizer 512 1507 912
Authorizer 1024 1584 1604

Memory is not a demonstrated repeatable cold-start fix. The authorizer's 256→512 MB comparison was almost unchanged in the first run and faster in the second. Core results also changed direction between runs. These small samples show substantial variation; they do not justify increasing production allocations. The synthetic warm handler does not measure real route work or its cost.

At the existing production allocations, the SDK-instrumented run recorded these component medians (ms; three samples each):

Entry Agent entry SDK evaluation Other application imports Extension init
App, 1024 MB 741 318 385 401
Developer, 1024 MB 634 243 253 281
Authorizer, 256 MB 621 219 58 289

SDK evaluation covers 33–34 runtime AWS SDK/Smithy CommonJS modules, with nested evaluations deduplicated. It is a subset of application import time, not an additional phase or a DynamoDB request. The runtime DynamoDB SDK was 3.1105.0 and the agent 14.7.0. SDK CPU time closely tracked its wall time. The agent's roughly 0.6–0.8 s startup and SDK evaluation explain substantial remaining work; the core bundles also spend roughly 0.25–0.4 s on other imports. No business handler or DynamoDB operation was invoked by these probes.

The retained profiles confirm native VM metrics are already disabled by the agent's serverless configuration. The dedicated ESM wrapper and slim layer are already in use. The extension's roughly 0.28–0.41 s initialization includes its licensing/setup path, so SSM cannot explain the entire 1.4–1.9 s initialization; the exact SSM network duration was not isolated. No monitoring feature was disabled to obtain these results. The pinned Rust extension is current, and the Node 14.8.0 release notes do not identify a cold-start fix to justify an unmeasured layer change for this issue.

The next code experiment should compare a tree-shaken SDK bundle against the same runtime SDK versions on ARM64, including artifact size and end-to-end cold-request latency. The earlier local SDK-bundling test used a different environment and found no consistent benefit; neither that result nor this decomposition proves a production win. Deferring imports into the first request would only move the same cost out of InitDuration.

The wider production REPORT comparison found the following cold-start counts and mean initialization times. Its before window covers the preceding day and its after window covers deployment+2 minutes through collection:

Entry Before requests / cold starts Before mean init After requests / cold starts After mean init
App 530 / 21 1563 ms 421 / 4 1771 ms
Developer 100 / 1 1741 ms 100 / 1 1761 ms
Authorizer 102 / 1 1704 ms 102 / 2 1420 ms

These are different traffic windows, and developer/authorizer cold samples are particularly sparse. They establish neither a stable p99 improvement nor a causal regression. The gateway's later rolling hour contained no data; retain the earlier post-deploy report as a separate observation rather than inventing an endpoint comparison.

New Relic NRQL confirmed the current production release's invocation telemetry for all three core functions (421 app, 100 developer, 102 authorizer). A later read-only query found 37 synthetic errors in AwsLambdaInvocationError with the source-map verification attribute, confirming error delivery as well as invocation delivery. Immediate per-run queries had initially returned zero; allow for ingestion delay before treating that as a monitoring failure. The final observer found 58 stored error stacks matching the fixed startup-source.ts:2: TypeScript location, without exporting any stack traces or request data. Counts increased as the existing probe telemetry finished ingesting; no new business request was needed.

Raw measurements, sample counts, source hashes and workflow provenance are in performance/api-startup-investigation-690.json. The completed workflows removed their temporary stacks, functions and log groups. Production memory, monitoring and concurrency remain unchanged by this investigation.

Post-deployment API integration jobs now upload an api-startup-<stage>-<attempt> artifact (two-day retention). It contains aggregate CloudWatch REPORT statistics for the hour before each function's last update and up to an hour after it, plus timestamped API Gateway latency p95/p99 and request counts. Rollout and ingestion margins are excluded. The report records the deployed release, memory allocation and whether each core function has the New Relic wrapper, layer and account configured. It does not change functions, monitoring or concurrency, and exports no secrets, environment dumps or request logs. Diagnostic failures do not block promotion. Run it with an existing AWS identity using node scripts/api-startup-report.mjs <dev|beta|prod> report.json. Short windows and sparse cold starts remain inconclusive; compare counts and traffic composition before attributing a p99 change to this release. Configuration checks do not prove that New Relic is ingesting events or mapping stack traces.

New Relic stays enabled with the pinned slim layer, ESM wrapper, request/error reporting and dependency spans. Provisioned concurrency is not used. The token authorizer remains uncached so revocation takes effect on the next request.

Two avoidable costs were found:

Local Node 24.19.0 / esbuild 0.28.2 measurements against 9a25a01 (nine fresh processes per entry; decimal KB; JS plus linked map, before ZIP compression):

Entry JS before → after (KB) Map before → after (KB) Import median before → after (ms)
App core 760.3 → 760.3 2,845.4 → 893.0 233.2 → 208.5
Developer core 670.8 → 670.9 2,554.5 → 765.2 219.2 → 207.1
Token authorizer 556.6 → 70.5 1,862.3 → 109.1 169.2 → 102.8

These isolate application imports: they exclude the Lambda platform, New Relic agent/extension, network and invocation work. They are not production p99 or a measurement of New Relic overhead. The raw report is in performance/api-startup-690.json. Bundling the AWS SDK was also tried; it increased package size and did not consistently improve local imports, so SDK externalization is retained. The existing dedicated ESM wrapper already avoids New Relic's documented legacy NEW_RELIC_USE_ESM loader workaround; no loader or monitoring switch is added.

Reproduce the import/size probe without AWS calls or credentials:

npm run profile:api-startup --workspace=@repo/infra -- --samples 9
# Compare a dependency-installed checkout of the baseline using its old map policy:
node apps/infra/scripts/profile-api-startup.mjs --source-root /path/to/baseline --embedded-sources --samples 9

After normal deployment, re-measure with New Relic on in all compared releases. Query the three core/authorizer log groups over comparable traffic:

filter @type = "REPORT"
| stats count(*) as requests, count(@initDuration) as coldStarts,
    avg(@initDuration) as initAvgMs, pct(@initDuration, 95) as initP95Ms,
    pct(@initDuration, 99) as initP99Ms, pct(@duration, 99) as handlerP99Ms,
    pct(@billedDuration, 99) as billedP99Ms by @log

Compare API Gateway Latency/IntegrationLatency p95/p99 as well: a developer request can pay authorizer and developer-core initialization in sequence. Use New Relic's existing aws.lambda.coldStart dashboard and dependency spans to distinguish initialization from first DynamoDB/SSM calls. The extension's SSM license-key lookup, runtime SDK loading, CPU allocation and container turnover remain possible contributors; no new production attribution is claimed without current logs. This execution environment had no AWS/New Relic identity, so live init/p99 validation remains a post-deployment check.

Resource Budget

Both are well under the 500-resource limit.

View Markdown source