A model in production answered 40,000 support questions last night. How many were throttled, which model got called, and what did each team spend? On a fresh Amazon Bedrock account the honest answer is that you cannot say. Bedrock keeps almost no history of what your models did unless you switch it on.
CloudWatch is the AWS monitoring service that stores metrics, logs, and alarms. Bedrock talks to it in two separate channels: a stream of numeric metrics that arrives automatically, and a detailed invocation log that is off until you configure it. This part is about wiring both, reading them, and turning the numbers into alarms and a per team cost breakdown.
Key takeaways
The bedrock-runtime endpoint publishes metrics to the AWS/Bedrock CloudWatch namespace with no setup: Invocations, InvocationLatency, InvocationThrottles, InvocationServerErrors, InputTokenCount, OutputTokenCount, plus TimeToFirstToken and an estimated quota gauge.
Model invocation logging is disabled by default. Once on, every call writes a JSON record with the prompt, the response, token counts, and any tags you attached, to CloudWatch Logs or S3 or both.
Metrics are for alarms and dashboards. Logs are for the forensics: who called which model, with what, and what it cost. You want both, and the logging bill is tiny next to the inference bill.
Two places Bedrock tells you what happened
Bedrock reports itself through two channels, and mixing them up wastes a lot of debugging time. The first is CloudWatch metrics: numeric time series like a count of invocations per minute or a latency in milliseconds, aggregated and cheap to keep for a long time. These arrive on their own the moment you make a call. The second is model invocation logging: a full event record per request, with the input, the output, token counts, and metadata. That one you have to turn on, and it carries the detail a metric can never hold.
The rule I use is simple. If the question is how much or how fast in aggregate, it is a metric. If the question is which request, which caller, or which prompt, it is a log. A latency spike shows up first as a metric, and you confirm the cause by reading the logs for that window. If you have only skimmed the stack overview in Part 1, the thing to hold onto is that both channels hang off the same bedrock-runtime endpoint you already call.
Runtime metrics worth an alarm
Open CloudWatch, pick the AWS/Bedrock namespace, and you will find the runtime metrics grouped by model id and by operation. Most of them you will glance at once and forget. A few belong on an alarm because they tell you the service is failing or about to. InvocationThrottles is the one I wire first, because a rising throttle count means you are hitting your account quota and requests are being rejected, which is a capacity problem you fix with a quota increase or cross region inference, both covered in Part 8. InvocationServerErrors means Bedrock itself faulted, which is rare and worth paging on. InvocationLatency is your speed signal for anything that waits on the full response.
Two newer metrics matter for anyone streaming or running near a quota. TimeToFirstToken measures milliseconds from request to first streamed token, and it is the number a chat user actually feels, so alarm on it rather than total latency for interactive apps. The estimated quota consumption gauge shows how close you are to your tokens per minute ceiling before throttling starts, which turns a capacity surprise into a gauge you can watch. Here is the short list I put on a dashboard.
| Metric | What it tells you | Alarm when |
|---|---|---|
| Invocations | Call volume per model | Drops to zero on a live path |
| InvocationThrottles | Requests rejected at quota | Any sustained non zero |
| InvocationServerErrors | Bedrock side faults | Any non zero, page it |
| InvocationLatency | Time to final token | p99 over your budget |
| TimeToFirstToken | Streaming responsiveness | p90 over what users tolerate |
| InputTokenCount, OutputTokenCount | Token throughput, cost proxy | Unexpected jump in output tokens |
Metric names from the AWS/Bedrock runtime metrics documentation. Availability of TimeToFirstToken and the estimated quota gauge varies by Region and operation, so confirm both are present in your Region [VERIFY].
Turn on model invocation logging
Metrics show you the shape of traffic. They cannot tell you which prompt drove a latency spike or which team spent the money, because a metric is a number with no request attached. For that you enable model invocation logging, which writes one JSON record per call. Each record holds invocation metadata, the input and output bodies, token counts, and any request metadata tags. It is off by default, and in more than one account I have opened, it was still off months into production, which meant nobody could answer a single what happened question.
You choose the destination: CloudWatch Logs, S3, or both, and both must sit in the same account and Region as the calls. There is a size rule that trips people. A log event up to 100 KB goes to CloudWatch Logs inline. Anything larger, or binary data like images, is written to an S3 bucket instead, so if you log image or video calls you need an S3 target configured for the overflow even when your main sink is CloudWatch Logs. You also pick which data types to record, text, images, embeddings, and so on, which lets you keep token counts while dropping raw image bytes if that is your policy. The command below sets it up from the CLI and reads it back.
aws bedrock put-model-invocation-logging-configuration
--region us-east-1
--logging-config '{
"cloudWatchConfig": {
"logGroupName": "/bedrock/invocations",
"roleArn": "arn:aws:iam::111122223333:role/BedrockLoggingRole",
"largeDataDeliveryS3Config": {
"bucketName": "my-bedrock-logs", "keyPrefix": "overflow/"
}
},
"s3Config": { "bucketName": "my-bedrock-logs", "keyPrefix": "raw/" },
"textDataDeliveryEnabled": true,
"imageDataDeliveryEnabled": false,
"embeddingDataDeliveryEnabled": true
}'
# Confirm it stuck
aws bedrock get-model-invocation-logging-configuration --region us-east-1Expected output: the get call returns your loggingConfig JSON. Failure mode: the put succeeds but no logs appear, almost always because the roleArn lacks logs:CreateLogStream and logs:PutLogEvents on the log group, or the S3 bucket policy does not allow the bedrock service to write. Logs arriving with a delay of a minute or two is normal, not a fault.
Query token usage with Logs Insights
Once logs land in CloudWatch, Logs Insights is where you turn them into answers. It is a query language over your log group that parses the JSON fields for you. The query I run most often groups token counts by model and by the calling principal, which is the fastest way to see where spend is going before any billing report catches up. Each record carries input, output, and where caching applies cache read and cache write token counts, so you can multiply by the per token rates from Part 6 and get a real cost, not an estimate.
fields identity.arn as principal, modelId,
input.inputTokenCount as inTok,
output.outputTokenCount as outTok
| stats sum(inTok) as inputTokens,
sum(outTok) as outputTokens,
count() as calls
by modelId, principal
| sort inputTokens desc
| limit 20Expected output: one row per model and caller with summed input tokens, output tokens, and a call count. Failure mode: empty results usually mean you are querying the wrong log group, or you logged to S3 only, in which case the fields live in the S3 JSON and you query them with Athena instead of Logs Insights.
Worked example
A support assistant runs 2,000,000 Converse calls a month, roughly 1,500 input tokens and 400 output tokens each. The logs show 1,600,000 calls went to Amazon Nova Lite and 400,000 to Claude 3.5 Sonnet. Multiply tokens by published rates and the split is stark. Nova Lite lands near 298 USD for the month. Claude Sonnet, on a quarter of the calls, lands near 4,200 USD because its per token rate is far higher.
Now price the observability itself. Two million JSON records at roughly 4 KB each is about 8 GB of ingestion. At 0.50 USD per GB that is 4 USD, storage at 0.03 USD per GB is cents, and a Logs Insights scan of the month is a few cents more. Observability costs about 4 USD against a 4,500 USD inference bill.
The lesson: the logs are what told you 89 percent of spend came from a quarter of the traffic. That routing decision is worth far more than the 4 USD it cost to see it. Rates are illustrative, confirm your own before you commit.
| Line item | Volume | Monthly (USD) |
|---|---|---|
| Nova Lite inference | 1.6M calls | ~298 |
| Claude 3.5 Sonnet inference | 0.4M calls | ~4,200 |
| CloudWatch Logs ingestion | ~8 GB | ~4 |
| Logs storage and queries | 8 GB plus scans | under 1 |
CloudWatch Logs Standard class ingestion is 0.50 USD per GB for the first 10 TB, storage 0.03 USD per GB per month, Logs Insights queries 0.005 USD per GB scanned. Inference rates are illustrative. Confirm current numbers on the pricing pages before you plan around them.
Request metadata for cost attribution
The token query above groups by IAM principal, which is useful, but many teams call Bedrock through one shared role, so every request looks like the same caller. Request metadata fixes that. You attach up to 16 key value tags to each call, and they land in the log record under a top level requestMetadata field, which lets you group spend by team, application, environment, or feature instead of by role. For the Converse API you set requestMetadata in the request body. For InvokeModel you send the X-Amzn-Bedrock-Request-Metadata header. Keys and values are capped at 256 characters each.
The catch is that metadata is only recorded when logging is on, in the Region where the call is made. Tag every call from day one even if you are not querying by it yet, because you cannot backfill a tag onto a request that already happened. Decide the two or three dimensions you will want to slice by, team and feature are the usual pair, and set them on every call in your client wrapper so no code path forgets.
In practice
Set request metadata in one place, the client wrapper every service shares, not in each feature. A single line that adds team and feature tags from the calling context means every future query can slice by them, and you never chase down the one code path that forgot. Add a metadata key for the app version too, so a cost or latency regression can be tied to the deploy that caused it.
What to alarm on, and what to ignore
An alarm on every metric is noise nobody reads. Three alarms carry most of the value. First, InvocationServerErrors above zero, because that is Bedrock faulting and it should page someone. Second, InvocationThrottles sustained above zero, because you are losing requests at the quota edge. Third, a latency alarm on the tail, p99 for batch style calls or p90 TimeToFirstToken for chat, set to the number your users actually notice rather than a round figure. Everything else belongs on a dashboard you look at, not an alarm that wakes you.
What I would not alarm on is raw token count. It moves with normal traffic and a spike is usually a busy day, not an incident, so watch it on a dashboard and let request metadata tell you which team drove it. The vendor neutral view of why you monitor grounding and safety, not just uptime, sits in the GenAI Series, and if you run across clouds the equivalent monitoring story lives in the Azure Gen AI Series.
My take
Logging is the setting I regret leaving off, never the one I regret turning on. The first serious question a stakeholder asks about a Bedrock app is who is spending what, and without logs from the start you answer with a shrug and a promise to instrument it next sprint. Turn it on before the first real traffic, tag every call, and the answer is a two minute query instead of a two week retrofit.
Retention, and the log group that quietly grows
CloudWatch log groups keep events forever unless you tell them otherwise. A busy Bedrock workload that writes full prompt and response bodies can push tens of gigabytes a month into a group nobody set a retention on, and you pay storage on all of it every month after. Set a retention period the day you create the group. For most observability I keep 30 to 90 days in CloudWatch Logs, long enough to debug an incident and attribute a billing cycle, and route anything I need for longer to S3 where storage is cheaper and lifecycle rules can tier it down.
The split matters because the two sinks have different economics. CloudWatch Logs is priced for query speed, S3 for cheap durable storage. My pattern is CloudWatch Logs as the hot store for Logs Insights over the last month, and S3 as the cold store for audit and long retention, with an S3 lifecycle rule that moves objects to a colder class after 30 days and expires them on your compliance clock. Log to both from the start and you get fast queries now and a cheap archive later, without moving data twice.
One more setting saves money quietly. If your policy does not need raw prompt and response bodies, disable the body data types and keep token counts and metadata. The records shrink by an order of magnitude, your Logs Insights scans get cheaper because there is less to read, and you still answer every cost and volume question. Turn body logging on only for the group and the window where you are actively debugging, then turn it back off. A log group with no retention and full bodies is the most common way a Bedrock observability setup becomes a line item nobody budgeted for.
Metrics for alarms, logs for cost attribution
Here is how I set up Bedrock observability on a new workload. Enable model invocation logging to CloudWatch Logs on the first day, with an S3 overflow target if any calls carry images or large payloads, and set a retention period that matches your data policy. Put three alarms on the runtime metrics, server errors, sustained throttles, and a tail latency figure, and leave the rest on a dashboard. Tag every call with team and feature through a shared client wrapper so the token query can attribute spend without a code change later.
When would I skip the logs? Almost never, but if a workload is regulated in a way that forbids storing prompt and response bodies, log with the body data types disabled so you still capture token counts and metadata without the sensitive content. The next part turns these numbers into a budget: cost governance and FinOps on AWS, where the metadata tags you set here become the dimensions of a real chargeback. It builds directly on the attribution query above, so wire the tags before you get there.
This week, run the put-model-invocation-logging-configuration command on one workload, make ten real calls, and run the token query. The row that shows where your spend actually goes is usually a surprise.
The first time you enable invocation logging on a workload that has been running blind, the token query almost always names a caller nobody expected. It is usually mundane, a retry loop or one team resending full context on every turn, and it is usually the biggest single line on the bill.
References
- Monitor model invocation using CloudWatch Logs and Amazon S3
- Monitor bedrock-runtime inference using CloudWatch metrics
- Per-request metadata tagging, Amazon Bedrock User Guide
- Amazon CloudWatch pricing
About The Author
DrJha