BOE Estimation Template

April 27, 2026#tech#boe#estimation#capacity-planning#infrastructure

First-pass engineering and capacity estimation template for production services

BOE Estimation Template

Purpose

Use this template to produce a first-pass engineering or capacity estimate for any production service.

This method is intentionally general. It applies to:

  • synchronous APIs
  • asynchronous consumers and workers
  • webhook handlers
  • scheduled jobs
  • orchestration services

The workload unit, scaling signal, and validation plan will change by service. The structure should not.

Output expectations

A usable BOE should answer:

  1. What exact workload is being measured?
  2. What does normal traffic look like?
  3. What does peak traffic look like?
  4. Which module is driven by which traffic stream?
  5. How does traffic translate into concurrency?
  6. How does concurrency translate into pod count and resource requests?
  7. What autoscaling signal should be used?
  8. What cluster footprint follows from the module-level plan?
  9. What node or hardware shape should run that footprint?

End-to-end flow

This is the core translation chain:

  1. traffic -> concurrency
  2. concurrency -> replicas
  3. replicas + per-pod CPU/memory request -> cluster demand
  4. cluster demand -> node / hardware choice

Each step answers a different question:

  • traffic -> concurrency What is active at the same time during the peak window?
  • concurrency -> replicas How many pods are needed to hold that in-flight work safely?
  • replicas + per-pod CPU/memory request -> cluster demand How much total CPU and memory must the cluster reserve?
  • cluster demand -> node / hardware choice What node family, node size, and node count can host the workload with headroom?

This sequencing matters. Do not start with instance size and try to force the workload into it.

Step 1. Define the workload unit

Pick one canonical workload unit and state it explicitly.

Examples:

  • API request
  • queue message
  • provider invocation
  • webhook callback
  • scheduled execution

Rules:

  • The workload unit must be countable from a stable source.
  • The workload unit must be close to what actually drives compute or queue occupancy.
  • If one user action fans out into multiple downstream actions, state whether the BOE is based on the user action or the fan-out unit.

Step 2. Select the minimum metric set

The metric set is part of the BOE, not an afterthought.

Volume estimation metrics

  • latest complete month
  • trailing 12-month average
  • peak complete month
  • month-over-month growth rate
  • peak hour in the selected burst window
  • burst factor relative to average hour
  • success count
  • failure count

Optional operational metrics

  • p95 service time
  • p99 service time
  • queue lag
  • backlog depth
  • timeout rate

Operational metrics are useful, but they are not the primary sizing inputs unless they directly affect occupancy or backlog behavior.

Step 3. Build the traffic model

Turn the raw metrics into a demand profile.

At minimum, capture:

  • steady state: trailing average
  • current state: latest complete month
  • growth: month-over-month change
  • peak state: peak month and peak hour
  • reliability context: success and failure counts

The goal is to describe both sustained load and burst load.

Step 4. Map traffic to modules

Do not size every module from the same top-line number.

Each deployable module needs its own traffic driver.

Examples:

  • ingress API: requests per second
  • worker: jobs per second
  • webhook service: callbacks per second
  • scheduler: tasks per interval
  • orchestrator: workflow steps per second

This step is where most weak BOEs fail. A clean traffic model is still not enough if it is applied to the wrong module.

Step 5. Convert traffic into concurrency

Use service time to estimate how many requests or jobs are active at the same time.

Useful formulas:

  • peak_rps = peak_hourly_count / 3600
  • required_concurrency ~= arrival_rate_per_second * service_time_seconds

Interpretation:

  • arrival rate describes how fast work enters the system
  • service time describes how long one unit occupies the module
  • concurrency is the amount of work the module must hold in flight

Notes:

  • This is where observed TIME_TAKEN or equivalent request duration becomes useful.
  • Duration helps estimate occupancy. It does not directly determine CPU request or memory request.
  • If the module is mostly waiting on external I/O, service time can be high while CPU stays low.

Step 6. Estimate single-pod capacity

This is the missing bridge between traffic and Kubernetes resources.

For one pod, determine:

  • max safe concurrency
  • CPU usage under representative load
  • memory floor
  • memory peak
  • degradation point under burst

Preferred inputs:

  • controlled load test
  • real production telemetry from a comparable service
  • benchmark results from a staging environment

Without single-pod capacity data, the BOE remains directional only.

Step 7. Produce the initial production allocation

For each module, document:

  • replicas
  • CPU request
    • the minimum CPU a pod asks the scheduler to reserve
  • memory request
    • the minimum memory a pod asks the scheduler to reserve
  • CPU limit
    • the max CPU the pod can use
  • memory limit
    • the max memory the pod can use before OOM risk
  • autoscaling enabled or disabled
  • scaling signal
    • this is the metric Kubernetes uses to decide when to scale a workload up or down
    • scaling signal = CPU means:
      • if CPU stays high, add replicas
      • if CPU stays low, reduce replicas
  • scaling min and max
    • these are the lower and upper bounds for autoscaling
    • min and max replicas

Guidance:

  • use at least two replicas for stateless production ingress unless there is a hard reason not to
  • keep a headroom target, usually 30% to 50%
  • size workers from concurrency and backlog tolerance, not just from CPU

Step 8. Choose the autoscaling signal

Pick the signal that matches the real bottleneck.

Common patterns:

  • CPU-based HPA for simple synchronous APIs
  • queue lag or backlog-based KEDA for asynchronous consumers
  • request concurrency or latency-based scaling for bursty ingress services
  • custom scaling for workflow orchestrators when CPU is not the binding constraint

Avoid scaling on a metric that only loosely correlates with user-visible pressure. Pick the metric that moves when the user experience degrades.

Example: scaling a queue consumer on CPU only.

Say research-consumer is waiting on external providers:

- each worker thread is busy for 5s
- CPU stays low because the process is mostly waiting on network I/O
- backlog in Kafka keeps growing
- users see delayed completions

If autoscaling looks only at CPU:

- CPU might sit at 25%
- autoscaler decides "no need to scale"
- but queue lag is exploding
- user-visible pressure is high even though CPU is low

So CPU is loosely correlated with the real problem.

Better signal:
- queue lag
- oldest message age
- in-flight work
- job completion delay

Another example:

- scaling webhook pods on memory
- memory is flat
- incoming callback rate spikes
- request latency and 5xx rise
- memory never tells you the ingress is overloaded

Better signal there might be:
- request concurrency
- request rate
- CPU, if the work is CPU-bound

Step 9. Roll up to cluster capacity

After each module has a pod-level estimate, compute the cluster footprint.

At minimum:

  • total CPU request
  • total memory request
  • total CPU limit
  • total memory limit
  • expected peak replica count
  • headroom above the initial estimate

Useful formula:

  • cluster_request = sum(module_replica_count * per_pod_request)
    • per_pod_request means the resources for one pod

This is the point where node shape and node count become visible.

From cluster demand to hardware choice

Once cluster demand is known, choose hardware based on the aggregate resource shape.

At minimum, evaluate:

  • total CPU request
  • total memory request
  • expected peak replica count
  • max pods per node
  • required headroom for failover and burst
  • cost of fewer large nodes vs more small nodes

Use allocatable capacity, not raw instance capacity. Node overhead, kube-system pods, logging, monitoring, and daemonsets all reduce what the workload can actually use.

General hardware guidance

Pick node families based on workload behavior, not just on what is already available in the cluster.

Common patterns:

  • general-purpose nodes Best when CPU and memory are reasonably balanced.
  • compute-optimized nodes Best when pods are CPU-heavy and memory-light.
  • memory-optimized nodes Best when pods hold large heaps, caches, or buffers.
  • storage-optimized nodes Relevant only if the workload is locally disk-sensitive.

Questions to answer before choosing node size:

  • Is the workload CPU-bound, memory-bound, or I/O-bound?
  • Does the service need many small pods or a few large pods?
  • Is the workload bursty enough that node autoscaling speed matters?
  • Does the workload need x86 specifically, or can it run on ARM?
  • What is the blast radius if one large node fails?

Node sizes

  • prefer medium/large/xlarge style balanced nodes first
  • avoid very large nodes unless you need them
  • smaller nodes give better spread and lower failure blast radius
  • larger nodes reduce overhead but make one-node failure hurt more

AWS-style mapping:

  • m* family: default starting point
  • c* family: CPU-heavy services
  • r* family: memory-heavy services
  • x* family: only for very large memory cases

What decides the family?

  • CPU request dominating memory request across the module set -> c*
  • memory request dominating CPU request -> r*
  • neither dominates strongly -> m*

What decides xlarge vs 2xlarge vs bigger?

  • total cluster demand
  • max pods per node
  • desired failure blast radius
  • autoscaler behavior
  • bin-packing efficiency

Practical tradeoff:

  • fewer large nodes Lower control-plane overhead, but larger failure blast radius
  • more smaller nodes Better spread and bin-packing flexibility, but more cluster overhead

The hardware decision should come last. The service BOE should first explain pod shape and replica count, then derive node family and node count from that result.

The clean rule is:
- service BOE decides replicas and per-pod shape
- cluster BOE decides node family and node size

Step 10. Define the validation plan

Every BOE should state how it will be corrected after rollout.

Required follow-up checks:

  • observed CPU and memory per module
  • observed concurrency or backlog
  • observed error rate
  • observed burst windows
  • variance from the original traffic estimate

The estimate is only complete when the first production telemetry closes the loop.

Use this outline for future BOE writeups:

  1. Scope
  2. Workload unit
  3. Metric contract
  4. Traffic model
  5. Module traffic mapping
  6. Concurrency model
  7. Single-pod capacity assumptions
  8. Initial production allocation
  9. Autoscaling approach
  10. Cluster rollup
  11. Risks and unknowns
  12. Validation plan

Common failure modes

  • using monthly totals without any peak analysis
  • sizing all modules from one top-line metric
  • ignoring service time and concurrency
  • assuming CPU usage scales linearly with external I/O-heavy work
  • copying a DevOps baseline without validating the workload model
  • skipping headroom
  • omitting the post-rollout validation plan

Practical rule

Volume metrics identify demand.

Service time and pod telemetry convert demand into concurrency.

Concurrency and per-pod capacity convert concurrency into replicas and cluster size.