NotCheap MAKE EVERY CREDIT COUNT
Analyze Your Costs

GPU rental costs: compare the job, the H100 variant and idle time

Let's start with the math that trips up a lot of budgets. Ten thousand GPU-hours at a listed $2.89 per GPU-hour cost $28,900. But if only 80% of your billed time is productive, those same 10,000 productive GPU-hours require 12,500 billed hours and cost $36,125 before storage or other charges. Here's how to see that gap coming before it shows up on an invoice.

H100 is a hardware family, not an interchangeable commodity

Here's a mistake I see good teams make all the time: they budget for "an H100" as if it were one SKU with one market price. In reality, an H100 PCIe card, an H100 SXM5 module, and an H100 NVL setup behave quite differently in memory bandwidth, interconnect speed, and thermal headroom. A bare PCIe card tops out at roughly 2.0 TB/s of VRAM bandwidth, while an SXM5 unit delivers over 3.3 TB/s and pairs with 900 GB/s NVLink interconnects for multi-GPU scaling. Say you're fine-tuning a 70B parameter model across an 8-GPU cluster. On PCIe without fast interconnects, your pricey tensor cores spend a lot of time sitting around waiting on gradient synchronization. The cheaper hourly rate looks great on paper, until your training run takes twice as long and the total bill ends up higher.

Observed listing or routeDisplayed basisWhat remains to compare
Runpod H100 PCIe, 80GB$2.89 per hour on the observed pricing pageSelected cloud/region, availability and attached resources
Lambda single-GPU H100 PCIe, 80GB$3.29 per GPU-hourInstance configuration and availability
Lambda single-GPU H100 SXM, 80GB$4.29 per GPU-hourWhether the workload benefits from the configuration
Vast marketplaceLive on-demand, interruptible or reserved offersHost, rental type, resources and the complete current quote

What to keep in mind: these are point-in-time listings, not benchmarked guarantees of cost-per-token or training throughput. The contract and execution environment matter as much as the silicon. A persistent dedicated cloud pod, a serverless container endpoint, and a committed multi-node cluster each come with different cold-start latencies, idle billing rules, and minimum commitments. So before you get attached to a number, check whether the advertised rate covers a standalone single GPU or requires provisioning an entire 8-way node upfront.

Productive compute time is your true budget lever

If I could get every team to look at just one number, it would be GPU utilization. It's usually the biggest gap between the spreadsheet and the monthly invoice. You pay for the clock time the VM or container stays assigned to your account, not just the seconds your PyTorch kernels are actually crunching matrices. Setup steps, pulling multi-gigabyte container images from Docker registries, downloading model weights from Hugging Face, tokenizer pre-processing, periodic checkpoints written to network volumes, and recovering from out-of-memory crashes all tick against your billed clock. (Yes, even the crash you're staring at while you sip your coffee.)

10,000 productive GPU-hours at $2.89Billed hoursCompute cost
100% productive10,000$28,900
80% productive12,500$36,125
50% productive20,000$57,800

Say your workflow hits 80% productive utilization, which is typical for a well-orchestrated pipeline with automated warm workers and pre-cached datasets. Delivering 10,000 productive hours then consumes 12,500 billed hours. At $2.89 per hour, your real compute cost moves from $28,900 to $36,125, an added $7,225 of overhead. If orchestration is looser, with manual notebook exploration or idle waiting between evaluation splits, utilization can easily slide to 50%. At that point you burn 20,000 billed hours and $57,800 for the exact same amount of delivered work. The good news is that engineering time spent on automated warm restarts, local SSD scratch caching, and fast checkpoint serialization will usually save far more than hunting for a fifteen-cent discount on the headline GPU rate.

Count GPU-hours and instance-hours correctly

Another easy trap is mixing up instance-hours and individual GPU-hours. If you provision an 8-GPU node on RunPod or Lambda and let it run uninterrupted for 24 hours, you haven't used 24 hours of compute. You've used 192 GPU-hours. At a $2.89 hourly rate per GPU, that single day costs $554.88, not $69.36. So it's worth double-checking whether a vendor's checkout screen quotes the cost per individual accelerator or the aggregate hourly cost for the whole physical host. Multiply an already-bundled chassis price by the accelerator count and you'll inflate your budget eightfold. Divide an accelerator rate across a cluster the wrong way and your accounting team gets a surprise.

Continuous uptime math catches people off guard too. Leave a single dedicated H100 instance on across a standard 30-day billing month and it racks up 720 hours. At $2.89 per hour, that baseline instance costs $2,080.80 every month before network storage or data egress. If your team only runs inference batches for four hours each afternoon, an always-on dedicated box costs five times more than serverless endpoints or instances that shut themselves off when queue depth hits zero.

Marketplace spots and interruptible capacity demand a recovery plan

Marketplaces like Vast.ai can offer big discounts on interruptible or unreserved GPU capacity, sometimes landing well below tier-1 cloud rates. The practical catch: the cheapest listing is only the cheapest completed job if your software stack is built to handle frequent interruptions. On a peer-to-peer or decentralized marketplace, a host can disappear, lose connectivity, or be reclaimed by a higher bidder with little to no warning.

If your training script doesn't save incremental checkpoints to detached, durable cloud storage every 15 to 30 minutes, an interruption can wipe out hours of uncommitted progress. That GPU time is gone, and you'll pay for more hours reloading weights and recompiling computational graphs. Before you bet a critical deadline on spot pricing, run a test that deliberately simulates host eviction. Measure how many minutes it takes to spin up a replacement node, mount your network volume, restore the optimizer state, and resume gradient descent. On Lambda and RunPod, it's also worth weighing spot savings against reserved cluster agreements: a 1-year commitment can unlock significant discounts for predictable long-term workloads, without the operational headache of constant eviction.

Benchmark a representative workload before signing a contract

Please don't pick a cloud GPU provider based only on a pricing table or synthetic FLOPS benchmarks. Instead, set up a quick, isolated proof-of-concept with your exact production stack: the same base model, precision format (such as FP8 or BF16), batch size, sequence length, and evaluation pipeline. Time the end-to-end run from initial container boot to final checkpoint upload, and log your sustained tokens-per-second, VRAM headroom, network transfer latencies, and total elapsed wall-clock time.

While you're at it, tally the ancillary fees that don't show up in the headline GPU rate: provisioned NVMe scratch disks, persistent network volume storage (often billed at $0.10 to $0.30 per GB per month), and outbound data egress if you frequently stream checkpoints off-platform. And as a final sanity check, ask whether you need dedicated GPUs at all. For many inference-heavy products, querying managed APIs, using our Qwen or DeepSeek cost frameworks as a starting point, can deliver equivalent capabilities with zero idle overhead and zero cluster maintenance.

Price productive GPU work

Starts with the observed $2.89 H100 PCIe listing and an assumed 80% productive share. Enter a per-GPU-hour rate. Your productive-hour target must already include the GPU count. Add storage, transfers or other billed extras to fixed payments.

12,500 billed GPU-hours estimated for this job.

$36,125.00 for this scenario · $3.6125 per productive GPU-hour

Billed GPU-hours = productive GPU-hours × billed/productive factor. Compute charge = billed GPU-hours × per-GPU-hour quote. At 80% productive share, the factor is 1 ÷ 0.8 = 1.25.

Compare a hosted model API with local operation →

Sources

PASS IT ON

Useful for your team?

Share this guide with a colleague.