Technical

In inference systems, there’s a tradeoff between total throughput (tokens/s) and interactivity (tokens/s/user). Batching more users increases utilization of the hardware but slows down the time per output token of any given user. As such, delivering faster tokens to the end user necessarily means that an inference provider is letting capacity be stranded, forcing those tokens to be priced higher. We see the economics play out in Fast Mode for Claude Opus 5, where the same model is priced more than five times higher per output token. ...

AWS Trainium ASICs are some of the highest volume inference accelerations deployed today alongside NVIDIA GPUs and Google TPUs. Anthropic has notably committed a significant amount of FLOPs to Trainium 2 and 3, collaborating with AWS on Project Rainier. Despite competitive perf/TCO and peak theoretical performance, a nascent software ecosystem leaves it relatively uncompetitive in the landscape of inference accelerators. ...