We gave models easy, medium, and hard AWS challenges and on the hardest challenge, the cheapest model cost 33x less than the most expensive.
33x cost spread among perfect scorers
7 models went 3-for-3 on the hard challenge
3 open-weight models in the top 4 by cost
Figure 1. What each model cost to complete the hard challenge. Cost derived from recorded tokens at OpenRouter rates as of 24 June 2026. Each line spans the three-run range. The dashed line separates models that passed all three runs from those that did not.
We set out to see how well a range of AI models would perform building AWS infrastructure and found high success across all cohorts on 3 tiers of challenges over multiple test runs. On our hardest AWS challenge, seven models went 3-for-3 with the cheapest model, MiMo v2.5, averaging $0.038 per run versus the most expensive, Claude Sonnet 4.6 which averaged $1.24 per run.
Is Michelangelo's ability necessary to paint your spare bedroom?
MiMo and Claude both went 3-for-3 on each challenge, except MiMo averaged $0.038 per run while Claude averaged $1.24, roughly a 33x difference in cost for the task at hand.
The 11 models did not form one smooth price curve, either. MiMo, DeepSeek, Grok, and Qwen made up a sub-$0.16 cluster. GPT averaged $0.54, Gemini $1.10, and Claude $1.24.
Figure 2. Why the clusters land where they do: output tokens are priced 4 to 10x higher than input tokens depending on the model. Verbose models pay a compounding penalty, three-tier challenge.
Figure 3. Cost vs score, all 11 models, hard challenge. Mean values across three runs, cost on a log scale, bubble size reflects mean turns to complete the challenge.
In our tests, we had each model perform 3 runs per challenge, and all models succeeded in the easy and medium challenges 100%.
The hard challenge finally broke the tie with 29 perfect runs out of 33, with seven models going 3-for-3. Even the unsuccessful attempts were remarkably close: three models missed a single role-name grader check just once, while one model encountered an unreachable service during only one trial.
Figure 4. Hard challenge results, AWS 3-tier architecture. 7 models were successful for all 3 runs. The other 4 were successful 2 out of 3 runs.
If you're not experimenting with a broader range of models for cloud infrastructure tasks yet, you should be. In our testing, we found that capabilities for reliably building cloud infrastructure are high across all models we tested, making cost the differentiating factor.
Resist defaulting to the familiar or cheapest-per-token options as your first consideration. A solid evaluation starts with defining the outcome and its acceptance checks, a skillset your cloud architect is well suited for. Run each candidate repeatedly, remove the ones that fail the checks, and compare spend across the ones that remain.
Evaluate models beyond just the popular choices for cloud infrastructure tasks. For example in our tests, we were surprised at how well some open-weight models performed, sparking speculation as to whether self-hosting could be a viable option for more cloud teams in the near future.
Figure 5. The self-hosting case in numbers: on the three-tier challenge, open-weight models averaged $0.378 per run against $0.605 for closed-weight models, a 1.6x gap.
Figure 6. Cost as the challenge gets harder. Mean modeled cost per run at each challenge tier, OpenRouter rates dated 24 June 2026. Grok and DeepSeek grow roughly 9-10x from easy to hard but stay under $0.16 throughout. GLM 5.1 grows 240x, from $0.0043 on easy to $1.03 on hard, ending pricier than all but two models.
To conduct this benchmark, we evaluated 11 well-known AI models across multiple cohorts: mainstream frontier models like GPT and Sonnet, "challenger" options like Grok and Kimi, and open-weight models such as MiMo and DeepSeek.
We then crafted three challenges tiering up in complexity from easy, to medium, to hard. These tiers are rooted in our experience with cloud architecture patterns commonly found throughout enterprise cloud environments.
Figure 7. The benchmark loop. A failed deploy or a failed check sends the model back to try again, up to the turn cap. Only a validated, running result counts as a pass.
Our benchmark tool orchestrated each model to generate Terraform from scratch based on specifications, inspecting and revising files upon failure. The orchestrator then deployed results to a live AWS account, where a deterministic grader queried resources and tested the running system to determine success. A failed deploy or a failed check sent the model back to try again, up to a turn cap. Only a validated, running result counted as a pass.
Figure 8. What that retry loop looked like in practice: turn counts on the three-tier challenge spread from 2 (Grok, DeepSeek) to 10. Sub-100 scorers needed more correction turns to get there.
Figure 9. Terraform and AWS provisioning is roughly constant across models and runs, averaging around 850 seconds. Total wall-clock time varies far more, driven by model generation, three-tier challenge.
Capability gaps are converging for cloud infrastructure. Across the 11 models in our tests, the tasks stopped being the differentiator, cost did. These cost differences were significant, with a 33x spread between models achieving the same outcome.
Models you might tend to select based on reputation or familiarity can cost dramatically more than other options that would also meet the requirements for the task. Our tests found that great results could be achieved without relying on the frontier models most teams likely default to.
Figure 10. The cost spread, by challenge tier. Spread is the priciest model's cost divided by the cheapest, among models that scored 100 at that tier. All costs at OpenRouter rates dated 24 June 2026.
The ability to define outcomes and validate them with deterministic acceptance tests is a valuable capability for cloud teams to develop and refine as the model landscape evolves. Teams that master this ability will be leaders in leveraging AI for business value without compromising quality of result.