Skip to main content

Select efficient accelerators and instance types for AI workloads

Applicable Role: Provider and Consumer

Description​

AI workloads such as training, fine-tuning, and inference require significant compute resources. The type of hardware used, including CPUs, GPUs, TPUs, and specialized accelerators, has a direct impact on energy efficiency and performance.

Different hardware options vary in their ability to execute AI workloads efficiently. Selecting appropriate hardware and compute resources, combined with intelligent workload orchestration across heterogeneous platforms, improves utilization, reduces execution time, and lowers overall energy consumption..

Solution​

  • Profile workloads based on latency, throughput, and execution characteristics before selecting compute resources
  • Choose hardware optimized for the specific workload, such as CPUs, GPUs, TPUs, NPUs, or other specialized accelerators
  • Maintain a catalogue of supported accelerator types and their suitability for different workload classes
  • Use orchestration and scheduling systems to automatically dispatch workloads to the most energy-efficient available compute platform
  • Implement closed-loop monitoring and resource allocation to continuously optimize workload placement and avoid over-provisioning
  • Monitor utilization and dynamically adjust resource allocation to improve efficiency
  • Evaluate performance-per-watt benchmarks and runtime telemetry when selecting hardware and instance types

SCI Impact​

SCI = (E × I) + M per R

E (Energy): Efficient hardware reduces compute time and energy consumption for AI workloads.

M (Embodied Carbon): Better utilization can reduce the number of required machines and associated embodied emissions.

Cost Impact​

  • Hardware costs: Instance type choice affects hourly compute rates significantly
  • Utilization efficiency: Better hardware-workload fit reduces per-inference cost
  • Reserved instance savings: Efficient hardware selection enables better RI negotiation
  • Power and cooling costs: Specialized accelerators may have lower operational energy costs
  • Trade-off: Premium or specialized accelerators (GPUs, TPUs, NPUs, neuromorphic processors, or in-memory computing systems) may cost more upfront but can deliver significantly lower cost-per-inference and energy consumption for suitable workloads

Assumptions​

  • Suitable hardware options are available for the workload
  • Performance benchmarks reflect real-world usage

Considerations​

  • Specialized hardware may not be available in all regions
  • Costs may vary across hardware options
  • Migration to different hardware may require changes in software or frameworks
  • Underutilized hardware can negate efficiency gains

References​