Text chat, live voice, image generation and overnight batch processing can all compete for the same GPU fleet. Fair scheduling helps, but some bursts require a stronger mechanism: preemption. Lower-priority work can be paused, checkpointed or evicted so real-time traffic receives capacity quickly enough to meet latency targets.

Define which workloads may be interrupted

Interactive voice and real-time agents usually have tight latency budgets, while offline embedding jobs or batch generations can wait. Classify workloads by interruption tolerance before designing scheduler rules.

A job that cannot safely resume should be harder to preempt because restarting from zero may waste more capacity than the interruption saves.

Estimate preemption cost

Evicting a model can require reloading weights later, and stopping a long generation may discard minutes of compute. The scheduler needs an estimate of restart cost rather than treating every queued job as interchangeable.

Checkpointable tasks are strong preemption candidates because they can resume close to where they stopped. Non-checkpointable work may be allowed to finish when it is already near completion.

Use priority bands instead of one queue

Separate real-time, near-real-time and batch classes provide clearer behavior than a continuously changing global score. Within each class, ordinary fairness and queue age can still determine order.

Priority bands also make product promises easier to explain. Premium live sessions can receive protected response latency while bulk generation is explicitly described as best effort.

Reserve a small non-preemptible floor

Some capacity should remain available for system health, critical control traffic and workloads that cannot tolerate interruption. Without a protected floor, cascading preemptions can make the entire fleet unstable during overload.

The size of this reserve can change by time of day and region, but it should be intentional rather than whatever capacity happens to remain unused.

Avoid preemption thrashing

If real-time load rises and falls rapidly, the scheduler can repeatedly stop and restart the same batch jobs. Hysteresis and minimum-run windows reduce this oscillation.

Metrics should track preemptions per job, wasted compute and time-to-completion so operators can see whether a latency optimization is creating excessive background inefficiency.

Coordinate with admission control and warm pools

Preemption should not be the first response to every spike. Warm pools reduce cold-start delay and admission control prevents the queue from growing beyond recoverable capacity.

Our article on AI inference admission control covers that boundary. Preemption becomes the mechanism for reallocating already-occupied capacity when priority changes after jobs have entered the system.

Mixed AI fleets need more than high average utilization. They need predictable behavior when interactive and batch demand collide. Preemption, checkpointing and service-aware priorities let the platform protect real-time experience without dedicating a separate GPU fleet to every workload.

Preemption policy should be reviewed together with customer promises. A batch customer that pays for guaranteed completion time may deserve reserved non-preemptible capacity, while an internal offline job can tolerate aggressive interruption. The scheduler should therefore understand service class, not only model priority. Finance teams also need visibility because repeated preemption can increase total GPU seconds if work must restart from scratch. The best policy protects interactive latency without quietly making batch economics worse, which is why checkpoint support, restart cost and completion deadlines belong in the same capacity model.

Preemption policy should be reviewed together with customer promises. A batch customer that pays for guaranteed completion time may deserve reserved non-preemptible capacity, while an internal offline job can tolerate aggressive interruption. The scheduler should therefore understand service class, not only model priority. Finance teams also need visibility because repeated preemption can increase total GPU seconds if work must restart from scratch. The best policy protects interactive latency without quietly making batch economics worse, which is why checkpoint support, restart cost and completion deadlines belong in the same capacity model.

Preemption policy should be reviewed together with customer promises. A batch customer that pays for guaranteed completion time may deserve reserved non-preemptible capacity, while an internal offline job can tolerate aggressive interruption. The scheduler should therefore understand service class, not only model priority. Finance teams also need visibility because repeated preemption can increase total GPU seconds if work must restart from scratch. The best policy protects interactive latency without quietly making batch economics worse, which is why checkpoint support, restart cost and completion deadlines belong in the same capacity model.

Preemption policy should be reviewed together with customer promises. A batch customer that pays for guaranteed completion time may deserve reserved non-preemptible capacity, while an internal offline job can tolerate aggressive interruption. The scheduler should therefore understand service class, not only model priority. Finance teams also need visibility because repeated preemption can increase total GPU seconds if work must restart from scratch. The best policy protects interactive latency without quietly making batch economics worse, which is why checkpoint support, restart cost and completion deadlines belong in the same capacity model.