Together AI has introduced Together Compiler, an agent-driven system for continually optimizing model inference. The November 4 release targets a familiar infrastructure problem: a serving configuration that works well for one model, traffic pattern or GPU may perform poorly when those conditions change.
The technical announcement describes agents generating optimization candidates and testing them against persistent inference engines. Keeping an engine alive allows candidate forward passes to be loaded without repeating the full startup sequence. Together reports up to 2.6 times higher token throughput than its selected optimized baselines, with gains depending on the model, hardware and workload.
Production use and customer availability
In a separate product announcement, Together says the compiler already runs on some popular serverless endpoints. End-customer access is in limited beta. A self-service option for optimizing dedicated endpoints is described as forthcoming, so it should not be treated as generally available today.
Together calls the broader approach Polymorphic Inference: adapting the serving system as models, hardware and traffic evolve. The compiler is an implementation of that agenda, rather than a model whose weights improve automatically during a user conversation.
Why the workload matters
Our analysis is that the strongest idea here is measuring optimizations where they will actually run. A kernel that is fast on random inputs can behave differently under real routing patterns, context lengths and concurrency. Faster experimentation is valuable when it preserves those relevant conditions.
Infrastructure teams should keep several measurements separate. Token throughput describes capacity, while time to first token and response latency describe the experience of an individual request. A configuration can improve one while making another worse. The right choice depends on the service’s actual objectives.
Correctness also needs its own gate. Generated GPU code must preserve the computation within the precision and numerical tolerances the application accepts. A performance win should be tested across representative inputs and failure cases before it becomes part of a production endpoint.
For a customer, the most useful trial would compare a stable baseline and an optimized configuration using the same request distribution. Record latency percentiles, cost per completed workload and error rates, then repeat when traffic changes. That reveals whether an adaptive process adds value beyond a one-time tuning exercise.
Together Compiler’s launch makes continual optimization a concrete product direction with a limited-beta route. Its reported speedups are promising vendor results; the broader significance is the attempt to make inference tuning track the conditions of a running service.