Inference Optimization Engineer (Sr Engineer / Sta5 Engineer / Principal Engineer)

Optimize and serve trained ML/LLM models to real-world users focusing on speed (TTFT, tokens/s), accuracy, and cost efficiency.

Apply model compression techniques (quantization, pruning, distillation) to reduce model size without accuracy loss.

Tune runtime and serving frameworks, manage memory/caching for long contexts and multiple users, and profile GPU/TPU bottlenecks for hardware co-optimization.

4+ years of experience in systems programming, HPC, or ML infrastructure.

Strong proficiency in Python or Go; familiarity with CUDA/Triton kernel development.

Production experience with inference engines like vLLM, TensorRT-LLM, or SGLang and profiling tools like Nsight.

Deep knowledge of transformer architectures and related hardware performance concepts.

Experienced in balancing inference latency, throughput, and accuracy trade-offs in ML deployment.

Skilled in low-level optimization including CUDA/Triton kernel tuning and distributed multi-GPU setups.

Hands-on with high-performance inference systems and tooling to identify and mitigate hardware bottlenecks.