SIte Reliability Engineer, AI Inference

TheAI Inference Engineerplays a critical role in the AI lifecycle by bridging the gap between high-performance model development and optimized deployment environments. This position focuses onoptimizingLarge Language Models (LLMs)for inference, serving diverse environments—from GPU-rich data centers to resource-constrained edge devices—with a strong emphasis on maximizing throughput, minimizing latency, andmaintainingmodel accuracy. 

This role is pivotal in advancing F5’s AI capabilities, ensuring enterprise-grade reliability byleveraginghardware acceleration, designing scalable infrastructure, andmonitoringsystem performance.

Key Responsibilities

High-Performance AI Serving

• Build andmaintainrobust inference engines using tools likevLLM,TGI (Text Generation Inference), andNVIDIA Triton, ensuring high performance at scale. 

• Handle deployment optimizations to deliver low-latency AI serving solutions for multiple business applications.

Hardware Acceleration and Optimization

• Profile andoptimizemodels for specialized hardware backends, includingNVIDIA GPUs(CUDA/TensorRT),Apple Silicon (CoreML), and AI accelerators likeTPUsandLPUs. 

• Collaborate with hardware teams to maximizeutilizationand performance across various computational environments.

Inference Orchestration and Scalability

• Design and implementauto-scaling architecturesfor online (real-time) and batch inference pipelines,leveragingKubernetesfor inference routing and orchestration. 

• Ensure software solutions areoptimizedfor peak performance during traffic spikes,maintainingreliability and scalability.

Performance Monitoring and Observability

• Establish robust observability frameworks tomonitorTime to First Token (TTFT), tokens per second, and memory bandwidthutilizationagainst service-level agreements (SLAs). 

• Build and executeperformance and load testing suitestoidentifybottlenecks and ensure consistent reliability at scale.

Technical Requirements

Required Skills:

• Programming Languages:Proficiencyin programming languages such asPython,C++,Rust, orGolangspecifically for high-performance AI workflows. 

• Inference Tools:Proven hands-on experience with tools likevLLM,TensorRT,Llama.cpp, andOllamafor inference development and optimization. 

• Infrastructure Expertise:Strong familiarity with infrastructure technologies, includingDocker,Kubernetes, and cloud platforms such asAWS,GCP, andAzure. 

• Hardware Optimization Expertise:Comprehensive understanding of GPU and AI hardware, including techniques for profiling andoptimizingperformance for accelerators like NVIDIA GPUs and TPUs.

Preferred Experience:

• Prior experience deployingLarge Language Models (LLMs)with advanced techniques likeSpeculative DecodingorPagedAttention. 

• Contributions toopen-source inference librariesor hardware-level kernel development (e.g., CUDA, Triton kernels). 

• Background inMLOpsorSREroles focused onhigh-performance AI endpointsand reliability during demand surges. 

• Proficiencyin designing scalable solutions for high-throughput inference environmentsoptimizedfor traffic bursts.

Success Metrics (KPIs):

• Latency Reduction:Continuously improve inference latency metrics, ensuring minimalTime to First Token (TTFT)andmaximumtokens per second. 

• Cost Efficiency:Achieve lower "Cost per 1K Tokens" through better resourceutilizationand hardware optimization. 

• Scalability:Maintainsystem stability and reliability duringtraffic spikes, ensuring performance consistency across environments. 

• Throughput Maximization:Deploy modelsoptimizedfor peak hardware usage and maximized process throughput.

Why Join F5?

F5 empowers you to push boundaries inAI optimizationandhigh-performance engineering. Joining our team means: 

• Collaborating withcutting-edgetechnologies and hardware solutions to support real-time AI applications. 

• Advancing your career in a fast-paced, multidisciplinary environment focused on innovation, scalability, and problem-solving. 

• Driving transformative projects that deliver real-time AI reliability to global customers whilemaintainingcost and efficiency standards. 

• Working on advancedMLOpssolutionsthat seamlessly scale enterprise AI systems and shape the future of intelligent deployment.

What Success Looks Like:

As anAI Inference Engineerat F5, success is measured by your ability to: 

• Combine technicalexpertiseand problem-solving skills to deliver low-latency, scalable, and high-performing AI prediction systems. 

• Collaborate efficiently across cross-functional teams,participatingin knowledge sharing and system refinement. 

• Demonstrate initiative by driving optimizations across hardware, tools, and orchestration processes, balancing immediate solutions with long-term architectural goals. 

• Translatecomplex AI and inference workflows into practical solutions that align with F5's strategicobjectives.

#LI-AK1