Senior Site Reliability Engineer, AI Inference

Job Title: AI Inference Engineer

Role Objective

TheAI Inference Engineerplays a critical role in the AI lifecycle by bridging the gap between high-performance model development and optimized deployment environments. This position focuses on optimizingLarge Language Models (LLMs)for inference, serving diverse environments—from GPU-rich data centers to resource-constrained edge devices—with a strong emphasis on maximizing throughput, minimizing latency, and maintaining model accuracy. 

This role is pivotal in advancing F5’s AI capabilities, ensuring enterprise-grade reliability by leveraging hardware acceleration, designing scalable infrastructure, and monitoring system performance.

Key Responsibilities

High-Performance AI Serving

• Build and maintain robust inference engines using tools likevLLM,TGI (Text Generation Inference), andNVIDIA Triton, ensuring high performance at scale. 

• Handle deployment optimizations to deliver low-latency AI serving solutions for multiple business applications.

Hardware Acceleration and Optimization

• Profile and optimize models for specialized hardware backends, includingNVIDIA GPUs(CUDA/TensorRT),Apple Silicon (CoreML), and AI accelerators likeTPUsandLPUs. 

• Collaborate with hardware teams to maximize utilization and performance across various computational environments.

Inference Orchestration and Scalability

• Design and implementauto-scaling architecturesfor online (real-time) and batch inference pipelines, leveragingKubernetesfor inference routing and orchestration. 

• Ensure software solutions are optimized for peak performance during traffic spikes, maintaining reliability and scalability.

Performance Monitoring and Observability

• Establish robust observability frameworks to monitorTime to First Token (TTFT), tokens per second, and memory bandwidth utilization against service-level agreements (SLAs). 

• Build and executeperformance and load testing suitesto identify bottlenecks and ensure consistent reliability at scale.

Technical Requirements

Required Skills:

• Programming Languages:Proficiency in programming languages such asPython,C++,Rust, orGolangspecifically for high-performance AI workflows. 

• Inference Tools:Proven hands-on experience with tools likevLLM,TensorRT,Llama.cpp, andOllamafor inference development and optimization. 

• Infrastructure Expertise:Strong familiarity with infrastructure technologies, includingDocker,Kubernetes, and cloud platforms such asAWS,GCP, andAzure. 

• Hardware Optimization Expertise:Comprehensive understanding of GPU and AI hardware, including techniques for profiling and optimizing performance for accelerators like NVIDIA GPUs and TPUs.

Preferred Experience:

• Prior experience deployingLarge Language Models (LLMs)with advanced techniques likeSpeculative DecodingorPagedAttention. 

• Contributions toopen-source inference librariesor hardware-level kernel development (e.g., CUDA, Triton kernels). 

• Background inMLOpsorSREroles focused onhigh-performance AI endpointsand reliability during demand surges. 

• Proficiency in designing scalable solutions for high-throughput inference environments optimized for traffic bursts.

Success Metrics (KPIs):

• Latency Reduction:Continuously improve inference latency metrics, ensuring minimalTime to First Token (TTFT)and maximum tokens per second. 

• Cost Efficiency:Achieve lower "Cost per 1K Tokens" through better resource utilization and hardware optimization. 

• Scalability:Maintain system stability and reliability duringtraffic spikes, ensuring performance consistency across environments. 

• Throughput Maximization:Deploy models optimized for peak hardware usage and maximized process throughput.

Why Join F5?

F5 empowers you to push boundaries inAI optimizationandhigh-performance engineering. Joining our team means: 

• Collaborating with cutting-edge technologies and hardware solutions to support real-time AI applications. 

• Advancing your career in a fast-paced, multidisciplinary environment focused on innovation, scalability, and problem-solving. 

• Driving transformative projects that deliver real-time AI reliability to global customers while maintaining cost and efficiency standards. 

• Working on advancedMLOpssolutionsthat seamlessly scale enterprise AI systems and shape the future of intelligent deployment.

What Success Looks Like:

As anAI Inference Engineerat F5, success is measured by your ability to: 

• Combine technical expertise and problem-solving skills to deliver low-latency, scalable, and high-performing AI prediction systems. 

• Collaborate efficiently across cross-functional teams, participating in knowledge sharing and system refinement. 

• Demonstrate initiative by driving optimizations across hardware, tools, and orchestration processes, balancing immediate solutions with long-term architectural goals. 

• Translatecomplex AI and inference workflows into practical solutions that align with F5's strategic objectives.