Site Reliability Engineer - Memphis
xAI · Southaven, United States +1 · 23d ago
SpaceXAI’s mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge. Our team is small, highly motivated, and focused on engineering excellence. This organization is for individuals who appreciate challenging themselves and thrive on curiosity. We operate with a flat organizational structure. All employees are expected to be hands-on and to contribute directly to the company’s mission. Leadership is given to those who show initiative and consistently deliver excellence. Work ethic and strong prioritization skills are important. All employees are expected to have strong communication skills. They should be able to concisely and accurately share knowledge with their teammates.
ABOUT THE ROLE:
As a Site Reliability Engineer focused on campus reliability, you will design what the campus watches and trusts, technically command cross-discipline SEVs, and build the guardrails that make the next incident smaller. You are the connective tissue across compute, network, storage, power, and cooling. This role demands calm incident leadership, fleet-scale observability judgment, and the ability to drive reliability work across software and facility boundaries. We are looking for candidates from power plants, nuclear power plants, data centers, or people who currently work or have worked in facilities like SpaceXAI with power, compute, cooling, and related plant systems.
RESPONSIBILITIES:
- Own monitoring architecture and signal quality: what we alert on, suppress, and trust. Consume NOC noise-disposition feedback to drive suppression and redesign. Treat alert noise as a design failure, not an operator failure.
- Provide SEV command support: technical incident leadership, bridge coordination with the NOC, and timeline and severity hygiene.
- Run blameless postmortems and drive corrective actions to closed, not filed.
- Lead cross-functional reliability projects spanning compute, network, storage, and facility signal boundaries.
- Build and maintain playbooks, run game days, and keep cross-discipline dependency maps current. Own runbook quality jointly with the NOC (SRE designs; NOC operates and corrects).
- Define error budgets and availability objectives at campus and service boundaries as adopted by the business.
- Participate in on-call rotations and incident response for SEV-class events in the Memphis / Southaven data center campus.
PROFILE SIGNALS:
- Experience in power plants, nuclear power plants, data centers, or facilities like SpaceXAI — current or prior — spanning power, compute, cooling, or related plant systems.
- Proven large-scale incident command experience and calm technical leadership on a bridge.
- Demonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality. Treats alert noise as a design failure, not an operator failure.
- Cross-team facilitation and systems engineering depth across software and facility boundaries.
SUCCESS MEASURED BY:
- MTTD / MTTR trend for SEV-class events
- % of SEVs with a blameless postmortem and closed action items
- Alert actionable ratio
- Recurrence rate of incident classes
- Monitoring coverage of critical dependencies across compute, network, storage, power, and cooling
BASIC QUALIFICATIONS:
- Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or a related field (or equivalent experience).
- 5+ years of experience in site reliability, systems engineering, plant operations, or large-scale production operations, preferably in power plants, nuclear power plants, high-performance computing, or data center environments.
- Proven large-scale incident command experience and calm technical leadership on a bridge.
- Demonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality.
- Experience working across at least two of: compute, network, storage, power, and cooling / facilities telemetry.
- Experience writing and operating playbooks or runbooks with a 24/7 operations, control room, or NOC partner.
- Proficiency in scripting (Python, Bash) for automation and analysis, plus general experience in at least one systems language (C, C++, Java, Go, Rust, or similar). Not required to be expert in all of them.
- Excellent problem-solving skills with a data-driven approach to reliability engineering.
- Ability to work collaboratively with cross-functional teams, including NOC, data center operations, and infrastructure engineering.
PREFERRED SKILLS AND EXPERIENCE:
- Experience in power plants, nuclear power plants, or other high-consequence industrial control environments.
- Current or prior work in data centers or facilities like SpaceXAI spanning power, compute, cooling, and related plant systems.
- Experience in AI/ML infrastructure or supercomputing environments.
- Hands-on definition and use of SLOs, SLIs, and error budgets at service or campus boundaries.
- Experience running game days, dependency mapping, and closed-loop corrective action programs.
- Familiarity with data center hardware and plant signals (servers, GPUs, networking, power, cooling) in addition to software telemetry.
- Prior work in a fast-paced startup or tech company like SpaceXAI.
SpaceXAI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.