Principal Site Reliability Engineer

Key Responsibilities

• Design and architect reliable, secure, scalable, and maintainable infrastructure and services. Take proactive steps to ensure solutions meet defined reliability and functionality requirements. 
• Establish technical direction, engineering standards, and operational best practices across complex infrastructure and application environments. 
• Identify system dependencies, operational risks, capacity constraints, performance issues, and potential failure points before they affect service. 
• Translate business, client, security, and application requirements into practical infrastructure and reliability solutions. 
• Lead the installation, configuration, deployment, and validation of applications across Windows Server and Linux environments. 
• Oversee structured builds and deployments using runbooks, scripts, readiness assessments, change controls, and post-deployment validation. 
• Troubleshoot complex operating system, service, application, installation, patching, permissions, certificate, and connectivity issues. 
• Define and improve monitoring, alerting, logging, observability, capacity planning, and service-health practices. 
• Develop and promote automation that reduces manual effort, improves consistency, and lowers operational risk. 
• Lead operating system, middleware, and application patching initiatives, including change planning, rollback preparation, execution, and validation. 
• Direct major incident response, root cause analysis, corrective-action planning, and prevention of recurring failures. 
• Partner with cybersecurity teams on vulnerability remediation, system hardening, STIG compliance, and other security-driven changes. 
• Evaluate emerging technologies and recommend solutions that improve reliability, resilience, security, and operational efficiency. 
• Create and maintain technical standards, architecture documentation, runbooks, deployment procedures, and troubleshooting guides. 
• Provide technical leadership, mentorship, and design guidance to engineers across multiple teams. 
• Communicate technical risks, dependencies, decisions, and recommendations clearly to leadership and stakeholders. 

Core Skills and Qualifications

Technical Leadership and Architecture

• Extensive experience in site reliability engineering, systems engineering, infrastructure architecture, production operations, or application hosting. 
• Demonstrated ability to design and support highly available, resilient, and secure enterprise systems. 
• Experience leading complex technical initiatives across engineering, operations, security, networking, and application teams. 
• Ability to make sound architectural decisions, evaluate tradeoffs, and communicate recommendations to technical and nontechnical stakeholders. 
• Experience defining engineering standards, operational controls, and reliability practices. 

Windows and Linux System Administration

• Advanced, hands-on experience administering Windows Server and/or Linux systems. 
• Ability to access deployed hosts and perform post-deployment configuration, troubleshooting, and validation. 
• Experience installing, configuring, and validating applications in Windows Server and Linux environments. 
• Ability to resolve operating-system-level, service-level, and application-level issues. 
• Strong knowledge of system services, permissions, configuration files, logs, processes, and resource utilization. 

Manual Build and Deployment Experience

• Experience leading structured build and deployment activities using runbooks, deployment guides, scripts, and technical procedures. 
• Ability to execute and troubleshoot scripts, validate outputs, and resolve build or configuration issues. 
• Experience with build handoffs, environment-readiness assessments, deployment validation, and post-build verification. 
• Ability to identify process gaps, document exceptions, and improve deployment procedures. 
• Experience managing complex or high-risk production changes. 

Troubleshooting and Operational Support

• Ability to investigate complex service failures, installation errors, patching failures, application startup problems, permissions issues, and connectivity incidents. 
• Experience reviewing logs, event viewers, service status, configuration files, ports, certificates, and access controls. 
• Strong analytical and problem-solving skills, with the ability to isolate root causes and implement sustainable solutions. 
• Experience leading major incident response and coordinating technical teams during business-critical outages. 
• Ability to document symptoms, findings, impact, corrective actions, and recommended next steps clearly. 
• Extensive experience supporting production or other mission-critical environments. 

Scripting and Automation

Advanced hands-on experience with one or more of the following:

• PowerShell 
• Bash 
• Python 
• Ansible 
• Chef 

Candidates should be able to create, modify, validate, and troubleshoot scripts and automation workflows. Experience identifying automation opportunities and establishing safe, repeatable operational processes is essential.

Patching and Software Maintenance

• Experience planning and executing operating system, middleware, and application patching. 
• Ability to troubleshoot patch failures, compatibility issues, and post-patch application problems. 
• Strong understanding of maintenance windows, change control, rollback planning, risk assessment, and post-change validation. 
• Experience coordinating patching and remediation activities across application, infrastructure, cybersecurity, and client teams. 

Cloud and Oracle Cloud Infrastructure

• Strong understanding of cloud-hosted and hybrid infrastructure. 
• Experience with Oracle Cloud Infrastructure or another major cloud platform. 
• Knowledge of cloud compute, storage, networking, identity, access management, load balancing, and environment provisioning.  
• Experience designing or supporting reliable, secure, and scalable cloud environments. 
• Familiarity with infrastructure-as-code and configuration-management practices. 

Network and Connectivity Troubleshooting

• Working knowledge of DNS, firewalls, routing, load balancers, ports, certificates, and network communication between systems.  
• Ability to identify and isolate host, application, certificate, firewall, DNS, and routing-related issues. 
• Familiarity with standard connectivity and network diagnostic tools. 

Cybersecurity and Compliance

• Experience with vulnerability remediation, system hardening, secure configuration, and compliance-driven infrastructure changes. 
• Familiarity with Security Technical Implementation Guides and federal cybersecurity requirements. 
• Ability to implement security remediation without disrupting application functionality or service availability. 
• Experience supporting federal, government-hosted, healthcare, or other regulated environments is highly valued. 

Documentation and Communication

• Ability to create and maintain architecture documentation, operational standards, technical procedures, runbooks, and change records.  
• Strong attention to detail when documenting completed work, risks, exceptions, decisions, and validation results. 
• Experience working in ticketing, incident-management, problem-management, and change-management systems. 
• Excellent written and verbal communication skills. 
• Ability to present complex technical issues, risks, and recommendations to engineers, clients, and senior leadership. 
• Demonstrated ability to mentor engineers and influence technical direction across teams. 

Preferred Qualifications

• Experience supporting Oracle Cloud Infrastructure environments. 
• Experience supporting Oracle Health, Millennium, or Cerner applications and related infrastructure. 
• Knowledge of federal cybersecurity workflows, STIG implementation, and compliance requirements. 
• Experience supporting federal clients or government-hosted environments. 
• Experience with Citrix technologies. 
• Experience supporting legacy infrastructure and business-critical legacy applications. 
• Advanced experience with infrastructure-as-code or configuration-management tools. 
• Experience with production support, incident command, problem management, and SRE operational practices. 
• Knowledge of service-level indicators, service-level objectives, error budgets, observability, and capacity planning. 
• Experience designing high-availability, disaster-recovery, backup, and service-continuity solutions.