Principal Engineer - Agentic Security Operations (remote)
First American · United States · 1d ago
Staff / Lead / PrincipalOn-siteSoftware Engineering
Who We Are
Join a team that puts its People First! Since 1889, First American (NYSE: FAF) has held an unwavering belief in its people. They are passionate about what they do, and we are equally passionate about fostering an environment where all feel welcome, supported, and empowered to be innovative and reach their full potential. Our inclusive, people-first culture has earned our company numerous accolades, including being named to the Fortune 100 Best Companies to Work For® list for eleven consecutive years. We have also earned awards as a best place to work for women, diversity and LGBTQ+ employees, and have been included on more than 50 regional best places to work lists. First American will always strive to be a great place to work, for all. For more information, please visit www.careers.firstam.com.What We Do
Our Security Operations Center (SOC) protects the enterprise by detecting, investigating, and responding to cybersecurity threats.As a member of our Threat Detection Engineering team operating within our SOC, you will build governed agent capabilities to support data onboarding, detection engineering, threat hunting, investigation, response, automation, and platform operations. Our goal is to expand threat visibility, accelerate security outcomes, reduce repetitive work, and improve operational consistency and resilience without compromising sound security judgment, human accountability, or control.
The Principal Engineer provides hands-on technical leadership to turn this strategy into secure, reliable, and supportable production capabilities across the SOC.
*We are open to remote candidates for this role
What You’ll Do
- Lead the architecture and hands-on implementation of Agentic Security Operations capabilities across the SOC.
- Define and evolve the reference architecture for governed data services, agent capabilities, workload routing, execution environments, and deterministic orchestration.
- Build reusable capabilities that translate operational intent into tested, reviewable, and auditable security outcomes.
- Establish engineering standards for creating, testing, registering, approving, releasing, monitoring, and retiring agents and their capabilities.
- Advance detection-as-code practices, including versioning, simulation, deployment, tuning, migration, and rollback.
- Build governed knowledge and context services that provide authoritative documentation, policies, operational history, citations, freshness information, and permission-aware retrieval.
- Design workload identity, least-privilege access, short-lived authorization, spend controls, approval gates, audit trails, rollback, and emergency shutdown mechanisms.
- Define autonomy levels that determine which activities may execute automatically, require human approval, or remain analyst led.
- Establish agent evaluation and observability through historical replay, synthetic scenarios, expected-versus-actual comparisons, policy tests, traces, outcome verification, and regression testing.
- Measure quality, security, reliability, latency, cost, analyst acceptance, and operational impact.
- Engineer reliable integrations across search, case management, workflow, communications, and response systems, including retries, idempotency, partial failures, and reconciliation.
- Ensure consequential response actions remain deterministic, attributable, reversible, and subject to appropriate human oversight.
- Design production reliability, continuity, recovery, and support models for agent capabilities and security data services.
- Lead proofs of concept and architecture evaluations using measurable criteria for security, fidelity, scale, reliability, portability, operating effort, and economic value.
- Collaborate with enterprise architecture, infrastructure, data, risk, privacy, legal, and compliance teams.
- Mentor engineers, establish technical standards, review designs, and influence Threat Detection Engineering strategy.
- Present complex technical decisions, risks, tradeoffs, and investment recommendations clearly to engineering and executive stakeholders.
- Partner across Threat Detection and Response groups, including engineering, threat hunting, security analysis, investigation, and incident response stakeholders on initiatives and to translate operational feedback into improvements within our agentic harness, evaluations, safeguards, workflows, and other platforms.
What You’ll Bring
- Extensive experience designing and operating enterprise security, software, data, or distributed systems at scale.
- Demonstrated success as a principal-level technical leader while remaining actively involved in architecture, prototyping, coding, testing, and production delivery.
- Hands-on experience delivering production AI-assisted or agentic systems that use tools, memory, retrieval, planning, and structured workflows.
- Strong understanding of security operations, including telemetry management, detection engineering, threat hunting, triage, investigation, response, and automation.
- Experience with high-volume streaming and batch data pipelines, object-based storage, elastic analytics, distributed query, schema management, and data-quality controls.
- Experience designing modular agents, reusable capabilities, workload routing, local and managed execution, and human-in-the-loop workflows.
- Deep understanding of grounding, retrieval, source attribution, context freshness, permission inheritance, and protection against untrusted or manipulated context.
- Experience evaluating nondeterministic systems using versioned test sets, historical replay, synthetic scenarios, regression testing, and measurable outcome criteria.
- Strong software-engineering practices, including source control, automated testing, continuous delivery, infrastructure automation, peer review, and rollback.
- Experience designing secure APIs, event-driven integrations, workload identities, authorization boundaries, secrets management, and auditable action execution.
- Knowledge of AI security risks such as prompt injection, data poisoning, excessive agency, sensitive-data exposure, insecure tool use, and model or workflow drift.
- Strong production-engineering experience covering observability, service objectives, capacity, failure isolation, continuity, recovery, and operational support.
- Ability to evaluate emerging capabilities critically, distinguish demonstrated functionality from product claims, and validate architectural assumptions through controlled testing.
- Excellent written and verbal communication skills, including the ability to document architectures, operating models, engineering standards, and executive recommendations.
- Ability to lead through influence and work effectively across organizational and technical boundaries.
- Hands-on experience with cloud-scale data and analytics platforms supporting security operations, including distributed query, high-volume data processing, object storage, schema management, data pipelines, and the operational patterns required to run security analytics at scale (e.g., Databricks, Snowflake, etc.)
- Experience operating and migrating security analytics and detection workloads across modern platforms, along with strong working knowledge of major public cloud environments (AWS, Azure, GCP) and the identity, networking, logging, data protection, and workload-security controls required to secure them.
- A minimum of 10 years’ experience in cybersecurity or relevant fields.
- Bachelor’s degree in computer science, Information Technology, Cybersecurity, a related field, or equivalent work experience.
- Preferred certifications such as, GASAE, GAIPS, GDSA, GCDA, CEH, OSCP, CISSP, etc.
Preferred Qualifications
- Experience modernizing a centralized security analytics environment into a decoupled architecture with customer-controlled data and replaceable compute or control layers.
- Experience with knowledge graphs, entity resolution, semantic normalization, retrieval-augmented systems, or multi-agent coordination.
- Experience migrating security content while validating semantic equivalence, historical fidelity, and operational parity.
- Background in detection engineering, incident response, threat hunting, adversary emulation, or security automation.
- Experience defining governance and autonomy frameworks for AI-enabled production systems.
- Experience building internal platforms or shared engineering services used by multiple security functions.
What Success Looks Like
- A leadership-approved Agentic Security Operations reference architecture and governance model.
- Governed agent prototypes progressing into reliable production capabilities.
- Reusable capabilities spanning data, detection, automation, and other SOC functions.
- Measurable reductions in engineering effort and operational cycle time without sacrificing security or human accountability.
- Continuous evidence that agent capabilities are accurate, secure, reliable, explainable, cost-effective, and safe to operate.
- A production platform with clear ownership, support procedures, recovery controls, and an intentional path away from vendor or architecture lock-in.
** Note that the following statements only apply to candidates who will be working from an unincorporated area within Los Angeles County. **
First American will consider for employment all qualified applicants, including those with arrest or conviction records, in a manner consistent with the requirements of applicable state and local laws (e.g., the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act).
First American intends to conduct a review of an applicant’s criminal history in connection with a conditional offer. First American reasonably believes that a criminal history may have a direct, adverse and negative relationship with the following material job duties for this position potentially resulting in the withdrawal of the conditional offer of employment: handling of confidential, proprietary or trade secret information belonging to First American or its customers, administrating or facilitating financial transactions, and the ability to meet customer-imposed criminal history requirements.