LLM Research & Validation Specialist
Abu Dhabi Islamic Bank · United Arab Emirates · 5d ago
Job title: LLM Research & Validation Specialist
Location: Abu Dhabi, UAE
Role purpose:
Lead frontier research, quantitative evaluation and independent validation of large language models, multimodal models, retrieval-augmented generation systems and agentic AI used or proposed by ADIB. Translate mathematical and scientific methods into reproducible validation tests, challenger analyses, runtime controls and decision-useful evidence for model governance.
The role combines deep technical research with second-line effective challenge.
It is expected to build validation toolkits and evaluation harnesses, independently assess conceptual soundness and production behaviour, and communicate material limitations clearly to technical teams, senior management and governance forums.
The role does not own model development or production approval.
Key accountabilities /responsibilities:
Lead independent validation of LLM, multimodal, RAG and agentic AI use cases across design, implementation, deployment and ongoing monitoring.
Assess transformer architecture, tokenisation, embeddings, attention, context-window behaviour, decoding, fine-tuning, alignment, quantisation and inference configuration.
Design reproducible evaluation harnesses, golden datasets, adversarial suites, counterfactual tests, canary sets and statistically defensible acceptance criteria.
Evaluate task performance, hallucination and factuality, calibration, robustness, stability, long-context behaviour, retrieval quality, grounding, citation faithfulness and uncertainty.
Perform deep testing of prompt injection, indirect injection, data leakage, tool-use safety, excessive agency, multi-step failure propagation, kill-switches and human oversight.
Apply probability, statistics, optimisation, information theory, numerical methods and experimental design to develop challenger tests and quantify uncertainty.
Review data provenance, representativeness, contamination, benchmark validity, leakage, drift and limitations of synthetic or LLM-generated evaluation data.
Build and maintain reusable Python-based validation tooling, automated test pipelines, experiment tracking, results repositories and technical documentation.
Conduct structured research on emerging model architectures, interpretability, mechanistic analysis, scalable oversight, model evaluation and AI safety methods.
Independently challenge model owners, vendors and developers, document findings, propose risk-based restrictions and track remediation without assuming first-line ownership.
Prepare validation reports, research notes, standards, committee papers and senior-management briefings that clearly distinguish evidence, judgement and residual uncertainty.
Mentor junior validators, improve team methodology and support knowledge transfer across Model Risk
Education and experience:
Master's degree in Theoretical Physics, Applied Physics, Mathematics, Applied Mathematics or a closely related quantitative discipline is required. A PhD or research-intensive master's is strongly preferred.
Typically, one to three years of relevant experience in AI research, machine learning, quantitative modelling, model validation, scientific computing or a closely related field. Exceptional research profiles may be considered based on demonstrated capability.
Deep understanding of probability, statistics, linear algebra, optimisation, numerical computation, experimental design and uncertainty quantification.
Strong understanding of transformers, LLM training and inference, embeddings, RAG, fine-tuning, alignment, evaluation, agentic systems and AI safety failure modes.
Advanced Python proficiency and experience with scientific and ML libraries. Exposure to PyTorch, Hugging Face, evaluation frameworks, experiment tracking, SQL, Git and cloud AI platforms is expected.
Ability to read research papers critically, reproduce methods, design-controlled experiments and convert findings into bank-grade validation evidence.
Experience with red teaming, adversarial testing, interpretability, calibration, robustness, privacy, security or model risk management is strongly advantageous.
Excellent technical writing and communication, including the ability to explain mathematical concepts, assumptions and limitations to non-specialist stakeholders.
Banking experience is advantageous but not mandatory. The role requires willingness to develop knowledge of financial services, Islamic banking, CBUAE expectations and ADIB governance.
Indicative success measures:
Validation conclusions are reproducible, evidence-based and proportionate to use-case risk.
Reusable evaluation assets and automation measurably improve validation coverage, consistency and efficiency.
Material LLM and agentic risks are identified early, clearly communicated and translated into actionable controls or use restrictions.
Research outputs strengthen ADIB validation methodology and remain traceable to tested evidence rather than unsupported claims.
Stakeholders receive constructive, independent challenges while second-line ownership and decision rights remain clear.