Data Engineering Architect, Senior
Bloomberg Industry Group · Arlington, United States · 24d ago
You bring recent, hands-on production experience with Databricks and will play a leading role in evolving our Databricks-based Lakehouse architecture, including the modernization and migration of existing data workloads. You enjoy translating complex product and user behavior into well-structured, reliable datasets that power analytics, experimentation, and decision-making. You can move comfortably between technical implementation and strategic architecture, communicating complex decisions clearly and influencing technical direction across teams.
Primary Responsibilities:
Partner with product analytics stakeholders to translate business-defined KPIs and data requirements into scalable, production-grade datasets.
Own the design, build, and operation of scalable data pipelines end-to-end (ingestion → transformation → serving).
Define and evolve the architecture of the Product Analytics Lakehouse, making technical decisions that improve scalability, performance, reliability, governance, and consistency across datasets and workloads.
Build and maintain production-grade, well-modeled datasets (Gold layer) that power analytics and AI use cases.
Define, implement, and drive adoption of reusable data engineering patterns, frameworks, standards, and guardrails that reduce duplication, improve engineering leverage, and make the right development patterns easier to adopt.
Own data quality and reliability for production datasets, including validation, monitoring, SLAs, and incident resolution.
Productionize and scale prototype datasets and logic developed by analytics partners into reliable, maintainable data pipelines.
Build governed, purpose-built datasets to support AI/ML use cases while enforcing controlled and secure data access patterns.
Lead the technical evolution of workloads into Databricks, evaluating existing architecture and determining appropriate migration, modernization, and coexistence strategies.
Make and communicate architectural tradeoffs across performance, cost, reliability, governance, maintainability, and developer experience.
Provide technical leadership and architectural guidance across Product Analytics, helping engineers and analytics partners make sound data architecture, modeling, and platform decisions.
Mentor and provide technical guidance to engineers and other technical contributors, raising engineering standards through hands-on leadership rather than formal authority.
Job Requirements:
Strong experience building and operating data pipelines using SQL and Python in a modern cloud environment.
Deep expertise in SQL, including complex transformations, data modeling, query optimization, and performance tuning at scale.
2+ years of recent, hands-on production experience with Databricks, including designing, building, optimizing, and operating production data workloads.
Strong hands-on experience with Spark/PySpark and distributed data processing in a production environment.
Strong understanding of modern data architecture patterns, including Lakehouse architecture, ELT, and layered data models (bronze/silver/gold).
Proven experience designing data models for analytics, including dimensional or domain-oriented approaches.
Experience driving database and data engineering best practices, including schema design, migrations, and performance optimization.
Demonstrated ability to own consequential architecture and engineering decisions and drive them from design through production in environments with limited structure or support.
Demonstrated experience establishing reusable data frameworks, standards, and guardrails that have been successfully adopted beyond an individual project or pipeline.
Experience owning production data systems, including monitoring, debugging, and resolving data pipeline failures.
Experience working closely with business stakeholders or analysts to translate ambiguous requirements into scalable data solutions.
Strong technical judgment in evaluating technologies and architecture patterns based on scalability, reliability, cost, maintainability, governance, and developer experience—not technology novelty alone.
Experience evolving or migrating legacy data platforms and workloads while maintaining continuity for downstream consumers.
Strong understanding of data governance, including metadata, lineage, access controls, data quality, and the role of governance in creating trusted, reusable data products.
Strong record of project execution and completion with experience with agile development practices.
Excellent written and verbal communication skills, with demonstrated ability to translate complex technical concepts for both technical and non-technical audiences, influence technical direction across teams, and build alignment around architecture and engineering standards.
Experience with developing source-controlled pipelines in a CI/CD environment.
Experience working with product analytics data (event tracking, user behavior, experimentation) is a plus.
Education and Experience:
Bachelor’s degree in computer science or a related discipline, or equivalent experience. Master’s degree preferred.
10+ years of experience in data engineering, data architecture, software engineering, or related fields, including significant experience designing and owning production data systems and architecture.
Hands-on experience building and maintaining scalable data pipelines and datasets in cloud-based environments with recent production experience in Databricks and cloud infrastructure.
Proven ability to design and evolve data architectures, data models, and production pipelines that support analytics at scale, including performance, reliability, governance, and cost optimization.
Experience taking data solutions from concept to production, including monitoring, debugging, and ongoing maintenance.
Demonstrated ability to operate independently at an architectural level, make high-impact technical decisions, influence engineering practices across teams, and raise data platform standards in environments with limited structure.
Compensation Range:
$180,000.00-$220,000.00Placement in the salary range will be decided upon completion of the interview process. Salary determination will be determined based on factors including but not limited to relevant experience, demonstrated skills related to the requirements of the role, education, certifications, and geographic location.
Equal Opportunity
Bloomberg Industry Group maintains a continuing policy of non-discrimination in employment. It is Bloomberg Industry Group’s policy to provide equal opportunity and access for all persons, and the Company is committed to attracting, retaining, developing, and promoting the most qualified individuals without regard to age, ancestry, color, gender identity or expression, genetic predisposition or carrier status, marital status, national or ethnic origin, race, religion or belief, sex, sexual orientation, sexual and other reproductive health decisions, parental or caring status, physical or mental disability, pregnancy or maternity/parental leave, protected veteran status, status as a victim of domestic violence, or any other classification protected by applicable law (“Protected Characteristic”). Bloomberg prohibits treating applicants or employees less favorably in connection with the terms and conditions of employment, in all phases of the employment process, because of one or more Protected Characteristics (“Discrimination”).