Databricks Platform Engineer - Technology Services Section, Global IT Strategy Office (GITSO)

Job Description:

Business Overview

The Technology Services Section within the Global IT Strategy Office (GITSO) is responsible for driving enterprise technology strategy, governance, and innovation initiatives across Rakuten Group.

The Data Platform Management Group enables enterprise-wide data-driven decision-making by developing and governing the organization's data platform ecosystem. The team is responsible for establishing scalable and secure data platform capabilities, data governance frameworks, security and compliance processes, and operational standards that support business growth and technology transformation.

The group works closely with Data Engineers, Data Platform Architects, Security teams, Enterprise Architects, Infrastructure teams, and business stakeholders to maximize the value of enterprise data while ensuring compliance, governance, and operational excellence.

Department Overview

The Data Platform Management Group serves as the strategic governance and management function for Rakuten's enterprise data platform initiatives.

The team is responsible for defining platform strategy, establishing governance frameworks, driving security and compliance activities, managing cross-functional programs, and enabling effective use of enterprise data across the organization.

Working across multiple organizations, the group facilitates alignment between business objectives, technology investments, architecture standards, and governance requirements while supporting the continuous evolution of the enterprise data platform.

Position:

Why We Hire

As Rakuten continues to expand its data-driven capabilities with a new Databricks-based data platform, we need a hands-on engineer who can contribute to the design, implementation, and maintenance of the platform's core automation systems and observability layer.

This role is critical for the success of the project: the data contract-driven deployment pipeline, the Declarative Automation Bundle generation system, and the monitoring infrastructure that ensures platform health and SLA compliance. It requires a Databricks Platform Engineer who combines deep Databricks and Python expertise with strong software engineering capabilities.

The successful candidate will proactively identify design challenges across systems with no internal precedent and propose well-reasoned solutions for team review. They will bring a methodical engineering approach to the full build-and-operate cycle, from contract processing to production monitoring, ensuring the sustainability and scalability of the platform's systems and processes.

Position Details

<Contract System & Deployment Pipeline>

・Design and implement the contract-driven Declarative Automation Bundle generator that translates Open Data Contract Standard YAML files into deployable Databricks assets.

・Build and maintain the Declarative Automation Bundle deployment pipeline, including environment parameterization, deployment atomicity, and rollback mechanisms.

・Develop and operate Azure Pipelines CI/CD workflows: multi-stage pipelines, environment approvals, variable groups, and service connections.

・Implement Unity Catalog governance automation: schema, table, tag, and grant management via Databricks SDK and REST APIs.

・Generate and execute SQL DDL statements for column masks and row filters as defined by data contracts.

・Design and maintain the contract index, including atomic maintenance and recovery mechanisms.

・Implement the dead-letter asset pattern: co-deployment generation and deployment atomicity.

・Design and enforce Git workflows: branching strategies (develop/staging/main), merge policies, and environment promotion gates.

・Develop schema validation, smoke tests, and idempotent deployment verification.

<Platform Observability & Monitoring>

・Build platform health dashboards and alerting from Databricks system tables.

・Design and implement freshness signals via Delta transaction log analysis.

・Implement SLA/SLO comparison logic against contract-declared commitments using the contract index.

・Build dead-letter accumulation SLI monitoring for mandated platform signals.

・Design platform health signal monitoring: SCIM sync latency, deployment ordering failure detection, break-glass activation alerting.

・Develop cost attribution reporting.

・Implement data quality monitoring using Unity Catalog quality constraints and expectation-based frameworks.

<Infrastructure Contribution>

・Contribute to Databricks workspace deployment and administration alongside the Cloud Platform Engineers.

Mandatory Qualifications

・5+ years of experience in data engineering, data platform engineering, or DataOps.

・3+ years of experience with the Databricks platform: Unity Catalog, workspace administration, Databricks SDK, and REST APIs.

・Strong proficiency in Python: automation scripting, YAML parsing, code generation, and API integration.

・Solid software engineering practices: modular design, code maintainability, error handling, testing, and documentation.

・Experience with Declarative Automation Bundles or equivalent infrastructure-as-code deployment frameworks for Databricks.

・Understanding of RBAC and governance models: schemas, grants, column masks, row filters.

・Experience with CI/CD pipelines (Azure DevOps and Azure Pipelines favored): multi-stage YAML pipelines, environment approvals, service connections.

・Experience building monitoring dashboards, alerting systems, and operational reporting.
 

Desired Qualifications

・Proficiency in SQL.

・Familiarity with the Open Data Contract Standard (ODCS) or similar data contract frameworks.

・Experience with Delta Lake internals: transaction log analysis, write audit trails, schema evolution.

・Experience with data quality frameworks and SLA/SLO monitoring.

・Knowledge of Azure Entra ID, SCIM provisioning, or workload identity federation.

・Experience with Git workflow design: branching strategies, merge policies, environment promotion.

・Experience in IT operations domains such as license management or asset management.

<Language Requirements>

・English: Business Level

・Japanese: Preferred

<Ideal Candidate>

・A hands-on engineer with a get-things-done attitude.

・Passionate about data-driven decision making, process improvement, and operational excellence.

・A collaborative problem-solver who builds shared understanding before committing to a technical direction.

・Understands how platform design decisions impact the domains and teams that depend on the platform.

・Rigorous about deployment safety: atomicity, rollback, idempotency, and testing.

・Comfortable working with high standards of governance.

・Understands the importance of measuring platform health signals and operational visibility from the start, not as an afterthought.

・Effective at balancing long-running platform build work with responsive operational monitoring duties.

#engineer

#infrastructureengineer

#technologymanagementdiv