Product Manager Evaluation & Data (EC AI)
Rakuten · Singapore · 1mo ago
Job Description:
Rakuten Group, Inc. is a global leader in internet services and has a diverse ecosystem spanning across e-commerce, fintech, communications and more serving approximately 1.8 billion members worldwide. Founded in Tokyo in 1997, the Group operates in over 30 countries and regions with more than 30,000 employees.
Based in Singapore's Central Business District, Rakuten Asia Pte. Ltd. serves as the regional headquarters for Asia, driving value through areas such as advertising product development, product strategy, and data management to support Rakuten Group's global ecosystem.
Learn more at: https://global.rakuten.com/corp/
As the Product Manager – Evaluation and Data, your mission is to make the quality of Rakuten's AI products measurable, repeatable, and actionable. You will own the evaluation strategy, process, and platform that enable teams to define success, prevent regressions, make confident release decisions, and continuously improve AI experiences through reliable data and insights.
High Impact Role: You'll establish how AI product quality is measured, directly influencing release decisions, user experience, and business growth.
Evaluation-Driven Culture: You'll turn product requirements and real-world failures into reliable evaluations with talented engineers, data scientists, and domain experts.
Ownership & Autonomy: You'll own the evaluation strategy and platform roadmap, making critical decisions on processes, capabilities, quality standards, and priorities across Rakuten's AI products.
Collaborative Environment: You'll work closely with product, engineering, data science, analytics, and business teams to define success and achieve shared goals.
Position Details
Evaluation Strategy & Criteria: Define success for Rakuten's AI products and translate requirements, user needs, and risks into measurable criteria.
Evaluation Quality: Select grading methods, calibrate automated graders with human judgment, and review transcripts and failures.
Platform Strategy: Define the platform vision, roadmap, and requirements with engineering and data teams.
Platform Capabilities: Prioritize dataset management, model/prompt comparison, automated and human grading, multi-turn evaluation, trace review, dashboards, and release integration.
Continuous Improvement: Maintain balanced suites, convert production issues into regression tests, and improve evaluation coverage and accuracy.
Adoption & Decisions: Establish governance, enable team and expert contributions, and connect evaluations with production monitoring, A/B testing, feedback, and business KPIs to guide priorities and releases.
Mandatory Qualifications:
Experience: 5+ years of product management experience in AI/ML, evaluation, data products, or platforms.Experience with consumer AI products is preferred.
Data Proficiency: Excellent analytical skills and experience working with large datasets.
Platform & Technical Acumen: Ability to define platform requirements and communicate model/prompt versioning, tracing, data pipelines, and release integration.
Experimentation & Quality: Experience with A/B testing, production monitoring, user feedback, and human evaluation.
Communication Skills: Excellent written and verbal skills, with the ability to present complex evaluation results clearly and concisely.
Collaboration Skills: Proven ability to lead cross-functional product, engineering, data science, and domain teams.
Rakuten is an equal opportunities employer and welcomes applications regardless of sex, marital status, ethnic origin, sexual orientation, religious belief, or age.