Principal Product Manager

This role is based in Redmond, Washington.

Compensation:

USD $139,900 - $274,800 per year

Overview:

We are looking for a deeply technical and forward-thinking Principal Product Manager to lead the Evaluation Strategy for Microsoft Agent 365 Tools, which includes MCP, Skills, etc. You will define how we measure, validate, and continuously improve the performance of 1P and 3P agentic tools.

Responsibilities:

• Define and own the evaluation strategy for all 1P and 3P Agentic tools.
• Develop a unified framework covering offline evals, online evals, and assertion-based rubric design.
• Partner with engineering to evolve internal platforms.
• Create grading frameworks and ground truth generation mechanisms.
• Design and maintain evaluation suites that capture model regressions.

Qualifications:

Required: Bachelor's Degree AND 8+ years experience in product/service/program management or software development.

Preferred: 12+ years experience in product management, with demonstrated technical depth across LLMs and proven experience leading AI/LLM evaluation strategy.