Job Url: https://www.remoterocketship.com/company/grafana-labs/jobs/staff-software-engineer-agentic-applications-grafana-ops-ai-ml-united-states-remote

Job Description: 
Grafana Labs
Website
LinkedIn
All Job Openings
Grafana Labs is a company that specializes in open-source observability technologies and solutions. It offers a comprehensive suite of tools for logging, metrics, tracing, and profile management with products like Grafana, Loki, Tempo, and Mimir. 

Their offerings are designed to help businesses visualize, monitor, and alert on data from various sources, providing capabilities such as anomaly detection, root cause analysis, and service level objective management using AI/ML insights. Grafana Labs provides both cloud-based and self-managed solutions, ideal for infrastructure, application, and frontend observability. 

Additionally, their platform supports integration with various data sources like Prometheus and OpenTelemetry, making them a key player in the observability and infrastructure monitoring space. 

Monitoring • Observability • Dashboards


501 - 1000 employees


Founded 2014

🏢 Enterprise

☁️ SaaS

🤖 Artificial Intelligence

Staff Software Engineer, Agentic Applications - Grafana Ops, AI/ML
Yesterday

🇺🇸 United States – Remote

💵 $148.5k - $178.2k / year

⏰ Full Time

🔴 Lead

🧑‍💻 Full-stack Engineer

🦅 H1B Visa Sponsor

Cloud

Grafana

Open Source

Apply Now


Receive Emails with Similar Jobs
Report problem
📋 Description
• Grafana Labs is a remote-first, open-source powerhouse.
• The Grafana AI teams play a key role in this mission by helping users make sense of complex observability data through AI-driven features.
• We are looking for an experienced engineer with expertise in evaluating Generative AI systems, particularly Large Language Models (LLMs).
• This role involves designing and scaling automated evaluation pipelines, integrating them into CI/CD workflows, and defining metrics that reflect both product goals and model behavior.
• As the team matures, there’s a broad opportunity to expand or redefine this role based on impact and initiative.

🎯 Requirements
• Experience designing and implementing evaluation frameworks for AI/ML systems.
• Familiarity with prompt engineering, structured output evaluation, and context-window management in LLM systems.
• High autonomy to collaborate and translate team goals into clear, testable criteria supported by effective tooling.
• Bonus Points For:
• Experience working in environments with rapid iteration and experimental development.
• A pragmatic mindset that values reproducibility, developer experience, and thoughtful trade-offs when scaling GenAI systems.
• A passion for minimizing human toil and building AI systems that actively support engineers.

🏖️ Benefits
• All of our roles include Restricted Stock Units (RSUs), giving every team member ownership in Grafana Labs' success.
• We believe in shared outcomes—RSUs help us stay aligned and invested as we scale globally.
• We operate a global annual leave policy of 30 days per annum.
• 3 days of your annual leave entitlement are reserved for Grafana Shutdown Days to allow the team to really disconnect.
• *We will comply with local legislation where applicable.