Observability Engineer

About the role

OmniOps is looking for an experienced Observability Engineer to design, deploy, and operate modern observability platforms for our clients. You will work directly with client teams to ensure their cloud‑native, on‑premises, and hybrid environments are fully observable and reliable.

Key responsibilities

• Deploy, configure, upgrade, and manage Grafana Alloy agents across Kubernetes, Linux, and Windows.
• Design and maintain observability back‑ends using Grafana, Loki, Mimir, Tempo, and Alertmanager, on‑premises or via Grafana Cloud.
• Build telemetry pipelines for logs, metrics, and traces, ensuring reliable collection and delivery.
• Manage retention policies, storage capacity, and platform health across client environments.
• Create dashboards, alerting rules, notification channels, and monitoring standards.
• Provide self‑monitoring capabilities to detect ingestion, agent, and backend health issues.
• Support client teams with application instrumentation and observability best practices.
• Define telemetry collection standards, SDK recommendations, and integration approaches.
• Validate data ingestion and assist clients in implementing alerting strategies aligned with SLOs.

Required profile

• Proven experience operating Grafana‑based observability platforms in production.
• Hands‑on experience with Grafana Alloy, OpenTelemetry Collector, Promtail, or similar tools.
• Strong background deploying monitoring agents on Kubernetes, Linux, and Windows.
• Solid understanding of Kubernetes concepts (DaemonSets, RBAC, manifests, Helm).
• Experience with Grafana Cloud tenant management and stack configuration.
• Deep knowledge of observability concepts: metrics, logs, traces, alerting, retention, capacity planning.
• Familiarity with PromQL, LogQL, and telemetry data modeling.
• Excellent communication and stakeholder management skills.

Required skills

• Grafana Alloy
• OpenTelemetry Collector
• Promtail / Grafana Agent
• Kubernetes (DaemonSets, RBAC, Helm)
• Linux and Windows environments
• Grafana Cloud
• Loki, Mimir, Tempo, Alertmanager
• PromQL
• LogQL