Lead Observability Engineer

About the role

OmniOps, a Riyadh-based technology solutions provider, is hiring a Lead Observability Engineer. You set the technical direction for how observability is delivered to clients across Grafana Alloy, the LGTM stack (Loki, Grafana, Tempo, Mimir) and OpenTelemetry. At intermediate level the job is to deploy and run the stack well; at senior/lead level you design the reference architecture, define the standards others follow, mentor engineers and act as the senior technical voice to clients, while staying hands-on with Alloy pipelines, cardinality issues and incident response.

Key responsibilities

• Own the reference architecture for client observability across on-prem LGTM, Grafana Cloud and hybrid setups, choosing the right model for each client and designing for scale, multi-tenancy and high availability.
• Set the OpenTelemetry strategy and track Grafana and OpenTelemetry roadmaps, running and deciding on PoCs such as Grafana Beyla/eBPF, Pyroscope and Faro.
• Define and maintain standard Alloy configurations for Kubernetes, Linux and Windows, and own the label and attribute taxonomy and cardinality budget.
• Drive infrastructure-as-code and GitOps, build the release and upgrade process for agents and backends, and set SLOs for the platform.
• Plan capacity and cost, run incident escalations and blameless postmortems, and embed security (SSO, RBAC, secrets, encryption, PII redaction).
• Act as senior technical lead on client engagements, run maturity assessments, lead migrations from tools such as Prometheus/Thanos, ELK, Datadog, New Relic, Splunk or legacy Grafana Agent/Promtail, and coach client app teams on instrumentation and SLOs.
• Mentor engineers, write runbooks and architecture decision records, and work with delivery, sales and account teams.

Required profile

• 6+ years in DevOps, SRE or platform engineering, with at least 3 years focused on observability in production.
• Deep production-scale experience with the Grafana stack (Grafana, Loki, Mimir, Tempo, Alertmanager) and tuning, scaling and troubleshooting.
• Proven technical leadership: architecture decisions, mentoring, incident leadership and client-facing communication.

Required skills

• Expertise with Grafana Alloy and/or the OpenTelemetry Collector, and strong OpenTelemetry knowledge (OTLP, semantic conventions, logs/metrics/traces correlation).
• Advanced Kubernetes, PromQL, LogQL, TraceQL and cardinality control.
• Infrastructure-as-code and GitOps (Terraform, Helm, Argo CD or Flux, Ansible) and CI/CD.
• Grafana Cloud administration, SLOs, error budgets and alerting.
• Nice to have: MSP experience, AWS/Azure/GCP telemetry, eBPF observability, Grafana k6, Grafana OnCall/PagerDuty, and certifications such as Grafana Certified Associate or CKA/CKAD.