Prometheus
Table of Contents
1. Abstract
- open-source monitoring and alerting toolkit built around a time-series database.
- excels at collecting and analyzing metrics from dynamic, cloud-native environments.
- see Monitoring
2. Characteristics
- Pull-based model
- metrics identified by key-value pairs (labels)
- PromQL: designed for manipulating and analyzing time-series data.
- Alertmanager integration : notification setup via multiple channels
- Exporters and integrations: ecosystem of exporters exists for collecting metrics from multiple sources (databases, message queues, etc.).
3. Use Cases
3.1. Infrastructure monitoring
Track CPU usage, memory consumption, disk space, and network traffic across servers and containers.
3.2. Application performance monitoring (APM)
Monitor request rates, response times, error rates, and other application-specific metrics.
3.3. Business metrics monitoring:
Track key performance indicators (KPIs) like orders per minute or user sign-up rate.
4. Sample Prod
4.1. Components:
4.1.1. Prometheus Servers:
- Role: Scrape metrics from targets, store them locally, and handle alerting.
- Scalability: Use a two-tier approach:
- Tier 1: Multiple Prometheus instances for scraping, each dedicated to specific services or teams. This ensures fault tolerance.
- Tier 2: Optional - A global Prometheus instance for long-term storage and querying across all data. Tools like Thanos or Cortex can back this tier.
4.1.2. Target Services:
- Role: Your applications, infrastructure components (databases, load balancers, etc.) exposing metrics in Prometheus format.
- Instrumentation: Use client libraries (like the official Prometheus client libraries) to instrument your applications and expose relevant metrics.
4.1.3. Service Discovery:
- Role: Dynamically discover and register new targets with Prometheus servers.
- Tools: Consul, Kubernetes Service Discovery, AWS Service Discovery, etc.
4.1.4. Alertmanager:
- Role: Receives alerts from Prometheus, handles deduplication, grouping, silencing, and routing to notification channels (email, PagerDuty, Slack, etc.).
- High Availability: Deploy in a cluster (minimum 3 nodes) for redundancy.
4.1.5. Grafana:
- Role: Visualization and dashboarding tool, querying data from Prometheus (and potentially other data sources).
- High Availability: Run multiple Grafana instances behind a load balancer for redundancy.
4.2. Data Flow:
- Target Expose Metrics: Applications and infrastructure expose metrics.
- Prometheus Scrapes: Prometheus servers scrape metrics from targets.
- Prometheus Stores: Data is stored in Prometheus's local time-series database.
- Long-Term Storage: Optionally, data is sent to a long-term storage solution (Thanos, Cortex) from Tier 1 Prometheus servers.
- Alertmanager Processing: Prometheus sends alerts to Alertmanager based on configured rules.
- Alerting: Alertmanager routes alerts to the appropriate channels.
- Grafana Querying: Users query and visualize data in Grafana dashboards, drawing data from Prometheus.
4.3. Additional Considerations:
- Monitoring Your Monitoring: Monitor the health and performance of your Prometheus and Grafana infrastructure itself!