Updated: 2021-10-24 02:12:19 +0800 +08

Prometheus Performance Tuning

Profiling

Overview

http://your.prometheus.host:9090/debug/pprof

Go Profiling

  1. go pprof http://your.prometheus.host:9090/debug/pprof/heap
  2. go pprof web

System

Adjust nofile

  1. find the current ulimit number of open file

    ulimit -a
    
  2. increase open files (file descriptor) limit

    ulimit -n 102400
    

Startup Flag

adjust query flag

  • query.timeout
  • query.max-samples
  • query.max-concurrency

Federation

Thanos

Reference

Why does Prometheus consume so much memory?@stackoverflow