Promethues 監控Tomcat 主要用的模塊

Promethus 負載抓取/存儲指標信息、并提供查詢功能
grafana 數據可視化
JMX exporter 提供JMX中JVM相關的metrics

1、利用JMX exporter，在JAVA進程內啟動一個小型的Http server

2、配置Prometheus抓取那個Http server提供的metrics。

3、配置Grafana連接Prometheus，配置Dashboard。

一、下載一個tomcat,下載jmx_exporter

1、獲取jmx_exporter有2種方法，自己編譯獲取jar包，獲取現成的jar包

wget  https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar

或者編譯
git clone https://github.com/prometheus/jmx_exporter
cd jmx_exporter
mvn package

2、安裝tomcat

wget http://mirrors.tuna.tsinghua.edu.cn/Apache/tomcat/tomcat-8/v8.5.51/bin/apache-tomcat-8.5.51.tar.gz
tar zxvf apache-tomcat-8.5.51.tar.gz
mv apache-tomcat-8.5.51 /usr/localt/tomcat_test

3、配置文件下載

wget https://github.com/prometheus/jmxexporter/blob/master/exampleconfigs/tomcat.yml

tomcat.yml的內容如下

---   
lowercaseoutputLabelNames: true
lowercaseOutputName: true
rules:
- pattern: 'Catalina<type=GlobalRequestProcessor, name="(w+-w+)-(d+)"><>(w+):'
  name: tomcat_$3_total
  labels:
    port: "$2"
    protocol: "$1"
  help: Tomcat global $3
  type: COUNTER
- pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
  name: tomcat_servlet_$3_total
  labels:
    module: "$1"
    servlet: "$2"
  help: Tomcat servlet $3 total
  type: COUNTER
- pattern: 'Catalina<type=ThreadPool, name="(w+-w+)-(d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
  name: tomcat_threadpool_$3
  labels:
    port: "$2"
    protocol: "$1"
  help: Tomcat threadpool $3
  type: GAUGE
- pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
  name: tomcat_session_$3_total
  labels:
    context: "$2"
    host: "$1"
  help: Tomcat session $3 total
  type: COUNTER

4、收集數據

4-1、Tomcat收集數據

mkdir /usr/local/tomcat_test/jmx/
cp jmx_prometheus_javaagent-0.12.0.jar  /usr/local/tomcat_test/jmx/
cp tomcat.yml  /usr/local/tomcat_test/jmx/

修改配置文件vim /usr/local/tomcat_test/bin/catalina.sh 
JAVA_OPTS="-javaagent:/usr/local/tomcat_test/jmx/jmx_prometheus_javaagent-0.12.0.jar=39081:/usr/local/tomcat_test/jmx/tomcat.yml"

4-2、JAR包運行（案例，關鍵文件jmxprometheusjavaagent-0.3.0.jar和tomcat.yml）

java -javaagent:./jmx_prometheus_javaagent-0.3.0.jar=9151:tomcat.yaml -jar yourJar.jar

5、測試是否收集到數據 curl -s http://localhost:39081/ | more

[root@Prometheus promethus]# curl -s http://localhost:39081/ | more 
# HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.12.0",name="jmx_prometheus_javaagent",} 1.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 3.99
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.582290765496E9
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 74.0

二、配置promethus 采集數據

1、文件引用 (本文采用) /usr/local/prometheus/prometheus.yml

- job_name: 'tomcat'
    file_sd_configs:
    - files: ['/usr/local/prometheus/conf/tomcat.yml']
      refresh_interval: 180s

/usr/local/prometheus/conf/tomcat.yml/tomcat.yml

- targets:
  - 114.67.116.119:39081
  labels:
    idc: test_idc
    service: tomcat_test

2、直接配置

scrape_configs:
  - job_name: 'java'
    static_configs:
    - targets: ['114.67.116.119:39081']

3、重載配置文件

kill -hup `ps -ef |grep prometheus|grep -v grep|awk '{print $2}'`
或者
kill -9 PID
/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml