PMM出问题排查

2017-09-29 阅读量259

看各种日志
monitoring service

1

2

3

4

[root@node4 ~]# ll /var/log/pmm-*

-rw-r--r--. 1 root root 1880 Sep 27 18:02 /var/log/pmm-linux-metrics-42000.log

-rw-r--r--. 1 root root 783 Sep 27 18:02 /var/log/pmm-mysql-metrics-42002.log

-rw-r--r--. 1 root root 7143 Sep 27 18:08 /var/log/pmm-mysql-queries-0.log

docker里的日志

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

[root@node4 log]# docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

e4916410b314 percona/pmm-server:latest "/opt/entrypoint.sh" 2 hours ago Up 31 minutes 0.0.0.0:80->80/tcp, 443/tcp pmm-server

948a9aeb047e percona/pmm-server:latest "/bin/true" 2 hours ago Created pmm-data

[root@node4 log]# docker logs e4916410b314

2017-09-27 08:39:47,175 CRIT Supervisor running as root (no user in config file)

2017-09-27 08:39:47,175 WARN Included extra file "/etc/supervisord.d/pmm.ini" during parsing

Unlinking stale socket /var/run/supervisor/supervisor.sock

2017-09-27 08:39:47,527 INFO RPC interface 'supervisor' initialized

2017-09-27 08:39:47,528 INFO supervisord started with pid 1

2017-09-27 08:39:48,536 INFO spawned: 'mysql' with pid 15

2017-09-27 08:39:48,543 INFO spawned: 'consul' with pid 16

2017-09-27 08:39:48,552 INFO spawned: 'grafana' with pid 17

2017-09-27 08:39:48,563 INFO spawned: 'nginx' with pid 18

2017-09-27 08:39:48,610 INFO spawned: 'cron' with pid 19

2017-09-27 08:39:48,612 INFO spawned: 'qan-api' with pid 20

进去容器看

1

2

3

4

docker exec -it e4916410b314 /bin/bash

/var/log 下面各种日志

/var/log/grafana/grafana.log

/var/log/prometheus.log

pmm-admin list是yes不代表没问题,check-network看看

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

[root@node4 ~]# pmm-admin check-network

PMM Network Status

Server Address | 172.16.83.103

Client Address | 172.16.83.103

* System Time

NTP Server (0.pool.ntp.org) | 2017-09-27 17:13:58 +0800 CST

PMM Server | 2017-09-27 09:13:58 +0000 GMT

PMM Client | 2017-09-27 17:13:58 +0800 CST

PMM Server Time Drift | OK

PMM Client Time Drift | OK

PMM Client to PMM Server Time Drift | OK

* Connection: Client --> Server

-------------------- -------

SERVER SERVICE STATUS

-------------------- -------

Consul API OK

Prometheus API OK

Query Analytics API OK

Connection duration | 551.133µs

Request duration | 2.467879ms

Full round trip | 3.019012ms

* Connection: Client <-- Server

-------------- ------ -------------------- ------- ---------- ---------

SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD

-------------- ------ -------------------- ------- ---------- ---------

linux:metrics node4 172.16.83.103:42000 OK YES -

mysql:metrics node4 172.16.83.103:42002 OK YES -