Prometheus does not scrape all metrics from PCP pmproxy

Question

On my laptop with Fedora 30 I have Performance Co-Pilot (PCP) daemons installed and running, and Prometheus installed from the package golang-github-prometheus-prometheus-1.8.0-4.fc30.x86_64. In PCP collector's config I specified the following metric namespaces:

# Performance Metrics Domain Specifications
# 
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
#root   1   pipe    binary      /var/lib/pcp/pmdas/root/pmdaroot
#pmcd   2   dso pmcd_init   /var/lib/pcp/pmdas/pmcd/pmda_pmcd.so
proc    3   pipe    binary      /var/lib/pcp/pmdas/proc/pmdaproc -d 3
#xfs    11  pipe    binary      /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
linux   60  pipe    binary      /var/lib/pcp/pmdas/linux/pmdalinux
#pmproxy    4   dso pmproxy_init    /var/lib/pcp/pmdas/mmv/pmda_mmv.so
mmv 70  dso mmv_init    /var/lib/pcp/pmdas/mmv/pmda_mmv.so
#jbd2   122 dso jbd2_init   /var/lib/pcp/pmdas/jbd2/pmda_jbd2.so
#kvm    95  pipe    binary      /var/lib/pcp/pmdas/kvm/pmdakvm -d 95

[access]
disallow ".*" : store;
disallow ":*" : store;
allow "local:*" : all;

When I visit the URL localhost:44323/metrics the output is very rich and covers many namespaces, ie. mem, network, kernel, filesys, hotproc etc., but when I scrape it with Prometheus where the job is defined as:

scrape_configs:
  - job_name: 'pcp'
    scrape_interval: 10s
    sample_limit: 0
    static_configs:
      - targets: ['127.0.0.1:44323']

I see the target status UP, but in the console only these two metric namespaces are available for querying: hinv and mem. I tried to copy other metric names from /metrics page, but queries result in the error: 'No datapoints found.' Initially I thought the problem could have been due to a limit on the number of samples per target or the sampling interval too small (I originally set it to 1s), but hinv and mem are not next to each other and there are other metrics (ie. filesys, kernel) in between them that are omitted. What could be the reason for that?

To know more about ITIL, join our ITIL course online. Also, If you wish to master the principles of ITIL Training to work on step-by-step tasks that have significant relevance for the world of business, industry professionals developed the ITIL syllabus.

Answer 1

I'm adding another answer because I just saw this problem in another situation where Prometheus v.2.19 was taking metrics from PCP v.5 on CentOS 7 servers using PMAPI. The scrape was defined as a single task with several metric domains in the Prometheus config file, ie:

The problem was solved by dividing the scrape operation into many jobs, each with only one target, for example:

- job_name: 'pcp'
  file_sd_configs:
  - files: [...]
  metric_path: '/metrics'
  params:
    target: ['kernel', 'mem', 'disk', 'network', 'mounts', 'lustre', 'infiniband']

Only kernel metrics were gathered when there was a difficulty with one metric domain, usually lustre or infiniband, due to a lack of appropriate hardware on the host.

The problem was solved by dividing the scrape operation into many jobs, each with only one target, for example:

- job_name: 'pcp-kernel'
  file_sd_configs:
  - files: [...]
  metric_path: '/metrics'
  params:
    target: ['kernel']

- job_name: 'pcp-mem'
  file_sd_configs:
  - files: [...]
  metric_path: '/metrics'
  params:
    target: ['mem']

[...]

This manner, even if one or all of the extra domains failed, metrics from the core domains were always scraped successfully. This design appears to be more resilient, but it makes the target status view busier due to the increased number of scrape processes.