What s the procedure for the GPU metrics reporting script setup

Question

Can somebody help me with the set up of GPU metrics reporting script?

Answer 1

To Set up the GPU metrics reporting script:

On each of your VM instances, check that you meet the following requirements:

1. Each VM instance must have GPUs attached.

2. Each VM instance must have a GPU driver installed.

3. Each VM instance must have the pip utility installed. On each of your VM instances, install the GPU metrics agent. 

4. To install the metrics agent, complete the following steps: 

• Download the GPU metrics reporting scripts.

git clone https://github.com/GoogleCloudPlatform/tensorflow-inference-tensorrt5-t4-gpu.git 

• Switch to the metrics_reporting folder.

cd tensorflow-inference-tensorrt5-t4-gpu/metrics_reporting

• Set up the installation environment for the metrics agent.

pip install -r ./requirements.txt

• Move the metric reporting script to your root directory.

sudo cp report_gpu_metrics.py /root/

• Enable the GPU metrics agent.

cat <<-EOH > /lib/systemd/system/gpu_utilization_agent.service
[Unit]
Description=GPU Utilization Metric Agent
[Service]
Type=simple
PIDFile=/run/gpu_agent.pid
ExecStart=/bin/bash --login -c '/usr/bin/python /root/report_gpu_metrics.py'
User=root
Group=root
WorkingDirectory=/
Restart=always
[Install]
WantedBy=multi-user.target
EOH

• Reload the system daemon.

systemctl daemon-reload

• Enable the gpu monitoring service.

systemctl --no-reload --now enable /lib/systemd/system/gpu_utilization_agent.service

git clone https://github.com/GoogleCloudPlatform/tensorflow-inference-tensorrt5-t4-gpu.git

systemctl --no-reload --now enable /lib/systemd/system/gpu_utilization_agent.service