Hi all when I add a rule file in prometheus yml file the promethus service will be stopped I checked the log but couldnt figure it out

0 votes

This is the prometheus.yml file. Before I configure any rules, It was running perfectly


# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
#  query_log_file: /var/log/prometheus/query.log

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 'localhost:9093'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - prometheus_rules.yml

This is the log

Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.562Z caller=main.go:842 msg="TSDB started"
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.562Z caller=main.go:969 msg="Loading configuration file" filename=/usr/local/bin/prometheus/prometheus.yml
Aug  5 05:38:03 localhost prometheus: level=error ts=2021-08-05T09:38:03.571Z caller=manager.go:956 component="rule manager" msg="loading groups failed" err="/usr/local/bin/prometheus/prometheus_rules.yml: group \"instance_down_alert_rule\", rule 1, \"InstanceDown\": invalid annotation name: d:wqescription"
Aug  5 05:38:03 localhost prometheus: level=error ts=2021-08-05T09:38:03.571Z caller=main.go:995 msg="Failed to apply configuration" err="error loading rules, previous rule set restored"
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:685 msg="Stopping scrape discovery manager..."
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:699 msg="Stopping notify discovery manager..."
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:721 msg="Stopping scrape manager..."
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:695 msg="Notify discovery manager stopped"
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=main.go:681 msg="Scrape discovery manager stopped"
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=manager.go:934 component="rule manager" msg="Stopping rule manager..."
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.571Z caller=manager.go:944 component="rule manager" msg="Rule manager stopped"
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.572Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.573Z caller=main.go:896 msg="Notifier manager stopped"
Aug  5 05:38:03 localhost prometheus: level=info ts=2021-08-05T09:38:03.573Z caller=main.go:715 msg="Scrape manager stopped"
Aug  5 05:38:03 localhost prometheus: level=error ts=2021-08-05T09:38:03.573Z caller=main.go:905 err="error loading config from \"/usr/local/bin/prometheus/prometheus.yml\": one or more errors occurred while applying the new configuration (--config.file=\"/usr/local/bin/prometheus/prometheus.yml\")"
Aug  5 05:38:03 localhost systemd: prometheus.service: main process exited, code=exited, status=1/FAILURE
Aug  5 05:38:03 localhost systemd: Unit prometheus.service entered failed state.
Aug  5 05:38:03 localhost systemd: prometheus.service failed.

"I am getting this same error in Docker prometheus container too"

Aug 5, 2021 in DevOps Tools by Khopithan

edited Mar 4, 2025 356 views

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP