Skip to content

Prometheus Alertmanager Fixes#218

Merged
Dmitry-Popovichev merged 8 commits into
masterfrom
prom_alert_fixes
Jun 3, 2025
Merged

Prometheus Alertmanager Fixes#218
Dmitry-Popovichev merged 8 commits into
masterfrom
prom_alert_fixes

Conversation

@khalford

@khalford khalford commented May 22, 2025

Copy link
Copy Markdown
Member

Includes various fixes and improvements for Prometheus and Alertmanager

  • Move Alertmanager and Prometheus logging from /opt/<service> into /var/log/<service>
  • Simplify Alertmanager and Prometheus install by merging most tasks into unarchive and copy modules
  • Fixes a doc typo

khalford added 2 commits May 22, 2025 14:35
Moving alertmanager logs to /var/log as this is more conventional
Moving prometheus logs to /var/log as this is more conventional
khalford added 3 commits May 22, 2025 15:28
We can remove a lot of tasks as they can be achieved by already used tasks.

In the old changes the following tasks run:
- Check if the binaries exist (stat)
- Download the binaries (get_url)
- Extract the binaries (unarchive)
- Create the binary directory (file)
- Move the extracted binaries into the folder (shell)

In the new changes the following tasks run:
- Unarchive
  - Check the downloaded files exists with "creates"
  - Download the files with "src" as a URL
  - Extract the files to destination "/tmp"
- Copy
  - Copy the extracted files into the binaries directory
  - Create that directory if it doesn't exist
The alert will trigger after 30 seconds not 10. Updated the description
to reflect that
@khalford khalford force-pushed the prom_alert_fixes branch 2 times, most recently from 26e8b58 to 4735e4f Compare May 23, 2025 06:14
jose-caballero
jose-caballero previously approved these changes May 23, 2025

@AlexCK-STFC AlexCK-STFC left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM, couple of questions ^

Comment thread chatops_deployment/ansible/roles/alertmanager/tasks/main.yml
Comment thread chatops_deployment/ansible/roles/prometheus/tasks/main.yml
Comment thread chatops_deployment/ansible/roles/alertmanager/tasks/main.yml Outdated
Comment thread chatops_deployment/ansible/roles/prometheus/tasks/main.yml Outdated
khalford added 2 commits June 2, 2025 10:58
Reducing the permission scope of the alertmanager and prometheus logs. The public do not need read access to the logs. Any user who would need to read the logs should already be in the prometheus/alertmanager group.
When running the playbook for a second time, if the binaries were already removed from `/tmp` (e.g. system reboot) then the binaries will be re-downloaded and extracted even if `/opt/<application>` already exists.  We can fix this by checking if `/opt/<application>` exists instead and only run the copy module if the unarchive module made changes.
AlexCK-STFC
AlexCK-STFC previously approved these changes Jun 2, 2025
When a task is run based on a previous task causing a change on the target host it should be run as a handler. Moving the handler tasks into handlers to fix the lint warning.

https://ansible.readthedocs.io/projects/lint/rules/no-handler/#problematic-code

@Dmitry-Popovichev Dmitry-Popovichev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, btw with "mode" you can type it as an octal number, i.e mode: 0774 or as a string mode: "774" and ansible will do the conversion. Doesn't really matter either way but just something I found today while reviewing

@Dmitry-Popovichev Dmitry-Popovichev merged commit 60cc6ed into master Jun 3, 2025
5 checks passed
@Dmitry-Popovichev Dmitry-Popovichev deleted the prom_alert_fixes branch June 3, 2025 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants