Red Hat Advanced Cluster Management for Kubernetes Alerts: How to Use It
With the observability provider enabled, you may use Red Hat Advanced Cluster Management for Kubernetes (RHACM) to advantage perception approximately and optimize your controlled clusters. If the controlled cluster is Red Hat OpenShift Container Platform (RHOCP) 4.8+ or KS cluster, you may see indicators from all of the controlled clusters withinside the hub cluster.
You can also configure ahead indicators with an outside notification system.
In this weblog post, I introduce the way to use amtool to manipulate RHACM indicators.
Amtool
The amtool is a CLI device for interacting with the alertmanager API. It’s bundled with all releases of Alertmanager. You can set up regionally with the subsequent command:
pass get github.com/prometheus/alertmanager/cmd/amtool
Note: If you are the consumer who can get entry to the observability-alertmanager-zero pod directly, you could use amtool, that is bundled with that pod. Use the amtool alert –alertmanager.url=http://localhost:9093 command to listing alerts.
Connect to RHACM Alertmanager
RHACM exposes the alertmanager API via a route. You can get the alertmanager URL with the aid of using the use of the subsequent command:
oc get route alertmanager -n open-cluster-management-observability -o jsonpath=”{.spec.host}”
Before you connect with alertmanager, you furthermore may want to byskip bearer-token to amtool. You can get the bearer token from RHACM console Configure consumer or fetch the token with the subsequent command: oc whoami -t when you have logged for your OCP cluster. You can create a config report in YAML layout from one in every of default config
locations: $HOME/.config/amtool/config.yml or /etc/amtool/config.yml. View the subsequent instance syntax:
alertmanager.url: https://alertmanager-open-cluster-management-observability.apps.xxx
http.config.file: $HOME/.config/amtool/http_config.ymlSpecify the record for the http.config.record parameter and in http_config format. View the subsequent instance syntax:
authorization:
type: Bearer
credentials: sha256~xxxxxxx
tls_config:
insecure_skip_verify: trueCONFIGURATION
You can use amtool to apprehend the cutting-edge alertmanager configuration. View the subsequent alertmanager configuration sample:
global:
resolve_timeout: 5m
http_config: {}
smtp_hello: localhost
smtp_require_tls: true
slack_api_url: <secret>
pagerduty_url: https://events.pagerduty.com/v2/enqueue
hipchat_api_url: https://api.hipchat.com/
opsgenie_api_url: https://api.opsgenie.com/
wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
route:
receiver: default-receiver
group_by:
- alertname
- cluster
repeat_interval: 45m
receivers:
- name: default-receiver
slack_configs:
- send_resolved: true
http_config: {}
api_url: <secret>
...Examples
Continue analyzing to learn the way you could use amtool to manipulate signals.
1. View all lively signals with the subsequent command:
$ amtool alert
Alertname Starts At Summary State
KubeCPUOvercommit 2021-10-27 07:47:32 UTC Cluster has overcommitted CPU resource requests. active
2. View all lively indicators with prolonged outputs via way of means of strolling the subsequent command:
$ amtool alert -o extended
Labels Annotations Starts At Ends At Generator URL State
alertname="KubeCPUOvercommit" cluster="cyang2-kind" severity="warning" description="Cluster has overcommitted CPU resource requests for Pods and cannot tolerate node failure." summary="Cluster has overcommitted CPU resource requests." 2021-10-27 07:47:32 UTC 2021-11-10 13:50:02 UTC http://prometheus-k8s-0:9090/graph?g0.expr=sum%28namespace_cpu%3Akube_pod_container_resource_requests%3Asum%29+%2F+sum%28kube_node_status_allocatable%7Bresource%3D%22cpu%22%7D%29+%3E+%28count%28kube_node_status_allocatable%7Bresource%3D%22cpu%22%7D%29+-+1%29+%2F+count%28kube_node_status_allocatable%7Bresource%3D%22cpu%22%7D%29&g0.tab=1 active
3. Silence a particular alert with the subsequent command:
$ amtool silence add alertname=KubeCPUOvercommit --comment=acked
290bb29e-6457-47b0-b10d-140b10418c4c
4. Silence all signals with the label matches.
RHACM provides the cluster label for every alert. RHACM makes use of this label to become aware of in which the alert is from. So you could silence all signals from 1 cluster with the aid of using the use of the subsequent commands:
$ amtool silence add cluster="local-cluster" --comment=acked
48ccecdc-abb1-4196-83fd-593ba010ddf3
$ amtool silence add alertname="KubeCPUOvercommit" cluster=~".+1" --comment=acked
18abf36d-b01e-46a0-ba1a-b814acb4bae0
Similarly, regex matching is likewise supported. The =~ syntax (much like Prometheus) is used to symbolize a regex match. Regex matching may be utilized in mixture with an immediate match. This announcement provides a silence that fits indicators with the alertname=”KubeCPUOvercommit”, and cluster is at stop of one label fee pairs set.
5. View silences with the subsequent command:
$ amtool silence query
ID Matchers Ends At Created By Comment
290bb29e-6457-47b0-b10d-140b10418c4c alertname="KubeCPUOvercommit" 2021-11-10 14:54:44 UTC chuyang acked
6. Expire a silence with the subsequent command:
$ amtool silence expire 290bb29e-6457-47b0-b10d-140b10418c4c
7. Expire all silences the usage of the subsequent command:
$ amtool silence expire $(amtool silence query -q)Conclusion
In conclusion, Red Hat Advanced Cluster Management for Kubernetes (RHACM) helps the use of amtool to control RHACM alerts. I wish this weblog is beneficial to you!
