Skip to content

Slack

Although a turnkey integration is available for email based notifications, organizations may wish to send notifications to other platforms in use internally. This recipe documents how organizations can customize the "Alert Manager" system addon configuration and send the alerts to Slack


Configure Slack

If you want to receive the notifications via Slack, you will need an existing Slack workspace or create a new one for testing here

To setup the alerting to your Slack workspace, you will need a Slack API URL

  • Go to Slack > Administration > Manage apps
  • Go to "Custom Integrations"
  • Search for "Incoming Webhooks" in App Directory and select the integrations

Slack

  • Add it to your Slack workspace

Slack

  • Then configure the channel where you would like to receive the notifications
  • Confirm the configuration and click Add Incoming Webhooks Integration

Slack

  • Copy the Webhook URL (Slack API URL) for the integration to use in the next steps

Slack


Configure Alert Manager Addon

Follow the below steps to create a customized "Alert Manager" System Addon to send notifications to the Slack channel from the prior step.

  • Navigate to Infrastructure -> Addons
  • Create a new addon with using "Customize System Addon" option.

Custom AlertManager

  • Create a new version for the custom system Alert Manager addon.

Custom AlertManager

  • Download the default configuration manifest.

Custom AlertManager

  • Modify the "global" section of the downloaded configuration to add the Slack API URL that you copied from the "Configure Slack Integration" steps:
global:
  resolve_timeout: 5m
  slack_api_url: https://hooks.slack.com/services/T01D6D4xxxxxxxxlhvzNg
  • Now, add the custom receiver for Slack notifications to the "receivers" section of the configuration:
receivers:
- name: default-receiver
- name: rafay-webhook
  webhook_configs:
  - http_config:
      tls_config:
        ca_file: /etc/config/certs/ca.crt
        cert_file: /etc/config/certs/client.crt
        insecure_skip_verify: true
        key_file: /etc/config/certs/client.key
        server_name: '{{{ .V2EventFrameworkFQDN }}}'
    url: https://{{{ .V2EventFrameworkFQDN }}}/event/v1/alert/raise
- name: myalerts
  slack_configs:
  - channel: '#alert'
    send_resolved: true
  • Optionally, you can further customize the "slack_configs" to make the notifications easier to read. An example is provided here.
- name: myalerts
  slack_configs:
  - channel: '#alert'
    send_resolved: true
    text: |-
      {{ range .Alerts -}}
      *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
      *Description:* {{ .Annotations.description }}
      *Details:*
        {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
        {{ end }}
      {{ end }}
    title: |-
      [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
      {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
        {{" "}}(
        {{- with .CommonLabels.Remove .GroupLabels.Names }}
          {{- range $index, $label := .SortedPairs -}}
            {{ if $index }}, {{ end }}
            {{- $label.Name }}="{{ $label.Value -}}"
          {{- end }}
        {{- end -}}
        )
      {{- end }}
  • Next add the custom receiver for Slack notifications to the routes configuration in "route" section
route:
  group_interval: 5m
  group_wait: 10s
  receiver: rafay-webhook
  repeat_interval: 3h
  routes:
  - continue: true
    match_re:
      severity: .*
    receiver: rafay-webhook
  - match_re:
      severity: .*
    receiver: myalerts
  • Upload the modified configuration to the addon
  • Save the custom Alert Manager addon version

Custom AlertManager


Update Cluster Blueprint

Follow the below steps to deploy the customized Alert Manager configuration for Slack notifications

  • Go to Infrastructure > Blueprints
  • Add this custom Alert Manager addon to the custom blueprint

Custom AlertManager

  • Update the new blueprint version with the customized Alert Manager addon to the clusters

Custom AlertManager

  • The new system Alert Manager addon will be deployed to the clusters

Custom AlertManager

In a few minutes, the blueprint will be published successfully and the customized Alert Manager addon is deployed to the clusters

Custom AlertManager


Verify Configuration

Once there are alerts generated in the clusters, you will receive notifications in your Slack channel for the alerts. An illustrative example is shown below.

Notifications in Slack