[FEATURE] Auto restart of prometheus-msteams pod when the entry from hashicorp vault is synced - prometheus-msteams

Hi @gouravsw thanks for filing your issue. I am trying to understand what your want to achieve. Is it correct that you use vault to store your MS Teams webhook url and inject that via the vault-agent sidecar to the config file? How did you deploy prometheus-msteams? If you've deployed it via the helm chart, then the config file is injected to the pod via a configmap. If the configmap changes, the prometheus-msteams deployment will be rolled updated as of this recent PR. This feature is available since helm chart version 0.7.1. Hence, you only need to make sure that the vault-agent sidecar updates that configmap. Does this solve your problem?

1

Answered Nov 22 '20 at 13:50

Knappek

@Knappek Hello Andy, Thank you for your response, actually config map for prometheus-msteams is static and generic and hence it would not be changing even though the entries in vault is changed (created, deleted or updated) and because of that I thought to ask for this feature, where if any changes seen in connector.yaml (@ /etc/config/ or /vault/secrets or at any other location), prometheus-msteams pod should be restarted.

1) Is it correct that you use vault to store your MS Teams webhook url and inject that via the vault-agent sidecar to the config file? A: Yes and Yes.

2) How did you deploy prometheus-msteams? A: Using helmfile which internally uses helm chartt.

3) Does this solve your problem? A: As the configmap is static, problem is not solved. I also tried to use reloader (restarts the pod when configmap or secret changes), but it did not worked as configmap is not changing.

Below I am pasting generic configmap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-msteams-config
  namespace: devops-monitoring
data:
  config.hcl: |
    "auto_auth" = {
      "method" = {
        "mount_path" = "auth/devops/k8s/dt-ue2"
        "config" = {
          "role" = "dt-ue2-read"
        }
        "type" = "kubernetes"
      }

      "sink" = {
        "config" = {
          "path" = "/home/vault/.token"
        }

        "type" = "file"
      }
    }

    "exit_after_auth" = false
    "pid_file" = "/home/vault/.pid"

    "template" = {
      "contents" = "{{- with secret \"devops/data/ms-teams-urls\" }}connectors:\n  {{- range $key, $value := .Data.data }}\n  - {{ $key }}: {{ $value }} {{end}}{{end}}"
      "destination" = "/vault/secrets/connectors.yaml"
    }

    "vault" = {
      "address" = "https://vault-test.dt.ue2.devops.com"
    }
  config-init.hcl: |
    "auto_auth" = {
      "method" = {
        "mount_path" = "auth/devops/k8s/dt-ue2"
        "config" = {
          "role" = "dt-ue2-read"
        }
        "type" = "kubernetes"
      }

      "sink" = {
        "config" = {
          "path" = "/home/vault/.token"
        }

        "type" = "file"
      }
    }

    "exit_after_auth" = true
    "pid_file" = "/home/vault/.pid"

    "template" = {
      "contents" = "{{- with secret \"devops/data/ms-teams-urls\" }}connectors:\n  {{- range $key, $value := .Data.data }}\n  - {{ $key }}: {{ $value }} {{end}}{{end}}"
      "destination" = "/vault/secrets/connectors.yaml"
    }

    "vault" = {
      "address" = "https://vault-test.dt.ue2.devops.com"
    }

1

Answered Nov 22 '20 at 14:39

gouravsw

So this is the configmap you use along the helmfile, correct?

Could you please also post the resulting configmap please? I am still trying to fully understand ho the vault-agent works, I haven't used that before. Thanks :)

1

Answered Nov 22 '20 at 18:02

Knappek

@Knappek Hello Andy, the above is the resulting configmap.

Using helm template command I have generated the above configMap.

1

Answered Nov 22 '20 at 19:23

gouravsw

I see. But this configmap is only used by the vault-agent sidecar, isn't it? How does the configmap for the prometheus-msteams config look like?

1

Answered Nov 22 '20 at 19:28

Knappek

As mention in prometheus-msteams config, Configmap of prometheus-msteams will create the connector.yaml file which will have the entries for msteams urls and this connectors.yaml is stored in some location either /etc/config/connectors.yaml or /vault/secrets/connectors.yaml, and this files is used by prometheus-msteams connector to send the alerts/messages to Microsoft team channels.

The generic config mentioned above will also save the connectors.yaml at location /vault/secrets and it is used by microsoft teams to send alertts.

1

Answered Nov 23 '20 at 03:34

gouravsw