Monitoring PX-Backup with Prometheus and Grafana

This document shows how you can monitor your PX-Backup cluster with Prometheus and Grafana.

Prerequisites

A PX-Backup cluster
You must have kubectl access to your PX-Backup cluster

Install and configure Prometheus

1. Enter the following combined spec and kubectl command to install the Prometheus Operator:

kubectl apply -f - <<'_EOF'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-operator
  namespace: px-backup
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-operator
subjects:
  - kind: ServiceAccount
    name: prometheus-operator
    namespace: px-backup
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-operator
  namespace: px-backup
rules:
  - apiGroups:
      - extensions
    resources:
      - thirdpartyresources
    verbs: ["*"]
  - apiGroups:
      - apiextensions.k8s.io
    resources:
      - customresourcedefinitions
    verbs: ["*"]
  - apiGroups:
      - monitoring.coreos.com
    resources:
      - alertmanagers
      - prometheuses
      - prometheuses/finalizers
      - servicemonitors
      - prometheusrules
      - podmonitors
      - thanosrulers
    verbs: ["*"]
  - apiGroups:
      - apps
    resources:
      - statefulsets
    verbs: ["*"]
  - apiGroups: [""]
    resources:
      - configmaps
      - secrets
    verbs: ["*"]
  - apiGroups: [""]
    resources:
      - pods
    verbs: ["list", "delete"]
  - apiGroups: [""]
    resources:
      - services
      - endpoints
    verbs: ["get", "create", "update"]
  - apiGroups: [""]
    resources:
      - nodes
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources:
      - namespaces
    verbs: ["list", "watch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-operator
  namespace: px-backup
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: prometheus-operator
  name: prometheus-operator
  namespace: px-backup
spec:
  selector:
    matchLabels:
      k8s-app: prometheus-operator
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: prometheus-operator
    spec:
      containers:
        - args:
            - --kubelet-service=kube-system/kubelet
            - --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1
          image: quay.io/coreos/prometheus-operator:v0.36.0
          name: prometheus-operator
          ports:
            - containerPort: 8080
              name: http
          resources:
            limits:
              cpu: 200m
              memory: 100Mi
            requests:
              cpu: 100m
              memory: 50Mi
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: prometheus-operator

2. To grant Prometheus access to the metrics API, create the ClusterRole, ClusterRoleBinding, Service, and ServiceAccount Kubernetes objects:

kubectl apply -f - <<'_EOF'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":
      {"annotations":{},"name":"px-backup-prometheus"},"rules":[{"apiGroups":[""],
      "resources":["nodes","services","endpoints","pods"],"verbs":["get","list",
      "watch"]},{"apiGroups":[""],"resources":["configmaps"],"verbs":["get"]},
      {"nonResourceURLs":["/metrics","/federate"],"verbs":["get"]}]}
  name: px-backup-prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  - /federate
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding",
      "metadata":{"annotations":{},"name":"px-backup-prometheus"},"roleRef":
      {"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole",
      "name":"px-backup-prometheus"},"subjects":[{"kind":"ServiceAccount",
      "name":"px-backup-prometheus","namespace":"px-backup"}]}
  name: px-backup-prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: px-backup-prometheus
subjects:
- kind: ServiceAccount
  name: px-backup-prometheus
  namespace: px-backup
---
apiVersion: v1
kind: Service
metadata:
  name: px-backup-prometheus
  namespace: px-backup
spec:
  type: ClusterIP
  ports:
    - name: web
      port: 9090
      protocol: TCP
      targetPort: 9090
  selector:
    prometheus: px-backup-prometheus
---
apiVersion: v1
kind: ServiceAccount
metadata:
   name: px-backup-prometheus
   namespace: px-backup

3. To specify the monitoring rules for PX-Backup, create a ServiceMonitor object by entering the following combined spec and kubectl command:

kubectl apply -f - <<'_EOF'
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
   namespace: px-backup
   name: px-backup-prometheus-sm
   labels:
     name: px-backup-prometheus-sm
spec:
   selector:
     matchLabels:
       app.kubernetes.io/name: px-backup
   namespaceSelector:
     any: true
   endpoints:
     - port: rest-api
       targetPort: 10001

4. Apply Prometheus specs for PX-Backup metrics:

kubectl apply -f - <<'_EOF'
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
   name: px-backup-prometheus
   namespace: px-backup
spec:
   replicas: 2
   logLevel: debug
   serviceAccountName: px-backup-prometheus
   serviceMonitorSelector:
     matchLabels:
       name: px-backup-prometheus-sm

Install and configure Grafana

1. Create a storage class for Grafana and persistent volumes with the grafana-data, grafana-dashboard, grafana-source-config, and grafana-extensions names:

kubectl apply -f - <<'_EOF'
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: px-grafana-sc
provisioner: kubernetes.io/portworx-volume
parameters:
   repl: "3"
   priority_io: "high"
allowVolumeExpansion: true
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
   name: grafana-data
   namespace: px-backup
   annotations:
     volume.beta.kubernetes.io/storage-class: px-grafana-sc
spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 1Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
   name: grafana-dashboard
   namespace: px-backup
   annotations:
     volume.beta.kubernetes.io/storage-class: px-grafana-sc
spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 1Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
   name: grafana-source-config
   namespace: px-backup
   annotations:
     volume.beta.kubernetes.io/storage-class: px-grafana-sc
spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 1Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
   name: grafana-extensions
   namespace: px-backup
   annotations:
     volume.beta.kubernetes.io/storage-class: px-grafana-sc
spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 1Gi

Note: In this storage class:

The provisioner parameter is set to kubernetes.io/portworx-volume. For details about the Portworx-specific parameters, refer to the Portworx Volume section of the Kubernetes website.
Three replicas of each volume will be created.

2. Enter the following command to install Grafana:

Note: If your cluster is on a cloud provider, then follow the instructions in Step 3 to export it to a nodeport or loadbalancer.

kubectl apply -n px-backup -f - <<'_EOF'
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  labels:
    app: grafana
spec:
  type: ClusterIP
  ports:
    - port: 3000
  selector:
    app: grafana
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  labels:
    app: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
       fsGroup: 2000
      containers:
        - image: grafana/grafana:6.1.6
          name: grafana
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 100m
              memory: 100Mi
            requests:
              cpu: 100m
              memory: 100Mi
          readinessProbe:
            httpGet:
              path: /login
              port: 3000
          volumeMounts:
            - name: grafana
              mountPath: /etc/grafana/provisioning/dashboard
              readOnly: false
            - name: grafana-dash
              mountPath: /var/lib/grafana/dashboards
              readOnly: false
            - name: grafana-source-cfg
              mountPath: /etc/grafana/provisioning/datasources
              readOnly: false
            - name: grafana-plugins
              mountPath: /var/lib/grafana/plugins
              readOnly: false
      volumes:
      - name: grafana
        persistentVolumeClaim:
          claimName: grafana-data
      - name: grafana-dash
        persistentVolumeClaim:
          claimName: grafana-dashboard
      - name: grafana-source-cfg
        persistentVolumeClaim:
          claimName: grafana-source-config
      - name: grafana-plugins
        persistentVolumeClaim:
          claimName: grafana-extensions

Note: In this deployment, the volumes section references the PVCs you created in the previous step.

3. Enter the following kubectl port-forward command to forward all connections made to localhost:3000 to svc/grafana:3000:

kubectl port-forward svc/grafana --namespace px-backup --address 0.0.0.0 3000

Alternatively, if your cluster is on a cloud provider, perform one of the following to export to the nodeport or loadbalancer:

To export to the nodeport, change the following parameter in the Install and configure Grafana procedure -> Step 2 code snippet:

type: NodePort

Note: The cluster node must be accessible using an external IP.

To export to the loadbalancer, create an ingress role:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: guestbook2
  namespace: px-backup
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: px-backup-prometheus 
          servicePort: 9090

4. Follow the instructions from the Grafana Support for Prometheus page of the Prometheus documentation, to create a Prometheus data source named px-backup.

5. Follow the instructions from the Importing a dashboard page of the Grafana documentation to import the PX-Backup dashboard JSON file.