We are bringing the best of Tectonic to Red Hat OpenShift to build the most secure, hybrid Kubernetes application platform.
Prometheus is an open-source metrics and monitoring application that is shipped as an open cloud service in Tectonic. The default Prometheus instance is used to monitor the Tectonic control plane, and a cluster-wide AlertManager aggregates the alerts across multiple Prometheus instances for greater visibility. Prometheus instances can be created through the Tectonic Console or using kubectl
.
In this tutorial we will:
Prometheus instances are comprised of a few Kubernetes objects:
kube-state-metrics
.Deploying a Prometheus instance requires several Kubernetes objects. First, create a ServiceAccount
and a ClusterRoleBinding
to ensure that Prometheus can scrape kube-state-metrics
to determine the number of replicas deployed by an application.
Copy the following YAML into a file named prometheus-serviceaccount.yaml
:
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus-hello
namespace: default
Then, copy the following YAML into a file named prometheus-crb.yaml
:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus-hello
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus-k8s
subjects:
- kind: ServiceAccount
name: prometheus-hello
namespace: default
Next, use the open cloud service to deploy the Prometheus instance. Prometheus as an open cloud service will manage the underlying StatefulSet
and Pod
s for Prometheus.
Copy the following into a file named prometheus.yaml
:
apiVersion: monitoring.coreos.com/v1alpha1
kind: Prometheus
metadata:
name: hello
labels:
prometheus: hello
namespace: default
spec:
replicas: 1
version: v1.7.1
serviceAccountName: prometheus-hello
serviceMonitorSelector:
matchLabels:
k8s-app: kube-state-metrics
ruleSelector:
matchLabels:
prometheus: hello
resources:
requests:
memory: 400Mi
alerting:
alertmanagers:
- namespace: tectonic-system
name: alertmanager-main
port: web
To tell this new Prometheus instance what to monitor, create the ServiceMonitor
and the ConfigMap
.
First, copy the following into a file named prometheus-servicemonitor.yaml
:
apiVersion: monitoring.coreos.com/v1alpha1
kind: ServiceMonitor
metadata:
name: kube-state-metrics
namespace: default
labels:
k8s-app: kube-state-metrics
spec:
endpoints:
- honorLabels: true
interval: 30s
port: http-metrics
targetPort: 0
jobLabel: simple-app
namespaceSelector:
matchNames:
- tectonic-system
selector:
matchLabels:
k8s-app: kube-state-metrics
Next, copy the following into a file named prometheus-configmap.yaml
:
kind: ConfigMap
apiVersion: v1
metadata:
name: simple-prom
labels:
prometheus: hello
namespace: default
data:
alerting.rules: |
# Alert if deployment missing
ALERT SimpleDeploymentMissing
IF kube_deployment_status_replicas{deployment="simple-deployment"} < 3
FOR 10m
LABELS {severity="critical"}
ANNOTATIONS {description="Prometheus could not find the Simple Deployment"}
Next, create a means to access the Prometheus UI. Copy the following into a file named prometheus-service.yaml
:
apiVersion: v1
kind: Service
metadata:
name: prometheus-hello-world
namespace: default
spec:
type: ClusterIP
ports:
- name: web
protocol: TCP
port: 9090
targetPort: 9090
selector:
prometheus: hello
Finally, create Ingress
to enable access to Prometheus from outside the cluster. Copy the following into a file named prometheus-ingress.yaml
:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: prom-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: "tectonic"
ingress.kubernetes.io/rewrite-target: /
ingress.kubernetes.io/ssl-redirect: "true"
ingress.kubernetes.io/use-port-in-redirects: "true"
spec:
rules:
- host: prometheus.ingress.example.com
http:
paths:
- path: /
backend:
serviceName: prometheus-hello-world
servicePort: 9090
Use kubectl
to push all of these files up to the cluster at the same time. Confirm that the working directory contains only the YAML files described above, then create all required objects with a single command:
$ kubectl create -f .
configmap "simple-prom" created
clusterrolebinding "prometheus-hello" created
service "prometheus-hello-world" created
ingress "prom-ingress" created
serviceaccount "prometheus-hello" created
servicemonitor "kube-state-metrics" created
prometheus "hello" created
Now, with the Prometheus instance online, and configured to alert based on the number of replicas of the app, access the Prometheus UI to watch the configured alerts fire.
Use the URL provided by Ingress
to access the Prometheus Alerts console: prometheus.ingress.example.com/alerts.
The alert configured in Prometheus will fire if the replicas
for the simple-deployment
is fewer than 3. Scale down the simple-deployment
:
$ kubectl scale deployment/simple-deployment --replicas 2
deployment "simple-deployment" scaled
By default, Prometheus scrapes the kube-state-metrics
every 30 seconds. After scaling the deployment, wait 30 seconds, then refresh prometheus.ingress.example.com/alerts to see that the alert is now firing, as the simple-deployment
has fewer than 3 replicas.
Scale the simple-deployment
back up to 3:
$ kubectl scale deployment/simple-deployment --replicas 3
deployment "simple-deployment" scaled