Elastic Fluentd Kibana on Kubernetes
There are lots of articles explaining the installations of EFK stack individually or via scripts which can solve the purpose of getting the Stack up and running. In this article I want to have a complete installation as a suite of EFK stack with X-pack features such as Authentication and SSL with Elastic 7.6.2. At the end of the article I have linked the git repository with complete set!
From official Elastic doc.
Elastic released some security features for free as part of the default distribution (Basic license) starting in Elastic Stack 6.8 and 7.1. This new feature offering includes the ability to encrypt network traffic using SSL, create and manage users, define roles that protect index and cluster-level access, and fully secure Kibana.
After this article you shall be able to install EFK on your cluster up and running. As per the project needs and scope of the usage you can make use of different subscriptions provided by Elastic.
Pre-requisites#
- Understanding of Elastic Search (Basic). I shall soon write how ES stack can be used.
- Basic understanding of fluentd and kibana to begin.
- Understanding of Kubernetes (Moderate). If working with cloud provider more understand would be better.
- Helm charts and some scripting.
Background
Here it is very nicely explained about the EFK manifests in Kubernetes.
Certificates#
We will be installing 3 Node Elastic cluster as StatefulSet, since we are using the SSL for elastic search we would want to generate the certificate for the elastic nodes! The Encryption is done within the cluster as well as external HTTP calls. (Later we will talk about the service names aka DNS names used in the certificates) For now this script will create certificate for us by executing the commands in the sample docker image of elastic.
#!/bin/bash
if || ; then
fi
||
#password=$$([ ! -z "$$ELASTIC_PASSWORD" ] && echo $$ELASTIC_PASSWORD || echo $$(docker run --rm $(ELASTICSEARCH_IMAGE) /bin/sh -c "< /dev/urandom tr -cd '[:alnum:]' | head -c20")) && \
&& \
&& && \
&& \
# Extract the CA certificate in PEM format for kibana to ssl verify elastic TLS
TL;DR
Scripts create 2 certificates in pkc12 format. one is CA and other is the certificate. The certificates are used for volume mounts on elastic nodes.
Detailed(Skip if you already know about certs..)
Elastic has already provided scripts to generate certificates for us. If you can exec the elastic pod/container you can find the scripts as below. More details here
Two certificates are created one is self signed CA(elastic-stack-ca.p12) and other is the Certificate(elastic-certificates.p12) signed out of this CA. They are in pkcs12 format which has the private key and protected with password. It is recommended to provide passwords to the certificates as argument to the script. Kibana needs the CA in pem format for verifying the elastic connection; it is also generated.
Configure Secrets#
- K8s Secret for elastic credentials with userId and password.
- K8s Secret for elastic certificate in p12 format. Needed to mount on the elastic nodes.
- K8s Secret for elastic CA in pem format. This is needed for kibana for ssl verify.
Configure helm values#
Next step is to configure the elastic properties via helm values file.
Elastic configurations#
Enable the xpack security and configure the keystore and truststore . Configure the Elastic user and password for authn. Configure the certificate paths and mount the certificates via secret. Set the password which is used while creating the certificates.(I have used random pass , recommended to change it).
Note : The default clustername is elasticsearch hence the master created for elastic will be elasticsearch-master.
extraEnvs:
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: ELASTIC_USERNAME
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
esConfig:
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path:
/usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path:
/usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path:
/usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.path:
/usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.keystore.password: $0meR@nd0mP@ss
xpack.security.transport.ssl.truststore.password: $0meR@nd0mP@ss
xpack.security.http.ssl.keystore.password: $0meR@nd0mP@ss
xpack.security.http.ssl.truststore.password: $0meR@nd0mP@ss
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs
Fluentd configurations#
Its straight and simple! provide the elasticsearch endpoint service and also the logs from the kube-system and istio-system are not included! This is optional and you can remove/add as per your needs.
extraConfigMaps: # Ignoring the logs from kube-system and istio-
system
out.conf: |-
<match kubernetes.var.log.containers.**_kube-system_**>
@type null
</match>
<match kubernetes.var.log.containers.**_istio-system_**>
@type null
</match>
<match kubernetes.var.log.containers.**_efk_**>
@type null
</match>
elasticsearch:
host: elasticsearch-master
auth:
enabled: true
user: "elastic" # this is just for testing. Use Kibana user
instead
password: "elastic$123"
port: 9200
scheme: "https"
sslVerify: false
Kibana Configurations#
It is again simple and straight! Configure the Elastic Search connections via secret and provide the elastic-ca.pem file as volume mount for ssl verifying the elastic service
extraEnvs:
- name: 'ELASTICSEARCH_USERNAME'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
- name: 'ELASTICSEARCH_PASSWORD'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: 'KIBANA_ENCRYPTION_KEY'
valueFrom:
secretKeyRef:
name: kibana
key: encryptionkey
kibanaConfig:
kibana.yml: |
elasticsearch.ssl:
certificateAuthorities:
/usr/share/kibana/config/certs/elasticsearch-ca.pem `
verificationMode: certificate # Elastic CA cert for SSL verify secretMounts:
- name: elastic-ca
secretName: elastic-ca-pem # Secret Names are hardcoded for
simplicity
path: /usr/share/kibana/config/certs
Helm Chart installations#
Create a simple helm chart and create a new requirements.yaml with all dependencies of efk.
---
dependencies:
- name: "elasticsearch"
repository: "@elastic"
version: "7.6.2"
- name: "fluentd-elasticsearch"
repository: "@kiwigrid"
version: "6.2.3"
- name: "kibana"
repository: "@elastic"
version: "7.6.2"
Now the values are ready and also the requirements. We just need to install the chart.
&&
This will create elasticservice-master as the cluster service and also headless service(elasticsearch-master-headless) for seeding the elastic hosts. The indivudual nodes are identified as FQDN elasticsearch-master-0.elasticsearch-master-headless, elasticsearch-master-1.elasticsearch-master-headless, elasticsearch-master-1.elasticsearch-master-headless . These are exactly the DNS names used in the certificates. Additionally elasticsearch-master is also added in the certificate since kibana would also use this certificate as http client.(I am using same certificate here for both transport layer security and http security)
The kibana is currently running as k8s service with type as clusterIP. The charts of kibana provide way to install ingress so the service can be available outside.!
You can port-forward the kibana pod and try with the login user we created.
Note:
- This article is a vanilla install. You can use sealed secrets for making it more secure
- The passwords can be securely mounted as keystore secret which will be picked by init container and added into the pods but this is not working currently.