Skip to content
This repository was archived by the owner on Dec 14, 2022. It is now read-only.

Commit 78f7e7b

Browse files
author
Chris Wiechmann
committed
Added documentation to monitor disk usage
1 parent 8dfd130 commit 78f7e7b

File tree

1 file changed

+60
-45
lines changed

1 file changed

+60
-45
lines changed

README.md

Lines changed: 60 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -528,6 +528,22 @@ This ensures that clients can use the available Elasticsearch nodes for a fail-o
528528

529529
<p align="right"><a href="#table-of-content">Top</a></p>
530530

531+
### Configure cluster UUID
532+
533+
This step is optional, but required to monitor your Filebeat instances as part of the stack monitoring. To obtain the Cluster UUID run the following in your browser:
534+
`https://elasticsearch1:9200/` (if you have already activated authentication you can use the elastic user here)
535+
536+
Take over the UUID into the .env file:
537+
`ELASTICSEARCH_CLUSTER_UUID=XBmL4QynThmwg0X0YN-ONA`
538+
539+
You may also configure the following parameters: `GATEWAY_NAME` & `GATEWAY_REGION` to make you Filebeat instances unique.
540+
541+
![Monitoring-Overview][Monitoring-Overview]
542+
543+
To activate these changes the Filebeat service must be restarted.
544+
545+
<p align="right"><a href="#table-of-content">Top</a></p>
546+
531547
### Setup API-Manager
532548

533549
Before a document is send to Elasticsearch, additional information for the processed API is requested by Logstash from the API-Manager through an API lookup. This lookup is handled by the API-Builder and performed against the configured API-Manager.
@@ -710,6 +726,42 @@ The monitoring users are used to send metric information to Elasticsearch to ena
710726

711727
<p align="right"><a href="#table-of-content">Top</a></p>
712728

729+
### Custom certificates
730+
731+
_If you are using an existing Elasticsearch cluster, you have to provide the required CA to the solution to allow certificate validation._
732+
733+
The project is shipped with sample long running certificates/keys that should help you to get started with the solution. For a production environment these certificates and keys should be replaced with custom certificates, as the sample certificates & key are public available on GitHub.
734+
735+
After you have created the corresponding certificates and keys based on your CA, you must save them in the folder: `certificates`.
736+
Afterwards these certificates must be configured in the `.env` file.
737+
```
738+
API_BUILDER_SSL_KEY=config/certificates/corporate-certificate.key
739+
API_BUILDER_SSL_CERT=config/certificates/corporate-certificate.crt
740+
API_BUILDER_SSL_KEY_PASSWORD=dfslkjaskljdklasjdlas
741+
ELASTICSEARCH_CA=config/certificates/corp-ca.crt
742+
ELASTICSEARCH_KEY=config/certificates/corporate-elasticsearch.key
743+
ELASTICSEARCH_KEY_PASSPHRASE=config/certificates/corporate-elasticsearch.crt
744+
ELASTICSEARCH_CRT=config/certificates/corporate-elasticsearch.key
745+
KIBANA_KEY=config/certificates/corporate-kibana.key
746+
KIBANA_CRT=config/certificates/corporate-kibana.crt
747+
```
748+
You can find more information about the individual certificates in the `.env` file.
749+
750+
<p align="right"><a href="#table-of-content">Top</a></p>
751+
752+
### Secure API-Builder Traffic-Monitor API
753+
The API-Builder project for providing access to Elasticsearch data has no access restrictions right now. To ensure only API-Gateway Manager users (topology administrators with proper RBAC role) or other users with appropriate access rights can query the log data, one can expose this API via API-Manager and add security here.
754+
755+
To import the API Builder application REST-API into your API-Manager, you can access the Swagger/OpenAPI definition here (replace docker-host and port appropriately for the container that is hosting the API-Builder project):
756+
https://docker-host:8443/apidoc/swagger.json?endpoints/trafficMonitorApi
757+
758+
<p align="right"><a href="#table-of-content">Top</a></p>
759+
760+
## Monitoring
761+
762+
It is important that the solution is monitored appropriately and by default Internal-Stack Monitoring is used for this purpose, which monitors the Elasticsearch cluster, Kibana, Logstash and Filebeat.
763+
You can alternatively use Metricbeat and you can find more guidance in this section on platform monitoring.
764+
713765
### Enable Metricbeat
714766

715767
In the default configuration, the solution uses the so-called self-monitoring. This means that components such as Logstash, Kibana, Filebeat, etc. independently send monitoring information (metrics) to Elasticsearch. However, this approach is not recommended by Elastic and is deprecated.
@@ -764,57 +816,20 @@ You can enable Application Performance Monitoring (APM) to monitor APIBuilder4El
764816

765817
![Service API-Builder4Elastic overview](imgs/apm/2_apm-apibuilder4elastic-overview.png)
766818

767-
768819
Learn how to set up and activate APM [here](apm).
769820

770-
### Configure cluster UUID
771-
772-
This step is optional, but required to monitor your Filebeat instances as part of the stack monitoring. To obtain the Cluster UUID run the following in your browser:
773-
`https://elasticsearch1:9200/` (if you have already activated authentication you can use the elastic user here)
774-
775-
Take over the UUID into the .env file:
776-
`ELASTICSEARCH_CLUSTER_UUID=XBmL4QynThmwg0X0YN-ONA`
777-
778-
You may also configure the following parameters: `GATEWAY_NAME` & `GATEWAY_REGION` to make you Filebeat instances unique.
779-
780-
![Monitoring-Overview][Monitoring-Overview]
781-
782-
To activate these changes the Filebeat service must be restarted.
783-
784-
<p align="right"><a href="#table-of-content">Top</a></p>
785-
786-
### Custom certificates
787-
788-
_If you are using an existing Elasticsearch cluster, you have to provide the required CA to the solution to allow certificate validation._
789-
790-
The project is shipped with sample long running certificates/keys that should help you to get started with the solution. For a production environment these certificates and keys should be replaced with custom certificates, as the sample certificates & key are public available on GitHub.
821+
### Disk-Usage monitoring
791822

792-
After you have created the corresponding certificates and keys based on your CA, you must save them in the folder: `certificates`.
793-
Afterwards these certificates must be configured in the `.env` file.
794-
```
795-
API_BUILDER_SSL_KEY=config/certificates/corporate-certificate.key
796-
API_BUILDER_SSL_CERT=config/certificates/corporate-certificate.crt
797-
API_BUILDER_SSL_KEY_PASSWORD=dfslkjaskljdklasjdlas
798-
ELASTICSEARCH_CA=config/certificates/corp-ca.crt
799-
ELASTICSEARCH_KEY=config/certificates/corporate-elasticsearch.key
800-
ELASTICSEARCH_KEY_PASSPHRASE=config/certificates/corporate-elasticsearch.crt
801-
ELASTICSEARCH_CRT=config/certificates/corporate-elasticsearch.key
802-
KIBANA_KEY=config/certificates/corporate-kibana.key
803-
KIBANA_CRT=config/certificates/corporate-kibana.crt
804-
```
805-
You can find more information about the individual certificates in the `.env` file.
823+
It is important that you monitor the disk usage of the Elasticsearch cluster and get alarmed accordingly.
824+
Elasticsearch also independently monitors disk usage against preconfigured thresholds and closes write operations when the high disk watermark index is exceeded. This means that no more new data can be written.
825+
To avoid this condition, your alerts should already warn below the Elasticsearch thresholds. The thresholds for Elasticsearch:
806826

807-
<p align="right"><a href="#table-of-content">Top</a></p>
827+
- Low watermark for disk usage: 85%.
828+
- High watermark for disk usage: 90%
808829

809-
### Secure API-Builder Traffic-Monitor API
810-
The API-Builder project for providing access to Elasticsearch data has no access restrictions right now. To ensure only API-Gateway Manager users (topology administrators with proper RBAC role) or other users with appropriate access rights can query the log data, one can expose this API via API-Manager and add security here.
811-
812-
To import the API Builder application REST-API into your API-Manager, you can access the Swagger/OpenAPI definition here (replace docker-host and port appropriately for the container that is hosting the API-Builder project):
813-
https://docker-host:8443/apidoc/swagger.json?endpoints/trafficMonitorApi
814-
815-
<p align="right"><a href="#table-of-content">Top</a></p>
830+
So your alerts should report a critical alert before 90%. For more information, please read here: [Disk-based shard allocation settings](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/modules-cluster.html#disk-based-shard-allocation)
816831

817-
### Lifecycle Management
832+
## Lifecycle Management
818833

819834
Since new data is continuously stored in Elasticsearch in various indexes, these must of course be removed after a certain period of time.
820835
Since version 2.0.0, the solution uses the Elasticsearch [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) feature for this purpose, which defines different lifecycle stages per index. The so-called ILM policies are automatically configured by the solution using [configuration files](apibuilder4elastic/elasticsearch_config) and can be reviewed in Kibana.

0 commit comments

Comments
 (0)