Skip to content
This repository was archived by the owner on Dec 14, 2022. It is now read-only.

Commit 9ffd6c0

Browse files
author
Chris Wiechmann
committed
[skip ci] Let the Audit-Index grow up to 10GB before rolling
1 parent c8b4103 commit 9ffd6c0

File tree

2 files changed

+4
-7
lines changed

2 files changed

+4
-7
lines changed

README.md

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -678,12 +678,11 @@ https://docker-host:8443/apidoc/swagger.json?endpoints/trafficMonitorApi
678678
### Lifecycle Management
679679

680680
Since new data is continuously stored in Elasticsearch in various indexes, these must of course be removed after a certain period of time.
681-
The solution uses the Elasticsearch [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) for this purpose, which defines different lifecycle stages per index. The so-called ILM policies are automatically configured by the solution using [configuration files](apibuilder4elastic/elasticsearch_config) and can be reviewed in Kibana.
681+
Since version 2.0.0, the solution uses the Elasticsearch [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) feature for this purpose, which defines different lifecycle stages per index. The so-called ILM policies are automatically configured by the solution using [configuration files](apibuilder4elastic/elasticsearch_config) and can be reviewed in Kibana.
682682
The indices pass through stages such as Hot, Warm, Cold which can be used to deploy different performance hardware per stage. This means that traffic details from two weeks ago no longer have to be stored on high-performance machines.
683683

684684
The configuration is defined here per data type (e.g. Summary, Details, Audit, ...). The following table gives an overview.
685685

686-
687686
| Data-Type | Description | Hot (Size/Days) | Warm | Cold | Delete | Total |
688687
| :--- |:--- | :--- | :--- | :--- | :--- | :--- |
689688
| **Traffic-Summary** | Main index for traffic-monitor overview and primary dashboard | 30GB / 7 days | 15 days | 30 days | 10 days | 62 days |
@@ -692,7 +691,7 @@ The configuration is defined here per data type (e.g. Summary, Details, Audit, .
692691
| **Traffic-Trace** | Trace-Messages belonging to an API-Request shown in Traffic-Monitor | 30GB / 7 days | 7 days | 10 days | 5 days | 29 days |
693692
| **General-Trace** | General trace messages, like Start- & Stop-Messages | 30GB / 7 days | 7 days | 10 days | 5 days | 29 days |
694693
| **Gateway-Monitoring** | System status information (CPU, HDD, etc.) from Event-Files | 30GB / 15 days | 15 days | 15 days | 15 days | 60 days |
695-
| **Domain-Audit** | Domain Audit-Information as configured in Admin-Node-Manager | 30GB / 270 days | 270 days| 720 days| 15 days | >3 years|
694+
| **Domain-Audit** | Domain Audit-Information as configured in Admin-Node-Manager | 10GB / 270 days | 270 days| 720 days| 15 days | >3 years|
696695

697696
Please note:
698697
:point_right: It's optional to use different hardware per stage
@@ -714,7 +713,7 @@ The following recommendations are based on our tests and is splitted by the desi
714713
#### 7 Days rentention period
715714

716715
Please note the following:
717-
The Standard Index Lifecycle Policy defines that an index can grow to 50 GB and rolls into a new after 30 days. For __7 days__, the indexes should be rolled after __2 days__ instead of 30 days.
716+
The relevant indices for the messaurment below are the Traffic-Summary & Traffic-Details indices that are driving the API-Gateway Traffic-Monitor. The Lifecycle Policy for these indices defines that an index grows to 30 GB or rolls into a new after 7 days (whatevery comes first).
718717
The recommendation contains only one ElasticSearch node, which provides no data redundancy if this node fails. If you need data redundancy, another ElasticSearch node must be added. After adding another node the data is automatically distributed between them.
719718

720719
| Volume | Components | Nodes | Shards | Comment |
@@ -731,7 +730,6 @@ The recommendation contains only one ElasticSearch node, which provides no data
731730
#### 14 Days rentention period
732731

733732
Please note the following:
734-
The Standard Index Lifecycle Policy defines that an index can grow to 50 GB and rolls into a new after 30 days. For __14 days__, the indexes should be rolled after __4 days__ instead of 30 days.
735733
The recommendation contains only one ElasticSearch node up to a volume of max. 25 million transactions. This means no data redundancy if this node fails. If you need data redundancy, another ElasticSearch node must be added. After adding another node the data is automatically distributed between them.
736734

737735
| Volume | Components | Nodes | Shards | Comment |
@@ -748,7 +746,6 @@ The recommendation contains only one ElasticSearch node up to a volume of max. 2
748746

749747
#### 30 Days rentention period
750748

751-
The Standard Index Lifecycle Policy is sufficient for 30 days data retention.
752749
The recommendation contains only one ElasticSearch node up to a volume of max. 10 million transactions. This means no data redundancy if this node fails. If you need data redundancy, another ElasticSearch node must be added. After adding another node the data is automatically distributed between them.
753750

754751
| Volume | Components | Nodes | Comment |

apibuilder4elastic/elasticsearch_config/apigw-domainaudit/ilm_policy.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
"actions": {
77
"rollover": {
88
"max_age": "270d",
9-
"max_size": "30gb"
9+
"max_size": "10gb"
1010
},
1111
"set_priority": {
1212
"priority": 100

0 commit comments

Comments
 (0)