Skip to content
This repository was archived by the owner on Dec 14, 2022. It is now read-only.

Commit 56ce6a1

Browse files
author
Chris Wiechmann
committed
#73 ILM-Policies and documentation adjusted - Details provided now for 30 days
1 parent 81f5418 commit 56ce6a1

File tree

6 files changed

+39
-27
lines changed

6 files changed

+39
-27
lines changed

‎CHANGELOG.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
99
- Now the Application-Id shown in Traffic-Monitor column: Subject resolves to the Application-Name [#69](https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/issues/69)
1010
- Now it is possible to perform a Full-Text search (search for a part of value) on the Subject-Column in Traffic-Monitor [#70](https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/issues/70)
1111

12+
### Changed
13+
- ILM policies optimized to reduce the required disk space [#73](https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/issues/73)
14+
1215
### Fixed
1316
- Indices are rolled over too often when an Index-Template is changed [#72](https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/issues/72)
1417

@@ -18,7 +21,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
1821
- Index-Rollover error when using regional indices [#66](https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/issues/66)
1922

2023
### Changed
21-
- ILM policies optimized for the ideal index sizes [#68](https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/issues/68)
24+
- ILM policies optimized for the ideal index sizes and number of shards [#68](https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/issues/68)
2225

2326
### Added
2427
- Initial version of Update instructions. See [UPDATE.md](UPDATE.md)

‎README.md‎

Lines changed: 24 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -735,11 +735,11 @@ The configuration is defined here per data type (e.g. Summary, Details, Audit, .
735735

736736
| Data-Type | Description | Hot (Size/Days) | Warm | Cold | Delete | Total |
737737
| :--- |:--- | :--- | :--- | :--- | :--- | :--- |
738-
| **Traffic-Summary** | Main index for traffic-monitor overview and primary dashboard | 30GB / 15 days | 15 days | 30 days | 10 days | 70 days |
739-
| **Traffic-Details** | Details in Traffic-Monitor for Policy, Headers and Payload reference | 30GB / 15 days | 7 days | 10 days | 5 days | 37 days |
740-
| **Traffic-Trace** | Trace-Messages belonging to an API-Request shown in Traffic-Monitor | 30GB / 60 days | 7 days | 10 days | 5 days | 82 days |
741-
| **General-Trace** | General trace messages, like Start- & Stop-Messages | 30GB / 60 days | 7 days | 10 days | 5 days | 82 days |
742-
| **Gateway-Monitoring** | System status information (CPU, HDD, etc.) from Event-Files | 30GB / 60 days | 15 days | 15 days | 15 days | 105 days |
738+
| **Traffic-Summary** | Main index for traffic-monitor overview and primary dashboard | 30GB / 15 days | 5 days | 10 days | 0 days | 30 days |
739+
| **Traffic-Details** | Details in Traffic-Monitor for Policy, Headers and Payload reference | 30GB / 15 days | 5 days | 10 days | 0 days | 30 days |
740+
| **Traffic-Trace** | Trace-Messages belonging to an API-Request shown in Traffic-Monitor | 30GB / 60 days | 5 days | 10 days | 0 days | 75 days |
741+
| **General-Trace** | General trace messages, like Start- & Stop-Messages | 30GB / 60 days | 5 days | 10 days | 0 days | 75 days |
742+
| **Gateway-Monitoring** | System status information (CPU, HDD, etc.) from Event-Files | 30GB / 60 days | 30 days | 15 days | 0 days | 105 days |
743743
| **Domain-Audit** | Domain Audit-Information as configured in Admin-Node-Manager | 10GB / 270 days | 270 days| 720 days| 15 days | >3 years|
744744

745745
Please note:
@@ -780,16 +780,23 @@ Please note:
780780
#### Retention period
781781

782782
The second important aspect for sizing is the retention period, which defines how long data should be available. Accordingly, disk space must be made available.
783-
The Traffic-Summary, Traffic-Details and Trace-Messages indicies play a particularly important role here. The solution is delivered with default values which you can read [here](#lifecycle-management). Based on the these default values which result in ap. 60 days the following disk space is required.
784-
785-
| Volume per day | Stored documents | Total Disk-Space | Comment |
786-
| :--- | :--- | :--- | :--- |
787-
| up to 1 Mio (~15 TPS) | 60 Mio. | 30 GB | 2 Elasticsearch nodes, each with 15 GB |
788-
| up to 5 Mio (~60 TPS) | 300 Mio. | 60 GB | 2 Elasticsearch nodes, each with 30 GB |
789-
| up to 10 Mio (~120 TPS) | 600 Mio. | 160 GB | 2 Elasticsearch nodes, each with 80 GB |
790-
| up to 25 Mio (~300 TPS) | 1.500 Bil. | 500 GB | 3 Elasticsearch nodes, each with 200 GB |
791-
| up to 50 Mio (~600 TPS) | 3.000 Bil. | 1 TB | 4 Elasticsearch nodes, each with 250 GB |
792-
783+
In particular the Traffic-Summary and Traffic-Details indicies become huge and therefore play a particularly important role here. The solution is delivered with default values which you can read [here](#lifecycle-management). Based on the these default values which result in ap. 30 days the following disk space is required.
784+
785+
| Volume per day | Total Disk-Space | Comment |
786+
| :--- | :--- | :--- |
787+
| up to 1 Mio (~15 TPS) | 60 GB | 2 Elasticsearch nodes, each with 50 GB |
788+
| up to 5 Mio (~60 TPS) | 250 GB | 2 Elasticsearch nodes, each with 150 GB |
789+
| up to 10 Mio (~120 TPS) | 500 GB | 2 Elasticsearch nodes, each with 250 GB |
790+
| up to 25 Mio (~300 TPS) | 1 TB | 3 Elasticsearch nodes, each with 500 GB |
791+
| up to 50 Mio (~600 TPS) | 2 TB | 4 Elasticsearch nodes, each with 500 GB |
792+
793+
If the required storage space is unexpectedly higher, then you can do the following:
794+
- add an additional Elasticsearch cluster node at a later time.
795+
- Elasticsearch will then start balancing the cluster by moving shards to this new node
796+
- this additional node will of course also improve the overall performance of the cluster
797+
- increase the disk space of an existing node
798+
- if the cluster state is green, you can stop a node, allocate more disk space, and then start it again
799+
- the available disk space is used automatically by allocating shards
793800

794801
### Test infrastructure
795802

@@ -801,6 +808,8 @@ The following test infrastructure was used to determine the [maximum capacity or
801808
| 4x | AWS EC2 t2.xlarge instance | 4 vCPUS | 16GB | 30GB | Logstash, API-Builder, Memcached | 7.10.0 | Logstash instances started as needed for the test. Logstash, API-Builder and Memcache always run together |
802809
| 5x | AWS EC2 t2.xlarge instance | 4 vCPUS | 16GB | 80GB | Elasticsearch | 7.10.0 | Elasticsearch instances started as needed. Kibana running on the first node |
803810

811+
There is no specific reason that EC2 t2.xlarge instances were used for the test setup. The deciding factor was simply the number of CPU cores and 16 GB RAM.
812+
804813
<p align="right"><a href="#table-of-content">Top</a></p>
805814

806815
## Updates

‎apibuilder4elastic/elasticsearch_config/apigw-monitoring/ilm_policy.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
}
1515
},
1616
"warm": {
17-
"min_age": "15d",
17+
"min_age": "0ms",
1818
"actions": {
1919
"allocate": {
2020
"number_of_replicas": 1,
@@ -31,7 +31,7 @@
3131
}
3232
},
3333
"cold": {
34-
"min_age": "15d",
34+
"min_age": "30d",
3535
"actions": {
3636
"set_priority": {
3737
"priority": 0

‎apibuilder4elastic/elasticsearch_config/trace-messages/ilm_policy.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
}
1515
},
1616
"warm": {
17-
"min_age": "7d",
17+
"min_age": "0ms",
1818
"actions": {
1919
"allocate": {
2020
"number_of_replicas": 1,
@@ -31,15 +31,15 @@
3131
}
3232
},
3333
"cold": {
34-
"min_age": "10d",
34+
"min_age": "5d",
3535
"actions": {
3636
"set_priority": {
3737
"priority": 0
3838
}
3939
}
4040
},
4141
"delete": {
42-
"min_age": "5d",
42+
"min_age": "10d",
4343
"actions": {
4444
"delete": {
4545
"delete_searchable_snapshot": true

‎apibuilder4elastic/elasticsearch_config/traffic-details/ilm_policy.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
}
1515
},
1616
"warm": {
17-
"min_age": "7d",
17+
"min_age": "0ms",
1818
"actions": {
1919
"allocate": {
2020
"number_of_replicas": 1,
@@ -31,15 +31,15 @@
3131
}
3232
},
3333
"cold": {
34-
"min_age": "10d",
34+
"min_age": "5d",
3535
"actions": {
3636
"set_priority": {
3737
"priority": 0
3838
}
3939
}
4040
},
4141
"delete": {
42-
"min_age": "5d",
42+
"min_age": "10d",
4343
"actions": {
4444
"delete": {
4545
"delete_searchable_snapshot": true

‎apibuilder4elastic/elasticsearch_config/traffic-summary/ilm_policy.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
}
1515
},
1616
"warm": {
17-
"min_age": "15d",
17+
"min_age": "0ms",
1818
"actions": {
1919
"allocate": {
2020
"number_of_replicas": 1,
@@ -31,10 +31,10 @@
3131
}
3232
},
3333
"cold": {
34-
"min_age": "30d",
34+
"min_age": "5d",
3535
"actions": {
3636
"set_priority": {
37-
"priority": 0
37+
"priority": 20
3838
}
3939
}
4040
},

0 commit comments

Comments
 (0)