Skip to content

Commit 9308e15

Browse files
[DOCS-12074] Add images and make packs public (#32790)
* add packs * add individual packs * edits * remove example logs * fix missing dashes * move packs section up * Apply suggestions from code review Co-authored-by: jeff-morgan-dd <jeff.morgan@datadoghq.com> * delete unknown files * add images and nav * fix typo * more typos * move packs in nav * add images and updates --------- Co-authored-by: jeff-morgan-dd <jeff.morgan@datadoghq.com>
1 parent d80cd69 commit 9308e15

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+183
-21
lines changed

config/_default/menus/main.en.yaml

Lines changed: 115 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5842,56 +5842,161 @@ menu:
58425842
parent: observability_pipelines_destinations
58435843
identifier: observability_pipelines_syslog
58445844
weight: 421
5845+
- name: Packs
5846+
url: observability_pipelines/packs/
5847+
parent: observability_pipelines
5848+
identifier: observability_pipelines_packs
5849+
weight: 5
5850+
- name: Akamai CDN
5851+
url: observability_pipelines/packs/akamai_cdn/
5852+
parent: observability_pipelines_packs
5853+
identifier: observability_pipelines_packs_akamai_cdn
5854+
weight: 501
5855+
- name: Amazon CloudFront
5856+
url: observability_pipelines/packs/amazon_cloudfront/
5857+
parent: observability_pipelines_packs
5858+
identifier: observability_pipelines_packs_amazon_cloudfront
5859+
weight: 502
5860+
- name: Amazon VPC Flow Logs
5861+
url: observability_pipelines/packs/amazon_vpc_flow_logs/
5862+
parent: observability_pipelines_packs
5863+
identifier: observability_pipelines_packs_amazon_vpc_flow_logs
5864+
weight: 503
5865+
- name: AWS CloudTrail
5866+
url: observability_pipelines/packs/aws_cloudtrail/
5867+
parent: observability_pipelines_packs
5868+
identifier: observability_pipelines_packs_aws_cloudtrail
5869+
weight: 504
5870+
- name: Cisco ASA
5871+
url: observability_pipelines/packs/cisco_asa/
5872+
parent: observability_pipelines_packs
5873+
identifier: observability_pipelines_packs_cisco_asa
5874+
weight: 505
5875+
- name: Cloudflare
5876+
url: observability_pipelines/packs/cloudflare/
5877+
parent: observability_pipelines_packs
5878+
identifier: observability_pipelines_packs_cloudflare
5879+
weight: 506
5880+
- name: F5
5881+
url: observability_pipelines/packs/f5/
5882+
parent: observability_pipelines_packs
5883+
identifier: observability_pipelines_packs_f5
5884+
weight: 507
5885+
- name: Fastly
5886+
url: observability_pipelines/packs/fastly/
5887+
parent: observability_pipelines_packs
5888+
identifier: observability_pipelines_packs_fastly
5889+
weight: 508
5890+
- name: Fortinet Firewall
5891+
url: observability_pipelines/packs/fortinet_firewall/
5892+
parent: observability_pipelines_packs
5893+
identifier: observability_pipelines_packs_fortinet_firewall
5894+
weight: 509
5895+
- name: HAProxy Ingress
5896+
url: observability_pipelines/packs/haproxy_ingress/
5897+
parent: observability_pipelines_packs
5898+
identifier: observability_pipelines_packs_haproxy_ingress
5899+
weight: 510
5900+
- name: Istio Proxy
5901+
url: observability_pipelines/packs/istio_proxy/
5902+
parent: observability_pipelines_packs
5903+
identifier: observability_pipelines_packs_istio_proxy
5904+
weight: 511
5905+
- name: Netskope
5906+
url: observability_pipelines/packs/netskope/
5907+
parent: observability_pipelines_packs
5908+
identifier: observability_pipelines_packs_netskope
5909+
weight: 512
5910+
- name: NGINX
5911+
url: observability_pipelines/packs/nginx/
5912+
parent: observability_pipelines_packs
5913+
identifier: observability_pipelines_packs_nginx
5914+
weight: 513
5915+
- name: Okta
5916+
url: observability_pipelines/packs/okta/
5917+
parent: observability_pipelines_packs
5918+
identifier: observability_pipelines_packs_okta
5919+
weight: 514
5920+
- name: Palo Alto Firewall
5921+
url: observability_pipelines/packs/palo_alto_firewall/
5922+
parent: observability_pipelines_packs
5923+
identifier: observability_pipelines_packs_palo_alto_firewall
5924+
weight: 515
5925+
- name: Windows XML
5926+
url: observability_pipelines/packs/windows_xml/
5927+
parent: observability_pipelines_packs
5928+
identifier: observability_pipelines_packs_windows_xml
5929+
weight: 516
5930+
- name: ZScaler ZIA DNS
5931+
url: observability_pipelines/packs/zscaler_zia_dns/
5932+
parent: observability_pipelines_packs
5933+
identifier: observability_pipelines_packs_zscaler_zia_dns
5934+
weight: 517
5935+
- name: Zscaler ZIA Firewall
5936+
url: observability_pipelines/packs/zscaler_zia_firewall/
5937+
parent: observability_pipelines_packs
5938+
identifier: observability_pipelines_packs_zscaler_zia_firewall
5939+
weight: 518
5940+
- name: Zscaler ZIA Tunnel
5941+
url: observability_pipelines/packs/zscaler_zia_tunnel/
5942+
parent: observability_pipelines_packs
5943+
identifier: observability_pipelines_packs_zscaler_zia_tunnel
5944+
weight: 519
5945+
- name: Zscaler ZIA Web Logs
5946+
url: observability_pipelines/packs/zscaler_zia_web_logs/
5947+
parent: observability_pipelines_packs
5948+
identifier: observability_pipelines_packs_zscaler_zia_web_logs
5949+
weight: 520
58455950
- name: Search Syntax
58465951
url: observability_pipelines/search_syntax/
58475952
parent: observability_pipelines
58485953
identifier: observability_pipelines_search_syntax
5849-
weight: 5
5954+
weight: 6
58505955
- name: Scaling and Performance
58515956
url: observability_pipelines/scaling_and_performance/
58525957
parent: observability_pipelines
58535958
identifier: observability_pipelines_scaling_and_performance
5854-
weight: 6
5959+
weight: 7
58555960
- name: Handling Load and Backpressure
58565961
url: observability_pipelines/scaling_and_performance/handling_load_and_backpressure/
58575962
parent: observability_pipelines_scaling_and_performance
58585963
identifier: observability_pipelines_handling_load_and_backpressure
5859-
weight: 601
5964+
weight: 701
58605965
- name: Best Practices for Scaling Observability Pipelines
58615966
url: observability_pipelines/scaling_and_performance/best_practices_for_scaling_observability_pipelines/
58625967
parent: observability_pipelines_scaling_and_performance
58635968
identifier: observability_pipelines_best_practices_for_scaling_observability_pipelines
5864-
weight: 602
5969+
weight: 702
58655970
- name: Monitoring and Troubleshooting
58665971
url: observability_pipelines/monitoring_and_troubleshooting/
58675972
parent: observability_pipelines
58685973
identifier: observability_pipelines_monitoring_and_troubleshooting
5869-
weight: 7
5974+
weight: 8
58705975
- name: Worker CLI Commands
58715976
url: observability_pipelines/monitoring_and_troubleshooting/worker_cli_commands/
58725977
parent: observability_pipelines_monitoring_and_troubleshooting
58735978
identifier: observability_pipelines_worker_cli_commands
5874-
weight: 701
5979+
weight: 801
58755980
- name: Monitoring Pipelines
58765981
url: observability_pipelines/monitoring_and_troubleshooting/monitoring_pipelines/
58775982
parent: observability_pipelines_monitoring_and_troubleshooting
58785983
identifier: observability_pipelines_monitoring_pipelines
5879-
weight: 702
5984+
weight: 802
58805985
- name: Pipeline Usage Metrics
58815986
url: observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/
58825987
parent: observability_pipelines_monitoring_and_troubleshooting
58835988
identifier: observability_pipelines_pipeline_usage_metrics
5884-
weight: 703
5989+
weight: 803
58855990
- name: Troubleshooting
58865991
url: observability_pipelines/monitoring_and_troubleshooting/troubleshooting/
58875992
identifier: observability_pipelines_troubleshooting
58885993
parent: observability_pipelines_monitoring_and_troubleshooting
5889-
weight: 704
5994+
weight: 804
58905995
- name: Guides
58915996
url: observability_pipelines/guide/
58925997
parent: observability_pipelines
58935998
identifier: observability_pipelines_guide
5894-
weight: 8
5999+
weight: 9
58956000
- name: Log Management
58966001
url: logs/
58976002
pre: log

content/en/observability_pipelines/packs/_index.md

Lines changed: 28 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,22 +9,39 @@ cascade:
99

1010
## Overview
1111

12-
When you set up a pipeline to send logs from a specific source to Observability Pipelines, you might have questions such as:
12+
{{< img src="observability_pipelines/packs/packs.png" alt="The packs section of Observability Pipelines" style="width:100%;" >}}
13+
14+
When setting up a pipeline to send logs from a specific source to Observability Pipelines, you often need to decide how to process and manage those logs.
15+
16+
Questions such as the following might come up:
1317

1418
- Which logs from this source are important?
15-
- Which logs from this source should be dropped?
16-
- Which logs should be retained?
17-
- Should logs be sampled?
18-
- Should quotas be added?
19+
- Which logs can safely be dropped?
20+
- Should repetitive logs be sampled?
21+
- Which fields should be parsed or formatted for the destination?
22+
23+
Making these decisions typically requires coordination across multiple teams and detailed knowledge of each log source.
24+
25+
Observability Pipelines Packs provide predefined configurations to help you make these decisions quickly and consistently. Packs apply Datadog-recommended best practices for specific log sources such as Akamai, AWS CloudTrail, Cloudflare, Fastly, Palo Alto Firewall, and Zscaler.
26+
27+
### What Packs do
28+
29+
Each Pack includes source-specific configurations that defines:
30+
31+
- **Fields that can safely be removed** to reduce payload size
32+
- **Logs that can be dropped**, such as duplicate events or health checks
33+
- **Logs that should be retained or parsed**, such as errors or security detections
34+
- **Formatting and normalization rules** to align logs across different destinations and environments
35+
36+
By using Packs, you can apply consistent parsing, filtering, and routing logic for each log source without creating configurations manually.
1937

20-
Often, you need to consult with different teams to answer these questions.
38+
### Why use Packs
2139

22-
Use Observability Pipelines Packs to help you set up and optimize Observability Pipelines without extensive manual configuration. Packs contain predefined configurations that are specific to a source and identify:
40+
Packs help teams:
2341

24-
- Log fields that can safely be removed
25-
- Logs that can be dropped, such as duplicated logs
26-
- Logs that need to be parsed
27-
- Logs that need to be formatted for the destination
42+
- **Reduce ingestion volume and costs** by filtering or sampling repetitive, low-value events
43+
- **Maintain consistency** in parsing and field mapping across environments and destinations
44+
- **Accelerate setup** by applying ready-to-use configurations for common sources
2845

2946
## Packs
3047

content/en/observability_pipelines/packs/akamai_cdn.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ description: Learn more about the Akamai CDN pack.
55

66
## Overview
77

8+
{{< img src="observability_pipelines/packs/akamai_cdn.png" alt="The Akamai pack" style="width:25%;" >}}
9+
810
Akamai logs show client requests and responses at the edge.
911

1012
What this pack does:

content/en/observability_pipelines/packs/amazon_cloudfront.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ description: Learn more about the AWS CloudFront pack.
55

66
## Overview
77

8+
{{< img src="observability_pipelines/packs/aws_cloudfront.png" alt="The Amazon CloudFront pack" style="width:25%;" >}}
9+
810
AWS CloudFront logs show requests, cache use, and edge activity.
911

1012
What this pack does:

content/en/observability_pipelines/packs/amazon_vpc_flow_logs.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ description: Learn more about the Amazon VPC Flow Logs pack.
55

66
## Overview
77

8+
{{< img src="observability_pipelines/packs/aws_vpc_flow_logs.png" alt="The Amazon VPC Flow Logs pack" style="width:25%;" >}}
9+
810
Amazon VPC Flow Logs capture network traffic between VPC resources.
911

1012
What this pack does:

content/en/observability_pipelines/packs/aws_cloudtrail.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ description: Learn more about the AWS CloudTrail pack.
55

66
## Overview
77

8+
{{< img src="observability_pipelines/packs/aws_cloudtrail.png" alt="The AWS CloudTrail pack" style="width:25%;" >}}
9+
810
AWS CloudTrail records API calls and account activity across AWS services.
911

1012
What this pack does:

content/en/observability_pipelines/packs/cisco_asa.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ description: Learn more about the Cisco ASA pack.
55

66
## Overview
77

8+
{{< img src="observability_pipelines/packs/cisco_asa.png" alt="The Cisco ASA pack" style="width:25%;" >}}
9+
810
Cisco ASA firewall logs capture syslog events for traffic, VPNs, and security alerts.
911

1012
What this pack does:

content/en/observability_pipelines/packs/cloudflare.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ description: Learn more about the Cloudflare pack.
55

66
## Overview
77

8+
{{< img src="observability_pipelines/packs/cloudflare.png" alt="The Cloudflare pack" style="width:25%;" >}}
9+
810
Cloudflare logs show edge traffic, performance, and security.
911

1012
What this pack does:

content/en/observability_pipelines/packs/f5.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ description: Learn more about the F5 pack.
55

66
## Overview
77

8+
{{< img src="observability_pipelines/packs/f5.png" alt="The F5 pack" style="width:25%;" >}}
9+
810
F5 logs capture traffic, security policy, and intrusion events.
911

1012
What this pack does:

content/en/observability_pipelines/packs/fastly.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ description: Learn more about the Fastly pack.
55

66
## Overview
77

8+
{{< img src="observability_pipelines/packs/fastly.png" alt="The Fastly pack" style="width:25%;" >}}
9+
810
Fastly CDN logs record client requests, cache states, and delivery performance.
911

1012
What this pack does:

0 commit comments

Comments
 (0)