Skip to content
Closed

Dev #223

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
577f696
Add feedback widget and modal for user feedback submission (#193)
Monil-KTX Oct 2, 2025
5b186e4
Feat/integration of news letter and feedback menu (#194)
ktx-krupa Oct 3, 2025
fb34251
Feature/grafana plugin integration (#197)
ktx-krupa Oct 3, 2025
2ca2e8d
feat: integrate reCAPTCHA for newsletter subscription form (#198)
ktx-krupa Oct 3, 2025
cb8b219
update regex-pattern.md (#185)
DebashisBorgohainO2 Oct 6, 2025
be05bd6
Fix/resolve feedback menu related issue (#200)
Monil-KTX Oct 6, 2025
d1072a4
fix: enhance newsletter form with submit button and reset functionali…
Monil-KTX Oct 6, 2025
2f8f603
updated sensitive-data-redaction.md (#203)
DebashisBorgohainO2 Oct 7, 2025
32cc254
Update documentation under /management to reflect latest UI changes (…
DebashisBorgohainO2 Oct 8, 2025
a4b7e68
Fix/make theme related changes (#207)
Monil-KTX Oct 10, 2025
173cd03
fix: refactor header layout and improve mobile menu functionality (#208)
Monil-KTX Oct 10, 2025
10d7c3a
Fix/change width related css for md content (#209)
Monil-KTX Oct 10, 2025
bfdf0a5
update traces documentation (#210)
DebashisBorgohainO2 Oct 10, 2025
29e1853
update the index page under /traces
DebashisBorgohainO2 Oct 10, 2025
3771ac3
Revert "update the index page under /traces"
DebashisBorgohainO2 Oct 10, 2025
bb54a2e
Traces updates (#211)
DebashisBorgohainO2 Oct 10, 2025
d2e4779
fix: update secondary sidebar styles for consistent height and scroll…
Monil-KTX Oct 10, 2025
b9b91ad
fix: remove empty versions.json file (#214)
Monil-KTX Oct 13, 2025
b629bd2
Create federated search documentation (#216)
DebashisBorgohainO2 Oct 14, 2025
d88c5f2
fix: comment out version provider in mkdocs configuration
Monil-KTX Oct 13, 2025
daa6405
fix: add header-list-style class for consistent list styling in heade…
Monil-KTX Oct 15, 2025
c1cc949
fix: update header-list-style for consistent spacing in lists
Monil-KTX Oct 15, 2025
5868337
Merge branch 'dev' into fix/v3/make-css-related-changes-for-header-an…
Monil-KTX Oct 15, 2025
870d0f3
Merge pull request #220 from openobserve/fix/v3/make-css-related-chan…
ktx-abhishek Oct 15, 2025
e1ada76
Update deploy-docs-staging.yaml
ktx-abhishek Oct 15, 2025
d4c5a40
Merge pull request #221 from openobserve/fix/s3-sync
ktx-abhishek Oct 15, 2025
77dc50d
Create federated search documentation (#216)
DebashisBorgohainO2 Oct 14, 2025
c0c84ee
fix: update header-list-style for consistent spacing in lists
Monil-KTX Oct 15, 2025
9276b59
fix: add header-list-style class for consistent list styling in heade…
Monil-KTX Oct 15, 2025
3e02a8f
Update deploy-docs-staging.yaml
ktx-abhishek Oct 15, 2025
22b9821
Update enterprise feature availability markings across documentation …
DebashisBorgohainO2 Oct 15, 2025
5bcfa72
Merge branch 'dev' into update-docs
DebashisBorgohainO2 Oct 15, 2025
5e4d485
Merge pull request #226 from openobserve/update-docs
ktx-abhishek Oct 15, 2025
fd1c449
docs: Add Pipeline Remote Destinations guide (#229)
DebashisBorgohainO2 Oct 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/deploy-docs-staging.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ jobs:
- name: Deploy to S3
run: |
aws s3 sync ./site s3://openobserve-website-staging/docs --exclude=".git/*"
aws s3 sync ./site s3://openobserve-website-staging/docs --exclude=".git/*" --delete
- name: Invalidate CloudFront cache
run: |
aws cloudfront create-invalidation --distribution-id E2GZJM0TJIDFRM --paths "/docs/*"
aws cloudfront create-invalidation --distribution-id E2GZJM0TJIDFRM --paths "/docs/*"
Binary file added docs/images/current-cluster-query-result.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/current-cluster-query.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/federated-search-multi-select.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/federated-search-result.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/federated-search.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/user-guide/.pages
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ nav:
- Management: management
- Profile: profile
- Performance: performance
- Federated Search: federated-search
- Best Practices: best-practices
- Migration: migration

Expand Down
5 changes: 5 additions & 0 deletions docs/user-guide/federated-search/.pages
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
nav:

- Federated Search Overview: index.md
- How to Use Federated Search: how-to-use-federated-search.md
- Federated Search Architecture: federated-search-architecture.md
146 changes: 146 additions & 0 deletions docs/user-guide/federated-search/federated-search-architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
---
title: Federated Search in OpenObserve - Architecture
description: Technical explanation of OpenObserve deployment modes, normal cluster query execution, and how federated search works across single and multiple clusters.
---
This document explains the technical architecture of OpenObserve deployments, how queries execute in normal clusters, and how [federated search](../) coordinates queries across clusters in a supercluster.

> This feature is available in Enterprise Edition.

## Understanding OpenObserve deployments
Before diving into how federated search works, you need to understand how OpenObserve can be deployed. OpenObserve scales from a single machine to a globally distributed infrastructure.

## Single node deployment
The simplest deployment: one instance of OpenObserve runs all functions on one machine. Data stores locally, and the node processes queries directly. This works for testing or small deployments.

## Single cluster deployment
When you need scale, multiple specialized nodes work together as a cluster. Each node type has a specific role:

- **Router**: Entry point that forwards queries to queriers
- **Querier**: Processes queries in parallel with other queriers
- **Ingester**: Receives and stores data in object storage
- **Compactor**: Optimizes files and enforces retention
- **Alertmanager**: Executes alerts and sends notifications

A single cluster handles more data and provides higher availability than a single node.

## Supercluster deployment
When you need to operate across multiple geographical regions, multiple clusters connect as a supercluster. This is where federated search becomes relevant.

!!! note "Key point"
Each cluster in a supercluster operates independently with its own data storage. Data ingested into one cluster stays in that cluster. However, configuration metadata synchronizes across all clusters, allowing unified management.

## Region and cluster hierarchy
In a supercluster, regions organize clusters geographically. A region may contain one or more clusters.
<br>
**Example:**
<br>

```bash
Region: us-test-3
├─ Cluster: dev3
└─ Cluster: dev3-backup

Region: us-test-4
└─ Cluster: dev4
```
Each cluster has independent data storage. Data stays where it was ingested.

## How queries execute
Understanding query execution helps you understand how federated search works whether querying one cluster or multiple clusters.

### Normal cluster query execution
This section explains how any OpenObserve cluster processes queries internally, regardless of whether it is a standalone cluster or part of a supercluster. Understanding this internal process is essential because:

- This is how standalone clusters work
- This is what happens when you query your current cluster in a supercluster without federated search coordination
- During federated search, each individual cluster uses this same internal process to search its own data

When a cluster receives a query:

1. Router forwards the query to an available querier.
2. That querier becomes the leader querier.
3. Leader querier parses SQL, identifies data files, creates execution plan.
4. Leader querier distributes work among available queriers. These queriers become worker queriers.
5. All worker queriers search their assigned files in parallel.
6. Worker queriers send results to the leader querier.
7. Leader querier merges results and returns final answer.

### Query execution for your current cluster in a supercluster
Your current cluster is the cluster you are logged into. When you select your current cluster from the Region dropdown, this is not federated search.
<br>
For example, if you are logged into Cluster A and you select Cluster A from the Region dropdown, the query executes using the normal cluster query execution process described above. No cross-cluster communication occurs, and no federated search coordination is needed.

### Federated search for one different cluster in a supercluster
When you select a different cluster from the Region dropdown, not the cluster you are logged into, federated search coordination is used:
<br>

**Step 1: Coordination setup**
<br>
Your current cluster becomes the leader cluster.
<br>

**Step 2: Query distribution**
<br>
Leader cluster sends the query to the selected cluster via gRPC.
<br>

**Step 3: Query processing**
<br>
The selected cluster processes the query using its normal cluster query execution process.
<br>

**Step 4: Result return**
<br>
The selected cluster sends its results back to the leader cluster.
<br>

**Step 5: Result presentation**
<br>
The leader cluster displays the results.

### Federated search for multiple clusters in a supercluster

When you select no cluster or multiple clusters from the Region dropdown, federated search extends the query across all selected clusters:
<br>

**Step 1: Coordination setup**
<br>
Your current cluster becomes the leader cluster. The leader cluster identifies all selected clusters, or all clusters if none selected, that contain data for the queried stream. These other clusters become worker clusters.
<br>

**Step 2: Query distribution**
<br>
The leader cluster sends the query to all worker clusters via gRPC. All clusters now have the same query to execute.
<br>

**Step 3: Parallel processing**
<br>
Each cluster processes the query using its normal cluster query execution process. The leader cluster searches its own data if it contains data for that stream. Worker clusters search their own data. All processing happens simultaneously.
<br>

**Step 4: Result aggregation**
<br>
Each cluster aggregates its own results internally using its leader querier and worker queriers. Worker clusters send their aggregated results to the leader cluster. The leader cluster merges all results from all clusters and returns the unified response.

## Metadata synchronization
In a supercluster, clusters share configuration and schema information in real-time while keeping actual data separate. This synchronization happens via NATS, a messaging system that coordinates communication between clusters.
<br>
While stream schemas are synchronized across all clusters in real-time, the actual data for a stream only exists in the cluster or clusters where it was ingested.

| **Synchronized across clusters** | **NOT synchronized (stays local)** |
|----------------------------------|-----------------------------------|
| Schema definitions | Log data |
| User-defined functions | Metric data |
| Dashboards and folders | Trace data |
| Alerts and notifications | Raw ingested data |
| Scheduled tasks and reports | Parquet files and WAL files |
| User and organization settings | Search indices |
| System configurations | |
| Job metadata | |
| Enrichment metadata | |

This design maintains data residency compliance while enabling unified configuration management.

## Limitations

**No cluster identification in results:** Query results do not indicate which cluster provided specific data. To identify the source, query each cluster individually.
76 changes: 76 additions & 0 deletions docs/user-guide/federated-search/how-to-use-federated-search.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
title: Federated Search in OpenObserve - How-to Guide
description: Step-by-step instructions for querying your current cluster and performing federated searches across one or more clusters in a supercluster setup.
---
This document explains how to query your current cluster and how to perform [federated searches](../) across one or more different clusters in a supercluster setup.
> This feature is available in Enterprise Edition.

## How to query your current cluster in a supercluster

Query your current cluster when you know the data is in your cluster or when you need the fastest query performance.

!!! note "What you need to know:"

- This is not federated search
- You are querying the current cluster.
- No cross-cluster communication occurs.
- Results will include data from the current cluster only.
<br>
**Steps:**
![current-cluster-query](current-cluster-query.png)

1. Navigate to the **Logs** page.
2. Enter your query in the SQL Query Editor.
3. Select a time range.
4. Select one specific cluster from the **Region** dropdown.
5. Select **Run query**.

> For detailed explanation, see **Normal cluster query execution** in the [Federated Search Architecture](../federated-search/federated-search-architecture/) page.
<br>

**Result**<br>
Data from the selected cluster only.
![current-cluster-query-result](current-cluster-query-result.png)


## How to query one or more different clusters in a supercluster

Use federated search when you need data from multiple clusters.

!!! note "What you need to know"

- Multiple clusters will process your query simultaneously.
- Results will combine data from all selected clusters.

**Steps**
<br>
![federated-search](federated-search.png)

1. Navigate to the **Logs** page.
2. Enter your query in the SQL Query Editor.
3. Select a time range.
4. Leave the **Region** dropdown unselected, or select multiple clusters.
5. Select **Run query**.

> For detailed explanation, see **Federated search for one different cluster** and **Federated search for multiple clusters** in the [Federated Search Architecture](../federated-search-architecture/) page.
<br>

**Result**<br>
Combined data from all selected clusters.
![federated-search-result](federated-search-result.png)
## Region selection reference

Use this quick reference to understand how region selection affects query execution:

| **Region/Cluster Selection** | **Behavior** | **Query Type** | **Communication** |
|------------------------------|--------------|----------------|-------------------|
| None selected | Queries all clusters | Federated search | Cross-cluster via gRPC |
| Your current cluster selected | Queries only your current cluster | Normal cluster query (NOT federated) | Internal only, no cross-cluster |
| One different cluster selected (same region) | Queries only that cluster | Federated search | Cross-cluster via gRPC |
| One different cluster selected (different region) | Queries only that cluster | Federated search | Cross-cluster via gRPC |
| Multiple clusters selected | Queries all selected clusters | Federated search | Cross-cluster via gRPC |


**Next step**

- [Federated Search Architecture](../federated-search-architecture/)
64 changes: 64 additions & 0 deletions docs/user-guide/federated-search/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
title: Federated Search in OpenObserve - Overview
description: Learn what federated search is, key concepts, prerequisites, and when to use it.
---
This document provides an overview of federated search in OpenObserve.

> This feature is available in Enterprise Edition.

## What is federated search?

Federated search enables querying across multiple OpenObserve clusters that are connected as a supercluster, all from one interface.
<br>

Without federated search, investigating issues across regions requires logging into each cluster separately, running the same query multiple times, and manually combining results. This wastes time during critical incidents.
With federated search, you query once and receive unified results from all clusters.

!!! note "Prerequisites"

- OpenObserve Enterprise edition
- Multiple clusters configured as a supercluster

## How to verify if your environment is in a supercluster
Check whether the Region dropdown appears on the Logs page. If visible, your clusters are configured as a supercluster.
![federated-search](../../images/federated-search.png)

## Key concepts in federated search

Before using federated search, understand these core concepts:

- **Node:** A single instance of OpenObserve running on one machine or server.
- **Cluster:** A group of OpenObserve nodes working together to handle data ingestion, storage, and querying. Each cluster has its own data storage.
- **Region:** A geographical location that contains one or more clusters. For example, Region us-east may contain cluster prod-east-1 and cluster prod-east-2.
- **Supercluster:** Multiple OpenObserve clusters across different geographical regions connected to work as a unified system. This enables federated search capability.
- **Data distribution:** Data ingested into a specific cluster stays in that cluster's storage. It is not replicated to other clusters. This ensures data residency compliance.
- **Metadata synchronization:** Configuration information such as schemas, dashboards, and alerts synchronize across all clusters in a supercluster. This allows unified management while keeping data distributed.
- **Federated search:** The capability to query data across different clusters in a supercluster. Federated search activates when you:

- Select one or more different clusters, meaning clusters other than your current cluster: The selected clusters' data is searched via federated coordination.
- Select none: All clusters search simultaneously via federated coordination and results are combined.

> **Important**: Querying your current cluster uses normal cluster query execution, not federated search architecture.

> For detailed technical explanations of deployment modes, architecture, and how queries execute, see the [Federated Search Architecture](../federated-search-architecture/) page.

## When to use federated search

| **Use case** | **Cluster selection** | **Reason** |
|--------------|----------------------|------------|
| Data is in one specific different cluster | Select that different cluster | Access only that cluster's data via federated search |
| Multi-region deployments | Select none or multiple clusters | Query all regions at once via federated search |
| Centralized search across teams | Select none or multiple clusters | Unified visibility across all clusters via federated search |


## When not to use federated search

| **Use case** | **Cluster selection** | **Reason** |
|--------------|----------------------|------------|
| Data is in your current cluster | Select your current cluster | Uses normal cluster query without cross-cluster communication |


**Next steps**

- [How to Use Federated Search](../how-to-use-federated-search/)
- [Federated Search Architecture](../federated-search-architecture/)
4 changes: 4 additions & 0 deletions input.css
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,7 @@ main {
-webkit-text-fill-color: transparent;
background-clip: text;
}

.header-list-style {
list-style: none;
}
8 changes: 8 additions & 0 deletions overrides/css/output.css
Original file line number Diff line number Diff line change
Expand Up @@ -944,6 +944,10 @@ video {
gap: 1.5rem;
}

.tw-gap-y-2 {
row-gap: 0.5rem;
}

.tw-space-x-0\.5 > :not([hidden]) ~ :not([hidden]) {
--tw-space-x-reverse: 0;
margin-right: calc(0.125rem * var(--tw-space-x-reverse));
Expand Down Expand Up @@ -1555,6 +1559,10 @@ main {
background-clip: text;
}

.header-list-style {
list-style: none;
}

.hover\:tw-border-gray-200:hover {
--tw-border-opacity: 1;
border-color: rgb(229 231 235 / var(--tw-border-opacity, 1));
Expand Down
Loading
Loading