Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ content/authentication @kowa-c8y @BeateRixen
# Edge
/content/edge* @bbyreddy @rthrippleton @BeateRixen
/static/files/edge* @bbyreddy @rthrippleton @BeateRixen
/content/datahub/running-datahub-on-the-edge* @bbyreddy @rthrippleton @BeateRixen

/content/streaming-analytics/ @BeateRixen @skom-c8y
# 'block-reference-bundle' is automatically and frequently committed to - don't bother _too many_ CODEOWNERS
Expand Down
2 changes: 1 addition & 1 deletion content/change-logs/edge/edge-2025.0.0-datahub.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ version: 2025.0.0
---
{{< product-c8y-iot >}} Edge now supports installing and upgrading the {{< product-c8y-iot >}} DataHub Edge using the Edge operator.

For more details, see [Setting up DataHub Edge](/datahub/running-datahub-on-the-edge/#setting-up-datahub-edge).
For more details, see [DataHub](/edge-kubernetes/datahub).
2 changes: 1 addition & 1 deletion content/change-logs/edge/edge-2025.0.8-CIE-5487.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ version: 2025.0.8
---
In previous versions of {{< product-c8y-iot >}} Edge, a race condition caused the Dremio component of DataHub to start up without authentication being configured, breaking almost all aspects of DataHub. This issue has now been fixed.

If you have a system that is suffering from this problem but otherwise functional, you do not need to do a fresh installation of Edge. Instead, disable DataHub in your current Edge installation, and then upgrade to the latest version of Edge. After the upgrade, re-enable DataHub, and the problem will be cleared up. See [Upgrading Edge](/edge-kubernetes/manage-edge/#version-upgrade) for instructions on upgrading. For disabling and enabling DataHub, see [More general configuration changes](edge-kubernetes/manage-edge/#more-general-configuration-changes). The value you will be changing in the Custom Resource looks like this:
If you have a system that is suffering from this problem but otherwise functional, you do not need to do a fresh installation of Edge. Instead, disable DataHub in your current Edge installation, and then upgrade to the latest version of Edge. After the upgrade, re-enable DataHub, and the problem will be cleared up. See [Upgrading Edge](/edge-kubernetes/manage-edge/#version-upgrade) for instructions on upgrading. For disabling and enabling DataHub, see [More general configuration changes](/edge-kubernetes/manage-edge/#more-general-configuration-changes). The value you will be changing in the Custom Resource looks like this:
```
dataHub:
enabled: true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ For your convenience, here is an overview of the contents of this document:
| [Setting up {{< product-c8y-iot >}} DataHub](/datahub/setting-up-datahub) | Set up {{< product-c8y-iot >}} DataHub and its components |
| [Working with {{< product-c8y-iot >}} DataHub](/datahub/working-with-datahub) | Manage offloading pipelines and query the offloaded results |
| [Operating {{< product-c8y-iot >}} DataHub](/datahub/operating-datahub) | Run administrative tasks |
| [Running {{< product-c8y-iot >}} DataHub on {{< product-c8y-iot >}} Edge](/datahub/running-datahub-on-the-edge) | Run the Edge edition of {{< product-c8y-iot >}} DataHub |
| [Integrating {{< product-c8y-iot >}} DataHub with other products](/datahub/integrating-datahub-with-other-products) | Learn how to integrate {{< product-c8y-iot >}} DataHub with other products |

The [change log](/change-logs/?component=.component-datahub) provides an overview on features, changes, and other relevant information.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

9 changes: 0 additions & 9 deletions content/datahub/running-datahub-on-the-edge.md

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -98,11 +98,7 @@ Value: Your key name, for example, `arn:aws:kms:eu-west-2:123456789012:key/071a8
**SSE-C**: The client specifies an base64-encoded AES-256 key to be used to encrypt and decrypt the data. **{{< product-c8y-iot >}} DataHub does not support this option.**

##### NAS {#nas}
**NAS** is a storage system mounted (NFS, SMB) directly into the Dremio cluster. It is only available for {{< product-c8y-iot >}} Edge installations. The following settings must be defined for this data lake:

|Settings|Description|
|:---|:---|
|Mount path|The mount path refers to a path in the local Linux file system on both the coordinator and executor containers. By default, the file system of {{< product-c8y-iot >}} Edge is mounted into /datalake inside the containers. To use some other folder, you must map the folder into both containers, for example, to /datalake inside the containers.|
**NAS** is a storage system mounted (NFS, SMB) directly into the Dremio cluster. It is only available on {{< product-c8y-iot >}} Edge installations.

#### Saving settings {#saving-settings}
Once all settings are defined, click **Save** in the action bar to the right. During the save process, the following steps are automatically conducted:
Expand Down
31 changes: 31 additions & 0 deletions content/edge-kubernetes/datahub.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
weight: 80
title: DataHub
layout: bundle
sector:
- edge_server
---

{{< product-c8y-iot >}} DataHub on Edge offers the same functionality as a cloud installation of {{< product-c8y-iot >}} DataHub, and is an optional component of Edge. The significant difference is that processes and data are entirely local to your network, rather than in the cloud. You can define offloading pipelines, which regularly move data from the Operational Store of {{< product-c8y-iot >}} into a data lake. In the Edge setup, a NAS or local disk is used as data lake. Dremio, the internal engine of {{< product-c8y-iot >}} DataHub, can access the data lake and run analytical queries against its contents, using SQL as the query interface.

To learn more about DataHub in general, see [DataHub overview](/datahub/datahub-overview). As an end user, DataHub on Edge appears and behaves much the same as DataHub in a cloud installation, subject to the limitations in the comparison table later in this section.

### Installing and using DataHub {#installing-and-using-datahub}

DataHub is an optional component of Edge, and can be enabled by updating the `spec.dataHub` field in the Edge custom resource (CR). For more details on the `spec.messagingService` field, refer to [Edge custom resource - DataHub](/edge-kubernetes/edge-custom-resource-definition/#k8-edge-datahub). For general guidance on configuring Edge, see the [Install Edge](/edge-kubernetes/installing-edge-on-k8/) and [Modify Edge](/edge-kubernetes/manage-edge/#modify-edge) sections.

The data lake and related storage will always be written to the host file system under the path `/datahub`, whatever is mounted there. You are expected to have a single shared NAS file system, such as NFS mounted at that path _on all nodes of the Kubernetes cluster that Edge is running on_. This is to ensure the resilience of your data lake contents.

In order to access Dremio, you must also make the domain `datahub-<domain_name>` resolvable, just as the configured domain name and `management-<domain_name>` were made resolvable in [Accessing Edge](/edge-kubernetes/installing-edge-on-k8/#accessing-edge).

### Comparison between DataHub Edge and DataHub Cloud {#comparison-between-datahub-edge-and-datahub-cloud}

| Area | {{< product-c8y-iot >}} DataHub Edge | {{< product-c8y-iot >}} DataHub Cloud |
| ----- | ----- | ----- | ----- |
| High availability | Depending on any underlying virtualization technology | Depending on the cloud deployment setup |
| Vertical scalability | Yes | Yes |
| Horizontal scalability | No | Yes |
| Upgrades with no downtime | No | No |
| Installation | Offline & Online | Online |
| Dremio cluster setup | 1 master, 1 executor | Minimum 1 master, 1 executor |
| Data lakes | NAS or local disk | Azure Storage, S3, (NAS) |
Loading