feat(ingest): add HBase source connector plugin #15257

btkcodedev · 2025-11-10T13:08:10Z

Summary

Adds an HBase metadata ingestion plugin to DataHub to extract namespaces, tables, column families and schema metadata via the HBase Thrift API. Includes backend source implementation, plugin registration, UI config stubs, docs and minimal wiring to surface the connector in the ingestion UI.

Motivation

Support for HBase metadata ingestion so users can index HBase namespaces/tables into DataHub and make them discoverable, searchable and lineage-able.

What changed

Added backend source implementation
Registered plugin entrypoint

The connector is a StatefulIngestionSourceBase and supports:
Containers: HBase namespaces
Schema metadata: row key + column families / qualifiers
Deletion detection via stateful ingestion
Platform instance and env config
Thrift-based connection (Thrift / happybase recommended)

Compatibility / Risk

No breaking changes.
Requires Thrift-related dependencies and (optionally) UI assets.
If Thrift libs missing, connector logs a clear error and fails to connect.

codecov · 2025-11-10T13:40:14Z

Codecov Report

❌ Patch coverage is 7.69231% with 24 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...eb-react/src/app/ingest/source/conf/hbase/hbase.ts	7.69%	12 Missing ⚠️
...-react/src/app/ingestV2/source/conf/hbase/hbase.ts	7.69%	12 Missing ⚠️

📢 Thoughts on this report? Let us know!

codecov · 2025-11-10T19:18:31Z

Bundle Report

Changes will increase total bundle size by 8.36kB (0.03%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	28.64MB	8.36kB (0.03%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
`assets/index-*.js`	3.87kB	19.01MB	0.02%
*`assets/hbaselogo-.png`** (New)	4.49kB	4.49kB	100.0% 🚀

Files in assets/index-*.js:

./src/app/ingestV2/source/builder/constants.ts → Total Size: 5.9kB
./src/app/ingest/source/builder/constants.ts → Total Size: 6.87kB
./src/app/ingest/source/builder/sources.json → Total Size: 35.11kB
./src/app/ingestV2/source/builder/RecipeForm/constants.ts → Total Size: 10.03kB
./src/app/ingestV2/source/builder/sources.json → Total Size: 34.27kB
./src/images/hbaselogo.png → Total Size: 45 bytes
./src/app/ingestV2/source/builder/RecipeForm/hbase.ts → Total Size: 2.67kB

sgomezvillamor · 2025-11-12T08:07:33Z

metadata-ingestion/setup.py

        "glue = datahub.ingestion.source.aws.glue:GlueSource",
        "sagemaker = datahub.ingestion.source.aws.sagemaker:SagemakerSource",
        "hana = datahub.ingestion.source.sql.hana:HanaSource",
+        "hbase = datahub.ingestion.source.sql.hbase:HBaseSource",


additional to the entry point, hbase dependencies should be set here

https://github.com/btkcodedev/datahub/blob/85649af65c0f1e3068d68633b3e983994710eea2/metadata-ingestion/setup.py#L404-L405

sgomezvillamor · 2025-11-12T08:10:13Z

datahub-web-react/src/images/hbaselogo.png

this rectangle shape may not fit well in the UI, have you checked?
a logo that fits better in a square/circle shape would look better

Resized to square

sgomezvillamor

Overall looks good

I miss:

unit/integration tests
some minimal user docs in https://github.com/btkcodedev/datahub/tree/85649af65c0f1e3068d68633b3e983994710eea2/metadata-ingestion/docs/sources

sgomezvillamor · 2025-11-12T13:32:57Z

Please, double check you followed steps defined here https://docs.datahub.com/docs/metadata-ingestion/adding-source Eg I miss updates in constant.ts

btkcodedev · 2025-11-13T06:39:54Z

Changes Made:

Backend:

Added dependencies (happybase, thrift) to setup.py
Fixed linting issues in hbase.py

Frontend:

Added HBase to the sources registry datahub-web-react/src/app/ingest/source/builder/sources.json with the default recipe
Created UI form fields datahub-web-react/src/app/ingestV2/source/builder/RecipeForm/hbase.ts
Updated constants for logo and URN mapping

Documentation:

Resized logo to square
Added setup guide and sample recipe

sgomezvillamor · 2025-11-14T09:52:45Z

metadata-ingestion/src/datahub/ingestion/source/hbase/hbase.py

hbase is not an sql database
so sources should not be in metadata-ingestion/src/datahub/ingestion/source/sql/

may you place them in metadata-ingestion/src/datahub/ingestion/source/hbase/?

Created dedicated src/datahub/ingestion/source/hbase/ directory

sgomezvillamor · 2025-11-14T09:53:55Z

metadata-ingestion/src/datahub/ingestion/source/sql/hbase.py

+            self.connection = Hbase.Client(protocol)
+
+            # Open connection
+            transport.open()


should this be closed?

revamped connection lifecycle and now it's properly managed

sgomezvillamor · 2025-11-14T09:57:33Z

metadata-ingestion/src/datahub/ingestion/source/sql/hbase.py

+    )
+    auth_mechanism: Optional[str] = Field(
+        default=None,
+        description="Authentication mechanism (None, KERBEROS, or custom)",


do we support kerberos in the current implementation?
what does custom mean?

Removed auth_mechanism field

sgomezvillamor · 2025-11-14T10:00:14Z

metadata-ingestion/src/datahub/ingestion/source/sql/hbase.py

+    - Table properties and configuration
+
+    HBase is a distributed, scalable, big data store built on top of Hadoop.
+    This connector uses the HBase Thrift API to extract metadata.


Code uses Thrift1 API but doesn't validate version. HBase supports both Thrift1 and Thrift2 with different APIs:

Thrift1: Older, limited namespace support

Thrift2: Newer, full namespace support
Current implementation works around Thrift1 limitations by parsing table names, but this is fragile.

Wouldn't it be better to go with Thrift2 directly?

also, have you considered happybase? it seems a very popular python lib for consuming hbase and it may save you from setting up thirft

Replaced raw Thrift implementation with happybase library

sgomezvillamor · 2025-11-14T10:01:17Z

metadata-ingestion/src/datahub/ingestion/source/sql/hbase.py

+        default=True,
+        description="Include column families as schema metadata",
+    )
+    max_column_qualifiers: int = Field(


is this being used?
I haven't found code for sampling in this source

Removed max_column_qualifiers field from configuration

sgomezvillamor

Left some comments

Still main blocker is testing:

unit tests for configuration validation
unit tests for namespace/table discovery and schema field generation
integration tests (with docker?)

sgomezvillamor · 2025-11-14T10:21:09Z

datahub-web-react/src/app/ingest/source/builder/constants.ts

v1 frontend was deprecated, so you can skip updating anything in datahub-web-react/src/app/ingest/source/

just keep udpates in datahub-web-react/src/app/ingestV2/

…sm, update deps

btkcodedev · 2025-11-17T03:53:31Z

Moved from sql/ to src/datahub/ingestion/source/hbase/
Switched to happybase library (replaced raw Thrift)
Removed unused max_column_qualifiers config field
Removed unsupported auth_mechanism config field
Added proper connection cleanup in close() method
Added 65+ test cases (20+ config, 35+ unit, 10+ integration)
Updated setup.py dependencies: happybase>=1.2.0

All tests are passing

btkcodedev · 2025-11-17T03:54:14Z

*Integration tests are in development - (facing some issues with docker)

btkcodedev · 2025-11-19T04:53:56Z

Integration tests are passing with mock

Add Hbase plugin

85649af

github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX community-contribution PR or Issue raised by member(s) of DataHub Community labels Nov 10, 2025

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Nov 10, 2025

vercel bot had a problem deploying to Preview November 10, 2025 13:15 Failure

deepgarg760 requested review from sgomezvillamor and treff7es November 11, 2025 16:19

sgomezvillamor reviewed Nov 12, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Nov 12, 2025

sgomezvillamor requested changes Nov 12, 2025

View reviewed changes

chore: add missing frontend components, add docs add deps

fe13472

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Nov 13, 2025

chore: resize logo to square

62adb62

vercel bot had a problem deploying to Preview November 13, 2025 03:20 Failure

btkcodedev requested a review from sgomezvillamor November 13, 2025 11:38

sgomezvillamor reviewed Nov 14, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Nov 14, 2025

sgomezvillamor requested changes Nov 14, 2025

View reviewed changes

sgomezvillamor reviewed Nov 14, 2025

View reviewed changes

fix: move core HBase directory, removed unwanted fields, auth mechani…

7bf79a3

…sm, update deps

vercel bot had a problem deploying to Preview November 17, 2025 04:00 Failure

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Nov 17, 2025

fix: integration test with mock

e0b79f1

btkcodedev requested a review from sgomezvillamor November 19, 2025 04:54

vercel bot had a problem deploying to Preview November 19, 2025 05:01 Failure

Merge branch 'master' into btkcodedev/hbaseplugin

67d5be5

vercel bot had a problem deploying to Preview November 20, 2025 02:49 Failure

Merge branch 'master' into btkcodedev/hbaseplugin

b8043f0

vercel bot had a problem deploying to Preview November 20, 2025 14:16 Failure

feat(ingest): add HBase source connector plugin #15257

Are you sure you want to change the base?

feat(ingest): add HBase source connector plugin #15257

Uh oh!

Conversation

btkcodedev commented Nov 10, 2025

Summary

Motivation

What changed

Compatibility / Risk

Uh oh!

codecov bot commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codecov bot commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle Report

Affected Assets, Files, and Routes:

Assets Changed:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sgomezvillamor left a comment

Choose a reason for hiding this comment

Uh oh!

sgomezvillamor commented Nov 12, 2025

Uh oh!

btkcodedev commented Nov 13, 2025

Changes Made:

Backend:

Frontend:

Documentation:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sgomezvillamor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

btkcodedev commented Nov 17, 2025

Uh oh!

btkcodedev commented Nov 17, 2025

Uh oh!

btkcodedev commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

codecov bot commented Nov 10, 2025 •

edited

Loading

codecov bot commented Nov 10, 2025 •

edited

Loading