Skip to content

Commit c50bc84

Browse files
authored
docs(ingestion): Added Google sheets connector support for fivetran (#15294)
1 parent 95ed3ea commit c50bc84

File tree

2 files changed

+86
-1
lines changed

2 files changed

+86
-1
lines changed

metadata-ingestion/docs/sources/fivetran/fivetran_pre.md

Lines changed: 80 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,24 @@ Source and destination are mapped to Dataset as an Input and Output of Connector
2727

2828
## Current limitations
2929

30-
Works only for
30+
### Supported Destinations
31+
32+
Works only for:
3133

3234
- Snowflake destination
3335
- Bigquery destination
3436
- Databricks destination
3537

38+
### Ingestion Limits
39+
40+
To prevent excessive data ingestion, the following limits apply per connector:
41+
42+
- **Sync History**: Maximum of 500 sync runs per connector (controlled by `history_sync_lookback_period`)
43+
- **Table Lineage**: Maximum of 120 table lineage entries per connector
44+
- **Column Lineage**: Maximum of 1000 column lineage entries per connector
45+
46+
When these limits are exceeded, only the most recent entries are ingested. Warnings will be logged during ingestion to notify you when truncation occurs.
47+
3648
## Snowflake destination Configuration Guide
3749

3850
1. If your fivetran platform connector destination is snowflake, you need to provide user details and its role with correct privileges in order to fetch metadata.
@@ -76,6 +88,73 @@ grant role fivetran_datahub to user snowflake_user;
7688
5. [Privileges documentation](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/privileges.html)
7789
5. Check the starter recipe below and replace `workspace_url` and `token` with your information from the previous steps.
7890

91+
## Configuration Options
92+
93+
### Fivetran REST API Configuration
94+
95+
The Fivetran REST API configuration is **required** for Google Sheets connectors and optional for other use cases. It provides access to connection details that aren't available in the Platform Connector logs.
96+
97+
To obtain API credentials:
98+
99+
1. Log in to your Fivetran account
100+
2. Go to **Settings****API Config**
101+
3. Create or use an existing API key and secret
102+
103+
```yaml
104+
api_config:
105+
api_key: "your_api_key"
106+
api_secret: "your_api_secret"
107+
base_url: "https://api.fivetran.com" # Optional, defaults to this
108+
request_timeout_sec: 30 # Optional, defaults to 30 seconds
109+
```
110+
111+
**Note**: If you're using Google Sheets connectors, you must provide `api_config`. Without it, Google Sheets connectors will be skipped with a warning.
112+
113+
## Google Sheets Connector Support
114+
115+
Google Sheets connectors require special handling because Google Sheets is not yet natively supported as a DataHub source. As a workaround, the Fivetran source creates Dataset entities for Google Sheets and includes them in the lineage.
116+
117+
### Requirements
118+
119+
- **Fivetran REST API configuration** (`api_config`) is required for Google Sheets connectors
120+
- The API is used to fetch connection details that aren't available in Platform Connector logs
121+
122+
### What Gets Created
123+
124+
For each Google Sheets connector, two Dataset entities are created:
125+
126+
1. **Google Sheet Dataset**: Represents the entire Google Sheet
127+
128+
- Platform: `google_sheets`
129+
- Subtype: `GOOGLE_SHEETS`
130+
- Contains the sheet ID extracted from the Google Sheets URL
131+
132+
2. **Named Range Dataset**: Represents the specific named range being synced
133+
- Platform: `google_sheets`
134+
- Subtype: `GOOGLE_SHEETS_NAMED_RANGE`
135+
- Contains the named range identifier
136+
- Has upstream lineage to the Google Sheet Dataset
137+
138+
### Limitations
139+
140+
- **Column lineage is disabled** for Google Sheets connectors due to stale metadata issues in the Fivetran Platform Connector (as of October 2025)
141+
- This is a workaround that will be removed once DataHub natively supports Google Sheets as a source
142+
- If the Fivetran API is unavailable or the connector details can't be fetched, the connector will be skipped with a warning
143+
144+
### Example Configuration
145+
146+
```yaml
147+
source:
148+
type: fivetran
149+
config:
150+
# Required for Google Sheets connectors
151+
api_config:
152+
api_key: "your_api_key"
153+
api_secret: "your_api_secret"
154+
155+
# ... other configuration ...
156+
```
157+
79158
## Advanced Configurations
80159

81160
### Working with Platform Instances

metadata-ingestion/docs/sources/fivetran/fivetran_recipe.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,12 @@ source:
4747
# sources_to_database:
4848
# connector_id: source_db
4949

50+
# Optional - Fivetran REST API configuration (required for Google Sheets connectors)
51+
# api_config:
52+
# api_key: "your_api_key"
53+
# api_secret: "your_api_secret"
54+
# base_url: "https://api.fivetran.com" # Optional
55+
# request_timeout_sec: 30 # Optional
5056
# Optional -- This mapping is optional and only required to configure platform-instance for source
5157
# A mapping of Fivetran connector id to data platform instance
5258
# sources_to_platform_instance:

0 commit comments

Comments
 (0)