You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: metadata-ingestion/docs/sources/fivetran/fivetran_pre.md
+80-1Lines changed: 80 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,12 +27,24 @@ Source and destination are mapped to Dataset as an Input and Output of Connector
27
27
28
28
## Current limitations
29
29
30
-
Works only for
30
+
### Supported Destinations
31
+
32
+
Works only for:
31
33
32
34
- Snowflake destination
33
35
- Bigquery destination
34
36
- Databricks destination
35
37
38
+
### Ingestion Limits
39
+
40
+
To prevent excessive data ingestion, the following limits apply per connector:
41
+
42
+
-**Sync History**: Maximum of 500 sync runs per connector (controlled by `history_sync_lookback_period`)
43
+
-**Table Lineage**: Maximum of 120 table lineage entries per connector
44
+
-**Column Lineage**: Maximum of 1000 column lineage entries per connector
45
+
46
+
When these limits are exceeded, only the most recent entries are ingested. Warnings will be logged during ingestion to notify you when truncation occurs.
47
+
36
48
## Snowflake destination Configuration Guide
37
49
38
50
1. If your fivetran platform connector destination is snowflake, you need to provide user details and its role with correct privileges in order to fetch metadata.
@@ -76,6 +88,73 @@ grant role fivetran_datahub to user snowflake_user;
5. Check the starter recipe below and replace `workspace_url` and `token` with your information from the previous steps.
78
90
91
+
## Configuration Options
92
+
93
+
### Fivetran REST API Configuration
94
+
95
+
The Fivetran REST API configuration is **required** for Google Sheets connectors and optional for other use cases. It provides access to connection details that aren't available in the Platform Connector logs.
96
+
97
+
To obtain API credentials:
98
+
99
+
1. Log in to your Fivetran account
100
+
2. Go to **Settings** → **API Config**
101
+
3. Create or use an existing API key and secret
102
+
103
+
```yaml
104
+
api_config:
105
+
api_key: "your_api_key"
106
+
api_secret: "your_api_secret"
107
+
base_url: "https://api.fivetran.com"# Optional, defaults to this
108
+
request_timeout_sec: 30# Optional, defaults to 30 seconds
109
+
```
110
+
111
+
**Note**: If you're using Google Sheets connectors, you must provide `api_config`. Without it, Google Sheets connectors will be skipped with a warning.
112
+
113
+
## Google Sheets Connector Support
114
+
115
+
Google Sheets connectors require special handling because Google Sheets is not yet natively supported as a DataHub source. As a workaround, the Fivetran source creates Dataset entities for Google Sheets and includes them in the lineage.
116
+
117
+
### Requirements
118
+
119
+
- **Fivetran REST API configuration** (`api_config`) is required for Google Sheets connectors
120
+
- The API is used to fetch connection details that aren't available in Platform Connector logs
121
+
122
+
### What Gets Created
123
+
124
+
For each Google Sheets connector, two Dataset entities are created:
125
+
126
+
1. **Google Sheet Dataset**: Represents the entire Google Sheet
127
+
128
+
- Platform: `google_sheets`
129
+
- Subtype: `GOOGLE_SHEETS`
130
+
- Contains the sheet ID extracted from the Google Sheets URL
131
+
132
+
2. **Named Range Dataset**: Represents the specific named range being synced
133
+
- Platform: `google_sheets`
134
+
- Subtype: `GOOGLE_SHEETS_NAMED_RANGE`
135
+
- Contains the named range identifier
136
+
- Has upstream lineage to the Google Sheet Dataset
137
+
138
+
### Limitations
139
+
140
+
- **Column lineage is disabled** for Google Sheets connectors due to stale metadata issues in the Fivetran Platform Connector (as of October 2025)
141
+
- This is a workaround that will be removed once DataHub natively supports Google Sheets as a source
142
+
- If the Fivetran API is unavailable or the connector details can't be fetched, the connector will be skipped with a warning
0 commit comments