Skip to content

Conversation

@kevin-induro
Copy link

Tell us what you do here

  • improving, documenting, or customizing an existing source (please link an issue or describe below)

Short description

The primary motivation is to switch the basic resources (i.e. those created from ALL_OBJECTS with the crm_objects function) from fetching all the Hubspot objects to fetching only those needed based on the incremental load date.

Additional Context

There are still a few different things that can be improved.

  • I noticed that the *_property_history tables have columns for all the properties of the object and null values for all those columns. It should really just have the columns _dlt_id, _dlt_load_id, object_id, property_name, source_id, source_type, timestamp, updated_by_user_id, value
  • I think there's room for improvement on the property history side as well. It'll likely be faster to use the search API to get the recent changes, then pass the IDs from that into the batch API rather than trying to get all of the objects histories every time.
  • I'm not happy with the settings properties. I think things can be generally put together better, and I think I made it worse while trying to follow the previous convention. I also think that there are more settings that can be configurable when declaring the source and refactoring the settings could allow that better.

Copy link
Collaborator

@dat-a-man dat-a-man left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary
Reviewed and confirmed schema-level backward compatibility, table names, columns, and keys remain unchanged.

Behaviorally, the switch to the Search API introduces a few differences:

  • Associations are fetched per record → may increase latency and rate-limit risk for large datasets.
  • 10k record limit → may trigger recursive fetch or full reload when timestamp overlaps occur.
  • Archived records always reload in full → gradual performance slowdown as archive grows.
  • Overall performance becomes variable (faster for small deltas, slower for heavy associations).

Everything else looks solid. Small improvements possible, mainly around improving the documentation to clearly describe these changes and set expectations.

PR looks good to approve.
@anuunchin, could you please have a passing look in case there’s anything else to refine or clarify?

Thanks for the great work @kevin-induro! 🙌

@dat-a-man dat-a-man requested a review from anuunchin November 6, 2025 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants