You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Porter is the PHP data importer. She fetches data from anywhere, from the local file system to third party online services, and returns an [iterator](#record-collections). Porter is a fully pluggable import framework that can be extended with [connectors](#connectors)for any protocol and [transformers](#transformers) to manipulate data immediately after import.
11
+
### Scalable and durable data imports for publishing and consuming APIs
12
12
13
-
Ready-to-use data [providers][Provider] include all the necessary connectors and other dependencies to access popular online services such as [Stripe][Stripe provider] for online payments, the [European Central Bank][ECB provider]for foreign exchange rates or [Steam][Steam provider] for its complete PC games library and more. Porter's provider library is limited right now, and some implementations are incomplete, but we hope the PHP community will rally around Porter's abstractions and become the de facto framework for publishing online services, APIs, web scrapers and data dumps. Porter's interfaces have undergone intensive scrutiny and several iterations during years of production use to ensure they are efficient, robust, flexible, testable and easy to implement.
13
+
Porter is the all-purpose PHP data importer. She fetches data from anywhere and serves it as a [record collection](#record-collections)for iterable data sets, to encourage processing one record at a time instead of loading entire data sets into memory at once. Her [durability](#durability) feature provides automatic and transparent recovery from intermittent network connectivity problems by default.
14
14
15
-
Porter's key [durability](#durability) feature ensures recoverable connection failures are transparently retried up to five times by default, with increasing delays between each attempt until the fetch is successful. This helps ensure intermittent network failures will not disrupt the entire import operation. Special care has been taken to ensure Porter's features are safe for concurrency, such that multiple imports can be paused and resumed simultaneously, which is especially important for iterators implemented with generators (which can be paused) as well as the upcoming asynchronous imports in v5.
15
+
Porter's interface trichotomy of [providers](#providers), [resources](#resources) and [connectors](#connectors) maps well to APIs. A typical API, for example GitHub, would define the Provider as GitHub, a resource as `GetUser` or `ListRepositories`and the Connector could be `HttpConnector`.
16
16
17
-
###### Quick links
17
+
Porter provides a dual API for synchronous and [asynchronous](#asynchronous) imports, both of which are concurrency safe, so multiple imports can be paused and resumed simultaneously. Asynchronous mode allows large scale imports across multiple connections to work at maximum efficiency without waiting for each network call to complete.
*Formally defines a structured data import framework with the following concepts: [providers](#providers) represent one or more [resources](#resources) that fetch data from [connectors](#connectors).
52
-
* Provides efficient in-memory data processing interfaces to handle large data sets one record at a time, via iterators, which can be implemented using generators.
53
-
*Offers post-import [transformations](#transformers), such as [filtering](#filtering) and [mapping][MappingTransformer], to transform third-party into data useful for first-party applications.
54
+
*Defines a formal structure for data import APIs: [providers](#providers) represent one or more [resources](#resources) that fetch data from [connectors](#connectors).
55
+
* Provides efficient data processing interfaces to handle large data sets one record at a time, via iterators, which can be implemented using generators.
56
+
*[Asynchronous](#asynchronous) imports offer highly efficient CPU-bound data processing for large scale imports across multiple connections concurrently.
54
57
* Protects against intermittent network failures with [durability](#durability) features.
58
+
* Offers post-import [transformations](#transformers), such as [filtering](#filtering) and [mapping][MappingTransformer], to transform third-party into data useful for first-party applications.
55
59
* Supports PSR-6 [caching](#caching), at the connector level, for each fetch operation.
56
60
* Joins two or more linked data sets together using [sub-imports][Sub-imports] automatically.
57
61
58
62
Quick start
59
63
-----------
60
64
61
-
To get started quickly, try our [quick start guide][Quickstart]. For a more thorough introduction, continue reading this document.
65
+
To get started quickly, try our [quick start guide][Quickstart]. For a more thorough introduction continue reading.
62
66
63
67
Understanding this manual
64
68
-------------------------
65
69
66
-
The first half of this manual covers Porter's main features and how to use them. The second half covers architecture, interface and implementation details for Porter developers. There's an intermission inbetween so you'll know where the division is!
70
+
The first half of this manual covers Porter's main API for consuming data services. The second half covers architecture, interface and implementation details for publishing data services. There's an intermission in-between so you'll know where the separation is!
67
71
68
72
Text marked as `inline code` denotes literal code, as it would appear in a PHP file. For example, `Porter` refers specifically to the class of the same name within this library, whereas *Porter* refers to this entire project as a whole.
69
73
@@ -74,7 +78,7 @@ Usage
74
78
75
79
Create a `new Porter` instance—we'll usually only need one per application. Porter's constructor requires a [PSR-11][PSR-11] compatible `ContainerInterface` that acts as a repository of [providers](#providers).
76
80
77
-
When integrating Porter into a typical MVC framework application, we'll usually have a service locator or DI container implementing this interface already. We can simply inject the entire container into Porter. Although it's probably safer to create a separate container just for Porter's providers, it usually doesn't matter.
81
+
When integrating Porter into a typical MVC framework application, we'll usually have a service locator or DI container implementing this interface already. We can simply inject the entire container into Porter, although it's best practice to create a separate container just for Porter's providers.
78
82
79
83
Without a framework, pick any [PSR-11 compatible library][PSR-11 search] and inject an instance of its container class. We could even write our own container since the interface is easy to implement, but using an existing library is beneficial, particularly since most support lazy-loading of services. If you're not sure which to use, [Joomla DI](https://github.com/joomla-framework/di) is fairly lightweight and straightforward.
80
84
@@ -126,7 +130,7 @@ The following data flow diagram gives a high level overview of Porter's main int
126
130
127
131
</div>
128
132
129
-
Our application calls `Porter::import()` with an `ImportSpecification` and receives `PorterRecords`in return. Everything else happens internally so we don't need to worry about it unless writing custom providers, resources or connectors.
133
+
Our application calls `Porter::import()` with an `ImportSpecification` and receives `PorterRecords`back. Everything else happens internally so we don't need to worry about it unless writing custom providers, resources or connectors.
130
134
131
135
Import specifications
132
136
---------------------
@@ -150,7 +154,7 @@ Record collections are `Iterator`s, guaranteeing imported data is enumerable usi
150
154
151
155
### Details
152
156
153
-
Record collections may be `Countable`, depending on whether the imported data was countable and whether any destructive operations were performed after import. Filtering is a destructive operation since it may remove records and therefore the count reported by a `ProviderResource` would no longer be accurate. It is the responsibility of the resource to supply the number of records in its collection by returning an iterator that implements `Countable`, such as `ArrayIterator` or `CountableProviderRecords`. When a countable iterator is detected, Porter returns `CountablePorterRecords` provided no destructive operations were performed.
157
+
Record collections may be `Countable`, depending on whether the imported data was countable and whether any destructive operations were performed after import. Filtering is a destructive operation since it may remove records and therefore the count reported by a `ProviderResource` would no longer be accurate. It is the responsibility of the resource to supply the total number of records in its collection by returning an iterator that implements `Countable`, such as `ArrayIterator`, or more commonly, `CountableProviderRecords`. When a countable iterator is used, Porter returns `CountablePorterRecords`, provided no destructive operations were performed.
154
158
155
159
Record collections are composed by Porter using the decorator pattern. If provider data is not modified, `PorterRecords` will decorate the `ProviderRecords` returned from a `ProviderResource`. That is, `PorterRecords` has a pointer back to the previous collection, which could be written as: `PorterRecords` → `ProviderRecords`. If a [filter](#filtering) was applied, the collection stack would be `PorterRecords` → `FilteredRecords` → `ProviderRecords`. Normally this is an unimportant detail but can sometimes be useful for debugging.
156
160
@@ -162,6 +166,26 @@ Since record collections are just objects, it is possible to define derived type
162
166
163
167
The result of a successful `Porter::import` call is always an instance of `PorterRecords` or `CountablePorterRecords`, depending on whether the number of records is known. If we need to access methods of the original collection, returned by the provider, we can call `findFirstCollection()` on the collection. For an example, see [CurrencyRecords][CurrencyRecords] of the [European Central Bank Provider][ECB] and its associated [test case][ECB test].
164
168
169
+
Asynchronous
170
+
------------
171
+
172
+
The new asynchronous API, introduced in version 5, is built on top of the fully programmable asynchronous framework, [Amp]. The synchronous API is not compatible with the asynchronous API so one must decide which to use. In general, the asynchronous API should be preferred for new projects because async can do everything sync can do, including emulating synchronous behaviour, but sync code cannot behave asynchronously without significant refactoring.
173
+
174
+
We must be inside the async event loop to begin programming asynchronously. Let's illustrate how to rewrite the [earlier example](#importing-data) asynchronously.
0 commit comments