Installing a Simple Data Pipe Connector

Introduction

The Simple Data Pipe example application includes connectors for a couple cloud data sources. However, additional connectors (e.g., Reddit AMA, SugarCRM) are available as add-ons. If a particular connector cannot be found it is also possible to create custom connectors for specific data sources.

Once the desired connector is found or created, installing it into the Simple Data Pipe is straightforward. This tutorial will step through how to install a connector into a deployment of Simple Data Pipe.

Depending on the Simple Data Pipe environment, there are two options for installation a connector:

• The remote edit approach where updates are made directly in the IBM Bluemix DevOps Services Web IDE then deployed.
• The local edit approach where updates are made locally then pushed and deployed to the IBM Bluemix environment.

Installing a Connector Remotely

To update a Simple Data Pipe deployed in IBM Bluemix remotely. The application would need to have been cloned and available in the Git repository that is part of IBM Bluemix DevOps Services. This is the case, if installation of the Simple Data Pipe was done via the Deploy to Bluemix button. To install a connector in this scenario:

1. Log into IBM Bluemix.

2. Go into the Simple Data Pipe application Overview page.

3. Go into the IBM Bluemix DevOps Services Web IDE.

   • If present, click on EDIT CODE in the Overview page to launch the Web IDE.
   • Else, log into IBM Bluemix DevOps Services and go into the Simple Data Pipe project.

4. Once in IBM Bluemix DevOps Services, click on EDIT CODE to enable live edit.

5. Click on package.json.

6. Edit the package.json to add the connector (or connectors) to the dependencies. list.

Example 1: To add the Reddit connector

"dependencies": {   
   "simple-data-pipe-connector-reddit": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-reddit.git",
   "async": "^1.2.1",
   ...
},

Example 2: To add the Reddit and SugarCRM connectors

"dependencies": {   
    "simple-data-pipe-connector-reddit": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-reddit.git",
    "simple-data-pipe-connector-sugarcrm": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-sugarcrm.git",
    "async": "^1.2.1",
    ...
},

Note: Be sure to follow proper JSON syntax when editing package.json.

7. Save the changes ( > File > Save with the Web IDE).
8. Click on the Deploy the App button (right arrow).

The application will be rebuild and deployed. Once deployed, Simple Data Pipe should now include the new connector.

Installing a Connector From Local

A Simple Data Pipe deployed in IBM Bluemix which does not include a Git repository that is part of IBM Bluemix DevOps Services requires a local clone of the Simple Data Pipe repository. The Cloud Foundry command line interface is also required. To install a connector in this scenario:

From a command line

1. Switch into Simple Data Pipe repository directory: cd simple-data-pipe.

2. Install the connector with npm.

   For example, with a connector in Github:

   npm install https://github.com/ibm-cds-labs/simple-data-pipe-connector-reddit.git
   

   or with a local connector (for local development):

   npm link [connectorPath]/simple-data-pipe-connector-reddit
   

   where [connectorPath] is the path to the connector

   Using npm link adds a symlink in the pipes/nodes_modules directory. This allows for updates to the connector to be immediately available in Simple Data Pipe. Alternatively, npm install can be used to install a copy of the connector in Simple Data Pipe but, when changing the connector npm install will need to be re-run so the update can be picked up.

3. Log into Bluemix with the appropriate [org] and [space]:

   cf login -a https://api.ng.bluemix.net -o [org] -s [space]
   

4. Push the Simple Data Pipe application:

   cf push -f manifest.yml
   

Once completed, the updated Simple Data Pipe should be available within IBM Bluemix with the new connector.

Troubleshooting

Connector does not install

• Check npm install output for errors or warnings
• Confirm connector url is correct
• Confirm the connector's package.json includes the parameter pipes-connector-name

Connector not available in Simple Data Pipes UI

• Check browser for JavaScript errors
• Check development environment console for errors when starting Simple Data Pipe
• Confirm connector JavaScript file name is index.js
• Confirm connector code is present under /pipes/node_modules

Conclusion

Additional Connectors can easily be added as needed to the Simple Data Pipe using either of the above approaches. For the connectors to be properly recognized and made usable within the Simple Data they will need to makes use of the simple-data-pipe-sdk Connector API.

See the Simple Data Pipe GitHub repo, the simple-data-pipe-sdk GitHub repo, the Simple Data Pipe landing page, and the Simple Data Pipe Connector page for additional information and resources.