Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Contributing to Streamz

## Streamz Conda Environment
For the convenience of the community streamz offers an environment file for contributors to create conda environments. A few basic quick start commands for creating a using a streamz conda environment are found below.

### Creating Conda Development Environment
Creating the streamz conda environment can be achieved by simply running ```conda env create -f ./conda/environments/streamz_dev.yml```

### Using Conda Environment
The streamz conda environment can be activated by running ```conda activate streamz_dev```
20 changes: 20 additions & 0 deletions conda/environments/streamz_dev.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: streamz_dev
channels:
- conda-forge
- defaults
dependencies:
- python>=3.7
- cython>=0.29,<0.30
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is the cython dep coming from?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cytoolz? I don't suppose we're building anything.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, if one of the deps needs it conda should figure that out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh ... my bad I meant to remove that. I was experimenting with implementing librdkafka with cython and forgot to remove it. I will remove them now.

On a side note that is working very well BTW and might open something up around that as well. Working on the best way to make cython not a hard dependency at the moment since I'm sure several people don't using streamz don't necessarily want it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working on the best way to make cython not a hard dependency

Typically you can include the C file for those that want to compile (source dist) and of course, if you build wheels and conda packages, those should not need cython.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused, do you mean that you are using librdkafka with cython or using cython to speed up streamz kafka bindings?

Copy link
Contributor Author

@jdye64 jdye64 Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused, do you mean that you are using librdkafka with cython or using cython to speed up streamz kafka bindings?

Actually both. I was replacing the current streamz kafka sources/bindsings with my own sources (AKA: not the confluent-python-kafka library) which were cython modules that interacted directly with librdkafka++. There are many advantages to that especially in the realm of partition balancing that the new KafkaConsumer offers (not implemented in the confluent python library yet) for distributed systems with many processes appearing/disappearing. I actually saw this #233 while messing around with it and think it could be helpful here as well.

In a nutshell I was seeing about 15-20X speedups in my consumers when using my cython implementations.

Actually this one as well would benefit #279

- pytest
- flake8
- black
- isort
- tornado
- toolz
- zict
- six
- librdkafka=1.2.2
- dask
- distributed
- pandas
- python-confluent-kafka=1.1.0