Skip to content

Commit 35b8e9c

Browse files
authored
Merge pull request #176 from CJ-Wright/dask_gotcha
add docs about crossing dask and stream nodes
2 parents 62ba067 + 3281779 commit 35b8e9c

File tree

1 file changed

+20
-0
lines changed

1 file changed

+20
-0
lines changed

docs/source/dask.rst

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,3 +117,23 @@ did.
117117
``source.emit``.
118118

119119
.. _Dask: https://dask.pydata.org/en/latest/
120+
121+
122+
Gotchas
123+
+++++++
124+
125+
126+
An important gotcha with ``DaskStream`` is that it is a subclass ``Stream``, and so can be used as an input
127+
to any function expecting a ``Stream``. If there is no intervening ``.gather()``, then the downstream node will
128+
receive Dask futures instead of the data they represent::
129+
130+
source = Stream()
131+
source2 = Stream()
132+
a = source.scatter().map(inc)
133+
b = source2.combine_latest(a)
134+
135+
In this case, the combine operation will get real values from ``source2``, and Dask futures.
136+
Downstream nodes would be free to operate on the futures, but more likely, the line should be::
137+
138+
b = source2.combine_latest(a.gather())
139+

0 commit comments

Comments
 (0)