Merge commit '18ed6790e53e124d2b562bf3b7a7c1714b538705' into docs/editorial-review-1

* commit '18ed6790e53e124d2b562bf3b7a7c1714b538705':
  Move PegdownMarkdownFormatter to docs class
  Fix setup.py dependency, fix rf_rasterize signature regression in unit test
  reverse sort
  Deterministic pymd file order
  Update docs readme
  Fix broken ref, attempt trigger circle build for docs/* branches

Author: metasim

.circleci/config.yml

@@ -191,6 +191,7 @@ workflows:
             branches:
               only:
                 - /feature\/.*docs.*/
+                - /docs\/.*/
 
   nightly:
     triggers:

docs/README.md

@@ -25,6 +25,21 @@ The build constructs in `<src-root>/docs` are (due to legacy reasons) the top-le
 }
 ```
 
+## Building the docs
+
+To build the static site locally:
+
+    sbt makeSite
+    
+The site will be at `<src-root>/docs/target/site/index.html`.
+    
+    
+To start an interactive server running the docs:
+
+    sbt previewSite
+    
+The sbt server log a message with an address to view the site.
+
 ## Content Development Process
 
 Start with one of the existing files in `<src-root>/pyrasterframes/src/main/python/docs` as a template. [`local-algebra.pymd`](../pyrasterframes/src/main/python/docs/local-algebra.pymd) is a good example. If the content will have code blocks you want evaluated an results injected into the output, use the file extension `.pymd`. If the content doesn't use evaluatable code blocks, use `.md`.
@@ -44,9 +59,9 @@ To set up an environment whereby you can easily test/evaluate your code blocks d
     [info] Set current project to RasterFrames (in build file:<src-root>/)
     sbt:RasterFrames>    
     ```  
-2. The first time you check out the code, or whenever RasterFrames code is updated, you need to build the project artifacts so they are available for Pweave.
+2. The first time you check out the code, or whenever RasterFrames code is updated, you need to build the project artifacts so they are available for Pweave. Some docs also refer to test resources, so the easiest way to do it is to run the unit tests.
     ```
-    sbt:RasterFrames> pyrasterframes/package
+    sbt:RasterFrames> pyrasterframes/test
     [info] Compiling 4 Scala sources to <src-root>/core/target/scala-2.11/classes ...
    ... lots of noise ...
    [info] PyRasterFrames assembly written to '<src-root>/pyrasterframes/target/python/deps/jars/pyrasterframes-assembly-0.8.0-SNAPSHOT.jar'
@@ -66,7 +81,7 @@ To set up an environment whereby you can easily test/evaluate your code blocks d
     sbt:RasterFrames> pyrasterframes/doc
     ```
     There's a command alias for this last step: `pyDocs`.
-4. To evaluate a single `.pymd` file, you pass the `-s` option and the filename relative to the `python` directory:
+4. To evaluate a single `.pymd` file, you pass the `-s` option and the filename relative to the `pyraterframes/src/main/python` directory. You can also specify the output [format](http://mpastell.com/pweave/formats.html) with the `-f` argument.
     ```
     sbt:RasterFrames> pyrasterframes/pySetup pweave -s docs/getting-started.pymd
     [info] Synchronizing 44 files to '<src-root>/pyrasterframes/target/python'
@@ -82,7 +97,7 @@ To set up an environment whereby you can easily test/evaluate your code blocks d
     Weaved docs/getting-started.pymd to docs/getting-started.md
     ```
 5. The _output_ Markdown files are written to `<src-root>/pyrasterframes/target/python/docs`. _Note_: don't edit any files in the `pyrasterframes/target` directory... they will get overwritten each time `sbt` runs a command.
-6. During content development it's sometimes helpful to see the output rendered as basic HTML. To do this, add the `-d html` option to the pweave command:
+6. During content development it's sometimes helpful to see the output rendered as basic HTML. To do this, add the `-f html` option to the pweave command:
     ```
     sbt:RasterFrames> pyrasterframes/pySetup pweave -f html -s docs/getting-started.pymd
     [info] Synchronizing 54 files to '<src-roog>/pyrasterframes/target/python'
@@ -103,8 +118,11 @@ To set up an environment whereby you can easily test/evaluate your code blocks d
 
 ## Notebooks
 
-The `rf-notebooks` sub-project creates a Docker image with Jupyter Notebooks pre-configured with RasterFrames. Any `.pymd` file under `.../python/docs/` is converted to an evaluated Jupyter Notebook and included as a part of the build (an additional bonus of the Pweave tool). 
+The `rf-notebooks` sub-project creates a Docker image with Jupyter Notebooks pre-configured with RasterFrames. Any `.pymd` file under `.../python/docs/` is converted to an evaluated Jupyter Notebook and included as a part of the build.
 
 ## Submission Process
 
-Submit new and updated documentation as a PR against locationtech/rasterframes. Make sure you've signed the Eclipse Foundation ECA and you ["Signed-off-by:"](https://stackoverflow.com/questions/1962094/what-is-the-sign-off-feature-in-git-for) each commit in the PR. The "Signed-off-by" address needs to be the exact same one as registered with the [Eclipse Foundation](https://wiki.eclipse.org/Development_Resources/Contributing_via_Git).
+Submit new and updated documentation as a PR against locationtech/rasterframes. Make sure you've signed the Eclipse Foundation ECA and you ["Signed-off-by:"](https://stackoverflow.com/questions/1962094/what-is-the-sign-off-feature-in-git-for) each commit in the PR. The "Signed-off-by" email address needs to be the exact same one as registered with the [Eclipse Foundation](https://wiki.eclipse.org/Development_Resources/Contributing_via_Git).
+
+If you are using circle CI, the circle configuration is set up to build the docs with `sbt makeSite` for branch names matching `feature/.*docs.*` or `docs/.*`
+

pyrasterframes/src/main/python/docs/__init__.py

@@ -21,6 +21,8 @@
 import os
 from pyspark.sql import DataFrame
 from pyrasterframes import RFContext
+from pweave import PwebPandocFormatter
+
 
 def docs_dir():
     return os.path.dirname(os.path.realpath(__file__))
@@ -38,3 +40,12 @@ def resource_dir():
 
 def resource_dir_uri():
     return 'file://' + resource_dir()
+
+
+class PegdownMarkdownFormatter(PwebPandocFormatter):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+    # Pegdown doesn't support the width and label options.
+    def make_figure_string(self, figname, width, label, caption=""):
+        return "![%s](%s)" % (caption, figname)

pyrasterframes/src/main/python/setup.py

@@ -47,16 +47,6 @@ def _divided(msg):
     return divider + '\n' + msg + '\n' + divider
 
 
-from pweave import PwebPandocFormatter
-
-class PegdownMarkdownFormatter(PwebPandocFormatter):
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-
-    # Pegdown doesn't support the width and label options.
-    def make_figure_string(self, figname, width, label, caption=""):
-        return "![%s](%s)" % (caption, figname)
-
 class PweaveDocs(distutils.cmd.Command):
     """A custom command to run documentation scripts through pweave."""
     description = 'Pweave PyRasterFrames documentation scripts'
@@ -67,6 +57,7 @@ class PweaveDocs(distutils.cmd.Command):
         ('quick=', 'q', 'Check to see if the source file is newer than existing output before building. Defaults to `False`.')
     ]
 
+
     def initialize_options(self):
         """Set default values for options."""
         # Each user option must be listed here with their default value.
@@ -96,6 +87,8 @@ def run(self):
         """Run pweave."""
         import traceback
         import pweave
+        from docs import PegdownMarkdownFormatter
+
         bad_words = ["Error"]
         pweave.rcParams["chunk"]["defaultoptions"].update({'wrap': False, 'dpi': 175})
         if self.format == 'markdown':
@@ -104,7 +97,7 @@ def run(self):
                 'description': 'Pegdown compatible markdown'
             }
 
-        for file in self.files:
+        for file in sorted(self.files, reverse=True):
             name = path.splitext(path.basename(file))[0]
             dest = self.dest_file(file)
 

pyrasterframes/src/main/python/tests/VectorTypesTests.py

@@ -97,7 +97,7 @@ def some_point(g):
             pandas_df_out.poly_len.values
         )
 
-    def test_rasterize(self):
+    def test_geometry_udf(self):
         from geomesa_pyspark.types import PolygonUDT
         # simple test that raster contents are not invalid
 
@@ -117,12 +117,25 @@ def area(g):
         area_result = area.collect()
         self.assertTrue(all([r[0] for r in area_result]))
 
-        cols = 194
-        rows = 250
-        with_raster = with_poly.withColumn('rasterized', rf_rasterize('poly', 'geometry', lit(16), cols, rows))
-        # expect a 4 by 4 cell
+    def test_rasterize(self):
+        from geomesa_pyspark.types import PolygonUDT
+
+        @udf(PolygonUDT())
+        def buffer(g, d):
+            return g.buffer(d)
+
+        # start with known polygon, the tile extents, **negative buffered**  by 10 cells
+        buf_cells = 10
+        with_poly = self.rf.withColumn('poly', buffer(self.rf.geometry, lit(-15 * buf_cells)))  # cell res is 15x15
+
+        # rasterize value 16 into buffer shape.
+        cols = 194  # from dims of tile
+        rows = 250  # from dims of tile
+        with_raster = with_poly.withColumn('rasterized',
+                                          rf_rasterize('poly', 'geometry', lit(16), lit(cols), lit(rows)))
         result = with_raster.select(rf_tile_sum(rf_local_equal_int(with_raster.rasterized, 16)),
                                     rf_tile_sum(with_raster.rasterized))
+        #
         expected_burned_in_cells = (cols - 2 * buf_cells) * (rows - 2 * buf_cells)
         self.assertEqual(result.first()[0], float(expected_burned_in_cells))
         self.assertEqual(result.first()[1], 16. * expected_burned_in_cells)