Feature docs v34 v2 (#192)

ronanstokes-db · web-flow · commit 3bfd5f67a520 · 2023-04-06T18:06:03.000-07:00
* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* doc changes only

* wip

* wip

* Update generating_column_data.rst

* Update generating_json_data.rst

* wip

* new document build

* adjusted comment banner at start of each doc file

* updated build

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* reverted code comment changes

* wip
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,12 +3,12 @@
 ## Change History
 All notable changes to the Databricks Labs Data Generator will be documented in this file.
 
-### Unreleased
+### Version 0.3.3post2
 
 #### Changed
 * Fixed use of logger in _version.py and in spark_singleton.py
 * Fixed template issues 
-* Document reformatting and updates
+* Document reformatting and updates, related code comment changes
 * Modified option to allow for range when specifying `numFeatures` with `structType='array'` to allow generation
   of varying number of columns
 * When generating multi-column or array valued columns, compute random seed with different name for each column
diff --git a/docs/source/TOBEADDED.rst b/docs/source/TOBEADDED.rst
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -44,7 +44,8 @@
     'sphinx.ext.viewcode',  # add links to source code
     #'numpydoc',  # handle NumPy documentation formatted docstrings. Needs to install
     'recommonmark',  # allow including Commonmark markdown in sources
-    'sphinx_rtd_theme'
+    'sphinx_rtd_theme',
+    'sphinx_copybutton'
 ]
 
 source_suffix = {
diff --git a/docs/source/extending_text_generation.rst b/docs/source/extending_text_generation.rst
@@ -1,7 +1,5 @@
 .. Databricks Labs Data Generator documentation master file, created by
    sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 Extending Text Generation
 =========================
diff --git a/docs/source/generating_cdc_data.rst b/docs/source/generating_cdc_data.rst
@@ -1,7 +1,5 @@
-.. Test Data Generator documentation master file, created by
-sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-You can adapt this file completely to your liking, but it should at least
-contain the root `toctree` directive.
+.. Databricks Labs Data Generator documentation master file, created by
+   sphinx-quickstart on Sun Jun 21 10:54:30 2020.
 
 Generating Change Data Capture Data
 ===================================
diff --git a/docs/source/generating_column_data.rst b/docs/source/generating_column_data.rst
@@ -1,3 +1,6 @@
+.. Databricks Labs Data Generator documentation master file, created by
+   sphinx-quickstart on Sun Jun 21 10:54:30 2020.
+
 Generating Column Data
 ======================
 
diff --git a/docs/source/generating_json_data.rst b/docs/source/generating_json_data.rst
@@ -1,7 +1,5 @@
-.. Test Data Generator documentation master file, created by
+.. Databricks Labs Data Generator documentation master file, created by
    sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 Generating JSON and Structured Column Data
 ==========================================
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -1,7 +1,5 @@
-.. Test Data Generator documentation master file, created by
+.. Databricks Labs Data Generator documentation master file, created by
    sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 Databricks Labs Data Generator Documentation
 ============================================
@@ -35,6 +33,7 @@ As it is installable via `%pip install`, it can also be incorporated in environm
    Generating Change Data Capture (CDC) data<generating_cdc_data>
    Using multiple tables <multi_table_data>
    Extending text generation  <extending_text_generation>
+   Use with Delta Live Tables <using_delta_live_tables>
    Troubleshooting data generation <troubleshooting>
 
 .. toctree::
diff --git a/docs/source/installation_notes.rst b/docs/source/installation_notes.rst
@@ -1,7 +1,5 @@
-.. Test Data Generator documentation master file, created by
+.. Databricks Labs Data Generator documentation master file, created by
    sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 Installation Notes
 ==================
diff --git a/docs/source/license.rst b/docs/source/license.rst
@@ -1,7 +1,5 @@
-.. Test Data Generator documentation master file, created by
+.. Databricks Labs Data Generator documentation master file, created by
    sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 License
 =======
diff --git a/docs/source/multi_table_data.rst b/docs/source/multi_table_data.rst
@@ -1,7 +1,5 @@
-.. Test Data Generator documentation master file, created by
-sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-You can adapt this file completely to your liking, but it should at least
-contain the root `toctree` directive.
+.. Databricks Labs Data Generator documentation master file, created by
+   sphinx-quickstart on Sun Jun 21 10:54:30 2020.
 
 Generating and Using Data with Multiple Tables
 ==============================================
diff --git a/docs/source/options_and_features.rst b/docs/source/options_and_features.rst
@@ -23,11 +23,9 @@ Finally type conversion is applied.
 The following table lists some of the common options that can be applied with the ``withColumn`` and ``withColumnSpec``
 methods.
 
-.. table:: Column creation options
-
-================  ==============================
+================  =========================================================================================
 Parameter         Usage
-================  ==============================
+================  =========================================================================================
 minValue          Minimum value for range of generated value. Alternatively, use ``dataRange``
 
 maxValue          Minimum value for range of generated value. Alternatively, use ``dataRange``
@@ -109,7 +107,7 @@ numFeatures       Synonym for `numColumns`
 
 structType        If set to `array`, generates array value from multiple columns.
 
-================  ==============================
+================  =========================================================================================
 
 
 .. note::
@@ -127,7 +125,7 @@ You may generate multiple columns with the same column generation spec by specif
 an integer value to generate a specific number of columns. The generated columns will be suffixed with a number
 representing the column - for example "email_0", "email_1" etc.
 
-If you specify the attribute `structType="array"`, the multiple columns will be combined into a single array valued
+If you specify the attribute ``structType="array"``, the multiple columns will be combined into a single array valued
 column.
 
 Generating random values
diff --git a/docs/source/repeatable_data_generation.rst b/docs/source/repeatable_data_generation.rst
@@ -1,7 +1,5 @@
-.. Test Data Generator documentation master file, created by
+.. Databricks Labs Data Generator documentation master file, created by
    sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 Repeatable Data Generation
 ==========================
diff --git a/docs/source/textdata.rst b/docs/source/textdata.rst
@@ -1,3 +1,6 @@
+.. Databricks Labs Data Generator documentation master file, created by
+   sphinx-quickstart on Sun Jun 21 10:54:30 2020.
+
 Generating and Manipulating Text Data
 =====================================
 
diff --git a/docs/source/troubleshooting.rst b/docs/source/troubleshooting.rst
@@ -1,7 +1,5 @@
-.. Test Data Generator documentation master file, created by
+.. Databricks Labs Data Generator documentation master file, created by
    sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 Troubleshooting
 ===============
diff --git a/docs/source/using_delta_live_tables.rst b/docs/source/using_delta_live_tables.rst
@@ -0,0 +1,203 @@
+.. Databricks Labs Data Generator documentation master file, created by
+   sphinx-quickstart on Sun Jun 21 10:54:30 2020.
+
+Using the Data Generator with Delta Live Tables
+===============================================
+
+Delta Live Tables is one of the newer Databricks platform features for creating reliable batch and streaming ingest
+pipelines using Python and SQL. It allows for creating ingest pipelines using Python code (and SQL) with annotations
+to define the tables and views produced.
+
+.. seealso::
+   You can find more information on Delta Live Tables at the following links:
+
+  * `Delta Live Tables <https://docs.databricks.com/delta-live-tables/index.html>`_
+  * `Delta Live Tables on Azure <https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/>`_
+  * `Change Data Capture with Databricks Delta Live Tables <https://www.databricks.com/blog/2022/04/25/simplifying-change-data-capture-with-databricks-delta-live-tables.html>`_
+
+You can use the Databricks Labs Data Generator inside of a Delta Live Tables to generate synthetic data sources for
+both batch and streaming sources. You can also use it from outside a Delta Live Tables pipeline to write data to a
+file that is subsequently read via `Autoloader` or a `spark.read` operation within a Delta Live Tables pipeline.
+
+Installing the data generator in a Delta Live Tables notebook
+-------------------------------------------------------------
+
+You can install the Databricks Labs Data Generator in a DLT notebook using the same command that you would use in a
+regular Python notebook
+
+.. code-block::
+
+   %pip install dbldatagen
+
+This must be the first cell in the notebook ahead of any initial markup cells.
+
+You can subsequently define a source based on the use of the data generator to generate a table, view or
+live / streaming table.
+
+Defining a DLT live table with synthetic data source
+----------------------------------------------------
+Once the library has been installed in the Delta Live Tables pipeline, you can use the DLT annotations
+combined with the data generator code to create a DLT live or live streaming sources.
+
+For example, the following code will generate a batch of device events indicating their state (`running`, `idle` etc)
+along with timestamp of change of state for a series of devices.
+
+.. code-block:: python
+
+   import dlt
+   import dbldatagen as dg
+
+   @dlt.table
+   def device_data_source():
+     DEVICE_STATES = ['RUNNING', 'IDLE', 'DOWN']
+     DEVICE_WEIGHTS = [10, 5,1 ]
+     SITES = ['alpha', 'beta', 'gamma', 'delta', 'phi', 'mu', 'lambda']
+     AREAS = ['area 1', 'area 2', 'area 3', 'area 4', 'area 5']
+     LINES = ['line 1', 'line 2', 'line 3', 'line 4', 'line 5', 'line 6']
+     TAGS = ['statusCode', 'another notification 1', 'another notification 2',
+             'another notification 3']
+     NUM_LOCAL_DEVICES = 20
+
+     STARTING_DATETIME="2022-06-01 01:00:00"
+     END_DATETIME="2022-09-01 23:59:00"
+     EVENT_INTERVAL = "10 seconds"
+
+     testDataSpec = (
+       dg.DataGenerator(spark, rows=1000000)
+      .withColumn("site", "string", values=SITES,  random=True, omit=True)
+      .withColumn("area", "string", values=AREAS, random=True, omit=True)
+      .withColumn("line", "string", values=LINES, random=True, omit=True)
+      .withColumn("local_device_id", "int", maxValue=NUM_LOCAL_DEVICES-1, omit=True, random=True)
+
+      .withColumn("local_device", "string", prefix="device",  baseColumn="local_device_id")
+
+      .withColumn("deviceKey", "string",
+                  expr = "concat('/', site, '/', area, '/', line, '/', local_device)",
+                  baseColumn=["local_device", "line", "area", "site"])
+
+      # tag name is name of device signal
+      .withColumn("tagName", "string", values=TAGS, random=True)
+
+      # tag value is state
+      .withColumn("tagValue", "string",
+                  values=DEVICE_STATES, weights=DEVICE_WEIGHTS,
+                  random=True)
+
+      .withColumn("tag_ts", "timestamp",
+                  begin=STARTING_DATETIME,
+                  end=END_DATETIME,
+                  interval=EVENT_INTERVAL,
+                  random=True)
+
+      )
+
+     dfTestData = testDataSpec.build()
+
+     return dfTestData
+
+
+To change this to a streaming live table, you only need change the parameters to the call to the `build` method.
+
+For example, replacing the build line with following code will change it to a DLT live streaming table.
+
+.. code-block:: python
+
+     dfTestData = testDataSpec.build(withStreaming=True, options={'rowsPerSecond': 50000}))
+
+You may define many streaming and non-streaming sources in the same Delta Live Tables pipeline notebooks
+allowing experimentation, benchmarking and exploration of different design approaches for your pipelines.
+
+Using the synthetic data sources in Delta Live Tables
+-----------------------------------------------------
+Once one or more synthetic data sources have been defined in a DLT pipeline, they can be used as sources for other
+downstreaming computations, views, as sources of joins etc.
+
+The following example illustrates the use of a synthetic data source to experiment with the creation of summary tables.
+
+For this example, we will enhance the source by materializing the ``site`` key and and an `event_date` so that
+they can be used for partitioning, and compute a derived summary table indicating the latest state for each device.
+
+First of all, here is the revised source
+
+.. code-block:: python
+
+   import dlt
+   import dbldatagen as dg
+
+   @dlt.table(name="device_data_source", partition_cols=["site", "event_date"])
+   def device_data_raw():
+     DEVICE_STATES = ['RUNNING', 'IDLE', 'DOWN']
+     DEVICE_WEIGHTS = [10, 5,1 ]
+     SITES = ['alpha', 'beta', 'gamma', 'delta', 'phi', 'mu', 'lambda']
+     AREAS = ['area 1', 'area 2', 'area 3', 'area 4', 'area 5']
+     LINES = ['line 1', 'line 2', 'line 3', 'line 4', 'line 5', 'line 6']
+     TAGS = ['statusCode', 'another notification 1', 'another notification 2',
+             'another notification 3']
+     NUM_LOCAL_DEVICES = 20
+
+     STARTING_DATETIME="2022-06-01 01:00:00"
+     END_DATETIME="2022-09-01 23:59:00"
+     EVENT_INTERVAL = "10 seconds"
+
+     rowsRequested = 10 * 1000000
+
+     testDataSpec = (
+       dg.DataGenerator(spark, rows=rowsRequested)
+      .withColumn("site", "string", values=SITES,  random=True)
+      .withColumn("area", "string", values=AREAS, random=True, omit=True)
+      .withColumn("line", "string", values=LINES, random=True, omit=True)
+
+      .withColumn("local_device_id", "int", maxValue=NUM_LOCAL_DEVICES-1, omit=True,
+                   random=True)
+      .withColumn("local_device", "string", prefix="device",
+                  baseColumn="local_device_id")
+
+      .withColumn("deviceKey", "string",
+                  expr = "concat('/', site, '/', area, '/', line, '/', local_device)",
+                  baseColumn=["local_device", "line", "area", "site"])
+
+      # tag name is name of device signal
+      .withColumn("tagName", "string", values=TAGS, random=True)
+
+      # tag value is state
+      .withColumn("tagValue", "string",
+                  values=DEVICE_STATES, weights=DEVICE_WEIGHTS,
+                  random=True)
+
+      .withColumn("tag_ts", "timestamp",
+                  begin=STARTING_DATETIME,
+                  end=END_DATETIME,
+                  interval=EVENT_INTERVAL,
+                  random=True)
+
+      .withColumn("event_date", "date", expr="to_date(tag_ts)", baseColumn="tag_ts")
+      )
+
+     dfTestData = testDataSpec.build()
+
+     return dfTestData.where("tagName = 'statusCode'")
+
+Now lets create a summary table that stores the latest state for each device.
+
+.. code-block:: python
+
+   import dlt
+   import dbldatagen as dg
+
+   @dlt.table(name="device_data_summary",
+              table_properties={"delta.checkpoint.writeStatsAsStruct" : "true",
+                                "delta.checkpoint.writeStatsAsJson" : "false",
+                                "delta.autoOptimize.optimizeWrite" : "true"
+                               }
+             )
+   def device_summary():
+     return spark.sql("""
+       with ranked as (select deviceKey, tagValue, tag_ts,
+              row_number() over (partition by deviceKey order by tag_ts desc) as rn
+              from LIVE.device_data_source)
+       select deviceKey, tagValue as state, tag_ts,rn from ranked
+              where rn == 1
+     """)
+
+As you can see in the example above, once the synthetic data source is created, it can be referred in the same way
+as any other DLT live or streaming table.
diff --git a/docs/source/using_streaming_data.rst b/docs/source/using_streaming_data.rst
@@ -1,7 +1,5 @@
-.. Test Data Generator documentation master file, created by
+.. Databricks Labs Data Generator documentation master file, created by
    sphinx-quickstart on Sun Jun 21 10:54:30 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 Using Streaming Data
 ====================
diff --git a/python/dev_require.txt b/python/dev_require.txt
@@ -32,4 +32,5 @@ recommonmark
 sphinx-markdown-builder
 rst2pdf==0.98
 Jinja2 < 3.1
+sphinx-copybutton
 

Original file line number	Diff line number	Diff line change
`@@ -44,7 +44,8 @@`
`44`	`44`	`'sphinx.ext.viewcode', # add links to source code`
`45`	`45`	`#'numpydoc', # handle NumPy documentation formatted docstrings. Needs to install`
`46`	`46`	`'recommonmark', # allow including Commonmark markdown in sources`
`47`		`- 'sphinx_rtd_theme'`
	`47`	`+ 'sphinx_rtd_theme',`
	`48`	`+ 'sphinx_copybutton'`
`48`	`49`	`]`
`49`	`50`
`50`	`51`	`source_suffix = {`