modified files to build for Databricks runtime 11.3 LTS compliant versions

Modified build and dependency files to build for Databricks runtime 11.3 LTS compliant versions as earlier versions
will not be supported beyond March 2025.

This allows for use of Python 3.9 / Apache Spark 3.3.0 as minimum versions and brings in important updates to streaming (that will avoid version specific unit tests for other commits)

Author: ronanstokes-db

.github/workflows/push.yml

@@ -31,10 +31,10 @@ jobs:
           sudo update-alternatives --set java /usr/lib/jvm/temurin-8-jdk-amd64/bin/java
           java -version
 
-      - name: Set up Python 3.8
+      - name: Set up Python 3.9.21
         uses: actions/setup-python@v5
         with:
-          python-version: '3.8.12'
+          python-version: '3.9.21'
           cache: 'pipenv'
 
       - name: Check Python version

.github/workflows/release.yml

@@ -24,10 +24,10 @@ jobs:
           sudo update-alternatives --set java /usr/lib/jvm/temurin-8-jdk-amd64/bin/java
           java -version
 
-      - name: Set up Python 3.8
+      - name: Set up Python 3.9.21
         uses: actions/setup-python@v5
         with:
-          python-version: '3.8.12'
+          python-version: '3.9.21'
           cache: 'pipenv'
 
       - name: Check Python version

CHANGELOG.md

@@ -8,6 +8,13 @@ All notable changes to the Databricks Labs Data Generator will be documented in
 #### Fixed 
 * Updated build scripts to use Ubuntu 22.04 to correspond to environment in Databricks runtime
 
+#### Changed
+* Changed base Databricks runtime version to DBR 11.3 LTS (based on Apache Spark 3.3.0)
+
+#### Added
+* Added support for serialization to/from JSON format
+
+
 ### Version 0.4.0 Hotfix 2
 
 #### Fixed

CONTRIBUTING.md

@@ -19,10 +19,7 @@ Dependent packages are not installed automatically by the `dbldatagen` package.
 
 ## Python compatibility
 
-The code has been tested with Python 3.8.12 and later.
-
-Older releases were tested with Python 3.7.5 but as of this release, it requires the Databricks 
-runtime 9.1 LTS or later. 
+The code has been tested with Python 3.9.21 and later.
 
 ## Checking your code for common issues
 
@@ -46,7 +43,7 @@ Our recommended mechanism for building the code is to use a `conda` or `pipenv`
 But it can be built with any Python virtualization environment.
 
 ### Spark dependencies
-The builds have been tested against Spark 3.2.1. This requires the OpenJDK 1.8.56 or later version of Java 8.
+The builds have been tested against Spark 3.3.0. This requires the OpenJDK 1.8.56 or later version of Java 8.
 The Databricks runtimes use the Azul Zulu version of OpenJDK 8 and we have used these in local testing.
 These are not installed automatically by the build process, so you will need to install them separately.
 
@@ -75,7 +72,7 @@ To build with `pipenv`, perform the following commands:
     - Run `make dist` from the main project directory
   - The resulting wheel file will be placed in the `dist` subdirectory
 
-The resulting build has been tested against Spark 3.2.1
+The resulting build has been tested against Spark 3.3.0
 
 ## Creating the HTML documentation
 
@@ -161,19 +158,19 @@ See https://legacy.python.org/dev/peps/pep-0008/
 
 # Github expectations
 When running the unit tests on Github, the environment should use the same environment as the latest Databricks
-runtime latest LTS release. While compatibility is preserved on LTS releases from Databricks runtime 10.4 onwards, 
+runtime latest LTS release. While compatibility is preserved on LTS releases from Databricks runtime 11.3 onwards, 
 unit tests will be run on the environment corresponding to the latest LTS release. 
 
-Libraries will use the same versions as the earliest supported LTS release - currently 10.4 LTS
+Libraries will use the same versions as the earliest supported LTS release - currently 11.3 LTS
 
 This means for the current build:
 
 - Use of Ubuntu 22.04 for the test runner
 - Use of Java 8
-- Use of Python 3.11
+- Use of Python 3.9.21 when testing / building the image
 
 See the following resources for more information
 = https://docs.databricks.com/en/release-notes/runtime/15.4lts.html
-- https://docs.databricks.com/en/release-notes/runtime/10.4lts.html
+- https://docs.databricks.com/en/release-notes/runtime/11.3lts.html
 - https://github.com/actions/runner-images/issues/10636
 

Pipfile

@@ -10,7 +10,7 @@ sphinx = ">=2.0.0,<3.1.0"
 nbsphinx = "*"
 numpydoc = "==0.8"
 pypandoc = "*"
-ipython = "==7.31.1"
+ipython = "==7.32.0"
 pydata-sphinx-theme = "*"
 recommonmark = "*"
 sphinx-markdown-builder = "*"
@@ -19,13 +19,13 @@ prospector = "*"
 
 [packages]
 numpy = "==1.22.0"
-pyspark = "==3.1.3"
-pyarrow = "==4.0.1"
-wheel = "==0.38.4"
-pandas = "==1.2.4"
-setuptools = "==65.6.3"
-pyparsing = "==2.4.7"
+pyspark = "==3.3.0"
+pyarrow = "==7.0.0"
+wheel = "==0.37.0"
+pandas = "==1.3.4"
+setuptools = "==58.0.4"
+pyparsing = "==3.0.4"
 jmespath = "==0.10.0"
 
 [requires]
-python_version = "3.8.12"
+python_version = "3.9.21"

README.md

@@ -83,8 +83,8 @@ The documentation [installation notes](https://databrickslabs.github.io/dbldatag
 contains details of installation using alternative mechanisms.
 
 ## Compatibility 
-The Databricks Labs Data Generator framework can be used with Pyspark 3.1.2 and Python 3.8 or later. These are 
-compatible with the Databricks runtime 10.4 LTS and later releases. For full Unity Catalog support, 
+The Databricks Labs Data Generator framework can be used with Pyspark 3.3.0 and Python 3.9.21 or later. These are 
+compatible with the Databricks runtime 11.3 LTS and later releases. For full Unity Catalog support, 
 we recommend using Databricks runtime 13.2 or later (Databricks 13.3 LTS or above preferred)
 
 For full library compatibility for a specific Databricks Spark release, see the Databricks 
@@ -155,7 +155,7 @@ The GitHub repository also contains further examples in the examples directory.
 
 ## Spark and Databricks Runtime Compatibility
 The `dbldatagen` package is intended to be compatible with recent LTS versions of the Databricks runtime, including 
-older LTS versions at least from 10.4 LTS and later. It also aims to be compatible with Delta Live Table runtimes, 
+older LTS versions at least from 11.3 LTS and later. It also aims to be compatible with Delta Live Table runtimes, 
 including `current` and `preview`. 
 
 While we don't specifically drop support for older runtimes, changes in Pyspark APIs or

dbldatagen/column_generation_spec.py

@@ -95,7 +95,7 @@ class ColumnGenerationSpec(SerializableToDict):
     # restrict spurious messages from java gateway
     logging.getLogger("py4j").setLevel(logging.WARNING)
 
-    def __init__(self, name, colType=None, minValue=0, maxValue=None, step=1, prefix='', random=False,
+    def __init__(self, name, colType=None, *, minValue=0, maxValue=None, step=1, prefix='', random=False,
                  distribution=None, baseColumn=None, randomSeed=None, randomSeedMethod=None,
                  implicit=False, omit=False, nullable=True, debug=False, verbose=False,
                  seedColumnName=DEFAULT_SEED_COLUMN,
@@ -529,18 +529,22 @@ def _setup_logger(self):
         else:
             self.logger.setLevel(logging.WARNING)
 
-    def _computeAdjustedRangeForColumn(self, colType, c_min, c_max, c_step, c_begin, c_end, c_interval, c_range,
+    def _computeAdjustedRangeForColumn(self, colType, c_min, c_max, c_step, *, c_begin, c_end, c_interval, c_range,
                                        c_unique):
         """Determine adjusted range for data column
         """
         assert colType is not None, "`colType` must be non-None instance"
 
         if type(colType) is DateType or type(colType) is TimestampType:
-            return self._computeAdjustedDateTimeRangeForColumn(colType, c_begin, c_end, c_interval, c_range, c_unique)
+            return self._computeAdjustedDateTimeRangeForColumn(colType, c_begin, c_end, c_interval,
+                                                               c_range=c_range,
+                                                               c_unique=c_unique)
         else:
-            return self._computeAdjustedNumericRangeForColumn(colType, c_min, c_max, c_step, c_range, c_unique)
+            return self._computeAdjustedNumericRangeForColumn(colType, c_min, c_max, c_step,
+                                                              c_range=c_range,
+                                                              c_unique=c_unique)
 
-    def _computeAdjustedNumericRangeForColumn(self, colType, c_min, c_max, c_step, c_range, c_unique):
+    def _computeAdjustedNumericRangeForColumn(self, colType, c_min, c_max, c_step, *, c_range, c_unique):
         """Determine adjusted range for data column
 
         Rules:
@@ -589,7 +593,7 @@ def _computeAdjustedNumericRangeForColumn(self, colType, c_min, c_max, c_step, c
 
         return result
 
-    def _computeAdjustedDateTimeRangeForColumn(self, colType, c_begin, c_end, c_interval, c_range, c_unique):
+    def _computeAdjustedDateTimeRangeForColumn(self, colType, c_begin, c_end, c_interval, *, c_range, c_unique):
         """Determine adjusted range for Date or Timestamp data column
         """
         effective_begin, effective_end, effective_interval = None, None, None
@@ -656,7 +660,7 @@ def _getUniformRandomSQLExpression(self, col_name):
         else:
             return "rand()"
 
-    def _getScaledIntSQLExpression(self, col_name, scale, base_columns, base_datatypes=None, compute_method=None,
+    def _getScaledIntSQLExpression(self, col_name, scale, base_columns, *, base_datatypes=None, compute_method=None,
                                    normalize=False):
         """ Get scaled numeric expression
 

dbldatagen/data_analyzer.py

@@ -92,7 +92,7 @@ def _displayRow(self, row):
 
         return ", ".join(results)
 
-    def _addMeasureToSummary(self, measureName, summaryExpr="''", fieldExprs=None, dfData=None, rowLimit=1,
+    def _addMeasureToSummary(self, measureName, *, summaryExpr="''", fieldExprs=None, dfData=None, rowLimit=1,
                              dfSummary=None):
         """ Add a measure to the summary dataframe
 
@@ -340,7 +340,7 @@ def _generatorDefaultAttributesFromType(cls, sqlType, colName=None, dataSummary=
         return result
 
     @classmethod
-    def _scriptDataGeneratorCode(cls, schema, dataSummary=None, sourceDf=None, suppressOutput=False, name=None):
+    def _scriptDataGeneratorCode(cls, schema, *, dataSummary=None, sourceDf=None, suppressOutput=False, name=None):
         """
         Generate outline data generator code from an existing dataframe
 

dbldatagen/data_generator.py

@@ -76,7 +76,7 @@ class DataGenerator(SerializableToDict):
 
     # logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.NOTSET)
 
-    def __init__(self, sparkSession=None, name=None, randomSeedMethod=None,
+    def __init__(self, sparkSession=None, name=None, *, randomSeedMethod=None,
                  rows=1000000, startingId=0, randomSeed=None, partitions=None, verbose=False,
                  batchSize=None, debug=False, seedColumnName=DEFAULT_SEED_COLUMN,
                  random=False,
@@ -782,7 +782,7 @@ def _checkColumnOrColumnList(self, columns, allowId=False):
                    f" column `{columns}` must refer to defined column")
         return True
 
-    def withColumnSpec(self, colName, minValue=None, maxValue=None, step=1, prefix=None,
+    def withColumnSpec(self, colName, *, minValue=None, maxValue=None, step=1, prefix=None,
                        random=None, distribution=None,
                        implicit=False, dataRange=None, omit=False, baseColumn=None, **kwargs):
         """ add a column specification for an existing column
@@ -842,7 +842,7 @@ def hasColumnSpec(self, colName):
         """
         return colName in self._columnSpecsByName
 
-    def withColumn(self, colName, colType=StringType(), minValue=None, maxValue=None, step=1,
+    def withColumn(self, colName, colType=StringType(), *, minValue=None, maxValue=None, step=1,
                    dataRange=None, prefix=None, random=None, distribution=None,
                    baseColumn=None, nullable=True,
                    omit=False, implicit=False, noWarn=False,
@@ -1058,7 +1058,7 @@ def withStructColumn(self, colName, fields=None, asJson=False, **kwargs):
 
         return newDf
 
-    def _generateColumnDefinition(self, colName, colType=None, baseColumn=None,
+    def _generateColumnDefinition(self, colName, colType=None, baseColumn=None, *,
                                   implicit=False, omit=False, nullable=True, **kwargs):
         """ generate field definition and column spec
 
@@ -1591,7 +1591,7 @@ def scriptTable(self, name=None, location=None, tableFormat="delta", asHtml=Fals
 
         return results
 
-    def scriptMerge(self, tgtName=None, srcName=None, updateExpr=None, delExpr=None, joinExpr=None, timeExpr=None,
+    def scriptMerge(self, tgtName=None, srcName=None, *, updateExpr=None, delExpr=None, joinExpr=None, timeExpr=None,
                     insertExpr=None,
                     useExplicitNames=True,
                     updateColumns=None, updateColumnExprs=None,

dbldatagen/text_generator_plugins.py

@@ -69,7 +69,7 @@ class _FnCallContext:
         def __init__(self, txtGen):
             self.textGenerator = txtGen
 
-    def __init__(self, fn, init=None, initPerBatch=False, name=None, rootProperty=None):
+    def __init__(self, fn, *, init=None, initPerBatch=False, name=None, rootProperty=None):
         super().__init__()
         assert fn is not None or callable(fn), "Function must be provided wiith signature fn(context, oldValue)"
         assert init is None or callable(init), "Init function must be a callable function or lambda if passed"
@@ -284,7 +284,7 @@ class FakerTextFactory(PyfuncTextFactory):
 
     _defaultFakerTextFactory = None
 
-    def __init__(self, locale=None, providers=None, name="FakerText", lib=None,
+    def __init__(self, *, locale=None, providers=None, name="FakerText", lib=None,
                  rootClass=None):
 
         super().__init__(name)