Add generation script for large CSV files

Author: gjbex

source-code/polars/README.md

@@ -10,4 +10,8 @@ Polars is an alternative to pandas that is designed to have better performance.
    directory with the same name.
 1. `polars_versus_pandas_benchmarks.ipynb`: Jupyter notebook that compares the
    performance of polars and pandas on a variety of operations.
+1. `create_csv_data.py`: Python script to generate one or more large CSV files
+   for benchmarking.
+1. `create_csv_data.slurm`: Slurm script to run `create_csv_data.py` on a
+   cluster.
 1. `data`: Directory containing the data used in the notebook.

source-code/polars/create_csv_data.slurm

@@ -0,0 +1,19 @@
+#!/usr/bin/env -S bash -l
+#SBATCH --account=lpt2_sysadmin
+#SBATCH --nodes=1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=1
+#SBATCH --mem=2G
+#SBATCH --time=01:00:00
+#SBATCH --mail-user=geertjan.bex@uhasselt.be
+#SBATCH --mail-type=FAIL,END
+
+module purge
+module load Python/3.11.3-GCCcore-12.3.0
+
+# This should generate a file of approximately 6 GB
+python ./create_csv_data.py \
+    --files 1 \
+    --cols 100 \
+    --rows 2500000 \
+    large_data