Merge branch 'gcarreno:main' into main

Author: synopse

.gitignore

@@ -4,9 +4,7 @@
 /profiling
 /results
 /entries/entries.json
-compile_all.*
-test_all.*
-run_all.*
+*_all.*
 
 # Compiled l10n files: .mo should be ignored
 *.mo
@@ -67,3 +65,7 @@ backup/
 
 # Castalia statistics file
 *.stat
+generator/Delphi/src/OSX64/
+generator/Delphi/src/Win32/
+generator/Delphi/src/Win64/
+generator/Delphi/src/Linux64/

CONTRIBUTING.md

@@ -3,7 +3,7 @@
 
 ## Formatting
 
-All entries should be formatted using the `Ctr-D` shortcut for both Lazarus and Delphi entries.
+All source code should be formatted using the default formatting rules for Pascal code, which your IDE should provide. Both Lazarus and Delphi uses `Ctr-D` as a shortcut to format your code.
 
 ## Folder name for the entry
 
@@ -13,11 +13,11 @@ For example, using _Gustavo Carreno_ for the name, the folder would be `entries/
 
 ## Name of the executable binary
 
-The executable binary follows the same rules has the entry folder above.
+The executable binary follows the same rules has the entry folder above; therefore, on Windows, the above example's executable would be `gcarreno.exe` and on Linux, just `gcarreno`.
 
 ## Placement of the executable binary
 
-The executable binary should be placed under a folder named `bin` below the root folder of this repository.
+The executable binary should be placed under a folder named `bin` below the root folder of this repository (`../../../bin` relative to your source).
 
 This folder is not present on the repository and is being ignored from the `.gitignore`.
 
@@ -57,7 +57,8 @@ The type is contained within the title and can be one of these types:
 Subjects should be no greater than 50 characters, should begin with a capital letter and do not end with a period.
 
 Use an imperative tone to describe what a commit does, rather than what it did. For example, use change; not changed or changes.
-The Body
+
+### The Body
 
 Not all commits are complex enough to warrant a body, therefore it is optional and only used when a commit requires a bit of explanation and context. Use the body to explain the what and why of a commit, not the how.
 

README.md

@@ -36,36 +36,37 @@ The task is to write an Object Pascal program which reads the file, calculates t
 ```
 
 ## Entering The Challenge
-Submissions will be via a `PR`( Pull Request ) to this repository. \
+Submissions will be via a `PR` (Pull Request) to this repository.  
 The challenge will run from the 10th of March until the 10th of May, 2024.
 
 When creating your entry, please do as follows:
 1. Create a folder under `entries` with your first initial and last name, e.g., for Gustavo Carreno: `entries/gcarreno`.
 2. If you're worried about anonymity, because the Internet stinks, feel free to use a fictional one: Bruce Wayne, Clark Kent, James Logan, Peter Parker, Diana of Themyscira. Your pick!
 3. Create a `README.md` with some content about your approach, e.g., `entries/gcarreno/README.md`.
 4. Put all your code under `entries/<your name>/src`, e.g., `entries/gcarreno/src`.
-5. If you need to provide a custom `.gitignore` for something not present in the main one, please do.
+5. Send your binary to the `bin` folder off the root of this repository. 
+6. If you need to provide a custom `.gitignore` for something not present in the main one, please do.
+7. Read the [CONTRIBUTING.md](./CONTRIBUTING.md) file for more details.
 
 This challenge is mainly to allow us to learn something new. This means that copying code from others will be allowed, under these conditions:
 1. You can only use pure Object Pascal with no calls to any operating system's `API` or external `C/C++` libraries. \
-  **There's been a bit of confusion about this restriction.** \
-  To clear that out: You can use any package/custom code you want. \
-  As long as it compiles cross-platform and itself is only pure Object Pascal. \
-  Anything from the `Jedi Project` or even `mORMmot` ( or anything else ), if it compiles, runs cross-platform it's allowed.
+  **There's been a bit of confusion about this restriction.**  
+  - To clear that out: You can use any package/custom code you want.
+  - As long as it compiles cross-platform and itself is only pure Object Pascal.
+  - Anything from the `Jedi Project` or even `mORMmot` ( or anything else ), if it compiles, runs cross-platform it's allowed.
 2. The code must have some sort of mention/attribution to the original author, in case you've used someone else's code.
 3. It's not a blatant copy just for the sake of submission.
 4. It adds something of value, not just a different code formatting.
 5. All code should be formatted with the `IDE`'s default formatting tool.
 
-**IMPORTANT** \
-This challenge can be entered even if you only have access to the Community Edition of RAD Studio. \
-I have a Windows VM, with RAD Studio installed, that will do the necessary cross compilation into my Linux host.
+**IMPORTANT**  
+This challenge can be entered even if you only have access to the Community Edition of RAD Studio. I have a Windows VM, with RAD Studio installed, that will do the necessary cross compilation into my Linux host.
 
 Submit your implementation and become part of the leader board!
 
 ## Rounding
 
-Székely Balázs has provided code for rounding towards positive infinity per the original challenge.\
+Székely Balázs has provided code for rounding towards positive infinity per the original challenge.  
 This will be the official way to round the output values:
 ```pas
 function TBaseline.RoundEx(x: Double): Double;
@@ -96,7 +97,7 @@ end;
 ```
 
 ## Generating the measurements.txt
-> **NOTE** \
+> **NOTE**  
 > We now have both a Lazarus version and a Delphi version of the generator for both 32b and 64b.
 
 In order to produce the One Billion Rows of text, we are providing the [source code](./generator) for the official generator, so we all have the same entry data.
@@ -110,7 +111,7 @@ In order to produce the One Billion Rows of text, we are providing the [source c
 | **-n** or **--line-count \<number\>** | The amount of lines to be generated ( Can use 1_000_000_000 ) |
 
 ## Baseline
-> **NOTE** \
+> **NOTE**  
 > This is still a bit in flux, still needing to get the Delphi version done.
 
 In order to verify the official output, we are providing the [source code](./baseline) for the official baseline.
@@ -178,7 +179,7 @@ These are the results from running all entries into the challenge on my personal
 | 4 | 1:16.059  | lazarus-3.0, fpc-3.2.2 | Richard Lawson    | Using 1 thread   | |
 | 5 | 12:40.179 | lazarus-3.0, fpc-3.2.2 | Iwan Kelaiah      | Using 1 thread   | |
 
-> ** NOTE **
+> **NOTE**
 >
 > After some tests performed by @paweld, it makes no sense to have an `HDD` run.
 > I've removed that from the results
@@ -206,7 +207,8 @@ A: Ubuntu 23.10 64b.
 I'd like to thank [@paweld](https://github.com/paweld) for taking us from my miserable 20m attempt, to a whopping ~25s, beating the [Python script](https://github.com/gunnarmorling/1brc/blob/main/src/main/python/create_measurements.py) by about 4 and a half minutes.\
 I'd like to thank [@mobius](https://github.com/mobius1qwe) for taking the time to provide the Delphi version of the generator.\
 I'd like to thank [@dtpfl](https://github.com/dtpfl) for his invaluable work on maintaining the `README.md` file up to date with everything.\
-I'd like to thank Székely Balázs for providing many patches to make everything compliant with the original challenge.
+I'd like to thank Székely Balázs for providing many patches to make everything compliant with the original challenge.\
+I'd like to thank [@corneliusdavid](https://github.com/corneliusdavid) for giving some of the information files a once over and making things more legible and clear.
 
 ## Links
 The original repository: https://github.com/gunnarmorling/1brc \

entries/dcornelius/README.md

@@ -0,0 +1,55 @@
+# David Cornelius
+
+An Entry to the One Billion Row Challenge in Object Pascal using Delphi 12 Athens by [David Cornelius](https://github.com/corneliusdavid).
+
+I wanted to see how different methods compared for ease of writing the code and speed of execution, so I solved this in three different ways:
+
+- **TDictionary** - as each line is read, create an object and add it to a `TDictionary` collection; if an entry already exists for a city, update it instead of adding. This was simple to implement but since `TDictionary` doesn't have a sort method, after this list is built, another list must be created to sort them.
+- **TStringList** - this is a really simple implementation but requires a lot of memory because the `LoadFromFile` method is used to read in all rows before processing them. Then a second TStringList is used to collate and sort the data. *NOTE: Using LoadFromFile resulted in an immediate Range Check Error when trying to read in the 1-billion line file! The default Stream created in `LoadFromFile` was the problem. When I switched to LoadFromStream and created my own Stream, it worked. However, it it's not near as fast as the `TDictionary` version.*
+- **In-Memory Table** - another approach I thought I'd try was to load all the data into an in-memory table and use local SQL to query the data. While I did learn some cool things about FireDAC, I also learned that this is by far, the most *inefficient* approach for this: after running for 26 *HOURS*, I killed the process! 
+
+## Compiler
+
+**Delphi 12 Athens** Enterprise Edition - which includes the ability to generate Linux console apps.
+
+### Dependencies
+
+There are no dependencies if run under one of the most recent versions of Delphi. The code should be backwards-compatible to Delphi 10.3 Rio (it uses inline variables and type inference introduced in that version) and further back with a few simple modifications. It uses `System.StrUtils`, `Generics.Collections`, and a few other run-time libraries in Delphi.
+
+### Conditional Compilation
+
+There are compiler directives to add some convenience when debugging. If built with the default *Debug* configuration, then the DEBUG compiler symbol is defined which turns on a few lines of code that give a little feedback and wait for Enter to be pressed so you can run this from the IDE without missing the quickly disappearing DOS box where the output is displayed.
+
+Thus, build in *Release* mode for the official challenge. 
+
+### Execution
+
+The program runs identical between Win32, Win64, and Linux64. If you run it without exactly two parameters, it will display the syntax and exit. The parameters are:
+
+- **Filename**: this is the input filename containing the measurements data as defined by the challenge.
+- *Method*: this specifies which method to use when reading and collating the data and correspond to the methods as discussed above:
+	- **DIC**: TDictionary
+	- **TSL**: TStringList; NOT an official entry--very slow.
+	- **TBL**: In-Memory Table; NOT an official entry, only test with *small* data sets
+
+#### Example
+
+To run the challenge, read from the `measurements.txt` file, and use the TDictionary method, run it like this:
+
+```
+C:> docrnelius measurements.txt dic
+```
+
+## Remarks
+
+I entered this challenge as a learning experience. I did not expect to be the fastest as I don't have time to implement multiple threads (which is clearly the road to victory here) but I had fun and learned a lot! 
+
+I now know more about how buffer size can affect stream reads significantly. I have not used streams much in Delphi before but after using `AssignFile`/`Reset`/`Readln`/`CloseFile` in my first attempts, noticing how fast `TStringList.LoadFromFile` was on small files and studying its implementation, I switched to using a `TStreamReader` and realized how much simpler and faster it is for reading text files.
+
+I also learned some things about a `TDictionary` and why it works so well for this particular situation. And, *after* I implemented this method, I looked at other entries and noticed the one by IWAN KELAIAH was very similar to mine. While the "ikelaiah" entry was submitted before mine, I did not look at or copy anything from that implementation. Chalk it up to great minds thinking alike, I guess!
+
+Finally, I learned that FireDAC has "LocalSQL" and uses the SQLite engine internally to query in-memory tables. It's not efficient for this long data set of two fields but could come in handy later, like when handling results from a REST service or something.  
+
+## History
+
+- Version 1.0: working version with `TDictionary`, `TStringList`, and `TFDMemTable` methods implemented.

entries/dcornelius/src/BuildDCorneliusWin32.cmd

@@ -0,0 +1,18 @@
+@echo off
+echo Build 1BRC Entry for DCornelius - Delphi 12 Athens, Win32, Release mode.
+echo Assumes Delphi 12 is in the path and can find the compiler for Win32 (dcc32.exe).
+pause
+
+REM RSVars is a batch supplied with Delphi and sets environment variables used in the compilation
+call RSVars
+
+REM -$L- : no debug symbols
+REM -$Y- : no symbol reference info
+REM -B   : build all units
+REM -Q   : quiet compile
+REM -TX  : set extension
+REM -D   : define compiler symbol
+REM -E   : output folder
+REM -CC  : console target
+REM -U   : unit folders
+dcc32.exe -$L- -$Y- --no-config -B -Q -TX.exe -DRELEASE -E..\..\..\bin -CC -U"%BDS%\lib\win32\release" dcornelius.dpr 

entries/dcornelius/src/BuildDCorneliusWin64.cmd

@@ -0,0 +1,18 @@
+@echo off
+echo Build 1BRC Entry for DCornelius - Delphi 12 Athens, Win64, Release mode.
+echo Assumes Delphi 12 is in the path and can find the compiler for Win64 (dcc64.exe).
+pause
+
+REM RSVars is a batch supplied with Delphi and sets environment variables used in the compilation
+call RSVars
+
+REM -$L- : no debug symbols
+REM -$Y- : no symbol reference info
+REM -B   : build all units
+REM -Q   : quiet compile
+REM -TX  : set extension
+REM -D   : define compiler symbol
+REM -E   : output folder
+REM -CC  : console target
+REM -U   : unit folders
+dcc64.exe -$L- -$Y- --no-config -B -Q -TX.exe -DRELEASE -E..\..\..\bin -CC -U"%BDS%\lib\Win64\release" dcornelius.dpr 

entries/dcornelius/src/dcornelius.dpr

@@ -0,0 +1,79 @@
+program dcornelius;
+(* as: OBRC_DCornelius.dpr
+ * by: David Cornelius
+ * on: March, 2024
+ * in: Delphi 12 Athens
+ * to: submit an entry in the One Billion Row Challenge
+ *
+ * Development/Testing was done on a 3.8 GHz Intel i7 desktop computer running Windows 11.
+ *
+ * NOTE: Build with 'Debug' configuration for messages, automatic timing, and a pause at the end.
+ *       Build with 'Release' configuration for the official submission.
+ *)
+
+{$APPTYPE CONSOLE}
+
+{$R *.res}
+
+uses
+  System.SysUtils, System.Classes,
+  {$IFDEF DEBUG}
+  System.Diagnostics,
+  {$ENDIF }
+  uChallengeWithDictionary in 'uChallengeWithDictionary.pas',
+  uChallengeCommon in 'uChallengeCommon.pas',
+  uChallengeWithStringList in 'uChallengeWithStringList.pas',
+  udmChallengeWithFireDAC in 'udmChallengeWithFireDAC.pas' {dmChallengeWithFireDAC: TDataModule};
+
+procedure DisplaySyntax;
+begin
+  Writeln('SYNTAX: ' + ExtractFileName(ParamStr(0)) + ' <filename> <method>');
+  Writeln('  where <filename> is a text file with weather station data');
+  Writeln('    and <method> is the algorytm for summarizing the data:');
+  Writeln('        TSL = read in all data to a TStringList (lots of memory needed)');
+  Writeln('        DIC = build a Dictionary, then sort after built');
+  Writeln('        TBL = load a FireDAC in-memory table - warning: takes several hours!');
+  Writeln;
+  {$IFDEF DEBUG}
+  Writeln('Running in Debug mode.');
+  {$ELSE}
+  Writeln('Running in Release mode.');
+  {$ENDIF}
+  Writeln('Press ENTER...');
+  Readln;
+end;
+
+begin
+  try
+    if ParamCount <> 2 then
+      DisplaySyntax
+    else begin
+      {$IFDEF DEBUG}
+      var StopWatch := TStopwatch.StartNew;
+      {$ENDIF}
+      ChallengeCommon := TChallengeCommon.Create(ParamStr(1));
+      try
+        var Method := ParamStr(2);
+        if SameText(Method, 'TSL') then
+          ChallengeWithStringList
+        else if SameText(Method, 'DIC') then
+          ChallengeWithDictionary
+        else if SameText(Method, 'TBL') then
+          ChallengeWithFireDAC
+        else
+          raise EArgumentException.Create('Invalid method');
+      finally
+        ChallengeCommon.Free;
+      end;
+      {$IFDEF DEBUG}
+      StopWatch.Stop;
+      var ms := StopWatch.ElapsedMilliseconds;
+      Writeln(Format('Elapsed Time milliseconds: %d, minutes: ~%d:%d', [ms, ms div 1000 div 60, (ms div 1000) mod 60]));
+      Readln;
+      {$ENDIF}
+    end;
+  except
+    on E: Exception do
+      Writeln(E.ClassName, ': ', E.Message);
+  end;
+end.