Skip to content

Commit 38e635d

Browse files
olavloiteolavloite
authored
chore: dotnet build scripts (#547)
* chore: add CreatePool and CreateConnection functions for SpannerLib Creates a spannerlib module and adds functions for CreatePool and CreateConnection. * chore: add Java wrapper for SpannerLib Adds a Java wrapper for SpannerLib and some simple tests for this wrapper. * chore: add Execute function for SpannerLib Adds an Execute function for SpannerLib that can be used to execute any type of SQL statement. The return type is always a Rows object. The Rows object is empty for DDL statements, it only contains ResultSetStats for DML statements without a THEN RETURN clause, and it contains actual row data for queries and DML statements with a THEN RETURN clause. The Execute function can also be used to execute client-side SQL statements, like BEGIN, COMMIT, SET, SHOW, etc. * chore: add transaction support for SpannerLib * chore: add ExecuteBatch to SpannerLib Adds an ExecuteBatch function to SpannerLib that supports executing DML or DDL statements as a single batch. The function accepts an ExecuteBatchDml request for both types of batches. The type of batch that is actually being executed is determined based on the statements in the batch. Mixing DML and DDL in the same batch is not supported. Queries are also not supported in batches. * chore: add WriteMutations function for SpannerLib Adds a WriteMutations function for SpannerLib. This function can be used to write mutations to Spanner in two ways: 1. In a transaction: The mutations are buffered in the current read/write transaction. The returned message is empty. 2. Outside a transaction: The mutations are written to Spanner directly in a new read/write transaction. The returned message contains the CommitResponse. * chore: add .NET wrapper for SpannerLib Adds a .NET wrapper and tests for SpannerLib. * chore: add grpc-server API for SpannerLib Adds a gRPC server that exposes the API of SpannerLib. This allows clients to connect to run SpannerLib as a child process and connect to it using gRPC. * chore: add Java wrapper for SpannerLib (#527) * chore: add Java wrapper for SpannerLib Adds a Java wrapper for SpannerLib and some simple tests for this wrapper. * chore: add Execute function for SpannerLib (#529) * chore: add Execute function for SpannerLib Adds an Execute function for SpannerLib that can be used to execute any type of SQL statement. The return type is always a Rows object. The Rows object is empty for DDL statements, it only contains ResultSetStats for DML statements without a THEN RETURN clause, and it contains actual row data for queries and DML statements with a THEN RETURN clause. The Execute function can also be used to execute client-side SQL statements, like BEGIN, COMMIT, SET, SHOW, etc. * chore: add transaction support for SpannerLib (#530) * chore: add transaction support for SpannerLib * chore: add ExecuteBatch to SpannerLib (#531) * chore: add ExecuteBatch to SpannerLib Adds an ExecuteBatch function to SpannerLib that supports executing DML or DDL statements as a single batch. The function accepts an ExecuteBatchDml request for both types of batches. The type of batch that is actually being executed is determined based on the statements in the batch. Mixing DML and DDL in the same batch is not supported. Queries are also not supported in batches. * chore: add WriteMutations function for SpannerLib (#532) Adds a WriteMutations function for SpannerLib. This function can be used to write mutations to Spanner in two ways: 1. In a transaction: The mutations are buffered in the current read/write transaction. The returned message is empty. 2. Outside a transaction: The mutations are written to Spanner directly in a new read/write transaction. The returned message contains the CommitResponse. * fix: check for error after calling Next() * chore: add .NET gRPC wrapper Adds a gRPC wrapper for .NET. This wrapper uses the gRPC API to connect to SpannerLib. The gRPC channel uses a Unix domain socket. * chore: dotnet build scripts Adds various build scripts for the .NET wrapper. * chore: re-build and push * chore: cleanup * fix: split scripts for building binaries and language-specific wrapper copying
1 parent ad05fde commit 38e635d

File tree

22 files changed

+313
-74
lines changed

22 files changed

+313
-74
lines changed

‎.github/workflows/spanner-lib-tests.yml‎

Lines changed: 36 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -87,82 +87,63 @@ jobs:
8787
os: [ubuntu-latest, macos-latest, windows-latest]
8888
runs-on: ${{ matrix.os }}
8989
steps:
90-
- name: Install dotnet
91-
uses: actions/setup-dotnet@v4
92-
with:
93-
dotnet-version: '9.0.x'
94-
- name: Install Go
95-
uses: actions/setup-go@v5
96-
with:
97-
go-version: ${{ matrix.go-version }}
9890
- name: Checkout code
9991
uses: actions/checkout@v4
10092
with:
10193
submodules: 'true'
102-
- name: Build shared library
103-
working-directory: spannerlib/shared
104-
run: go build -o spannerlib.so -buildmode=c-shared shared_lib.go
105-
- name: Copy to .NET wrapper
106-
working-directory: spannerlib
94+
- name: Install dotnet
95+
uses: actions/setup-dotnet@v4
96+
with:
97+
dotnet-version: '8.0.x'
98+
- name: Create temporary global.json
99+
# Force the use of .NET 8.0.x by writing that version to a global.json file.
100+
# Otherwise, .NET will default to the latest version on the host.
101+
# .NET 8 has a much smaller chance of crashing when using a native library written in Go
102+
# than .NET 9 has, so using .NET 8 allows us to run the tests using the native library as well.
107103
run: |
108-
mkdir -p wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/any
109-
if [ "$RUNNER_OS" == "Windows" ]; then
110-
cp shared/spannerlib.so wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/any/spannerlib.dll
111-
else
112-
cp shared/spannerlib.so wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/any/spannerlib
113-
fi
114-
shell: bash
115-
- name: Build .NET native library package
116-
run: dotnet pack
117-
working-directory: spannerlib/wrappers/spannerlib-dotnet/spannerlib-dotnet-native
104+
echo '{"sdk":{"version": "8.0.0", "rollForward": "latestFeature"}}' > ./global.json
118105
shell: bash
119-
- name: Add .NET package source
120-
run: |
121-
if [ "$RUNNER_OS" == "Windows" ]; then
122-
echo ${GITHUB_WORKSPACE}"\spannerlib\wrappers\spannerlib-dotnet\spannerlib-dotnet-native\bin\Release"
123-
dotnet nuget add source ${GITHUB_WORKSPACE}"\spannerlib\wrappers\spannerlib-dotnet\spannerlib-dotnet-native\bin\Release" --name local
124-
else
125-
dotnet nuget add source "$PWD"/bin/Release --name local
126-
fi
127-
working-directory: spannerlib/wrappers/spannerlib-dotnet/spannerlib-dotnet-native
106+
- name: .NET Version check
107+
working-directory: spannerlib/wrappers/spannerlib-dotnet
108+
run: dotnet --version
128109
shell: bash
129-
- name: Build gRPC server
130-
working-directory: spannerlib/grpc-server
131-
run: |
132-
go build -o grpc_server server.go
133-
chmod +x grpc_server
134-
- name: Copy gRPC server to .NET wrapper
135-
working-directory: spannerlib
110+
- name: Install Go
111+
uses: actions/setup-go@v5
112+
with:
113+
go-version: ${{ matrix.go-version }}
114+
# Install compilers for cross-compiling between operating systems.
115+
- name: Install compilers
136116
run: |
137-
mkdir -p wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/any
117+
echo "$RUNNER_OS"
138118
if [ "$RUNNER_OS" == "Windows" ]; then
139-
cp grpc-server/grpc_server wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/any/grpc_server.exe
119+
echo "Windows does not yet support cross compiling"
120+
elif [ "$RUNNER_OS" == "macOS" ]; then
121+
brew tap SergioBenitez/osxct
122+
brew install x86_64-unknown-linux-gnu
123+
brew install mingw-w64
140124
else
141-
cp grpc-server/grpc_server wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/any/grpc_server
125+
sudo apt-get update
126+
sudo apt install -y g++-mingw-w64-x86-64 gcc-mingw-w64-x86-64
127+
sudo apt-get install -y gcc-arm-linux-gnueabihf
142128
fi
143129
shell: bash
144-
- name: Build .NET gRPC server package
145-
run: dotnet pack
146-
working-directory: spannerlib/wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server
147-
shell: bash
148-
- name: Add .NET package source
130+
- name: Build the .NET wrapper using the build script
131+
working-directory: spannerlib/wrappers/spannerlib-dotnet
149132
run: |
150-
if [ "$RUNNER_OS" == "Windows" ]; then
151-
dotnet nuget add source ${GITHUB_WORKSPACE}"\spannerlib\wrappers\spannerlib-dotnet\spannerlib-dotnet-grpc-server\bin\Release" --name local-grpc-server
152-
else
153-
dotnet nuget add source "$PWD"/bin/Release --name local-grpc-server
154-
fi
155-
working-directory: spannerlib/wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server
133+
echo "$RUNNER_OS"
134+
./build.sh
156135
shell: bash
157136
- name: Restore dependencies
158137
run: dotnet restore
159138
working-directory: spannerlib/wrappers/spannerlib-dotnet
160139
shell: bash
161140
- name: Build
162-
run: dotnet build --no-restore
141+
run: dotnet build --no-restore -c Release
163142
working-directory: spannerlib/wrappers/spannerlib-dotnet
164143
shell: bash
165144
- name: .NET Unit Tests
166145
working-directory: spannerlib/wrappers/spannerlib-dotnet/spannerlib-dotnet-tests
167-
run: dotnet test --no-build --verbosity normal
146+
run: |
147+
dotnet --version
148+
dotnet test --no-build --verbosity normal -c Release
168149
shell: bash

‎spannerlib/.gitignore‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@ spannerlib.dll
55
grpc_server
66
grpc_server.exe
77
vendor/bundle
8-
shared/
98
*.gem
109
.DS_Store
1110
*.swp

‎spannerlib/grpc-server/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
binaries

‎spannerlib/grpc-server/build-dotnet.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Builds the gRPC server binary for darwin/arm64, linux/x64, and windows/x64
2+
# and copies the binaries to the appropriate folders of the .NET wrapper.
3+
4+
./build-executables.sh
5+
6+
mkdir -p ../wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/any
7+
rm ../wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/any/grpc_server 2> /dev/null
8+
9+
mkdir -p ../wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/osx-arm64
10+
cp binaries/osx-arm64/grpc_server ../wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/osx-arm64/grpc_server
11+
12+
mkdir -p ../wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/linux-x64
13+
cp binaries/linux-x64/grpc_server ../wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/linux-x64/grpc_server
14+
15+
mkdir -p ../wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/win-x64
16+
cp binaries/win-x64/grpc_server.exe ../wrappers/spannerlib-dotnet/spannerlib-dotnet-grpc-server/binaries/win-x64/grpc_server.exe

‎spannerlib/grpc-server/build-executables.sh‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Builds the gRPC server binary for darwin/arm64, linux/x64, and windows/x64.
2+
# The binaries are stored in the following files:
3+
# binaries/osx-arm64/grpc_server
4+
# binaries/linux-x64/grpc_server
5+
# binaries/win-x64/grpc_server.exe
6+
7+
mkdir -p binaries/osx-arm64
8+
GOOS=darwin GOARCH=arm64 go build -o binaries/osx-arm64/grpc_server server.go
9+
chmod +x binaries/osx-arm64/grpc_server
10+
11+
mkdir -p binaries/linux-x64
12+
GOOS=linux GOARCH=amd64 go build -o binaries/linux-x64/grpc_server server.go
13+
chmod +x binaries/linux-x64/grpc_server
14+
15+
mkdir -p binaries/win-x64
16+
GOOS=windows GOARCH=amd64 go build -o binaries/win-x64/grpc_server.exe server.go
17+
chmod +x binaries/win-x64/grpc_server.exe

‎spannerlib/shared/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
binaries

‎spannerlib/shared/build-binaries.sh‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Builds the shared library binaries and copies the binaries to OS/arch specific folders.
2+
# Binaries can be built for linux/x64, darwin/arm64, and windows/x64.
3+
# Which ones are actually built depends on the values of the following variables:
4+
# SKIP_MACOS: If set, will skip the darwin/arm64 build
5+
# SKIP_LINUX: If set, will skip the linux/x64 build that uses the default C compiler on the system
6+
# SKIP_LINUX_CROSS_COMPILE: If set, will skip the linux/x64 build that uses the x86_64-unknown-linux-gnu-gcc C compiler.
7+
# This compiler is used when compiling for linux/x64 on MacOS.
8+
# SKIP_WINDOWS: If set, will skip the windows/x64 build.
9+
10+
# The binaries are stored in the following files:
11+
# binaries/osx-arm64/spannerlib.dylib
12+
# binaries/linux-x64/spannerlib.so
13+
# binaries/win-x64/spannerlib.dll
14+
15+
echo "Skip macOS: $SKIP_MACOS"
16+
echo "Skip Linux: $SKIP_LINUX"
17+
echo "Skip Linux cross compile: $SKIP_LINUX_CROSS_COMPILE"
18+
echo "Skip windows: $SKIP_WINDOWS"
19+
20+
if [ -z "$SKIP_MACOS" ]; then
21+
echo "Building for darwin/arm64"
22+
mkdir -p binaries/osx-arm64
23+
GOOS=darwin GOARCH=arm64 CGO_ENABLED=1 go build -o binaries/osx-arm64/spannerlib.dylib -buildmode=c-shared shared_lib.go
24+
fi
25+
26+
if [ -z "$SKIP_LINUX_CROSS_COMPILE" ]; then
27+
# The following software is needed for this build, assuming that the build runs on MacOS.
28+
#brew tap SergioBenitez/osxct
29+
#brew install x86_64-unknown-linux-gnu
30+
echo "Building for linux/x64 (cross-compile)"
31+
mkdir -p binaries/linux-x64
32+
CC=x86_64-unknown-linux-gnu-gcc GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -o binaries/linux-x64/spannerlib.so -buildmode=c-shared shared_lib.go
33+
elif [ -z "$SKIP_LINUX" ]; then
34+
# The following commands assume that the script is running on linux/x64, or at least on some system that is able
35+
# to compile to linux/x64 with the default C compiler on the system.
36+
echo "Building for linux/x64"
37+
mkdir -p binaries/linux-x64
38+
GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -o binaries/linux-x64/spannerlib.so -buildmode=c-shared shared_lib.go
39+
fi
40+
41+
if [ -z "$SKIP_WINDOWS" ]; then
42+
# This build requires mingw-w64 or a similar Windows compatible C compiler if it is being executed on a
43+
# non-Windows environment.
44+
# brew install mingw-w64
45+
echo "Building for windows/x64"
46+
mkdir -p binaries/win-x64
47+
CC=x86_64-w64-mingw32-gcc GOOS=windows GOARCH=amd64 CGO_ENABLED=1 go build -o binaries/win-x64/spannerlib.dll -buildmode=c-shared shared_lib.go
48+
fi

‎spannerlib/shared/build-dotnet.sh‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Builds the shared library and copies the binaries to the appropriate folders for
2+
# the .NET wrapper. Binaries can be built for linux/x64, darwin/arm64, and windows/x64.
3+
# Which ones are actually built depends on the values of the following variables:
4+
# SKIP_MACOS: If set, will skip the darwin/arm64 build
5+
# SKIP_LINUX: If set, will skip the linux/x64 build that uses the default C compiler on the system
6+
# SKIP_LINUX_CROSS_COMPILE: If set, will skip the linux/x64 build that uses the x86_64-unknown-linux-gnu-gcc C compiler.
7+
# This compiler is used when compiling for linux/x64 on MacOS.
8+
# SKIP_WINDOWS: If set, will skip the windows/x64 build.
9+
10+
mkdir -p ../wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/any
11+
rm ../wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/any/* 2> /dev/null
12+
13+
./build-binaries.sh
14+
15+
if [ -z "$SKIP_MACOS" ]; then
16+
mkdir -p ../wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/osx-arm64
17+
cp binaries/osx-arm64/spannerlib.dylib ../wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/osx-arm64/spannerlib.dylib
18+
fi
19+
20+
if [ -z "$SKIP_LINUX_CROSS_COMPILE" ] || [ -z "$SKIP_LINUX" ]; then
21+
mkdir -p ../wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/linux-x64
22+
cp binaries/linux-x64/spannerlib.so ../wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/linux-x64/spannerlib.so
23+
fi
24+
25+
if [ -z "$SKIP_WINDOWS" ]; then
26+
mkdir -p ../wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/win-x64
27+
cp binaries/win-x64/spannerlib.dll ../wrappers/spannerlib-dotnet/spannerlib-dotnet-native/libraries/win-x64/spannerlib.dll
28+
fi

‎spannerlib/wrappers/spannerlib-dotnet/build.sh‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
2+
# Builds the .NET wrapper components that depend on the Go shared library.
3+
# This script executes the following steps:
4+
# 1. Determine which binaries should be built (darwin/arm64, linux/x64, win/x64).
5+
# The default is to build all. This requires the system that is running the script to be able to
6+
# compile for all those operating systems and architectures.
7+
# 2. Build the binaries for the gRPC server and copy these to the appropriate folders for the .NET wrapper.
8+
# 3. Build the binaries for the shared library and copy these to the appropriate folders for the .NET wrapper.
9+
# 4. Generate a 'snapshot' version number based on the current date/time for the .NET gRPC and shared library wrappers.
10+
# 5. Update all references to the gRPC and shared library wrappers to use the new generated version number.
11+
# 6. Pack the .NET gRPC and shared library wrappers and register the build directories of these as local nuget sources.
12+
# This allows the other projects in the solution to pick up these 'snapshot' versions locally instead of looking for
13+
# them in the central nuget repository.
14+
# Note that we use package references instead of project references for the .NET wrappers that contain the binary
15+
# files of the shared library, because .NET does not support dynamically loading the correct native library version
16+
# for project references.
17+
18+
# Determine which builds to skip when the script runs on GitHub Actions.
19+
if [ "$RUNNER_OS" == "Windows" ]; then
20+
# Windows does not support any cross-compiling.
21+
export SKIP_MACOS=true
22+
export SKIP_LINUX=true
23+
export SKIP_LINUX_CROSS_COMPILE=true
24+
elif [ "$RUNNER_OS" == "macOS" ]; then
25+
# When running on macOS, cross-compiling is supported.
26+
# We skip the 'normal' Linux build (the one that does not explicitly set a C compiler).
27+
export SKIP_LINUX=true
28+
elif [ "$RUNNER_OS" == "Linux" ]; then
29+
# Linux does not (yet) support cross-compiling to MacOS.
30+
# In addition, we use the 'normal' Linux build when we are already running on Linux.
31+
export SKIP_MACOS=true
32+
export SKIP_LINUX_CROSS_COMPILE=true
33+
fi
34+
echo "Skip macOS: $SKIP_MACOS"
35+
echo "Skip Linux: $SKIP_LINUX"
36+
echo "Skip Linux cross compile: $SKIP_LINUX_CROSS_COMPILE"
37+
echo "Skip windows: $SKIP_WINDOWS"
38+
39+
# Remove existing builds
40+
rm -r ./spannerlib-dotnet-native/libraries 2> /dev/null
41+
rm -r ./*/bin 2> /dev/null
42+
rm -r ./*/obj 2> /dev/null
43+
44+
# Build gRPC server
45+
echo "Building gRPC server..."
46+
cd ../../grpc-server || exit 1
47+
./build-dotnet.sh
48+
49+
# Build shared library
50+
echo "Building shared library..."
51+
cd ../shared || exit 1
52+
./build-dotnet.sh
53+
54+
# Build nuget packages
55+
cd ../wrappers/spannerlib-dotnet || exit 1
56+
57+
echo "Building .NET packages..."
58+
59+
# Add the build folders as local nuget package sources.
60+
# This allows the references to these generated versions to be picked up locally.
61+
62+
dotnet nuget remove source local-native-build 2>/dev/null
63+
if [ "$RUNNER_OS" == "Windows" ]; then
64+
# PWD does not work on Windows
65+
dotnet nuget add source "${GITHUB_WORKSPACE}\spannerlib\wrappers\spannerlib-dotnet\spannerlib-dotnet-native\bin\Release" --name local-native-build
66+
else
67+
dotnet nuget add source "$PWD"/spannerlib-dotnet-native/bin/Release --name local-native-build
68+
fi
69+
dotnet nuget remove source local-grpc-server-build 2>/dev/null
70+
if [ "$RUNNER_OS" == "Windows" ]; then
71+
# PWD does not work on Windows
72+
dotnet nuget add source "${GITHUB_WORKSPACE}\spannerlib\wrappers\spannerlib-dotnet\spannerlib-dotnet-grpc-server\bin\Release" --name local-grpc-server-build
73+
else
74+
dotnet nuget add source "$PWD"/spannerlib-dotnet-grpc-server/bin/Release --name local-grpc-server-build
75+
fi
76+
77+
# Create packages for the two components that contain the binaries (shared library + gRPC server)
78+
find ./**/bin/Release -type f -name "Alpha*.nupkg" -exec rm {} \;
79+
cd spannerlib-dotnet-native || exit 1
80+
dotnet pack
81+
cd .. || exit 1
82+
cd spannerlib-dotnet-grpc-server || exit 1
83+
dotnet pack
84+
cd .. || exit 1
85+
86+
# Restore the packages of all the projects so they pick up the locally built packages.
87+
dotnet restore

‎spannerlib/wrappers/spannerlib-dotnet/global.json‎

Lines changed: 0 additions & 7 deletions
This file was deleted.

0 commit comments

Comments
 (0)