chore: add simple token-based parser (#285)

* chore: add simple token parser

* chore: add multi-byte support

* chore: add benchmark

* chore: find parameters with comments in place (#424)

* chore: find parameters with comments in place

Find query parameters in the SQL string while keeping comments in place. This both
improves performance, as we need to copy the string fewer times, but more imporantly;
it keeps the comments in the SQL string that is sent to Spanner. The latter is
important for the PostgreSQL dialect, as query hints in that dialect are embedded into
comments.

* build: run tests pull requests for all branches (#426)

* chore: ignore lint errors

* perf: add a cache for parsed statements (#425)

* perf: add a cache for parsed statements

The functions for determining the type of statement and which query parameters
are in a statement are deterministic and do not change over time. These can
therefore safely be cached. This improves the execution speed of frequently
executed SQL strings, as the same parsing does not need to happen for each
execution.

* chore: go mod tidy

* chore: create statementParser struct and make it dialect-aware (#427)

* chore: create statementParser struct and make it dialect-aware

Refactors the statement parser into a struct that contains a field for the database
dialect that is being used. This dialect field will be used in follow-up pull request
to implement dialect-specific parsing.

* chore: remove duplicate quoted string parsing (#428)

* chore: remove duplicate quoted string parsing

The calculateFindParams function contained its own logic for skipping over quoted string literals.
This code was redundant, as it is also implemented in the skip function. This change removes this
duplication and simplifies the calculateFindParams implementation significantly.

This also ensures that we only have one place where string literals are found and skipped in the
parser, which will make it easier to make this code dialect-aware.

* chore: add support for PostgreSQL-style comments (#429)

* chore: add support for PostgreSQL-style comments

Adds support for PostgreSQL-style comments and sets some of the tests up for multi-dialect
testing.

* chore: support PostgreSQL-style parameters (#431)

* chore: support PostgreSQL-style parameters

Adds support for both recognizing PostgreSQL-style query parameters, and
converting positional parameters to PostgreSQL-style query parameters.

When using the PostgreSQL-dialect, the driver accepts both GoogleSQL
and PostgreSQL-style named parameters (so both @param and $1). This to
be consistent with other PostgreSQL drivers in Go (e.g. pgx).

* chore: add PostgreSQL quoting rules (#432)

* chore: add PostgreSQL quoting rules

Add PostgreSQL quoting rules to the parser. This means:
1. Backslashes do not start an escape sequence.
2. Repeating the same quote twice inside a string literal is an escaped quote.
3. Backtick is not a valid quote character.
4. Triple-quoted strings are not supported. However, if the start of a string consists
   of three consequtive quotes, then that means 'start of a string' and then 'an escaped
   quote'.
5. Linefeeds are allowed in string literals.

This change does not add support for dollar-quoted strings. That will be added in a
follow-up pull request.

* chore: add support for dollar-quoted strings (#433)

* chore: add support for dollar-quoted strings

Adds support for dollar-quoted strings.

Also fixes an issue in the skipWhitespaces function, where multi-byte characters would be
skipped as-if they were spaces.

* test: add some more tests for PostgreSQL parsing (#434)

* test: add some more tests for PostgreSQL parsing

* feat: detect database dialect and support PostgreSQL databases (#435)

* feat: detect database dialect and support PostgreSQL databases

Automatically detect the dialect of the database that the driver connects to,
and modify the way that query parameters are recognized based on the dialect
of the database. This makes it possible to use this driver with Spanner databases
that use the PostgreSQL dialect.

* perf: cache client-side SQL parsing (#436)

Cache the outcome of parsing a potential client-side SQL statement to prevent
the same regular expressions to be tested repeatedly. Also, the initial check
whether a statement is a client-side statement has been replaced with a keyword
based check: Instead of trying to match the SQL statement against the list of
all possible client-side statements, the parser first checks whether the first
keyword is a keyword that could potentially be a client-side statement. If it
is not, then the function returns early.

Benchmarks:

```
BenchmarkDetectStatementTypeWithCache-8      12530068     942 ns/op
BenchmarkDetectStatementTypeWithoutCache-8    2861143    4182 ns/op
```

* chore: rename function to skipWhitespacesAndComments

Author: olavloite

benchmarks/go.mod

@@ -38,6 +38,7 @@ require (
 	github.com/google/s2a-go v0.1.9 // indirect
 	github.com/googleapis/enterprise-certificate-proxy v0.3.6 // indirect
 	github.com/googleapis/gax-go/v2 v2.14.2 // indirect
+	github.com/hashicorp/golang-lru v0.5.1 // indirect
 	github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
 	github.com/spiffe/go-spiffe/v2 v2.5.0 // indirect
 	github.com/zeebo/errs v1.4.0 // indirect

benchmarks/go.sum

@@ -828,6 +828,7 @@ github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFb
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0/go.mod h1:hgWBS7lorOAVIJEQMi4ZsPv9hVvWI6+ch50m39Pf2Ks=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.11.3/go.mod h1:o//XUCC/F+yRGJoPO/VU0GSB0f8Nhgmxx0VIRUvaC0w=
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/iancoleman/strcase v0.2.0/go.mod h1:iwCmte+B7n89clKwxIoIXy/HfoL7AsD47ZCWhYzw7ho=
 github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=

conn.go

@@ -204,6 +204,7 @@ type SpannerConn interface {
 var _ SpannerConn = &conn{}
 
 type conn struct {
+	parser        *statementParser
 	connector     *connector
 	closed        bool
 	client        *spanner.Client
@@ -724,7 +725,7 @@ func (c *conn) Prepare(query string) (driver.Stmt, error) {
 
 func (c *conn) PrepareContext(_ context.Context, query string) (driver.Stmt, error) {
 	execOptions := c.options()
-	parsedSQL, args, err := parseParameters(query)
+	parsedSQL, args, err := c.parser.parseParameters(query)
 	if err != nil {
 		return nil, err
 	}
@@ -733,7 +734,7 @@ func (c *conn) PrepareContext(_ context.Context, query string) (driver.Stmt, err
 
 func (c *conn) QueryContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Rows, error) {
 	// Execute client side statement if it is one.
-	clientStmt, err := parseClientSideStatement(c, query)
+	clientStmt, err := c.parser.parseClientSideStatement(c, query)
 	if err != nil {
 		return nil, err
 	}
@@ -754,11 +755,11 @@ func (c *conn) queryContext(ctx context.Context, query string, execOptions ExecO
 		return pq.execute(ctx, execOptions.PartitionedQueryOptions.ExecutePartition.Index)
 	}
 
-	stmt, err := prepareSpannerStmt(query, args)
+	stmt, err := prepareSpannerStmt(c.parser, query, args)
 	if err != nil {
 		return nil, err
 	}
-	statementType := detectStatementType(query)
+	statementType := c.parser.detectStatementType(query)
 	// DDL statements are not supported in QueryContext so fail early.
 	if statementType.statementType == statementTypeDdl {
 		return nil, spanner.ToSpannerError(status.Errorf(codes.FailedPrecondition, "QueryContext does not support DDL statements, use ExecContext instead"))
@@ -797,7 +798,7 @@ func (c *conn) queryContext(ctx context.Context, query string, execOptions ExecO
 
 func (c *conn) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error) {
 	// Execute client side statement if it is one.
-	stmt, err := parseClientSideStatement(c, query)
+	stmt, err := c.parser.parseClientSideStatement(c, query)
 	if err != nil {
 		return nil, err
 	}
@@ -812,7 +813,7 @@ func (c *conn) execContext(ctx context.Context, query string, execOptions ExecOp
 	// Clear the commit timestamp of this connection before we execute the statement.
 	c.commitTs = nil
 
-	statementInfo := detectStatementType(query)
+	statementInfo := c.parser.detectStatementType(query)
 	// Use admin API if DDL statement is provided.
 	if statementInfo.statementType == statementTypeDdl {
 		// Spanner does not support DDL in transactions, and although it is technically possible to execute DDL
@@ -824,7 +825,7 @@ func (c *conn) execContext(ctx context.Context, query string, execOptions ExecOp
 		return c.execDDL(ctx, spanner.NewStatement(query))
 	}
 
-	ss, err := prepareSpannerStmt(query, args)
+	ss, err := prepareSpannerStmt(c.parser, query, args)
 	if err != nil {
 		return nil, err
 	}

driver.go

@@ -22,6 +22,7 @@ import (
 	"io"
 	"log/slog"
 	"math/big"
+	"os"
 	"regexp"
 	"runtime"
 	"runtime/debug"
@@ -33,6 +34,7 @@ import (
 	"cloud.google.com/go/civil"
 	"cloud.google.com/go/spanner"
 	adminapi "cloud.google.com/go/spanner/admin/database/apiv1"
+	"cloud.google.com/go/spanner/admin/database/apiv1/databasepb"
 	"cloud.google.com/go/spanner/apiv1/spannerpb"
 	"github.com/google/uuid"
 	"github.com/googleapis/gax-go/v2"
@@ -48,6 +50,10 @@ const userAgent = "go-sql-spanner/1.13.2" // x-release-please-version
 const gormModule = "github.com/googleapis/go-gorm-spanner"
 const gormUserAgent = "go-gorm-spanner"
 
+const DefaultStatementCacheSize = 1000
+
+var defaultStatementCacheSize int
+
 // LevelNotice is the default logging level that the Spanner database/sql driver
 // uses for informational logs. This level is deliberately chosen to be one level
 // lower than the default log level, which is slog.LevelInfo. This prevents the
@@ -91,6 +97,19 @@ var spannerDriver *Driver
 func init() {
 	spannerDriver = &Driver{connectors: make(map[string]*connector)}
 	sql.Register("spanner", spannerDriver)
+	determineDefaultStatementCacheSize()
+}
+
+func determineDefaultStatementCacheSize() {
+	if defaultCacheSizeString, ok := os.LookupEnv("SPANNER_DEFAULT_STATEMENT_CACHE_SIZE"); ok {
+		if defaultCacheSize, err := strconv.Atoi(defaultCacheSizeString); err == nil {
+			defaultStatementCacheSize = defaultCacheSize
+		} else {
+			defaultStatementCacheSize = DefaultStatementCacheSize
+		}
+	} else {
+		defaultStatementCacheSize = DefaultStatementCacheSize
+	}
 }
 
 // ExecOptions can be passed in as an argument to the Query, QueryContext,
@@ -210,6 +229,16 @@ type ConnectorConfig struct {
 	Instance string
 	Database string
 
+	// StatementCacheSize is the size of the internal cache that is used for
+	// connectors that are created from this ConnectorConfig. This cache stores
+	// the result of parsing SQL statements for query parameters and the type of
+	// statement (Query / DML / DDL).
+	// The default size is 1000. This default can also be overridden by setting
+	// the environment variable SPANNER_DEFAULT_STATEMENT_CACHE_SIZE.
+	StatementCacheSize int
+	// DisableStatementCache disables the use of a statement cache.
+	DisableStatementCache bool
+
 	// AutoConfigEmulator automatically creates a connection for the emulator
 	// and also automatically creates the Instance and Database on the emulator.
 	// Setting this option to true will:
@@ -335,6 +364,7 @@ type connector struct {
 	adminClient    *adminapi.DatabaseAdminClient
 	adminClientErr error
 	connCount      int32
+	parser         *statementParser
 }
 
 func newOrCachedConnector(d *Driver, dsn string) (*connector, error) {
@@ -480,6 +510,11 @@ func createConnector(d *Driver, connectorConfig ConnectorConfig) (*connector, er
 			connectorConfig.IsolationLevel = val
 		}
 	}
+	if strval, ok := connectorConfig.Params[strings.ToLower("StatementCacheSize")]; ok {
+		if val, err := strconv.Atoi(strval); err == nil {
+			connectorConfig.StatementCacheSize = val
+		}
+	}
 
 	// Check if it is Spanner gorm that is creating the connection.
 	// If so, we should set a different user-agent header than the
@@ -574,6 +609,7 @@ func openDriverConn(ctx context.Context, c *connector) (driver.Conn, error) {
 	connId := uuid.New().String()
 	logger := c.logger.With("connId", connId)
 	connection := &conn{
+		parser:      c.parser,
 		connector:   c,
 		client:      c.client,
 		adminClient: c.adminClient,
@@ -614,12 +650,33 @@ func (c *connector) increaseConnCount(ctx context.Context, databaseName string,
 		if c.clientErr != nil {
 			return c.clientErr
 		}
+		c.logger.Log(ctx, LevelNotice, "fetching database dialect")
+		closeClient := func() {
+			c.client.Close()
+			c.client = nil
+		}
+		if dialect, err := determineDialect(ctx, c.client); err != nil {
+			closeClient()
+			return err
+		} else {
+			// Create a separate statement parser and cache per connector.
+			cacheSize := c.connectorConfig.StatementCacheSize
+			if c.connectorConfig.DisableStatementCache {
+				cacheSize = 0
+			} else if c.connectorConfig.StatementCacheSize == 0 {
+				cacheSize = defaultStatementCacheSize
+			}
+			c.parser, err = newStatementParser(dialect, cacheSize)
+			if err != nil {
+				closeClient()
+				return err
+			}
+		}
 
 		c.logger.Log(ctx, LevelNotice, "creating Spanner Admin client")
 		c.adminClient, c.adminClientErr = adminapi.NewDatabaseAdminClient(ctx, opts...)
 		if c.adminClientErr != nil {
-			c.client = nil
-			c.client.Close()
+			closeClient()
 			c.adminClient = nil
 			return c.adminClientErr
 		}
@@ -630,6 +687,26 @@ func (c *connector) increaseConnCount(ctx context.Context, databaseName string,
 	return nil
 }
 
+func determineDialect(ctx context.Context, client *spanner.Client) (databasepb.DatabaseDialect, error) {
+	it := client.Single().Query(ctx, spanner.Statement{SQL: "select option_value from information_schema.database_options where option_name='database_dialect'"})
+	defer it.Stop()
+	for {
+		if row, err := it.Next(); err != nil {
+			return databasepb.DatabaseDialect_DATABASE_DIALECT_UNSPECIFIED, err
+		} else {
+			var dialectName string
+			if err := row.Columns(&dialectName); err != nil {
+				return databasepb.DatabaseDialect_DATABASE_DIALECT_UNSPECIFIED, err
+			}
+			if dialect, ok := databasepb.DatabaseDialect_value[dialectName]; ok {
+				return databasepb.DatabaseDialect(dialect), nil
+			} else {
+				return databasepb.DatabaseDialect_DATABASE_DIALECT_UNSPECIFIED, fmt.Errorf("unknown database dialect: %s", dialectName)
+			}
+		}
+	}
+}
+
 // decreaseConnCount decreases the number of connections that are active and closes the underlying clients if it was the
 // last connection.
 func (c *connector) decreaseConnCount() error {

driver_test.go

@@ -26,6 +26,7 @@ import (
 	"time"
 
 	"cloud.google.com/go/spanner"
+	"cloud.google.com/go/spanner/admin/database/apiv1/databasepb"
 	"cloud.google.com/go/spanner/apiv1/spannerpb"
 	"github.com/google/go-cmp/cmp"
 	"github.com/google/go-cmp/cmp/cmpopts"
@@ -533,7 +534,12 @@ func TestConn_StartBatchDml(t *testing.T) {
 }
 
 func TestConn_NonDdlStatementsInDdlBatch(t *testing.T) {
+	parser, err := newStatementParser(databasepb.DatabaseDialect_GOOGLE_STANDARD_SQL, 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
 	c := &conn{
+		parser:            parser,
 		logger:            noopLogger,
 		autocommitDMLMode: Transactional,
 		batch:             &batch{tp: ddl},
@@ -568,7 +574,12 @@ func TestConn_NonDdlStatementsInDdlBatch(t *testing.T) {
 }
 
 func TestConn_NonDmlStatementsInDmlBatch(t *testing.T) {
+	parser, err := newStatementParser(databasepb.DatabaseDialect_GOOGLE_STANDARD_SQL, 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
 	c := &conn{
+		parser: parser,
 		logger: noopLogger,
 		batch:  &batch{tp: dml},
 		execSingleQuery: func(ctx context.Context, c *spanner.Client, statement spanner.Statement, tb spanner.TimestampBound, options ExecOptions) *spanner.RowIterator {
@@ -605,8 +616,12 @@ func TestConn_NonDmlStatementsInDmlBatch(t *testing.T) {
 func TestConn_GetBatchedStatements(t *testing.T) {
 	t.Parallel()
 
+	parser, err := newStatementParser(databasepb.DatabaseDialect_GOOGLE_STANDARD_SQL, 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
 	ctx := context.Background()
-	c := &conn{logger: noopLogger}
+	c := &conn{logger: noopLogger, parser: parser}
 	if !reflect.DeepEqual(c.GetBatchedStatements(), []spanner.Statement{}) {
 		t.Fatal("conn should return an empty slice when no batch is active")
 	}
@@ -649,8 +664,13 @@ func TestConn_GetBatchedStatements(t *testing.T) {
 }
 
 func TestConn_GetCommitTimestampAfterAutocommitDml(t *testing.T) {
+	parser, err := newStatementParser(databasepb.DatabaseDialect_GOOGLE_STANDARD_SQL, 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
 	want := time.Now()
 	c := &conn{
+		parser:            parser,
 		logger:            noopLogger,
 		autocommitDMLMode: Transactional,
 		execSingleQuery: func(ctx context.Context, c *spanner.Client, statement spanner.Statement, tb spanner.TimestampBound, options ExecOptions) *spanner.RowIterator {
@@ -677,7 +697,12 @@ func TestConn_GetCommitTimestampAfterAutocommitDml(t *testing.T) {
 }
 
 func TestConn_GetCommitTimestampAfterAutocommitQuery(t *testing.T) {
+	parser, err := newStatementParser(databasepb.DatabaseDialect_GOOGLE_STANDARD_SQL, 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
 	c := &conn{
+		parser: parser,
 		logger: noopLogger,
 		execSingleQuery: func(ctx context.Context, c *spanner.Client, statement spanner.Statement, tb spanner.TimestampBound, options ExecOptions) *spanner.RowIterator {
 			return &spanner.RowIterator{}
@@ -693,7 +718,7 @@ func TestConn_GetCommitTimestampAfterAutocommitQuery(t *testing.T) {
 	if _, err := c.QueryContext(ctx, "SELECT * FROM Foo", []driver.NamedValue{}); err != nil {
 		t.Fatalf("failed to execute query: %v", err)
 	}
-	_, err := c.CommitTimestamp()
+	_, err = c.CommitTimestamp()
 	if g, w := spanner.ErrCode(err), codes.FailedPrecondition; g != w {
 		t.Fatalf("error code mismatch\n Got: %v\nWant: %v", g, w)
 	}