Skip to content

Commit d0173dc

Browse files
djthorpedjthorpe
committed
Updated README
1 parent 7bdc23a commit d0173dc

File tree

1 file changed

+70
-204
lines changed

1 file changed

+70
-204
lines changed

‎README.md‎

Lines changed: 70 additions & 204 deletions
Original file line numberDiff line numberDiff line change
@@ -1,223 +1,89 @@
11
# go-sqlite
22

3-
This module provides an alternative interface for sqlite, including:
3+
This module provides an interface for sqlite, including:
44

5-
* Opening in-memory databases and databases by file path;
5+
* Opening in-memory databases and persistent file-based databases;
66
* Transactions (committing changes and rolling back on errors);
7-
* Reading results into a struct, map or slice.
7+
* Adding custom functions, authentication and authorization;
88
* Reflection on databases (schemas, tables, columns, indexes, etc);
9-
* Attaching and detaching databases by schema name;
109
* Executing arbitrary statements or building statements programmatically;
10+
* A pool of connections to run sqlite in a highliy concurrent environment such as a webservice;
11+
* A backend REST API for sqlite;
12+
* A generalized importer to import data from other data sources in different formats;
13+
* A generalized file indexer to index files in a directory tree and provide a REST API
14+
to search them;
15+
* A frontend web application to explore and interact with databases.
1116

1217
Presently the module is in development and the API is subject to change.
1318

14-
## Opening and creating databases
15-
16-
You can create a new in-memory database using the `New` method, or you can open an existing file-based database using the `Open` method.
17-
Both methods take an optional `*time.Location` argument which allows interpretation of time values with time zone. For example,
18-
19-
```go
20-
package main
21-
22-
import (
23-
"github.com/djthorpe/go-sqlite/pkg/sqlite"
24-
)
25-
26-
func main() {
27-
db, err := sqlite.New() // Open in-memory database with local time zone
28-
if err != nil {
29-
// ...
30-
}
31-
defer db.Close()
32-
33-
path := // ...
34-
db, err := sqlite.Open(path,time.UTC) // Open file database with UTC time zone
35-
if err != nil {
36-
// ...
37-
}
38-
defer db.Close()
39-
}
40-
```
41-
42-
Use `db.Close()` to release database resources.
43-
44-
## Executing queries and transactions
45-
46-
Statements are executed using the `db.Exec` method on a query. In order to create
47-
a query from a string, use the `Q()` method (see below for information on building
48-
SQL statements). You can include bound parameters after the query:
49-
50-
For example,
51-
52-
```go
53-
package main
54-
55-
import (
56-
"github.com/djthorpe/go-sqlite/pkg/sqlite"
57-
. "github.com/djthorpe/go-sqlite/pkg/lang"
58-
)
59-
60-
func main() {
61-
db, err := sqlite.New() // Open in-memory database with local time zone
62-
if err != nil {
63-
// ...
64-
}
65-
defer db.Close()
66-
67-
if result,err := db.Exec(Q("CREATE TABLE test (a TEXT,b TEXT)")); err != nil {
68-
// ...
69-
} else {
70-
fmt.Println(result)
71-
}
72-
}
19+
| If you want to... | Folder | Documentation |
20+
|--------------------------------------|-----------------|---------------|
21+
| Use the lower-level sqlite3 bindings similar to the [C API](https://www.sqlite.org/capi3ref.html) | [sys/sqlite3](https://github.com/djthorpe/go-sqlite/tree/master/sys/sqlite3) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/sys/sqlite3/README.md) |
22+
| Use high-concurrency high-level interface including statement caching and connection pool | [pkg/sqlite3](https://github.com/djthorpe/go-sqlite/tree/master/pkg/sqlite3) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/pkg/sqlite3/README.md) |
23+
| Implement a REST API to sqlite3 | [plugin/sqlite3](https://github.com/djthorpe/go-sqlite/tree/master/plugin/sqlite3) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/plugin/sqlite3/README.md) |
24+
| Develop a front-end web service to the REST API backend | [npm/sqlite3](https://github.com/djthorpe/go-sqlite/tree/master/npm/sqlite3) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/npm/sqlite3/README.md) |
25+
| Use an "object" interface to persist structured data | [pkg/sqobj](https://github.com/djthorpe/go-sqlite/tree/master/pkg/sqobj) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/pkg/sqobj/README.md) |
26+
| Use a statement builder to programmatically write SQL statements | [pkg/lang](https://github.com/djthorpe/go-sqlite/tree/master/pkg/lang) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/pkg/lang/README.md) |
27+
| Implement a generalized data importer from CSV, JSON, etc | [pkg/importer](https://github.com/djthorpe/go-sqlite/tree/master/pkg/importer) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/pkg/importer/README.md) |
28+
| Implement a search indexer | [pkg/indexer](https://github.com/djthorpe/go-sqlite/tree/master/pkg/indexer) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/pkg/indexer/README.md) |
29+
| Tokenize SQL statements for syntax colouring (for example) | [pkg/tokenizer](https://github.com/djthorpe/go-sqlite/tree/master/pkg/tokenizer) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/pkg/tokenizer/README.md) |
30+
| See example command-line tools | [cmd](https://github.com/djthorpe/go-sqlite/tree/master/cmd) | [README.md](https://github.com/djthorpe/go-sqlite/blob/master/cmd/README.md) |
31+
32+
## Requirements
33+
34+
* A sqlite3 installation, with library and header files;
35+
* go1.17 or later
36+
* Tested on Debian Linux (32- and 64- bit) and macOS with ARM
37+
and x64 architectures.
38+
39+
## Building
40+
41+
This module does not include a full
42+
copy of __sqlite__ as part of the build process, but expect a `pkgconfig`
43+
file called `sqlite3.pc` to be present (and an existing set of header
44+
files and libraries to be available to link against, of course).
45+
46+
In order to locate the correct installation of `sqlite3` use two environment variables:
47+
48+
* `PKG_CONFIG_PATH` is used for locating `sqlite3.pc`
49+
* `DYLD_LIBRARY_PATH` is used for locating the dynamic library when testing and/or running
50+
51+
On Macintosh with homebrew, for example:
52+
53+
```bash
54+
[bash] brew install sqlite3 npm
55+
[bash] git clone git@github.com:djthorpe/go-sqlite.git
56+
[bash] cd go-sqlite
57+
[bash] SQLITE_LIB="/usr/local/opt/sqlite/lib"
58+
[bash] PKG_CONFIG_PATH="${SQLITE_LIB}/pkgconfig" DYLD_LIBRARY_PATH="${SQLITE_LIB}" make
7359
```
7460

75-
The result object returned (of type SQResult) contains fields `LastInsertId` and `RowsAffected`
76-
which may or may not be set depending on the query executed. To return data, a result set is
77-
returned which should allows you to iterate across the results as a map of values, a slice
78-
of values:
79-
80-
81-
```go
82-
package main
83-
84-
import (
85-
"github.com/djthorpe/go-sqlite/pkg/sqlite"
86-
. "github.com/djthorpe/go-sqlite/pkg/lang"
87-
)
88-
89-
func main() {
90-
db, err := sqlite.New() // Open in-memory database with local time zone
91-
if err != nil {
92-
// ...
93-
}
94-
defer db.Close()
95-
96-
rs,err := db.Select(Q("SELECT a,b FROM test WHERE a=?"),"foo")
97-
if err != nil {
98-
// ...
99-
}
100-
defer rs.Close()
101-
for {
102-
row := rs.Next()
103-
if row == nil {
104-
break
105-
}
106-
// ...
107-
}
108-
}
109-
```
110-
111-
You can also create a block of code, which when returning any error will rollback
112-
any database snapshot, or else commit the snapshot if no error occurred:
113-
114-
```go
115-
package main
116-
117-
import (
118-
"github.com/djthorpe/go-sqlite/pkg/sqlite"
119-
. "github.com/djthorpe/go-sqlite/pkg/lang"
120-
)
121-
122-
func main() {
123-
db, err := sqlite.New() // Open in-memory database with local time zone
124-
if err != nil {
125-
// ...
126-
}
127-
defer db.Close()
128-
129-
db.Do(func (txn sqlite.SQTransaction) error {
130-
_, err := txn.Exec(Q("..."))
131-
if err != nil {
132-
// Rollback any database changes
133-
return err
134-
}
135-
136-
// Perform further operatins here...
137-
138-
// Return success, commit transaction
139-
return nil
140-
})
141-
}
142-
```
61+
On Debian Linux you shouldn't need to locate the correct path to the sqlite3 library, since
62+
only one copy is installed:
14363

144-
## Supported Column Types
145-
146-
The following types are supported, and expect the declared column types:
147-
148-
| Scalar Type | Column Type |
149-
| ------------| ----------------------|
150-
| int64 | INTEGER |
151-
| float64 | FLOAT |
152-
| string | TEXT |
153-
| bool | BOOL |
154-
| time.Time | TIMESTAMP or DATETIME |
155-
| []byte | BLOB |
156-
157-
If you pass other integer and unsigned integer types into the `Exec` and `Query` functions then they are converted to one of the above types. You can also define methods `MarshalSQ` and `UnmarshalSQ` in order to convert your custom types into
158-
scalar types. For example, the following methods convert between supported scalar types:
159-
160-
```go
161-
type CustomParam struct {
162-
A, B string
163-
}
164-
165-
func (c CustomParam) MarshalSQ() (interface{}, error) {
166-
if data, err := json.Marshal(c); err != nil {
167-
return nil, err
168-
} else {
169-
return string(data), err
170-
}
171-
}
172-
173-
func (c *CustomParam) UnmarshalSQ(v interface{}) error {
174-
if data, ok := v.(string); ok {
175-
return json.Unmarshal([]byte(data), c)
176-
} else {
177-
return fmt.Errorf("Invalid type: %T", v)
178-
}
179-
}
64+
```bash
65+
[bash] sudo apt install libsqlite3-dev npm
66+
[bash] git clone git@github.com:djthorpe/go-sqlite.git
67+
[bash] cd go-sqlite
68+
[bash] make
18069
```
18170

182-
## Attaching databases by schema name
183-
184-
You can load additional databases to a database by schema name. Use `Attach` and `Detach` to attach and detach databases. For example,
185-
186-
187-
188-
## Database reflection
71+
There are some examples in the `cmd` folder of the main repository on how to use
72+
the package. The various make targets are:
18973

190-
## Building statements programmatically
191-
192-
A statement builder can be used for generating SQL statements programmatially. It is intended you use
193-
the following primitves to build your statements:
194-
195-
* `P` is a placeholder for a value, which binds to the corresponding placeholder in `Query` or `Exec` methods;
196-
* `V()` is the value function;
197-
* `N()` is the name function, which corresponds to a table or column name;
198-
* `Q()` is the quote function, which allows insertation or execution or arbitary queries;
199-
* `S()` is the select function, which builds up a SELECT statement;
200-
201-
In order to use these primitives within your code, it is suggested you import the laguage namespace directly into
202-
your code. For example:
203-
204-
```go
205-
package main
206-
207-
import (
208-
. "github.com/djthorpe/go-sqlite/pkg/lang"
209-
)
210-
211-
func main() {
212-
s := S(N("a"),N("b").Distinct().Where(N("a").Is(P))
213-
fmt.Println(s) // Prints SELECT DISTINCT * FROM a,b WHERE a=?
214-
}
215-
```
74+
* `make all` will perform tests, build all examples, the backend API and the frontend web application;
75+
* `make test` will perform tests;
76+
* `make server plugins` will install the backend server and required plugins in the `build` folder;
77+
* `make npm` will compile the frontend web application in a 'dist' folder for each npm module located in the `npm` folder;
78+
* `make clean` will remove all build artifacts.
21679

217-
If the symbols P,V,N,Q or S clash with any symbols in your code namespace, you can import the package
218-
without the dot prefix and refer to the sumbols prefixed with `lang.` instead.
80+
## Contributing & Distribution
21981

220-
## Reading results into a struct, map or slice
82+
__This module is currently in development and subject to change.__
22183

84+
Please do file feature requests and bugs [here](https://github.com/djthorpe/go-sqlite/issues).
85+
The license is Apache 2 so feel free to redistribute. Redistributions in either source
86+
code or binary form must reproduce the copyright notice, and please link back to this
87+
repository for more information:
22288

223-
## Importing data
89+
> Copyright (c) 2021, David Thorpe, All rights reserved.

0 commit comments

Comments
 (0)