|
2 | 2 |
|
3 | 3 | This section gives a comprehensive overview of the Web Service API for launching Bullet JSON queries. |
4 | 4 |
|
5 | | -The JSON API is the actual Query format that is expected by the backend. [The BQL API](api-bql.md) is a more |
6 | | -user-friendly API which can also be used - the Web Service will automatically detect the BQL query and convert the |
7 | | -query to this JSON format before submitting it to the backend. |
| 5 | +The JSON API is the actual Query format that is expected by the backend. [The BQL API](api-bql.md) is a more user-friendly API which can also be used - the Web Service will automatically detect the BQL query and convert the query to this JSON format before submitting it to the backend. With the addition of Post Aggregations and Expressions, |
| 6 | +it is a lot easier to use BQL rather than construct the JSON. The Bullet Web Service also provides [an API](https://github.com/bullet-db/bullet-service/releases/tag/bullet-service-0.4.2) to convert BQL to JSON if you so desire. |
8 | 7 |
|
9 | 8 | * For info on how to use the UI, see the [UI Usage section](../ui/usage.md) |
10 | 9 | * For examples of specific queries see the [Examples](examples.md) section |
11 | 10 |
|
| 11 | +## Constituents of a Bullet Query |
| 12 | + |
12 | 13 | The main constituents of a Bullet JSON query are: |
13 | 14 |
|
14 | 15 | * __filters__, which determine which records will be consumed by your query |
15 | 16 | * __projection__, which determines which fields will be projected in the resulting output from Bullet |
16 | | -* __aggregation__, which allows users to aggregate data and perform aggregation operations |
| 17 | +* __aggregation__, which allows you to aggregate data and perform aggregation operations |
| 18 | +* __postAggregations__, which allows you to perform post aggregations before the result is returned |
17 | 19 | * __window__, which can be used to return incremental results on "windowed" data |
18 | 20 | * __duration__, which determines the maximum duration of the query in milliseconds |
19 | 21 |
|
20 | | -Fields inside maps can be accessed using the '.' notation in queries. For example, |
21 | | - |
22 | | -`myMap.key` |
23 | | - |
24 | | -will access the "key" field inside the "myMap" map. There is no support for accessing fields inside Lists yet. Only the entire object can be operated on for now. |
25 | 22 |
|
26 | 23 | The main constituents of a Bullet query listed above create the top level fields of the Bullet query: |
27 | 24 | ```javascript |
28 | 25 | { |
29 | 26 | "filters": [{}, {}, ...], |
30 | 27 | "projection": {}, |
31 | | - "aggregation": {}. |
| 28 | + "aggregation": {}, |
| 29 | + "postAggregations": [{}, {}, ...], |
32 | 30 | "window": {}, |
33 | 31 | "duration": 20000 |
34 | 32 | } |
35 | 33 | ``` |
36 | 34 |
|
37 | | -We will describe how to specify each of these top-level fields below: |
| 35 | +### Accessing Complex Fields |
| 36 | + |
| 37 | +Fields inside maps and lists can be accessed using the '.' notation in queries. |
| 38 | + |
| 39 | +| Complex Field Type | Example | |
| 40 | +| ------------------------ | ---------------------- | |
| 41 | +| Map of Primitive | `myMap.key` | |
| 42 | +| Map of Map of Primitive | `myMap.myInnerMap.key` | |
| 43 | +| List of Map/Primitive | `myList.0` | |
| 44 | +| List of Map of Primitive | `myListOfMaps.4.key` | |
| 45 | + |
| 46 | +We will now describe how to specify each of these top-level fields below: |
38 | 47 |
|
39 | 48 | ## Filters |
40 | 49 |
|
@@ -102,13 +111,15 @@ The format for a Relational filter is: |
102 | 111 | "operation": "== | != | <= | >= | < | > | RLIKE | SIZEIS | CONTAINSKEY | CONTAINSVALUE" |
103 | 112 | "field": "record_field_name | map_field.subfield", |
104 | 113 | "values": [ |
105 | | - { "kind": "VALUE", "value": "foo"}, |
106 | | - { "kind": "FIELD", "value": "another_record_field_name"} |
| 114 | + { "kind": "VALUE", "type": "BOOLEAN | INTEGER | LONG | FLOAT | DOUBLE | STRING | MAP | LIST", "value": "foo"}, |
| 115 | + { "kind": "FIELD", "type": "BOOLEAN | INTEGER | LONG | FLOAT | DOUBLE | STRING | MAP | LIST", "value": "another_record_field_name"} |
107 | 116 | ] |
108 | 117 | } |
109 | 118 | ``` |
110 | 119 |
|
111 | | -Note that you may specify ```VALUE``` or ```KIND``` currently for the ```kind``` key in the entries in the ```values``` field above, denoting the type of value this is. As a shortcut, you can also specify the following format for ```VALUE``` kind. |
| 120 | +Note that you may specify ```VALUE``` or ```KIND``` currently for the ```kind``` key in the entries in the ```values``` field above, denoting the type of value this is. The ```type``` field is a *optional* and is provided to change the type of the provided ```kind``` (value or field) to the provided type. If you do not provide this type, the value or field provided here will be *casted* to the type of the field (the LHS of the filter). |
| 121 | + |
| 122 | +As a shortcut, you can also specify the following format for ```VALUE``` kind. |
112 | 123 |
|
113 | 124 | ```javascript |
114 | 125 | { |
@@ -303,6 +314,122 @@ The following attributes are supported for ```TOP K```: |
303 | 314 |
|
304 | 315 | Note that the ```K``` in ```TOP K``` is specified using the ```size``` field in the ```aggregation``` object. |
305 | 316 |
|
| 317 | +## Post Aggregations |
| 318 | + |
| 319 | +Post Aggregations allow you to perform some final operations on the aggregated data before it is returned, as the name suggests. It is **optional** and it is performed for each window. For example, you can cast your result field into another type or perform some math. |
| 320 | + |
| 321 | +| Post Aggregation | Meaning | |
| 322 | +| ---------------- | ------- | |
| 323 | +| ORDER BY | Orders your result by your specified fields in ascending or descending order | |
| 324 | +| COMPUTATION | Specify an expression (can be nested expressions) to do math with or cast fields in your result | |
| 325 | + |
| 326 | +The ```"postAggregations"``` field takes a list of these Post Aggregation entries. The __order__ of the various post aggregations in this list determines how they are evaluated. Post aggregations can refer to previous results of post aggregations in the list to chain them. |
| 327 | + |
| 328 | +### ORDER BY |
| 329 | + |
| 330 | +This orders result records based on given fields (in ascending order by default). To sort the records in descending order, use the ```DESC``` ```direction```. You can specify any fields in each record or from previous post aggregations. Note that the ordering is fully typed, so the types of the fields will be used. If multiple fields are specified, ties will be broken from the list of fields from left to right. |
| 331 | + |
| 332 | +```javascript |
| 333 | +{ |
| 334 | + "type": "ORDERBY", |
| 335 | + "fields": ["A", "B"], |
| 336 | + "direction": "DESC" |
| 337 | +} |
| 338 | +``` |
| 339 | + |
| 340 | +### COMPUTATION |
| 341 | + |
| 342 | +This lets you perform arithmetic on the results in a fully nested way. We currently support ```+```, ```-```, ```*``` and ```/``` as operations. The format for this is: |
| 343 | + |
| 344 | +```javascript |
| 345 | +{ |
| 346 | + "type": "COMPUTATION", |
| 347 | + "expression": {} |
| 348 | +} |
| 349 | +``` |
| 350 | + |
| 351 | +#### Expressions |
| 352 | + |
| 353 | +For future extensibility, the ```expression``` in the post aggregation is free form. Currently, we support binary arithmetic operations that can be nested (implying parentheses). This forms a tree of expressions. The leaves of this tree resolve atomic values such as fields or constants. So, there are two kinds of expressions. |
| 354 | + |
| 355 | +##### Binary Expressions |
| 356 | + |
| 357 | +```javascript |
| 358 | +{ |
| 359 | + "operation": "+", |
| 360 | + "left": {}, |
| 361 | + "right": {}, |
| 362 | + "type": "INTEGER | FLOAT | BOOLEAN | DOUBLE | LONG | STRING" |
| 363 | +} |
| 364 | +``` |
| 365 | +, where ```left``` and ```right``` are themselves expressions and ```type``` is used for force cast the result to the given type. |
| 366 | + |
| 367 | +##### Unary Expressions |
| 368 | + |
| 369 | +```javascript |
| 370 | +{ |
| 371 | + "value": { |
| 372 | + "kind": "FIELD | VALUE", |
| 373 | + "value": "foo.bar", |
| 374 | + "type": "INTEGER | FLOAT | BOOLEAN | DOUBLE | LONG | STRING" |
| 375 | + } |
| 376 | +} |
| 377 | +``` |
| 378 | + |
| 379 | +These is the same definition value used for filtering mentioned above and can be used to extract fields from the record as your chosen type or use constants as your chosen type. |
| 380 | + |
| 381 | +If casting __fails__ in any of the expressions, the expression is ignored. |
| 382 | + |
| 383 | +Putting all these together, here is a complete example of post aggregation. This first force computes a new field C, which is the result of doing ```(CAST(foo.bar, LONG) + CAST((CAST(1.2, DOUBLE)/CAST(1, INTEGER)), FLOAT)``` or (C: foo.bar + (1.2/1) for each record in the result window and then orders the result by foo.baz first then by the new the field C. |
| 384 | + |
| 385 | +##### Post Aggregation Example |
| 386 | + |
| 387 | +```javascript |
| 388 | +{ |
| 389 | + "postAggregations":[ |
| 390 | + { |
| 391 | + "type":"COMPUTATION", |
| 392 | + "expression":{ |
| 393 | + "operation":"+", |
| 394 | + "left":{ |
| 395 | + "value":{ |
| 396 | + "kind":"FIELD", |
| 397 | + "value":"foo.bar", |
| 398 | + "type":"LONG" |
| 399 | + } |
| 400 | + }, |
| 401 | + "right":{ |
| 402 | + "operation":"/", |
| 403 | + "left":{ |
| 404 | + "value":{ |
| 405 | + "kind":"VALUE", |
| 406 | + "value":"1.2", |
| 407 | + "type":"DOUBLE", |
| 408 | + } |
| 409 | + }, |
| 410 | + "right":{ |
| 411 | + "value":{ |
| 412 | + "kind":"VALUE", |
| 413 | + "value":"1", |
| 414 | + "type":"INTEGER" |
| 415 | + } |
| 416 | + }, |
| 417 | + "type":"FLOAT" |
| 418 | + }, |
| 419 | + "newName":"C" |
| 420 | + } |
| 421 | + }, |
| 422 | + { |
| 423 | + "type":"ORDERBY", |
| 424 | + " fields":[ |
| 425 | + "foo.baz", "C" |
| 426 | + ], |
| 427 | + "direction":"ASC" |
| 428 | + } |
| 429 | + ] |
| 430 | +} |
| 431 | +``` |
| 432 | + |
306 | 433 | ## Window |
307 | 434 |
|
308 | 435 | The "window" field is **optional** and allows you to instruct Bullet to return incremental results. For example you might want to return the COUNT of a field and return that count every 2 seconds. |
|
0 commit comments