Query result inconsistency btwn 5.2.0 and 5.1.6 on sp2b

[nguyenm100]

Current Behavior

sp2b.txt

we run sp2b perf test against our software nightly and noticed a performance discrepancy while trying out rdf4j 5.2.0. You can read about sp2b here: https://observablehq.com/@lomoramic/sp2b-benchmark-results

effectively, it's a set of benchmark sparql queries and includes a data generator. there about 15 different queries in the test.

many of the queries are returning equivalent number of results but there are a few that return different number of results btwn 5.2.0 and 5.1.6

In particular:

PREFIX rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns#
PREFIX rdfs: http://www.w3.org/2000/01/rdf-schema#
PREFIX foaf: http://xmlns.com/foaf/0.1/
PREFIX dc: http://purl.org/dc/elements/1.1/
PREFIX dcterms: http://purl.org/dc/terms/

SELECT ?yr ?name ?document
WHERE {
?class rdfs:subClassOf foaf:Document .
?document rdf:type ?class .
?document dcterms:issued ?yr .
?document dc:creator ?author .
?author foaf:name ?name
OPTIONAL {
?class2 rdfs:subClassOf foaf:Document .
?document2 rdf:type ?class2 .
?document2 dcterms:issued ?yr2 .
?document2 dc:creator ?author2
FILTER (?author=?author2 && ?yr2<?yr)
} FILTER (!bound(?author2))
}

Is returning 229 triples in 5.2.0 consuming 1037ms and 36 triples in 5.1.6 consuming 147ms against a corpus of ~10000 triples.

I will dig around to see if there are other parity issues btwn other queries.

include 10k triples for local testing.

Expected Behavior

Is this expected behavior for 5.2.0?

Steps To Reproduce

No response

Version

5.2.0

Are you interested in contributing a solution yourself?

None

Anything else?

No response

[kenwenzel]

@nguyenm100 Can you share your repository configuration?

[nguyenm100]

will send along tomorrow when in the office but using lmdb with nothing special.. I see there's an optional statement in the sparql and thought I read there was some fixes to optional handling in one of the upgrades so perhaps smth there?

[kenwenzel]

BTW, the problem with the query is that it creates a very large interrnediate result because there are no common variables between the mandatory and optional part. Wouldn't it be better to use the same variable for ?author and ?author2?

[nguyenm100]

just part of the sp2b benchmark set of queries. some intentionally written to push 3stores perf and optimizers.

[kenwenzel]

@hmottestad Do we have an optimizer for ?author=?author2?

[nguyenm100]

so do we think 5.2.0 fixed a bug and is producing the correct results over 5.1.6 and prior? that's prolly more important right now tbh

[kenwenzel]

Yes, it is correct in 5.2.0
Take a look in this paper (query 6):
https://arxiv.org/pdf/0806.4627

[nguyenm100]

ah ok missed that in the paper. will go through and check the numbers and close out the issue if everything looks good. tx

[nguyenm100]

checked numbers for 10k triples and 1m triples and it matches! Note: at 1m triples, Q6 times out (we set a timeout of 5m)

[kenwenzel]

@nguyenm100 Are you running the benchmark on a rather slow CI machine?

[nguyenm100]

we have a perf cluster of decent quality vm 96gb mem 16 cpu...

[hmottestad]

@hmottestad Do we have an optimizer for ?author=?author2?

I don't think we do. Issue is that we don't know if ?author must be an IRI or if it can be a literal. I know Stardog had an annotation that could be used to tell the optimiser that it's always an IRI and it would merge the two variables into one. We don't support query hints, so we can't do the same.

To be honest though, cross joins are rare in production and I don't know if there is much point in optimising for them since the user can so easily fix this themselves by rewriting the query.

@nguyenm100 would you mind trying to run your benchmarks on this branch here to see if it's any faster? #5549 You need to use isolation level READ_COMMITTED or lower.