The raw() method always returns the first 250 results

[bessone]

Scout Version

10.19.0

Scout Driver

Typesense

Laravel Version

12.28.1

PHP Version

8.4

Database Driver & Version

No response

SDK Version

No response

Meilisearch CLI Version

No response

Description

When performing a search that returns more than 250 results (for example, 300), the raw() method returns only the first 250 results.

The Typesense documentation (https://typesense.org/docs/29.0/api/search.html#pagination-parameters) states that using the per_page parameter returns a maximum of 250 hits.
"NOTE: Only up to 250 hits (or groups of hits when using group_by) can be fetched per page."

The problem with scout is that it doesn't return the results from the current page, but always the first 250.

For example, when paging 50 results per page, the 300 results returned only display hits on the first 5 pages; the sixth page won't work.

Steps To Reproduce

`$results= Model::search($query)->paginate(50)->withQueryString();

$raw = $search->raw();

// Add hits to model result
foreach ($results &$result) {
foreach($raw['hits'] as $hit) {
if ($hit['document']['id'] == (string) $result->id) {
$case->hit = $hit;
break;
}
}
}`

[tharropoulos]

Currently investigating. Will report soon

[tharropoulos]

The behavior you’re seeing comes from calling raw() without page context and from the repro using an undefined variable.

In your snippet, $search is never defined. raw() belongs to the Scout Builder you get from Model::search($query), not to the paginator returned by paginate(). If you call raw() on a new builder without page and per_page, Typesense defaults to page = 1 with per_page capped at 250, so you always get the first 250 hits.

To fetch raw results that match the current paginator page, you can do either of the following.

use the raw-aware paginator:

$paginator = Model::search($query)->paginateRaw(50); // or ->simplePaginateRaw(50)

$raw = $paginator->items(); // raw response for the current page
// $raw['hits'] contains the hits for this page

reuse the paginator’s page and per page when calling raw():

$models = Model::search($query)->paginate(50);

$raw = Model::search($query)->options([
  'per_page' => $models->perPage(),
  'page' => $models->currentPage(),
])->raw();

align raw hits to Eloquent models efficiently:

$hitsById = collect($raw['hits'])->keyBy('document.id');

foreach ($models as $model) {
  $model->hit = $hitsById[(string) $model->getKey()] ?? null;
}

This is the pattern that causes the mismatch:

$models = Model::search($query)->paginate(50);
$raw = Model::search($query)->raw(); // new builder defaults to page 1, so first 250 hits only

Typesense still limits per_page to 250. If you are on page 6 with 50 per page, make sure the raw request includes page => 6 so you get the correct slice.

[bessone]

Thanks for your clear answer.
Showing hits is a very useful feature, but I didn't find much information about it, so I did what I could with my knowledge.

As for my snippet, I simplified it to make it simpler, but I overdid it. In fact, my code includes the $search variable, from which I retrieve $results with paginate() and after the raw().

Using the raw-aware paginator I got just what I need, for every page with 50 results the 50 correct hits.
So I no longer need to call raw(), with $paginator->items() I have exactly what I need.

I really appreciate the time you took to find this solution. Grazie!

[tharropoulos]

@bessone Sorry for the ping, but have you resolved this issue? If so, could you close this down?

[jprehm]

@tharropoulos I'm having the same issue, and seeing that highlights arrays are empty. This is because the returned items do not exist in the search results.

$items = Event::search($request->string('search'))->get();

dump($items);

$data = Event::search($request->string('search'))->raw();
dd($data);

The first collection is from the dump, which contains the search results. Which has 175 items, but the raw() is showing 2500/2500 matches.

When viewing the hits, there are no highlights for any of the hits.

[jprehm]

I found this is caused by using $request->string($search)->raw() updating to $request->input($search)->raw()` returns the expected results.

When using $request->string($search) the query is not converted to a string, but is a stringable object.