feat: Scout Search Implementation

[frknakk]

Fixes #3077

@yajra
What do you think? Feel free to let me know if i should change some parts.
disableOrdering js script: We could either integrate that into laravel-datatables-html or add it into some docs for people to implement on their own (if they want to). I could also leave that part out completely. What do you think?

If it's possible to somehow add custom scss/js code into laravel-datatables-html i would also do 2 more pull requests for nice optimizations that i'm using since a long time on my own projects.

Also it would be good to add an example usage into docs:

public function dataTable(QueryBuilder $query): EloquentDataTable
{
	return (new EloquentDataTable($query))
		// Enable scout search for eloquent model
		->enableScoutSearch(Product::class)
		
		// Add filters to scout search
		->scoutFilter(function (string $keyword) {
			return 'region IN ["Germany", "France"]'; // Meilisearch
                        return 'region:Germany OR region:France'; // Algolia
		})
		
		// Add filters to default search
		->filter(function (QueryBuilder $query, bool $scout_searched) {
			if (!$scout_searched)
			{
				// Filter already added for scout search
				$query->whereIn('region', ['Germany', 'France']);
			}

			// Stock is not indexed so it has to be filtered after the initial scout search
			$query->where('stock', '>', 50);
		}, true)

		->setRowId('id');
}

---

Disable ordering (JS)
When the scout search was performed, the code sets a disableOrdering: true flag in the ajax response. The JS part has to fetch that and disable the user ordering. Otherwise the user gets confused because the ordering is overwritten by the relevancy ordering of the search results.

$('#products-table').on('xhr.dt', function (e, settings, json, xhr) {
	if (json == null || !('disableOrdering' in json)) return;

	let table = LaravelDataTables[$(this).attr('id')];
	if (json.disableOrdering) {
		table.settings()[0].aoColumns.forEach(function(column) {
			column.bSortable = false;
			$(column.nTh).removeClass('sorting_asc sorting_desc sorting').addClass('sorting_disabled');
		});
	} else {
		let changed = false;
		table.settings()[0].aoColumns.forEach(function(column) {
			if (column.bSortable) return;
			column.bSortable = true;
			changed = true;
		});
		if (changed) {
			table.draw();
		}
	}
});

[yajra]

Was able to test and works well 🎉. However, it seems like the scout search is failing when the global search is off but it seems intended since meilisearch will basically do the wild search for us?

		->filter(function (QueryBuilder $query, bool $scout_searched) {

		}, false)

[frknakk]

Was able to test and works well 🎉. However, it seems like the scout search is failing when the global search is off but it seems intended since meilisearch will basically do the wild search for us?

		->filter(function (QueryBuilder $query, bool $scout_searched) {

		}, false)

Yes, that's intended. Adding column search for scout would make the code a mess and difficult to maintain. Additionally i don't think there is a relevant use case for that. In 99% of the cases setting searchableAttributes on meilisearch/algolia will probably be enough.

[yajra]

Where do we put the disableOrdering js script?

I think we should not force disable the ordering function and let the developer handle it via javascript ordering: false.

In most of our cases, our users want a specific ordering but it will be a big boost in performance if we can delegate the searching via meilisearch.

[frknakk]

Where do we put the disableOrdering js script?

I think we should not force disable the ordering function and let the developer handle it via javascript ordering: false.

In most of our cases, our users want a specific ordering but it will be a big boost in performance if we can delegate the searching via meilisearch.

I think you misunderstood the purpose of the script.

When searching via Meilisearch, it doesn't make any sense to keep the user ordering (e.g. sort by name asc), because then there will be dozens of unrelevant entries before you get the desired result. Because of that, when searching via Meilisearch the user ordering gets ignored serverside and it forces the order of meilisearch results (= entries that are matching best are shown first).

The disableOrdering js script is just a frontend script, that temporarily hides the ordering symbols when meilisearch was used. Because otherwise you can click on all columns to change ordering but nothing will change. The script also enables the ordering again when the search was deleted or meilisearch is not active.

[yajra]

When searching via Meilisearch, it doesn't make any sense to keep the user ordering (e.g. sort by name asc), because then there will be dozens of unrelevant entries before you get the desired result. Because of that, when searching via Meilisearch the user ordering gets ignored serverside and it forces the order of meilisearch results (= entries that are matching best are shown first).

Good point, I was thinking only of search results (discarding relevance) but with ordering as defined by the users.

The disableOrdering js script is just a frontend script, that temporarily hides the ordering symbols when meilisearch was used. Because otherwise you can click on all columns to change ordering but nothing will change. The script also enables the ordering again when the search was deleted or meilisearch is not active.

I was trying to do a script for this and can't figure it out ^_^. Do you have a sample script for this?

[frknakk]

I was trying to do a script for this and can't figure it out ^_^. Do you have a sample script for this?

Yes, it's above in my initial message 😄

[yajra]

OMG! Apologies for overlooking that script 😅. The functionality makes full sense to me now.

I think it would be nice if we could have that script integrated into the HTML builder but with docs too for those not using the builder. Maybe we can add new script template for scout to integrate this?

scout.blade.php

$(function(){
    window.{{ config('datatables-html.namespace', 'LaravelDataTables') }}=window.{{ config('datatables-html.namespace', 'LaravelDataTables') }}||{};
    window.{{ config('datatables-html.namespace', 'LaravelDataTables') }}["%1$s"]=$("#%1$s").DataTable(%2$s);

    $('#%1$s').on('xhr.dt', function (e, settings, json, xhr) {
        if (json == null || !('disableOrdering' in json)) return;

        let table = LaravelDataTables[$(this).attr('id')];
        if (json.disableOrdering) {
            table.settings()[0].aoColumns.forEach(function(column) {
                column.bSortable = false;
                $(column.nTh).removeClass('sorting_asc sorting_desc sorting').addClass('sorting_disabled');
            });
        } else {
            let changed = false;
            table.settings()[0].aoColumns.forEach(function(column) {
                if (column.bSortable) return;
                column.bSortable = true;
                changed = true;
            });
            if (changed) {
                table.draw();
            }
        }
    });
});

And then maybe a shortcut builder method to set the template to scout?

->setTableId('products-table')
->template('scout');

[yajra]

I also created a poc repo to test this. Thanks!

[frknakk]

@yajra Do you have any more suggestions for improvement? Looks good to me :)

Last thing missing is a docs section that has a quick start with the usage examples from both this pull request and the new one in laravel-datatables-html.

Feel free to test, fix and uncomment the missing database drivers. The code that's there should work theoretically, but i couldn't test it yet.

[yajra]

The changes look good! Will test on other DB drivers.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[yajra]

Confirmed working on SQLite. The CASE WHEN SQL doesn't seem to work with Oracle but it's ok.

Will also try PGSQL by tomorrow.

[yajra]

PostgreSQL does not work. However was able to do a code that works for both pgsql and oracle. Can't test on sqlsrv though.

    protected function applyFixedOrderingToQuery(string $keyName, array $orderedKeys)
    {
        $connection = $this->getConnection();
        $driverName = $connection->getDriverName();

        // Escape keyName and orderedKeys
        $keyName = $connection->getQueryGrammar()->wrap($keyName);
        $orderedKeys = collect($orderedKeys)
            ->map(function ($value) use ($connection) {
                return $connection->escape($value);
            });

        switch ($driverName) {
            case 'mysql':
                $this->query->orderByRaw("FIELD($keyName, ".$orderedKeys->implode(',').')');
                return true;

            case 'pgsql':
            case 'oracle':
                $this->query->orderByRaw(
                    "CASE "
                    .
                    $orderedKeys
                        ->map(fn($value, $index) => "WHEN $keyName=$value THEN $index")
                        ->implode(' ')
                    .
                    " END"
                );
                return true;

            case 'sqlite':
            case 'sqlsrv':
                $this->query->orderByRaw(
                    "CASE $keyName "
                    .
                    $orderedKeys
                        ->map(fn($value, $index) => "WHEN $value THEN $index")
                        ->implode(' ')
                    .
                    " END"
                );
                return true;

            default:
                return false;
        }
    }

Merging this and will apply the patch in a new PR. Thanks a lot for your patience and contribution! 🍻 🎉

[yajra]

Released on v10.11.0 🚀 Thanks!