Examine Facets proposal

[nikcio]

Examine Facets proposal

Linked PR	#311
Linked PR	#312
Linked PR	#313

What is faceted search?

Faceted search is a technique that involves augmenting traditional search techniques with a faceted navigation system, allowing users to narrow down search results by applying multiple filters based on faceted classification of the items. It is sometimes referred to as a parametric search technique. A faceted classification system classifies each information element along multiple explicit dimensions, called facets, enabling the classifications to be accessed and ordered in multiple ways rather than in a single, pre-determined, taxonomic order. (Source)

Description

This proposal is on the implementation of faceted search in Examine. The proposal is mostly based on finding reasonable interface abstractions the best approach for building the feature and less on specific implementation details.

Previous implementation

Facets are available for Examine when targeting .NET framework via the Examine.Facets package by Callum Whyte. This proposal is based on the implementation of that package.

Motivation

I'm currently working on a project which would have a great use for Examine facets.

Approach

1. Externalize the Faceting features

The first approach is to externalize the Faceting features to a separate Nuget package in the same way Examine.Facets works a POC of this approach can be seen here: POC: Examine.Facets

2. Internalize the Faceting features (I think this is the best approach)

The second approach is to make faceting available directly in the existing Examine package and existing classes this would make it possible to avoid creating the following implementations and instead add the features to the existing classes:

• FacetBooleanOperation
• FacetQuery
• FacetSearchExecutor
• FacetSearchQuery
• FacetSearchResults
• FacetSearcher

This approach, therefore, allows the default searcher to do a faceted search and this will lower the barrier to entry because a developer wouldn't have to register a separate searcher and explicitly use this searcher for faceted searching. As seen in this example:

Example 1 (From the existing Examine.Facets package by Callum)

// Setup
if (_examineManager.TryGetIndex("CustomIndex", out IIndex index))
{
    if (index is LuceneIndex luceneIndex)
    {
        var searcher = new FacetSearcher(
            "FacetSearcher",
            luceneIndex.GetIndexWriter(),
            luceneIndex.DefaultAnalyzer,
            luceneIndex.FieldValueTypeCollection
        );

        _examineManager.AddSearcher(searcher);
    }
}

// Fetching a searcher
_examineManager.TryGetSearcher("FacetSearcher", out ISearcher searcher);

Example 2 (From a test in the POC)

TrackingIndexWriter writer = indexer.IndexWriter;
var searcherManager = new SearcherManager(writer.IndexWriter, true, new SearcherFactory());
var searcher = new FacetSearcher(nameof(FacetSearcher), searcherManager, analyzer, indexer.FieldValueTypeCollection);

Example source

Structure

Bases

Searching

public interface IFacetField
{
    /// <summary>
    /// The field name
    /// </summary>
    string Field { get; }

    /// <summary>
    /// The field to get the facet field from
    /// </summary>
    string FacetField { get; set; }
}

Searching results

public interface IFacetValue
{
    /// <summary>
    /// The label of the facet value
    /// </summary>
    string Label { get; }

    /// <summary>
    /// The occurrence of a facet field
    /// </summary>
    float Value { get; }
}

public interface IFacetResult : IEnumerable<IFacetValue>
{
    /// <summary>
    /// Gets the facet for a label
    /// </summary>
    /// <param name="label"></param>
    /// <returns></returns>
    IFacetValue Facet(string label);
}

Example facet result

Tags:

Software (121)
People (20)
Packages (2)

public interface IFacetResults
{
    /// <summary>
    /// Facets from the search
    /// </summary>
    IDictionary<string, IFacetResult> Facets { get; }
}

Extensions

/// <summary>
/// Get the values for a particular facet in the results
/// </summary>
public static IFacetResult GetFacet(this ISearchResults searchResults, string field)
{
    // Implementation
}

/// <summary>
/// Get all of the facets in the results
/// </summary>
public static IEnumerable<IFacetResult> GetFacets(this ISearchResults searchResults)
{
    // Implementation
}

Types of facets

Sources of information about Lucene's facet search are:

• Facets with Lucene
• Apache Lucene Faceted Search User's Guide

String Facet

Allows for counting the documents that share the same string value.

New FieldDefinitionTypes:

• FacetFullText
• FacetFullTextSortable

Extends the existing FullText and FullTextSortable type and adds the required SortedSetDocValuesFacetField to the indexed document. Without this field, SortedSetDocValuesFacetCounts will not work.

New query methods

On IQuery

/// <summary>
/// Add a facet string to the current query
/// </summary>
IFacetQueryField Facet(string field);

/// <summary>
/// Add a facet string to the current query, filtered by value
/// </summary>
IFacetQueryField Facet(string field, string value);

/// <summary>
/// Add a facet string to the current query, filtered by multiple values
/// </summary>
IFacetQueryField Facet(string field, string[] values);

public interface IFacetQueryField : IBooleanOperation
{
    /// <summary>
    /// Maximum number of terms to return
    /// </summary>
    IFacetQueryField MaxCount(int count);

    /// <summary>
    /// Sets the field where the facet information will be read from
    /// </summary>
    IFacetQueryField FacetField(string fieldName)
}

New IFacetField

public interface IFacetFullTextField : IFacetField
{
    /// <summary>
    /// Maximum number of terms to return
    /// </summary>
    int MaxCount { get; set; }

    /// <summary>
    /// Filter values
    /// </summary>
    string[] Values { get; set; }
}

Facets config / New index methods - Optional addition. Properly not the most used feature

FacetsConfig allows for setting some values in the index which are useful for faceting API docs

On LuceneIndexOptions

public FacetsConfig FacetConfig { get; set; }

This will make it possible to set the facet configuration on the specific index and reuse it when searching.

Methods to change the field used when reading facets (default is $facets which is where all facet values are indexed if FacetsConfig.SetIndexFieldName(dimName, indexFieldName) is not called.):

See IFacetQueryField (It's not possible to specify the reading field in range facets)

This will make it possible to set the faceting field per facet field giving the most flexibility when composing a query,

Note: The FacetConfig will also need to be available at search time in the searchExecutor to be used in the constructor when using Taxonomy

Numeric Range Facet

Used with numbers to build range facets. For example, it would group documents of the same price range.

Double Range

New FieldDefinitionTypes:

• FacetDouble
• FacetFloat

Extends the existing Double and Float type and adds the required DoubleDocValuesField and SingleDocValuesField respectively, aswell as the SortedSetDocValuesFacetField to enable string like faceting, to the indexed document. Without the fields, DoubleDocValuesField and SingleDocValuesField faceting will not work.

New query methods

On IQuery

/// <summary>
/// Add a range facet to the current query
/// </summary>
IFacetRangeQueryField Facet(string field, DoubleRange[] doubleRanges);

public interface IFacetDoubleRangeQueryField : IBooleanOperation
{
    /// <summary>
    /// Sets if the range query is on <see cref="float"/> values
    /// </summary>
    /// <param name="isFloat"></param>
    /// <returns></returns>
    IFacetDoubleRangeQueryField IsFloat(bool isFloat);
}

New IFacetField

public interface IFacetDoubleField : IFacetField
{
    DoubleRange[] DoubleRanges { get; set; }
}

Long Range / Numeric range

New FieldDefinitionTypes:

• FacetInt
• FacetLong
• FacetDateTime
• FacetDateYear
• FacetDateMonth
• FacetDateDay
• FacetDateHour
• FacetDateMinute

Extends the existing types and adds the required NumericDocValuesField, aswell as the SortedSetDocValuesFacetField to enable string like faceting, to the indexed document. Without the fields, NumericDocValuesField faceting will not work.

New query methods

On IQuery

/// <summary>
/// Add a range facet to the current query
/// </summary>
IFacetRangeQueryField Facet(string field, Int64Range[] longRanges);

public interface IFacetLongRangeQueryField : IBooleanOperation
{
}

New IFacetField

public interface IFacetLongField : IFacetField
{
    Int64Range[] LongRanges { get; set; }
}

Taxonomy Facet

Doing Taxonomy requires using a speciffic writer (DirectoryTaxonomyWriter) and is therefore out of the scope of this proposal.

See more at: https://norconex.com/facets-with-lucene/

---

What now

• Decide on the best approach for the feature.
• Implement the API proposal.
• Add documentation on the new facet API.
• Refine the API proposal to be acceptable if it's not already.
• Release version with the proposal.

[bergmania]

Wow @nikcio 🤩

Good work on this proposal :)

From a user point of view, I also like the approach 2 the most, but it requires more of the implementations.

Sorry for my lack of knowledge, but in what case will IFacetValue.Value not be an integer/long?

[nikcio]

@bergmania in Apache Lucene Faceted Search User's Guide there's written the following under 2.2 Facet Associations

So far we've discussed categories as binary features, where a document either belongs to a category, or not.

While counts are useful in most situations, they are sometimes not sufficiently informative for the user, with respect to deciding which subcategory is more important to display.

For this, the facets package allows to associate a value with a category. The search time interpretation of the associated value is application dependent. For example, a possible interpretation is as a match level (e.g., confidence level). This value can then be used so that a document that is very weakly associated with a certain category will only contribute little to this category's aggregated weight.

So it's possible to use this feature to get floating values. I'm not quite sure myself excatly how this is configured and used so I mostly just based the value type off the FacetResult type from the Lucene.Net.Facet package because it's the output of the different facet readers in Lucene.NET.

[nikcio]

@Shazwazza I've created some PR's that could implement this proposal and add some great functionality / Documentation to Examine. Please let me know if I can do anything to help the PR's along.

PRs:
#311 (Facet implementation)
#312 (XML docs on facets and in the project where missing)
#313 (Nullable feature for project and facets feature)

[Shazwazza]

@nikcio Sorry to keep you waiting on this one, I have all of these starred in my inbox and will get to them soon, just a bit swamped this week.

[nzdev]

Added support for efficient deep paging (SearchAfter) for faceted and non faceted search #321.

[Shazwazza]

@nikcio I started having a look through all this yesterday and so far all I can say is what an amazing job you've done so far. I'll keep reviewing over this week and we can determine if there are any tweaks necessary. @nzdev also thanks a ton for your recent PRs and help. Hopefully can get this all merged in for xmas/NY and get what will most likely be a new major release out.

[nzdev]

WIP https://github.com/nzdev/Examine/tree/v3/feature/facet-taxonomy Facet Taxonomy Index support. Needed for Hierarchical facets. Also is something like 20%-25% faster according to Lucene.Net docs.

[dealloc]

Has there been an update on this?

[nikcio]

@dealloc I believe #311 is very close to being done. Just waiting for the stars to align😅🤞

(I'm a little unsure what we are waiting on to be honest 😬😅)

[nikcio]

Status:

Here is how I see the status of the Examine repo. What do you think @Shazwazza and @nzdev?

Merged PRs (Release/4.0)

• Implementation of Facets (#310) #311
• Facets XML (#311) #312
• Facets nullable (#311 & #307) #313
• Support for efficient deep paging in lucene.net #320
• Support for deep paging for Facets #321
• Taxonomy Index for Facets #323
• Delete ReaderStatus #336

Needs to be merged from the Release/3.0 branch - This is done in #345

• test: Add .NET 7 target to tests #318
• CI - Add your build.yml change to the release branch 3.0 #343

Still needs to be merged into Release/4.0

• Merges the changes from the release/3.0 branch to release/4.0 branch #345
• Additional ServiceCollection extension methods #339 - This gives better backwards compatibility for Umbraco

Still needs to be done

[This list is based on #345 being merged into release/4.0]

The massive amount of work that has been going on have created some warnings in the project where some are more important than others. Here is a run down of the ones I think we should look at before a stable 4.0 release.

• Null reference warnings
  • LuceneIndex.cs - line 1212
  • LuceneIndex.cs - line 1288
• XML warnings
  • There's a few places where there's still missing a little bit of XML docs.

Other PRs that could properly come in another future release:

• Suggestions and Spellchecking #326
• V4 Similarity support #327
• Spatial API #328
• Update packages #332
• Test case for nested range query #334
• Draft: feat: added custom score queries #338
• Allow mix of single objects and enumerables in valueset #340

Stale PRs?

• Remove azure directory #226
• build(deps): Bump nokogiri from 1.13.9 to 1.13.10 in /docs #319

Preview/Alpha/Beta release

I think the best way to get some kind of feel of what still needs to be done is only possible by making an early release and hear around the community for people to test it out. - This can be done when the current PRs in the "Still needs to be merged into Release/4.0" and #345 are merged in. @Shazwazza