Full Text Searching

Full Text Indexes in Marten are built based on GIN or GiST indexes utilizing Postgres built in Text Search functions. This enables the possibility to do more sophisticated searching through text fields.

WARNING

To use this feature, you will need to use PostgreSQL version 13 or above, as this is the minimum version supported by Marten - this is also the data type that Marten use to store it's data.

Defining Full Text Index through Store options

Full Text Indexes can be created using the fluent interface of StoreOptions like this:

• one index for whole document - all document properties values will be indexed

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex();
});

^{snippet source | anchor}

INFO

If you don't specify language (regConfig) - by default it will be created with 'english' value.

• single property - there is possibility to specify specific property to be indexed

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex(d => d.FirstName);
});

^{snippet source | anchor}

• single property with custom settings

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex(
        index =>
        {
            index.Name = "mt_custom_italian_user_fts_idx";
            index.RegConfig = "italian";
        },
        d => d.FirstName);
});

^{snippet source | anchor}

• multiple properties

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex(d => d.FirstName, d => d.LastName);
});

^{snippet source | anchor}

• multiple properties with custom settings

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex(
        index =>
        {
            index.Name = "mt_custom_italian_user_fts_idx";
            index.RegConfig = "italian";
        },
        d => d.FirstName, d => d.LastName);
});

^{snippet source | anchor}

• more than one index for document with different languages (regConfig)

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>()
        .FullTextIndex(d => d.FirstName) //by default it will use "english"
        .FullTextIndex("italian", d => d.LastName);
});

^{snippet source | anchor}

Defining Full Text Index through Attribute

Full Text Indexes can be created using the [FullTextIndex] attribute like this:

• one index for whole document - by setting attribute on the class all document properties values will be indexed

[FullTextIndex]
public class Book
{
    public Guid Id { get; set; }

    public string Title { get; set; }

    public string Author { get; set; }

    public string Information { get; set; }
}

^{snippet source | anchor}

• single property

public class UserProfile
{
    public Guid Id { get; set; }

    [FullTextIndex] public string Information { get; set; }
}

^{snippet source | anchor}

INFO

If you don't specify regConfig - by default it will be created with 'english' value.

• single property with custom settings

public class UserDetails
{
    private const string FullTextIndexName = "mt_custom_user_details_fts_idx";

    public Guid Id { get; set; }

    [FullTextIndex(IndexName = FullTextIndexName, RegConfig = "italian")]
    public string Details { get; set; }
}

^{snippet source | anchor}

• multiple properties

public class Article
{
    public Guid Id { get; set; }

    [FullTextIndex] public string Heading { get; set; }

    [FullTextIndex] public string Text { get; set; }
}

^{snippet source | anchor}

INFO

To group multiple properties into single index you need to specify the same values in IndexName parameters.

• multiple indexes for multiple properties with custom settings

public class BlogPost
{
    public Guid Id { get; set; }

    public string Category { get; set; }

    [FullTextIndex] public string EnglishText { get; set; }

    [FullTextIndex(RegConfig = "italian")] public string ItalianText { get; set; }

    [FullTextIndex(RegConfig = "french")] public string FrenchText { get; set; }
}

^{snippet source | anchor}

Text Search

Postgres contains built in Text Search functions. They enable the possibility to do more sophisticated searching through text fields. Marten gives possibility to define full text indexes and perform queries on them. Currently four types of full Text Search functions are supported:

• regular Search (to_tsquery)

var posts = (await session.Query<BlogPost>()
    .Where(x => x.Search("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

• plain text Search (plainto_tsquery)

var posts = (await session.Query<BlogPost>()
    .Where(x => x.PlainTextSearch("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

• phrase Search (phraseto_tsquery)

var posts = (await session.Query<BlogPost>()
    .Where(x => x.PhraseSearch("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

• web-style Search (websearch_to_tsquery, supported from Postgres 11+

var posts = (await session.Query<BlogPost>()
    .Where(x => x.WebStyleSearch("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

All types of Text Searches can be combined with other Linq queries

var posts = (await session.Query<BlogPost>()
    .Where(x => x.Category == "LifeStyle")
    .Where(x => x.PhraseSearch("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

They allow also to specify language (regConfig) of the text search query (by default english is being used)

var posts = (await session.Query<BlogPost>()
    .Where(x => x.PhraseSearch("somefilter", "italian"))
    .ToListAsync());

^{snippet source | anchor}

Weighted Full Text Indexes and Relevance Ranking 9.31

By default a full text index concatenates its members into one flat vector, so a match in a title is exactly as relevant as a match in a long description. PostgreSQL can do better: setweight() labels each member, and ts_rank() then scores a title match above a body match.

Weighting the index

opts.Schema.For<Achievement>().WeightedFullTextIndex(idx => idx
    .Weighted(a => a.Title, TextSearchWeight.A)
    .Weighted(a => a.Tagline, TextSearchWeight.B)
    .Weighted(a => a.Description, TextSearchWeight.C));

which produces:

create index mt_doc_achievement_idx_fts on public.mt_doc_achievement using gin ((
  setweight(to_tsvector('english', coalesce(data ->> 'Title', '')), 'A') ||
  setweight(to_tsvector('english', coalesce(data ->> 'Tagline', '')), 'B') ||
  setweight(to_tsvector('english', coalesce(data ->> 'Description', '')), 'C')
));

Note that each member is converted to a vector separately and the vectors are concatenated. That is what setweight requires, and it is why weighting is a distinct method rather than an option on FullTextIndex() — the DDL is a different shape, not a configured variant of the same one.

There are four weight labels, A through D. They carry no numbers of their own; the rank function supplies those, defaulting to {D, C, B, A} = {0.1, 0.2, 0.4, 1.0}. D is what an unlabelled vector already means, so it is the default.

WARNING

A weighted index needs at least two different weights. Weighting expresses a relative ordering, so one weight — or the same weight on every member — ranks nothing, and Marten refuses it at configuration time rather than emitting an index that looks weighted and is not.

Ordering by relevance

OrderByTextRank() sorts by ts_rank(), highest first:

var results = await session
    .Query<Achievement>()
    .Where(a => a.WebStyleSearch(term))
    .OrderByTextRank(term, TextSearchFunction.WebStyle)
    .ToListAsync();

Use ThenByTextRank() to add relevance after another ordering.

The search function is explicit and has no default. Ranking with a different tsquery function than the Where clause used is always a mistake, and a friendly default would be occasionally wrong and silently so — so pass the one that matches your filter: Plain, Phrase, WebStyle or Raw.

The rank resolves the same tsvector the Where clause matched on, read from the index definition. That is deliberate rather than incidental: a rank computed over a different vector than the filter matched on returns rows in an order that looks plausible and means nothing, which is a far quieter failure than returning the wrong rows.

TIP

The search term is passed as a parameter rather than interpolated into the SQL.

Costs worth knowing before you use it

GIN indexes cannot order. ts_rank is a post-filter sort over everything the @@ matched, so ranking a broad query sorts a lot of rows. Narrow the filter before reaching for the rank.

Adding weights to an existing index is not a free migration. Weights change the index expression, so Weasel drops and recreates the index on the next schema apply. On a large table that is an outage rather than a migration — consider building it out of band, as described in the ignore-indexes documentation.

One index per text search configuration. If a document has two full text indexes sharing a regConfig, Marten cannot tell which one a search means and will refuse the query rather than pick one. Give them different regConfig values, or register a single index covering every member you want to search.

Partial text search in a multi-word text (NGram search)

Marten provides the ability to search partial text or words in a string containing multiple words using NGram search. This is quite similar in functionality to NGrams in Elastic Search. As an example, we can now accurately match rich com text within Communicating Across Contexts (Enriched). NGram search uses English by default. NGram search also encompasses and handles unigrams, bigrams and trigrams. This functionality is added in v5.

var result = await session
    .Query<User>()
    .Where(x => x.UserName.NgramSearch(term))
    .ToListAsync();

^{snippet source | anchor}

var store = DocumentStore.For(_ =>
{
    _.Connection(Marten.Testing.Harness.ConnectionSource.ConnectionString);
    _.DatabaseSchemaName = "ngram_test";

    // This creates an ngram index for efficient sub string based matching
    _.Schema.For<User>().NgramIndex(x => x.UserName);
});

await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync();

await using var session = store.LightweightSession();

string term = null;
for (var i = 1; i < 4; i++)
{
    var guid = $"{Guid.NewGuid():N}";
    term ??= guid.Substring(5);

    var newUser = new User(i, $"Test user {guid}");

    session.Store(newUser);
}

await session.SaveChangesAsync();

var result = await session
    .Query<User>()
    .Where(x => x.UserName.NgramSearch(term))
    .ToListAsync();

^{snippet source | anchor}

var result = await session
    .Query<User>()
    .Where(x => x.Address.Line1.NgramSearch(term))
    .ToListAsync();

^{snippet source | anchor}

INFO

Especially when writing unit tests which uses NGramSearch ensure to call await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync(); which will add the required system functions to database for using it.

WARNING

When you call NGramSearch, ensure to call it on a string property/field of the document rather than on the document itself.

• await session.Query<User>().Where(x => x.Address.Line1.NgramSearch(term)) - accessing NGramSearch on any string property is the right usage. ✅
• await session.Query<User>().Where(x => x.NgramSearch(term)) Don't target the NGramSearch on the document User, it won't work. ❌
• await session.Query<User>().Where(x => x.Name.ToLower().NgramSearch(term)) Don't target the NGramSearch on a computed property on the document User i.e. x.Name.ToLower(), it won't work. ❌

NGram search on non-English text 7.39.5

If you want to use NGram search on non-English text, Marten provides a mechanism via an opt-in storeOptions.Advanced.UseNGramSearchWithUnaccent = true which uses Postgres unaccent extension for applying before creating ngrams and on search input for a better multilingual experience. Check the sample code below:

var store = DocumentStore.For(_ =>
{
   _.Connection(Marten.Testing.Harness.ConnectionSource.ConnectionString);
   _.DatabaseSchemaName = "ngram_test";
   _.Schema.For<User>().NgramIndex(x => x.UserName);
   _.Advanced.UseNGramSearchWithUnaccent = true;
});

await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync();

await using var session = store.LightweightSession();
//The ngram uðmu should only exist in bjork, if special characters ignored it will return Umut
var umut = new User(1, "Umut Aral");
var bjork = new User(2, "Björk Guðmundsdóttir");

//The ngram øre should only exist in bjork, if special characters ignored it will return Chris Rea
var kierkegaard = new User(3, "Søren Kierkegaard");
var rea = new User(4, "Chris Rea");

session.Store(umut);
session.Store(bjork);
session.Store(kierkegaard);
session.Store(rea);

await session.SaveChangesAsync();

var result = await session
   .Query<User>()
   .Where(x => x.UserName.NgramSearch("uðmu") || x.UserName.NgramSearch("øre"))
   .ToListAsync();

^{snippet source | anchor}

INFO

Especially when writing unit tests which uses NGramSearch ensure to call await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync(); which will add the required system functions as well the unaccent extension to database for use (when UseNGramSearchWithUnaccent is set to true).

NGram Search Across Multiple Properties 7.39.5

In many cases, you may want to perform partial text search across multiple fields like UserName, FirstName, and LastName.

A naive approach might be to apply individual Ngram indexes and search each field separately:

Don't do this

This results in multiple indexes per document and requires complex LINQ queries to combine the results — inefficient and hard to maintain.

// Inefficient and verbose
var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // Too many indexes
    _.Schema.For<User>().NgramIndex(d => d.UserName);
    _.Schema.For<User>().NgramIndex(d => d.FirstName);
    _.Schema.For<User>().NgramIndex(d => d.LastName);
});

var result = await session
    .Query<User>()
    .Where(x => x.UserName.NgramSearch(term) 
             || x.FirstName.NgramSearch(term) 
             || x.LastName.NgramSearch(term))
    .ToListAsync();

Instead, define a computed property that concatenates the values into a single field, and index that:

public class User
{
    public Guid Id { get; set; }
    public string UserName { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }

    // Combine searchable fields
    public string SearchString => $"{UserName} {FirstName} {LastName}";
}

Then configure the Ngram index on that property:

    _.Schema.For<User>().NgramIndex(x => x.SearchString);

This simplifies querying:

var result = await session
    .Query<User>()
    .Where(x => x.SearchString.NgramSearch(term))
    .ToListAsync();

Sorting NGram Results by Relevance 8.29

Use OrderByNgramRank() to sort ngram search results by relevance using PostgreSQL's ts_rank() function. Results are ordered by highest relevance first (descending):

var results = await session
    .Query<User>()
    .Where(x => x.SearchString.NgramSearch("search term"))
    .OrderByNgramRank(x => x.SearchString, "search term")
    .ToListAsync();

This generates SQL like:

SELECT d.data FROM mt_doc_user d
WHERE mt_grams_vector(d.data ->> 'SearchString', FALSE) @@ mt_grams_query($1, FALSE)
ORDER BY ts_rank(mt_grams_vector(d.data ->> 'SearchString', FALSE), mt_grams_query('search term', FALSE)) DESC

OrderByNgramRank() can be combined with Select(), Take(), and other LINQ operators:

var topResults = await session
    .Query<User>()
    .Where(x => x.SearchString.NgramSearch(term))
    .OrderByNgramRank(x => x.SearchString, term)
    .Take(10)
    .Select(x => new { x.Id, x.SearchString })
    .ToListAsync();