Full Text Searching

Full Text Indexes in Marten are built based on GIN or GiST indexes utilizing Postgres built in Text Search functions. This enables the possibility to do more sophisticated searching through text fields.

WARNING

To use this feature, you will need to use PostgreSQL version 13 or above, as this is the minimum version supported by Marten - this is also the data type that Marten use to store it's data.

TIP

Full text search matches words; it misses a paraphrase that shares none of them. To combine keyword relevance with embedding similarity, see hybrid search in the pgvector support package. Its keyword half uses the full text indexes described on this page. The other Critter Stack stores have their own versions: full text search in Polecat and full text search in Fisher.

Defining Full Text Index through Store options

Full Text Indexes can be created using the fluent interface of StoreOptions like this:

• one index for whole document - all document properties values will be indexed

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex();
});

^{snippet source | anchor}

INFO

If you don't specify language (regConfig) - by default it will be created with 'english' value.

• single property - there is possibility to specify specific property to be indexed

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex(d => d.FirstName);
});

^{snippet source | anchor}

• single property with custom settings

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex(
        index =>
        {
            index.Name = "mt_custom_italian_user_fts_idx";
            index.RegConfig = "italian";
        },
        d => d.FirstName);
});

^{snippet source | anchor}

• multiple properties

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex(d => d.FirstName, d => d.LastName);
});

^{snippet source | anchor}

• multiple properties with custom settings

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>().FullTextIndex(
        index =>
        {
            index.Name = "mt_custom_italian_user_fts_idx";
            index.RegConfig = "italian";
        },
        d => d.FirstName, d => d.LastName);
});

^{snippet source | anchor}

• more than one index for document with different languages (regConfig)

var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // This creates
    _.Schema.For<User>()
        .FullTextIndex(d => d.FirstName) //by default it will use "english"
        .FullTextIndex("italian", d => d.LastName);
});

^{snippet source | anchor}

Defining Full Text Index through Attribute

Full Text Indexes can be created using the [FullTextIndex] attribute like this:

• one index for whole document - by setting attribute on the class all document properties values will be indexed

[FullTextIndex]
public class Book
{
    public Guid Id { get; set; }

    public string Title { get; set; }

    public string Author { get; set; }

    public string Information { get; set; }
}

^{snippet source | anchor}

• single property

public class UserProfile
{
    public Guid Id { get; set; }

    [FullTextIndex] public string Information { get; set; }
}

^{snippet source | anchor}

INFO

If you don't specify regConfig - by default it will be created with 'english' value.

• single property with custom settings

public class UserDetails
{
    private const string FullTextIndexName = "mt_custom_user_details_fts_idx";

    public Guid Id { get; set; }

    [FullTextIndex(IndexName = FullTextIndexName, RegConfig = "italian")]
    public string Details { get; set; }
}

^{snippet source | anchor}

• multiple properties

public class Article
{
    public Guid Id { get; set; }

    [FullTextIndex] public string Heading { get; set; }

    [FullTextIndex] public string Text { get; set; }
}

^{snippet source | anchor}

INFO

To group multiple properties into single index you need to specify the same values in IndexName parameters.

• multiple indexes for multiple properties with custom settings

public class BlogPost
{
    public Guid Id { get; set; }

    public string Category { get; set; }

    [FullTextIndex] public string EnglishText { get; set; }

    [FullTextIndex(RegConfig = "italian")] public string ItalianText { get; set; }

    [FullTextIndex(RegConfig = "french")] public string FrenchText { get; set; }
}

^{snippet source | anchor}

Text Search

Postgres contains built in Text Search functions. They enable the possibility to do more sophisticated searching through text fields. Marten gives possibility to define full text indexes and perform queries on them. Five full text search operators are supported, one per PostgreSQL query function plus a prefix form built on to_tsquery:

• regular Search (to_tsquery)

var posts = (await session.Query<BlogPost>()
    .Where(x => x.Search("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

• plain text Search (plainto_tsquery)

var posts = (await session.Query<BlogPost>()
    .Where(x => x.PlainTextSearch("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

• phrase Search (phraseto_tsquery)

var posts = (await session.Query<BlogPost>()
    .Where(x => x.PhraseSearch("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

• web-style Search (websearch_to_tsquery, supported from Postgres 11+

var posts = (await session.Query<BlogPost>()
    .Where(x => x.WebStyleSearch("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

• prefix Search (to_tsquery with the :* prefix operator on every word)

var results = (await session.Query<BlogPost>()
    .Where(x => x.PrefixSearch("Priced"))
    .ToListAsync());

^{snippet source | anchor}

PrefixSearch rewrites the term before it reaches PostgreSQL: the words are split on spaces and each becomes a prefix, joined with &, so "Priced idea" is sent as Priced:* & idea:*. That is the difference from Search("Priced"), which asks for the whole lexeme and does not match PricedIdeaScreening. Reach for it when the indexed text is an identifier rather than prose: enum values stored as strings, concatenated codes, type-ahead over names. Because the words are joined with &, every word must prefix-match something in the document.

Two things follow from the rewrite. A prefix is matched against the stemmed lexemes the index holds, so with the default english configuration "screen" matches Screening but a prefix that only exists in the unstemmed word may not. And the term reaches to_tsquery as query syntax, so a word containing &, |, !, (, ) or : is a syntax error rather than a search for that character; strip or quote such input before calling it.

Ranking a prefix search needs the rewritten form spelled out. OrderByTextRank takes the term and the query function as you would pass them to PostgreSQL, so pair PrefixSearch("Priced idea") with OrderByTextRank("Priced:* & idea:*", TextSearchFunction.Raw) (see Ordering by relevance below).

All types of Text Searches can be combined with other Linq queries

var posts = (await session.Query<BlogPost>()
    .Where(x => x.Category == "LifeStyle")
    .Where(x => x.PhraseSearch("somefilter"))
    .ToListAsync());

^{snippet source | anchor}

They allow also to specify language (regConfig) of the text search query (by default english is being used)

var posts = (await session.Query<BlogPost>()
    .Where(x => x.PhraseSearch("somefilter", "italian"))
    .ToListAsync());

^{snippet source | anchor}

When no index matches the regConfig 9.37

Every search operator takes a regConfig, defaulting to english, and it selects which index the search runs against. If the document has no index for that configuration, Marten does not fail — it falls back to to_tsvector(regConfig, d.data) over the whole stored document.

The rows that come back are still correct. What changes is worth knowing about:

• No index can serve that expression, so it is a sequential scan that re-parses every document's JSON on every query. It is fine in development and degrades with table size.
• Every string in the document becomes matchable, not just the members you indexed, so a term occurring in an unrelated field now matches.

Since 9.37 Marten logs a warning the first time this happens for a given document type and regConfig, naming both what it looked for and what is actually indexed:

Full text search on BlogPost looked for a 'english' index and found none. Indexed
configurations: italian. Falling back to an unindexed scan of the whole document, which
cannot use any index and searches every string in the document rather than the indexed
members. Register an index for 'english', or pass one of the configured values as the
search's regConfig.

The usual cause is an index declared with one configuration and a search left on the default — easy to hit through HybridSearchAsync, whose RegConfig defaults to english and which callers rarely set explicitly.

TIP

A document with no full text index at all does not warn. Searching the whole stored document is the intended behavior there, not a mistake, so warning about it would fire on correct code.

The warning goes to the ILogger Marten was given — the one registered through AddMarten()'s service provider — and fires once per document type and regConfig for the life of the store, rather than on every query.

Session shortcuts

Each operator also has a one-call form on IQuerySession for the common case of "every document of a type matching this text": SearchAsync<T>, PlainTextSearchAsync<T>, PhraseSearchAsync<T>, WebStyleSearchAsync<T> and PrefixSearchAsync<T>, each taking the term, an optional regConfig (default english) and a cancellation token. They are exactly Query<T>().Where(x => x.XxxSearch(term, regConfig)).ToListAsync() and add nothing else, so switch to the LINQ form the moment you need another predicate, an ordering or a page.

var results = await session.PrefixSearchAsync<BlogPost>("Priced idea");

^{snippet source | anchor}

Weighted Full Text Indexes and Relevance Ranking 9.31

By default a full text index concatenates its members into one flat vector, so a match in a title is exactly as relevant as a match in a long description. PostgreSQL can do better: setweight() labels each member, and ts_rank() then scores a title match above a body match.

Weighting the index

opts.Schema.For<Achievement>().WeightedFullTextIndex(idx => idx
    .Weighted(a => a.Title, TextSearchWeight.A)
    .Weighted(a => a.Tagline, TextSearchWeight.B)
    .Weighted(a => a.Description, TextSearchWeight.C));

which produces:

create index mt_doc_achievement_idx_fts on public.mt_doc_achievement using gin ((
  setweight(to_tsvector('english', coalesce(data ->> 'Title', '')), 'A') ||
  setweight(to_tsvector('english', coalesce(data ->> 'Tagline', '')), 'B') ||
  setweight(to_tsvector('english', coalesce(data ->> 'Description', '')), 'C')
));

Note that each member is converted to a vector separately and the vectors are concatenated. That is what setweight requires, and it is why weighting is a distinct method rather than an option on FullTextIndex() — the DDL is a different shape, not a configured variant of the same one.

There are four weight labels, A through D. They carry no numbers of their own; the rank function supplies those, defaulting to {D, C, B, A} = {0.1, 0.2, 0.4, 1.0}. D is what an unlabelled vector already means, so it is the default.

WARNING

A weighted index needs at least two different weights. Weighting expresses a relative ordering, so one weight — or the same weight on every member — ranks nothing, and Marten refuses it at configuration time rather than emitting an index that looks weighted and is not.

Ordering by relevance

OrderByTextRank() sorts by ts_rank(), highest first:

var results = await session
    .Query<Achievement>()
    .Where(a => a.WebStyleSearch(term))
    .OrderByTextRank(term, TextSearchFunction.WebStyle)
    .ToListAsync();

Use ThenByTextRank() to add relevance after another ordering.

The search function is explicit and has no default. Ranking with a different tsquery function than the Where clause used is always a mistake, and a friendly default would be occasionally wrong and silently so — so pass the one that matches your filter: Plain, Phrase, WebStyle or Raw.

The rank resolves the same tsvector the Where clause matched on, read from the index definition. That is deliberate rather than incidental: a rank computed over a different vector than the filter matched on returns rows in an order that looks plausible and means nothing, which is a far quieter failure than returning the wrong rows.

TIP

The search term is passed as a parameter rather than interpolated into the SQL.

Costs worth knowing before you use it

GIN indexes cannot order. ts_rank is a post-filter sort over everything the @@ matched, so ranking a broad query sorts a lot of rows. Narrow the filter before reaching for the rank.

Adding weights to an existing index is not a free migration. Weights change the index expression, so Weasel drops and recreates the index on the next schema apply. On a large table that is an outage rather than a migration — consider building it out of band, as described in the ignore-indexes documentation.

One index per text search configuration. If a document has two full text indexes sharing a regConfig, Marten cannot tell which one a search means and will refuse the query rather than pick one. Give them different regConfig values, or register a single index covering every member you want to search.

Partial text search in a multi-word text (NGram search)

Marten provides the ability to search partial text or words in a string containing multiple words using NGram search. This is quite similar in functionality to NGrams in Elastic Search. As an example, we can now accurately match rich com text within Communicating Across Contexts (Enriched). NGram search uses English by default. NGram search also encompasses and handles unigrams, bigrams and trigrams. This functionality is added in v5.

var result = await session
    .Query<User>()
    .Where(x => x.UserName.NgramSearch(term))
    .ToListAsync();

^{snippet source | anchor}

var store = DocumentStore.For(_ =>
{
    _.Connection(Marten.Testing.Harness.ConnectionSource.ConnectionString);
    _.DatabaseSchemaName = "ngram_test";

    // This creates an ngram index for efficient sub string based matching
    _.Schema.For<User>().NgramIndex(x => x.UserName);
});

await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync();

await using var session = store.LightweightSession();

string term = null;
for (var i = 1; i < 4; i++)
{
    var guid = $"{Guid.NewGuid():N}";
    term ??= guid.Substring(5);

    var newUser = new User(i, $"Test user {guid}");

    session.Store(newUser);
}

await session.SaveChangesAsync();

var result = await session
    .Query<User>()
    .Where(x => x.UserName.NgramSearch(term))
    .ToListAsync();

^{snippet source | anchor}

var result = await session
    .Query<User>()
    .Where(x => x.Address.Line1.NgramSearch(term))
    .ToListAsync();

^{snippet source | anchor}

INFO

Especially when writing unit tests which uses NGramSearch ensure to call await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync(); which will add the required system functions to database for using it.

WARNING

When you call NGramSearch, ensure to call it on a string property/field of the document rather than on the document itself.

• await session.Query<User>().Where(x => x.Address.Line1.NgramSearch(term)) - accessing NGramSearch on any string property is the right usage. ✅
• await session.Query<User>().Where(x => x.NgramSearch(term)) Don't target the NGramSearch on the document User, it won't work. ❌
• await session.Query<User>().Where(x => x.Name.ToLower().NgramSearch(term)) Don't target the NGramSearch on a computed property on the document User i.e. x.Name.ToLower(), it won't work. ❌

NGram search on non-English text 7.39.5

If you want to use NGram search on non-English text, Marten provides a mechanism via an opt-in storeOptions.Advanced.UseNGramSearchWithUnaccent = true which uses Postgres unaccent extension for applying before creating ngrams and on search input for a better multilingual experience. Check the sample code below:

var store = DocumentStore.For(_ =>
{
   _.Connection(Marten.Testing.Harness.ConnectionSource.ConnectionString);
   _.DatabaseSchemaName = "ngram_test";
   _.Schema.For<User>().NgramIndex(x => x.UserName);
   _.Advanced.UseNGramSearchWithUnaccent = true;
});

await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync();

await using var session = store.LightweightSession();
//The ngram uðmu should only exist in bjork, if special characters ignored it will return Umut
var umut = new User(1, "Umut Aral");
var bjork = new User(2, "Björk Guðmundsdóttir");

//The ngram øre should only exist in bjork, if special characters ignored it will return Chris Rea
var kierkegaard = new User(3, "Søren Kierkegaard");
var rea = new User(4, "Chris Rea");

session.Store(umut);
session.Store(bjork);
session.Store(kierkegaard);
session.Store(rea);

await session.SaveChangesAsync();

var result = await session
   .Query<User>()
   .Where(x => x.UserName.NgramSearch("uðmu") || x.UserName.NgramSearch("øre"))
   .ToListAsync();

^{snippet source | anchor}

INFO

Especially when writing unit tests which uses NGramSearch ensure to call await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync(); which will add the required system functions as well the unaccent extension to database for use (when UseNGramSearchWithUnaccent is set to true).

NGram Search Across Multiple Properties 7.39.5

In many cases, you may want to perform partial text search across multiple fields like UserName, FirstName, and LastName.

A naive approach might be to apply individual Ngram indexes and search each field separately:

Don't do this

This results in multiple indexes per document and requires complex LINQ queries to combine the results — inefficient and hard to maintain.

// Inefficient and verbose
var store = DocumentStore.For(_ =>
{
    _.Connection(ConnectionSource.ConnectionString);

    // Too many indexes
    _.Schema.For<User>().NgramIndex(d => d.UserName);
    _.Schema.For<User>().NgramIndex(d => d.FirstName);
    _.Schema.For<User>().NgramIndex(d => d.LastName);
});

var result = await session
    .Query<User>()
    .Where(x => x.UserName.NgramSearch(term) 
             || x.FirstName.NgramSearch(term) 
             || x.LastName.NgramSearch(term))
    .ToListAsync();

Instead, define a computed property that concatenates the values into a single field, and index that:

public class User
{
    public Guid Id { get; set; }
    public string UserName { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }

    // Combine searchable fields
    public string SearchString => $"{UserName} {FirstName} {LastName}";
}

Then configure the Ngram index on that property:

    _.Schema.For<User>().NgramIndex(x => x.SearchString);

This simplifies querying:

var result = await session
    .Query<User>()
    .Where(x => x.SearchString.NgramSearch(term))
    .ToListAsync();

Sorting NGram Results by Relevance 8.29

Use OrderByNgramRank() to sort ngram search results by relevance using PostgreSQL's ts_rank() function. Results are ordered by highest relevance first (descending):

var results = await session
    .Query<User>()
    .Where(x => x.SearchString.NgramSearch("search term"))
    .OrderByNgramRank(x => x.SearchString, "search term")
    .ToListAsync();

This generates SQL like:

SELECT d.data FROM mt_doc_user d
WHERE mt_grams_vector(d.data ->> 'SearchString', FALSE) @@ mt_grams_query($1, FALSE)
ORDER BY ts_rank(mt_grams_vector(d.data ->> 'SearchString', FALSE), mt_grams_query('search term', FALSE)) DESC

OrderByNgramRank() can be combined with Select(), Take(), and other LINQ operators:

var topResults = await session
    .Query<User>()
    .Where(x => x.SearchString.NgramSearch(term))
    .OrderByNgramRank(x => x.SearchString, term)
    .Take(10)
    .Select(x => new { x.Id, x.SearchString })
    .ToListAsync();