Skip to content

Rebuilding Projections

Projections can be completely rebuilt with the async daemon subsystem. Both inline and asynchronous projections can be rebuilt with the async daemon.

cs
public class DistanceProjection: EventProjection
{
    public DistanceProjection()
    {
        ProjectionName = "Distance";
    }

    // Create a new Distance document based on a Travel event
    public Distance Create(Travel travel, IEvent e)
    {
        return new Distance {Id = e.Id, Day = travel.Day, Total = travel.TotalDistance()};
    }
}

snippet source | anchor

cs
StoreOptions(x => x.Projections.Add(new DistanceProjection(), ProjectionLifecycle.Async));

var agent = await StartDaemon();

// setup test data
await PublishSingleThreaded();

// rebuild projection `Distance`
await agent.RebuildProjectionAsync("Distance", CancellationToken.None);

snippet source | anchor

Optimized Projection Rebuilds 7.30

TIP

This optimization will be turned on by default in Marten 8, but we didn't want to force anyone using Marten 7 to have to upgrade their database without the explicit opt in configuration.

Marten can optimize the projection rebuilds of single stream projections by opting into this flag in your configuration:

cs
var builder = Host.CreateApplicationBuilder();
builder.Services.AddMarten(opts =>
{
    opts.Connection("some connection string");

    // Turn on the PostgreSQL table partitioning for
    // hot/cold storage on archived events
    opts.Events.UseArchivedStreamPartitioning = true;

    // Use the *much* faster workflow for appending events
    // at the cost of *some* loss of metadata usage for
    // inline projections
    opts.Events.AppendMode = EventAppendMode.Quick;

    // Little more involved, but this can reduce the number
    // of database queries necessary to process inline projections
    // during command handling with some significant
    // caveats
    opts.Events.UseIdentityMapForInlineAggregates = true;

    // Opts into a mode where Marten is able to rebuild single
    // stream projections faster by building one stream at a time
    // Does require new table migrations for Marten 7 users though
    opts.Events.UseOptimizedProjectionRebuilds = true;
});

snippet source | anchor

In this mode, Marten will rebuild single stream projection documents stream by stream in the reverse order that the streams were last modified. This was conceived of as being combined with the FetchForWriting() usage with asynchronous single stream projections for zero downtime deployments while trying to create less load on the database than the original "left fold" / "from zero" rebuild would be.

Rebuilding a Single Stream 7.28

A long standing request has been to be able to rebuild only a single stream or subset of streams by stream id (or string key). Marten now has a (admittedly crude) ability to do so with this syntax on IDocumentStore:

cs
await theStore.Advanced.RebuildSingleStreamAsync<SimpleAggregate>(streamId);

snippet source | anchor

Released under the MIT License.