Removing Protected Information 7.31

INFO

The Marten team is at least considering support for crypto-shredding in Marten 8.0, but no definite plans have been made yet.

For a variety of reasons, you may wish to remove or mask sensitive data elements in a Marten database without necessarily deleting the information as a whole. Documents can be amended with Marten's Patching API. With event data, you now have options to reach into the event data and rewrite selected members as well as to add custom headers. First, start by defining data masking rules by event type like so:

var builder = Host.CreateApplicationBuilder();
builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // By a single, concrete type
    opts.Events.AddMaskingRuleForProtectedInformation<AccountChanged>(x =>
    {
        // I'm only masking a single property here, but you could do as much as you want
        x.Name = "****";
    });

    // Maybe you have an interface that multiple event types implement that would help
    // make these rules easier by applying to any event type that implements this interface
    opts.Events.AddMaskingRuleForProtectedInformation<IAccountEvent>(x => x.Name = "****");

    // Little fancier
    opts.Events.AddMaskingRuleForProtectedInformation<MembersJoined>(x =>
    {
        for (int i = 0; i < x.Members.Length; i++)
        {
            x.Members[i] = "*****";
        }
    });
});

^{snippet source | anchor}

That's strictly a configuration time effort. Next, you can apply the masking on demand to any subset of events with the IDocumentStore.Advanced.ApplyEventDataMasking() API. First, you can apply the masking for a single stream:

public static Task apply_masking_to_streams(IDocumentStore store, Guid streamId, CancellationToken token)
{
    return store
        .Advanced
        .ApplyEventDataMasking(x =>
        {
            x.IncludeStream(streamId);

            // You can add or modify event metadata headers as well
            // BUT, you'll of course need event header tracking to be enabled
            x.AddHeader("masked", DateTimeOffset.UtcNow);
        }, token);
}

^{snippet source | anchor}

As a finer grained operation, you can specify an event filter (Func<IEvent, bool>) within an event stream to be masked with this overload:

public static Task apply_masking_to_streams_and_filter(IDocumentStore store, Guid streamId, CancellationToken token)
{
    return store
        .Advanced
        .ApplyEventDataMasking(x =>
        {
            // Mask selected events within a single stream by a user defined criteria
            x.IncludeStream(streamId, e => e.EventTypesAre(typeof(MembersJoined), typeof(MembersDeparted)));

            // You can add or modify event metadata headers as well
            // BUT, you'll of course need event header tracking to be enabled
            x.AddHeader("masked", DateTimeOffset.UtcNow);
        }, token);
}

^{snippet source | anchor}

TIP

Regardless of what events you specify, only events that match a pre-registered masking rule will have the header changes applied.

To apply the event data masking across streams on an arbitrary grouping, you can use a LINQ expression as well:

public static Task apply_masking_by_filter(IDocumentStore store, Guid[] streamIds)
{
    return store.Advanced.ApplyEventDataMasking(x =>
        {
            x.IncludeEvents(e => e.EventTypesAre(typeof(QuestStarted)) && e.StreamId.IsOneOf(streamIds));
        });
}

^{snippet source | anchor}

Finally, if you are using multi-tenancy, you can specify the tenant id as part of the same fluent interface:

public static Task apply_masking_by_tenant(IDocumentStore store, string tenantId, Guid streamId)
{
    return store
        .Advanced
        .ApplyEventDataMasking(x =>
        {
            x.IncludeStream(streamId);

            // Specify the tenant id, and it doesn't matter
            // in what order this appears in
            x.ForTenant(tenantId);
        });
}

^{snippet source | anchor}

WARNING

ForTenant() scopes both the read and the write, but that was only true of the read before the fix for #5234.

Under Events.UseTenantPartitionedEvents each tenant draws from its own event sequence, so seq_id is not unique across tenants. Masking and stream compaction keyed their UPDATE and DELETE on seq_id alone, which meant masking one tenant also overwrote the same-numbered event in every other tenant, and compacting one tenant's stream deleted the other tenants' events and left the calling tenant's Compacted<T> snapshot in their place. Nothing threw.

If you ran masking or stream compaction against a store with UseTenantPartitionedEvents enabled on an affected version, the damage is already written and no code change can reverse it — restore the affected tenants' events from a backup or your archival storage. Stores that never enabled UseTenantPartitionedEvents are unaffected, because a single global sequence makes seq_id store-unique there.

WARNING

Masking reaches archived streams, and skipped events, because neither is erased by being set aside. Every source does: both kinds of IncludeStream(), and IncludeEvents() unless your filter rules them out itself with IsArchived.

Before this was fixed, masking read through the same paths as everything else and left those rows out, so protected data stayed in place while the pass reported success. If you relied on masking for data that was archived, or skipped, when you ran it, run the same ApplyEventDataMasking() call again after upgrading. That also runs your rules again over everything the first pass reached, so if a rule isn't safe to run twice, such as one that draws a new pseudonym, add x.IsArchived to the filter of an IncludeEvents() that still names the same subject, like IncludeEvents(x => x.StreamId == streamId && x.IsArchived). Never use x.IsArchived as the whole filter for this, which masks every archived event in the store.

Here's a couple more facts you might need to know:

• The masking rules can only be done at configuration time (as of right now)
• You can apply multiple masking rules for certain event types, and all will be applied when you use the masking API
• The masking has absolutely no impact on event archiving or projected data -- unless you rebuild the projection data after applying the data masking of course