Open Telemetry and Metrics 7.10

Marten has built in support for emitting Open Telemetry spans and events, as well as for exporting metrics using System.Diagnostics.Metrics. This information should be usable with any monitoring system (Honeycomb, DataDog, Prometheus, SigNoz, Project Aspire, etc.) that supports Open Telemetry.

Exporting Marten Data

Heads up, .NET processes will not actually emit any metrics or Open Telemetry data they are collecting unless there are both configured exporters and you have explicitly configured your application to emit this information. And note that you have to explicitly export both metrics and Open Telemetry activity tracing independently. That all being said, here's a sample of configuring the exporting -- this case just exporting information to a Project Aspire dashboard in the end:

// This is passed in by Project Aspire. The exporter usage is a little
// different for other tools like Prometheus or SigNoz
var endpointUri = builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"];
Console.WriteLine("OLTP endpoint: " + endpointUri);

builder.Services.AddOpenTelemetry().UseOtlpExporter();

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing.AddSource("Marten");
    })
    .WithMetrics(metrics =>
    {
        metrics.AddMeter("Marten");
    });

^{snippet source | anchor}

Note, you'll need a reference to the OpenTelemetry.Extensions.Hosting Nuget for that extension method.

Connection Events

WARNING

This connection tracking is bypassed by some uncommonly used operations, but will apply to basically every common operation done with IQuerySession or IDocumentSession.

It's often important just to track how many connections your application is using, and how long connections are being used. To that end, you can opt into emitting Open Telemetry spans named marten.connection with this option:

using var store = DocumentStore.For(opts =>
{
    opts.Connection("some connection string");

    // Track Marten connection usage
    opts.OpenTelemetry.TrackConnections = TrackLevel.Normal;
});

^{snippet source | anchor}

This option will also tag events on the activity for any exceptions that happened from executing database commands. At a minimum, this option might help you spot performance issues due to chattiness between your application servers and the database.

There is also a verbose mode that will also tag Open Telemetry activity events for all the Marten operations (storing documents, appending events, etc.) performed by an IDocumentSession.SaveChangesAsync() call immediately after successfully committing a database transaction. This mode is probably most appropriate for troubleshooting or performance testing where the extra information being emitted might help you spot database usage issues. That option is shown below:

using var store = DocumentStore.For(opts =>
{
    opts.Connection("some connection string");

    // Track Marten connection usage *and* all the "write" operations
    // that Marten does with that connection
    opts.OpenTelemetry.TrackConnections = TrackLevel.Verbose;
});

^{snippet source | anchor}

Event Store Metrics

You can opt into exporting a metrics counter for the events appended to Marten's event store functionality with this option:

using var store = DocumentStore.For(opts =>
{
    opts.Connection("some connection string");

    // Track the number of events being appended to the system
    opts.OpenTelemetry.TrackEventCounters();
});

^{snippet source | anchor}

This is adding a Counter metric named marten.event.append. This metric has tags for:

• event_type - the Marten name for the type of the event within its configuration
• tenant.id - the tenant id for which the event was captured. If you are not using multi-tenancy, this value will be "DEFAULT" and can be just ignored

FetchForWriting Metrics 9.24

The cost of a command handler that uses FetchForWriting is mostly the cost of rebuilding the aggregate, and that in turn is however many events Marten had to read back and fold. How many that is depends entirely on the aggregate's projection lifecycle, and it is not something you can see from the outside. Opt into measuring it with:

using var store = DocumentStore.For(opts =>
{
    opts.Connection("some connection string");

    // How many events does each FetchForWriting() call actually replay?
    opts.OpenTelemetry.TrackFetchForWritingMetrics();
});

This adds a Histogram named marten.fetch_for_writing.events_replayed, recorded once per FetchForWriting() call, with tags for:

• aggregate.type - the .NET type name of the aggregate that was fetched
• fetch.plan - which plan served the call: Live, Inline or Async

Reading it is straightforward, because each plan has a characteristic shape:

fetch.plan	What the histogram shows
Live	The full stream length, every single time — the aggregate is rebuilt from scratch on each fetch
Async	Only the events the async daemon has not projected yet, so this doubles as a per-fetch view of daemon lag
Inline	Always zero — the stored snapshot is current by construction

The histogram's own count gives you fetch frequency for free, so a sum by (aggregate.type) tells you which aggregates dominate your event replay, and the Live percentiles tell you what switching one of them to Inline would actually save before you commit to that migration.

TIP

Nothing is created until TrackFetchForWritingMetrics() is called, so leaving it off costs a null check on the fetch path and nothing more. Both tag values are cached per aggregate type, so a recorded fetch allocates nothing on the heap either.

Async Daemon Metrics and Spans

See Open Telemetry and Metrics within the Async Daemon documentation.