Migrate Mocha from 16.7 to 16.8

Update every Mocha.* package in the application to version 16.8 before applying these changes.

Entries that name RabbitMQ, the Postgres outbox, or Azure Service Bus apply to applications that use them. All other entries apply to every transport.

API breaking changes#

RabbitMQ topology resources are provisioned with their connection#

IRabbitMQResource.ProvisionAsync and the ProvisionAsync methods of RabbitMQQueue, RabbitMQExchange, RabbitMQBinding, RabbitMQQueueBinding, and RabbitMQExchangeBinding take the connection that owns the channel as a new second parameter:

Diff
-await queue.ProvisionAsync(channel, cancellationToken);
+await queue.ProvisionAsync(channel, connection, cancellationToken);

Assemblies that call or implement these methods must be recompiled against 16.8.

The RabbitMQ transport requires RabbitMQ.Client 7.2.2#

Mocha.Transport.RabbitMQ references RabbitMQ.Client 7.2.2. Version 16.7 referenced 7.2.0. An application that pins RabbitMQ.Client to a lower version gets a package downgrade error (NU1605) at restore. Raise the pinned version to 7.2.2 or later.

Behavioral breaking changes#

RabbitMQ sends and publishes wait for the broker's confirmation#

A send or publish through the RabbitMQ transport completes when RabbitMQ has confirmed the message. In 16.7 it completed when the message was written to the connection, and a message that RabbitMQ did not accept was lost without an error. The change covers SendAsync, PublishAsync, RequestAsync, replies, faults, and the dispatches of the outbox and the scheduler.

What you will observe:

  • A call that returns means that RabbitMQ has the message. A message for an exchange reached every queue that the exchange routes to. When no binding matches, RabbitMQ drops the message and the call still completes, as in 16.7. A message for a queue address reached that queue.
  • Every call takes at least one round trip to the broker. A loop that awaits each call sends one message per round trip. To send faster, start several calls and await them together.
  • While RabbitMQ cannot be reached, or blocks the connection during a memory or disk alarm, a call waits until its cancellation token is cancelled. See A RabbitMQ send or publish waits while the broker is unavailable.
  • A handler that sends or publishes keeps its concurrency slot while the call waits.

See Publisher confirms.

RabbitMQ sends and publishes fail when the broker does not accept the message#

In 16.7 a send or publish completed whether or not RabbitMQ accepted the message. In 16.8 the answer of the broker decides how the call ends. Mocha does not send a failed message again.

What happensOutcome
RabbitMQ rejects the message with a negative acknowledgement. This can happen when a queue is at its length limit and uses x-overflow: reject-publish.The call throws the PublishException of the RabbitMQ client. RabbitMQ rejects a published event when one of the queues of its exchange rejects it, and the other queues keep their copy.
The broker closes the channel with a reply code, for example 406 PRECONDITION_FAILED for a message above the size limit, 404 NOT_FOUND for a missing exchange, or 403 ACCESS_REFUSED.The call throws the exception of the RabbitMQ client (AlreadyClosedException or OperationInterruptedException) with the reply code and text of the broker.
The connection or the channel is lost before the confirmation arrives.The call throws the exception of the RabbitMQ client. RabbitMQ may or may not have the message.
The headers of the message do not fit into one frame of the connection.The call throws InvalidOperationException before the message is sent.

What to do:

  • Handle these exceptions where the application sends or publishes.
  • Sending a message again after a lost connection can deliver it twice. Publishing a rejected event again delivers it once more to every queue that accepted it. Make handlers safe to run more than once for the same message, or configure the transactional inbox in the consuming service.
  • For a subscriber queue that must not miss events, use x-overflow: drop-head or a dead-letter exchange instead of reject-publish.
  • Through the Postgres outbox, each of these outcomes counts as an attempt, and the outbox dispatches the message again. See The Postgres outbox no longer stops retrying a message.

A message for a RabbitMQ queue that does not exist is returned#

This entry applies to messages whose destination is a queue address, such as queue:orders, and to messages that Mocha moves to an error or skipped queue. RabbitMQ now returns such a message when the queue does not exist. In 16.7 RabbitMQ dropped the message and the call completed.

  • With auto-provisioning enabled for the queue, Mocha declares the queue and its bindings again and sends the message once more. A queue that is deleted while a sending application runs is created again by the next send.
  • With auto-provisioning disabled for the queue, for an exclusive queue, or when the queue is still missing after it was declared, the call throws InvalidOperationException with the text The broker returned the message because no queue named '<queue>' exists.

Messages for an exchange and replies are not returned. RabbitMQ drops a message for an exchange that no binding matches, and a reply whose requester has stopped, as in 16.7.

See Missing queues and bindings.

A failed RabbitMQ message is acknowledged only after it was moved#

When a handler fails, Mocha sends a fault to the requester or moves the message to the error queue of its endpoint. This send now waits for the broker's confirmation. In 16.7 the received message was acknowledged as soon as the fault or the moved message was written to the connection.

When the send fails:

  1. The log shows FaultDeliveryFailed at Error with the exception.
  2. Mocha moves the received message to the skipped queue of its endpoint, and the log shows MessageSkipped (Warning).
  3. When that send fails as well, the received message returns to its source queue and RabbitMQ delivers it again.

Each of these sends waits while RabbitMQ cannot be reached or blocks the connection, and the message keeps its concurrency slot for that time.

RabbitMQ receive endpoints attach again on their own#

In 16.7 a receive endpoint whose channel was closed while the connection stayed open did not consume again until the connection was replaced or the application restarted. This can happen when a handler exceeds the broker's delivery acknowledgement timeout or when the queue is deleted.

In 16.8 every receive endpoint attaches again on its own, with delays of about 1 to 30 seconds between attempts. With auto-provisioning enabled it declares its queue and bindings again first. Mocha also reconnects a lost connection itself and keeps trying until the application stops, whatever the AutomaticRecoveryEnabled setting of the connection factory is.

What you will observe:

  • The log shows ConsumerChannelLost or ConsumerCancelled (Warning) and then ConsumerAttached (Information). For a lost connection it shows ConnectionLost (Warning) and then ConnectionReady (Information).
  • A queue that is deleted while its consuming application runs is created again when auto-provisioning is enabled.
  • The health check reports an endpoint that is not attached. See The message bus health check reports transport health.
  • When the broker refuses the connection, a declaration, or the consumer the same way on every attempt, the attempts continue and are logged at Error (ConnectionAttemptFailed, ConsumerAttachFailed). This can happen when credentials changed or when the configuration of a queue differs from the existing queue. Sends and publishes wait while the dispatcher connection is not ready.
  • A network path that fails without closing the connection is detected through heartbeats, after two to three minutes with the client's default interval of 60 seconds. Sends and publishes wait during that time. See Heartbeats for a shorter interval.

See Connection recovery.

RabbitMQ handlers are cancelled when their connection or channel is lost#

When a connection or channel is lost, Mocha cancels the CancellationToken of every handler that runs for a message of that channel. RabbitMQ delivers these messages again. This includes a message whose handler had finished its work but whose acknowledgement could no longer be sent. In 16.7 the token was cancelled only when the endpoint stopped.

What you will observe:

  • A handler that passes its token on ends with an OperationCanceledException when the channel is lost, and runs again for the redelivered message. Make handlers safe to run more than once for the same message, or configure the transactional inbox.
  • A handler that ignores its token keeps its MaxConcurrency slot until it returns. When such handlers hold every slot, the endpoint cannot attach again. The health check reports The consumer waits for a handler of a lost channel to return. for its queue, and after 30 seconds the log shows ConsumerBlockedByHandlers at Error. Pass the token to database queries, HTTP requests, and other asynchronous operations.

See Message handling after a connection loss.

The Postgres outbox no longer stops retrying a message#

In 16.7 the Postgres outbox stopped dispatching a message after 10 attempts and delayed each attempt twice as long as the one before. In 16.8 the outbox dispatches a message until the transport accepts it.

  • A message that fails is dispatched again after 10 seconds. The delay does not grow with the number of attempts.
  • The TimesSent column still counts the attempts. It no longer limits them.
  • A message that failed 10 times in 16.7 stays in the outbox table and is dispatched again after the update.
  • Every failed dispatch is logged. When the attempt count of a message is a multiple of 10, the processor also writes a Critical entry in the category Mocha.Outbox.PostgresOutboxProcessor:
Outbox message <id> (message ID <message id>, type <type>) to <destination> failed on attempt <n>. The message is retried.
  • The 16.7 Critical entry Message with ID <id> exhausted all 10 attempts and will not be retried. is no longer written. Change alert rules that match it to the new text.
  • A message that cannot be read, has an unknown message type, or matches no endpoint is retried in the same way. Its Critical entry shows (null) for the message ID, the type, and the destination when the stored envelope cannot be read.
  • A failed dispatch no longer ends the processing run. The processor logs the exception at Error with the text The dispatch of outbox message <id> (message ID <message id>, type <type>) to <destination> failed on attempt <n>. The message is retried. and continues with the next message. In 16.7 the failure was logged as An unexpected error occurred while processing outbox event. Change alert rules that match that text for a failed dispatch.

A Postgres outbox dispatch keeps its transaction open until the broker confirms#

An outbox worker claims a message in a database transaction, dispatches it, and deletes the row in the same transaction. With RabbitMQ a dispatch now waits for the broker's confirmation, so the transaction, its row lock, and its database connection are held for the whole dispatch. While RabbitMQ cannot be reached or blocks the connection, the dispatch waits. In 16.7 a dispatch to RabbitMQ completed when the message was written to the connection.

The session is idle in transaction while the dispatch runs. When idle_in_transaction_session_timeout in PostgreSQL, or the idle transaction timeout of a connection pooler, is shorter than a dispatch, the database ends the session during the dispatch. If the broker accepts the message, the row cannot be deleted and the message is dispatched again. Make handlers safe to run more than once for the same message, or configure the transactional inbox.

The Postgres scheduler no longer stops retrying a scheduled message#

This entry applies to the Postgres-backed scheduled message store that UsePostgresScheduling() registers. With RabbitMQ it stores scheduled messages and redeliveries.

In 16.7 the scheduler stopped dispatching a scheduled message after 10 attempts and delayed each attempt twice as long as the one before. In 16.8 the scheduler dispatches a message until the transport accepts it.

  • A message that fails is dispatched again after 10 seconds. The delay does not grow with the number of attempts.

  • After a failed dispatch the scheduler pauses for 10 seconds before it dispatches the next message. Other due messages can be up to 10 seconds late while a message fails.

  • The times_sent column still counts the attempts. The max_attempts column is no longer applied.

  • A message that failed 10 times in 16.7 stays in the table and is dispatched after the update. Delete the rows that must not be sent before you update:

    SQL
    DELETE FROM scheduled_messages
    WHERE times_sent >= max_attempts;
  • A scheduled message can be cancelled at any attempt count. In 16.7 a message that had used all attempts could not be cancelled.

  • When the attempt count of a message is a multiple of 10, the dispatcher writes the Critical entry Scheduled message <id> (message ID <message id>, type <type>) to <destination> failed on attempt <n>. The message is retried. The 16.7 entry Scheduled message <id> exhausted all 10 retry attempts and will not be retried. is no longer written. Change alert rules that match it.

  • The index on scheduled_time no longer has the filter "times_sent" < "max_attempts". Add an EF Core migration to apply the change. Without it the scheduler works, but its queries do not use the index.

  • The scheduler wakes at the time the next message is due. In 16.7 it woke 1 second late for a new message and later for a retry.

See Retry behavior.

A RabbitMQ send or publish waits while the broker is unavailable#

While RabbitMQ cannot be reached, a send or publish waits for the next connection until its cancellation token is cancelled. Then it throws OperationCanceledException. In 16.7 a call that found no connection threw the exception of the RabbitMQ client after five connection attempts.

What to do:

  • A caller that must not wait, such as an HTTP request, passes a cancellation token with a time limit.
  • A call that is cancelled while it waits for its confirmation throws OperationCanceledException, and the broker may or may not have the message.

At most 100 RabbitMQ sends and publishes are in flight per transport#

Each send or publish uses a channel of its own on the dispatcher connection until RabbitMQ has answered. One transport instance runs at most 100 sends and publishes at the same time, or fewer when the broker allows fewer channels per connection. Further calls wait for a free channel. 16.7 had no such limit and kept at most 10 idle channels.

What you will observe:

  • The dispatcher connection shows up to 100 channels in the RabbitMQ management UI. The channels stay open for the next sends.
  • Code that starts many calls at once, for example with Task.WhenAll over a large batch, has at most 100 of them in flight. The other calls wait for a free channel.
  • In measurements against a single RabbitMQ node, throughput rose up to about 32 calls in flight and not beyond.

RequestAsync cancels a dispatch that outlasts the response timeout#

The response timeout of RequestAsync, two minutes, now also ends the dispatch of the request. When it elapses while the request is still being dispatched, the dispatch is cancelled and RequestAsync throws ResponseTimeoutException. In 16.7 RequestAsync waited for the dispatch to return first. See How to handle timeouts.

RabbitMQ requests wait for the reply consumer#

A request is sent only while the reply queue of the service instance exists and its consumer is attached. While the reply consumer attaches again, for example after a connection loss, RequestAsync waits. The wait counts towards the response timeout.

A request that is sent before the transport has started or after it has stopped throws InvalidOperationException with the text The RabbitMQ reply endpoint is not started. Send requests only while the host runs.

RabbitMQ startup retries for up to two minutes#

When RabbitMQ cannot be reached at startup, the transport retries with delays of about 1 to 10 seconds until StartTimeout elapses, two minutes by default. Then the start fails. In 16.7 the start failed after five attempts, which took about 15 seconds plus the time of the attempts.

The start fails at once, without further attempts, when the broker refuses the connection or the topology: refused credentials, a failed TLS handshake, denied access to the virtual host, or a queue or exchange declaration that the broker rejects. A virtual host that has reached its connection limit refuses the connection the same way, which can happen during a rolling update. 16.7 made five attempts in these cases as well.

What you will observe:

  • The host does not finish starting while the transport retries. Check that startup probes and deployment timeouts allow for StartTimeout, or lower it:

    C#
    builder.Services
        .AddMessageBus()
        .AddRabbitMQ(transport => transport.StartTimeout(TimeSpan.FromSeconds(30)));
  • TimeSpan.Zero makes one attempt. Timeout.InfiniteTimeSpan retries until the host cancels the start.

  • Each failed attempt is logged as ConnectionAttemptFailed (Warning). A failed start is logged as ConnectionFailed (Error) with the exception.

StartTimeout applies to startup only. After startup Mocha reconnects until the application stops. See Connection recovery.

The generated RabbitMQ reply queue expires instead of being auto-delete#

The reply queue that the transport generates per service instance (response-<instance id>) is declared differently:

Property16.716.8
Auto-deleteYesNo
Expiry without consumers (x-expires)30 minutes5 minutes
Queue type when the transport default is quorumChosen by the brokerQuorum, with x-quorum-initial-group-size if configured
RemovedBy RabbitMQ when its consumer goes awayBy Mocha when the reply endpoint stops, otherwise by RabbitMQ after the expiry

What you will observe:

  • The queue stays during a short disconnection, and replies wait in it until Mocha has reconnected. In 16.7 RabbitMQ deleted the queue when the connection was lost.
  • After a process that ended without stopping the bus, the queue remains on the broker for 5 minutes.
  • With auto-provisioning enabled, Mocha deletes the queue when the reply endpoint stops, also when the runtime is disposed without a stop. It waits at most 5 seconds for the deletion.
  • Queue policies, dashboards, and alerts that select auto-delete queues no longer match the reply queue.

See Temporary receive endpoints.

RabbitMQ receive endpoints wait for running handlers on shutdown#

When the host stops, every receive endpoint stops taking new messages and waits for its running handlers until the shutdown timeout of the host elapses. Only then does Mocha cancel the CancellationToken of the handlers that still run, and RabbitMQ delivers their messages again. In 16.7 the token of every handler was cancelled as soon as its endpoint stopped.

What you will observe:

  • A shutdown takes as long as the longest running handler, up to the shutdown timeout (HostOptions.ShutdownTimeout, 30 seconds by default). Closing channels and deleting the reply queue can take a few seconds longer.
  • A handler that relied on its token being cancelled at the start of a shutdown now runs until it finishes or the shutdown timeout elapses.
  • Set HostOptions.ShutdownTimeout to the time handlers may take, and keep the termination grace period of the platform, such as terminationGracePeriodSeconds in Kubernetes, above it.
  • The log shows ConsumerDrainStarted (Information), ConsumerDrainTimedOut (Warning, when the timeout elapsed first), and ConsumerStopped (Information) for each endpoint.

See Graceful shutdown.

Receive endpoints start and stop in a different order#

This entry applies to every transport, including custom transports. 16.7 started and stopped the receive endpoints of a transport one after the other, without regard to their kind.

  • On start, a transport starts its reply endpoints first and then its other receive endpoints.
  • On stop, a transport stops its other receive endpoints at the same time and then its reply endpoints.
  • When the start of a transport fails, the transport stops what it has started and rethrows the failure.

For a custom transport:

  • ReceiveEndpoint.OnStopAsync runs for several endpoints of the transport at the same time.
  • The cancellation token of ReceiveEndpoint.OnStopAsync and MessagingTransport.OnAfterStopAsync bounds the wait for running handlers. It can already be cancelled when the method is called. Release the resources of the endpoint, including temporary broker resources, without it.
  • When the start of the transport fails, MessagingTransport.OnBeforeStopAsync and MessagingTransport.OnAfterStopAsync run, also when OnBeforeStartAsync did not complete.
  • Register an IMessageBusHealthCheck in the services of the message bus to report the state of the transport to the health check. TransportStartedHealthCheck reports a transport or receive endpoint that is not started.

Disposing the runtime without stopping it cancels running handlers#

When the messaging runtime is disposed while it is still started, for example because the service provider is disposed without a host stop, it stops with a cancelled token. Running handlers get no time to finish: their cancellation tokens are cancelled at once. In 16.7 the dispose stopped the runtime with a token that was never cancelled.

Cleanup still runs, each step with its own time limit:

TransportCleanupLimit
RabbitMQDeletes the generated reply queue.5 seconds
PostgresReleases its leased messages and removes its consumer registration.5 seconds each
Azure Service BusRemoves the resources of temporary endpoints.30 seconds

To let handlers finish, stop the host before the service provider is disposed. A host that stops normally does this already.

An Azure Service Bus stop that is cancelled no longer fails#

When the shutdown timeout of the host elapses while an Azure Service Bus receive endpoint waits for its handlers, the endpoint now disposes its processor and completes the stop. In 16.7 the stop failed with an OperationCanceledException, and the resources of a temporary endpoint were not removed.

What you will observe:

  • For a temporary endpoint, the forwarding subscriptions and the queue are removed after the processor is disposed. When this takes longer than 30 seconds, the log shows the Warning Deleting the temporary resources of Azure Service Bus queue '<queue>' did not complete within 00:00:30 and the stop continues.
  • Disposing the processor waits for running handlers. A handler that ignores its CancellationToken delays the stop until it returns.

Health check, log, and trace changes#

The message bus health check is no longer tagged live#

AddMessageBus() registers the health check with the tag ready. In 16.7 it registered the tags ready and live.

  • A health endpoint that filters on the live tag no longer runs the message bus check. With no other check tagged live, it answers Healthy while the application can serve the request.

  • A health endpoint without a filter still runs the check, so it fails during a broker outage and during every re-attach of a consumer. Do not use it as a liveness probe.

  • To keep the tags of 16.7, pass them:

    Diff
     builder.Services
         .AddHealthChecks()
    -    .AddMessageBus();
    +    .AddMessageBus(tags: ["ready", "live"]);

See Readiness and liveness.

The message bus health check times out after 10 seconds#

The check gives the bus 10 seconds to answer the health request. In 16.7 the check had no timeout of its own: an unanswered health request ended after the response timeout of two minutes with a ResponseTimeoutException.

What you will observe:

  • A health request that is not answered in time fails the check with the description A timeout occurred while running check.

  • The health request travels through the broker and waits in a queue of the service like every other request. This can fail the check when the receive endpoint that handles the health request has a backlog of more than 10 seconds. Raise the timeout, or target a specific endpoint:

    C#
    builder.Services
        .AddHealthChecks()
        .AddMessageBus(timeout: TimeSpan.FromSeconds(30));
  • A health request that was not handled within the timeout expires and is dropped when it is received.

  • A failed check reports the failure status of its registration. AddMessageBus() has the new optional parameters name, failureStatus, tags, and timeout. The default failure status is Unhealthy, as in 16.7.

See Configure the health check.

The message bus health check reports transport health#

Before the check sends the health request, it runs the health checks that the transports register. When a transport is unhealthy, the check fails at once with the reason as its description, and the Data of the entry lists each unhealthy transport with its reason. In 16.7 the check only sent the health request.

ReasonReported
Transport '<name>' is not started.Before the runtime has started and after it has stopped.
Receive endpoint '<name>' of transport '<name>' is not started.While a receive endpoint of a started transport is not started.
The dispatcher connection is unavailable.RabbitMQ: while the dispatcher connection is lost.
The consumer connection is unavailable.RabbitMQ: while the consumer connection is lost.
The broker blocks the dispatcher connection.RabbitMQ: during a memory or disk alarm of the broker.
Queue '<queue>': The consumer is not attached.RabbitMQ: from the loss of a channel until the endpoint consumes again. This includes the short time of every re-attach.
Queue '<queue>': The consumer waits for a handler of a lost channel to return.RabbitMQ: while handlers of a lost channel hold every concurrency slot of the endpoint.

What to do:

  • A readiness probe with a failure threshold of 1 fails during every re-attach of a consumer. Use a failure threshold of 2 or more.
  • Change alerts that match the 16.7 description Message Bus is unhealthy. to the new descriptions.
  • The Postgres and Azure Service Bus transports report only whether they and their receive endpoints are started. An outage of their broker or database shows as an unanswered health request.

See Add the health check to ASP.NET Core.

RabbitMQ connections have names#

The two connections of a RabbitMQ transport are named <service> <transport> consumer and <service> <transport> dispatcher. The service is the service name of the host, or the process name when there is none. The transport is the name of the transport, rabbitmq by default. The names appear in the RabbitMQ management UI and in the broker's log.

  • When the connection factory sets ClientProvidedName, both connections keep that name.
  • With a custom IRabbitMQConnectionProvider, the provider names the connections.
  • Change dashboards, alerts, and scripts that select the connections of the application by name.

See Two connections per broker transport.

RabbitMQ log categories, event names, and texts changed#

The RabbitMQ transport writes its log entries under two new categories, and every entry has a new event name and a new text:

16.7 category16.8 category
Mocha.Transport.RabbitMQ.RabbitMQConsumerManagerMocha.Transport.RabbitMQ.Consumer
Mocha.Transport.RabbitMQ.RabbitMQDispatcherMocha.Transport.RabbitMQ.Dispatcher

What to change:

  • Log level settings and filters that name a 16.7 category. A filter on the prefix Mocha.Transport.RabbitMQ keeps matching.

  • Alert rules that match a 16.7 text. Match the event name instead:

    16.7 text16.8 event name
    Connection shutdown detected. ... (Warning)ConnectionLost (Warning)
    Failed to create connection (attempt ...). Retrying in ... (Warning), Reconnection attempt failed (Error), Connection recovery failed. Will retry. ... (Error)ConnectionAttemptFailed (Warning, or Error when the broker refuses the connection the same way on every attempt)
    Failed to create connection after 5 attempts (Error)ConnectionFailed (Error), written only when the start fails
    Failed to reconnect consumer for queue ... (Error)ConsumerAttachFailed (Warning, or Error when the broker refuses the consumer the same way on every attempt)
    Connection blocked: ... (Warning)ConnectionBlocked (Warning)
  • After startup Mocha does not give up on a connection or a consumer, so no entry reports that. To alert on a long outage, use the Elapsed property of ConnectionAttemptFailed and ConsumerAttachFailed. See What to alert on.

Every entry carries properties that identify the service, the connection, and the broker in logger scopes. Keep IncludeScopes = true in the OpenTelemetry logging configuration to export them.

Recovery logs lists every entry with its level, text, and event name.

The receive path logs faults, retries, and redeliveries#

The receive pipeline of every transport writes log entries that 16.7 did not write:

Event nameLevelCategoryWritten
MessageFaultedErrorMocha.Middlewares.ReceiveFaultMiddlewareOnce for every message whose processing failed, with the exception.
FaultDeliveredInformationMocha.Middlewares.ReceiveFaultMiddlewareWhen the fault was sent to the requester or the message was moved to the error endpoint.
FaultDeliveryFailedErrorMocha.Middlewares.ReceiveFaultMiddlewareWhen that send failed.
RetryingMessageInformationMocha.ConsumerRetryMiddlewareFor every retry of a handler, with the exception.
RedeliveringMessageWarningMocha.ReceiveRedeliveryMiddlewareWhen a redelivery is prepared, with the exception.
MessageRedeliveredInformationMocha.ReceiveRedeliveryMiddlewareWhen the redelivery was dispatched.
MessageDiscardedWarningMocha.ConsumerRetryMiddleware, Mocha.ReceiveRedeliveryMiddlewareWhen an exception policy discards a message.
MessageSkippedWarningMocha.Middlewares.ReceiveDeadLetterMiddlewareWhen a message that was not consumed is moved to its skipped or error endpoint.

What you will observe:

  • The log volume rises with the failure rate. Every faulted message adds one Error entry with its exception.
  • Alerts that count Error entries of the application now count faulted messages. Exclude the event name MessageFaulted, or set the log level of its category, where that is not wanted.
  • Every entry carries MessageId, MessageType, CorrelationId, ConversationId, Endpoint, and Transport in a logger scope.
  • Every other entry written while a message is received, including the entries of your handlers, carries the same logger scope.

Circuit breakers log their state changes#

Both circuit breakers of the receive pipeline now log when they open (CircuitOpen, Warning, with the last exception), test recovery (CircuitHalfOpen, Information), and close (CircuitClosed, Information). In 16.7 they logged nothing.

Circuit breakerCategoryText starts with
Receive-levelMocha.ReceiveCircuitBreakerMiddlewareConsumer circuit for endpoint
Transport-levelMocha.TransportCircuitBreakerMiddlewareTransport circuit for endpoint

An open circuit pauses the processing of its endpoint. Alert on CircuitOpen. See Configure the circuit breaker.

The receive span carries the conversation ID#

The tag messaging.message.conversation_id of the receive span now holds the conversation ID of the message. In 16.7 it held the correlation ID. Change trace queries that look up receive spans by correlation ID through this tag. See How correlation works.

Edit this page on GitHub
Maintained by ChilliCream. Last updated on by Tobias Tengler