Can I have more than one outbox in the same application, to have different retry policies?

[lburja]

I would like to have more than one transactional outbox in my application, so that I can configure each one with different attempt frequency and number of attempts.

For example, some operations need to be retried for long a period of time (e.g., attempt for up to 7 days). And for others, a shorter retry policy is desirable (e.g., give up after a few minutes or after a few retries).

I was wondering if the outboxes would interfere with each other in unexpected ways (since they work on the same TXNO_OUTBOX table).

Or if there is there a recommended way to achieve what I want, i.e. to have different retry policies for different operations?

[badgerwithagun]

Interesting idea. There is no easy way to do this.

I think what we probably want is a new option for parameterised calls, e.g.

  outbox.with().retryBehaviour(attempts, timeBetweenAttempts).schedule(MyService.class).doSomething()`

And add overrideMaxAttemptCount and overrideTimeBetweenAttempts nullable fields to TXNO_OUTBOX. to support it.

It's not a tricky change.

[lburja]

I made a pull request following this suggestion

[badgerwithagun]

For now, I don't think there will be any problems running two instances of TransactionOutbox in the same app with different configurations. but I don't think it will achieve what you want, because both instances will compete to process the retries for each others' jobs. The effect you will get is that all jobs randomly use a mixture of the two behaviours depending on which instance does the work.

[lburja]

If the two instances are competing for retries, it won’t be helpful in my case. I had a feeling this could happen, so thanks for confirming it.

I like the other proposal, but I understand it requires development. I’ll take a look at the code, but if it involves adding new columns, then I suspect it won’t be as straightforward as it sounds.

[badgerwithagun]

@lburja and @nishiol: apologies for the delay in responding two these two PRs. #739 and #790

@victokoh also wants to adapt the retry approach, but in their case they want to allow for exponential backoff. I think we need a unified approach here.

My suggestion is to have an interface something like this:

public interface RetryStrategy<T extends RetryOptions> {
  void setClockProvider(ClockProvider clockProvider);
  void setOptions(T options);
  void setNextAttemptTime(TransactionOutboxEntry entry);
  void handleFailure(TransactionOutboxEnty entry, Exception e);
}

public interface RetryOptions {
  Class<? extends RetryStrategy<?>> retryStrategy();
}

And implement the current behaviour like this:

@Value
public class DefaultRetryOptions implements RetryOptions {
  int blockAfterAttempts;
  Duration attemptFrequency;

  @Override
  Class<DefaultRetryStrategy> retryStrategy() {
     return DefaultRetryStrategy.class;
  }
}

@NoArgsConstructor
public class DefaultRetryStrategy implements RetryStrategy<DefaultRetryOptions> {

  @Setter
  private ClockProvider clockProvider;

  @Setter
  private DefaultRetryOptions options;

  @Override
  public void setNextAttemptTime(TransactionOutboxEntry entry) {
    entry.setNextAttemptTime(clockProvider.get().instant().plus(options.getAttemptFrequency()).truncatedTo(MILLIS));
  }
  
  @Override
  public  void handleFailure(TransactionOutboxEnty entry, Exception e) {
     entry.setAttempts(entry.getAttempts() + 1);
     var blocked = (entry.getTopic() == null) && (entry.getAttempts() >= blockAfterAttempts);
     entry.setBlocked(blocked);
  }
}

You can then just pass an instance of RetryStrategy to your ParameterisedTransactionOutboxBuilder:

outbox
  .with()
  .retryOptions(new DefaultRetryOptions(3, Duration.ofMinutes(5))
  .schedule(SomeClass.class)

Then TransactionOutbox uses Instantiator to create an instance of RetryStrategy from RetryOptions.retryStrategy() (this allows us to serialise the request but allow the user to create a RetryStrategy that uses their preferred DI framework if they wish):

var retryStrategy = instantiator.createInstance(retryOptions.retryStrategy());
retryStrategy.setClockProvider(clockProvider.get());
retryStrategy.setOptions(retryOptions);
retryStrategy.handleFailure(entry, e);

You can also then add a defaultRetryOptions property on TransactionOutbox which allows a custom implemention to be provided on setup. The existing attemptFrequency and blockAfterAttempts properties can just be used to set up the default implementation.

null on a TransactionOutboxEntry can be assumed to mean "use the default from TransactionOutbox.

How does this sound?

[badgerwithagun]

An exponential backoff strategy could look like this:

@NoArgsConstructor
public class ExponentialBackoffStrategy implements RetryStrategy<DefaultRetryOptions> {
  
  @Setter
  private ClockProvider clockProvider;

  @Setter
  private DefaultRetryOptions options;
  
  @Override
  public void setNextAttemptTime(TransactionOutboxEntry entry) {
    entry.setNextAttemptTime(
      clockProvider.instant().plus(
        Duration.ofMillis(Math.pow(options.getAttemptFrequency().toMillis(), entry.getAttempts))
      ).truncatedTo(MILLIS) 
    );
  }
  
  @Override
  public  void handleFailure(TransactionOutboxEnty entry, Exception e) {
     entry.setAttempts(entry.getAttempts() + 1);
     var blocked = (entry.getTopic() == null) && (entry.getAttempts() >= blockAfterAttempts);
     entry.setBlocked(blocked);
  }
}

[badgerwithagun]

@victokoh and I are talking this over now to implement a starter solution