__transaction_buffer_snapshot corruption recovery

[mrtworo]

Hey All!

We are looking for the best practices in terms of recovery in below situation, thanks!

We have a ledger marked as under-replicated:

INFO  org.apache.bookkeeper.tools.cli.commands.autorecovery.ListUnderReplicatedCommand - 25082
INFO  org.apache.bookkeeper.tools.cli.commands.autorecovery.ListUnderReplicatedCommand - 	Ctime : 1731526130312
INFO  org.apache.bookkeeper.tools.cli.commands.autorecovery.ListUnderReplicatedCommand - Under replicated ledger count: 1

It belongs to a system topic:

persistent://U/VTN/__transaction_buffer_snapshot

Which is in following state:

{
  "version": 42,
  "creationDate": "2024-11-07T06:56:46.373Z",
  "modificationDate": "2024-12-30T13:01:25.318Z",
  "ledgers": [
    {
      "ledgerId": 25081,
      "entries": 1,
      "size": 189,
      "timestamp": 1731520179032,
      "isOffloaded": false
    },
    {
      "ledgerId": 25141,
      "entries": 1,
      "size": 190,
      "timestamp": 1731534988489,
      "isOffloaded": false
    },
    {
      "ledgerId": 504253,
      "timestamp": 1735651074124,
      "isOffloaded": false
    }
  ],
  "cursors": {
    "__compaction": {
      "version": 27,
      "creationDate": "2024-11-07T06:56:46.403Z",
      "modificationDate": "2024-12-30T13:01:23.988Z",
      "cursorsLedgerId": -1,
      "markDelete": {
        "ledgerId": 25015,
        "entryId": 0
      },
      "properties": {
        "CompactedTopicLedger": 25082
      }
    }
  }
}

CompactedTopicLedger is the ledger marked as under-replicated and we cannot recover it.

As far as I understand this is the transaction buffer snapshot topic and it is required to be available for topics recovery to work.

Also have multiple topics unavailable
e.g.:

WARN  org.apache.pulsar.broker.transaction.buffer.impl.TopicTransactionBuffer -
 Closing topic persistent://U/VTN/m-3-test-input due to read transaction buffer snapshot while recovering the transaction buffer throw exception

Is there a way to recover from this state without recreating the affected topics?
Could setting autoSkipNonRecoverableData to true help in this case?