Q: about bad chunks

[boelle]

my friend in the UK has this Q....

if a chunk fails, can we work out where that is against the incremental backups and recover that chunk or the files on or affected by that chunk

we plan to do periodic checks of our backups to detect if they become corrupt,

we do nightly short self test and the long test once a month, and before a drive is put in to service we test the entire disk for bad blocks and the smart self test, i dont think we can do much more as drives are concerned... is that somewhat a corect view?

as for the memory we plan to do memtest every 6 months... and while that runs we check the drives again for bad blocks

we use mergerfs and it seems that in some edge cases where disks are near full can interrupt a backup in a way that is not desired, so as an experiment we have reduced the chunk size to be max 100mb

internet connection could also somewhat have a say if the connection is broken at the wrong point

but... lets say we get a bad chunk but data access is still good, what is the best way to move forward? do a check with --repair, then add another backup and do a another check?

and is it possible to know what files are affected by a bad chunk if there are no access to the data?

[ThomasWaldmann]

borg check (repository part) might find defect chunks based on a crc32 mismatch - in such a case it will tell you the chunkid and the segment/offset where it happened.

borg check (archives part) will read all archives and check whether all referred chunks exist - if you add --verify-data it will also do a cryptographic verification of the chunk contents (which is much slower, but more thorough).

borg check --repair will delete defect chunks and change all references to them to refer to all-zero replacement chunks of same size. it will still remember the chunkid of the correct/missing chunk inside the archive metadata.

if you then do another backup and that backup creates the missing chunk again, the next borg check --repair will heal all the places that referred to that chunk.

borg list REPO::ARCHIVE has a {health} placeholder that will show for each file whether it is ok or has replacement chunks.

you're doing quite a lot to make sure your hw works ok. asking myself whether you might like zfs in a raid or mirrored setup - it does quite a lot to make sure data is correct and nothing gets lost: it has strong checksums for all data (so it knows which part of a inconsistent mirror is correct), it does a weekly scrub to check whether all data reads ok.

i can't say anything about mergerfs, never used it myself. but i can say that a correctly and reliably working fs is essential for a borg repository.

if a borg backup gets interrupted in the middle (machine crash or breakdown of borg client/server ssh connection), usually nothing really bad should happen: the next time you start borg, it will notice the incomplete transaction and roll back to last consistent state.

[boelle]

what we use mergerfs for is to "merge" all disk as one big one, like jbod on windows, with the difference that each still can be read individual as where raid normal breaks if a disk (or to many) bites the dist, even worse if the controller goes bad

snapraid then kind of adds the missing link in that it takes the biggest disc and use that for parity

i will try and look at zfs later in the evening

but at least i got the "for dummies" explanation on how it handles defect within the repro

[ThomasWaldmann]

worse if the (RAID) controller goes bad

That's another big plus of zfs: you don't need a RAID controller (they even strongly advise against it), just any "dumb" SATA, SAS, NVME or ... adapter - much easier to replace in case of a controller/adapter defect.

[boelle]

i had a quick look and not figured it yet, but what i one disk fails with zfs? i can still recover right? but what if more than that fails? will i need more parity space? its the same with snapraid, it can only recover one drive with 1 drive of parity, if you want to recover from 2 failed drives then you need another parity drive

[ThomasWaldmann]

mirror and raidz1 offers 1 redundant disk, raidz2 offers 2.