btrfs data integrity checksum under mdadm raid

Question

I found several places suggesting to use btrfs under mdraid (mdadm) or LVM Raid. But nobody expands on how that works when you have data checksum issues!

My main concern of not using btrfs native raid, is that brtfs have native data checksum. And if I ever have a problem on any of those setup, the two bits diverging will be outside of brtfs domain (either on dmcrypt, or lvm or dm-raid, etc)... How do I go about "fixing" it?

Despite the amount of opinions to not use btrfs raid, I could not find good articles on actual alternatives. My Goal is to have any Mirror, Data integrity/checksum (btrfs or otherwise), and snapshots (for the goal of making incremental remote backups over ssh). Encryption is not required, but not a problem either. Also btrfs is also not required, i might do with ext4 after looking at the options that already cover all that before the fs.

This is about late boot data drives, not root partition.

(Also, note that LUKS+integrity can always be replaced with plain dm-integrity. Just covering 2 cases at once)

option 1:

2 HDD -> mdadm raid1 -> LUKS+integrity -> btrfs

Option 2:

2 HDD -> LUKS+integrityx2 -> mdadm raid1 -> btrfs

Option 3:

2 HDD -> LVM PVx2 -> VGx1 -> LVx1 raid1+raidintegrity -> btrfs

Option 4:

2 HDD -> LUKS+integrityx2 -> LVM PVx2 -> VGx1 -> LVx1 raid1 + raidintegrity -> btrfs

Option 5:

2 HDD -> LVM PVx2 -> VGx2 -> LVx2 linear -> mdadm raid1 -> btrfs

Option 6:

2 HDD -> MBRx2 -> partitionx2 -> dm-integrityx2 -> mdadm raid1 -> btrfs

Option 7:

2 HDD -> mdadm raid1 -> dm-integrity -> btrfs

...Many, many more combinations!

Does any of those non-btrfs raids/integrity will help me deal with btrfs checksum complaining about data integrity mismatch? Or the other way, if mdadm have a /sys/block/mdX/md/mismatch_cnt and I need to pick one of the two HDDs in a RAID1, will brtfs data help me deal with it?

(pls, don't suggest openzfs, already had too many questions closed for "Y for X questions" reasons :)

Answer 1

Does any of those non-btrfs raids/integrity will help me deal with btrfs checksum complaining about data integrity mismatch?

Nope, if you have any kind of duplicity, and a layer that checks whether the read data is correct, that checking layer must be the one selecting the "effective" duplicate. Since the file system always sits "on top" in all these options, that's not possible.

So, you'll want a file system with the integrity check built into its understanding of duplicity.

The closest you get to that is some form of

disk 1 -> [error correcting layer ->] checksumming layer -\
                                                           >RAID 1 -> filesystem
disk 2 -> [error correcting layer ->] checksumming layer -/

because the checksumming layer (a role that LUKS+integrity can play) will signal a failure "upwards", forcing the RAID 1 layer to pick the other copy.

Is that a great solution? Hm. Doing the cryptography twice certainly does sound like a performance downside. (I don't actually know whether you can tell cryptsetup luksFormat that you'd like checksums, but not cryptography? Then you could do the cryptography on top of the RAID 1 layer, below the filesystem layer, should you want encryption. Yay! More Layers! More layers are always good for reliability.)

Honestly, without btrfs or ZFS stock Linux is simply lacking behind other distributed storage solutions (including these in the MS Windows world). But, I don't think that in this time and age we need to count these two options out. btrfs RAID 1 doesn't seem to be in its infacy anymore (the thing where the selection which disk to read from was purely based on whether the reading PID was even or odd are over…), and it seems to be in productive use all over the place, in non-RAID5/6 configurations, at least (it seems better btrfs RAID 6 only arrived with Linux 6.2 (Changelog)).