Keep two entire Linux desktop systems in sync

Question

I have a laptop PC that I use for everyday tasks. I mainly use my PC for software development (so lots of git repos + IDE + development tools), browsing... lots of different stuff actually. I'm using Debian testing with almost all software coming from package repositories (apt).

I would like to build another system (a desktop PC, same CPU architecture, i.e. amd64) that shall have the same data and system configuration: that is, I would like to seamlessly (not necessarily in real time though) switch from the laptop to the desktop (and back) and continue my work there.

I did some research and I found several challenges:

• different hardware configuration might lead to some stuff being different (kernel version and/or modules, /etc/modprobe.d, /etc/modules-load.d and so on)
• not everything in the system state might be reflected by the file system (unless I sync a "cold state", e.g. during boot and shutdown)
• conflict management
• if the sync is done during boot, what to do if it fails?
• how to handle installed packages? (i.e. via "manifest files" + redo installs/removals vs sync the actual package contents in the file system + /var/lib/dpkg)

I don't need version tracking, just that the two systems are in sync at some point in time (with possibly only one of them being active at any given time - a third "bridge" system would be needed of course).

Some thoughts, off the top of my head:

• maybe handle user data (e.g. /home) and the system (/usr, /var, /etc) with separate tools (I'm thinking git vs rsync)
• near-realtime sync ("hot" sync) vs sync on boot/shutdown ("cold" sync - probably safer?)
• system-dependant and ephemeral stuff should be excluded somehow

I don't want to use remote desktop: the purpose of this exercise is also disaster recovery, so both systems should be ready to use at a moment's notice.

Of course classic configuration management tools (Puppet, Ansible, etc.) are not suitable for this IMHO because the state of the system changes often (because it's a personal computer for personal use).

Am I overengineering it?

Answer 1

Starting from the bottom:

maybe handle user data (e.g. /home) and the system (/usr, /var, /etc) with separate tools (I'm thinking git vs rsync)

Right approach! In fact, most distros, debian including, stick to some standards for directories; that allows you to get the same systems if you keep the apt-installed software in sync, the configuration in /etc selectively in sync (there's things like your machine name, some storage configuration, /etc/fstab, host name, your power settings… configured there, and you might want to sync some, but not all of it); /var/lib for mutable but persistent state (from containers to editor plugins).

When you have some non-apt-installed software in /usr/local, you need to look at that individually; generally, /usr/local/etc, and /usr/local/var/lib might be things that are worth syncing.

I don't know whether everything in home (excluding ~/.cache, and similar to /etc, treating ~/.config selectively) is appropriately handled by git – actually, no, it's not. But rsync, restic etc are suitable methods for synchronization.

near-realtime sync ("hot" sync) vs sync on boot/shutdown ("cold" sync - probably safer?)

dangerous for things outside of /home/, because the services of your system will already be accessing files; you don't want to randomly install stuff during bootup or shut down (especially not the latter, because how do you even notice if something went wrong?).

So, I'd go with: "hot" sync user data, but leave software / system sync up to dedicated reboots. Split system sync into two jobs: making sure you have the same packages installed as the other machine, that's a job for apt-get, and syncing /etc and /var/lib, that's a job for rsync.

I'd probably just create a file (e.g., /var/lib/syncnextboot) and have a systemd unit activated at boot, that checks for the existence of said file, exits successfully if it isn't there, and otherwise fetches the package list from wherever, runs apt-get to install that exact list of packages, then rsyncs over the right parts of /etc and of /var/lib, and only after all that has been successful, deletes the sentinel file and exits successfully; otherwise it just exits unsuccessfully (your system is then in what sytemctl would call "degraded state", but that's not a problem other than not being synced).

The /home rsyncing can be done opportunistically as service that depends on Network.target and is launched when online, and checks for reachability of the coordination server.

---

This all sounds like you might like layered, immutable-base operating systems, which make stronger guarantees on the ability to take all changes to a system and transport them anywhere.

In the Fedora world, such an OS would be Fedora Silverblue. I'm not sure whether there's an equivalent debian-based OS. Wikipedia suggests "Endless OS", but I'm always wary of distros that I haven't heard of; they typically have problems that noone has heard of before, too ;) But you could give it a try if all the software you need comes with it or is available as flatpak, or if you're not married to debian, try Silverblue (which, unlike Endless not only allows you to install software via flatpak, but also to define software layers, which you can modify to install "normally" packaged (i.e., installed via dnf/rpm, fedora's apt-get/dpkg) software).

---

different hardware configuration might lead to some stuff being different (kernel version and/or modules, /etc/modprobe.d, /etc/modules-load.d and so on)

mostly not a concern, usually solved by simply selectively excluding the packages bringing the graphic cards drivers for the laptop on the workstation and vice versa.

not everything in the system state might be reflected by the file system (unless I sync a "cold state", e.g. during boot and shutdown)

But that's also not state that makes sense to replicate, so, don't care!

conflict management

Hard because hard. I'd advocate for having the self-discipline to not uninstall software using apt on one device while installing it on another. The rest might be fairly rsyncable.

how to handle installed packages? (i.e. via "manifest files" + redo installs/removals vs sync the actual package contents in the file system + /var/lib/dpkg)

Yes! Core problem! Luckily, as linked to above, keeping the packages in sync is the core solution to the core problem, and apt-get and dpkg make that easy.

Answer 2

A modern addition to what has been said: You can build upon the UBlue project, which itself builds upon Fedora's atomic desktop images (aka Fedora Silverblue or Kinoite etc.).

Of course data and configs may need to be synced with a method you wish (e.g. for user dotfiles/configuration using a GitHub repository or special apps like chezmoi or dotz), but for system apps and system adjustments you can easily built your own system image with UBlue.

So for any system adjustments you might want, you can use a cloud-native approach and fork a GitHub project, where everything to building an image is automated. You can then share and use this image with all your systems. Furthermore, you can also flexibly create and build it upon different base images like Bazzite for gaming. And you will keep all (security) updates form "upstream" as it just works with a Dockerfile building upon existing images.

If you're not just an "end-user" and want to do it more low-level, you can of course also define your own "bootable containers" with bootc, which is the tool this is all based upon.

Answer 3

For user data synchronization, I have found that syncthing works well. It practically instantly detects changed files, so it is possible to rapidly switch from one computer to another and continue working on the same files.

Myself I have found syncing user files sufficient and haven't bothered synchronizing the system. This does result in some extra maintenance burden, so I understand the desire.

---

For system files, I would recommend declaring one system as primary, and do all the major system changes there. If you use btrfs or zfs, you can setup snapshots of the system state. To synchronize the two systems, you would have a script that does:

1. In initial setup, create prev-sync on primary system from current state, and use btrfs-send to transfer it in full to the secondary system. Copy prev-sync on secondary to active and configure system to use that as the root partition.

Then for each incremental synchronization:

1. Take a new snapshot on primary, next-sync (replacing any interrupted one if such exists).
2. Use btrfs-send to transmit difference between next-sync and prev-sync to secondary system.
3. Rename next-sync to prev-sync on both systems.
4. On secondary system, rename old active to prev-active and copy prev-sync to active.
5. Secondary system will continue to run on the old state until next reboot. You don't have to reboot immediately.

Benefit of btrfs-send or zfs-send is that it doesn't have to spend time searching for differences in files. The differences are readily available thanks to the copy-on-write mechanism in the snapshots. For send/receive to work, you always need to have the unmodified prev-sync available on the receiving side. If you do local modifications to active on the secondary system, they will only be present until next sync.

With snapshots, you can have a second Grub boot option that instructs kernel to mount prev-active instead. That way you always have a quick way to recovery if a sync fails.

Answer 4

In the details, laptops and desktops are different (not the same processor, not the same optimizations, perhaps not the same kernel and compiler versions, often not the same graphical cards and screen sizes).

But as a software developer you mostly use some version control (e.g. git) on the software you are coding.

I'm in the same position as you and just use git to maintain the same source code base on both machines.

Of course it really depends on the kind of software you code. For an example OpenCL source code is extremely dependent on the hardware.

And for GUI or web development, you could have different fonts installed on the two computers. And different preferred window sizes or colors.

Also two different Linux distributions have different kernels, different libraries versions, different compilers or debuggers etc.

For multithreaded applications the number of cores also matters.

PS.People ask me about my afiliation. I am retired but do want to teach part-time. And am coding RefPerSys. It is none-sense that stackoverflow cannot be contacted by email or web forms.

Answer 5

Possibly define the whole thing as a VM and migrate it between the two as needed. This would abstract the hardware differences away to an extent. Not sure how well X/Wayland work virtualised...