bmapfs

Rsync seems to have problems syncing huge files (e.g. VM images).

The idea of Bmapfs is to represent the huge VM image as a directory of smaller files that can be syncronized using Rsync. It maps a file (or block device) into smaller files representing individual blocks of the underlying file (or block device). It works for read and write access.

Build instructions are place near the end.

Usage

1. Mount the image or block device you want to sync:

   bmapfs [options] SRC MOUNTPOINT
   

   If you intend to write to the mapped files, the option -rw is mandatory.

   Alternatively you can use the mount(8) command, if bmapfs is installed as mount.bmapfs in your search path:

   mount -t bmapfs SRC MOUNTPOINT
   

2. Backup the bmapped device or image, e.g. using Rsync:

   rsync --inplace -a MOUNTPOINT/ BACKUP/
   

   The --inplace option is not required, but may improve the speed by reducing file copy actions on the destination side.

3. Restore to the bmapped device or image, e.g. using Rsync:

   rsync --inplace -a BACKUP/ MOUNTPOINT/
   

   Using --inplace is mandatory, as otherwise Rsync's method of replacing files isn't compatible with how Bmapfs works (the directory containing the mapped blocks is readonly.)

Build

To build Bmapfs you need a decent Go (I used 1.19.5). Just run the usual go build command inside the sources. The built binary can be installed wherever you want, you can even use go install to let Go decide where to install it.

Timing

• 100GiB image (LV), 1GiB network link, 4MiB bmapfs blocks

  • initial sync: 15 minutes

    sent 107,401,795,796 bytes  received 486,447 bytes  114,807,356.75 bytes/sec
    total size is 107,374,182,400  speedup is 1.00
    
    real    15m35,207s
    user    4m54,115s
    sys     3m29,334s
    

  • immediate re-sync including checksumming: 14 minutes (both sides have to re-read their data)

    sending incremental file list
    
    sent 810,743 bytes  received 12 bytes  968.07 bytes/sec
    total size is 107,374,182,400  speedup is 132,437.27
    
    real    13m56,968s
    user    0m20,070s
    sys     0m54,800s
    
    

  • resync w/o checksumming after some changes (~1GiB) on the source volume

    sent 226,288,995 bytes  received 315,078,483 bytes  1,185,909.04 bytes/sec
    total size is 107,374,182,400  speedup is 198.34
    
    real    7m35,609s
    user    1m14,816s
    sys     0m55,372s
    

    If only a small part of the data changed, the most time is consumed for comparision of both sides.

Known issues

• For known issues see the issue tracker.