Virtual block device backed by multiple files

Question

On Linux (say, Ubuntu/Debian) I would like to create a virtual block device (let's say /dev/mapper/myvbd ) that is backed by a bunch of files on the user's home (say /home/myuser/myvbdfiles/file[1...100] ).

If it were a single file I could do it trivially using losetup, but what I would like to do is writing an application or a kernel module that, while in execution, creates the virtual block device and maps I/O requests made by the user on that device to arbitrary positions on any of the files on disk (according to an algorithm that I want to write, maybe provided by a library).

I have written a test proof of concept using FUSE and Python, but I would like to do it in C. What do you think is the best way to do it? Any hint or resource I can look at?

Answer 1

If the mapping is fixed, i.e. 512-byte sector i is always in file f_i sector b_i, then you can use a device mapper table device via dmsetup.

Each line in the table file specifies a mapping. In your case, they could be simply

    i   1   linear   f_i   b_i

---

It might be interesting to create a derivative of dm-switch, say dm-chunkmap, that redefines region_table as a per-chunk mapping (with a configurable number, 2ⁿ 512-byte sectors per chunk): low bits specify the device/file, and high bits the target sector on that device.

Using 32 bits per table entry, the maximum size would be two terabytes (but 8 MiB of kernel RAM would be needed per gigabyte for the mapping); using groups of eight sectors 16 TiB (and only a megabyte of kernel RAM per gigabyte of block device would be needed).

Something along the lines of

/* Each chunk has device index in low bits,
   and target sector number in upper bits. */
typedef unsigned chunk_t;

/*
 * Context block for a dm chunk-mapping device.
 */
struct chunkmap_ctx {
    struct dm_target *ti;
    unsigned nr_paths;        /* Number of paths (devices) in path_list. */
    unsigned dev_shift;      /* nr_paths <= 1 << dev_shift */
    unsigned dev_mask;       /* (1 << dev_shift) - 1 */
    unsigned chunk_shift;    /* 1 << chunk_shift sectors per region */
    unsigned chunk_mask;     /* (1 << chunk_shift) - 1 */
    unsigned long nr_chunks; /* Number of regions making up the device */
    chunk_t *chunk_table;
    struct dm_dev *dev_list[]; /* Devices */
};

static int chunkmap_map(struct dm_target *ti, struct bio *bio)
{
    struct chunkmap_ctx *rctx = ti->private;
    sector_t offset = dm_target_offset(ti, bio->bi_iter.bi_sector);
    chunk_t chunk = READ_ONCE(rctx->chunk_table[offset >> rctx->chunk_shift]);
    bio_set_dev(bio, rctx->dev_list[chunk & rctx->dev_mask]);
    bio->bi_iter.bi_sector = (sector_t)(chunk >> rctx->dev_shift) << rctx->chunk_mask
                           + (offset & rctx->chunk_mask);
}

static struct target_type chunkmap_target = {
    .name = "chunkmap",
    .version = {1, 1, 0},
    .module = THIS_MODULE,
    .ctr = /* chunkmap_ctr */,
    .dtr = /* chunkmap_dtr */,
    .map = chunkmap_map,
    .message = /* chunkmap_message */,
    .status = /* chunkmap_status */,
    .prepare_ioctl = /* chunkmap_prepare_ioctl */,
    .iterate_devices = /* chunkmap_iterate_devices */,
};

which allows using chunks of 2^chunk_shift sectors (2^{9 + chunk_shift} bytes) per chunk, and up to 2^dev_shift target devices/paths, with the chunks ordered on the target devices completely arbitrarily.

It should be noted that the structure does allow mapping different sectors to the same target device sector, but this should be considered an error, since it can result in errors. In other words, it is a good idea to only allow unique chunk_table entries.

If there is an use case for something like this, outside of one-off experiment, I'm pretty sure it could be pushed into mainline kernels via dm-devel mailing list.