Small Grant: Sia Virtual Block Device (sia_vbd)

Grants Complete
21

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

22

Hey @rrauch, this is a reminder to submit your progress report for January.

23

Apologies for the delay, I spent the last few weeks in heavy crunch mode in order to get the release out.

Here is my belated progress report:

January 2025 Progress Report

What progress was made on your grant this month?

  • Developed a data serialization format that serves as the foundation for all permanently stored data.
  • Implemented the Write-Ahead Log (WAL) with support for transactional writes.
  • Created the Inventory system to keep track of all components.
  • Built Chunks with Zstd compression and indexing.
  • Implemented the Repository for centralized storage.
  • Updated the renterd_client Rust library, adding necessary functionality for the Repository.
  • Added automatic crash recovery.
  • Enabled automatic syncing of the repository.
  • Introduced WAL Garbage Collection.
  • Rewrote Chunk indexing to overcome HTTP header size limitations by introducing Manifests.
  • Testing, testing and more testing.
  • Developed a user-friendly CLI for easy volume creation and deletion.
  • Added support for Docker and systemd.
  • Released Milestone 2.

Links to repos worked on this month:

What will you be working on this month?

  • Implement caching, which is the most critical feature still missing.
  • If time allows, add Chunk Garbage Collection.

These last few weeks have been extremely busy, and I had to deal with a significant setback.
But in the end, a lot of progress was made. The difficult parts are all done, and I don’t expect any major issues with the remaining functionality.

24

Milestone 2 Released

Milestone 2 of sia_vbd, the first beta release, is now available!

This version is almost fully functional:

  • NBD (Network Block Device) support
  • Cross-Platform: Runs on every platform where renterd is available.
  • Immutability: Writes never modify existing data; any change leads to a new overall state (Snapshot). Previously
    held data remains available (until eventual garbage collection).
  • Content-Addressed Storage: All data is hashed and identified by its content ID for integrity and deduplication.
  • Content Compression: Transparently compresses content (Zstd) before uploading.
  • Transactional Writes: Atomic writes with automatic rollback on failure.
  • Write-Ahead Logging: Records transactions in a local, durable WAL before being committed to eventual storage.
  • Crash Tolerant: Detects when the local, WAL-recorded state is ahead of the committed backend state.
  • Background Synchronization: Continuously uploads new data to the backend in the background, allowing fast writes
    and avoids blocking reads.
  • Multiple Block Devices and Backends: Supports multiple block devices, across one or more renterd instances.
  • Single Binary, Single Process: Delivered as a single, self-contained binary that runs as a single
    process, making deployment easy and straightforward.
  • Highly Configurable: While coming with reasonable default settings, sia_vbd offers many additional options to
    configure and fine-tune.
  • CLI Interface: Includes an easy-to-use CLI for common operations.
  • Docker and systemd support,

However, some functionality is still missing:

  • Caching: Caching is not yet implemented. Without caching, most data must be re-read multiple times from
    renterd, resulting in very slow performance due to the high latency of each read operation. Performance
    will improve significantly once caching is in place.
  • Garbage Collection: Garbage collection is currently not available, causing volumes to grow indefinitely.
    Implementing GC will allow obsolete data to be deleted over time.
  • Resizing: Block Devices can not be resized for the time being.
  • Branching CLI Support: Although branching functionality has been implemented, users currently cannot interact with
    it. CLI functions will be added to enable branch operations.
  • Tags: Tagging is not currently supported.

Test Drive

A Docker image is available to give it a quick try:

docker pull ghcr.io/rrauch/sia_vbd

docker run -it --rm ghcr.io/rrauch/sia_vbd --help

This release lacks caching, so performance will be much slower compared to the upcoming release.

More details about how to use sia_vbd can be found here:

1 Like
25

Milestone 3 Released

Milestone 3 of sia_vbd, the second beta release, was released a few days ago!

This version added the most important functions that where still missing in the previous release:

  • Caching: The previous release lacked any sort of caching, so performance was very slow. M3 contains a persistent caching layer for both, metadata as well as block data. The cache is configurable and is structured into 2 levels: L1 (in-memory) and L2 (on-disk).
  • Garbage Collection: Due to the lack of GC in the previous release, volumes would grow indefinitely. In this release automatic garbage collection is performed in the background. Unreferenced data will be deleted eventually.

Milestone 3 is feature complete with the exception of the following:

  • Resizing: Block Devices can not be resized for the time being.
  • Branching CLI Support: Although branching functionality has been implemented, users currently cannot interact with it. CLI functions will be added to enable branch operations.
  • Tags: Tagging is not currently supported.

The next release will contain these missing functions and will be feature complete.

Test Drive

The Docker image has been updated and is available here:

docker pull ghcr.io/rrauch/sia_vbd

docker run -it --rm ghcr.io/rrauch/sia_vbd --help

Detailed usage instructions and examples can be found here:

26

February 2025 Progress Report

What progress was made on your grant this month?

  • A persistent caching layer was added to significantly improve performance.
  • Implemented two-level caching: L1 (in-memory) & L2 (on-disk)
  • Added configurability for the cache. Resource limits and file system path can be configured on a per-volume basis.
  • Enabled automatic tracking of unreferenced (obsolete) data and metadata.
  • Introduced automatic background garbage collection
  • More testing was performed
  • Additional sections have been added to the README, with a detailed list of all configuration options, as well as explanations of the concepts behind sia_vbd and the terminology used.
  • Released Milestone 3.

Links to repos worked on this month:

What will you be working on this month?

  • Implement the last missing features: Resizing, Branching & Tagging
  • Release!
27

Version 0.4.0 Released

Version 0.4.0 of sia_vbd is out!

Progress Since the Previous Release

This release adds all remaining features that were still missing in the previous release:

Branching

Volumes can have more than a single branch. New branches can be created from any existing branch, tag or commit. Branches can be instantiated, modified and deleted without affecting the state of any other branch. Please note: Only one branch can be active at any given time.

sia_vbd branch --help

Tagging

Tags are very similar to branches and can also be created from any existing branch, tag or commit. The main difference is that tags cannot be instantiated. However, they can be used as a source of a new branch. Any data associated with an existing tag is guaranteed to not be garbage collected.

sia_vbd tag --help

Resizing

Existing Volumes can be freely resized with the CLI. Resizing only works while the Volume is stopped. Please be careful when shrinking: any data beyond the shrink-point will be lost!
Resizing only affects the selected branch, so it’s possible to create a tag or branch before resizing and roll back in case of accidental data loss.

sia_vbd volume resize --help

Get sia_vbd

sia_vbd is available from its Github Repository:

The Docker image has been kept up-to-date and is available here:

docker pull ghcr.io/rrauch/sia_vbd

docker run -it --rm ghcr.io/rrauch/sia_vbd --help

Detailed usage instructions, including configuration options and examples can be found in the Readme.

Caveat

sia_vbd does currently NOT support the recently released renterd version 2 due to a number of breaking API changes.

28

March 2025 Progress Report

What progress was made on your grant this month?

  • Implemented all remaining features: Resizing, Branching, and Tagging
  • Released the feature-complete version 0.4.0

Links to repos worked on this month:

What will you be working on this month?

  • I consider this grant to be complete as everything outlined in the proposal has been implemented and released.

Note

Both sia_vbd and my previous project sia_nfs are currently not working properly with renterd v2 due to breaking API changes.
I had planned to update both projects in March to keep them working, but progress is currently blocked by renterd issue 1862.
Once this issue is resolved, I’ll try to make time to update both projects, along with the shared renterd_client library they both use.

29

Thanks for your final report and congrats on the completion of your grant! We’ll reach out with offboarding information.