Introduction
Project Name: Sombrero
Name of the organization or individual submitting the proposal: Michael Bulanov
Describe your project
Sombrero is the former SiaSMB project rebranded according to the Sia Foundation’s branding guidelines. The name was chosen by analogy with Samba, the software suite implementing the SMB protocol support for Linux and MacOS.
Now that indexd has been officially released, and the source code is publicly available, any remaining open items related to the integration shall be finalized. These include:
- authenticating different indexer accounts with the SMB server, so each account could only access their own files
- packing incomplete slabs and uploading the packed slabs in the background
- routinely checking the uploaded packed slabs for fragmentation and repacking/re-uploading them when the threshold is exceeded
- enabling “public” folders, where each account belonging to a workgroup could see the data uploaded by another account in the same workgroup
Who benefits from your project?
Mostly users who want to access the decentralized storage the way they are used to: by mounting Sia as a remote share and doing anything they would usually do with such share. Possibly also developers who want to use Sombrero as a building block.
How does the project serve the Foundation’s mission of user-owned data? What problem does your project solve?
The project contributes to lowering the friction for using Sia as the storage layer by integrating into the user’s usual work environment, with minimum to zero additional setup steps required.
Are you a resident of any jurisdiction on the restriction list? No
Will your payment bank account be located in any jurisdiction on that list? No
Grant Specifics
Amount of money requested and justification with a reasonable breakdown of expenses: 24,000 USD for the total duration of 4 months, 100% of which shall constitute the developer fee.
What is the high-level architecture overview for the grant, i.e. how does your project build on Sia?
Sombrero continues to support both renterd- and indexd-backed remote shares. An arbitrary number of shares of either type can be registered. Workgroups and accounts created therein can be granted a customized access to each share.
Upon connecting to a remote share, the user’s device connects to the SMB server. This server can be running on a public host or locally, on the same device or on the LAN. When the user tries to upload or download any files, the server issues write or read requests, which are processed by the indexd SDK connected to either a local or a remote indexer. The file and directory metadata are stored on the same machine where the SMB server is running.
The SMB server also serves an admin API, which allows to register and delete accounts and shares and customize access to each share. This API is only available locally.
What security best practices are you following?
Most of the security-related part has already been implemented in SiaSMB. User authentication occurs via the NTLMv2 protocol, which is not the most secure one, but it is quite robust and does not require any additional setup steps on the user’s side. The other available authentication protocol, Kerberos, is more secure, but is also more difficult to set up, because it requires operating an Active Directory domain controller (KDC) and is therefore more suitable for enterprise users.
The admin API is served on the port 9999 (default), which should never be exposed.
Timeline with measurable objectives and goals
Milestone 1 (1 month)
- Extend the authentication to support multiple
indexdaccounts.
Currently, the server can only register one account at eachindexdinstance. This means that each user account or workgroup registered with the SMB server will share the same storage quota. The proposed refactor shall enable multiple accounts each using its own quota. - Enable public folders on the individual workgroups.
The idea is to allow users of the same workgroup create shared folders within the workgroup’s workplace (e.g.public), where the users could place common files and/or directories. - Add missing API endpoints or rewrite existing ones.
The changed database schema will almost surely require new and/or modified API handlers (see above). - Extend the unit tests to cover the changed database schema.
Milestone 2 (1 month)
- Implement packing incomplete slabs and uploading them.
The existing code stores the written data locally chunk-wise. Every complete slab is then uploaded to the Sia network by a background thread. Any incomplete slabs stay local. The proposed change shall implement packing incomplete slabs and uploading the full slabs by another background thread. - Ensure correct downloads from the mixed slabs.
Whenever a slab contains data from several different files, that data needs to be correctly pulled by the download code. - Extend the test coverage.
Milestone 3 (1 month)
- Implement slab fragmentation check.
Users connected to an SMB share shall use that share like a normal remote share. Under those conditions, files get modified or deleted very often. There needs to be a thread running that will be continuously checking the stored metadata for any slabs that got fragmented beyond a certain threshold. - Repack and re-upload fragmented slabs.
If any slab gets fragmented beyond a certain threshold, it needs to be repacked and re-uploaded, and the metadata updated accordingly. - Update the public test server and announce the new round of testing.
Milestone 4 (1 month)
- Collect and incorporate the community feedback from the testing.
- Implement re-pinning metadata from one indexer to another (or from
renterdtoindexd).
This ensures that the users can always switch from one indexer to another whenever they like, or move from the data stored inrenterdto the indexer-based model.
Who is the target user for your project?
- users who want to access the decentralized storage the way they are used to: by mounting Sia as a remote share and doing anything they would usually do with such share
- frontend developers who want to use Sombrero as the backend
What are your plans for this project following the grant?
- enable the option of running Sombrero as a “lite” version, for those who still want to run a local SMB server backed by
renterd(without needing to install PostgreSQL) - add multi-channel support (currently only supported by Windows SMB clients)
- consider exporting the file metadata in some usable format (e.g. SQL) to enable backups
- continue maintaining the project; monitor any bugs discovered or any significant changes in the
indexdcode
Potential risks that will affect the outcome of the project:
- Re-pinning metadata from one indexer to another might not work out as expected. As this is the final milestone, it should not affect the rest of the project.
Development Information
Will all of your project’s code be open-source? Yes
Leave a link where code will be accessible for review.
GitHub - mike76-dev/siasmb: SMB protocol implementation for Sia cloud storage
SMB protocol implementation for Sia cloud storage
(the repo will be renamed to sombrero as soon as the work starts)
Do you agree to submit monthly progress reports? Yes
Contact Information
The contact details of the project developer are already known to the Foundation.