Small Grant: SIA FUSE Filesystem

Grants Approved
1

Project Name:
SIA FUSE Filesystem (aka siafs)

Name of the organization or individual submitting the proposal:
OKay INC
Luis Daniel Lucio Quiroz (CEO & Developer). Small business owner if you prefer.

Describe your project.
The siafs, as its name suggests, will allow the creation of a filesystem mount through the libfuse library. Like any FUSE file system, users will interact with the hosted SIA files as if they were local.

How does the projected outcome serve the Foundation’s mission of user-owned data?

The siafs will have the following positive impacts:

  • It will allow SIA unaware massive-storage applications to use SIA transparently.
  • It will improve the dynamics and economics of the SIA marketplace.
  • It will allow other businesses to include SIA in their IT strategy. Therefore, they will be required to buy more SC, and the value of the SC will go up.

Grant Specifics

A total of 4,500 USD will cover the following:

  • 250 USD to buy a refurbished (or second-handed) desktop computer with at least 16 GB of RAM and 256 GB SDD.
  • 4250 USD for my Salary (2 months).

A testnet wallet will be used through the Faucet.

Timeline

We plan to deliver two versions:

  • First delivery: version 0.1 will be the basic siafs without a caching engine. @30th day (or before).
  • Second delivery: version 0.2 will be the siafs with the LRU caching engine (read below). @60th day (or before).

Project features

  • Finished code published on GitHub under the GPL3 license. Developers/Sysadmins with the skills can compile it by following the directions.
  • Binary (RHEL 9 RPM) and wrappers will be provided. The RPM will be published in my RHEL repo. URL to the RPM will be published.
  • LRU caching engine (if Memcached and local storage is available). The FUSE filesystem uses a lot of callback routines that may translate into burning a lot of SCs. A caching engine will help to reduce these calls. The code will be ready for other developers to include different caching policies.

What are the goals of this small grant?
To allow any SIA holder to use SIA storage on any application. If this project becomes popular, the demand for storage will increase; consequently, hosts will earn more and renters will start buying SC (SC price may go up).

Potential risks that will affect the outcome of the project:
Unforeseen changes in the Renterd API.
API incompatibilities: we will integrate the FUSE API and the SIA Renterd API. So far, at this point, I do not see any issue.

Development Information

Will all of your project’s code be open-source?
Yes, under GPL3 license.

Leave a link where the code will be accessible for review.
siafs, the repository will be created when the grant is approved.

Do you agree to submit monthly progress reports?
Yes

Contact info

Email: luis.daniel.lucio@gmail.com, dlucio@okay.com.mx

Any other preferred contact methods:
LinkedIn: https://www.linkedin.com/in/dlucio/
Telegram: daniel_lucio (please email first to disclose your Telegram ID)

Related Past Projects

2

Hello Daniel,

Does it will work on windows, like a usb key ?

Have a nice day

3

Hi,

I doubt it. I haven’t used Windows since 1999 (only to run a few apps/games). I do not know Windows besides using the mouse. After this is finished, any Windows developer may add/fork to WinSIA-software. There is this project GitHub - billziss-gh/winfuse: FUSE for the Windows kernel that aims to give any FUSE filesystem Windows support.

But for the sake of this proposal, its scope is only on Linux.

1 Like
4

Hi @daniel-lucio, thanks for submitting your grant proposal!

Regarding your budget: “A total of 4,500/5,000 USD will cover the following:”

Please provide a precise budget request, as the committee will not approve a range.

5

Fixed :slight_smile:

Thanks for pointing out that.

6

Based on a recent discord discussion, I am unsure how this effort would be better/different than Rclone.

I would recommend looking at the sia rclone support and seeing how it can be improved. Rclone itself is basically a filesystem abstraction framework with fuse mounts for any possible service.

Kudos.

7

I am a C coder, not a GO coder. Be my guest if you know GO.

8

Thanks for your proposal to the Sia Foundation Grants Program! The committee appreciated this grant and is requesting changes in order to further consider it.

  • Rather than requesting $500 to fund a Siacoin purchase (something we try to avoid when possible) you can use the Zen testnet as an alternative. Please remove this item from your budget and feel free to re-adjust your budget items/request amount as necessary as well.

Once this is complete the proposal will be returned to the committee for consideration.

9

It is fixed. Please review.

10

As previously mentioned, Sia already has direct Rclone support and s3-fuse support. I don’t really see the benefit of the project.

11

You may be right, or not.

I didn’t have the idea, it was on the Sia - Grants page. It just happens that I know how to create Fuse filesystems and after giving a few reviews to the renterd API I realized it is possible.

I understand that the S3 protocol is slow and gets worse when an object ages.

As for rclone, it is designed to be used with siad not with renterd. There are quite API differences. I am almost positive, it won’t run. As I have said, I am not a GO coder; be my guest if you know GO.

My siafs will have caching at different levels. The FUSE API makes a lot of repetitive calls even when you do an ls /some/directory. Caching these calls not only will speed it up (as data won’t change if you pull the same meta-data twice in less than a second), but it will also prevent unnecessary SC burning (if you download many times a small/medium file).

Also, this project will give light to another one that I was advised not to disclose until this is done.

12

The renting side, which would use FUSE, is renterd, not hostd. As for rclone, Redsolver made a fork for renterd, but it was abandoned after S3 support was added.

If you think you can make something more performant, that would have merit. However, I also think it’s better to try to improve what exists rather than start over unless it is really required.

13

I wanted to say renterd (I was reading something about hostd while writing).

As I have said, I am not a GO coder, but if you are, be my guest and help.

I also have said before: caching, caching, caching.

I spent some moments reading the s3fs-fuse,
https://github.com/s3fs-fuse/s3fs-fuse/blob/c97f7a2a1373c9ff1f91bb2257aa67fd03889211/src/fdcache_entity.cpp#L1108 and here, where the HTTP queries are triggered (I think), I see no caching. I may be wrong, but I don’t see code verifying if an object is cached or storing it (the code exists, but I don’t find any reference to caching).
https://github.com/s3fs-fuse/s3fs-fuse/blob/c97f7a2a1373c9ff1f91bb2257aa67fd03889211/src/s3fs.cpp#L3414 This is the function that libfuse calls when accessing any file. No caching reference at all.
https://github.com/s3fs-fuse/s3fs-fuse/blob/c97f7a2a1373c9ff1f91bb2257aa67fd03889211/src/s3fs.cpp#L3276 This has caching reference, but it seems it only caches metadata.

I remember using s3fs-fuse for a non-crypto-related project (VoIP) with AWS. The software hung until the file was retrieved (WAV files, 10-30 MB each). I can’t confirm or deny that siafs may be faster, the only thing I can confirm is that siafs will have caching in different levels, preventing unnecessary calls to renterd.

After this is done, I have other ideas that will speed up siafs. Including another like this one that will help other developers to adopt SIA in their projects.

14

Would you consider adding Finder/Windows Explorer Integrations to your original grant request or as another milestone?

As part of your caching strategy, I think this would be a welcomed feature to be able to sync/cache selected individual files/folders with a secondary click and keep some other files ‘online’ on the Sia Network, a little bit like Dropbox does it.

15

Not a Windows developer, sorry.

The caching strategy that will be implemented is LRU.

I have another idea for a middleware that will do object caching, that will be proposed after this proposal is accepted and delivered.

16

Thank you for the latest edits to your proposal. The committee has voted to approve this grant!

We’ll reach out to your provided email address to begin onboarding. It can take a couple of weeks to get new grantees set up, so please adjust your timelines accordingly. We’ll contact you after onboarding is complete to get the updated timeline and milestone target dates.

1 Like
17

For those who are following this project, we are getting there.

18

Please take this as the report for May 2nd.

What progress was made on your grant this month?

  • Refurbished CPU purchase and installation,
  • Installed renterd and synced with the testnet chain,
  • C coding.

Release 0.1 has been published.

This release was focused on the logic. The following actions were tested locally and worked:

  • copy, delete, and overwrite of files
  • directory creation/delete

What will you be working on next month?
The next release 0.2 will be focused on:

  • Speed improvements,
  • Caching.
  • Creation of the first RPM package and published in my repository.
  • README file update to reflect directions about how to use it.
19

Hello @daniel-lucio,

Thank you for providing your progress report in the template format, it helps us a lot! However, please shift your progress report deliverables to the 2nd of each month, covering the work done in the prior month.
If you’d like to provide additional info you can do that after those reporting requirements are met.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

20

I wrote an email asking about this and Frances answered that I could send the report before the second. So here it is.