As previously mentioned, Sia already has direct Rclone support and s3-fuse support. I don’t really see the benefit of the project.
You may be right, or not.
I didn’t have the idea, it was on the Sia - Grants page. It just happens that I know how to create Fuse filesystems and after giving a few reviews to the renterd API I realized it is possible.
I understand that the S3 protocol is slow and gets worse when an object ages.
As for rclone, it is designed to be used with siad not with renterd. There are quite API differences. I am almost positive, it won’t run. As I have said, I am not a GO coder; be my guest if you know GO.
My siafs will have caching at different levels. The FUSE API makes a lot of repetitive calls even when you do an ls /some/directory. Caching these calls not only will speed it up (as data won’t change if you pull the same meta-data twice in less than a second), but it will also prevent unnecessary SC burning (if you download many times a small/medium file).
Also, this project will give light to another one that I was advised not to disclose until this is done.
The renting side, which would use FUSE, is renterd, not hostd. As for rclone, Redsolver made a fork for renterd, but it was abandoned after S3 support was added.
If you think you can make something more performant, that would have merit. However, I also think it’s better to try to improve what exists rather than start over unless it is really required.
I wanted to say renterd (I was reading something about hostd while writing).
As I have said, I am not a GO coder, but if you are, be my guest and help.
I also have said before: caching, caching, caching.
I spent some moments reading the s3fs-fuse,
https://github.com/s3fs-fuse/s3fs-fuse/blob/c97f7a2a1373c9ff1f91bb2257aa67fd03889211/src/fdcache_entity.cpp#L1108 and here, where the HTTP queries are triggered (I think), I see no caching. I may be wrong, but I don’t see code verifying if an object is cached or storing it (the code exists, but I don’t find any reference to caching).
https://github.com/s3fs-fuse/s3fs-fuse/blob/c97f7a2a1373c9ff1f91bb2257aa67fd03889211/src/s3fs.cpp#L3414 This is the function that libfuse calls when accessing any file. No caching reference at all.
https://github.com/s3fs-fuse/s3fs-fuse/blob/c97f7a2a1373c9ff1f91bb2257aa67fd03889211/src/s3fs.cpp#L3276 This has caching reference, but it seems it only caches metadata.
I remember using s3fs-fuse for a non-crypto-related project (VoIP) with AWS. The software hung until the file was retrieved (WAV files, 10-30 MB each). I can’t confirm or deny that siafs may be faster, the only thing I can confirm is that siafs will have caching in different levels, preventing unnecessary calls to renterd.
After this is done, I have other ideas that will speed up siafs. Including another like this one that will help other developers to adopt SIA in their projects.
Would you consider adding Finder/Windows Explorer Integrations to your original grant request or as another milestone?
As part of your caching strategy, I think this would be a welcomed feature to be able to sync/cache selected individual files/folders with a secondary click and keep some other files ‘online’ on the Sia Network, a little bit like Dropbox does it.
Not a Windows developer, sorry.
The caching strategy that will be implemented is LRU.
I have another idea for a middleware that will do object caching, that will be proposed after this proposal is accepted and delivered.
Thank you for the latest edits to your proposal. The committee has voted to approve this grant!
We’ll reach out to your provided email address to begin onboarding. It can take a couple of weeks to get new grantees set up, so please adjust your timelines accordingly. We’ll contact you after onboarding is complete to get the updated timeline and milestone target dates.
1 Like
For those who are following this project, we are getting there.
Please take this as the report for May 2nd.
What progress was made on your grant this month?
- Refurbished CPU purchase and installation,
- Installed renterd and synced with the testnet chain,
- C coding.
Release 0.1 has been published.
This release was focused on the logic. The following actions were tested locally and worked:
- copy, delete, and overwrite of files
- directory creation/delete
What will you be working on next month?
The next release 0.2 will be focused on:
- Speed improvements,
- Caching.
- Creation of the first RPM package and published in my repository.
- README file update to reflect directions about how to use it.
Hello @daniel-lucio,
Thank you for providing your progress report in the template format, it helps us a lot! However, please shift your progress report deliverables to the 2nd of each month, covering the work done in the prior month.
If you’d like to provide additional info you can do that after those reporting requirements are met.
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
I wrote an email asking about this and Frances answered that I could send the report before the second. So here it is.
What progress was made on your grant this month?
- Huge files upload issue resolved,
- L1 meta cache working,
- Small code optimizations,
- FUSE 3 preferred over FUSE 2 when both exist,
- Partial L2 data cache implementation.
What will you be working on next month?
The next release 0.2 will be focused on the following:
- Speed improvements,
- Caching.
- Creation of the first RPM package and published in my repository.
- README file update to reflect directions about how to use it.
Unfortunately, I had 7 days without Internet. I will deliver version 0.2 in the following week or so.
Hello @daniel-lucio
Thank you for your progress report!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Hello @daniel-lucio
This is a reminder that your June progress report is now late.
Progress reports are due by the end of the 2nd day of the month, in this instance your June progress report was due on July 2nd.
Please be sure to follow the Progress Report Template and submit your progress report so we may continue funding your grant.
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
Sorry, I was out and forgot when I came back. Doing it now.
What progress was made on your grant this month?
- Partial L2 data cache implementation. There is a core dump that is hard to debug. This buggy code hasn’t been uploaded to GitHub.
What will you be working on next month?
The next release 0.2 will be focused on the following:
- Caching: finishing L2
- Creation of the first RPM package and published in my repository.
- README file update to reflect directions about how to use it.
I am very happy to report that basic L2 tests have been working so far. Need more testing, but it looks good.
Hello @daniel-lucio
This is a reminder that your progress report is due.
Please be sure to follow the Progress Report Template and submit your progress report by the 2nd of each month. Thanks!
Regards,
Kino on behalf of the Sia Foundation and Grants Committee
What progress was made on your grant this month?
- L2 cache implemented,
- Fix a core dump.
Release 0.1.1 has been published (stable-ish) with L1 support only. L2 support is only available in the master branch for now.
What will you be working on next month?
- Keep under local use. If no core dumps or misbehaviour are detected, release 0.2 will be published.
What progress was made on your grant this month?
- 0.2 published!
What will you be working on next month?
I believe the project scope is done.