Grant Proposal: Sia Satellite II - Additional Scope

Grants Complete
1

I was advised by the Committee to submit this as a new proposal, so this is what I am doing now.

Project Name: Sia Satellite II - Additional Scope
Name of the organization or individual submitting the proposal: Michael Bulanov

Project Description

The Sia Satellite project is in the end phase of its development. Almost all features have been implemented and tested, such as:

  • Accepting fiat as well as SC payments
  • Different payment plans (pre-payment and invoicing)
  • Contract formations and renewals managed by renterd but facilitated by a satellite
  • Contract formations and renewals managed by a satellite
  • Backups and retrievals of the file metadata
  • Automatic file repairs
  • Switching between different satellites (still being tested)

All these features are supposed to lower the entry barrier of utilizing the Sia network, because they do not only enable renting decentralized storage with a fiat currency, but also enable delegating the low-level activities to a satellite and focusing only on accessing the data, whenever the user wants.

However, a couple of new features have been introduced in renterd meanwhile: S3 compatibility and upload packing. Both haven’t been in the initial scope of the project, but they would be very nice-to-have features supported by Sia Satellite, especially given the fact that the upload packing is now enabled in renterd by default. Enabling the upload packing also broke some of the features of Sia Satellite, so fixing that is very important.

Who Benefits from Sia Satellite?

Users who want to rent decentralized cloud storage but do not have crypto currency. Also those who do not want or have no possibility to run a renterd node 24/7.

How Does the Project Serve the Foundation’s Mission of User-Owned Data?

Sia Satellite lowers the entry barrier to rent storage on the Sia network and solves some of the inherent issues, like the need to run a node constantly or very often to prevent the stored data from degrading and getting lost.

Requested Budget

The project requests a budget of 16,000 USD, which shall constitute the full-time salary of the developer, over the period of two months, which shall cover the additional work required in the scope.

Project Timeline

  • Upload packing support: by Nov 15, 2023
  • Proxying uploads through the satellite (included into the original scope but delayed due to the upload packing introduction in renterd: by Nov 30, 2023
  • S3 support: by Dec 15, 2023
  • Testing: by Dec 31, 2023

Potential Risks That Will Affect the Outcome of the Project

The development of Sia Satellite has shown so far that all risks put in the estimation are negligible or can be successfully mitigated. The only remaining risk is that new breaking changes are introduced in renterd. The project developer takes this risk and shall not request any additional funding beyond the project timeline.

Open Source Commitment

The project is and will continue to be open-source.

Link to the Project Repository

Commitment to Post Monthly Progress Reports

The progress reports shall be posted monthly in this forum thread.

Contact Information

The contact information is already known to the Sia team.

2 Likes
3

both S3 Support and upload packing are important features

4

Hello @mike76

Congratulations, the committee has approved this proposal!

We’ll reach out via email to set up grant program onboarding. Onboarding typically takes a week or two, so free to adjust your internal timelines as appropriate.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

1 Like
5

Progress Report November

What progress was made on your grant this month?

  • Implemented upload packing support.
  • Implemented proxying uploads through the satellite. When this option is enabled, renterd uploads the file directly to the satellite, which places the file in a temporary buffer. Every 10 minutes, the buffered files are uploaded to the Sia network. During the next autopilot loop iteration, renterd imports the uploaded file metadata back.
  • Enabled encrypting data, filenames, and other sensitive metadata before sending to the satellite. This way, the satellite operator can no longer see what is uploaded. The user can still see the metadata and download the files in the portal dashboard by providing the encryption key from renterd. Naturally, this key is not transferred to the satellite.
  • Enabled saving file metadata from S3 multipart uploads. During the project work, a bug was discovered in renterd that prevented the uploaded parts from being assembled in the correct order, and the information was submitted to the core team on how to reproduce it. The bug has been fixed, and the feature works now.

Links to repos worked on this month:

What will you be working on this month?

S3 uploads are now supported via renterd. This means that also Vup (S5) can utilize the satellite now. What is left is enabling S3 via proxying through the satellite. This is what I am going to do the coming month followed by some cleanup and writing docs.

6

Progress Report December

What progress was made on your grant this month?

  • Implemented S3 support for all satellite usage modes: uploads via renterd, proxy uploads via the satellite, with or without additional data encryption.
    The challenge with the S3 support turned out to be that if the individual parts of a multipart upload were encrypted, and the metadata was sent to the satellite, it was impossible to decrypt the original file back, because the satellite had no idea about the sizes of the individual parts.
    I tried a workaround where each part would be prepended with its size as an uint64. Unfortunately, that brought another challenge, because the size of the encrypted file didn’t match the decrypted file, and the HTTP downloads errored due to the partial content.
    Finally, a solution was found where the sizes of the encrypted parts were to be stored in satellite.json together with the other satellite-related data on the client side. This still brings one trade-off, though: whenever the user desires to switch back to the original renterd, the file metadata may become incompatible.
  • The data encryption is now optional.
  • Made numerous web portal improvements:
    – Optional Google-facilitated login/signup
    – Improved file explorer
    – Improved dashboard
    – Help/FAQ sections
  • Made some bug fixes

Links to repos worked on this month:

What will you be working on this month?

This is the final report. The project will still be maintained, though, so any feedback is welcome, preferably via Discord/email.