Grant Proposal: Sia Satellite II

Grants Complete
1

Project Name: Sia Satellite

Project Lead: Michael Bulanov
Developer: Michael Bulanov

Current Status

Sia Satellite has reached the public beta stage, and the first Satellite node is launched (https://ss-alpha.online). The purpose of the testing, in addition to removing any revealed bugs, is to gain the statistical data as per what the operational costs of running a Satellite node shall be.

Despite running in the test mode, the service is fully operational. The renters can pay for the service in fiat money on the portal, and then use the Satellite to upload and download files with the renting software such as renterd. All crypto currency-related aspects (except the contract details, which are reported in Siacoins (SC) for the convenience) remain hidden from the renter.

With this proposal, the project lead intends to address further unmet needs in the decentralized storage space.

  1. The project lead believes that the only one obligation of a customer (read: a renter) should be to pay in time for the services they receive. Currently, the renters need to keep their nodes running in order to maintain the health of their data. This is inconvenient, so the renters could delegate more responsibilities to a Satellite, which could then be monitoring the health of the data and re-uploading the shards to the new hosts if the data redundancy dropped below a certain threshold.

  2. The same as above, the renting software running continuously takes care of the contracts that are expiring and renews them. A Satellite could perform automatic renewals of such contracts, so the renters would only need to maintain a positive fiat balance with the Satellite. This feature is already partially implemented: the RequestContracts RPC sends the contracts that the Satellite has formed, to the renter.

  3. If a renter accidentally loses the metadata of their files, then these files are usually gone. A Satellite could keep a copy of the metadata in its database allowing the renter to recover the files.

  4. A Satellite could act as a proxy when the renters upload their files to multiple hosts. Instead, the renters could upload the files to the Satellite only, thus saving the bandwidth.

All the options above shall be on the opt-in basis, because they imply certain levels of trust.

  1. Once multiple Satellite nodes are deployed, the renters should be able to switch from one Satellite to another as seamlessly as possible.

Further action items may arise based on the community feedback during the testing phase.

Open-Source Commitment

Like always, the project source code shall be maintained in the public repository located at GitHub - mike76-dev/sia-satellite: A network service that allows credit card payment for Sia storage..

Project Timeline

The project shall have the following milestones:

  • Automatic contract renewals (May 23)
  • Renter metadata backup (Jun 23)
  • Upload proxying (Jul-Aug 23)
  • Automatic repairs (Jul-Sep 23)
  • Deployment of the second Satellite (Aug 23)
  • Common satellites database and migratable user accounts (Sep-Oct 23)

In addition to the milestones mentioned above, some improvements to the existing codebase are planned:

  • General cleanup of the satellite code
  • Optimization of the HostDB code
  • Support of the upcoming hostd
  • Redesign of the web portal

Risks

There are a few external risks to the project:

  • Denial of services from the cloud server provider(s) due to the violation of some of their policies (anti-spam, anti-abuse, etc.) For instance, Hetzner was notoriously mentioned for being very much intolerant to their customers’ activities; for this reason, OVH was chosen as the provider for the first Satellite.
    Severity: medium, likelihood: low, mitigation: easy.
  • Denial of services from the payment processor (Stripe) due to the violation of their policies (mainly high-risk activities). Some work has already been done to mitigate this, and a general course of action for the Satellite operators has been developed. In the worst-case scenario, it may become necessary to switch to more tolerant payment processors (e.g. PaymentCloud).
    Severity: medium, likelihood: medium, mitigation: difficult.
  • Denial of services from the mail services provider(s) due to the violation of their anti-spam policies or due to the implementation of more stringent security measures. Can be easily mitigated by choosing another provider. Severity: low, likelihood: low, mitigation: easy.
  • Unavailability of the exchange rate providers (e.g. in certain regions). Can be easily mitigated by switching to another provider. Severity: low, likelihood: low, mitigation: easy.

All the risks listed above are more related to running a Satellite node rather than to the project itself. There are also internal risks related to the scalability of the satellite. So far, it is unknown how the satellite will perform under load. The tests will show that.

Budget

The project is requesting 48,000 USD for the following six months. The requested amount shall constitute the full-time salary of the developer, paid monthly at the beginning of each month.
The costs of the server infrastructure and of the testing of the services within the project timeline have already been covered by the previous grant, and their reimbursement is not requested in this proposal.

Reporting

The progress of the development shall be reported on a monthly basis in the Forum and in the community Discord.

Conclusion

The project lead is kindly asking the Foundation to review this proposal. He is also encouraging the community to ask as many questions as possible.

1 Like
2

The committee has approved this proposal! They noted potential issues maintaining forks and a desire to work with you to possibly avoid that for sustainability’s sake, but are otherwise excited to see your expanded work on Sia Satellite.

We’ll contact you through our existing channels to set up the new grant. Thanks!

1 Like
3

Thanks Steve and the Committee for this gift of trust!

I am also inclined to minimize the maintenance effort for the core team, so looking forward to working together!

4

Hi All,

This is the monthly progress update on Sia Satellite II.

What was done:

  • The user can now choose if the contract renewals should be initiated by the autopilot of renterd or by the Satellite. For this, a switch was added to the Satellite configuration tab in the UI. If automatic renewals are enabled, renterd requests any renewed contracts from the Satellite at the beginning of each maintenance loop. In addition, renewed contracts are requested at the startup.
  • The web portal was redesigned. I hope that it looks better now.
  • The changes were made into a new release of the Satellite (0.2.0-beta) and of the renterd fork (0.3.0-alpha).

What comes next:

  • Now I am busy with a deep refactoring and optimization of the Satellite code. Once this part is done, I will move on to the next milestone.
5

Thanks for the monthly report Mike!

6

Hi All,

This is the June monthly progress update on Sia Satellite II.

This has been a rough month, because I finally decided to do a major refactoring of the legacy siad code that was the base of Sia Satellite. Please don’t get me wrong, I like the modular structure of siad, because I find it so much straightforward. But I do admit that it is a thing of the past, so I re-implemented all four legacy siad modules: Gateway, Consensus, TransactionPool, and Wallet.

The major factor that pushed me to do the refactoring was that I got tired of the type conversion between siad and core, but that was not the only thing. I wanted to have a better compatibility with the new renterd.

  • All four modules use core types instead of siad types
  • JSON or BoltDB stores are all moved to MySQL
  • Wallet uses 12-word BIP39 seed phrases
  • Wallet interface is more compatible with renterd, which should simplify forming and renewing contracts.

Unfortunately, I was mislead by a so called red herring and spent good two weeks on trying to optimize the legacy Consensus code. I tried to implement a few caches to speed up reading from the database, but it finally turned out that the bottleneck was writing to the database. I do believe that it is possible to optimize that, but this is clearly beyond the scope of this project.

The new code is available at GitHub - mike76-dev/sia-satellite at refactor.

7

Hi All,

This is the July monthly progress update on Sia Satellite II.

I finished refactoring the Satellite code and merged the refactor branch into master. A lot has been done on improving the stability of the new code.

I also added a new feature: backups of the file metadata. Now, if the renter opts in, the metadata is saved on the Satellite on each file upload. If the renter deletes the file after that, or if the data becomes corrupt at the renter side, the renter can just restart the node, and the Satellite will export the file metadata to the node. It comes with a caveat: the Satellite charges a fee for saving, retrieving, and storing the metadata. The renter can manage the saved metadata on the Satellite portal.

With the new code and the new features implemented, a new version of Sia Satellite was released, 0.3.0-beta. In the following few days, it will be pushed to https://ss-alpha.online.

As the next step, I am planning to implement another new feature: automatic file repairs. For that, I need to look into how renterd uploads and downloads files, and how the slab migrations work. Wish me good luck, although after having re-implemented the consensus and the wallet code, I’m quite positive that I can do that, too :slight_smile:

2 Likes
8

Hello @mike76,

Thank you for your July progress report! It looks great.

The Committee has voted to create a progress report template which should make it easier for both you, the Grantee, to provide monthly updates as well as to enable the Committee to quickly review progress without spending too much time on minute details.

For August’s progress report (which should be provided in September), please follow the template outlined in this post: About the Grants category - #8

Thank you for your help and cooperation.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

1 Like
9

Progress Report Form

What progress was made on your grant this month?
Please summarize your progress in 3-5 sentences or bullet points:

  • The automatic file repair code has been implemented and is now being tested. As usual, the user has to opt in for this functionality. If they do, all contract formations and renewals are handled by the satellite. The satellite also monitors the health of all uploaded slabs and migrates them, if it drops below a certain threshold (the same way it is done in renterd). When the user turns on their renterd node (as well as during each autopilot loop iteration), all new or renewed contracts as well as all changed file metadata are fetched from the satellite.
  • Made some web portal improvements and bug fixes.

Links to repos worked on this month:

What will you be working on this month?
Please summarize your development goals into a few sentences or bullet points:

While I am testing the file repairs, I can start working on the next feature, which is transferring contracts and files between satellites. The goal is to enable a user to seamlessly switch from one satellite to another, without having to form new contracts and re-upload files. The user still has to register an account on the other satellite and have it funded.

10

Progress Report September

What progress was made on your grant this month?

  • The automatic file repairs have been thoroughly tested and are working fine. This means that, after having opted in, the user may switch off their renterd node, come back in a year, and the files will still be there. The only thing the user has to care about is to maintain a positive balance with the satellite. NOTE: upload packing is not supported yet.
  • Implemented RPCShareContracts, which exports the contract set of renterd to the satellite when the satellite is enabled or changed. This makes the contracts independent of the satellite that formed them and enables a seamless change of the satellite by the user.
  • Implemented a ComboBox component in the renterd UI to select from the known satellites or to enter a new satellite.
  • Made a lot of improvements on the web portal:
    – Created a page that pops up on logging in and explains how to rent storage using the satellite.
    – Enabled opting out of the opt-in features from the dashboard. Opting in is still done in renterd, because of the private key and/or metadata transfer.
    – Implemented file download directly from the dashboard.

Links to repos worked on this month:

What will you be working on this month?

  • Upload packing support
  • S3 support
  • Enabling SC payments
11

The Sia Satellite project is in the end phase of its development. Almost all features have been implemented and tested, such as:

  • Contract formations and renewals managed by renterd but facilitated by a satellite
  • Contract formations and renewals managed by a satellite
  • Backups and retrievals of the file metadata
  • Automatic file repairs
  • Switching between different satellites (still being tested)

All these features are supposed to lower the entry barrier of utilizing the Sia network, because they do not only enable renting decentralized storage with a fiat currency, but also enable delegating the low-level activities to a satellite and focusing only on accessing the data, whenever the user wants.

However, a couple of new features have been introduced in renterd meanwhile: S3 compatibility and upload packing. Both haven’t been in the initial scope of the project, but they would be very nice-to-have features supported by Sia Satellite, especially given the fact that the upload packing is now enabled in renterd by default.

I heard from @PJ (looks like he is not on this forum) that the upload and download code of renterd was going to be finalized coming month, and there were still a couple of breaking changes to be expected.

Therefore, I am kindly asking the Committee and the Foundation to extend this grant by two months with an additional budget of 16,000 USD (which shall constitute the full-time salary of the developer, i.e. myself). All other provisions of the original proposal shall remain.

12

Progress Report October

What progress was made on your grant this month?

  • Enabled SC payments for those who don’t feel like paying in fiat but still want to take advantage of the features provided by the satellite.
  • Added a new payment plan: Invoicing. Unlike the pre-payment, invoicing allows a user to have a negative account balance; at the end of each month, any negative balance is settled using the credit card details provided by the user; this part is handled by Stripe. In order to be able to switch to Invoicing, a user needs to have made at least one payment into their account.
  • Made satellite fees configurable via the client (they were hard-coded originally) and added the /dashboard/fees endpoint that allows querying the prices, e.g. from the web portal.
  • Implemented sending email notifications when the satellite’s wallet balance becomes too low, as well as monthly activity reports.
  • Enabled switching maintenance mode on and off; this pauses the contract maintenance and blocks RPC and some API requests.
  • Enabled putting announcements on the web portal, this can be done via the client. Announcements pop up as a banner on the top of the page and can be easily dismissed on a mouse click, except when in the maintenance mode.
  • Made some improvements and bug fixes.
  • Enabled saving file metadata in non-default buckets.
  • Started working on the upload packing. Managed to have files containing partial slabs saved on the satellite and downloaded from the dashboard. Still have to catch a few bugs.

Links to repos worked on this month:

What will you be working on this month?

This is my last progress report in this thread. Any new reports will be posted here: Grant Proposal: Sia Satellite II - Additional Scope.