Grant Proposal: Sia Satellite

Grants Complete
1

Project Name: Sia Satellite

Project Lead: Michael Bulanov

Purpose

Sia is a decentralized cloud storage platform where the renters form storage contracts with the hosts in an open marketspace and pay them in Siacoins (SC). The open marketspace ensures that the renters pay a competitive price for the data they store but can also expect a reasonably good quality of the service provided by the hosts. The decentralized nature of Sia ensures that there is no single entity controlling the renter’s data which, combined with a sufficient redundancy, means that the data will remain available as long as it is paid for, even if a part of the hosts goes offline or disappears entirely.

Unfortunately, you can currently only use the Sia storage platform if you have Siacoins. For most of the users, those who have just heard about crypto currencies but never actually used them, this means an additional barrier. To purchase SC, one first needs to open an account with a crypto exchange, often go through a KYC process, deposit fiat money, buy Bitcoins (because on most exchanges SC can only be traded for BTC), and exchange them for Siacoins. All this significantly hinders a broader adoption of Sia.

Sia Satellite is a business model that can help overcome this barrier, at the cost of some centralization. A satellite is a network service that forms contracts with the hosts on behalf of the renter, manages these contracts, keeps track of the spendings, and pays the hosts with SC. The renter uploads their data directly to the hosts and downloads from them, thus reducing the load on the satellite. At the end of each period (usually one month), the renter pays for the service with their credit card. An upfront payment is also possible. An upfront-paying user can enjoy certain benefits, like setting price limits or selecting the hosts to store their data with.

When using Sia Satellite, a renter does not need to own SC to use Sia storage, nor do they need to know about SC at all.

Sia Satellite consists of the following three parts:

  • The satellite service, run on a remote server by a third party (the satellite owner)
  • The tweaked renter software, run locally by a user
  • The web portal, used to register user accounts, provide usage analytics, and pay for the service

When the service is fully developed, anyone will be able to run their own satellite. The name of the project was intentionally chosen very generic, so that those who later want to run their own satellite node can choose whatever name they like.

Running many satellite nodes distributed all over the world can mitigate the centralization piece, similarly to public Skynet portals. The owners of the satellites can charge reasonable fees for providing their services. The more satellites there are, the stronger the competition between them, and the better the price/quality ratio of the services.

Open-Source Commitment

The project source code shall be maintained in a public repository located at GitHub - mike76-dev/sia-satellite: A network service that allows credit card payment for Sia storage..

Project Timeline

The minimum viable prototype, a satellite daemon capable of connecting to the peers and syncing to the blockchain, is already running instead of the Sia daemon on https://mike76.ddns.net/pulse.

The project shall have the following milestones:

  • Development of the satellite service (Dec 22 - Jan 23)
  • Development of the customized renter software (Jan - Mar 23)
  • Development of the web portal (Mar 23)
  • Testing of the product (Mar - May 23)

Risks

The project will be using renterd software as the base for the customization. The progress of Phase 2, and consequently Phase 4, will therefore depend on it. This is the risk that the project lead shall take, and he shall request no further funding beyond the proposed duration of the project.

Budget

The project is requesting 26,000 USD, paid in Siacoins, for the following six months. The spendings can be itemized as follows:

  • 24,000 USD shall constitute the salary of the developer, paid monthly or quarterly, at the discretion of the Foundation, at the beginning of each period.
  • 1,000 USD shall be used to pay for the project infrastructure (remote server, subscriptions, etc.) for the duration of the project, paid upon completion of Phase 1.
  • 1,000 USD shall be used for testing of the product (mostly for paying the hosts that the software will form contracts with), paid upon completion of Phase 2.

Reporting

The progress of the development shall be reported on a monthly basis in the community Discord.

Conclusion

The project lead is kindly asking the Foundation to review this proposal. He is also encouraging the community to ask as many questions as needed to improve the clarity on the project scope and the expected outcome.

6 Likes
2

I have got quite a limited feedback so far, but also a couple of useful questions/suggestions. In order to keep the discussion in one place, I will post here some clarification.

I am thinking of the following features (not limited to these) as the continuation of the project:

  • Proxying renter uploads in order to save their bandwidth
  • File repair
  • Communicating with other satellite nodes
  • …

All these features are outside the scope of the current proposal. I do not want to overpromise and underdeliver.

Thanks to those who have provided their feedback. Looking forward to more questions or suggestions.

3

Hi Mike, this is a great proposal and exactly the type of thing we should be building for Sia.

I love the network and have supported it for a few years, but it is hard as a code-illiterate person to actually use it. Something that is very user-friendly and payment friendly would help materialize the powerful utility this technology is providing.

I don’t have any expertise in how to do this, so I can’t comment on the engineering of it, but to me this is a project that definitely should get funded several times over.

1 Like
4

I guess I have a couple major concerns/questions:

  1. What experience do you have coding large projects like this? Some resume/portfolio would be good to post here to show you are capable of working on projects like this.
  2. Without repair, storing on Sia is effectively useless. So is the plan to do repair through the renterd software?
  3. Will this tweaked renter use a remote or local consensus server? Because without full light nodes this also has very questionable usability.
5

Thanks for your questions. I will try to answer them here.

  1. Special thanks for this one. Having a portfolio is not a prerequisite to apply for a grant, and if I had one, I would rather apply for a developer position at Sia Foundation, would it not make more sense?
    Anyway, it would probably be fair that the community knew a bit more about who I am. I am not a professional coder. In fact, I am a chemist, PhD. However, coding has always been my passion, since the 80’s. I used to code in Basic, Pascal, C/C++, Assembly. When web development became a thing, it absolutely blew my mind away. I tried myself in JavaScript, TypeScript, React framework, Python, and Go (which I am now using for this project). I am a bit new with Github, there is still a lot I need to learn. By the way, mike76-dev, my username on Github, was not chosen because I wanted to show everyone that I was a cool developer, but because mike76, the nickname I had always had, was already taken, and mike76-dev was suggested by Github.
    Back to the original question. Even though I am not a professional software developer, I do have an experience in project management. I am not sure what exactly you mean with “large project like this”, but the scope of the project was outlined earlier. It is quite measurable, for a defined amount of money and within a defined time, the community shall receive a result described in the proposal.
    Some people from the community may remember my SkyLearn project, which I am unfortunately not maintaining due to the issues with Skynet. There was also some work I did for @DaWe on his SkyLive project. I do not know what else you would consider a sufficiently good portfolio for endorsing this proposal.
  2. File repair is planned but not within this proposal, as I already mentioned. The ground concept is that the renter should not need to keep their node running 24/7. I still need to figure out how it should best be done.
  3. As far as I know, renterd can use either a local or a distributed consensus. The project, however, focuses on whatever is available at this point. So, right now a renter would have to use a local consensus and hence to run a full node. Light nodes are still on the way, it is just a question of time.

Speaking about usability, it is not doing great at the moment. We need to do something in order to change that. It will not come by itself :slight_smile:

6
  1. Fair enough. The reason I speak of scope isn’t that the goals themselves are nebulous or anything, just that the implementation of what you are saying is technically complex.
  2. Sounds good.
  3. ^

I do want to say that being able to use a light node in tadem with some other node doing repair and contract management is exactly what I’ve wanted the Sia project to be able to do since I got here. The thought of being able to use a large, decentralized, open market as a CDN is incredibly exciting for me (I am a nerd after all). I want to make clear I am supportive of this initiative I just wanted to clear up some things first.

I still need to figure out how it should best be done.

Imo the only good solution for this is for the contracting node to do repair. The only tasks that should be passed to the client are downloading directly from the hosts and (possibly) uploading directly to the hosts (though this one has pros and cons).

1 Like
7

Conditionally Approved: The committee approves this grant, subject to the following conditions:

  • Scope: We ask that you narrow the scope of this proposal to exclude modifying
    any renter software. The Foundation’s core developers will work with you to
    ensure that the service is directly compatible with renterd.
  • Budget: We ask that you receive your salary in USD or another fiat currency.
    Funds used for forming contracts can be provided in SC.

Furthermore, we make the following recommendations:

  • Once the software has been developed and tested, we recommend applying for a
    separate grant focused on operating a satellite service. This would include
    writing documentation, FAQs, etc. for prospective operators, and providing
    funding to subsidize operating costs.
1 Like
8

Thanks a lot for approving my proposal. This is not only a financial aid, but also a huge development opportunity for me.
I never liked the idea to modify the existing renter software, but I had to make assumptions when I applied, including no support from the other developers, and under those assumptions, the only possible scenario implied modifying renterd. I will be more than happy to collaborate with the core developer team.
I am also fine with receiving my salary in EUR, as well as the infrastructure-related costs, because they have to be paid in fiat anyway.
And I really appreciate the recommendations. I will keep them in mind for the future development.

2 Likes
9

Merry Christmas everyone!

This is the first monthly progress update on Sia Satellite project. As a reminder, the source code is available on GitHub - mike76-dev/sia-satellite: A network service that allows credit card payment for Sia storage..

I had to change my initial planning and focus on the user accounts and the payment processing first. The users can now create accounts, log in, change their passwords, and delete their accounts if they like. When they log in, they are redirected to the dashboard (still work in progress) where they can see their balance and some Sia network averages and create a payment plan based on their storage requirements.

For the payments, Stripe was chosen as the integration partner. The plans include the support of various payment methods besides credit cards, also in multiple currencies. Actually, Stripe reached out to me asking for more information, apparently because they see cryptocurrencies as a high-risk business, and I was able to show that the end-users will not be buying crypto with their credit cards, so now we are good to go.

What is planned next:

  • Adding a Stripe webhook to update the user balance after payment, followed by moving from test mode to live mode
  • Moving HostDB from JSON to MySQL and optimizing the payment calculation
  • Creating a command-line client to improve the control of the daemon
  • Moving on to forming contracts with the hosts

The service (as it is up to date) is available live on Sia Satellite. I hope it will survive your testing until I get back from my vacation in two weeks :blush:

10

Thanks! The committee appreciated the update and is pleased with your progress, they just had one comment regarding mux: you are welcome to get started with the package you are currently using and will be able to switch with minimal changes once the Foundation moves SiaFoundation/hostd/mux into SiaFoundation/mux.

For your next report, please submit it here by February 2nd. Thank you!

1 Like
11

Hi All,

This is my second monthly update report. Before talking about specific achievements, I would like to describe briefly how the Satellite is designed (this design idea has been crystallizing and getting a more solid shape over the last couple of months).

The Satellite is essentially based on siad, which saves me a lot of time, because I can leverage the already existing code. Gateway, Consensus, TransactionPool, and Wallet modules are all imported from go.sia.tech/siad packages. Further, two new modules are introduced: Satellite and Portal. Portal provides the API that can be accessed via the web portal, while Satellite is where all the magic will happen.

A Satellite node will be an intermediary between the renters and the hosts, so it essentially consists of two major parts. The host-facing part is called Manager, and it inherits a lot from the Renter module of siad. It has a HostDB and a Contractor subsystems, which are responsible for the host selection and the contract formation. The renter-facing part is called Provider, and it will interact with the renters by accepting the contract formation requests and sending ready contracts. A renter using a Satellite node does not have to run their own Wallet module; this one will be provided by the Satellite.

Which brings us to the question: How will the renter pay for the contracts without a Wallet module? Quite simple - via the Portal (and the web portal). A renter who wants to use a Satellite node registers an account at the web portal, creates a payment plan, and commits a payment in a fiat currency. After a successful payment, they receive an email containing the renter seed, which they need to put in the config file of their renterd software (still to be figured out). This seed is generated deterministically from the user email address and the wallet seed of the Satellite node, so it is guaranteed to be unique.

What has been achieved during the last month:

  1. Users can now pay with the real credit cards, and their balances will be updated on the dashboard. As the service in not implemented yet, a proper warning was posted on the web portal.
  2. HostDB now uses MySQL to persist the host entries. Loading the entries on startup was moved to a goroutine, so the startup takes now much shorter than before.
  3. A command-line client, satc, was made similarly to siac.
  4. A custom Contractor was implemented. The major difference with go.sia.tech/siad/modules/renter/contractor is that in most methods, two public keys are passed (for the host and the renter) instead of just the host public key. The Contractor also uses MySQL to persist the contracts.

Now, the Manager part is mostly done for an MVP, and I will be focusing on the Provider part now.

As usual, the repo is located on GitHub - mike76-dev/sia-satellite: A network service that allows credit card payment for Sia storage., and the current state of the development can be tested live on Sia Satellite.

And thanks to The Foundation for the T-shirt! :slight_smile:

sia-t-shirt
sia-t-shirt1920×2560 477 KB

12

A short update. I found by accident that my app password for Google Mail stopped working, so it was not possible to register a new account on Sia Satellite for a couple of weeks without me knowing about that. Sorry for that! The issue is now fixed, and it is now possible to register.

13

Thanks mike76! Going forward, please submit your monthly report ideally by the 2nd of every month. Thank you!

14

Hi All,

This is my February monthly update report. There has been a lot of progress over the last month. For instance, the contracts can now be formed involving renterd and the Satellite. One of such contracts can be found here: Explorer - Transaction 3c0c58....

What was done:

What comes next:

  • Contract renewals
  • Further integration with renterd
15

I’m a hosting and web builder dev for my customers. I’d like to invest to something like skynet portal.

How could I start?

16

The committee has reviewed your report and is pleased with your progress. Keep it up!

1 Like
17

Hello, as it seems your question may have been overlooked, I would suggest you ask in the community discord.

Thanks.

18

Hi All,

This is my March progress report. The work is progressing very well, and there is not a lot left to do till the release.

What was done:

  • FormContracts RPC was rewritten to leverage the core code. Now it does not count the already existing contracts against the requested number, it just provides as many new contracts as requested, because
  • RequestContracts RPC was introduced, specifically for the case when the renterd node has accidentally lost some or all of its contracts.
  • RenewContracts RPC now works over RHP3.
  • UpdateRevision RPC was created to inform the satellite about eventual uploads, downloads, or any other actions going on between renterd and the hosts directly. This was done just for the purpose of a better accounting mid-contract, because the satellite will fetch the final revision anyway when the contract expires.
  • Uploads via renterd are working (RHP2).
  • Downloads via renterd are working (RHP3).
  • Previously, the renter’s public key was generated by the satellite and had to be immutable, because the renter needed to identify themselves with the satellite. This presented a security issue, because a group of malicious hosts could potentially track down the data uploaded by the same renter and even recreate the files. Now the public key used for each contract formation is derived from the renter seed and the host’s public key, like it is done in renterd.
  • The contracts on the dashboard can now be expanded to show the detailed information and collapsed. The can also now be sorted by the start or end height.
  • Various bug fixes.

What comes next:

  • Finish the integration with renterd (especially the new UI).
  • Clean out some remaining bugs, especially in the accounting part.
  • Prepare for the release!
2 Likes
19

This is awesome! Glad to see this work being done

1 Like
20

The committee is pleased with your progress. Keep it up!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee