Grant Proposal: Sia Satellite

Grants Complete
3

Hi Mike, this is a great proposal and exactly the type of thing we should be building for Sia.

I love the network and have supported it for a few years, but it is hard as a code-illiterate person to actually use it. Something that is very user-friendly and payment friendly would help materialize the powerful utility this technology is providing.

I don’t have any expertise in how to do this, so I can’t comment on the engineering of it, but to me this is a project that definitely should get funded several times over.

1 Like
4

I guess I have a couple major concerns/questions:

  1. What experience do you have coding large projects like this? Some resume/portfolio would be good to post here to show you are capable of working on projects like this.
  2. Without repair, storing on Sia is effectively useless. So is the plan to do repair through the renterd software?
  3. Will this tweaked renter use a remote or local consensus server? Because without full light nodes this also has very questionable usability.
5

Thanks for your questions. I will try to answer them here.

  1. Special thanks for this one. Having a portfolio is not a prerequisite to apply for a grant, and if I had one, I would rather apply for a developer position at Sia Foundation, would it not make more sense?
    Anyway, it would probably be fair that the community knew a bit more about who I am. I am not a professional coder. In fact, I am a chemist, PhD. However, coding has always been my passion, since the 80’s. I used to code in Basic, Pascal, C/C++, Assembly. When web development became a thing, it absolutely blew my mind away. I tried myself in JavaScript, TypeScript, React framework, Python, and Go (which I am now using for this project). I am a bit new with Github, there is still a lot I need to learn. By the way, mike76-dev, my username on Github, was not chosen because I wanted to show everyone that I was a cool developer, but because mike76, the nickname I had always had, was already taken, and mike76-dev was suggested by Github.
    Back to the original question. Even though I am not a professional software developer, I do have an experience in project management. I am not sure what exactly you mean with “large project like this”, but the scope of the project was outlined earlier. It is quite measurable, for a defined amount of money and within a defined time, the community shall receive a result described in the proposal.
    Some people from the community may remember my SkyLearn project, which I am unfortunately not maintaining due to the issues with Skynet. There was also some work I did for @DaWe on his SkyLive project. I do not know what else you would consider a sufficiently good portfolio for endorsing this proposal.
  2. File repair is planned but not within this proposal, as I already mentioned. The ground concept is that the renter should not need to keep their node running 24/7. I still need to figure out how it should best be done.
  3. As far as I know, renterd can use either a local or a distributed consensus. The project, however, focuses on whatever is available at this point. So, right now a renter would have to use a local consensus and hence to run a full node. Light nodes are still on the way, it is just a question of time.

Speaking about usability, it is not doing great at the moment. We need to do something in order to change that. It will not come by itself :slight_smile:

6
  1. Fair enough. The reason I speak of scope isn’t that the goals themselves are nebulous or anything, just that the implementation of what you are saying is technically complex.
  2. Sounds good.
  3. ^

I do want to say that being able to use a light node in tadem with some other node doing repair and contract management is exactly what I’ve wanted the Sia project to be able to do since I got here. The thought of being able to use a large, decentralized, open market as a CDN is incredibly exciting for me (I am a nerd after all). I want to make clear I am supportive of this initiative I just wanted to clear up some things first.

I still need to figure out how it should best be done.

Imo the only good solution for this is for the contracting node to do repair. The only tasks that should be passed to the client are downloading directly from the hosts and (possibly) uploading directly to the hosts (though this one has pros and cons).

1 Like
7

Conditionally Approved: The committee approves this grant, subject to the following conditions:

  • Scope: We ask that you narrow the scope of this proposal to exclude modifying
    any renter software. The Foundation’s core developers will work with you to
    ensure that the service is directly compatible with renterd.
  • Budget: We ask that you receive your salary in USD or another fiat currency.
    Funds used for forming contracts can be provided in SC.

Furthermore, we make the following recommendations:

  • Once the software has been developed and tested, we recommend applying for a
    separate grant focused on operating a satellite service. This would include
    writing documentation, FAQs, etc. for prospective operators, and providing
    funding to subsidize operating costs.
1 Like
8

Thanks a lot for approving my proposal. This is not only a financial aid, but also a huge development opportunity for me.
I never liked the idea to modify the existing renter software, but I had to make assumptions when I applied, including no support from the other developers, and under those assumptions, the only possible scenario implied modifying renterd. I will be more than happy to collaborate with the core developer team.
I am also fine with receiving my salary in EUR, as well as the infrastructure-related costs, because they have to be paid in fiat anyway.
And I really appreciate the recommendations. I will keep them in mind for the future development.

2 Likes
9

Merry Christmas everyone!

This is the first monthly progress update on Sia Satellite project. As a reminder, the source code is available on GitHub - mike76-dev/sia-satellite: A network service that allows credit card payment for Sia storage..

I had to change my initial planning and focus on the user accounts and the payment processing first. The users can now create accounts, log in, change their passwords, and delete their accounts if they like. When they log in, they are redirected to the dashboard (still work in progress) where they can see their balance and some Sia network averages and create a payment plan based on their storage requirements.

For the payments, Stripe was chosen as the integration partner. The plans include the support of various payment methods besides credit cards, also in multiple currencies. Actually, Stripe reached out to me asking for more information, apparently because they see cryptocurrencies as a high-risk business, and I was able to show that the end-users will not be buying crypto with their credit cards, so now we are good to go.

What is planned next:

  • Adding a Stripe webhook to update the user balance after payment, followed by moving from test mode to live mode
  • Moving HostDB from JSON to MySQL and optimizing the payment calculation
  • Creating a command-line client to improve the control of the daemon
  • Moving on to forming contracts with the hosts

The service (as it is up to date) is available live on Sia Satellite. I hope it will survive your testing until I get back from my vacation in two weeks :blush:

10

Thanks! The committee appreciated the update and is pleased with your progress, they just had one comment regarding mux: you are welcome to get started with the package you are currently using and will be able to switch with minimal changes once the Foundation moves SiaFoundation/hostd/mux into SiaFoundation/mux.

For your next report, please submit it here by February 2nd. Thank you!

1 Like
11

Hi All,

This is my second monthly update report. Before talking about specific achievements, I would like to describe briefly how the Satellite is designed (this design idea has been crystallizing and getting a more solid shape over the last couple of months).

The Satellite is essentially based on siad, which saves me a lot of time, because I can leverage the already existing code. Gateway, Consensus, TransactionPool, and Wallet modules are all imported from go.sia.tech/siad packages. Further, two new modules are introduced: Satellite and Portal. Portal provides the API that can be accessed via the web portal, while Satellite is where all the magic will happen.

A Satellite node will be an intermediary between the renters and the hosts, so it essentially consists of two major parts. The host-facing part is called Manager, and it inherits a lot from the Renter module of siad. It has a HostDB and a Contractor subsystems, which are responsible for the host selection and the contract formation. The renter-facing part is called Provider, and it will interact with the renters by accepting the contract formation requests and sending ready contracts. A renter using a Satellite node does not have to run their own Wallet module; this one will be provided by the Satellite.

Which brings us to the question: How will the renter pay for the contracts without a Wallet module? Quite simple - via the Portal (and the web portal). A renter who wants to use a Satellite node registers an account at the web portal, creates a payment plan, and commits a payment in a fiat currency. After a successful payment, they receive an email containing the renter seed, which they need to put in the config file of their renterd software (still to be figured out). This seed is generated deterministically from the user email address and the wallet seed of the Satellite node, so it is guaranteed to be unique.

What has been achieved during the last month:

  1. Users can now pay with the real credit cards, and their balances will be updated on the dashboard. As the service in not implemented yet, a proper warning was posted on the web portal.
  2. HostDB now uses MySQL to persist the host entries. Loading the entries on startup was moved to a goroutine, so the startup takes now much shorter than before.
  3. A command-line client, satc, was made similarly to siac.
  4. A custom Contractor was implemented. The major difference with go.sia.tech/siad/modules/renter/contractor is that in most methods, two public keys are passed (for the host and the renter) instead of just the host public key. The Contractor also uses MySQL to persist the contracts.

Now, the Manager part is mostly done for an MVP, and I will be focusing on the Provider part now.

As usual, the repo is located on GitHub - mike76-dev/sia-satellite: A network service that allows credit card payment for Sia storage., and the current state of the development can be tested live on Sia Satellite.

And thanks to The Foundation for the T-shirt! :slight_smile:

sia-t-shirt
sia-t-shirt1920×2560 477 KB

12

A short update. I found by accident that my app password for Google Mail stopped working, so it was not possible to register a new account on Sia Satellite for a couple of weeks without me knowing about that. Sorry for that! The issue is now fixed, and it is now possible to register.

13

Thanks mike76! Going forward, please submit your monthly report ideally by the 2nd of every month. Thank you!

14

Hi All,

This is my February monthly update report. There has been a lot of progress over the last month. For instance, the contracts can now be formed involving renterd and the Satellite. One of such contracts can be found here: Explorer - Transaction 3c0c58....

What was done:

What comes next:

  • Contract renewals
  • Further integration with renterd
15

I’m a hosting and web builder dev for my customers. I’d like to invest to something like skynet portal.

How could I start?

16

The committee has reviewed your report and is pleased with your progress. Keep it up!

1 Like
17

Hello, as it seems your question may have been overlooked, I would suggest you ask in the community discord.

Thanks.

18

Hi All,

This is my March progress report. The work is progressing very well, and there is not a lot left to do till the release.

What was done:

  • FormContracts RPC was rewritten to leverage the core code. Now it does not count the already existing contracts against the requested number, it just provides as many new contracts as requested, because
  • RequestContracts RPC was introduced, specifically for the case when the renterd node has accidentally lost some or all of its contracts.
  • RenewContracts RPC now works over RHP3.
  • UpdateRevision RPC was created to inform the satellite about eventual uploads, downloads, or any other actions going on between renterd and the hosts directly. This was done just for the purpose of a better accounting mid-contract, because the satellite will fetch the final revision anyway when the contract expires.
  • Uploads via renterd are working (RHP2).
  • Downloads via renterd are working (RHP3).
  • Previously, the renter’s public key was generated by the satellite and had to be immutable, because the renter needed to identify themselves with the satellite. This presented a security issue, because a group of malicious hosts could potentially track down the data uploaded by the same renter and even recreate the files. Now the public key used for each contract formation is derived from the renter seed and the host’s public key, like it is done in renterd.
  • The contracts on the dashboard can now be expanded to show the detailed information and collapsed. The can also now be sorted by the start or end height.
  • Various bug fixes.

What comes next:

  • Finish the integration with renterd (especially the new UI).
  • Clean out some remaining bugs, especially in the accounting part.
  • Prepare for the release!
2 Likes
19

This is awesome! Glad to see this work being done

1 Like
20

The committee is pleased with your progress. Keep it up!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

21

Hi All,

This is my April progress report.

What was done:

What comes next:

  • In addition to fixing any bugs revealed during the testing of the Satellite, I need to discuss with the Sia team about how to integrate the Satellite with renterd. Unlike L2 services, which could be run as standalone modules, Sia Satellite is more like L1.5, which makes it difficult to separate the Satellite code from renterd completely.
  • A follow-up proposal (Grant Proposal: Sia Satellite II) with an extended scope of the Satellite features was submitted.
1 Like
22

Thank you for the report, everything looks good so far!