Core Development: Host

Foundation Core Development Proposal: Host Module (WIP)

  • Contributing Authors(s):
  • Start Date: 2020-01-13
  • Category: Foundation / Technical / Host
  • Tracking Issue: -

Summary

This a open formal proposal that aims to discuss the future of the host module on the Sia network. With the inception of the foundation, new resources will soon become available that will
enable additional development, fixes and improvements of the host module that were previously not economically viable for the parent company Skynet Labs to execute. Although development
of the host module will only be supported on siad for the time being to more efficiently allocated the available resources, a proactive plan should be compiled that enables growth of the host module beyond general improvements or singular venture-based use cases. Effectively bringing the host module closer to the foundation. We propose an implementation that can be tweaked and optimized for different hardware and use cases the host wants to support on the Sia network.

Motivation

The goal of Sia is to create a decentralized, permission-less and high performance global data storage network that can serve as the foundation for many different web3 (e.g. Skynet) and other applications.
As such, the host module is a crucial building block for the Sia network and directly affects any data related interactions. Being a for-profit company, the primary maintainer of the
host module, Skynet Labs (previously Nebulous Labs) has expressed the inability to maintain and improve the host module beyond what moves the needle financially for the company. With the
inception of the foundation it is therefore logical that the foundation dedicates a significant amount of resources to maintain and improve the host module in a way that is beneficial for
the entire Sia Ecosystem
.

Stakeholders

Current and future participants of the Sia network encompassing unknown network participants and known participants including Skynet Labs & Ecosystem, Filebase, StoreWise and Siacentral.

Feedback will be gathered by sharing this proposal in various Discord & Forum channels.

Detailed Explanation

Improvements

Some improvements include:

  • Faster sync times (utreexo)
  • Less lock-ups and timouts
  • Lower latency for requests
  • Faster transfer speeds
  • Better issue reporting
  • Lower memory, IO and CPU footprint
  • Better scalability for datacenter/enterprise systems

Currently the Sia Foundation and Skynet Labs share the IP for the host and renter modules (source: discord). For the foreseeable future the siad host module will remain the
primary host implementation. This is a logical decision considering the already existing expertise within Skynet Labs and the time it takes to onboard and train the new foundation engineers. As new foundation resources become available, we request the foundation dedicate some available resources to improve upon the current host implementation.

Standardized / Modular host implementation

As more projects start building on Sia, new use cases and requests will arise for specific host features. These host features might (1) be too specialized and cause
conflict with the Skynet Labs vision and goal for the siad host; (2) too many features can make the host implementation unusable for low powered hardware devices.

Therefore we propose the foundation together with Skynet Labs begin work on designing a modular host implementation that is scalable, cross compatible and flexible enough to support the addition or removal of specific modules without causing host-renter interaction conflict. We envision hosts having the ability to, depending on the hardware available, activate
profiles optimized for specific use cases. For low powered devices (i.e. a RPI) a lightweight default implementation can be made available.

Drawbacks

Many unknowns are still present for a modular host implementation: (1) how will renters known which host has which module activated?, (2) are there better strategies available to build
a host package that can grow outside of the scope of individual use cases or constraints?

Rationale and Alternatives

The rationale for these changes:

  1. Support different host implementations with different optimizations and use cases.
  2. Maintain a lean and high flexibility approach to the host package components.
  3. Guarantee cross-compatibility between host implementations and prevent renter - host interaction issues.
  4. Prevent lock-in to one single host implementation.
  5. Support datacenter and enterprise hosts.

Deployment Impact

We believe that host improvements will have a positive impact in driving growth of the Sia network. We also believe that a modular host will create a vivid ecosystem of highly optimized
hosts ready to provide the best service possible with their available infrastructure.

Success Metrics

  1. More stable and performant host-renter interactions.
  2. Growth of the number of Sia hosts.
  3. Growth of data on the network.

I agree with a lot of this but wanted to pull out a few things that I think are important:

  • The Sia network should really only have one host protocol. Having more than that means fragmenting the storage ecosystem and complicating things in significant ways. And by ‘host protocol’, I mean that every host should support every RPC, even if not every host has the same hardware characteristics or contract length.

  • Sia nodes should not be restricted to lightweight hardware. Though it’s pretty cool that you can run a host with a raspi today, the long term vision is for hosts to be high-octane, and as the amount of revenue for hosts on the Sia network goes up, I think it’s reasonable for the minimum hardware requirements to also go up.

More on minimum specs:
We’ve more or less always held that a high end host should have its main metadata on a RAID1 (mirrored) SSD array. If I recall correctly, the main metadata needs to be about 1/1,000 the size of the main data. So if you have a 40 TB array, you’re going to need your SSDs to be 40 GB large each at least.

The host is going to become increasingly RAM heavy. Supporting things like user-side upload and improving access latency is going to move more and more into RAM. We’ve typically held that you want 1 GB of RAM per 1 TB of storage. I think in the long run, this is likely to drop by a factor of 5 to 10, but I wouldn’t want it to be a design goal to make the host much more lightweight than that, as the RAM is very useful in a lot of places.

Especially with Ephemeral Accounts - doubly so if we start adding privacy features to these EAs - the host is getting more crypto heavy. To keep up with requests, expecting large hosts to have a quad core or better CPU is also going to be reasonable.

You have to remember that the host is an open marketplace, and larger hosts get better economies of scale. Supporting ultralight hosts like raspi based hosts is going to significantly increase development effort while not really adding any competitive edge to the Sia network in the long run. I really think we should be okay with raspi’s being edged out of the Sia hosting network once the network is doing another order of magnitude in revenue. Dev resources will be more valuable in other places - like improving the accounting, reducing downtime, and fixing longstanding bugs, and of course just generally improving performance.

Other stuff:

Currently the Sia Foundation and Skynet Labs share the IP for the host and renter modules (source: discord)

The host today is completely open source, the IP is “owned” by everyone, and everyone has the legal authority to fork the host and make any modifications they wish to it, and to present those changes to the community and petition the existing hosts on the Sia network today to upgrade.

1 Like