HostScore - a Host Benchmarking Tool

Grants Approved
1

Project Name: HostScore
Project Lead: Michael Bulanov

Project Description:

With SiaStats gone, there is currently a need in a benchmarking tool for the hosts on the Sia network. There is an official Sia explorer (https://siascan.com), but it is currently lacking in functionality.

Another unmet need is in a service that would score the hosts and provide a set of selected hosts to the renters according to their preferences, be it performance, price, or geolocation.

As a result of this project, the following objectives shall be met:

  • A host benchmarking tool with at least 3 nodes located in different parts of the world

  • A central web portal to display the benchmarking data

  • An API with a clear and comprehensive documentation

The project would benefit the following stakeholders:

  • Sia hosts would get an understanding how they score on the network

  • Renters on Sia would get a tool to select the hosts based on certain criteria

If using external host providers will be implemented in renterd later on, the tool shall be made compatible with it.

In overall, the objectives stated above shall serve the Foundation’s mission of user-owned data by further improving the hosting and renting experience on Sia.

Project Budget:

The project lead is requesting 52,000 USD for the period of 6 months, paid monthly starting from January 2024. The requested budget can be broken down as follows:

  • 48,000 USD shall constitute a full-time salary of the developer (the project lead).

  • 3,000 USD shall be used to pay for the infrastructure (benchmarking servers and the portal server) for 12 months; this is to ensure that the service is functional even after the project ends.

  • 1,000 USD shall be used to pay for the storage contracts.

Project Timeline:

The project shall have the following milestones:

  1. Development of the benchmarking server
    Stack: Golang, utilizing certain parts of the Sia Satellite code.
    Measurable outcome: 3 benchmarking servers running in different parts of the world (East-US, EU, and Asia) together with the setup documentation and a demo portal that allows to view the benchmarks.
    Timeline: Mar 2024

  2. Development of the web portal
    Stack: React.JS (frontend), Golang (backend)
    Measurable outcome: Fully functional web portal with the option to sort and view hosts based on the user-chosen metrics.
    Timeline: Mar-May 2024

  3. Development of the API
    Stack: Golang
    Measurable outcome: A functional RESTful API with the comprehensive documentation published on the web portal.
    Timeline: May-Jun 2024

  4. Ensuring of the Post-Hardfork Functioning
    This work will be carried out in parallel to the rest of the project to ensure that the service will remain functional post-hardfork.
    Timeline: Jan-Jun 2024, will depend on the development of the Anagami testnet

Potential Risks:

There are three risks to the project timeline that could be identified:

  • Denial of service from the hosting provider(s). This is a negligible risk, because the experience of working with OVH has shown that this is a reliable provider.

  • Upcoming hardfork, which will require modifying the consensus code. To mitigate this, the service shall be running on the testnet with all necessary modifications implemented and tested once the v2 testnet is available.

  • Implementation of the external host provider API in renterd that could lead to the changes in the service API.

Open Source Commitment:

The code shall be fully open-source and available on GitHub - mike76-dev/hostscore: A Sia host benchmarking and scoring tool.

Tracking Progress:

The project lead commits to submitting monthly progress reports.

Contact Information:

The contact details of the project lead are already known to the Foundation.

2

Dunno if this is appropriate, I’ll delete if its not, but I wonder if you’ve considered another name for this tool. I first quickly read this as GagMe, I think because I’m used to the spelling Gauge instead of Gage.

3

TBH, while I see value here, especially for Lume, I would have thought this would be named GaugeMe. GageMe doesn’t make as much sense?

Regardless, I think this grant is a good idea, but I do think it needs a better name :upside_down_face: :man_shrugging:

4

Thanks all for the feedback!

Indeed, I suspected there would be an issue. I’m not good at naming projects after all. I just wanted to avoid spawning another SiaStuff :slight_smile:

I renamed the project.

5

Will the website also be open sourced along with the benchmark code? This would help the community keeping the site alive later on.

What would the three regions be? East US, Europe, and Asia? Assuming these can run on cheap VPSes found on sites like lowendtalk/lowendbox - I believe your budget could allow a few more locations. There are reliable hosts than are on par with OVH but a lot cheaper. At the least I would want to see east and west USA locations.

Can you elaborate what would go into performance metrics? I would think just simply the speed of which files can be retrieved is the only metric.

Will each benchmark node test each host or would it just test the closest/lowest ping? Personally I’d want to know how all hosts compare to say a us east location but I understand if an Asian host is pinged less often.

It wasn’t explicitly stated but uptime/reliability and longevity would be an important metric on the host selection.

I would also appreciate hosts who willingly contribute their information and contact for additional transparency and trustworthiness.

I think this is a needed project

6

I would also consider looking at Akash too for this and not just traditional hosting.

7

Thanks for the insights.

Yes, the web portal will absolutely be open-source, and the code will be available in the same GitHub repository. First, it is a prerequisite for submitting a grant proposal. Second, it is the meaning of the project, to make the service available to the community even if I decide to step down at some point.

The three regions you mentioned are the minimum. I will also be considering West US, Australia, and South America (maybe not limiting to those).
I will take a look at the hosting providers you suggested, thanks for that. Reliability is quite important here as is a certain degree of tolerance (imagine that each node is going to continuously spam the network with RPCs).

There will be two kinds of scans run by each node. The first one will be a usual scan to fetch the host’s settings/price table and to measure the ping time. The second will be the upload/download test, which will produce the upload/download speeds, TTFB, and other similar metrics. The second one will be run less often.

Uptime and Age are two of the seven metrics that renterd keeps, so it is quite obvious that HostScore should keep them, too. Others are Prices, Collateral, Remaining Storage, Interaction History, and Version. I will think about other metrics to include.

The information about the Sia hosts is public, it is not possible to hide it away and remain a host. However, I will work on implementing VPN detection as a part of the geolocation metric.

Thanks again.

8

Hello @mike76,

Thanks for your proposal! The committee is requesting more information regarding this grant proposal:

They would like you to provide more info in your Project Timeline section, better detailing the work that will take place and the expected results. Additionally, the committee would like to see demonstrable milestones for your Standard Grant request.

Thanks again, and let us know if you have any questions!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

9

Thanks for the feedback!

I have edited my initial proposal to include the requested information.

10

Hello @mike76,

Thanks for providing us with the requested updates to your proposal. The additional information and items you listed under “measured outcomes” were very helpful during our assessment. After review, the committee has approved this grant!

We’ll reach out via email to get you set up. Onboarding typically takes a week or two, so feel free to adjust your internal timelines as appropriate.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

11

Thank you very much Kino and the Committee!

I have already started working on the project, but it feels much better to know that it will be funded!

12

Progress Report January

What progress was made on your grant this month?

  • Established basic code infrastructure and ensured that it works on all 3 networks (Mainnet, Zen, and Anagami).
  • Implemented a single-address wallet based on the core consensus.
  • Implemented the host database with the recognition of v1 and v2 host announcements.
  • Implemented host scanning and benchmarking.
  • Set up the first benchmarking node (currently working on Zen only).
  • Set up a demo-portal server.
  • Created the frontend of the demo-portal.

Links to repos worked on this month:

What will you be working on this month?

  • Continue testing of the node and the portal.
  • Adjust the benchmark frequency and intensity if needed.
  • Deploy more benchmarking nodes.
13

Progress Report February

What progress was made on your grant this month?

  • Testing has shown that the currently used benchmark frequency is optimal.
  • The recent spike in the Siacoin price emphasized the need for limiting the USD-bound host settings. This was implemented.
  • Another benchmarking node (East-US) was deployed in addition to the EU node.
  • The work on the portal frontend has started.
  • Instead of running two separate services, one for the Mainnet and one for the Testnet, the decision was made to combine these two services in one. That shall bring the advantage of only benchmarking one host at a time, therefore reducing the error. This work continues in the merge-networks branch.

Links to repos worked on this month:

What will you be working on this month?

  • Finish refactoring the code and merge it into master.
  • Deploy the third benchmarking node (AP) and enable Mainnet on all three nodes.
  • Continue working on the portal frontend.
14

Hello @mike76,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

15

Progress Report March

What progress was made on your grant this month?

  • One more benchmarking node (AP) was deployed. Now HostScore combines three benchmarking nodes located in different parts of the world (EU, East-US, and AP).
  • A demo portal (HostScore - Demo Portal) was released. Both Mainnet and Zen are supported.
  • Various improvements and bug fixes.

Links to repos worked on this month:

What will you be working on this month?

  • Release the full portal.
  • Start working on the public API.
16

Progress Report April

What progress was made on your grant this month?

  • Deployed the public portal (HostScore). Collected the community feedback and incorporated it.
  • Developed and published a (preliminary) API specification.
  • Various bug fixes.

Links to repos worked on this month:

What will you be working on this month?

  • Finalize the public API.
1 Like
17

Hello @mike76,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee