The Hosting Handbook


  • admins

    This thread is not for troubleshooting. If your host is not working, please ask in a different thread. Off topic posts in this thread will be deleted by the moderator.

    This thread used to be the profit maximization thread, however it has turned into the hosting handbook thread. This post aims to always contain up to date information about features and pitfalls of hosting, and to act as a handbook when trying to figure out how best to manage your host.

    It's worth noting that not all of the core components of the hosting network are in place yet. Sia does not currently do geographic filtering, does not have proof of burn, does not measure latency, and a few other things that we will be adding over the coming months. This means that, to remain a competitive host, you should check this thread at least once a month for changes.


    Host Scoring

    Perhaps the most important part of hosting is understanding the scoring system that renters use when picking hosts. Renters scan hosts, evaluate them, assign them a score, and then select hosts probabilistically based on that score. If you have a better score, you have a higher chance of being selected as a host. Today, the top hosts are much, much better than the bottom hosts. The median host almost never gets selected to participate in a batch of contracts. Conversely, hosts in the top 10% are selected to be in pretty much every single batch of contracts. If the average host were more competitive, there would be a more fair distribution of contracts.

    Renters use a scoring system when picking hosts. Each host has a score. Every point counts as 1 ticket in a lottery. All the tickets go into a hat, and then the renter will pick 50 unique hosts from the hat (hosts cannot be selected multiple times). If you have twice the score of another host, you will have twice the chance of being selected.

    Renters use their own criteria for ranking hosts. It's important to understand that it's impossible to see a global ranking for this reason. Some renters will avoid all hosts in certain regions. Some renters will strongly prefer hosts with low bandwidth costs. Some renters will strongly prefer hosts with low storage costs. As we continue to evolve Sia, the variety of renters will grow, which means more opportunities for hosts to find their strengths and focus on selling those. Though there's no single chart you can just look at, you can use some siad tools, the explorers like siapulse.com and siahub.info, and general market sentiment and advice to keep yourself well informed and competitive.

    If you are running a renter (this is the default - if you aren't, you probably know it), you can see a full list of hosts by running siac hostdb -v. This will display the scores of the hosts in your database. Your database is going to look a bit different than everyone else's, but it will give you a general feel for the landscape. You can see in the score column what sorts of scores you have given everyone. Note that these scores are all relative - one of the hosts is set to score '1', and all of the other hosts are given a score relative to this. That's because the actual score is between 10^70 and 10^80 for each host, and that's not very readable. So, in the chart, if your own host has a score of '2000', it means that you are 2000x as likely to be selected as the guy with a score of '1'. And the guy with a score of '4000' is 2x as likely as you to be selected during the host lottery.

    Your goal of course is to make as much money as possible. You can increase your score in a number of ways (reviewed below), but the biggest is always price. Lower price means better score and getting selected in the lottery more, but of course it also means you get paid less. So you want to pick a score high enough that you are making as much money as possible, but low enough that you win the lottery regularly and get lots of clients/renters contracting you.

    To see the full breakdown of a host, and why they have a certain score, run the command siac hostdb view [pubkey], using the pubkey of the host. This will display things like their price, their uptime, their collateral, and various penalties.

    Small Ways to Boost your Score

    There are a couple of things that you can be penalized for as a host which are easy enough to avoid. We penalize for them because it makes you a lot less useful and a lot more risky to a renter.

    • Version. We penalize the older versions because they are usually buggy, slower, or don't scale as well. It also means the host is likely not paying as much attention, and therefore is just generally less likely to be a valuable host. Some versions that have known security vulnerabilities or known significant bugs can receive large penalties. Hosts running v1.1.2 will have their score cut to 1/8th for running an old host, and hosts older than v1.0.0 will have a 1000x penalty on for not upgrading to stable host code.

    • Storage Remaining: It's a legitimate problem for renters when they form contracts with hosts who then run out of storage space. So hosts with less storage remaining receive penalties. (we will eventually be replacing this penalty with the proof of burn mechanic, but it's not ready yet). In the most recent version, hosts with more than 4 TB of storage receive no penalty at all. If you have less than 1 TB of storage, there is at least a 32x penalty. This may sound severe, but hosts running out of storage space was a significant problem on the network for a while. You can get a big advantage just by making sure you are offering enough storage to the network.

    • Age: Hosts that are very new to the network are more likely to leave/quit/desert than hosts who have been around for a while. So you get a penalty just for being new to the network. If it is your first day on the network, you are going to have a 128x penalty. 'First day' is counted from the moment you made your first host announcement. Subsequent host announcements will not reset your age. By the time you are 1 week, old, the penalty has reduced to only 16x. And, hosts who have been on the network for more than 6 weeks will have no penalty at all. This one can be frustrating for newcomers, but it's been really useful for making sure people only end up with good hosts. If you are new, you can help fight the age bias with low prices that you raise later.

    How Uptime Affects Score

    There is no penalty at all if your uptime is more than 98%. The penalty is very small if your uptime is more than 95% (score is multiplied by 90%). It starts to get nasty after that though. A host with 90% uptime gets a 2x penalty, and a host with 80% uptime gets a 30x penalty. 70% uptime has more than a 1000x penalty and it gets increasingly severe. Most of the penalties follow step functions, but the uptime penalties follow a smooth curve. That is, a host with 91% uptime will have a higher score than a host with 90% uptime. If you can stay above 95% you basically don't have to worry about it.

    95% uptime is actually very forgiving. It means that you need to have a host that is running 24/7, but frequent downtime is okay. You do not want to have downtime exceeding 24 hours, and you also do not want to spend more than about 36 hours of downtime total in any 30 day period. But that means you have plenty of time for reboots, for debugging, and other types of maintenance. And it means you can survive events like power outages without a huge impact on your uptime score.

    How Price Affects Score

    Every time you cut your price in half, your score will go up by 16x. Price is a smooth function, so cutting your price by 1% will have an effect on your score. Renters all look at price slightly different though, based on their usage patterns. Small renters will favor a lower contract price. Renters with lots of archive storage will favor having a low storage price. Renters trying to set up filesharing or a CDN will probably favor a low bandwidth price. Basically, each renter estimates their usage and then looks at the host prices to obtain a 'total cost', and then scores based on the assumed total cost. If you cut your total cost in half, your score will go up by 16x. But, which price variables need to be adjusted to affect total cost the most will be different for each renter.

    This essentially boils down to specialization. If you have an abundance of bandwidth, set your bandwidth price very low. If your bandwidth is being over-used and your storage is under-used, raise your bandwidth price and drop your storage price. Every host has different strengths, and you want to set your prices to play to your strengths. But the general strategy is pretty simple: if you have a resource that is under-utilized, set a lower price. If you have a resource that is constrained or saturated, set a higher price.

    The contract price serves two purposes. The primary purpose is to pre-pay the transaction fees that the host will need when making the storage proof. The default used to be 30 SC and then 20 SC. We've since lowered the fees on the network by a lot, so now the recommended price is 5 SC. The second reason is to prevent small renters from forming contracts with you, but the network is not big enough at this time for this to be meaningful. Mostly, you just want to make sure it's higher than the transaction fees you will be paying.

    How Collateral Affects Score

    Collateral actually has two effects. The first is that it improves your score because renters like to see collateral. But the second is that it reduces your score, because renters have to pay a siafund fee on the collateral. The price penalty only really becomes relevant if your collateral is more than 5x your total cost. Generally speaking you want to target 2-3x collateral vs. price.

    Most renters today use linear scoring for the collateral. That is, every time you double your collateral, your score doubles as well. 3.9% of your collateral contributes to the total price, which has a 16x multiplier on it when you double the total price. So if doubling your collateral is enough to move the total price by more than about 20%, it'll actually cause your score to drop instead of increase.


    Explaining Host Configuration Options

    This section walks through some of the options and parameters of the host that you can adjust.

    Max Collateral

    Max collateral is the maximum amount of collateral you will put into a single contract. This can affect how much data is uploaded over a single contract. If your collateral is 100 SC / TB / mo, and your max collateral is 600 SC, that means you will allow up to 6 terabyte-months of storage to be saved in the contract. Most contracts last 3 months, so the contract could hold a maximum of 2 TB of data.

    This parameter exists to prevent people from creating contracts with you that drain all of your collateral. If you do not have a lot of money to put into collateral, it's typically better to aim at having a larger number of smaller contracts.

    Collateral Budget

    This is the total amount of collateral that the host is willing to put into contracts. Setting a sane value here will prevent the host from draining your wallet by forming too many contracts. However, running out of collateral budget means you will not be able to continue forming contracts, so this value should not be too conservative.


    I think that's a good starting point. I will continue to update this post as the ecosystem changes, and I welcome people to discuss their strategies, observations, and overall sentiment regarding the scoring system. The system is complex, but hosts who understand it well and work to optimize their score with both have higher profits, and also be contributing to the network in a more meaningful way.



  • @Taek
    Thanks for the informative write-up. Two things:

    • this thread should be pinned so it does not get lost after all the "missing/lost coins" threads. Or better, there should be a separate "Hosting" category as this is not really "general discussion" and is relevant only to hosters.

    • You should put a link to the host parameters documentation.
      For example I'm struggling to fully understand all the parameters regarding collateral- how are "collateral", "collateralbudget" and "maxcollateral" related to each other and to other parameters and variables such as "maxduration", "storageprice" and available siacoins in the hosting wallet. I think all these are interdependent and could result in unexpected behavior (scoring) if one of them is set incorrectly or unrealistic.


  • admins

    Thanks. I have moved the thread to a 'hosting' forum, stickied it, and also changed it from a profit maximization guide to a hosting handbook. I've added a small explanation for collateral budget and max collateral.



  • This post is deleted!


  • I would like to see a simple chart with lists of penalty and multiplier functions summarizing the scoring of hosts.



  • Thank you Taek for the explanation. I was guessing in the right direction, now I understand.
    I remember I have seen somewhere the descriptions for other parameters too but can't find that page...

    The most strange parameter for me was "collateralbudget". The name is obvious and explaining enough but I wondered about the default value 1MS. It looks insane high for beginners (like me).
    Basically the upper limit for total locked collateral is the SC amount in wallet or the collateralbudget parameter- whichever is lower.

    I think the next tip in this thread should be about backup and disaster recovery for the hosters- what to do to be able to recover the hosting contracts and data. For example, does it make sense to back up all the shared storage folders and siad "application folders" (host, consensus, transactionpool, etc.)? What happens if the recovery takes some time as new hardware has to be bought, installed and configured? And the renter makes changes, uploads new and deletes old data in that time? In other words- can a hosting node recover itself by syncing active contracts after a 1-2 day downtime? It should, but is it so? And here we come back to collateral... How much collateral could be "lost" in such scenario? Would the renter notice that the redundancy is/was reduced (temporarily)? Or would the renter just take all the collateral and make a hosting contract with another node which is running at that time?


  • admins

    The most strange parameter for me was "collateralbudget". The name is obvious and explaining enough but I wondered about the default value 1MS. It looks insane high for beginners (like me).

    That default is a leftover from when 1 MS was worth close to $100. Those days are of course long behind us. We'll have to update the defaults.

    For example, does it make sense to back up all the shared storage folders and siad "application folders" (host, consensus, transactionpool, etc.)?

    It's a good idea to keep the actual wallet folders themselves (the host, wallet, consensus, etc. directories) running on raid10, or maybe even triple mirroring. If you lose those, you lose the whole host. The storage folders aren't as important to keep backed up, the host can gracefully handle losing one. That said, RAID5 across 5 or 6 disks is probably not a bad idea.

    What happens if the recovery takes some time as new hardware has to be bought, installed and configured? And the renter makes changes, uploads new and deletes old data in that time? In other words- can a hosting node recover itself by syncing active contracts after a 1-2 day downtime? It should, but is it so?

    The renters will tolerate about 2 days of downtime before moving to other hosts. If you have less than 24 hours of downtime, you aren't at risk of losing collateral. Sia is generally pretty generous about letting you take the hosts offline for maintenance/upgrades. But you want to complete it within a few hours ideally, and you don't want to have to do it too often (once a month or so).

    The renter will notice that you are offline, but as the renter is using a ton of hosts to store data redundantly, the renter is going to forgive that downtime as long as the other hosts are still running. Over the course of 30 days, try to have >95% uptime, and over the course of 1 week try not to have more than 24 hours of downtime.



  • @Taek
    Thanks again!
    So I conclude from your post that it is better to have a copy (mirror) of the live system data than a daily or hourly snapshot. That's a bit more complicated than taking snapshots (one of zfs features I like most) as usually mirrors are on drives attached locally in the same host.
    About backing up the hosted folders.
    I don't agree it's not so important to back them up, as the folders can be pretty large, and if you imagine that you need to restore say a terabyte by re-downloading it from the internet...
    My main concern is about loosing sync between the hosting contract database and the actual contents in the hosted folders. What happens if I don't loose the folder but the contents are different from what host process expect to see (restored from a snapshot copy).
    Anyway, do I understand right that for disaster recovery (lost folder recovery) it is advisable to add a bunch of smaller folders to the host instead of one large? What would be "the optimal" size for the host to handle?

    In between. From where would a hosting node recover a lost folder?
    The renter? Would it impose upload fees to the renter? Technically it would be an upload for the renter but requesting to pay for that wouldn't be fair (not to name it robbery).
    Other hosting nodes (redundancy hosts)? Then the before failed host should pay the download fee to the hosts from where the folders are being restored. But does a host know where are the other copies of its data (from where to download)?



  • Hello,

    I am interested in your Project Business Model & its future implications on data storage industry.

    I would like to know that what is the recommended max collateral to be put at stake for becoming a host?

    Though I have seen insane amount of 50,000/- SC, on this forum I would like to know that is it true? Also, Iin my opinion in USD terms USD 50 to 100 is a fair enough collateral to start with. Am I right?



  • @mimran play with it, and see how it affects your income! I don't think there are enough hosts on the network yet to have formed a baseline on what is right and wrong for any of the parameters affecting the price of storage / the income of hosters.

    However one thing should be clear: there's no one size fits all approach to this question. Sia is all about differentiation and optimization. If you want to host large files for a long time (a la Amazon Glacier) then your config is going to be different from the hoster who is optimizing for website file distribution (a la Amazon CloudFront).



  • Hi, Can I both host files and dual mine Ether/Siacoins on the same mining rig? Sorry my ignorance I'm a newby here



  • Hi all,
    I should be getting my QNAP NAS online tonight with a few TB's to offer up (in RAID5).

    My question centers around illegal content. How can the blockchain / Developers monitor this, obviously MS/Amazon have teams dedicated to removing suich content and shutting down the accounts of those who upload it. While I'm sure 99% of users are legit, I absolutely do not want that stuff anywhere near my life / family / NAS.

    I had a quick search but couldn't see anyone asking this, I'm sure they have though.

    Thanks

    Dave



  • @ds2000
    all data stored by others on your host is encrypted and only partial. So you can't know what you are storing.
    There is only one having access to all parts and keys for the file- that is the renting node.

    If you can't live with that in mind, don't host (share) your storage space.



  • Hello.

    Parts about how price affects Score needs update because it does not correct after this code update 2 month ago:

    https://github.com/NebulousLabs/Sia/commit/a85353d8cec915ff2632eec32283a12dafcf0107

    -priceExponentiation = 4
    +priceExponentiation = 5

    Those parts are incorrect:
    Every time you cut your price in half, your score will go up by 16x.
    If you cut your total cost in half, your score will go up by 16x.
    The price penalty only really becomes relevant if your collateral is more than 5x your total cost. Generally speaking you want to target 2-3x collateral vs. price.

    That is, every time you double your collateral, your score doubles as well. 3.9% of your collateral contributes to the total price, which has a 16x multiplier on it when you double the total price. So if doubling your collateral is enough to move the total price by more than about 20%, it'll actually cause your score to drop instead of increase



  • @Taek said in The Hosting Handbook:

    If you are running a renter (this is the default - if you aren't, you probably know it), you can see a full list of hosts by running siac hostdb -v. This will display the scores of the hosts in your database.

    It doesn't show a score for me. Were there any changes made to the implementation since this thread was opened 5 months ago or am I doing something wrong?



  • @pachisi456
    I wondered about this too.
    Now I can tell the hostdb -v method is not reliable.

    The hostdb is maintained by the renter module. As your renter will not rent from your host (itself) it does not include it in the hostdb.
    So, to get your hosts score you need to run a separate node (different seed) and look in that nodes hostdb.
    And there is another catch. The renter module queries only the top 200 of hosts regularly for settings, the others only occassionally.
    For example my host settings got refreshed by the renter only once in two weeks as my host is not in the top 200.

    So the easiest and in my opinion most reliable way to check the hosts score is looking for the host in the Siahub host directory. I think Siahub uses a modified/customized module that queries all hosts.



  • @reinisp
    Thanks! Good answer.
    Just realised that the way I've put my question was a bit misleading. What I've meant is that when I run siac.exe hostdb -v I don't see ANY scores, just the list of hosts with prices etc. How can I see the actual score number of a any host?



  • @pachisi456
    The hostdb -v output gives you the host list sorted according to score.
    So the last one you see has the best score.
    The overall score is shown in the column following the announced address. and before the column showing the price.

    In the column named "Pubkey" you see the hashes of the nodes (beginning with the "ed25519:").
    If you want to read a hosts settings and score breakdown you should copy the whole hash, then
    siac hostdb view <pubkey of a particular host>
    gives you the settings of that.

    For example:
    siac hostdb view ed25519:181be570a6b1b80e07775f0cb26ae92d654c74daa8d11e4a043b8289409ec2b9
    gives you

    Host information:
      Public Key: ed25519:181be570a6b1b80e07775f0cb26ae92d654c74daa8d11e4a043b8289409ec2b9
      Block First Seen: 124977
    
      Host Settings:
        Accepting Contracts:            true
        Total Storage:                  10899 GB
        Remaining Storage:              10446 GB
        Offered Collateral (TB / Mo):   200 SC
    
        Contract Price:            2 SC
        Storage Price (TB / Mo):   13 SC
        Download Price (1 TB):     200 SC
        Upload Price (1 TB):       0 H
        Version:                   1.3.0
    
      Score Breakdown:
        Age:           0.500
        Burn:          1.000
        Collateral:    37.333
        Interaction:   0.655
        Price:         1085.014
        Storage:       1.000
        Uptime:        1.000
        Version:       0.700
    
      Scan History Length: 53
      Overall Uptime:      1.000
    

Log in to reply