December 2016 Update + Roadmap


  • admins

    Core Development

    This month we ended up dropping a lot of the work that we've been doing on the wallet. It was decided that the core usability bottleneck for Sia was the uploading speeds, and also Sia up until this point is not actually very proactive about punishing bad hosts. We've spent a large part of November and will continue to spend December on these things, because they really should be our strengths.

    We have some really exciting news. We re-wrote the uploading algorithm, and uploads are now going at full speed all the time. Our home connection is pretty slow, only 15mbps, but my tests on the live network were getting between 12 and 16mpbs constantly. This can be compared to our v1.0.4 release, which had the connection inching along at something closer to 0.5mpbs constantly. So, at the very least it's a 10x - 20x speedup, but my guess is that it's actually a lot faster - our home connection is the bottleneck here, not the Sia network.

    It's not ready for release yet, there are some clear bugs with the new algorithms. But it's super promising, and fixing the bugs will only speed things up further.

    We've also made big changes to the download code. One of the biggest features is that downloads no longer need to wait 30+ minutes to get started, they will usually start in 60 seconds or less. Downloads have been overhauled just like uploads, and are also seeing significant speedups not just in latency but also in throughput.

    Release Schedule

    v1.1.0

    We're hoping to have v1.1.0-RC1 out in the next week or so. The primary feature v1.1.0 is going to be improved upload and download speeds, however there will also be updated code to better detect and utilize superior hosts - meaning generally faster speeds, better redundancy (6x redundancy over good hosts is much better than 6x redundacy over mediocre hosts), all while providing lower prices. v1.1.0 is going to reduce the default redundancy from 6x to 4x because of our algorithm improvements, which means lower prices and faster upload times, on top of the improvements that are already there.

    The full v1.1.0 will likely be released the first or second week of January. The code has changed in significant ways and that means we will have to do a substantial amount of testing. We will be aiming to release one RC per week until the software is proven stable.

    v1.2.0

    This release is target for around February and will feature upgrades to the host and wallet. Hosts will be faster, will be able to handle larger contracts and renewals, and will be performing proof-of-burn as the final securty countermeasure in Sia. This will also feature upgrades to the host plugin in the UI, giving users access to more of the host's settings, and also giving users a tool to gauge how well they are doing and what steps they could be taking to improve the amount of data on their machine.

    The wallet has been receiving incremental changes in a separate development branch for several months now, and those changes will finally be released in v1.2.0. Most notably the wallet will unlock instantly, however we will also fix errors related to the 'transaction is too big for the transaction pool' class of errors, we will be updating the seed-recovery code to not need a restart to find your balance, and we'll be fixing a few other stability issues as well.

    There's a very good chance that either v1.1.0 or v1.2.0 will have the ability to repair files without having the file locally. Meaning, once a file gets past about 1.5x redundancy, it'll be safe to delete from your local machine and Sia will still be able to (though more slowly) get the file to its full 4x redundancy. It should be noted that Sia can only repair files if it is running, so even though you will be able to delete the file locally you will still have to keep Sia on in the background for a few hours per week to do repairs.

    v1.3.0

    This release is farther away, slated likely for April or May. There are a few features that we are considering for v1.3.0, but we haven't finalized the roadmap. Here's a list of potential things we will be adding:

    • General speed and scalability improvements.
    • Ability to share files.
    • Ability to set price vs. performance preferences, including abiltiy to blacklist or whitelist hosts.
    Within 2017

    Here are a few things that we may have by the end of 2017:

    • Ability to recover your files using only your wallet seed.
    • Support for mobile wallets.

    Other Noteworthy Developements

    Minebox Presale

    For those unfamiliar with https://minebox.io, Minebox is a storage device that integrates tightly with the Sia network. You can use it like any traditional NAS, however any unused storage space can be rented out over the Sia network for profitability, and all data stored locally on the Minebox can easily be backed up to the Sia network.

    Nebulous Hiring

    Nebulous is currently looking to expand the team! You can find a full description of the available positions at https://sia.tech/jobs

    Overall Health of Sia

    Despite what the price may indicate, things are going very well at Sia. On the development side Sia is making substantial improvements. Faster uploads, faster downloads, and an ever growing set of tests to improve stability. On the business side Sia is making progress evangelizing to enterprises, and while we can't share anything yet there are several large companies in our pipeline. The Sia team has also received more invitations to speak at conferences, including places like London.


  • Global Moderator

    Very exciting times ahead and I am thrilled about the speed improvements!

    One point which was not well explained is this:

    @Taek said in December 2016 Update + Roadmap:

    ... and will be performing proof-of-burn as the final securty countermeasure in Sia.

    To what threat(s) is this countermeasure necessary, and how is proof-of-burn necessary?


  • admins

    When proof of burn is released, we will have a full guide explaining what it is and why it's necessary. It boils down to proving that you are real though. If I am giving data to a host, I need to know that they are a real host interested in propping up the network.

    Otherwise hosts can perform Sybil attacks. We have a few mechanisms already in place to defend against these, however nothing is as complete or as absolute as proof-of-burn. If you add 100 TB of storage to the network, we would like to know that the 100 TB is real storage, and that the person offering the storage has real intentions to profit from it off of the network. It might sound counterintuitive, but the most secure way to express those intentions is to burn money.

    The amount of money that you need to burn is not all that high. Maybe 2-5% of the cost of the hardware. So, given that a 4 TB drive costs about $100 today, you'd burn maybe $3 worth of siacoins when you host that 4TB to prove that you are "real". This prevents a bunch of ghosts from saying they have 4 TB when actually they do not. To successfully pretend, the ghosts would need to burn $3 each, which is a pretty big deal if you have no intention of ever making that money back, and not that big a deal at all if your expected revenue is going to be a lot more than the original $100 that you spend on the drives.



  • You are bang on the money in focussing on transmission speed, which is the Achilles heel of all cloud storage solutions.

    The v1.1 improvements advance Sia from demo speeds into a production tool - with performance throttled at network bandwidth.

    But network bandwidth is still a serious choke point and will remain so for many of us outside the largest conurbations.

    Could Sia architecture support Rsync to squeeze more from existing bandwidth?

    Doing website backups Rsync over SSH affords me speeds 12 times my maximum network bandwidth by transmitting only changed data blocks.

    Changed blocks are detected by comparing hashes, which should please blockchain fans!



  • And what about the ability to recover all contracts from network if host somehow crashes (hdd failure for example)? - Backups are great - but if you host let's say 1Tb of data - backup is something datacenter grade like tape LTO or so. It's OK for a big datacenters or hosting providers - but not all SIA hosts may have this systems? - Or it will be a musthave for hosts and recovery from blockchain/SIA network is impossible by design? - If we talk about 6x redundancy - I don't see a problem to completely recover host "from zero" only using a seed/wallet file. No need to have external backup storage when SIA Network is a backup for us all :)

    Why am I asking - after some failures of my ESXi server running my projects VMs and SIA host I had to recover the last one some times - even RAID10 not helping much if a power failure occurs (Power Supply died and raid controller battery backup one month earlier causing ESXi to loose datastore with all VMs) - I had to remove all data dirs and redownload a blockchain and contracts were lost even when all hosted files were in place... Something with this must be done.



  • @Steve1943 said in December 2016 Update + Roadmap:

    Could Sia architecture support Rsync to squeeze more from existing bandwidth?

    Doing website backups Rsync over SSH affords me speeds 12 times my maximum network bandwidth by transmitting only changed data blocks.

    Changed blocks are detected by comparing hashes, which should please blockchain fans!

    Even more - it could use bittorent-like p2p technology to download and upload files between hosts or even download blockchain for the first run.



  • This covers my wishlist.

    No need to mirror data locally.
    Possiblity to recover data/files from seed only

    If I could add one more "nice to have" then it would be.

    Monthly email reports, usage, funds, etc. Basically I would like to have a remainder when its time to top up the SC funds.

    I totally see this may be bloat, possibly such reports could be scripted using cron or so, instead.


Log in to reply