March 2017 Update + Roadmap


  • admins

    We are introducing a new, ongoing roadmap for Sia. It's a public Trello board listing all of our current activities and future planned activities. You can also vote for your favorite cards/tasks. Check it out here: https://trello.com/b/Io1dDyuI/sia-public-roadmap

    Recent Development

    The past few months have been very busy for Sia. One of our biggest accomplishments was the release of v1.1.0 and v1.1.1, which each provided substantial speedups to uploads and downloads on the renter. We have also released a competitive website https://rankings.sia.tech - a just-for-fun place to show off how much you have uploaded.

    The new uploading code was so efficient that many of our hosts have hit full capacity, and this has resulted in a new type slowdown. Renters who are newly forming contracts know to avoid hosts without much available storage, but renters who have already formed contracts do not currently recognize that they need to form new ones when their hosts are full. Our next major set of upgrades for the renter is going to be focusing on host selection, and making sure that the renter is continuously making sure it has the best possible selection of hosts. This will also help with the occasional download issues that users are seeing.

    Upcoming Development

    Below is the list of things that we'll be working on over the next 4-8 weeks.

    Instant Wallet Unlocking

    We've promised instant wallet unlocking a few times, but we're finally finishing and releasing instant wallet unlocking. @nemo is currently fully focused on finishing the upgrades that we need, and then we will be able to release v.1.2.0 with instant wallet unlocking. The instant wallet unlocking code will also enable us to reduce the memory usage of Sia by about 500 MB.

    Beyond instant wallet unlocking, the wallet will also be upgraded to allow loading from a wallet seed without needing to restart Sia, and will also now find all coins. The current version does not always find all of the coins, we've had to help people make manual adjustments to their configs to get everything working. That will no longer be needed with the new wallet code.

    Host Performance and Reliability Upgrade

    Unfortunately, hosts have not been correctly submitting storage proofs. v1.1.2 will have a hotfix for two of the issues relating to hosts incorrectly handling their storage contracts and this should resolve 90% of the problems. The other 10% will be resolved through upgraded monitoring and testing that will be launched in v1.2.0.

    There is a second host bug where Windows hosts will occasionally corrupt and lose all of their contracts following a power failure. The fix is unfortunately more substantial, and will not be ready until v1.2.0.

    v1.2.0 will see substantial changes to the host, particularly to the file management. In addition to fixing the corruption bug, the host will have greatly improved performance and scalability, and Windows users will no longer need to run their hosts as the admin user (Linux and Mac users already do not need to run their hosts as admin).

    The upgrade from v1.1.1 to v1.2.0 for hosts could take a while. If you have hundreds of GBs on your host, the upgrade could even take a few hours. While unfortunate, it should be the most expensive upgrade for a long time, and it is well worth it. The new host code is much faster, much more scalable, much more reliable, and overall better for your filesystem.

    Renter Performance Upgrade

    We are always continuously monitoring renter performance, and right now the biggest problem is clearly host quality. When hosts fill up with storage, or get overwhelmed with bandwidth requests, the renter is not very good about identifying that the host has lost quality and needs to be replaced. Our next major upgrade for the renter is therefore ongoing monitoring of host quality, and making sure that new hosts are brought in to replace faulty hosts.

    This should help a lot, both with performance and with file reliability.

    Proof of Burn

    Related to host quality and host performance, we will be adding proof of burn to Sia, likely in v1.2.0. The proof of burn will close the final major security gap in the Sia code, more or less 'completing' the core design for the storage protocol. There is of course much to do beyond that in terms of performance and features, but Sia's base will essentially be complete.

    When the feature is released, we will also release an updated hosting guide explaining the proof-of-burn process.

    File Sharing

    File sharing will probably not be in v1.2.0. But it may be in v1.2.1 or v1.2.2, and it's something we're very excited to be bringing back to the community. The file sharing we will be adding today will be a simplified type of sharing, where all that is traded is the .sia files. If the receiver of the file is already connected to enough hosts to download the file, then the file can just be downloaded without hassle. Otherwise, the receiver will pick some of the cheaper hosts and form new file contracts with them, allowing them to download that file, and any future files they receive that are shard on that host.

    Those file contracts that get created will not be used for uploading, only for downloading. They will also not be renewed.

    Improved File Recovery

    Today, if your renter folder corrupts, you will more or less lose all of your files. The same code that enables file sharing however will also enable file recovery (essentially you can share your files with yourself). You will be able to create a backup of all your Sia files, and if something goes wrong and you lose your renter directory, you will have a few weeks (4-6) to re-download your files and get them back. While this is not as good as being able to recover your files using just a seed, it is a significant improvement to losing everything. Furthermore, it means you can back up multiple TBs of data using a usb stick that's just a few GBs.

    This is of course not the end of the improvements, as time continues we will make it increasingly easy to recover your files in the event of disaster.

    Auto-Repair

    I am almost embarrassed that things are still this way, but currently on Sia if you delete the file locally, the redundancy will constantly degrade, and performance can also suffer substantially. So largely, you need to keep your files locally even as you upload them to Sia.

    Either v1.2.1 or v1.2.2 will have an upgrade to fix this. You will be able to delete your files locally once they are uploaded beyond 1.5x redundancy, and Sia will be able to keep them safely online after that.


    That's all the time I have for today, however I hope to make a follow up post with some updates on our long term mission and our business developments at some point soon. In short, our long term mission remains to replace all existing centralized cloud storage solutions and CDNs, and our business development is going well.



  • Awesome! File-sharing (doubling as file recovery) is the feature I'm looking forward to most. Auto-Repair will also be a great step forward.



  • I'm keen to have the h"ost performance and reliability fixed".

    I have had a lot of issues, where SIAd (due to unstable HW or instand shutdown of the PC) can't start and where you have to mingle with JSON files to have to re started with out loss of contracts.

    If not fixed very quickly it feels like broken by design for a storage platform! - it must be able to self recover! This point is absolutely key...


Log in to reply