July 2016 Update + Roadmap

  • admins

    June 28th saw the release of our v1.0.0, the first decentralized storage app
    to make it into production. Following the release, we saw huge growth numbers.
    The number of people on our mailing list doubled, the number of people in our
    slack rose dramatically, the amount of traffic on our webpage rose

    We saw the number of hosts increase to 75 hosts serving more than 300TB of
    data, and we had an influx of requests from potential business customers who
    have taken an interest in Sia.

    That said, the release was not without its problems, and I apologize for being
    quiet on these for so long. Many farmers were having problems keeping their
    hosts online, and many renters were having trouble uploading, and downloading
    their files. We spent most of the past week talking to users, reading logs,
    and trying to figure out exactly what was going on with the network.

    There is good news. Most of the problems are small in scope even if large in
    effect. By and large, the architecture of Sia is sound and does not need to be
    rewritten. Most of the issues are things that we can fix in a few weeks, and
    while I'm sure they will not be the last speedbumps, they are exactly that -
    speed bumps. We need to slow down a little, increase the testing, fix the
    bugs, and then we can keep going on our way.

    The vast majority of the problems seem to be related to a single bug. When a
    renter creates file contracts with hosts, the renter creates a long chain of
    interdependent transactions. It might look like this:

    fc1 -> fc2 -> fc3 -> fc4

    If one of those transactions, say fc2, ends up being invalid, then you lose
    all of the following transactions, even though only one of them is invalid.

    fc1 -> X -> X -> X

    Perhaps more significantly, even if they are all valid, the host on the other
    side of the connection may not be receiving all of the dependent transactions,
    meaning the host will just see an invalid transaction even if the transaction
    is complete.

    ? -> ? -> X -> X

    Ultimately, fixing these problems is not conceptually very difficult. We first
    need to make sure that the renter is not creating dependent transactions until
    their parents have been confirmed valid, and we need to make sure that the
    host is properly getting all dependencies during the contract formation

    Though this was the core problem, talking to users revealed about 30 bugs
    overall, around half of which have been fixed up to this point. The other half
    are still being worked on, however none of the bugs are very scary - we just
    need to put in the hours and we should have all currently known issues patched
    up in the next week.

    And we need to be expanding our testing to make sure that we're properly
    catching these bugs before they get into production. We released v1.0.0
    because we felt that the core architecture was complete, but as we have seen
    the testing was insufficient and ultimately I think we can say that Sia was
    released too early. The good news is that we should be able to spend most of
    the next month improving our testing and our testing strategies, and by the
    end of the month have a release that is much stronger.

    The current plan is to have siad v1.0.1 and sia-ui v1.0.2 out on July 19th
    with all of the major bugs patched up. Throughout July and much of August we
    will be focused largely on testing, and sometime mid-August you can expect
    another release that has bugfixes for everything that shows up during our
    improved testing and hardening.

    Jordan, one of the Nebulous team members, will be adding business development
    and marketing to his list of responsibilities. This will include building and
    sourcing infographics, interacting with enterprise customers, growing our
    third party app community, expanding our meetup communities, and overall
    pushing the business interests of Sia.

    For those of you looking for ways to help, there's one thing that Sia needs
    more than anything else right now - testing. If you are not a developer, the
    best thing to do is to download and run the app, and report all problems or
    annoyances that you run into. If you are a developer, we could especially used
    improved test coverage on our API. Here is a screenshot of our current API

    Sia API Code Coverage

    As you can see, some of our endpoints are covered by tests, but most of them
    are not. In the screenshot, there is partial coverage for the API call that
    allows you to add a storage folder to the host, but no coverage at all for the
    API call that allows you to resize a storage folder.

    There is a guide for getting started here. Please do not hesistate to ask
    (on either the forum or on the slack) for guidance, we're happy to help, and happy to code review.

    The Sia team is as excitied as ever to be partipating in the decentralization
    movement. All parts of the ecosystem are seeing lots of growth, and while
    there's still a long road ahead of us we're all much further down that road
    than we were a year ago, and the future looks bright.

  • Thanks for the update. Sounds great!

  • For those of you looking for ways to help, there's one thing that Sia needs
    more than anything else right now - testing. If you are not a developer, the
    best thing to do is to download and run the app, and report all problems or
    annoyances that you run into.

    I'd like to draw special attention to GitHub user david60, who has given detailed bug reports on a number of important issues:
    https://github.com/NebulousLabs/Sia/issues?q=is%3Aissue author%3Adavid60
    As our community grows (especially the developer community), I hope to see more users contributing at this level of quality. It goes a long way towards making Sia more robust.

  • admins

    I wanted to provide everyone with an update as we've been pretty quiet since the launch of the v1.0.0.

    We have been, slowly but surely, increasing the amount of testing that we are doing and fixing the bugs that we run into along the way. We have found a large number of bugs, some of which I will describe below. We are in the middle of setting up a proper, long-running testnet. We have improved the control mechanics of some of the modules of our code so that we can more easily run local tests that simulate dozens of nodes (currently most of our tests are either on a single node or between two nodes).

    Some of the bugs that have been fixed so far:

    • The host was not properly protecting file contracts as they were being revised. In particular, if the host was trying to perform an action such as a storage proof while a renter was uploading data, the host would potentially corrupt the file contract. This has been revised.

    • There was low tolerance for propagation differences between the renter and the host. Because Sia is a global network, it is frequently the case that a renter will have more blocks than the host it is talking to by a small number. When this was the case, the renter would potentially be unable to form file contracts. This issue has been resolved by having the host wait/sleep if there seems to be missing transactions or missing blocks.

    • The renter was not always adding a high enough transaction fee to the file contracts that it was creating, causing the host to reject all attempts to form file contracts.

    • The host was supposed to auto-disable if the host was having disk troubles. We've since realized that this is not the best way to respond to disk troubles. Furthermore, the host would frequently disable due to non-disk related issues, meaning that most hosts would have trouble staying online for more than a few hours because they would keep auto-disabling for the wrong reasons. The auto-disable code has been removed.

    • Miners were frequently having coins stolen because they were exposing their ports over the open internet, meaning that anyone with your IP address could talk to your wallet and ask for your coins. If you are a miner, your IP address is going to be visible to everyone via the node list, so it was easy for some attacker to scan the network and steal from any miner with an insecure network connection. This really isn't a bug, as miner's should have known better. However, enough people were doing it that we added some extra safety features to make sure miners wouldn't do it on accident. Now, before you can serve your API over the open internet, you must pass a special flag to allow you to do so, and you must also authenticate your API via an http password. It should be noted that this still only provides weak security - the API authentication is plaintext http, which means anyone who can snoop your connection will be able to steal the api pasword and then your wallet funds. This is significantly harder however than scanning the network and checking for unauthenticated and exposed nodes.

    • The host would occasionally reject renter file contracts in a way that the renter found confusing, so the renter would report the wrong error in the logs. This has been fixed.

    • The renter, when doing a revision, would corrupt the file contract revision if the connection to the host was disrupted at just the wrong time. Once the file contract revision was corrupted, the renter could no longer upload data anymore, and potentially certain data would also be unavailable for download. This has been fixed.

    • When trying to download files, the renter would incorrectly fetch the list of contracts/hosts that protect the files. This would often cause downloads to fail. This has been fixed.

    The graphical client has seen a similar list of fixes, though in general they are less critical to the functioning of the network. By and large, most of these problems were able to make it into the release because of insufficient testing. We believe that a handful of bugs still remain, but our local tests for hosting and renting are now much more successful.

    We will be releasing siad v1.0.1 and Sia-UI v1.0.2 in the next week or two. The first release candidate for these should be out in the next day or two. It may still take a long time to form contracts, and wallets with a heavy use history may take a very long time or may not be able to form contracts at all. Once contracts have been formed, there should be far fewer issues with uploading and downloading. I strongly recommend that everyone upgrade as soon as the release is available, as there are many bug fixes, including security fixes.

    I would like to thank everyone for their patience. We have delayed the release of long-desired features such as instant-wallet-unlocking in favor of writing more testing and fixing the features that we've already implemented. As best we can tell, there are no major architectural issues, merely a lot of bugs that we need to work through. We have already worked through many of them, and will continue working through them over the next month.

Log in to reply