July 2016 Update + Roadmap

[Taek]

June 28th saw the release of our v1.0.0, the first decentralized storage app
to make it into production. Following the release, we saw huge growth numbers.
The number of people on our mailing list doubled, the number of people in our
slack rose dramatically, the amount of traffic on our webpage rose
dramatically.

We saw the number of hosts increase to 75 hosts serving more than 300TB of
data, and we had an influx of requests from potential business customers who
have taken an interest in Sia.

That said, the release was not without its problems, and I apologize for being
quiet on these for so long. Many farmers were having problems keeping their
hosts online, and many renters were having trouble uploading, and downloading
their files. We spent most of the past week talking to users, reading logs,
and trying to figure out exactly what was going on with the network.

There is good news. Most of the problems are small in scope even if large in
effect. By and large, the architecture of Sia is sound and does not need to be
rewritten. Most of the issues are things that we can fix in a few weeks, and
while I'm sure they will not be the last speedbumps, they are exactly that -
speed bumps. We need to slow down a little, increase the testing, fix the
bugs, and then we can keep going on our way.

The vast majority of the problems seem to be related to a single bug. When a
renter creates file contracts with hosts, the renter creates a long chain of
interdependent transactions. It might look like this:

fc1 -> fc2 -> fc3 -> fc4

If one of those transactions, say fc2, ends up being invalid, then you lose
all of the following transactions, even though only one of them is invalid.

fc1 -> X -> X -> X

Perhaps more significantly, even if they are all valid, the host on the other
side of the connection may not be receiving all of the dependent transactions,
meaning the host will just see an invalid transaction even if the transaction
is complete.

? -> ? -> X -> X

Ultimately, fixing these problems is not conceptually very difficult. We first
need to make sure that the renter is not creating dependent transactions until
their parents have been confirmed valid, and we need to make sure that the
host is properly getting all dependencies during the contract formation
process.

Though this was the core problem, talking to users revealed about 30 bugs
overall, around half of which have been fixed up to this point. The other half
are still being worked on, however none of the bugs are very scary - we just
need to put in the hours and we should have all currently known issues patched
up in the next week.

And we need to be expanding our testing to make sure that we're properly
catching these bugs before they get into production. We released v1.0.0
because we felt that the core architecture was complete, but as we have seen
the testing was insufficient and ultimately I think we can say that Sia was
released too early. The good news is that we should be able to spend most of
the next month improving our testing and our testing strategies, and by the
end of the month have a release that is much stronger.

The current plan is to have siad v1.0.1 and sia-ui v1.0.2 out on July 19th
with all of the major bugs patched up. Throughout July and much of August we
will be focused largely on testing, and sometime mid-August you can expect
another release that has bugfixes for everything that shows up during our
improved testing and hardening.

Jordan, one of the Nebulous team members, will be adding business development
and marketing to his list of responsibilities. This will include building and
sourcing infographics, interacting with enterprise customers, growing our
third party app community, expanding our meetup communities, and overall
pushing the business interests of Sia.

For those of you looking for ways to help, there's one thing that Sia needs
more than anything else right now - testing. If you are not a developer, the
best thing to do is to download and run the app, and report all problems or
annoyances that you run into. If you are a developer, we could especially used
improved test coverage on our API. Here is a screenshot of our current API
coverage:

As you can see, some of our endpoints are covered by tests, but most of them
are not. In the screenshot, there is partial coverage for the API call that
allows you to add a storage folder to the host, but no coverage at all for the
API call that allows you to resize a storage folder.

There is a guide for getting started here. Please do not hesistate to ask
(on either the forum or on the slack) for guidance, we're happy to help, and happy to code review.

The Sia team is as excitied as ever to be partipating in the decentralization
movement. All parts of the ecosystem are seeing lots of growth, and while
there's still a long road ahead of us we're all much further down that road
than we were a year ago, and the future looks bright.

[halpmon]

Thanks for the update. Sounds great!

[nemo]

For those of you looking for ways to help, there's one thing that Sia needs
more than anything else right now - testing. If you are not a developer, the
best thing to do is to download and run the app, and report all problems or
annoyances that you run into.

I'd like to draw special attention to GitHub user david60, who has given detailed bug reports on a number of important issues:
https://github.com/NebulousLabs/Sia/issues?q=is%3Aissue author%3Adavid60
As our community grows (especially the developer community), I hope to see more users contributing at this level of quality. It goes a long way towards making Sia more robust.

[Taek]

I wanted to provide everyone with an update as we've been pretty quiet since the launch of the v1.0.0.

We have been, slowly but surely, increasing the amount of testing that we are doing and fixing the bugs that we run into along the way. We have found a large number of bugs, some of which I will describe below. We are in the middle of setting up a proper, long-running testnet. We have improved the control mechanics of some of the modules of our code so that we can more easily run local tests that simulate dozens of nodes (currently most of our tests are either on a single node or between two nodes).

Some of the bugs that have been fixed so far:

• The host was not properly protecting file contracts as they were being revised. In particular, if the host was trying to perform an action such as a storage proof while a renter was uploading data, the host would potentially corrupt the file contract. This has been revised.

• There was low tolerance for propagation differences between the renter and the host. Because Sia is a global network, it is frequently the case that a renter will have more blocks than the host it is talking to by a small number. When this was the case, the renter would potentially be unable to form file contracts. This issue has been resolved by having the host wait/sleep if there seems to be missing transactions or missing blocks.

• The renter was not always adding a high enough transaction fee to the file contracts that it was creating, causing the host to reject all attempts to form file contracts.

• The host was supposed to auto-disable if the host was having disk troubles. We've since realized that this is not the best way to respond to disk troubles. Furthermore, the host would frequently disable due to non-disk related issues, meaning that most hosts would have trouble staying online for more than a few hours because they would keep auto-disabling for the wrong reasons. The auto-disable code has been removed.

• Miners were frequently having coins stolen because they were exposing their ports over the open internet, meaning that anyone with your IP address could talk to your wallet and ask for your coins. If you are a miner, your IP address is going to be visible to everyone via the node list, so it was easy for some attacker to scan the network and steal from any miner with an insecure network connection. This really isn't a bug, as miner's should have known better. However, enough people were doing it that we added some extra safety features to make sure miners wouldn't do it on accident. Now, before you can serve your API over the open internet, you must pass a special flag to allow you to do so, and you must also authenticate your API via an http password. It should be noted that this still only provides weak security - the API authentication is plaintext http, which means anyone who can snoop your connection will be able to steal the api pasword and then your wallet funds. This is significantly harder however than scanning the network and checking for unauthenticated and exposed nodes.

• The host would occasionally reject renter file contracts in a way that the renter found confusing, so the renter would report the wrong error in the logs. This has been fixed.

• The renter, when doing a revision, would corrupt the file contract revision if the connection to the host was disrupted at just the wrong time. Once the file contract revision was corrupted, the renter could no longer upload data anymore, and potentially certain data would also be unavailable for download. This has been fixed.

• When trying to download files, the renter would incorrectly fetch the list of contracts/hosts that protect the files. This would often cause downloads to fail. This has been fixed.

The graphical client has seen a similar list of fixes, though in general they are less critical to the functioning of the network. By and large, most of these problems were able to make it into the release because of insufficient testing. We believe that a handful of bugs still remain, but our local tests for hosting and renting are now much more successful.

We will be releasing siad v1.0.1 and Sia-UI v1.0.2 in the next week or two. The first release candidate for these should be out in the next day or two. It may still take a long time to form contracts, and wallets with a heavy use history may take a very long time or may not be able to form contracts at all. Once contracts have been formed, there should be far fewer issues with uploading and downloading. I strongly recommend that everyone upgrade as soon as the release is available, as there are many bug fixes, including security fixes.

I would like to thank everyone for their patience. We have delayed the release of long-desired features such as instant-wallet-unlocking in favor of writing more testing and fixing the features that we've already implemented. As best we can tell, there are no major architectural issues, merely a lot of bugs that we need to work through. We have already worked through many of them, and will continue working through them over the next month.