Deliverable: Utreexo hardfork code implemented, tested, and released
Manpower: 1-3 Foundation engineers, 3-4 contracted engineers
Budget: 300MS (does not include Foundation engineer salaries)
Timeline: Jan 2021 — Jan 2022
This is a formal proposal for implementing a new consensus algorithm for Sia based on Utreexo. Utreexo is a proposal for Bitcoin that replaces the UTXO set with a compact, hash-based accumulator. This greatly lowers the resource requirements for operating a full node, at the cost of increased bandwidth. Another significant feature of Utreexo is that, due to the small size of the accumulator, fully-validating nodes can be “bootstrapped” using a short message or even a QR code. (Okay, technically they’re more like “forward-validating” nodes, but you get the idea.) Thanks to this bootstrapping, you’ll be able to run a node on a phone, Raspberry Pi, or other low-power device. You’ll also be able to help your friends bootstrap: instead of telling them to start
siad and wait a day or two for it to download the entire blockchain, you can just give them your accumulator, and they’ll be instantly synced with no loss of security (as long as they trust you, of course).
Adopting the Utreexo model in Sia is a major undertaking: I expect it to be the largest single project that the Foundation pursues in 2021. Much of the existing consensus and networking code will need to be heavily modified or replaced entirely. This code has served us well for years now, but we have learned many lessons during that time. Utreexo offers a chance for us to start fresh on an extremely robust foundation. My hope is that the resulting codebase not only meets our own needs, but also serves as a template and an inspiration for the wider crypto community—something that everyone can point to as an example of excellent blockchain engineering.
Work on integrating Utreexo has already begun. I have been making slow progress throughout 2020 towards “minimum viable Utreexo” – that is, the simplest possible blockchain that contains all the core Utreexo functionality. As a result of this work, we’ve learned a lot about what Utreexo consensus looks like in practice, and how to adapt it to Sia’s requirements. The next step in the process is the development of a comprehensive specification, laying out in detail every aspect of Utreexo-based Sia. This includes the accumulator’s Merkle tree structure, the transaction format, new consensus rules, changes to the transaction pool and wallet, changes to the gateway protocol, the hardfork upgrade procedure, and much more. The spec will be developed in cooperation with Skynet Labs.
I expect the specification to be complete in Q1 2021. At this time, we will publish the spec and announce a timeline for the implementation, consisting of anticipated deadlines for various milestones. During the implementation phase, the Foundation will work closely with Skynet engineers to turn the agreed-upon spec into actual code. The Foundation will be publishing monthly updates as to the status of Utreexo (among other projects), allowing the community to track our progress towards future milestones. Once all of the major pieces are in place, we will need to begin a massive testing effort; after all, this will constitute the most significant hardfork in Sia’s history. At this time, we may also solicit professional security audit(s) for the new consensus code. When the code has been tested and audited to our satisfaction, we will deploy it to a public testnet for community testing. Assuming the testnet operates without serious bugs or failures for a reasonable period of time, we will set an activation height for the hardfork and release the hardfork binaries.
I am allotting a period of 1 year for this effort. Software deadlines are notoriously over-optimistic, and so despite having made substantial progress already, I anticipate that there will be many unforeseen challenges and setbacks during the integration process. It’s possible that the work will be complete well before December 2021, but it’s also possible that we barely squeak by and the hardfork does not actually activate until 2022.
I will lead the overall effort, with the assistance of any in-house devs that can be found and trained in time. Our salaries will fall under the larger Core Development budget, so are not counted here. I am budgeting a further 300 MS for contracting, security audits, and other associated costs. This is based on an estimate of $100k for a high-quality security audit and $500k for contracting 3-4 Skynet engineers part-time for one year.