Standard Grant: HTLC Upgrade for Sia for use in Atomic Swaps

Hello @alrighttt

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

What progress was made on your grant this month?

Progress has been made on porting transaction encoding and signing code to Sia Rust. 

Security audit of atomic swap consensus logic within Sia Go has begun. At least one significant issue has been found. 

Summarize any problems that you ran into this month and how you’ll be solving them.

Lacking documentation for "index mode" will soon be a blocker.

Pain points with using the Sia walletd API to broadcast v2 transactions were identified and remedied with Nate.

Significant issues were discovered with Sia's core encoding. The porting the fixes have potential to be a blocker in the coming weeks.

Would like to set a meeting with Nate to better understand which types, if any, of v1 consensus code are required to be ported.

Links to repos worked on this month:

https://github.com/alrighttt/core - added additional unit tests
https://github.com/KomodoPlatform/komodo-defi-framework/pull/2108/commits - draft PR is open for some additional encoding functionality

What will you be working on next?

Work will continue on the security audit of the relevant atomic swap spend policies. We will validate any fixes from the Sia developers.

We will continue with the rust port of v2 transaction encoding and signing. As this progressed, we will continue with informational wallet functionalities.

Hello @alrighttt

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

As per our SoW document a demonstration of “Information Wallet Functionality” was expected to be delivered today.

However, due to previous blockers regarding ‘Index mode,’ development focus shifted from the HTTP client within the Komodo DeFi Framework to furthering the progress of the Sia Rust implementation. Significant progress has been made on the porting of relevant Sia Go types to our rust codebase. This work was originally planned further along into the stated timeline in our SoW.

We will provide an addendum to our SoW document detailing of our efforts thus far and timelines going forward alongside the monthly update July 2.

What progress was made on your grant this month?

  • Requested and received a script from Nate to generate an “Event” of each type. This is useful for developing the API client within the DeFi Framework.

  • Implemented all remaining data types necessary for V1 and V2 functionality. Work remains on implementing decoding and deserialization for some types involving file contracts and SiaFunds.

  • Significant effort was spent porting the “Event” type to Rust as this involves a complex Go interface and custom serialization.

  • Significant effort was spent testing and porting the parser of the miniscript-like SpendPolicy serialization. Testing was performed to analyze the security of the UnlockConditions backwards compatibility code within v2 consensus. Some feedback regarding this was provided to Nate via DM.

  • A bug in the serialization of the PolicyTypeUnlockConditions Go type was reported to Nate. The bug was fixed here: Change unlock conditions policy encoding to be unambiguous by n8maninger · Pull Request #173 · SiaFoundation/core · GitHub

  • The basic “Tier Nolan” atomic swap protocol has been adapted to Sia’s SpendPolicy idioms.

  • Many additional unit tests were published for both Go and Rust implementations.

  • An addendum to the original SoW document for the grant has been published detailing the current overall progress and new timelines for upcoming tasks.

Summarize any problems that you ran into this month and how you’ll be solving them.

  • Encountered undefined behaviors in various encoding methods in Sia core. These instances have been reported to the Sia development team. The Sia Rust encoding must be refactored. Previously we followed the assumption that encoding of a Transaction object will always succeed. We must now refactor this encoding to return Rust Results.

  • We identified the lack of a critical Walletd API endpoint. This was communicated to Nate. We requested and received the api/events/:id API endpoint enabling the DeFi Framework to check confirmations of arbitrary transactions.

  • A refactor of the UnlockKey type within Sia Rust was required to account for PR#173. Previously it was an acceptable assumption that any UnlockKey encountered over the API would be ed25519 and therefore have a 32 byte public key. The refactor was required to account for arbitrary length public keys and arbitrary algorithm Specifiers.

Links to repos worked on this month:

Working branch:

Demonstration of undefined behavior within Sia Go encoding:

Go unit tests:

SpendPolicy adaptation of Tier Nolan atomic swap protocol:

SoW Addendum:

What will you be working on next?

The team will shift focus almost entirely to the remaining work for the “Sia Rust” library in the coming month. This will enable us to rapidly integrate any remaining endpoints into the DeFi Framework’s Walletd API client.

1 Like

Hello @alrighttt

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Hello @alrighttt,

This is a reminder that your progress report is due.

Please be sure to follow the Progress Report Template and submit your progress report by the 2nd of each month. Thanks!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Progress Report Form
What progress was made on your grant this month?

Summarize any problems that you ran into this month and how you’ll be solving them.
Please summarize your issues into a few sentences or bullet points:

  • SpendPolicy parser whitespace handling proved an ongoing issue. This is now fully remedied by walletd API allowing JSON serialization of the SpendPolicy type.

Links to repos worked on this month:

What will you be working on next?
Please summarize your development goals into a few sentences or bullet points:

  • HD wallet support within the DeFi Framework API. Seed generation and path derivation inline with BIP39.

  • Transaction creation helpers and finalizing wallet operations support.

The item detailed in the SoW Addendum, “Interactive Wallet Functionality”, will be included alongside our monthly report on September 2. We aren’t satisfied with the level of completion for this task, and we prefer to give a more polished demonstration than we are capable of providing as of today.

Progress Report Form
What progress was made on your grant this month?

Sia Rust now acts a standalone Walletd client capable of running fully within a browser via WASM.

The Komodo Wallet frontend now supports activating SIA, viewing transaction history, sending and receiving funds.

A persistent public testnet has been established with a public indexer node hosted at https://sia-wallet.komodo.earth
GitHub - KomodoPlatform/walletd at komodo-testnet must be used to sync this testnet

We are prepping a public demo of a Komodo Wallet instance with Sia integrated. This will be made available shortly.

Summarize any problems that you ran into this month and how you’ll be solving them.
Please summarize your issues into a few sentences or bullet points:

  • No major technical hurdles.

  • This past month involved introducing additional team members to Sia for the first time. Our Frontend developers and QA team members have now begun significant efforts in each of their departments.

  • Some minor feedback regarding Walletd was provided resulting in Walletd pull requests #168, #170 and #172. Thanks, Nate!

Links to repos worked on this month:

1 Like

Hello, we are working on making the demo publicly available. We have identified issues with our HTTP client within the WASM version of Sia rust, and we are refactoring a major portion of it.

Progress can be tracked at the above repos. If any team members would like a technical overview of our current progress, please reach out here or on discord.

1 Like

Hello @alrighttt

Thank you for your Progress Report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

Progress Report Form
What progress was made on your grant this month?

Summarize any problems that you ran into this month and how you’ll be solving them.

Links to repos worked on this month:

What will you be working on next?
Please summarize your development goals into a few sentences or bullet points:

Final delivery of the grant including release candidates of sia-rust and Komodo DeFi Framework with Sia fully integrated.

Hi, we’re nearing final delivery of the grant. We understand today is the stated conclusion date. We are tying up loose ends in our GUI, our “Sia Rust” library and Komodo DeFi Framework.

We have outstanding items that must be addressed prior to producing a true “release candidate”. We anticipate significant activity through the weekend and early next week. We will also continue to make incremental improvements as we see fit past the conclusion of the grant. We will strive for a polished UX ti be ready day 1 of Sia’s V2 consensus activation through continued internal testing and ongoing user testing via the Anagami testnet.

For anyone interested in tracking progress, we have activity happening in each of our three related repos. Sia related feature branches have “sia” in their branch names:

This komodo-wallet repository is the newly open source front end hosted at https://app.komodoplatform.com/ . We will be adding a testnet coin connected to Sia’s Anagami testnet.

An in-depth progress report detailing this last month of development will be provided on November 2.

1 Like

Hello @alrighttt

Thank you for your progress report!

As this is expected to be your final progress report, the team will be reaching out to you for the offboarding process and next-steps.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

@alrighttt Thanks again for the update so far.

An in-depth progress report detailing this last month of development will be provided on November 2.

Do you have an idea of when this will be delivered?

Apologies for the awful communication these past weeks. We would like to request an extension until Monday December 1. There are no major issues related to the project. I have been in charge of this project within the Komodo team, and I have neglected to keep our team well informed or well organized as I was forced to take personal leave. I am back to full capacity as of today.

I will post a progress report by the end of the day today. To summarize, our frontend needs better CI coverage, and the backend needs to implement recent serde changes introduced in Sia’s v2 consensus logic.

Hello, this report doesn’t follow the exact format of others due to covering more than a month and summarizing any remaining blocking items. The following list includes only major items as documenting individual changes would be extremely verbose for these past few weeks of development.

For anyone interested in a more exhaustive changelog see the following git diffs:

Currently we are facing a blocker due to two issues.

Firstly, our rust implementation of Sia consensus types needs to be updated to account for recent changes to Sia V2 consensus protocols.

Secondly, our CI/CD for komodo-wallet and komodo-defi-framework are still running the older version of the consensus protocol. Another round of manual QA testing will need to take place after these changes are fully integrated.

We must integrate the following changes to the repositories komodoplatform/sia-rust, komodoplatform/komodo-defi-framework, komodoplatform/komodo-wallet, komodoplatform/sia-core, komodoplatform/sia-core-utils

As per Nate:

  • State elements had their ID field removed
  • All of the types that embedded a state element now have a state element and ID field
  • v2 file contracts added a capacity field
  • JSON-wise
    • All prefixes were removed except for public keys with ed25519: prefix
    • Siacoin/Siafund/V2FileContract element types are now {"id": "", "stateElement": {}}

The following are “TODO items” which describe some inefficiency or desire for improvement that will continue development past the “release candidate” release.

  • Only encoding of the V2Transaction types were ported to rust in order to provide the ability to generate signatures. As a result of this, KDF is sending the V2Transaction type over the wire in a different encoding format than Sia core would produce. Realistically this has no practical effect, but it’s an inefficiency we would like to remedy. We will fully implement V2Transaction decoding.

  • HD wallet support is under active development within Komodo DeFi Framework. There is no blocker for this within the Sia module, but the feature itself within KDF is still being actively developed. It’s yet to be determined how to provide a clean UX if a Sia user wants to import an existing seed or export a seed generated via KDF to Walletd.

  • Transaction miner fee calculation is static in some cases. We will change this to dynamic transaction fees based on the api/txpool/fee prior to the Sia hardfork. This has been put off because it requires significant refactoring of our V2TransactionBuilder type. For the time being, fees are static for transactions with variable inputs and outputs lengths.

  • Much of the documentation is inline cargo doc comments. Not much attention has been given to how these will be formatted when shown via the webpage generated via cargo doc. The format will be standardized prior to publishing the sia-rust crate to crates.io.

  • We were unaware of the existence of the Walletd endpoint, /api/outputs/siacoin/:id, due to it not appearing in the documentation. Utilizing this endpoint can greatly simplify some of the logic with the KDF Sia module, and we intend to optimize these cases.

To summarize, “hand off” is obviously not as smooth as we had hoped, but rest assured we are continuing dedicated work on this project to deliver a polished open source product. We are extremely excited to have Komodo Wallet and any forks ready and thoroughly user tested well before the deployment of the Anagami update. We value open communication and transparency, so please don’t hesitate to ask any questions or provide blunt feedback.

This diff can hopefully shine some light on the work involved to incorporate the changes from:

We now need to do a round of review on these changes to our sia-rust library, incorporate these changes into Komodo DeFi Framework and Komodo Wallet

We weren’t aware of these changes until early November.

Hi, we’re preparing the final handover. We want to give some idea of what this entails.

The current plan for delivery is to provide a hosted instance of our frontend, Komodo Wallet, with Sia testnet(s) active. We’ll run this for at least a few weeks in hopes of getting user feedback. It’s probably not an issue for us to run this until the hardfork activation especially if we are receiving good user feedback. We would encourage the Sia team or community to run their own instances as well.

We’ll provide a general handoff document which will summarize our efforts and detail each relevant codebase and where to find the documentation.

Assuming no major changes to Sia Core, we can handle a few hours here and there to integrate any changes made between now and mainnet activation.

It is critically important that we receive notification of any changes to Sia Core’s data types, encoding or serialization. We don’t intend to maintain our fork of Sia Core indefinitely. If possible, we are requesting that these tests be integrated into CI or otherwise ran on a regular basis by the Sia developers. We are happy to restructure them or otherwise clean them up if running them in CI is agreeable. They serve as a crticial check that our Rust implementation of Sia data types matches Sia Core. If there were ever a mismatch, it could lead to bugs or exploits.