Standard Grant Proposal: HTLC Upgrade for Sia for use in Atomic Swaps

Grants Approved
9

Here is the Scope of Work for Milestone #1

1 Like
10

The first progress report has been uploaded to github:

No major hiccups. Progress is within the expectations of the Scope of Work document.

Please direct any questions to “alrighttt” on Discord.

1 Like
11

Thanks for your progress report. Please create a new post with the content directly in the forum as opposed to linking out to Github. We prefer this for archival purposes. Thanks!

12

Progress Report #1

This report was initially published on github on February 23, 2024.

Sia Skeleton Protocol

A minimal implementation of the Sia protocol was pushed to the sia-dev branch of GitHub - KomodoPlatform/komodo-defi-framework: This is the official Komodo DeFi Framework repository

This represents the foundation for the Sia protocol within the Komodo DeFi Framework and serves as a template for all future Sia related work.

This enables the integration of a Sia-based coin within the Komodo DeFi Framework API via the newly created task::enable_sia::init API endpoint.

Usage:

Start the Defi Framework API with the following within coins file:

  {
    "coin": "TSIA",
    "asset": "TSIA",
    "fname": "TSIA",
    "rpcport": 59842,
    "mm2": 0,
    "protocol": {
      "type": "SIA",
      "protocol_data": {
      }
    }
  }

Send a POST request with the following body:

{
	"userpass": "{{userpass}}",
    "mmrpc": "2.0",
    "method": "task::enable_sia::init",
    "params": {
        "ticker": "TSIA",
        "activation_params": {"http_url" : "http://localhost:9980/", "http_auth": "password"}
    }
}

This enables further commands such as my_balance. All endpoints currently provide mocked data or intentionally cause a thread to crash via the unimplemented!() Rust macro.

The relevant code for this “skeleton implementation” can be viewed in this git comparsion: Comparing af571608c34f38d82e3c421f9bd61c5265db0cf8...sia-dev · KomodoPlatform/komodo-defi-framework · GitHub

Docker Testnet Framework

The basis for the Docker unit test framework has also been established within the Komodo DeFi Framework. This is based on the Dockerfile provided by the Sia team within the walletd repo. It has been adapted to establish a “komodo” testnet instead of the anagami testnet.

The foundation address of this testnet has been changed to one controlled by the Komodo API team. This testnet has its POW difficulty lowered to allow blocks to be created on demand.

Moving forward, another testnet will be established in a similar fashion. One testnet will work as a persistent internal testnet for the Komodo API team. The other testnet implementation will facilitate unit tests within the Komodo DeFi Framework and will not be persistent.

The relevant code can be found within this commit: init basic sia docker node · KomodoPlatform/komodo-defi-framework@32fdb85 · GitHub

Usage:

cargo test --features "run-docker-tests" --package mm2_main --test docker_tests_main docker_tests::dummy -- --exact

Walletd CPU miner changes

The need to create an arbitrary number of blocks with the walletd CPU miner was identified. A new parameter, blocks, was added to the mine command of walletd. This new parameter allows the user to specify how many blocks the CPU miner should mine before exiting. This is crucial for future unit tests within the Komodo DeFi Framework.

A pull request adding this parameter was opened: add blocksToMine to testnet miner by Alrighttt · Pull Request #53 · SiaFoundation/walletd · GitHub

Walletd /util/generateaddress Endpoint

A new walletd endpoint has been created to enable address generation from an arbitrary SpendPolicy. This was identified as a necessary first step towards testing the newly introduced consensus mechanisms that will allow for HLTC-like consensus mechanisms.

The code is simple and relies on already established mechanisms for serializing and deserailizing the Go types within walletd. This endpoint is unlikely to be merged upstream or be made available to users unless requested.

The code for this endpoint can be found in this commit: add generateaddress util method; change initial POW on anagami · SiaFoundation/walletd@0b53f40 · GitHub

Usage:

curl -u :password --location 'http://localhost:9980/api/utils/generateaddress/' \
--header 'Content-Type: text/plain' \
--data '"thresh(1,[thresh(2,[after(1708697003),pk(0x0102030000000000000000000000000000000000000000000000000000000000)]),thresh(2,[h(0x0102030000000000000000000000000000000000000000000000000000000000),pk(0x0405060000000000000000000000000000000000000000000000000000000000)])])"'

Next Steps

The Komodo API team will commence with manual testing of the newly introduced Sia consensus mechanisms.

ed25519 signature creation within the Komodo DeFi Framework has been identified as a useful first step in regards to this testing. Given arbitrary ed25519 signing using the DeFi Framework’s private key handler, the API team can begin to produce complex HLTC transactions within walletd.

As this consensus testing goes forward, various endpoints will be added to both the Komodo DeFi Framework and walletd APIs.

The interface between both APIs will be established as needed in order to facilitate this manual testing.

Unit tests utilizing the newly established walletd Docker framework within the Komodo DeFi Framework will be established in tandem with any new code for the Sia protocol integration.

13

Progress Report #2

As per the Scope of Work document, a test report regarding the manual testing processes of Sia’s new “v2” consensus mechanisms is expected as of today. Much of the time in the past weeks was dedicated to establishing the “Sia Rust” library. This task was identified as a necessary step towards further in-depth consensus mechanism testing. A test report will be produced in the coming weeks.

This report has also been published to github: https://github.com/KomodoPlatform/sia-grant/blob/main/ProgressReport2.md

Komodo Testnet walletd Rebase

The Komodo testnet branch has been rebased to SiaFoundation’s latest master branch after previously being some weeks behind due to using the “its-happening” branch as upstream.

This was identified as a critical step forward as the DeFi Framework will require use of walletd API’s newly introduced “indexer mode” and its new endpoints such as api/addresses/{address}/balance.

ref: GitHub - Alrighttt/walletd at komodo-testnet-rebase

Komodo DeFi Framework Draft Pull Request

The initial pull request with all Sia related changes thus far has been opened within the Komodo DeFi Framework github repository. This is a draft pull request. It will be finalized and will be reviewed by the API team before being merged into the dev branch.

As of today, this pull request helps easily visualize all newly added Sia functionality to the DeFi Framework.

ref: feat(sia): Initial Sia integration by Alrighttt · Pull Request #2086 · KomodoPlatform/komodo-defi-framework · GitHub

Sia API Client Interface within DeFi Framework

The initial interface between the Komodo DeFi Framework API and Sia’s walletd API has been established. This is currently only serves as an example on how to implement walletd API endpoints within the DeFi Framework.

Two endpoints have been integrated, /api/consensus/tip and api/addresses/{address}/balance.

ref: https://github.com/KomodoPlatform/komodo-defi-framework/blob/d180505b43f8167bd733263e73804ea60d4c1632/mm2src/coins/sia/http_client.rs#L102

walletd API WASM Research

Some research was performed to assess the feasibility of using the WASM object produced by Sia’s core/walletd within the DeFi Framework API.

The assessment is that it is not feasible to use this WASM object in any capacity. No further related work will be performed.

A bare bones example of how to build and utilize this WASM object was published.

ref: GitHub - Alrighttt/sia-wasm-web-example: A simple demonstration on how to utilize the WASM object outputted by Sia's walletd

Sia Rust

The beginnings of the Sia Rust library have been established.

All functionality necessary for generating addresses from arbitrary UnlockConditions or SpendPolicy objects has been established.

A series of unit tests have been established in both the Go and Rust codebases to easily identify any potential future discrepancies.

The tests found in the above “rust_port_test.go” will immediately identify any changes that must be incorporated into Sia Rust if the Sia developers change any fundamental data types or hashing or encoding functions for these data types.

Next Steps

The testnet setup will be reorganized in the coming weeks as manual testing of the consensus mechanisms continue. The Komodo persistent testnet will be re-established in the coming weeks.

Further unit tests will be established within both the Go and Rust codebases as additional data types are ported to Sia Rust. The current goal is to port all data structures necessary to create and serialize Sia transactions.

The draft pull request within the DeFi Framework will be finalized, reviewed and merged into the dev branch.

The initially planned “Sia Consensus Model Testing Report” will be delivered as the necessary tooling is developed to produce Sia transactions from with the DeFi Framework. The SpendPolicy and UnlockConditions data structures being ported are a major step towards this.

14

Hi @alrighttt, please shift your progress report deliverables to the 2nd of each month, covering the work done in the prior month. Also, while we welcome additional info, please follow the report format outlined here: About the Grants category - #8

Make sure you use that template, and if you’d like to provide additional info you can do that after those reporting requirements are met.

15

Hello @alrighttt,

This is a reminder that your April progress report is now late.

Progress reports are due by the end of the 2nd day of the month, in this instance your April progress report was due on May 2nd.

Please be sure to follow the Progress Report Template and submit your progress report so we may continue funding your grant.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

16

Apologies, I am preparing the report for later today. Will not happen again.

17

Progress Report Form

What progress was made on your grant this month?

The first pull request to the Komodo DeFi Framework was created, reviewed and merged into the dev branch.

The API client within the Komodo DeFi Framework has been refactored significantly since this pull request.

Further progress has been made on Sia Rust. Work has begun on implementing serialization and encoding of the Sia v1 transaction format.

A docker test suite was established to allow running a custom walletd instance in the DeFi Framework CI. This will be heavily utilized to closely mimic the real world Sia consensus model.

Summarize any problems that you ran into this month and how you’ll be solving them.

Please summarize your issues into a few sentences or bullet points:

We had anticipated further integration of the Sia API endpoints within the Komodo DeFi Framework. However, the “index mode” was not made available until recently. We instead shifted our focus to Sia Rust.

What will you be working on next?

Please summarize your development goals into a few sentences or bullet points:

  • Next code sprint pull request has been drafted. This will be reviewed and iterated upon next week.

  • Test Sia walletd pull request #112 to ensure the wallet balance and UTXO fetching mechanisms are sufficient for our needs.

  • Integrating new index mode API endpoints to the DeFi Framework.

  • Complete Sia v1 transaction support.

Links to repos worked on this month:

18

Hello @alrighttt

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

19

What progress was made on your grant this month?

Progress has been made on porting transaction encoding and signing code to Sia Rust. 

Security audit of atomic swap consensus logic within Sia Go has begun. At least one significant issue has been found.

Summarize any problems that you ran into this month and how you’ll be solving them.

Lacking documentation for "index mode" will soon be a blocker.

Pain points with using the Sia walletd API to broadcast v2 transactions were identified and remedied with Nate.

Significant issues were discovered with Sia's core encoding. The porting the fixes have potential to be a blocker in the coming weeks.

Would like to set a meeting with Nate to better understand which types, if any, of v1 consensus code are required to be ported.

Links to repos worked on this month:

https://github.com/alrighttt/core - added additional unit tests
https://github.com/KomodoPlatform/komodo-defi-framework/pull/2108/commits - draft PR is open for some additional encoding functionality

What will you be working on next?

Work will continue on the security audit of the relevant atomic swap spend policies. We will validate any fixes from the Sia developers.

We will continue with the rust port of v2 transaction encoding and signing. As this progressed, we will continue with informational wallet functionalities.
20

Hello @alrighttt

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

21

As per our SoW document a demonstration of “Information Wallet Functionality” was expected to be delivered today.

However, due to previous blockers regarding ‘Index mode,’ development focus shifted from the HTTP client within the Komodo DeFi Framework to furthering the progress of the Sia Rust implementation. Significant progress has been made on the porting of relevant Sia Go types to our rust codebase. This work was originally planned further along into the stated timeline in our SoW.

We will provide an addendum to our SoW document detailing of our efforts thus far and timelines going forward alongside the monthly update July 2.

22

What progress was made on your grant this month?

  • Requested and received a script from Nate to generate an “Event” of each type. This is useful for developing the API client within the DeFi Framework.

  • Implemented all remaining data types necessary for V1 and V2 functionality. Work remains on implementing decoding and deserialization for some types involving file contracts and SiaFunds.

  • Significant effort was spent porting the “Event” type to Rust as this involves a complex Go interface and custom serialization.

  • Significant effort was spent testing and porting the parser of the miniscript-like SpendPolicy serialization. Testing was performed to analyze the security of the UnlockConditions backwards compatibility code within v2 consensus. Some feedback regarding this was provided to Nate via DM.

  • A bug in the serialization of the PolicyTypeUnlockConditions Go type was reported to Nate. The bug was fixed here: Change unlock conditions policy encoding to be unambiguous by n8maninger · Pull Request #173 · SiaFoundation/core · GitHub

  • The basic “Tier Nolan” atomic swap protocol has been adapted to Sia’s SpendPolicy idioms.

  • Many additional unit tests were published for both Go and Rust implementations.

  • An addendum to the original SoW document for the grant has been published detailing the current overall progress and new timelines for upcoming tasks.

Summarize any problems that you ran into this month and how you’ll be solving them.

  • Encountered undefined behaviors in various encoding methods in Sia core. These instances have been reported to the Sia development team. The Sia Rust encoding must be refactored. Previously we followed the assumption that encoding of a Transaction object will always succeed. We must now refactor this encoding to return Rust Results.

  • We identified the lack of a critical Walletd API endpoint. This was communicated to Nate. We requested and received the api/events/:id API endpoint enabling the DeFi Framework to check confirmations of arbitrary transactions.

  • A refactor of the UnlockKey type within Sia Rust was required to account for PR#173. Previously it was an acceptable assumption that any UnlockKey encountered over the API would be ed25519 and therefore have a 32 byte public key. The refactor was required to account for arbitrary length public keys and arbitrary algorithm Specifiers.

Links to repos worked on this month:

Working branch:

Demonstration of undefined behavior within Sia Go encoding:

Go unit tests:

SpendPolicy adaptation of Tier Nolan atomic swap protocol:

SoW Addendum:

What will you be working on next?

The team will shift focus almost entirely to the remaining work for the “Sia Rust” library in the coming month. This will enable us to rapidly integrate any remaining endpoints into the DeFi Framework’s Walletd API client.

1 Like
23

Hello @alrighttt

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

24

Hello @alrighttt,

This is a reminder that your progress report is due.

Please be sure to follow the Progress Report Template and submit your progress report by the 2nd of each month. Thanks!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

25

Progress Report Form
What progress was made on your grant this month?

Summarize any problems that you ran into this month and how you’ll be solving them.
Please summarize your issues into a few sentences or bullet points:

  • SpendPolicy parser whitespace handling proved an ongoing issue. This is now fully remedied by walletd API allowing JSON serialization of the SpendPolicy type.

Links to repos worked on this month:

What will you be working on next?
Please summarize your development goals into a few sentences or bullet points:

  • HD wallet support within the DeFi Framework API. Seed generation and path derivation inline with BIP39.

  • Transaction creation helpers and finalizing wallet operations support.

26

The item detailed in the SoW Addendum, “Interactive Wallet Functionality”, will be included alongside our monthly report on September 2. We aren’t satisfied with the level of completion for this task, and we prefer to give a more polished demonstration than we are capable of providing as of today.

27

Progress Report Form
What progress was made on your grant this month?

Sia Rust now acts a standalone Walletd client capable of running fully within a browser via WASM.

The Komodo Wallet frontend now supports activating SIA, viewing transaction history, sending and receiving funds.

A persistent public testnet has been established with a public indexer node hosted at https://sia-wallet.komodo.earth
GitHub - KomodoPlatform/walletd at komodo-testnet must be used to sync this testnet

We are prepping a public demo of a Komodo Wallet instance with Sia integrated. This will be made available shortly.

Summarize any problems that you ran into this month and how you’ll be solving them.
Please summarize your issues into a few sentences or bullet points:

  • No major technical hurdles.

  • This past month involved introducing additional team members to Sia for the first time. Our Frontend developers and QA team members have now begun significant efforts in each of their departments.

  • Some minor feedback regarding Walletd was provided resulting in Walletd pull requests #168, #170 and #172. Thanks, Nate!

Links to repos worked on this month:

1 Like
28

Hello, we are working on making the demo publicly available. We have identified issues with our HTTP client within the WASM version of Sia rust, and we are refactoring a major portion of it.

Progress can be tracked at the above repos. If any team members would like a technical overview of our current progress, please reach out here or on discord.

1 Like