Project Name: Sia Enterprise Tooling 2024
Name of the organization or individual submitting the proposal: Bustedware LLC
Describe your project.
The goal of this project is to store Terraform state files on the Sia network and build a custom Terraform binary which natively integrates with the Sia network.
Terraform is an infrastructure as code (IaC) tool that allows you to define and provision your infrastructure and services in a declarative manner, enabling automated and consistent deployment and management of resources across various cloud providers and on-premises environments, using HashiCorps Terraform Language (HCL) code.
Terraform works by comparing the current infrastructure in the real world vs the infrastructure you have defined with HCL using a file which contains the infrastructure “state”. On each run of terraform it will use this state file to compare against the existing infrastructure and adjust accordingly.
The state file contains a relatively small JSON object. Developers working on the same infrastructure need to share this state file and can only run a single apply for a given state file at any given time. HashiCorp offers Terraform Enterprise at $500 per workspace to share state files among team members who are working on the same HCL and manage locks to ensure only a single developer is working on state at any given time. By moving the state file onto the Sia network, community users of Terraform can benefit from an enterprise like environment where they can share state without needing to invest in a heavy platform like Terraform enterprise, plus existing users of Terraform enterprise can save on cost by switching over to the version of Terraform built for the Sia network.
Sia Grafana will likely require regular maintenance as community adopts the dashboards and provide feedback. Throughout the year I will support the siagrafana dashboards and automation codebase while taking in community feedback, further enhancing the monitoring already available.
Who benefits from your project?
The entire Sia ecosystem could benefit by opening the doors for Terraform users to migrate their state files to the Sia network. This will increase demand on Siacoin, hostd servers will earn more, renters would likely see growing space on the network. Most of all theres big cost savings moving from an platform like Terraform Enterprise.
My continued community support and contributions for the Sia Grafana monitoring project will continue to benefit Sia Network Operators, Developers and Engineers, Enterprise Users, Data Analysts, Community Members, and Application Developers.
The dashboards have only been available for a short time but we have already seen some community activity/interest: Discord link1, link2, link3
How does the project serve the Foundation’s mission of user-owned data?
Developers will be able to more easily share their state files using the Sia network. With Sias renterd users can already store their files securely on the network that only they have exclusive access to.
The hostd, renterd, and walletd dashboards provide a single pane of glass view for the Sia networks storage infrastructure, improving the availability of user-owned data, and insights into performance.
Grant Specifics
- $72,000 for full time salary
Q1 Feb-Apr:
- Fork terraform code base, perform build, and test built from source binary.
(release will be provided in forked repo with a different checksum than the official release) - Start building discord bot to listen to Sia discord for prometheus questions and use chat gpt / ai integration to train and provide customized responses. It should be notifying me of any discussions related to prometheus for training data
- Optimize hostd, renterd, walletd prometheus endpoints and dashboards.
- finalize prometheus endpoints in the renterd and walletd forked repos which should be refactored similar to the encoder that’s used in hostd.
considerations in work for refactor and PRs for renterd and walletd
-
after combing through all endpoints and json objects, there’s a considerable amount of json that I didn’t include (intentionally) in prometheus endpoints for all services. some endpoints can also be optimized with considerations to labels and prometheus indexing/dbms files system resources. things like frequently changing timestamps in labels for example is extremely inefficient on the prometheus storage layer. during the hostd PR there was also collaboration about various metrics on what was needed vs not needed like the
/accountsendpoint which could have been millions of accounts and not valuable for a prometheus endpoint. things like this will come up during the renterd and walletd refactor -
dashboards are just developed by myself without community feedback. allowing a burn in period to gauge community interest and implement any community feedback would be extremely valuable.
Q2 May-Jul:
- Begin integrating renterd codebase for renting storage into terraform codebase.
- Terraform forked repo can be checked for commits/releases.
- Discord bot is able to provide at least 1 basic response to some well recognized prometheus related question.
Q3 Aug-Oct:
- First tests of terraform storing state files on the Sia network are performed.
- Terraform plans can be performed and state file can be retrieved from sia network.
- Terraform global configuration can specify a renterd seed for interacting with contracts.
- Potentially create middleware layer to interact with Sia for terraform on its behalf.
- With repo commits and releases users should be able to perform steps in README to run their own terraform plans. Focus optimizing renterd prometheus endpoints and dashboards. Refer to Q1 bullet point 2 regarding optimization focus
Q4 Nov-Jan:
- More testing with terraform and ensuring any bugs are worked out.
- Ensure that multiple hosts can access the same state file by sharing renterd seed / contracts.
Potential risks that will affect the outcome of the project
Time is the biggest limiting factor as this will be a second full time endeavor, on top of also supporting the sia grafana effort for the year. I believe working full time for 12 months straight the project can be completed to showcase minimum viable product( showing that Terraform state files can integrate with the Sia network and that dashboards/prometheus collection is optimized).
Will all of your project’s code be open-source?
Yes
Leave a link where code will be accessible for review
https://github.com/bustedware/terraform
Do you agree to submit monthly progress reports?
Yes
Contact info
Email: jbills@bustedware.com
Discord: @bustedware