Grant Proposal: Sia data sources for Grafana

Grants Proposed
1

Introduction

Project Name: Sia data sources for Grafana

Name of the organization or individual submitting the proposal: Bustedware LLC

Describe your project.
We seek funding to develop a Grafana data source tailored specifically for Sia hostd, renterd, and walletd services. This project aims to enhance monitoring and observability within the Sia ecosystem, providing users with a powerful tool to visualize and analyze key metrics from these products. Our data source will seamlessly integrate with Grafana, offering users an intuitive and unified platform to monitor the health, performance, and utilization of their Sia hostd and renterd services.

By creating this dedicated Grafana data source, we aspire to empower Sia users with insights into the behavior of their storage infrastructure. With your support, we are excited to bring this project to life and contribute to the advancement of Sia’s monitoring capabilities, ultimately strengthening the ecosystem’s resilience and growth.

Who benefits from your project?
A Sia Grafana data source plugin would benefit a wide range of stakeholders within the Sia network and ecosystem, including:

  1. Sia Network Operators: Hosts and renters on the Sia network would greatly benefit from the plugin. Host operators can monitor their hardware and network performance, track storage utilization, and optimize their offerings based on data insights. Renters can analyze their storage usage and reliability of hosts, helping them make informed decisions about their data storage strategies.
  2. Developers and Engineers: Those responsible for maintaining and optimizing the Sia network infrastructure would find the plugin invaluable for monitoring the health and performance of hostd. It could help them identify bottlenecks, track trends, and optimize the overall system for better efficiency.
  3. Enterprise Users: Enterprises utilizing Sia for decentralized cloud storage can leverage the plugin to ensure that their storage infrastructure is performing as expected. They can monitor usage, uptime, and overall system health, providing them with the confidence to use Sia for critical data storage needs.
  4. Data Analysts: The plugin can provide valuable data for analysts who want to analyze trends, usage patterns, and other metrics related to Sia’s storage services. This can lead to better insights into user behavior and contribute to the improvement of the Sia ecosystem.
  5. Community Members: Enthusiasts and community members interested in contributing to the Sia network can use the plugin to gain a deeper understanding of its performance and contribute to its growth by making informed suggestions and optimizations.
  6. Application Developers: Developers building applications that utilize the Sia network can use the plugin to monitor the storage aspect of their applications and ensure their users have a seamless experience.

Overall, the Sia Grafana data source plugin has the potential to benefit anyone involved with the Sia ecosystem, from individual users to enterprises, by providing valuable insights and data for more effective monitoring, management, and decision-making.

How does the project serve the Foundation’s mission of user-owned data?
The Grafana data source plugin for Sias hostd and renterd would provide a single pane of glass view for your entire storage infrastructure, improving the availability of user-owned data, and insights into performance.

Grant Specifics

  • $12,000 for part time salary

Timeline with measurable objectives and goals:
Month 1: Grafana datasource is installed, authenticated, and connected to hostd. Basic dashboard is built and displaying hostd metrics
Month 2: Grafana datasource is installed, authenticated, and connected to renterd. Basic dashboard is built and displaying renterd metrics
Month 3: Grafana datasource is installed, authenticated, and connected to walletd. Basic dashboard is built and displaying walletd metrics

Potential risks that will affect the outcome of the project:
I have developed grafana plugins in the past and I know an extremely customized data source can complicate and increase the lead time to project completion. I have been running siad primarily, there’s also getting over the initial learning curve for running hostd, renterd, and walletd services.

Development Information

Will all of your project’s code be open-source?
Yes

Leave a link where code will be accessible for review.
https://github.com/bustedware/siagrafana

Do you agree to submit monthly progress reports?
Yes

Contact info

Email: jbills@bustedware.com

2

Hello @sc2bigjoe,

Thanks for submitting this proposal. The committee has a few concerns they’d like to address before making a decision.

Using the Sia software as a data source relies on that data being stored locally in databases. While this is the case for hostd, renterd and walletd do not currently do that. What would be your solution here?

Feasibility aside, what other metrics or interesting data would you intend to pull for the software? If you have a list of preferred metrics for each app, our team can identify which are currently possible and which we might be able to build endpoints for.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

3

Using the Sia software as a data source relies on that data being stored locally in databases.

Not necessarily. Although I think that it would be a good extension of the plugin to be able to source data which is stored on the Sia Network… but that was not originally part of the vision for this effort.

I want this particular Grafana plugin to leverage the APIs for hostd, renterd, and walletd found here:
hostd: hostd
renterd: renterd
walletd: walletd

I haven’t fully reviewed all the available API endpoints in each service but as an example the metrics endpoint in hostd has lots of good information that could be displayed in a dashboard hostd

Absolutely if we need to plot the values over time it would need to be persisted somewhere but that doesn’t necessarily have to be the Sia Network.

If the foundation is interested, I can also develop a telegraf OUTPUT plugin which would store data on the Sia Network but I think we would need to double the amount of time and double the requested capital that I originally asked for as part of this grant. I am the original author for the telegraf MongoDB output plugin. I was planning to make a separate grant proposal for the telegraf effort but I’m happy to combine it with this one. The telegraf output plugin would be all encompassing, meaning anything telegraf can pickup would be stored on the Sia Network. I’m not certain if the Sia Network can persist data that frequently so I would probably need to batch/throttle writes for it.

Let me know if you need any additional information or if the foundation is interested in setting up a conference call to discuss further.

4

This is interesting. Can you point me to a repo?

5
1 Like
6

Hello @sc2bigjoe,

Thanks for the updates to your proposal. I know we’ve already reached out to clarify some questions that the committee had, but I wanted to post a formal response to that effect.

The committee wanted to potentially correct a misunderstanding regarding one of our responses, and clarify other elements of your proposal. Once you discuss this with our team, we’ll post a summary of that discussion here as well as next steps regarding this grant.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee