Grant Proposal: SiaShare

Grants Complete
1

Introduction

Project Name: SiaShare

Name of the organization or individual submitting the proposal: mjmay08

Purpose:
This proposal is based on the idea from @parox for an end-to-end encrypted file sharing site that is very simple to set up and use: E2EE file sharing app

The plan is to create something very similar to https://wormhole.app/ but with data stored on Sia using renterd. A user can visit the site, drop some files on the page, and then get a link to share those files. Files will expire after a certain time period or a specific number of downloads.

The goal is to create a lightweight web service that anyone can run alongside renterd with minimal setup and configuration. The client application will use Uppy (https://uppy.io/) for the file upload experience, wormhole-crypto (GitHub - SocketDev/wormhole-crypto: Streaming encryption for Wormhole.app, based on Encrypted Content-Encoding for HTTP (RFC 8188)) for the file and metadata encryption, and the Uppy Tus integration to provide resumable file upload to the backend. The backend will be a simple webservice with a sqlite database and files will be temporarily cached on disk while they are being uploaded to the Sia network using renterd.

Grant Specifics

Project Timeline
In total I expect this proposal to only take a few months at most. The scope is limited but as I am only working on this part time I don’t want to promise anything shorter than that.
I have been working on a proof of concept already so that I could be certain this proposal would be a success. I hope to have a basic working version for the community to start testing a few weeks after the grant is approved.
After that I will work on customization(expiration, file size/count limitations, etc), UI improvements, documentation, and testing the scale of the service.

In addition there are many other enhancements that could be made as time allows such as:

  • Packing files that are uploaded together into one to minimize the impact of Sia’s minimum sector size
  • File preview in the browser without downloading
  • Enabling Uppy integrations for services like Dropbox, Google Drive, and OneDrive so that user’s can import and share a file directly from one of those services.

Budget
I am requesting $5000 for the duration of the proposal to cover the part-time salary of myself as well as infrastructure costs to host an instance of this service for the next few months for the community to use while it is in development.

Potential risks that will affect the outcome of the project:
As mentioned above I have already been working on this project for a little while so that I could be confident in my ability to deliver it. The only risk I can see right now is that since I am building on top of renterd which is still beta, any issues with that integration could push back the timeline (however so far I have had no issues with the integration).

Development Information

Open-Source Commitment
The project source code shall be maintained in a public repository located at: GitHub - mjmay08/SiaShare
All code will be MIT licensed.

Progress reports
Yes, I agree to submit monthly progress reports. In addition I plan to provide more freqent updates on the Sia discord as this proposal is only expected to take a few months at most.

Contact info

Email: Will provide in DM on approval
Any other preferred contact methods: Sia Discord @mjmay08

Conclusion

I’m excited to contribute to the Sia ecosystem and hope that the grant committee will approve my proposal. I welcome any questions on details I may have missed above.

1 Like
2

Hi @mjmay08, very excited to see this!

I it would be really cool if the tool could replicate the wormhole feature where the receiver does not need to wait for the upload to finish to start downloading.

I forgot to add it to the Grant Idea but this is also very similar to @redsolver’s SkySend which also had an extremely simple user-friendly UX: GitHub - redsolver/skysend: Decentralized end-to-end encrypted file-sharing - that really made it one of the most useful/used Skynet apps. I think you are familiar with it, but definitely worth checking it out for further inspiration. Maybe @redsolver will port that to work with S5 and then we will have 2 great easy to use file sharing options on the Sia stack :grin:.

Other notes:

  • Also its pretty cool how wormhole supports direct p2p transfers in certain situations.
  • wormhole FAQ says they might open source the full app, if the code would be a useful reference for the streaming bits or integration with the wormhole-crypto library then it might be worth reaching out and asking about it.

:rocket:

3

This feels somewhat similar to ideas I’ve had around BitTorrent and general filesharing of data on sia by knowing the sector and host data + encryption key.

I would recommend that you try to stick to the data formats S5 is creating which are based on blake3 for interoperability. With that, this could probably be further integrated into the ecosystem in the future.

Kudos!

4

If I understand correctly I think both of these asks are solved by using WebTorrent? For sure that is how they do the p2p transfer, but that must also be how they accomplish the “instant file streaming”? I haven’t used WebTorrent before but that looks like something I could attempt as well.

Cryptography is not a strong suit of mine and the only reason I considered submitting this proposal is that this library (GitHub - SocketDev/wormhole-crypto: Streaming encryption for Wormhole.app, based on Encrypted Content-Encoding for HTTP (RFC 8188)) makes that part of the application pretty straightforward. I’d rather not risk doing the encryption incorrectly by implementing it myself using blake3. Maybe someone would be willing to make that change after my initial proposal is complete?

5

The blake3 stuff already exists. However, blake3 is not an encryption library, it’s a hashing library (md5, sha1, sha256, etc). It is being used to create content-based file IDs (CID) like what IPFS does with S5 and Lume.

6

Thanks for the proposal! I just wanted to reply quickly to let you know that we received this proposal a bit too close to our meeting tomorrow to take a proper look at it. We will review it in full at our next meeting on May 30th!

1 Like
7

Hello Mjmay08,

The committee has reviewed and approved this proposal, congratulations! We will need a better estimate on the timeframe involved when possible, as this will be required for payment.

The committee also noted that running a public instance of a file-sharing service can produce a whole set of burdens and responsibilities regarding content monitoring, and recommends that you consider limiting the focus of the proposal to development only.

Please reach out on the Sia Foundation community Discord server or provide an email address to get in touch with us so we can provide you with the documents to begin your exciting project!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

8

I’m excited to hear that you’ve approved my proposal.
Here is a better estimate of the timeframe for the project. I expect it to take 3 months with the following milestones:

June
-Working MVP where the user can select files to upload, they are encrypted in the browser, sent to the server, uploaded to Sia, and the user is given a link to share the files. The link can then be used to download the files and they will be decrypted in the browser.

July
-“Instant file streaming” using WebTorrent
-Implement file expiration and cleanup of old files

August
-Improve the UI
-Document everything so that others can set it up easily
-Make everything configurable (expiration times, file size/count limits, etc)
-Address any community feedback based on previous milestones

I agree that I don’t want to run a public instance of a file-sharing service. I was thinking of running a temporary instance for the community to test out, but maybe I will exclude that for now.

9

Here is my first monthly progress report for SiaShare (I will be on vacation for the next week and a half so sending a few days early).

Reminder that the project source is available at GitHub - mjmay08/SiaShare.

As was my goal for June, I have completed an MVP version of the application. A user can select files to upload in their browser where they are encrypted and then uploaded via TUS to the server. Then the files are uploaded to Sia and the user is given a link which can then be used to download the files which are then decrypted in the browser.

I was focused on getting the functionality working so the UI is very basic right now. I will also try to update the README soon in case anyone wants to play around with it.

What’s Next?

-Document how to run SiaShare
-Investigate using WebTorrent to enable “instant file streaming”
-Improve the UI

10

July Progress Report
This month was completely focused on adding instant file streaming functionality. I didn’t originally include this in my proposal but added it at the request of @parox. I probably should have done a little more research before agreeing to add it to my proposal as it is technology that I was unfamiliar with. It has taken me longer than I expected to get this functionality working, but I think I have it mostly complete.
I haven’t pushed the code yet as it required a pretty big refactor and the code is a bit of a mess right now. I should have it cleaned up soon and will push a new version then.
With the bad news out of the way, here are the things I accomplished in July:
-Integrated WebTorrent with SiaShare and successfully enabled instant file streaming allowing the recipient to download the file before it is even uploaded to the server/Sia.
-Integrated a bittorrent-tracker server with the backend allowing a swarm of peers to discover each other without relying on a third-party tracker.

Currently I am working on the last piece of the puzzle which is adding a web seed to the torrent so that if the original uploader stops seeding the file (for example closing their browser after the upload is complete) then it will gracefully fallback to downloading the file from the server/Sia. This work is probably 75% complete. After this is complete I will focus again on the other tasks I had been hoping to accomplish in July.

3 Likes
11

Hey thats awesome that you got it working as it is certainly not a trivial feature, very cool!

12

Hello @mjmay08,

Thank you for your July progress report! It looks great.

The Committee has voted to create a progress report template which should make it easier for both you, the Grantee, to provide monthly updates as well as to enable the Committee to quickly review progress without spending too much time on minute details.

For August’s progress report (which should be provided in September), please follow the template outlined in this post: About the Grants category - #8

Thank you for your help and cooperation.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

13

Progress Report (August)

Progress Report Form

What progress was made on your grant this month?

Please summarize your progress in 3-5 sentences or bullet points:

  • Finished WebTorrent integration. Downloads should now seamlessly fetch from both WebRTC peers as well as the server/Sia network.

  • Complete UI overhaul. The existing very basic UI has been replaced with a much better looking and more functional UI.

  • SSL support so that SiaShare can be run on something other than localhost. Includes automatic self-signed cert generation for development.

  • Added instructions on how to run SiaShare to the README. Also added a section explaining how SiaShare works.

Links to repos worked on this month:

What will you be working on this month?

Please summarize your development goals into a few sentences or bullet points:

This does conclude the original timeline from my grant proposal, however there is still some work remaining. As I mentioned in last month’s update, the instant file streaming work ended up taking much more time than expected.

At a high level the remaining work includes: file expiration and cleanup, file count/size enforcement, addressing some important TODOs in the code, and better error handling and code cleanup.

Now that SiaShare is in a more complete state I would love it if some people would test it out and provide feedback. I would also like to know if the grant committee is open to extending this grant for a couple more months or if I will just have to continue working on SiaShare in my free time.

14

Unfortunately I never received any response to my progress report last month so I’m a little unsure of the status of my grant. I will provide an update for September anyways.

Progress Report Form

What progress was made on your grant this month?

Please summarize your progress in 3-5 sentences or bullet points:

Worked on all of the feedback I received from @parox including:

  • Adding production build scripts, a Dockerfile, and added to documentation for running in production.

  • Added github action to publish a Docker image for each master commit making it easier for others to quickly run the latest version on SiaShare.

  • Made changes to allow running without SSL in case the user wants to run behind a reverse proxy.

Links to repos worked on this month:

What will you be working on this month?

Please summarize your development goals into a few sentences or bullet points:

  • Work on @parox’s suggestion for adding basic auth for uploads.

  • Implement file expiration/cleanup.

  • Address some TODOs in the code.

1 Like
15

Great job on SiaShare so far! I’ve been successfully using it as a private wormhole alternative for a while now.

16

I would love to see the grant extended for continued feature development and refinement. You should share it on Discord and try to get some feedback from the community, especially now that you have improved the setup process.

17

Thank you both for the positive feedback!
I was hoping the grant committee would respond to my question about extending the grant, but they’ve met 3 times since I asked that in my August progress report and I’ve gotten no response. Do I need to submit another proposal just to get the extension?

18

Sorry about that. Seems like we overlooked that question in our response summaries. If you’d be interested in writing a guest blog post about SiaShare, DM me on Discord.