Standard Grant: S5 Network 2024

Grants Approved
12

Hello @redsolver

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

13

Hi, I unfortunately won’t be able to submit this month’s progress report in time due to being busy with putting out some fires in the infrastructure of SkyFeed, the other project I’m working on. X (Twitter) was blocked in Brazil a few days ago, and as a result over a million people signed up on Bluesky and caused 10x to 20x higher activity than usual. SkyFeed is hosting over 40,000 custom feeds and serving about 500,000 users daily there. My servers couldn’t keep up, so I was busy trying to get them back online (and set up new ones) over the past few days (and nights).

I’ll try to finish up and submit my progress report and spec updates as soon as possible after the situation cools down a bit, hopefully later this week.

14

Hello @redsolver,

Thank you for the update on your situation, it seems like a good problem to be having at the very least!

We look forward to your progress report sometime soon.

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

15

S5 Progress Report (August)

What progress was made on your grant this month?

Wrote the S5 v1 specifications. They might however still see some small changes based on feedback from other developers and S5 users.

  • Blobs

  • Registry

  • Streams

  • Accounts

  • Upload

  • API Interface

  • Identity

  • Key Derivation

  • Encryption

  • File System (FS5) (still missing key management section and encryption fields, will be updated together with the Dart implementation)

All specs can be read directly on GitHub: docs/src/spec at main · s5-dev/docs · GitHub

In August, I also implemented an API for secure end-to-end-encrypted messaging with MLS (Messaging Layer Security, RFC 9420) over S5 Streams, for Vup Chat, which Covalent was working on: GitHub - vup-app/vup-chat: Chat app for bsky powered by S5

What will you be working on next?

  • Update the S5 Dart and Flutter libraries to fully match the S5 v1 specifications

  • Get feedback from other developers on the S5 specifications (including from other content-addressed network projects) and based on that make small changes to the specs if needed

  • Assist developers in adding the new S5 v1 Dart and Flutter libraries to their app, and eliminate remaining developer experience pain points in the process

16

S5 Progress Report (September)

What progress was made on your grant this month?

  • Published the S5 v1 specifications on web (hosted on S5 via DNSLink): Blobs - S5 Network Docs
  • Updated some parts of the Dart library to match the S5 v1 specs
    • S5APIProvider base class (API Interface)
    • CryptoImplementation base class (API Interface)
    • New BlobCID (Blobs)
    • util/derive_hash.dart (Key Derivation)

Links to repos worked on this month:

What will you be working on next?

  • Build the S5 TypeScript/JS library based on the v1 specs, to enable developers building apps for the most popular development ecosystem (the web and native JS) to fully utilize all S5 features (also needed for the Vup Web app in November of this grant)
  • In parallel to building the TS lib, finish migrating the Dart lib to v1
  • Get feedback from other developers on the S5 specifications (including from other content-addressed network projects) and based on that make small changes to the specs if needed
  • Assist developers in adding the new S5 v1 Dart and Flutter libraries to their app, and eliminate remaining developer experience pain points in the process
2 Likes
17

S5 Progress Report (October)

What progress was made on your grant this month?

Started building the S5 TypeScript/JS library based on the v1 specs. Adapted some of the APIs to better match the web platform, like using Blob objects for large file uploads. Progress has been slower than expected, nevertheless the lib is a requirement for the Vup Web app, so it will be finished in a few weeks at most.

Links to repos worked on this month:

What will you be working on next?

  • Hire the professional designer for the Vup V1 UI, so it can already be worked on in parallel while I finish the parts of the S5 TS/JS library required for Vup Web
  • After that’s done, I’ll start working on Vup Web with the features S5 identity creation and recovery, storage service account management and using the decentralized file system (FS5)
  • Get developers to use the S5 v1 libs in their projects and collect feedback
18

Hello @redsolver

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

19

S5 Progress Report (November)

What progress was made on your grant this month?

  • Found and hired a professional designer for the new Vup web app and UI design. He’s currently busy working out a consistent design language and will soon start with implementing different components for the app directly in HTML+CSS using https://storybook.js.org/ Storybook makes it easy to collaborate because after the components are ready, I can directly use them in the web app together with the S5 TypeScript library
  • Got a new domain for the S5 project! s5.pro for S5 Protocol. The docs are already there: https://docs.s5.pro/ and a landing page on the main domain will be added soon. It replaces the previous sfive.net domain
  • Created a new blobsum command line utility for easily and efficiently calculating S5 Blob Identifiers (CIDs) of files on your system. It’s a fork of b3sum

Links to repos worked on this month:

What will you be working on next?

  • Continue working together with the designer on the new UI and components
  • Continue working on the S5 TS/JS library
  • After the first components are ready, start implementing all the different S5 file system features and the overall Vup web app (S5 identity creation and recovery, storage service accounts, …)
1 Like
20

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

21

S5 Progress Report (December)

What progress was made on your grant this month?

  • The designer has been busy working on different components, concepts/views and utilities for the Vup Web UI, while I gave some feedback. Vup Web is slightly behind schedule now, but should finally get a nice MVP ready this month with all basic file management features working

  • Bluesky now has over 26M users which also resulted in significantly more load on my feed infrastructure and kept me quite busy putting out fires. So this month I built a new powerful open-source indexer in Rust to keep up with the network growth and have less issues in the future. The new indexer will also play a crucial role in bridging the AT Protocol and Bluesky data and media over to the S5 protocol as part of the archiver milestone planned later in 2025 for this grant

  • Attended the 38th Chaos Communication Congress and had some interesting discussions there around S5, the AT Protocol and other projects building decentralized protocols and networks

Links to repos worked on this month:

What will you be working on next?

  • Use the first finished design components to build the Vup Web UI with all basic features implemented and working

  • Continue working on the S5 TS/JS library and implementing all v1 specs

  • Design a landing page for the S5 protocol which gives a quick overview over which features are possible with S5 v1 and onboards developers to use it in their project (to be deployed on https://s5.pro)

22

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

23

Timeline Correction

The timeline listed in the second post of this thread is off by one month due to the grant taking a few weeks to be approved and onboarded, which resulted in the official timeline in the grant agreement being shifted by one month. Unfortunately I can’t edit the timeline in this thread anymore due to it being the second post (instead of the first), so instead I’m just posting the correct timeline here for reference:

Milestones and Timeline

Finish the S5 v1 specifications (2024-09-02)

Design, define and write v1 specifications for the different S5 data structures and features:

  • Blobs, already done: Blobs - S5 Network Docs
  • Registry
  • Encryption
  • File System (including metadata formats)
  • Identity System
  • Streams

In July, I will attend IPFS Camp 2024 (https://2024.ipfs.camp/) for exchanging ideas with the wider IPFS ecosystem and getting feedback on the final S5 specs, to make them the best they can be.

Adapt the S5 Dart and Flutter libraries to the new v1 specifications (2024-10-02)

In August and likely already earlier while working on the specs, I will adapt the existing Dart/Flutter libraries for S5 to use the new specifications. This will be very important for validating if everything works as expected and adding support for S5 to different apps like GitHub - LinwoodDev/Butterfly: 🎨 Powerful, minimalistic, cross-platform, opensource note-taking app (in collaboration with the developers), to see if the APIs are well-designed and the data structures efficient.

TypeScript Web Library (2024-11-02)

Build a TypeScript/JS library based on spec v1 in collaboration with @pcfreak to ensure the spec is sound and make all S5 features available for the most popular development ecosystem (the web and native JS). The libraries will use Rust via WASM for performance-critical things like hashing, signing and encryption (just like in the Dart libraries).

The TypeScript library will be designed to “just work” (like the s5 dart package), so devs can easily add it to their app to get features like a decentralized end-to-end-encrypted file system (compatible with Vup) and real-time communication between multiple users.

Build Vup Web (2025-01-02)

Using the new TypeScript libraries, build a lightweight web version of the Vup Cloud Storage app (fully compatible with the native version of course). This will also be the first implementation of the new user-friendly Vup UI designed by a professional designer.

Features

  • View and download shared directories and files (both public and encrypted)
  • Sign in with a S5 Identity
  • Upload files and manage the encrypted file system
  • Media Thumbnails

RHP4 support for S5 web apps (when available)

From https://sia.tech/roadmap:

RHP4 will also enable storage consumers to download and upload data directly in the browser without installing additional software.

When this feature lands in Sia, I will add it to the web-based and native S5 libraries (like TypeScript and Dart), to support more efficient direct downloads from the Sia Network.

Improvements to the S5 Node needed for Vup v1 (2025-01-02)

  • Improved Pinning API (for example background mass-pinning)
  • More powerful Account and Admin APIs, to make managing larger nodes with many users easier
  • Performance and scalability improvements

Get Vup to version v1 (native) (2025-03-02)

  • Update the native Vup App to match the new professional UI design of the web app
  • Migrate all data structures to the S5 specification v1 and the new libraries
  • Advanced two-way sync with file deletion and multi-device support
  • Advanced file sharing permission systems (for example read-only/write per file)
  • Add built-in audio player
  • Release on Linux, Windows, macOS, Android and iOS

Vup Collaborative Productivity apps (2025-04-02)

Webxdc (https://webxdc.org/) is a standard to bring privacy-friendly web apps to messenger chats in a simple file format containing HTML5, CSS, JavaScript and other asset files. But apart from that use case, this is also the ideal format to provide many different collaborative web apps for Vup (Web and Native) and other S5-powered apps. S5 can both be used as the transport layer (instead of a chat room) for real-time collaboration like drawing on a whiteboard or editing text, and it can also store files you edited permanently in the S5 file system. There are already many Webxdc apps made for https://delta.chat and other messengers and most of them should just work with Vup+S5.

From the Webxdc specification:

All authentication, identity management, social discovery and message transport is outsourced to the host messenger which runs a webxdc app container file and relays application update messages between app users, letting each app inherit offline-first and end-to-end encryption capabilities implemented by the hosting messenger.

In this case, the Vup app is the “hosting messenger”!

Some use cases possible with this:

  • Select a text file in your Vup App and share it for end-to-end-encrypted collaborative editing with other users by simply sharing a link with them. They don’t need to install anything due to the app being web-based and just need to open the link in their browser and start editing with you! In regular intervals, your local Vup App will persist the changes to your encrypted file system.
  • Same with task list editing, whiteboard apps, calendars, board games and more! Any productivity app that runs on the web and could use a file system.

The biggest advantage of this approach over just implementing collaborative productivity apps/features myself in Vup is that it saves me a ton of work and also makes the whole platform easily extensible by third-party developers with new and exciting apps.

Vup P2P (2025-05-02)

Add Iroh support (https://iroh.computer/) to Vup for keeping true p2p connectivity between multiple devices and users. (only trusted devices by default)

Use Cases

  • You want to stream a video file and one of your other devices already has it cached locally. Instead of needing to reach out to the Internet, your local Vup app will just connect directly to your other device and stream the file from there
  • If enabled, you can connect directly with other users you are editing a text file (or something else, see Webxdc above) with for improved latency and privacy (no server involved)

Archiver (2025-08-02)

Archiving makes sense for personal archive use cases, for preserving history and satisfying your data hoarder needs. It’s also a fantastic way to onboard massive amounts of useful data to the network.

The existing S5 Archiver will be extended to support all the following data sources natively:

  • Websites
  • YouTube Videos
  • AT Protocol data repositories and blobs (for example the entirety of Bluesky data)
  • GIT repositories
  • Twitch VODs and media.ccc.de

After that’s done, I will deploy a large Sia storage cluster with multiple S5 Nodes. This archival infrastructure will then be used to archive massive amounts of Creative-Commons-licensed YouTube Videos, different websites and CommonCrawl archives. It will serve as both a scalability test of the software and network, and as a useful archive.

All parts of the archive with a public license that permits it (like Creative Commons), will be publicly browsable using a S5-powered web app.

1 Like
24

S5 Progress Report (January)

What progress was made on your grant this month?

Links to repos worked on this month:

What will you be working on next?

  • Finish implementing full identity and account support in the S5 TypeScript library (currently ~70% done)

  • Continue working on Vup Web after the next components from the designer are available

  • Design a landing page for the S5 protocol which gives a quick overview over which features are possible with S5 v1 and onboards developers to use it in their project (to be deployed on https://s5.pro)

25

S5 Progress Report (February)

What progress was made on your grant this month?

  • Implemented pretty much all missing parts in the S5 TypeScript library
    • full identity implementation, with seed phrase generation, verification and recovery, key derivation
    • added identity-backed api, including upload, portal account registration and login
    • file system: added full identity support, plaintext upload, directory and file creation, home file system
    • registry write support
    • hidden db implementation for legacy compat with Vup portal accounts
    • added example script which mirrors all apps from the webxdc app store to a new directory in the s5 file system
  • My designer made solid progress on the Vup Web design components, quite a lot of new ones were added to the repository
  • Started sketching some ideas on how the S5 Protocol landing page could look like and speaking with @Julian who will work on it (I’ve been already working together with him in 2020 and 2021 for different Skynet apps (like for example the https://vup.app/ website), so really looking forward to doing it again).

Links to repos worked on this month:

What will you be working on next?

  • Add tests to the S5 TypeScript library in order to ensure everything is implemented correctly
  • Get the first proper release of Vup Web v1 out. The TypeScript library now already includes everything needed so that shouldn’t cause any delays, and all of the different components and designs are also finally coming together
  • In Vup Web, implement initial support for browsing and using webxdc apps (this is the first step for collaborative productivity apps)
  • Initiate work on the S5 landing page
  • Get the existing native Flutter-based Vup to version 1 and add features like advanced two-way sync (this milestone is now delayed due to Vup Web taking longer than expected)
1 Like
26

Due to poor time management on my part, I am unfortunately unable to publish my progress report for March on time today. I would like to apologize for this and will post it as soon as possible (no later than April 5th)

27

Hi @redsolver, thanks for the update. What’s the current status of your report?

28

S5 Progress Report (March)

What progress was made on your grant this month?

Links to repos worked on this month:

What will you be working on next?

  • Implement sharing in Vup Web, polish the user experience and add more features

  • In Vup Web, implement initial support for browsing and using webxdc apps (this milestone is now delayed)

  • Publish the new S5 landing page

  • Get the existing native Flutter-based Vup to version 1 and add features like p2p connectivity and advanced two-way sync

29

Hello,

Thank you for your progress report!

Regards,
Kino on behalf of the Sia Foundation and Grants Committee

30

For reference, here are the current estimated completion dates for my milestones:

  • Collaborative Productivity apps (delayed to 2025-05-02)
  • Get Vup (Flutter) to version 1 (delayed to 2025-06-02)
  • Vup P2P (delayed to 2025-06-02)
  • Archiver (2025-08-02, still as planned)

This is not my progress report, the April report will be posted on 2025-05-02 as usual

31

S5 Progress Report (April)

What progress was made on your grant this month?

  • Published an initial version of the new S5 landing page on https://s5.pro

  • I attended the first ever AT Proto / Bluesky conference in Europe and gave a talk (redsolver at AHOY! European Social Web Day) about SkyFeed and its early beginnings in the Sia Skynet ecosystem. I also presented and discussed a new content-addressed architecture for the SkyFeed Indexer and AppView which can scale beyond the current (already massive) Bluesky network size by storing time-sharded snapshots of records on S5-compatible storage networks like Sia.

Links to repos worked on this month:

What will you be working on next?