Questions about planned 100 file limitation
-
I moved this discussion to another thread as it seemed we were moving slightly off topic.
@in-cred-u-lous I'd rather release something that acknowledges the actual limits of Sia than something that's going to have a user uploading >100 files and run into a bunch of unexpected problems. Obviously it's a fix until the scalability solutions are in place. If we could get the scalability stuff finished in 1 day we would release it but there's a lot of code that needs to be written and tested to make that happen. We ARE dealing with the problem head-on.
Always great to hear :-)
Further, it's not the network that's being crippled by all of the files, it's your local machine. siad starts to struggle at 100 files because the process that the uploader uses to scan and upload files is intolerant of large file volumes. A rogue uploader isn't the problem, it's your local state.
Ok, I misunderstood the problem in that case. I recall there being scalability issues with the number of concurrent uploads being rate limited due to what can fit inside blocks. But this is different.
It seems what you're saying here is that there are resource/memory issues with the renter that limit the number of files it can manage. But, could you be a bit more specific?
Are you planning a 100 file cap on the total number of files in the renter, or a cap on the number of concurrent uploads? Concurrent uploads is presumably much more resource intensive than managing the same number of 100% redundant files on the network. Or, am I wrong about that?
-
Glad this has it's own thread, was just about to make one.
I'm still trying to figure out exactly what's going on, but there are a few problems that Sia has run into. And they are related to the number of people using Sia and the number of files that each person is uploading to Sia.
Right now, we don't batch files into a single file contract. Right now, each file causes 24 new file contracts to be created, and then 24 new file contract revisions to be created. It gets more intense when I establish that each file contract has a precursor transaction that gets created so that the file contract can be made without any refund outputs. (Refund outputs would cause complications in the current wallet logic, and would severely inhibit the wallet's ability to daisy-chain transactions, which is a requirement for making a bunch of file contracts all at once). Therefore, each time you upload a file, 72 transactions get put on the blockchain. That's per file. This puts a lot of strain on the network, and also a lot of strain on your local client. Furthermore, the repair loop is inefficient, and get stuck frequently on slower hosts, meaning that it can take 100+ minutes for the renter to do a single cylce of the repair loop (which is responsible for both uploading files and for repairing files that have lost hosts/redundancy).
Putting a cap at 8 concurrent uploads in the renter would probably be sufficient. This will prevent the repair loop from choking, and it will keep the total number of file contracts going out to the network to a minimum. People who want to upload more files all at once are encouraged to zip things up until we've got file contract batching ready to go, which is definitely our highest priority right now (though must be balanced against our current fundraising and the 3 clients we're trying to sell Sia to).
Another problem that's really slowing down the network is that a lot of file contracts and file contract revisions are being rejected because the transaction pool is full. Because file contracts and file contract revisions don't add fees, they are actually getting stuck when the transaction pool hits 1/4 full, because that's the point when the transaction pool stops accepting free transactions.
So, what's our game plan:
- Start adding fees by default to all transactions.
- Start batching multiple files into each file contract. Which means changing the user behavior of the renter - all uploads will need to be pay-in-advance uploads. This has been unavoidable from the start, but we've been waiting to make the transition until the code is ready.
I'm not sure exactly how long all of this is going to take. I want to say that we can have it all done by the end of the week, but that's going to require luck and a lot of effort. Really we should have focused on some of these issues a while ago, but I feel like I say that continuously. I think the reality is that there's a ton of coding to do and we're slowly running into the next-biggest and next-biggest problems as the network continues to evolve. We've come a long way in the past 6 months, and we're going to keep going at high speed.
-
@Taek Thanks for clarifying this. Limitations are ok as long as they are communicated to the end-user. My renter has been syncing 100+ files simultaneously (mean size >200MG each) and I see now its clearly been a strain on my system. I've dialed down the queuing of files and will see how this works for the next few days.
What about total renter size? Repairing files presumably also involves creating new file contracts (as repairs using new hosts needs to be paid for). Should there be a cap also on the total number of files being repaired concurrently, or does this require much less resources (locally and on the network) than initial uploads?
