4. Extremely slow upload?

Extremely slow upload?

  • W

    I have a file that I started uploading yesterday and it is taking an extremely long time. My internet connection is quite fast, 50+ mb/s but this 110gb file is all but stalled. I checked it this morning and it was 23.98% uploaded. This evening I checked. . . it is now 24% uploaded. How does one go about debugging this? Do I need to purchase more coin? Can I "search for more peers or hosts? I feel a little frustrated at the moment. This is such a great technology/idea but if I can't use it. . . grr. . .

    0
  • Global Moderator

    Jesus 110 gb file? I can barely get a 2GB file to upload right. I think the problem you're having is reasonable for this stage in the project.

    Make sure you tag me @bryan if you need to me respond.
    Forum Mod. I cannot fix transactions. I can't help with Mac or Linux.
    alt text
    Unofficial Web Based Siacoin Wallet. No more syncing.

    0
  • W

    With the videos and documentation talking about terabytes of data. . . 110 gb did not seem like too much to ask. I just wish I understood the technology. Am I simultaneously uploading to all of the hosts I have contracts with or did it just pick one and I'm at the mercy of their download speed?

    0
  • R

    @Wegg
    That looks bad, less than 1% (~1GB) per day.
    I'm not sure you can do much to speed up. But you could try at least to see if that is your "wallet" slowing the upload or the network.
    First, look at your RAM usage. Sia gets terribly slow if not enough RAM available and it starts swapping.
    I think there was a possibility to speed up a bit by restarting Sia (or computer).

    I'm not an authoritative person (so I'm not sure if this is true), but technically the file is split into many small chunks of data, then uploaded simultaneously to a bunch of hosts.
    When you rent storage, you make ~50 contracts (to ~50 hosts), thus the initial "big fee". Then your data chunks are distributed between all hosts you have contracts with. Remember that you upload redundantly. I think, every chunk is being uploaded to 3 different hosts, so technically you are uploading not 110GB but at least 330GB. "At least" because I don't think the data is getting smaller as it is encrypted before uploading...

    Which OS is your Sia running on?
    You could look at your netstat output (look at connections to port 9982). On linux you can see a "Send-Q" column.
    There should be something (performance monitor?) to see the send queue or output queue for Windows too, but I don't know what tool gives you that information.
    So, if the "send queue" looks relatively big for some hosts and near zero for other, that means some hosts are slowing down your upload. If you see big numbers on all connections, it means your internet connection gets saturated and you can't upload faster.

    0
  • W

    I have 32 gigs of Ram in my system and currently I am using 4.76 gig so. . . I don't think it is a ram thing.

    Wouldn't the first host that receive a chunk share it with the others to remove bottlenecks in the pipeline? Would be weird to have the system build redundancy only from the original source. . .

    I'm on the latest Ubuntu.

    The Send-Q looks like. . . this.

    tcp6 0 0 :::9982 :::* LISTEN 20449/siad

    I have no idea what that tells you.

    0
  • R

    @Wegg
    That doesn't tell anything indeed...
    I'm not on linux, so you may have to use other flags/options, but my output looks like:

    netstat -n

    TCP: IPv4
    Local Address Remote Address Swind Send-Q Rwind Recv-Q State

    192.168.1.230.9981 63.143.31.231.50091 64768 0 1049740 0 ESTABLISHED
    192.168.1.230.42470 188.242.52.10.9986 185856 0 1049800 0 ESTABLISHED
    192.168.1.230.9981 123.119.75.129.41430 184832 11 1049444 0 ESTABLISHED
    192.168.1.230.9981 73.32.177.107.61498 805120 29861 1049740 0 ESTABLISHED
    192.168.1.230.9981 98.213.126.243.59533 261888 0 1049740 0 ESTABLISHED
    192.168.1.230.9981 79.8.124.99.58866 66680 11 1049796 0 ESTABLISHED
    ...

    From what I conclude, my node has some active connections, and for instance 73.32.177.107 has a relatively big send queue...
    But I am hosting (not renting). You should have more connections with send queues when uploading. If there aren't, restart your siad and look if that changes.

    Wouldn't the first host that receive a chunk share it with the others to remove bottlenecks in the pipeline? Would be weird to have the system build redundancy only from the original source. . .

    That was my first thought too. But as I understand, currently only the renter is uploading anything. Hosting nodes don't know where their redundant copies are. I think because one host should not know anything about renters other contracts. Privacy, you know, so it's harder to track the renter and his file locations.
    Maybe later some sort of "relaying" could be added, but that makes things more complicated. For instance, the renter pays the upload fee for every redundant copy. And it would create a vulnerability, if a "upload paid by someone else" were allowed.

    0
  • W

    I appear to have completely stalled. To test this I tried to upload some photos last night and now 12 hours later each of the 1mb files is only 3.33% uploaded. The large 110gb file is only 26% uploaded. I have somehow log-jammed my network of people I have contracts with. It's like I'm trying to shove hundreds of gigs of data through a 56k modem at this point. I don't know if it is too many files being uploaded at once, a setting I have wrong or a limitation to this technology but. . . I'm going to have to find another solution. :-(

    0
Log in to reply