3. SIA upload size problem

SIA upload size problem

  • L

    Hello,
    I have some questions about the upload with SIA (V1.2.1).
    I currently run SIA deamon on GNU/Linux server (Fedora) and I use SIA storage to save my server backup.
    Currently my allowance is about 40KSC, I start to upload the most smaller backup (136MB for 68 files) and everything is done at this point.

    But I try to upload another backup of 47GB for 127 564 files (it huge step ^^) and the problems coming :/
    At this point everything seem to be normal (the command "siac renter uploads" and "siac renter list -v" return an normal result, and an network monitoring show an big network activites) but after 70 hours of upload its isn't complet (The upload server speed is about 1.5 MB/s).

    So after more research I found (according to the "siac renter uploads" command) that the total percentage of upload is about ~10% and the size printed in all contracts reach the sum of 394GB. By this way, that means that the final upload size is ~4TB ?!

    In this case I can't use SIA for this application :/ (because QOS on SIA will expand the upload time to 100 days)

    What is the reason of this huge size expansion ? (I read on the wiki that SIA use reed solomon erasure and pieces of chunk)
    It's solvable issue ?

    Thanks for your help.

    0
  • H

    Each file is padded to 40MB and uploaded 3 times in the network.
    So maybe you should consider tar or zipping them.
    Another idea would be to tar and split the tar in 40MB chunks and upload each one of them.
    Another idea if your files can be separated and you want to access them individually, would be to create groups of 40MB each:

    the below software can help via Wine:
    burn to the brim
    sizeme

    0
  • L

    Hello,
    Thanks for your help !
    My fear become reality :/
    So, I need to write my own backup software, because the backup system should be automatic and efficient (in terms of network bandwith and memory management).
    I probably write the program soon, and I think that it is good idea to share it on GitHub in the future.

    1
  • L

    Hello,
    I finally writen the backup program on GitHub, it an alpha but it should work (I have just made couple of small tests at the moment).
    So I would to share it, I past the link on this post : https://github.com/0xlouis/SIA-Chunk-Backup

    0
Log in to reply