3. Corrupt Host Folder paths

Corrupt Host Folder paths

  • B

    In my setup, I have 6 1TB volumes available for SIA hosting.They are formatted as NTFS, and initially I added each root directory directory to SIA:
    M:
    N:
    P:
    Q:
    R:
    S:
    And all seemed to be good - Sia was making files directly on the root of each drive.

    One day, after restarting Sia-UI (the Files view had turned completely black and I could not get it back), all my folders started displaying like this:
    P:
    P:
    P:
    S:
    S:
    S:
    And in the past 48 hours since that happened, I have gotten a few new contracts, but no uploads at all, whereas before there was a few contracts that had been adding files at about 3Mb/s.

    I added additional storage and mounted them as:
    C:\Sia\Sia1
    C:\Sia\Sia2
    C:\Sia\Sia3
    C:\Sia\Sia4
    C:\Sia\Sia5
    C:\Sia\Sia6
    and added them to Sia as:
    C:\Sia\Sia1\Storage
    C:\Sia\Sia2\Storage
    C:\Sia\Sia3\Storage
    C:\Sia\Sia4\Storage
    C:\Sia\Sia5\Storage
    C:\Sia\Sia6\Storage\

    I am now trying to remove the corrupted folders. I was able to remove one of the "P:" and one of the "S:", and there are no files left on those two drives. However, if I try to remove the other two P and S entries, I get the error:
    "Could not remove folder: could not successfully offload specified number of sectors from storage folder"
    And to be clear, this is not an issue with insufficient storage, but rather because Sia can't find the original files anymore.

    The symlinks in "AppData\Roaming\Sia-UI\sia\host\storagemanager" are all still correct, and M, N, Q, and R still have all the files from Sia contained in them.

    I am looking for instructions on how to either fix the folder info for the remaining 4 corrupt folders, or manually move and relink the files onto my new storage mount locations.

    0
  • admins

    Hmm, I'm really not sure what is going on here, and it sounds to me like a problem with NTFS, as strange as that is. I don't think anything that Sia is doing should be able to rename / remap your drives.

    I am not sure what to do here, I'll have to think about it more.

    0
  • B

    I agree it is a bizarre problem. As I said, the symlinks are still pointing to the right locations. It seems to me that path sia is remembering is not derived from the symlink, that it's storing the path it expects in its database, and the stored value and the actual location of the symlink are out of sync. If this is true, then there is definitely a bug somewhere that resulted in the database values being changed.

    0
  • B

    I have just looked at Sia and my folder names/paths have changed again! To be clear, this is what I saw ~12 hours ago, the last time I looked at Sia:

    Storage Folders:
    Used Capacity % Used Path
    3.813 GB 1.0230 TB 0.37 P:
    3.813 GB 1.0230 TB 0.37 P:
    3.813 GB 1.0230 TB 0.37 S:
    3.813 GB 1.0230 TB 0.37 S:
    3.813 GB 1.0000 TB 0.38 C:\Sia\Sia1\Storage
    3.515 GB 1.0000 TB 0.35 C:\Sia\Sia2\Storage
    3.515 GB 1.0000 TB 0.35 C:\Sia\Sia3\Storage
    3.515 GB 1.0000 TB 0.35 C:\Sia\Sia4\Storage
    3.515 GB 1.0000 TB 0.35 C:\Sia\Sia5\Storage
    3.515 GB 1.0000 TB 0.35 C:\Sia\Sia6\Storage

    (note that the forum strips all trailing slashes from the paths when I input them)

    And in the last 12 hours, I left Sia-UI open (the only open window on the machine) and did not touch the machine other than to lock it. Now it shows:

    Storage Folders:
    Used Capacity % Used Path
    4.077 GB 1.0000 TB 0.41 C:\Sia\Sia1\Storage
    4.060 GB 1.0000 TB 0.41 C:\Sia\Sia2\Storage
    4.060 GB 1.0000 TB 0.41 C:\Sia\Sia3\Storage
    4.060 GB 1.0000 TB 0.41 C:\Sia\Sia4\Storage
    4.060 GB 1.0000 TB 0.41 C:\Sia\Sia5\Storage
    4.056 GB 1.0000 TB 0.41 C:\Sia\Sia6\Storage
    4.056 GB 1.0000 TB 0.41 C:\Sia\Sia6\Storage
    4.056 GB 1.0000 TB 0.41 C:\Sia\Sia6\Storage
    4.056 GB 1.0000 TB 0.41 C:\Sia\Sia6\Storage
    4.056 GB 1.0000 TB 0.41 C:\Sia\Sia6\Storage

    To put it in words, my extra P: and S: have turned into extra copies of "C:\Sia\Sia6\Storage" - and this happened with Sia-UI open, and the machine untouched.

    0
  • B

    I updated to 1.04 Sia-UI, and now if I try to remove one of these extra/corrupt folders, I get:
    "
    Error Removing Folder
    ESOCKETTIMEDOUT
    "

    I'm sure I could resolve this by fixing the database, although I know that is a very risky operation to do manually. But this disconnect between where the symlinks connect to, and what the sia server thinks they connect to, is quite bothersome. It seems to me like there was some database update committed at one point whose "WHERE" conditions were overly broad. The only thing that's preventing me right now is that I don't know what type of database is being used.

    0
  • admins

    The timeout is not surprising to me, I think we forgot to increase the timeouts in the GUI for expensive host operations. Sorry about that. In the meantime, you can use the terminal plugin. The command host folder remove [foldername] should remove the folder just fine without causing timeouts.

    To fix the problem manually, you need to do two things (both while siad is NOT running):

    1. You need to update the json file at $SIADIR/host/storagemanager/storagemanager.json - it's got a list of storage folders with a "Path" and a "UID" value. Change the "Path" value to the correct location.
    2. Change the symlinks in the folder to the right place. The "UID" value is a base64 string, but it's corresponding symlink is in hex. So you'll need to covert it to figure out which symlink should be pointing to the right location. Based on the bugs you're reporting, the symlinks may actually already be pointing to the right place.

    This page can help you with the base64 <-> hex conversion: http://tomeko.net/online_tools/base64.php?lang=en

    0
  • B

    1. exited Sia-UI (Quit from taskbar icon), waited for all processes to end
    2. made a copy of %appdata%/host/storagemanager/storagemanager.json
    3. Checked that all the symlinks were correct
    4. Opened storagemanager.json in Notepad++
    5. Noted that the first entry was 100% correct, but the remaining entries all had the same UID, which also did not correspond to any of the symlinks
    6. converted the missing symlinks from hex to Base64 using http://tomeko.net/online_tools/hex_to_base64.php?lang=en , and replaced the erroneous entries with the proper UID and Path. Note that I did not change any of the other values, like the Size, SizeRemaining, etc
    7. saved the file and opened Sia-UI. Unlocked Wallet. Everything looks good in the UI, except the SizeRemaining of the folders being inaccurate.
    8. From the terminal (in SIA-UI) I ran host folder remove N:\ and could see (via Resource Monitor) that the files in N:\ were being copied to C:\Sia1\Storage\ - however after this was finished, all the files were still in N: (and a quick check showed that all the files appeared to be in C:\Sia1\Storage\ as well), and the terminal gave the error: Could not remove folder: could not successfully offload specified number of sectors from storage folder - this cannot actually be due to the lack of available space as there is over 930GB available in C:\Sia1\Storage\.
    9. Replicated this with the other 3 erroneous folders, same results (only tried one at a time of course)
    0
Log in to reply