How the file health loop actually functions

I asked this question and got a great answer from @Sevey, but David thought it would be a good idea to move this topic to here so it’s a tad more immortalized. So here’s the quesiton:

Could anyone knowledgeable fill me in how file health works in siad ?(I don’t feel like going through the codebase right now). I’d assume that the renter just keeps a list of hosts each file is on, and then the renter node spot checks which hosts are good by periodically downloading random files and checking the hash against the locally stored one. With this setup it’d be very easy to calculate file health on the fly by just doing (maxhosts/hostsWithFIle) for each file recursively. But I also wouldn’t be surprised if it was a bit more sophisticated than this

I’ve just been trying to think of a way to do file health checks without having to waste a bunch of bandwidth and compute for papyrus(obv manually checking each file would be a terrible idea).

Here is my response from Discord.

siafiles, referring to any .sia file which includes skyfiles, contains a map in the metadata that links host pubkeys to piece roots. This map is used to calculate the health of a file. See Health in the siafile package for the math, but simply put it checks if the host pubkey corresponds with a contract that is available. The renter then has a few background threads that enable siac renter to show the MinRedundancy for the renter for example.

First loop is the Health Loop. This is a loop that goes over the filesystem and makes sure that the metadata stored in the .siadir files is up to date. If a .siadir file has been updated in the last hour then it is considered up to date.

If the Health Loop finds a directory that is out of date it will update it by calling Bubble on that directory. Bubble calculates the metadata for the directory, updates the .siadir file on disk, and then calls Bubble on the parent directory until it reaches the root of the renter’s filesystem /renter/fs . This is why the .siadir files contain directory metadata and aggregate metadata. The aggregate metadata is the metadata that represents the subtree of the filesystem where the current directory would be the root. So things like worst health, most out of data metadata, total size, number of files, etc.

If the Bubble encounters a file that needs to be repaired it triggers the repair loop to start.

The reason the renter can use the siafile’s hostpubkey map to calculate the health of the siafile based on the it’s known contracts is because the contractor is ensuring the list of contracts is updated with which hosts are online and still good for renew via threadedContractMaintenance . So the renter doesn’t need to spend money doing test downloads to check the health of the file.

Okay, so how it works is effectively what I described with more details on how it works right?
Basically how it works is this:

  • Contract maintenance script downloads a random file from each contract on a prescribed timeframe to check if the contract is healthy.
  • Then each individual file is assumed to be healthy if all of the contracts it is contained within are healthy.

Do I understand correctly?

There is no downloading necessary.
The contractor has sufficient information from previous scans or attempted interactions with the host/contract to determine if the contract is still good.

managedMarkContractsUtility is the method that goes through all the checks. It is probably easier for you to read through that method and ask questions then for me to try and explain every check.

Contract maintenance occurs every block, since contracts are transactions it makes sense to have that synced up. The health loop is what runs on a time interval and is only concerned with the filesystem metadata on disk.

Your second point is correct.

1 Like