Hacker News

I like the log message "BadBadBadChunkRecord". I wonder what BadChunkRecord and BadBadChunkRecord are? Is there a VeryBadBadBadChunkRecord?

I've been trying to replace all my various backups (Time Machine, Backblaze, CCC) to use a single tool (Arq - https://www.arqbackup.com/)

The Backblaze client is the next to go. To be honest, I haven't had too many issues with it, but the restore interface in particular is pretty poor and slow.

I'm speculating here based on my experience working on a storage fabric product in the past.

First - it's sensible to have limits to the size of the block list. These systems usually have a metadata layer and a block layer. In order to ensure a latency SLO, some reasonable limits to the number of blocks need to be imposed. This also prevents creating files with many blocks of small sizes. More below.

Second - Usually blocks are designed to match some physical characteristics of the storage device, but 10 MB doesn't match anything. 8MB might be a block size for an SSD or a HDD, or 72MB=9*8MB might be a block size with 9:16 erasure coding, which they likely have for backup use-cases. That being said, it's likely the block size is not fixed to 10 MB and could be increased (or decreased). Whether or not the client program that does the upload is aware of that, is a different problem.

This seems at the same time reasonable (no files larger than 1TB sounds fair, I don’t think anyone reads ‘no file size restrictions’ and imagines a one terabyte file), and not, because support should realize that that might have something to do with the issue and get someone technical involved.

Yev here from Backblaze -> This is something that our engineering team is taking a look at. We've been able to successfully upload files over 1TB internally, so we're trying to chase down what's triggering those error messages. From what I can gather the client should certainly upload files over 1TB, but something wonky is going on this case. We're looking at it though!

The Backblaze computer backup app is kind of a mess in general. The only reason I still use it is because of the unbeatable price, there are many better alternatives if cloud storage prices aren't an issue for you.

I've had many issues with it, and poking around at its logs and file structures makes it pretty obvious that it's very bad code. There's numerous different inconsistent naming conventions and log formats.

I once tried to edit the custom exclusions config file (bzexcluderules) and gave up shortly after because the format of the file is both completely nonsensical and largely undocumented - whoever implemented didn't know how to do proper wildcard matching and was likely unaware of the existence of regular expressions, so they came up with an unholy combination of properties such as ruleIsOptional="t" (no idea what this means), skipFirstCharThenStartsWith=":\Users\" (no comment on this one), contains_1 and contains_2 (why not contains_3? sky's the limit!), endsWith="\ntuser.dat" and hasFileExtension="dat" (because for some reason checking the extension with endsWith isn't enough?), etc.

The log suggests it doesn’t immediately stop upon noticing the file is over 1 TB—the second entry comes two minutes later when it’s trying to upload the 100,001st block. This makes me wonder whether 100,000 blocks is an intentional limit on file size or whether it’s some sort of internal threshold that isn’t meant to prevent uploading files of that size but is buggy. Perhaps files bigger than 1 TB have slightly different behavior by design and that behavior was broken?

Since 2 minutes isn’t long enough to upload 1 TB, it’s either looking at the blocks out of order and skipping ahead to block 100,001 for some reason or noticing that’s the first block that hasn’t been uploaded yet.

Another comment suggests 10 MB isn’t a fixed limit. In that case, I wonder if 100,000 blocks is the limit and the intention of the designers is that users with such large files would increase their block size. If so, that should be documented of course, and a scenario. Although it’s still a bit strange that it would apparently try to upload the 100,001st block and not immediately warn the user that the file size is incompatible with their block size.

I'm going to ask the wrong question here, but... What files are >1TB? Raw video footage?

The question is, where that limit comes from.. It sounds weird. 40bit file size record? like why?

I recently was fixing my own tool here and had to shufle some fields. I settled up at 48bit file sizes, so 256TB for single file. Should be enough for everyone ;)

[dead]

Honestly? Be a good customer. Help them help you and help yourself in the meanwhile

Having a 1TB file sucks in way more ways than just "your backup provider doesn't support it". Believe me.