Hi all. I am copying an older WD 8tb drive to a new WD 8tb drive. An example of what happens consistemtly is that a folder that was 3gb actual with ~3gb size on disk (on the older drive) becomes 3gb actual with ~9gb size on disk (on the new drive). Therefore, I can only copy a fraction of the older drive to the new drive. Does anybody know how to eliminate that problem? Thank you.
Short answer: Cluster Size
There is a good (if very old) detailed explanation here: https://www.howtogeek.com/180369/why-is-there-a-big-difference-between-size-and-size-on-disk/
SuperUser contributor Bob has the answer for us:
I will be assuming that you are using the FAT/FAT32 file system here, since you mention this is an SD card. NTFS and exFAT behave similarly with regards to allocation units. Other file systems might be different, but they aren’t supported on Windows anyway.
If you have a lot of small files, this is certainly possible. Consider this:
50,000 files
32 KB cluster size (allocation units), which is the max for FAT32
Ok, now the minimum space taken is 50,000 * 32,000 = 1.6 GB (using SI prefixes, not binary, to simplify the maths). The space each file takes on the disk is always a multiple of the allocation unit size – and here we’re assuming each file is actually small enough to fit within a single unit, with some (wasted) space left over.
If each file averaged 2 KB, you’d get about 100 MB total – but you’re also wasting 15x that (30 KB per file) on average due to the allocation unit size.
In-Depth Explanation
Why does this happen? Well, the FAT32 file system needs to keep track of where each file is stored. If it were to keep a list of every single byte, the table (like an address book) would grow at the same speed as the data – and waste a lot of space. So what they do is use “allocation units”, also known as the “cluster size”. The volume is divided into these allocation units, and as far as the file system is concerned, they cannot be subdivided – those are the smallest blocks it can address. Much like you have a house number, but your postman doesn’t care how many bedrooms you have or who lives in them.
So what happens if you have a very small file? Well, the file system doesn’t care if the file is 0 KB, 2 KB, or even 15 KB, it’ll give it the least space it can – in the example above, that’s 32 KB. Your file is only using a small amount of this space, and the rest is basically wasted, but still belongs to the file – much like a bedroom you leave unoccupied.
Why are there different allocation unit sizes? Well, it becomes a trade-off between having a bigger table (address book, e.g. saying John owns a house at 123 Fake Street, 124 Fake Street, 666 Satan Lane, etc.), or more wasted space in each unit (house). If you have larger files, it makes more sense to use larger allocation units – because a file doesn’t get a new unit (house) until all others are filled up. If you have lots of small files, well, you’re going to have a big table (address book) anyway, so may as well give them small units (houses).
Large allocation units, as a general rule, will waste a lot of space if you have lots of small files. There usually isn’t a good reason to go above 4 KB for general use.
Fragmentation?
As for fragmentation, fragmentation shouldn’t waste space in this manner. Large files may be fragmented, i.e. split up, into multiple allocation units, but each unit should be filled before the next one is started. Defragging might save a little space in the allocation tables, but this isn’t your specific issue.
Possible Solutions
As gladiator2345 suggested, your only real options at this point are to live with it or reformat with smaller allocation units.
Your card might be formatted in FAT16, which has a smaller limit on table size and therefore requires much larger allocation units in order to address a larger volume (with an upper limit of 2 GB with 32 KB allocation units). Source courtesy of Braiam. If that is the case, you should be able to safely format as FAT32 anyway.
Basically you are going to need to reformat with a different allocation.
OK. Thanks. I never did anything with either drive, other than plugging them in. So what I think this means is that, out of the box, two drives coming from the same manufacturer (WD) are formatted with different cluster sizes?..
Are they different model numbers?
They are: WDBFJK0080HBK-NESN and WDBBGB0080HBK-NESN.
Yeah - those are two different generation products. It’s very possible that something changed between them.
Just to follow up on this, it appears that not only do they likely have different sector sizes we have also changed file systems for this generation product. New My Book’s (the one with the bumps on it) are now formatted in exFAT which has the advantage of being compatible with both newer PCs and Macs out of the box.
Thanks. I am finding that exFAT can be formatted at a minimum of 512k. In order to get down to 4k, I had to reformat under NTFS.
