Maximum total drive storage allowed on My Cloud (v2)?

Greetings!

I know that this topic has been discussed before, but since the most recent postings I’ve found date back to March 2014, I decided to post this again, just in case any new information has become available since then.

Question:
What is the maximum total drive storage, (both internal and external via USB), that can be attached to a My Cloud device?

Mine is a 6T Gen2 cloud device.


I know from reading other articles in other places that a particular system can only access a certain amount of disk storage based on things like the way the firmware is written, the amount of memory installed, etc. etc. etc.

I had a Gen-1 HP Media Vault NAS device that would support a maximum of 1.2T of drive space as a combined total of the internal drive as shipped, any drive installed in the internal expansion bay, and anything connected via the USB connector. This was explained as a limitation of the device itself due to, (I believe), internal memory constraints.

I also know that exceeding a device’s maximum storage capacity limitation can rapidly result in a borked system when the file system wraps-around on itself. (I’ve done that with Windows XP, and it’s not a pretty sight!)

So. . . . .

Does anyone know of any published spec on the maximum total drive storage space allowed on a My Cloud - particularly as applies to the Gen-2 (2.n firmware) versions?

Thanks!

Jim (JR)

As far as I can tell, that limitation would be either the LBA type, partition type, or filesystem type dependent.

The 6TB version already clearly supports GPT partitioning, which has a maximum partition size limit of 6.4ZiB (Zebibytes) on disks with 512 byte sectors. This is much larger than the current hardware tech bottleneck, which is 48bit LBA-- It has a max addressable block limit that will limit 512byte sector based devices to 128PiB (pebibyte)

The file system you choose to use could impose a bottleneck before that–

If you use an NTFS formatted storage volume, then the max size appears to be 16EiB.

The Mycloud gen2 supports these filesystem types:

FAT
EXT3 and EXT4
NTFS
HFS+
XFS (not documented)

The single bay MyClouds are not really meant to be taken apart, which is the only way to switch out the drive inside (a not straight forward process), which means you are limited to the size of disk WD ships inside unless you feel like voiding the warranty. However, the max size of the external USB disk is controlled by the above information.

It sounds like what you are REALLY asking, is what is the maximum RECOMMENDED size. We can derive that based on the amount of system ram installed in the system, against the max recommended sizes of other offers. It looks like WD is subtly suggesting that one have at least 125mb of ram for each GB of storage, if you work out the numbers given on their product comparison chart.
https://support.wdc.com/knowledgebase/answer.aspx?ID=11425

There is also one wildcard at play with the single bay My Cloud units. The USB port. There have been complaints in other threads of people having trouble getting more than five USB drives operating at the same time. So even though you could put the largest hard drives possible into a USB enclosure if you cannot get seven external USB hard drives drives working with the single bay My Clouds, which is what the single bay My Cloud manuals indicate is the maximum number of USB attached drives the devices are capable of accessing, then you most likely wouldn’t hit what ever limit there is on the total storage size of both internal and external drives.

Indeed. That would be a limitation on the USB driver stack though. Technically speaking, USB’s spec theoretically allows up to 256 serially addressed devices per root hub. You are going to saturate the USB bus with protocol traffic long before you reach that though. (the speed of the port is going to be a serious limiting factor in putting lots of storage on the device, at least as far as reliability is concerned.)

jharris1993: The issue with XP wrapping its filesystem was due to LBA. When it shipped, XP only supported 28bit LBA. There were disks that needed 48bit LBA to address the full drive. The lba register would overflow in the windows driver when trying to access these disks, unless you set a registry hack, and installed an updated atapi.sys. (Needs SP1 installed) In addition, there is a limit imposed by the old partition scheme, MBR-- which could only use 32bit LBAs. This limits the size of a disk to 2TB. (or 137GB, with the atapi issue.)

The Mycloud NAS uses linux internally. (Gen2 is still technically linux, even though it is a weaksauce busybox shrinkwrap embedded version) Linux knows all about GPT, the new partitioning schema, and has known about 48bit LBA for over a decade. There is not a reasonable technological reason why the Mycloud would be unable to see even a truly enormous sata disk attached to it.

Bennor,

I wish it were that simple.

Especially with embedded devices - like these stand-alone NAS devices - there may be hardware limitations with controllers and/or available memory that would limit the max storage possible. The fact that the USB connector has issues with more than “X” number of devices reinforces my fear of there being some kind of internal limit.

(BTW, has this been explored on a “device size” basis? i.e. can you add more 250 gig devices than 2T devices?)

The rapidly expanding size of both hard drives themselves, as well as the corresponding enclosure capacities and stand-alone device sizes, makes it easy to rapidly and (somewhat), inexpensively add huge volumes of storage.

Assuming that I can only add two or three devices of any size, it is not unreasonable to expect that someone could add 16, 32, or even 64 Tb of storage to the device. Note that I said it’s not unreasonable to expect, not that it is likely to happen in the next 15 minutes.

Example, on Micro Center’s web site, there is a Western Digital My Book Duo 16TB (2 x 8TB) SuperSpeed USB 3.0 3.5" RAID External Hard Drive for $500 US.
(Viz.: http://tinyurl.com/m7e369z )

On the same page is a Seagate Backup Plus Hub 8TB External Desktop Hard Drive for $200 US, (Viz.: http://tinyurl.com/ljx2jjv )

At both $500 and $200 US a pop, these might seem a bit extravagant right now. In six months, a year, (or at a “going out of business sale”), the price of these items could well be within a reasonable price-range. It wasn’t long ago when one or two terabyte drives were horrifically expensive.

Now 8T drives are becoming (somewhat) reasonable in price, to the point that sometime soon I may want to post a “teardown and upgrade” article on how to upgrade a My Cloud of size “X” to 8T. I strongly suspect that within a year, or less, 16T drives will become available. Expensive? Of course. Available if you have the $$$? Absolutely.

So, using the 16T device as an example, if someone were insane enough to connect three of those beasts to a My Cloud, you’d have 48T of storage in addition to whatever the capacity of the internal drive might be.

An even more likely scenario is someone adding a relatively inexpensive 8T external device to an 8T My Cloud - total capacity, 16T. Too big? Dunno.

Q.E.D. I believe that this question has merit and somehow or other should be addressed.

Of course, the absence of a “shutdown” button within the web interface, or the inability to turn it off safely from the front panel, or a real WORKING USB 3.n interface, or a [insert name of stupid and annoying issue] has been kicked around here for eons, and should have been addressed too.

Unfortunately, I am not holding my breath and I fear we won’t find out until someone borks his box by adding too much storage to it.

What say ye?

Jim “JR”

As I said to Benor, it’s not that simple.

Using my previous example, the HP Media Vault - despite running a busy-box version of Linux - was spec’d to a hard limit of 1.2T due to memory constraints.

Though we can “assume” from the previous post linking to the WD spec page, (which is woefully incomplete), that there are 512 megs of memory in the beast, (and a small amount of swap, which helps), we don’t know anything about how this particular version of Linux might have been modified, how much of that memory is allocated to device storage pointers/buffers/memory, or what the address width of the various controllers might be, so I would really hesitate to assume I could hook up a drive of arbitrarily humongous size to the device.

What would happen - hard disk wrap-around or a file-system crash? Both are absolutely undesirable outcomes, and depending on how the system caves in on itself the results could be anything from “oh rats!” all the way to “JESUS *&%#-*ing CHRIST!!!” on the Richter scale.

Ergo, I strongly suspect that there is an upper limit out there somewhere, just waiting for some ignorant fool to mess with it. Since it is very unlikey that the My Cloud can support devices in the Exabyte range, we should know what the limit is.

What say ye?

Jim “JR”

I would tend to agree that thinking you can attach any arbitrarily huge drive to the system and have it work flawlessly is a pipedream, especially with these single bay consumer grade NAS appliances. They ONLY have 512mb of ram in them, for the gen2! (Gen1 has 256mb!!) If you have lots of users hammering the device, the system is going to run out of ram to use as FS cache very quickly, and will start having cache misses, which will then degrade its performance. The same may be true of write caching, or for having sufficient buffers for incomming connection requests.

At some point it becomes “absurd” to pretend the box is anything other than what it actually is-- a device designed and intended for light consumer use. (which would be the most likely reason somebody would try saddling it with that much storage, even with the trend for ever larger personal media collections.) When you start trying to use it as a local alternative to Netflix, for a family of 5 or more, all trying to stream 4k content at the same time, you are quickly going to overwhelm this little thing. Especially if you are doing it with something like Plex server. At that point, you really should be considering one of the multibay offerings that have upgradable RAM and other nice perks.

For single nerdy losers like myself, the single bay offering is cheap and effective, but you need to know what you are buying, and why you are buying it. In my not so humble opinion, the single bay MyCloud does not belong in small to midsize businesses-- at all.

Sir,

I entirely agree, with the exception of the “nerdy losers like myself” part. It’s us “nerdy losers” who pave the way for those other “ignorant fools” to have the technology they enjoy.

As far as heavy-duty servers are concerned, I solve that problem with a discarded system case, a power supply and MoBo from my parts box, a S-load of hard drives, and a copy of Linux Mint.

The result is about 16T of RAID 5/1 that gets backed up to two single-drive external enclosures nightly by a cron job.

The problem comes to the fore when I got shipped overseas, needed my server’s content, and had a very limited amount of cargo and weight I can bring. Ergo - the 6T My Cloud with a 3T external drive connected to it. It’s small and light, and (almost) carries the entire content of my “big” file server. (I’m considering expanding the external device to 6T, and adding the rest of my server’s content to it.)

I’d really like to have a bigger RAID box, multiple backup paths, and the winning numbers for the next 200 bazillion dollar Powerball! ( :grin: )

However, cost, weight, and size are constraints that I have to work within. so I am glad that the 6T My Cloud was available when I needed it.

What say ye?

Jim “JR”

Interesting observation:
If I enable “streaming media” mode for those paths where I have media files, I don’t need Plex. All I have to do is install VLC’s media player, attach it to the device, find the file I want, and stream away.

This works both on my (relatively) high-powered laptop with the i7 processor in it, all the way down to my woefully under-powered Android smartphone and tablet. I don’t think I have any 4k media, but then again, I don’t have a 300 foot TV screen either. With what I have, (which I think maxes out at 1080i), I don’t have any problems at all.

Of course I don’t have a half-dozen people trying to stream at the same time either.

As Sherlock would say: “Ingenuity, Watson - ingenuity!”

What say ye?

Jim “JR”

I read your other thread, and better understand the direction of your questioning. Based on some quick and dirty calculations, I can safely presume that this hardware will definitely support a drive up to 64tb in size, based on the number of significant bits required for a 6tb drive to be fully addressable. (it’s 36bits, in case you wondered.)

It is likely that the box will not have sufficient ram to properly cache the filesystem in a condition such as that, so the ram limitation is the most likely cause for concern, rather than the sata controller’s significant/active address bits.

SIgh. . . . .

Words like “always”, “definitely”, “never”, and their ilk, “always” ( :grin: ) make the hair on the back of my neck stand on end.

“Safely” presume? With all due respect, I’d really like to know how. I don’t “safely presume” that my data is even still there until I power it up again. Even then I like to have MD5 or SHA-1 hashes for anything critical.

[NB: HashCalc by SlavaSoft is a wonderful and tiny checksum calculator for Windows. It calculates every checksum I’ve ever heard of, and a bunch I haven’t, incredibly rapidly. It’s spec’d for Windows XP, but works wonderfully on my Windows 7 64 bit system. The fact that it’s freeware is even better.]

In this case, have you actually seen spec-sheets or data sheets for the enclosed hardware? How about memory allocation mapping? Unfortunately, I am not privy to any of that information so I have to assume the worst. If you have that info, and would like to share, I’d be eternally in your debt.

Jim “JR”

Update from Wired_w on the other thread:

And I quote:

Here he makes a valid point, which clarifies his earlier statement that though it will absolutely, positively address the entirety of a 64T drive, we don’t know if it has the memory or other resources to do it.

I stand corrected.

Jim “JR”

Another potential caveat is that drives larger than 3tb tend to be “advanced sector format” drives, with 4k sectors, instead of 512byte sectors.

I have not interrogated my system to see if it contains an advanced sector format drive, or a traditional sector format drive. Most (but not all!!) advanced sector format drives will (attempt!) to run with 512 byte virtual lba, if the controller does not support 4k block addressing. However, this is an embedded device. We cannot presume it knows what 4k sectors are, unless we observe it using one in the wild. (and vise versa!!)

hdparm should tell me this, but it will have to wait until I get home from work.

OK, I have interrogated the installed disk with hdparm.

/dev/sda:

ATA device, with non-removable media
        Model Number:       WDC WD40EFRX-68WT0N0                    
        Serial Number:      WD-[Deleted-Privacy]
        Firmware Revision:  82.00A82
        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
        Supported: 9 8 7 6 5 
        Likely used: 9
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors: 7814037168
        Logical  Sector size:                   512 bytes
        Physical Sector size:                  4096 bytes
        Logical Sector-0 offset:                  0 bytes
        device size with M = 1024*1024:     3815447 MBytes
        device size with M = 1000*1000:     4000787 MBytes (4000 GB)
        cache/buffer size  = unknown
        Nominal Media Rotation Rate: 5400

Looks like the device knows about 4k physical sectors, and works with virtual/logical 512byte sector simulation. Until somebody tries a 4k native drive in one, the 4k native format disk support is unknown.

Update:

I am doing an upgrade to my 6TB Gen-2 system where I am expanding the external storage from 3T to 6T, and updating the internal storage from 6T to 8T.

My game plan is to do a physical byte-by-byte copy using DD-Rescue (ddrescue - on Ubuntu/clones it is installed as gddrescue and is invoked as ddrescue)

After the copy using ddrescue, I plan to use GParted to expand the partition to its max size and re-install.

I am expecting the external drive upgrade to work w/o problems. The internal drive might be a horse of a different hue.

I will report results when done.

Jim (JR)