Why the inefficient file format?

My Cloud OS 3 Personal & Network Attached Storage My Cloud
1

Just installed a 3TB drive.  I am stunned that the file system is some odd (to me) variant WHICH I CANNOT CHANGE!!!  The file system is like a FAT system, in that tons of disk space is lost if you have a lot of small files, plus “properties” (which can be associated with an NTFS file) cannot be copied to the WD drive!  The worst part (for me) of the latter case is that you get an error popup requiring you to accept this limitation, which means you cannot start a big copy and walk away, expecting it to be done when you get back.  Infuriating!

Questions:

  • Is there a way around this problem?
  • If not, is there a competitor’s product which has (or can be changed to) a better file system?

Thanks!

2

http://wdc.custhelp.com/app/answers/detail/a_id/1391/~/reformatting-a-wd-nas-drive-as-fat32,-ntfs,-or-a-mac-file-system

3

Thanks for the quick response.  However, I already saw that webpage, which lists a bunch of models (including mine), and then says “The Western Digital units mentioned above use a proprietary file system and cannot be reformatted as FAT32, NTFS, or a Mac File System.”  I called the support line (surprisingly quick answer), who confirmed that the webpage was correct.

4

JerryDoerr wrote:

… plus “properties” (which can be associated with an NTFS file) cannot be copied to the WD drive!  The worst part (for me) of the latter case is that you get an error popup requiring you to accept this limitation, which means you cannot start a big copy and walk away, expecting it to be done when you get back.  Infuriating!

It’s a NAS.  ALL NASes behave similarly.

What you wanted was a USB drive.

5

Thanks, Tony.

Wow, it’s too bad that they all work that way, because that way is crummy (system bleeped my attempt to say “s@cks” – really?) for my purposes:

  • So much space wasted for small files – eg, I have copied 38.1 GB of actual data from an NTFS drive to the NAS drive, and it takes 38.2 GB of space on the NTFS drive, but 65.9 GB of space on the NAS drive!
  • Plus it’s such a pain to do large copies which have uncopiable “properties”.

(If anyone has time to waste, I’m all ears for WHY all NAS drives work that way – what advantages outweigh the big (to me) disadvantages that I have mentioned?)

So maybe there is another way?  You mentioned USB drives – we already have one per computer, but sharing isn’t sufficient because one of the computers can be out of the house for a day/week/month.  But I notice that my new router has 2 USB ports, which (per the documentation I have just now read) seem to be intended for USB drives (of various flavors, including NTFS).  Is that my answer? – so that I should just return my 3TB NAS drive and buy a 3TB USB drive to replace it?  Are there gotchas if I go down that path?

6

For large storage devices, sector/cluster size is a compromise between indexing space and small file requirements.  I guess WD may have optimized for media files.

 Plus it’s such a pain to do large copies which have uncopiable “properties”.

Which properties can’t you copy, and how are you copying?

I edited a DOS batch backup script to copy all my media and files to the MC.  That barfed due to /O ownership flag.  Not surprising when trying to copy to a shared access disk that has no knowledge of the PC users. Remove that flag and everything copies without trouble, as do all File Manager operations between XP PC and NAS.  I don’t use WD’s app on the PC because it’s completely unnecessary; just create persistent drive mappings and use it like any other disk.  In fact, I have multiple drive mappings for different parts of the NAS.

7

 But I notice that my new router has 2 USB ports

Missed this…

Yes, I’d always recommend investigating the capabilities of the kit you already have.

I have a Technicolor 582n ADSL router, which has a USB port it can use as NFS and DLNA media server, but I got it after the MyCloud.  And, although it’s supposed to support NTFS as an option, mine seems not to have that option…  It will serve out a FAT32 disk, though.  Check first before buying something else that isn’t right…  FAT32 only supports 2TB per partition, IIRC…

8

JerryDoerr wrote:

Thanks, Tony.

 

Wow, it’s too bad that they all work that way, because that way is crummy (system bleeped my attempt to say “s@cks” – really?) for my purposes:

  • So much space wasted for small files – eg, I have copied 38.1 GB of actual data from an NTFS drive to the NAS drive, and it takes 38.2 GB of space on the NTFS drive, but 65.9 GB of space on the NAS drive!
  • Plus it’s such a pain to do large copies which have uncopiable “properties”.

(If anyone has time to waste, I’m all ears for WHY all NAS drives work that way – what advantages outweigh the big (to me) disadvantages that I have mentioned?)

 

So maybe there is another way?  You mentioned USB drives – we already have one per computer, but sharing isn’t sufficient because one of the computers can be out of the house for a day/week/month.  But I notice that my new router has 2 USB ports, which (per the documentation I have just now read) seem to be intended for USB drives (of various flavors, including NTFS).  Is that my answer? – so that I should just return my 3TB NAS drive and buy a 3TB USB drive to replace it?  Are there gotchas if I go down that path?

You bought the poor man’s NAS i.e. the MYCLOUD.  If you want support and better hardware, pay the price and get a synology.

Why use proprietary formats?  Because that’s the only competitive advantage NAS makers have is the software.  If the raspberry PI had SATA support and gigabit lan, no one would buy MYCLOUD devices.

Unless you bought a brand new router that cost over $200 USD, using a USB port on it is going to be much worse.  At best you will be able to read data back to maybe one client at a time.

9

cpt_paranoia –

Re your first message, I’m aware of the sector/cluster vs indexing compromise.  NTFS (my main experience) accommodates small files well (very little lost space), but if that means that it needs an inordinate amount of indexing space, it’s not obvious to the user.  In any case, TonyPh12345 above says that it’s not just WD – rather, that all NAS drives work that way.  Perhaps NAS drives can be REALLY big (many TB, for corporate use rather than personal use), and therefore the file system must be more worried about index space – just guessing here.

I can’t tell you what the troublesome properties are, because the user is not told what properties cannot be copied.  In response to your question about how I am copying, I am simply trying to copy/paste using IE Version 11 under Windows 8.1 – though I have gotten the same error for several years, in earlier Windows versions and earlier IE versions.  Here is the error from IE:

Error copying properties.PNG

More details about this error – sorry for the length.  When I click Yes, there is no apparent (to me) problem with the resulting copy.  Even so, there are still 2 problems with this error, only one of which I have already whined about:

  • As per previous whines, it means that you cannot come back to your computer and find that your copy request has finished.  And if you are copying a lot of files with properties, you will have a long wait for completion after clicking Yes.
  • Also (new whine), it means that the files with “properties” must be processed twice to accomplish the copy.  That is, the file copy algorithm goes thru all the files you have requested, copying the ones without properties, but just keeping a list of the files with properties.  Only after going thru all the files once does it present the above error to the user (luckily, so that you can respond Yes once for all such files).  If the user clicks “Do this for all current items” and then clicks Yes, the algorithm goes back thru all the files with properties and actually copies them this time.

These might not be a big deal when copying a small number of files – depends on the exact size, order, and properties of the files you are copying.  But with 10’s of thousands of files with “properties”, plus 10’s of thousands of files without “properties”, there is a long delay before you finally see the popup error, and then it takes a non-trivial amount of extra time just to go thru all files with properties again (not even counting the time to do the actual copies, which is presumably the same as it would have been if the copies had been done on the first pass). 

Re your second message, I have a brand new LinkSys AC1900 (AKA EA6900) router.  (Which, by the way, meets my expectations, so I don’t whine about everything.)  I don’t know how I can check for compatible USB drives before I buy, since the pre-purchase info is unlikely to tell me what the file structure is, nor whether I can reformat the drive.  Good point about the partition size – just checked Wikipedia, which says “With Windows 8 and Windows Server 2012, the maximum file size implemented [for NTFS] is 256 TB”, so presumably a drive can be that large also.

10

BC1111  –

Would Synology get me NTFS?

Your proprietary formats discussion makes sense … but does that mean that each manufacturer rolls its own?  If so, is there one which provides NTFS compatibility?

Router is a brand new LinkSys AC1900 (AKA EA6900), $150 thru Amazon.  But I don’t know how long it has been available – the beefier (and newer?) LinkSys has problems (per reviews), so I passed on it.  I just want “local cloud” (oxymoron?)storage, as backup and sharing for 2 computers – so one client at a time is fine 99.9% of the time.

11

It’s not only the underlying file system that’s at issue (NTFS vs EXT4) – it’s the protocol that’s used to communicate between systems  (Samba, or CIFS).  

Samba (the open-source implementation of CIFS/SMB protocol) doesn’t support some NTFS extended attributes.

Since most NAS devices (not all) rely on EXT4 and Samba, you’ll have to be careful.

If you absolutely rely on those attributes, you might be better choosing a NAS which is built around a WIndows platform instead of LInux and uses NTFS as the store.

12

JerryD

How are you actually doing the copy? you mention using IE11 which may be possiable it is not recomended

Is this local or remote? If local use file explorer, not Internet explorer. if remote use the WD desktop app

A NAS is a seperate computer that shares its disks so the underlying format is not visable and not relevent. Some may handle the atribute better then others it could take a lot of trials to find one.

I have not seen any attribute issues and your screen shot is not visable yet so I have no suggestions for this

13

TonyPh12345 –

It could be that I don’t need the troublesome properties/attributes – hard to tell, since my main use is for backups, so I’ll never know until/unless I need to restore.  But I have done a small test, and can’t see anything wrong with one of those copied-without-properties files.  However, it’s the copy process itself which is ALWAYS an issue for me, because the error popup interrupts and lengthens the copying process, as I described earlier.

I don’t know how to follow your suggestion to “be careful” to get an NAS which is build around Windows and NTFS – is there a secret code in the advertising literature?

larryg0 –

SORRY! – I misspoke.  It’s a local copy, and I should have said that I use File Explorer, not IE.

I don’t have the persistence to go thru a lot of trials to find a compatible NAS.  That’s why I like the suggestion of a USB drive attached to my router – but the only feedback so far on this was pretty negative.

Sorry about the screen shot – it shows OK when I look in my post.  Here’s what it says (from File Explorer):

    In title bar:

        1 Interrupted Action

   

    In body:

        Are you sure you want to copy this file without its properties?

       

        The file xxxxxxx has properties that can’t be copied to the new location.

   

        <Some details about the file here, which is usually (always?) a .eml file.

          Details shown for a .eml file are From, Subject, and Date received.

          So the copy process knows about the internals of this file type.

        >

   

          Do this for all current items

       

             Yes    Skip    Cancel    <these are 3 click targets>

If there is a way to tell File Explorer to always do the copy anyway, with no error popup, I would shut up and go away.

14

A USB drive attached to the router is effectivly creating a NAS ans you will probably have the same issues. A USB on a local PC would not have the same issues.

If you are ok with loosing the attributes you can try another copy methode, I use Windows commander and if it even complains it should have an option to apply for all. you could also try xcopy which is what I use for my primary PC backup. I have a little batch file i call from task scheduler

15

It might be significant (I don’t know) that the example you gave is an .eml email file.  No wonder you’re losing disk space (assuming your emails aren’t bloated HTML format…).

Rather than use File Explorer, you could create a backup script (a .bat file), and stick a shortcut on your desktop, and just hit that when you want to do a backup. It saves having to trawl around file hierarchies, and makes simple backups (copy, not mirror) very simple. Here’s the script I use to back up my media:

echo off
REM mymedia backup script 2014/10/27
REM added disk presence checks 2012/10/02
REM assumes presence of following drives:
REM C: Local disk
REM E: Media (Dane-Elec 1.5TB)
REM M: NAS (WD MyCloud)

REM DOS command to copy all files newer than already exist
REM xcopy
REM /C	continue past errors
REM /D	copy newer files
REM /E	include empty directories
REM /S	include subdirectories except empty ones
REM /F	display full pathnames
REM /H	copy hidden files
REM /I	assume destination is a directory if it doesn't exist, and copying multiple files
REM /K	copy attributes
REM /O	copy ownership attributes
REM /R	override read-only destination
REM /U	copies only files that already exist (update?)
REM /V	verify copies
REM /Y	suppress overwrite request
REM /L	list what would be copied

set ERROR=FALSE

echo Starting MyMedia backup > C:\Backup\log\NAS\MyMedia.log

REM backup my stuff to MyCloud NAS
if exist M:\NAS.drv (
	echo Backup to N: >> C:\Backup\log\NAS\MyMedia.log
	xcopy /C /D /S /F /H /I /K /R /V /Y /EXCLUDE:C:\Backup\exclude\backup.txt C:\Backup N:\Backup >> C:\Backup\log\NAS\MyMedia.log
	xcopy /C /D /S /F /H /I /K /R /V /Y C:\User N:\User >> C:\Backup\log\NAS\MyMedia.log
	echo 'Documents and Settings' to N: >> C:\Backup\log\NAS\MyMedia.log
	del /F /Q "N:\Documents and Settings"
	xcopy /C /D /S /F /H /I /K /R /V /Y /EXCLUDE:C:\Backup\exclude\documents.txt "C:\Documents and Settings" "N:\Documents and Settings" >> C:\Backup\log\NAS\Documents.log
	xcopy /C /D /S /F /H /I /K /R /V /Y "C:\Documents and Settings\All Users\Application Data\F-Secure" "N:\Documents\F-Secure" >> C:\Backup\log\NAS\Documents.log
) else (
	echo NAS N: drive not found
	set ERROR=TRUE
)

REM backup my stuff to Dane-Elec 1.5TB
if exist E:\media.drv (
	echo Backup to E: >> C:\Backup\log\NAS\MyMedia.log
	xcopy /C /D /S /F /H /I /K /R /V /Y /EXCLUDE:C:\Backup\exclude\backup.txt C:\Backup E:\Backup >> C:\Backup\log\NAS\MyMedia.log
	xcopy /C /D /S /F /H /I /K /R /V /Y C:\User E:\User >> C:\Backup\log\NAS\MyMedia.log
	echo 'Documents and Settings' to E: >> C:\Backup\log\NAS\MyMedia.log
	del /F /Q "E:\Documents and Settings"
	xcopy /C /D /S /F /H /I /K /R /V /Y /EXCLUDE:C:\Backup\exclude\documents.txt "C:\Documents and Settings" "E:\Documents and Settings" >> C:\Backup\log\NAS\Documents.log
	xcopy /C /D /S /F /H /I /K /R /V /Y "C:\Documents and Settings\All Users\Application Data\F-Secure" "E:\Documents\F-Secure" >> C:\Backup\log\NAS\Documents.log
) else (
	echo MyMedia E: drive not found
	set ERROR=TRUE
)

REM backup my music to Dane-Elec 1.5TB
if exist M:\NAS.drv (
	if exist E:\media.drv (
		echo scripts and artwork to M: >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y C:\Music\scripts M:\Music\scripts >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y C:\Music\artwork M:\Music\artwork >> C:\Backup\log\NAS\MyMedia.log
		echo scripts and artwork to E: >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y C:\Music\scripts E:\Music\scripts >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y C:\Music\artwork E:\Music\artwork >> C:\Backup\log\NAS\MyMedia.log
		echo MyMusic to E: >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y /EXCLUDE:C:\Backup\exclude\music.txt M:\Music E:\Music >> C:\Backup\log\NAS\MyMedia.log
		echo Audiobook to E: >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y /EXCLUDE:C:\Backup\exclude\audiobook.txt M:\AudioBook E:\AudioBook >> C:\Backup\log\NAS\MyMedia.log
	) else (
		echo MyMedia E: drive not found
		set ERROR=TRUE
	)
) else (
	echo NAS M: drive not found
	set ERROR=TRUE
)

REM backup my pictures & video to Dane-Elec 1.5TB
if exist M:\NAS.drv (
	if exist E:\media.drv (
		echo Pictures\open to E: >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y P:\open E:\Pictures\open > C:\Backup\log\NAS\MyMedia.log
		echo Video\open to E: >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y V:\open E:\Video\open > C:\Backup\log\NAS\MyMedia.log
		echo Pictures\closed to E: >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y C:\Pictures\closed E:\Pictures\closed >> C:\Backup\log\NAS\MyMedia.log
		echo Video\closed to E: >> C:\Backup\log\NAS\MyMedia.log
		xcopy /C /D /S /F /H /I /K /R /V /Y C:\Video\closed E:\Video\closed >> C:\Backup\log\NAS\MyMedia.log
	) else (
		echo MyMedia E: drive not found
		set ERROR=TRUE
	)
) else (
	echo NAS M: drive not found
	set ERROR=TRUE
)

if %ERROR%==TRUE (
	echo Backup incomplete: check for missing or incorrect disks
	echo Backup incomplete: check for missing or incorrect disks >> C:\Backup\log\NAS\mymedia.log
	echo You can assign drive letters using 'My Computer/Manage/Storage/Disk Management'
	date /t >> C:\Backup\log\NAS\mymedia.log
	time /t >> C:\Backup\log\NAS\mymedia.log
	pause
) else (
	echo Backup succesful >> C:\Backup\log\NAS\mymedia.log
	date /t >> C:\Backup\log\NAS\mymedia.log
	time /t >> C:\Backup\log\NAS\mymedia.log
)

That might look a bit fierce, but it’s doing a lot of things, and backing up multiple locations to multiple drives.  Fundamentally, it’s based around the DOS xcopy command:

xcopy /C /D /S /F /H /I /K /R /V /Y /EXCLUDE:<EXCLUDE file=“”> <source location> <backup location> >> <logfile>

There are a few simple error checks to make sure that drives are present before trying to access them; I leave little “I’m here” files (e.g. media.drv) on the drives, and the script checks it can find them; crude, but effective.

The /C flag says ‘continue past errors’ (which might be useful in your case), and the /D flag says ‘copy any files that are newer than the version on the destination drive’. This ensures your backup contains the latest copy of files on the source.

I stick a shortcut to this script on my desktop, and hit it on a regular basis, and leave the script to do its work…

[had to do a major edit after a tiny edit on the iPad messed up the entire post… Hope it’s the same as it used to be…]

16

I have a little batch file i call from task scheduler

That’s a good idea; it might be a nice idea to schedule my script to run every time I shut the machine down.

[goes off to investigate adding scheduled tasks: ah, in Control Panel|Schedule Tasks|Add Scheduled Task.  Simple…  oh, but it only offers run at start, not run at shutdown.  My poor old PC has enough to do when it starts up, running virus checks, checking software updates without adding to its burden]

17

Using xcopy in a .bat (.cmd) file – wow, I haven’t done that for years … decades probably.  I’m mildly surprised it is still supported.  It’s a good idea – thanks for the extensive example to start with, and all the inline documentation.

But before I resort to that, I think I will try a USB drive attached to my router – if that works, it solves all of my problems.  I have a high-capacity thumb drive and a disk drive (both are USB and NTFS), so I can test without investing in new hardware.  If I’m a good boy, I will report back here re my success or failure.

18

cpt

I have never tried it but there is a command line shutdown (shutdown /? for details) command that you could add to the end of your script then use this to shutdown the system instead of the shutdown menu

I leave my computer on all the time and just let it sleep, it should wake up to run tasks although it may be an option in task scheduler

19

 I have never tried it but there is a command line shutdown

Good idea.  I’ll give it a try…

20

I’ve finally gotten back to this.  This post attempts to document what I found – and may surpass anyone’s interest.

As review, I had 3 issues about the WD NAS disk:

  • It was abysmally slow.
  • It would not accept certain “properties” (of unknown sort) of files that I wanted to copy to it.
  • It reported that WAY MORE space is used for my backed-up files than is used for those files on my primary NTFS disk.

Re speed, that turned out to be my network setup.  After installing a gigabit router, and after connecting my computer via direct-wired Ethernet rather than via Wi-Fi, speed is no longer an issue.  The different backup approaches that I tested each had different strengths – some were better at large files and some were better at small files.  But overall, for the mix that I commonly back-up (4339 items, and 1.31 GB), speed was pretty close to the same for all of them.  For reference, to establish an upper limit on what could be expected, I did the backup to a Seagate USB 3.0 drive, formatted as NTFS, directly connected to my computer – ie, no network involvement. It took 1:28 (ie, 1 minute and 28 seconds), and gave no “properties” error popup.  Then I got these results doing the backup across the network:

  • To the WD NAS drive: 3:49, with “properties” error.
  • To the Seagate drive, but connected to the router’s USB 3.0 port: 3:55, with “properties” error.
  • To the Seagate drive, but connected to a second Ethernet-wired computer on the network: 3:45, with no “properties” error.

So when I tell you that before the network speed improvements, the backup would take more than an hour, you will understand how happy I am with under 4 minutes.

Re the file “properties” problem, I think I will just live with it.  Interestingly, the problem is NOT just with the WD NAS drive.  As someone in previous posts predicted, the problem seems to be independent of the drive. As indicated above, when using the Seagate drive connected to the router, the properties problem still occurred, even though that drive is NTFS.  In any case, the problem seems not to be critical for my backups, and the speed improvement actually helps the properties problem, indirectly.  That’s because the previous slow speed caused such a long delay before the properties error popped up (more than 30 minutes), but you must wait for the popup before you can tell it to continue with the copy.  Now, with the speed improvement, it only takes 2:40 to get the error popup, which is OK.

Re the disk space problem (which gave rise to the Subject I used for this thread), it seems severe.  For my 1.31 GB case above, it takes up 1.33 GB on an NTFS disk, but takes up 4.89 GB on the WD NAS drive!  However, I don’t know if those numbers can be trusted, because of this bizarre (to me) observation: in the case of the NTFS Seagate drive connected to the router’s USB port, even pre-existing files were reported as using much more space than they really used on the NTFS drive!!!  So the NAS interface does not really report the actual space used on the disk!  The only way to get around this problem is apparently to use an NTFS disk connected to another computer on the network.  But I think I will just live with this problem also.

Which does lead to another question: how can I determine how much space is used (even though it might be a lie) on the WD NAS disk?  Right-clicking on the drive doesn’t even produce a “Properties” choice (the normal way to see disk usage), and choosing “Properties” for the root-level folders reports 0 bytes for each folder.