MyCloud 3TB how to resolve possible index database corruption

I have an older 3TB MyCloud that is reporting data that no longer exists. The home page graph says 2.9 TB free but the sidebar items report 1.6 TB of video files, 380 GB of music, 22 GB of pictures and 2 billion kilobytes of “Other” data. All user data was removed via deletion over the network and the deletion was confirmed by logging in with SSH and checking the disk usage and free space. No user files remain. The shares were then removed and re-added. There is no trace of user data remaining.

The total usage on the hard drive is around 60 GB. The Cache folder has about 50GB of data in it.

I have rebuilt the database via the GUI twice but it is still reporting the non-existent files.

It would seem the GUI has two sources of file usage data. I assume one is the OS and the other is Twonky. Which part of the graph has which data?

It there any way to clean up the erroneous data?

Do you have a USB hard drive attached to the My Cloud? If so make sure USB Content Availability is disabled in the My Cloud Dashboard > Settings.

If one has used WD Sync then they may have a hidden folder on their My Cloud using up space. This folder won’t be found using a file explorer. One has to use SSH and search for the hidden WD Sync folder.

No, I have never connected a USB drive to the NAS. Never used cloud services/WDSync either. The residual metadata does reflect what has traditionally been on the NAS hard drive data partition (4), but none of the actual data is still present.

I am fairly sure that some database needs to be purged or rebuilt. The questions are: which database, where is it, and how can I clean it?.

I am in the process of replacing the hard drive. It has been having a lot of “unexpected network errors” that leave undeletable files on the disk. SMART reporting says it will soon be an ex hard drive. It has over 13000 hours of spin time.

I would like it to start clean, so clearing out the bogus data is a priority.

I did not want to use the images out on the Internet as their partition schemas are wrong (non-GPT) and can’t be resized in the Ubuntu 20 Disks app, so I am making an image from the old disk before it finally suffers a catastrophic failure. When I left for work it was still making the compressed image. It was done with the system partitions but still had to work through 2.7TB of zeros on the data partition. ( I zeroed the free space to make it compressible )

How did you “rebuild” the database via the GUI? Are you talking about the Rebuild option for the DLNA Database found on the WD My Cloud Dashboard > Settings > Utilities page? If so that typically has nothing to do with what information is shown on the Capacity chart/graphic. Rather it only affects the Twonky media server’s database.

If one has backed up their data from the My Cloud, or removed all their data. One can perform a Quick Restore via the My Cloud Dashboard > Settings > Utilities page. That will delete all user data and reset the My Cloud options/settings to default values. A Full Restore will take significantly longer since it will attempt to securely erase all user data on the drive before resetting the options to default values.

For what ever reason, from time to time, the Capacity graphic will show incorrect information. Sometimes a system reset will fix the issue.

Yes, it was done by the GUI where there is an option to rescan or rebuild. It’s not up now and I am not at home so I can’t repeat the steps but that sounds about right. I didn’t notice if it referred to DLNA or not, all I noticed was the database part.

The metadata displayed is certainly something that Twonky would need to index and the OS would not particularly care about. That’s why I tend to think the circular graph is from the OS and the categorized totals are from Twonky.

Was trying to avoid a complete reset in order to keep the users and network config. I can do that, though, on the new copy assuming the imaging works.

Generally the Twonky database is separate from what you see in the Capacity section because one can turn off the Media Serving option on each Share so the Twonky and iTunes server on the My Cloud does not scan the Share for media files. And one can further turn off Twonky and iTunes media servers all together in the Dashboard.

One can access the Twonky administration page and perform further actions on the Twonky server. Example: http://WDMyCloud:9000

Much more on the My Cloud Twonky Media Server.

Thanks for the link to the Twonky FAQ. At the moment, the image is still being applied. Even though it compressed to 4.7 GB, it still has to write out 2.7TB of zeros. Didn’t see a good way around that, or at least any that I wanted to mess with. Work a little, wait a lot seemed easier than work a lot, wait a little less.

It will probably take most of the night. When (if) the new drive comes up I will apply the advice provided.

Thanks for your help.

Some data points to show that the Twonky media server (and it’s database) typically do not factor in to the Capacity calculations.

With Twonky and iTunes media server enabled. The Capacity section showing some files on the My Cloud.

Twonkyturnedon

Now I go and turn off Twonky and iTunes media server in the Dashboard > Settings > Media. Then add a bunch of additional media files to the My Cloud.

CapacitywithTwonkyiTunesTurnedOff

No change in the Twonky/iTunes section indicating new files were added. Note the zero music tracks added yet Capacity section now shows 300MB of music tracks.

One service which DOES affect what is shown on the Capacity charge is the wdmcserverd service. Disable that through SSH (issue: /etc/init.d/twonky stop via SSH to the My Cloud) and several of the Capacity section values no longer are displayed, they’re shown as zeros.

Capacitywdmcserverdstopped

One can restart the wdmcserverd service by issing the following command via SSH to the My Cloud: /etc/init.d/wdmcserverd start

There is some discussion of removing the hidden .wdmc directories after stopping the wdmcserverd service that may be relevant.

Well, the reimaging didn’t go so well. Using dd was a complete failure so I tried Clonezilla. That did a much better job of reproducing the filesystem but it resulted in an unstable system. It came up OK and I could SSH into it, but trying to load the management web page only resulted in crashes.

Checking dmesg on the imaged drive via SSH showed that the web server was killed due to lack of memory.

There was no swap space on the cloned drive. It looks like fdisk identified partition 3 as Microsoft Basic Data when it should have been swapfs and Clonezilla just went with that.

I logged in with ssh and executed mkswap /dev/sda3 ; swapon /dev/sda3 ; reboot - and that worked. Oddly, sda3 is physically the first partition on the hard drive.

I did find a lot of residual references in /DataVolume/shares/.wdmc to media files that used to exist. I think that may have been where the phantom data was hiding.

After much gnashing of teeth, cursing, swilling of beer, and throwing of heavy objects, the 4TB drive is in and working.

I now have time to read your excellent write-up.

Unsolicited advice to all: If you want to upgrade your drive don’t wait until it fails. The 2T and 3T images available for download are not GPT and will probably only result in a red led condition. That’s all they did on mine. My hard drive was tired and growing bad sectors in the user data area, but the system area was still good so I was able to image it and create a replacement drive.

Clonezilla will make a good backup copy of your drive. It’s not the easiest software to use, but it does work.

Since technology and software can change over time, it’s possible that there have been updates or new solutions available since then.