Clean OS (Debian), OpenMediaVault and other "firmwares"

@Fox_exe - Фантастическая работа, спасибо. Fantastic work, thank you. Ваш английский хорошо.

I am an amateur photographer and some days I used to try to upload 200 photos at a time to the My Cloud, and the ensuing scans would make it go off line. Not good for a Network Attached Storage device to disappear from the network. Now, with a clean install of Debian, for the first time, it works properly.

Removing thumbnails produced by the WD “firmware”

As I’m sure most of you know, the default WD software scans ALL the data on your drive. It makes up to four thumbnails of every video and picture it finds (regardless of whether or not the containing directories were marked as containing media files), and stashes them in hidden directories called .wdmc. It will have done this even on web pages downloaded in html format to a private share - and bear in mind that some web sites can use around 20 images to render a single page. And yes, the WD wastes its resources and your disk space making these jpg and jpeg thumbnails (which are only used by their software). An install of clean Debian (or any other install enabled by Fox_exe) leaves these files in with your data.

I wrote a script (initially for WD) to identify these but I never heard from them again on the subject. Here is a modified version of it, and this allows you to remove them all if you wish - or just locate them from the files created. It is pretty slow, taking around 7 minutes to finish searching and calculating disk usage on my system which has about 700GB in use with the results shown in the screenshot.

Script to remove thumbnails - screenshot

If this is usefule to anyone, good. It is supplied “as is,” and I accept no responsibility for any errors, loss of data, divorce or death of your dog should you choose to run it. The assumption is made that there is nothing else, important or otherwise, in the .wdmc directories. These will be irretrievably deleted by this script if you select the “y” option at the end. Yes, this could all be done on one line with the find command, but as I’d already writtent the bulk of this for WD I thought maybe someone may find it useful.

#!/bin/bash
# Script to find and remove the .wdmc directories and thumbnails created by WD on the My Cloud NAS
# KM 2014-07-05. V0.2 Added basic instructions
root="/nfs/shares/"
abort() { echo -e "\nAborted. Files \"wdmc_dirs\" \"totaldirs\" \"jpgs\" and \"jpegs\" NOT deleted.\nThey are located in $(pwd).\n";exit 1; }
fmt() { printf "%'d\n" $1; }
w() { echo ", please wait."; }
[[$EUID -ne 0]] && { echo "This script must be run by root. Aborted.";exit 1; }

cd "$root" || { echo -e "\n\nDirectory not found. Please ensure you are running the script on the My Cloud and, if necessary, change the cd command on line 4 of the script to match the mount point of your Data volume.\n";exit 1; }
 
clear;echo -e "This script searches for the \".wdmc\" directories and\nthe thumbnails created by the WD. If any are found you\nwill be shown how many, and the disk space they occupy.\nYou will then be given the opportuntiy to remove them.\n\nYou need to run it as root on the NAS server via SSH.\n\nYOU NEEDTO CHANGE LINE 4 OF THIS SCRIPT TO MATCH THE\nMOUNT POINT OF YOUR DATA. If you haven't done this yet,\ndo it now and run the script again.\n\nThis script is supplied strictly \"as is.\"\n";read -p "Press the Enter key to start, <CTRL-C> to abort."
clear;echo -n "Searching for \".wdmc\" directories";w
>wdmc_dirs
find "$root" -type d -name ".wdmc" >wdmc_dirs
dirs=$(wc -l<wdmc_dirs)
[[! -s wdmc_dirs]] && { echo "No directories named \".wdmc\" found."; rm wdmc_dirs;exit 0; }
echo "Found $(fmt $dirs) directories."

echo -ne "\nLooking for subdirectories";w
>totaldirs
while read line; do find "$line" -type d -mindepth 1 >>totaldirs; done<wdmc_dirs
totaldirs=$(wc -l<totaldirs)
echo "The .wdmc directories contain $(fmt $[$totaldirs-$dirs]) subdirectories, making a total of $(fmt $totaldirs)."

echo -ne "\nLooking for jpg thumbnails";w
>jpgs
while read line; do find "$line" -type f -name "*.jpg" >>jpgs; done<wdmc_dirs
jpgs=$(wc -l<jpgs)
echo "The directories contain $(fmt $jpgs) jpg files."

echo -ne "\nLooking for jpeg thumbnails";w
>jpegs
while read line; do find "$line" -type f -name "*.jpeg" >>jpegs; done<wdmc_dirs
jpegs=$(wc -l<jpegs)
echo -e "The directories contain $(fmt $jpegs) jpeg files.\n\nA total of $(fmt $[$jpegs+$jpgs]) thumbnails found.\n"

echo -n "Calculating space that would be freed";w
subt=0
while read line; do size=$(du -ac "${line}" | tail -1 | cut -f 1); subt=$[${size}+${subt}]; done<wdmc_dirs
echo -e "Done. Disk space used is $(fmt $[${subt}/1024])KB.\n"

read -rp "Do you want to remove these directories and all their contents [y/n]?" ans
[[$ans =~ (y|Y)]] || abort

read -r -p "Are you sure? [y/n] " ans
[[$ans =~ (y|Y)]] || abort

echo -n "Deleting $jpg files";w
while read line; do rm -r "$line"; done<wdmc_dirs
rm wdmc_dirs totaldirs jpgs jpegs
echo "Done"

Re my last post - sadly, the image I uploaded says it is “AWAITING APPROVAL.” My apologies for this.  

Easy way to delete all “.wdmc” folders:

find /DataVolume -name ".wdmc" -type d -exec rm -r {} +

* Its scans all Data folders and remove .wdmc folders

Also good thing to delete “thumbnails.db” files (Windows thumbnails database. Same as .wdmc, but only for windows, generated by Windows :D)

How to set, so the WD MYDrive go to slepp move when not in use…  i have installed Clean debian and OpenMediaVault on WDMyCloud!

I only use FTP & Mini DNLA on the Server…

When i check the drive, i can hear it’s running, Green Light…

DKBjorn wrote:

 

When i check the drive, i can hear it’s running, Green Light…

Sleep mode works perfect, but i dont add support for led color change.

Maybe i add it later.

I dont want to change LED light…   just say it was Green…

how does sleep mode work then…?

every time i listen to the drive it’s spinning and it’s warm…  

Maybe it the cooling  fN there runnng, but why then…?

with original soft it didnt run…

Fox_exe wrote:> Sleep mode works perfect, but i dont add support for led color change.

Maybe i add it later.

Hi Fox_exe.

I am really interested in this. 

I have installed the clean debian from your image in November. Back then, disk was never going offline. 

I do not want to reinstall debian it works very well.

Could you share how can i configure sleep mode?  

Should i stop some services? Mount  folders to ramfs…? Something else?

Thanks

I don’t remember what exactly I did of all this, but I began from this: http://info4admins.com/tips-to-spindown-your-hard-disk-in-debian-or-ubuntu/

Hmm… I can miss Ramlog package. Check it is installed: /etc/init.d/ramlog status

More info:  http://www.tremende.com/ramlog/

After installing the OMV 1.12 using the install.sh script, I cannot open the GUI. 404 Not Found is the message. I can ssh into the MyCloud though. Thanks for any help.

Oh, I just logged in after a few more tries. Sorry and thank you.

kowloon_chan wrote:

Oh, I just logged in after a few more tries. Sorry and thank you.

Next time - clear browser cache first :smiley:

Fox_exe wrote:

Autoinstall script makes 2nd raid and install .img in hem. Old raid is accesable too (If you dont run “Run_me_after_reboot.sh”).

For switch betwin 2 OS’s - just flash config_mdx.img

cat /proc/cmdline | awk -F= ‘BEGIN{RS=" "}{ if ($1==“root”) print $2 }’

ITs show you what of MD’s yoused now. (Flash config for another md to switch betwin os)…

 

Hi,

When you say “flash config_mdx.img”, if I’m using md0 now, does it mean to do a dd command for md1? Then do I just reboot?

Thanks.

Hi Fox_exe,

thaks for link, now it works. Disk is spinning down just well.

Here is the list of steps for the others:

  1. add noatime for  ext3 and ext4 in fstab 

  2. Install ramlog. Make sure that it is executed as a first service. I had a issue with that. Insserv was ignoring boot order of

services. 

  1. follow  http://info4admins.com/tips-to-spindown-your-hard-disk-in-debian-or-ubuntu/ to configure samba (also works for samba4)

  2. install and configure hdparms

  3. you are done

Note: Don;t be confused. Seems that WD has it’s own minimal spindown period. I do not find out what is the period. Anyway, if you set the spindown period too short, WD will follow own spindown period.

kowloon_chan wrote:> When you say “flash config_mdx.img”, if I’m using md0 now, does it mean to do a dd command for md1? Then do I just reboot?

Now - need upload old kernel too. (Because i use 3.2.40, in official - 3.2.26 + PageSize=64k)

Maybe you can use backup kernel from partition 6.

Hi fox_exe,

I saw you have this speed

Samba v4 + Comcerto/WD Patches + Tuning + PageSize=64kb = 80-105 MB/s

Can you give me how to config this?

Regards,

Hi all

I’m using Clean Debian and all I’ve installed is Syncthing - nothing else, not even OMV. I must have done something wrong because I have no swap space - and apparently no room to create it. Here is why I have arrived at this conclusion:

root@MyCloud:~# swapon -s
Filename Type Size	Used	Priority
root@MyCloud:~# free -m
             total used free shared buffers cached
Mem: 226 157 68 0 4 66
-/+ buffers/cache: 85 140
Swap: 0 0 0
root@MyCloud:~# parted -l
Model: ATA WDC WD20EFRX-68E (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
 3 15.7MB 528MB 513MB primary
 1 528MB 2576MB 2048MB ext3 primary raid
 2 2576MB 4624MB 2048MB ext3 primary raid
 5 4624MB 4724MB 99.6MB primary
 6 4724MB 4824MB 101MB primary
 7 4824MB 4826MB 1049kB primary
 8 4826MB 4828MB 2097kB primary
 4 4828MB 2000GB 1996GB ext4 primary


Model: Linux Software RAID Array (md)
Disk /dev/md0: 2048MB
Sector size (logical/physical): 512B/4096B
Partition Table: loop

Number Start End Size File system Flags
 1 0.00B 2048MB 2048MB ext3


Model: Linux Software RAID Array (md)
Disk /dev/md1: 2048MB
Sector size (logical/physical): 512B/4096B
Partition Table: loop

Number Start End Size File system Flags
 1 0.00B 2048MB 2048MB ext3
                                               
root@MyCloud:~#

 Does anyone know which partition(s) I can resize to make room for a swap file, or do I have to reinstall? That would mean loads of work, unfortunately…

Thanks in advance.

Run “/ruin_me_after_install.sh” or use this command:

swapon -f /dev/sda3

Thanks for the rapid reply, Fox. Unfortunately  I ran the wrong script - run_me_after_reboot.sh. Now I have a screen about disk arrays :

I have no idea how to exit from this. I should have just run the swapon command. I’m sorry, Fox. Please can you help me again?

Update:

I logged into the NAS on another terminal and I have the following at the end of the dmesg file:

[68216.516343] md1: unknown partition table

Looks like I will have a brick if I try to reboot it so I’m trying to copy all my data off it.

Scooby-2 wrote: 

[68216.516343] md1: unknown partition table

Its not a error. Same warning in official firmware (Mdadm tries to recognize md array as a physical disk instead of Partition)

About configuration screen when run my script - just press “Next” (Or choose ‘all/auto’)