Large File Problem eg. Backup-Jobs

LarsMaschke · February 18, 2011, 11:27am

Hello WD Community,

the My Book World drives keeps crashing when I tried to transfer large files like backup jobs. So I made some tests and I think I get it working by adding swap to the memory.

I did the following:

Activate the SSH Console.

I created a startup-file /etc/init.d/S99swap with this code:

#!/bin/sh

CREATE SWAP FILE

case “$1” in
    start)
        if [-f /swap.img] ; then
          echo “SWAP AVAILABLE” >> /tmp/messages
        else
          echo “SWAP NOT AVAILABLE - CREATED” >> /tmp/messages
          dd if=/dev/zero of=/swap.img bs=1M count=500
          mkswap /swap.img
        fi

        swapon /swap.img
    ;;
    stop)
        swapoff /swap.img
    ;;
    *)
     echo $“Usage: $0 {start|stop}”
     RETVAL=1
esac

Then I set the rights with:

chmod 755 /etc/init.d/S99swap

After a reboot everything looks OK and the backups work very well.

Maybe I can help someone with this workaround.

Greets from Germany

Lars Maschke

oxter · February 21, 2011, 8:00pm

Guten Tag!!!

Thank you for sharring this.

LarsMaschke · March 1, 2011, 11:53am

Furthermore there is another little trick

in /etc/init.d/S90smbd remove the “nice” command from smbd in the start section and simple use:

start)
        echo -n "Starting SMB services: "
        rm -f /var/spool/samba >/dev/null 2>&1
        ln -sf /DataVolume/jewab /var/spool/samba >/dev/null 2>&1
        chmod 1777 /var/spool/samba >/dev/null 2>&1
        /usr/sbin/smbd -D
        RETVAL=$?

I found out that this “nice” can crashes the drive.

doug809 · March 5, 2011, 1:07am

Lars,

Thank you very much for this post, it seems to have been the answer on my drive to the freezing/disappearing and probably everyone else’s too. After many hours/days troubleshooting this issue, I saw many of the solutions regarding disabling certain services (twonky, inetd, etc), checking/increasing swap files, etc … they helped a bit (presumably because they reduced resource usage), but they didn’t solve the problem.

The real problem seems all along to have been the smbd process being set to such a high priority (negative niceness). Your solution (which sets the process priority back to the default 0) really is the correct one I believe, and I noticed an immediate improvement in reliability and even speed. I could see the benefit by using the ‘top’ command to watch the processes, and now smbd never goes above 75% cpu. It’s kind of amazing actually, in testing I could simultaneously transfer several large files, play music from iTunes, stream a movie to Media player, browse thumbnails of a photo directory and have a few SSH sessions open.

I really suspect if WD makes this one simple change to the default configuration on the Mybooks, they will reduce their support costs tremendously!

Cheers,

Doug

jdz2k · March 6, 2011, 10:58pm

are you saying that right after the start) programming code above…you have what’s below?

stop)
echo -n "Shutting down SMB services: "

I dont want to remove too much…lol.

doug809 · March 7, 2011, 7:33am

Actually, there’s only 1 line to change (the one in bold in that code block above):

That is, find the line that says " nice -n -20 /usr/sbin/smbd -D"

and remove “nice -n -20” so that it just says " /usr/sbin/smbd -D"

After saving the file, type “reboot” and the new setting will take effect after the WD finishes restarting.

jdz2k · March 7, 2011, 9:05pm

ok thanks. ive implemented this and disabled mionet as it definetely causes my book to freeze up also.

030811 update: the 64GB upload was a success!! i think we may be on to something here. here is a snapshot of what i did.

followed LarsMaschke post to remove “nice -n -20” so that it just says " /usr/sbin/smbd -D" in /etc/init.d/S90smbd.

link: http://community.wdc.com/t5/My-Book-World-Edition/Large-File-Problem-eg-Backup-Jobs/m-p/148222#M4093

followed Vlado post to disable everything mionet (I didnt do mv /etc/init.d/S97twonkyserver /root which deals w/ twonky) and i also edited /usr/mionet/monitorCVM.sh to make Mionet obey startup flag.

link: http://community.wdc.com/t5/My-Book-World-Edition/Drive-Becomes-Un-Accessible-After-short-use-II/td-p/38659/highlight/true

reboot through putty and/or web interface.

i had undone everything else i’d tried on the forums here so truly mionet is the only thing disabled.

arzach · March 8, 2011, 3:36pm

I’ve removed the “nice” option too, I’ve disabled everything except Twonky and reduced its buffer size… but my 2T Mirror MBWE is still freezing after some time.

And I do not transfer big files, just listen some mp3 from twonky and then left it for a couple of days.

jdz2k · March 8, 2011, 7:29pm

uh oh. that isnt good news at all arzach. ill wait until it happens and report back.

doug809 · March 9, 2011, 5:29am

@arzach

If you need twonky, maybe the same “nice” logic will work there too?

Go to /etc/init.d/
Edit “S97twonkyserver”
Find the following line and remove the “nice -n -20” part

nice -n -20 “$TWONKYSRV” -D -appdata /DataVolume/jewab -logfile /DataVolume/jewab/twonkymedia.log

stop & start the twonky server (or simply use the reboot command to restart the mybook)

(Fingers crossed).

arzach · March 9, 2011, 9:05am

ok thanks!!!

removed nice from twonky.

now let’s see…

jdz2k · March 9, 2011, 3:51pm

i just checked and my twonkyserver didnt have a nice option in front of it. its 01.01.18 f/w. hopefully this will do the trick for ya arzach. im still holding on to hope myself.

arzach · March 10, 2011, 2:20pm

update: freezed again.

I’ve read lots of files for two days, with samba and twonky, kepping an eye on the resources (top command in the shell).

I’ve seen that after twonky start the memory runs very very low but it never ends (there is a process called ksoftirq that I guess it does that job).

Anyway, I’ve read some mp3 this morning (through samba), and then stop them… then after one hour the disk was freezed again.

It does’t freezed when it’s in use… just when I’m looking somewhere else!

doug809 · March 11, 2011, 1:26am

arzach

I’m not sure what the difference in our configrations is exactly, but i turned the twonky back on and ran some load tests.

twonky streaming mp3s (also tried video) from the mybook
windows media player streaming an AVI from the mybook
quicktime streaming an MOV from the mybook
iTunes playing audio from the mybook
Copying a 5GB folder from the mybook to my local drive
SSH session with top command running
Random browsing on mybook from Windows Explorer

After 2.5 hours, still couldn’t cause anything to crash. The video on the twonky stream stopped sometimes, but audio kept going. That was the only real problem, the other video streams kept going fine.

The top command would typically alternate between twonky & smbd taking the most resources.

Interestingly, I downloaded and installed “Twonky Manager” so that I could use the playlists instead of playing mp3’s individually, but when i tried to connect to the Mybook, it said “Your server appears to be busy, please check the server and try again”

So … I checked my config and have indicated everything i think might be relevant below, but if we can’t spot the difference, then I think i’m out of ideas. :mansad:

Firmware Version 01.02.06 (are you using the same firmware?)

System Uptime 6 days, 19:08

Advanced | Network | Services

UPnP for Web Access & FTP = Off

Web Access HTTP/HTTPS = Default

FTP = off

NFS = off

AFP = off

Disks configured as JBOD (no RAID)

HDD Standby = off

TOP command …

Mem: 124472K used, 2356K free, 0K shrd, 64K buff, 92124K cached

Load average: 0.43, 0.42, 0.22 (State: S=sleeping R=running, W=waiting)

PID USER STATUS RSS PPID %CPU %MEM COMMAND

3927 nobody S 3848 3410 20.2 3.0 smbd

59 root SW< 0 2 2.1 0.0 kswapd0

5117 root R 804 4016 1.9 0.6 exe

3418 root S 1760 1 0.1 1.3 nmbd

4014 root S 1644 3366 0.1 1.2 sshd

3593 root S 6864 3518 0.0 5.4 twonkymediaserv

3595 root S 6864 3518 0.0 5.4 twonkymediaserv

3585 root S 6864 3518 0.0 5.4 twonkymediaserv

5312 root S 6864 3518 0.0 5.4 twonkymediaserv

3518 root S 6864 3502 0.0 5.4 twonkymediaserv

4135 root S 6864 3518 0.0 5.4 twonkymediaserv

3502 root S 6864 3499 0.0 5.4 twonkymediaserv

3580 root S 6864 3518 0.0 5.4 twonkymediaserv

3590 root S 6864 3518 0.0 5.4 twonkymediaserv

5307 root S 6864 3518 0.0 5.4 twonkymediaserv

3573 root S 6864 3518 0.0 5.4 twonkymediaserv

3582 root S 6864 3518 0.0 5.4 twonkymediaserv

4134 root S 6864 3518 0.0 5.4 twonkymediaserv

4136 root S 6864 3518 0.0 5.4 twonkymediaserv

3814 root S 3500 1 0.0 2.7 service_ctcs.ph

5186 root S 3224 3410 0.0 2.5 smbd

3841 root S 3168 3840 0.0 2.4 dn_scd.php

/etc/init.d # ls

S01auto_mountd S9iTune

S10portmap S9lltd

S40network S9mDNSResponder

S50sshd _S60inetd

S55mini_httpd _SK01controlled_power_down

S85ntpdated _SK02upgrade

S90smbd _SK50sshd

S94fuse _SK80nfsd

S95windbindd _SK85ntpdated

S96wixalarm _SK91atalk

S97twonkyserver _SK9C_cds

S98logrotate _SK9iTune

S99sxminit rcH

S9M_mionet rcS

S9WiAutoConfig

I’m not sure if there’s anything else you can turn off in that list above (aside from twonky lol)?

… And just in case the client software is causing a problem for you, I am running Win 7 and Google Chrome for the browser.

Doug

ShaileshJKumar · March 16, 2011, 12:31am

Where would this create the swap file. I did exactly as you said in the post and it earsaed the attached USB disk completly.

arzach · March 26, 2011, 12:38pm

thank you doug809, my set up is very similar to yours, but i’m still having problems if twonky is running.

I’m giving up becouse I’m really fed up with this buggy product

I will forget twonky and I will use my disks only for lan backup.

and I will never ever buy a WD products again.

jdz2k · March 26, 2011, 3:24pm

I have had success with the changes i made and have been running for at least 28 days except for being through a couple power outages(they only lasted for a minute…weird). Im FINALLY able to use this NAS as a regular NAS!!!

Sorry to hear your situation arzach. i had given up for months and was a powercycling fool but came back a month ago and finally feel my solution has arrived!!

philbell · March 28, 2011, 2:55am

Excellent solution!!!

I had the same problem - whether or not there was any load on the system. Just leaving it standing there caused a hang after 2 or 3 hours.

I did these things:

firmware upgrade
remove the nice part of the smbd and twonky daemon commands in init.d
used the exact swap script quoted in the original post.
reboot.

It’s been running for 8 hours of backup. The web interface response time is MUCH quicker now too!

I hope it lasts. If I don’t post anything else on this topic, that means the drive is performing well.

UPDATE

One month of UNINTERRUPTED uptime.

Briswolf · March 28, 2011, 10:23am

Hi LarsMaschke,

This looks like a good idea but I am wondering where exactly the root filesystem is stored? Is it on the hard disks or is it in some flash memory?

If it is in the flash then using it for swap is probably not a great idea as it could cause the flash to wear faster if there is high swap usage.

I have no idea though where / lives physically and I don’t really know how to find out. I think I will try your solution here and put the swap file in my data store. Might be a bad idea to put the swap on a RAID 1 volume. I’m not sure.

Sadly I have stumbled to these forums only 16 hours after buying my new MBWEII from the shops, imagine my disappointment Maybe I should just get a refund. I used to trust WD products

Wayne

doug809 · March 30, 2011, 5:18am

@arzach

Sorry to hear it didn’t work out and i feel bad because i know it is possible eventually (i haven’t had to reboot since we last spoke).

System Uptime

19 days, 6:15

Is there any chance there was a typo or something in one of the fixes you tried? Maybe WD will swap your box and you’d have better luck with a replacement unit (after modifiying the ‘nice’ option again of course)?

Hopefully WD will learn from this!

Doug