Large File Problem eg. Backup-Jobs


#1

Hello WD Community,

the My Book World drives keeps crashing when I tried to transfer large files like backup jobs. So I made some tests and I think I get it working by adding swap to the memory.

I did the following:

Activate the SSH Console.

I created a startup-file /etc/init.d/S99swap with this code:

!/bin/sh

CREATE SWAP FILE

case "$1" in
    start)
        if [-f /swap.img] ; then
          echo "SWAP AVAILABLE" >> /tmp/messages
        else
          echo "SWAP NOT AVAILABLE - CREATED" >> /tmp/messages
          dd if=/dev/zero of=/swap.img bs=1M count=500
          mkswap /swap.img
        fi

        swapon /swap.img
    ;;
    stop)
        swapoff /swap.img
    ;;
    *)
     echo $"Usage: $0 {start|stop}"
     RETVAL=1
esac

Then I set the rights with:

chmod 755 /etc/init.d/S99swap

After a reboot everything looks OK and the backups work very well.

Maybe I can help someone with this workaround.

Greets from Germany

Lars Maschke


#2

 Guten Tag!!!!

Thank you for sharring this.

:smiley:


#3

Furthermore there is another little trick

in /etc/init.d/S90smbd remove the "nice" command from smbd in the start section and simple use:

start)
        echo -n "Starting SMB services: "
        rm -f /var/spool/samba >/dev/null 2>&1
        ln -sf /DataVolume/jewab /var/spool/samba >/dev/null 2>&1
        chmod 1777 /var/spool/samba >/dev/null 2>&1
        /usr/sbin/smbd -D
        RETVAL=$?

I found out that this "nice" can crashes the drive.


#4

Lars,

Thank you very much for this post, it seems to have been the answer on my drive to the freezing/disappearing and probably everyone else's too.  After many hours/days troubleshooting this issue,  I saw many of the solutions regarding disabling certain services (twonky, inetd, etc), checking/increasing swap files, etc ... they helped a bit  (presumably because they reduced resource usage), but they didn't solve the problem.  

The real problem seems all along to have been the smbd process being set to such a high priority (negative niceness).  Your solution (which sets the process priority back to the default 0) really is the correct one I believe, and I noticed an immediate improvement in reliability and even speed.  I could see the benefit by using the 'top' command to watch the processes, and now smbd never goes above 75% cpu.  It's kind of amazing actually, in testing I could simultaneously transfer several large files, play music from iTunes, stream a movie to Media player, browse thumbnails of a photo directory and have a few SSH sessions open.

I really suspect if WD makes this one simple change to the default configuration on the Mybooks, they will reduce their support costs tremendously!

Cheers,

Doug


#5

are you saying that right after the start) programming code above...you have what's below?

 stop)
        echo -n "Shutting down SMB services: "

I dont want to remove too much...lol.


#6

Actually, there's only 1 line to change (the one in bold in that code block above):

That is, find the line that says " nice -n -20 /usr/sbin/smbd -D"

and remove "nice -n -20" so that it just says " /usr/sbin/smbd -D"

After saving the file, type "reboot"  and the new setting will take effect after the WD finishes restarting.


#7

ok thanks. ive implemented this and disabled mionet as it definetely causes my book to freeze up also.

030811 update: the 64GB upload was a success!! i think we may be on to something here. here is a snapshot of what i did.

1) followed LarsMaschke post to remove "nice -n -20" so that it just says " /usr/sbin/smbd -D"  in /etc/init.d/S90smbd.

 link: http://community.wdc.com/t5/My-Book-World-Edition/Large-File-Problem-eg-Backup-Jobs/m-p/148222#M4093

2) followed Vlado post to disable everything mionet (I didnt do mv /etc/init.d/S97twonkyserver /root which deals w/ twonky) and i also edited /usr/mionet/monitorCVM.sh to make Mionet obey startup flag.

link: http://community.wdc.com/t5/My-Book-World-Edition/Drive-Becomes-Un-Accessible-After-short-use-II/td-p/38659/highlight/true

3) reboot through putty and/or web interface.

  

i had undone everything else i'd tried on the forums here so truly mionet is the only thing disabled.


#8

I've removed the "nice" option too, I've disabled everything except Twonky and reduced its buffer size.. but my 2T Mirror MBWE is still freezing after some time.

And I do not transfer big files, just listen some mp3 from twonky and then left it for a couple of days.


#9

uh oh. that isnt good news at all arzach. ill wait until it happens and report back.


#10

@arzach

If you need twonky, maybe the same "nice" logic will work there too?

  1. Go to  /etc/init.d/

  2. Edit "S97twonkyserver"

  3. Find the following line and remove the "nice -n -20" part

 nice -n -20 "$TWONKYSRV" -D -appdata /DataVolume/jewab -logfile /DataVolume/jewab/twonkymedia.log

  1. stop & start the twonky server (or simply use the reboot command to restart the mybook)

(Fingers crossed). :smile:


#11

ok thanks!!!!

removed nice from twonky.

now let's see..


#12

i just checked and my twonkyserver didnt have a nice option in front of it. its 01.01.18 f/w. hopefully this will do the trick for ya arzach. im still holding on to hope myself.


#13

update: freezed again.

I've read lots of files for two days, with samba and twonky, kepping an eye on the resources (top command in the shell).

I've seen that after twonky start the memory runs very very low but it never ends (there is a process called ksoftirq that I guess it does that job).

Anyway, I've read some mp3 this morning (through samba), and then stop them.. then after one hour the disk was freezed again.

It does't freezed when it's in use.. just when I'm looking somewhere else!

:frowning:


#14

arzach

I'm not sure what the difference in our configrations is exactly, but i turned the twonky back on and ran some load tests. 

  • twonky streaming mp3s (also tried video)  from the mybook
  • windows media player streaming an AVI from the mybook
  • quicktime streaming an MOV from the mybook
  • iTunes playing audio from the mybook
  • Copying a 5GB folder from the mybook to my local drive
  • SSH session with top command running
  • Random browsing on mybook from Windows Explorer

After 2.5 hours, still couldn't cause anything to crash.    The video on the twonky stream stopped sometimes, but audio kept going. That was the only real problem, the other video streams kept going fine.

The top command would typically alternate between twonky & smbd taking the most resources.  

Interestingly, I downloaded and installed "Twonky Manager" so that I could use the playlists instead of playing mp3's individually, but when i tried to connect  to the Mybook, it said "Your server appears to be busy, please check the server and try again"

So ... I checked my config and have indicated everything i think might be relevant below, but if we can't spot the difference, then I think i'm out of ideas.  :mansad:

Firmware Version 01.02.06  (are you using the same firmware?)

System Uptime 6 days, 19:08

Advanced | Network | Services

UPnP for Web Access & FTP = Off

Web Access HTTP/HTTPS = Default

FTP = off

NFS = off

AFP = off

Disks configured as JBOD (no RAID)

HDD Standby = off

TOP command ...

Mem: 124472K used, 2356K free, 0K shrd, 64K buff, 92124K cached

Load average: 0.43, 0.42, 0.22    (State: S=sleeping R=running, W=waiting)

  PID USER     STATUS   RSS  PPID %CPU %MEM COMMAND

 3927 nobody   S       3848  3410 20.2  3.0 smbd

   59 root     SW<        0     2  2.1  0.0 kswapd0

 5117 root     R        804  4016  1.9  0.6 exe

 3418 root     S       1760     1  0.1  1.3 nmbd

 4014 root     S       1644  3366  0.1  1.2 sshd

 3593 root     S       6864  3518  0.0  5.4 twonkymediaserv

 3595 root     S       6864  3518  0.0  5.4 twonkymediaserv

 3585 root     S       6864  3518  0.0  5.4 twonkymediaserv

 5312 root     S       6864  3518  0.0  5.4 twonkymediaserv

 3518 root     S       6864  3502  0.0  5.4 twonkymediaserv

 4135 root     S       6864  3518  0.0  5.4 twonkymediaserv

 3502 root     S       6864  3499  0.0  5.4 twonkymediaserv

 3580 root     S       6864  3518  0.0  5.4 twonkymediaserv

 3590 root     S       6864  3518  0.0  5.4 twonkymediaserv

 5307 root     S       6864  3518  0.0  5.4 twonkymediaserv

 3573 root     S       6864  3518  0.0  5.4 twonkymediaserv

 3582 root     S       6864  3518  0.0  5.4 twonkymediaserv

 4134 root     S       6864  3518  0.0  5.4 twonkymediaserv

 4136 root     S       6864  3518  0.0  5.4 twonkymediaserv

 3814 root     S       3500     1  0.0  2.7 service_ctcs.ph

 5186 root     S       3224  3410  0.0  2.5 smbd

 3841 root     S       3168  3840  0.0  2.4 dn_scd.php

/etc/init.d # ls

S01auto_mountd              S9iTune

S10portmap                  S9lltd

S40network                  S9mDNSResponder

S50sshd                     _S60inetd

S55mini_httpd               _SK01controlled_power_down

S85ntpdated                 _SK02upgrade

S90smbd                     _SK50sshd

S94fuse                     _SK80nfsd

S95windbindd                _SK85ntpdated

S96wixalarm                 _SK91atalk

S97twonkyserver             _SK9C_cds

S98logrotate                _SK9iTune

S99sxminit                  rcH

S9M_mionet                  rcS

S9WiAutoConfig

I'm not sure if there's anything else you can turn off in that list above (aside from twonky lol)?

... And just in case the client software is causing a problem for you, I am running Win 7 and Google Chrome for the browser.

Doug


#15

Where would this create the swap file. I did exactly as you said in the post and it earsaed the attached USB disk completly.


#16

thank you doug809, my set up is very similar to yours, but i'm still having problems if twonky is running.

I'm giving up becouse I'm really fed up with this buggy product

I will forget twonky and I will use my disks only for lan backup.

and I will never ever buy a WD products again.


#17

I have had success with the changes i made and have been running for at least 28 days except for being through a couple power outages(they only lasted for a minute...weird). Im FINALLY able to use this NAS as a regular NAS!!!

Sorry to hear your situation arzach. i had given up for months and was a powercycling fool but came back a month ago and finally feel my solution has arrived!!


#18

Excellent solution!!!

I had the same problem - whether or not there was any load on the system. Just leaving it standing there caused a hang after 2 or 3 hours.

I did these things:

1) firmware upgrade

2) remove the nice part of the smbd and twonky daemon commands in init.d

3) used the exact swap script quoted in the original post.

4) reboot.

It's been running for 8 hours of backup. The web interface response time is MUCH quicker now too!

I hope it lasts. If I don't post anything else on this topic, that means the drive is performing well.

UPDATE

One month of UNINTERRUPTED uptime.


#19

Hi LarsMaschke,

This looks like a good idea but I am wondering where exactly the root filesystem is stored? Is it on the hard disks or is it in some flash memory?

If it is in the flash then using it for swap is probably not a great idea as it could cause the flash to wear faster if there is high swap usage.

I have no idea though where / lives physically and I don't really know how to find out. I think I will try your solution here and put the swap file in my data store. Might be a bad idea to put the swap on a RAID 1 volume. I'm not sure.

Sadly I have stumbled to these forums only 16 hours after buying my new MBWEII from the shops, imagine my disappointment :frowning: Maybe I should just get a refund. I used to trust WD products :frowning:

  • Wayne

#20

@arzach

Sorry to hear it didn't work out and i feel bad because i know it is possible eventually (i haven't had to reboot since we last spoke).

System Uptime 19 days, 6:15

Is there any chance there was a typo or something in one of the fixes you tried?   Maybe WD will swap your box and you'd have better luck with a replacement unit (after modifiying the 'nice' option again of course)?

Hopefully WD will learn from this!

Doug