HELP: UFRAW-BATCH running for 9 days straight. Device completely unusable

My Cloud OS 3 Personal & Network Attached Storage My Cloud
41

Did you really expect WD to fix anything?

42

Yeah; no-one could have anticipated that response, could they
?

:wink:

43

thanks sergey and paranoia
SSH is too advanced for me 

too bad that WD doesn’t update the firmware for V4, i think in fact they have obsoleted V4 already
i have completely no idea what mycloud is doing


44

Of course I was. I have read in this forum a lot of people returning their devices because of this same problem. They are losing costumers because of this. I would care about it. If WD doesn’t fix, who will?

45

Do people not having plex server, also facing this fdraw issue?

46

This problem started for me today.
I do not have Plex server.

Mem: 973472K used, 63840K free, 0K shrd, 28736K buff, 373152K cached
CPU:  0.1% usr  3.3% sys 96.3% nic  0.0% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 3.04 2.85 2.57 6/201 18038
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 5669     1 root     S    83872  8.0   1 36.6 /usr/local/wdmcserver/bin/wdphotodbmerger
18013 18007 root     R N   110m 10.8   0 32.6 /usr/local/wdmcserver/bin/ufraw-batch --silent --create-id=also --out-type=jpeg --out-depth=16 --output=/shares
18014 18006 root     R N   110m 10.8   0 28.7 /usr/local/wdmcserver/bin/ufraw-batch --silent --create-id=also --out-type=jpeg --out-depth=16 --output=/shares
 4923     1 root     S     4992  0.4   1  0.7 system_daemon
...
47

Same problem here.

I spent some time today investigating the ufraw-batch process and tried to find a way to bypass it for cr2 files, but didn’t get very far. I “may or may not have” experimented with unmapping the cr2 association in the imagemagick policy.xml file in a clone of /usr/local/modules/localwdmcserver (since that’s write-protected somehow) and “may or may not have” gotten imagemagick to return with failure (unidentified file type) rather than sitting in delegated ufraw-batch forever. Don’t know how to make such a change to the active localwdmcserver directory. I don’t know how the write protection works; it’s more complicated than the usual file attributes that you can usually change with chmod. So I gave up before trying anything that would almost certainly compromise my warranty.

Instead, I manually interrupted the pair of hung ufraw-batch processes and saw that ufraw-batch was reinstantiated twice, this time with convert accessing the next two cr2 files. Cool, so the background processing moves on even if it fails to convert any single image. I wrote a script to play whack-a-mole with the ufraw-batch processes. It gives each instantiation of ufraw-batch 2 minutes to try and complete nominally before interrupting it and waiting for the next instance. That 2 minutes is a guess; I have no idea it’s too much or too little. Haven’t figured out how long ufraw-batch takes to process a cr2 file when it actually completes successfully.

The thumbnail generation processing had been stuck on the first pair of cr2 files for days if not weeks. I’ve just nudged it forward 180 cr2 files over three hours. So far so good.

Posted the script on github if anyone wants to try it (at your own risk, blah blah):

48

Finally, a day and a half later, my My Cloud Mirror finished generating thumbnails with the help of this script nudging the processing through the cr2 processing, and now my NAS is perfectly silent when I’m not directly accessing files.

The cloud interface shows blank thumbnails for all of my cr2 files. I had hoped that the thumbnail generation would have worked for some of my cr2 files (those shot in standard/landscape orientation) but it appears that it did not. It’s possible that thumbnail generation would have succeeded for some files if I had configured the script to allow more than 2 minutes before interrupting.

In any case, now that the background processing is done (for now), I’ll just need to remember to start the script again when I push more cr2 files to the server.

49

The “read only”-ness of that folder has to do with it being a cramfs container that gets mounted at early system init. There is MUCH to despise about the way WD decided to handle the Mycloud gen2’s file system.

At best you can abuse fox_exe’s wdcrack to give you a user-level init to replace that file with a symlink on persistent storage. (changes to cramfs mounts are backed by tmpfs, and are lost on reboot.) If you do this, be sure to halt then restart the scan daemon.

50

Hi,

I’m really interested in trying your script, this is the least one would expect from a decent code - if a process times out processing a file, just skip it
 Not so hard but apparently WD doesn’t really care. Your script seems to be the only real solution for us so far!

Anyway, our friend @PEMAMETAL suggests here that it takes about 5 minutes per CR2 picture. Would you consider trying to make the timeout 10 minutes instead of 2 minutes? I personally don’t mind waiting a few days to get all the scannable pictures scanned (as opposed to waiting forever!) My device has been stuck with those ufraw processes for more than a month now (literally!) I’m a photographer and I bought this device specifically for back up but now it’s extremely slow and almost useless
 And their “engineers” suggest we zip our CR2 files 
 Total joke!

I have zero knowledge in writing code, but could you please explain this (from your script description) :
“The effective allowable run time for each process is dictated by the check period and allowed number of cycles, as follows:
allowableDuration = waitSecBetweenChecks * persistence
i.g. if persistence==4 and waitSecBetweenChecks==30 then allowableDuration=120 sec”

What is the persistence in this case? What I understood is that you allow 4 cycles of checks, each cycle takes 30 seconds. Is that correct? So what happens if an image is processed in under 2 minutes? Is the process stuck there until the 2 minutes are complete? I.e. does this method disable automatic progression to the next file? This is what I inferred when you said here “That 2 minutes is a guess; I have no idea it’s too much or too little” Because if auto progression to the next file is allowed, 2 minutes cannot be “too much” since the process will just go to the next file if the first is processed in <2 minutes, correct? So in short, my question is: does this script disable automatic progression of the ufraw process and restricts it to the time you specify (e.g. 4x30 sec.) ?
An ideal case would be for the script to let the ufraw process work with automatic progression and only terminate the process when it is stuck trying to process the same file for an x amount of time. I’d suggest to make that time 7~10 minutes instead of 2.
A final question, would this also affect generation of thumbnails from JPEG files too? Or is the ufraw (as the name suggests) only deals with RAW files?

In summary, it would be super to let the WD handle thumbnail generation normally (as it does that well for JPEG) and ONLY kill the ufraw-batch process(es) if stuck at the same file(s) for 10 minutes. Please let me know what you think.

Thanks a lot everyone, hopefully we’ll reach a viable solution soon!!

51

Hey man.

Let me contribute with my opinion. In my case, even the CR2 files that the ufraw-batch process could get over, I didn’t get thumbnails of any of them. So it is a useless process , the most efficient solution would be kill the ufraw-batch process and let the WD My Cloud scan just music and .jpg

53

Do you mean the “problematic” CR2 files (in portrait orientation)? Yeah those ones should be skipped altogether. However, it would be nice to still have at least the landscape photos scanned with thumbnails. Or did you mean that WD couldn’t generate any thumbnails for any CR2 file at all? In such case, why would it be stuck only on the “portrait” ones?

So I did run the script after modifying the “waitSecBetweenChecks” to 150 seconds. Surprisingly (?) ufraw-batch did not restart after it was killed! Even more surprising is that I went to the directory where the "problematic’ files were located and they had thumbnails! I’m using a mac, so could those thumbnails be the computer-generated ones? Is there such a thing? If so, are the thumbnails generated by WD itself only for their app/web
etc ? In such case, it would be really nice to kill all instances of thumbnail generation and just be over with it since 90% of the time I only use my computer to access those files. I am aware that the OS normally generates thumbnails for all kinds of local files but I don’t know how that works with a NAS device.

Alternatively, could one actually use a computer program that generates the same kind of thumbnails that WD would, and store them at the same location? Or is that directory unaccessible? I have a raspberry pi laying around, if this would be possible maybe I can run a script through it and let it handle the thumbnail generation on behalf of the WD cloud after killing all the native thumbnail processes. Needless to say, I have zero idea if that is possible, or how to do it if it is! Just an idea, if someone wants to try!

54

Thanks guys for keeping this discussion going.

Wierd_w, thanks for the explanation of the read-only behavior. Sounds like there is a potential workaround that someone could work out based on wdcrack. I currently have a valid warranty that I’m trying to keep intact and the wdcrack option feels like a step too far, although I may be kidding myself that WD would not claim that my warranty is already void by having ran anything at all over ssh. I’ll keep this option in mind and may look into it later.

@astrolabe, thanks for the pointer to @PEMAMETAL’s comment about the thumbnail generation nominally taking ~5 mins to complete. I’ve read that post many times but missed that part. I could experiment with this today and post back later if it’s worthwhile. The script basically checks for active ufraw processes at a regular interval (waitSecBetweenChecks) and counts how many times each active PID has been observed during previous check cycles. If that count for any of the active PIDs exceeds the “persistence” parameter, then the script kills that PID. If an instance of ufraw finishes in under 2 minutes (or waitSecBetweenChecks*persistence) then the script does nothing since the count would not exceed the persistence threshold. In that case, the background processing will move on to the next file, and in theory, you’ll have a valid thumbnail for the cr2 file that ufraw finished processing. This script does not affect jpeg thumbnail generation since the background processing calls “convert” (imagemagick) which delegates conversion of various non-native file types to subordinate processes. The ufraw-batch processor is mapped to handle cr2, crw, and dng files, whereas I think the convert program handles jpegs natively and my script doesn’t pay attention to that.

By the way, sorry the script is so convoluted. It serves my intention of trying to allow ufraw to finish processing cr2 files nominally and aborting if the processing takes too long, although it hasn’t been tuned to allow the nominal case to succeed.

@PEMAMETAL, I’m not sure I understood your suggestion fully; I think you’re saying that it’s pointless for the script to wait for ufraw-batch to complete because in the end, none of the non-portrait cr2 files completed anyway, so we might as well simply kill every instance of ufraw-batch immediately? That makes sense, although I think the reason that thumbnail generation fails even for non-portrait cr2’s is that the script doesn’t give ufraw-batch enough time to finish processing even when ufraw is working correctly. As you noted previously, it takes ~5 mins for the background processing to nominally complete for non-portrait cr2 files. You can adapt this script to do precisely what I think you’re suggesting by setting persistence to 0 or 1 to kill each instance of ufraw immediately (within waitSecBetweenChecks seconds), if you’re not interested in having thumbnails for non-portrait cr2’s. Come to think of it, I like that approach because 5-10 mins of cpu/ram/disk/fan activity for each individual cr2 thumbnail is probably not worth it. If you do want thumbnails of your non-portrait cr2’s then we can tweak the persistence and waitSecBetweenChecks parameters to allow the processing enough time to complete nominally for files that ufraw is able to process.

Several options to consider:

  • Tune the ufraw monitor script to allow enough time for ufraw-batch to nominally finish
    • Requires experimentation to confirm 10 mins is sufficient.
    • 10 mins of processing per cr2 file is very expensive!
  • Interrupt processing of all cr2 files, per @PEMAMETAL’s suggestion
    • Set persistence to 0 in the script to interrupt all instances of ufraw-batch
    • Or write a much simpler version of the script to do this
      • This is my preference.
  • Interrupt all ufraw-batch instances AND run custom routine to generate thumbnails for cr2’s
    • May be possible to process internally with ufraw-batch using different parameters. Complicated.
    • Per @astrolabe’s suggestion, probably could generate thumbnails externally (i.g. using The Gimp) and push them to the NAS. If feasible, this would probably be the best, least-wear, solution if you need thumbnails of all of your cr2’s.
      • Good idea! But it won’t be as easy as just pushing the files to the server. Would need to update file/thumbnail mapping.
      • Thumbails are stored in a hidden directory with similar structure to your shares mapping:
        • For example, for cr2 file /shares/Volume_1/my_photos/raw/myphoto.cr2,
          there is a corresponding thumbnail here (with some type of unique 33 character hash):
          /shares/Volume_1/.wdmc/my_photos/raw/transcoded/myphoto.[unique_hash].png
      • And there must be some index somewhere that maps files to thumbnails, probably in wdmc.db in the same directory as the transcoded thumbnails.
      • Actually, maybe this isn’t too hard. When we interrupt ufraw-batch, the background processing generates a png file and probably still maps the invalid png thumbnail to the original file. It shouldn’t be difficult to get the list of (invalid) png’s with the unique hashes and rename our externally-generated thumbnails with these filenames.
  • Check dimensions of the cr2 file that ufraw is processing and interrupt only if height>width
    • Checking dimensions is easy but the script gets even more complicated
  • Override ufraw-batch mapping for cr2 files.
    • Probably requires wdcrack per Wierd_w’s suggestion
    • Requires deeper understanding of imagemagick delegation

I just moved some more cr2 files to my NAS, both in portrait and landscape orientations. Oddly, ufraw-batch succeeded with both files in ~60 sec each. So, I don’t know why it fails sometimes. I wonder if spaces in path names might be part of the problem when ufraw-batch fails, or something else that we haven’t thought of.

55

Yes that is exactly what I mean. I didn’t get thumbnails at all. The ufraw-batch process can get over the CR2 on landscape, but no thumbnails are generated. I don’t know why it gets stuck only on the portrait ones, but that is what I found out.

If you are seen thumbnails on Mac finder, those are thumbnails generated by your Mac. If you want to see WD thumbnails you have to access by web www.mycloud.com, or by the mobile app.

If you use only your computer to access files and you do it inside your home network, there is no need to even enable cloud access, because you are at local network, your are accessing it via afp. I think you are confusing some things. The WD thumbnails are created to be seen on the mobile app, and when you are remotely accessing your device on any computer in the world by www.mycloud.com.

56

Thanks @ryan_k and @PEMAMETAL for the explanations.

It is funny that I had looked into this ufraw issue somewhat extensively but I completely missed the part about those thumbnails being for cloud access ONLY! -Thinking about all the time I had wasted on nothing
 But to add insult to injury, I remember that I had to enable cloud access to use WD Sync which was the ONLY way to shut down my device using a Mac!! (Aside from SSH which I didn’t want to use at the time) When I contacted WD support I was told that there is no option to shutdown since “being network-based, they are designed for uninterrupted use over the network.” (!!!) Off topic, but I had to vent 

Anyway, I will disable cloud access for now and see how it goes.

@ryan_k if that method for externally generating thumbnails actually work I might try it in the future, I may need to access my files remotely at some point so if we get this fixed then why not! It does look quite involved though.

57

Hello. I think I was not clear in my reply (Sorry I am from Brazil, I am not a native English speaker). What I am trying to say is that I am not using any script. I am talking about the native UFRAW-batch process, in my analysis what I found out is that the CR2 file being scanned by the UFRAW changed about every 5 minutes if it is a landscape photo, but if it is a portrait it gets stuck forever. After that diagnoses, I converted every portrait CR2 file to DNG so the UFRAW-batch could complete, BUT in the end, after all the process was complete there was no thumbnails at all, no CR2 thumbnails and no DNG thumbnail, but at least the device was quiet. Thats what I am trying to tell you. I think no matter what script you try, this device seems unable to generate thumbnails for RAW files. I searched for information at the time and RAW files are not on the list of compatible files.

This is what I get for all CR2 or DNG files.

11
112880×1800 824 KB

58

Did you check if the thumbnail were generated? or just finished the process? As I stated in my last reply, I was never able to get thumbnails from RAW files. It doesn’t matter if it is CR2, DNG, portrait or non portrait. I accepted the fact I won’t have thumbnail for RAW files a long time ago. And to be honest, it doesn’t matter because the files I really want thumbnails are the equivalent jpg’s. The CR2 are just to store and eventually use again on Adobe Lightroom if needed. The big pain in the ass to me is the slowness and ■■■■■■ performance of the device when the UFRAW-batch gets stuck on portrait CR2, and then you have to zip them or convert each one to DNG. That takes a lot of work and time. I am seriously thinking of selling this NAS and get a MY cloud HOME, WD support replied me saying that they don’t use UFRAW-batch, and they are CR2 compatible.
https://support.wdc.com/knowledgebase/answer.aspx?ID=18573

59

Where is ufraw-batch located? I can’t find it on either of my gen1 or gen2 My Clouds. Is it part of some other
package?

60

@PEMAMETAL, Oh I see. Sorry I misunderstood what you meant by the thumbnails not being generated correctly even for non-portrait raw files. Your English is perfectly fine, I just misread your post.

I just checked on those cr2 files that I mentioned ufraw was able to finish processing within 1 minute each. You’re right, even though the processing finished reasonably quickly, the cloud interface does not show valid thumbnails for any cr2 files whatsoever, so there’s no point allowing ufraw to run.

I rewrote my script to simply kill each instance of ufraw-batch as soon as it finds it.

1 Like
61

@rac8006

Where is ufraw-batch located? I can’t find it on either of my gen1 or gen2 My Clouds. Is it part of some other
package?

On my Gen 2 My Cloud Mirror, the ufraw-batch executable is located here:
/usr/local/wdmcserver/bin/ufraw-batch

Is that what you meant?