Can I add entries in there which will exclude an entire directory tree? If I can then that will solve my own SafePoint back-up issues in one go as I have large gigabyte files on there that DO NOT need backing-up.
SafePoint does use that exclusion when perform the rsync operations.
if it’s the standard rsync that’s used, than all you’d need to do should be to add the top folder of the tree you’d like to exclude, trailed by a “/”. From the rsync man page: The patterns can take several forms. The rules are: if the pattern starts with a / then it is matched against the start of the filename, otherwise it is matched against the end of the filename. Thus /foo would match a file called foo at the base of the tree whereas foo would match any file called foo anywhere in the tree. if the pattern ends with a / then it will only match a directory, not a file, link or device. if the pattern contains a wildcard character from the set *?[ then expression matching is applied using the shell filename matching rules. Otherwise a simple string match is used. if the pattern contains a / (not counting a trailing /) then it is matched against the full filename, including any leading directory. If the pattern doesn’t contain a / then it is matched only against the final component of the filename. Furthermore, if the pattern includes a double asterisk “**” then all wildcards in the pattern will match slashes, otherwise they will stop at slashes. if the pattern starts with "+ " (a plus followed by a space) then it is always considered an include pattern, even if specified as part of an exclude option. The "+ " part is discarded before matching. if the pattern starts with "- " (a minus followed by a space) then it is always considered an exclude pattern, even if specified as part of an include option. The "- " part is discarded before matching. if the pattern is a single exclamation mark ! then the current exclude list is reset, removing all previous exclude patterns. The +/- rules are most useful in exclude lists, allowing you to have a single exclude list that contains both include and exclude options. Here are some examples: --exclude “*.o” would exclude all filenames matching *.o --exclude “/foo” would exclude a file in the base directory called foo --exclude “foo/” would exclude any directory called foo --exclude “/foo/*/bar” would exclude any file called bar two levels below a base directory called foo --exclude “/foo/**/bar” would exclude any file called bar two or more levels below a base directory called foo --include “*/” --include “*.c” --exclude “*” would include all directories and C source files --include “foo/” --include “foo/bar.c” --exclude “*” would include only foo/bar.c (the foo/ directory must be explicitly included or it would be excluded by the “*”)
after editing the config file accordingly, I would also run the rsync command once with “–exclude-from ‘config file name’” and the “-n” option, which will produce a list of files rsync would synchronize without actually doing it.
One more thing to note, rsync will always treat the “–exclude” path as relative, meaning that when specifying “/dir1” or “/dir1/dir2”, etc., it will always look for “dir1” under the specified source directory rather than the actual root directory.
It only excluded the Publc share from the SafePoint backup but another folder named Public elsewhere got backed-up. Also surgically excluded the Goodmans firmware update directory tree.
I just ran a test, adding to your example a source directory “test” (which contains the folderstructure of your example) and a destination directory “testsync” as well as an exclude file “rsyncex” (in the same root folder as “test” and “testsync”) containing the line “dir3/”, the command
rsync -r --exclude-from rsyncex test/ testsync
would produce the following result in “testsync”:
dir1/dir2
dir5/dir6
The folder dir3 and all below (i.e. dir4) got excluded.
yes, that’s the correct result for the syntax applied.
If the first line in “exclude.txt” would read “Public/” instead (w/o the “/” ahead of “Public”), the result would be that any folder called “Public” anywhere in the directory tree of the source directory would get omitted.
Is the rsync command now working as you expect/need it?
Sussed it. The Syntax (tested it with dry runs) had worked flawlessly. It’s excluded ONLY the directories with their associated sub-directories and files from the rsync operation. I needed to specify an absolute path with the source being the root directory for the purpose of the rsync operation.
Folders named “Public” elsewhere get processed as normal. The “Public” share itself is 100% excluded and one directory and it’s sub-directories and files under the share “Repository” is also correctly excluded.
Perfect. Exactly what I want.
This means that static stuff I already have backed up on DVD and elsewhere does not get backed-up to my back-up NAS with is a 500Gb NAS with the MBL being 1Tb.
Tony’s advise, ages ago, was to purchase another 1Tb drive. I’ve flat refused to go along that path and have proven my theories were 100% valid and correct. I’m happy!
Now, it would be nice to have this simple functionality coded into the Dashboard UI so those uninitiated to an operating system command line can enjoy the same. It would be an awesome feature because then people will be able to buy smaller external drives to back-up ONLY what is essential to themselves.
Yes, being able to edit this via the Dashboard (kind of similar to defining the directories Twonky scans) would indeed be very helpful.
Until than, I’ll also be using the command line to (re-) configure my SafePoint backups. Reminds me of my good ol’ Linux days, which were actually years rather …
Worked perfectly. Now, how and where to edit /usr/local/NSPT/WDSafe/lib/CopyHandler.pm so when it performs the the dry run it also takes into account /etc/nas/NSPT/exclude.txt.
tjbbjcn wrote:
yes, that’s the correct result for the syntax applied.
If the first line in “exclude.txt” would read “Public/” instead (w/o the “/” ahead of “Public”), the result would be that any folder called “Public” anywhere in the directory tree of the source directory would get omitted.
Is the rsync command now working as you expect/need it?
. . . to get the dry-run to provide the correct statistics to the rest of the perl script.
I think I may have found a minor bug with SafePoint. Where the actual rsync that moves files does exclude certain files and directories that the rsync dry-run that’s performed before the live rsync processes and tallies everything.
I have a suspicion that even with my desire to exclude certain directories the dry-run will tally everything to be excluded and if the amount is greater than the capacity of the target then the SafePoint backup will not run even though there would be enough space on the target because of the exclusions.
I’m not exactly familiar with the way SafePoint works quite yet (and can’t check it out either, since I am currently sending back the device to have it replaced), but in principle, when it is possible to tweak the config files so rsync will take exclusions for the actual copying process, it should also be possible to get it to do the same for the dry run.
I guess SafePoint is written by WD, isn’t it? Any chance to talk to one of their developers to clarify your suspicion?
I tested the new rsync exclusions file using --dry-run. The bug I’ve fund is minor. There may be enough space on the target for the Safepoint but the dry-run will state there is not and prevent a Safepoint back-up from running.
Now, my exclusion of entire directory trees make the gap a lot wider. For example, the most resent Safepoint back-up excluded about 61Gb of data but the dry-run would have included that figure.
What I see is lazy programming and by profession I am a program and also quality control. It is a but of a worry where if there is laziness displayed there, where else has laziness been employed?
I recently seen the symptoms of another issue where files that the Apache2 uses were created in the root of the OS’s file system (all 0 bytes in length) and a file named /2 which is a STDOUT redirection that’s gone wrong. A scripting error in some other script.
W.D. needs a bit more quality enforced on it’s quality control.
Anyway, I now have Safepoint working exactly the way I want it to work. I just don’t need every single file backing-up. Just the stuff that matters. Why do I need to backup data that is already stored securely on other off-line drives, CDs and DVDs? (Existing on my MBL for the sheer sake of convenience.)
certainly agree with you there, and glad that you now have SafePoint working exactly the way you want it to work!
Once I’ve got my replacement MBL, I’ll be trying to make it work for my purpose as well (that is mainly getting it to accept certain exclusions). If it acts up too much or requires too much time to sort it all out, I’ll just resort to a more userfriendly alternative (ChronoSync, FoldersSynchronizer, Carbon Copy Cloner, etc.). Just don’t have the time at the moment to mess with it invinitely, even so I always have and really do enjoy fiddling with the (unix based/linux) system.
I’ve also read your post on these strange symptoms you discovered in the log. Sounds quite scary, and not like something that actually should happen on a correctly working system! Will be interesting to see how that experience develops.
The changes to the Safepoint back-up stuff has been so seriously minor to get it to exclude directories. To add one line to the perl script. Nothing else within the script is changed. If you have to edit anything else in the script then you’re doing something wrong. I actually copied and pasted the one line from another part of the same code so it’s sort of impossible to even make a typo!!!
As to the exclude.txt file, it’s simply adding the additional exclusion rules to the top of what’s already there.
That’s it. No more to that than that. The extras blurb in this thread was me testing/proving the theory without disturbing the official code and configuration. Obviously make a copy of the original files so you can put it back to the originals BEFORE performing a manual firmware upgrade from file for when W.D. release a firmware upgrade.
thanks for clarifying, and the advice on copying the config file. I’ll make sure I do that! First I need to wait for the replacement (of my MBL) though.
I’ve checked my modification to the perl script and exclude.txt. Also, thank-you for the assistamce. The Safepoint back-up works exactly as required and this is tested by an actual back-up.
Yes. The excude patterns are correctly processed and the addition to the perl script also works AND even corrects a bug that the W.D. firmware development team have missed.
I’ve also saved myself about 100 pounds NOT needing to get another NAS to match the capacity of the my MyBook Live. :robothappy:
