Theoretically it can be done. It will void your warranty to make the changes needed.
It is not something easy to implement though, not for the novice, and will work on the underlying filesystem, not with SAMBA.
I can confirm that [quotaon], [quotacheck] and [repquota] exist as commands on a gen2, and seem to work. [edquota] is missing, as is [quota]. This means you will have to manually create and maintain quota files using the built-in editor [vi]. Existence of repquota might allow you to set a cron job to alert you if a user has gone over the quota limit, but sendmail is not present on my gen2. Some enterprising people in the EX2 community figured out that you can use phpmailer to send emails with bash commands though.
At least on my Gen 2, userquota and groupquota are turned on as mount options for the persistent volume. This means they (WD) turned on the underlying FS support, but just arent putting it to use. Judicious use of [mount -o bind] on a non-shared folder, at an empty mount point using a user script, that passes mount options to turn on user quotas, with appropriate quota files, might give you what you want. (On a Gen1, just make appropriate pokes at /etc/fstab)
By setting linux FS quotas on the underlying file system samba gives access to (assuming samba is configured for the linux user accounts you set the quotas on…) you will effectively get what you want.
If you are not good at linux-fu, need to retain your warranty, or have vital data on your mycloud, DO NOT ATTEMPT.
If you choose to embark down that path on your own anyway, by all means, tell us how it goes-- but it is NOT a supported configuration, and I would not be surprised if there are un-forseen issues trying it.
I am now tempted to try turning it on (I have long since voided my warranty) and seeing if I can make it work with the stock firmware using something like fun-plug. (Since hey, I managed to get iscsi turned on, and all that jazz.)
