Hand Brake is great.
I use VidCoder myself (vidcoder.codeplex.com). Same engine, but I find it easier to use.
If Your source of FLV is YouTube, and your banwidth (internet) is not an issue, you should try and get hem again, as MP4 from YouTube - use JDownloader (jdownloader.org) as it gives all the options for a link (720p, 1080p, etc).
FLVs are already a very compressed format. Compressing it again even when using x264, just for the sake of changing the container (MP4) is not a quality option. I would not recoomend it just because “some” files skip/stutter. Same argument for XViD. Why? as WDTV-Live-SMP is quite capable of playing either just fine.
Using High-Profile (and x264 in general) makes sense for high quality material (DVD-MPG2, BluRay, 720P/1080p HDTV). If you have a couple of 700MB AVI that stutter, do you really want to degrade them (smaller is never better) just so they play over WiFi?
I have observed that 720p AVI (XViD/DivX) work better from a MKV container.
To make MKV, just use MKVToolnix and it’s all just a matter of how fast you cand copy form one place to another (no conversion). Advantage is that MKV is much more versatile (for freeware tools) - think subtitles.
I have a Core2Duo 6550 (old stuff) and an AMD5850 (no use since HandBrake/VidCoder only uses NVidia CUDA).
For 1h30m x264 @ 720x400 resolution and decent settings, it takes roughly the same time (less than 2h)
If you have a i7 and a CUDA video card (NVidia with many cores) that can be done in 10-15 minutes easy.
There are advanced settings that can add significantly to that time, but fom a YouTube FLV I doubt you need HiQuality.
See my other post here.
