If you want the file compressed (with no real apparent loss in visual quality), as opposed to just multiplexing the MPEG-2 stream into a .mkv container, then it’s basically a 2-step process.
First, you need DVDShrink, or DVDDecrypter, or DVDFab, or any of the other DVD rippers to create a copy of the DVD without the CSS encryption. It doesn’t really matter whether your copy is .iso or VIDEO_TS.
Then, the second step is to open Handbrake, choose the .iso or VIDEO_TS folder you just made in step one as your input file, select the “High Profile” preset, change the output to .mkv (and add/remove any audio/subtitle tracks that you want/don’t want) and then “Encode”.
