Whenever I try to set up any kind of keyframes to change what area is being rotoscoped throughout my video, the area being rotoscoped goes insane and jumps around between the keyframes. Any idea what I'm doing wrong?

Edit: I've also realized that this doesn't happen if I only manipulate the large dots and not the smaller dots that are connected to them. Problem sort of solved, but I still want to know why this is.

The larger dots move the line where it is connected to them. Each large dot has 2 small dots (diamonds) coming off it, these dots change the curve of the line coming from the large dot. Couple of screenshots attached, as this may not make sense.

First shows a "Normal" crop with straightish lines. Notice how the diamonds are all aligned with the lines

Second shows the same crop, but with the diamonds moved, notice how the lines curve according to where the diamonds are pointing

HTH

Rotoscope mode Alpha for me not working. Only RGB and Luma working. I have 0.8.3 version and Ubuntu 11.10 . Maybe I do something wrong?

Alpha mode only writes into the alpha channel, so to see anything, you must add an effect which uses alpha, like composite.
(Or just inspect the alpha channel using display options in alpha operations)

Thanks for response Marko.