I am trying to rotoscope from layer 2 to layer 1 by applying a mask to video layer 2 around the area I want to remove.

However when I add composite to the clip to transition between the layers, it simply seems to simply remove the layer 2 image and just show layer one. Going through various transfer options just gets me various weird colour effects of combined layers, or black holes where by rotoscoped layer should be.

I[m using Alpha in rotoscoping, and composite with Over for the transition.

"various weird colour effects of combined layers, or black holes "

Yeah. I find this too. But I also find that this weirdness is just in the preview. The rendered version does not have this.

See this item in the manual where I describe how you can get the preview to work properly too.

http://userbase.kde.org/Kdenlive/Manual/Effects/Alpha_manipulation#Hint_for_previewing_the_alpha_manipulation_effects