I have below construction where the second layer clip has the rotoscope effect, and the bottom composite has opacity changed.



Below are stills from the beginning (where there is no composite yet) and the end of the clip. You can see that the rotoscope mask is showing through the composite. (I've lowered the opacity further to make the mask show up more clearly.)

Is this a bug, or should I be doing something different to achieve what I'm trying here? If nothing else comes up I can just render the bottom two rotoscoped clips separately and replace them with the render during the final cut. But if I can do it cleanly directly within kdenlive that would off course be better.

Try with the Feather width and Feathering passes in rotoscoping to smear the hard edges. It’s not a bug.

Ok, so it's not a bug and the rotoscope IS supposed to affect the other clips as well.

But making the hard edges softer won't solve my problem, what I want is to not see the darker polygon in the second still at all. Basically to have the rotoscope mask not affect the other clips on top.

(To make things a little clearer: the rotoscope is tracing a path on the map in dark blue. In the first still you can see a little of the trace, on the second you see all of it. Unlike in the first still, which is perfectly like I wanted it, in the second you see the rotoscope mask I used as a darker polygon around the blue trace on the map, which I obviously do not want. The "cross" you see in the second picture is caused by the four clips on top forming a grid with (upturned, to make the problem more clear) opacity turned on, so that's not a part of my problem.)

Off course, it might be that what I'm wanting isn't possible (I'm new to kdenlive and don't yet fully understand how its composite transitions work), and I'll have to bite the bullet and render the rotoscoped clips separately and combine them with the transparent clips on top in the final cut.

I'm not entirely sure what it it you want to achieve, could you make a picture of it a describe it precisely ? you might want to change the track for the composite effect ? i usually can do everything i want with compisite so if you explain more clearly i should be able to help !

Sure.

This is what I'm trying to accomplish (the opacity was changed from 90% to 50% to match the "wrong" picture):



This is what I'm getting instead (the rotoscape mask is visible):


I achieved the first picture by rendering the rotoscope (first two layers) separately and then replacing those two layers by the render.

ok so after some tries, here's what i found out. There's definitely an issue with composite... so now it depends on what you're doing with the composite. If you're not using it for a wipe, you can use composite and transform or affine instead, else it depends. and i realised i have no idea how to add images so my screenshots are useless haha
I ususally prefer composite and transform btw

Indeed, "composite and transform" works perfectly fine. So seems a bug in the "composite" transition after all.

Thanks for the help!

No problem, glad i could help !