I have a problem finding a non-used colour for the blue screen effect. My overlay scene contains many different high-saturated colours.

How does the effect determine distance in colour-space between two pixels? What does the distance look like as a function of two rgb-vectors? Given that function, I can probably find something far away from the other colours.

Alpha manipulation > Color selection
has a choice of different color spaces and subspace shapes, with independent tolerances along each axis. It can be cascaded to build more complex shapes in the color space.

To clean up the resulting selection, you can use "alpha shapes" for simple in- and out- masks, or rotoscoping for more complex masking.

Ok, where can I find information on ABI and HCI? I found something on HCL.

See here:

http://en.wikipedia.org/wiki/HSL_color_space

under "hue and chroma".

The alpha and beta from here are the A and B coordinates in ABI

I is the intensity as described a bit down the same page.

thanks