From the manual I guess it allows you to transforem a group layer with all the child layers at once, but that action is still possible when the Recursive Transfrom(The little spider icon) is not pressed.

It seems it only makes a difference when dealing with layers with masks, because you might want to transform those seperately sometimes.

It doesn't seem to have any effect on mask layers too.

probably caused by a bug. There were some adjustments to it in 4.2 where someone noticed there was a bug.