When you hold the Ctrl key the freehand tool starts working like the polyline tool. Is there a way i can turn this off?

I don't know of any way to turn that behaviour off.
What would you want it to do when Ctrl is pressed?

I have the Alt and Ctrl buttons swapped, so Ctrl is remove selection. But it still acts like a poly line.

There seems to be a difference between 'pre-click' modifier actions and 'post-click' modifier actions.
As far as I can tell, the pre-click modifier actions are swapped when "Switch Ctrl/Alt Selection Modifiers" is enabled but the post-click modifier actions are not swapped over.
Hence the Ctrl key always makes the Freehand Selection tool behave as a polyline when used as a post-click modifier.

However, the pre-click use of Ctrl is applied and the post-click Ctrl key forces the selection action to be Subtract, i.e. the unswapped Alt key action.
This does seem like a bug.
Please report this at https://bugs.kde.org/ (sign up and sign in are needed) then post a link to the bug report in a reply here.

Bug report sent! Thank you for the input!