Would be smoother if I can press and hold a key to invoke the palette and release the key to dismiss it. Having to click outside the palette is kinda weird, so I avoid using the palette altogether.

I have the same feeling. I am always afraid of clicking outside the pallete and ruin anything on canvas.

I'm always trying to find keys to toggle it off.

I would also consider that users could have the option of switching the pallete off or replacing it for something optional.

+1

You can easily assign a key to the palette via configure krita > canvas input settings > Show popup palette.
I also didn't like that and assigned a keyboard shortcut there. As far as I can tell, the only difference with what you suggest is that the palette stays there until you select something or press the key again.

I would like to add that it would be nice to have the "hold/release" behaviour (at least optionally) on more popup elements.
For instance, I use "color selector" popup constantly and it would be way more comfortable (UX wise) having the hold/release behaviour.
And same goes for "eraser mode" by the way!