A button on my pen for my tablet switches to eraser for me which is fine. But for some reason when i press the button it also uses the Pan tool? Which is a real inconvenience especially when i want to constantly switch back and fourth from pen to eraser. Since it's constantly forcing me to move from the spot of my work i'm specifically looking at. I'm not sure if this is just a setting in my tablet settings or in the program. If i check if Pan tool is binded to anything it doesn't show anything. Even changing the ket bind on my Tablet settings it continues to be used as pan tool.

You can configure the pan action in Krita in settings/configure krita/canvas input settigns -- by default it's associated with the space key.