Hi,
I was wondering if it were possible to save the input vectors from the mouse of pen and then use them, along with keyframes to regenerate the image instead of having them continuously rasterized.

This should enable lots of things, like trying a stroke again but with a different brush or colour, editing the paths created by the input so a slight mistake in drawing doesn't mean you have to redraw bits again you can just adjust the input vectors and undoing a specific bit of drawing from the undo stack. it should also be possible to apply filters in a similar manner so filters are just stored as their inputs and applied in realtime with keyframes.

It probably is possible, but would require a huge amount of work.

For filters - have you tried filter masks or filter layers? It's basically what you're asking, the only difference is that it is in the layers stack instead of on the undo stack (that could be overwritten at any time). And to apply it in the normal way, you just need to merge it with the layer.