I like using the "Organize files" feature of Amarok, but when it has to put music in a differently-named folder, it doesn't just rename the folder, but creates a new folder, then moves the songs to it. Because of this, I still have the old folder which holds things like album art and CC license files. Is there a way to make these move with the songs?

Currently not. You might want to add a wish in the bug database, maybe it gets added to Amarok 2.0.