If you build any shape with the rectangle tool, ellipse tool, polygon tool or polyline tool and skew it or rotate it, the calligraphy tool's follow selected path option will not work properly. Even after converting the shapes to path.

The Calligraphy Tool does need some attention.
It is actually following the path but it's not paying any attention to the transform which has been applied to the path.

This needs a bug report at https://bugs.kde.org/ so please do that and then paste a link to the bug report here.

The workaround is to follow the newly created vector shape with the calligrahy tool and then apply any rotation, transform to the calligraphy stroke object.

Edit:Add: There's a better workaround: Apply the transform to the vector object using a transform mask then use the calligraphy tool to follow the vector object.

Link to the bug report https://bugs.kde.org/show_bug.cgi?id=433288