Some scanners mail the scanned pdf file as an application/octet-stream with pdf extension. These are indeed pdf files. Currently, they are not opened as pdf, but I get the application chooser dialog, because no application is associated to application/octet-stream.

I would like the system to (try and) open such files as pdf files, i.e., fall back to filename extension-based detection for application/octet-stream mimetyped files. Is this possible to configure in the file associations kcm or in another way?

An octet-stream is random binary data. You can assign it to eg. okular, but that's "wrong" since it's random binary data.
What needs to happen is that whatever opens the file (kmail?) either prefers the suffix over the mimetype (ugghhh...) or (much better omits the (particular) mimetype, so that the system is forced to detect it. -> File a bug against whatever opens it.

A shorthand "solution" would likely be to assign octet-stream data to always be openened by "kioclient[4|5] exec %U" (which redirects the opening and scratches the mimetype), but I've never tried that. => "kcmshell[4|5] filetypes"