Simply put :

Layer
└ Filter Mask A
└ Filter Mask B

Does this mean the effect is applied as in I apply the Filter A first to the layer -> and then apply Filter B to its result?
Or is it the opposite?

It seems that the mask lowest in the stack is applied to the image first and then the next one up is applied to the result, and so on working up to the top mask. Then any filter or transform layer above the image layer is applied to that end result.

In your case, Filter Mask B will be applied to the image first and then Filter Mask A applied to that result.

If you use a filter mask such as Edge Detection as part of a mask stack, the end result can be very different depending on the order of the masks, which is to be expected. Sometimes, the end result is just strange and you can only get what you think is a reasonable result by applying the edge detection mask as a filter layer instead.

If you're having problems due to what you think are incorrect results, please describe them in more detail because I believe there's a bug in this process and I'm close to raising a formal bug report.

ahabgreybeard wrote:If you're having problems due to what you think are incorrect results, please describe them in more detail because I believe there's a bug in this process and I'm close to raising a formal bug report.

No I wasn't having any particular problem. I just wanted to make sure how it operates cause the results varied in different orders.

If I encounter any bug-ish problem later I'll post it here.