Reddit mentions of Modeling the Imaging Chain of Digital Cameras (SPIE Tutorial Text Vol. TT92) (Tutorial Texts in Optical Engineering)

If you use 3/4 Nyquist it is indeed correct that it won't be monotonic - this is the weakness of single points on the curve of a function trying to convey all of the necessary information. On the other hand, if you're curious what's going on at very fine scales - the place to look /is/ what's going on near Nyquist. Even if there are nonidealities. And you can design a metric that glosses those over, for example integrating between (say) 3/4 and Nyquist, which would be much better conditioned. The wider the bandwidth the better conditioned it is, until you just end up at SQF.

If you want to capture "sharpness," it really is optimal to just use SQF. Perceptual sharpness was simply figured out a long time ago. Hultgren's modified SQF is superior to the original, in my opinion.

The more complicated behavior I can't help you with (perhaps look in my two reference books on this topic...)

https://www.amazon.com/Modeling-Imaging-Digital-Tutorial-Engineering/dp/0819483397/ref=sr_1_3?keywords=remote+sensing+image+chain&qid=1564974865&s=gateway&sr=8-3

https://www.amazon.com/Remote-Sensing-Image-Chain-Approach/dp/0195178173/ref=sr_1_1?keywords=remote+sensing+image+chain&qid=1564974865&s=gateway&sr=8-1

Composites of smear and jitter are easy to model (even multiple smears) - just convolve several jitters or smears together with the .conv syntax. If you want to learn prysm, I would recommend keeping an eye on the v0.17 release notes. This release is due in the next little while (a few weeks, probably) and will break a lot of way the library was used in <= v0.16 to reduce the amount of vocabulary you need to know to use the library.