Reddit mentions of Hearing in Time: Psychological Aspects of Musical Meter

Check out The Time of Music by Jonathan Kramer. I'm working my way through it right now and I'm almost done - it's a fantastic book. There are parts I agree with more than others, but Kramer is pretty clear that he intends the book as a way to prompt discussion/thought/debate rather than as the word of god descended from the heavens, and it's definitely done that for me.

EDIT: Here's a couple other related books that have ended up on my reading list but I haven't gotten to yet.

Justin London - Hearing in Time

Christopher Hasty - Meter as Rhythm

Well anything that falls under "cross-cultural" usually winds up with the ethnomusicologists.

You sound like you'd be more interested in the music cognition side of music theory (not my area at all). Again, Music Perception is going to be your best bet here. You might also check out David Huron's book, Sweet Anticipation, which deals with anticipation in music from a psychological perspective. Juslin and Sloboda's Handbook of Music and Emotion has a lot of the cognitive side of things as well, if you're interested in that aspect. Finally, David Temperley's Cognition of Basic Musical Structures or Justin London's Hearing in Time might be worth a look.

Like a lot of things in music, it's largely a cultural phenomenon. Consider 12 bar blues. At a metrical level, consider that the "perfect" meter in liturgical music of the middle ages was compound triple.

If you're really serious about getting good information on this, you'll read Christopher Hasty's Meter as Rhythm, William Rothstein's Phrase Rhythm in Tonal Music, James London's Hearing In Time, and Trevor de Clercq's MTO article, Measuring a Measure.