But no one has the "right" to tell me I HAVE to put those things in my yard or that I have to sit and listen to prayers in public (taxpayer's) buildings, stadiums etc, and if I don't "like" it I can leave.
No one will try and force you to put crap in your yard cept for those trying to sneak political signs there and you are free to remove those. As for whether or not someone can pray in a public place such as stadiums, public buildings, parks, sidewalks, bus stop, etc, they can and actually the remedy is that if you don't like it then you can get out of earshot as you have no legal right to try and stop them from doing so..
As for what you said about screaming in the praying person's face, that would not be advisable as that is a form of assault and getting "in someone's face" can be deemed as a threatening act where the person would have the right to use physical force to remove you from their personal space. Anyway, that's one of those things that could get a person a tail whoopin and sent to jail all in the same day so it's better to leave the person alone and mind your own business or simply go somewhere that you don't hear it if it's too bad. I assume if you wanna scream but stay out of the person's face while screaming that it would be OK to do in most instances.