Sam's never pushed a limit that was too far for Dean. They've tried stuff Dean wasn't sure he would like, but it was never anything that made him balk and question Sam's sanity or anything like that.
Dean chalked it up to them just having pretty similar tastes.
Over the years, they've talked about different things that they've tried, what they liked, what they didn't, what they've seen in porn that they'd want to do and things they wouldn't.
They're in 100% agreement about bodily fluids boundaries. Anything that isn't spit or semen is never anything they would want to purposefully include in their sex life.
They're not really keen on doing it anywhere risky or weird, either. I mean sure, they've done it in a few bar bathrooms, the car, even occasionally in a graveyard after an easy salt and burn. But they're not really interested in the risk of getting caught, or causing trouble over it. Not when there are plenty of other more opportune (and comfortable) places.
Once Dean's little… interest for BDSM-flavored activities came to Sam's attention, they had a pretty thorough discussion about it. It wasn't as awkward as Dean feared it would be, and was actually pretty insightful into the things Sam was into and not into as well. He was pretty pleased his brother's tastes ran more exciting than he originally thought.
Light-hearted teasing was fine, actual degradation and humiliation was not. Dean didn't mind dropping the occasional "sir" in the mix when Sam would give him orders, but that was pretty much the only formality he would feel comfortable using. "Master" was just too much, and they both had way too many parent issues for the "daddy" dynamic to be even remotely appealing.
Dean liked a bit of pain with his pleasure, but nothing beyond what Sam could do with his hands; slapping, scratching, pinching. Any combination of the above was cool in moderation, and thankfully Sam wasn't the sort to get hyped up on the idea of actually hurting Dean, which worked well for both of them.
Other than those hard and fast rules, Dean was a pretty adventurous type. To his surprise, Sam was too, and it always made Dean's day to pull some new, sexy quirk out of Sam's head.
Trying them out was even better, and Dean had a growing list scribbled in a notebook that they were always crossing off and adding to as the opportunity arose.
As Dean became fond of saying, you gotta get your kicks where you can.