The Mystery Spot is where it started for Sam. Whilst for Dean he instantly knew he was in love with his angel when he first felt the effects of the bond, Sam didn't. All Sam felt for his Angel (That he didn't know was an Angel) in these early days was sympathy. A sympathy for the run away. Sam knew the look the Trickster was showing when he thought no one was looking. It was a look of loneliness. It was the look of someone who had left their family behind. Sam knew that look all too well, he'd seen it in the mirror one too many times.
When Loki let them out of the time loop, Sam felt gratitude (He also felt proud of the Trickster but wasn't entirely sure why). But he killed Dean on that Wednesday and nearly destroyed any feeling Sam was developing. Sam spent months searching for the Angel (who he still didn't know was an angel, let alone his Angel) and he convinced him to bring Dean back.
When Sam woke again on that Wednesday morning with "Back in Time" playing he was ecstatically happy (and quite a bit of that was directed back at the Trickster). So Loki had saved their future by not being a complete dick for once (It is worth mentioning here that no matter what had happened the bond would never be broken, Gabriel may never have gotten Sam but he would never have been satisfied with anyone else).
Sam would never admit that the reason he wanted to talk to Loki was because he had developing feelings for the Trickster. Because that definitely wasn't the reason, or at least the whole reason.
Sam had admitted to himself that he was at least starting to like like Loki, although he would never say that out loud.
The main reason for his wanting to talk to Loki was that, the Trickster is a powerful being and they could use that kind of power to fight Lucifer.
Lucifer, who wanted to wear him to prom. Lucifer, who would use his family and guilt against him. Lucifer, who was the second oldest angel.
Lucifer who just so happened to be Lok-sorry Gabriel's older brother.
Sam remembered his sympathy for the run away here and it only grew stronger as he realised gabriel was a runaway younger brother who just couldn't stand the way his older brothers behaved.
Then Sam remembered that he was starting to like like the Trickster, that turned out to be an angel. He also realised that he was more than just starting to like like the Angel, he did like like the Angel.
This ultimately, led to Sam remembering that he and Dean planned to send the Angel on a suicide mission to kill his own brother.
Sam was just happy that when he realised this he was still the Impala. He would have cried if he was human.
Gabriel actually did the suicide mission.
He realised he loved him.
Sam realised he didn't like like the Angel.
Now Gabriel was dead. Sam had nothing.
All he could do is live his life and never know what would have happened if he had been able to be with Gabriel. His Angel.