Jess knows Sam's mother died on November second, and that every following year his father went on a bender starting on Halloween and didn't sober up for a week. So it makes perfect sense that he doesn't have any happy memories of Halloween.
Now if he'd just relax a little, so he and she can make new memories...