Howard's father left him and his mother. People think this is where Howard's low self esteem comes from, but that isn't true. His father wasn't a very nice person. To him, Howard was an inconvenience, and an embarrassment to the Wolowitz name. Howard didn't play sports, didn't have a lot of friends and was a total nerd. He was never happy with 'little Howie' only having good grades. He felt he should at least be good at a sport. That would look good on a college application. Whenever his father would tell him this, Howard would point out that he was too young to be thinking about college.
Sheldon's comments about him only having a master's degree did bother him, but what bothered him more was his father. He told Howard that he was utterly disappointed that Howard was a lowly engineer and believed he could have done better. Howard hated that his father could make him feel so bad and so good about himself at the same time.
Penny had a similar problem with her father. If she had been a boy, then he would have paid Penny more attention. Hell, he would have actually given her some attention! Penny's father wanted a boy so much he all but ignored Penny. She never had a relationship with her father until she moved to Pasadena. Even then, he still called her slugger.
Howard had always been attracted to Penny. Penny could tell, even without the pervy comments. But Penny hadn't always been attracted to Howard. In her mind, she woke up one day, and just... Just noticed him. It weirded her out a little bit.