Well i got this theory there are so many clues hidden around that he was special and not a Lannister, ill just post a few of them, there are more than this but with company christmas party hangovers me brain is not computing.
Its Tyrion and The Red Viper talking
Red Viper "It was when i visited Casterly Rock with my mother, her consort, and my sister Elia. I was, oh, fourteen, fifthteen, thereabouts, Elia was a year older. Your brother and sister were eight or nine, as i recall and you had just been born,"
Tyrion "
A Queer time to come visiting."His mother had died giving him birth, so the Martells would have found the Rock deep in Mourning.....
So why would the Martells come to visit at the worst possible time, there was bad blood yet between the families. So perheps Tyrion was special.
They even describe a omen that said that Tyrion´s birth would bring Famine, plague and war, and winter and the long that night that never ends and the Viper adds, ohh and the fall of your father.
So as we all know some of these things have already come true and could Tyrion be a bastard son og Taegaryen dynasty
Jon is chatting with Ygritte.
Page 599 in A Storm of Swords :Steel and Snow
Jon Snow "
No he Thought,
The Wall protects the realm. From the others.....and from you and your kind as well. Sweetling.
Ygritte "I had another friend of mine who dreamed of Dragons. A Dwarf. he told me....."
This is just a few, im sure Boaz will fill in the blanks.