[the smiling weirwood]

Okay, Varys did not make a mistake to land him where he is. He was sold into slavery to some sicko magician guy and then tossed out into the streets. That says inescapable circumstances to me as opposed to bad life-choices.