He was the son of a rail splitter, so to speak.

There were quite few rumors about Robert Todd Lincoln’s sexuality when he was a young man in the White House. While most of his contemporizes were on the battlefield he stayed behind to study law. Although he was eventually given a staff commission on General Grants staff at the end of the war, there was a constant drumbeat of criticism that the President’s son was not in uniform. President Lincoln always said it was because of his wife who became demented at the thought of losing another son as she had almost gone mad after the death of their beloved Thad. So young Mr. Lincoln stayed in the background shuffling papers and creating the North American John Hay Love association.
(Alice Roosevelt Longworth Doesn’t Live Here Anymore: The sexuality of political Children by Doris Kearns Goodwin, Playboy Press 2008)