As a child, a babysitter named Shirley stole him from his mother. Shirley also took his two older sisters, Renee and Pepper.
Dateline NBC reported Renee and Pepper's journey in March, in an episode called "The Girl Who Didn't Exist."
Shirley took the three children from their home in the early 1970s. Shirley gave Pepper away to a couple who then illegally adopted her, gave Ray to a friend named Anna Brown and kept Renee to herself.
Three years later, she went back and kidnapped Pepper from her adoptive parents.
The three proceeded to move around the country, often living in cheap motels, until the girls grew up and Shirley died.
After Dateline's story aired, the girls' mom called the news magazine saying she was their mother and she also had a son.
Dateline then found Ray Smith living in Loveland, Colo.
"I don't know why, couldn't have been any money in it, I always lived in poor conditions just seems like a little bit of insanity to me," Smith said about his kidnapping.
He grew up with Anna Brown, until she gave him away to a different family and then a different family.
He bounced between her and others for a while, growing up as Jim Brown.
He learned his real name by accident, by eventually finding a birth certificate. But he never went looking for his real mom.
"Don't really know how to look for people," Smith, a custodian at Colorado State University, said. "I didn't think I'd ever meet my parents. But I did. That's great."
Last week, Dateline flew Smith to California to meet his mother Jehri Coleman and sisters, Renee Vasquez and Pepper Smith.
They all met for the first time in more than 30 years.
"I don't have so many questions that I used to have," Ray Smith said. "It's really good to know who your biological family is. Not so much of a mystery anymore, no so much weight on my shoulders I suppose."
The entire story airs on Dateline Friday at 9 p.m. For more information, visit http://www.msnbc.msn.com/id/3032600/.
(KUSA-TV © 2011 Multimedia Holdings Corporation)