Elijah P. Marrs was born a slave in January 1840 in Shelby County, Kentucky to Andrews and Frances Marrs. His father, Andrew, had been granted his freedom by his master before Elijah was born, but his mother was still a slave, and with her sons was owned by a man named Jesse Robinson. He had one brother, Henry C. Marrs. He converted to the Baptist religion at the age of eleven by Rev. Charles Wells and received basic education studying at night under Ham Graves, a black man, and later at Sunday schools. 

Marrs picked up on Graves' ability to teach and began teaching fellow slaves how to read as well. This passion for teaching and helping others continued even when the American Civil War (1861–1865) started, Marrs read newspapers to other slaves, keeping people informed about the progress of the war. In 1879, Elijah and Henry co-founded Simmons College of Kentucky. Elijah served as the first president and Henry was the principal, but these roles were in flux, and in 1879 and 1880, he also served as business manager. The school opened on November 24, 1879.