X doesn't have any memories. He was a failed experimental specimen with no memories. This outcast was rescued and raised as a human by the people of Bellmare fishing village. This might be a happy ending for X, but instantly, he went mad and massacred the entire village with his exploding energy. Being heartbroken, he fled to the abyss of Nadarzel to commit suicide but was saved by Kaos' arrow.
"Take care of your head carefully when you stand before the Best Assassin of Covenant!"