The mother letters are:

These letters are mothers because they are primary: א is the first letter of the Alef-Bet, מ is the middle, and ש is next to last. (ת, Tau, isn't used here because it is one of the double letters.)

Being primary, they are the forces that create space: א is the up/down axis, מ the east/west axis, and ש the north/south axis. They are creators, representing logical principles:

• Shin (ש) is thesis, first principle. It represents the right pillar, Mercy
• Mem (מ) is antithesis, contrary principle. It represents the left pillar, Severity.
• Alef (א) is synthesis, balancing principle. It represents the middle pillar, Mildness [1].

Sefer Yetzirah says this quite clearly at the beginning of Chapter II: The three mother letters A, M, Sh are the foundations of the whole; and resemble a balance, the good in one scale, the evil in the other, and the oscillating tongue of the balance between them.

These also represent both Hebrew and alchemical elements: Shin is fire, sulfur; Mem is water, salt; and Alef is air, mercury. We are further told that these letters are the roots of the letters YVH of the Tetragrammaton, יהוה: Yod derives from Mem, Heh from Shin, and Vau from Alef.

The relationship between the mothers, doubles, and singles isn't obvious in the Tree as we are taught to view it. However, the logic becomes much more apparent in an alternate layout you can see in the Gra tree. The mother letters are horizontal paths communicating between the left and right Pillars; the double letters are forces holding the Sephiroth on the individual pillars apart from one another; and the simple letters communicate between adjacent Pillars.

Read more about:

Alef-Bet | Sefer Yetzirah | letter classes | double letters | simple letters