How Yizkor began
Praying for the dead goes back to ancient times. Not simply because we want to remember but because it is good for the dead to be remembered: the Sifrei (209) informs us that the dead need atonement.
Giving charity as part of the memorial service is good for the deceased (Shulchan Aruch Orach Chayyim 621:6) since charity is a means of atonement and it shows that we follow the ways of those who now cannot give charity themselves.
Yizkor on Yom Kippur is endorsed by the commentaries on the Yom Kippur morning reading which relates, “After the death of the two sons of Aaron…” (Lev. 16).
The saying of Yizkor started as a Yom Kippur practice which then spread to the other festivals except Rosh HaShanah.
Some people leave the synagogue before Yizkor if their parents are still alive but though this seems to be supported by the Kol Bo who says that excessive weeping and grief disturbs the festival spirit, it is far better for people not to leave, which in any case creates a disturbance and results in chatting and playing outside the synagogue which is a worse desecration of the spirit of the day.
It is better to remain in the synagogue and say a quiet prayer of gratitude for the lives of one’s parents.
No-one leaves when Kaddish is said; why should they leave during Yizkor?
Bear in mind also that the Yizkor prayers mention the heroes and martyrs who gave their lives for our people and for Israel, making every one of us a bereaved survivor.