Good question. From the way you phrased it, I can see that you understand the main idea behind sakkāya diṭṭhi.
– But the keyword there is diṭṭhi.
– Sakkāya diṭṭhi is the wrong view that there is a “soul-like entity” in the rebirth process, i.e., that there is something permanent that goes from life to life.
Even after you SEE that there is no such “soul-like entity” you have the PERCEPTION (saññā) of “me” and “mine”.
– That WRONG saññā REDUCES AT THE Anagami stage and GOES AWAY only at the Arahant stage.
– In fact, there are THREE types of “vipallasa”: diṭṭhi, saññā, and citta.
– Both saññā, and citta vipallasa go away at the Arahant stage with the elimination of avijja and mana. So, “mana” (sometimes called “asmi mana”) is the “perception of a “me” and it is harder to remove than sakkāya diṭṭhi.
– See, “Vipallāsa (Diṭṭhi, Saññā, Citta) Affect Saṅkhāra”
Also, if you have not read the revised post on “Sakkāya Diṭṭhi – Wrong View of “Me” and “Mine”” please read that too.