Yes. There is a difference between kama and kama raga. There may be a trace of kama left in an Anagami, but that is not enough to lead to a birth in kama loka.
That trace of kama is likely to be associated with rupa raga (and avijja) which are two samyojanas removed at the Arahant stage.
You asked: ” In the days of the Buddha, did people listening to His discourses go from anariya to Anagami or Arahant right away, bypassing the Sotapanna stage?”
Yes. There are many accounts in the Tipitaka, of people attaining any of the four stages of magga phala while listening to a desana or even a single verse.
– But that always happens in the sequence: Sotapanna, Sakadagami, Anagami, Arahant. It is just that even all four can take place within a short time, ending in the Arahant stage. So, it is not really bypassing any intermediate stage.