Yes, or just the xenophobic dislike of non-human sapients. It was fairly publically well known that Palpatine was a Sith Lord, after the war got into full swing. Fear and oppression do drive people to insane loyalty though, you are right.

[Luke shakes his head, remembering how two duels with the Emperor]

Still, he's gone. Nothing left but a few Planetary governors and fleet commanders desperately seeking alliances outside the galaxy.