Why polling doesn't require context switch?

Question

When we have a microproccessor and we must service the peripherals, the theory suggests that we can use either interrupts (context switch required) or polling (no context switch required). On the first occasion what we do when we receive the interrupt is finish our current command and then:

1. Switch context (program counter, flags, registers etc)
2. Call the interrupt service routine (ISR)
3. Restore the previous context
4. Return to our program's next command

But if instead of interrupts we use polling what we do is essentially execute our program normally and scan periodically for peripherals that may need service. So if we detect a peripheral that needs service wouldn't we have to switch context in order to execute the ISR? So what am I missing here?