I have this question lingering in my mind since I’m new to RTOS concepts.
Let’s suppose a task/thread and an ISR are using a semaphore resource. Now Interrupt triggers and ISR tries to acquire a semaphore resource but is used by another task. Does it lead to blocking condition for an ISR? Generally we are not supposed to block an ISR. How is this scenario handled by an OS/RTOS? Are there any solutions/APIs inculcated either FreeRTOS or Linux Kernel?
You should not use blocking calls in an interrupt context. Most RTOS will emit an error (at least in a debug build) and halt or otherwise prevent you from doing that.
The thing is of course that ISR's run to completion. If you block to wait on a semaphore, nothing will run to actually give it. You can only take a semaphore that has previously been given.
If you do take a semaphore in an ISR it must use non-blocking semantics, i.e. immediately return with and indication of whether the semaphore was available or not. Depending on the specific RTOS, that may be a special version of the sem take, or simply require a zero timeout argument. However such a use of a semaphore has no advantage in most cases over a simple shared variable of an atomic type.
That said, interrupts are asynchronous and a semaphore is a synchronisation primitive. It is hard to see a circumstance where you would makes sense to take a semaphore in an ISR. I would question your design if you find yourself doing that. The more likely usage is for the ISR to give (or increment) the semaphore to signal a waiting task.
It is not hard to avoid - the ISR can simply do nothing other than signal a thread context (with a semaphore or event flag), then that thread context can wait on a semaphore to synchronise with some other thread context or other ISR if necessary. That is to say that if handling an interrupt event involves a need to wait on some other context, then the handling should be deferred to a thread context that allows blocking. See for example this on deferred interrupt handling in FreeRTOS.
RTOS's vary in how they handle what APIs can be called from an IST. Some simply require non-blocking (zero timeout) use of standard APIs where others have ISR specific versions. For example FreeRTOS has xSemaphoreTakeFromISR() which does not block. I would not take the presence of such a function as an indication that it makes any sense to use it however! A lot of FreeRTOS is like that ;-)