If the CPU has some new features, how does the Linux kernel handle this ? From what I remembered, PAE does not need to rebuild the entire the system and Linux seems like just installing the driver and things work.
Say, if the CPU provides more execution modes, Do we need to rebuild the entire system?
If so, why does PAE do not need to rebuild while this one may need?
The types of new features which a Linux kernel could support without rebuilding the kernel depend on the instruction set and the nature of the features. Since x86 provides a direct means to save processor state (as well as indicating how much state exists), even features which add application-level state could be used with a new kernel.
Some ISAs provide a software layer that can be somewhat independent of the operating system; for example, Alpha had PAL (Privileged Architecture Library). Such a layer can abstract the hardware from the operating system (somewhat like how BIOS was used), allowing the operating system to use this software to perform implementation-specific operations.
Obviously even without a ISA-specified interface, Linux could be extended to support the use of a module to provide a similar abstraction. Because Linux is open source there is less incentive to provide such an interface, though there might be some advantage to having a single system image with multiple system specializations. In principle dynamic kernel patching could provide an even broader range of extensions, though it is primarily intended for critical (security/reliability-oriented) updates. Generally, if one is changing hardware, one will be rebooting and can update the software.
In order to use such an interface, the subsystems that could be affected would need to be know at the time of kernel compilation. If abstracting a new feature from the existing kernel requires large abstraction layer software, then just using a new kernel would generally make more sense. If supporting the new feature requires frequent calls back and forth between the abstraction layer software and the existing kernel, the performance would tend to suffer. (Linux is monolithic-oriented in design philosophy — preferring performance over abstraction — so requiring what could eventually become a hypervisor layer is unlikely to be attractive even if the added complexity was miniscule.)
Without such an abstraction layer, changes that add state or otherwise require new privileged operations cannot be added without changing the kernel. In principle, new modes which only require privileged enabling of the mode but did not add new state (e.g., the mode indicator uses a reserved bit which the kernel writes as read) and which are safe for other applications to have enabled (because the context switch code would not be modified) could be supported with a relatively simple kernel module.
For example, a kernel module could be used to support an entirely new instruction encoding indicated with a previously reserved bit in the page table entry of pages of code using the new encoding. Since the change is limited to the specific pages, no special support would be needed from the core of the kernel
Because PAE used a different page table entry format, adding support required a new kernel. However, non-PAE-aware subsystems could be upgraded to support PAE through kernel modules as long as the core kernel supported PAE.
New unprivileged instructions which do not add state or generate new types of interrupts/exceptions (that enter privileged mode), do not require kernel support to be used by application software.
(I am not even a programmer much less an operating system developer, so I do not know if Linux abstracts exception handling in such a way that a kernel module could add new exception handling code. In theory, an ISA could define a larger exception handler table, reserving unused entries to support future exceptions, in which case, allowing a module to add handling of a new exception type might be practical. Of course, if the core kernel allows arbitrary interrupt handlers to be changed by a module, any such interrupt — which could include system calls — could be directed to the module's code and then call core kernel functions to handle unchanged aspects. Allowing a module to hijack the kernel in this manner seems inadvisable.)