I upgraded to faux kernel not a long time ago and I am having a lot of issues with it. I want to go back to the original kernel. I even tried restoring to factory defaults but it didn't restore the kernel. I don't have a backup of the old kernel. I am running 4.4 on a nexus 4
Here is a list with stock kernels for nexus 4. As said before, simply download it and flash it like any other kernel.
You should look around XDA-developers. Someone there will be able to point you as to where the kernel is located. Once you get your hands on it, you can flash it like you did your current kernel.