Today I wrote a piece of code that hooks MessageBoxA API to redirect execution to another function that will call MessageBoxA with different arguments.
This is the traditional way to hook APIs , there are other methods used for instance IAT hooking.
C++ Visual Studio Express 2010 : http://pastebin.com/ny3NGn93
The 1st step was to Hotpatch MessageBoxA function. In other words some functions in Windows DLLs are ready to be hooked that's why a "mov edi,edi" (which represent a 2 bytes NOP) which will be patched to a short jump is the first instruction executed in each of these functions.
The question that you may ask is why not simply using 2 NOPs instead of "mov edi,edi" , the answer is related to performance : each NOP takes 1 clock cycle so 2 NOPs will take 2 clock cycles. But "mov edi,edi" instruction will take only 1 clock cycle.
Each "mov edi,edi" is preceded by 5 NOPs which are never being executed. Hotpatching consists of overwriting "mov edi,edi" by a short 5 bytes jump back. And then inserting our hook by overwriting the 5 NOPs with a long jump instruction that will execute our desired function, this requires us to calculate a relative address.
When the hook is no longer needed it is sufficient to restore only "mov edi,edi" instruction as the 5 bytes before it are never executed.
See you soon :)