After executing the binary it prints : "Input Key:" and waits for us to enter the flag. The routine printing the "Input Key:" message is executed at initialization alongside a sub-routine implementing the ptrace anti-debugging trick. Since we're going to debug the binary, I patched the anti-debugging sub-routine's address with nullsub_1.
After setting up remote debugging under IDA and supplying some random input to the binary we see a call to some code that was stored in executable memory.
IDA sometimes has trouble displaying memory under its debugger, so let's setup a manual memory region and set it as an executable 64-bit segment.
Now we should be able to view the entirety of the bytes copied to memory.
In the figure below, the code that is executed starts by checking if the 4th bit of input[0xC] is set. If it's not set, the message ":(" is printed and the process exits.
However, if the bit is set the code proceeds to decrypt the next block and also XOR the subsequent blocks that are still encrypted. (see figure below)
There's also a second test, implemented in some blocks, which involves the NOT instruction (figure below). This means that the 3rd bit of input[0x11] must not be set.
The amount of code executed is huge and doing this manually is impossible. So, we have two options :
1. Either dump the encrypted data, decrypt it statically and then build the flag by automatically reading the disassembly.
2. Automate the IDA debugger to save the index plus the bits that must be set and guarantee that everything will be executed by continually patching ECX in order not to take the JECXZ jump.
Even if the 2nd attempt would take longer to complete, I chose to implement it. If I ever do the first one, I'll be sharing it here too :)
So, what the script below does is create a dictionary where the key is the character's position and the value is an array of bits that must be set. I simply ignore the NOT case, since we only care about the bits that must be set.
For example if the character at index 2 needs to have bits : 0, 4 and 7 set, the dictionary entry would look like this : {2: [1, 16, 80]}
After the process terminates, we proceed to OR the numbers of each array which gives us the flag in the end.
Here's the script that must be executed under IDA to get the flag.
Script runtime : approx. 15 minutes
Full script : here