Often times, when I stumble upon IDAPython scripts, I notice that they are using inefficient / incorrect IDAPython APIs to disassemble or decode instructions (for instance using idc.GetMnem() or idc.GetDisasm()). Therefore, in this blog post, I am going to illustrate how to use IDA’s instruction decoding functions from IDAPython in order to write a very simple x86 emulator. The goal is to demonstrate the correct use of instruction decoding IDAPython APIs. By the end of this post, you should be able to solve similar problems using IDAPython.

Table of Contents

Overview
Disassembling the program
Writing the emulator
Quick intro to instruction decoding
Scoping challenge functions
Emulating instructions

 

Overview

Let’s get started with the following sample program which contains a table of 12 challenge functions that get called in a loop and the result is displayed. Our goal is to write an emulator that can compute the challenge response value statically without using a 3rd party emulator.

#include <stdio.h>
#include <cstdint>
#include <time.h>
#include <stdlib.h>

typedef uint64_t (*challenge_proto_t)(uint32_t, uint32_t);

//-------------------------------------------------------------------------
uint64_t challenge_1(uint32_t c1, uint32_t c2) // 39 operation(s)
{
    uint64_t result;
    __asm
    {
        pushad
        mov eax, [c1]
        mov edx, [c2]
        not eax
        dec edx
        xor edx, eax
        xor edx, eax
        inc eax
        not eax
        sub edx, 0x27c12466
        inc eax
        dec edx
        not edx
        inc eax
        add eax, 0x273804ca
        xor edx, 0xaa5a1584
        sub eax, edx
        not edx
        xor eax, 0xf94f7d8c
        dec edx
        dec eax
        sub eax, edx
        not edx
        dec edx
        sub edx, 0xd7b41b83
        xor eax, 0xa551a9c7
        add eax, eax
        dec eax
        inc eax
        not eax
        add edx, 0xa551b974
        inc edx
        dec edx
        not edx
        xor eax, 0x200d519
        not edx
        not eax
        sub edx, 0xeb15b7ef
        xor eax, 0xb2558b8c
        xor eax, 0xda288d90
        not eax
        not edx
        mov dword ptr[result], eax
        mov dword ptr[result + 4], edx

        popad
    }
    return result;
}

// .
// .
// challenge_2 .. challenge_12
// .
// .

challenge_proto_t challenge_funcs[] = {
    challenge_1,
    challenge_2,
    challenge_3,
    challenge_4,
    challenge_5,
    challenge_6,
    challenge_7,
    challenge_8,
    challenge_9,
    challenge_10,
    challenge_11,
    challenge_12
};

//-------------------------------------------------------------------------
int main(int argc, char *argv[])
{
    if (argc < 4)
    {
        printf("challenge func[0..%d] challenge1-32 challenge2-32 -> result-64\n", _countof(challenge_funcs));
        return -1;
    }

    uint32_t f = atol(argv[1]) % _countof(challenge_funcs);

    uint32_t c1 = atol(argv[2]);
    uint32_t c2 = atol(argv[3]);

    printf("challenge_%d(%d, %d) -> %016I64X\n", f, c1, c2, challenge_funcs[f](c1, c2));

    return 0;
}

In real life, one can find him/herself in a similar situation wanting to treat a function as a blackbox instead of reversing it back into pseudo-code. In such cases, one has many choices:

• Use Appcall during runtime and query the challenge function directly.
• Use a decompiler to extract the algorithm in C pseudo-code and recompile it then use it.
• Extract the challenge function assembly listing, reassemble and use it.
• Use an emulation framework (the Unicorn engine for example) to emulate the challenge function.

Let’s compile this program and run it with c1=123 and c2=456:

C:\>for /l %a in (0, 1, 11) do @test.exe %a 123 456
challenge_0(123, 456)  -> 8FDCE2E203FCAAF2
challenge_1(123, 456)  -> E0317E1AB061ED8B
challenge_2(123, 456)  -> A0A0E0C2279BE734
challenge_3(123, 456)  -> 5D18D0A79D07D7D8
challenge_4(123, 456)  -> 2583B4EEB62E6042
challenge_5(123, 456)  -> D5261E0275AB9805
challenge_6(123, 456)  -> F2B3282E143F7927
challenge_7(123, 456)  -> 9B9B3CBB0169F4CD
challenge_8(123, 456)  -> EF51086C5D1AF235
challenge_9(123, 456)  -> FC8A97125C0EA232
challenge_10(123, 456) -> EEAE8BEB7996D2E7
challenge_11(123, 456) -> 4F36E6A65AB03929

Disassembling the program

Let’s disassemble the test program and locate the challenge_funcs table and the first challenge function:

;
; The challenge functions as referenced from main()
; (12 functions)
;
.rdata:0041749C 00 10 40 00             challenge_funcs dd offset sub_401000
.rdata:0041749C                                   ; DATA XREF: _main+57↑r
.rdata:004174A0 90 10 40 00             dd offset sub_401090
.rdata:004174A4 30 11 40 00             dd offset sub_401130
.rdata:004174A8 D0 11 40 00             dd offset sub_4011D0
.rdata:004174AC 80 12 40 00             dd offset sub_401280
.rdata:004174B0 30 13 40 00             dd offset sub_401330
.rdata:004174B4 A0 13 40 00             dd offset sub_4013A0
.rdata:004174B8 30 14 40 00             dd offset sub_401430
.rdata:004174BC C0 14 40 00             dd offset sub_4014C0
.rdata:004174C0 30 15 40 00             dd offset sub_401530
.rdata:004174C4 E0 15 40 00             dd offset sub_4015E0
.rdata:004174C8 80 16 40 00             dd offset sub_401680

;
; The first challenge function
;
.text:00401000                         ; int __cdecl sub_401000(int c1, int c2)
.text:00401000                         sub_401000 proc near
.text:00401000 ; CODE XREF: _main+65↓p
.text:00401000 ; DATA XREF: .rdata:challenge_funcs↓o
.text:00401000
.text:00401000                         var_8= dword ptr -8
.text:00401000                         var_4= dword ptr -4
.text:00401000                         c1= dword ptr  8
.text:00401000                         c2= dword ptr  0Ch
.text:00401000
.text:00401000 55                      push    ebp
.text:00401001 8B EC                   mov     ebp, esp
.text:00401003 83 EC 08                sub     esp, 8
.text:00401006 53                      push    ebx
.text:00401007 56                      push    esi
.text:00401008 57                      push    edi
.text:00401009 60                      pusha
.text:0040100A 8B 45 08                mov     eax, [ebp+8]
.text:0040100D 8B 55 0C                mov     edx, [ebp+c2]
.text:00401010 F7 D0                   not     eax
.text:00401012 4A                      dec     edx
.text:00401013 33 D0                   xor     edx, eax
.text:00401015 33 D0                   xor     edx, eax
.text:00401017 40                      inc     eax
.text:00401018 F7 D0                   not     eax
.text:0040101A 81 EA 66 24 C1 27       sub     edx, 27C12466h
.text:00401020 40                      inc     eax
.text:00401021 4A                      dec     edx
.text:00401022 F7 D2                   not     edx
.
.
.
.text:00401076 F7 D2                   not     edx
.text:00401078 89 45 F8                mov     [ebp+var_8], eax
.text:0040107B 89 55 FC                mov     [ebp+var_4], edx
.text:0040107E 61                      popa
.text:0040107F 8B 45 F8                mov     eax, [ebp+var_8]
.text:00401082 8B 55 FC                mov     edx, [ebp+var_4]
.text:00401085 5F                      pop     edi
.text:00401086 5E                      pop     esi
.text:00401087 5B                      pop     ebx
.text:00401088 8B E5                   mov     esp, ebp
.text:0040108A 5D                      pop     ebp
.text:0040108B C3                      retn
.text:0040108B
.text:0040108B                         sub_401000 endp
.text:0040108B

We could easily locate the challenge_funcs table because it is referenced from main(). The first challenge function, like all of the others, have a very distinct format/pattern on which we will base the emulator design.

We can distinctly see a pusha instruction (at 0x401009), followed by two instructions that load the initial values (at 0x40100A and 0x40100D), then a series of operations (between 0x401010 and 0x401076) on those registers and finally we see the results being copied back into local variables (at 0x401078 and 0x40107B) before popa is used to restore all registers.

We will use this code pattern to write a small function that identify the boundaries of the instructions that do the computation. We will then write another function that emulates code within a given range and returns the result.

Writing the emulator

In this section, we will implement two functions:

• scope_challenge_function(): this function finds the instructions boundaries to be emulated.
• emulate_challenge_function(): this function emulates instructions within a given range.

Before we get started, let’s define some global variables needed by the script:

import idc, idautils, idaapi

challenge_funcs_tbl = 0x41749C
challenge_funcs_tbl_size = 12

RESULTS = (
    0x8FDCE2E203FCAAF2,
    0xE0317E1AB061ED8B,
    0xA0A0E0C2279BE734,
    0x5D18D0A79D07D7D8,
    0x2583B4EEB62E6042,
    0xD5261E0275AB9805,
    0xF2B3282E143F7927,
    0x9B9B3CBB0169F4CD,
    0xEF51086C5D1AF235,
    0xFC8A97125C0EA232,
    0xEEAE8BEB7996D2E7,
    0x4F36E6A65AB03929)

We deduced the challenge functions table and its size from the disassembly above. We also define the RESULTS variable containing the output of calling each challenge function with c1=123 and c2=456. We will use that table to verify the emulation after we are done.

To enumerate all the challenge functions in the table, we can do something like:

for i in range(0, challenge_funcs_tbl_size):
    func = idc.Dword(challenge_funcs_tbl +  i * 4)
    print(">%x: challenge #%d" % (func, i + 1))

…and the output is:

>401000: challenge #1
>401090: challenge #2
>401130: challenge #3
>4011d0: challenge #4
>401280: challenge #5
>401330: challenge #6
>4013a0: challenge #7
>401430: challenge #8
>4014c0: challenge #9
>401530: challenge #10
>4015e0: challenge #11
>401680: challenge #12

Quick intro to instruction decoding

To decode instructions using IDAPython, use the idautils.DecodeInstruction() function:

# .text:0040101A 81 EA 66 24 C1 27       sub     edx, 27C12466h
inst = idautils.DecodeInstruction(0x40101A)

If decoding fails, then this function returns None. If decoding succeeds, we get an instruction object containing information about the instruction and its operands.

These are the important instruction attributes:

• inst.itype: this is an integer representing the instruction type. Different opcodes have the same itype and hence opcode != itype.
• inst.size: this is the size of the decoded instruction.
• inst.Operands[]: this is a zero based array containing operands information.
• inst.Op1 .. OpN: these are 1-based aliases into the Operands array.
• inst.ea: the linear address of the decoded instruction.

You might be wondering what is the relationship between an opcode and its itype? The answer is simple. In IDA, the open database’s processor module is responsible for filling the itype field based on the opcode. In the IDA SDK, you can find a header file called allins.hpp. This header file contains enums for all supported processor modules along with enum members for each supported instruction:

// Excerpt from allins.hpp

// x86/x64 itypes
enum
{
NN_null = 0,            // Unknown Operation
NN_aaa,                 // ASCII Adjust after Addition
NN_aad,                 // ASCII Adjust AX before Division
NN_aam,                 // ASCII Adjust AX after Multiply
NN_aas,                 // ASCII Adjust AL after Subtraction
.
.
.
NN_jz,                  // Jump if Zero (ZF=1)
NN_jmp,                 // Jump
NN_jmpfi,               // Indirect Far Jump
NN_jmpni,               // Indirect Near Jump
NN_jmpshort,            // Jump Short (not used)
NN_lahf,                // Load Flags into AH Register
.
.
.
// Pentium III Pseudo instructions

NN_cmpeqps,             // Packed Single-FP Compare EQ
NN_cmpltps,             // Packed Single-FP Compare LT
NN_cmpleps,             // Packed Single-FP Compare LE
NN_cmpunordps,          // Packed Single-FP Compare UNORD
.
.
.
}

I don’t know why, but the NN_ prefix is used for all the instruction types on the x86/x64 processor.

Here’s a simple example on how to decode and check the instruction type:

# .text:00402085 74 09 jz short loc_402090
inst = idautils.DecodeInstruction(0x402085)
print("YES" if inst.itype == idaapi.NN_jz else "NO")

One can intuitively check what the decoded instruction is by comparing against one of the idaapi.NN_xxxx constants.

As for operands, one can access them via inst.Operands[] or inst.OpN. To get the number of operands used by the decoded instruction, you should not rely on the length of the Operands array because it will always resolve to UA_MAXOP == 8 (see ida.hpp). Instead, iterate over each operand and see if its type is o_void.

An instruction operand is defined using the op_t structure type defined in the ua.hpp header file.

Some of the operand members are:

• op.flags: operands flags.
• op.dtype: operand type. One of the dt_xxx consts. One can use this field to tell the size of the operand (1 == dt_byte, 2 == dt_word, etc.).
• op.type: operand type. One of the o_xxx consts.
• specflag1.. specflag4: processor specific flags.

These are the supported operand types (o_xxx):

• o_void: no operand present.
• o_reg: operand is a  register (al, ax,es,ds…).
• o_mem: direct memory reference (DATA).
• o_phrase: memory Reference [Base Reg + Index Reg].
• o_displ: memory Reg [Base Reg + Index Reg + Displacement].
• o_imm: immediate value.
• o_far: immediate far Address (CODE).
• o_near: immediate near Address (CODE).
• o_idpspec0 .. o_idpspec5: processor specific flags.

There are additional operand members whose meaning differ based on the operand’s type:

• op.reg: the register number (o_reg).
• op.phrase: index register with memory accessing operands (o_phrase).
• op.value: the immediate value (o_imm) or the outer displacement (o_displ).
• op.addr: memory address used by the operand (o_mem, o_far, o_displ, o_near).

When the operand type is o_reg or o_phrase, then the op.reg/op.phrase values contain the register’s enum value. Like the NN_xxx terminology, the IDA SDK also provides the register constant names and their values; however this is only true for the x86/x64 processor module. Here’s an excerpt from the header file intel.hpp:

enum RegNo
{
  R_ax = 0,
  R_cx,
  R_dx,
  R_bx,
  R_sp,
  R_bp,
  R_si,
  R_di,
  R_r8,
  R_r9,
  R_r10,
  R_r11,
  R_r12,
  R_r13,
  R_r14,
  R_r15,
.
.
.
}

Unfortunately though, those enums are not exposed to IDAPython, but at least we know enough to define something like the following:

REG_EAX = 0
REG_EDX = 2
REG_EBP = 5
.
.
.
REG_NAMES = { REG_EAX: 'eax', REG_EDX: 'edx', REG_EBP: 'ebp' ...}

Here’s another example on how we can fully ‘disassemble’ an instruction:

# .text:0040106F 35 90 8D 28 DA xor     eax, 0DA288D90h
out = ''
inst = idautils.DecodeInstruction(0x40106F)
out += "XOR "     if inst.itype == idaapi.NN_xor else ""
out += "EAX"      if (inst.Op1.type == idaapi.o_reg and inst.Op1.reg == 0) else ""
out += ", 0x%08X" % inst.Op2.value if (inst.Op2.type == idaapi.o_imm) else ""

print(out)

# Outputs: XOR EAX, 0xDA288D90

That covers the instruction decoding principles. Please refer to header files intel.hpp, allins.hpp, ua.hpp and idp.hpp for more information.

Scoping the challenge functions

Earlier, we figured out how to go over the challenge functions table and retrieve the address of each challenge function. Let’s now write a function that uses the instruction decoder to find the boundaries of the instructions to be emulated.

Please note that I can use IDAPython’s FindBinary() but that defies the purpose of this article. For demonstration purposes, I want to find the code pattern in question using instruction decoding only:

def scope_challenge_function(func_ea):
    f = idaapi.get_func(func_ea)
    if f is None:
        return (False, "No function at address!")
        
    emu_start, emu_end = f.startEA, f.endEA
    
    ea = emu_start

    #    
    # Find the start of the emulation pattern
    #
    stage = 0
    while ea <= emu_end:
        inst = idautils.DecodeInstruction(ea)
        if inst is None:
            return (False, "Could not decode")
            
        # Advance to next instruction
        ea += inst.size
        
        # mov (eax|edx), [ebp+?]
        if (inst.itype == idaapi.NN_mov) and (inst.Operands[0].type == idaapi.o_reg) and \
           (inst.Operands[1].type == idaapi.o_displ) and (inst.Operands[1].phrase == REG_EBP):
            # mov eax, [ebp+8]
            if (stage == 0) and (inst.Operands[0].reg == REG_EAX) and (inst.Operands[1].addr == 8):
                stage = 1
            # mov edx, [ebp+0xC]
            elif (stage == 1) and (inst.Operands[0].reg == REG_EDX) and (inst.Operands[1].addr == 0xC):
                stage = 2
                emu_start = ea
        elif (stage == 2) and (inst.itype == idaapi.NN_popa):
            # Let's decode backwards twice and double check the pattern
            ea2 = idc.PrevHead(ea)
            
            # Disassemble backwards
            for _ in range(0, 2):
                ea2 = idc.PrevHead(ea2)

                inst = idautils.DecodeInstruction(ea2)
                if (inst.itype == idaapi.NN_mov) and (inst.Op1.type == idaapi.o_displ) and \
                   (inst.Op1.reg == 5):
                    if inst.Op2.reg == 2 and stage == 2:
                        stage = 3
                    elif inst.Op2.reg == 0 and stage == 3:
                        stage = 4
                        emu_end = ea2
                        break
                   
            break
            
       
    if stage != 4:
        return (False, "Could not find markers")
            
    return (True, (emu_start, emu_end))

The basic pattern when decoding instructions is to advance the decoding address (the ea variable in this case) by inst.size after each successful decoding. Afterwards, one should check the instruction’s itype, then inspect the operands accordingly.

Note that at stage #2, I start decoding backwards. To go backwards in a proper disassembly listing, one can use the idc.PrevHead() function to retrieve the start address of the previously defined instruction (see line 37). Let’s test this function:

Python>ok, info = scope_challenge_function(0x401000)
Python>if ok: print("start=%08X end=%08X" % (info[0], info[1]))
start=00401010 end=00401078

Emulating instructions

In the previous step, we managed to retrieve a start and end address for the emulation boundaries. Now, let’s write a simple emulation function that only supports a limited set of instructions (NOT, DEC, INC, XOR, SUB and ADD):

def emulate_challenge_function(info, c1, c2, dbg = False):
    emu_start, emu_end = info
    if dbg:
        print("Emulating from %x to %x (%d, %d)" % (emu_start, emu_end, c1, c2))

    # Reset registers    
    regs = { 
      REG_EAX: c1,
      REG_EDX: c2
    }
    
    def get_opr_val(inst, regs):
        if inst.Op2.type == o_imm:
            return (True, inst.Op2.value)
        elif inst.Op2.type == idaapi.o_reg:
            return (True, regs[inst.Op2.reg])
        else:
            return (False, 0)
            
    ea = emu_start
    while ea < emu_end:
        out = ">%x: " % ea
        ok = True
        inst = idautils.DecodeInstruction(ea)
        ea += inst.size
        if inst.itype == idaapi.NN_not:
            out += "NOT"
            regs[inst.Op1.reg] = ~regs.get(inst.Op1.reg, 0) & 0xffffffff
        elif inst.itype == idaapi.NN_dec and inst.Op1.type == idaapi.o_reg:
            out += "DEC"        
            regs[inst.Op1.reg] = (regs.get(inst.Op1.reg, 0) - 1) & 0xffffffff
        elif inst.itype == idaapi.NN_inc and inst.Op1.type == idaapi.o_reg:
            out += "INC"        
            regs[inst.Op1.reg] = (regs.get(inst.Op1.reg, 0) + 1) & 0xffffffff
        elif inst.itype == idaapi.NN_xor:
            ok, val = get_opr_val(inst, regs)
            regs[inst.Op1.reg] = (regs.get(inst.Op1.reg, 0) ^ val) & 0xffffffff
            out += "XOR %08X" % val
        elif inst.itype == idaapi.NN_sub:
            ok, val = get_opr_val(inst, regs)
            regs[inst.Op1.reg] = (regs.get(inst.Op1.reg, 0) - val) & 0xffffffff
            out += "SUB %08X" % val
        elif inst.itype == idaapi.NN_add:
            ok, val = get_opr_val(inst, regs)
            regs[inst.Op1.reg] = (regs.get(inst.Op1.reg, 0) + val) & 0xffffffff
            out += "ADD %08X" % val
        else:
            ok = False

        # Dump registers
        for k, v in regs.items():
            out += (" [%s: %08X] " % (REG_NAMES.get(k, "%x" % k), v))

        if not ok:
            return (False, "Emulation failed at %08X" % ea)

        if dbg:            
            print(out)
    
    return (True, (regs[REG_EDX] << 32) | regs[REG_EAX])

When the function starts, it populates the regs dictionary with the initial values of the registers. We use the op.reg as the key into that dictionary. Any uninitialized register will contain the value zero. The emulation function then enters a loop and decodes each instruction. For each instruction, it inspects its type (to know what operation to emulate) and its operands (to know how to retrieve the needed values). At the end of the loop, a 64-bit value is returned.

We can verify if the emulator is correct by comparing the results returned from the emulator against the results we captured earlier:

for i in range(0, challenge_funcs_tbl_size):
    func = idc.Dword(challenge_funcs_tbl +  i * 4)
    
    ok, info = scope_challenge_function(func)
    if ok:
        ok, val = emulate_challenge_function(info, 123, 456, dbg)
        if (val != RESULTS[i]):
            print("Mistmatch #%d: %16X vs %16X" % (i, val, RESULTS[i]))
            break
        
    else:
        print("Failed to scope challenge function #%d" % i)

I hope you found this article useful. Please do not hesitate to ask question and/or point out mistakes in this article. You can download the files used in this article from here:

You might also like:

• Blizzard CTF 2017: The LichKing Reverse Engineering challenge walkthrough
• Batchography: Number counting in Batch files