Now that we have the basic knowledge provided, we can now start working on reverse engineering small programs. Reverse engineering techniques can fall under two categories. The first would be static analysis and the second is dynamic analysis. With static analysis you find as much as you can about the program without even running it. Dynamic analysis involves running the program and tracing through it to get a better idea of its behavior. This tutorial will use a mix of both techniques together.

I’ll demonstrate how to perform reverse engineering using tools that already exist on practically every Linux computer. There are other tools that you need install, but since I intend for this to be a fast introduction to concepts, I’d rather not wait on people following to install tools. Also, some of the better tools cost money and I want to show that learning basic reverse engineering can still be done with a minimal budget. I’ll provide in later posts introductions on other tools that I use. For now, I am sticking with objdump and gdb.

GDB basic commands

To open a program in gdb you run gdb program. While there there are a few commands that we use that we should be familiar with. In gdb there most commands have a short version. Since I don’t like typing long commands, I usually use the shortcut versions. For convenience here is a list of some shortcuts I use with some descriptions. For each of the above commands I recommend checking the relevant help section by typing help [COMMAND] within gdb.

Shortcut Command Description
b break [location] Set a breakpoint at [location]
c continue continue after stopped at a break point
i r info registers display the registers
si stepi step to the next instruction, going into subroutines
ni nexti step to the next instruction, going through subroutines
disass disassemble print a disassembly of a section
x/FMT ADDRESS examine Used to examine memory contents at ADDRESS using the format FMT.

Before we continue, lets make sure that gdb is setup to display Intel syntax. To do so, you need the following contents in your ~/.gdbinit file

set disassembly-flavor intel

Starting example

To start off, let us write a simple program and see how it looks like when compiled.

#include <stdio.h>

int main(int argc, char** argv)
    printf("b01ler up!\n");
    return 0;

Go ahead and compile this program

$ gcc main.c -o b01lerup

Next we want to get a disassembly for the compiled program. For that, we need to use objdump.

$ objdump -D -M intel b01lerup

What this does is it tells objdump to disassemble all sections of the program, and output in Intel assembly syntax. By default Unix utilities use AT&T syntax which can be hard to read. And honestly, most modern tools use Intel syntax anyway ;).

First thing you’ll notice is that there is a LOT of output. I’ll parse out the relevant sections here for you.

000000000000067a <main>:
 67a:   55                      push   rbp
 67b:   48 89 e5                mov    rbp,rsp          ; Prologue
 67e:   48 8d 3d 9f 00 00 00    lea    rdi,[rip+0x9f]        # 724 <_IO_stdin_used+0x4>
 685:   e8 c6 fe ff ff          call   550 <puts@plt>   ; Call print
 68a:   b8 00 00 00 00          mov    eax,0x0
 68f:   5d                      pop    rbp              ; Epilogue
 690:   c3                      ret    

The first line is simply tells us that this is the symbol for the main function at that address. After that there we see two instructions that are used as the function prologue.

The next instructions directly correspond to the printf call. Here we load into a register the first argument for printf, which the compiler happily optimized into a puts call for us, then we call the correct function.

 67e:   48 8d 3d 9f 00 00 00    lea    rdi,[rip+0x9f]        # 724 <_IO_stdin_used+0x4>
 685:   e8 c6 fe ff ff          call   550 <puts@plt>

The curious reader would wonder where is our string located? If you look at the printf man page you will notice that the argument is a pointer to a format string: int printf(const char *format, ...);

The first line with an lea instruction tells the CPU to load the effective address that is 0x9f bytes ahead of the next instruction pointer rip. Thankfully, objdump was smart enough to calculate what that address is for us. If we search for that address in the disassembly we can see the following segment.

Disassembly of section .rodata:

0000000000000720 <_IO_stdin_used>:
 720:   01 00                   add    DWORD PTR [rax],eax
 722:   02 00                   add    al,BYTE PTR [rax]
 724:   62                      (bad)
 725:   30 31                   xor    BYTE PTR [rcx],dh
 727:   6c                      ins    BYTE PTR es:[rdi],dx
 728:   65 72 20                gs jb  74b <__GNU_EH_FRAME_HDR+0x1b>
 72b:   75 70                   jne    79d <__GNU_EH_FRAME_HDR+0x6d>
 72d:   21 00                   and    DWORD PTR [rax],eax

The address 0x724 is in the .rodata, or data, section of the program. If we look at the bytes starting from 0x724 onwards we can actually see our string represented byte by byte as ASCII values ending with 00. To prove that, I will convert the characters in an ugly python one liner.

>>> [chr(x) for x in [0x62,0x30,0x31,0x6c,0x65,0x72,0x20,0x75,0x70,0x21,0x00]]
['b', '0', '1', 'l', 'e', 'r', ' ', 'u', 'p', '!', '\x00']

This string ends with a null byte as in C strings are null terminated. Another way to identify the content of strings is a useful Linux utility conveniently called strings. If you run strings b01lerup you will see an output of every null terminated sequence of characters in the program among them is the text b01ler up!. Using this utility can also give you some insight as the contents of a program even before you run a disassembler! Warning though, there have been exploits attacking strings so be careful.

As a fun note, you can hot-patch programs in a hex editor to change its behavior. Try it by changing the string itself to whatever you want and running the program!

After the return from the printf/puts call, we get the assembly for our return 0; line.

 68a:   b8 00 00 00 00          mov    eax,0x0
 68f:   5d                      pop    rbp
 690:   c3                      ret    

Here we set the register EAX to 0 as the calling convention defines the AX register to be used for returns. Next, the program runs the epilogue part of our function, and then return to the calling function. Yes, even the main function has a caller.

We were able to identify the behaviour of our program purely statically.

Passing Arguments and Calling Conventions

We will be using gdb for the remainder of the examples. Lets start by displaying the different calling conventions of x86 and x86_64. Compile the following program with gcc -m32 to get a 32 bit executable.

#include <stdio.h>
int main() {
    printf("%d, %d, %d, %d, %d, %d, %d, %d\n", 1,2,3,4,5,6,7,8);
    return 0;

Open the program with gdb and disassemble the main function as follows:

(gdb) disassemble main
Dump of assembler code for function main:
   0x0000053d <+0>:	lea    ecx,[esp+0x4]
   0x00000541 <+4>:	and    esp,0xfffffff0
   0x00000544 <+7>:	push   DWORD PTR [ecx-0x4]
   0x00000547 <+10>:	push   ebp
   0x00000548 <+11>:	mov    ebp,esp
   0x0000054a <+13>:	push   ebx
   0x0000054b <+14>:	push   ecx
   0x0000054c <+15>:	call   0x589 <>
   0x00000551 <+20>:	add    eax,0x1aaf
   0x00000556 <+25>:	sub    esp,0xc
   0x00000559 <+28>:	push   0x8          ; Start pushing arguments for printf
   0x0000055b <+30>:	push   0x7
   0x0000055d <+32>:	push   0x6
   0x0000055f <+34>:	push   0x5
   0x00000561 <+36>:	push   0x4
   0x00000563 <+38>:	push   0x3
   0x00000565 <+40>:	push   0x2
   0x00000567 <+42>:	push   0x1
   0x00000569 <+44>:	lea    edx,[eax-0x19ec]
   0x0000056f <+50>:	push   edx
   0x00000570 <+51>:	mov    ebx,eax
   0x00000572 <+53>:	call   0x3d0 <printf@plt>
   0x00000577 <+58>:	add    esp,0x30
   0x0000057a <+61>:	mov    eax,0x0
   0x0000057f <+66>:	lea    esp,[ebp-0x8]
   0x00000582 <+69>:	pop    ecx
   0x00000583 <+70>:	pop    ebx
   0x00000584 <+71>:	pop    ebp
   0x00000585 <+72>:	lea    esp,[ecx-0x4]
   0x00000588 <+75>:	ret
End of assembler dump.

If you are wondering about the call at 0x0000054c, that indicates that my systems version of gcc default compiles with position-independent code. For the purposes of this post that information is irrelevant but do note that sometimes systems have different compilation options that introduce differences in the compiled output. The basis stays the same.

Starting from main+28 we can see the arguments for the printf call being pushed. Lets set a breakpoint and inspect memory around the function call.

Reading symbols from a.out...(no debugging symbols found)...done.
(gdb) break *main+28
Breakpoint 1 at 0x559
(gdb) break *main+53
Breakpoint 2 at 0x572
(gdb) r
Starting program: /home/gh0s1/a.out

Breakpoint 1, 0x56555559 in main ()
(gdb) info registers esp ebp
esp            0xffffccb4	0xffffccb4
ebp            0xffffccc8	0xffffccc8
(gdb) c

Breakpoint 2, 0x56555572 in main ()
(gdb) i r esp ebp
esp            0xffffcc90	0xffffcc90
ebp            0xffffccc8	0xffffccc8
(gdb) x/12xw $esp
0xffffcc90:	0x56555614	0x00000001	0x00000002	0x00000003
0xffffcca0:	0x00000004	0x00000005	0x00000006	0x00000007
0xffffccb0:	0x00000008	0xffffcd74	0xffffcd7c	0x56555551
(gdb) x/s 0x56555614
0x56555614:	"%d %d %d %d %d %d %d %d \n"
(gdb) q

What happened here was that we set two break points, started running the program. When I hit the first breakpoint, the instruction where the fist argument was being passed, we can see the addresses stored by the base pointer, and stack pointer. Continuing over to the next breakpoint, we see that the stack “grew” since the stack pointer has a smaller address (stack grows downwards).

Examining the next 12 words in hexadecimal representation, we see the arguments all placed in memory nicely. The first argument is a pointer to the format string provided. This is because in a Linux 32 bit x86 system follows the CDECL calling convention.

Recompile the program now without the -m32 flag so we can see how 64 bit x86 is different.

Reading symbols from a.out...(no debugging symbols found)...done.
(gdb) disass  main
Dump of assembler code for function main:
   0x000000000000068a <+0>:	push   rbp
   0x000000000000068b <+1>:	mov    rbp,rsp
   0x000000000000068e <+4>:	sub    rsp,0x8
   0x0000000000000692 <+8>:	push   0x8
   0x0000000000000694 <+10>:	push   0x7
   0x0000000000000696 <+12>:	push   0x6
   0x0000000000000698 <+14>:	mov    r9d,0x5
   0x000000000000069e <+20>:	mov    r8d,0x4
   0x00000000000006a4 <+26>:	mov    ecx,0x3
   0x00000000000006a9 <+31>:	mov    edx,0x2
   0x00000000000006ae <+36>:	mov    esi,0x1
   0x00000000000006b3 <+41>:	lea    rdi,[rip+0x9a]        # 0x754
   0x00000000000006ba <+48>:	mov    eax,0x0
   0x00000000000006bf <+53>:	call   0x560 <printf@plt>
   0x00000000000006c4 <+58>:	add    rsp,0x20
   0x00000000000006c8 <+62>:	mov    eax,0x0
   0x00000000000006cd <+67>:	leave
   0x00000000000006ce <+68>:	ret
End of assembler dump.
(gdb) break *main+53
Breakpoint 1 at 0x6bf
(gdb) r
Starting program: /home/gh0s1/a.out

Breakpoint 1, 0x00000001000006bf in main ()
(gdb) i r
rax            0x0	0
rbx            0x0	0
rcx            0x3	3
rdx            0x2	2
rsi            0x1	1
rdi            0x100000754	4294969172
rbp            0x7fffffffdb40	0x7fffffffdb40
rsp            0x7fffffffdb20	0x7fffffffdb20
r8             0x4	4
r9             0x5	5
r10            0x8	8
r11            0x1	1
r12            0x100000580	4294968704
r13            0x7fffffffdc20	140737488346144
r14            0x0	0
r15            0x0	0
rip            0x1000006bf	0x1000006bf <main+53>
eflags         0x212	[ AF IF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
(gdb) x/s $rdi
0x100000754:	"%d %d %d %d %d %d %d %d \n"

As you can see, the first 6 arguments are passed through registers instead of memory with the remaining arguments being passed on through the stack following the System V ABI.

One important note is that the stack memory is not cleared upon the return of a function. As an example from Dennis Yurichev’s reverse engineering for beginners is the following code segment.

#include <stdio.h>

void f1()
    int a=1, b=2, c=3;

void f2()
    int a, b, c;
    printf("%d, %d, %d\n", a, b, c);

int main()

This program will print 1, 2, 3 when ran. But as we see in f2 the function does not initialize any of the variables?

When disassembled we get the following:

(gdb) disassemble main
Dump of assembler code for function main:
   0x00000000000006cd <+0>:	push   rbp
   0x00000000000006ce <+1>:	mov    rbp,rsp
   0x00000000000006d1 <+4>:	mov    eax,0x0
   0x00000000000006d6 <+9>:	call   0x68a <f1>
   0x00000000000006db <+14>:	mov    eax,0x0
   0x00000000000006e0 <+19>:	call   0x6a6 <f2>
   0x00000000000006e5 <+24>:	mov    eax,0x0
   0x00000000000006ea <+29>:	pop    rbp
   0x00000000000006eb <+30>:	ret    
End of assembler dump.
(gdb) disassemble f1 
Dump of assembler code for function f1:
   0x000000000000068a <+0>:	push   rbp
   0x000000000000068b <+1>:	mov    rbp,rsp
   0x000000000000068e <+4>:	mov    DWORD PTR [rbp-0xc],0x1
   0x0000000000000695 <+11>:	mov    DWORD PTR [rbp-0x8],0x2
   0x000000000000069c <+18>:	mov    DWORD PTR [rbp-0x4],0x3
   0x00000000000006a3 <+25>:	nop
   0x00000000000006a4 <+26>:	pop    rbp
   0x00000000000006a5 <+27>:	ret    
End of assembler dump.
(gdb) disassemble f2
Dump of assembler code for function f2:
   0x00000000000006a6 <+0>:	push   rbp
   0x00000000000006a7 <+1>:	mov    rbp,rsp
   0x00000000000006aa <+4>:	sub    rsp,0x10
   0x00000000000006ae <+8>:	mov    ecx,DWORD PTR [rbp-0x4]; not set in this scope
   0x00000000000006b1 <+11>:	mov    edx,DWORD PTR [rbp-0x8]
   0x00000000000006b4 <+14>:	mov    eax,DWORD PTR [rbp-0xc]
   0x00000000000006b7 <+17>:	mov    esi,eax
   0x00000000000006b9 <+19>:	lea    rdi,[rip+0xb4]        # 0x774
   0x00000000000006c0 <+26>:	mov    eax,0x0
   0x00000000000006c5 <+31>:	call   0x560 <printf@plt>
   0x00000000000006ca <+36>:	nop
   0x00000000000006cb <+37>:	leave  
   0x00000000000006cc <+38>:	ret    
End of assembler dump.

We can see that function f1 sets its memory during its call frame to the correct values, and function f2 reads from the same addresses during its call frame. The stack is reset to the same area among the two function calls.

Control flow

When was the last time you wrote a program that consisted solely of sequential instructions? Real program consist of many control flow branches and as a reverse engineer you need to be able to identify patterns of such branches. To demonstrate a few patterns, here are a few small programs that have different control flow branches based on C control constructs.


#include <stdio.h>
int main()
    for(int i = 0; i < 10; i++) {
        printf("i is %d which is ", i);

Compiled it looks like the following. I included comments on the side to help understand the program as well as split up the disassembly by blocks.

Dump of assembler code for function main:
   0x000000000000068a <+0>:	push   rbp
   0x000000000000068b <+1>:	mov    rbp,rsp
   0x000000000000068e <+4>:	sub    rsp,0x10

   0x0000000000000692 <+8>:	mov    DWORD PTR [rbp-0x4],0x0 ; initialize i
   0x0000000000000699 <+15>:	jmp    0x6b5 <main+43>         ; jump to start of loop block

   0x000000000000069b <+17>:	mov    eax,DWORD PTR [rbp-0x4] ; Get the value of i
   0x000000000000069e <+20>:	mov    esi,eax                  ; Pass as second argument to printf
   0x00000000000006a0 <+22>:	lea    rdi,[rip+0xad]        # 0x754 ; Pass string as first arg
   0x00000000000006a7 <+29>:	mov    eax,0x0
   0x00000000000006ac <+34>:	call   0x560 <printf@plt>

   0x00000000000006b1 <+39>:	add    DWORD PTR [rbp-0x4],0x1 ; i++

   0x00000000000006b5 <+43>:	cmp    DWORD PTR [rbp-0x4],0x9 ; Check for the break condition
   0x00000000000006b9 <+47>:	jle    0x69b <main+17>         ; Restart loop

   0x00000000000006bb <+49>:	mov    eax,0x0
   0x00000000000006c0 <+54>:	leave  
   0x00000000000006c1 <+55>:	ret    
End of assembler dump.


#include <stdio.h>
#include <stdlib.h>
int main(int argc, char** argv)
    int i = strtol(argv[1], NULL, 10);

    if (i == 1)
    else if (i == 2)
            printf("Some other number\n");

Compiling in gcc and then disassembling this in gdb yields the following:

Dump of assembler code for function main:
   0x000000000000071a <+0>:	push   rbp
   0x000000000000071b <+1>:	mov    rbp,rsp
   0x000000000000071e <+4>:	sub    rsp,0x20
   0x0000000000000722 <+8>:	mov    DWORD PTR [rbp-0x14],edi
   0x0000000000000725 <+11>:	mov    QWORD PTR [rbp-0x20],rsi
   0x0000000000000729 <+15>:	mov    rax,QWORD PTR [rbp-0x20]
   0x000000000000072d <+19>:	add    rax,0x8
   0x0000000000000731 <+23>:	mov    rax,QWORD PTR [rax]
   0x0000000000000734 <+26>:	mov    edx,0xa
   0x0000000000000739 <+31>:	mov    esi,0x0
   0x000000000000073e <+36>:	mov    rdi,rax
   0x0000000000000741 <+39>:	call   0x5f0 <strtol@plt>
   0x0000000000000746 <+44>:	mov    DWORD PTR [rbp-0x4],eax
   0x0000000000000749 <+47>:	cmp    DWORD PTR [rbp-0x4],0x1
   0x000000000000074d <+51>:	jne    0x762 <main+72>
   0x000000000000074f <+53>:	lea    rdi,[rip+0xbe]        # 0x814
   0x0000000000000756 <+60>:	mov    eax,0x0
   0x000000000000075b <+65>:	call   0x5e0 <printf@plt>
   0x0000000000000760 <+70>:	jmp    0x787 <main+109>
   0x0000000000000762 <+72>:	cmp    DWORD PTR [rbp-0x4],0x2
   0x0000000000000766 <+76>:	jne    0x77b <main+97>
   0x0000000000000768 <+78>:	lea    rdi,[rip+0xa9]        # 0x818
   0x000000000000076f <+85>:	mov    eax,0x0
   0x0000000000000774 <+90>:	call   0x5e0 <printf@plt>
   0x0000000000000779 <+95>:	jmp    0x787 <main+109>
   0x000000000000077b <+97>:	lea    rdi,[rip+0x9a]        # 0x81c
   0x0000000000000782 <+104>:	call   0x5d0 <puts@plt>
   0x0000000000000787 <+109>:	mov    eax,0x0
   0x000000000000078c <+114>:	leave
   0x000000000000078d <+115>:	ret
End of assembler dump.

There are a few things that appear in this disassembly.

First of all, you can see that here we have argv and argc being passed to main. These arguments are immediately copied to local variables from the registers edi and rsi. Next up you see see the code which calls strtol(argv[1], NULL, 10)

We know that argv is an array of strings and we need to do some pointer manipulations to get the argument to the function argv[1]. What happens is we get the address to the start of the array, add the length of a pointer to it since its argv is an array of pointers, then dereference that address to get the value of argv[1]. This is visible in the snippet below.

   0x0000000000000729 <+15>:	mov    rax,QWORD PTR [rbp-0x20] ; Start of the array
   0x000000000000072d <+19>:	add    rax,0x8                  ; Get the offset from argv
   0x0000000000000731 <+23>:	mov    rax,QWORD PTR [rax]      ; Dereference that to get argv[1]

If you look at main+47 you will see the logic for the if case. Here we compare the value of our variable being checked, if it doesn’t match, then we jump to location of the else if block. At the end of if block, we jump to the code for the end of the program.

   0x0000000000000749 <+47>:	cmp    DWORD PTR [rbp-0x4],0x1 		; if(i == 1)
   0x000000000000074d <+51>:	jne    0x762 <main+72>			; jump to else if block
   0x000000000000074f <+53>:	lea    rdi,[rip+0xbe]        # 0x814
   0x0000000000000756 <+60>:	mov    eax,0x0
   0x000000000000075b <+65>:	call   0x5e0 <printf@plt>
   0x0000000000000760 <+70>:	jmp    0x787 <main+109>

   0x0000000000000762 <+72>:	cmp    DWORD PTR [rbp-0x4],0x2 		; if (i == 2) 
   0x0000000000000766 <+76>:	jne    0x77b <main+97>			; jump to else block
   0x0000000000000768 <+78>:	lea    rdi,[rip+0xa9]        # 0x818
   0x000000000000076f <+85>:	mov    eax,0x0
   0x0000000000000774 <+90>:	call   0x5e0 <printf@plt>
   0x0000000000000779 <+95>:	jmp    0x787 <main+109>

Lastly, note that both if cases end in jmp 0x787 <main+109> to avoid executing the else block.

Switch statements

Let’s translate the above code to use switch statements.

#include <stdio.h>
#include <stdlib.h>
int main(int argc, char** argv)
    int i = strtol(argv[1], NULL, 10);

    switch (i) {
        case 1:
        case 2:
            printf("Some other number\n");

Which compiles to the following.

Dump of assembler code for function main:
   0x00000000000006ca <+0>:	push   rbp
   0x00000000000006cb <+1>:	mov    rbp,rsp
   0x00000000000006ce <+4>:	sub    rsp,0x20
   0x00000000000006d2 <+8>:	mov    DWORD PTR [rbp-0x14],edi
   0x00000000000006d5 <+11>:	mov    QWORD PTR [rbp-0x20],rsi
   0x00000000000006d9 <+15>:	mov    rax,QWORD PTR [rbp-0x20]
   0x00000000000006dd <+19>:	add    rax,0x8
   0x00000000000006e1 <+23>:	mov    rax,QWORD PTR [rax]
   0x00000000000006e4 <+26>:	mov    edx,0xa
   0x00000000000006e9 <+31>:	mov    esi,0x0
   0x00000000000006ee <+36>:	mov    rdi,rax
   0x00000000000006f1 <+39>:	call   0x5a0 <strtol@plt>
   0x00000000000006f6 <+44>:	mov    DWORD PTR [rbp-0x4],eax
   0x00000000000006f9 <+47>:	mov    eax,DWORD PTR [rbp-0x4]
   0x00000000000006fc <+50>:	cmp    eax,0x1              ; case 1:
   0x00000000000006ff <+53>:	je     0x708 <main+62>
   0x0000000000000701 <+55>:	cmp    eax,0x2              ; case 2:
   0x0000000000000704 <+58>:	je     0x716 <main+76>
   0x0000000000000706 <+60>:	jmp    0x724 <main+90>
   0x0000000000000708 <+62>:	lea    rdi,[rip+0xb5]        # 0x7c4
   0x000000000000070f <+69>:	call   0x590 <puts@plt>
   0x0000000000000714 <+74>:	jmp    0x730 <main+102>
   0x0000000000000716 <+76>:	lea    rdi,[rip+0xab]        # 0x7c8
   0x000000000000071d <+83>:	call   0x590 <puts@plt>
   0x0000000000000722 <+88>:	jmp    0x730 <main+102>
   0x0000000000000724 <+90>:	lea    rdi,[rip+0xa1]        # 0x7cc
   0x000000000000072b <+97>:	call   0x590 <puts@plt>
   0x0000000000000730 <+102>:	mov    eax,0x0
   0x0000000000000735 <+107>:	leave
   0x0000000000000736 <+108>:	ret
End of assembler dump.

Next up, we have the cases for the switch. Our variables result is saved to a register first which makes sense since in a switch statement, we will be comparing the value often. That register is then compared directly to the case conditions. After the comparison, control flow jumps to segments which perform our operations. For the default case we still have a layer of indirection. This happens because our compiler generates a jump table to follow for switch statements. Different number of cases would generate different assembly for switch statements. For more information about the various optimizations in switch conditions, here is a good read.

Now you should be able to recognize some common patterns in compiled programs. I hope that this post has helped you start with some crackmes and entry CTF challenges. If you have any questions, feel free to ask me :)

Useful resources