Pierluigi Paganini
April 23, 2016
Software reverse engineering is frequently mentioned in the context of illegal activity: the stealing of IP, fraud with software licenses, and so forth.
At the same time, reversing has legal applications, the most known of which is malware research. It can be also used to provide improved compatibility with the closed platforms, enhanced applications, and advanced OS features (see more legal software reverse engineering tasks). In particular, in the United States, there is a legal rule allowing reverse engineering of an object if the license for it was obtained in a legal way and the results won’t be used in any illegal activity.
You can find a number of questions around software reverse engineering process on the Q&A portals. A significant part of them is about the reversing process and initial knowledge. In this short post, we’ll try to provide some basic software reverse engineering tips, how to start, and what a reverser should know.
Basic knowledge: Reverse engineering books
To get a comprehensive impression about software reverse engineering and disassembling in particular, I would recommend starting with a classic book “Hacker Disassembling Uncovered: Powerful Techniques To Safeguard Your Programming” by Kris Kaspersky. Disassembling is frequently called software reversing itself: it supposes the reconstruction of the software source code by its executables.
Great description of practical disassembling techniques is also provided in the “The IDA Pro Book: The Unofficial Guide to the World’s Most Popular Disassembler” by Chris Eagle; IDA Pro is a great disassembling tool, I would say, a reverser must-have for now – as we’ll say below.
The book “Hacking: The Art of Exploitation” by Jon Erickson will provide you a good description of different approaches on how to investigate the software functioning on the system level: how it interacts with the OS and its different components, what processes are involved, etc. It is an important stage of software reverse engineering and research, as we will discuss in next paragraphs.
You could also search for software reverse engineering tutorials on the web – some pretty good works can be found.
Basic knowledge: General programming
To reverse engineer a software object you should be familiar with the ways it is built and functions – just that simple. If it is a Windows driver, you should know about drivers, Windows driver specifics, the way they interact with the systems, etc. if it is a network communication subsystem, you should know about network layers, network exchange, building a distributed applications, etc.
When reversing a software piece, you should be familiar with the typical patterns and translation / compilation nuances of the code in the particular language, as C++ disassembling differs a lot from C# or Java disassembling, for example.
Software reverse engineering process: Main steps
There is no formalized software reverse engineering process with stages and ready recipes. It is a very creative and adaptive process of investigation of a software piece from different perspectives, depending on its specifics and task priorities.
I will try to briefly describe some major steps – typical stages of software research process.
Usually research starts with investigating the process of a software piece functioning. If it is malware research, obviously starting it on your machine is not a good idea, but you can use a virtual machine.
After initial start and observation, you proceed to the in-depth functioning research. To reconstruct software functioning step-by-step, you need to attach a debugger to it.
Attaching a debugger
Debugger is one of the basic reverser tools, as you can hardly conduct any kind of process of software reverse engineering without the possibility to pause a software piece execution to see what is happening. WinDBG and OllyDBG are popular Windows debuggers, and you can use llbd to debug Mac OS / iOS software.
Attaching debugger is not always a trivial task. While malware rarely cares about the protection from research, commercial software usually applies various anti-debugging and other anti-reversing techniques. There are quite a few of them, and each needs its own approach to overcome. The good news for reversers is that virtually any anti-debug protection can be neutralized; it’s only the question of time and efforts. On the other hand, sometimes it needs a lot of resources.
You can learn more it in this article about anti-debug protection techniques and ways to bypass them.
Researching functioning
After you managed to attach a debugger, you can now see what this software piece changes in the surrounding world at each step. When researching, you may be interested in different aspects: what system API this software uses, how it works with the network, what system resources are used, etc. Various tools can help you with it:
There are much more tools you would need: unpackers, stack viewers, module-scheme-builders, etc. You can get more information about various applications to help you when researching software in this article.
Disassembling
Disassembling is one of the core software reverse engineering process steps and is frequently named “reversing” itself as it supposes the restoration of the source code (may be not “word-by-word” but down to the steps, functions, and ideas of method implementation). Disassembling needs you to know Assembly language and main principles of code translation / building as well as general principles of building software starting from OOP and up to the details of typical software architectures.
To work on this stage, you need a disassembler. As I mentioned above, the most powerful and popular disassembler nowadays is IDA by Hex Rays. OllyDBG and WinDBG include in-built disassemblers, which are not that feature-rich but can work for some tasks.
The process of disassembling is like a detective investigation and needs a lot of creativity, intuition, and patience.
Software reverse engineering is an interesting process requiring all your knowledge and software development talents. Though sometimes unjustly labeled as an exclusively illegal practice, it helps to fight malware and improve software systems. But enjoying a hacking charm of it, remember to use your talents for good.
Written by: Dennis Turpitka
Author Bio: Dennis Turpitka, CEO of the Apriorit, is an expert within Digital Security solution business design and development, Virtualization and Cloud Computing R&D projects, establishment and management of Software Research direction. Successful entrepreneur, who organized several security start-ups.
[adrotate banner=”9″]
Edited by Pierluigi Paganini
(Security Affairs –Software Reverse Engineering, hacking)
