Rootkits are an awesome technology, maintaining access to a victims system after they have been compromised. But how does a rootkit work? First of all the rootkit must be categorised. There are five types of rootkits; library, application, firmware, kernel and virtualised.
This post deals with Application level rootkits, which are the most basic. Unsurprisingly this makes it the easiest to detect. On a Linux machine the terminal application is stored at /bin/bash (/bin/sh is also a linux shell, though we will ignore that for the moment.) By removing the legitimate terminal application, and replacing it with the binary (compiled version of the rootkit application) that is conveniently called bash, we can trick the user into running that rootkit, when they think they are running the terminal. A major problem with just replacing the binary is dependency; if another application calls your binary is it robust enough to handle it? Generally to avoid anything conspicuous the legitimate code will also be run, and therefore this problem is worked around. Often this will be done in the form of patching, where the malicious rootkit code will be ether spliced into the application, or there will be some form of execution subversion so that the rootkits code will be run. The Issue with patching is that it increases file size. Ideally the way around this is to use a detour patch to jump out of the program into another, execute the malicious code, and then jump back into the original code. Generally speaking a detour patch will contain two jumps out of the application; the first to initialize data and execute the patch, the second to clean up to avoid knocking the application/system over.
Again, by keeping the rootkit in userland we avoid the danger of knocking over the system. However, as anti-virus is userland based they are far more likely to detect the rootkit (ether through a recognition of the binary signature or through heuristic (behaviour) analysis. Other types of rootkit, though more difficult to code and implement have the advantage of being far harder to detect and remove.