And just when you didn’t think it could get worse, the bad guys learn that encryption is a great way of hiding their payload. Hiding the payload from real-time protection, making you think the computer is clean when it is in fact infected. At this point, you might be wondering what the solution is. There isn’t one. At least, not a solution that will protect you from getting infected. More on that later. First, I want to start by explaining what, in this context, encrypted malware is.
If you came here thinking this was an article about ransomware, then I am sorry to have to tell you that this isn’t about ransomware. At least not specifically. Neither is this about malware that uses SSL to encrypt traffic. Again, at least not specifically. This article will focus on malware that is itself encrypted in order to avoid detection. What the actual payload is will not be the focus of this, but rather how encryption is used to evade real-time protection, which is the main defense for most users. I also want to touch on how we currently try to fight it and why this will keep being a problem with the current solutions.
Hidden by crypters
So how do the malware actors hide their code from real-time protection? You might have guessed that it is hidden by encrypting the malware, making it unreadable by traditional file scans and real-time protection. But the answer is a bit more complicated than that, because how would the machine know how to execute the code if it is encrypted? It’s not like the machine can magically know what is in the malicious file while the AV (Anti-virus) doesn’t. This is where the stub comes in.
A stub is a small piece of code that will decrypt the malware before execution. These can be divided into two types: Scantime and Runtime. Scantime stubs will decrypt the malware onto the disk, load it into memory and execute the code. A runtime stub, on the other hand, will load and decrypt the malware directly in memory, often using various Windows API calls. Runtime stubs offer more resilient protection against detection than scantime stubs, as most AVs will only check for malware code in the files on disk, and not in the code loaded directly into memory.
This is a bit simplified (there are some intermediary steps done to help obscure the decryption and loading of the malware), but this is roughly how stubs are used to get the encrypted malware running on the victim machine without the AV detecting anything. The process of encrypting and creating the stubs is done by software which we categorize as crypters. These tools are used both for illegitimate and legitimate use, as some software vendors try to protect their code from reverse engineering by encrypting their executables.
Detecting the stub
Some of you might start to think that the solution must be to scan for the stub code or signature and use it to detect that something is afoul. However, while this might work on known stubs generated by static crypters, the engine would need to have encountered that specific stub before. Additionally, things get more complicated when you get to polymorphic crypters. Polymorphic crypters use a wide range of techniques to alter the resulting stub so that it will be difficult to detect. These techniques range from inserting junk code and shuffling code to generating random variable names. In the end, the stub you get will be different each time one runs the crypter. Therein lies the challenge for AVs to accurately detect the stub.
Now to some of you, it might come as a surprise that crypters has been around for well over a decade. And AV vendors have done their best to try and combat the issue. And some have managed to create countermeasures. Techniques like scanning the memory space for known malicious code is the most widespread solution. However, this isn’t something that is done in real-time. Often this is part of a “Deep Scan” or a “Full Scan”, which not many people do on a regular basis. Also, even if you do scan regularly, you probably don’t scan so often as to catch the malware before it is able to execute itself and do what it was meant to do.
In more recent years, AV vendors have started to experiment with behavior analysis and machine learning while monitoring app behavior during execution, in an attempt to look for suspicious behavior. This will probably improve with time, but as of now, it will probably not be a reliable method of detection.
Why none of these matter
All the solutions I have mentioned have one thing in common, which is also the thing that in my opinion makes them lacking: They can only detect the malware after the infection.
It might be too much to expect that AVs should protect against infection, and I am not stating that with any sarcasm. It might be impossible to create real-time protection against encrypted malware, and we will forever be forced to rely on detection post-infection. This is a big problem, especially if we only rely on memory scans, even if we do run them once a day. During that 24h window, there is plenty of time for malware to execute its code and either spread or gain persistence.
Do I have a solution? No. Not besides training users to be more careful and cautious while the use their computers. In the end, the user is the vanguard in defending against cyber attacks. Without that vanguard, most systems don’t stand a chance, even if you had the latest security software.