Malware Analysis

Malware classification is the foundational step in any analysis workflow. Understanding the category Of a specimen determines the analyst’s tooling, analysis strategy, and threat assessment posture. Modern malware frequently blends categories, but each type has distinct structural signatures and Behavioral profiles.

File Infector Viruses attach to host executables by prepending, appending, or cavity-infecting The target binary. Prepending viruses place their code at the beginning of the host and redirect Execution flow via a modified entry point in the PE header’s AddressOfEntryPoint field. Appending Viruses attach to the end and modify the entry point accordingly. Cavity infection fills unused Sections within the PE file, preserving the original file size, which makes detection by file-size Heuristics unreliable.

Boot Sector Viruses target the Master Boot Record at sector 0x0000 of the physical disk or the Volume Boot Record of individual partitions. The MBR contains the partition table (offset 0x01BE Through 0x01FD) and a small bootstrap program (446 bytes). A boot sector virus overwrites this Bootstrap code, ensuring execution before any operating system loader. Modern UEFI systems mitigate This through Secure Boot, which verifies a cryptographic signature chain from the firmware through The bootloader.

Macro Viruses exploit the programmability of document formats, primarily Microsoft Office’s VBA (Visual Basic for Applications) runtime. The malicious macro is stored within the OLE2 Compound Document structure in streams such as Macros/VBA/ThisDocument. Defense relies on disabling macros By default, which Group Policy enforces through the registry key HKLM\SOFTWARE\Microsoft\Office\<version>\<app>\Security\VBAWarnings set to 4.

Polymorphic Viruses encrypt their body with a variable key and prepend a mutating decryption Routine. The decryption engine changes its code structure on each infection through instruction Substitution, register reassignment, and code reordering, while preserving semantic equivalence. The Encrypted payload remains identical across generations, but the decryption stub has no stable Signature. Detection requires generic decryption engines that emulate the stub in a sandboxed Environment.

Metamorphic Viruses go further by transforming their entire body, not just an encryption Wrapper. Techniques include register renaming, instruction substitution (e.g., xor eax, eax Replaced by sub eax, eax or mov eax, 0), code permutation through nop insertion and basic-block Reordering, and control flow flattening where the logical sequence is encoded into a state machine Dispatched via a switch-like structure.

Stealth Viruses actively intercept system calls to conceal their presence. In DOS-era Implementations, the virus would hook INT 21h to filter directory reads, removing its own entry from Listings. Modern equivalents hook Windows API functions through Import Address Table patching or Inline hooking, where the first bytes of a function are overwritten with a jump to the malware’s Handler.

Worms are self-replicating programs that propagate across networks without requiring a host file or User interaction. Their replication logic is autonomous. Propagation mechanisms include exploiting Network service vulnerabilities (buffer overflows in SMB, RPC, or web services), copying to Removable media via autorun.inf, sending themselves as email attachments, and scanning for open Network shares.

The Stuxnet worm (discovered 2010) represents the most sophisticated worm ever analyzed. It Targeted Siemens S7-300 and S7-400 PLCs controlling uranium enrichment centrifuges at Natanz, Iran. Stuxnet exploited four Windows zero-day vulnerabilities: CVE-2010-2568 (LNK shortcut parsing), CVE-2010-2729 (print spooler), CVE-2010-3337 (task scheduler), and a WinCC project file Vulnerability. Its propagation path was USB drives via the LNK vulnerability, which executed the Worm merely by displaying the file icon in Windows Explorer, without requiring the user to open the File. The payload injected malicious STEP 7 blocks into the PLC that caused centrifuge speed to Oscillate between 1,410 Hz and 2,106 Hz while reporting nominal 1,064 Hz operation, causing physical Destruction.

Trojans masquerade as legitimate software to deceive the user into executing them. Unlike viruses And worms, they do not self-replicate.

Backdoors open a network listener that accepts commands from a remote attacker. Primitive Backdoors bind a shell (cmd.exe or /bin/sh) to a TCP port. Advanced implementations use covert Channels, such as ICMP tunneling or encoding data within DNS queries, to evade firewall egress Filtering.

Remote Access Trojans (RATs) provide comprehensive system control: file system browsing, screen Capture, webcam activation, keystroke logging, process management, and remote shell access. Notable Families include DarkComet, Poison-Ivy, and njRAT. RATs communicate over HTTP(S) or custom TCP protocols with configurable C2 servers.

Downloaders are lightweight stubs whose sole purpose is to retrieve and execute secondary Payloads from a remote server. This staged approach keeps the initial infection vector small and Allows the attacker to update the payload without re-infecting the host. The downloader may use a Simple HTTP GET request or a more sophisticated protocol with encryption and integrity verification.

Bankers specialize in stealing financial credentials. They inject malicious code into the Browser process (via DLL injection or BHO extension) to capture login forms, modify transaction data In transit, and intercept two-factor authentication tokens. They target specific banking websites Using configuration files that map URL patterns to webinject scripts, which are JavaScript snippets That modify the rendered page to capture additional fields or redirect transactions.

Ransomware encrypts the victim’s files and demands payment for the decryption key. Modern ransomware Operates as a service (Ransomware-as-a-Service or RaaS), where the ransomware authors lease their Infrastructure to affiliates.

The encryption process follows this sequence: enumerate files on all mounted drives, Excluding critical system files to maintain system operability; generate a per-victim or per-file Encryption key using a cryptographically secure random number generator; encrypt file contents using A symmetric cipher (AES-256 in CBC or CTR mode); encrypt the symmetric key with an RSA public key Embedded in the binary; delete the original files or overwrite them; and drop a ransom note with Payment instructions.

Payment mechanisms have evolved from wire transfers to cryptocurrencies, primarily Bitcoin and Monero. Bitcoin provides pseudo-anonymity with a public ledger, while Monero offers stronger privacy Through ring signatures, stealth addresses, and confidential transactions.

WannaCry (2017) exploited the EternalBlue SMB vulnerability (MS17-010) to propagate across Networks with an NSA-developed exploit leaked by the Shadow Brokers. It encrypted files using AES-128 in CBC mode with RSA-2048 key exchange. A kill-switch domain (iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com) was hardcoded in the binary; if the domain Resolved, encryption was aborted. A security researcher registered this domain, inadvertently Halting the initial spread.

NotPetya (2017) was disguised as ransomware but functioned as a destructive wiper. It used the EternalBlue exploit for lateral movement and the Mimikatz tool for credential harvesting. Unlike WannaCry, the encryption was irreversible: it overwrote the MBR and encrypted the Master File Table Using a fixed Salsa20 key embedded in the binary. The “decryption key” displayed in the ransom note Was computationally useless, as the MFT corruption made data recovery impossible through normal Means.

Ryuk (2018) is a targeted ransomware family that focuses on large enterprises, healthcare Organizations, and government agencies. It enters through TrickBot or Emotet botnet Infections and employs manual lateral movement by the operators before deploying the ransomware. The Encryption uses RSA-2048 and AES-256, and ransom demands range from hundreds of thousands to Millions of dollars.

LockBit operates as RaaS and has been one of the most prolific ransomware families. LockBit 3.0 Introduced a bug bounty program and implemented a self-propagation mechanism that automatically Spreads across the network. It uses AES-256 and RSA-2048 for encryption and employs intermittent Encryption to speed up the process on large files.

Rootkits are malware designed to maintain persistent, stealthy access to a system by subverting the OS’s integrity mechanisms.

User-mode rootkits operate in ring 3 and hook application-level APIs. Techniques include Import Address Table hooking, where the malware modifies the IAT of a target process to redirect API calls To malicious code; Detours-style inline hooking, where the first bytes of the target function are Overwritten with a jump instruction to the hook handler; and DLL search order hijacking, where a Malicious DLL placed in the application’s directory is loaded instead of the legitimate system DLL.

Kernel-mode rootkits operate in ring 0 and are significantly harder to detect. Techniques Include modifying the System Service Dispatch Table (SSDT) to redirect system calls; hooking the Interrupt Descriptor Table (IDT) to intercept hardware interrupts; Direct Kernel Object Manipulation (DKOM) to remove process entries from the EPROCESS linked list, making them invisible to task Managers; and filtering I/O request packets through a filesystem filter driver to hide files and Registry keys. Modern kernel-mode rootkits may use signed vulnerable drivers (BYOVD — Bring Your Own Vulnerable Driver) to load their payload, exploiting the fact that Windows requires driver Signing and legacy signed drivers may contain known vulnerabilities.

Detection of kernel-mode rootkits requires analysis from outside the compromised system, By booting from a known-clean OS and examining the disk image, or using hardware-assisted Virtualization to monitor the guest OS from a hypervisor-level vantage point that the rootkit cannot Tamper with.

Keyloggers capture keystrokes by setting a Windows hook via SetWindowsHookEx with the WH_KEYBOARD_LL parameter, or by polling the keyboard device through GetAsyncKeyState. Kernel-mode keyloggers use a filter driver on the keyboard class device stack to intercept IRP_MJ_READ requests, capturing keystrokes before they reach the user-mode application. Advanced Keyloggers may capture window titles and clipboard contents to provide context for the logged Keystrokes.

Screen Scrapers periodically capture the screen contents using the Windows Graphics Device Interface (GDI) or the Desktop Duplication API (IDXGIOutputDuplication). The captured frames are Compressed and transmitted to the C2 server. Some implementations are event-driven, capturing the Screen only when specific applications (e.g., banking websites) are in the foreground.

Information Stealers harvest credentials, session tokens, cryptocurrency wallets, and browser Data. They target specific file locations: Chrome stores credentials in an SQLite database at %LocalAppData%\Google\Chrome\User Data\Default\Login DataEncrypted with DPAPI using the user’s Windows password. Cryptocurrency wallets (Bitcoin Core, Electrum, MetaMask) store private keys in Known file paths. Session cookies are extracted from browser SQLite databases to enable session Hijacking without requiring credential entry. Notable info-stealer families include RedLine, Raccoon, and Vidar.

Adware displays unwanted advertisements, by injecting code into the browser or by Modifying DNS settings to redirect search queries. PUPs include browser toolbars, system optimizers That report fictitious problems, and software bundlers that install additional applications without Clear consent. While not inherently malicious, PUPs often exhibit behaviors that overlap with Malware: persistence mechanisms, browser modification, and data collection. Detection and removal Involve identifying the installed components through registry analysis and file system enumeration.

Packing compresses or encrypts the PE file’s code and data sections, replacing them with a small Decompression stub. At runtime, the stub allocates memory, decrypts the original payload into it, And transfers execution. Packers reduce file size and eliminate static signatures. Common packers Include UPX, Themida, VMProtect, and custom packers used by advanced threat actors. Detection Involves entropy analysis (packed sections exhibit entropy greater than 7.0 on the Shannon scale), Import table anomalies (few or no imports beyond kernel32.dll and ntdll.dll), and section Characteristics that are unusual for their content (e.g., executable and writable simultaneously).

Crypting applies a layer of encryption to the payload with a decryption routine that varies per Build. Unlike packing, the goal is not compression but signature evasion. Crypters are often sold as Services to other malware authors. Multi-layer crypting applies encryption recursively: the Outermost layer decrypts to reveal another encrypted payload, and so on, requiring multiple rounds Of emulation or manual unpacking to reach the original code.

Anti-debugging techniques detect or disrupt debugger attachment. IsDebuggerPresent checks the BeingDebugged flag in the Process Environment Block. NtQueryInformationProcess with ProcessDebugPort queries the debug port handle. CheckRemoteDebuggerPresent checks for a remote Debugger. Timing checks compare QueryPerformanceCounter or RDTSC values before and after Suspected debugging operations; significant delays indicate a debugger is intercepting instructions. The INT 2D instruction triggers an exception that a debugger handles differently from the OS, Allowing the malware to detect its presence. Structured Exception Handling can be abused by Installing an exception handler and then executing an invalid instruction; if the handler is Invoked, no debugger is present (a debugger would intercept the exception first).

Anti-VM techniques fingerprint the execution environment to detect virtualization. CPUID with Leaf 0x1 returns a hypervisor-present bit in ECX bit 31. Checking for VMware-specific registry keys (HKLM\SOFTWARE\VMware, Inc.\VMware Tools), MAC address prefixes (00:0C:29, 00:50:56 for VMware; 08:00:27 for VirtualBox), or driver files (vmmouse.sys``vboxguest.sys) reveals the VM. Hardware Fingerprinting examines the CPU vendor string, number of reported cores, and available memory Against expected values. Malware may also check for artifacts of VM snapshot tools, debugger Breakpoints, or monitoring drivers.

Anti-sandbox techniques detect automated analysis environments. Sandbox evasion includes Checking the uptime (GetTickCount less than 30 minutes suggests a fresh VM), the username (user sandbox``malware are common sandbox usernames), the computer name, the number of processors (sandboxes often run with 1-2 cores), and the available disk space. Delayed execution (sleeping for Minutes or hours) causes the sandbox to time out before the malicious payload activates. Mouse Movement detection via GetCursorPos in a loop identifies headless environments where no user Interaction occurs. Environment-aware malware only exhibits malicious behavior when the checks pass, Remaining dormant in analysis environments.

Persistence ensures the malware survives system reboots and user session changes. Multiple Mechanisms exist, and advanced malware implements several in parallel for redundancy.

Registry Run Keys are the most common persistence mechanism on Windows. Malware adds a value to HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run or the HKCU equivalent, pointing to its Executable. More subtle locations include RunOnce (executed once and then deleted), RunServices RunServicesOnceAnd the Winlogon keys (Shell``Userinit). The Image File Execution Options Registry key (HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\<executable>) With a Debugger value causes Windows to launch the specified debugger whenever the target Executable is run, which malware exploits by setting itself as the “debugger” for a commonly-run Application like sethc.exe or utilman.exe.

Scheduled Tasks provide persistence through schtasks.exe or the Task Scheduler COM API. Malware creates a task that triggers on user logon, system startup, or at regular intervals. Tasks Can be configured to run with SYSTEM privileges regardless of which user is logged in, and the task Definition is stored in an XML file at %SystemRoot%\System32\Tasks\.

Windows Services allow malware to run as a background process with configurable privileges. The Malware installs a service using CreateService (or sc.exe) and configures it to start Automatically on boot. Service executables run in the context of the specified account, which can be LocalSystem for maximum privileges. Malware often names its service to resemble legitimate Windows Services (e.g., svchost.exe is a common target for name spoofing, though the actual service Executable path is stored in the registry).

DLL Search Order Hijacking exploits the order in which Windows searches for DLLs when a process Loads. If an application specifies a DLL by name without a full path, Windows searches the Application’s directory first. By placing a malicious DLL with the same name as a missing or Delayed-loaded dependency in the application’s directory, the malware’s DLL is loaded instead. This Is particularly effective when the application runs with elevated privileges, as the malicious DLL Inherits those privileges.

WMI Event Subscriptions provide a fileless persistence mechanism. Malware registers an event Consumer (a script or executable to run) and an event filter (a WQL query that triggers the Consumer). The subscription is stored in the WMI repository and survives reboots. WMI persistence is Difficult to detect because no files are modified on disk and the WMI repository is not monitored by Most file integrity checking tools.

Pass-the-Hash attacks reuse the NTLM hash of a user’s password to authenticate to remote systems Without knowing the plaintext password. The attacker obtains the hash from memory (via Mimikatz or LSASS dump) or from the SAM database. The hash is then used with tools like psexec``wmicOr the Impacket library’s smbexec to authenticate to SMB services on remote machines. Kerberos Authentication mitigates this by using tickets rather than password hashes, but NTLM fallback may Still be available.

WMI (Windows Management Instrumentation) provides a powerful lateral movement channel. An Attacker can execute arbitrary commands on remote systems via the Win32_Process class’s Create Method through DCOM/RPC. The command executes under the security context of the WMI service, which Runs as SYSTEM by default. WMI lateral movement is attractive because it uses legitimate OS Functionality, generates no new files on disk (if PowerShell or WMI event subscriptions are used), And is rarely monitored by traditional security tools.

PSExec is a Sysinternals tool that copies a service executable to the remote system’s admin Share and creates a service to execute it. Attackers use PSExec or reimplementations in frameworks Like Metasploit and CrackMapExec to run arbitrary code on remote systems. Detection involves Monitoring for service creation events (Event ID 7045) and admin share access.

RDP (Remote Desktop Protocol) abuse involves brute-forcing or credential-stuffing RDP access, Then using the remote desktop session to execute commands manually or via scripts. RDP tunneling Through tools like ngrok or reverse SSH tunnels allows the attacker to bypass network segmentation And firewall rules that block inbound RDP.

Polling-based C2 has the malware contact the server at regular intervals (seconds to hours) Using HTTP(S) GET or POST requests. The response may contain encoded commands, URLs for secondary Payload downloads, or configuration updates. To blend with legitimate traffic, malware may use Popular domains as cover (domain fronting through CDN edge servers), mimic user-agent strings of Common browsers, and embed commands within seemingly innocuous responses (e.g., within image Metadata or cookie values).

Domain Generation Algorithms (DGAs) produce a large list of pseudo-random domain names from a Seed value (date, keyword, or embedded constant). The malware generates and attempts to resolve Hundreds of domains daily. The operator registers one of the generated domains in advance, creating A rendezvous point that changes daily. This makes domain blocklisting ineffective because the active Domain is unpredictable without knowledge of the DGA algorithm and seed.

DNS Tunneling encodes data within DNS queries and responses to exfiltrate information or receive Commands while bypassing firewall rules that allow outbound DNS traffic. The malware Encodes data as subdomain labels (e.g., encoded_data.malware.com) and the authoritative DNS server For the malicious domain decodes the query and returns the response in TXT or CNAME records. Bandwidth is limited by DNS packet size ( 255 bytes per label) and query rate, but it is Sufficient for command delivery and small data exfiltration.

Fast-Flux Networks use a pool of compromised hosts (proxies) with rapidly changing IP addresses Mapped to a single domain via short DNS TTL values. When the malware resolves the C2 domain, it Receives a different IP address each time, all of which are proxies that forward traffic to the Actual C2 server. This provides resilience against IP-based blocking and makes takedown operations Difficult because the actual server’s IP is never exposed.

Process Hollowing creates a legitimate process in a suspended state, unmaps its memory, injects Malicious code, and resumes execution. The steps are: CreateProcess with CREATE_SUSPENDED to Launch a legitimate process (e.g., svchost.exe); NtUnmapViewOfSection to free the original code Section; VirtualAllocEx to allocate memory in the target process; WriteProcessMemory to write The malicious payload; SetThreadContext to modify the RIP register of the suspended thread to Point to the injected code; and ResumeThread to begin execution. The process appears legitimate in The process list (correct name, parent process, and command line) but runs arbitrary code.

DLL Injection forces a target process to load a malicious DLL. The classic method uses VirtualAllocEx to allocate memory in the target process, WriteProcessMemory to write the DLL Path, and CreateRemoteThread with LoadLibrary as the start routine to load the DLL in the Target’s address space. More advanced methods include reflective DLL injection, where the DLL maps Itself into memory without calling LoadLibraryAnd process Doppelganging, which exploits NTFS Transactions to create a process from a modified file without writing to disk.

Reflective DLL Loading enables a DLL to load itself into a process’s address space without using The standard Windows loader (LoadLibrary). The technique involves parsing the PE header of the DLL In memory, allocating memory for each section, resolving import dependencies by walking the PEB’s Loaded module list, processing relocations, and calling the DLL’s entry point (DllMain). This Avoids API hooks on LoadLibrary and GetProcAddress and does not create entries in the loader’s Module list, making the loaded DLL invisible to tools that enumerate loaded modules.

The first step in static analysis is establishing the specimen’s identity and provenance.

Cryptographic Hashes provide a unique fingerprint. MD5 produces a 128-bit digest, SHA-256 Produces a 256-bit digest. While MD5 is cryptographically broken for collision resistance, it Remains useful for lookup against legacy threat intelligence databases. SSDeep generates Context-triggered piecewise hashes (CTPH) that can identify similar malware variants through fuzzy Matching, as opposed to exact hash matching which fails when the binary is modified. IMPHash Computes a hash of the PE file’s imported functions, providing a signature that is resilient to code Section modifications but sensitive to the set of API functions used.

File Type Identification uses magic bytes at the beginning of the file. A PE file begins with The DOS stub signature MZ (0x4D, 0x5A) at offset 0, with the PE signature PE\0\0 at the offset Specified in the e_lfanew field (offset 0x3C, 4 bytes, little-endian). ELF files begin with 0x7F ELF. Identifying the correct file type prevents misanalysis of polyglot files or mislabeled Extensions.

PE Structure Analysis examines the Portable Executable format in detail. The DOS Header (64 Bytes) contains the e_lfanew pointer to the PE signature. The PE Signature is 4 bytes: PE\0\0. The COFF File Header (20 bytes) specifies the machine type (0x8664 for x64, 0x014C for x86), number of Sections, and timestamp. The Optional Header contains the AddressOfEntryPoint (the RVA where Execution begins), the ImageBase (preferred load address), and the DataDirectory entries (import Table, export table, resource table, etc.). The Section Headers define the memory layout: .text For code, .rdata for read-only data, .data for writable data, .rsrc for resources. Analyzing Section entropy, sizes, and characteristics (executable, readable, writable flags) reveals packing And anomalous structures.

Extracting human-readable strings from the binary reveals URLs, IP addresses, file paths, error Messages, registry keys, mutex names, and embedded credentials. The strings utility extracts ASCII Strings of minimum length 4 by default; the FLOSS tool additionally decodes obfuscated strings found In stack-based string references, which are common in malware that constructs strings on the stack At runtime to avoid static detection. Strings analysis is quick and often yields immediate Intelligence about the malware’s functionality and targets.

The Import Address Table lists the Windows API functions that the binary calls. Analyzing imports Provides a functional profile of the malware before any code is examined.

Import analysis also identifies the specific libraries referenced. A binary that imports only kernel32.dll and ntdll.dll is suspiciously minimal, suggesting the imports are resolved Dynamically via GetProcAddress or LdrGetProcedureAddress to evade static import analysis. Delay-loaded imports (specified in the PE optional header’s DataDirectory index 13) are resolved Only when the imported function is first called, which may never happen in a sandbox environment.

Disassembly converts binary machine code into assembly language mnemonics, enabling the analyst to Understand the program’s logic at the instruction level.

X86/x64 instructions consist of an optional prefix byte (lock, rep, segment override), the opcode (1-3 bytes), the ModR/M byte (encoding the addressing mode and register operands), the SIB byte (Scale-Index-Base for complex addressing modes), and an optional displacement (1, 2, or 4 bytes) and Immediate value (1, 2, 4, or 8 bytes). Understanding this encoding is essential for recognizing Hand-crafted assembly or shellcode that may not follow compiler-generated patterns.

Control flow reconstruction identifies the program’s structure by tracing jumps and calls. Conditional jumps (je``jne``jg``jlEtc.) encode if-else and loop structures. call Instructions mark function boundaries and return points. ret instructions indicate function Epilogues. Indirect jumps (jmp rax) are used for switch statements, virtual function calls, and Obfuscation (jump tables, computed gotos). Identifying and resolving indirect jumps is critical for Building an accurate control flow graph.

IDA Pro (Hex-Rays) is the industry-standard commercial disassembler and decompiler. It supports Over 100 processor architectures and file formats, provides interactive disassembly with graph views Of control flow, and offers the Hex-Rays decompiler plugin that produces C-like pseudocode from Assembly. IDAPython enables scripting and plugin development. Its cross-reference analysis (xrefs) Is among the most comprehensive, tracking data and code references across the entire binary.

Ghidra (NSA) is a free, open-source reverse engineering suite with a built-in decompiler, Disassembler, and scripting support (Java, Python via Jython, and a C-based scripting API). Ghidra Supports collaborative reverse engineering through a server-based project model. Its decompiler Output quality is comparable to Hex-Rays for most binaries, and its extensibility through plugins And scripts makes it a powerful alternative to IDA Pro.

radare2 is an open-source command-line framework for reverse engineering. It provides Disassembly, debugging, analysis, and scripting capabilities. Its command-line interface is Efficient for batch processing and pipeline integration. R2ghidra integrates Ghidra’s decompiler Into the radare2 workflow. Radare2 is particularly useful for automated analysis and integration Into malware analysis pipelines.

Binary Ninja is a commercial reverse engineering platform with a modern API-driven architecture. Its intermediate language (BNIL) enables analysis plugins that operate independently of the target Architecture. The API is accessible through Python and C++, making it suitable for custom analysis Tooling. Its graph views and type system provide an intuitive interface for navigating complex Binaries.

Dynamic analysis executes the malware in an isolated, instrumented environment to observe its Runtime behavior.

Cuckoo Sandbox is an open-source automated malware analysis system. It executes the specimen in A virtual machine ( Windows) while monitoring file system, registry, network, and process Activity through a suite of monitors. The analysis results are compiled into a structured report With screenshots, network PCAP data, dropped files, and behavioral signatures. Cuckoo’s modular Architecture allows customization of the analysis environment, network routing, and reporting Format.

Joe Sandbox provides hybrid analysis combining static and dynamic techniques. It executes the Specimen in multiple environments (different OS versions, geographic locations) and applies deep Analysis to extracted artifacts. Joe Sandbox’s HTML and PDF reports include behavioral indicators, MITRE ATT&CK mappings, and threat intelligence correlations.

ANY.RUN offers interactive malware analysis where the analyst can interact with the malware in Real-time within a browser-based sandbox. This is particularly valuable for analyzing malware that Requires user interaction (e.g., ransomware that waits for a specific trigger) and for investigating Behaviors that automated sandboxes may miss due to timeout limitations.

Behavioral monitoring captures the malware’s actions during execution, providing a functional Profile without requiring code-level understanding.

File system monitoring tracks created, modified, moved, and deleted files. Of particular Interest are dropped files (secondary payloads), modified system files, and files in unusual Locations. File system changes are monitored through Windows API hooks (on CreateFile WriteFile``DeleteFile) or through filesystem filter drivers for kernel-level monitoring.

Registry monitoring captures modifications to the Windows registry, including new keys and Values, modifications to existing values, and key deletions. Registry changes reveal persistence Mechanisms, configuration modifications, and system settings manipulation. Monitoring is performed Through API hooks on RegSetValueEx``RegCreateKeyEx``RegDeleteValueAnd related functions.

Process monitoring tracks process creation and termination, DLL loading, thread creation, and Memory allocation. Process relationships (parent-child) reveal the malware’s execution chain. Process injection is detected by monitoring for VirtualAllocEx with PAGE_EXECUTE_READWRITE WriteProcessMemory to a remote process, and CreateRemoteThread with a start address in unbacked Memory.

Network monitoring captures DNS queries, HTTP(S) requests, TCP/UDP connections, and data Exfiltration. Network behavior reveals C2 communication patterns, data theft, and propagation Attempts. Monitoring is performed through network interface tapping or by hooking socket-related APIs (connect``send``recv``WSASend``WSARecv).

Memory forensics examines the contents of a system’s RAM to detect artifacts that are not visible on Disk, such as injected code, decrypted payloads, and transient data structures.

The Volatility Framework is the primary tool for memory forensics. Volatility 3, written in Python, supports analysis of memory dumps from Windows, Linux, and macOS systems. Key plugins Include windows.pslist (listing processes), windows.psscan (scanning for terminated or hidden Processes), windows.dlllist (listing loaded DLLs per process), windows.malfind (detecting Injected code by finding memory regions with PAGE_EXECUTE_READWRITE protection that are not backed By a file on disk), windows.dumpfiles (extracting files from memory), and windows.hivescan (recovering registry hives).

Process Memory Dumping extracts the complete memory space of a specific process, enabling Analysis of the running malware’s state. This includes the decrypted payload (for packed malware), The import address table with resolved function addresses, loaded modules, heap data, and stack Contents. Tools like procdump (Sysinternals) or Volatility’s memmap plugin facilitate this Extraction.

Injection Detection identifies code that has been injected into a process. Indicators include Memory regions with PAGE_EXECUTE_READWRITE permissions that are not backed by a file (unbacked Executable memory), threads whose start address resides in unbacked memory, and modified VAD (Virtual Address Descriptor) entries. Volatility’s malfind plugin automates this detection by Scanning process memory for regions matching these criteria and disassembling the injected code.

PCAP Analysis examines captured network traffic using tools like Wireshark (GUI) and tshark (CLI). Analysis focuses on identifying C2 communication patterns, data exfiltration, and lateral Movement. Key artifacts include DNS queries to suspicious domains, HTTP requests with unusual User-agent strings or headers, beacon patterns (regular intervals between connections), and large Data transfers to external IP addresses.

DNS Query Analysis examines DNS requests for indicators of malicious activity. DGA-generated Domains exhibit high character entropy and are queried in bulk. DNS tunneling produces high volumes Of queries with encoded data in subdomain labels. Fast-flux networks produce frequent queries for The same domain returning different IP addresses. DNS monitoring is often the first indicator of C2 Communication, as malware resolves the C2 domain before establishing a connection.

HTTP Traffic Analysis examines request and response bodies, headers, and patterns. Malware C2 Communication over HTTP often uses specific headers for command identification, encoded payloads in POST bodies, and steganographic techniques to embed data within image or document responses. SSL/TLS Inspection (via MITM proxying in the sandbox) is necessary to analyze HTTPS-based C2 channels, Though certificate pinning implemented by the malware may prevent this.

X86-64 architecture extends the 32-bit x86 ISA with 64-bit registers, a larger address space, and Additional registers. The 16 general-purpose registers are RAX``RBX``RCX``RDX``RSI``RDI RBP``RSPAnd R8 through R15. The lower 32 bits are accessible as EAX``EBXEtc., and The lower 16 and 8 bits have their own names (AX``AH``AL). RIP is the instruction pointer, And RFLAGS contains status flags (zero flag, carry flag, sign flag, etc.).

The instruction set includes data movement (mov``lea``push``pop), arithmetic (add``sub mul``div``inc``dec``neg), bitwise operations (and``or``xor``not``shl``shr), Comparison (cmp``test), control flow (jmp``je``jne``call``ret), and memory operations. lea (Load Effective Address) computes an address without accessing memory, commonly used for Arithmetic: lea rax, [rbx + rcx*4 + 0x10] computes rbx + rcx*4 + 16 and stores the result in rax.

Memory operands use the format [base + index*scale + displacement]Where base and index are Registers, scale is 1, 2, 4, or 8, and displacement is a 32-bit signed constant. This addressing Mode supports array access, structure field access, and stack variable access.

Calling conventions define how function arguments are passed, how the stack is managed, and who is Responsible for cleanup. Understanding these is essential for reconstructing function prototypes From disassembly.

cdecl (used in 32-bit C code): arguments are pushed right-to-left onto the stack; the caller Cleans the stack after the call (add esp, N). Variadic functions (like printf) require cdecl Because only the caller knows how many arguments were passed.

stdcall (used in Win32 API): arguments are pushed right-to-left; the callee cleans the stack (ret N). This reduces code size slightly compared to cdecl because the cleanup instruction appears Once in the callee rather than at every call site.

Microsoft x64 Calling Convention (used in 64-bit Windows): the first four arguments are passed In RCX``RDX``R8``R9 (for integer and pointer types); additional arguments are pushed onto The stack right-to-left. The caller allocates 32 bytes of shadow space on the stack before the call (even if fewer than four arguments are passed), which the callee can use to spill register Arguments. RAX holds the return value. Registers RBX``RBP``RDI``RSI``RSP``R12-R15 Are non-volatile (callee-saved); all others are volatile (caller-saved). Stack alignment must be Maintained at 16-byte boundaries before a call instruction.

Conditionals: The cmp instruction subtracts the source from the destination and sets flags Without storing the result. The test instruction performs a bitwise AND and sets flags. Conditional jumps follow, using the flag state: je/jz (jump if equal / zero flag set), jne/jnz (jump if not equal / zero flag clear), jg/jnle (jump if greater, signed), jl/jnge (jump if less, signed), ja/jnbe (jump if above, unsigned), jb/jnae (jump if Below, unsigned). An if-else structure compiles to cmp; jcc false_branch; ... true_branch ...; jmp end; false_branch: ...; end:.

Loops: Counted loops use ecx/rcx as a counter with loop (decrement and jump if not zero). Compiled loops use cmp or dec/inc with a conditional jump. do-while loops place The condition check at the end (jump back to the body). while loops check the condition at the Beginning (jump past the body if false). for loops combine initialization, condition, and Increment into a single structure.

Switch statements: Compilers implement switch statements using jump tables (an array of target Addresses indexed by the switch variable) for dense case values, or chained conditional jumps for Sparse case values. A jump table implementation loads the switch variable, verifies it is within the Valid range, and then uses an indirect jump: lea rax, [jump_table]; movsxd rcx, dword [rax + rdi*4]; add rcx, rax; jmp rcx. Identifying jump Tables in disassembly is important for understanding the control flow of switch-heavy code, which is Common in protocol parsers and command dispatchers.

Functions: The call instruction pushes the return address onto the stack and jumps to the Target. The ret instruction pops the return address and jumps to it. Stack-based buffer overflows Exploit the fact that the return address is stored on the stack; overwriting it with a controlled Value redirects execution when the function returns. Compiler security features include stack Canaries (a random value placed between local variables and the return address, checked before ret), Address Space Layout Randomization (ASLR), and Data Execution Prevention (DEP/NX).

Identifying cryptographic algorithms in malware is critical for understanding data protection and Potentially recovering encrypted data.

AES (Advanced Encryption Standard) implementations are identifiable by the S-box lookup table, Which contains the 256-byte substitution box beginning with bytes 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76. The inverse S-box is also commonly present. AES operations include SubBytes (S-box lookup), ShiftRows (byte rotation), MixColumns (Galois field multiplication in GF(2^8)), and AddRoundKey (XOR With round key). AES-NI instructions (aesenc``aesdec``aeskeygenassist) indicate Hardware-accelerated AES, which is common in modern ransomware.

RC4 implementations are identifiable by the Key Scheduling Algorithm (KSA), which initializes a 256-byte state array, and the Pseudo-Random Generation Algorithm (PRGA), which produces the Keystream. The state array initialization loop runs 256 iterations, and the swap operation swap(S[i], S[j]) is a distinctive pattern. RC4 is common in older malware due to its simplicity And small code footprint.

RSA implementations are identifiable by modular exponentiation routines using large integer Arithmetic (multi-precision multiplication and division). Key indicators include large constant Arrays (the public exponent, 65537 or 0x10001, and the modulus), Montgomery multiplication For efficient modular reduction, and the Chinese Remainder Theorem optimization for private key Operations. RSA key sizes of 2048 or 4096 bits are standard in ransomware.

XOR-based encryption is the simplest and most common in basic malware. Single-byte XOR uses a Single key byte to XOR each byte of the plaintext. Detection involves frequency analysis (the most Common byte in the ciphertext, when XORed with the expected most common plaintext byte, reveals the Key). Multi-byte XOR uses a repeating key. Identifying XOR loops in assembly is straightforward: Look for a loop body containing an xor instruction operating on array elements with a single-byte Or multi-byte key.

Traditional Antivirus relies on signature-based detection, comparing file hashes and byte Sequences against a database of known malware signatures. This approach is effective against known Threats but fails against novel, polymorphic, or metamorphic malware that lacks a stable signature. Heuristic analysis augments signatures by analyzing code characteristics: API call sequences, code Structure patterns, and behavioral heuristics that indicate malicious intent. Generic signatures Detect malware families by matching structural attributes (e.g., specific section names, import Combinations) rather than exact byte sequences.

Endpoint Detection and Response (EDR) represents a paradigm shift from reactive detection to Continuous monitoring and response. EDR agents instrument the endpoint’s operating system to collect Telemetry on process creation, file system access, registry modifications, network connections, and Memory allocations. This telemetry is analyzed in real-time using behavioral rules, machine learning Models, and threat intelligence feeds. When suspicious activity is detected, the EDR platform can Automatically isolate the endpoint from the network, kill malicious processes, quarantine files, and Initiate a forensic investigation.

EDR solutions such as CrowdStrike Falcon, Microsoft Defender for Endpoint, Carbon Black, and SentinelOne provide capabilities including exploit prevention (blocking known exploit techniques Like heap spraying and return-oriented programming), behavioral detection (identifying sequences of Actions that constitute an attack chain), fileless attack detection (monitoring for PowerShell Scripts, WMI event subscriptions, and reflective DLL loading), and automated threat hunting (proactively searching endpoints for indicators of compromise associated with known threat actors).

Cloud-delivered protection extends endpoint defenses by offloading analysis to cloud-based Services. When a suspicious file is encountered on the endpoint, metadata (hash, header information, Behavioral context) is sent to the cloud for rapid analysis. The cloud service can run the file in a Sandbox, check against the latest threat intelligence, and return a verdict within seconds. This Approach provides near-real-time protection against emerging threats without requiring signature Updates to be distributed to every endpoint.

YARA is a pattern-matching tool used for identifying and classifying malware based on textual or Binary patterns. YARA rules consist of metadata (author, description, date, reference), a set of Strings (hex patterns, regular expressions, or plain text), and a boolean condition that determines When the rule matches.

rule Suspicious_Packed_Binary {
    meta:
        description = "Detects packed PE files with high entropy"
        author = "malware-analysis-team"
    condition:
        uint16(0) == 0x5A4D and
        entropy(1) > 7.0 and
        pe.number_of_imports < 5
}

YARA rules are used at multiple points in the defense pipeline. On endpoints, YARA scans incoming Files, running processes, and memory dumps. On network gateways, YARA inspects file transfers and Email attachments. In forensic investigations, YARA searches disk images and memory dumps for Indicators of compromise associated with specific threat actors or campaigns.

Advanced YARA techniques include using modules for PE file analysis (the pe module provides access To imports, exports, sections, and resources), the math module for numerical comparisons, and the hash module for matching cryptographic hashes. Regular expressions with wildcard matching enable Detection of obfuscated strings and encoded data. Rules can be organized into hierarchical Namespaces and shared across organizations through threat intelligence platforms.

Indicators of Compromise are observable artifacts that indicate a security breach or malware Infection. IOCs are the output of malware analysis and the input to detection and response systems.

Host-based IOCs include file hashes (MD5, SHA-1, SHA-256) of malware binaries and dropped Payloads; file names and paths used by the malware; registry keys and values created or modified for Persistence or configuration; scheduled tasks and services created by the malware; mutex names used For single-instance enforcement; and memory artifacts such as injected code patterns or suspicious RWX memory regions.

Network-based IOCs include IP addresses of C2 servers; domain names including DGA-generated Domains and legitimate domains used for command relay; URLs used for payload delivery; DNS query Patterns (high-volume queries to suspicious domains, unusual record types); HTTP headers and User-agent strings used in C2 communication; SSL/TLS certificate fingerprints (JA3 hash) used for Server or client identification; and network port numbers used for listening or outbound Connections.

Behavioral IOCs describe sequences of actions that constitute an attack pattern. Unlike atomic IOCs (a single hash or IP), behavioral IOCs capture the intent and technique of the attacker. For Example, a behavioral IOC might describe the sequence: “Process A creates a suspended process of B, Unmaps B’s memory, writes executable code, and resumes B’s execution” (process hollowing). Behavioral IOCs map to the MITRE ATT&CK framework, which provides a standardized taxonomy of Adversary tactics, techniques, and procedures.

IOC management requires a structured repository (often a STIX/TAXII-compliant threat intelligence Platform) that stores, correlates, and distributes indicators across security tools. IOC quality Degrades over time: IP addresses are reassigned, domains expire, and hashes are modified By recompilation. Behavioral indicators and TTP-based detection (technique-based rather than Indicator-based) provide more durable detection coverage.

The incident response lifecycle follows the NIST SP 800-61 framework: preparation, detection and Analysis, containment, eradication, recovery, and post-incident activity (lessons learned).

Preparation involves establishing the incident response team, defining roles and Responsibilities, deploying monitoring and detection tools, and developing playbooks for common Incident types (ransomware, data breach, business email compromise). The team must have access to Forensic tooling (disk imaging, memory acquisition, log analysis), communication channels (secure Out-of-band communication for coordination during active incidents), and legal and compliance Guidance for evidence handling and breach notification requirements.

Detection and Analysis begins with alert triage from security monitoring tools (SIEM, EDR, IDS). The analyst validates the alert, assesses scope (number of affected systems, data exposure), and Identifies the threat actor and their objectives. Analysis techniques include examining EDR Telemetry for the attack chain, reviewing network logs for C2 communication and lateral movement, Analyzing memory dumps for injected code and credential harvesting, and correlating IOCs with threat Intelligence to identify known threat actors and their typical TTPs.

Containment aims to limit the damage and prevent further spread. Short-term containment isolates Affected systems from the network (EDR network isolation, firewall rules, VLAN segmentation) while Preserving evidence. Long-term containment implements temporary fixes that allow business operations To continue while the root cause is addressed. Containment strategies must balance the urgency of Stopping the attack against the need to preserve forensic evidence and maintain business continuity.

Eradication removes the threat from the environment. This includes removing malware from Affected systems, deleting unauthorized accounts, revoking compromised credentials, patching Exploited vulnerabilities, and removing persistence mechanisms (registry keys, scheduled tasks, Services, WMI subscriptions). Eradication must be thorough; incomplete removal allows the attacker To regain access through remaining footholds.

Recovery restores affected systems to normal operation. This includes restoring systems from Known-good backups (verified to be free of malware), rebuilding compromised systems from clean Images, re-enrolling systems in management platforms, and implementing additional monitoring on Recovered systems to detect re-infection attempts. Recovery should be gradual and monitored, with Systems returning to production only after verification.

Post-Incident Activity captures lessons learned to improve future response. A post-mortem review Examines the timeline of the incident, evaluates the effectiveness of detection and response, Identifies gaps in defenses, and produces actionable recommendations. Metrics include mean time to Detect (MTTD), mean time to respond (MTTR), scope of impact, and root cause analysis. Findings are Incorporated into updated playbooks, detection rules, and security architecture improvements.

Threat intelligence transforms raw data about threats into actionable information that supports Decision-making across the security organization.

Strategic intelligence provides high-level context about the threat landscape: adversary Motivations, geopolitical factors driving cyber operations, emerging attack trends, and risk Assessments for the organization’s industry and geographic exposure. Strategic intelligence is Consumed by executive leadership and informs security investment and risk management decisions.

Tactical intelligence describes adversary tactics, techniques, and procedures (TTPs). It answers The question “how” rather than “what” and is mapped to frameworks such as MITRE ATT&CK. Tactical Intelligence enables technique-based detection that is resilient to the adversary changing their Tools and infrastructure. For example, knowing that an adversary uses living-off-the-land techniques (using built-in Windows tools like PowerShell, WMI, and certutil for execution) allows defenders to Monitor for abuse patterns rather than specific file hashes.

Operational intelligence provides context about specific impending attacks against the Organization: campaign details, target industries, intended victims, and timing. This intelligence Enables proactive defense measures such as blocking specific infrastructure, implementing targeted Detection rules, and escalating monitoring on high-value assets.

Technical intelligence consists of specific indicators (IOCs) such as file hashes, IP addresses, Domain names, and YARA rules. Technical intelligence has the shortest shelf life but is immediately Actionable in detection tools. Automated ingestion of technical feeds into SIEM, EDR, and firewall Platforms enables near-real-time blocking and alerting.

Threat intelligence platforms aggregate data from multiple sources (commercial feeds, open-source Intelligence, information sharing communities such as ISACs, internal telemetry, and dark web Monitoring) and correlate it to produce actionable intelligence products. Effective intelligence Programs close the feedback loop: operational outcomes (detected incidents, blocked attacks) inform The relevance and accuracy of consumed intelligence, and gaps in coverage drive collection Requirements.

Application Whitelisting permits only approved executables to run on a system. Windows AppLocker And WDAC (Windows Defender Application Control) enforce policies based on file attributes (path, Publisher signature, hash). Application whitelisting is highly effective against unknown malware Because any unapproved binary is blocked regardless of its behavior, but it requires careful Management of the allow list and can interfere with legitimate software updates.

Exploit Mitigation technologies increase the difficulty of successfully exploiting Vulnerabilities. Address Space Layout Randomization (ASLR) randomizes the base addresses of Executables and DLLs each time the system boots, defeating return-to-libc and return-oriented Programming attacks that rely on known addresses. Data Execution Prevention (DEP) marks non-code Memory pages as non-executable, preventing shellcode execution on the stack or heap. Control Flow Guard (CFG) validates indirect branch targets against a bitmap of valid destinations, preventing Control flow hijacking through corrupted function pointers. Arbitrary Code Guard (ACG) prevents Dynamic code generation, blocking JIT compilers and reflective DLL injection. These mitigations are Enforced at the hardware and OS level, requiring no application changes.

Network Segmentation limits the blast radius of a compromise by dividing the network into zones With controlled communication paths. Micro-segmentation extends this principle to individual Workloads, enforcing least-privilege network access at the VM or container level. Effective Segmentation prevents lateral movement by ensuring that a compromised host in one segment cannot Reach hosts in other segments except through controlled choke points where monitoring and inspection Are applied.

Privilege Restriction follows the principle of least privilege. User accounts should not have Administrative rights for daily operations. Local Administrator Password Solution (LAPS) randomizes And rotates the local administrator password on each endpoint, preventing pass-the-hash lateral Movement using a common local admin credential. Just-In-Time (JIT) privileged access management Grants administrative rights only when needed and only for the duration of the approved task, with Full logging of all privileged actions.

Email Security addresses the primary initial access vector for many attacks. Domain-based Message Authentication, Reporting, and Conformance (DMARC) prevents email spoofing by aligning the Sender’s domain with the domain in the email’s From header. Sender Policy Framework (SPF) specifies Which mail servers are authorized to send email for a domain. DomainKeys Identified Mail (DKIM) adds A cryptographic signature to outbound email. Email filtering sandboxes analyze attachments and URLs In incoming messages, blocking malicious content before it reaches the user’s inbox. Security Awareness training reduces the effectiveness of phishing attacks by teaching users to recognize Suspicious emails, verify sender identity, and report potential threats.

1. Writing in the present tense about historical events. Use the past tense consistently.

2. Confusing causes, events, and consequences. Be clear about chronological and causal relationships.

3. Failing to evaluate the reliability and provenance of sources before using them as evidence.

4. Presenting a one-sided argument without considering alternative interpretations or counter-evidence.

The key principles covered in this topic are linked in the sub-pages above. Focus on understanding the definitions, applying the formulas or frameworks, and evaluating strengths and limitations of each approach.

Worked examples demonstrating the application of key concepts are covered in the detailed sub-pages linked above.