{"id":386,"date":"2021-07-22T06:16:08","date_gmt":"2021-07-22T06:16:08","guid":{"rendered":"https:\/\/reversea.me\/?p=386"},"modified":"2021-07-22T06:16:08","modified_gmt":"2021-07-22T06:16:08","slug":"writing-a-volatility-3-plugin","status":"publish","type":"post","link":"https:\/\/reversea.me\/index.php\/writing-a-volatility-3-plugin\/","title":{"rendered":"Writing a Volatility 3 Plugin"},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 6<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span>\n<p><strong>TL;DR<\/strong> We explain how to write a Volatility 3 plugin. In addition, we also explain how to manually install symbol files. If you are interested in this excellent memory forensic framework and want to write your own analysis tools, read on!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Introduction<\/h2>\n\n\n\n<p><a href=\"https:\/\/github.com\/volatilityfoundation\/volatility3\">Volatility 3<\/a> is the newest (and largely anticipated) version of the most popular memory forensic tool. Volatility was actually born as <a href=\"https:\/\/www.blackhat.com\/presentations\/bh-dc-07\/Walters\/Paper\/bh-dc-07-Walters-WP.pdf\">Volatools in a presentation at BH USA 2007<\/a>, quickly becoming the <em>de facto<\/em> tool for memory dump analysis. Initially developed in Python 2, the entire framework has been completely rewritten and redesigned in Python 3. In addition, no more <code>--profile<\/code> is needed! Volatility 3 automatically detects the profiles for all supported operating systems (better late than never, right?). Plugin development has also been improved: plugins are now versioned and can call other plugins directly (<code>ConfObject<\/code> no longer required!). You can <a href=\"https:\/\/www.youtube.com\/watch?v=ozeedYjv5Lw\">check this video of Richard Davis on Youtube to know more about Volatility 3 and its new features<\/a>.<\/p>\n\n\n\n<p>In this post, we&#8217;ll learn how to write a Volatility 3 plugin. So first things first &#8212; you need to download the Volatility 3 source code :). To do so, simply open a terminal on your computer, go to your preferred folder for development work, and clone the official repository:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">git clone https:\/\/github.com\/volatilityfoundation\/volatility3.git<\/code><\/pre>\n\n\n\n<p>To run it, you need Python 3.5.3 or later. You&#8217;ll almost certainly need to have the following additional Python packages on your system:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>pefile 2017.8.1 (or later)<\/li><li>pdbparse<\/li><li>jsonschema<\/li><\/ul>\n\n\n\n<p>Once downloaded, try running it to see if it works. You can also download <a href=\"http:\/\/webdiis.unizar.es\/~ricardo\/sbc-2021\/adicional\/volcados\/alina1G.elf.tar.gz\">this memory dump<\/a> for testing purposes. In the terminal, type the following:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">python3 vol.py -f ~\/volcados\/alina1G.elf windows.info<\/code><\/pre>\n\n\n\n<p>Works? Congratulations! You can skip the next section. Does not? Then keep reading.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Manual Installation of Windows Symbol Files<\/h2>\n\n\n\n<p>Otherwise, some errors will have arisen. You will surely see an output similar to this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">Volatility 3 Framework 1.0.1\nWARNING  volatility3.framework.plugins: Automagic exception occurred: volatility3.framework.exceptions.InvalidAddressException: Offset outside of the buffer boundaries\n\nUnsatisfied requirement plugins.Info.nt_symbols: Windows kernel symbols\n\nA symbol table requirement was not fulfilled.  Please verify that:\n    You have the correct symbol file for the requirement\n    The symbol file is under the correct directory or zip file\n    The symbol file is named appropriately or contains the correct banner\n\nUnable to validate the plugin requirements: ['plugins.Info.nt_symbols']<\/code><\/pre>\n\n\n\n<p>Another big change in Volatility 3 is that it requires the symbol file (in a special JSON file format) from the operating system of the memory dump under analysis to parse it appropriately. By default, Volatility 3 tries to locate this file locally. When it cannot find it, it retrieves the symbol file from the Microsoft Symbol server, converts it to the JSON file format, and stores it locally for future use. In particular, these JSON files are stored in the folder <code>volatility3\/symbols<\/code>. Inside this folder there is another folder for each operating system supported by Volatility (<code>windows<\/code>, <code>mac<\/code>, and <code>linux<\/code>), and within them these JSON files are saved in separate folders. A good read on this topic is available <a href=\"https:\/\/volatility3.readthedocs.io\/en\/latest\/symbol-tables.html\">in the official Volatility 3 documentation<\/a>.<\/p>\n\n\n\n<p>We can drill down into running errors with the output verbosity argument <code>-v<\/code>. This is a count argument, so you can add more <code>v<\/code> to increase verbosity as much as you want. Run the above command again with <code>-vvv<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">python3 vol.py -f ~\/volcados\/alina1G.elf -vvv windows.info\nVolatility 3 Framework 1.0.1\nINFO     volatility3.cli: Volatility plugins path: ['\/Users\/ricardo\/volatility3\/volatility3\/plugins', '\/Users\/ricardo\/volatility3\/volatility3\/framework\/plugins']\n[... omitted ...]\nINFO     volatility3.framework.symbols.windows.pdbconv: Download PDB file...\nDEBUG    volatility3.framework.symbols.windows.pdbconv: Attempting to retrieve http:\/\/msdl.microsoft.com\/download\/symbols\/ntkrnlmp.pdb\/00625D7D36754CBEBA4533BA9A0F3FE22\/ntkrnlmp.pdb\nLevel 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Info.nt_symbols0F3FE22\/ntkrnlmp.pdb\nWARNING  volatility3.framework.plugins: Automagic exception occurred: volatility3.framework.exceptions.InvalidAddressException: Offset outside of the buffer boundaries<\/code><\/pre>\n\n\n\n<p>As shown, there is an error downloading and parsing the PDB file. To solve this problem, we will download and install it manually. First, we need to download the PDB file. You can download it with your preferred web browser or via a terminal with <code>wget<\/code> or <code>curl<\/code>. For instance:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">wget http:\/\/msdl.microsoft.com\/download\/symbols\/ntkrnlmp.pdb\/00625D7D36754CBEBA4533BA9A0F3FE22\/ntkrnlmp.pdb<\/code><\/pre>\n\n\n\n<p>Now we need to create the JSON file. To do this, you need to run a Python script provided with Volatility 3. In the Volatility 3 root folder, run this command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">PYTHONPATH=\".\" python3 volatility3\/framework\/symbols\/windows\/pdbconv.py -f ..\/ntkrnlmp.pdb -g 00625D7D36754CBEBA4533BA9A0F3FE22<\/code><\/pre>\n\n\n\n<p>Note that the value of the <code>-g<\/code> argument is the file identifier used in the download link. As a result, a file called <code>00625D7D36754CBEBA4533BA9A0F3FE2-2.json.xz<\/code> will be created in the current working directory. Now we need to move this file to the folder <code>volatility3\/symbols\/windows\/ntkrnlmp.pdb<\/code> (first you will need to create a folder with the name <code>ntkrnlmp.pdb<\/code>, which corresponds to the name of the downloaded PDB file):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">mkdir volatility3\/symbols\/windows\/ntkrnlmp.pdb\nmv 00625D7D36754CBEBA4533BA9A0F3FE2-2.json.xz volatility3\/symbols\/windows\/ntkrnlmp.pdb\/.<\/code><\/pre>\n\n\n\n<p>Once you complete these steps, the execution of the above command will be successful:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">python3 vol.py -f ~\/volcados\/alina1G.elf windows.info\nVolatility 3 Framework 1.0.1\nProgress:  100.00        PDB scanning finished\nVariable    Value\n\nKernel Base    0x82805000\nDTB    0x185000\nSymbols    file:\/\/\/Users\/ricardo\/volatility3\/volatility3\/symbols\/windows\/ntkrnlmp.pdb\/00625D7D36754CBEBA4533BA9A0F3FE2-2.json.xz\nIs64Bit    False\nIsPAE    False\nprimary    0 WindowsIntel\nmemory_layer    1 Elf64Layer\nbase_layer    2 FileLayer\nKdDebuggerDataBlock    0x82926c28\nNTBuildLab    7601.17514.x86fre.win7sp1_rtm.10\nCSDVersion    1\nKdVersionBlock    0x82926c00\nMajor\/Minor    15.7601\nMachineType    332\nKeNumberProcessors    1\nSystemTime    2019-09-21 12:07:14\nNtSystemRoot    C:\\Windows\nNtProductType    NtProductWinNt\nNtMajorVersion    6\nNtMinorVersion    1\nPE MajorOperatingSystemVersion    6\nPE MinorOperatingSystemVersion    1\nPE Machine    332\nPE TimeDateStamp    Sat Nov 20 08:42:46 2010<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Writing your Own Plugin<\/h2>\n\n\n\n<p>We are going to develop a simple plugin called <code>MyFirstPlugin<\/code> to show the plugin writing process in Volatility 3. The plugin will work similar to <code>PsList<\/code>, simply showing for each process the PID, the process name, and a boolean value to know if the process is Wow64. In particular, we will use the <code>PsList<\/code> plugin to retrieve the list of processes acquired in the memory dump.<\/p>\n\n\n\n<p>In Volatility 3, our plugin class has to inherit from <code>PluginInterface<\/code>. Volatility automatically finds all plugins in the <code>plugins<\/code> folder and imports every plugin that inherits from <code>PluginInterface<\/code>. Move to the folder <code>volatility3\/plugins\/windows<\/code> inside the Volatility 3 root folder and create a file with name <code>myplugin.py<\/code> with this content:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"python\" class=\"language-python\">from typing import List\n\nfrom volatility3.framework import renderers, interfaces\nfrom volatility3.framework.configuration import requirements\nfrom volatility3.framework.interfaces import plugins\nfrom volatility3.plugins.windows import pslist\n\nclass MyFirstPlugin(plugins.PluginInterface):\n    _required_framework_version = (1, 0, 0)<\/code><\/pre>\n\n\n\n<p>Apart from the import libraries that we will need when completing the source code, we have defined that the minimum required version of Volatility for this plugin to work will be 1.0.0. Next, we need to define the requirements of our plugin. The plugin requirements define the arguments necessary for our plugin to function appropriately. Optional arguments can also be defined. All of this is defined in the <code>get_requirements<\/code> method:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"python\" class=\"language-python\">@classmethod\n    def get_requirements(cls) -&gt; \n                    List[interfaces.configuration.RequirementInterface]:\n        return [\n            requirements.TranslationLayerRequirement(name = 'primary',\n                           description = 'Memory layer for the kernel',\n                           architectures = [\"Intel32\", \"Intel64\"]),\n            requirements.SymbolTableRequirement(name = \"nt_symbols\",\n                            description = \"Windows kernel symbols\"),\n            requirements.BooleanRequirement(name = 'onlywow64',\n                            description = \"Only show WoW64 processes\",\n                            default = False,\n                            optional = True)\n        ]<\/code><\/pre>\n\n\n\n<p>Note that this is a class method. This method will be invoked before the instantation of an object of the class. In this case, our plugin will only work with Intel architecture dumps (both 32-bit and 64-bit) and will need the Windows kernel symbols. As an optional argument, <code>onlywow64<\/code> has been defined (default is <code>False<\/code>).<\/p>\n\n\n\n<p>In Volatility 3 you have to define a <code>run<\/code> method, which will be called by Volatility after loading the memory dump. This method returns an object of type <code>TreeGrid<\/code>, which, as in Volatility 2, serves to facilitate obtaining an output format determined by the user (unified output, we will explain this later). We are going to add the following code to our plugin:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"python\" class=\"language-python line-numbers\">    def run(self):\n        tasks = pslist.PsList.list_processes(self.context,\n                                                self.config['primary'],\n                                                self.config['nt_symbols'])\n        wow64 = self.config['onlywow64']\n        if wow64:\n            tasks = self.onlyWow64(tasks)\n\n        return renderers.TreeGrid([(\"PID\", int), (\"Image\", str),\n                                   (\"WoW64\", int)], self._generator(tasks))<\/code><\/pre>\n\n\n\n<p>This method first retrieves the list of processes (through the plugin <code>PsList<\/code>). Afterwards, it is checked if the <code>onlywow64<\/code> argument is defined to, in this case, filter the processes and keep only those that are WoW64 processes. Finally, we are creating a <code>TreeGrid<\/code> object, which follows the same definition as in Volatility2: it needs the tuple of values (strings and type) of the returned data and as a second parameter, the data generator.<\/p>\n\n\n\n<p>The two remaining methods to be defined, <code>onlyWow64<\/code> and <code>_generator<\/code>, would be as shown below:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"python\" class=\"language-python line-numbers\">    def _generator(self, data):\n        for task in data:\n            yield (0, [\n                    int(task.UniqueProcessId),\n                    task.ImageFileName.cast(\"string\", \n                            max_length = task.ImageFileName.vol.count, \n                            errors = 'replace'),\n                    int(task.get_is_wow64())\n                ])\n\n    def onlyWow64(self, tasks):\n        for task in tasks:\n            if task.get_is_wow64():\n                yield task <\/code><\/pre>\n\n\n\n<p>With all this, we can proceed to the execution and see that it runs smoothly:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">python3 vol.py -f ~\/volcados\/alina1G.elf windows.myfirstplugin.MyFirstPlugin\nVolatility 3 Framework 1.0.1\nProgress:  100.00        PDB scanning finished\nPID    Image   WoW64\n\n4    System  0\n268    smss.exe    0\n348    csrss.exe   0\n384    wininit.exe 0\n[... omitted ...]<\/code><\/pre>\n\n\n\n<p>Note how we have refer to our plugin: we need to indicate the name of the plugin class, preceded by the directory hierarchy where the plugin is located. If we run the plugin with the argument <code>onlywow64<\/code>, we see that no process appears (since the memory dump comes from a 32-bit Intel architecture):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">python3 vol.py -f ~\/volcados\/alina1G.elf windows.myfirstplugin.MyFirstPlugin --onlywow64\nVolatility 3 Framework 1.0.1\nProgress:  100.00        PDB scanning finished\nPID    Image   WoW64<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Unified Output<\/h2>\n\n\n\n<p>Since Volatility version 2.5, <strong>unified output<\/strong> was introduced, which allows a user to use a plugin without worrying about the output format: the user may want the output in CSV, JSON, or even SQLite, and get it just by specifying how she want it. Optional in Volatility 2, unified output is now mandatory in Volatility 3 (in fact, it is the only possible output definition).<\/p>\n\n\n\n<p>Unified output is based on a <code>TreeGrid<\/code> object, whose constructor requires two parameters: the first parameter is a tuple list, which defines the string and the type of returned data, while the second parameter is the generator of data.<\/p>\n\n\n\n<p>In our plugin, we can request the output in JSON format very easily:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">python3 vol.py -f ~\/volcados\/alina1G.elf -r json windows.myfirstplugin.MyFirstPlugin\nVolatility 3 Framework 1.0.1\nProgress:  100.00        PDB scanning finished\n[\n  {\n    \"Image\": \"System\",\n    \"PID\": 4,\n    \"WoW64\": 0,\n    \"__children\": []\n  },\n  {\n    \"Image\": \"smss.exe\",\n    \"PID\": 268,\n    \"WoW64\": 0,\n    \"__children\": []\n  },\n  {\n    \"Image\": \"csrss.exe\",\n    \"PID\": 348,\n    \"WoW64\": 0,\n    \"__children\": []\n  },\n[... omitted ...]\n}<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p><em>And that&#8217;s all, folks! In this post we explained how to write your own analysis plugins in Volatility version 3. In addition, we also detailed how to manually install symbol files, which are now required for analyzing memory dumps in Volatility 3. If you are interested in malware analysis and memory forensics, <a href=\"http:\/\/webdiis.unizar.es\/~ricardo\/sbc-2021\/\" data-type=\"URL\" data-id=\"http:\/\/webdiis.unizar.es\/~ricardo\/sbc-2021\/\">you can read more about this topic in this training<\/a> that I&#8217;ve prepared for <a href=\"https:\/\/www.incibe.es\/summer-bootcamp\" data-type=\"URL\" data-id=\"https:\/\/www.incibe.es\/summer-bootcamp\">Summer Boot Camp 2021<\/a>, an international workshop organized by the <a href=\"https:\/\/www.incibe.es\/en\" data-type=\"URL\" data-id=\"https:\/\/www.incibe.es\/en\">Spanish National Cybersecurity Institute<\/a> (only in Spanish, sorry!). At the moment, there are few <a href=\"https:\/\/github.com\/volatilityfoundation\/community3\">community plugins available for Volatility 3<\/a>, but this list is sure to grow in the future. I hope this post helps you get started writing your own analysis plugins, helping to increase the analysis tools of the forensics community!<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 6<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span>TL;DR We explain how to write a Volatility 3 plugin. In addition, we also explain how to manually install symbol files. If you are interested in this excellent memory forensic framework and want to write your own analysis tools, read on! Introduction Volatility 3 is the newest (and largely anticipated) version of the most popular [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16,18,15],"tags":[13,36,24,14],"class_list":["post-386","post","type-post","status-publish","format-standard","hentry","category-forensics","category-ms-windows-internals","category-tools","tag-memory-forensics","tag-plugin-development","tag-volatility","tag-windows","no-featured-image"],"_links":{"self":[{"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/posts\/386","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/comments?post=386"}],"version-history":[{"count":9,"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/posts\/386\/revisions"}],"predecessor-version":[{"id":398,"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/posts\/386\/revisions\/398"}],"wp:attachment":[{"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/media?parent=386"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/categories?post=386"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/reversea.me\/index.php\/wp-json\/wp\/v2\/tags?post=386"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}