Skip to content

Single-shot parser for ESXi logs that produces annotated timelines of interesting events.

Notifications You must be signed in to change notification settings

Psmths/esxi-log-parser

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 

Repository files navigation

ESXi Log Parser

ESXi Log Parser is a forensic tool designed to ingest and analyze logs from VMware ESXi systems. It parses logs located in /scratch/logs, extracts meaningful events, and generates an annotated CSV timeline that can aid in investigations or post-incident analysis.


🚀 Features

  • Parses logs from VMware ESXi systems
  • Outputs a detailed forensic timeline in CSV format
  • Annotates events of interest

📦 Installation

No installation required — simply clone the repository and run the script.


🛠 Usage

Positional Arguments:

  • logdir – Directory containing the extracted ESXi logs (e.g. from /scratch/logs)

Optional Arguments:

  • -o, --output – Filename for the output CSV timeline
  • -v, --verbose – Increase logging verbosity to DEBUG level

ESXi Forensic Notes

Gathering Logs

The fastest way to gather logs from an ESXi server is to run the vm-support command. This will collect a variety of system information in addition to logs and package it as a .tgz archive. The command will print out the exact location of this generated support bundle.

SSH Logon and Logoff Activity

There are two great places to look for evidence of this activity, in hostd.log SSH activity presents as follows:

2025-02-19T18:21:13.088Z Hostd[526002]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 832 : SSH session was opened for '[email protected]'.
2025-02-19T18:21:16.199Z Hostd[526009]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 833 : SSH session was closed for '[email protected]'.

As you can see, both the logon and logoff times are reflected in this log. In this example, the SSH session came from 10.100.40.16. Additionally, in auth.log:

2025-02-19T18:21:09.170Z sshd[1711235]: Connection from 10.100.40.16 port 58232
2025-02-19T18:21:13.082Z sshd[1711235]: Accepted keyboard-interactive/pam for root from 10.100.40.16 port 58232 ssh2

Though it seems that auth.log does not record logoff events for ssh.

File Transfers through SFTP

In the syslog.log file, evidence of SFTP file transfers can be seen (in this case, it was a file download, an SFTP upload will show 0 bytes read and the file size in the bytes written field):

2025-02-19T18:21:13.152Z sftp-server[1711241]: open "/scratch/logs.tar" flags READ mode 0666
2025-02-19T18:21:16.183Z sftp-server[1711241]: close "/scratch/logs.tar" bytes read 80038912 written 0

Though this log doesn't provide an exact correlation to a specific SSH session, context clues can be used to determine which session was responsible for the activity.

Shell and Command Line Activity

There is a log file called shell.log that records commands executed through SSH sessions. An example of such activity:

2025-02-19T18:15:31.369Z shell[1711193]: Interactive shell session started
2025-02-19T18:15:33.891Z shell[1711193]: [root]: cd /scratch/log/
2025-02-19T18:15:34.198Z shell[1711193]: [root]: ls

WebUI Access, Logins, etc.

The WebUI for ESXi is exposed through an internal proxy called envoy, and therefore access to this webpage can be found in envoy-access.log. A good indication that the WebUI has been accessed would be a request for /ui/welcome.txt, which would show up in the log as follows:

2025-01-29T17:01:29.024Z envoy-access[525191]: GET /ui/welcome.txt HTTP/2 404 via_upstream - 0 0 0 0 0 10.100.40.16:36810 10.100.40.251:443 127.0.0.1:43497 127.0.0.1:8309 -

The above log line indicates that the endpoint at 10.100.40.16 accessed the WebUI on the ESXi host (which has the IP 10.100.40.251). This is a request that is repeated periodically, so if a client stays on the login page there will be a number of these events. Authentications to the WebUI show up in the hostd.log file as follows:

2025-01-29T17:01:29.049Z Hostd[525986]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxui-3b60-1260 sid=52979845] Event 722 : User [email protected] logged in as Mozilla/5.0 (X11; Linux x86_64; rv:134.0) Gecko/20100101 Firefox/134.0

Some additional activity can also be identified. For instance, when a user leverages the WebUI to control a virtual machine using the web console (virtual KVM), the following requests get logged to envoy-access.log:

2025-02-19T18:16:55.911Z envoy-access[525191]: GET /screen?id=39&ts=1739989015899 HTTP/2 200 via_upstream - 0 3759 68 68 0 10.100.40.16:40870 10.100.40.251:443 127.0.0.1:45254 127.0.0.1:8309 -

A rather important note, WebUI logon events that have the user agent set to VMware-client, or have a blank username are probably related to the ESXi host being managed by a vCenter Server Appliance (VCSA). These logs are plentiful and possibly worth filtering out.

Local Account Creation and Deletion

These events are tracked in hostd.log:

2025-02-19T18:38:20.201Z Hostd[525974]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxui-ed44-1ea6 sid=52d445a4 user=root] Event 835 : Account testuser was created on host esx02.example.local
2025-02-19T18:45:44.670Z Hostd[526006]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxui-fbe6-203b sid=523f6561 user=root] Event 844 : Account testuser was removed on host esx02.example.local

VIB Acceptance Level Modifications

VIBs (vSphere Installation Bundles) are sometimes used by threat actors to gain persistance on an ESXi host. By default, only trusted VIBs can be installed, but this setting can be changed, leading to the following log lines in hostd.log:

2025-02-19T18:39:28.267Z Hostd[525997]: [Originator@6876 sub=Vimsvc.TaskManager opID=esxui-ff63-1f89 sid=523f6561 user=root] Task Created : haTask-ha-host-vim.host.ImageConfigManager.updateAcceptanceLevel-2271474
2025-02-19T18:39:30.883Z Hostd[526008]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 843 : Host acceptance level changed from partner to community
2025-02-19T18:39:30.971Z Hostd[526009]: [Originator@6876 sub=Vimsvc.TaskManager opID=esxui-ff63-1f89 sid=523f6561 user=root] Task Completed : haTask-ha-host-vim.host.ImageConfigManager.updateAcceptanceLevel-2271474 Status success

Execution Prevention

Related to the aforementioned VIBs and trust levels, some versions of ESXi will prevent execution of binaries not tied to an installed VIB. Such preventions are logged as follows in hostd.log:

2025-02-19T18:49:40.003Z Execution Prevented, Hostd[525974]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 847 : Execution of unknown (non VIB installed) binary './example' prevented. Unknown content can cause malware attacks similar to Ransomware.

Shutdown and Startup

There are several logs that can indicate the ESXi host has been shutdown, rebooted, or started up. The log most likely to contain the most history would be vmksummary.log, which would contain the following for these events:

2024-10-04T19:30:14.949Z No(13) bootstop[527550]: Host is powering off
2024-10-04T19:32:17.323Z No(13) bootstop[526553]: Host has booted

The hostd.log file will also show evidence of shutdown events such as:

2025-02-19T18:29:36.501Z Hostd[525974]: [Originator@6876 sub=Vimsvc.TaskManager opID=esxui-bb62-7382 sid=52b95137 user=root] Task Created : haTask-ha-host-vim.HostSystem.shutdown-146676

DHCP Leases

If the ESXi host is using DHCP to obtain an IP address assignment for its management interface, these logs will be found in the dhclient.log file. The following example shows a DHCP lease assignment:

2024-09-20T00:17:42.003Z In(30) dhclient-uw[524801]: DHCPDISCOVER on vmk1 to 255.255.255.255 port 67 interval 6
2024-09-20T00:17:42.020Z In(30) dhclient-uw[524801]: DHCPOFFER from 10.100.20.53
2024-09-20T00:17:42.020Z In(30) dhclient-uw[524801]: DHCPREQUEST on vmk1 to 255.255.255.255 port 67
2024-09-20T00:17:42.020Z In(30) dhclient-uw[524801]: DHCPACK from 10.100.20.53
2024-09-20T00:17:42.020Z In(30) dhclient-uw[524801]: ipv4: Setting address: 10.100.20.3/255.255.255.0
2024-09-20T00:17:42.035Z In(30) dhclient-uw[524801]: ipv4: Bounding to DHCP server: 10.100.20.53
2024-09-20T00:17:42.068Z In(30) dhclient-uw[524801]: ipv4: Setting default gateway: 10.100.20.254

In this example, the ESXi host in question had its IP set to 10.100.20.3/24.

A Note on pyVmomi and esxcli

It seems that running the esxcli command will cause a hostd logon event originating from localhost:

2025-01-12T20:45:53.845Z shell[1388052]: [root]: esxcli software vib list
2025-01-12T20:45:54.020Z Hostd[525982]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxcli-b5-c88b sid=525e5904] Event 644 : User [email protected] logged in as pyvmomi Python/3.8.16 (VMkernel; 8.0.1; x86_64)

About

Single-shot parser for ESXi logs that produces annotated timelines of interesting events.

Topics

Resources

Stars

Watchers

Forks

Languages