The VMware ESX kernel is able to write a so called core dump if it crashes. This core dump records the state of the process at the time of the crash. gdb can read such a core dump and get information out of it. First you must unpack the vmware-core.gz file because it’s in a tgz archieve. Now it is time to start gdb to see what exactly happened when I try to start a VM on ESX 3.5 in Workstation 6.
[root@esx35 tmp]# ls
cimserver_start.conf hsperfdata_root vmware-core
cimxml.socket vmhsdaemon-0 vmware-core.gz
esx-install.sh vmkdump.log vmware-root
[root@esx35 tmp]# gdb
GNU gdb Red Hat Linux (6.3.0.0-1.138.el3rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
(gdb) core-file vmware-core
Program terminated with signal 11, Segmentation fault.
#0 0x000b8a6a in ?? ()
(gdb) backtrace
#0 0x000b8a6a in ?? ()
#1 0x000b8a6a in ?? ()
#2 0x00004020 in ?? ()
#3 0x00006484 in ?? ()
#4 0x00006400 in ?? ()
#5 0xb039b400 in ?? ()
#6 0x00000000 in ?? ()
(gdb)
What is a segmentation fault? When ESX 3.5 runs, it has access to certain portions of memory. First, you have local variables in each of the functions; these are stored in the stack. Second, ESX 3.5 may have some memory, allocated during runtime (using either malloc, in C, or new, in C++), stored on the heap (you may also hear it called the "free store"). ESX 3.5 is only allowed to touch memory that belongs to it -- the memory previously mentioned. Any access outside that area will cause a segmentation fault. Segmentation faults are commonly refered to as segfaults.
