Hacking .vmx files to change the virtual PCI slot for a NIC

It’s possible to change the virtual PCI port that a virtual NIC uses.

This allows a tester to simulate using host profiles with differing hardware, and in this case, it’s possible to use host profiles even if the PCI bus trees are slightly different

eg, in the .vmx file for VM1

ethernet0.pciSlotNumber = “32”
ethernet1.pciSlotNumber = “34”
ethernet2.pciSlotNumber = “35”
ethernet3.pciSlotNumber = “36”
ethernet4.pciSlotNumber = “37”
ethernet5.pciSlotNumber = “38”

leads to:

~ # lspci | grep -i vmnic
000:002:00.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic0]
000:002:02.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic1]
000:002:03.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic2]
000:002:04.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic3]
000:002:05.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic4]
000:002:06.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic5]

 

But if we change the PCI slot for VM2, we get a slightly different result:

ethernet0.pciSlotNumber = “32”
ethernet1.pciSlotNumber = “34”
ethernet2.pciSlotNumber = “35”
ethernet3.pciSlotNumber = “36”
ethernet4.pciSlotNumber = “37”
ethernet5.pciSlotNumber = “39”

which gives:

~ # lspci | grep -i vmnic
000:002:00.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic0]
000:002:02.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic1]
000:002:03.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic2]
000:002:04.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic3]
000:002:05.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic4]
000:002:07.0 Network controller: Intel Corporation PRO/1000 MT Single Port Adapter [vmnic5]

 

2:07 = (2*16 + 7 == 39)

 

Advertisements

Quick way of balancing fixed paths for multiple LUNs in ESXi

#!/bin/ash

for f in `cd /dev/disks ; ls -1 naa* | grep -v :`
do
LUN=`esxcli nmp fixed getpreferred --device $f`
LUNNUM=`echo $LUN | sed s/.*:L//`
if [ $((LUNNUM % 2)) -eq 0 ]; then
NEWLUN=`echo $LUN | sed 's/vmhba1/vmhba0/'`
else
NEWLUN=`echo $LUN | sed 's/vmhba0/vmhba1/'`
fi
esxcli nmp fixed setpreferred --device $f --path $NEWLUN
printf "$NEWLUN $f\n"
done

Quick way of locating BIOS extensions using Linux

From Wikipedia:

In order to discover memory-mapped option ROMs during the boot process, PC BIOS implementations scan real memory from 0xC0000 to 0xF0000 on 2 KiB boundaries, looking for a ROM signature: 0xAA55 (0x55 followed by 0xAA, since the x86 architecture is little-endian). In a valid expansion ROM, this signature is immediately followed by a single byte indicating the number of 512-byte blocks it occupies in real memory. The next byte contains an offset describing the option ROM’s entry point, to which the BIOS immediately transfers control. At this point, the expansion ROM code takes over, using BIOS services to register interrupt vectors for use by post-boot applications, provide a user configuration interface, or display diagnostic information.

Easy way to find these in Linux:

# curr=$((0xc0000)) ; incr=2048 ; while [ $curr -lt $((0xfffff)) ]; do od -t x1 -A x -j $curr -N 3 /dev/mem | head -1; let curr=curr+$incr; done | grep " 55 aa "

0c0000 55 aa 40
0c8000 55 aa 0f
0ca000 55 aa 08
0dc000 55 aa 20
0e0000 55 aa 20

These appear to be:

0xc0000 - 0xc7fff IBM PC BIOS           (0x40 512B blocks)
0xc8000 - 0xc9eff PCI???                (0x0f 512B blocks)
0xca000 - 0xcafff VMware BIOS           (0x08 512B blocks)
0xdc000 - 0xdffff unknown               (0x20 512B blocks)
0xe0000 - 0xe3fff Phoenix BIOS / VMware (0x20 512B blocks)

ESXi : Working with tar, vmtar, vmtgz and ramdisks

The following script demonstrates how to create a .vgz file which can be loaded into memory.

It also demonstrates the fact that these act as union mounts:

# mkdir /foo

# echo "This file in tar disk" > /foo/bar

# tar cvf /tmp/foo.tar /foo
tar: removing leading '/' from member names
foo/
foo/bar

# rm -rf /foo

# vmtar -c /tmp/foo.tar -o /tmp/foo.vmtar

# rm /tmp/foo.tar

# vmtar -t < /tmp/foo.vmtar
drwxr-xr-x 0/0          0 2010-09-04 13:35 foo/
-rw-r--r-- 0/0         22 2010-09-04 13:35 foo/bar

# gzip < /tmp/foo.vmtar > /tmp/foo.vgz

# rm /tmp/foo.vmtar

# vmkramdisk /tmp/foo.vgz

# echo "Not in tardisk" > /foo/file2

# vdf
Tardisk              1k-blocks      Used
SYS1                    204548    204548
SYS2                     56724     56724
SYS3                     34724     34724
SYS4                       100       100
foo.vgz                      4         4
-----
Ramdisk              1k-blocks      Used Available Use% Mounted on
MAINSYS                  32768      2416     30352   7% --
tmp                     196608       120    196488   0% --
updatestg               768000         8    767992   0% --

# ls -al /foo
drwxr-xr-x    1 root     root                512 Oct  4 13:38 .
drwxr-xr-x    1 root     root                512 Oct  4 13:37 ..
-rw-r--r--    1 root     root                 22 Oct  4 13:35 bar
-rw-r--r--    1 root     root                 15 Oct  4 13:38 file2

# vdu /foo
For '/foo/file2':
                                 heap :         2048
For '/foo/bar':
                       tardisk foo.vgz:          512
For '/foo':
                                 heap :         2048            (    2 inodes)
                       tardisk foo.vgz:          512            (    1 inodes)

# vmkramdisk -u foo.vgz

# ls -al /foo
drwxr-xr-x    1 root     root                512 Oct  4 13:38 .
drwxr-xr-x    1 root     root                512 Oct  4 13:37 ..
-rw-r--r--    1 root     root                 15 Oct  4 13:38 file2

# vdu /foo
For '/foo/file2':
                                 heap :         2048
For '/foo':
                                 heap :         2048            (    2 inodes)

More ESXi kickstarting – firstboot scripts

An example kickstart file could like look this, including a first boot script

vmaccepteula
rootpw letmein
autopart --firstdisk --overwritevmfs
install cdrom
network --bootproto=dhcp --device=vmnic0

%firstboot --unsupported --interpreter=busybox --level=900

vim-cmd hostsvc/enable_local_tsm
vim-cmd hostsvc/start_local_tsm

vim-cmd hostsvc/enable_remote_tsm
vim-cmd hostsvc/start_remote_tsm

%post --unsupported --interpreter=python --ignorefailure=true
import time
stampFile = open('/finished.stamp', mode='w')
stampFile.write( time.asctime() )

And during the kickstart process, it will create the /vmfs/volumes/Hypervisor1/onetime.tgz file

You can look inside the ontime.tgz file, and you can see that it will include the firstboot scripts, which then get later cleaned up, including the /etc/vmware/init/init.d/900.firstboot_001 script which was contained within the kickstart :

# tar tzvf /vmfs/volumes/Hypervisor1/onetime.tgz
drwxr-xr-x 0/0         0 2010-10-04 10:26:56 etc/
drwxr-xr-x 0/0         0 2010-10-04 10:26:56 etc/vmware/
drwxr-xr-x 0/0         0 2010-10-04 10:26:56 etc/vmware/init/
drwxr-xr-x 0/0         0 2010-10-04 10:26:56 etc/vmware/init/init.d/
-r----x--t 0/0       135 2010-10-04 10:26:56 etc/vmware/init/init.d/900.firstboot_001
-r----x--t 0/0        61 2010-10-04 10:26:56 etc/vmware/init/init.d/999.firstboot_remove
-r----x--t 0/0        63 2010-10-04 10:26:56 etc/vmware/init/init.d/999.firstboot_password
-rw-r--r-T 0/0     27052 2010-10-04 10:26:56 etc/vmware/esx.conf
-rw-r--r-T 0/0         0 2010-10-04 10:26:56 etc/resolv.conf
-rw-r--r-T 0/0       183 2010-10-04 10:26:56 etc/hosts
drwxr-xr-x 0/0         0 2010-10-04 10:26:56 var/
drwxr-xr-x 0/0         0 2010-10-04 10:26:56 var/lib/
drwxr-xr-x 0/0         0 2010-10-04 10:26:56 var/lib/firstboot/
-r----x--t 0/0       596 2010-10-04 10:26:56 var/lib/firstboot/999.firstboot_remove.py
-r----x--t 0/0       316 2010-10-04 10:26:56 var/lib/firstboot/999.firstboot_password.py
drwxr-xr-x 0/0         0 2010-10-04 10:26:56 var/log/
-rw-r--r-- 0/0     16164 2010-10-04 10:26:56 var/log/esxi_install.log

Installing ESXi under VMware Fusion – the quick way!

From the previous post, since we already have the IMAGEDD file, we might as well just call that the .VMDK-flat file, and create the associated .VMDK file with the correct geometry.

macos# bzip2 -d < IMAGEDD.BZ2 > SCSI0_0-flat.vmdk
macos# cat > SCSI0_0.vmdk

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
isNativeSnapshot="no"
createType="monolithicFlat"

# Extent description
RW 1843200 FLAT "SCSI0_0-flat.vmdk" 0

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "7"
ddb.longContentID = "21ba0fb6dd5c184536c27ff9fffffffe"
ddb.uuid = "60 00 C2 91 3d 48 5e 62-57 0d da cc 64 5e 9a 23"
ddb.geometry.cylinders = "900"
ddb.geometry.heads = "64"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

Installing ESXi imagedd.bz2

Now that I’ve managed to mount the CDROM, I can manually DD the image onto the destination disk.

I can do this will the following python script, all within the ESXi installer itself. I can scp this file in, and run it [and write any other preformatted .bz2 file onto the hard disk]:

#! /usr/bin/env python

# File: writeesxi.py

import bz2
import os

buffsize = 1024 * 1024 # 1MB blocks

i_filename = "/vmfs/volumes/mpx.vmhba32:C0:T0:L0/IMAGEDD.BZ2"
#i_filename = "IMAGEDD.BZ2"
o_filename = "/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0"
#o_filename = "IMAGEDD"

bz2_file = bz2.BZ2File(i_filename)
#out_file = os.open(o_filename,os.O_RDWR|os.O_TRUNC|os.O_CREAT)
# ESXi doesn't like TRUNC or CREAT. Assume the file exists:
out_file = os.open(o_filename,os.O_RDWR)

while True:
  b = bz2_file.read(buffsize)
  if len(b) == 0 :
    break
  os.write(out_file,b)
  os.write(1,".")

os.close(out_file)
bz2_file.close()
print

Now that I can arbitarily write images onto the hard disk, I should be able to make a cut down 64MB ESXi server.