Most of the time I work with SMB sized customers. Most of them don’t want or need to know the ends and outs of their vSwitch configuration (or much of the underlying configuration at all) because they don’t intend on needing to change it. If they want to modify something they will send in a support request and one of us at Eagle will reply to them.
Because most SMB shops only have maybe 3 ESXi hosts doing all of this via an SSH session into the ESXi shell tends to be quicker and provide better overall documentation for clients. It also helps with support if we ever have to reload a host. We don’t have to pull down any special Virtual Appliances or walk people though configuring a usb drive to do something special. It’s very clear and easy for folks to understand.
Keeping it simple also helps get rid of errors and makes things much more consistent. This script takes advantage of variables which are kept in the head of the script so that things are made simpler and so you don’t have to go hunting for all the places to change things.
In this post I look at the basic configuration file I use to prepare clients ESXi hosts for use. This script has been optimized for vSphere 5.0. Much of it is not backwards compatible although I left the commands in for 4.1 hosts in most cases. It’s also worth noting that I don’t have the actual script file up for download. You’ll need to put the parts in a file yourself.
Before we get started I should make it clear if you use anything discussed in this blog it is at your your own risk.
It should first be noted that any thing preceded by a # symbol is a comment.
First we start off with the standard stuff that you would find in the header of the file…
###################################################################
#
# 10-23-11 Script created by Tony Foster
# 10-28-11 Fixed some typos, moved jumbo frames for mgmt to end so
# V.1.1 there is no disconect of the SSH session. Also added
# Syslog setup.
# 12-11-11 Changed the iSCSI detection settings. In old script
# V.1.2 it would detect hardware iSCSI HBA's as well as
# VMware's software HBA.
# 12-14-11 Added a network refresh before changing network bindings
# V.1.3 in Section 09.02. Otherwise it may fail to config the
# management network accordingly.
# 05-08-12 Changed the default syslog to SSL port to 1514. Built a
# V.1.4 vMotion vSwitch Config. Started work on createing a
# vLAN build option ran into snag with arrays.
# Also changed out the esxcfg to esxcli commands.
# 06-21-12 V.1.4 did not work correctly re-writing to make it work.
# V.1.5 Added vlan support and updated to new command set.
#
#
#
# Install Script for vSphere 5.0
# Eagle Software, Inc V.1.5
#
# (C) Copyright 2012 EAGLE Software, Inc. in Association with
# Anthony (Tony) Foster. All rights reserved.
#
###################################################################
This section contains all the variables. Comments are out to the side that describe what they do. I strongly recommend you just change the values of the variables to match what you want them to do.
#################Document what you are building#################### ## For host: <Put the host name here> ## File build date: <Put the build date here> ## Pound signs (#) represent comments. ## ## Enter information in the variables below to configure your host. ## Settings: # Management Settings MyMgtVswitch="vSwitch0" #Management vSwitch name (Optional) MgtNicOne='vmnic0' #Management Nic one's identifier MgtNicTwo='vmnic4' #Management Nic two's identifier MgtVMk='vmk0' #Management Kernel identifier MyConsoleName='Service Console' #Name for the connection to the ESXi host (Optional) MyMgtNetName='Management Network' #Name for the management network (Optional) MyMgtVLAN='0' #vLAN for the management network leave at 0 for default (optional) MyMgtMTU='9000' #MTU size for management /*# vMotion config MyvMotVswitch="vSwitch0" #Management vSwitch name (Optional) MyvMotName='vMotion' #Name for base vMotion Kernel vMotNicOne='vmnic0' #Managment Nic one's identifier vMotNicTwo='vmnic4' #Managment Nic two's identifier MyvMotOneIP='x.x.x.x' #IP for vMotion1 MyvMotTwoIP='x.x.x.x' #IP for vMotion2 MyvMotSubNet='255.255.255.0' #Subnet mask for vMotion MyvMotVMkOne='vmk3' #First VM Kernel for vMotion (Optional) MyvMotVMkTwo='vmk4' #Second VM Kernel for vMotion (Optional) MyVMotVLAN='0' #vLAN for the vMotion network leave at 0 for default (optional) MyvMotPortCount='16' #vMotion vSwitch Port count MyvMotMTU='9000' #MTU size for vMotion /*# Production Settings MyProdVswitch="vSwitch1" #Production vSwitch name (Optional) ProdNicOne='vmnic1' #Production Nic one's identifier ProdNicTwo='vmnic5' #Production Nic two's identifier MyProdNetName='Production Network' #Production network name (Optional) MyProdVlanNum="0" #Production vLAN id 0 for none (Optional) MyVlanBaseName="vLAN" #Base name for any additional vLANs created IE vLAN 10 MyVlanArray="0" #String array of additional vLANs to add each seperated by a space set to 0 for none MyProdMTU='9000' #MTU size for Production # iSCSI Settings MyiSCSIVswitch="vSwitch2" #iSCSI vSwitch name (Optional) MyiSCSIKernName='iSCSI' #iSCSI base kernel Name iSCSINicOne='vmnic2' #iSCSI Nic one's identifier iSCSINicTwo='vmnic6' #iSCSI Nic two's identifier MyiSCSINetName='iSCSI Network' #iSCSI network name (Optional) MyiSCSIOneIP='y.y.y.y' #IP for iSCSI1 MyiSCSITwoIP='y.y.y.y' #IP for iSCSI2 MyiSCSISubNet='255.255.255.0' #Subnet mask for vMotion MyiSCSIVMkOne='vmk1' #First VM Kernel for iSCSI (Optional) MyiSCSIVMkTwo='vmk2' #Second VM Kernel for iSCSI (Optional) MyiSCSIVLAN='0' #vLAN for the iSCSI network leave at 0 for default (optional) MyiSCSIMTU='9000' #MTU size for iSCSI # Host Settings EditMyHostFile='0' #Set the value to 1 to edit the host file MyDomainName=".domain.com" #Domain name of the hosts MyHostBaseName="ESXi0" #The base form of the host name MyHostIPBase="x.x.x." #The first 3 octets of the hosts IP address MyHostBaseIP="51" #The LAST octet of the ip, this will be the first host MyTotalNumOfHosts='3' #The number of hosts you want to put into the file MyStartingHostNum='1' #The first host to start with # vCenter Settings MyvCenterName="vCenter" #The short name of the vCenter MyvCenterIP="x.x.x.x" #The IP address of the vCenter # General Settings (designed for EqualLogic Arrays) MyiSCSIArrayOneIP='x.x.x.x' #IP address of iSCSI array MyNTPServerOne='time.nist.gov' #Time Server one MyNTPServerTwo='' #Time Server Two (Optional) MySyslogServer="ssl://x.x.x.x:1514" #Syslog server name or IP : port SSL 1514 MySyslogLocPath='/scratch/log' #Local path to save logs to (Optional) MyDumpServerIP='x.x.x.x' #Dump Server ip address MyDumpServerVMK='vmk0' #Dump Server vmKernel number
This very first section is just a safety check to make sure you’re not running vMotion across the production network. If you need this explained as to why this is a bad idea you may want to consult some other documentation before continuing with this blog.
# Section Error Checking #Create vMotion vSwitch if needed if [ $MyvMotVswitch == $MyProdVswitch ]; #Warn if vMotion and production are on the same network then clear screen echo ""; echo ""; echo "" echo "STOP!!! This is not a supported configuration DO NOT CONTINUE" echo "Consult VMware for information" echo ""; echo ""; echo "" else clear screen echo ""; echo ""; echo "" echo "NO ERRORS FOUND in the configuration" echo ""; echo ""; echo "" fi
Now we start on the meat of the script. This first part changes the naming of the management kernel and the management network. The naming scheme has always driven me nuts. Don’t call it a network if there is only one thing is using it.
# Section 00 # Clean up port settings vim-cmd hostsvc/net/portgroup_set --portgroup-name="$MyConsoleName" $MyMgtVswitch "Management Network" vim-cmd hostsvc/net/portgroup_set --portgroup-name="$MyMgtNetName" $MyMgtVswitch "VM Network" #Reverse the decision #vim-cmd hostsvc/net/portgroup_set --portgroup-name="VM Network" $MyMgtVswitch "Management Network" #vim-cmd hostsvc/net/portgroup_set --portgroup-name="Management Network" $MyMgtVswitch "Service Console"
The next part starts by creating a vSwitch for all of the other network segments we need to create. You will also note that we are sizing the switches accordingly. To many ports and we are wasting space to few and we can cause problems for our design.
# Section 01.01
# create a vSwitch
# the -p represents how many ports the vSwitch has
# 8 ports are held back for internal use, so subtract 8 to get usable ports
# valid values are 16, 32, 64, 128, 248, 504, 1016, 2040, 4088
esxcli network vswitch standard add -v $MyProdVswitch -P 128
#create a Production vSwitch
esxcli network vswitch standard add -v $MyiSCSIVswitch -P 32 #create an iSCSI vSwitch
echo "*******************************************************"
#Modify vSwitch0
if [ $MyMgtVswitch != $MyProdVswitch ]; #dont shrink the ports if Prod and Management vSwitchs are together
then
vim-cmd hostsvc/net/vswitch_setnumports $MyMgtVswitch 32
echo "Updated management network port count"
fi
echo "*******************************************************"
#Create vMotion vSwitch if needed
if [ $MyMgtVswitch != $MyvMotVswitch ]; #dont create the vSwitch if Management and vMot vSwitchs are the same
then
esxcli network vswitch standard add -v $MyvMotVswitch -P $MyvMotPortCount
echo "Created vMotion vSwitch"
fi
In the following section we get all of the parts setup for our vSwitchs. This includes things like jumbo frames, nics, CDP, and the like.
# Section 02.01 # set vSwitchs to use jumbo frames #esxcfg-vswitch -m 9000 $MyMgtVswitch # esxcli network vswitch standard set -m $MyProdMTU -v $MyProdVswitch esxcli network vswitch standard set -m $MyiSCSIMTU -v $MyiSCSIVswitch #Enable Jumbo Frames on vMotion vSwitch if needed echo "*******************************************************" if [ $MyMgtVswitch != $MyvMotVswitch ]; #dont enable unless the vSwitch for Management and vMot vSwitchs are diff then esxcli network vswitch standard set -m $MyvMotMTU -v $MyvMotVswitch echo "Set frame size for vMotion" fi # Section 02.02 # turn on CDP #esxcfg-vswitch -B both $MyvMotVswitch esxcli network vswitch standard set -c both -v $MyMgtVswitch esxcli network vswitch standard set -c both -v $MyProdVswitch esxcli network vswitch standard set -c both -v $MyiSCSIVswitch esxcli network vswitch standard set -c both -v $MyvMotVswitch # Section 02.03 # add a nic to a vSwitch # esxcfg-vswitch -L $vMotNicTwo $MyvMotVswitch esxcli network vswitch standard uplink add -v $MyProdVswitch -u $ProdNicOne esxcli network vswitch standard uplink add -v $MyProdVswitch -u $ProdNicTwo esxcli network vswitch standard uplink add -v $MyiSCSIVswitch -u $iSCSINicOne esxcli network vswitch standard uplink add -v $MyiSCSIVswitch -u $iSCSINicTwo #Add nics if needed to vMotion vSwitch echo "*******************************************************" if [ $MyMgtVswitch != $MyvMotVswitch ]; #dont add nics unless Management and vMot vSwitchs are differant then esxcli network vswitch standard uplink add -v $MyvMotVswitch -u $vMotNicOne esxcli network vswitch standard uplink add -v $MyvMotVswitch -u $vMotNicTwo echo "Added nics to vMotion vSwitch" fi # Section 02.04 # add nic's to default vSwitch esxcli network vswitch standard policy failover set -a "$ProdNicOne,$ProdNicTwo" -v $MyProdVswitch esxcli network vswitch standard policy failover set -a "$iSCSINicOne,$iSCSINicTwo" -v $MyiSCSIVswitch esxcli network vswitch standard policy failover set -a "$MgtNicOne,$MgtNicTwo" -v $MyMgtVswitch echo "*******************************************************" if [ $MyMgtVswitch != $MyvMotVswitch ]; #dont add nics unless Management and vMot vSwitchs are differant then esxcli network vswitch standard policy failover set -a "$vMotNicOne,$vMotNicTwo" -v $MyvMotVswitch echo "Added nics to vMotion vSwitch" fi
Section 3 lets you turn on flow control if your hosts don’t already do so.
# Section 03.01
# Turn on flow control if neccesary (optional)
# on by default in most cases
#ethtool --pause vmnic0 tx on rx on
#ethtool --pause vmnic1 tx on rx on
#ethtool --pause vmnic2 tx on rx on
#ethtool --pause vmnic3 tx on rx on
#ethtool --pause vmnic4 tx on rx on
#ethtool --pause vmnic5 tx on rx on
#ethtool --pause vmnic6 tx on rx on
#ethtool --pause vmnic7 tx on rx on
The following section starts building the useable components of our network specifically Port Groups.
# Section 04.01
# build port groups
#esxcfg-vswitch -A "iSCSI2" $MyiSCSIVswitch
esxcli network vswitch standard portgroup add -p "$MyProdNetName" -v $MyProdVswitch
esxcli network vswitch standard portgroup add -p "$MyiSCSINetName" -v $MyiSCSIVswitch
esxcli network vswitch standard portgroup add -p "$MyiSCSINetName 1" -v $MyiSCSIVswitch
esxcli network vswitch standard portgroup add -p "$MyiSCSINetName 2" -v $MyiSCSIVswitch
esxcli network vswitch standard portgroup add -p "$MyiSCSIKernName""1" -v $MyiSCSIVswitch
esxcli network vswitch standard portgroup add -p "$MyiSCSIKernName""2" -v $MyiSCSIVswitch
esxcli network vswitch standard portgroup add -p "$MyvMotName""1" -v $MyvMotVswitch
esxcli network vswitch standard portgroup add -p "$MyvMotName""2" -v $MyvMotVswitch
In the next section we start building our VMkernels
# Section 05.02 # Set portgroups for a specific vLAN #esxcfg-vswitch -v $MyProdVlanNum -p "$MyProdNetName" $MyProdVswitch esxcli network vswitch standard portgroup set -p "$MyMgtNetName" -v $MyMgtVLAN esxcli network vswitch standard portgroup set -p "$MyProdNetName" -v $MyProdVlanNum esxcli network vswitch standard portgroup set -p "$MyiSCSINetName" -v $MyiSCSIVLAN esxcli network vswitch standard portgroup set -p "$MyiSCSINetName 1" -v $MyiSCSIVLAN esxcli network vswitch standard portgroup set -p "$MyiSCSINetName 2" -v $MyiSCSIVLAN esxcli network vswitch standard portgroup set -p "$MyiSCSIKernName""1" -v $MyiSCSIVLAN esxcli network vswitch standard portgroup set -p "$MyiSCSIKernName""2" -v $MyiSCSIVLAN esxcli network vswitch standard portgroup set -p "$MyvMotName""1" -v $MyVMotVLAN esxcli network vswitch standard portgroup set -p "$MyvMotName""2" -v $MyVMotVLAN sleep 5 # Section 05.03 # Build Production vLANs #MyProdVswitch="vSwitch1" #Production vSwitch name (Optional) #MyVlanBaseName="vLAN" #Base name for any additional vLANs created IE vLAN 10 #MyVlanArray="12 14 92 105 75" #String array of additional vLANs to add each seperated by a space set to 0 for none if [ "$MyVlanArray" != "0" ]; then for MyVlanProc in $MyVlanArray; do esxcli network vswitch standard portgroup add -p "$MyVlanBaseName ${MyVlanProc}" -v $MyProdVswitch esxcli network vswitch standard portgroup set -p "$MyVlanBaseName ${MyVlanProc}" -v ${MyVlanProc} done fi sleep 5
Now we’ll configure our VMkernels
# Section 06.01 # Convert port group to vmKernel with Jumbo Frames #MyMgtMTU='9000' #MTU size for management #MyvMotMTU='9000' #MTU size for vMotion #MyProdMTU='9000' #MTU size for Production #MyiSCSIMTU='9000' #MTU size for iSCSI #esxcfg-vmknic -a -i $MyiSCSIOneIP -n $MyiSCSISubNet -m 9000 "iSCSI1" esxcli network ip interface add -m $MyiSCSIMTU -p "$MyiSCSIKernName""1" esxcli network ip interface add -m $MyiSCSIMTU -p "$MyiSCSIKernName""2" esxcli network ip interface add -m $MyvMotMTU -p "$MyvMotName""1" esxcli network ip interface add -m $MyvMotMTU -p "$MyvMotName""2" # Section 06.02 # set vmKernel IP #esxcfg-vmknic -a -i $MyiSCSIOneIP -n $MyiSCSISubNet -m 9000 "iSCSI1" esxcli network ip interface ipv4 set -I $MyiSCSIOneIP -N $MyiSCSISubNet -i $MyiSCSIVMkOne -t static esxcli network ip interface ipv4 set -I $MyiSCSITwoIP -N $MyiSCSISubNet -i $MyiSCSIVMkTwo -t static esxcli network ip interface ipv4 set -I $MyvMotOneIP -N $MyvMotSubNet -i $MyvMotVMkOne -t static esxcli network ip interface ipv4 set -I $MyvMotTwoIP -N $MyvMotSubNet -i $MyvMotVMkTwo -t static
Now we’ll start configuring our iSCSI network.
# Section 07.01 # create iSCSI adapter esxcli iscsi software set --enabled=true # Section 07.02 # Verify that its on. # Turn on iSCSI http://kb.vmware.com/kb/1029301 #esxcfg-swiscsi -e #esxcfg-swiscsi -q esxcli iscsi software get sleep 10 # Section 07.03 # Refresh the network vim-cmd hostsvc/net/refresh # Section 07.04 # list vmhba# echo `esxcli iscsi adapter list | grep vmhba | grep 'iSCSI Software Adapter' |awk '{print $1}'` # Section 07.05 # save it to a variable # Choose one of the two options, comment out the other MyHBA=$( esxcli iscsi adapter list | grep vmhba | grep 'iSCSI Software Adapter' |awk '{print $1}') # Section 08.01 # Add iSCSI array to iSCSI discovery vmkiscsi-tool -D -a $MyiSCSIArrayOneIP $MyHBA sleep 5 # Section 09.01 # Put nics in correct binding orders #iSCSI esxcli network vswitch standard portgroup policy failover set -u -p "$MyiSCSINetName" esxcli network vswitch standard portgroup policy failover set -a "$iSCSINicOne" -s "" -p "$MyiSCSIKernName""1" esxcli network vswitch standard portgroup policy failover set -a "$iSCSINicTwo" -s "" -p "$MyiSCSIKernName""2" esxcli network vswitch standard portgroup policy failover set -a "$iSCSINicTwo" -s "$iSCSINicOne" -p "$MyiSCSINetName 1" esxcli network vswitch standard portgroup policy failover set -a "$iSCSINicOne" -s "$iSCSINicTwo" -p "$MyiSCSINetName 2" #Management esxcli network vswitch standard portgroup policy failover set -u -p "$MyMgtNetName" esxcli network vswitch standard portgroup policy failover set -a "$MgtNicOne" -s "$MgtNicTwo" -p "$MyConsoleName" #vMotion esxcli network vswitch standard portgroup policy failover set -a "$vMotNicOne" -s "$vMotNicTwo" -p "$MyvMotName""1" esxcli network vswitch standard portgroup policy failover set -a "$vMotNicTwo" -s "$vMotNicOne" -p "$MyvMotName""2" sleep 5 # Section 09.02 # Refresh the network vim-cmd hostsvc/net/refresh # Section 11.01 # bind VMkernels to VMhba’s esxcli iscsi networkportal add -A $MyHBA -n $MyiSCSIVMkOne esxcli iscsi networkportal add -A $MyHBA -n $MyiSCSIVMkTwo sleep 5 # Section 12.01 # Rescan for new storage esxcli iscsi adapter discovery rediscover -A $MyHBA
I’ve left out the configuration rules for setting up an EqualLogic array. Its another post for another time. When I write it I’ll link to it here. Here we set our vMkernel for VMotion.
# Section 14.01 # Set the VMotion kernel for vMotion http://kb.vmware.com/1006989 vim-cmd hostsvc/vmotion/vnic_set $MyvMotVMkOne vim-cmd hostsvc/vmotion/vnic_set $MyvMotVMkTwo
Now we’ll set our NTP settings. For these changes to take affect you will want to restart the ESXi host so it will re-read its time servers.
# Section 15.01 # Set the time servers for NTP # for Windows TS see http://kb.vmware.com/kb/1318 for details # found the orginal idea for this at http://zenhat.org/2010/09/11/how-to-sample-kickstart-file-for-vmware-esxi-4-1/ cat >> /etc/ntp.conf << EOF server $MyNTPServerOne server $MyNTPServerTwo EOF chkconfig ntpd on
The next part configures syslogging and dump logging.
# Section 16.01 # Configure the Syslog service for vCenter # List config esxcli system syslog config get # Set the Syslog esxcli system syslog config set --logdir="$MySyslogLocPath" --loghost="$MySyslogServer" --logdir-unique=true --default-rotate=8 --default-size=1024 # Reload syslog esxcli system syslog reload # List config esxcli system syslog config get # Section 16.02 # Enable Firewall entries # Open the firewall for syslog vim-cmd hostsvc/firewall_enable_ruleset syslog # Section 17.01 # Configure remote dump host # List current dump configuration esxcli system coredump network get # Configure dump server connection esxcli system coredump network set --interface-name $MyDumpServerVMK --server-ipv4 $MyDumpServerIP --server-port 6500 # Enable network core dump esxcli system coredump network set --enable true # List current dump configuration esxcli system coredump network get
In some cases its beneficial to register the ESXi host’s in the host file. This can help in some instances where DNS is not available to respond to name resolution requests. We put entries in the ESXi hosts in the following section.
# Section 18.02
# Register the ESXi Hosts in the /etc/hosts file
if [ $EditMyHostFile -eq 1 ]; then
while [ $MyStartingHostNum -le $MyTotalNumOfHosts ]; do
#echo "Counting up: $MyStartingHostNum"
echo -e "$MyHostIPBase""$MyHostBaseIP $MyHostBaseName""$MyStartingHostNum $MyHostBaseName""$MyStartingHostNum""$MyDomainName" >> /etc/hosts
MyHostBaseIP=`expr $MyHostBaseIP + 1`
MyStartingHostNum=`expr $MyStartingHostNum + 1`
done
echo -e "$MyvCenterIP $MyvCenterName $MyvCenterName""$MyDomainName" >> /etc/hosts
fi
Now we just have a couple of more things left to do. Firs we are going to put the host in maintinance mode so its ready for a reboot of the host.
# Section 19.01
# put the host in maintenance mode for a reboot
vim-cmd hostsvc/maintenance_mode_enter
Once that’s done we’ll take care of the management configuration. We don’t want to change this till we get to the end. In some instances I’ve seen networks get cranky when we flip over to jumbo frames and make other changes to the management network. That’s why this is done at the end of the script.
# Section 20.01
# set the management vSwitch to use jumbo frames
#MyMgtMTU='9000' #MTU size for management
#MgtVMk='vmk0' #Management Kernel identifier
esxcli network vswitch standard set -m $MyMgtMTU -v "$MyMgtVswitch"
esxcli network ip interface set -m $MyMgtMTU -i $MgtVMk
Now on to the very last thing we will do to our ESXi host. I expect that you didn’t type all of this stuff in while sitting at the console of your ESXi host… I bet you SSH’ed into it and either used winscp or putty with some copy and paste magic and dropped all of this code on to your system. Let seal up that loop hole so SSH is disabled. It should be noted that this is not permanent and you will probably want to go in and set a permanent rule for the SSH service through the gui.
# Section 21.01
# Turn off SSH
vim-cmd hostsvc/disable_ssh
vim-cmd hostsvc/stop_ssh
vim-cmd hostsvc/net/refresh
sleep 5
And that is a basic script minus some small parts that I use to build ESXi hosts in a hurry. I’m sure there are ways to improve it so that its more operationally efficient and I’m sure it could be ported over to a KS file or power-shell or any other form you want to use. You could probably get really inventive about how to deploy this quickly.
Remember this is for ESXi 5.0 hosts. If you run this on hosts prior to 5.0 you may get unpredictable results. Many of the section have the 4.x commands in the header if you need them however it should be noted that they are not complete and will not result in a full configuration if run.
If you have questions or comments please let me know and I will try my best to answer them for you.
