Networking

UPDATE: Connections and Ports in ESX and ESXi

Mr. Dudley Smith has updated his PDF diagram with some minor corrections and additions.  Get the latest, most up-to-date version here (click the graphic) …

connections-ports-esx-v3

He also updated “the brain” which can be found at it’s new home http://webbrain.com/brainpage/brain/89EFA582-2C35-F6A2-9ED1-7AD4810266C2/.  Make sure you update your bookmark accordingly.

Post to Twitter Post to Delicious Post to Digg Post to StumbleUpon

The E.T.D.F. Series – Setting up the Network and Dedicated Remote Access (Part 3)

    OK, we are *almost* done getting our network set up properly for VDI, but we’ve got a few more things to do.  Specifically, we need to address:

  1. Handling our external, Internet facing, dynamic IP address.
  2. DHCP and DNS
  3. External VPN access

Dynamic External IP

Most ISP’s (not all) provide a single, dynamic IP address for consumer grade service (i.e. home use).  But when we’re trying to connect to our virtual desktops from somewhere out on the public Internet, how do we know which IP address to connect to?

Generally speaking, connecting to an IP address is bad practice because it’s inflexible.  Instead, we should connect to a Fully Qualified Domain Name (FQDN).  OK fine, so we’ll set up a DNS entry and use a FQDN to connect to our desktops.  But what happens when our dynamic IP address changes and the DNS entry is still mapped to the old IP address?

What we need is an external Dynamic DNS (aka DDNS) service which will allow us to programmatically update our IP address whenever it changes.  There are a number of both free and paid-for DNS providers out there that can deliver DDNS services.  Personally, I use EditDNS (www.editdns.net).  They have a ton of functionality and they’ve been rock solid for the past few years I’ve been using their services, so I’m quite happy with them.

Now, many home use routers these days have the capability to update a DDNS provider.  But in my experience, the functionality is somewhat limited.  What if, for example, I want aaron.sweemer.com and desktop.sweemer.com to be dynamic entries and www.sweemer.com to be a static entry, pointing to my blog server hosted somewhere else?  In reality, I’ve got about 20 FQDN’s that I need to be dynamically updated and about 100 that I want static.  So instead, I created a script that will:

  1. Query my external IP address (check this out, a free tool from Whatismyip.com)
  2. Compare the result of the query with the IP obtained from the previous query
  3. If the IP is the same, or contains something other than an IP (e.g. HTTP error), the scirpt exits.
  4. If the IP is different, the script updates my DDNS entries via the EditDNS API, then updates a log file documenting the change, and finally adds the new IP to the last line of a file called previous_ips.

If you’d like to use the script I wrote, you’ll first need to do the following:

  1. If don’t already have one, set up an account with EditDNS and make sure you have properly configured the domain name(s) you own.
  2. Verify your linux distro has lynx (a command line, text only, www client)
  3. Verify your linux distro has curl (a tool to transfer data using HTTP)
  4. Create a directory (anywhere you have rwx access is fine) for the script and its files to live
  5. In this directory, create a text file called editdns.sh.  Paste the content (below) into it.
  6. Replace XXXXXXXX with your EditDNS password.
  7. Make editdns.sh executable (chmod +x /path/to/editdns/editdns.sh)
  8. Create another text file called records and, one per line, enter the FQDN’s of the DDNS entries you wanted updated (e.g. mydesktop.mydomain.com)
  9. Add the editdns.sh script to your crontab to run at regular intervals (e.g. mine runs every five minutes and the entry in cron looks like this:    */5 * * * * cd /usr/local/editdns; ./editdns.sh)
    And here’s a copy of the actual script …

#!/bin/bash

EDITDNSPASS=”XXXXXXXX”
LYNX=`which lynx`
TIME=`date`
CIP=`curl -s http://www.whatismyip.com/automation/n09230945.asp | awk –re-interval ‘$1 ~ /^([0-9]{1,3}\.){3}[0-9]{1,3}$/ {print}’`
PIP=`tail -1 ./previous_ips`

if [ "$CIP" != "$PIP" && –n "$CIP" ]; then
cat ./records | while read FQDN; do
$LYNX -source “http://DynDNS.EditDNS.net/api/dynLinux.php?p=$EDITDNSPASS&r=$FQDN”
done
echo “IP Change!  New IP is $CIP.  Editdns.net was updated at $TIME.” >> ./editdns.log
echo $CIP >> ./previous_ips
else
exit 0
fi

If you have any issues or questions, feel free to email me.  Also, keep in mind, this a quick and dirty script that accomplishes what I want it to accomplish.  Feel free to make it more robust (e.g. error handling or better logging) to suit your needs.

DNS and DHCP

It’s quite possible you’ll want to skip this section and opt for setting up DHCP and DNS via Microsoft’s built in DHCP and DNS services that come out of the box with their server products.  To properly set up VMware View, we’ll need to set up Active Directory anyway, and quite frankly, it’s far easier to set up a Microsoft server with DHCP and DNS than it is to set up a Linux server.  So feel free to skip this section and leverage Microsoft for these services.  If, however, you’re a gluten for punishment then by all means, read on.

Let’s first start with DNS.  Here too we need Dynamic DNS because as we’re handing out IP addresses via DHCP, we want our DNS server to properly reflect current information as IP addresses change.  So, if you don’t already have bind9 (the DNS server), go ahead and install it (sudo apt-get install bind9 should work on Ubuntu / Debian distros).

The default configuration for bind9 is to act as a caching server, so the first thing we need to do is configure our DNS to forward all unknown DNS requests to another DNS server.  These should be provided to you from your ISP.  Edit the forwarders {} section of your named.conf.options file (usually located in /etc/bind/) to look like this …

asweemer@cincylab-rtr1:/etc/bind$ more named.conf.options
options {
directory “/var/cache/bind”;

forwarders {
1.2.3.4;
5.6.7.8;
};

auth-nxdomain no;    # conform to RFC1035
listen-on-v6 { any; };
};

Obviously, you’ll need to change 1.2.3.4 and 5.6.7.8 to the IP addresses given to you by your ISP.

Next, we need to modify our master named.conf to allow dynamic updates to DNS.  Add the following entry to the bottom of your named.conf file.

controls {
inet 127.0.0.1 allow {127.0.0.1; 192.168.9.25; 10.10.7.1; 10.10.7.2; } keys {“rndc-key”;};
};

This tells the DNS server to allow updates from the IP address located between the {}.  Notice the first three IP addresses are local IP addresses.  The fourth IP address is a slave DNS server, which I have yet to set up.  The rndc-key is the default key generated during installation of bind9 and it’s used to authorize the updating of DNS records.  If you’re using Ubuntu, then you’ll likely find the key in the file /etc/bind/rndc.key …

asweemer@cincylab-rtr1:/etc/bind$ sudo cat rndc.key
key “rndc-key” {
algorithm hmac-md5;
secret “QZ5jOmcr/OW3nzksR5q0Hw==”;
};
asweemer@cincylab-rtr1:/etc/bind$

Note the file is a text file named rndc.key, and the actual key is called rndc-key located within the text file.

OK, next we need to define our zones in the named.conf.local file.  For each domain you’re using (probably just one), you’ll need two entries:  one for the domain and one for the reverse lookup of the domain.  I have two domains I’ll be updating, so my named.conf.local file looks like this …

asweemer@cincylab-rtr1:/etc/bind$ cat named.conf.local
//
// Do any local configuration here
//

include “/etc/bind/rndc.key”;

zone “mydomain.com” {
type master;
file “/etc/bind/zones/mydomain.com.db”;
allow-update { key “rndc-key”; };
allow-transfer {10.10.7/24; };
};

zone “7.10.10.in-addr.arpa” {
type master;
file “/etc/bind/zones/rev.7.10.10.in-addr.arpa”;
allow-update { key “rndc-key”; };
allow-transfer {10.10.7/24; };
};

zone “dmz.mydomain.com” {
type master;
file “/etc/bind/zones/dmz.mydomain.com.db”;
allow-update { key “rndc-key”; };
allow-transfer {192.168.9/24; };
};

zone “9.168.192.in-addr.arpa” {
type master;
file “/etc/bind/zones/rev.9.168.192.in-addr.arpa”;
allow-update { key “rndc-key”; };
allow-transfer {192.168.9/24; };
};

A couple points to note here:

  • I created a subdirectory called “zones” under /etc/bind/ where I put all my zone files.  This isn’t the default location, and in addition, this isn’t necessary as the zone files can be located anywhere you’d like.  But be aware the configuration file above reflects the location of my files.
  • Notice the include “/etc/bind/rndc.key” on the first line and the all-update directive within each zone definition?  This should be self explanatory at this point.
  • The allow-transfer directive within each zone definition explicitly limits zone transfers (copy) to the IP(s) defined.  This is an important security feature since, by default, DNS allows transfers to anyone, and the info contained within a DNS zone file can really give hackers visibility into your network.

Now we need to create the zone files we just defined above, which will contain our actual DNS records.  Here is the zone file for our dmz.mydomain.com …

asweemer@cincylab-rtr1:/etc/bind/zones$ cat dmz.mydomain.com.db
$TTL 3600       ; 1 hour
dmz.mydomain.com             IN SOA  master.dmz.mydomain.com. root.master.dmz.mydomain.com. (
2009060514 ; serial
86400      ; refresh (1 day)
86400      ; retry (1 day)
2419200    ; expire (4 weeks)
3600       ; minimum (1 hour)
)
NS      master.dmz.mydomain.com.
NS      slave.dmz.mydomain.com.
A       192.168.9.25
MX      10 mail.dmz.mydomain.com.
MX      20 mail-spool.dmz.mydomain.com.
computer-1              A       192.168.9.247
TXT     “317bf41a2c5b70fd9ca4e283d364dcddd5″
computer-2              A       192.168.9.250
TXT     “00cf6242f693ebbf1d545159548e44ab81″
computer-3              A       192.168.9.243
TXT     “31a0cb7e096a96c63dc998d2db3be6e450″
mail                    A       192.168.9.25
mail-spool              A       192.168.9.26
master                  A       192.168.9.25
www                     CNAME   master

A couple important things to point out here:

  • The entries for computer-1, 2 and 3 are dynamic entries there were generated by the DHCP server.  The TXT record that follows these entries is a unique identifier which is also generated by the DHCP server and is used to ensure it won’t overwrite existing DNS records that were generated by another process/server.
  • You’ll obviously need to change the domain names and IP addresses to match your environment.
  • If you haven’t worked with bind9 before, this file probably looks pretty cryptic to you.  If so, I would recommend taking a look at http://www.zytrax.com/books/dns/ch8/soa.html, which gives a pretty good overview of the SOA (defined in the first part of the file).  The balance of the file (i.e. the record definitions) is pretty straight forward.

The reverse zone should look like this …

asweemer@cincylab-rtr1:/etc/bind/zones$ more rev.9.168.192.in-addr.arpa
$ORIGIN 9.168.192.IN-ADDR.ARPA.
$TTL    1h;
IN      SOA     master.dmz.mydomain.com. root.master.dmz.mydomain.com. (
2009060501      ;
1d              ;
1d              ;
4w              ;
1h              ;
)
IN      NS      master.dmz.mydomain.com.
IN      NS      slave.dmz.mydomain.com.
25      IN      PTR     master.dmz.mydomain.com.
26      IN      PTR     slave.dmz.mydomain.com.

I mixed it up just a bit in this file to point out a few different ways to configure a zone file.  In this file, notice the following differences:

  • The $ORIGIN directive sets the domain name to be appended to any unqualified records.  If the $ORIGIN directive doesn’t exist (as it doesn’t in the first config file), then it is implicitly defined by the zone name.
  • The time variables can be defined with d (day), w (week), h (hour), etc.

That’s about it for DNS.  Once you’ve got your bind9 server configured, restart your bind9 server (sudo /etc/init.d/bind9 restart).  And of course, be sure to test your configurations by using the standard DNS tools (e.g. dig, nslookup).  If you get errors, pay careful attention to your local syslog file (probably located at /var/log/syslog) as that’s where DNS and DHCP errors typically write their error messages.

OK, next up is configuring our DHCP server.  And once again, this post is starting to get way to long, so it looks like I’ll need a fourth and (hopefully) final post to this section.

Post to Twitter Post to Delicious Post to Digg Post to StumbleUpon

View 3.1 USB Redirection Improvements

As you probably have seen, View 3.1 GA’d yesterday.  One of the improvements listed in the release notes was:

  • USB Improvements – View 3.1 offers more reliable and broader device support with reduced bandwidth consumption. A separate TCP/IP stream is used.

From what I understand in talking to some people is that a lot of time was spent on the USB redirection stack to further optimize and tune it.

ALSO, USB redirection traffic is now split out onto it’s own traffic stream.  USB redirection traffic will now communicate from the client to the host vm on TCP port 32111.  I imagine this opens up a few new opportunities to do some USB specific traffic prioritization/trottling.  Very interesting!   In previous versions, the USB traffic was inside of the RDP stream (virtual channel).  This prevented us from ever REALLY seeing the USB specific traffic or having any control over it.  Simply put, now we do.  Gotta love progress!

Post to Twitter Post to Delicious Post to Digg Post to StumbleUpon

Scripted ESX Installation: Reconfiguring COS Networking with Kickstart

Frequently customers have specific NICs (like onboard NICs) that they’d like assigned to the COS, leaving the other NICs for VM traffic.  This is difficult, however, when using our automated kickstart deployment scripts as there is no way to explicitly define the vmnic assigned to the COS.  And to make matters worse, the VMkernel is not yet available to us during the %post section of the kickstart script, which makes COS networking configuration difficult! Recently I had a customer who was getting frustrated because …

  1. They would “rack and stack” a physical server and wire up their NICs accordingly (i.e. onboard NICs on the management VLAN, remaining NICs on production VLANs)
  2. PXE boot the server
  3. When kickstart completed, they’d lose connection to the COS.

This happens because during installation, ESX just assigns vmnic0 to the lowest PCI number, and then assigns vmnic0 to the COS. And this is often not the NIC the admin wants used for their COS. Of course, they could go back after the fact and reconfigure the COS networking, but this kind of defeats the purpose of a completely hands-free, automated deployment.

Here is one possible solution to the problem.  Below is a script I wrote to append to the %post section of a kickstart file.  Obviously, you’ll need to make modifications for your environment.

## This script should be appended to the %post section of an ESX kickstart file.
## For more info on kickstart and scripted ESX installations, see Appendix B of
## http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_installation_guide.pdf

##
##
Essentially, this is a “script that creates a script.” Because the VMkernel is
## not yet available to us during the %post section of the scripted install, we use
## %post to generate a script called /tmp/post_esx_install.sh that will launch via
## rc.local upon first boot (and only first boot).
##
## The post_esx_install.sh will first make a backup copy of esx.conf and then
## reconfigure the COS networking.  Please see the in-line comments below for
## tweaking post_esx_install.sh for your environment.
##
## If you have any questions, please email aaron [at] sweemer [dot] com.

%post

cat > /tmp/esx_post_install.sh << EOF
#!/bin/bash
cp /etc/vmware/esx.conf /etc/vmware/esx.conf.backup
/usr/sbin/esxcfg-vswitch -U vmnic0 vSwitch0
/usr/sbin/esxcfg-vswif -d vswif0

## If your kickstart file has vmportgroup=1, you *might* want to uncomment the
## next line

## /usr/sbin/esxcfg-vswitch -D “VM Network”

/usr/sbin/esxcfg-vswitch -A “VMkernel” vSwitch0

## You’ll need to find which physical NICs you want assigned to your COS.  From
## the command line of an already installed ESX server, execute
## “/usr/sbin/esxcfg-nics -l” as root and look for something unique about the
## NICs.  For example, this could be the word “Broadcom” or it could be the
## actual PCI number.  In the next line, replace “search term” with this
## text.

/usr/sbin/esxcfg-nics -l | awk ‘\$0 ~ /search term/ {print \$1}’ | xargs –n 1 /usr/sbin/esxcfg-vswitch vSwitch0 –L

## Note: if you want to test the line above from the command-line, you’ll need
## to remove the leading “\” in front of $0 and $1. The \’s need to be here so
## the esx_post_install.sh script gets properly written by kickstart. But when
## executing directly on a command line, the \’s need to be removed.

## Replace the x.x.x.x after -i with the IP address and after -n with the
## subnet mask for your COS.

/usr/sbin/esxcfg-vswif -a vswif0 -p “Service Console” -i x.x.x.x  -n x.x.x.x

## Replace the x.x.x.x after -i with the IP address and after -n with the subnet
## mask for your VMkernel port group.

/usr/sbin/esxcfg-vmknic -a -i x.x.x.x -n x.x.x.x VMkernel

## Replace x.x.x.x with the default gateway for the COS in both of the next two lines.
route add default gw x.x.x.x
echo “GATEWAY=x.x.x.x” >> /etc/sysconfig/network

mv /etc/rc.d/rc.local.save /etc/rc.d/rc.local
EOF

chmod +x /tmp/esx_post_install.sh
cp /etc/rc.d/rc.local /etc/rc.d/rc.local.save

cat >> /etc/rc.d/rc.local << EOF
cd /tmp/
/tmp/esx_post_install.sh
EOF

As an example, in my environment I have server with 4 NICs and by default, ESX assigns vmnic0, which is mapped to PCI 02:00.00, to the service console. However, what is actually physically wired to my management network is vmnic3, which is mapped to PCI 02:03.00.  In the script above, I simply searched for the number 3 (i.e. replaced search term with 3) and now my scripted ESX installation works properly.

Below is the configuration of my server before I redeployed with kickstart.  The line in red is the NIC I want assigned to the COS.  The lines in black are what ESX assigns the COS by default.

BEFORE (without %post section)


[root@vesx7 root]# esxcfg-nics -l
Name    PCI      Driver      Link Speed    Duplex MTU    Description
vmnic1  02:01.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM

vmnic2  02:02.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic3  02:03.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic0  02:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM


[root@vesx7 root]# esxcfg-vswitch -l
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks

vSwitch0       64          4           64                1500    vmnic0

PortGroup Name      VLAN ID  Used Ports  Uplinks
VM Network          0        0           vmnic0

Service Console     0        1           vmnic0

Now, here is the same output after I redeployed the server with my modifications to the %post section of the kickstart file. The scripted deployment of ESX now properly assigns vmnic3 to my service console.

AFTER (with %post section)

[root@vesx7 root]# esxcfg-nics -l
Name    PCI      Driver      Link Speed    Duplex MTU    Description
vmnic1  02:01.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic2  02:02.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic0  02:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic3  02:03.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM

[root@vesx7 root]# esxcfg-vswitch -l
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks

vSwitch0       64          5           64                1500    vmnic3

PortGroup Name      VLAN ID  Used Ports  Uplinks
Production          0        0           vmnic3

Service Console     0        1           vmnic3

I hope this was helpful.  Let me know if you have any questions.

Well, I’d better sign off and start packing because I leave for Omaha, NE in a few hours.



Post to Twitter Post to Delicious Post to Digg Post to StumbleUpon

A Slick USB Over IP Solution That I use Every Day

Let me show you a slick application I stumbled across about 6 months ago.  My HP all-in-one USB printer is certainly a handy device, but being bound to the thing via a USB cable was driving me nuts.  I wanted to have full control (which meant a print server wasn’t going to cut it) and I wanted that control from anywhere in my house over my wireless network.  Now, there are a number of products on the market that can do this.  As an example, AnywhereUSB from Digi is one such product which seems to have a good reputation and from what I’ve seen, it works well.  But, I was in one of my moods and I was bound and determined to find a free solution.  In my searches I found a few software products and tried the 30 day demos.  They all worked well but they weren’t free and they were all products for Windows.  And since my WindowsXP desktop is actually a VMware Workstation virtual machine running on top of Ubuntu Linux, what I really needed was a solution for Linux.

I knew that if I could find something for Linux I would kill two birds with one stone.  Because not only would I be able to connect USB devices to my Linux OS, but as an added benefit, my WindowsXP virtual machine would see the connected device just as if it were actually connected … no additional software needed for Windows!  Eventually I stumbled across USB Server (+ USB Client) for Linux Beta which is offered as a freeware product from IncentivesPro (http://www.incentivespro.com).  After playing with the product a bit and creating a few custom scripts to further automate the connecting and disconnecting of USB devices, I found the solution I was looking for!  And I can tell you, I use the product on a daily basis.  Here’s a quick look at my setup.

1) I have an HP all-in-one USB printer connected to a Linux server running USB Server for Linux.

2) On my laptop, running Ubuntu 8.04 I have the USB Client for Linux installed.

3) To automate the process of connecting to the server and attaching the USB devices, I created a Bash script called connect_usb.  Simply running this script produces the following output …

asweemer@cowbuntu:~$ connect_usb
Restarting USB Server on sweemserv … Success!
Restarting local USB Server … Success!
Connecting to USB Server on sweemserv … Success!
Looking for USB Devices on sweemserv … Success!
Found the following USB Devices on sweemserv:

1: USB Server on sweemserv:32032 status: [connected]
‘–> 5: USB Device:   Officejet 5600 series  HP  – Composite USB Device
busid: 1-1       hwid: 03f0-4f11
speed: [full]    status: [device is connected]

Connecting to the USB Devices on sweemsrv … Success!
asweemer@cowbuntu:~$

After I see this, I have full control of the device just as if I had the USB cable plugged directly into my laptop.

4)  I already mentioned that I run my VMware corporate XP desktop as a Workstation 6.5 instance.  The USB Client presents the USB devices as local, so I connect to them in Workstation as if they were directly connected devices.  Check out the following screenshot …

See the last line “Hewlett-Packard Officejet 5600 series”?  That’s the printer.  And believe me, it’s not local like the other devices in the list, but VMware Workstation doesn’t know the difference.  And when I connect the device to the virtual machine, WindowsXP doesn’t know the difference either.

6) That’s it!  Works like a charm everytime :)

If you’d like a copy of the Bash script, let me know and I’ll update the post.  Also, I’d be interested in any other unique ways to handle USB Redirection, so please comment if you have a solution.

Post to Twitter Post to Delicious Post to Digg Post to StumbleUpon

Bring on the 10Gig Ethernet!

VMware recently updated its networking performance tests to see if the ESX hypervisor could efficiently leverage the ever-expanding bandwidth available at the Ethernet level. In short, it sure can! A single VM can effectively saturate a 10Gbps link when jumbo frames are enabled. But that’s not to say it can’t perform well with multiple virtual machines. Things scaled nicely and equitably for all VM’s. This type of scalable performance is reassuring as customers continue to raise consolidation ratios within their datacenters and virtualize the largest of workloads.

To save you some reading, here is the summary from the whitepaper, which can be found at: http://www.vmware.com/pdf/10GigE_performance.pdf

Conclusion:The results presented in the previous sections show that virtual machines running on ESX 3.5 Update 1 can efficiently share and saturate 10Gbps Ethernet links. A single uniprocessor virtual machine can push as much as 8Gbps of traffic with frames that use the standard MTU size and can saturate a 10Gbps link when using jumbo frames. Jumbo frames can also boost receive throughput by up to 40 percent, allowing a single virtual machine to receive traffic at rates up to 5.7Gbps.

Our detailed scaling tests show that ESX scales very well with increasing load on the system and fairly allocates bandwidth to all the booted virtual machines. Two virtual machines can easily saturate a 10Gbps link (the practical limit is 9.3Gbps for packets that use the standard MTU size because of protocol overheads), and the throughput remains constant as we add more virtual machines. Scaling on the receive path is similar, with throughput increasing linearly until we achieve line rate and then gracefully decreasing as system load and resource contention increase.

Thus, ESX 3.5 Update 1 supports the latest generation of 10Gbps NICs with minimal overheads and allows high virtual machine consolidation ratios while being fair to all virtual machines sharing the NICs and maintaining 10Gbps line rates.

Post to Twitter Post to Delicious Post to Digg Post to StumbleUpon