Networking
UPDATE: Connections and Ports in ESX and ESXi
Sep 22nd
Mr. Dudley Smith has updated his PDF diagram with some minor corrections and additions. Get the latest, most up-to-date version here (click the graphic) …
He also updated “the brain” which can be found at it’s new home http://webbrain.com/brainpage/brain/89EFA582-2C35-F6A2-9ED1-7AD4810266C2/. Make sure you update your bookmark accordingly.
The E.T.D.F. Series – Setting up the Network and Dedicated Remote Access (Part 3)
Jun 8th
- OK, we are *almost* done getting our network set up properly for VDI, but we’ve got a few more things to do. Specifically, we need to address:
- Handling our external, Internet facing, dynamic IP address.
- DHCP and DNS
- External VPN access
Dynamic External IP
Most ISP’s (not all) provide a single, dynamic IP address for consumer grade service (i.e. home use). But when we’re trying to connect to our virtual desktops from somewhere out on the public Internet, how do we know which IP address to connect to?
Generally speaking, connecting to an IP address is bad practice because it’s inflexible. Instead, we should connect to a Fully Qualified Domain Name (FQDN). OK fine, so we’ll set up a DNS entry and use a FQDN to connect to our desktops. But what happens when our dynamic IP address changes and the DNS entry is still mapped to the old IP address?
What we need is an external Dynamic DNS (aka DDNS) service which will allow us to programmatically update our IP address whenever it changes. There are a number of both free and paid-for DNS providers out there that can deliver DDNS services. Personally, I use EditDNS (www.editdns.net). They have a ton of functionality and they’ve been rock solid for the past few years I’ve been using their services, so I’m quite happy with them.
Now, many home use routers these days have the capability to update a DDNS provider. But in my experience, the functionality is somewhat limited. What if, for example, I want aaron.sweemer.com and desktop.sweemer.com to be dynamic entries and www.sweemer.com to be a static entry, pointing to my blog server hosted somewhere else? In reality, I’ve got about 20 FQDN’s that I need to be dynamically updated and about 100 that I want static. So instead, I created a script that will:
- Query my external IP address (check this out, a free tool from Whatismyip.com)
- Compare the result of the query with the IP obtained from the previous query
- If the IP is the same, or contains something other than an IP (e.g. HTTP error), the scirpt exits.
- If the IP is different, the script updates my DDNS entries via the EditDNS API, then updates a log file documenting the change, and finally adds the new IP to the last line of a file called previous_ips.
If you’d like to use the script I wrote, you’ll first need to do the following:
- If don’t already have one, set up an account with EditDNS and make sure you have properly configured the domain name(s) you own.
- Verify your linux distro has lynx (a command line, text only, www client)
- Verify your linux distro has curl (a tool to transfer data using HTTP)
- Create a directory (anywhere you have rwx access is fine) for the script and its files to live
- In this directory, create a text file called editdns.sh. Paste the content (below) into it.
- Replace XXXXXXXX with your EditDNS password.
- Make editdns.sh executable (chmod +x /path/to/editdns/editdns.sh)
- Create another text file called records and, one per line, enter the FQDN’s of the DDNS entries you wanted updated (e.g. mydesktop.mydomain.com)
- Add the editdns.sh script to your crontab to run at regular intervals (e.g. mine runs every five minutes and the entry in cron looks like this: */5 * * * * cd /usr/local/editdns; ./editdns.sh)
And here’s a copy of the actual script …
#!/bin/bash
EDITDNSPASS=”XXXXXXXX”
LYNX=`which lynx`
TIME=`date`
CIP=`curl -s http://www.whatismyip.com/automation/n09230945.asp | awk –re-interval ‘$1 ~ /^([0-9]{1,3}\.){3}[0-9]{1,3}$/ {print}’`
PIP=`tail -1 ./previous_ips`if [ "$CIP" != "$PIP" && –n "$CIP" ]; then
cat ./records | while read FQDN; do
$LYNX -source “http://DynDNS.EditDNS.net/api/dynLinux.php?p=$EDITDNSPASS&r=$FQDN”
done
echo “IP Change! New IP is $CIP. Editdns.net was updated at $TIME.” >> ./editdns.log
echo $CIP >> ./previous_ips
else
exit 0
fi
If you have any issues or questions, feel free to email me. Also, keep in mind, this a quick and dirty script that accomplishes what I want it to accomplish. Feel free to make it more robust (e.g. error handling or better logging) to suit your needs.
DNS and DHCP
It’s quite possible you’ll want to skip this section and opt for setting up DHCP and DNS via Microsoft’s built in DHCP and DNS services that come out of the box with their server products. To properly set up VMware View, we’ll need to set up Active Directory anyway, and quite frankly, it’s far easier to set up a Microsoft server with DHCP and DNS than it is to set up a Linux server. So feel free to skip this section and leverage Microsoft for these services. If, however, you’re a gluten for punishment then by all means, read on.
Let’s first start with DNS. Here too we need Dynamic DNS because as we’re handing out IP addresses via DHCP, we want our DNS server to properly reflect current information as IP addresses change. So, if you don’t already have bind9 (the DNS server), go ahead and install it (sudo apt-get install bind9 should work on Ubuntu / Debian distros).
The default configuration for bind9 is to act as a caching server, so the first thing we need to do is configure our DNS to forward all unknown DNS requests to another DNS server. These should be provided to you from your ISP. Edit the forwarders {} section of your named.conf.options file (usually located in /etc/bind/) to look like this …
asweemer@cincylab-rtr1:/etc/bind$ more named.conf.options
options {
directory “/var/cache/bind”;forwarders {
1.2.3.4;
5.6.7.8;
};auth-nxdomain no; # conform to RFC1035
listen-on-v6 { any; };
};
Obviously, you’ll need to change 1.2.3.4 and 5.6.7.8 to the IP addresses given to you by your ISP.
Next, we need to modify our master named.conf to allow dynamic updates to DNS. Add the following entry to the bottom of your named.conf file.
controls {
inet 127.0.0.1 allow {127.0.0.1; 192.168.9.25; 10.10.7.1; 10.10.7.2; } keys {“rndc-key”;};
};
This tells the DNS server to allow updates from the IP address located between the {}. Notice the first three IP addresses are local IP addresses. The fourth IP address is a slave DNS server, which I have yet to set up. The rndc-key is the default key generated during installation of bind9 and it’s used to authorize the updating of DNS records. If you’re using Ubuntu, then you’ll likely find the key in the file /etc/bind/rndc.key …
asweemer@cincylab-rtr1:/etc/bind$ sudo cat rndc.key
key “rndc-key” {
algorithm hmac-md5;
secret “QZ5jOmcr/OW3nzksR5q0Hw==”;
};
asweemer@cincylab-rtr1:/etc/bind$
Note the file is a text file named rndc.key, and the actual key is called rndc-key located within the text file.
OK, next we need to define our zones in the named.conf.local file. For each domain you’re using (probably just one), you’ll need two entries: one for the domain and one for the reverse lookup of the domain. I have two domains I’ll be updating, so my named.conf.local file looks like this …
asweemer@cincylab-rtr1:/etc/bind$ cat named.conf.local
//
// Do any local configuration here
//include “/etc/bind/rndc.key”;
zone “mydomain.com” {
type master;
file “/etc/bind/zones/mydomain.com.db”;
allow-update { key “rndc-key”; };
allow-transfer {10.10.7/24; };
};zone “7.10.10.in-addr.arpa” {
type master;
file “/etc/bind/zones/rev.7.10.10.in-addr.arpa”;
allow-update { key “rndc-key”; };
allow-transfer {10.10.7/24; };
};zone “dmz.mydomain.com” {
type master;
file “/etc/bind/zones/dmz.mydomain.com.db”;
allow-update { key “rndc-key”; };
allow-transfer {192.168.9/24; };
};zone “9.168.192.in-addr.arpa” {
type master;
file “/etc/bind/zones/rev.9.168.192.in-addr.arpa”;
allow-update { key “rndc-key”; };
allow-transfer {192.168.9/24; };
};
A couple points to note here:
- I created a subdirectory called “zones” under /etc/bind/ where I put all my zone files. This isn’t the default location, and in addition, this isn’t necessary as the zone files can be located anywhere you’d like. But be aware the configuration file above reflects the location of my files.
- Notice the include “/etc/bind/rndc.key” on the first line and the all-update directive within each zone definition? This should be self explanatory at this point.
- The allow-transfer directive within each zone definition explicitly limits zone transfers (copy) to the IP(s) defined. This is an important security feature since, by default, DNS allows transfers to anyone, and the info contained within a DNS zone file can really give hackers visibility into your network.
Now we need to create the zone files we just defined above, which will contain our actual DNS records. Here is the zone file for our dmz.mydomain.com …
asweemer@cincylab-rtr1:/etc/bind/zones$ cat dmz.mydomain.com.db
$TTL 3600 ; 1 hour
dmz.mydomain.com IN SOA master.dmz.mydomain.com. root.master.dmz.mydomain.com. (
2009060514 ; serial
86400 ; refresh (1 day)
86400 ; retry (1 day)
2419200 ; expire (4 weeks)
3600 ; minimum (1 hour)
)
NS master.dmz.mydomain.com.
NS slave.dmz.mydomain.com.
A 192.168.9.25
MX 10 mail.dmz.mydomain.com.
MX 20 mail-spool.dmz.mydomain.com.
computer-1 A 192.168.9.247
TXT “317bf41a2c5b70fd9ca4e283d364dcddd5″
computer-2 A 192.168.9.250
TXT “00cf6242f693ebbf1d545159548e44ab81″
computer-3 A 192.168.9.243
TXT “31a0cb7e096a96c63dc998d2db3be6e450″
mail A 192.168.9.25
mail-spool A 192.168.9.26
master A 192.168.9.25
www CNAME master
A couple important things to point out here:
- The entries for computer-1, 2 and 3 are dynamic entries there were generated by the DHCP server. The TXT record that follows these entries is a unique identifier which is also generated by the DHCP server and is used to ensure it won’t overwrite existing DNS records that were generated by another process/server.
- You’ll obviously need to change the domain names and IP addresses to match your environment.
- If you haven’t worked with bind9 before, this file probably looks pretty cryptic to you. If so, I would recommend taking a look at http://www.zytrax.com/books/dns/ch8/soa.html, which gives a pretty good overview of the SOA (defined in the first part of the file). The balance of the file (i.e. the record definitions) is pretty straight forward.
The reverse zone should look like this …
asweemer@cincylab-rtr1:/etc/bind/zones$ more rev.9.168.192.in-addr.arpa
$ORIGIN 9.168.192.IN-ADDR.ARPA.
$TTL 1h;
IN SOA master.dmz.mydomain.com. root.master.dmz.mydomain.com. (
2009060501 ;
1d ;
1d ;
4w ;
1h ;
)
IN NS master.dmz.mydomain.com.
IN NS slave.dmz.mydomain.com.
25 IN PTR master.dmz.mydomain.com.
26 IN PTR slave.dmz.mydomain.com.
I mixed it up just a bit in this file to point out a few different ways to configure a zone file. In this file, notice the following differences:
- The $ORIGIN directive sets the domain name to be appended to any unqualified records. If the $ORIGIN directive doesn’t exist (as it doesn’t in the first config file), then it is implicitly defined by the zone name.
- The time variables can be defined with d (day), w (week), h (hour), etc.
That’s about it for DNS. Once you’ve got your bind9 server configured, restart your bind9 server (sudo /etc/init.d/bind9 restart). And of course, be sure to test your configurations by using the standard DNS tools (e.g. dig, nslookup). If you get errors, pay careful attention to your local syslog file (probably located at /var/log/syslog) as that’s where DNS and DHCP errors typically write their error messages.
OK, next up is configuring our DHCP server. And once again, this post is starting to get way to long, so it looks like I’ll need a fourth and (hopefully) final post to this section.
View 3.1 USB Redirection Improvements
May 28th
As you probably have seen, View 3.1 GA’d yesterday. One of the improvements listed in the release notes was:
- USB Improvements – View 3.1 offers more reliable and broader device support with reduced bandwidth consumption. A separate TCP/IP stream is used.
From what I understand in talking to some people is that a lot of time was spent on the USB redirection stack to further optimize and tune it.
ALSO, USB redirection traffic is now split out onto it’s own traffic stream. USB redirection traffic will now communicate from the client to the host vm on TCP port 32111. I imagine this opens up a few new opportunities to do some USB specific traffic prioritization/trottling. Very interesting! In previous versions, the USB traffic was inside of the RDP stream (virtual channel). This prevented us from ever REALLY seeing the USB specific traffic or having any control over it. Simply put, now we do. Gotta love progress!
Scripted ESX Installation: Reconfiguring COS Networking with Kickstart
May 27th
Frequently customers have specific NICs (like onboard NICs) that they’d like assigned to the COS, leaving the other NICs for VM traffic. This is difficult, however, when using our automated kickstart deployment scripts as there is no way to explicitly define the vmnic assigned to the COS. And to make matters worse, the VMkernel is not yet available to us during the %post section of the kickstart script, which makes COS networking configuration difficult! Recently I had a customer who was getting frustrated because …
- They would “rack and stack” a physical server and wire up their NICs accordingly (i.e. onboard NICs on the management VLAN, remaining NICs on production VLANs)
- PXE boot the server
- When kickstart completed, they’d lose connection to the COS.
This happens because during installation, ESX just assigns vmnic0 to the lowest PCI number, and then assigns vmnic0 to the COS. And this is often not the NIC the admin wants used for their COS. Of course, they could go back after the fact and reconfigure the COS networking, but this kind of defeats the purpose of a completely hands-free, automated deployment.
Here is one possible solution to the problem. Below is a script I wrote to append to the %post section of a kickstart file. Obviously, you’ll need to make modifications for your environment.
|
## This script should be appended to the %post section of an ESX kickstart file. %post
cat > /tmp/esx_post_install.sh << EOF ## If your kickstart file has vmportgroup=1, you *might* want to uncomment the /usr/sbin/esxcfg-vswitch -A “VMkernel” vSwitch0 ## You’ll need to find which physical NICs you want assigned to your COS. From
/usr/sbin/esxcfg-nics -l | awk ‘\$0 ~ /search term/ {print \$1}’ | xargs –n 1 /usr/sbin/esxcfg-vswitch vSwitch0 –L ## Note: if you want to test the line above from the command-line, you’ll need ## Replace the x.x.x.x after -i with the IP address and after -n with the ## Replace the x.x.x.x after -i with the IP address and after -n with the subnet ## Replace x.x.x.x with the default gateway for the COS in both of the next two lines. mv /etc/rc.d/rc.local.save /etc/rc.d/rc.local
chmod +x /tmp/esx_post_install.sh cat >> /etc/rc.d/rc.local << EOF |
As an example, in my environment I have server with 4 NICs and by default, ESX assigns vmnic0, which is mapped to PCI 02:00.00, to the service console. However, what is actually physically wired to my management network is vmnic3, which is mapped to PCI 02:03.00. In the script above, I simply searched for the number 3 (i.e. replaced search term with 3) and now my scripted ESX installation works properly.
Below is the configuration of my server before I redeployed with kickstart. The line in red is the NIC I want assigned to the COS. The lines in black are what ESX assigns the COS by default.
|
BEFORE (without %post section)
PortGroup Name VLAN ID Used Ports Uplinks |
Now, here is the same output after I redeployed the server with my modifications to the %post section of the kickstart file. The scripted deployment of ESX now properly assigns vmnic3 to my service console.
|
AFTER (with %post section)
[root@vesx7 root]# esxcfg-nics -l
[root@vesx7 root]# esxcfg-vswitch -l
PortGroup Name VLAN ID Used Ports Uplinks |
I hope this was helpful. Let me know if you have any questions.
Well, I’d better sign off and start packing because I leave for Omaha, NE in a few hours.
A Slick USB Over IP Solution That I use Every Day
Nov 20th
Let me show you a slick application I stumbled across about 6 months ago. My HP all-in-one USB printer is certainly a handy device, but being bound to the thing via a USB cable was driving me nuts. I wanted to have full control (which meant a print server wasn’t going to cut it) and I wanted that control from anywhere in my house over my wireless network. Now, there are a number of products on the market that can do this. As an example, AnywhereUSB from Digi is one such product which seems to have a good reputation and from what I’ve seen, it works well. But, I was in one of my moods and I was bound and determined to find a free solution. In my searches I found a few software products and tried the 30 day demos. They all worked well but they weren’t free and they were all products for Windows. And since my WindowsXP desktop is actually a VMware Workstation virtual machine running on top of Ubuntu Linux, what I really needed was a solution for Linux.
I knew that if I could find something for Linux I would kill two birds with one stone. Because not only would I be able to connect USB devices to my Linux OS, but as an added benefit, my WindowsXP virtual machine would see the connected device just as if it were actually connected … no additional software needed for Windows! Eventually I stumbled across USB Server (+ USB Client) for Linux Beta which is offered as a freeware product from IncentivesPro (http://www.incentivespro.com). After playing with the product a bit and creating a few custom scripts to further automate the connecting and disconnecting of USB devices, I found the solution I was looking for! And I can tell you, I use the product on a daily basis. Here’s a quick look at my setup.
1) I have an HP all-in-one USB printer connected to a Linux server running USB Server for Linux.
2) On my laptop, running Ubuntu 8.04 I have the USB Client for Linux installed.
3) To automate the process of connecting to the server and attaching the USB devices, I created a Bash script called connect_usb. Simply running this script produces the following output …
asweemer@cowbuntu:~$ connect_usb
Restarting USB Server on sweemserv … Success!
Restarting local USB Server … Success!
Connecting to USB Server on sweemserv … Success!
Looking for USB Devices on sweemserv … Success!
Found the following USB Devices on sweemserv:
1: USB Server on sweemserv:32032 status: [connected]
‘–> 5: USB Device: Officejet 5600 series HP – Composite USB Device
busid: 1-1 hwid: 03f0-4f11
speed: [full] status: [device is connected]
Connecting to the USB Devices on sweemsrv … Success!
asweemer@cowbuntu:~$
After I see this, I have full control of the device just as if I had the USB cable plugged directly into my laptop.
4) I already mentioned that I run my VMware corporate XP desktop as a Workstation 6.5 instance. The USB Client presents the USB devices as local, so I connect to them in Workstation as if they were directly connected devices. Check out the following screenshot …
See the last line “Hewlett-Packard Officejet 5600 series”? That’s the printer. And believe me, it’s not local like the other devices in the list, but VMware Workstation doesn’t know the difference. And when I connect the device to the virtual machine, WindowsXP doesn’t know the difference either.
6) That’s it! Works like a charm everytime
If you’d like a copy of the Bash script, let me know and I’ll update the post. Also, I’d be interested in any other unique ways to handle USB Redirection, so please comment if you have a solution.
Bring on the 10Gig Ethernet!
Nov 17th
VMware recently updated its networking performance tests to see if the ESX hypervisor could efficiently leverage the ever-expanding bandwidth available at the Ethernet level. In short, it sure can! A single VM can effectively saturate a 10Gbps link when jumbo frames are enabled. But that’s not to say it can’t perform well with multiple virtual machines. Things scaled nicely and equitably for all VM’s. This type of scalable performance is reassuring as customers continue to raise consolidation ratios within their datacenters and virtualize the largest of workloads.
To save you some reading, here is the summary from the whitepaper, which can be found at: http://www.vmware.com/pdf/10GigE_performance.pdf
Conclusion:The results presented in the previous sections show that virtual machines running on ESX 3.5 Update 1 can efficiently share and saturate 10Gbps Ethernet links. A single uniprocessor virtual machine can push as much as 8Gbps of traffic with frames that use the standard MTU size and can saturate a 10Gbps link when using jumbo frames. Jumbo frames can also boost receive throughput by up to 40 percent, allowing a single virtual machine to receive traffic at rates up to 5.7Gbps.
Our detailed scaling tests show that ESX scales very well with increasing load on the system and fairly allocates bandwidth to all the booted virtual machines. Two virtual machines can easily saturate a 10Gbps link (the practical limit is 9.3Gbps for packets that use the standard MTU size because of protocol overheads), and the throughput remains constant as we add more virtual machines. Scaling on the receive path is similar, with throughput increasing linearly until we achieve line rate and then gracefully decreasing as system load and resource contention increase.
Thus, ESX 3.5 Update 1 supports the latest generation of 10Gbps NICs with minimal overheads and allows high virtual machine consolidation ratios while being fair to all virtual machines sharing the NICs and maintaining 10Gbps line rates.

