Cisco about 5 years ago-ish got into the server business when they entered the market with Cisco UCS Servers. If you go to the CCNA Datacenter YouTube channel playlist I talk about the different models of UCS servers (C, B, and E series). In this blog post I’ll walk you through configuration of a E series server that I had to deploy recently.
UCS E blades have a whole different concept that you might be used to. Since Cisco released their NX-OS and UCS lines, everything is becoming more virtualized and because of this, we can’t think anymore in terms of just physical. The case is the same here with UCS E and how it interacts with VMware (Hyper V too but I deployed a VMware setup in my case). I’m sure the C and B series do the same.
Aside from the Cisco documentation, of course hit up Ciscolive.com and search for any UCS E breakouts. I liked the BRKVIR-2003 “Unified Computing System E-Series: virtualization ready branch server platform for ISR G2” At 45 minutes the speaker explains how internally the UCS E blade talks to the host router.
Cisco’s Getting Started Guide could use some work. Its not too clear on what the terms mean, and the explanations are too brief on what MGF (multi-gigabit fabric) is and what UCSE2/0 and 2/1 are. the Cisco Live breakout session really helps with this.
Luckily, your reading this post!
ANATOMY OF THE UCS-E BLADE
The first thing to understand with the UCSE is that its a blade in a ISR G2 router. Meaning that instead of having to have a standalone server sitting at a site, you can now insert the server into a ISR G2 router and managing it all from one spot. If anyone used the SRE modules from the ISR G1s, some of this you might know already. http://www.cisco.com/c/en/us/products/interfaces-modules/services-ready-engine-sre-modules/index.html
This of course means new terminology and configuration.
In my case, I have a 2951 ISR2 router. The blade has already been built in so when I do a #sh ip int b I can see in here two new interfaces.
ucse2/0 unassigned YES unset up up
ucse2/1 unassigned YES unset up up
This is where you gotta make a mental leap of faith with me. Currently the Cisco documentation isn’t the best on explaining what these two interfaces do. Cisco says slotx/0 is the Layer 3 Console “port”, and the slot x/1 is the Layer 2 MGF VLAN interface. …..huh?
Taken straight from Cisco’s site, the image on the left does an OK job in explaining how the UCSE series works with the router and the actual blade itself. Picture the bottom of the diagram the actual blade itself. In the front of the blade you got 3 ports. M, GE2, and GE3. For some reason, GE1 and M port are interchangeable in the Cisco world. GE1/M is listed as Port #3 on the diagram. This port is physical, or “external”.
GE2 and 3 are listed in the diagram as Port 4 and 5 on the diagram, respectively.
The thing that you need to wrap your head around are these logical interfaces. Similar to BVI interfaces, USCE has UCSE interfaces.
Those UCSE2/0 and USCE21 interfaces are our logical “BVI” interfaces, or loopbacks — to stretch the analogy. Below is a diagram taken from the Cisco Live breakout session that I think explains little better than the diagram used in the documentation of how PCIe and MGF interfaces work.
PCIe is the L3 USE2/0 interface. MGF is the L2 interface. These two interfaces are used to talk to the “backplane of the router”. Basically a way that the router can talk to the UCSE blade sitting in its chassis. Thats what the diagram above is trying to show.
So the whole point of UCSE is to have a server in a router. You can choose from Linux, VMware, HyperV, Citrix Xenserver, but in my example I chose VMware 5.1
So you have the option of choosing if you want to get to your vms using either the PCIe interface (l3), the MDF (l2 interface), or use one of the external ports. In my case, I will be using the PCIe internal port.
CISCO INTEGRATED MANAGMENT CONTROLLER (CIMC)
Another piece of the puzzle you need to know about is the CIMC, which is basically Cisco’s version of HP’s iLO and Dell’s DRAC. Its a way to remotely manage and configure the host running off the blade. Its a GUI interface but for the die-hard CLI fans there is away to configure the blade through the CLI although as a CLI fan, I must say this GUI is really straightforward and after working with ILO and DRAC in the past, is a quick and easy way to configure the blade. There’s not much to it, so learning the CLI syntax might be counterprodutive unless the GUI is hosed, and even then theres not much commands for the IMC.
Here one can do the normal things you’d can you on a DRAC/ILO, restart the server, launch a console session, check serial numbers, logs, network settings, upgrade BIOS/firmware among other things. A nice plus if you run UCS Manager, you can add this standalone CIMC session to it so you dont need to manage it one by one.
I’ll start off with assuming you already have the UCS E blade installed and when you run the #sh ip int b command you see the internal ucse2/0 and ucse2/1 interfaces. I will be using the PCIe port for communication between VMware ESXi and the router.
1. Configure the slot x/0 interface.
Since I will be creating a l3 interface to get to my VM host, let’s do the following config…
ip address 10.56.204.2 255.255.255.0
imc ip address 10.56.100.101 255.255.255.0 default-gateway 10.56.100.1
imc access-port dedicated
description Internal switch interface connected to Service Module
no ip address
So what does this all mean? First, remember we said ucse2/0 is our layer 3 interface to get to the host. But before that, I need to get an OS on my blade before I even GET to the host. This is where CIMC comes in. CIMC will be the way I install the host’s OS. On the blade there is a green port called M. In reality this port is GE1 external. You once again have a few options to configure how to get to the CIMC. Call me old school, but I like a physical cable going for the mgmt port. You can optionally use the backplane of the router to get to host but this way works as well.
Thats great and all, but where do I plug the other end to? Well, if it isnt a router, its probably going to be a switch! On the other end of my switch I simply configure the port an access and put it into VLAN 10 which is our management vlan.
Now going back to the ucse2/0 config, the command imc ip address 10.56.10.101 255.255.255.0 default-gateway 10.56.10.1 simply means give the CIMC the IP address of 10.56.10.1/24 and its default gateway for the mgmt vlan that I had before will be 10.56.10.1. Simple!
The next command, imc access-port dedicated means to “hardcode” the M Port for just CIMC access which is what I want. If you want a “shared” setup, you can choose other options instead of dedicated like lom-shared or console.
Finally, the IP address which is ip address 10.56.204.2. This will be assigned so that the host can see the l3 interface inside of the blade and with the dedicated command, will have an exit point to the physical world. This diagram should help clear this up…the M port connects to the real world but internally, the ucse2/0 port is in a 10.56.204.0/24 subnet with .2 and that will be the IP that the host will use as its default gateway. Now, not only is CIMC setup, but I can now also get to the host. More on this later…
Once we have those commands put in, we can now access our CIMC!
2. CIMC Config
Default credentials for the UCS is admin and password. When you first login you get prompted to change the password.
Once in, first thing you should do is build the RAID array. Go to SERVER > RAID > and you get prompted to create an array. If not, click under Actions > Create. UCS E blades only get 1-3 drives, so its pretty straight forward what kind of RAID you can do. In my case, I only had 2 drives, so RAID1 it is!
With the RAID out of the way, now you can do housekeeping items like changing the hostname, make the login for CIMC Active Directory aware, upgrade the firmware, etc.
What if I want to install Windows?
f you want to use Windows, you don’t need any custom image. Instead, you need to download from the Cisco site the right RAID firwamre drivers. See Downloads > Products > Servers – UC > Cisco UCS E-Series Software > Cisco UCS E_your_model Software > Drivers
3. Install an OS (Vmware ESXi 5.1)
Next, is to install the actual OS. Like I said before, I will be installing ESXi5.1. If your doing a remote install, I recommend you drag the ISO to a local server or desktop to avoid having the CIMC reading an ISO thousands of miles away! Make sure you download the custom image from VMware site for Cisco.
Once you got it and you put it as close to the CIMC IP as you can to avoid latency, bring up the KVM. KVM is that little keyboard icon at the top of CIMC.
Once in the KVM, click on the Virtual Media tab and click ADD IMAGE. Add in the iso for VMware and click the checkbox on the MAPPED column. Now just restart the host by either going back to CIMC and hitting POwer Cycle Server to on the KVM go to Macro > CTRL+ATL+DEL
If you find issues booting from the CD and it keeps going through PXE/DHCP loop, ensure your BOOT ORDER is correct. Do this by going to SERVER > BIOSand click CONFIGUE BOOT ORDER.
4. OS Installation
This part everyone should be familiar with, go through the prompts like you would do any other vm. Once the install is complete, give it the static IP config and/or hostname, DNS, password, etc you would give any other ESXi host. Refer to step 1 visio diagram for how the ESXi talks to the ucse 2/0 interface.
5. Final steps
If everything went OK, you can now ping the ESXi host from your LAN. vSphere to the IP you gave the ESXi host and manage it like you would normally. If your having issues, check to make sure you selected the “M Port” for your Management Interface on the vmware side. Alas this part is trial and error since its “lowest MAC address will map to vmnic0”. Cisco’s documentation states this, so I just find it easier to disconnect all cables except for the M port. That’ll narrow down your choices.
Hope this blog post helps someone out there going down the same route I did!