Hadoop Starter Kit and EMC Solutions Insiders Episode 2

EMC Solutions Insiders is a tool to help enable and educate EMC and Partner field technical pre-sales people.

We were lucky enough to catch up with James Ruddy (@darth_ruddy) with the EMC Office of the CTO.

James created a pretty nice starter kit to help people get up and going with Hadoop on VMware using the new Big Data Extensions and of course the simplest way to attach data nodes with native HDFS on Isilon.

Check out the show page on the EMC Community Network to get more details and links. Additionally you need to go check out the Big Data Blog at EMC.

 

EMC Solutions Insiders – Episode 1 SDDC

EMC Solutions Insiders is a webcast for EMC SE’s and EMC Partner SE’s. The purpose is to enable the technical field on what is pertinent to our customers.

EMC Solutions Insiders on EMC Community Network

In our first attempt we were able to get an MVP, Aaron Chaisson to talk about the Software Defined Data Center (SDDC). Check out the Show page for additional links and information.

If you are an EMC customer (or want to be one) I would love to hear what you think. If you want more information contact your local EMC SE or EMC Partner SE.

Thanks.

Virtual Storage Integrator – VSI 5.5

Actually using the Virtual Storage Integrator (VSI) from EMC will simplify the complex environments used to create virtual data centers and private clouds. Wanted to bring to your attention some of the new features in the VSI

What is new?

vsi5.5whatsnew_png

A little deeper

wpid1252-media_1372694601114.png

Demo from my good friend Itzik Reich @itzikr

Itzik shows off how to use the Recoverpoint Point in Time with VMware SRM and the VSI plugin to recover from a bookmark that is not the latest. Always WAS a problem if you could only restore using SRM to a corrupted point in the data. Now you can choose and SRM will failover to that point in time. Read Iztik’s blog post on RecoverPoint 4.0 – http://itzikr.wordpress.com/2013/04/30/recoverpoint-4-0-finally/

wpid1253-media_1372694638201.png

Restore a VM from a VNX snapshot or RecoverPoint using the VSI plugin. Pretty cool.

Working on the demo for this now. I will post it in a few days.

 

VDI Calculator for VNX

Biggest question around sizing your VDI usually comes down to sizing the storage.

Some of the solutions team created a pretty cool sizing whitepaper a few months back. Which inspired me to create this web based calculator. It is not meant to do everything in the whole world.

Just give a quick and easy VNX setup.

http://vdi.2vcps.com

The source is on GitHub so please have fun with that.

Sample Output:
calc-shot

More on the EMC, Cisco and Citrix 5000 on vSphere CVD (Cisco Validated Design) in 30 minutes

UPDATED May 26 – New links included Mike Brennan’s latest blog going into the testing and details of the CVD – http://blogs.cisco.com/?p=114587

Scaling CVD for XenDesktop and EMC on Cisco <—-NEWEST Version of the paper!!!!

Back a few months ago I shared about the joint reference architecture that was able to boot and login 5000 users in 30 minutes. Using the Cisco UCS running VMware vSphere, with Citrix XenDesktop and an EMC VNX 7500. Cisco and Citrix answer a few questions with Craig Chapman on the joint validated design in this video:

http://youtu.be/Yw0P8J9e_vo

I really like this layout and it usually takes pretty pictures to get me interested. Here is a high level overview of the design.

 

 

media_1360240163674.png

 

Some of the original CVD links:

CVD Paper – 5000 User VDI with EMC, Cisco, Citrix and VMware
Mike Brennan’s blog about 5000 VDI users

Updated May 2013
Solution Brief from Cisco 4000 Users

Update May 24, 2013 – More links:

Mike Brenner blog on the CVD
Scaling CVD for XenDesktop and EMC on Cisco <—-NEWEST Version of the paper!!!!

 

 

VMware View Stretched Cluster

The last few days I have been considering the best way to stretch a cluster of VMware View resources. After digging and talking to people smarter than me I figured out there is a lot of things to consider and that means lots of ways to solve this. In this first post I want to highlight the first overall solution that was inspired by an actual customer. This design came from one of the fine EMC SE’s and it inspired me to share further. I stole his picture. It is very storage centric (imagine that) so most of what I share will give some detail to the VDI and VMware portion.

VMware View and VNX and Isilon

wpid1163-media_1360942508658.png

Probably more detail then you need. Important things to remember. The VPLEX will keep the Volume in sync across distance at each site. All the benefits of FAST Cache will still be in place for each site.
In this solution each location will have file data redirected to the Isilon for SMB shares. I will use the VMware View pools and entitlements to force users to each side. Group Policy (GPO) or Third Party Persona Management will direct the users to their data. We are active/active in the sense that workloads are live at each site. Active Passive for the File portion as we will only kick users to site B in the event of a planned or unplanned event.
In another post I will discuss what I learned to make it complete non-persistent site to site active active everything. There is some cool stuff coming here.
First I used Resource pools to map to the VMware View Pools I created to. In the picture below the “Dell-Blades” cluster hosts 1-3 are in site A and hosts 3-6 are site B. One problem How to make sure each pool is pinned to each location?

wpid1164-media_1360944054169.png

Create the VM Group and Host Groups first!

wpid1165-media_1360944271778.png

Create the VM Site A and B group first. Then Create the Host Groups. As simple as editing the settings in your cluster and clicking DRS Groups Manager. One gotcha is you have to have hosts and VM’s first before making the groups. This may be an issue you have have not provisioned your View desktops first (I would wait). Just use some dummy VM’s at first to get the rules created.

With the Groups created Create the Rule

wpid1166-media_1360944500319.png

Remember these rules should say “Should run on hosts in group” (big thanks to @VirtualChappy). If you don’t have the rules right failover won’t work in case of a site going away for whatever reason.

Useful Script for setting DRS Host Affinity for all VM’s in a Resource Pool

wpid1162-media_1360597971375.png

This script I located on the community forums from the amazing LucD and at fix from another community user "GotMoo" I love the VMware Community.

What is so cool is I can run this after provisioning all of my desktops to get them in the right DRS VM Group and since usually in VMware View Environments you might create and destroy desktop VM’s regularly this helps a ton.

$vCenterServer = "vcenter.domain.lab"
#authenticating and Connecting to the VC

$CurrentUserName = [Security.Principal.WindowsIdentity]::getcurrent().name
$cred = Get-Credential $CurrentUserName
Write-Output "Connecting to vCenter. Please stand by..."
Connect-VIServer -Server $vCenterServer -Credential $Cred

 

#Function for updating the Resource VM Groups
function updateDrsVmGroup ($clusterName,$resourcePoolName,$groupVMName){
$cluster = Get-Cluster -Name $clusterName
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$groupVM = New-Object VMware.Vim.ClusterGroupSpec
#Operation edit will replace the contents of the GroupVMName with the new contents seleced below.
$groupVM.operation = "edit"

$groupVM.Info = New-Object VMware.Vim.ClusterVmGroup
$groupVM.Info.Name = $groupVMName

# Perform your VM selection here. I use resource pools per cluster to identify group members,
# but you could use any method to select your VM's.
get-cluster $clusterName | Get-ResourcePool $resourcePoolName | get-vm | %{
$groupVM.Info.VM += $_.Extensiondata.MoRef
}
$spec.GroupSpec += $groupVM

#Apply the settings to the cluster
$cluster.ExtensionData.ReconfigureComputeResource($spec,$true)
}

# Calling the function. I've found the group names to be case sensitive, so watch for that.
#updateDrsVmGroup ("ClusterName") ("ResourcePool Name") ("DRS VM Groupname")
updateDrsVmGroup ("UCS") ("DesktopsA") ("VM Site A")
updateDrsVmGroup ("UCS") ("DesktopsB") ("VM Site B")
# updateDrsVmGroup ("Cluster_STAGE") ("Group A") ("Group A VMs (Odd)")
# updateDrsVmGroup ("Cluster_STAGE") ("Group B") ("Group B VMs (Even)")
Disconnect-VIServer -Confirm:$False

More to come…

Finally this is a quick look at setting up View to be cross location. Of course other considerations about web load balancers, networking, number of View Connection Managers all need to be decided for your environment. Next post will include some of the stuff I found about keeping the users data live in both sites. Things like Windows DFS (Isilon can be a member), Atmos, VNX replication, and something called Panzura.

Cisco, EMC and Citrix 5000 Users in 30 minutes – Cisco Validated Design

Cisco + EMC + VMware + Citrix goodness.If you are working on deploying XenDesktop on VMware (the best way to do XenDesktop). Check out this article and Cisco Validated Design.

media_1360240163674.png
http://www.cisco.com/en/US/docs/unified_computing/ucs/UCS_CVDs/citrix_emc_ucs_scaleVDI.pdf
http://blogs.cisco.com/datacenter/zero-to-5000-citrix-vdi-users-logged-in-and-working-in-just-30-minutes/Give it a read and see how it all works.

If you are like me and looking for where they get “30 minutes” from goto page 11.
“We were able to ramp (log in and start workloads) up to steady state in 30 minutes without pegging the processor, exhausting memory or storage subsystems.”

Build IP Management Server with Ubuntu, GIT and Nmap

Have 25 Vlans in your lab and Storage, Servers, Switches and other equipment all over the place?

Build a good Ubuntu box

wpid1098-media_1358298686321.png
If you need more help on this leave some comments. It is pretty straight forward.
Be sure to select openssh server as a role.
After you login:
Remember to
$sudo apt-get update
$sudo apt-get upgrade
#If you are adventurous
$sudo apt-get dist-upgrade

Install VMware Tools (you ARE doing this in a VM right?)

wpid1099-media_1358299813754.png
From your VI Client set the VMware tools install to start, this will insert the disk for VMware Tools, but unfortunately doesn’t automate the install.
$sudo -s
#apt-get install build-essential
#apt-get install
#apt-get install apache2
The last two steps might not be needed but I do them out of habit. They allow the VMware tools to build the Kernel modules needed for VMXNET drivers.#mount /dev/cdrom /mnt/cd
#cp /mnt/cd/VMwareTools-8.6.0-425873.tar.gz
#tar xvzf /tmp/VMwareTools-8.6.0-425873.tar.gz
#cd vmware-tools-distrib
#./vmware-install.pl
Now Run through the install of VMware tools.#apt-get install nmap
#apt-get install gitweb
#exit
You want to do some of the next steps as your normal user not root.
$cd ~
$mkdir nmaps
$git init nmaps
$sudo vi /etc/gitweb.conf
Modify the project root var to match your path to you nmaps.
wpid1100-media_1358302137956.png
Now restart apache.
$sudo /etc/init.d/apache2 force-reload
wpid1101-media_1358302508614.png
Ok great, the webpage will be up at http://yourhost/gitweb
Now to add your network information:
$nmap -sP 10.10.10.0/25 -oG /pathtoyour/nmapsdirectory/file.map
<update> The above command is to scan a subnet with pings then dump the output to a file.</update>

Nmap – great network scanner, the -sP sets it to “Ping” mode, you don’t need to crawl ports or anything so ping is fine, it will resolve the DNS if that is working.
my example:
$nmap -sP 172.10.0.0/16 -oG /home/lab-admin/nmaps/vlan10.map

For what it is worth a /16 network takes forever. If you know the usable ip’s are a way smaller subset you can specify a range.
$nmap -sP 172.10.0.1-100 -oG /home/lab-admin/nmaps/vlan10.map

Next on the first time you must add the file to the git repository
$cd nmaps
$git add /home/lab-admin/nmaps/vlan10.map
$git commit -m “My Network Maps”

Now we have some content on the gitweb page.

wpid1102-media_1358344480271.png
Click tree to see the files you have added.
wpid1103-media_1358344539639.png
Click on blob to see your IP’s in use.

Let’s automate it with crontab

wpid1104-media_1358345701192.png
As you can see it is fairly easy to get rolling. Before you finish, run the nmap command for each subnet and git add each file to the repository.Now we will tell cron to run the nmap scans on a regular interval and git commit the changes.
$vi ip-scanscript.sh
In this file put each ip scan on its own line. As seen above. Save the changes and exit.
$chmod +x ip-scanscript.sh
$./ip-scanscript.sh
This will create all the files you want in one swoop.
$cd nmaps
$git add *
This will add all the new files in the nmaps directory to the git repository.Edit the script one more time.
$vi ip-scanscript.sh
wpid1105-media_1358346482687.png
By adding the git commit line to the script you can now set this up to automatically run with cron.
$crontab -e
[note]I was having issues with the range line in the screenshot above. Still testing.
[2nd note] change the git commit line to include a ‘-a’ example:
git commit -a -m “My IP Lists”
If you need help with crontab this is where I go when I can’t remember.
http://www.adminschoice.com/crontab-quick-reference
http://www.thegeekstuff.com/2011/07/cron-every-5-minutes/
I set mine to run every 4 hours.Hope this is helpful, not much reason to buy a fancy product when this can be up and running so quickly.

More References:
http://gitref.org/basic/

You down with VDP? Yeah you know me!

I had to be the first one to make a really bad joke.

Everyone will admit, how to efficiently back up your VM’s is a hot topic. Remember VDP is VMware’s product, but a lot of EMC technical people should be able to let you know right away how it works. VDP will be an excellent fit for a lot of customers with environments where they can’t spend extra on “virtual” backups.

Here are some of my favorite things in the new VDP.

  1. First it is built right into the new vSphere Web Client
  1. A simple wizard guides you through making the jobs.
  2. VDP uses Change block tracking to accelerate full restores.
  3. Integrated Self-service File level restore. What is better than file level restore? No one opening a ticket to ask you to do it!

The other stuff

Someone will eventually ask what is the difference between VDP and Avamar?

VDP

  • Max # VMs: 100
  • Storage Pool: up to 2TB
  • Replication (DR): None
  • Image-Level backup only

Avamar

  • Max # VMs: Unlimited
  • Storage Pool: up to 124TB *
  • Replication (DR): Included
  • Image-Level backup
  • Guest-Level backup
  • Extensive application support
  • Physical & Virtual backup
  • NAS/NDMP support
  • Desktop/Laptop backup
  • Extended retention to VTL/tape
  • Enterprise management
  • Purpose-Built Backup Appliances
    • Avamar Data Store
    • Data Domain integration *

Get Ready for the vHunt at VMworld with some bonus prizes

I am very excited this year for VMworld. As is tradition there will be an In-n-Out run. More details on this as my calendar is more packed in previous years. It might need to be a late night run.

One other thing I am excited about is this years vHunt twitter game. In addition to all of the fun prizes EMC will be providing (a new iPad for one). I will be carrying out at the show some very special prizes thanks to VMware Press.

20120824-111946.jpg

  • Automating vSphere with VMware vCenter Orchestrator by Cody Bunch
  • The Official VCP 5 Certification Guide by Bill Ferguson
  • VMware vSphere 5 Building a Virtual Datacenter by Eric Maille and Rene-Francois Mennecier
  • Managing and Optimizing VMware vSphere Deployment by Sean Crookston and Harley Stagner
  • Administering VMware Site Recovery Manager 5.0 by Mike Laverick
  • While supplies last and if you find will need to find me OUTSIDE of the EMC Booth to get the prize. Once you find me tweet a picture with me and your new book with the #vHunt hastag and copy @VMwarepress.