vmware | CrashLoopBackoff

Not the Same Ol’ Sessions from Pure Storage at VMworld

I am really excited to be going to VMworld once again. I will be wearing my Orange Nike so most likely my feet won’t hurt quite as bad. Also expect the Pure Orange Superman to make an appearance.

More about the sessions. So I will be attending VMworld San Francisco, and speaking in EMEA.

STO2996-SPO – The vExpert Storage Game Show

The session I am stoked to be a part of is STO2996-SPO – The vExpert Storage Game Show. It will be a fun and informative time about next generation storage architectures presented in the form of a game show. PLUS, two members of the audience will join the session to help the vExpert teams. I know everyone will want to be on my team in EMEA.

STO3000-SPO – Flash Storage Best Practices and Technology Preview

This very exciting session with Vaughn and Cody (super-genius vExperts) will go into what to consider when moving your datacenter to all flash. Plus previews of the Pure VVOLs. If you think you are not ready for all flash, come to this session and learn how Flashy you can be.

STO2999-SPO – Customers Unplugged: Real-World Results with VMware on Flash

I wish I had thought of this. Customers using All Flash with VMware. All Tech, No Slides.

STO1965 – Virtual Volumes Technical Deep Dive

Dive into Virtual Volumes with Rawlinson Rivera – VMware, Suzy Visvanathan – VMware and Vaughn Stewart – Pure Storage. So many customers have asked me what will VVOLS actually do over the last 3 years. This will be a great chance to find that out.

VAPP2132 – Virtualizing Mission Critical Applications on All Flash Storage

How does Pure storage enable that final 10% of critical applications that just a few years ago people said would be impossible? Meet my friend Avi Nayek from Pure and Mohan Potheri from VMware and learn how flash eliminates storage as the road block to critical applications becoming virtual.

MGT1265 – Improving Cloud Operations Visibility with Log Management and vCenter Log Insight

Cody Hosterman, Did I tell you he is smart? Yeah. He is. Join Cody and Dominic Rivera from US Bank and Bill Roth from VMware on how to increase your Cloud Operations Visibility.

SDDC2754-SPO – New Kids on the Storage Block, File and Share: Lessons in Storage and Virtualization

Lessons from all the upstarts in the storage industry. Most of them are not “startups” anymore. Finding new ways to solve the issues of using Virtualization with legacy storage. Pure Storage, Nimble Storage, Tintri, Tegile, Coho Data, Data Gravity and moderated by Howard Marks from DeepStorage.net.

STO2496-SPO – vSphere Storage Best Practices: Next-Gen Storage Technologies

The Chad and Vaughn show. Now with Rawlinson Rivera! Storage is changing. Did I say that yet?

More information on Pure Storage Sessions

Customers Unplugged at VMworld: All Tech, No Slides

Coming Soon: Support for VMware VVOLs
Pure Storage set to paint VMworld 2014 orange!

VAAI and XCOPY with Pure Storage

VAAI has been around (almost 4 years now)for a while now and this is one thing I don’t often hear customers or others talking about very often. When your vSphere hosts detect that Hardware Acceleration is compatible. The host will attempt to send VAAI compatible commands to the storage device. As we describe it usually Full Copy is explained as if you need to clone or Storage vMotion a VM the ESXi host issues a command to move the storage device to move the blocks. So when describing this in the past it was a very simple, the Host issue the command and the blocks move. Set it and forget it, right?

Not so fast, my friend!

As good ol’ Lee Corso would say, “Not so fast, my Friend!”

The VAAI Xcopy command tells the storage device to move 4096 KB (AKA 4MB) at a time. So every 4MB is a new command. Not a big deal for disk based xcopy because the blocks could only move from spindle to spindle so fast. Still way more efficient than before but sometimes not actually faster at all.

Along came the Flash Array.

The FlashArray, XCOPY and VAAI

The Pure Storage snapshot technology is used for XCOPY commands. No matter where they are coming from. This results in just a metadata pointer change in order to move the data. The blocks don’t actually move anywhere since they are stored once and mapped in metadata. This enables zero impact snaps and clones that can be created as fast as I can click the button in the GUI.
What does this all mean?
Since the ESXi host is telling the FlashArray to move 4MB at a time the copy function does not reach the full potential of what the FlashArray can really do. It is like using a freight train to move cargo across the country but only putting one box in each car.

Pure Storage recommendation

This is why Pure recommends changing the MaxHWTransferSize (the setting that controls the size of the transfer) to the maximum allowed 16384 (or 16MB).

Default is 4096
Commands to help you change the setting via the CLI

esxcfg-advcfg -g /DataMover/MaxHWTransferSize
Value of MaxHWTransferSize is 4096

Set the transfer size to the Pure Storage best practice:

esxcfg-advcfg -s 16384 /DataMover/MaxHWTransferSize
Value of MaxHWTransferSize is 16384

…but wait there is more!

So the Pure Storage FlashArray is cool with cloning multi TB volumes using xcopy with no impact on performance or space usage. So the question is why only 16MB at a time? (real answer should come from someone way smarter than me at VMware).

I am curious to try out a Storage vMotion or cloning persistent View desktops that fully use the power of the array.
Until then, still better than spinning disk or no VAAI at all.

What happened while getting 100% Virtualized

I often think about how many people have stalled around getting to 100% virtual. I know you are thinking I need to find some fun things to do. You are probably right.

The first thing I thought when I deployed my very first virtual infrastructure project back in the day was, “Man, I want to see if I can virtualize EVERYTHING.” This is before I knew much about storage, cloud, and management. I may be naive but I think there is real potential out there to achieve this goal. There is low hanging fruit still out there depending how you deploy your infrastructure. Having attended VMware Partner Exchange (PEX) I know how the ecosystem is built around your journey to virtualization. The biggest slide to resellers and other partners is the one VMware shows off that says, “Every $1 a customer spends on VMware they buy $9-11 in infrastructure.” Which I fully believe is the reason many customers never saw the FULL cost savings they could have when going virtual.

Roadblocks

I believe we all ran into a couple of different kinds of roadblocks on our path. First were organizational. Line of business owners, groups within IT and other political entities made traveling the road very difficult. Certain groups didn’t want to share. Others started to think VM’s were free and went crazy with requests. Finally the very important people who own the very important application didn’t want to be virtual because somehow virtualization was a downgrade from dedicated hardware.

Then if we were able to dodge the roadside problems organizationally, there were technical problems. Remember that $11 of drag? The big vendors made an art of refreshing and updating you with new technology. I know, I helped do it. So performance was a problem? Probably buy more disk or servers. Then every 3-5 years they were back, with something new to fix what the previous generation did not deliver on. This “spinning drag” in the case of storage slowed you from getting to your goal. 100%.

Disillusionment

At some point you lose the drive to be 100% virtual. The ideal has been beaten out of you. Well at least my vendor takes me for steak dinner and I get to go to VMworld and pretend I am a big shot every year. This is where you settle. Resign yourself to the fact that everything is so complicated and hard it will never get done. The big vendors make a huge living on keeping you there. Changing the name from VI, to Private Cloud, Hybrid super happy land or whatever some marketing guys that have never opened the vCenter client think of next.

Distractions

So trying to rebuild Amazon in your data center? Probably lots of other things to fix first. Using more complicated abstraction layers may help in the long run to building a cloud. I see more customers continue to refresh wasteful infrastructure with new infrastructure while they are still trying to figure this out. What we need is a quick an easy win. Make things better and save money right away. Then maybe we can keep working on building the utopian cloud.

The low hanging fruit

When we first started to virtualize we looked for the easy wins. To get you rolling again down the path we need to identify the lowest hanging fruit in the data center. We found all the web servers running at 1% CPU and 300MB of Ram (if that) and virtualized those so quick the app owner didn’t even know it happened. Just like a room of 1000 servers all running at 2% CPU usage there are giant tracks of heat generating spinning waste covering the data center. You had to get so many of them and stripe so wide just to make performance serviceable. You wasted weeks of your life in training classes to learn how to tweak and tune these boat anchors because it was always YOUR fault it didn’t do what the vendor said it would.

Take that legacy disk technology and consolidate to a system made to make sure it is not the roadblock on the way to being 100% virtual. I remember taking pictures of the stacks of servers getting picked up by the recycling people and now is the time to send off tons of refrigerator sized boxes of spinning dead weight. I am not in marketing so I don’t want to sound like a sales pitch. I am seeing customers realize their goal of virtualization with simple and affordable flash storage. No more data migrations or End of Life forklift upgrades. No more having to decide if the maintenance is so high I should just buy a new box. Just storage that performs well all the time and is fine running virtual Oracle and VDI on the same box.

How we do it

How is Pure Storage able to replace disk with Flash (SSD)? Mainly, we created a system from the ground up just for Flash. We created a company that believes the old way of doing business needs to disappear. Customers say, “You actually do what you said, and more.” (Biggest reason I am here). Also, do it all at the price of traditional 15k disk. Not there on SATA, yet.

Make it ultra simple. No more tweaking, moving, migrating or refreshing. If you can give a volume a name and a size you can manage Pure Storage.
Make it efficient. No more wasted space due to having to short stroke drives, no more wasted space because you created a RAID 10 pool and now have nowhere to move things so you can destroy and recreate it.
Make it Available. Support that is awesome because things do happen. Most likely though most of your downtime is planned when it comes to migrating or upgrading code. Pure Storage will allow zero performance hit and zero outage to reboot a controller to upgrade the firmware/code (whatever you want to call it). Pretty nice for an environment that needs ultimate it uptime.
Make sure it alway performs. Imagine going to the DBA’s and saying, “everything is under 1ms latency, How about you stop blaming storage and double check your SQL code?” Now that is something as an administrator I wanted to say for a long long long time.

Once you remove complicated storage from the list of things preventing you from thing preventing 100% virtual you can focus on getting the applications working right, the automation to make life easier and maybe make it to your kid’s soccer games on Saturday.

No Spindles Bro

I was assisting one of my local team members the other day with sizing a VM for Microsoft SQL. I usually always fall back to this guide from VMware. So I started out with the basic seperation of Data and Logs and TempDB.

Make it look like this:

VM Disk Layout

LSI SCSI Adapter
C: – Windows

Paravirtual SCSI Adapter
D: – Logs
E: – Data
F: – TempDB

Which is pretty standard. Then someone said, “Why do we need to do that?” I thought for a second or five. Why DO we need to do that? I knew the answer in the old school. Certain raid types were awesomer at the types of data written by the different parts of the SQL Database. We are in a total post-spindle count world. No Spindles Bro! So what are some reasons to still do it this way for an All Flash Array?

1. Disk Queues
I think of these like torpedo tubes. The more tubes the less people are waiting in line to load torpedoes. You can fire more, so to speak. Just make sure the array on the other end is able to keep up. Having 30 queues all going to one 2 Gbps Fiber Channel port would be no good. See number 3 for paths.

2. Logical Separation and OCD compliance (if using RDMs)
Don’t argue with the DBA. Just do it. If something horrifically bad happens the logs and data will be in different logical containers. So maybe that bad thing happens to one or the other, not both. I am not a proponent of RDM’s. SO much more to manage. If you can’t win or don’t want to fight that fight at least with RDM’s you will be able to label the LUN on the array “SQLSERVER10 Logs D” so you know the LUN matches to something in Windows. This also makes writing snapshot scripts much easier.

3. Paths
Each Datastore or RDM has its own paths, if you are using Round Robin (recommended for Pure Flash Array) more IO on more paths equals better usage of the iSCSI or FC interconnects. If you put it all on one LUN, you only get those queues (see #1) and those paths. Remember do what you can to limit waiting.
Am I going down the right path? How does this make it easier? Are there other reasons to separate the logs and data for a database other than making sure the Raid 10 flux capacitor is set correctly for 8k sequential writes? I don’t want to worry about that anymore. Pretty sure plenty other VM Admins and DBA’s don’t either.

For me a good exercise in questioning why I did things one way and if I should still do them this way now.

Getting Ready for VMworld in Barcelona

My planing for VMworld in Barcelona began right after Speed 2 Lead Megalaunch from EMC. First came the meetings about who should be staffing the booth. In years past EMC would staff the booth with mainly BU (business unit) experts. Great people, but this really made for a product centric style in the booth. Some vSpecialists would be floating around the booth having conversations that usually spanned across multiple EMC business units. (example: How do I configure iSCSI multipathing for VNX while backing up with Avamar and Data Domain?) This year our goal was to flip it around. Where there would be mainly pre-sales people that loved VMware and EMC together. Complimented by a few experts from the BU’s to answer the deepest of questions.

We found this worked great in San Francisco so it is back again. With all new specialists from the EMEA region. These awesome Advanced Software SE’ and vSpecialists will be able to lead conversations on:
– SDDC (Software Defined Datacenter)
– SDS (Software-defined Storage)
– Mission Critical Applications (and how to virtualize them)
– Management and Orchestration
– End User Computing
– Trusted IT
– Tech Previews (Next Generation type things)

After getting all the right people in place came the time to get all the right demos. While some of the demos are the same from San Francisco, expect to see a some really cool new things.

None of this would have worked without the help of several AWESOME people. Jim Sanzone and Paul Manning are amazing guys to work with and it is a pleasure to learn from you two.

Make sure to come see me and my friends in the EMC booth. It should not be hard to find.

As an add on project I will be participating with the EMCElect to he curate things at the show.

VSI 6.0 vSphere Web Client

I didn’t get a chance to post this to the blog earlier this week. I wanted to share a demo I worked on this week showing how to install the new Web Client based Virtual Storage Integrator (VSI). We all know that the windows only client is in its last days (lived about a year too long in my opinion). So the new plugin to provision storage for the vSphere admin is a welcome addition. This very first version supports storage that is being Softwared-defined via ViPR.

Watch Full screen to see it better.

VSI is no extra charge and you can download if you have a valid EMC support contract from http://support.emc.com

VMware vCenter Appliance 5.5 – Tour

So you have ESXi up and running. What is next? Get the vCenter appliance running. I downloaded the OVA and imported in just a few minutes.

After getting the appliance all booted go the https://<your-ip>:5480

Setup vCenter Options

I selected custom so I could go through all the options.

Oracle is also an option.

Fill in if external. Embedded you just need to choose a password for the Administrator.

Setup your Active Directory authentication. You can do this later if you don’t have the right information now. One thing I learned is the hostname of the appliance MUST be set to a FQDN for this to work.

NTP rocks!

Now you have a ESXi all ready and added. Start being Virtually awesome.

Installing VMware vSphere 5.5 – Quick Tour

So if you haven’t gone through it in your lab, what is better than getting an idea of how to install vSphere 5.5 with a few screenshots. For the beginners out there I just wanted to walk through the process really quick like.

Boot from the media!

Still looks very familar if you have done this before. Of course if you are so awesome why are you still reading?

Look! It’s vSAN

Is it VSAN, vSAN or Vsan?

I always use password123 – just so it is easy. Just kidding. SRSLY!

By the way a note to VMware: Hitting F11 is not awesome on a Mac. Just hold every key on the bottom left side of the keyboard and hit volume down key for those that have always been mac people and thought F11 is some kind of Air Force project. Actually just FN +F11

Woot! Now you are a pro. Go take the VCP. Oh and study a bunch first.

Now it is time to add it to your vCenter.

VMworld 2013 – In-N-Out Burger Meetup

The annual 2vcps run to In-n-Out burger! This kid from Southern California now living in Atlanta, Georgia needs to get some burgers. So let’s meet up get a Double Double and talk a bit about all the cool things going on at VMworld.

Meetup
Where: at the In-n-out in San Francsico
When: Tuesday 8/27 – 11pm – after the EMC Party – Great way to finish off or begin the rest of the night.
What: Eat Double Double, Fries

At 11pm I will be getting my Double Double, Fries and Chocolate Shake. Nothing like 1200 Calories at 11pm (only 2am EST).

UPDATE: OSX on Fusion VM

I won’t fully apolgize for saying I wanted to punch Apple. Mainly, this is made much harder than it needs to be. If this doesn’t make sense read my previous rant about getting the install image for OSX Mountain Lion to run a VM legally and licensed on my Mac Mini purchased with Mountain Lion pre-installed.

In that article I cite an article that ended up having the answer if I would just follow the intructions. Go there I will re-describe here with the steps to getting the image installed in Fusion.

Get the BaseSystem.dmg

First we are going to mount the Apple_Boot Recoery HD. The attach the baseSystem.dmg image as a drive.

From the list we now enter:

>diskutil mount readOnly /dev/disk0s3
and
>hdutil attach "/Volumes/Recovery HD/com.apple.recovery.boot/BaseSystem.dmg"

The result of the hdutil attach command opens this window in finder. This is where I got it all wrong. Instead of following the directions I thought. Well instead of copying everyting in the download and force quiting like the instructions. I will just get the VM to boot from the BaseSystem.dmg. This seemed to be working fine.

On a side note. The BaseSystem.dmg is invisible in finder. From the CLI you can see it and make a copy from the Recovery HD to your main partition. If you want to be able to actually view the file use the command for the CLI:
>chflags nohidden ./BaseSystem.dmg
All this will do is get you this error during the install process with the menacing eyes of the Mountain Lion and the barely QA’d error message saying you didn’t buy Lion. No duh I didn’t buy Lion my mac CAME WITH Mountain Lion. Oh well I digress.

Back to instructions that will actually help you.

Install OSX Mountain Lion

Show All Disks

Use your handy EMC Elect 64GB USB drive

Seriously my 8GB USB was too small the next size up I own is 64GB (thanks @mjbrender)

Now we wait – then Force QUIT

So with a large enough USB drive you click install. The installer will now download 4+GB of installtion files into a InstallESD.dmg image on that USB drive. Now you have to read carefully. You must Force Quit when the installer is finished downloading and ready to reboot. Don’t Reboot. ALT + right click (or two finger’d, whatever). If you just normally quit the app in the dock it will clean itself up and delete the InstallESD.dmg file. Some reported seeing the file in the “Trash” after a normal quit, that just not how I roll. Force that junk!

Now copy that file somewhere else for fusion install.

The USB drive in Finder

In Fusion create a new VM

I am using Fusion 5 so this probably won’t work in previous versions with OSX 10.8
Click Continue without disc.

Choose a disc or disc image…

Now select the InstallESD.dmg image. Now Fusion will detect that you are installing Mac OSX 10.8 and roll on. Amazingly enough now it does not ask for your AppleID to verify the installation was purchased from the App Store.

Install the VMware tools and now you have a happy little OSX VM.

Turning rant back on:
I started messing with this morning and now it is 3:45pm. Thanks for taking a Saturday Apple. Urgh.