Ephemeral or Persistent? The Storage Choices for Containers (Part 3)

In this, the final part of a 3-part series, I cover the latest developments in ephemeral storage. Part 1 covered traditional ephemeral storage and Part 2 covered persistent storage.

CSI Ephemeral Storage

With the release of Kubernetes 1.15, there came the ability for CSI drivers that support this feature, the ability to create ephemeral storage for pods using storage provisioned from external storage platforms. Within 1.15 a feature gate needed to be enabled to allow this functionality, but with 1.16 and the beta release of this feature, the feature gate defaulted to true.

Conceptually CSI ephemeral volumes are the same as emptyDir volumes that were discussed above, in that the storage is managed locally on each node and is created together with other local resources after a Pod has been scheduled onto a node. It is required that volume creation has to be unlikely to fail, otherwise, the pod gets stuck at startup. 

These types of ephemeral volumes are currently not covered by the storage resource usage limits of a Pod, because that is something that kubelet can only enforce for storage that it manages itself and not something provisioned by a CSI provisioner. Additionally, they do not support any of the advanced features that the CSI driver might provide for persistent volumes, such as snapshots or clones.

To identify if an installed CSI supports ephemeral volumes just run the following command and check supported modes:

# kubectl get csidriver
NAME       ATTACHREQUIRED   PODINFOONMOUNT   MODES                  AGE
pure-csi   true             true             Persistent,Ephemeral   28h

With the release of Pure Service Orchestrator v6.0.4, CSI ephemeral volumes are now supported by both FlashBlade and FlashArray storage. 

The following example shows how to create an ephemeral volume that would be included in a pod specification:

 volumes:
  - name: pure-vol
    csi:
      driver: pure-csi
      fsType: xfs
      volumeAttributes:
        backend: block
        size: "2Gi"

This volume is to be 2GiB in size, formatted as xfs and be provided from a FlashArray managed by Pure Service Orchestrator.

Even though these CSI ephemeral volumes are created as real volumes on storage platforms, they are not visible to Kubernetes other than in the description of the pod using them. There are no associated Kubernetes objects and are not persistent volumes and have no associated claims, so these are not visible through the kubectl get pv or kubectl get pvc commands.

When implemented by Pure Storage Orchestrator the name of the actual volume created on either a FlashArray or FlashBlade does not match the PSO naming convention for persistent volumes.

A persistent volume has the naming convention of:

<clusterID>-pvc-<persistent volume uid>

Whereas a CSI ephemeral volumes naming convention is:

<clusterID>-<namespace>-<podname>-<unique numeric identifier>

Generic Ephemeral Storage

For completeness, I thought I would add the next iteration of ephemeral storage that will become available.

With Kubernetes 1.19 the alpha release of Generic Ephemeral Volumes was made available, but you do need to enable a feature gate for this feature to be capable.

These next generation of ephemeral volumes will again be similar to emptyDir volumes but with more flexibility, 

It is expected that the typical operations on volumes that are implementing by the driver will be supported, including snapshotting, cloning, resizing, and storage capacity tracking.

Conclusion

I hope this series of posts have been useful and informative. 

Storage for Kubernetes has been through many changes over the last few years and this process shows no sign of stopping. More features and functionality are already being discussed in the Storage SIGs and I am excited to see what the future brings to both ephemeral and persistent storage for the containerized world.

Ephemeral or Persistent? The Storage Choices for Containers (Part 2)

In this, the second part of a 3-part series, I cover persistent storage. Part 1 covered traditional ephemeral storage.

Persistent Storage

Persistent storage as the name implies is storage that can maintain the state of the data it holds over the failure and restart of an application, regardless of the worker node on which the application is running. It is also possible with persistent storage to keep the data used or created by an application after the application has been deleted. This is useful if you need to reutilize the data in another application, or as enable the application to restart in the future and still have the latest dataset available. You can also leverage persistent storage to allow for disaster recovery or business continuity copies of the dataset. 

StorageClass

A construct in Kubernetes that has to be understood for storage is the StorageClass. A StorageClass provides a way for administrators to describe the “classes” of storage they offer. Different classes might map to quality-of-service levels, or different access rules, or any arbitrary policies determined by the cluster administrators.

Each CSI storage driver will have a unique provisioner that is assigned as an attribute to a storage class and instructs any persistent volumes associated with that storage class to use the named provisioner, or CSI driver when provisioning the underlying volume on the storage platform.

Provisioning

Obtaining persistent storage for a pod is a three-step process:

  1. Define a PersistentVolume (PV), which is the disk space available for use
  2. Define a PersistentVolumeClaim (PVC), which claims usage of part or all of the PersistentVolume disk space
  3. Create a pod that references the PersistentVolumeClaim

In modern-day CSI drivers, the first two steps are usually combined into a single task and this is referred to as dynamic provisioning. Here the PersistentVolumeClaim is 100% if the PersistentVolume and the volume will be formatted with a filesystem on first attachment to a pod.

Manual provisioning can also be used with some CSI drivers to import existing volumes on storage devices into the control of Kubernetes by converting the existing volume into a PersistentVolume. In this case, the existing filesystem on the original volume is kept with all existing data when first mounted to the pod. An extension of this is the ability to import a snapshot of an existing volume, thereby creating a full read-write clone of the source volume the snapshot derived from.

When a PV is created it is assigned a storageClassName attribute and this class name controls many attributes of the PV as mentioned earlier. Note that the storageClassName attribute ensures the use of this volume to only the PVCs that request the equivalent StorageClass. In the case of dynamic provisioning, this is all managed automatically and the application only needs to call the required StorageClass the PVC wants storage from and the volume is created and then bound to a claim.

When the application is complete or is deleted, depending on the way the PV was initially created, the underlying volume construct can either be deleted or retained for use by another application, or a restart of the original application. This is controlled by the reclaimPolicy in the storageClass definition. In dynamic provisioning the normal setting for this is delete, meaning that when the PVC is deleted the associated PV is deleted and the underlying storage volume is also deleted. 

By setting the reclaimPolicy to retain this allows for manual reclamation of the PV.

On deletion of the PVC, the associated PV is not deleted and can be reused by another PVC with the same name as the original PVC. This is the only PVC that can access the PV and this concept is used a lot with StatefulSets.

It should be noted that when a PV is retained a subsequent deletion of the PV will result in the underlying storage volume NOT being deleted, so it is essential that a simple way to ensure orphaned volumes do not adversely affect your underlying storage platforms capacity.

At this point, I’d like to mention Pure Service Orchestrator eXplorer which is an Open Source project to provide a single plane of glass for storage and Kubernetes administrator to visualize how Pure Service Orchestrator, the CSI driver provided by Pure Storage, is utilising storage. One of the features of PSOX is its ability to identify orphaned volumes from a Kubernetes cluster.

Persistent Volume Granularity

There are a lot of options available when it comes to how the pod can access the persistent storage volume and these are controlled by Kubernetes. These different options are normally defined with a storageClass.

The most common of these is the accessMode which controls how the data in the PV can be accessed and modified. There are three modes available in Kubernetes:

  • ReadWriteMany (RWX) – the volume can be mounted as read-write by many nodes
  • ReadWriteOnce (RWO) – the volume can be mounted as read-write by a single node
  • ReadOnlyMany (ROX) – the volume can be mounted read-only by many nodes

Additional controls for the underlying storage volume can be provided through the storageClass include mount options, volume expansion, binding mode which is usually used in conjunction with storage topology (also managed through the storageClass). 

A storageClass can also apply specific, non-standard, granularity for different features a CSI driver can support.

In the case of Pure Service Orchestrator, all of the above-mentioned options are available to an administrator creating storage classes, plus a number of the non-standard features.

Here is an example of a storageClass definition configured to use Pure Service Orchestrator as the CSI provisioner:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: example
provisioner: pure-csi
parameters:
  iops_limit: "30000"
  bandwidth_limit: "10G"
  backend: block
  csi.storage.k8s.io/fstype: xfs
  createoptions: -q
mountOptions:
  - discard
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.purestorage.com/rack
        values:
          - rack-0
          - rack-1
allowedVolumeExpansion: true

This might look a little complex, but simplistically this example ensures that PersistentVolumes created through this storageClass will have the following attributes:

  • Quality of Service limits of 10Gb/s bandwidth and 30k IOPs
  • Volumes are capable of being expended in size
  • One first use by a pod the volume will be formatted with the xfs filesystem and mounted with the discard flag
  • The volume will only be created by an underlying FlashArray found in either rack-0 or rack-1 (based on labels defined in the PSO configuration file)

Pure Service Orchestrator even allows the parameters setting to control the NFS ExportRules of PersistentVolumes created on a FlashBlade.

Check back for Part 3 of this series, where I’ll discuss the latest developments in ephemeral storage in Kubernetes.

Ephemeral or Persistent? The Storage Choices for Containers (Part 1)

In this series of posts, I’ll cover the difference between ephemeral and persistent storage as far as Kubernetes containers are concerned and discuss the latest developments in ephemeral storage. I’ll also occasionally mention Pure Service Orchestrator™ to show how this can provide storage to your applications do matter what type is required.

Back in the mists of time when Kubernetes and containers, in general, were young storage was only ephemeral. There was no concept of persistency for your storage and the applications running in container environments were inherently ephemeral themselves and therefore there was no need for data persistency.

Initially, with the development of FlexDriver plugins and lately CSI compliant drivers, persistent storage has become a mainstream offering to enable applications that need or require state for their data. Persistent storage will be covered in the second blog in this series.

Ephemeral Storage

Ephemeral storage can come from several different locations, the most popular and simplest being emptyDir. This is, as the name implies, an empty directory mounted in the container that can be accessed by one or more pods in the container. When the container terminates, whether that be cleanly or through a failure event, the mounted emptyDir storage is erased and all its contents are lost forever. 

emptyDir

You might wonder where this “storage” used by emptyDir comes from and that is a great question. It can come from one of two places. The most common is actually from the actual physical storage available to the Kubernetes nodes running the container, usually from the root partition. This space is finite and completely dependent on the available free capacity of the disk partition the directory is present on. This partition is also used for lots of other dynamic data, such as container logs, image layers, and container-writable layers, so it is potentially an ever-decreasing resource.

To create this type of ephemeral storage for a pod(s) running in a container, ensure the pod specification has the following section:

 volumes:
  - name: demo-volume
    emptyDir: {}

Note that the {} states that we are not providing any further requirements for the ephemeral volume. The name parameter is required so that pods can mount the emptyDir volume, like this:

   volumeMounts:
    - mountPath: /demo
      name: demo-volume

If multiple pods are running in the container they can all access the same emptyDir if they mount the same volume name.

From the pods perspective, the emptyDir is a real filesystem mapped to the root partition, which is already part utilised, so you will see it in a df command, executed in the pod, as follows (this example has the pod running on a Red Hat CoreOS worker node):

# df -h /demo
Filesystem                Size      Used Available Use% Mounted on
/dev/mapper/coreos-luks-root-nocrypt
                        119.5G     28.3G     91.2G  24% /demo

If you want to limit the size of your ephemeral storage this can be achieved by adding resource limits to the container in the pod as follows:

      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi"

Here the container has requested 2GiB of local ephemeral storage, but the container has a limit of 4GiB of local ephemeral storage.

Note that if you use this method and you exceed the ephemeral-storage limits value the Kubernetes eviction manager will evict the pod, so this is a very aggressive space limit enforcement method.

emptyDir from RAM

There might be instances that you only need a minimal scratch space area for your emptyDir and you don’t want to use any of the root partition. In this case, resources permitting, you can create this in RAM. The only difference in the creation of the emptyDir is that more information is passed during its creation in the pod specification as follows:

 volumes:
  - name: demo-volume
    emptyDir:
      medium: Memory

In this case, the default size of the mounted directory is half of the RAM the running node has and is mounted on tmpfs. For example, here the worker node has just under 32GB of RAM and therefore the emptyDir is 15.7GB, about half:

# df -h /demo
Filesystem                Size      Used Available Use% Mounted on
tmpfs                    15.7G         0     15.7G   0% /demo

You can use the concept of sizeLimit for the RAM-based emptyDir but this does not work as you would expect (at the time of writing). In this case, the sizeLimit is used by the Kubernetes eviction manager to evict any pods that exceed the sizeLimit specified in the emptyDir

Check back for Part 2 of this series, where I’ll discuss persistent storage in Kubernetes.

Migrate Persistent Data into PKS with Pure vVols

While I discussed in my VMworld session this week some of the architectural decisions to be made while deploying PKS on vSphere my demo revolved around once it is up and running how to move existing data into PKS.

First, using the Pure FlashArray and vVols we are able to automate that process and quickly move data from another k8s cluster into PKS. It is not limited to that but this is the use case I started with.

Part 1 of the demo shows taking the persistent data from a deployment on and cloning it over the vVol that is created by using the vSphere Cloud Provider with PKS. vVols are particularly important because they keep the data in a native format and make copy/replication and snapshotting much easier.

Part 2 is the same process just scripted using Python and Ansible.

Demo Part 1 – Manual process of migrating data into PKS

Demo Part 2 – Using Python and Ansible to migrate data into PKS

How to automate the Migration with some Python and Ansible

The code I used is available from code.purestorage.com. Which also links to the GitHub repo https://github.com/PureStorage-OpenConnect/k8s4vvols

They let me on a stage. Again. 🙂

UNMAP – Do IT!

Pretty sure my friend Cody Hosterman has talked about this until he turned blue in the face.  Just a point I want to quickly re-iterate here for the record. Run unmap on your vSphere Datastores.

Read this if you are running Pure Storage, but even if you run other arrays (especially all-flash) find a way to do UNMAP on a regular basis:

http://www.codyhosterman.com/2016/01/flasharray-unmap-script-with-the-pure-storage-powershell-sdk-and-poweractions/

Additionally, start to learn the ins-n-outs of vSphere 6 and automatic unmap!

http://blog.purestorage.com/direct-guest-os-unmap-in-vsphere-6-0-2/

Speaking of In-n-out…. I want a double double before I start Whole 30.

in-n-out

Pure ELK Dashboards

Previously I blogged about getting PureELK installed with Docker in just a couple of minutes. After setting up your intitial array’s you may ask what is next?

Loading a preconfigured PureELK Dashboard

media_1450716163834.png

Click load saved dashboard and select one of the Pure Storage dashboards.

media_1452264626343.png

Remember there is a 2nd page page of Dashboards.

Pure Main Dashboard

media_1452264823321.png

Top 10 Volumes

media_1452264957200.png

Alert Audit

media_1452265011249.png

Max vs Average Pure Performance

media_1452265066174.png

Pure Space Analysis

media_1452265121910.png

Space Top – Bar Charts

media_1452265175795.png

Volume List View – Space and Performance

media_1452265229517.png

You can see there are several pre-made dashboards that you can take advantage of. What if you wanted to make your own Dashboard.

Create your Own Dashboard

media_1452265517171.png

To create your own Dashboard:
1. Click the Plus to Add a Visualization
2. Select a visualization
3. Once you have all of your Visualizations added you can click the Save icon and keep your new Dashboard for later use.

Some tips is you can resize and place the visualization anywhere you like on the dashboard. Just remember to click save. Also, You can can use the powerful seach feature to create tables of useful information that you are looking for.

 

Easy Storage Monitoring – Setting Up PureELK with Docker

[UPDATE June 2016: Appears this works with Ubuntu only, maybe a debian flavor. I am hearing RHEL is problematic to get the dependencies working.]

I have blogged in the past about setting up vROPS (vCOPS) and Splunk to monitor a Pure Storage FlashArray using the REST API. Scripts and GETs and PUTs are fun and all but what if there was a simple tool you can install to have your own on site monitoring and analytics of your FlashArrays?

Enter Pure ELK. Some super awesome engineers back in Mountain View wrote this integration for Pure and ELK and packaged it an amazingly easy insatllation and released it on Github! Open Source and ready to go!
https://github.com/pureelk

and

http://github.com/pureelk/pureelk

Don’t know Docker? Cool we will install it for you. Don’t know Kibana or elasticsearch? Got you covered. One line in a fresh Ubuntu install (I used Ubuntu but I bet your favorite flavor will suffice).

go ahead and try:

curl -s https://raw.githubusercontent.com/pureelk/pureelk/master/pureelk.sh | bash -s install

(fixed url to reflect no longer in Dev)

This will download and install docker, setup all the dependencies for Pure ELK and let you know where to go from your browser to config your FlashArrays.

I had one small snag:

Connecting to the Docker Daemon!

media_1450716022076.png

My user was not in the right group to connect to docker the first time. The Docker install when it is not automated actually tells you to add your user to the “docker” group in order to

$sudo usermod -aG docker [username]

Log out and back in that did the trick. If you know a better way for the change to be recognized without logging out let me know in the comments.

I re-ran the install
curl -s https://raw.githubusercontent.com/pureelk/pureelk/dev/pureelk.sh | bash -s install

In about 4 minutes I was able to hit the management IP and start adding FlashArrays!

Quickly add all your FlashArrays

media_1450715719804.png

Click the giant orange PLUS button.

This is great if you have more than one FlashArray. If you only have one it still works. Everyone should have more Flash though right?

media_1450715771293.png

Fill in your FlashArray information. You can choose your time-to-live for the metrics and how often to pull data from the FlashArray

Success!

media_1450715937834.png

I added a couple of arrays for fun and then clicked “Go to Kibana”
I could have gone to
https://[server ip]:5601

Data Already Collecting

media_1450716109188.png

This is just the beginning. The next post I will share some of the pre-packaged dashboards and also some of the cusotmizations you can make in order to visualize all the Data PureELK is pulling from the REST API. Have fun with this free tool. It can be downloaded and setup in less than 10 minutes on a linux machine, 15 minutes if you need to build a new VM.

Top 5 – Pure Storage Technical Blog Posts 2014

Today I thought it would be pretty cool to list out my favorite 5 technical blog posts that pertain to Pure Storage. These are posts that I use to show customers how to get things done without re-inventing the wheel. Big thanks to Barkz and Cody for all the hard work they put in this year. Looking forward to even more awesomeness this year.

SQL Server 2014 Prod/Dev with VMware PowerCLI and Pure Storage PowerShell Toolkit – Rob “Barkz” Barker

Enhanced UNMAP script using with PowerCLI and RESTful API – Cody Hosterman

VMware PowerCLI  and Pure Storage – Cody Hosterman
Check out the great script to set all the vSphere Best Practices for the Pure Storage Flash Array.

Pure Storage PowerShell Toolkit Enhancements – Rob “Barkz” Barker

PowerActions – The PowerCLI Plugin for the vSphere Web Client with UNMAP – Cody Hosterman

JO-Unicorn-Rainbow

Not the Same Ol’ Sessions from Pure Storage at VMworld

I am really excited to be going to VMworld once again. I will be wearing my Orange Nike so most likely my feet won’t hurt quite as bad. Also expect the Pure Orange Superman to make an appearance.
IMG_2992
More about the sessions. So I will be attending VMworld San Francisco, and speaking in EMEA.

STO2996-SPO – The vExpert Storage Game Show

The session I am stoked to be a part of is STO2996-SPO – The vExpert Storage Game Show. It will be a fun and informative time about next generation storage architectures presented in the form of a game show.  PLUS,  two members of the audience will join the session to help the vExpert teams. I know everyone will want to be on my team in EMEA.

STO3000-SPO – Flash Storage Best Practices and Technology Preview 

This very exciting session with Vaughn and Cody (super-genius vExperts) will go into what to consider when moving your datacenter to all flash. Plus previews of the Pure VVOLs.  If you think you are not ready for all flash, come to this session and learn how Flashy you can be.

STO2999-SPO – Customers Unplugged: Real-World Results with VMware on Flash

I wish I had thought of this. Customers using All Flash with VMware. All Tech, No Slides.

STO1965 – Virtual Volumes Technical Deep Dive

Dive into Virtual Volumes with Rawlinson Rivera – VMware, Suzy Visvanathan – VMware and Vaughn Stewart – Pure Storage. So many customers have asked me what will VVOLS actually do over the last 3 years. This will be a great chance to find that out.

VAPP2132 – Virtualizing Mission Critical Applications on All Flash Storage 

How does Pure storage enable that final 10% of critical applications that just a few years ago people said would be impossible? Meet my friend Avi Nayek from Pure and Mohan Potheri from VMware and learn how flash eliminates storage as the road block to critical applications becoming virtual.

MGT1265 – Improving Cloud Operations Visibility with Log Management and vCenter Log Insight

Cody Hosterman, Did I tell you he is smart? Yeah. He is. Join Cody and Dominic Rivera from US Bank and Bill Roth from VMware on how to increase your Cloud Operations Visibility.

SDDC2754-SPO – New Kids on the Storage Block, File and Share: Lessons in Storage and Virtualization

Lessons from all the upstarts in the storage industry. Most of them are not “startups” anymore. Finding new ways to solve the issues of using Virtualization with legacy storage. Pure Storage, Nimble Storage, Tintri, Tegile, Coho Data, Data Gravity and moderated by Howard Marks from DeepStorage.net.

STO2496-SPO – vSphere Storage Best Practices: Next-Gen Storage Technologies

The Chad and Vaughn show. Now with Rawlinson Rivera! Storage is changing. Did I say that yet?

More information on Pure Storage Sessions

Coming Soon: Support for VMware VVOLs
Pure Storage set to paint VMworld 2014 orange!

I am now at Pure Storage

Thought that after 2 weeks I would put it on my blog. It is long past official as I have already done “New Hire” and I am officially part of the Puritan family. My Orange pants are on order. One thing I am excited about is getting to install the array for my customers. Not just talking about how awesome it is but getting to see it. This should definitely inspire blog posts to share what I learn along the way.
I know many people probably already knew this, but someday I would like my blog to be a FLASH of the progression through my career.