network | CrashLoopBackoff

Finishing some setup for my Unifi Dream Machine Pro

I wanted a better home router. During the learning from home phase of the 2020 pandemic I learned I could not have advanced security features of the USG (Unifi Security Gateway) turned on and get sufficient bandwidth for 3 Kids and myself to stream and zoom. So I wanted an upgrade. I went with the Unifi Dream Machine Pro.
https://store.ui.com/collections/unifi-network-routing-switching/products/udm-pro
For reals though file this post under, I need to remember what I changed in case I have to do it again.

OpenDNS

First thing that I did on my older routers was to configure opendns as the external DNS for my networks. In order for OpenDNS so apply my content filtering settings it must know the source IP for my home. This can change because most ISP’s use DHCP to assign the IP’s. Although it seems that my ISP likes to reassign the same IP, I can’t trust that will always be true.

So first, make sure you sign up for an opendns and dns-o-matic account.

Log into the UDM UI

Click on the Settings Gear…

Click on Advanced Features -> Advanced Gateway Settings

Click Create new Dynamic DNS

For DNS-o-matic the settings look like:
Hostname: all.dnsomatic.com
Username: [Your DNS-o-matic user]
Password: [Your DNS-o-matic password]
Server: updates.dnsomatic.com/\/nic/update?hostname=%h&myip=%i

Links Below were very helpful

https://community.ui.com/questions/OpenDNS-not-working-with-UDM-Pro/c9d5589b-c14e-4c86-8470-4c228b0b5282

Very helpful link for getting the server URL. Also contains a few for some other services.
https://community.ui.com/questions/UDM-DynDNS-Google-Domains/fe9ba35d-66c3-437d-8323-debe2af55879#answer/2181146e-79b8-485c-8042-eb975c291242

https://community.ui.com/questions/Any-way-to-get-DNS-O-Matic-to-work-with-UDM-Pro-to-enable-OpenDNS-Home-with-dynamic-IP/ede30618-663c-43e0-b198-0f2cf2805e1d

DDClient

Another thing I want to do, is set a DNS A record. I could probably use some form of the settings above to inform my Google Name Service to update the record with the dynamic IP. But why be boring? Lets run the DDClient perl program in a container on my K3s cluster.

First, read the google domains documentation for dynamic records. I created a dynamic record and it generates the host record along with a username and password that can be used via the API to update the IP associated to the Domain Name.

Next, why create the container if I don’t need to?

https://hub.docker.com/r/linuxserver/ddclient/tags?page=1&ordering=last_updated
My k3s is on some Raspberry Pi’s so I choose the arm image.

Then another nice person built the deployment. Check out that blog for full detail. Without getting too distracted by kubesail and setting up k8s. I skipped to the YAML:
https://kubesail.com/template/loopDelicious/ddclient

Save this as ddclient-secret.yaml changing the info necessary for your google account.

apiVersion: v1
kind: Secret
metadata:
  name: ddclient-secret
  labels:
    app: ddclient
stringData:
  ddclient.conf: |
    daemon=300
    syslog=yes
    protocol=dyndns2
    use=web
    server=domains.google.com
    ssl=yes
    login=<google generated login>
    password=<google generated password> 
    your.domain.record.com

Now save this as ddclient.yaml, remember to modify the image for the type of arch your Kubernetes is running on.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ddclient-deployment
  labels:
    app: ddclient
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  replicas: 1
  selector:
    matchLabels:
      app: ddclient
  template:
    metadata:
      labels:
        app: ddclient
    spec:
      volumes:
        - name: ddclient-config-file
          secret:
            secretName: ddclient-secret
      containers:
        - name: ddclient
          image: linuxserver/ddclient:arm64v8-version-v3.9.1
          imagePullPolicy: Always
          volumeMounts:
            - mountPath: /config
              name: ddclient-config-file
          resources:
            requests:
              cpu: 10m
              memory: 64Mi
            limits:
              cpu: 50m
              memory: 128Mi

This deployment will use the secret for the settings and deploy the small container to update the Google Domain record with the new IP from the host.

kubectl create ns ddclient
kubectl -n ddclient apply -f ddclient-secret.yaml
kubectl -n ddclient apply -f ddclient.yaml

Some DNS stuff I might try later

This is interesting repo that updates dns with static hostnames. Unfortunately the UDM does not have this built in. I would suggest Ubiquiti to build pi-hole into the UDM Pro to integrate with its DHCP server and also provide some abilities to block bad DNS names for ads/phishing/malware.

Some NSX-T Things I learned

The last few months I have done a lot of work with NSX-T. I have not done so much networking since my CCNA days. I wanted to share a couple of things that were really helpful I found out on the web.

I was using NSX-T 2.4.2 and sometimes some troubleshooting guides were not very helpful as they were very specific to other versions.

This was great for removing some things I broke.
Deleting NSX-T Objects

Since the end goal with NSX was PKS this is where I spent time over and over.
https://docs.pivotal.io/pks/1-5/nsxt-prepare-env.html

After everything worked installing PKS I got this error when deploying a PKS cluster.
The deployment was failing and I kept seeing references to: pks-nsx-t-osb-proxy
https://community.pivotal.io/s/article/pks-nsx-t-osb-proxy-job-fails-while-installing-PKS
https://cormachogan.com/2019/02/05/reviewing-pks-logs-and-status/
https://community.pivotal.io/s/question/0D50P00003xMeUBSA0/installing-pivotal-container-service-failed-with-pksnsxtosbproxy

Some helpful information in those few links. Main thing is when you create certificates for NSX-T Manager, you should apply them too.

Also, make sure the NIC’s on your ESXi hosts are all setup the same way. I had 4 nics and 4 different VLAN/Trunk configs, no bueno. Also as VXLAN wants the frames to be at least 1600 MTU. I set everything to 9000 just for fun. That worked much better.

Lead the Transfomation

Speaking with customers everyday the most common thing I see the infrastructure teams struggle with is how do we get from X to Z. We are virtualizing first. Evaluating tier 1 apps as VM’s. Migrating non-essential services to the cloud. As an overall strategy how do I get from what I have, to where I want to go?
While there are many topics to get you going on this path, from management and orchestration to improved monitoring and security. One thing we as infrastructure guys often forget to ask is, “Are our applications ready for the future?” Many of the off the shelf applications are just fine for many of our use cases today will they still be viable in 5-10 years? Can we take a design that was created from a physical silo, virtualize it, and hope to be cloud ready? Maybe. How can we think in a new way about our applications?
Currently we take our application and think of it this way:

It consumes parts of these buckets. Physical or Virtual, the application is bound by the contraints of a general purpose OS accessing some sort of physical resources that are bound a physical RU in a datacenter.
So even as we look to build out like this:

We take the same solutions that we used in the physical world in order to provide scale and high availability. Mainly clustering. Does clustering provide actually cloud enabled applications? Most likely not. We look to the new bubble of dot com innovators for solutions to the boxes the old guard of application vendors have locked us in. I am not going negative on any current application but rather trying to challenge us to think beyond the way we have always done things.
So if we want to move towards a new model, public, private or hybrid cloud. It would be in the best interest of the infrastructure teams to lead the charge and provide thought leadership when moving applications to a cloud. I would argue in the near future you do not want to be the one that is seemingly hugging your infrastructure. It is always better to be leading the change than
roadblocking it, especially when the change will drive the business to the next level of service capabilities.

So a few questions to start asking:

What does my data actually look like?

How does the business use the rows and columns to function?

Are all application decisions made in a room walled in by current capabilites?

How are we going to deliver applications with a new model? (think mobile)

At what point do we need to expand the foundation of what we are built on in order to increase our effectiveness in the market?

In 5 years do I want to be working on plumbing of the datacenter or have enough agility and scalability built in where I can drive innovation rather than daily maintenance?

vSphere Metro Stretched Clusters – Some Info/Links

A lot of questions lately about vSphere Clusters across distance. I really need to learn for myself so I collected some good links.

Make sure you understand what “Only Non-uniform host access configuration is supported” means. Someone correct me if I have this wrong but your device that enables the distributed virtual storage needs to be sure that hosts in site A are writing to their preferred volumes in site A and vice versa in Site B. Probably way over simplifying it.

LINKS

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007545

http://virtualgeek.typepad.com/virtual_geek/2011/10/new-vmware-hcl-category-vsphere-metro-stretched-cluster.html

http://www.yellow-bricks.com/2011/10/07/vsphere-metro-storage-cluster-solutions-what-is-supported-and-what-not/

http://www.yellow-bricks.com/2011/10/05/vsphere-5-0-ha-and-metro-stretched-cluster-solutions/

Big thanks to Scott Lowe for clearing the details on this topic.

Trunks – Dell Power Connect and Cisco

I recently needed to install a stack of Dell 6224 Power Connect switches. The core of the network was actually a Cisco 3560 (no G). While there are already posts existing from Scott Lowe about using the “General” mode to keep VLAN 1 untagged and also have other VLAN’s tagged. Dell’s General mode traditionally works just like a default dot1q trunk in Cisco. However when VLAN 1 is in use I secretly grumble because I know the fact that Dell’s general mode is finicky when interoperating with some devices. Most of the time general mode works like a charm but not on this day.

Dell’s “trunk” mode worked fine. Any tagged VLAN would pass fine to the Cisco. Except that pesky native VLAN 1. We HAD to have VLAN 1 passed down to the ESX servers. So after kicking around wondering what I did wrong I decided to just work around the problem. I tagged vlan 1 on the Dell port and changed the native vlan on that specific trunk on the Cisco to another vlan (not being used on the Dell). BAM it worked.

Note: Dell was running their newest firmware on that day – 3.2.0.9 (they have since released 3.2.0.10)
Note 2: I am all about auto-negotiation at Gigabit but still like 100Mbps switch links to be hard coded.

Cisco 3560 (no G).

interface FastEthernet 0/24
speed 100
duplex Full
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,10,11
swtichport trunk native vlan 8
switchport mode trunk

Dell 6224

interface Ethernet 1/g24
no negotiation
speed 100
duplex full
switchport mode trunk
switchport trunk allowed vlan add 1,10,11

Firewalls are not Routers

I am no network super-genius but I do enough with networking to be able to get by. Two common mistakes I find many times are flat networks and firewalls as the default gateway. A flat network is when generally switches are connected to one another without any configuration. There is one broadcast domain which means every packet that the switch does not have an entry in the MAC address table, is sent out all the ports but the originating port. This repeats across all of the switches until the layer 2 destination is found. Now, this means your expensive Cisco switches are barely better than hubs. You don’t have collisions like you would on a hub and once the switch learns where the MAC address lives it keeps that information for a certain amount of time. Then again in this network setup the logs are most likely not monitored so if there where collisions and other errors it goes unnoticed.
That is not the title of this post though. Although related to a flat network using the firewall is a different issue. Using the firewall as the router works just fine when you have a flat network. You may never notice the problem in a small network, but as your network grew you noticed how problems can come up when there is just one big network. So someone smart said use vlans to segment the network, create smaller broadcast domains. Then when you try to fix or change the flat network with subnets and vlans can you find out the new vlans can not reach the rest of the original network.

The current flat network with switches and the firewall used as the default gateway or router.

The problem comes when you add subnets that are different than the interface ip of the firewall. Firewalls in general have issue with redirecting traffic bound for other networks back out of the same interface. So in the picture above traffic from vlan 1 that is using the firewall as the default gateway trying to reach the subnet on vlan 10. Since the host on vlan 1 does not know where that network lives it sends the traffic to the default gateway. Even if you added a static route to the firewall the traffic will often fail. That is because firewalls are not meant to route but rather send traffic between trusted and untrusted networks and vice-versa. So the question becomes how do you actually fix your flat network that has the firewall as the router. There is of course more complicated solutions to provide high availability using VRRP or HSRP.
First get a real layer 3 device. That is a router or a switch capable of routing between multiple vlans. The good news is many of your newer switches are capable of layer 3, it is included in many Dell and HP switches, it may still be an add-on with Cisco. I haven’t used a new switch in the last year that did not have layer 3.
Next important step is use the layer 3 device (switch or router) to route everything. Set a default route in the layer 3 device to send only outbound traffic to the firewall and bam everything works. Why is this so hard. Many times there is hundreds of servers and desktops already configured to use the firewall as their router. We will do a lot of work to avoid having to do a bunch of manual work.

Now you are using a router to route and the firewall to block bad things and maybe even do NAT. (note: If you are doing NAT be sure to add your new VLANs to your NAT rules so the new networks can reach the outside of your firewall.)

Ask Good Questions

This happened a long time ago. I arrived at a customer site to install View Desktop Manager (may have been version 2). This was before any cool VDI sizing tools like Liquidware Labs. I am installing ESX and VDM I casually ask, “What apps will you be running on this install?” The answer was, “Oh, web apps like youtube, flash and some shockwave stuff.” I thought “ah dang” in my best Mater voice. This was a case of two different organizations thinking someone else had gathered the proper information. Important details sometimes fall through the cracks. Since that day, I try to at least uncover most of this stuff before I show up on site.

Even though we have great assessment tools now, remember to ask some questions and get to know what is your customers end goal.

Things I learned that day. As related to VDI.

1. Know what your client is doing, “What apps are you going to use?”

2. Know where your client wants to do that thing from, “So, what kind of connection do you have to that remote office with 100+ users?”

This is not the full list of questions I would ask, just some I learned along the way.

VMware View and Xsigo

*Disclaimer – I work for a Xsigo and VMware partner.

I was in the VMware View Design and Best practices class a couple weeks ago. Much of the class is built on the VMware View Reference Architecture. The picture below is from that PDF.

It really struck me how many IO connections (Network or Storage) it would take to run this POD. Minimum (in my opinion) would be 6 cables per host with ten 8 host clusters that is 480 cables! Let’s say that 160 of those are 4 gb Fiberchannel and the other 320 are 1 gb ethernet. The is 640 gb for storage and 320 for network.

Xsigo currently uses 20 gb infiniband and best practice would be to use 2 cards per server. The same 80 servers in the above cluster would have 3200 gb of bandwidth available. Add in the flexibility and ease of management you get using virtual IO. The cost savings in the number director class fiber switches and datacenter switches you no longer need and the ROI I would think the pays for the Xsigo Directors. I don’t deal with pricing so this is pure contemplation. So I will stick with the technical benefits. Being in the datacenter I like any solution that makes provisioning servers easier, takes less cabling, and gives me unbelievable bandwidth.

So just in the way VMware changed the way we think about the datacenter. Virtual IO will once again change how we deal with our deployments.

ESX Commands: esxcfg-vswif

The esxcfg-vswif command allows you to create and modify Service Console ports and their IP information. Many times I have to change stuff after the install process is complete and the only place is via the direct service console because network communication is not possible. This usually happens when the network team changes a vlan in the middle of the install or they change a subnet. Not to disparage network teams many times I am the network team and the virtualization team.
Create a new vswif:
#first add a port group with esxcfg-vswitch esxcfg-vswitch -A "Service Console Test" vSwitch-Test #then use esxcfg-vswif to create a new vswif esxcfg-vswif -a -i 172.16.50.40 -n 255.255.255.0 -p "Service Console Test" vswif1 #List your vswifs esxcfg-vswif - l #Example: [root@esx3 root]# esxcfg-vswif -l Name Port Group IP Address Netmask Broadcast Enabled DHCP vswif0 Service Console 172.16.50.50 255.255.255.0 172.16.50.255 true false vswif1 Service Console Test172.16.50.40 255.255.255.0 172.16.50.255 true false

Modify your Service Console network information:
esxcfg-vswif -i 172.16.50.41 -n 255.255.255.0 vswif1 #example [root@esx3 root]# esxcfg-vswif -i 172.16.50.41 -n 255.255.255.0 vswif1 Setting IP config Nothing to flush. [root@esx3 root]# esxcfg-vswif -l Name Port Group IP Address Netmask Broadcast Enabled DHCP vswif0 Service Console 172.16.50.50 255.255.255.0 172.16.50.255 true false vswif1 Service Console Test172.16.50.41 255.255.255.0 172.16.50.255 true false

ESX Commands – esxcfg-firewall

I have really forgot to keep up on my VCDX study path. So today a quick tidbit on the esxcfg-firewall command.
Many of us today will use the vCenter Client to change firewall ports on the ESX. One instance where I exclusively mess with the firewall from the command line using esxcfg-firewall is when I install Dell OpenManage. I am already in the console to install the agents so I might as well open the firewall from the console too.
This really applies to any kind of agent or software you add to your ESX installation. So if you find yourself already in the console why not save a step and do it from the cli?

Lets look at the command

# esxcfg-firewall -o 1311,tcp,in,OpenManageRequest

First is the command, esxcfg-firewall, -o is for openport, the 1311 is the port number, tcp is protocol, in is the direction and the final part is the name of the service.

Now if you want to see all of your esxcfg-firewall settings try:
esxcfg-firewall -q

Show if specifig service is enabled.
esxcfg-firewall -q [service name]

Of course typing esxcfg-firewall -h gives lots of good help.

Some links: (You can google and find a ton more)

ESX Guide
VMware Land
Yellow Bricks
Vritualization Admin