13 March 2019

Data Domain Mtree cannot be deleted


Sometimes Data Domain Mtree cannot be deleted i.e. due to lack of proper license,

This can be caused if you replicate VTL Mtree to Data Domain without VTL license and you can get error as below:

mtree delete /data/col1/VTL_mtree
**** MTree "/data/col1/VTL_mtree" contains a VTL Pool, cannot be deleted.

There is workaround to to resolve the issue:
1. Create empty mtree.
2. Run fastcopy to overwrite the VTL mtree with an empty mtree.
3. Restart file system.
4. Delete the VLT mtree and the empty mtree.


Example command list:

mtree create /data/col1/empty_mtree
mtree list
filesys fastcopy source /data/col1/empty_mtree destination /data/col1/VTL_mtree
mtree list
filesys restart
mtree delete /data/col1/empty_mtree
mtree delete /data/col1/VTL_mtree


Example output:

sysadmin@dd-01# mtree create /data/col1/empty_mtree
MTree "/data/col1/empty_mtree" created successfully.
Quota soft limit: none, hard limit: none(*)
 (*) Quota is disabled. Quota limits not enforced.


sysadmin@dd-01# mtree list

Name                                 Pre-Comp (GiB)   Status
----------------------------        --------------         ------
/data/col1/VTL_mtree              5860.0             RW
/data/col1/backup                           0.0             RW
/data/col1/empty_mtree                  0.0             RW
----------------------------         --------------        ------
 D    : Deleted
 Q    : Quota Defined
 RO   : Read Only
 RW   : Read Write
 RD   : Replication Destination
 RLGE : Retention-Lock Governance Enabled
 RLGD : Retention-Lock Governance Disabled
 RLCE : Retention-Lock Compliance Enabled


sysadmin@dd-01# filesys fastcopy source /data/col1/empty_mtree destination /data/col1/VTL_mtree


Destination "/data/col1/VTL_mtree" already exists.

Proceeding will overwrite its content with "/data/col1/empty_mtree".
        Are you sure? (yes|no|?) [no]: yes

ok, proceeding.


Creating snapshot "FASTCOPY-2019-03-13-04-28-17" with one-hour retention period...done

Use this snapshot to recover in case of a mistake.

(00:00) Waiting for fastcopy to complete...

Fastcopy status: fastcopy /data/col1/empty_mtree to /data/col1/VTL_mtree: deleted 1046 files, 1 directory in 1.05 seconds


sysadmin@dd-01# mtree list

Name                              Pre-Comp (GiB)   Status
----------------------------   --------------           ------
/data/col1/VTL_mtree                      0.0        RW
/data/col1/backup                              0.0       RW
/data/col1/empty_mtree                    0.0        RW
----------------------------   --------------           ------
 D    : Deleted
 Q    : Quota Defined
 RO   : Read Only
 RW   : Read Write
 RD   : Replication Destination
 RLGE : Retention-Lock Governance Enabled
 RLGD : Retention-Lock Governance Disabled
 RLCE : Retention-Lock Compliance Enabled


sysadmin@dd-01# filesys restart

This action will restart the file system.
Applications may experience interruptions
while the file system is restarted.
        Are you sure? (yes|no) [no]: yes

ok, proceeding.

Disabling filesystem:
Please wait..............
The filesystem is now disabled.
Enabling filesystem:
Please wait...............

The filesystem is now enabled.

sysadmin@dd-01# mtree delete /data/col1/empty_mtree

Deleting an MTree will destroy all data in the MTree.
If there are any NFS exports, CIFS shares, VTL tapes, DDBOOST storage-units, Replication Contexts or Physical Capacity Measurements configured for this MTree, they need to be manually removed/edited. Do you wish to continue?
        Are you sure? (yes|no) [no]: yes

ok, proceeding.

MTree "/data/col1/empty_mtree" deleted successfully.


sysadmin@dd-01# mtree delete /data/col1/VTL_mtree

Deleting an MTree will destroy all data in the MTree.
If there are any NFS exports, CIFS shares, VTL tapes, DDBOOST storage-units, Replication Contexts or Physical Capacity Measurements configured for this MTree, they need to be manually removed/edited. Do you wish to continue?
        Are you sure? (yes|no) [no]: yes

ok, proceeding.

MTree "/data/col1/VTL_mtree" deleted successfully.


sysadmin@dd-01# mtree list
Name                           Pre-Comp (GiB)   Status
----------------------------   --------------        ------
/data/col1/VTL_mtree                      0.0      D
/data/col1/backup                              0.0     RW
/data/col1/empty_mtree                    0.0     D
----------------------------   --------------        ------
 D    : Deleted
 Q    : Quota Defined
 RO   : Read Only
 RW   : Read Write
 RD   : Replication Destination
 RLGE : Retention-Lock Governance Enabled
 RLGD : Retention-Lock Governance Disabled
 RLCE : Retention-Lock Compliance Enabled

12 March 2019

Data Domain - suddenly monitoring shows disk unknown


Sometimes when you monitor data domain your monitoring software can suddenly report disk as "UNKNOWN" however data domain works as normal.
This is caused by data domain management services misbehavior.
To resolve it you should run below commands:

1. SSH to your data domain

2. Enter SE mode:

system show serialno
(make note of the serial number)
priv set se
(use the serial number as the password)

3. Run the following:

reg set config.crontab.diskinfo_reset = "*/5 * * * * root /bin/killall -9 ssm"

4. Wait for at least 10 minutes and after that time run this:

reg removekey config.crontab.diskinfo_reset

28 February 2019

Avamar - Enable Client Migration Tool

Sometimes you have to migrate clients/ backups to new Avamar grid. To enable migrate on source grid follow below procedure.



On source avamar server make below changes

admin@grid1:~/>: vi /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

find "migrate"

add line: <entry key="migrate_feature_enabled" value="true" />

example of migrate section

 <node name="migrate">
            <map>
              <entry key="migrate_feature_enabled" value="true" />
              <entry key="src_driven_migrate" value="true" />
              <entry key="migrate_enabled" value="true" />
              <entry key="clients_only_in_workorders" value="false" />
              <entry key="secured_port_offset" value="2000" />
              <entry key="migrate_sched_priority" value="0" />
              <entry key="num_members_per_workorder" value="-1" />
              <entry key="internal_accounts" value="AVI_BACKUPS,EM_BACKUPS,MC_BACKUPS,EBR_BACKUPS" />
              <entry key="encryption" value="high" />
              <entry key="base_port" value="27000" />
              <entry key="internal_account_repl_window_sec" value="300" />
              <entry key="migrate_window_hours" value="24" />
              <entry key="parent_cancel_job_priority" value="6000" />
              <entry key="migrate_adhoc_priority" value="5000" />
            </map>
          </node>

restart mcs

admin@grid1:~/>: mcserver.sh --restart

start scheduler

admin@grid1:~/>: dpnctl start sched




23 February 2019

How to decript Avamar passwords

If you forgor all your Avamar software password you can follow below steps to recover them.
This do not apply to Avamar OS passwords like root & admin.

Decript Avamar passwords

/usr/local/avamar/bin/mccipher decrypt --all
grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

root@grid-01:~/#: grep AP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="backuprestoreAP" value="backuprestore1" />
                <entry key="backuponlyAP" value="backuponly1" />
                <entry key="MCUSERAP" value="MCUser1" />
                <entry key="replonlyAP" value="replonly1" />
                <entry key="restoreonlyAP" value="restoreonly1" />
                <entry key="rootAP" value="AvamarRootPassword" />
                <entry key="viewuserAP" value="viewuser1" />

Remember to encript the Avamar passwords

/usr/local/avamar/bin/mccipher encrypt --all 

14 February 2019

Upgrade Cohesity Software


1. Go to Cohesity download page and

  • Copy link to upgrade package, or
  • Download upgrade package to local hard drive


2. Login to Cohesity Web UI, go to "Admin" -> "Upgrade Cluster"


3. Click "Get New Package", then you have 2 options:

  • Provide download link to upgrade package


  • Upload upgrade package from local hard drive


Then click "Get Package and Upgrade"

4. Page will refresh automatically, you can click "Show Subtask" to monitor upgrade progress


If you upgrade single node Virtual Coheity it will be unavailable for few minutes, on multi-node clustered Cohesity, one node will be upgraded at a time and you still will be able to use it as normal.
On one point you will be logged out to to refresh the Web UI

5. Refresh web page and login to Cohesity Web UI, go to "Admin" -> "Upgrade Cluster" to confirm new software has been installed



Cohesity Virtual Edition - expand disk capacity

If you run out of space on your virtual Cohesity below are steps which allow you to extend its capacity.

1. In Cohesity Web UI you can see capacity 534.6 GB



2. SSH to the Cohesity, switch to iris_cli and run: "disk ls" to determine disks size, as you can see we have 50 GB metadata disk and 502 GB data disk


3. Login to ESX or vCentre server and confirm current drives size



4. Expand "Hard Disk 2" metadata disk form 50 to 70 GB and "Hard Disk 3" data disk from 512 to 1024 GB, then restart Cohesity appliance to ensure it can properly recognize the drives new size.



5. SSH to Cohesity, switch to iris_cli, run:

  • disk ls - make note of "CURRENT NODE ID"
  • disk extend node-id=CURRENT_NODE_ID
Now wait few minutes to allow system to expand disks


6. Run "disk ls" to confirm drives have new sizes as expected


8. Run "cluster restart all-services=true" to allow Cohesity to utilize new capacity


9. While all services are restarted login to Cohesity Web UI to confirm new available capacity


Deploy Cohesity Virtual Edition


1. Login to Cohesity support portal and download OVA file


2. Deploy Cohesity OVA to your ESXi / vSphere environment


3. Give it a name and choose OVA file


4. Choose datastore for Cohesity OS drive


5. Choose network the Cohesity should be connected to


6. Configure Cohesity initial IP settings


7. Click Next to deploy Cohesity appliance

8. Edit appliance configuration and add 2 hdd, one for metadata (best on SSD datastore) and second for data (max 20 x metadata drive size)



9. If you deploy Cohesity appliance to ESXi host IPs will not be applied and you have to do it manually. Open Cohesity appliance console, login as admin/admin, you have to put the credential twice, then run below commands:

cohesity>cluster status
Error: Node having ID 1195873274425422 is not part of a cluster

cohesity> node configure-network node-ips=192.168.12.101 subnet-mask=255.255.255.0 subnet-gateway=192.168.12.1 node-ids=1195873274425422

10. Login to Cohesity Web UI as admin/admin


11. Follow onscreen instructions to complete initial configuration


12. Complete all the relevant information, remember that appliance has to be able to resolve its fqdn to the IP, appliance needs access to NTP servers - otherwise installation will fail


13. Wait for cluster to be configured


14. Login to Cohesity Web UI, accept the license agreement


15. Apply License Key


16. Cohesity appliance deployment is complete now.


12 February 2019

Cohesity C2605 initial setup

Easy way to deploy physical Cohesity cluster.

Remember to read installation prerequisites prior to starting an installation.

1. Rack your Cohesity, cable as in Cohesity manual and configure your switches


2. Connect Monitor to one node and check the IP


3. Configure i.e. laptop with 169.254.0.1/16 IP address and connect to the same network Cohesity is on.
4. Brows to Cohesity Node IP https://169.254.4.16


5. Choose cluster nodes (3 minimum)


6. Configure nodes IP and IPMI


7. Configure cluster settings


For VIP Address I use one IP per node so for this cluster type Node 1 IP, then Node 3 IP - just last octet and click "Add VIP".
Remember Cluste Hostname has to be resolvable and each Node has to have access to DNS & NTP server, otherwise installation fails.

8. Click "Create Cluster", wait few minutes and Cohesity cluster should be up & running.

11 February 2019

Data Domain network speed test with iperf


If you want to test network speed between your Data Domain (i.e. to troubleshoot replication issue) use iperf tool.

Below example test speed from ddve-02 (source) to ddve-01 (destination).

For iperf to work you have to ensure the TCP port 5001 is open on destination Data Domain.

On destination Data Domain you have to start iperf as server
sysadmin@ddve-01# net iperf server

On source Data Domain start iperf as client
sysadmin@ddve-02# net iperf client 192.168.12.21 duration 60 interval 10

192.168.12.21 - destination Data Domain IP
duration 60 - test duration 60 seconds
interval 10 - report speed results every 10 seconds

Example iperf output

sysadmin@ddve-01# net iperf server
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  4] local 192.168.12.21 port 5001 connected with 192.168.12.22 port 46551
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.0 sec   229 GBytes  32.8 Gbits/sec
^C
sysadmin@ddve-01#


sysadmin@ddve-02# net iperf client 192.168.12.21 duration 60 interval 10
------------------------------------------------------------
Client connecting to 192.168.12.21, TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  3] local 192.168.12.22 port 46551 connected with 192.168.12.21 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  38.7 GBytes  33.2 Gbits/sec
[  3] 10.0-20.0 sec  31.5 GBytes  27.1 Gbits/sec
[  3] 20.0-30.0 sec  39.7 GBytes  34.1 Gbits/sec
[  3] 30.0-40.0 sec  41.4 GBytes  35.6 Gbits/sec
[  3] 40.0-50.0 sec  39.9 GBytes  34.3 Gbits/sec
[  3] 50.0-60.0 sec  38.2 GBytes  32.8 Gbits/sec
[  3]  0.0-60.0 sec   229 GBytes  32.8 Gbits/sec
sysadmin@ddve-02#

09 February 2019

Avamar client installation CLI

Avamar client for Windows

C:\Avamar> msiexec.exe /qn /I “C:\Avamar\AvamarClient-windows-x86_64-7.5.100-183.msi” SERVER=”grid1.avamar.local” DOMAIN=”/clients” GROUP=”/clients/backup_policy” UICOMPONENT=1 PROGRESSBAR=false BALLOONMESSAGE=false BACKUPREMINDER=never

Register client with the grid

C:\Program Files\avs\bin>avregister.bat “grid1.avamar.local” “/clients”

Degraded VD on Avamar storage node

Troubleshooting of degraded VD on Avamar storage node

Command list:

ssh-agent bash
ssh-add /home/admin/.ssh/dpnid
status.dpn
avmaint nodelist | grep ‘nodetag\|gendesc’
mccli event show –unack
mapall –noerror –user=root ‘grep -h “MRMON151\|MRMON195\|MRMON153\|MRMON194” /var/log/messages | sort’
mapall –noerror –user=root ‘grep -i “scsi\|medium\|media\|fatal\|critical\|sense\|memory” /var/log/messages’
mapall –noerror –all hostname

Login to the affected node:
ssn –user=root 0.x
grep -h “MRMON151\|MRMON195\|MRMON153\|MRMON194” /var/log/messages | sort
avsysreport pdisk controller=0
grep -i “scsi\|medium\|media\|fatal\|critical\|sense\|memory” /var/log/messages
sudo ipmitool fru print 0

Manual Garbage Collection

Manual GC should be performed by EMC staff as by doing it you may invalidate the warranty and/ or delete backups.

Login to grid Utility Node as admin.

1. Kill any current sessions

avmaint sessions --full | grep sessionid | cut -d'”' -f2 | xargs -L1 avmaint kill

2. Stop the Backup Scheduler

dpnctl stop sched

3. Stop the Maintenance Window Schedule

dpnctl stop maint

4. Take a new checkpoint

avmaint checkpoint --ava

5. Check and wait for the completion of the checkpoint

status.dpn | grep ^Checkpoint

6. Disable the Index Cache

avmaint config --ava indexcacheallowed=2

7. Initiate a manual Garbage Collection job

avmaint garbagecollect --ava --kill=0 --maxpass=0 --refcheck=true --throttlelevel=0 --usehistory=false --maxtime=18000

8. Re-enable the Index Cache

avmaint config --ava indexcacheallowed=1

9. Start the Maintenance Window Scheduler

dpnctl start maint

10. Start the Backup Scheduler

dpnctl start sched

11. Confirm all systems are up and operational

dpnctl status all


Command list:

avmaint sessions --full | grep sessionid | cut -d'”' -f2 | xargs -L1 avmaint kill
dpnctl stop sched
dpnctl stop maint
avmaint checkpoint --ava
status.dpn | grep ^Checkpoint
avmaint config –ava indexcacheallowed=2
avmaint garbagecollect --ava --kill=0 --maxpass=0 --refcheck=true --throttlelevel=0 --usehistory=false --maxtime=18000
avmaint config --ava indexcacheallowed=1
dpnctl start maint
dpnctl start sched
dpnctl status all

Remove partial/ incomplete backup

admin@avamar:~/#: avmgr getb –path=/clients/avamar-client –incpartials –format=xml
1 Request succeeded
<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?>
<backuplist version=”3.0″>
<backuplistrec flags=”16777233″ labelnum=”1″ label=”STG Windows FS_with VSS_no Exc-1504327839115″ created=”1504537525″ roothash=”467534c908cb977867b75ba6eaa2a9a301356064″ totalbytes=”1803647582208.00″ ispresentbytes=”0.00″ pidnum=”3001″ percentnew=”96″ expires=”1505142325″ created_prectime=”0x1d3258f3ccc0cf2” partial=”1″ retentiontype=”none” backuptype=”Full” ddrindex=”0″ locked=”0″ direct_restore=”1″/>
</backuplist>

admin@avamar:~/#: dpnctl stop maint
dpnctl: INFO: Suspending maintenance windows scheduler…

admin@avamar:~/#: avmaint session | grep path

admin@avamar:~/#: avmaint config asynccrunching=false –ava
<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?>
<gsanconfig asynccrunching=”true”/>

admin@avamar:~/#: avmgr delb –path=/clients/avamar-client –incpartials –incpartial –date=0x1d3258f3ccc0cf2
1 Request succeeded

admin@avamar:~/#: avmaint config asynccrunching=true –ava
<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?>
<gsanconfig asynccrunching=”false”/>

admin@avamar:~/#: dpnctl start maint
dpnctl: INFO: Resuming maintenance windows scheduler…
dpnctl: INFO: maintenance windows scheduler resumed.

admin@avamar:~/#: avmgr delb –path=/clients/avamar-client –incpartials –incpartial –date=0x1d3258f3ccc0cf2
admin@avamar:~/#: avmgr getb –path=/clients/avamar-client –incpartials –format=xml
1 Request succeeded



Command list:

avmgr getb –path=/clients/avamar-client –incpartials –format=xml
dpnctl stop maint
avmaint session | grep path
avmaint config asynccrunching=false –ava
avmgr delb –path=/clients/avamar-client –incpartials –incpartial –date=0x1d3258f3ccc0cf2
avmaint config asynccrunching=true –ava
dpnctl start maint
avmgr delb –path=/clients/avamar-client –incpartials –incpartial –date=0x1d3258f3ccc0cf2
avmgr getb –path=/clients/avamar-client –incpartials –format=xml

Useful commands

Restart MCS with new probe.xml

avamaint networkconfig /usr/local/avamar/var/probe.xml –avamaronly

WARNING: gc is suspended permanently
WARNING: hfscheck is suspended permanently.

avmaint sched resume cp –permanent –ava

avmaint sched resume gc –permanent –ava

avmaint sched resume hfscheck –permanent –ava

How to retrieve Avamar Serial Numbers via SSH

mapall --noerror --user=root --nodes=all+ --quiet 'echo `hostname -i`": "`/usr/bin/ipmitool fru print 0 | grep "Product Asset Tag" | sed "s/^.*:\s\(.*$\)/\1/"`'

PowerCLI - How to get cluster resource information

Get cluster available resources and usage

Get-Cluster -Name ClusterName | get-vmhost | select Name, NumCpu, CpuUsageMhz, CPUTotalMhz, MemoryUsageGB, MemoryTotalGB | Export-Csv -NoTypeInformation -UseCulture -Path c:\test\vent1.csv

Get all datastores per cluster

foreach ($cluster in Get-Cluster){Add-content fi $cluster.Name; Get-cluster -name $cluster.Name | Get-Datastore | select Name, FreeSpaceGB, CapacityGB}

Get all datastores per cluster and export to txt file

foreach ($cluster in Get-Cluster){Add-content "C:\test\test.txt" "$cluster"; Add-Content "C:\test\test.txt" "Name, FreeSpaceGB, CapacityGB"; $d=Get-cluster -name $cluster.Name | Get-Datastore | select Name, FreeSpaceGB, CapacityGB; $d | % {add-content "c:\test\test.txt" "$($_.Name), $($_.FreeSpaceGB), $($_.CapacityGB)"}}

Get count of all hosts cpu cores

Get-Cluster -Name SharedEnterprisePlus01 | Get-VMHost | Measure-Object -Property NumCpu -Sum | select -ExpandProperty Sum

Get list of all NOT empty resource pools in the cluster based on pool's VM's cpu count

$ResourcePools = Get-Cluster -Name ClusterName | Get-ResourcePool
foreach ($Pool in $ResourcePools){$a = Get-ResourcePool -Name $Pool.Name | Get-VM | Measure-Object -Property NumCpu -Sum | select -ExpandProperty Sum; if($a -gt 0) {$Pool.Name}}

Get list of all VMs which have "prod" in their name and summary information about disk size

$a = get-vm | where{$_.Name -like "*prod*"}; foreach ($v in $a) {Write-Host $v.Name; Get-HardDisk -vm $v.name | Measure-Object -Property CapacityGB -Sum | select -ExpandProperty Sum}

Get list of all VMs which have "prod" in their name and summary information about disk size and write to txt file. I've used Write-Host commands to display output on the screen too.

$a = get-vm; foreach ($v in $a) {$s = Get-HardDisk -vm $v.name | Measure-Object -Property CapacityGB -Sum | select -ExpandProperty Sum; $s | %{Add-Content "d:\iso\vm.txt" "$($v.Name), $($_)"}; Write-Host $v.Name; Write-Host $s}

PowerCLI – How to get Vmware tools version

Get vmtools version for all VMs


get-vm | % {get-view $_.id} | select Name,@{Name="Tools Version"; Expression={$_.Config.tools.toolsversion}}, @{ Name="Tool Status"; Expression={$_.Guest.ToolsVersionStatus}}

Get vmtools version for all VMs from a cluster

get-cluster -Name ClusterName | get-vm | % {get-view $_.id} | select Name, @{Name="Tools Version"; Expression={$_.Config.tools.toolsversion}}, @{ Name="Tool Status"; Expression={$_.Guest.ToolsVersionStatus}}

Get vmtools version for all powered on VMs

get-vm | where {$_.powerstate -eq "PoweredOn"} | % {get-view $_.id} | select Name, @{Name="Tools Version"; Expression={$_.Config.tools.toolsversion}}, @{ Name="Tool Status"; Expression={$_.Guest.ToolsVersionStatus}}

List all VMs with out of date vmtools

get-vm | where {$_.Guest.ToolsVersionStatus -ne "guestToolsCurrent"} | % {get-view $_.id} | select Name, @{Name="Tools Version"; Expression={$_.Config.tools.toolsversion}}, @{ Name="Tool Status"; Expression={$_.Guest.ToolsVersionStatus}}

Get vmtools for all VMs with Microsoft operating system

get-cluster -Name ClusterName | get-vm | % {get-view $_.id} | where {$_.config.guestfullname -like "*Microsoft*"} | select Name, @{Name="OS Version"; Expression={$_.config.guestfullname}}, @{Name="Tools Version"; Expression={$_.Config.tools.toolsversion}}, @{Name="Tools Status"; Expression={$_.Guest.ToolsVersionStatus2}}

Get vmtools version for all VMs exported to csv file

get-vm | % {get-view $_.id} | select Name, @{Name="Tools Version"; Expression={$_.Config.tools.toolsversion}}, @{ Name="Tool Status"; Expression={$_.Guest.ToolsVersionStatus}} | Export-Csv -NoTypeInformation -UseCulture -Path C:\Tools\ToolsVersion.csv

PowerCLI - iSCSI Configuration

iSCSI storage configuration

Create vdSwitch for iSCSI traffic. Switch will have 2 uplinks, and MTU 9000 (Jumbo frames)

$Datacentre = Get-Datacenter -Name Datacentre_01
New-VDSwitch -Name vdSwitch_iSCSI -Location $Datacentre -NumUplinkPorts 2 -MaxPorts 256 -Version "6.0.0" -Mtu 9000 | Out-Null


The iSCSI vdSwitch will have 2 iSCSI port groups associated with vlan 230. vdPG_iSCSI_1 will only use dvUplink1 and vdPG_iSCSI_2 will only use dvUplink2.

Get-VDSwitch -Name vdSwitch_iSCSI | Get-VDPortgroup | Get-VDUplinkTeamingPolicy | Set-VDUplinkTeamingPolicy -ActiveUplinkPort dvUplink1 -UnusedUplinkPort dvUplink2
Get-VDSwitch -Name vdSwitch_iSCSI | New-VDPortgroup -Name vdPG_iSCSI1 -NumPorts 120 -VLanId 230 -PortBinding Ephemeral
Get-VDSwitch -Name vdSwitch_iSCSI | New-VDPortgroup -Name vdPG_iSCSI2 -NumPorts 120 -VLanId 230 -PortBinding Ephemeral
Get-VDSwitch -Name vdSwitch_iSCSI | Get-VDPortgroup -Name vdPG_iSCSI2 | Get-VDPortgroupOverridePolicy | Set-VDPortgroupOverridePolicy -UplinkTeamingOverrideAllowed $true
Get-VDSwitch -Name vdSwitch_iSCSI | Get-VDPortgroup -Name vdPG_iSCSI2 | Get-VDUplinkTeamingPolicy | Set-VDUplinkTeamingPolicy -ActiveUplinkPort dvUplink2 -UnusedUplinkPort dvUplink1


Add esx host to iSCSI switch

Get-VDSwitch -Name vdSwitch_iSCSI | Add-VDSwitchVMHost -VMHost esx-01.localdomain.local
$nic2 = Get-VMHost -Name esx-01.localdomain.local | Get-VMHostNetworkAdapter -Physical -Name vmnic2
$nic3 = Get-VMHost -Name esx-01.localdomain.local | Get-VMHostNetworkAdapter -Physical -Name vmnic3
Get-VDSwitch -Name $vdSwitch_iSCSI_Name | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic2 -Confirm:$false
Get-VDSwitch -Name $vdSwitch_iSCSI_Name | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic3 -Confirm:$false


Add to iSCSI vdSwitch 2 vmkernel ports for iSCSI traffic (this has to be done for each esx host)

New-VMHostNetworkAdapter -VMHost esx-01.localdomain.local -VirtualSwitch vdSwitch_iSCSI  -PortGroup vdPG_iSCSI1 -IP 192.168.101.1 -SubnetMask 255.255.255.0 -Mtu 9000
New-VMHostNetworkAdapter -VMHost esx-01.localdomain.local -VirtualSwitch vdSwitch_iSCSI  -PortGroup vdPG_iSCSI2 -IP 192.168.101.101 -SubnetMask 255.255.255.0 -Mtu 9000


Enable software iSCSI storage adapter on esx host

Get-VMHostStorage -VMHost $HostName | Set-VMHostStorage -SoftwareIScsiEnabled $true


Add iSCSI vmkernel ports to iSCSI vdSwitch

$iSCSI1_nic = Get-VMHostNetworkAdapter -VMHost esx-01.localdomain.local | where {$_.IP -like "192.168.101.1"} | select Name
$iSCSI2_nic = Get-VMHostNetworkAdapter -VMHost esx-01.localdomain.local | where {$_.IP -like "192.168.101.101"} | select Name
$HBANumber = Get-VMHostHba -VMHost esx-01.localdomain.local -Type iSCSI | %{$_.Device}
$esxcli = Get-EsxCli -VMHost esx-01.localdomain.local
$esxcli.iscsi.networkportal.add($HBANumber,$true,$iSCSI1_nic.Name)
$esxcli.iscsi.networkportal.add($HBANumber,$true,$iSCSI2_nic.Name)


Add iSCSI target and rescan for storage

$HBANumber = Get-VMHostHba -VMHost esx-01.localdomain.local -Type iSCSI | %{$_.Device}
New-IScsiHbaTarget -IScsiHba $HBANumber -Address "192.168.101.201"
Get-VMHostStorage -VMHost esx-01.localdomain.local -RescanAllHba
Get-VMHostStorage -VMHost esx-01.localdomain.local -RescanVmfs


Additionally you may want to change iSCSI software adapter Advances Settings. Typically you will disable "DelayedAck" and i.e. for Equallogic array increase "LoginTimeout". I couldn't find any direct command to do this so I have to use Get-View. More about storage manipulation you can find here.

$HostView = Get-VMHost esx-01.localdomain.local | Get-View
$HostStorageSystemID = $HostView.configmanager.StorageSystem
$HBANumber = ($HostView.config.storagedevice.HostBusAdapter | where {$_.Model -match "iSCSI Software"}).device

$options = New-Object VMWare.Vim.HostInternetScsiHbaParamValue[] (2)
$options[0] = New-Object VMware.Vim.HostInternetScsiHbaParamValue 
$options[0].key = "DelayedAck" 
$options[0].value = $false
$options[1] = New-Object VMware.Vim.HostInternetScsiHbaParamValue 
$options[1].key = "LoginTimeout" 
$options[1].value = 60

$HostStorageSystem = Get-View -ID $HostStorageSystemID 
$HostStorageSystem.UpdateInternetScsiAdvancedOptions($HBANumber , $null, $options)


And finally I have put all commands together to make easy script to configure iSCSI on a cluster

$VIUser="administrator"
$VIPassword="password"

$DatacentreName="Datacentre_01"
$vdSwitch_iSCSI_Name="vdSwitch_iSCSI"
$vdPG_iSCSI1="vdPG_iSCSI_1"
$vdPG_iSCSI2="vdPG_iSCSI_2"
$iSCSI1_Net="192.168.101."
$iSCSI1_StartIP=1
$iSCSI1_NetMask="255.255.255.0"
$iSCSI2_Net="192.168.101."
$iSCSI2_StartIP=101
$iSCSI2_NetMask="255.255.255.0"
$iSCSI_Targets=@("192.168.101.201","192.168.101.202")
$HostNames=@("esx-01.localdomain.local","esx-02.localdomain.local")

Connect-VIServer vcentre.localdomain.local -User $VIUser -Password $VIPassword | Out-Null

$Datacentre = Get-Datacenter -Name $DatacentreName
New-VDSwitch -Name $vdSwitch_iSCSI_Name -Location $Datacentre -NumUplinkPorts 2 -MaxPorts 256 -Version "6.0.0" -Mtu 9000 | Out-Null

Get-VDSwitch -Name $vdSwitch_iSCSI_Name | Get-VDPortgroup | Get-VDUplinkTeamingPolicy | Set-VDUplinkTeamingPolicy -ActiveUplinkPort dvUplink1 -UnusedUplinkPort dvUplink2
Get-VDSwitch -Name $vdSwitch_iSCSI_Name | New-VDPortgroup -Name $vdPG_iSCSI1 -NumPorts 120 -VLanId 230 -PortBinding Ephemeral
Get-VDSwitch -Name $vdSwitch_iSCSI_Name | New-VDPortgroup -Name $vdPG_iSCSI2 -NumPorts 120 -VLanId 230 -PortBinding Ephemeral
Get-VDSwitch -Name $vdSwitch_iSCSI_Name | Get-VDPortgroup -Name $vdPG_iSCSI2 | Get-VDPortgroupOverridePolicy | Set-VDPortgroupOverridePolicy -UplinkTeamingOverrideAllowed $true
Get-VDSwitch -Name $vdSwitch_iSCSI_Name | Get-VDPortgroup -Name $vdPG_iSCSI2 | Get-VDUplinkTeamingPolicy | Set-VDUplinkTeamingPolicy -ActiveUplinkPort dvUplink2 -UnusedUplinkPort dvUplink1

$options = New-Object VMWare.Vim.HostInternetScsiHbaParamValue[] (2)
$options[0] = New-Object VMware.Vim.HostInternetScsiHbaParamValue 
$options[0].key = "DelayedAck" 
$options[0].value = $false
$options[1] = New-Object VMware.Vim.HostInternetScsiHbaParamValue 
$options[1].key = "LoginTimeout" 
$options[1].value = 60

foreach ($HostName in $HostNames){
    $iSCSI1_IP=$iSCSI1_Net+$iSCSI1_StartIP
    $iSCSI2_IP=$iSCSI2_Net+$iSCSI2_StartIP
    Get-VDSwitch -Name $vdSwitch_iSCSI_Name | Add-VDSwitchVMHost -VMHost $HostName
    $nic2 = Get-VMHost -Name $HostName | Get-VMHostNetworkAdapter -Physical -Name vmnic2
    $nic3 = Get-VMHost -Name $HostName | Get-VMHostNetworkAdapter -Physical -Name vmnic3
    Get-VDSwitch -Name $vdSwitch_iSCSI_Name | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic2 -Confirm:$false
    Get-VDSwitch -Name $vdSwitch_iSCSI_Name | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic3 -Confirm:$false
    New-VMHostNetworkAdapter -VMHost $HostName -VirtualSwitch $vdSwitch_iSCSI_Name  -PortGroup $vdPG_iSCSI1 -IP $iSCSI1_IP -SubnetMask $iSCSI1_NetMask -Mtu 9000 | Out-Null
    $iSCSI1_nic = Get-VMHostNetworkAdapter -VMHost $HostName | where {$_.IP -like $iSCSI1_IP} | select Name
    New-VMHostNetworkAdapter -VMHost $HostName -VirtualSwitch $vdSwitch_iSCSI_Name  -PortGroup $vdPG_iSCSI2 -IP $iSCSI2_IP -SubnetMask $iSCSI2_NetMask -Mtu 9000 | Out-Null
    $iSCSI2_nic = Get-VMHostNetworkAdapter -VMHost $HostName | where {$_.IP -like $iSCSI2_IP} | select Name

    Get-VMHostStorage -VMHost $HostName | Set-VMHostStorage -SoftwareIScsiEnabled $true  | Out-Null

    $HBANumber = Get-VMHostHba -VMHost $HostName -Type iSCSI | %{$_.Device}

    $esxcli = Get-EsxCli -VMHost $HostName
    $esxcli.iscsi.networkportal.add($HBANumber,$true,$iSCSI1_nic.Name)
    $esxcli.iscsi.networkportal.add($HBANumber,$true,$iSCSI2_nic.Name)

    $HostView = Get-VMHost $HostName | Get-View
    $HostStorageSystemID = $HostView.configmanager.StorageSystem
    $HostStorageSystem = Get-View -ID $HostStorageSystemID 
    $HostStorageSystem.UpdateInternetScsiAdvancedOptions($HBANumber , $null, $options)

    foreach ($iSCSI_Target in $iSCSI_Targets){
        New-IScsiHbaTarget -IScsiHba $HBANumber -Address $iSCSI_Target
    }

    Get-VMHostStorage -VMHost $HostName -RescanAllHba
    Get-VMHostStorage -VMHost $HostName -RescanVmfs
}

Disconnect-VIServer -Confirm:$false

PowerCLI - Cluster Configuration

I've just decided to create some step by step guide how to create VMware cluster via PowerCLI
Please keep in mind that this guide will not show you how to configure all parameters but should give you an idea how to start.

Prerequisite for this guide:

  • vCentre server has to be installed and static IP configured
  • All ESX hosts should have hypervisor installed and static management IP configured
  • There has to be network connectivity between vCentre and ESX hosts

Connect to your vCentre server

Connect-VIServer vcentre.localdomain.local

Get default folder list, we will use it to create datacentre

Get-Folder

PowerCLI C:\> Get-Folder

Name                           Type

----                           ----
Datacenters                    Datacenter
vm                             VM
network                        Network
host                           HostAndCluster
datastore                      Datastore
Discovered virtual machine     VM

Create new datacentre "Datacentre_01"

New-Datacenter -Location Datacenters -Name Datacentre_01

Create new cluster in datacentre "Datacentre_01"

New-Cluster -Name Cluster_01 -Location "Datacentre_01"

On "Cluster_01" enable HA, virtual machines in the cluster should not start if they violate availability constraints, cluster should tolerate 1 host failure, VM swap file should be stored wit VM files, VM restart priority in case of host failure should be "Low", in case of host isolation all running VMs should remain online

Get-Cluster -Name Cluster_01 | Set-Cluster -Name Cluster_01 -HAEnabled:$true -HAAdmissionControlEnabled:$true -HAFailover 1 -VMSwapfilePolicy "WithVM" -HARestartPriority "Low" -HAIsolationResponse "DoNothing" -Confirm:$false

On "Cluster_01" enable DRS and set to be fully automated

Set-Cluster -Cluster Cluster_01 -DRSEnabled:$true -DRSAutomationLevel "FullyAutomated" -Confirm:$false

On "Cluster_01" enable EVC for intel-nehalem

Set-Cluster -Cluster Cluster_01 -EVCMode "intel-nehalem" -Confirm:$false

If you want to disable EVC run:

Set-Cluster -Cluster Cluster_01 -EVCMode $null -Confirm:$false

Add host esx-01 to "Cluster_01"

Add-VMHost -Name esx-01.localdomain.local -Location Cluster_01 -User root -Password password

If you want to remove host from cluster run:

Set-VMHost -VMHost esx-01.localdomain.local -State Disconnected
Remove-VMHost esx-01.localdomain.local -Confirm:$false

Put your host in maintenance mode

Set-VMHost -VMHost esx-01.localdomain.local -State Maintenance

Get esx host physical nic list

Get-VMHost -Name esx-01.localdomain.local | Get-VMHostNetworkAdapter | where {$_.DeviceName -like "*nic*"} | select Name

PowerCLI C:\>Get-VMHost -Name esx-01.localdomain.local | Get-VMHostNetworkAdapter | where {$_.DeviceName -like "*nic*"} | select Name
Name
----
vmnic0
vmnic1
vmnic2
vmnic3

Confirm that there is only one nic attached to default vSwitch0

Get-VMHost -Name esx-01.localdomain.local | Get-VirtualSwitch -Name vSwitch0 | select Nic

PowerCLI C:\>Get-VMHost -Name esx-01.localdomain.local | Get-VirtualSwitch -Name vSwitch0 | select Nic
Nic
---
{vmnic0}

Add 2nd physical nic "vmnic1" to vSwitch0

$nic = Get-VMHost -Name esx-01.localdomain.local | Get-VMHostNetworkAdapter -Physical -Name vmnic1
Get-VMHost -Name esx-01.localdomain.local | Get-VirtualSwitch -Name vSwitch0 | Add-VirtualSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic -Confirm:$false

Create new VM port group on vSwitch0

Get-VMHost -Name esx-01.localdomain.local | Get-VirtualSwitch -Name vSwitch0 | New-VirtualPortGroup -Name vPortgroup

Add vlan id 120 to port group "vPortgroup"

Get-VMHost -Name esx-01.localdomain.local | Get-VirtualSwitch -Name vSwitch0 | Get-VirtualPortGroup -Name vPortgroup | Set-VirtualPortGroup -name test -VLanId 120

Create new VM port group on vSwitch0 with vlan id 120

Get-VMHost -Name esx-01.localdomain.local | Get-VirtualSwitch -Name vSwitch0 | New-VirtualPortGroup -Name vPortgroup -VLanId 120

Create new vdSwitch with 2 uplinks

$Datacentre = Get-Datacenter -Name Datacentre_01
New-VDSwitch -Name vdSwitch0 -Location $Datacentre -NumUplinkPorts 2 -LinkDiscoveryProtocol "LLDP" -LinkDiscoveryProtocolOperation "Listen" -MaxPorts 256 -Version "6.0.0"

Set uplink teaming policy for vdSwitch0: dvUplink1 as active and dvUplink2 as standby

Get-VDSwitch -Name vdSwitch0 | Get-VDPortgroup | Get-VDUplinkTeamingPolicy | Set-VDUplinkTeamingPolicy -ActiveUplinkPort dvUplink1 -StandbyUplinkPort dvUplink2

Create new port group Production on vdSwitch0 with 16 ports, vlan id 200 and port binding Ephemeral

Get-VDSwitch -Name vdSwitch0 | New-VDPortgroup -Name Production -NumPorts 16 -VLanId 200 -PortBinding Ephemeral

Create new port group vMotion on vdSwitch0 with 16 ports, vlan id 110 and port binding Ephemeral

Get-VDSwitch -Name vdSwitch0 | New-VDPortgroup -Name vMotion -NumPorts 16 -VLanId 110 -PortBinding Ephemeral

Overwrite uplink teaming policy for vMotion port group: dvUplink2 as active and dvUplink1 as standby

Get-VDSwitch -Name vdSwitch0 | Get-VDPortgroup -Name vMotion | Get-VDPortgroupOverridePolicy | Set-VDPortgroupOverridePolicy -UplinkTeamingOverrideAllowed $true
Get-VDSwitch -Name vdSwitch0 | Get-VDPortgroup -Name vMotion | Get-VDUplinkTeamingPolicy | Set-VDUplinkTeamingPolicy -ActiveUplinkPort dvUplink2 -StandbyUplinkPort dvUplink1

Add esx-01 host to vdSwitch0

Get-VDSwitch -Name vdSwitch0 | Add-VDSwitchVMHost -VMHost esx-01.localdomain.local

Add vmnic2 and vmnic3 of esx-01 host to vdSwitch0 uplinks

$nic2 = Get-VMHost -Name esx-01.localdomain.local | Get-VMHostNetworkAdapter -Physical -Name vmnic2
$nic3 = Get-VMHost -Name esx-01.localdomain.local | Get-VMHostNetworkAdapter -Physical -Name vmnic3
Get-VDSwitch -Name vdSwitch0 | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic2 -Confirm:$false
Get-VDSwitch -Name vdSwitch0 | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic3 -Confirm:$false

Add vmkernel port to vdSwitch0 to vMotion port group and configure it for vMotion traffic with static IP address

New-VMHostNetworkAdapter -VMHost esx-01.localdomain.local -VirtualSwitch vdSwitch0 -PortGroup vMotion -IP 192.168.0.1 -SubnetMask 255.255.255.0 -VMotionEnabled:$true

Remove your host from maintenance mode

Set-VMHost -VMHost esx-01.localdomain.local -State Connected



And finally I have put all commands together to make easy script to configure a cluster

$VIUser="administrator"
$VIPassword="password"
$HostUser="root"
$HostPassword="hostpassword"

$DatacentreName="Datacentre_01"
$ClusterName="Cluster_01"
$vSwitchName="vSwitch0"
$vdSwitchName="vdSwitch0"
$vMotionNet="192.168.0."
$VMotionStartIP=1
$VMotionNetMask="255.255.255.0"$HostNames=@("esx-01.localdomain.local","esx-02.localdomain.local")

Connect-VIServer vcentre.appv4u.com -User $VIUser -Password $VIPassword

New-Datacenter -Location Datacenters -Name $DatacentreName
New-Cluster -Name $ClusterName -Location $DatacentreName
Get-Cluster -Name $ClusterName | Set-Cluster -Name $ClusterName -HAEnabled:$true -HAAdmissionControlEnabled:$true -HAFailover 1 -VMSwapfilePolicy "WithVM" -HARestartPriority "Low" -HAIsolationResponse "DoNothing" -Confirm:$false
Set-Cluster -Cluster $ClusterName -DRSEnabled:$true -DRSAutomationLevel "FullyAutomated" -Confirm:$false
Set-Cluster -Cluster $ClusterName -EVCMode "intel-nehalem" -Confirm:$false

$Datacentre = Get-Datacenter -Name $DatacentreName
New-VDSwitch -Name $vdSwitchName -Location $Datacentre -NumUplinkPorts 2 -LinkDiscoveryProtocol "LLDP" -LinkDiscoveryProtocolOperation "Listen" -MaxPorts 256 -Version "6.0.0"

Get-VDSwitch -Name $vdSwitchName | Get-VDPortgroup | Get-VDUplinkTeamingPolicy | Set-VDUplinkTeamingPolicy -ActiveUplinkPort dvUplink1 -StandbyUplinkPort dvUplink2
Get-VDSwitch -Name $vdSwitchName | New-VDPortgroup -Name Production -NumPorts 16 -VLanId 200 -PortBinding Ephemeral
Get-VDSwitch -Name $vdSwitchName | New-VDPortgroup -Name vMotion -NumPorts 16 -VLanId 110 -PortBinding Ephemeral
Get-VDSwitch -Name $vdSwitchName | Get-VDPortgroup -Name vMotion | Get-VDPortgroupOverridePolicy | Set-VDPortgroupOverridePolicy -UplinkTeamingOverrideAllowed $true
Get-VDSwitch -Name $vdSwitchName | Get-VDPortgroup -Name vMotion | Get-VDUplinkTeamingPolicy | Set-VDUplinkTeamingPolicy -ActiveUplinkPort dvUplink2 -StandbyUplinkPort dvUplink1

foreach ($HostName in $HostNames){

 $VMotionIP=$vMotionNet+$VMotionStartIP
 Add-VMHost -Name $HostName -Location $ClusterName -User $HostUser -Password $HostPassword | Out-Null

 Set-VMHost -VMHost $HostName -State Maintenance | Out-Null

 $nic1 = Get-VMHost -Name $HostName | Get-VMHostNetworkAdapter -Physical -Name vmnic1
 Get-VMHost -Name $HostName | Get-VirtualSwitch -Name vSwitch0 | Add-VirtualSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic1 -Confirm:$false

 Get-VDSwitch -Name $vdSwitchName | Add-VDSwitchVMHost -VMHost $HostName
 $nic2 = Get-VMHost -Name $HostName | Get-VMHostNetworkAdapter -Physical -Name vmnic2
 $nic3 = Get-VMHost -Name $HostName | Get-VMHostNetworkAdapter -Physical -Name vmnic3
 Get-VDSwitch -Name $vdSwitchName | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic2 -Confirm:$false
 Get-VDSwitch -Name $vdSwitchName | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $nic3 -Confirm:$false
 New-VMHostNetworkAdapter -VMHost $HostName -VirtualSwitch $vdSwitchName  -PortGroup vMotion -IP $VMotionIP -SubnetMask $VMotionNetMask -VMotionEnabled:$true

 Set-VMHost -VMHost $HostName -State Connected
 $VMotionStartIP++
 $VMotionIP=""
}

Disconnect-VIServer -Confirm:$false

PowerCLI - Datastore manipulation

Get all LUNs connected to host

Get-Cluster -Name ClusterName | Get-VMHost -Name HostName | Get-ScsiLun

Get all "disk" LUNs connected to host

Get-Cluster -Name ClusterName | Get-VMHost -Name HostName | Get-ScsiLun -LunType disk

Get alldatastores name, free space, capacity

Get-View -ViewType Datastore -Property Name, Info | Get-VIObjectByVIView

Set MultiPath Policy to "RoundRobin" on all LUNs which have not "RoundRobin" already

Get-Cluster -Name ClusterName | Get-VMHost -Name HostName | Get-ScsiLun -LunType disk | Where {$_.MultiPathPolicy -ne "RoundRobin"} | Set-ScsiLun -MultiPathPolicy "RoundRobin"

Get full information of all LUNs connected to host (you can find here SAN device Model which will be used in next command)

Get-Cluster -Name ClusterName | Get-VMHost -Name HostName | Get-ScsiLun | fl

Set MultiPath Policy to "RoundRobin" on all LUNs where SAN model is "VRAID"

Get-Cluster -Name ClusterName | Get-VMHost -Name HostName | Get-ScsiLun -LunType disk | Where {$_.MultiPathPolicy -ne "RoundRobin" -and $_.Model -eq "VRAID"} | Set-ScsiLun -MultiPathPolicy "RoundRobin"

Get Canonical Name ans Capacity of all LUNs attached to each host in the cluster

foreach ($v in $(get-cluster -Name ClusterName | Get-VMHost)) {$v.name; Get-VMHost -name $v.Name | Get-ScsiLun | select CanonicalName,CapacityGB}

Get Canonical Name and Prefered Path of all LUNs connected to host where Path Policy is set to "Fixed" and SAN model is not "VRAID"

Get-Cluster -Name ClusterName | Get-VMHost -Name HostName | Get-ScsiLun -LunType disk | Where {$_.MultiPathPolicy -eq "Fixed" -and $_.Model -ne "VRAID"} | Get-ScsiLunPath | select ScsiCanonicalName, Name | Export-Csv -NoTypeInformation -UseCulture -Path c:\test\zone1_path.csv

Get all Equallogic datastore path policy and iop to switch path

Get-VMHost | Get-ScsiLun | Where {$_.Vendor -eq "EQLOGIC"} | select VMHost, MultipathPolicy, CommandsToSwitchPath

Get all Equallogic datastore path policy to "RoundRobin" and iop to switch path to "3"

Get-VMHost | Get-ScsiLun | Where {$_.Vendor -eq "EQLOGIC"} | Set-ScsiLun -MultipathPolicy RoundRobin -CommandsToSwitchPath 3

Display all datastores attached to ESX hosts with their Canonical Names

get-vmhost -Name HostName | Get-Datastore | Where-Object {$_.ExtensionData.Info.GetType().Name -eq "VmfsDatastoreInfo"} | ForEach-Object { if ($_) {$Datastore = $_; $Datastore.ExtensionData.Info.Vmfs.Extent | Select-Object -Property @{Name="Name";Expression={$Datastore.Name}},DiskName}}

PowerCLI - VM manipulation

How to get vCPU count of all powered on VMs of particular ESX host

Get-VMHost -Name HostName | get-vm | Where {$_.PowerState -eq "PoweredOn"} | Measure-Object -Property NumCpu -Sum | select -ExpandProperty Sum

How to get count of all powered on VMs of particular ESX host

$a = Get-VMHost -Name HostName | get-vm | Where {$_.PowerState -eq "PoweredOn"} | select NumCpu; $a.count

How to get summary of VM snapshot size

$vms = get-vm; foreach ($vm in $vms) {$snap = get-vm -Name $vm.Name | Get-Snapshot | Measure-Object -Property SizeGB -Sum | Select Name -ExpandProperty Sum; if ($snap -gt 0) {$vm.Name, $snap}}

Get Canonical Name and Capacity of all Raw Devices (RDM) attached to each VM in the cluster

foreach ($v in $(get-cluster -Name Cluster Name | Get-VM)) {$v.Name; Get-VM -Name $v.Name | Get-HardDisk -DiskType RawPhysical | select ScsiCanonicalName,CapacityGB}

EMC Avamar firewall Workaround


Wanted to share a workaround for an EMC Avamar bug I recently came across. I was doing a new install of a couple of Avamar servers and ran across an issue were I could not perform a backup or replicate between sites without encryption turned on. Turns out this is a known bug with Avamar OS 7.1.0.  Even after turning off the “avfirewall” at the target and source sites I was still unable to connect on the unencrypted port (port 27000).

The issue is with new installs of 7.1.  The “gsan-port” file is missing from the “/usr/local/avamar/lib/admin/security” directory.  Here are the steps to recreate this file.

su to the dpn user.

su – dpn
ssh-agent bash
ssh-add ~/.ssh/dpnid

create the “gsan-port” config file.

vi /usr/local/avamar/lib/admin/security/gsan-port

Insert this on the first line.

GSAN_PLAIN_TEXT=’27000,’

Save and close the config file. If using a multi-node grid; this must be created on the utility node and all the storage nodes.

You can quickly copy the configuration file to all the nodes with scp on the internal network or use the “mapall” command.

cd /usr/local/avamar/lib/admin/security

scp gsan-port 192.168.255.2:/usr/local/avamar/lib/admin/security
scp gsan-port 192.168.255.3:/usr/local/avamar/lib/admin/security
scp gsan-port 192.168.255.4:/usr/local/avamar/lib/admin/security

Restart the firewall services.

service avfirewall restart

Make sure to restart the services on the storage nodes as well if using a multi-node grid. Here is how to do all the nodes at once.

mapall –noerror –all+ –user=root ‘service avfirewall restart’

To quickly test that it worked, open a telnet session on port 27000 from a utility node to a local storage node or from the utility node to the target node. Once a connection is confirmed, retry an unencrypted backup. Hopefully this will save you a few hours of head banging wondering why your backup is busted.

Use Avtar to manage/ run backups


Avtar

Display backup list for server.virtworld.link


avtar --backups--id=Avamar_user --ap=Avamar_user_password --account=/clients/server.virtworld.link

Run backup


Run backup on Windows server server.virtworld.link using "Windows File System" plugin, store backup on Data Domain with index 1, keep backup for 365 days, mark it as retention type - daily, backup type incremental_full

avtar --create --id=Avamar_user --ap=Avamar_user_password --account=/clients/server.virtworld.link --label=Backup_Label_1 --pluginid-list=3001 --ddr=true --ddr-index=1 --expires=365 --retention-type=daily --backup-type=incremental_full

To avoid putting all the common parameters in the command line you can use avtar flag file.

my-avtart-flag.file

--id=Avamar_user
--ap=Avamar_user_password
--ddr=true
--ddr-index=1
--expires=365
--retention-type=daily
--backup-type=incremental_full

Backup command:

avtar --create --flagfile=my-avtart-flag.file --account=/clients/server.virtworld.link --label=Backup_Label_1 --pluginid-list=3001

Plugin

mccli plugin show
1001 Linux File System
3001 Windows File System

avtar --help


Usage: avtar <Task> [Options...]

Tasks:

-c, --create Create a new backup
-x, --extract Extract files from a backup
-t, --list Show contents of a backup
--backups List available client backups
--delete Delete one backup from the server
--showlog Show session log of a backup
--validate Verify integrity of a backup

Options:

--account=PATH, --path, --acnt Specify the Avamar server account path.
--after=DATE Process backups that were created on or after the DATE. Specify date by using the format: YYYY-MM-DD HH:MM:SS.FF.
--backup-type=TYPE Specify the backup type. Values for TYPE are incremental, differential, level0_full, incremental_full, differential_full, synthetic_full, metadata, co_incremental, co_full, co_incremental_full, co_incremental_co_full or incremental_co_full.
--backup_mounted_vhds Backup Natively Attached Virtual Hard Disks on Windows7 and Windows 2008 R2. [True]
--backupboost-config=MNTLIST Configure file system mounts specified in MNTLIST.
--backupboost-convert Convert linux file system backup to BackupBoost backup.
--backupboost-list List all candidate mounts for BackupBoost
--backupboost-unconfig=MNTLIST Unconfigure file system mounts specified in MNTLIST.
--backupsystem Backup Windows System State.
--backupsystem-ignore-version-check Skip Windows System State Backup via File System Plug-in on Windows 2008, Windows 7 and later.
--bbjournalmaint Perform BackupBoost journal clean up.
--bbjournalmaintdate=DATE Perform cleanup for BackupBoost journal logs that were created on and before DATE. Specify the date by using the format: YYYY-MM-DD HH:MM:SS
--bbjournalmaintlist=MNTLIST Perform BackupBoost journal clean up for a list of mounts specified in MNTLIST.
--before=DATE Select backups that were created before DATE. Specify the date by using the format: YYYY-MM-DD HH:MM:SS.FF.
--browse_filter_threshold_value=entries Minimum number of entries that activates the browse filter [10000]
--cachepaxstream Enable the caching for PAX stream format. (Primary used by the Web Service plug-in.)
--checkisalreadyrestored Check whether a MAPI item has already been restored once.
--count=NUM Specify the maximum number of records to display with a backup command.
--ddr-compressed-restore Enable Data Domain restore/read compression.
--ddr-exempt=PATTERN Prevent files that match the PATTERN from being sent to the Data Domain system.
--ddr-exempt-from=FILE, --ddrblock_from Prevent files that match patterns listed in FILE from being sent to the Data Domain system.
--deflateofficexml Deflate office XML files before restore. [True]
--dereference, -h Include all files pointed to by symbolic links.
--destination-data=NAME Replication data destination storage space. (same, ddr, ads). same
--directory=PATH, -C Change to this directory before the backup or restore.
--exclude=PATTERN Exclude files that match the PATTERN from a backup or restore.
--exclude-from=FILE, -X, --exclude_from Specify patterns in FILE. Files that match the patterns are excluded from the backup or restore.
--exclude-pluginid-list=STRING, --excludepluginids Excludes backups originally taken with one or more plug-ins listed in STRING.
--exclude_cachefile Exclude cache files from backup. [True]
--existing-dir-aclrestore, --aclrestore Replace security settings on pre-existing directories during a restore.
--existing-file-overwrite-option=SETTING, --overwrite Specify when to overwrite existing files on restore where SETTING is always, modified, never, newest, or newname. [never]
--expires=DAYS or TIMESTAMP or STRING Backup expiration date as a number of days (N or DAYS=N), weeks (WEEKS=N), months (MONTHS=N), years (YEARS=N), or an absolute timestamp from the date the backup finishes.
--flagfile=FILE Specify a file that contains a list of options.
--force Force traversal of network file system (NFS) and loopback file system (LOFS).
--forcefs=FSLIST Force traversal of the file system types specified in FSLIST.
--from-stdin Get backup stream from standard input (stdin) only.
--hbe-timeout=SECONDS Terminate HBE if it does not complete in the given time.
--help, --he, --hel Print this message.
--id=USER@DOMAIN/HOMEACNT Specify the authentication user ID.
--ignorefs=FSLIST Do not traverse the specified file system types.
--include=PATTERN Include files that match the PATTERN.
--include-from=FILE, --include_from Include files that match the patterns listed in FILE.
--informationals Set informational level.
--label=LABEL, -f Specify backup label.
--libavctl_path=PATH, --libavctl-path Load the shared library, libavctl. Include the file extension.
--logfile=FILE, --log Specify the full path and file name of the log file.
--logtimeformat=FORMAT, --logfmt Specify date format for logs (default, iso). default
--logtimezone=ZONE, --logzone Specify the time zone for log timestamps (default, local, gmt/utc). default
--max-entries=N, --maxentries Specify the maximum entries to display in a listing.
--media-in=NAME Read data from the media specified by this XML stream description.
--media-in-size=N Specifies the default size of the data read from --media-in [262144]
--media-out=NAME Write data to the media specified by this XML stream description.
--no-recursion Do not automatically descend into directories during a backup or restore.
--noinformationals Disable all status messages.
--nostdout Disable output to standard output (stdout).
--nowarnings Disable warning messages.
--one-file-system, -l Only backup files located on the targeted file system or volume.
--onlinefsrestore Indicate an online file system restore. (Homebase only).
--open-file-restore-option=SETTING Specify how to restore open files. Values for SETTING are deferred, never, newest, or newname. (Windows only.) [never]
--operation=TYPE, --op Specify the operation to perform. Values for TYPE are browse, backup, or restore.
--paging-cache Enable paging cache for large file systems or databases.
--paging-cache-limit=% Set the paging cache as a percentage of the filesystem. [2]
--paging-cache-limit-enable Enable limitation in the size of the paging cache.
--password=PSWD, --ap, --pswd Specify the authentication password for the Avamar users.
--pathmap=STRING Defines mapped paths.
--pausetimeout=SECS Specify the number of seconds to pause before automatically resuming a backup. [1800]
--pluginid-list=STRING, --pluginids Specify the Avamar plug-in ID list.
--preserve-hardlinks Preserve hard links during backup.
--preservepaths Retain original directory structure during a restore.
--protect-profile=NAME, --protect-system Protect the system profile using HomeBase Embedded. [disabled]
--quiet Suppress informational and warning messages.
--repaircache Verify client side hashes on the Avamar server, and then remove invalid ones.
--reparse-limit=N, --reparse_limit Specify the maximum number of levels to traverse in the NTFS reparse points. [1]
--reparse-restore-option=N Indicates how to restore reparse points: linksandtargets, linksonly, linksasfilesfolders [linksandtargets]
--restore-acls-only Restore only ACLs to existing file on disk.
--restore-destination=TYPE, --restore-option Specify the type of restore to perform. Values for TYPE are none, original, single, or multiple. none
--restore-interceptor-check Check for interceptors during restore. (Windows only.)
--restore-profile=NAME, --restore-system Restore a HomeBase Embedded profile on the system. [disabled]
--restore-sparse-threshold=KB Specify the threshold size in KB to use to create sparse regions during a restore. [128]
--restoreatreboot Enable restore for Windows-protected files during reboot processing.
--restorehidden Restore files that have the hidden attribute set. [True]
--restoreshortnames Attempt to set short (8.3) names during restore.(Windows only.)
--restoresystem Restore files that have the system attribute set.
--restorewfp Attempt to restore files under Windows File Protection (WFP).
--retention-type=TYPE, --retentiontype Specify a retention type: none, daily, weekly, monthly, or yearly.
--run-after-freeze=SCRIPT, --run_after_freeze Run this script after the volume is locked. (Backups only.)
--run-after-freeze-clauses=STRING, --run_after_freeze_clauses Use these clauses (STRING) to start the run-after-freeze script. [desc=run-after-freeze]
--run-at-end=SCRIPT, --run_at_end Run this script after avtar finishes.
--run-at-end-clauses=STRING, --run_at_end_clauses Use these clauses (STRING) to start the run-at-end script. [desc=run-at-end]
--run-at-start=SCRIPT, --run_at_start Run this script when avtar starts.
--run-at-start-clauses=STRING, --run_at_start_clauses Use these clauses (STRING) to start the run-at-start script. [desc=run-at-start]
--saveset-id=NAME Specify the savesetid of the backup.
--sequencenumber=N, --labelnumber Specify backup sequence number.
--server=IPADDR, --hfsaddr Specify the Avamar server IP address or fully qualified hostname. (HFSADDR)
--showlog, --showcomment Show session log for the selected backup.
--skip-high-latency Skip high-latency (offline) files during backup.
--streamformat=N Backup/Restore data from/to a stream of this format [none]
--streamformat-in=N Specifies the format of the data read from --media-in [none]
--streamformat-out=N Spedifies the format of the data to write to --media-out [none]
--system-profile-path=PATH Path where the HomeBase Embedded profile will be saved.
--systemstate-timeout=N Specify Windows System State process timeout. [3600]
--systemstatefile=FILE Name of Windows System State file to create.
--target=PATH Specify the directory for the restored backup.
--testwfp Test files protected with Windows File Protection. (WFP)
--to-stdout, -O Extract files to standard output (stdout) only.
--totals Show total size of backup at end of listing (list command only)
--tryagain Wait and try again later. [True]
--usage Print a list of available flags. Flag descriptions are not included.
--verbose, -v Enable all status and warning messages. Set verboseness level by specifying -verbose=N, where N is the level of verbosity.
--version Display build version.
--windows-optimized-backup, --win-opt Perform a Windows Optimized backup (Windows8 or higher)
--windows-rehydrate-limit=N The maximium number of Windows Deduplicated files to re-hydrate during a restore. Default of 500. (Windows8 or higher) [500]
--windows-restore-hydrated Restore Windows Deduplicated Files in 'Rehydrated' form (Windows8 or higher)
--within=DAYS Process backups that were created within the days specified.

Useful scripts for MCCLI server


Below sctipt restores files from first VM backup taken 2 days ego and restores to C:/_Restore folder

#!/bin/bash

searchdate=$(date --date='2 day ago' +%Y%m%d)
domain=/vcentre.virtworld.link/VirtualMachines
avamarserver=grid1.virtworld.link
avamaruser=backup_user
avamarpass=password123
vm=Win_Test
vmuser=backup
vmpass=vmpass123
vmfilesource=C:/Users/Administrator/Downloads/AvamarDB2-solaris10-sparc64-7.1.102-21.pkg
vmfiledest=C:/_Restore

echo "Search date: " $searchdate

arr=($(/usr/local/avamar/7.5.0-183/bin/mccli backup show --domain=$domain --name=$vm --mcsaddr=$avamarserver --mcsuserid=$avamaruser --mcspasswd=$avamarpass | tail -n +4 | awk '{print $1" "$4}'))

for (( i=0; i<${#arr[@]}; i=i+2));
do
datearr=$(date -d ${arr[$i]} +%Y%m%d)
if [ $datearr -eq $searchdate ];
then
echo "Backup Date: " $(date -d ${arr[$i]} +%Y%m%d)
echo "Backup Label: " ${arr[$i+1]}
restorecmd="/usr/local/avamar/7.5.0-183/bin/mccli backup restore --domain=$domain --name=$vm --labelNum="${arr[$i+1]}" --plugin=3019 --restore-vm-to=flr --dest-client-username="$vmuser" --dest-client-password="$vmpass" --data="$vmfilesource" --dest-dir="$vmfiledest" --mcsaddr="$avamarserver" --mcsuserid="$avamaruser" --mcspasswd="$avamarpass
$restorecmd
nobackup=false
break
fi
done

if [ "$nobackup" == "" ];
then
echo "No backup found for: "$vm
else
echo "Restore requested."
fi

Avamar Exchange VSS plugin - backup failing with error: avtar Error 7042: Failed to create snapview


Exchange Backup Failures

Exchange VSS plug-in DAG or individual Exchange node backup is failing (Failed to create snapview):

Avamar Exchange VSS plugin - backup failing with error: avtar Error <7042>: Failed to create snapview

2015-01-21 09:37:21 avtar Info <7209>: Finished reading makeview input
2015-01-21 09:37:21 avtar Info <7210>: Finished generating makeview tables
2015-01-21 09:37:21 avtar Info <16522>: Starting copy of paths in account '/clients/XXXXXXXX' label number 3
2015-01-21 09:37:21 avtar Error <7042>: Failed to create snapview, as one of the elements required for path "XXXXXXXXX.edb" not found in backup account '/XXXXXX' lablenum 3 (only found 'XXXX\E:' of path 'XXXX.edb').
2015-01-21 09:37:21 avtar Info <9772>: Starting graceful (staged) termination, unable to find paths in snapview (wrap-up stage)
2015-01-21 09:37:21 avtar Info <0000>: Entering the 'final' phase of termination, unable to find paths in snapview)
2015-01-21 09:37:21 avtar FATAL <5155>: Backup aborted due to earlier errors. No backup created on the server.
2015-01-21 09:37:21 avtar Info <7883>: Finished at 2015-01-21 09:37:21 Eastern Standard Time, Elapsed time: 0000h:00m:01s
2015-01-21 09:37:21 avtar Info <6149>: Error summary: 2 errors: 5155, 7042

This is a case-mismatch of Exchange Databases between two subsequent backups.
For example : If during Backup1, the database directory of the database was "C:\mailbox" and gets changed to "C:\MAILBOX", the next subsequent backup will fail with the above error message.

Resolution is to disable the use of case-sensitive paths

Setting the following flag inside "avexvss.cmd" filen in local VAR directory of all Exchange Servers as well as in shared VAR folder of the DAG client:

--use-case-sensitive-paths=false

If the above flag does not resolve the issue, then try backing up each Exchange database individually. The error happens when the snapview avtar process tries to consolidate the backups of each of the DBs into one single label number.

Popular