21 December 2015

Upgrade 10 SPARC LDoms to Oracle Solaris 11.3 in 20 Minutes

Basics first
Oracle Solaris 11 includes a powerful Upgrade Feature based on IPS (Image Packaging System)
and ZFS. When Upgrading the Operating System a new Boot Environment is created and updated.
No impact on the currently running System. The new Environment is activated using a reboot. This allows to failback to the old Boot Environment if Applications don't run as expected on the new Solaris Release.

Find more about Upgrading in the Oracle Solaris 11.3 Manual
https://docs.oracle.com/cd/E53394_01/html/E54739/pkgupdate.html#scrolltoc

Upgrading using VDCF
The VDCF Management Framework for Oracle Solaris includes a feature to upgrade multiple Nodes in parallel using one command. The operation to use is "node -c upgrade name=nodeA,nodeB build=targetSRU". This upgrade operation does not only upgrade the OS to a predefined build, it also first checks if enough disk space is available. After a successful OS Upgrade the VDCF Repository is updated with the new Boot Environment and OS Update.

Sample of Upgrading 10 LDoms in parallel
The VDCF Framework is installed in a small LDom running on a SPARC T4-1 Server. The 10 Target LDoms are running Solaris 11.2 on a SPARC T4-2 System. Some of the LDoms host Solaris Zones,
which are also upgraded in parallel.



VDCF uses the Oracle Solaris pkg command with the following options
-C 5                                 to update the Zones in parallel
--be-name s11.3.3.0.6.0              to set a meaningful BootEnv Name
entire@0.5.11,5.11-0.175.3.3.0.6.0   to select a specific Solaris SRU


Upgrade Output
-bash-4.1$ time node -c upgrade reboot force build=s11u3-sru3 name=g0042,g0068,g0070,g0084,g0092,g0095,g0096,g0097,g0102,g0103
Node Upgrade for multiple Nodes started ...
doing a 'pkg update  -C 5 --accept --be-name s11.3.3.0.6.0 entire@0.5.11,5.11-0.175.3.3.0.6.0' now ...
 Startup: Refreshing catalog 'jomasoft' ... Done
 Startup: Refreshing catalog 'solaris' ... Done
Planning: Solver setup ... Done
Planning: Running solver ... Done
Planning: Finding local manifests ... Done
Planning: Package planning ... Done
Planning: Merging actions ... Done
Planning: Checking for conflicting actions ... Done
Planning: Consolidating action changes ... Done
Planning: Evaluating mediators ... Done
Planning: Planning completed in 71.79 seconds
------------------------------------------------------------
Package: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.175.3.3.0.5.0:20151209T174724Z
License: lic_OTN

You acknowledge that your use of this Oracle Solaris software product
is subject to (i) the license terms that you accepted when you obtained
the right to use Oracle Solaris software; or (ii) the license terms that
you agreed to when you placed your Oracle Solaris software order with
Oracle; or (iii) the Oracle Solaris software license terms included with
the hardware that you acquired from Oracle; or, if (i), (ii) or (iii)
are not applicable, then, (iv) the OTN License Agreement for Oracle
Solaris (which you acknowledge you have read and agree to) available at
http://www.oracle.com/technetwork/licenses/solaris-cluster-express-license-167852.html.
Note: Software downloaded for trial use or downloaded as replacement
media may not be used to update any unsupported software.



            Packages to remove:  28
           Packages to install:  82
            Packages to update: 403
           Mediators to change:   6
       Create boot environment: Yes
Create backup boot environment:  No
Download:     0/21108 items    0.0/510.5MB  0% complete
Download:  1076/21108 items   79.0/510.5MB  15% complete (15.8M/s)
Download:  2761/21108 items  116.8/510.5MB  22% complete (11.6M/s)
Download:  4915/21108 items  152.6/510.5MB  29% complete (7.2M/s)
Download:  5998/21108 items  279.5/510.5MB  54% complete (16.1M/s)
Download:  8060/21108 items  293.4/510.5MB  57% complete (14.2M/s)
Download: 10200/21108 items  316.2/510.5MB  61% complete (3.7M/s)
Download: 12273/21108 items  334.7/510.5MB  65% complete (4.1M/s)
Download: 14134/21108 items  371.7/510.5MB  72% complete (5.5M/s)
Download: 15502/21108 items  421.0/510.5MB  82% complete (8.6M/s)
Download: 17555/21108 items  434.2/510.5MB  85% complete (6.2M/s)
Download: 19566/21108 items  476.6/510.5MB  93% complete (5.5M/s)
Download: Completed 510.50 MB in 58.96 seconds (8.6M/s)
 Actions:     1/33566 actions (Removing old actions)
 Actions:  5284/33566 actions (Installing new actions)
 Actions: 12840/33566 actions (Installing new actions)
 Actions: 14860/33566 actions (Installing new actions)
 Actions: 16981/33566 actions (Installing new actions)
 Actions: 18796/33566 actions (Installing new actions)
 Actions: 20043/33566 actions (Installing new actions)
 Actions: 20175/33566 actions (Installing new actions)
 Actions: 20763/33566 actions (Updating modified actions)
 Actions: 22038/33566 actions (Updating modified actions)
 Actions: 22681/33566 actions (Updating modified actions)
 Actions: 23478/33566 actions (Updating modified actions)
 Actions: 25056/33566 actions (Updating modified actions)
 Actions: 27103/33566 actions (Updating modified actions)
 Actions: 27951/33566 actions (Updating modified actions)
 Actions: 28741/33566 actions (Updating modified actions)
 Actions: 29399/33566 actions (Updating modified actions)
 Actions: 30202/33566 actions (Updating modified actions)
 Actions: 30751/33566 actions (Updating modified actions)
 Actions: 30954/33566 actions (Updating modified actions)
 Actions: 31219/33566 actions (Updating modified actions)
 Actions: 31772/33566 actions (Updating modified actions)
 Actions: 32463/33566 actions (Updating modified actions)
 Actions: 32885/33566 actions (Updating modified actions)
 Actions: Completed 33566 actions in 112.43 seconds.
 Done
 Done
 Done
 Done
 Done

A clone of s11.2.15.0.4.0 exists and has been updated and activated.
On the next boot the Boot Environment s11.3.3.0.6.0 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

 Done
BE             Flags Mountpoint Space  Policy Created
--             ----- ---------- -----  ------ -------
s11.2.15.0.4.0 N     /          142.0K static 2015-12-14 15:27
s11.3.3.0.6.0  R     -          9.61G  static 2015-12-21 10:43
GDom g0102 updated to entire@0.5.11,5.11-0.175.3.3.0.6.0.
GDom g0102 is rebooting now.



<snipp / other 8 Guest Domains Output removed>



WARN:  There are 3 active vServer(s) on Node g0097.
doing a 'pkg update  -C 5 --accept --be-name s11.3.3.0.6.0 entire@0.5.11,5.11-0.175.3.3.0.6.0' now ...
 Startup: Linked image publisher check ... Done
 Startup: Refreshing catalog 'jomasoft' ... Done
 Startup: Refreshing catalog 'solaris' ... Done
Planning: Solver setup ... Done
Planning: Running solver ... Done
Planning: Finding local manifests ... Done
Planning: Package planning ... Done
Planning: Merging actions ... Done
Planning: Checking for conflicting actions ... Done
Planning: Consolidating action changes ... Done
Planning: Evaluating mediators ... Done
Planning: Planning completed in 62.17 seconds
------------------------------------------------------------
Package: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.175.3.3.0.5.0:20151209T174724Z
License: lic_OTN

You acknowledge that your use of this Oracle Solaris software product
is subject to (i) the license terms that you accepted when you obtained
the right to use Oracle Solaris software; or (ii) the license terms that
you agreed to when you placed your Oracle Solaris software order with
Oracle; or (iii) the Oracle Solaris software license terms included with
the hardware that you acquired from Oracle; or, if (i), (ii) or (iii)
are not applicable, then, (iv) the OTN License Agreement for Oracle
Solaris (which you acknowledge you have read and agree to) available at
http://www.oracle.com/technetwork/licenses/solaris-cluster-express-license-167852.html.
Note: Software downloaded for trial use or downloaded as replacement
media may not be used to update any unsupported software.



            Packages to remove:  28
           Packages to install:  82
            Packages to update: 403
           Mediators to change:   6
       Create boot environment: Yes
Create backup boot environment:  No
Planning: Linked images: 0/3 done; 3 working: zone:v0127 zone:v0138 zone:v0158
Planning: Linked image 'zone:v0158' output:
|  Packages to remove:  27
| Packages to install:  72
|  Packages to update: 233
| Mediators to change:   3
|  Services to change:   9
`
Planning: Linked images: 1/3 done; 2 working: zone:v0127 zone:v0138
Planning: Linked image 'zone:v0127' output:
|  Packages to remove:  27
| Packages to install:  72
|  Packages to update: 233
| Mediators to change:   3
|  Services to change:   9
`
Planning: Linked images: 2/3 done; 1 working: zone:v0138
Planning: Linked image 'zone:v0138' output:
|  Packages to remove:  27
| Packages to install:  72
|  Packages to update: 233
| Mediators to change:   3
|  Services to change:   9
`
Planning: Finished processing linked images.
Download:     0/21108 items    0.0/510.5MB  0% complete
Download:    19/21108 items    0.4/510.5MB  0% complete (17.5M/s)
Download:   185/21108 items   22.0/510.5MB  4% complete (2.7M/s)
Download:   457/21108 items   35.5/510.5MB  6% complete (3.7M/s)
Download:   478/21108 items   35.7/510.5MB  6% complete (3.3M/s)
Download:   592/21108 items   58.7/510.5MB  11% complete (2.4M/s)
Download:   667/21108 items   63.6/510.5MB  12% complete (3.0M/s)
Download:   708/21108 items   64.6/510.5MB  12% complete (1.5M/s)
Download:   747/21108 items   66.5/510.5MB  13% complete (299k/s)
Download:   776/21108 items   67.2/510.5MB  13% complete (297k/s)
Download:   826/21108 items   68.7/510.5MB  13% complete (258k/s)
Download:   864/21108 items   69.6/510.5MB  13% complete (247k/s)
Download:   879/21108 items   69.7/510.5MB  13% complete (208k/s)
Download:   903/21108 items   69.8/510.5MB  13% complete (146k/s)
Download:   930/21108 items   70.3/510.5MB  13% complete (82.2k/s)
Download:   947/21108 items   70.9/510.5MB  13% complete (83.5k/s)
Download:   999/21108 items   74.3/510.5MB  14% complete (406k/s)
Download:  1107/21108 items   80.4/510.5MB  15% complete (1016k/s)
Download:  1167/21108 items   82.3/510.5MB  16% complete (949k/s)
Download:  1258/21108 items   82.9/510.5MB  16% complete (264k/s)
Download:  1352/21108 items   84.3/510.5MB  16% complete (173k/s)
Download:  1598/21108 items   92.6/510.5MB  18% complete (1.2M/s)
Download:  1738/21108 items   97.2/510.5MB  19% complete (1.4M/s)
Download:  1782/21108 items   98.6/510.5MB  19% complete (1017k/s)
Download:  1848/21108 items  102.1/510.5MB  20% complete (499k/s)
Download:  1912/21108 items  106.8/510.5MB  20% complete (857k/s)
Download:  1943/21108 items  107.4/510.5MB  21% complete (672k/s)
Download:  2051/21108 items  108.6/510.5MB  21% complete (175k/s)
Download:  2118/21108 items  109.6/510.5MB  21% complete (207k/s)
Download:  2149/21108 items  110.5/510.5MB  21% complete (163k/s)
Download:  2347/21108 items  114.4/510.5MB  22% complete (565k/s)
Download:  2739/21108 items  116.7/510.5MB  22% complete (622k/s)
Download:  3137/21108 items  118.4/510.5MB  23% complete (323k/s)
Download:  3336/21108 items  119.5/510.5MB  23% complete (236k/s)
Download:  3648/21108 items  120.6/510.5MB  23% complete (160k/s)
Download:  3902/21108 items  126.2/510.5MB  24% complete (610k/s)
Download:  3940/21108 items  129.2/510.5MB  25% complete (848k/s)
Download:  4014/21108 items  132.0/510.5MB  25% complete (726k/s)
Download:  4060/21108 items  135.5/510.5MB  26% complete (625k/s)
Download:  4260/21108 items  139.7/510.5MB  27% complete (789k/s)
Download:  4765/21108 items  145.4/510.5MB  28% complete (1018k/s)
Download:  5005/21108 items  217.1/510.5MB  42% complete (7.8M/s)
Download:  5273/21108 items  271.9/510.5MB  53% complete (12.9M/s)
Download:  5998/21108 items  279.5/510.5MB  54% complete (6.2M/s)
Download:  6708/21108 items  285.3/510.5MB  55% complete (1.4M/s)
Download:  7226/21108 items  289.4/510.5MB  56% complete (1016k/s)
Download:  7766/21108 items  292.4/510.5MB  57% complete (727k/s)
Download:  8599/21108 items  295.6/510.5MB  57% complete (659k/s)
Download:  9222/21108 items  298.9/510.5MB  58% complete (628k/s)
Download:  9429/21108 items  302.4/510.5MB  59% complete (675k/s)
Download:  9584/21108 items  307.9/510.5MB  60% complete (900k/s)
Download:  9891/21108 items  314.6/510.5MB  61% complete (1.2M/s)
Download: 10242/21108 items  317.0/510.5MB  62% complete (1020k/s)
Download: 10590/21108 items  319.2/510.5MB  62% complete (463k/s)
Download: 10860/21108 items  321.2/510.5MB  62% complete (428k/s)
Download: 11202/21108 items  324.8/510.5MB  63% complete (560k/s)
Download: 11868/21108 items  331.4/510.5MB  64% complete (1.0M/s)
Download: 12389/21108 items  335.7/510.5MB  65% complete (1.1M/s)
Download: 12990/21108 items  348.8/510.5MB  68% complete (1.7M/s)
Download: 13379/21108 items  358.4/510.5MB  70% complete (2.3M/s)
Download: 14185/21108 items  372.9/510.5MB  73% complete (2.4M/s)
Download: 15042/21108 items  406.5/510.5MB  79% complete (4.9M/s)
Download: 15609/21108 items  421.9/510.5MB  82% complete (4.9M/s)
Download: 16793/21108 items  429.3/510.5MB  84% complete (2.3M/s)
Download: 18063/21108 items  438.9/510.5MB  85% complete (1.7M/s)
Download: 18600/21108 items  447.5/510.5MB  87% complete (1.9M/s)
Download: 19184/21108 items  464.0/510.5MB  90% complete (2.4M/s)
Download: 19963/21108 items  486.1/510.5MB  95% complete (3.8M/s)
Download: 20850/21108 items  508.1/510.5MB  99% complete (4.4M/s)
Download: Completed 510.50 MB in 370.95 seconds (1.4M/s)
Download: Linked images: 0/3 done; 3 working: zone:v0127 zone:v0138 zone:v0158
Download: Linked images: 1/3 done; 2 working: zone:v0127 zone:v0158
Download: Linked images: 2/3 done; 1 working: zone:v0127
Download: Finished processing linked images.
 Actions:     1/33566 actions (Removing old actions)
 Actions:  5284/33566 actions (Installing new actions)
 Actions: 11908/33566 actions (Installing new actions)
 Actions: 12753/33566 actions (Installing new actions)
 Actions: 14587/33566 actions (Installing new actions)
 Actions: 15854/33566 actions (Installing new actions)
 Actions: 16867/33566 actions (Installing new actions)
 Actions: 17664/33566 actions (Installing new actions)
 Actions: 18794/33566 actions (Installing new actions)
 Actions: 19698/33566 actions (Installing new actions)
 Actions: 20175/33566 actions (Installing new actions)
 Actions: 20763/33566 actions (Updating modified actions)
 Actions: 21821/33566 actions (Updating modified actions)
 Actions: 22152/33566 actions (Updating modified actions)
 Actions: 22597/33566 actions (Updating modified actions)
 Actions: 23185/33566 actions (Updating modified actions)
 Actions: 23756/33566 actions (Updating modified actions)
 Actions: 24160/33566 actions (Updating modified actions)
 Actions: 24830/33566 actions (Updating modified actions)
 Actions: 25335/33566 actions (Updating modified actions)
 Actions: 25977/33566 actions (Updating modified actions)
 Actions: 27002/33566 actions (Updating modified actions)
 Actions: 27400/33566 actions (Updating modified actions)
 Actions: 27717/33566 actions (Updating modified actions)
 Actions: 28130/33566 actions (Updating modified actions)
 Actions: 28588/33566 actions (Updating modified actions)
 Actions: 29018/33566 actions (Updating modified actions)
 Actions: 29846/33566 actions (Updating modified actions)
 Actions: 30304/33566 actions (Updating modified actions)
 Actions: 30698/33566 actions (Updating modified actions)
 Actions: 30944/33566 actions (Updating modified actions)
 Actions: 31099/33566 actions (Updating modified actions)
 Actions: 31473/33566 actions (Updating modified actions)
 Actions: 31904/33566 actions (Updating modified actions)
 Actions: 32435/33566 actions (Updating modified actions)
 Actions: 32878/33566 actions (Updating modified actions)
 Actions: Completed 33566 actions in 190.06 seconds.
 Done
 Done
 Done
 Done
Finalize: Linked images: 0/3 done; 3 working: zone:v0127 zone:v0138 zone:v0158
Finalize: Linked images: 1/3 done; 2 working: zone:v0138 zone:v0158
Finalize: Linked images: 2/3 done; 1 working: zone:v0158
Finalize: Finished processing linked images.
 Done

A clone of s11.2.15.0.4.0 exists and has been updated and activated.
On the next boot the Boot Environment s11.3.3.0.6.0 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

 Done
BE             Flags Mountpoint Space  Policy Created
--             ----- ---------- -----  ------ -------
s11.2.15.0.4.0 N     /          85.25M static 2015-12-14 15:31
s11.3.3.0.6.0  R     -          12.15G static 2015-12-21 10:51
GDom g0097 updated to entire@0.5.11,5.11-0.175.3.3.0.6.0.
GDom g0097 is rebooting now.

Node Upgrade successful (10) for :  g0102 g0103 g0042 g0096 g0068 g0092 g0095 g0084 g0070 g0097

real    20m10.390s
user    1m15.991s
sys     2m19.544s



CPU Usage

The CPU Usage on the T4-1 Management LDom "g0076" increases to 20% while Upgrading.


-bash-4.1$ rcmon -c show cpu node=g0076 from="2015-12-21 10:55" to="2015-12-21 11:07"
                      ---------- Pool ----------   --- CpuShr --   --- CpuSys --   --- CpuUsr --   --- CpuAll --   -- CpuSAll --
DateTime              ID/Type  Max    Cur    All   Min /Avg /Max   Min /Avg /Max   Min /Avg /Max   Min /Avg /Max   Min /Avg /Max   Name
2015-12-21 10:55:04   00/dflt  65536  8      100%  -    100% -     -    0.7% -     -    0.2% -     -    0.9% -     -    0.9% -     g0076
2015-12-21 10:56:04   00/dflt  65536  8      100%  -    100% -     -    6.8% -     -    4.8% -     -    12%  -     -    12%  -     g0076
2015-12-21 10:57:04   00/dflt  65536  8      100%  -    100% -     -    5.1% -     -    7.4% -     -    12%  -     -    12%  -     g0076
2015-12-21 10:58:04   00/dflt  65536  8      100%  -    100% -     -    6.4% -     -    12%  -     -    19%  -     -    19%  -     g0076
2015-12-21 10:59:04   00/dflt  65536  8      100%  -    100% -     -    6.2% -     -    12%  -     -    18%  -     -    18%  -     g0076
2015-12-21 11:00:04   00/dflt  65536  8      100%  -    100% -     -    5.9% -     -    12%  -     -    18%  -     -    18%  -     g0076
2015-12-21 11:01:04   00/dflt  65536  8      100%  -    100% -     -    6.4% -     -    13%  -     -    19%  -     -    19%  -     g0076
2015-12-21 11:02:04   00/dflt  65536  8      100%  -    100% -     -    8.4% -     -    14%  -     -    22%  -     -    22%  -     g0076
2015-12-21 11:03:04   00/dflt  65536  8      100%  -    100% -     -    6.0% -     -    13%  -     -    19%  -     -    19%  -     g0076
2015-12-21 11:04:04   00/dflt  65536  8      100%  -    100% -     -    7.3% -     -    12%  -     -    20%  -     -    20%  -     g0076
2015-12-21 11:05:04   00/dflt  65536  8      100%  -    100% -     -    2.4% -     -    3.2% -     -    5.6% -     -    5.6% -     g0076
2015-12-21 11:06:04   00/dflt  65536  8      100%  -    100% -     -    1.8% -     -    1.7% -     -    3.5% -     -    3.5% -     g0076



Summary
This blog post shows how to Upgrade multiple Oracle Solaris LDoms to Solaris 11.3.
Using the VDCF Management Framework 10 SPARC LDoms are done in 20 Minutes.
The T4-Hardware is 4 years old. Just imagine how fast such Upgrades will run on the current SPARC M7 Systems!