Basics first
Oracle Solaris 11 includes a powerful Upgrade Feature based on IPS (Image Packaging System)
and ZFS. When Upgrading the Operating System a new Boot Environment is created and updated.
No impact on the currently running System. The new Environment is activated using a reboot. This allows to failback to the old Boot Environment if Applications don't run as expected on the new Solaris Release.
Find more about Upgrading in the Oracle Solaris 11.3 Manual
https://docs.oracle.com/cd/E53394_01/html/E54739/pkgupdate.html#scrolltoc
Upgrading using VDCF
The VDCF Management Framework for Oracle Solaris includes a feature to upgrade multiple Nodes in parallel using one command. The operation to use is "node -c upgrade name=nodeA,nodeB build=targetSRU". This upgrade operation does not only upgrade the OS to a predefined build, it also first checks if enough disk space is available. After a successful OS Upgrade the VDCF Repository is updated with the new Boot Environment and OS Update.
Sample of Upgrading 10 LDoms in parallel
The VDCF Framework is installed in a small LDom running on a SPARC T4-1 Server. The 10 Target LDoms are running Solaris 11.2 on a SPARC T4-2 System. Some of the LDoms host Solaris Zones,
which are also upgraded in parallel.
VDCF uses the Oracle Solaris pkg command with the following options
-C 5 to update the Zones in parallel
--be-name s11.3.3.0.6.0 to set a meaningful BootEnv Name
entire@0.5.11,5.11-0.175.3.3.0.6.0 to select a specific Solaris SRU
Upgrade Output
-bash-4.1$ time node -c upgrade reboot force build=s11u3-sru3 name=g0042,g0068,g0070,g0084,g0092,g0095,g0096,g0097,g0102,g0103
Node Upgrade for multiple Nodes started ...
doing a 'pkg update -C 5 --accept --be-name s11.3.3.0.6.0 entire@0.5.11,5.11-0.175.3.3.0.6.0' now ...
Startup: Refreshing catalog 'jomasoft' ... Done
Startup: Refreshing catalog 'solaris' ... Done
Planning: Solver setup ... Done
Planning: Running solver ... Done
Planning: Finding local manifests ... Done
Planning: Package planning ... Done
Planning: Merging actions ... Done
Planning: Checking for conflicting actions ... Done
Planning: Consolidating action changes ... Done
Planning: Evaluating mediators ... Done
Planning: Planning completed in 71.79 seconds
------------------------------------------------------------
Package: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.175.3.3.0.5.0:20151209T174724Z
License: lic_OTN
You acknowledge that your use of this Oracle Solaris software product
is subject to (i) the license terms that you accepted when you obtained
the right to use Oracle Solaris software; or (ii) the license terms that
you agreed to when you placed your Oracle Solaris software order with
Oracle; or (iii) the Oracle Solaris software license terms included with
the hardware that you acquired from Oracle; or, if (i), (ii) or (iii)
are not applicable, then, (iv) the OTN License Agreement for Oracle
Solaris (which you acknowledge you have read and agree to) available at
http://www.oracle.com/technetwork/licenses/solaris-cluster-express-license-167852.html.
Note: Software downloaded for trial use or downloaded as replacement
media may not be used to update any unsupported software.
Packages to remove: 28
Packages to install: 82
Packages to update: 403
Mediators to change: 6
Create boot environment: Yes
Create backup boot environment: No
Download: 0/21108 items 0.0/510.5MB 0% complete
Download: 1076/21108 items 79.0/510.5MB 15% complete (15.8M/s)
Download: 2761/21108 items 116.8/510.5MB 22% complete (11.6M/s)
Download: 4915/21108 items 152.6/510.5MB 29% complete (7.2M/s)
Download: 5998/21108 items 279.5/510.5MB 54% complete (16.1M/s)
Download: 8060/21108 items 293.4/510.5MB 57% complete (14.2M/s)
Download: 10200/21108 items 316.2/510.5MB 61% complete (3.7M/s)
Download: 12273/21108 items 334.7/510.5MB 65% complete (4.1M/s)
Download: 14134/21108 items 371.7/510.5MB 72% complete (5.5M/s)
Download: 15502/21108 items 421.0/510.5MB 82% complete (8.6M/s)
Download: 17555/21108 items 434.2/510.5MB 85% complete (6.2M/s)
Download: 19566/21108 items 476.6/510.5MB 93% complete (5.5M/s)
Download: Completed 510.50 MB in 58.96 seconds (8.6M/s)
Actions: 1/33566 actions (Removing old actions)
Actions: 5284/33566 actions (Installing new actions)
Actions: 12840/33566 actions (Installing new actions)
Actions: 14860/33566 actions (Installing new actions)
Actions: 16981/33566 actions (Installing new actions)
Actions: 18796/33566 actions (Installing new actions)
Actions: 20043/33566 actions (Installing new actions)
Actions: 20175/33566 actions (Installing new actions)
Actions: 20763/33566 actions (Updating modified actions)
Actions: 22038/33566 actions (Updating modified actions)
Actions: 22681/33566 actions (Updating modified actions)
Actions: 23478/33566 actions (Updating modified actions)
Actions: 25056/33566 actions (Updating modified actions)
Actions: 27103/33566 actions (Updating modified actions)
Actions: 27951/33566 actions (Updating modified actions)
Actions: 28741/33566 actions (Updating modified actions)
Actions: 29399/33566 actions (Updating modified actions)
Actions: 30202/33566 actions (Updating modified actions)
Actions: 30751/33566 actions (Updating modified actions)
Actions: 30954/33566 actions (Updating modified actions)
Actions: 31219/33566 actions (Updating modified actions)
Actions: 31772/33566 actions (Updating modified actions)
Actions: 32463/33566 actions (Updating modified actions)
Actions: 32885/33566 actions (Updating modified actions)
Actions: Completed 33566 actions in 112.43 seconds.
Done
Done
Done
Done
Done
A clone of s11.2.15.0.4.0 exists and has been updated and activated.
On the next boot the Boot Environment s11.3.3.0.6.0 will be
mounted on '/'. Reboot when ready to switch to this updated BE.
Done
BE Flags Mountpoint Space Policy Created
-- ----- ---------- ----- ------ -------
s11.2.15.0.4.0 N / 142.0K static 2015-12-14 15:27
s11.3.3.0.6.0 R - 9.61G static 2015-12-21 10:43
GDom g0102 updated to entire@0.5.11,5.11-0.175.3.3.0.6.0.
GDom g0102 is rebooting now.
<snipp / other 8 Guest Domains Output removed>
WARN: There are 3 active vServer(s) on Node g0097.
doing a 'pkg update -C 5 --accept --be-name s11.3.3.0.6.0 entire@0.5.11,5.11-0.175.3.3.0.6.0' now ...
Startup: Linked image publisher check ... Done
Startup: Refreshing catalog 'jomasoft' ... Done
Startup: Refreshing catalog 'solaris' ... Done
Planning: Solver setup ... Done
Planning: Running solver ... Done
Planning: Finding local manifests ... Done
Planning: Package planning ... Done
Planning: Merging actions ... Done
Planning: Checking for conflicting actions ... Done
Planning: Consolidating action changes ... Done
Planning: Evaluating mediators ... Done
Planning: Planning completed in 62.17 seconds
------------------------------------------------------------
Package: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.175.3.3.0.5.0:20151209T174724Z
License: lic_OTN
You acknowledge that your use of this Oracle Solaris software product
is subject to (i) the license terms that you accepted when you obtained
the right to use Oracle Solaris software; or (ii) the license terms that
you agreed to when you placed your Oracle Solaris software order with
Oracle; or (iii) the Oracle Solaris software license terms included with
the hardware that you acquired from Oracle; or, if (i), (ii) or (iii)
are not applicable, then, (iv) the OTN License Agreement for Oracle
Solaris (which you acknowledge you have read and agree to) available at
http://www.oracle.com/technetwork/licenses/solaris-cluster-express-license-167852.html.
Note: Software downloaded for trial use or downloaded as replacement
media may not be used to update any unsupported software.
Packages to remove: 28
Packages to install: 82
Packages to update: 403
Mediators to change: 6
Create boot environment: Yes
Create backup boot environment: No
Planning: Linked images: 0/3 done; 3 working: zone:v0127 zone:v0138 zone:v0158
Planning: Linked image 'zone:v0158' output:
| Packages to remove: 27
| Packages to install: 72
| Packages to update: 233
| Mediators to change: 3
| Services to change: 9
`
Planning: Linked images: 1/3 done; 2 working: zone:v0127 zone:v0138
Planning: Linked image 'zone:v0127' output:
| Packages to remove: 27
| Packages to install: 72
| Packages to update: 233
| Mediators to change: 3
| Services to change: 9
`
Planning: Linked images: 2/3 done; 1 working: zone:v0138
Planning: Linked image 'zone:v0138' output:
| Packages to remove: 27
| Packages to install: 72
| Packages to update: 233
| Mediators to change: 3
| Services to change: 9
`
Planning: Finished processing linked images.
Download: 0/21108 items 0.0/510.5MB 0% complete
Download: 19/21108 items 0.4/510.5MB 0% complete (17.5M/s)
Download: 185/21108 items 22.0/510.5MB 4% complete (2.7M/s)
Download: 457/21108 items 35.5/510.5MB 6% complete (3.7M/s)
Download: 478/21108 items 35.7/510.5MB 6% complete (3.3M/s)
Download: 592/21108 items 58.7/510.5MB 11% complete (2.4M/s)
Download: 667/21108 items 63.6/510.5MB 12% complete (3.0M/s)
Download: 708/21108 items 64.6/510.5MB 12% complete (1.5M/s)
Download: 747/21108 items 66.5/510.5MB 13% complete (299k/s)
Download: 776/21108 items 67.2/510.5MB 13% complete (297k/s)
Download: 826/21108 items 68.7/510.5MB 13% complete (258k/s)
Download: 864/21108 items 69.6/510.5MB 13% complete (247k/s)
Download: 879/21108 items 69.7/510.5MB 13% complete (208k/s)
Download: 903/21108 items 69.8/510.5MB 13% complete (146k/s)
Download: 930/21108 items 70.3/510.5MB 13% complete (82.2k/s)
Download: 947/21108 items 70.9/510.5MB 13% complete (83.5k/s)
Download: 999/21108 items 74.3/510.5MB 14% complete (406k/s)
Download: 1107/21108 items 80.4/510.5MB 15% complete (1016k/s)
Download: 1167/21108 items 82.3/510.5MB 16% complete (949k/s)
Download: 1258/21108 items 82.9/510.5MB 16% complete (264k/s)
Download: 1352/21108 items 84.3/510.5MB 16% complete (173k/s)
Download: 1598/21108 items 92.6/510.5MB 18% complete (1.2M/s)
Download: 1738/21108 items 97.2/510.5MB 19% complete (1.4M/s)
Download: 1782/21108 items 98.6/510.5MB 19% complete (1017k/s)
Download: 1848/21108 items 102.1/510.5MB 20% complete (499k/s)
Download: 1912/21108 items 106.8/510.5MB 20% complete (857k/s)
Download: 1943/21108 items 107.4/510.5MB 21% complete (672k/s)
Download: 2051/21108 items 108.6/510.5MB 21% complete (175k/s)
Download: 2118/21108 items 109.6/510.5MB 21% complete (207k/s)
Download: 2149/21108 items 110.5/510.5MB 21% complete (163k/s)
Download: 2347/21108 items 114.4/510.5MB 22% complete (565k/s)
Download: 2739/21108 items 116.7/510.5MB 22% complete (622k/s)
Download: 3137/21108 items 118.4/510.5MB 23% complete (323k/s)
Download: 3336/21108 items 119.5/510.5MB 23% complete (236k/s)
Download: 3648/21108 items 120.6/510.5MB 23% complete (160k/s)
Download: 3902/21108 items 126.2/510.5MB 24% complete (610k/s)
Download: 3940/21108 items 129.2/510.5MB 25% complete (848k/s)
Download: 4014/21108 items 132.0/510.5MB 25% complete (726k/s)
Download: 4060/21108 items 135.5/510.5MB 26% complete (625k/s)
Download: 4260/21108 items 139.7/510.5MB 27% complete (789k/s)
Download: 4765/21108 items 145.4/510.5MB 28% complete (1018k/s)
Download: 5005/21108 items 217.1/510.5MB 42% complete (7.8M/s)
Download: 5273/21108 items 271.9/510.5MB 53% complete (12.9M/s)
Download: 5998/21108 items 279.5/510.5MB 54% complete (6.2M/s)
Download: 6708/21108 items 285.3/510.5MB 55% complete (1.4M/s)
Download: 7226/21108 items 289.4/510.5MB 56% complete (1016k/s)
Download: 7766/21108 items 292.4/510.5MB 57% complete (727k/s)
Download: 8599/21108 items 295.6/510.5MB 57% complete (659k/s)
Download: 9222/21108 items 298.9/510.5MB 58% complete (628k/s)
Download: 9429/21108 items 302.4/510.5MB 59% complete (675k/s)
Download: 9584/21108 items 307.9/510.5MB 60% complete (900k/s)
Download: 9891/21108 items 314.6/510.5MB 61% complete (1.2M/s)
Download: 10242/21108 items 317.0/510.5MB 62% complete (1020k/s)
Download: 10590/21108 items 319.2/510.5MB 62% complete (463k/s)
Download: 10860/21108 items 321.2/510.5MB 62% complete (428k/s)
Download: 11202/21108 items 324.8/510.5MB 63% complete (560k/s)
Download: 11868/21108 items 331.4/510.5MB 64% complete (1.0M/s)
Download: 12389/21108 items 335.7/510.5MB 65% complete (1.1M/s)
Download: 12990/21108 items 348.8/510.5MB 68% complete (1.7M/s)
Download: 13379/21108 items 358.4/510.5MB 70% complete (2.3M/s)
Download: 14185/21108 items 372.9/510.5MB 73% complete (2.4M/s)
Download: 15042/21108 items 406.5/510.5MB 79% complete (4.9M/s)
Download: 15609/21108 items 421.9/510.5MB 82% complete (4.9M/s)
Download: 16793/21108 items 429.3/510.5MB 84% complete (2.3M/s)
Download: 18063/21108 items 438.9/510.5MB 85% complete (1.7M/s)
Download: 18600/21108 items 447.5/510.5MB 87% complete (1.9M/s)
Download: 19184/21108 items 464.0/510.5MB 90% complete (2.4M/s)
Download: 19963/21108 items 486.1/510.5MB 95% complete (3.8M/s)
Download: 20850/21108 items 508.1/510.5MB 99% complete (4.4M/s)
Download: Completed 510.50 MB in 370.95 seconds (1.4M/s)
Download: Linked images: 0/3 done; 3 working: zone:v0127 zone:v0138 zone:v0158
Download: Linked images: 1/3 done; 2 working: zone:v0127 zone:v0158
Download: Linked images: 2/3 done; 1 working: zone:v0127
Download: Finished processing linked images.
Actions: 1/33566 actions (Removing old actions)
Actions: 5284/33566 actions (Installing new actions)
Actions: 11908/33566 actions (Installing new actions)
Actions: 12753/33566 actions (Installing new actions)
Actions: 14587/33566 actions (Installing new actions)
Actions: 15854/33566 actions (Installing new actions)
Actions: 16867/33566 actions (Installing new actions)
Actions: 17664/33566 actions (Installing new actions)
Actions: 18794/33566 actions (Installing new actions)
Actions: 19698/33566 actions (Installing new actions)
Actions: 20175/33566 actions (Installing new actions)
Actions: 20763/33566 actions (Updating modified actions)
Actions: 21821/33566 actions (Updating modified actions)
Actions: 22152/33566 actions (Updating modified actions)
Actions: 22597/33566 actions (Updating modified actions)
Actions: 23185/33566 actions (Updating modified actions)
Actions: 23756/33566 actions (Updating modified actions)
Actions: 24160/33566 actions (Updating modified actions)
Actions: 24830/33566 actions (Updating modified actions)
Actions: 25335/33566 actions (Updating modified actions)
Actions: 25977/33566 actions (Updating modified actions)
Actions: 27002/33566 actions (Updating modified actions)
Actions: 27400/33566 actions (Updating modified actions)
Actions: 27717/33566 actions (Updating modified actions)
Actions: 28130/33566 actions (Updating modified actions)
Actions: 28588/33566 actions (Updating modified actions)
Actions: 29018/33566 actions (Updating modified actions)
Actions: 29846/33566 actions (Updating modified actions)
Actions: 30304/33566 actions (Updating modified actions)
Actions: 30698/33566 actions (Updating modified actions)
Actions: 30944/33566 actions (Updating modified actions)
Actions: 31099/33566 actions (Updating modified actions)
Actions: 31473/33566 actions (Updating modified actions)
Actions: 31904/33566 actions (Updating modified actions)
Actions: 32435/33566 actions (Updating modified actions)
Actions: 32878/33566 actions (Updating modified actions)
Actions: Completed 33566 actions in 190.06 seconds.
Done
Done
Done
Done
Finalize: Linked images: 0/3 done; 3 working: zone:v0127 zone:v0138 zone:v0158
Finalize: Linked images: 1/3 done; 2 working: zone:v0138 zone:v0158
Finalize: Linked images: 2/3 done; 1 working: zone:v0158
Finalize: Finished processing linked images.
Done
A clone of s11.2.15.0.4.0 exists and has been updated and activated.
On the next boot the Boot Environment s11.3.3.0.6.0 will be
mounted on '/'. Reboot when ready to switch to this updated BE.
Done
BE Flags Mountpoint Space Policy Created
-- ----- ---------- ----- ------ -------
s11.2.15.0.4.0 N / 85.25M static 2015-12-14 15:31
s11.3.3.0.6.0 R - 12.15G static 2015-12-21 10:51
GDom g0097 updated to entire@0.5.11,5.11-0.175.3.3.0.6.0.
GDom g0097 is rebooting now.
Node Upgrade successful (10) for : g0102 g0103 g0042 g0096 g0068 g0092 g0095 g0084 g0070 g0097
real 20m10.390s
user 1m15.991s
sys 2m19.544s
CPU Usage
The CPU Usage on the T4-1 Management LDom "g0076" increases to 20% while Upgrading.
-bash-4.1$ rcmon -c show cpu node=g0076 from="2015-12-21 10:55" to="2015-12-21 11:07"
---------- Pool ---------- --- CpuShr -- --- CpuSys -- --- CpuUsr -- --- CpuAll -- -- CpuSAll --
DateTime ID/Type Max Cur All Min /Avg /Max Min /Avg /Max Min /Avg /Max Min /Avg /Max Min /Avg /Max Name
2015-12-21 10:55:04 00/dflt 65536 8 100% - 100% - - 0.7% - - 0.2% - - 0.9% - - 0.9% - g0076
2015-12-21 10:56:04 00/dflt 65536 8 100% - 100% - - 6.8% - - 4.8% - - 12% - - 12% - g0076
2015-12-21 10:57:04 00/dflt 65536 8 100% - 100% - - 5.1% - - 7.4% - - 12% - - 12% - g0076
2015-12-21 10:58:04 00/dflt 65536 8 100% - 100% - - 6.4% - - 12% - - 19% - - 19% - g0076
2015-12-21 10:59:04 00/dflt 65536 8 100% - 100% - - 6.2% - - 12% - - 18% - - 18% - g0076
2015-12-21 11:00:04 00/dflt 65536 8 100% - 100% - - 5.9% - - 12% - - 18% - - 18% - g0076
2015-12-21 11:01:04 00/dflt 65536 8 100% - 100% - - 6.4% - - 13% - - 19% - - 19% - g0076
2015-12-21 11:02:04 00/dflt 65536 8 100% - 100% - - 8.4% - - 14% - - 22% - - 22% - g0076
2015-12-21 11:03:04 00/dflt 65536 8 100% - 100% - - 6.0% - - 13% - - 19% - - 19% - g0076
2015-12-21 11:04:04 00/dflt 65536 8 100% - 100% - - 7.3% - - 12% - - 20% - - 20% - g0076
2015-12-21 11:05:04 00/dflt 65536 8 100% - 100% - - 2.4% - - 3.2% - - 5.6% - - 5.6% - g0076
2015-12-21 11:06:04 00/dflt 65536 8 100% - 100% - - 1.8% - - 1.7% - - 3.5% - - 3.5% - g0076
Summary
This blog post shows how to Upgrade multiple Oracle Solaris LDoms to Solaris 11.3.
Using the VDCF Management Framework 10 SPARC LDoms are done in 20 Minutes.
The T4-Hardware is 4 years old. Just imagine how fast such Upgrades will run on the current SPARC M7 Systems!