Don't remove Data Disks from Solaris Zpools if performance is important

Solaris 11.4 delivers a new feature to remove data disks from existing zpools.
zpool remove myzpool <disk>

We used this feature a few times without problems on test environments.

But it has performance impact if the removed disk had data on it.
Especially if read performance is important. We know customers with oracle databases
where latency around 1ms is expected. After removing a disk from a
large zpool the performance was terrible and the only solution was to
re-create the zpool.

Important to understand there is an expected performance impact while the disk is removing.
Sure. The data needs to be copied to the remaining disks.
But even after the remove there can be a major performance impact when the data (from the removed disk) must be read, because the data copy added additional internal redirections.

The recommendation is to use this feature only after an accidentally add
of a disk to a wrong zpool. there is no performance impact if the
removed disk has no data.

Find the details of this recommendation in the Solaris 11.4 ZFS Manual:
https://docs.oracle.com/cd/E37838_01/html/E61017/remove-devices.html

 

To avoid such troubles we disabled the dataset -c remdisk feature for data disks
by default in VDCF Version 8.1.8

More about our VDCF Solaris Management product can be found on
https://www.jomasoft.ch/vdcf/

Did You Know Oracle Solaris Includes Ksplice?

Look what we have here:

-bash-5.0$ pkg list ksplice
NAME (PUBLISHER)     VERSION                    IFO
system/ksplice       11.4-11.4.29.0.1.82.3      i--


Ksplice supports online Kernel Updates.

Oracle Support delivers in rare cases of Kernel issues
an IDR which are installed online using ksplice.

For a Solaris Admin such an IDR is handled like other IDRs.
It can be installed as usual with the pkg command.


Here a sample:

# pkg info -g ./idr4712.1.p5p idr4712
          Name: idr4712
       Summary: To back out This IDR : # /usr/bin/pkg uninstall -r idr4712
   Description: sparc IDR built for release : Solaris 11.4 SRU # 29.82.3
         State: Not installed
     Publisher: solaris
       Version: 1
        Branch: None
Packaging Date: February 12, 2021 at 10:22:38 AM
          Size: 4.08 kB
          FMRI: pkg://solaris/idr4712@1:20210212T102238Z


-bash-5.0$ pkg list -g ./idr4712.1.p5p -af
NAME (PUBLISHER)         VERSION                      IFO
idr4712                  1                            ---
system/kernel/platform   11.4-11.4.29.0.1.82.3.4712.1 ---
system/ksplice           11.4-11.4.29.0.1.82.3.4712.1 ---
system/osnet-splice      11.4-11.4.29.0.1.82.3.4712.1 ---


# pkg set-publisher -g file:///var/tmp/idr4712.1.p5p solaris

# pkg install idr4712
          Packages to install:   2
            Packages to update:   2
            Services to change:   3
       Create boot environment:  No
Create backup boot environment: Yes
..
..
..


Using spliceadm you can verify the installed splices.

# spliceadm
ID        STATE        CVE             BUGID
471201    applied      N/A             32407818


in case of a problem you can even revert the fix

# spliceadm reverse 471201
Splice 471201 reversed successfully on Fri Apr 23 13:15:20.

# spliceadm status
ID        STATE        CVE             BUGID
471201    not-applied  N/A             32407818


Another powerful and easy to use Solaris Feature

Online Events about Oracle Solaris 04/2021

April 13th, 2021   9:00 am -  11:00 am PT
 

Oracle Webinar: Systems Engineering Forum Oracle Solaris and SPARC

Modernizing Enterprise Infrastructure with Oracle Solaris and SPARC
Simplifying Security and Compliance with Oracle Solaris (Incl. Demo)
Continuous Observability of Systems and Applications on Oracle Solaris (Incl. Demo)

April 27th, 2021    9:00 AM IST | 11:30 AM SGT | 1:30 PM AEST

Oracle Webinar: Infrastructure Modernization Forum: Oracle SPARC & Solaris

Oracle Compute Platforms for On-Prem Deployments

During this virtual event you will learn the vision and strategy of
Oracle Solaris and SPARC servers and more.

Do you have the overview of your Solaris server patch levels?

To have updated systems is very important for security and stability reasons.

Oracle releases patch updates each quarter for Solaris 10 and Solaris 11.3 till 01/2024.
This patches only fix the most important issues, but not all.
To receive this patches for the older Solaris releases you need Extended Support.

It is highly recommended to update to the current Solaris 11.4 release.
For Solaris 11.4 there is a SRU (Support Repository Update) each month.

Make sure you know which Solaris server are not yet on the recommended level.

With our JomaSoft VDCF management tool for Solaris you have a good realtime overview of the Levels.

Learn more about VDCF
bitly.com/jomasoft

 

Online Events about Oracle Solaris 03/2021

Mar 10, 2021 09:00 - 10:00 CET

Oracle Webinar: Oracle Solaris Update

This webinar will give an update on the Oracle Solaris operating system recent advancements. There is a specific focus on the observability in Solaris and compliance in Solaris. These will help you to improve the performance and secure compliance to regulatory requirements like PCI DSS for example. The webinar consists of a presentation and demonstrations of these features. 

 

 

Mar 16, 2021 19:00 - 20:30 CET

FRAOSUG - Frankfurt Area OpenSolaris User Group: Virtual Datacenter Cloud Framework

German/Deutsch

Das “Virtual Datacenter Cloud Framework” (VDCF ) ist seit 2006 verfügbar, inzwischen in der Version 8.1. Es deckt den ganzen Life Cycle von Solaris Systemen ab und unterstützt die SPARC und x86 Platform. Mittels einer Handvoll intuitiver Befehle können virtuelle Solaris Umgebungen auch von unerfahrenen Solaris Administratoren erstellt und betrieben werden.

Oracle LGWR is not running at highest priority

The Oracle database does automatically increase the priority on Solaris if you run
the database version 11.2.0.4 or later. On older versions this does not happen.

If you are on Oracle 12.1.0.2 it could be because of Bug
27092821 - IN 12C CRITICAL DATABASE PROCESSES ARE NOT RUNNING WITH HIGHER PRIORITIES IN FSS
It is fixed in 12.1.0.2.99

If you run the Oracle database in a Solaris Zone, make sure you have 'proc_priocntl' set in limitpriv in your zonecfg. This is required to allow the change of the priority inside the zone.

Ideally the ora_lwgr_x process should run in the FX class with prio 60.
This can be verified using

ps -efcZ | grep lgwr

Does your system contain the fixes for CVEs or are you vulnerable?

On Solaris 11 the software packages are stored in a IPS repository.
One of the packages is 'solaris-11-cpu'.

If a CVE is fixed Oracle adds the CVE info into this package as metadata.

That's great. You can easly search for a CVE to find out which
package and Solaris Update contains the fix.

Current sample for CVS-2020-14871 "component: Pluggable authentication module"

-bash-5.0$ pkg search :CVE-2020-14871:
INDEX          ACTION VALUE              PACKAGE
CVE-2020-14871 set    pkg://solaris/consolidation/osnet/osnet-incorporation@11.4,5.11-11.4.25.0.1.75.3 pkg:/support/critical-patch-update/solaris-11-cpu@2020.12-1
CVE-2020-14871 set    pkg://solaris/consolidation/osnet/osnet-incorporation@11.4,5.11-11.4.25.0.1.75.3 pkg:/support/critical-patch-update/solaris-11-cpu@2020.9-2
CVE-2020-14871 set    pkg://solaris/consolidation/osnet/osnet-incorporation@11.4,5.11-11.4.25.0.1.75.3 pkg:/support/critical-patch-update/solaris-11-cpu@2020.11-1
CVE-2020-14871 set    pkg://solaris/consolidation/osnet/osnet-incorporation@11.4,5.11-11.4.25.0.1.75.3 pkg:/support/critical-patch-update/solaris-11-cpu@2020.10-2
CVE-2020-14871 set    pkg://solaris/consolidation/osnet/osnet-incorporation@11.4,5.11-11.4.27.0.1.82.2 pkg:/support/critical-patch-update/solaris-11-cpu@2020.12-1
CVE-2020-14871 set    pkg://solaris/consolidation/osnet/osnet-incorporation@11.4,5.11-11.4.27.0.1.82.2 pkg:/support/critical-patch-update/solaris-11-cpu@2020.11-1
CVE-2020-14871 set    pkg://solaris/system/library@0.5.11,5.11-0.175.3.36.0.22.0    pkg:/support/critical-patch-update/solaris-11-cpu@2020.6-2
CVE-2020-14871 set    pkg://solaris/system/library@0.5.11,5.11-0.175.3.36.0.22.0    pkg:/support/critical-patch-update/solaris-11-cpu@2020.12-1
CVE-2020-14871 set    pkg://solaris/system/library@0.5.11,5.11-0.175.3.36.0.22.0   pkg:/support/critical-patch-update/solaris-11-cpu@2020.8-2
CVE-2020-14871 set    pkg://solaris/system/library@0.5.11,5.11-0.175.3.36.0.22.0   pkg:/support/critical-patch-update/solaris-11-cpu@2020.9-2
CVE-2020-14871 set    pkg://solaris/system/library@0.5.11,5.11-0.175.3.36.0.22.0    pkg:/support/critical-patch-update/solaris-11-cpu@2020.11-1
CVE-2020-14871 set    pkg://solaris/system/library@0.5.11,5.11-0.175.3.36.0.22.0   pkg:/support/critical-patch-update/solaris-11-cpu@2020.10-2
CVE-2020-14871 set    pkg://solaris/system/library@0.5.11,5.11-0.175.3.36.0.22.0   pkg:/support/critical-patch-update/solaris-11-cpu@2020.7-2

This means you have the fix installed if you are on
Solaris 11.4  CPU 2020-09 or later which is S11.4 SRU 25

Users of our JomaSoft VDCF tool can list the systems centrally
where the required package is already installed

-bash-5.0$ vpkgadm -c show_server id=solaris/system/library@0.5.11-0.175.3.36.0.22.0

Package: system/library - Core system libraries
PKG-ID : solaris/system/library@0.5.11-0.175.3.36.0.22.0
Version: 0.5.11-0.175.3.36.0.22.0 is installed on:
   Name  Type     PatchLevel                GroupPkg       Comment
  g0062  Node     3.36.0.23.0 (U3.SRU36)    large-server   ZFS Cloning / Shared DS
  v0123  vServer  3.36.0.23.0 (U3.SRU36)    mini-server    ZFS Clones
  v0143  vServer  3.36.0.23.0 (U3.SRU36)    mini-server    Shared dataset

If the fix is not installed, it does not mean you are vulnerable in this special case,
because the bug has no impact on Solaris 11.1 or later.

This is documented here:
https://www.oracle.com/security-alerts/cpuoct2020.html#AppendixSUNS

Anyway ... make sure you are up-to-date ...