sos-collector for OpenStack nodes

2021-08-08 494 words 3 minutes

sos-collector is a tool to collect sosreports from multiple nodes simultaneously.

As of sos version > 4.0 sos-collector has now been merged into the sos tool as sos collect with sos-collector acting as a redirect.

Prerequisites

Install the sos-collector package or just sos if your distribution supplies sos version > 4.0
sos-collector requires a connection to each node you will be collecting a SoS report from. This can be a workstation with the correct routing in place or in this case I will be using the RHOSP director node as it already has ssh keys setup for each of the overcloud nodes.

sos-collector Options

sos-collector \
  --ssh-user heat-admin \
  --insecure-sudo \
  --cluster-type=none \
  --no-local \
  --nodes=192.168.24.8,192.168.24.24,192.168.24.13 \
  --timeout 1800

Breaking down the options:

--ssh-user heat-admin: The user we will be connecting to the remote nodes with.
--insecure-sudo: The heat-admin user has password-less sudo assess so we can pass this option to disable the need to enter a password
--cluster-type=none: sos-collector is designed to be run on a supported cluster environment, we can pass none or jbon (just a bunch of nodes) to bypass this check.¹
--no-local: Disables collecting a SoS report on the director node itself, if this is required you can remove this option.
--nodes=controller0,controller1,controller2: The remote nodes we will be collecting SoS reports from, this can be IP address or hostname. The default amount of nodes that can be collected from simultaneously is 4, this can be overridden with --jobs=²
--timeout=1800: This option is depended on your environment and what options you pass to sos running on the nodes. The default is no where near enough time for sos to run on an openstack controller, 1800 seconds is a minimum.

Example

Output from running the default sosreport command on the overcloud nodes:

[stack@dell-430-36 ~]$ sos-collector \
  --ssh-user heat-admin \
  --insecure-sudo \
  --cluster-type=none \
  --no-local \
  --nodes=192.168.24.8,192.168.24.24,192.168.24.13 \
  --timeout 1800

The following is a list of nodes to collect from:
                     
        192.168.24.13
        192.168.24.24
        192.168.24.8 

Please enter the case id you are collecting reports for: 123456

Connecting to nodes...

Beginning collection of sosreports from 3 nodes, collecting a maximum of 4 concurrently

overcloud-controller-0 : Generating sosreport...
overcloud-controller-1 : Generating sosreport...
overcloud-controller-2 : Generating sosreport...
overcloud-controller-0 : Retrieving sosreport...
overcloud-controller-0 : Successfully collected sosreport
overcloud-controller-2 : Retrieving sosreport...
overcloud-controller-2 : Successfully collected sosreport
overcloud-controller-1 : Retrieving sosreport...
overcloud-controller-1 : Successfully collected sosreport

Successfully captured 3 of 3 sosreports
Creating archive of sosreports...

The following archive has been created. Please provide it to your support team.
    /var/tmp/sos-collector-123456-2021-08-08-wgupj.tar.gz

[stack@dell-430-36 ~]$ tar -tvf /var/tmp/sos-collector-123456-2021-08-08-wgupj.tar.gz
-rw------- stack/stack 129503200 2021-08-08 14:41 sos-collector-123456-2021-08-08-wgupj/sosreport-overcloud-controller-0-123456-2021-08-08-ufpncih.tar.xz
-rw------- stack/stack        33 2021-08-08 14:41 sos-collector-123456-2021-08-08-wgupj/md5/sosreport-overcloud-controller-0-123456-2021-08-08-ufpncih.tar.xz.md5
-rw------- stack/stack 123922000 2021-08-08 14:55 sos-collector-123456-2021-08-08-wgupj/sosreport-overcloud-controller-1-123456-2021-08-08-ujbbvtp.tar.xz
-rw------- stack/stack        33 2021-08-08 14:55 sos-collector-123456-2021-08-08-wgupj/md5/sosreport-overcloud-controller-1-123456-2021-08-08-ujbbvtp.tar.xz.md5
-rw------- stack/stack 105984656 2021-08-08 14:54 sos-collector-123456-2021-08-08-wgupj/sosreport-overcloud-controller-2-123456-2021-08-08-qpvkbvs.tar.xz
-rw------- stack/stack        33 2021-08-08 14:54 sos-collector-123456-2021-08-08-wgupj/md5/sosreport-overcloud-controller-2-123456-2021-08-08-qpvkbvs.tar.xz.md5
-rw------- stack/stack     19946 2021-08-08 14:55 sos-collector-123456-2021-08-08-wgupj/logs/sos-collector.log
-rw------- stack/stack      2290 2021-08-08 14:55 sos-collector-123456-2021-08-08-wgupj/logs/ui.log

This is a great alternative for openstack overcloud support report collect which is currently broken in RHOSP16³