Sunday, April 29, 2012

Parsing the Multipath

Having worked a little with XIV I have come to like the "xiv_devlist" tool.  "xiv_devlist" lists all the attached disks on the SAN and the number of paths that are currently active.  This is very useful when you want to ensure that all the paths to storage are active.

Find latest up-to-date code for this post on my Github profile: https://github.com/linuxplayground/mpath_devlist

It surprised me to find that there are no useful tools to parse the output from Multipath.

Here is an example:

multipath -ll
mpath2 (3600507680191014a3800000000000100) dm-7 IBM,2145                       
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]                       
\_ round-robin 0 [prio=100][active]                                            
 \_ 0:0:1:8  sdaa 65:160 [active][ready]                                       
 \_ 1:0:1:8  sdbk 67:224 [active][ready]                                       
\_ round-robin 0 [prio=20][enabled]                                            
 \_ 1:0:0:8  sdas 66:192 [active][ready]                                       
 \_ 0:0:0:8  sdi  8:128  [active][ready]                                       
mpath0 (3600507680191014a380000000000010a) dm-17 IBM,2145                      
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]                       
\_ round-robin 0 [prio=100][active]                                            
 \_ 1:0:1:0  sdbc 67:96  [active][ready]                                       
 \_ 0:0:1:0  sds  65:32  [active][ready]                                       
\_ round-robin 0 [prio=20][enabled]                                            
 \_ 1:0:0:0  sdak 66:64  [active][ready]                                       
 \_ 0:0:0:0  sda  8:0    [active][ready]                                       
mpath18 (3600507680191014a38000000000000ff) dm-24 IBM,2145                     
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]                       
\_ round-robin 0 [prio=100][active]                                            
 \_ 1:0:0:7  sdar 66:176 [active][ready]                                       
 \_ 0:0:0:7  sdh  8:112  [active][ready]                                       
\_ round-robin 0 [prio=20][enabled]                                            
 \_ 1:0:1:7  sdbj 67:208 [active][ready]                                       
 \_ 0:0:1:7  sdz  65:144 [active][ready]                                       
mpath17 (3600507680191014a38000000000000fe) dm-23 IBM,2145                     
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]                       
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:6  sdaq 66:160 [active][ready]
 \_ 0:0:0:6  sdg  8:96   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:6  sdbi 67:192 [active][ready]
 \_ 0:0:1:6  sdy  65:128 [active][ready]
mpath16 (3600507680191014a3800000000000109) dm-22 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:5  sdap 66:144 [active][ready]
 \_ 0:0:0:5  sdf  8:80   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:5  sdbh 67:176 [active][ready]
 \_ 0:0:1:5  sdx  65:112 [active][ready]
mpath9 (3600507680191014a380000000000010c) dm-14 IBM,2145
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:15 sdah 66:16  [active][ready]
 \_ 1:0:1:15 sdbr 68:80  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:15 sdaz 67:48  [active][ready]
 \_ 0:0:0:15 sdp  8:240  [active][ready]
mpath15 (3600507680191014a3800000000000108) dm-21 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:1:4  sdbg 67:160 [active][ready]
 \_ 0:0:1:4  sdw  65:96  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:4  sdao 66:128 [active][ready]
 \_ 0:0:0:4  sde  8:64   [active][ready]
mpath8 (3600507680191014a380000000000010b) dm-13 IBM,2145
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:14 sday 67:32  [active][ready]
 \_ 0:0:0:14 sdo  8:224  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:14 sdag 66:0   [active][ready]
 \_ 1:0:1:14 sdbq 68:64  [active][ready]
mpath14 (3600507680191014a3800000000000107) dm-20 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:3  sdan 66:112 [active][ready]
 \_ 0:0:0:3  sdd  8:48   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:3  sdbf 67:144 [active][ready]
 \_ 0:0:1:3  sdv  65:80  [active][ready]
mpath7 (3600507680191014a3800000000000105) dm-12 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:13 sdax 67:16  [active][ready]
 \_ 0:0:0:13 sdn  8:208  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:13 sdaf 65:240 [active][ready]
 \_ 1:0:1:13 sdbp 68:48  [active][ready]
mpath13 (3600507680191014a3800000000000106) dm-19 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:1:2  sdbe 67:128 [active][ready]
 \_ 0:0:1:2  sdu  65:64  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:2  sdam 66:96  [active][ready]
 \_ 0:0:0:2  sdc  8:32   [active][ready]
mpath6 (3600507680191014a3800000000000104) dm-11 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:12 sdae 65:224 [active][ready]
 \_ 1:0:1:12 sdbo 68:32  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:12 sdaw 67:0   [active][ready]
 \_ 0:0:0:12 sdm  8:192  [active][ready]
mpath12 (3600507680191014a38000000000000fd) dm-18 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:1  sdal 66:80  [active][ready]
 \_ 0:0:0:1  sdb  8:16   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:1  sdbd 67:112 [active][ready]
 \_ 0:0:1:1  sdt  65:48  [active][ready]
mpath5 (3600507680191014a3800000000000103) dm-10 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:11 sdav 66:240 [active][ready]
 \_ 0:0:0:11 sdl  8:176  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:11 sdad 65:208 [active][ready]
 \_ 1:0:1:11 sdbn 68:16  [active][ready]
mpath11 (3600507680191014a380000000000010e) dm-16 IBM,2145
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:17 sdaj 66:48  [active][ready]
 \_ 1:0:1:17 sdbt 68:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:17 sdbb 67:80  [active][ready]
 \_ 0:0:0:17 sdr  65:16  [active][ready]
mpath4 (3600507680191014a3800000000000102) dm-9 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:10 sdac 65:192 [active][ready]
 \_ 1:0:1:10 sdbm 68:0   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:10 sdau 66:224 [active][ready]
 \_ 0:0:0:10 sdk  8:160  [active][ready]
mpath10 (3600507680191014a380000000000010d) dm-15 IBM,2145
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:16 sdba 67:64  [active][ready]
 \_ 0:0:0:16 sdq  65:0   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:16 sdai 66:32  [active][ready]
 \_ 1:0:1:16 sdbs 68:96  [active][ready]
mpath3 (3600507680191014a3800000000000101) dm-8 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:9  sdat 66:208 [active][ready]
 \_ 0:0:0:9  sdj  8:144  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:9  sdab 65:176 [active][ready]
 \_ 1:0:1:9  sdbl 67:240 [active][ready]

So if you wanted to check that there are 4 active / ready paths to each "mpath" disk, you would have to visually scan through the whole output above.

A simple perl script that takes the output from multipath -ll as its input, can make this task much simpler.

dave@davidpc:~/devel/paths$ cat paths.txt | ./mpath_list.pl 
00      PATH    UID                                     DM-ID   ACTIVE  RESULT
2       mpath2  3600507680191014a3800000000000100       dm-7    4 of 4  OK
0       mpath0  3600507680191014a380000000000010a       dm-17   4 of 4  OK
18      mpath18 3600507680191014a38000000000000ff       dm-24   4 of 4  OK
17      mpath17 3600507680191014a38000000000000fe       dm-23   4 of 4  OK
16      mpath16 3600507680191014a3800000000000109       dm-22   4 of 4  OK
9       mpath9  3600507680191014a380000000000010c       dm-14   4 of 4  OK
15      mpath15 3600507680191014a3800000000000108       dm-21   4 of 4  OK
8       mpath8  3600507680191014a380000000000010b       dm-13   4 of 4  OK
14      mpath14 3600507680191014a3800000000000107       dm-20   4 of 4  OK
7       mpath7  3600507680191014a3800000000000105       dm-12   4 of 4  OK
13      mpath13 3600507680191014a3800000000000106       dm-19   4 of 4  OK
6       mpath6  3600507680191014a3800000000000104       dm-11   4 of 4  OK
12      mpath12 3600507680191014a38000000000000fd       dm-18   4 of 4  OK
5       mpath5  3600507680191014a3800000000000103       dm-10   4 of 4  OK
11      mpath11 3600507680191014a380000000000010e       dm-16   4 of 4  OK
4       mpath4  3600507680191014a3800000000000102       dm-9    4 of 4  OK
10      mpath10 3600507680191014a380000000000010d       dm-15   4 of 4  OK
3       mpath3  3600507680191014a3800000000000101       dm-8    4 of 4  OK

And the perl script that creates this output:
EDIT: Revised script (2012-04-29) Added size, Removed flag variable from script.
#!/usr/bin/env perl
use Term::ANSIColor;

$count = 0;
print "00\tPATH\tUID\t\t\t\t\tDM-ID\tSIZE\t\tACTIVE\tRESULT\n";
while(<STDIN>) {
        my ($line) = $_;
        chomp($line);
        if( $line =~ /^mpath(\d+)/ ) {
                my( $id ) = $1;
                /^(\w*) \((\w*)\)\s*(dm-\d*)/;
                if( $count > 0 ) { print $count, " of 4\t";
                        if( $count == 4) {
                                print  color 'bold green'; print "OK\n"; print color 'reset';
                        } else {
                                print color 'bold red'; print "PROBLEM\n"; print color 'reset';
                        }
                        $count = 0;
                }
                my ($pname) = $1;
                my ($puid)  = $2;
                my ($pdmid) = $3;
                print $id,"\t",$pname,"\t",$puid,"\t",$pdmid,"\t";
        }
        if ( $line =~ m/^\[(.*?)\]/ ) {
                print $1,"\t";
        }
        if ( $line =~ /^\s\\_.*active\]\[ready/ ) {
                $count += 1;
        }
}
if( $count > 0 ) { print $count, " of 4\t";
        if( $count == 4) {
                print  color 'bold green'; print "OK\n"; print color 'reset';
        } else {
                print color 'bold red'; print "PROBLEM\n"; print color 'reset';
        }

        $count = 0;
}
print color 'reset';
exit 0;

TO DO:
  1. Call "multipath -ll" from inside the perl script thus remove the requirement to pipe the results of multipath -ll into the script.
  2. Make sure multipath -ll is a valid system call.
  3. Add a summary of the results, IE: 18 paths found, ALL GOOD.
  4. Add a check to ensure user is root at the beginning of the script and exit with message if not.
  5. Sort output results in order of mpath number.
Post a Comment