How do I setup multiple LUNs on Red Hat Enterprise Linux?

How do I setup multiple LUNs on Red Hat Enterprise Linux?

 SOLUTION 已验证 - 已更新 2018年五月23日22:38 - 

English 

环境

• Red Hat Enterprise Linux
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 3

问题

• How do I setup multiple LUNs on Red Hat Enterprise Linux?

• receive this message:

  Raw

  kernel: scsi: On host 3 channel 0 id 1 only 511 (max_scsi_report_luns) of 522 luns reported, 
                   try increasing max_scsi_report_luns
  

• need to see more than the default setup of 511 luns (RHEL5,6)

• need to see more than the default setup of 16,383 luns (RHEL7)
• how do I configure max_luns and max_report_luns for the scsi_mod kernel module?
• In releases of RHEL 5 and prior, the scsi_mod module parameters such as, max_luns and max_report_luns could be changed by modifying /etc/modprobe.conf, how do I do the same in RHEL 6 and RHEL 7?
• How to increase the max_luns parameter?

决议

• There is a parmeter of SCSI core module to control the number of multiple LUNs.
• Listed below are the steps to configure multiple LUNs.

For Red Hat Enterprise Linux 7

NOTE: The default max_report_luns value is 16,383 within RHEL 7. Increasing this value typically is no longer required.

In order to change the values for either max_report_luns or max_luns, the kernel options line in /etc/default/grub needs to be changed. The parameter to be changed should use the naming convention found in /usr/share/doc/kernel-doc-3.10.0/Documentation/kernel-parameters.txt; namely <module-name>.<parameter>=<value>. The specific kernel option line within /etc/default/grub that needs to be changed starts with GRUB_CMDLINE_LINUX=. NOTE: Do not edit /boot/grub2/grub.cfg directly.

1. If the number (count) of LUNs in any returned list from the scsi REPORT LUNS command is greater than the default of 16,383 (cat /sys/module/scsi_mod/parameters/max_report_luns), increasing the report luns entry value will be needed. Append the change to your GRUB_CMDLINE_LINUX= line within the /etc/default/grub file, changing xxx to be at least the number of LUNs any single storage target will return.

   Raw

   GRUB_CMDLINE_LINUX=" ... rd.lvm.lv=vg_rhel7/root crashkernel=auto scsi_mod.max_report_luns=xxx ..."
   

2. It would be very unusual for storage not to support the REPORT LUNS command, so typically changing the max_luns parameter is not necessary. The max_luns is only used to poll for devices -- lun 0, 1, ... max_luns -- if storage returns a failure status when sent a REPORT LUNS command. If storage does not support the SCSI command REPORT LUNS and the maximum lun identifier number presented to the host is greater than the default of 512 (cat /sys/module/scsi_mod/parameters/max_luns), increasing the maximum lun identifier allowed is needed.

   Append the following to your GRUB_CMDLINE_LINUX= line within the /etc/default/grub file:

   Raw

   GRUB_CMDLINE_LINUX=" ... rd.lvm.lv=vg_rhel7/root crashkernel=auto ... scsi_mod.max_luns=xxx ... "
   

3. Since changes have been made within /etc/default/grub, the grub2 menu files within the boot partition need to be rebuilt:

   Rebuild the /boot/grub2/grub.cfg file by running the grub2-mkconfig -o command as follows:

   • On BIOS-based machines: ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
   • On UEFI-based machines: ~]# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

4. Reboot the system. After the reboot, the additional LUNs should appear. The changes will be persistent across reboots.

For Red Hat Enterprise Linux 6

The scsi_mod is now built in to the kernel and is no longer a loadable module as in version 5 and earlier. See the kernel documentation (/usr/share/doc/kernel-doc-2.6.32/Documentation/kernel-parameters.txt) which states:

Raw

 Module parameters for modules that are built into the kernel image  
 are specified on the kernel command line with the module name plus  
 '.' plus parameter name, with '=' and value if appropriate, such as:  

           usbcore.blinkenlights=1

Because the scsi_mod is now compiled into the kernel, in order to change the values for either max_report_luns or max_luns the kernel options line within /boot/grub/grub.conf needs to be changed using the naming convention found in the above kernel documentation; namely <module-name>.<parameter>=<value>.

1. If the number (count) of LUNs in any returned list from the scsi REPORT LUNS command is greater than the default of 511 (cat /sys/module/scsi_mod/parameters/max_report_luns), increasing the report luns entry value will be needed. If so, also append it to your grub.conf kernel line, changing xxx to be at least the number of LUNs any single storage target will return.

   Raw

   scsi_mod.max_report_luns=xxx
   

2. It would be very unusual for storage not to support the REPORT LUNS command, so typically changing the max_luns parameter is not necessary. The max_luns is only used to poll for devices -- lun 0, 1, ... max_luns -- if storage returns a failure status when sent a REPORT LUNS command. If storage does not support the SCSI command REPORT LUNS and the maximum lun identifier number presented to the host is greater than the default of 512 (cat /sys/module/scsi_mod/parameters/max_luns), increasing the maximum lun identifier allowed is needed.

   Append the following to your grub.conf 'kernel' line:

   Raw

   scsi_mod.max_luns=xxx
   

3. Reboot the system. After the reboot, the additional LUNs should appear. The changes will be persistent across reboots.

Example:

Raw

 kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg_rhel60-lv_root 
    rd_LVM_LV=vg_rhel60/lv_root rd_LVM_LV=vg_rhel60/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM 
    LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto 
    rhgb quiet scsi_mod.max_report_luns=<yourvalue>  scsi_mod.max_luns=<yourvalue> 

For Red Hat Enterprise Linux 3, 4, 5

1. Modify the kernel module configuration file.

   In Red Hat Enterprise Linux 4 and 5, /etc/modprobe.conf is used.

   Raw

   options scsi_mod max_report_luns=xxx
   options scsi_mod max_luns=xxx
   

   In Red Hat Enterprise Linux 3, you need modify /etc/modules.conf, please add the following line.

   Raw

   options scsi_mod max_scsi_luns=xxx
   

2. Build the initial ramdisk to implement the change

   Raw

   # mkinitrd -f /boot/newimage-2.6.xx 2.6.xx
   

   (add the exact kernel version you have in place of xx)

3. Reboot the system. After the reboot, the LUNs should appear.

 

 

 

---

Note: you may also need to increase the max luns value for your HBA within its driver.

For example, see Red Hat Enterprise Linux reports lun has a LUN larger than allowed by the host adapter for instructions on doing so for the Emulex or Qlogic fibre channel drivers.

诊断步骤

You can verify the current numbers after reboot via:

cat /sys/module/scsi_mod/parameters/max_luns
cat /sys/module/scsi_mod/parameters/max_report_luns

Often it is helpful to clearly identify which set of lun values is being presented and exported to the host. We can retrieve the list of luns via the scsi REPORT LUNS command (sg_luns) available from the sg3_utils package.

You only have to perform the sg_luns against unique h:c:t:* storage target combinations, but the following collects all the data needed for review.

[root@host ~]# lsscsi > sg_luns.log
[root@host ~]# ls -1c /dev/sd*[!0-9] | sort >> sg_luns.log
[root@host ~]# ls -1c /dev/sd*[!0-9] | sort | xargs -I {} sg_luns -vv  {} &>> sg_luns.log

Review the sg_luns output to verify all the expected luns are present. For example, if sda is scsi address 1:0:0:0 and we retrieve the following information from that storage target (1:0:0) :

[root@host ~]# sg_luns -vv /dev/sda
open /dev/sda with flags=0x802
# sg_luns -vvd /dev/sdi
open /dev/sdi with flags=0x802
    report luns cdb: a0 00 00 00 00 00 00 01 00 00 00 00 
Lun list length = 80 which imples 10 lun entries           << max_report_luns refers to this number (list entries)

Output response in hex
 00     00 00 00 50 00 00 00 00  00 02 00 00 00 00 00 00                    
 10     00 03 00 00 00 00 00 00  00 04 00 00 00 00 00 00                    
 20     00 05 00 00 00 00 00 00  00 06 00 00 00 00 00 00                    
 30     00 07 00 00 00 00 00 00  00 08 00 00 00 00 00 00                    
 40     00 09 00 00 00 00 00 00  00 0a 00 00 00 00 00 00                    
 50     00 0b 00 00 00 00 00 00                                             
Report luns [select_report=0]:
    0002000000000000
      Peripheral device addressing: lun=2
    0003000000000000
      Peripheral device addressing: lun=3
    0004000000000000
      Peripheral device addressing: lun=4
    0005000000000000
      Peripheral device addressing: lun=5
    0006000000000000
      Peripheral device addressing: lun=6
    0007000000000000
      Peripheral device addressing: lun=7
    0008000000000000
      Peripheral device addressing: lun=8
    0009000000000000
      Peripheral device addressing: lun=9
    000a000000000000
      Peripheral device addressing: lun=10
    000b000000000000
      Peripheral device addressing: lun=11             << max_luns refers to this max identifier number

Then that storage target is presenting 10 luns, 2-11, to this host. However, if we were expecting 15 luns from storage then there is a storage side issue with the lun presentation to this host that needs to be addressed first.

The sg_luns data can be used to determine the maximum lun identifier value and the maximum number of luns within any one report luns list. Moreover, the returned lun list can be used to ascertain how many luns are missing or have been missed by the host and what luns numbers are missing from the host configuration.