Reference Information › CA AppLogic Support Knowledge Base › Overview of Support Knowledge Base › AppLogic Knowledge Library - Grid Node Boot Routine

AppLogic Knowledge Library - Grid Node Boot Routine

This article introduces the grid node boot up routine in Applogic 3.x

The entrance point is /etc/rc.d/init.d/applogic. It's is launched when node is up and porcess following functions.

Major functionalities

1. Discover the network layout
2. Create bonding(bond0 and bond1)
3. Update Xen configuration file, startup Xen and link Xen bridge interface(xenbr0 and xenbr1) to bonding(bond0 and bond1)
4. Create vrmd configuration file and startup vrmd
5. Startup controller if server role is primary
6. Prepare hoop, nbd and md device
7. Initialize time sync
8. Startup software watchdog
9. Enable the hard disk write back cache
10. Enable the hard disk NCQ
11. Enable SMART
12. Enable IPMI control
13. Mount NFS (from 3.5)

Details

1. Discover the network layout
   1. a. Discovery all NIC(ethX and Mac address) , connected switches(name, model, protocol, connectivity), which one are connected to backbone and which one connected to external network.
   2. b. Discovered network layout is stored in /etc/applogic.d/network.conf
   3. c. Applogic load kernel module sd_filter.ko and run following scripts to perform discovery

      /usr/local/apl-srv/bin/net_discover.sh -> /usr/local/apl-srv/bin/nwd.pl perform network layout discovery
      

2. Create bonding(bond0 and bond1)
   1. a. Parse /etc/applogic.d/network.conf to find out ethX connected to backbone and external network, create bond0 for backbone network and bond1 for external network upon related ethX.
   2. b. 3tbond is used for bonding

      /usr/local/bin/3tbond info bondX get bondX information
      /usr/local/bin/3tbond destroy bondX destroy bondX
      /usr/local/bin/3tbond create bond0 nics=eth0,eth1 create bond0 on eth0 and eth1
      /usr/local/bin/3tbond create bond1 nics=eth2,eth3 create bond1 on eth2 and eth3
      

   3. c. It also output Mac address of active/passive NIC connected to backbone/external network to following files

      /var/local/applogic/ha_backbone.*
      /var/local/applogic/ha_external.*
      

3. 3. Update Xen configuration file, startup Xen and link Xen bridge interface(xenbr0 and xenbr1) to bonding(bond0 and bond1)
   1. Background knowledge of Xen bridge interface(http://wiki.xensource.com/xenwiki/XenNetworking)
   2. Disable memory balloon by modifying Xen configuration file /etc/xen/xend-config.sxp
   3. Startup Xen using following command in sequence

      /etc/rc.d/init.d/xencommons start 
      /etc/rc.d/init.d/xend start
      

   4. Link Xen bridge interface(xenbr0 and xenbr1) upon bonding(bond0 and bond1) using below script

      /usr/local/apl-srv/bin/xenbr.sh
      

   The xenbr.sh invokes following commands to attach bonding device to Xen bridge interface

   brctl show show all Xen bridge interface 
   brctl stp xenbrX off turn off stp on Xen bridge interface xenbrX
   brctl addif xenbrX bondX attach the bondX to Xen bridge interface xenbrX
   brctl delif xenbrX bondX remove bondX from Xen bridge interface xenbrX
   

4. Create vrmd configuration file and startup vrmd
   1. Based on vrmd template(/usr/local/apl-srv/templates/vrmd.conf.tmpl), collect node information and create vrmd configuration file(/etc/applogic.conf)
      • BIOS
      • Mother board
      • CPU
      • RAM
      • Disk
      • Hypervisor type
      • HVM/PV
      • VM

      Following scripts are invoked

      /usr/local/apl-srv bin/fill_template.pl 
      /usr/local/vrm/scripts/vrmcfggen.sh
      

   2. Run following script to start up the vrmd based on vrmd configuration file(/etc/applogic.conf)

      /usr/local/vrm/scripts/vrm_ctl.sh - - > /usr/local/vrm/vrmd launch the vrmd
      | -> /usr/local/vrm/smnd.sh monitors VRM and reboots the server if VRM crashes or unloads.
      | --> /usr/local/vrm/scripts/srv_reboot_main.sh reboots the server
      | -> /usr/local/vrm/scripts/srv_get_infod.sh get server hardware and vm info if hypervisor is VMware 
      | -> /usr/local/vnp/vnp_load load vnp driver(vnp.ko) and passed a few ethX/bondX/xenbrX to it.
      

5. Startup controller by startup recovery_master service if server role is primary

   /etc/rc.d/init.d/recovery_master 
   

   This script perform following operation

   • Run “arping -D -c 3 -I xenbrX <controller IP>” to detect duplicated controller ip.
   • Assign controller public IP to xenbr1 which is associated to controller public NIC and add external gateway to local route table
   • Assign controller private IP to xenbr0 which is associated to controller private NIC
   • Invoke /usr/local/apl-srv /bin/ctlb_ctl2.sh to start up controller followed by some preparation tasks, e.g. fsck the boot and meta volume, mount swap volume, configure vethX, load VNP driver for vLAN

     /usr/local/apl-srv /bin/ctlb_ctl2.sh -- > /usr/local/apl-srv bin/fill_template.pl create ctlb configuration file(/var/applogic/boot/ctlb.conf)
     |-> /usr/local/apl-srv/bin/ctlb_prep.sh preparation before controller startup: create metadata of controller volumes; create cluster descriptor; set recovery GUI password
     |-> /usr/local/apl-srv/bin/ctlb_conn_mon.sh connect to all physical node which has controller volume stream
     |-> /usr/local/ctlb/ctlb ??????
     |-> /usr/local/apl-srv/bin/ctl_vm_ctl.sh startup controller VM and verify by wget controller hpptd url.
     

6. Prepare hoop and md device
   1. Load hoop driver hoop.ko to enable hoop
   2. Create mdX using “mknod /dev/mdX”
7. Initial time sync
   1. Invoke following script to configure timing server in /etc/ntpd.conf, run ntpdate to sync time and startup ntpd service

      /usr/local/apl-srv/bin/ntp_ctl.sh start
      

8. Startup software watchdog
   1. Load driver softdog.ko
   2. Copy configuration file from /usr/local/apl-srv/misc/watchdog.conf to /etc/watchdog.conf
   3. Startup watchdog service
9. Enable the hard disk writeback cache using below command

   /usr/bin/sdparm -s WCE=1 /dev/sdaX
   

10. Enable the hard disk NCQ using below script and set queue length as 128 by default

    /var/local/applogic/ncq.op turn on or turn off NCQ at node startup stage
    /var/local/applogic/ncq.sda NCQ queue depth
    /usr/local/apl-srv/bin/ncq based on above 2 files, it perform NCQ configuration by echoing NCQ queue depth to /sys/block/sdX/device/queue_depth
    

11. Enable SMART by using below script

    /usr/local/applogic/bin/3tsmartcfg create SMART configuration file(/etc/smartd.conf)
    

12. Enable IPMI control by loading following drivers

    ipmi_msghandler.ko 
    ipmi_devintf.ko 
    ipmi_si.ko 
    

13. Mount NFS using following script

    /usr/local/apl-srv/bin/nfsctl.sh
    

    Used Env files

    /etc/applogic.env
    /usr/local/vrm/scripts/vrm_sc_defs.sh