[案例分享]PowerVC在部署虚拟机完成后,虚拟机第一次启动慢,怀疑是cloud-init在配置相关参数导致。

故障现象:

PowerVC在部署虚拟机完成后,虚拟机第一次启动慢。第一次启动大概要10分钟,怀疑是cloud-init在配置相关参数导致。

系统环境:

PowerVC 1.3.1.2

VIO Server 2.2.3.4

HMC 8.4

问题分析:

         由于部署完虚拟机后,会自动调用cloud-init进行相关参数的配置,估计问题应该出现在cloud-init与AIX结合上。

         经过与IBM同事沟通,得知在部署虚拟机后,需要在新建虚拟机第一次启动前,把/etc/resolv.conf文件删除,cloud-init在启动时会使用dns进行相关的查询,因此删除/etc/resolv.conf会缩短启动时间。具体操作为在部署虚拟机的界面中,有一个激活设置的地方,填写rm -rf /etc/resolv.conf。

         但在删除完resolv.conf后,启动还是很慢,因此诊断虚拟机上的cloud-init.log,从log里可以看到每次有两分钟的等待。但在等待的过程中,命令执行也没有什么问题。

  

  

2016-12-19 09:24:26,585 - util.py[DEBUG]: Writing to  /opt/freeware/var/lib/cloud/instances/180c7383-e478-4d8a-a36f-d3c88b0a03c6/sem/config_set_hostname  - wb: [420] 23 bytes

  

2016-12-19 09:24:26,587 - helpers.py[DEBUG]: Running  config-set_hostname using lock ()

  

2016-12-19 09:24:26,588 - cc_set_hostname.py[DEBUG]:  Setting the hostname to node1.localdomain (node1)

  

2016-12-19 09:24:26,588 - util.py[DEBUG]:  Running command ['/usr/sbin/chdev', '-l', 'inet0', '-a', 'hostname=node1']  with allowed return codes [0] (shell=False, capture=True)

  

2016-12-19 09:27:04,963 - util.py[DEBUG]:  Running command ['/usr/bin/uname', '-S', 'node1'] with allowed return codes  [0] (shell=False, capture=True)

  

2016-12-19 09:27:04,975 - __init__.py[DEBUG]:  Non-persistently setting the system hostname to node1

  

2016-12-19 09:27:04,976 - util.py[DEBUG]:  Running command ['hostname', u'node1'] with allowed return codes [0]  (shell=False, capture=True)

  

2016-12-19 09:29:49,308 - helpers.py[DEBUG]:  Running config-update_etc_hosts using lock ()

  

2016-12-19 09:29:49,309 - cc_update_etc_hosts.py[DEBUG]:  Configuration option 'manage_etc_hosts' is not set, not managing /etc/hosts  in module update_etc_hosts

  

         最后只能打开虚拟机操作系统启动时的debug模式,从debug信息来看,在启动时有两次pending比较久的事件,但在命令执行前后依然没有什么异常。

  

  

exec(/usr/sbin/umount,/tmp/tmpixQxmf){5177574,5505240}
  exec(/usr/sbin/chdev,-l,inet0,-a,hostname=client3){5964012,5505240}
  exec(/bin/sh,-c,/usr/lib/methods/chginet  -l inet0 -a "hostname=client3 " ){5046330,5964012}
  exec(/usr/lib/methods/chginet,-l,inet0,-a,hostname=client3  ){5046330,5964012}
  exec(/bin/hostname,client3){4718726,5046330}
  exec(/usr/bin/sh,-c,/usr/lib/lpd/pio/etc/piodmgr  -h >/dev/null 2>&1){5177576,4718726}
  exec(/usr/lib/lpd/pio/etc/piodmgr,/usr/lib/lpd/pio/etc/piodmgr,-h){5177576,4718726}
  exec(/usr/bin/egrep,^[  ]*hostname[ ]*=,/var/spool/lpd/pio/@local/piocfg){4522184,5177576}
  exec(/usr/bin/hostname){4522186,5177576}
  exec(/usr/bin/host,client3){4980912,4522188}
  exec(/usr/bin/sh,-c,/usr/sbin/lsvg  2>/dev/null){5439552,4456466}
  exec(/usr/sbin/lsvg){5439552,4456466}
  exec(/usr/bin/sh,-c,/usr/sbin/lqueryvg  -CX -g `/usr/sbin/getlvodm -v rootvg 2>/dev/null` >/dev/null  2>&1){5439554,4456466}
  exec(/usr/sbin/getlvodm,-v,rootvg){5898450,5439554}
  exec(/usr/sbin/lqueryvg,-CX,-g,00f6aa9a00004c0000000158b97eb3a5){5439554,4456466}
  exec(/usr/bin/sh,-c,/usr/sbin/lsvg  rootvg > /usr/tmp/lsvgtmp001 2> /dev/null){5439556,4456466}
  exec(/usr/sbin/lsvg,rootvg){5439556,4456466}
  exec(/usr/bin/sh,-c,/usr/sbin/lsfs  2>/dev/null){5439558,4456466}
  exec(/usr/sbin/lsfs){5439558,4456466}
  exec(/usr/bin/sh,-c,/usr/sbin/lsps  -a 2>/dev/null){5439560,4456466}
  exec(/usr/sbin/lsps,-a){5439560,4456466}
  exec(/usr/bin/sh,-c,/usr/sbin/getlvodm  -l hd6){4915388,5439560}
  exec(/usr/sbin/getlvodm,-l,hd6){4915388,5439560}
  exec(/usr/bin/sh,-c,/usr/sbin/lslv  -L -m 00f6aa9a00004c0000000158b97eb3a5.2 | /usr/bin/sed '1,2d' | /usr/bin/awk  '{print $3}' | /usr/bin/sort | /usr/bin/uniq -c |/usr/bin/awk  '{printf("%s %03d\n",$2,$1)}'){4915390,5439560}
  exec(/usr/sbin/lslv,-L,-m,00f6aa9a00004c0000000158b97eb3a5.2){5373954,4915390}
  exec(/usr/bin/sed,1,2d){5111998,4915390}
  exec(/usr/bin/awk,{print  $3}){6226110,4915390}
  exec(/usr/bin/sort){6291648,4915390}
  exec(/usr/bin/uniq,-c){6357186,4915390}
  exec(/usr/bin/awk,{printf("%s  %03d\n",$2,$1)}){4915390,5439560}
  exec(/usr/bin/sh,-c,/usr/sbin/getlvodm  -b 00f6aa9a00004c0000000158b97eb3a5.2){5373956,5439560}
  exec(/usr/sbin/getlvodm,-b,00f6aa9a00004c0000000158b97eb3a5.2){5373956,5439560}
  exec(/usr/bin/sh,-c,/usr/sbin/lquerylv  -L 00f6aa9a00004c0000000158b97eb3a5.2 -s){5373958,5439560}
  exec(/usr/sbin/lquerylv,-L,00f6aa9a00004c0000000158b97eb3a5.2,-s){5373958,5439560}
  exec(/usr/bin/sh,-c,/usr/sbin/getlvodm  -l hd6){5373960,5439560}
  exec(/usr/sbin/getlvodm,-l,hd6){5373960,5439560}
  
  <在此pending了很久>

  

  

exec(/usr/bin/cat,-){4522190,5177576}
  exec(/usr/bin/odmget,sm_cmd_opt){4522192,5177576}
  exec(/usr/bin/odmget,sm_cmd_hdr){4522194,5177576}
  exec(/usr/bin/odmget,sm_name_hdr){4522196,5177576}
  exec(/usr/bin/odmget,sm_menu_opt){4522198,5177576}
  exec(/usr/bin/sed,-e,s#\/#\\/#g){4522200,5177576}
  exec(/usr/bin/sed,-e,s/\_/\\_/g,-e,s/`/\\`/g,-e,s/\~/\\~/g,-e,s/"/\\"/g,-e,s/\'/\\'/g,-e,s/\./\\./g,-e,s/\!/\\!/g,-e,s/\@/\\@/g,-e,s/\$/\\$/g,-e,s/\^/\\^/g,-e,s/\&/\\&/g,-e,s/\*/\\*/g,-e,s/\[/\\[/g,-e,s/\]/\\]/g,-e,s/\|/\\|/g,-e,s/\?/\\?/g){4522202,5177576}
  exec(/usr/bin/sed,-e,s/\_/\\_/g,-e,s/`/\\`/g,-e,s/\~/\\~/g,-e,s/"/\\"/g,-e,s/\'/\\'/g,-e,s/\./\\./g,-e,s/\!/\\!/g,-e,s/\@/\\@/g,-e,s/\$/\\$/g,-e,s/\^/\\^/g,-e,s/\&/\\&/g,-e,s/\*/\\*/g,-e,s/\[/\\[/g,-e,s/\]/\\]/g,-e,s/\|/\\|/g,-e,s/\?/\\?/g){4522204,5177576}
  exec(/usr/bin/ex,/tmp/sm_cmd_opt_5177576){4522206,5177576}
  exec(/usr/bin/ex,/tmp/sm_cmd_hdr_5177576){4522208,5177576}
  exec(/usr/bin/ex,/tmp/sm_name_hdr_5177576){4522210,5177576}
  exec(/usr/bin/ex,/tmp/sm_menu_opt_5177576){4522212,5177576}
  exec(/usr/bin/odmcreate,-c,/tmp/socdefs_5177576.cre){4522214,5177576}
  exec(/usr/bin/chgrp,printq,/var/spool/lpd/pio/@local/smit/sm_cmd_hdr,/var/spool/lpd/pio/@local/smit/sm_cmd_hdr.vc,/var/spool/lpd/pio/@local/smit/sm_cmd_opt,/var/spool/lpd/pio/@local/smit/sm_cmd_opt.vc,/var/spool/lpd/pio/@local/smit/sm_menu_opt,/var/spool/lpd/pio/@local/smit/sm_menu_opt.vc,/var/spool/lpd/pio/@local/smit/sm_name_hdr,/var/spool/lpd/pio/@local/smit/sm_name_hdr.vc){4522216,5177576}
  exec(/usr/bin/chmod,664,/var/spool/lpd/pio/@local/smit/sm_cmd_hdr,/var/spool/lpd/pio/@local/smit/sm_cmd_hdr.vc,/var/spool/lpd/pio/@local/smit/sm_cmd_opt,/var/spool/lpd/pio/@local/smit/sm_cmd_opt.vc,/var/spool/lpd/pio/@local/smit/sm_menu_opt,/var/spool/lpd/pio/@local/smit/sm_menu_opt.vc,/var/spool/lpd/pio/@local/smit/sm_name_hdr,/var/spool/lpd/pio/@local/smit/sm_name_hdr.vc){4522218,5177576}
  exec(/usr/bin/odmadd,/tmp/sm_cmd_opt_5177576){4522220,5177576}
  exec(/usr/bin/odmadd,/tmp/sm_cmd_hdr_5177576){4522222,5177576}
  exec(/usr/bin/odmadd,/tmp/sm_name_hdr_5177576){4522224,5177576}
  exec(/usr/bin/odmadd,/tmp/sm_menu_opt_5177576){4522226,5177576}
  exec(/usr/bin/mv,-f,/var/spool/lpd/pio/loopback,/var/spool/lpd/pio/client3){4522228,5177576}
  exec(/usr/bin/ed,/var/spool/lpd/pio/@local/piocfg){4522230,5177576}
  exec(/usr/bin/rm,-f,/tmp/socdefs_5177576.cre,/tmp/sm_cmd_opt_5177576,/tmp/sm_cmd_hdr_5177576,/tmp/sm_name_hdr_5177576,/tmp/sm_menu_opt_5177576){4522232,5177576}
  exec(/bin/sh,-c,/usr/sbin/savebase  ){5046332,5964012}
  exec(/usr/sbin/savebase){5046332,5964012}
  exec(/usr/bin/compress){5177578,5046332}
  exec(/usr/bin/uname,-S,client3){5964014,5505240}
  exec(/usr/bin/hostname,client3){5964016,5505240}
  exec(/usr/bin/sh,-c,/usr/lib/lpd/pio/etc/piodmgr  -h >/dev/null 2>&1){4718728,5964016}
  exec(/usr/lib/lpd/pio/etc/piodmgr,/usr/lib/lpd/pio/etc/piodmgr,-h){4718728,5964016}
  exec(/usr/bin/egrep,^[  ]*hostname[ ]*=,/var/spool/lpd/pio/@local/piocfg){5177580,4718728}
  exec(/usr/bin/hostname){5177582,4718728}
  exec(/usr/bin/host,client3){2883770,5177584}
  
  <在此pending了很久>
  
  exec(/usr/bin/sh,-c,/usr/sbin/dumpctrl  -k >/dev/null 2>/dev/null){4522234,5243056}
  exec(/usr/sbin/dumpctrl,-k){4522234,5243056}
  exec(/usr/bin/ssh-keygen,-t,rsa,-N,,-f,/etc/ssh/ssh_host_rsa_key){5964018,5505240}
  Invalid operation  interrupt.
  D222ACC4 vcipher  vr0,vr0,vr0
  KDB(8)>
  KDB(8)>
  KDB(8)>
  KDB(8)>
  KDB(8)> g
  exec(/usr/bin/ssh-keygen,-t,dsa,-N,,-f,/etc/ssh/ssh_host_dsa_key){5964020,5505240}
  Invalid operation  interrupt.
  D222ACC4 vcipher  vr0,vr0,vr0
  KDB(12)>
  KDB(12)>
  KDB(12)>
  KDB(12)>
  KDB(12)> g

  

         结论,在本个案例中看出目前cloud-init和AIX结合并不完美,还有待开发人员进行完善。

参与10

4同行回答

cuizengshuncuizengshun系统运维工程师民生银行
现在是host=local, bind,和这个有关系吗?显示全部

现在是host=local, bind,和这个有关系吗?

收起
银行 · 2017-01-06
浏览4344
Henry2017Henry2017研发工程师金融行业
崔总,请教一个问题,如果创建的时候给虚机分配两个网络,比如是公网和内网,公网有网关,内网没有网关,启动的时候cloud-init会为两个网卡配置上ip,但是公网的网关没有配置上,cloud-init日志里没看到异常,是aix系统不允许还是cloud-init本身对于两个网络就没有配置网关?...显示全部

崔总,请教一个问题,如果创建的时候给虚机分配两个网络,比如是公网和内网,公网有网关,内网没有网关,启动的时候cloud-init会为两个网卡配置上ip,但是公网的网关没有配置上,cloud-init日志里没看到异常,是aix系统不允许还是cloud-init本身对于两个网络就没有配置网关?

收起
金融其它 · 2017-08-02
浏览4181
shawleoshawleo系统架构师IBM
如果环境里面没有DNS,就不要bind.显示全部

如果环境里面没有DNS,就不要bind.

收起
互联网服务 · 2017-01-06
浏览4397
shawleoshawleo系统架构师IBM
在镜像的  /etc/netsvc.conf 里面加一行:hosts=local4重新抓取镜像,再次部署试试。显示全部

在镜像的  /etc/netsvc.conf 里面加一行:

hosts=local4

重新抓取镜像,再次部署试试。

收起
互联网服务 · 2017-01-06
浏览4251

提问者

cuizengshun
系统运维工程师民生银行
擅长领域: 云计算服务器iaas

问题来自

相关问题

相关资料

相关文章

问题状态

  • 发布时间:2017-01-05
  • 关注会员:3 人
  • 问题浏览:8440
  • 最近回答:2017-08-02
  • X社区推广