saltstact 执行 sls 文件卡住问题请教。 - V2EX
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
fanne
V2EX    DevOps

saltstact 执行 sls 文件卡住问题请教。

  •  
  •   fanne 2017-05-24 17:14:47 +08:00 6574 次点击
    这是一个创建于 3114 天前的主题,其中的信息可能已经有所发展或是发生改变。

    初次使用 saltstack,若有语法什么的问题还请各位指点下。 sls 文件内容( https://github.com/fanne/salt)

    目录结构:

    [root@salt_master ~]# tree -Cv /srv/ /srv/ ├── README.md ├── pillar │ └── base │ ├── jboss7.sls │ └── top.sls └── salt └── base ├── bashrc_init │ └── bashrc_init.sls ├── cp_file.sls ├── del_file.sls ├── jboss7 │ ├── file │ │ ├── jboss-as-7.1.0.Final.zip │ │ ├── jboss-as-standalone │ │ ├── jboss-as.conf │ │ └── standalone.conf │ └── jboss7.sls ├── jdk_7_79 │ ├── file │ │ ├── env_jdk.txt │ │ └── jdk-7u79-linux-x64.rpm │ └── jdk_7_79.sls ├── jdk_8_121 │ ├── file │ │ ├── env_jdk.txt │ │ └── jdk-8u121-linux-x64.rpm │ └── jdk_8_121.sls └── top.sls 11 directories, 18 files 

    现在对 salt_node_02 节点安装 jdk 和 jboss 内容。 安装 jdk 前,salt_node_02 上的 jdk 信息

    [root@salt_node_02 ~]# java -version -bash: java: command not found 

    在 master 端执行:

    [root@salt_master ~]# salt 'salt_node_02' state.sls jdk_7_79.jdk_7_79 -l debug 

    输出结果:

    [DEBUG ] LazyLoaded nested.output [DEBUG ] LazyLoaded nested.output [DEBUG ] LazyLoaded nested.output [DEBUG ] LazyLoaded nested.output salt_node_02: ---------- ID: jdk-install Function: file.managed Name: /usr/local/src/jdk-7u79-linux-x64.rpm Result: True Comment: File /usr/local/src/jdk-7u79-linux-x64.rpm updated Started: 16:59:37.933787 Duration: 13352.812 ms Changes: ---------- diff: New file mode: 0644 ---------- ID: jdk-install Function: cmd.run Name: cd /usr/local/src && rpm -ivh jdk-7u79-linux-x64.rpm Result: True Comment: Command "cd /usr/local/src && rpm -ivh jdk-7u79-linux-x64.rpm" run Started: 16:59:51.287182 Duration: 27559.508 ms Changes: ---------- pid: 2009 retcode: 0 stderr: stdout: Preparing... ################################################## jdk ################################################## Unpacking JAR files... rt.jar... jsse.jar... charsets.jar... tools.jar... localedata.jar... jfxrt.jar... ---------- ID: jdk-evn Function: file.append Name: /etc/profile Result: True Comment: Appended 7 lines Started: 17:00:18.846921 Duration: 153.225 ms Changes: ---------- diff: --- +++ @@ -76,3 +76,10 @@ unset i unset -f pathmunge +export PATH=$PATH:/usr/local/bin +#set java environment +JAVA_HOME=/usr/java/jdk1.7.0_79 +CLASSPATH=.:$JAVA_HOME/lib/tools.jar +PATH=$JAVA_HOME/bin:$PATH +LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH +export JAVA_HOME CLASSPATH PATH LD_LIBRARY_PATH ---------- ID: jdk-evn Function: cmd.run Name: source /etc/profile Result: True Comment: Command "source /etc/profile" run Started: 17:00:19.000263 Duration: 202.819 ms Changes: ---------- pid: 2203 retcode: 0 stderr: stdout: Summary for salt_node_02 ------------ Succeeded: 4 (changed=4) Failed: 0 ------------ Total states run: 4 Total run time: 41.268 s [DEBUG ] jid 20170524165954616622 found all minions set(['salt_node_02']) 

    此时,salt_node_02 上的 jdk 信息

    [root@salt_node_02 ~]# java -version java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode) 

    然后安装 jboss7 内容,同样在 master 端执行命令: 然后 debug 日志一直打印以下信息

    [root@salt_master ~]# salt 'salt_node_02' state.sls jboss7.jboss7 -l debug [DEBUG ] Reading configuration from /etc/salt/master [DEBUG ] Missing configuration file: /root/.saltrc [DEBUG ] Configuration file path: /etc/salt/master [WARNING ] Insecure logging configuration detected! Sensitive data may be logged. [DEBUG ] Reading configuration from /etc/salt/master [DEBUG ] Missing configuration file: /root/.saltrc [DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc [DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear') [DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc [DEBUG ] LazyLoaded local_cache.get_load [DEBUG ] Reading minion list from /var/cache/salt/master/jobs/58/51c7350809d96364ef260721ee618e6765bf47830c9db82440105eeb919ced/.minions.p [DEBUG ] get_iter_returns for jid 20170524170231748034 sent to set(['salt_node_02']) will timeout at 17:02:36.830281 [DEBUG ] Checking whether jid 20170524170231748034 is still running [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear') [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode' [DEBUG ] Checking whether jid 20170524170231748034 is still running [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear') [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode' [DEBUG ] Checking whether jid 20170524170231748034 is still running [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear') [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode' [DEBUG ] Checking whether jid 20170524170231748034 is still running [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear') [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode' [DEBUG ] Checking whether jid 20170524170231748034 is still running [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear') [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode' 

    master 端一直在执行,没法结束。

    在 node 端查看信息,发现 jboss 已经安装上了:

    [root@salt_node_02 ~]# ps -ef|grep java root 2358 2305 1 17:02 ? 00:00:06 /usr/java/jdk1.7.0_79/bin/java -D[Standalone] -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms32m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.cOnfig=standalone.xml -Dorg.jboss.boot.log.file=/usr/local/src/jboss-as-7.1.0.Final/standalone/log/boot.log -Dlogging.cOnfiguration=file:/usr/local/src/jboss-as-7.1.0.Final/standalone/configuration/logging.properties -jar /usr/local/src/jboss-as-7.1.0.Final/jboss-modules.jar -mp /usr/local/src/jboss-as-7.1.0.Final/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/usr/local/src/jboss-as-7.1.0.Final -c standaloe.xml root 2648 1968 0 17:12 pts/2 00:00:00 grep java [root@salt_node_02 ~]# netstat -ntpl|grep java tcp 0 0 127.0.0.1:9999 0.0.0.0:* LISTEN 2358/java tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 2358/java tcp 0 0 0.0.0.0:4447 0.0.0.0:* LISTEN 2358/java tcp 0 0 127.0.0.1:9991 0.0.0.0:* LISTEN 2358/java [root@salt_node_02 ~]# 

    然后 master 端就是没法退出来,问题是出现哪里的?

    第 1 条附言    2017-05-24 18:02:42 +08:00

    所有文件在:https://github.com/fanne/salt/blob/master/salt/ jboss7.sls内容:

    install-jboss: file.managed: - source: salt://jboss7/file/jboss-as-7.1.0.Final.zip - name: {{ pillar['jboss-dir']['Jboss_Dir'] }}/jboss-as-7.1.0.Final.zip - user: root - group: root - mode: 644 cmd.run: - name: cd {{ pillar['jboss-dir']['Jboss_Dir'] }} && unzip -q jboss-as-7.1.0.Final.zip && rm -rf jboss-as-7.1.0.Final.zip - unless: test -d {{ pillar['jboss-dir']['Jboss_Dir'] }}/jboss-as-7.1.0.Final /etc/jboss-as: file.directory: - user: root - group: root - dir_mode: 755 - file_mode: 644 - force: True - recurse: - user - group - mode - require: - file: install-jboss /etc/jboss-as/jboss-as.conf: file.managed: - source: salt://jboss7/file/jboss-as.conf - user: root - group: root - mode: 644 - require: - file: install-jboss - template: jinja /etc/init.d/jboss-as-standalone: file.managed: - source: salt://jboss7/file/jboss-as-standalone - user: root - group: root - mode: 755 - require: - file: install-jboss - template: jinja {{ pillar['jboss-dir']['Jboss_Dir'] }}/jboss-as-7.1.0.Final/bin/standalone.conf: file.managed: - source: salt://jboss7/file/standalone.conf - user: root - group: root - mode: 644 - require: - file: install-jboss - template: jinja {{ pillar['jboss-dir']['Jboss_Dir'] }}/jboss-as-7.1.0.Final/bin/standalone.sh: file.managed: - user: root - group: root - mode: 755 - require: - file: install-jboss - template: jinja jboss-as-standalone: cmd.run: - name: chkconfig --add jboss-as-standalone - unless: chkconfig --list|grep jboss-as-standalone - require: - file: install-jboss service.running: - enable: True 
    22 条回复    2017-06-28 15:06:42 +08:00
    yujianwjj
        1
    yujianwjj  
       2017-05-24 17:59:34 +08:00
    jboss7.sls 贴出来看一下
    fanne
        2
    fanne  
    OP
       2017-05-24 18:03:03 +08:00
    @yujianwjj #1 贴了,在附言里了。
    yujianwjj
        3
    yujianwjj  
       2017-05-24 19:32:36 +08:00
    {{ pillar['jboss-dir']['Jboss_Dir'] }}/jboss-as-7.1.0.Final/bin/standalone.sh:
    这个 source 在哪。。
    fanne
        4
    fanne  
    OP
       2017-05-24 20:36:50 +08:00
    @yujianwjj #3
    貌似不是 source 的问题,我把 source 加上,也是一样的情况。

    {{ pillar['jboss-dir']['Jboss_Dir'] }}/jboss-as-7.1.0.Final/bin/standalone.sh:
    file.managed:
    - source: salt://jboss7/file/standalone.sh
    - user: root
    - group: rootbr /> - mode: 755
    - require:
    - file: install-jboss
    - template: jinja


    [root@salt_master base]# tree -Cv jboss7/
    jboss7/
    ├── file
    │ ├── jboss-as-7.1.0.Final.zip
    │ ├── jboss-as-standalone
    │ ├── jboss-as.conf
    │ ├── standalone.conf
    │ └── standalone.sh
    └── jboss7.sls
    yujianwjj
        5
    yujianwjj  
       2017-05-24 21:04:42 +08:00 via iPhone
    你的 jboss 似乎没有安装吧,只有解压 zip 和删除 zip 的操作。
    fanne
        6
    fanne  
    OP
       2017-05-25 09:47:25 +08:00
    @yujianwjj #5 安装了的,这个只要解压出来就可以的,而且在 node 端也看到 jboss 已经启动的

    [root@salt_node_02 ~]# ps -ef|grep jboss
    root 1915 1 0 May24 ? 00:00:00 su - root -c LAUNCH_JBOSS_IN_BACKGROUND=1 JBOSS_PIDFILE=/var/run/jboss-as/jboss-as-standalone.pid /usr/local/src/jboss-as-7.1.0.Final/bin/standalone.sh -c standalone.xml
    root 1923 1915 0 May24 ? 00:00:00 /bin/sh /usr/local/src/jboss-as-7.1.0.Final/bin/standalone.sh -c standalone.xml
    root 1974 1923 0 May24 ? 00:00:43 /usr/java/jdk1.7.0_79/bin/java -D[Standalone] -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.cOnfig=standalone.xml -Dorg.jboss.boot.log.file=/usr/local/src/jboss-as-7.1.0.Final/standalone/log/boot.log -Dlogging.cOnfiguration=file:/usr/local/src/jboss-as-7.1.0.Final/standalone/configuration/logging.properties -jar /usr/local/src/jboss-as-7.1.0.Final/jboss-modules.jar -mp /usr/local/src/jboss-as-7.1.0.Final/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/usr/local/src/jboss-as-7.1.0.Final -c standalone.xml
    root 20626 20610 0 09:47 pts/1 00:00:00 grep jboss
    [root@salt_node_02 ~]#
    fanne
        7
    fanne  
    OP
       2017-05-25 11:37:39 +08:00
    @yujianwjj #5 卡在了最后的启动那一步,去掉那一步就正常结束,把启动的加上,就会卡住,还不知啥处理。
    yesono
        8
    yesono  
       2017-05-25 11:50:45 +08:00
    可能启动脚本没写好导致。。。去掉这个看看是否被卡。。
    fanne
        9
    fanne  
    OP
       2017-05-25 12:07:43 +08:00
    @yesono #8 是卡在启动这一块的,启动脚本用 jboss 自带的脚本 jboss-as-standalone
    而且我直接在 node 上执行这个启动脚本,是正常可以启动的

    这是在 node 下手动执行的情况
    [root@salt_node_02 src]# service jboss-as-standalone start
    Starting jboss-as: [ OK ]
    [root@salt_node_02 src]# /etc/init.d/jboss-as-standalone stop
    Stopping jboss-as: *** JBossAS process (2750) received TERM signal ***
    [ OK ]
    [root@salt_node_02 src]# ps -ef|grep java
    yujianwjj
        10
    yujianwjj  
       2017-05-25 12:08:55 +08:00
    service.running:
    - enable: True

    这个应该是使用系统的包管理方式来启动 jbos,比如,如果你的系统是 centos6,那么会使用 service start jboss-as-standalone 这个方式来启动 jboos。
    fanne
        11
    fanne  
    OP
       2017-05-25 12:11:17 +08:00
    @yujianwjj #10 是可以用 service 启动的

    [root@salt_node_02 src]# service jboss-as-standalone stop
    Stopping jboss-as: [ OK ]
    [root@salt_node_02 src]# service jboss-as-standalone start
    Starting jboss-as: [ OK ]
    [root@salt_node_02 src]#
    fanne
        12
    fanne  
    OP
       2017-05-25 12:11:36 +08:00
    @yesono #8 可以用 service 启动的

    [root@salt_node_02 src]# service jboss-as-standalone stop
    Stopping jboss-as: [ OK ]
    [root@salt_node_02 src]# service jboss-as-standalone start
    Starting jboss-as: [ OK ]
    [root@salt_node_02 src]#
    yujianwjj
        13
    yujianwjj  
       2017-05-25 13:04:17 +08:00
    jboss-as-standalone:
    cmd.run:
    - name: chkconfig --add jboss-as-standalone
    - unless: chkconfig --list|grep jboss-as-standalone
    - require:
    - file: install-jboss
    - require:
    - file: /etc/init.d/jboss-as-standalone
    service.running:
    - enable: True
    - require:
    - file: /etc/init.d/jboss-as-standalone

    改成上面这样再试试
    fanne
        14
    fanne  
    OP
       2017-05-25 13:56:36 +08:00
    @yujianwjj #13 一样情况,也是会一直卡着的。

    debug 日志
    [DEBUG ] Checking whether jid 20170525133145602469 is still running
    [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
    [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
    [DEBUG ] Checking whether jid 20170525133145602469 is still running
    [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
    [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
    [DEBUG ] Checking whether jid 20170525133145602469 is still running
    [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
    [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
    [DEBUG ] Checking whether jid 20170525133145602469 is still running
    [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
    [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
    fanne
        15
    fanne  
    OP
       2017-05-25 16:04:10 +08:00
    @yujianwjj #13 又写了一个 nginx 的,https://github.com/fanne/salt/tree/master/salt/base/nginx_1_10
    也是正常执行成功。
    julyclyde
        16
    julyclyde  
       2017-05-25 17:23:46 +08:00
    java 流派的很多程序默认不带 daemonize 功能吧
    tongtf
        17
    tongtf  
       2017-05-26 16:24:14 +08:00
    jboss-as-standalone 脚本里面启动的那行改成 echo "nohup xxxxxxx &" | at now 应该就可已了,启动脚本没有返回,所以卡住了
    需要提前开启 atd 服务
    tongtf
        18
    tongtf  
       2017-05-26 16:30:37 +08:00
    if [ ! -z "$JBOSS_USER" ]; then
    if [ -x /etc/rc.d/init.d/functions ]; then
    daemon --user $JBOSS_USER LAUNCH_JBOSS_IN_BACKGROUND=1 JBOSS_PIDFILE=$JBOSS_PIDFILE $JBOSS_SCRIPT -c $JBOSS_CONFIG 2>&1 > $JBOSS_CONSOLE_LOG &
    else
    su - $JBOSS_USER -c "LAUNCH_JBOSS_IN_BACKGROUND=1 JBOSS_PIDFILE=$JBOSS_PIDFILE $JBOSS_SCRIPT -c $JBOSS_CONFIG" 2>&1 > $JBOSS_CONSOLE_LOG &
    fi
    fi


    应该是这段。后面 添加 |at now,不过结果只会是成功
    fanne
        19
    fanne  
    OP
       2017-05-26 17:10:03 +08:00
    @tongtf #18 脚本没有返回是咋意思?
    我今天又是了一个 mysql 的,用 yum 安装的,也是在启动步骤卡住了

    https://segmentfault.com/q/1010000009566820
    fanne
        20
    fanne  
    OP
       2017-05-26 17:18:23 +08:00
    @tongtf #18 添加了,也是一样,没有返回。

    一直处于这个状态

    [DEBUG ] Checking whether jid 20170526171632118302 is still running
    [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
    [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
    [DEBUG ] Checking whether jid 20170526171632118302 is still running
    [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
    [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
    [DEBUG ] Checking whether jid 20170526171632118302 is still running
    [DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
    [DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
    fanne
        21
    fanne  
    OP
       2017-05-26 17:19:32 +08:00
    @yesono #8 下午写了 mysql 的,也是这个情况,yum 安装 mysql 的
    fanne
        22
    fanne  
    OP
       2017-06-28 15:06:42 +08:00
    @tongtf @yujianwjj
    此问题已解决
    90 行:daemon --user $JBOSS_USER LAUNCH_JBOSS_IN_BACKGROUND=1 JBOSS_PIDFILE=$JBOSS_PIDFILE $JBOSS_SCRIPT -c $JBOSS_CONFIG 2>&1 > $JBOSS_CONSOLE_LOG 2>&1 &
    92 行:su - $JBOSS_USER -c "LAUNCH_JBOSS_IN_BACKGROUND=1 JBOSS_PIDFILE=$JBOSS_PIDFILE $JBOSS_SCRIPT -c $JBOSS_CONFIG" 2>&1 > $JBOSS_CONSOLE_LOG 2>&1 &

    mysql 的问题同样解决。
    关于     帮助文档     自助推广系统     博客     API     FAQ     Solana     1115 人在线   最高记录 6679       Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 25ms UTC 17:41 PVG 01:41 LAX 09:41 JFK 12:41
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86