最近在已部署MySQL Enterprise Monitor的服务器上新增了MySQL实例，导致MySQL Enterprise Monitor异常宕机了，无法重新启动成功。收到了Bind on TCP/IP port: Address already in use的错误提示。下面是这个问题的解决办法，供大家参考。

1、故障现象
[root@SZAPP03 init.d]# ./mysql-monitor-server start
Starting mysql service  [ OK ]
150127 09:57:34 mysqld_safe Logging to '/opt/mysql/enterprise/monitor/mysql/runtime/mysqld.log'.
150127 09:57:35 mysqld_safe Starting mysqld daemon with databases from /opt/mysql/enterprise/monitor/mysql/data/
./mysql-monitor-server : tomcat  (pid 28303) already running
150127 09:57:42 mysqld_safe mysqld from pid file /opt/mysql/enterprise/monitor/mysql/runtime/mysqld.pid ended

2、故障分析
#查看日志，提示为Bind on TCP/IP port: Address already in use，地址在使用，如下
[root@SZAPP03 init.d]# tail -100 /opt/mysql/enterprise/monitor/mysql/runtime/mysqld.log |grep "ERROR" -A5
2015-01-27 09:57:36 30753 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
2015-01-27 09:57:36 30753 [ERROR] Do you already have another mysqld server running on port: 13306 ?
2015-01-27 09:57:36 30753 [ERROR] Aborting

2015-01-27 09:57:36 30753 [Note] Binlog end
2015-01-27 09:57:36 30753 [Note] Shutting down plugin 'partition'
2015-01-27 09:57:36 30753 [Note] Shutting down plugin 'ARCHIVE'
2015-01-27 09:57:36 30753 [Note] Shutting down plugin 'BLACKHOLE'
2015-01-27 09:57:36 30753 [Note] Shutting down plugin 'ARCHIVE'
2015-01-27 09:57:36 30753 [Note] Shutting down plugin 'BLACKHOLE'

#根据上述错误提示，核查缺省的mysql enterprise monitor 端口号13306，如下，并没有被占用
[root@SZAPP03 init.d]# netstat -nltp|grep mysql
tcp        0      0 :::3306                    :::*                        LISTEN      8734/mysqld        
tcp        0      0 :::3307                    :::*                        LISTEN      9489/mysqld 

3、故障解决

#故障现象里有一个提示为tomcat  (pid 28303) already running
#这个引起了我的注意，于是尝试先kill到tomcat相关进程
[root@SZAPP03 init.d]# ps -ef|grep 28303|grep -v grep
mysqlmem 28303 28302  4 Jan20 ?        07:35:03 jsvc.exec -java-home /opt/mysql/enterprise/monitor/java 
 -user mysqlmem -pidfile /opt/mysql/enterprise/monitor/apache-tomcat/temp/catalina.pid -wait 10 -outfile 
 /opt/mysql/enterprise/monitor/apache-tomcat/logs/catalina-daemon.out -errfile &1 -classpath 
 /opt/mysql/enterprise/monitor/apache-tomcat/bin/bootstrap.jar:
 /opt/mysql/enterprise/monitor/apache-tomcat/bin/commons-daemon.jar:
 /opt/mysql/enterprise/monitor/apache-tomcat/bin/tomcat-juli.jar -Dnop -Xmx768M -Xms768M 
 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/mysql/enterprise/monitor/apache-tomcat/temp 
 -XX:+UseParallelOldGC -XX:MaxPermSize=512M -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
 -Djava.endorsed.dirs= -Dcatalina.base=/opt/mysql/enterprise/monitor/apache-tomcat
 -Dcatalina.home=/opt/mysql/enterprise/monitor/apache-tomcat 
 -Djava.io.tmpdir=/opt/mysql/enterprise/monitor/apache-tomcat/temp org.apache.catalina.startup.Bootstrap

[root@SZAPP03 init.d]# kill -9 28303
#再次检查是否有tomcat相关进程存在，逐一kill tomcat相关进程
[root@SZAPP03 init.d]# ps -ef|grep tomcat
[root@SZAPP03 init.d]# kill -9 28302
[root@SZAPP03 init.d]# kill -9 30867
# Author : Leshami
# Blog  : http://www.linuxidc.com

#再次重新启动mysql em，此时启动正常
[root@SZAPP03 init.d]# ./mysql-monitor-server start
Starting mysql service  [ OK ]
150127 10:16:08 mysqld_safe Logging to '/opt/mysql/enterprise/monitor/mysql/runtime/mysqld.log'.
150127 10:16:08 mysqld_safe Starting mysqld daemon with databases from /opt/mysql/enterprise/monitor/mysql/data/
Starting tomcat service  [ OK ]

#检查mysql em的状态
[root@SZAPP03 init.d]# ./mysql-monitor-server status
MySQL Enterprise MySQL is running
MySQL Enterprise Tomcat is running

#检查mysql 的端口号
[root@SZAPP03 init.d]# netstat -nltp|grep mysql
tcp        0      0 :::3306                    :::*                        LISTEN      8734/mysqld        
tcp        0      0 :::3307                    :::*                        LISTEN      9489/mysqld        
tcp        0      0 :::13306                    :::*                        LISTEN      31584/mysqld

--------------------------------------分割线 --------------------------------------

Ubuntu 14.04下安装MySQL http://www.linuxidc.com/Linux/2014-05/102366.htm

《MySQL权威指南(原书第2版)》清晰中文扫描版 PDF http://www.linuxidc.com/Linux/2014-03/98821.htm

Ubuntu 14.04 LTS 安装 LNMP Nginx\PHP5 (PHP-FPM)\MySQL http://www.linuxidc.com/Linux/2014-05/102351.htm

Ubuntu 14.04下搭建MySQL主从服务器 http://www.linuxidc.com/Linux/2014-05/101599.htm

Ubuntu 12.04 LTS 构建高可用分布式 MySQL 集群 http://www.linuxidc.com/Linux/2013-11/93019.htm

Ubuntu 12.04下源代码安装MySQL5.6以及Python-MySQLdb http://www.linuxidc.com/Linux/2013-08/89270.htm

MySQL-5.5.38通用二进制安装 http://www.linuxidc.com/Linux/2014-07/104509.htm

--------------------------------------分割线 --------------------------------------

本文永久更新链接地址：http://www.linuxidc.com/Linux/2015-01/112457.htm