Getting rid of ports in "CLOSE_WAIT" state
Posted: Sun May 29, 2005 12:28 am
[Problem]
Today someone had an interesting problem with some deamon allocating a port. The daemon crashed, and left the port in the "CLOSE_WAIT" state, yet the PID of the deamon was already killed.
So, you can't kill it again, but the Port is still stuck in "CLOSE_WAIT", as "netstat -a | grep CLOSE" shows you. Damn! When you try to restart it, it can't bind on the same Port again, since its still virtualy used. In this case, the port was 9000.
[Solution]
In this case you can use 'socklist|grep 9000' to find out which child or parent process of the already killed zombie is still locking the port. Note that these "zombie orphans" usualy dont show up when you use 'ps' !
You get output like:
tcp 9000 0 0 0 0
tcp 9000 0 0 0 0
tcp 9000 32460321 0 7774 10 sc_serv
Oh! There it is, PID 7774, still locking the socket! Try to 'kill -SIGTERM' it, or use SIGHUP/-9 if SIGTERM does not work.
In our case, another zombie orphan pid turned up after killing 7774, so run 'socklist|grep 9000" again, and hope nothing shows up. If so, you should also kill those to liberate the port of the tyrant zombie task that has its claws on it.
Presto! Port is clear, you can rerun the deamon/service/program/whatever, and it will be able to bind to its port again.
Today someone had an interesting problem with some deamon allocating a port. The daemon crashed, and left the port in the "CLOSE_WAIT" state, yet the PID of the deamon was already killed.
So, you can't kill it again, but the Port is still stuck in "CLOSE_WAIT", as "netstat -a | grep CLOSE" shows you. Damn! When you try to restart it, it can't bind on the same Port again, since its still virtualy used. In this case, the port was 9000.
[Solution]
In this case you can use 'socklist|grep 9000' to find out which child or parent process of the already killed zombie is still locking the port. Note that these "zombie orphans" usualy dont show up when you use 'ps' !
You get output like:
tcp 9000 0 0 0 0
tcp 9000 0 0 0 0
tcp 9000 32460321 0 7774 10 sc_serv
Oh! There it is, PID 7774, still locking the socket! Try to 'kill -SIGTERM' it, or use SIGHUP/-9 if SIGTERM does not work.
In our case, another zombie orphan pid turned up after killing 7774, so run 'socklist|grep 9000" again, and hope nothing shows up. If so, you should also kill those to liberate the port of the tyrant zombie task that has its claws on it.
Presto! Port is clear, you can rerun the deamon/service/program/whatever, and it will be able to bind to its port again.