This is short and sweet, but something we spent almost two weeks trying to resolve. The environment was setup using my documentation on Load Balancing Cells but the Remote console was not load balancing correctly. It worked if one cell was removed from the pool but not if both were live in the load balancer. As it turns out we saw the following error:
2012-03-26 09:43:10,470 | WARN | consoleproxy | ConnectionTracker | Closing connection java.nio.channels.SocketChannel[closed] as it has not initialized in time |
We suspect we see this message because the initial connection has left vCloud Cell 1 with a time stamp, and by the time it arrives to vCloud Cell 2 that time stamp was more than 2 minutes old. Which is well above the 2 second allowed skew. vCloud Cell 2 ignores the request because its more than the time allowed. The result is just an inconsistent drop in connections on the Remote Console connections.
I had run into other issues with NTP and vCD listed in the Related Posts Section as well that were new tolerances between 1.0.1 and 1.5 if you want to take a look. It seems NTP is one of those things you always want to check first just to rule it out on the cells. Be sure to read the other NTP article as well to see some other symptoms.