Cross-Post from Stack-Overflow: ColdFusion 10 instance/Tomcat dying at predictable intervals (white screen of death) - Stack Overflow
---
For the last few days out team has been struggling with an ongoing issue where at very predictable intervals one ColdFusion instance has been white-screen-of-death-ing.
Every three hours the site would simply start returning a blank white page for any url. We would then restart the instance and everything would be great... for another three hours, almost to the minute. Of course this happened on a Friday, so all weekend people were taking turns re-booting the instance every time it died.
As best as I can discern, no one made any changes to either ColdFusion or our server environment right before this started happening. Before this the instance was running fine.
Since then we've seen that the isapi_redirect.log file for this instance is filled with Tomcat/connection errors.
We followed the excellent instructions at Resolve Stability Problems and SPEED UP ColdFusion 10 » Web Trenches and adjusted our connector settings as recommended. While this may have very well helped out general performance, and changed the timeframe from 3 to 3.5 hours between crashes, it has not resolved it.
Before that we even tried moving the site from one of our virtual servers to another with no luck.
We tried re-booting IIS and even re-booting the entire server the one night to see if that would help, and still nothing.
Below is as much information as I can provide from what we are seeing in our logs and our configurations. Any help would be very very much appreciated and please let me know what other details I can provide that would be useful.
---
We are running IIS v7.5.7600.16385
This is the only website/IIS record bound to this instance and it's bound specifically to it, not "All websites".
When the problem occurs, I do not think any requests makes it to the instance... the IIS logs show that connections are still happening, but the http.log files for the instance just stop.
I am not sure if the tomcat related errors are the problem or a symptom.
The server runs fine when the problem occurs, we have several other CF instances running along side this one that have no issues.
The CF admin for the instance in question loads and is completely responsive during the problem (This has not often, for me, been the case for other past issues with an instance).
Again, no one changed anything with our code, CF instance configuration, or server configuration directly prior to this problem starting as far as we can tell.
---
Server Product: ColdFusion
Version: 10,0,13,287689
Tomcat Version: 7.0.23.0
Edition: Enterprise
Operating System: Windows Server 2008 R2
OS Version: 6.1
Update Level: chf10000013.jar
Adobe Driver Version: 4.1 (Build 0001)
---
workers.properties:
worker.list=Instance_Codebase
worker.Instance_Codebase.type=ajp13
worker.Instance_Codebase.host=localhost
worker.Instance_Codebase.port=8014
worker.Instance_Codebase.max_reuse_connections=250
worker.Instance_Codebase.connection_pool_size=250
worker.Instance_Codebase.connection_pool_timeout=60
---
A sample of our isapi_redirect.log. A full chunk of it can be viewed at http://trasper.com/files/isapi_redirect.log.txt.
The problem (in this example) happened right about at 11:41pm as far as we can tell.
[Wed Jun 25 23:40:34.503 2014] [10012:912] [info] ajp_send_request::jk_ajp_common.c (1658): (Instance_Codebase) all endpoints are disconnected, detected by connect check (27), cping (0), send (0)
[Wed Jun 25 23:40:34.504 2014] [10012:1396] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1313): (Instance_Codebase) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8014) is down (errno=54)
[Wed Jun 25 23:40:34.820 2014] [10012:1396] [error] ajp_get_reply::jk_ajp_common.c (2190): (Instance_Codebase) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Jun 25 23:40:34.823 2014] [10012:1396] [info] ajp_service::jk_ajp_common.c (2692): (Instance_Codebase) sending request to tomcat failed (recoverable), (attempt=1)
[Wed Jun 25 23:40:34.708 2014] [10012:7880] [error] ajp_get_reply::jk_ajp_common.c (2190): (Instance_Codebase) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Jun 25 23:40:40.477 2014] [10012:2296] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1047): Failed opening socket to (127.0.0.1:8014) (errno=61)
[Wed Jun 25 23:40:40.364 2014] [10012:8256] [error] ajp_service::jk_ajp_common.c (2711): (Instance_Codebase) connecting to tomcat failed.
[Wed Jun 25 23:40:40.825 2014] [10012:7060] [error] HttpExtensionProc::jk_isapi_plugin.c (2309): service() failed with http error 503
[Wed Jun 25 23:40:40.877 2014] [10012:10364] [error] ajp_send_request::jk_ajp_common.c (1669): (Instance_Codebase) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Jun 25 23:40:40.965 2014] [10012:10364] [info] ajp_service::jk_ajp_common.c (2692): (Instance_Codebase) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[Wed Jun 25 23:40:40.857 2014] [10012:1020] [error] HttpExtensionProc::jk_isapi_plugin.c (2309): service() failed with http error 503