Understanding WSO2 ESB timeout configurations (ESB Story - 2)

When the message flowing through the ESB, It's having two main paths.

• Message In sequence low (Client connection)
• Message out sequence flow(back-end connection)

Aforementioned both flows having multiple timeout configurations and these configurations help to manage the connections properly in several situations. It is worth of know understand what are those timeout values and how it correctly configure to get reliable throughout form the WSO2 ESB.

To explain this story I am talking sample proxy service and how it going through the WSO2 ESB.

As I explained in my previous blog post (How WSO2 Enterprise Service Bus Works (ESB Story - 1), When the client application or the external client invoke the proxy service, That request message arrive to the Transport layer. From this point ESB engaged with four main point which are handle the connections.

• Message listener (Http Listener)
• Message Sender (Http Sender)
• Endpoint timeout config
• Global timeout config

[Please refer below diagram]

Message Listner/Sender Timeouts

When client sending message to the ESB Passthrough transport creating a connection. Same way another connection will be created from ESB to backend to sent request to the back end server. Timing out of either connection will be happen based on the the "http.socket.timeout" set in the passthru-http.properties file located at /repository/conf directory.

ex : http.socket.timeout=180000

Let say, client sent the request to the ESB and ESB sent again same to the backend. In that case back end talking long time to respond back. In such a case you will see source handler (which is in the request side) will get timeout as below. But it is expected to timeout ESB to backend connection (TargetHandler).

WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-33855 {org.apache.synapse.transport.passthru.SourceHandler} 

Problem here is, both connection uses same timeout configuration. Once message received to the ESB from the external client SourceHandler timeout value will start counting and then message goest to the target handler. then only TargetHandler time out counter start. IIn such a case source handler will be timed out before TargetHandler timeout becase both are uses same timeout value. But the actual timeout value happens in the Targethandler side. This is most important situation when you are tracing down the timeout issues in ESB.

[Please refer previous blog post "How Messages are Flow Inside the WSO2 ESB (ESB Story - 3)" to understand SourceHandler and TargetHandler]

Endpoint timeout config

When the proxy service or the sequence having multiple endpoints, each endpoint can be set to separate timeout values. Once the timeout is reached, esb can trigger the fault sequence as a response action.Timeout value can be set as a "duration" parameter and fault sequence can be configure the "responseAction" parameter in endpoint configurations.




Tip :  Endpoint timeout value always need to be set to smaller value than the "http.socket.timeout" value configured in "passthru-http.properties" file

Global timeout config

Synapse which is the underlying mediation engine of WSO2 ESB is a complete asynchronous messaging engine which does not block its worker threads on network io. Instead of waiting the threads for responses, it registers a Calllback for a particular request and return. When the response is received, the registered callback is used to correlate it with the request and do the further processing. If the back end server not respond back, it is required to clear the registered Callbacks after sometime preventing possible memory leaks. It is done by a timer task called TimeoutHandler. ‘synapse.global_timeout_interval’ parameter represents the time duration which a Callback should be kept in the Callback store. If we have configured a timeout duration at the endpoint level, this global timeout value is not taken into consideration for that particular endpoint. For all the other endpoints which doesn’t have a timeout configuration, this global parameter value is considered as the timeout duration.

synapse.global_timeout_interval can be configured from the synapse.properties file. synapse.properties file is located at /repository/conf directory. Default value is 120 seconds.