Wednesday, May 31, 2017

Understanding WSO2 ESB timeout configurations (ESB Story - 2)


When the message flowing through the ESB, It's having two main paths.
  • Message In sequence low (Client connection)
  • Message out sequence flow(back-end connection)

Aforementioned both flows having multiple timeout configurations and these configurations help to manage the connections properly in several situations. It is worth of know understand what are those timeout values and how it correctly configure to get reliable throughout form the WSO2 ESB.

To explain this story I am talking sample proxy service and how it going through the WSO2 ESB.

As I explained in my previous blog post (How WSO2 Enterprise Service Bus Works (ESB Story - 1), When the client application or the external client invoke the proxy service, That request message arrive to the Transport layer. From this point ESB engaged with four main point which are handle the connections.
  • Message listener (Http Listener)
  • Message Sender (Http Sender)
  • Endpoint timeout config
  • Global timeout config
[Please refer below diagram]






Message Listner/Sender Timeouts


When client sending message to the ESB Passthrough transport creating a connection. Same way another connection will be created from ESB to backend to sent request to the back end server. Timing out of either connection will be happen based on the the "http.socket.timeout" set in the passthru-http.properties file located at /repository/conf directory.

ex : http.socket.timeout=180000

Let say, client sent the request to the ESB and ESB sent again same to the backend. In that case back end talking long time to respond back. In such a case you will see source handler (which is in the request side) will get timeout as below. But it is expected to timeout ESB to backend connection (TargetHandler).

WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-33855 {org.apache.synapse.transport.passthru.SourceHandler} 


Problem here is, both connection uses same timeout configuration. Once message received to the ESB from the external client SourceHandler timeout value will start counting and then message goest to the target handler. then only TargetHandler time out counter start. IIn such a case source handler will be timed out before TargetHandler timeout becase both are uses same timeout value. But the actual timeout value happens in the Targethandler side. This is most important situation when you are tracing down the timeout issues in ESB.

[Please refer previous blog post "How Messages are Flow Inside the WSO2 ESB (ESB Story - 3)" to understand SourceHandler and TargetHandler]

Endpoint timeout config


When the proxy service or the sequence having multiple endpoints, each endpoint can be set to separate timeout values. Once the timeout is reached, esb can trigger the fault sequence as a response action.Timeout value can be set as a "duration" parameter and fault sequence can be configure the "responseAction" parameter in endpoint configurations.




Tip :  Endpoint timeout value always need to be set to smaller value than the "http.socket.timeout" value configured in "passthru-http.properties" file

Global timeout config


Synapse which is the underlying mediation engine of WSO2 ESB is a complete asynchronous messaging engine which does not block its worker threads on network io. Instead of waiting the threads for responses, it registers a Calllback for a particular request and return. When the response is received, the registered callback is used to correlate it with the request and do the further processing. If the back end server not respond back, it is required to clear the registered Callbacks after sometime preventing possible memory leaks. It is done by a timer task called TimeoutHandler. ‘synapse.global_timeout_interval’ parameter represents the time duration which a Callback should be kept in the Callback store. If we have configured a timeout duration at the endpoint level, this global timeout value is not taken into consideration for that particular endpoint. For all the other endpoints which doesn’t have a timeout configuration, this global parameter value is considered as the timeout duration.

synapse.global_timeout_interval can be configured from the synapse.properties file. synapse.properties file is located at /repository/conf directory. Default value is 120 seconds.












3 comments:

Anonymous said...

I setted EI configure and endpoint timeout but it have below response.

HTML response : 504 Gateway Time-out , The server didn't respond in time.

What should I pay attention to if I want to increase endpoint timeout?

There are 3 steps to set timeout but it appear fault....

1) Global timeout defined in synapse.properties (\repository\conf\synapse.properties)

synapse.global_timeout_interval=18000000

2) Socket timeout defined in the passthru-http.properties (ESB_HOME\repository\conf\passthru-http.properties )

http.socket.timeout=18000000

3) Also set timeout in API.





18000000
fault


Anonymous said...

3) Also set timeout in API.





18000000
fault


Minh Trần said...

Great post Chamara,

In my case, I did a test with calling an API. Following is my selected logs:

2020-10-05 15:38:43,585 - sThroughMessageProcessor-12 - DEBUG g.apache.synapse.core.axis2.Axis2FlexibleMEPClient: [] Setting Timeout for endpoint : Endpoint [IDGen--v1.0_APIproductionEndpoint], URI : http://localhost:8082/sequence/batchId/next?length=20 to static timeout value : 15000
...
2020-10-05 15:39:03,660 - sThroughMessageProcessor-13 - DEBUG .apache.synapse.core.axis2.SynapseCallbackReceiver: [] Callback removed for request message id : urn:uuid:be12d50b-503b-4095-a296-ee36f9964d29. Pending callbacks count : 0
2020-10-05 15:39:03,661 - sThroughMessageProcessor-13 - DEBUG .apache.synapse.core.axis2.SynapseCallbackReceiver: [] Synapse received an asynchronous response message

You could see that, we set an endpoint timeout to 15s. And, the logs show that the callback was created at 15:38:43, and removed at 15:39:03, so it takes 20 seconds.

During monitoring, I found it works correctly but sometimes I works incorrectly.

Could you suggest any way I could investigate the root cause ?

Thank you,