Batch request failures

[ahhda]

Some of our batch calls are timing out when one of the calls within a batch call fails. Need some help understanding the logic behind batch calls.

Scenario:

We sent a batch request of size 49, all of these requests were sent to Alchemy which has been set up as secondary upstream in our case.

48 requests received a response, and 1 took ~10s, leading to HTTP Code error 499 (client closed request).

Dshackle logged only 48 of the 49 requests and our server timed out (timeout: 10s) waiting for the response.

All the failed batch requests have the same scenario mentioned above, requests get forwarded to secondary/fallback upstream, one of the requests fails/takes time, and the batch request fails.

Questions

• Why wasn't the request sent to our primary upstream? From our metrics, it wasn't behind more than 1 block.
• Is it possible not to split batch requests and send them to a single upstream?
• Is there a retry mechanism for when a request within a batch request fails?
• Is there anything we can do to prevent this? Any recommendations?

Access Logs from Dshackle (showing 48 out of 49 requests with request id, 347c69aa-70a1-4986-a845-cbdbdd2b1fae):
dshackle-logs-347c69aa-70a1-4986-a845-cbdbdd2b1fae.csv

Request that we sent from our server:
sent_request.txt

[{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000000ac22d66c43449baa7fe1b57d18db596430b72bd\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325492},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000003784e41b630bb89a575b336b6f4687f9b968eeec\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325493},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000ba12222222228d8ba445958a75a0704d566bf2c8\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325494},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000022de9399beedada74e155f82864759146a633381\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325495},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000ab99f7d53c6ed0d4a286d7b4fb290b7a7f48a762\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325496},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000007e8353c37b1a868f070e15fc20eeb32d006df5ff\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325497},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000006afa3c3eba7a6deb221c4e0e3e58a8f17f8baff1\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325498},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000994eda45b90096c1b5f98bb68ad8441ff030b623\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325499},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000dfdbd168726c94cbe40ccd4a0ffd9fde1ba740d3\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325500},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000dd16642f73362241f69a4114f23588e6381e12cf\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325501},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000001db45e131e1275664f2fb8a327d917b4d561ce09\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325502},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000079ded6e4472931aa3ae232ab940856c9c12b2ee1\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325503},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000061991a562fa5f16e668386229879041281ef42d6\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325504},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000049bf13357002be8d559514e1a9ca6f8cff1448cd\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325505},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000f0323234b34617e3dd8a3851903745efca63cad9\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325506},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000a9d409e2cfce9b14223a145a74614263b4bc53d8\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325507},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000842a0e42be46cb1f9aab006108dde5feccfe4ca3\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325508},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000008ea7ab60932d63e77bffab167206f1165e8e4639\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325509},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000e07a7526ebb336ea7d35d176fbd901b02a66c754\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325510},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000007ecfb50895ae952cbab223d22d500ebc6bbec251\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325511},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000f9786c0d6819ab1230c1eb8afebca0978cffd39f\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325512},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000a899d2fa5383f9e64e16cf4ee69d939153e0d7b5\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325513},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000e27d608fa95a3837ef6bded4ecea02395bc05147\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325514},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000dea79f5582ce8892b21043109bef7a181a91b93e\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325515},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000005b3eb8d9e70f3e4d8f2d62efead07cddeff3c1c9\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325516},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000001a022a833d2a874a2f3bd288789e45c1b23c108d\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325517},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000085c5a55ad0d6d8b9eb2a267fd13bd386092ed38\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325518},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000008f7c5569c12b2ec34e47dd3871c79094c9e26e4a\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325519},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000438acd7587ee5ac9c4074b99b64656cac389e9c1\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325520},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000300116093f01dadbff42d1010d4ae632c7cc0b4f\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325521},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000049051e824c6fcbd9b1135efce143cfa107e95086\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325522},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000001354ab1b760d5a8d2bc1c2f45d9c66a8842aba83\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325523},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000034946b384eaec86f8033d057cac93e939e1304fb\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325524},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000325f0b42c62159d29456ef7d660d347949515c26\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325525},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000001e42277ab89d6e4be2d9fbd7203e802728a44469\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325526},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000036b08494516f0497cd8810b87a714be7f0b59967\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325527},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000009ef03bd9caf23958cfd049d6e0b848ec3e9ba6e4\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325528},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000002f1187ef55f51b7b996b2583433aba28a3084b2c\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325529},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000b83732f456be276583f614bd6884fcdad47dce75\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325530},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000005839fa97a6042859a6ee7a196d9df51e0d9cb91e\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325531},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000b239d39af42e6dc2e984f7061d769a5fa68b2961\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325532},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000050479d727718c66f375381ec28239a8a09703bd7\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325533},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000069bc81c0fdcfb2a2b1926a3cf07799240356e681\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325534},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000a753acc10e8ade9621d9d55c3871670b21d1c9b4\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325535},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000025c85e06e84e0fb1cfe2d1c8388ed6d0bbce9113\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325536},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000dc9fc4c325b11814f14e885575939c70c8ef800c\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325537},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a0823100000000000000000000000034579d20e66c400cee0b523f533b457a8087f844\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325538},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a08231000000000000000000000000f63c52d252a229bd4bd8d56e032ae2b304e2cdc6\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325539},{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"data\":\"0x70a082310000000000000000000000009bb04eed717eacf28b6deface1721721552e0adc\",\"to\":\"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c\"},\"latest\"],\"id\":253325540}]

One request from the batch that was failed/delayed (format ... REQUEST_TIME <REQUEST> RESPONSE):

172.19.0.2 -  [30/Oct/2022:04:24:26 +0000] "POST /alchemy HTTP/1.1" 499 0 "" "ReactorNetty/1.0.11" 9.936 <"{"jsonrpc":"2.0","id":1,"method":"eth_call","params":[{"data":"0x70a082310000000000000000000000007ecfb50895ae952cbab223d22d500ebc6bbec251","to":"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c"},"latest"]}" >""

One of the requests from the batch that was successful:

172.19.0.2 -  [30/Oct/2022:04:24:17 +0000] "POST /alchemy HTTP/1.1" 200 102 "" "ReactorNetty/1.0.11" 0.493 <"{"jsonrpc":"2.0","id":1,"method":"eth_call","params":[{"data":"0x70a082310000000000000000000000000ac22d66c43449baa7fe1b57d18db596430b72bd","to":"0xc4c96166c46874a108a9d288bebd1d8c804a5d6c"},"latest"]}" >"{"jsonrpc":"2.0","id":1,"result":"0x0000000000000000000000000000000000000000000000000000000000000000"}"