POST requests are rejected (KeepAlive is not working?)
Cross post from Discord
Environment (Mainnet, test network, build from source, ...)
Confirmed this issue on self-hosting nodes Florencenet and mainnet.
Using tezos/tezos:v9.1 docker image
Summary
Hi, I have trouble regarding tezos-node RPC.
I develop a Java application, which connects to tezos-node (florencenet) v9.1 running on docker swarm. I noticed POST requests are failing while GET requests are successful. I believe rpc and p2p ports are exposed properly. The node is fully synced. I did not see this issue with tezos-node v8.x. Is there any change applied to tezos-node regarding RPC between v8 and v9?
This is all I could notice from change logs, but I do not know if it matters. https://tezos.gitlab.io/CHANGES.html#version-9-1
Some RPCs now send their response in chunked transfer encoding. Additionally, the implementation allows for more concurrency internally: it allows RPC requests to be treated even if a request is currently being treated. This leads to some improved response times on some RPC requests.Then, I did some experiments for troubleshooting. I posted the same POST requests to public nodes using the same java client, which was successful.
- florencenet.smartpy.io (https)
- florence-tezos.giganode.io (https)
Besides, I sent the same POST request via curl to my hosting node, which was successful too. This confused me. This means my node is okay and my Java implementation is bad. I have not changed the java client for a while. So, I suspected something changed on tezos-node side.
Expected behavior
POST requests should be accepted
Actual behavior
POST requested were rejected
Steps to reproduce
I cannot share my java code, but basically our application is built on Spring Boot. Http requests are handled by RestTemplate, which is a standard library for REST client. We do not customize it much.
Instead, I can share curl command:
$ cat tezos-forge-post-body.json
{"branch":"BLwpfQXCi9C4CkJPUTaixCw4DQPVkZ6m1vmb13PCTxRqxsBiDkR","contents":[{"kind":"origination","source":"tz1aSEPhU8TqhuZhLiSpQMfRozcQHVRD1rJ8","fee":"2840","counter":"146661","gas_limit":"10600","storage_limit":"60000","balance":"0","script":{"code":[{"prim":"parameter","args":[{"prim":"or","args":[{"prim":"unit","annots":["%default"]},{"prim":"pair","args":[{"prim":"pair","args":[{"prim":"nat","annots":["%counter"]},{"prim":"or","args":[{"prim":"pair","args":[{"prim":"mutez","annots":["%amount"]},{"prim":"contract","args":[{"prim":"unit"}],"annots":["%dest"]}],"annots":[":transfer"]},{"prim":"or","args":[{"prim":"option","args":[{"prim":"key_hash"}],"annots":["%delegate"]},{"prim":"pair","args":[{"prim":"nat","annots":["%threshold"]},{"prim":"list","args":[{"prim":"key"}],"annots":["%keys"]}],"annots":["%change_keys"]}]}],"annots":[":action"]}],"annots":[":payload"]},{"prim":"list","args":[{"prim":"option","args":[{"prim":"signature"}]}],"annots":["%sigs"]}],"annots":["%main"]}]}]},{"prim":"storage","args":[{"prim":"pair","args":[{"prim":"nat","annots":["%stored_counter"]},{"prim":"pair","args":[{"prim":"nat","annots":["%threshold"]},{"prim":"list","args":[{"prim":"key"}],"annots":["%keys"]}]}]}]},{"prim":"code","args":[[[[{"prim":"DUP"},{"prim":"CAR"},{"prim":"DIP","args":[[{"prim":"CDR"}]]}]],{"prim":"IF_LEFT","args":[[{"prim":"DROP"},{"prim":"NIL","args":[{"prim":"operation"}]},{"prim":"PAIR"}],[{"prim":"SWAP"},{"prim":"DUP"},{"prim":"DIP","args":[[{"prim":"SWAP"}]]},{"prim":"DIP","args":[[[[{"prim":"DUP"},{"prim":"CAR"},{"prim":"DIP","args":[[{"prim":"CDR"}]]}]],{"prim":"DUP"},{"prim":"SELF"},{"prim":"ADDRESS"},{"prim":"CHAIN_ID"},{"prim":"PAIR"},{"prim":"PAIR"},{"prim":"PACK"},{"prim":"DIP","args":[[[[{"prim":"DUP"},{"prim":"CAR","annots":["@counter"]},{"prim":"DIP","args":[[{"prim":"CDR"}]]}]],{"prim":"DIP","args":[[{"prim":"SWAP"}]]}]]},{"prim":"SWAP"}]]},[[{"prim":"DUP"},{"prim":"CAR","annots":["@stored_counter"]},{"prim":"DIP","args":[[{"prim":"CDR"}]]}]],{"prim":"DIP","args":[[{"prim":"SWAP"}]]},[[{"prim":"COMPARE"},{"prim":"EQ"}],{"prim":"IF","args":[[],[[{"prim":"UNIT"},{"prim":"FAILWITH"}]]]}],{"prim":"DIP","args":[[{"prim":"SWAP"}]]},[[{"prim":"DUP"},{"prim":"CAR","annots":["@threshold"]},{"prim":"DIP","args":[[{"prim":"CDR","annots":["@keys"]}]]}]],{"prim":"DIP","args":[[{"prim":"PUSH","args":[{"prim":"nat"},{"int":"0"}],"annots":["@valid"]},{"prim":"SWAP"},{"prim":"ITER","args":[[{"prim":"DIP","args":[[{"prim":"SWAP"}]]},{"prim":"SWAP"},{"prim":"IF_CONS","args":[[[{"prim":"IF_NONE","args":[[{"prim":"SWAP"},{"prim":"DROP"}],[{"prim":"SWAP"},{"prim":"DIP","args":[[{"prim":"SWAP"},{"prim":"DIP","args":[{"int":"2"},[[{"prim":"DIP","args":[[{"prim":"DUP"}]]},{"prim":"SWAP"}]]]},[[{"prim":"DIP","args":[{"int":"2"},[{"prim":"DUP"}]]},{"prim":"DIG","args":[{"int":"3"}]}],{"prim":"DIP","args":[[{"prim":"CHECK_SIGNATURE"}]]},{"prim":"SWAP"},{"prim":"IF","args":[[{"prim":"DROP"}],[{"prim":"FAILWITH"}]]}],{"prim":"PUSH","args":[{"prim":"nat"},{"int":"1"}]},{"prim":"ADD","annots":["@valid"]}]]}]]}]],[[{"prim":"UNIT"},{"prim":"FAILWITH"}]]]},{"prim":"SWAP"}]]}]]},[[{"prim":"COMPARE"},{"prim":"LE"}],{"prim":"IF","args":[[],[[{"prim":"UNIT"},{"prim":"FAILWITH"}]]]}],{"prim":"DROP"},{"prim":"DROP"},{"prim":"DIP","args":[[[[{"prim":"DUP"},{"prim":"CAR"},{"prim":"DIP","args":[[{"prim":"CDR"}]]}]],{"prim":"PUSH","args":[{"prim":"nat"},{"int":"1"}]},{"prim":"ADD","annots":["@new_counter"]},{"prim":"PAIR"}]]},{"prim":"NIL","args":[{"prim":"operation"}]},{"prim":"SWAP"},{"prim":"IF_LEFT","args":[[[[{"prim":"DUP"},{"prim":"CAR"},{"prim":"DIP","args":[[{"prim":"CDR"}]]}]],{"prim":"UNIT"},{"prim":"TRANSFER_TOKENS"},{"prim":"CONS"}],[{"prim":"IF_LEFT","args":[[{"prim":"SET_DELEGATE"},{"prim":"CONS"}],[{"prim":"DIP","args":[[{"prim":"SWAP"},{"prim":"CAR"}]]},{"prim":"SWAP"},{"prim":"PAIR"},{"prim":"SWAP"}]]}]]},{"prim":"PAIR"}]]}]]}],"storage":{"prim":"Pair","args":[{"int":"0"},{"prim":"Pair","args":[{"int":"2"},[{"string":"edpku5Ax6DT1UgtU8CcYhoEsvFPjzYLRCXpVouJGWLLia6upeZBEYq"},{"string":"edpku4oPk1uDgRKJG6q94pWwRvymGSyTk1jqGUyAaeRPFfNEZgXGB7"},{"string":"edpkuXLqzhPyctR8pXt8PqQCN5XBa4VvuUVXvnGGGmR8cXLHuzoCyw"}]]}]}}}]}
$ curl -vvv -H "Content-Type: application/json" -d @tezos-forge-post-body.json http://<masked>:18732/chains/main/blocks/head/helpers/forge/operationsHistory mode
full by default
Logs
Failing example (Java Http Client)
it appears HTTP connection was shut down right after the request was posted even with KeepAlive is enabled
Successful log (curl command)