ERROR mpool push: failed to push message

[pk-controller]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

Daemon: 1.31
lotus version 1.31.0

./lotus-shed send-csv --from <FIL source address> <filename>.csv

Repro Steps

1. ./lotus-shed send-csv --from .csv
2. It happens whenever running a batch send, batch approval, or batch proposal
3. after 10 messages or so, this varies
4. ERROR mpool push: failed to push message: failed to add locked: too many pending message for actor
   ...

Describe the Bug

So whenever we are sending a batch proposal, approval, or batch send. It will execute about 10 messages and then we get the error. We have seen this error now on 3 different computers.

Logging Information

N/A

[rvagg]

@pk-controller did this used to work but is now not working? I guess you're using a public API, like glif. The "untrusted" mpool push method has a limit of 10 messages at a time, to prevent spam attacks. A "trusted" path has 1000, but that would require not going through the gateway which imposes this limit, but running your own node or getting access to a node directly.

We could change send-csv to do smarter batching, or detect this error and adjust, but it would slow it down.

[jennijuju]

thanks rod

We could change send-csv to do smarter batching, or detect this error and adjust, but it would slow it down.

I don’t think we would be able to prioritize this as this is not a main use case of lotus.

@pk-controller Id suggest you to consider to run your own node, or reach out to @ArseniiPetrovich from Protofire for node service/api services.

[pk-controller]

@rvagg yes this used to work no issue, for the last 4 years, we would run up to 300-400. it just changed to the 10 maybe a month or two ago. I have always been using a lite node, it may have changed, but not recently. I use the glif lite node:
FULLNODE_API_INFO=wss://wss.node.glif.io/apigw/lotus lotus daemon --lite

[ArseniiPetrovich]

Hm, we haven't changed anything related to this on our end. The only thing is that without access key it is nowadays not possible to send over 100 requests per minute, but this was the case for many months now, if not for a year. Not sure how it was working on the 300-400 scale then at all.

[rvagg]

Oh, I recall now. In #12431 I did a couple of things, the main thing was plugging a hole whereby eth_sendTransactionRaw was using an untrusted path so it was possible to spam a gateway with messages and not be limited, but I was also fixing a hole with MpoolPush that was completely ignoring the fact that a push should be "untrusted" in certain circumstances, thereby undoing the whole point of having a trusted/untrusted path split for gateway vs raw access. You can see this fix in chain/messagepool/messagepool.go. I didn't call attention to it in the PR but we had discussed this internally and agreed to ship the fix quietly.

So what you're now bumping in to is the proper, intended limitations of the original code, you just happened to be bypassing it and were being allowed to spam glif with your messages where you probably shouldn't. I think the right way to do this is to use an API key with glif and/or negotiate with @ArseniiPetrovich to get access to a raw managed node or some other kind of un-capped API.