Issue with gas fee spikes and value leakage

Aztec
1

There is a significant disconnect between the current system Aztec relies on for sending information to L1, which creates a misalignment of incentives.

The node just sends a transaction and spams the exact same transaction with higher fees and just lets it go. Or outright fail to submit the transactions. Depending on the node configuration, or if the node is connected to an RPC service, and depending on how that service processes transactions, those transactions get dropped if they cannot make it in time. I do not see any attempt from the node to check, monitor, or try to submit them with adjusting the fee.

The rewards are all fixed at the moment. Currently, there is also no on-chain activity or anything; we have empty blocks, but after the TGE and especially as apps get deployed, we cannot afford these issues. While I am writing this, gas spiked again and I know satrk.eth is already run out of gas there and will not able to send proof with high gas. (Update) While I complete this writing he just fund the prover account 4-5mins ago. Since I write this I also want to send my thanks to satrk.eth as he didn’t leave us alone. I don’t know how long satrk.eth or us can keep up. We run prover for mina at a loss for 5 years. If it’s just hardware we would somehow manage it but with the gas cost it becomes hard to maintain/fund.

There is a misalignment of incentives when fee spikes happen; the majority of actors purposefully just skip participating, and there is no punishment for that.

For example, on epoch 2987, we are the only ones who sent the epoch root proof. If ours could not make it to the chain, though we make a lot of adjustments at the client level so our transactions can make it in, if that didn’t happen we would need to prune the last two epochs. For this proof, we spent 112 USD (https://etherscan.io/tx/0x15e9c5b52ede0ddf99c7ad45d9941c6d26fd99b03291390f1741225fe609d68e). While this was happening, a sequencer paid 153 USD for the block proposal https://etherscan.io/tx/0xe8fa2394d95007d149cdc17c590d7c00704cd42457da4ca82809d82bb377611c . If our proof did not make it to the chain, then this sequencer’s block proposal was also going to be for nothing. Epoch get pruned and all sequencers going to get slashed.

The issues here are that the sequencer rewards are already fixed to Aztec, so they lose a lot of money on these. Provers also lose because their rewards depend on block proposals and the proposal rate; when fee spikes occur, the rate drops to around 20 to 50 percent, so 50 to 80 percent of blocks are not getting proposed.

Every epoch, there are about 20 to 40 provers, multiples of which are funded by the same address, as can easily be seen on chain. They do not participate over a certain fee purposefully. So actually, we only have a few provers that are going to submit proofs when a spike happens, and because of how the node handles transactions, they also cannot make it to the chain. The chain is much more fragile than it looks because of that, and we cannot afford these halts or rollbacks when there are actually users doing transactions and settling DeFi transactions, etc.

I am outlining all of this because, as I mentioned at the start, it creates a misalignment of incentives. It is much clearer and more significant with provers, while it also exists with sequencers. It is much harder to collect data on sequencers and do analysis because an attester’s address can be any address, and we cannot see them when they are not proposing. This creates a situation where a few actors just extract most of the value from the layer and also make the layer fragile. These extractor actors already do not help at all, and we have dropped to a few that actually provide proofs every epoch for a long time, like satrk.eth. Rewards being low because of extractors, most of the long-time provers stopped running. Both at the sequencer and prover level, we need to have a better way to handle these and limit the value extraction from the layer, as that is just out of the pocket of everyone who believes in the Aztec vision and supports it in various ways, like holding tokens, running sequencers, or developing apps. While in a short time they are not that important, in the long term, these leaks become significant amounts. After the TGE, considering users and exchanges and everything else, it is much more important for the layer to be resilient and reliable.

5 Likes
2

can you share the estimate cost of running the proover so far please?

3

Is it by ETH gas only, or are hardware, electricity, and other costs included? Currently, blocks are empty, so it is much easier to run a prover and create the proof in the necessary time. We can probably generate 10 proofs in a single epoch, but if you do not cap the gas cost like how I outlined here, you are just digging yourself a bigger hole. As I outlined above, you do not actually contribute anything other than lining your pockets and extracting the leaking value from the layer. After the TGE, if there is no constant activity on the L2, proving high activity times will require significant hardware, which you burn money constantly to maintain. While you prove only a few epochs where there is significant activity and take the high reward, for the rest of the epoch, rewards are going to be divided by many duplicate ones, which makes it significantly unsustainable for actually doing the work for the chain. We have spent years optimizing the proving process to make it economically sustainable and support and invest in proving networks like zk-cloud and Fermah. Proving is not constant, but hardware cost is, so if you prove for multiple sources, you can get closer to economic viability. If we go down that path, it becomes much more complicated.

My focus here is that first, nodes do not have a good way or process for following up on transactions and trying to submit them to L1. After that, there is the gas spike issue which exists for both provers and sequencers, but I am not going to spend hours collecting and analyzing that data. I do not have time for that. In the long run, if we have a structure against this leakage, there will be times when it is just a question of whether you keep losing money or not. Then we will have a full ecosystem of value extractors, and the foundation will need to run some of the infrastructure themselves or pay for an infrastructure provider to run it. Every single chain out there ended up the same after a long while because the majority of the participants act the same, or they are the same participants from other places. So, I am just bringing this up early as I do everywhere else so people can discuss it and possibly come up with solutions. The only thing I can think of right now is that there will be some kind of gas metric relayed or proved, and enforcement can be more aggressive when it is supported by that data. It is a complex problem that needs to consider many variables.

4 Likes
4

Hi @emrepiconbello,

Thanks for raising this. You’re pointing at real pain points around gas volatility, transaction reliability, and incentive alignment. I want to be upfront that what we have today is not perfect. That said, many of the issues you describe are already accounted for in the protocol and client design, even if the current configuration and tooling can clearly be improved.

This will be a longer reply because the topic deserves a precise and technical response.

Point #1 : “The node just sends a transaction and spams the exact same transaction with higher fees and just lets it go…I do not see any attempt from the node to check, monitor, or try to submit them with adjusting the fee”

The node does actively price, monitor, and retry L1 transactions. This logic lives in l1_tx_utils.ts which you can find here

Initial transaction pricing

The initial priority fee is set as the maximum of:

  1. The p75 priority fee observed in the mempool
  2. The Ethereum node’s eth_maxPriorityFeePerGas estimate
  3. The median of the 75th percentile fees from the last 5 blocks

The intent is to avoid underpricing while also not blindly overpaying.

Node operators can further tune this behavior using the following parameters:

Env Var Default Effect on Initial TX
L1_GAS_LIMIT_BUFFER_PERCENTAGE 20% Safety margin on gas limit
L1_PRIORITY_FEE_BUMP_PERCENTAGE 20% Bump on the p75 analysis priority fee
L1_MINIMUM_PRIORITY_FEE_PER_GAS_GWEI 0 Floor for priority fee
L1_GAS_PRICE_MAX 2000 Hard cap on maxFeePerGas
L1_BLOB_FEE_PER_GAS_MAX 3000 Hard cap on blob fee

Monitoring and retries

Transactions are continuously monitored via monitorTransaction. A transaction is considered stalled based on elapsed L1 time since submission, and the node issues up to three speed-up attempts using the same nonce while bumping the priority fee.

Node operators can further tune this behavior using the following parameters:

Env Var Default Effect on Speed-up
L1_PRIORITY_FEE_RETRY_BUMP_PERCENTAGE 50% Bump percentage per speed-up attempt
L1_TX_MONITOR_MAX_ATTEMPTS 3 Max speed-up attempts before timeout
L1_TX_MONITOR_CHECK_INTERVAL_MS 1000ms How often to poll tx status
L1_TX_MONITOR_STALL_TIME_MS 12000ms When to treat a tx as “stalled”

Retries are spaced by L1_TX_MONITOR_STALL_TIME_MS.

Cancellation strategy

If retries fail, the node can issue a cancel transaction (0-ETH self-transfer using the same nonce, aggressively priced). This helps unblock the account and is usually cheaper than continuing contract interactions.

Node operators can further tune this behavior using the following parameters:

Env Var Default Effect
L1_TX_MONITOR_TX_TIMEOUT_MS 120000ms Max time before giving up on regular tx
L1_TX_MONITOR_CANCEL_TX_ON_TIMEOUT true Whether to send cancel tx on timeout
L1_TX_MONITOR_TX_CANCELLATION_TIMEOUT_MS 288000ms How long to wait for cancel tx
L1_TX_MONITOR_TX_UNSEEN_CONSIDERED_DROPPED_MS 72000ms Time before considering the cancel tx dropped from mempool and releasing the nonce

Cancel transactions themselves are retried (with a fixed 150% bump, not settable on the node) before the nonce is released.

So while there are certainly improvements to be made, the client is not “fire and forget.” It already implements fee estimation, monitoring, retries, and cancellation.

Point #2: “There is a misalignment of incentives when fee spikes happen; the majority of actors purposefully just skip participating, and there is no punishment for that.”

You are right that gas spikes stress the system. However, incentives do exist, particularly for provers.

Prover incentives

Provers are rewarded via an activity-weighted reward share mechanism. Rewards are not split evenly per epoch. Instead, each prover has an activity score that increases with consistent proof publication and decays when participation drops.

image
image1448×622 71.2 KB

Over time, provers who skip epochs see their reward share shrink, while consistent provers receive a disproportionate share of future rewards. This mechanism is explicitly designed to encourage participation even during temporarily unprofitable periods.

image
image1448×628 66.9 KB

The rate of decay and recovery is governance-controlled and adjustable. We need real data from provers to tune these parameters correctly, and that feedback is valuable. Please read here for a full walkthrough of the prover reward boost mechanism.

Concrete example

To make this more tangible, consider a simplified system with two provers, A and B.

Assume:

  • Total prover rewards for an epoch: 100 AZTEC
  • Both provers are starting from the same baseline
  • Each prover starts out with equal share of 0.1 as their activity score is constant below a certain threshold.

At this point, total shares in the system are 0.2, so rewards are split evenly:

  • A’s reward = 100 \times \frac{0.1}{0.2} = 50
  • B’s reward = 100 \times \frac{0.1}{0.2} = 50

Now assume there is a prolonged spike in L1 gas fees. During this period:

  • Prover B stops publishing proofs
  • Prover A continues publishing consistently

As a result, prover B’s activity score decays over time, while prover A’s share increases relative to the system. When gas conditions normalize and prover B resumes publishing, the total epoch rewards are still 100 AZTEC, but the share distribution has changed. For illustration, assume:

  • A’s share has increased such that it now captures ~90% of total shares
  • B’s share has dropped to ~10%

Rewards for that epoch now look like:

  • A’s reward ≈ 100×0.9=90
  • B’s reward ≈ 100 \times 0.1 = 10

The key point is that prover participation is not evaluated in isolation per epoch. Activity is rewarded over time, and periods of non-participation have a lasting cost. This is intentional: the mechanism is designed to favor provers who remain online and absorb short-term volatility, rather than those who opportunistically enter only during favorable conditions.

How fast prover B’s share drops as a result of their decision to not publish proofs during the gas spike is an updateable parameter (via a governance vote). In fact the entire prover rewards score mechanism is updateable. We need empirical data and research from provers and sequencers in order to set an acceptable mechanism with optimal parameters.

Sequencer incentives

Sequencers lose block rewards and fees when they choose not to propose. More importantly, they carry collective responsibility for proof inclusion, which leads to the next point.

Point #3: Provers also lose because their rewards depend on block proposals

So you’re right, let me rephrase what you are saying. There are two chain parameters that matter here:

blockReward: 400e18 # block reward is 400 $AZTEC
sequencerBps: 7000 # sequencer gets 70% of block rewards, provers get 30%

Each finalized block contributes rewards, 70% to sequencers and 30% to provers. Fewer blocks means fewer rewards, even though prover costs are fixed.

What is often missed is this:

If no proof lands for an epoch, the epoch is pruned, and the entire committee is slashed.

This means the epoch committee (and by extension the entire Aztec sequencer set) has a strong incentive to ensure proofs are delivered, even during gas spikes. They can do this by:

  • Proposing higher-value blocks
  • Adjusting reward parameters via governance
  • Coordinating off-chain payments to provers in extreme cases

In your epoch 2987 example, by submitting the only proof, you effectively prevented every committee member from being slashed 2,000 AZTEC. That is a significant amount of value protected. Today, the protocol lacks a native mechanism to route that value back to the prover who saved the epoch. That gap is real and on our roadmap.

We are very open to community efforts to design coordination mechanisms between sequencers and provers here. It is certainly on the Aztec core dev long term roadmap.

In Closing

I re-iterate my statement from the very top of this long post that some mechanisms to address incentives during periods of base layer congestion are already in place, others clearly need iteration, and governance will play a major role in tuning them correctly.

Some concrete areas we are actively thinking about:

  1. Publisher rotation when transactions stall
  2. Separate L1 submission profiles for provers vs sequencers
  3. Better economic modeling of prover and sequencer profitability
  4. Further tuning or redesign of the prover reward boost
  5. Better documentation and education around how these mechanisms work

Some mechanisms are already in place, others clearly need iteration, and governance will play a major role in tuning them correctly.

We move faster and make better decisions when provers, sequencers, and token holders share data, propose concrete changes, and pressure-test assumptions together. I strongly encourage that kind of engagement and am very open to specific proposals.

Thanks again for raising this early.

1 Like
5

This was mainly focused on prover comment. Sequencers; they do not have long slots, so they do not have much of a window for monitoring and adjusting, but a prover does, and from my observation, behavior is not reflected that way. For resiliency, we need it because the time window is very wide and, as I outlined, there are already only a few provers that provide consistent proof no matter the gas cost. Others fail, and while ours did for the epoch, ours also failed the following few epochs; thankfully, other provers did not fail the following epoch after that, which was also a single prover epoch.

I know that the system exists; the issue is that the proofs are every 40 minutes. Most spikes do not take that long, so they do not actually miss many activity points. There will always be people who game the system, but currently, they are the majority, which makes this situation problematic.

Because there are only a few that actually submit proofs at all gas prices and others fail, technically our prover takes all the reward, while the point I am making is that there is, again, fragility. The transaction submission for the prover does not try well for the 40 minute window; a lot of proofs are generated in under 3 to 5 minutes. If a proof is somehow failing, it can even try to generate the proof again; these are lacks on the software side. The second problem is that the punishment for low participation is not aggressive or long enough. I will go through the numbers and come up with a proposal if I can find the time. These two issues currently cause the number of “always participating” provers to dwindle, and with a low number, the software not trying hard enough on submission within the 40 minute window makes the system fragile. Currently, we have three consistent provers, but because of some gas estimation and other kinds of errors, sometimes they cannot submit; however, let us take the example that in that epoch, three of them submit. Our proof went in with a 12 gwei cost while gas was flying; for the rest of the epoch, gas was in the 20 to 30 range, so the other two submitted at around 25 on average. They would also burn 250 USD and get about 700 Aztec. In our epoch, it was about 2k; for the next epoch, the proof pool rewards were about half of ours, somewhere around 1,100 to 1,200, because of gas spikes where a lot of sequencers did not propose. On normal epochs, it is about 3.8k. These are significant reductions. Can we imagine a chain that skips like 60 to 70% of its blocks? Can we think about the UX from a user and product perspective? That is very harmful. That is why I believe the sequencer issue from a gas price perspective is important on the prover side; while I do not believe rewarding a prover for saving an epoch is that critical, sustainability for the consistent provers is more important because that will result in consistent provers for all epochs if we sort out submission inconsistencies too. In earlier times, we had seven to nine provers that consistently produced proofs every epoch; now we have three because we have 20 to 30 of them just picking low gas ones and diluting the rewards of the consistent ones. Three to five consistent ones is actually a good number, but the current state makes it unsustainable for them, and the node also does not try well enough for the 40 minute proving window; some errors just terminate the whole process while it could try many times in that window.

What does proposing higher value blocks mean; are we talking about transaction fees? I do not get these aspects in general. Is there an option in the prover selecting that if an epoch does not have this many rewards, it just does not produce the proof? The sequencer part is significantly more problematic. A sequencer paid 150 USD and, at max, that sequencer is going to get 20 Aztec from that proposal. This makes it so much worse if the sequencers propose multiple blocks in that time frame, especially for solo stokers or small validators. You are picked very rarely, and when you are picked, you propose three blocks, burn 500 to 700 USD, and get 60 Aztec or, in the case of their own stake, 900 Aztec in return; then you get into the committee two weeks later again to get another 20 to 60 Aztec or 300 to 900. So, what happens is they just exit the queue and stake with a provider, which makes the layer more centralized, or they just limit the gas fee. Everyone does that, so when there is a lot happening on L2, a gas spike cuts all the blocks and causes very bad UX for everyone in the ecosystem. I am throwing out random numbers, but since there is no direct punishment, they get punished if the prover does not do it, and even in that case, 2k is not that much of a punishment considering the costs they are taking when they propose. These disconnects create a gap and, while the ecosystem develops, we need to narrow those at a fast pace

Ethereum is not like a high frequency network; why do we need to rotate publishers on a stall? If it is stalling because of their node RPC, it will not recover with rotation. If it is a gas issue, just sending a higher fee transaction is a faster way to resolve it over canceling and submitting again. I do not get the reasoning behind this. Is it for high availability? Instead of multiple attesters, they should use one address as the main and rotate to a secondary only when they get stuck and to avoid wasting gas on every single proposal.

A separate profile for provers is 100% needed.

I believe currently tuning it will reduce the gap a bit, but because of the submission profile also being unreliable, if we go aggressive we could also punish the consistent ones; as I said above, if I can find free time, I would like to come up with some numbers and open it to discussion.

I do not want further incentives for provers because then it can be a very high-margin profit thing; then again, it becomes extraction of another kind. Reducing the “gas-limited ones’” participation or making them not viable will probably result better in the long run.

This is also a very impactful aspect. It will drop the participation barrier. When you bounce around a bunch of places to understand something because there is no clear structure, you waste time and at some point they move to the next thing. Good docs and information mean more people know about these aspects, which means more people can come up with solutions and suggestions.

Thanks for giving a very thorough and detailed answer. Our team is currently very occupied, and I am no longer actively participating there as I had other things to do. If someone has time internally, they will do it, as they need to decide whether to continue or not after running out of ETH.

3 Likes