Things Hidden Since the Foundation of Blockspace
Oct 19, 2022
Originally published at https://www.aniccaresearch.tech on October 19, 2022.
“When the whole world is globalized, you’re going to be able to set fire to the whole thing with a single match.” -Rene Girard
Special thanks to Travis for the help on the data, and @AutomataEmily, @llllvvuu for the feedback on the draft.
Introduction
How much do you really pay for a transaction?
The answer may not be as straightforward as it seems. Transaction costs in the traditional finance encompass explicit costs and implicit costs. Explicit costs are the direct costs of trading, such as transaction fees and taxes. Implicit costs are indirect. The bid-ask spread, market impact, delay, and unfilled trades all contribute to implicit costs.
The implicit transaction cost is a well-known concept in traditional finance and is often rigorously examined in market microstructure research. Implicit cost is also pervasive on blockchains, but much less covered.
In this article, we discuss the fundamental cause of the implicit cost of transactions in crypto, which is inherently rooted in the characteristics of the blockspace microstructure. We use observations from several case studies of discrete on-chain events, including notable NFT mints, stablecoin meltdowns, and airdrops to extract common patterns. In particular, we analyze their impacts on network fees, how the impacts decay, as well as the empirical negative externalities on L1s and rollups. As a coda, we discuss the long-term effects of implicit costs of blockspace for on-chain companies and projects, as well as the importance of gas expenditure for the frequent users of blockspace.
The Hidden Cost of Blockspace
Why are there implicit costs in crypto? To understand this, we must first understand blockspace.
Blockspace has been well-established to be the core commodity of crypto, serving as the true “real estate” of blockchains by providing the actual resources for the execution and storage of state changes. More granularly, blockspace is an amalgamation of finite resources provided by full nodes: CPU time, RAM, power consumption, bandwidth, and disc usage. All on-chain actions compete for these resources. Even though different on-chain activities consume different resources differently, they are treated uniformly and priced in the same units of gas. Gas is the single meter of value and time-preference of these resources. In a sense, it is regarded as a measurement apparatus that aggregates all private estimates of the value of blockspace at any given point in time.
Gas also sets the bound for the amount of resources that can be consumed at each given time. The higher the limit is, the harder it is to run a full node. The blockspace limit is intended to protect the decentralized nature of monolithic blockchains from centralization. Some chains increase the gas limit to increase throughput artificially, but this makes it harder for others to run a full node by increasing the resource load.
While gas provides a simple abstraction for the many resources that go into blockspace production, its market behavior is a bit less so. This is largely a reflection of the complexity of pricing blockspace itself. Intuitively, the value of blockspace differs greatly across chains; its value is largely a function of demand, quirks in the underlying chain’s consensus engine, and several exogenous market factors. Less intuitively, the value of blockspace between blocks and even within the same block on a chain can differ greatly. This is largely due to a very important characteristic of blockspace: it has a time premium.
The final execution price of a transaction remains implicit until the transaction gets included in a block and gets appended to the blockchain. Consumers of blockspace tend to have varying time preferences, but generally, users prefer settling their transactions on the blockchain as soon as possible. A user who simply uses the chain to transfer funds from one address to another usually would not care wherein a block their transaction was included, however many transactions on-chains are time-sensitive interactions with DEXes and lending protocols.
In these cases, users generally have high time preferences as having your transaction get included later in the block puts you at risk of the chain state-changing adversely. For example, when everyone is competing for an NFT series, wouldn’t you rather be certain that your mint transaction goes through before another user can mint and potentially snag the last NFT? Turns out, these types of transactions where users have high time preference take up a majority of blockspace on chains.
The top users of blockspace on Ethereum over the past month; most are exchanges.
The notion of blockspace’s time premium is most significant for MEV. Generally, algorithmic agents will compete to take advantage of the ability to order transactions, usually by including their transactions before anyone else, to capitalize on profitable on-chain opportunities like liquidations and sandwiching DEX transactions.
An astonishing sandwich attack took place across 8 DEX transactions (Credits to Bert Miller)
The time premium is not always strictly a top-to-bottom phenomenon for MEV activities. For instance, in the case of sandwiches, the most prime blockspace is not necessarily at the top of the block, but before and after a large victim transaction. Similarly for liquidation, the most valuable blockspace may be immediately after an oracle transaction rather than at the very top of the block. To bring back the real estate analogy, being *neighbors *with a suddenly attractive piece of real estate or auction may be extremely valuable.
MEV will keep growing. In the future, we expect the majority of the blockspace will be filled with MEV activities. Blockspace is a new kind of market, a market of constant action, of waves and fades, of tricks and ruses.
So how does this relate to implicit costs? The complexity of blockspace pricing, as a function of the presence of MEV, and the usage of gas as a catch-all unit for resource pricing leads to interesting implications on actual transaction costs, specifically:
Gas estimates are hard to consistently keep accurate given high variance in demand.
Certain MEV transactions have a non-local impact on the fee market (e.g. flash loans, Searchers bundles etc.)
Large on-chain events (e.g. NFT mints) severely impact fee markets on varied time horizons and varied scales.
In the next section, we look at how these factors empirically influence gas fees across a variety of different scenarios.
Blockspace Microstructure
Market microstructure studies the cost of trading and the impact of trading costs on the short-run behavior of the prices, specifically how the market reacts to an execution. Does the market stay where it is after an execution? Or does it move further away? Or does it come back to some degree?
The value of blockspace is liable to drastic changes during various high-impact market events. In turn, this can drastically alter the behavior of fee markets and related markets on both L1s and related rollups. Through observing historical events, we can find patterns of behavior across market events of the same type.
When events occur that either present the opportunity for significant financial gain on-chain or necessitate users to make many urgent transactions on-chain to avoid financial loss, blockspace becomes more valuable than it usually is as a result of its associated time period. During these time periods, chains become congested and network fees spike, which can price out retail users and even institutions.
One might be tempted to circumvent this issue by changing the way blockspace is bid on (fee markets) or adding some sort of way to deterministically order transactions. In practice, however, this has largely been difficult. Changing the structural nature of fee markets, as was the case with EIP1559, usually only helps alleviate some portion of the problem (i.e. reducing the variance of fees and the time it takes to revert to the mean). Removing fees entirely is out of the equation, as it would mean a lack of sustainable incentives for network security, and statically pricing blockspace allows for spamming of networks and potential Denial-of-Service (DoS) attacks, which we will see in the following example.
A notable example of a case where deterministic transaction sequencing and fixed transaction costs have failed is the Solana blockchain; Solana’s Gulf Stream transaction-forwarding protocol and Proof-of-History protocol in tandem were meant to serialize transactions to help avoid the complications arising from the fee markets present on chains like Ethereum. Instead, Solana’s fees are purely a function of recently processed transactions and deterministic fee parameters set by the validator set.
However, in many cases, users have worked around this and attempted to snag blockspace faster by spamming the network with transactions, with the hopes that one of those transactions will go through sooner than later. This, at times, has had devastating consequences for the network.
In September 2021, the GRAPE protocols IDO led to some validators receiving over 300,000 transactions per second from bots attempting to snag tokens, which overwhelmed the hardware of network validators, causing the network to halt and restart via hard-fork. A similar incident occurred more recently at the beginning of May, with bots trying to spam validators to mint NFTs. This ended up crashing the network for 7 hours, with some validators reporting upwards of 100 GB/s in network ingress, once again overwhelming validators. Since then, Solana has moved to implement fee markets to curtail this attack vector.
A graph depicting Solana slots per second; shortly after the GRAPE IDO the network crashed due to spam. (Credits to Jump Crypto)
Ultimately, any method that enables users to get favorable transaction ordering will contribute to blockspace time premium, even if on-chain methods do not exist. As aforementioned, this is critical to understanding why blockspace behaves the way it does, how we might better anticipate implied costs, and how we might counter them.
Case Studies
What trends do we observe from this phenomenon of blockspace time premium? In the following sections, we analyze a few such scenarios and their implications.
Otherdeed and other NFT Mints:
A staple of the NFT community, Bored Ape Yacht Club has continued to be one of the most valuable and sought-after NFT collections even in light of recent market events. With the BAYC ecosystem now encompassing both ApeCoin and plans for a play-to-earn MMORPG that integrates other Yuga Labs’ owned IP (like Crypto Punks), it was no surprise when they announced, an NFT collection that would give users rights to scarce digital land in the BAYC universe. However, the collection quickly got flak for clogging the Ethereum network and spiking the price of gas at the time of mint. As users rushed to mint valuable Otherdeed NFTs, blockspace on the Ethereum network got dramatically bid up. Let’s take a look at some trends.
Gas price by fee tiers
The Otherdeed mint started around block 14688900; we took a look at the gas prices between block 14688900 to block 14689800. A few insights can be gleaned from this:
While the spike in gas price was near-instantaneous (likely as a function of bots waiting to mint by bidding high gas fees), it took nearly 1000 blocks (approximately 3 hours!) for gas prices to revert to the mean; during this time ETH network fees for all users were incredibly high.
Top-of-block space (95th percentile priority fee) decayed at a slower rate than lower percentile priority fee tiers; while it is difficult to pinpoint causation here, it would likely be a fair assumption that during a high-value mint like this one, market agents looking for urgent transaction inclusion (i.e. the mint transaction) are more likely to consistently bid on blockspace between blocks rather than wait for network prices to revert.
We also make some observations about the volatility of fees at different priority tiersPercentile fee priority
As we can see, higher percentile fee tiers (blockspace closer to the top of the block) tended to have much lower volatility than lower percentile fee tiers. Again, this is most likely due to a function of the characteristics of blockspace buyers at different tiers; agents bidding for high percentile fees are likely to sustain their bids until they get filled (their transaction gets included), whereas those who are not aiming for urgent inclusion are likely to wait to transact later if their transactions don’t go through as opposed to continuing to pay high fees for bids.
In observing other high-value mints at inception, we can see similar trends. For example, these charts for the Tubby Cats NFT mint exhibit the same trends as the Otherdeed mint:
Gas price by fee tier for Tubby Cats NFT mint
The volatility of gas fees for Tubby Cats NFT mint
Surprisingly, even in Dutch auction formats, we see similar trends. had its initial mint in the format of a Dutch Auction with a starting price at 1ETH. The auction cleared in three minutes; over that period of time higher fee priority tiers once again took the longest to decay, with the volatility of the higher fee priority tiers being generally lower than the lower fee tiers.
Priority fee tiers during Azuki Dutch Auction mint
The volatility of fee tiers during Azuki Dutch Auction mint
Stablecoin Meltdown (UST and Maker Crises):
Now let’s look at the impacts to the value of blockspace over more prolonged market events. Over three days from May 7th to May 9th, Terra’s native stablecoin deviated significantly from peg before completely collapsing after the 9th. The collapse of the UST stablecoin had widespread consequences, including mass liquidations and arbitrage opportunities on both centralized platforms and decentralized platforms, including those on Ethereum and Avalanche. Where there exist such phenomena, there exist users trying to limit their risk in panic and arbitrageurs attempting to profit from the chaos.
The result is naturally a sustained increase in demand for blockspace. Below are charts from Etherscan and Snowtrace showing average gas prices during May on Ethereum and Avalanche respectively.
Gas prices on Ethereum and Avalanche during May 2022.
Looking specifically at Ethereum, which has a lion’s share of value locked in and flowing through DeFi protocols:
During this market downturn, we can observe a few things:
The time series is particularly spiky, especially for top-of-block (higher priority fee percentiles); as opposed to one-off NFT mints, prolonged market events are a bit more step-wise; periods of high demand and thus high fees are followed by periods of relative calm. This intuitively makes sense; under such market meltdowns, there is usually a lot of upfront activity with smaller peaks of activity to follow as participants attempt to get out over the course of days.
It still takes higher fee tiers more time to revert to the mean.
Unlike NFT mints, higher fee tiers have much greater volatility. This is likely in part due to the first observation; we can assume a lot of on-chain activity (i.e. liquidations, sandwiches, and other urgent transactions) occurs early on as the market downturn is going in full swing (hence the biggest spike at the beginning of the graph) and at some smaller lower peaks, with periods of relative calm. During periods of high on-chsain activity, blockspace demand is unusually high, with many people preferring urgent transaction inclusion. Relatively, less-urgent transactions (such as sends) are not likely to see much fluctuation in fee pricing.
We can replicate this same analysis for another notorious stablecoin meltdown, the MakerDAO liquidation crisis in March 2020. Looking at the data:
We can observe the exact same trends in blockspace value across both crises, which occurred nearly two years apart.
The Optimism Airdrop:
Layer-2 solutions such as Optimism and Arbitrum are generally cheaper to transact on than Ethereum Mainnet thanks to the ability to amortize L1 data storage costs over a batch of transactions and the efficiency of the sequencer model. While optimistic rollups have shown to be effective in reducing the costs associated with transacting networks, they are still as liable as ever to sharp changes in the demand for blockspace and fee congestion. The fees users pay for an L2 are the sum of execution costs for the transactions and the costs of storing the data for the proof on the underlying L1; correspondingly, this means that congestion at the L1 that increases L1 fees corresponds to increases in L2 fees, and congestion at the L2 layer will similarly increase L2 fees just like with base Ethereum.
The most notable example of this was the Optimism token airdrop on May 31st, an airdrop for the most active users of Optimism and Ethereum. As soon as the airdrop was claimable many users rushed to sell their airdrop for other assets, creating a massive uptick in transaction activity.
Correspondingly, fees hit all-time highs.
However, if we look at the ETH saved per transaction over the course of Optimism’s history, we notice an interesting trend when comparing the usage spike from Optimism’s airdrop to a spike earlier in the month.
Around the middle of May, we noticed a spike in transactions led to a similar spike in ETH-saved per day. It seems to accurately reflect the main proposition of a rollup: bundling transactions together into a single proof leads to economies of scale. However, the gas savings around the airdrop, which was of much greater magnitude, showed at best slightly above average savings over regular usage.
As Alex Beckett of Celestia covers, this is only because rollups experience economies of scale-up until the marginal cost of including a transaction equals the average total cost of transactions in the batch when considering all fees. In other words, even L2s reach a point where the cost of an additional transaction (execution + storage) begins eating into the economic savings brought forward by L2 scalability.
Accordingly, L2s are also liable to congestion and a sharp uptick in fees during large market events. In some cases, this means the network needs to get shut down, as was the case with Arbitrum’s airdrop ). Additionally, most L2s currently have a single rollup operator/sequencer handling transaction ordering, batching, and proof generation and have nascent ecosystems that have not warranted substantial MEV activity. It’s highly likely that as such activity grows on the L2s that such MEV activity would accordingly increase; in this case, we can expect “blockspace” on the L2 to get even more valuable *and *for “blockspace” on L2s to exhibit similar time premiums (and perhaps even direct relays of transactions to rollup operators similar to Flashbots).
As such we can see that rollups also exhibit similar trends to underlying L1s, even with the economic savings they bring.
Conclusion
The behavior of blockspace is a critical determinant in the user experience, and by extension, adoption, of blockchains.
As commercial activity on-chain continues to explode over the long run, the cost of blockspace will only grow. While this is mitigated by the many advances in scaling the underlying infrastructure (i.e. L2s and modular settlement layers), the high variance that we have observed with fee markets will still persist, largely as a consequence of how nearly all such markets are structured in a similar way.This still poses an issue to both on-chain services and businesses and their users, as most on-chain services/businesses either: a) subsidize the costs of transaction fees out of pocket, which makes cost management hard, given said variance, or b) pass costs onto users, which can lead to unstable usage of protocols, again given said variance. As an example, Yuga Labs issued the following statement after the Otherdeed mint:
Yuga Labs apologized for failed transactions, for which they paid gas fees, and promised to refund the costs of those transactions.
In the future, the paradigm of gas cost management (or the lack thereof) is likely not sustainable if mass adoption is the goal. This suggests the need for products and services that enable frequent consumers of blockspace to hedge their gas expenditure costs, much like how airlines hedge against fuel swaps. In the follow-up to this paper, we will explore this topic in great depth.