We are thrilled to share how we’re thinking about the work we’re doing to build the new proposed proof of space format. Along with the new proof of space, we are developing two new plot formats, one for HDDs and another for SSDs.

While the original proof of space was a significant step forward from other consensus mechanisms, the tech landscape has continued to evolve and we need an updated approach.

We’ve collaborated with leading experts in cryptography and blockchain technology to develop this new proof of space format. Our new solution will harness the latest advancements while staying true to the Chia blockchain’s commitment to decentralization and environmental responsibility.

Below we’ve included the latest info and updates from DrNick on the plot format updates.

Hello Chia Community, I’m Nicolas Halper (aka Dr Nick), and I spent over 2 years in research and development on Chia’s proof of space before releasing DrPlotter in February 2024 that introduced the first “compressed” k32 plots under 25GiB in size for a space savings of over 75%. Shortly after its release, I joined CNI to work on a new plot format that overcomes a lot of the shortcomings of the original format. For the past few months, I have been working closely with Bram Cohen, Krzysztof Pietrzak from IST Austria (co-author with Bram on the Beyond Hellman’s paper on proofs of space), as well as team member Harold Brenes (Bladebit) and consultations with Dan Boneh (professor in applied cryptography at Stanford). This combination of various backgrounds and talent has led us to structure a new proof of space that varies in a few subtle but substantial ways from the original, and I’m excited to say that the results have surpassed initial expectations.

First, I’ll recap the issues with the original plot format that led to this new development. You’ll then be introduced to the proposed new format and its expected requirements for plotting and harvesting, as well as the timeline for its release. For those of you who want to dive deeper into the technical changes for the newly proposed proof of space, there is a details section at the end. I’ll round up this post with answers to some of your most pressing questions!

Problems with the original format

Chia’s goal has long been to reduce the energy consumed by Proof-of-Work networks such as Bitcoin, while maintaining the same security guarantees. In order to achieve this, most of the cost and resources are used up front in constructing the plot files, with little energy then used to retrieve the proofs to satisfy the challenges from the network. However, with the advent of DrPlotter and other third party vendors, many farmers began to use GPUs to farm with “compressed” plots, which requires significantly more ongoing energy than what was originally intended.

More pressingly, rental attacks – where an attacker can rent thousands of GPUs online and grind full proofs with little or no storage, with intent to gain the majority of the network space– became a foreseeable risk. DrPlotter technology can do a phase 1 (the part of plotting where you gather all the proofs together) in under 1 second on an 8 x H100 cluster. This would allow such a cluster to spoof almost 1 PiB of netspace. While an 8 x H100 cluster would cost over $300k to purchase today, and is much more expensive than purchasing the 1 PiB in storage, it can be rented by the hour. To spoof 10 EiB could be done by renting approximately 10,000 of these clusters. Depending on Chia’s current netspace and who has the fastest timelord, an attacker could win the majority of blocks with under 30,000 of these clusters. While getting this many clusters online today would be practically infeasible (both in terms of availability and the upwards pressure on price), given the quickly growing AI segment these clusters will only continue to grow and become more cost effective. If availability and pricing becomes possible at $30/hr for H100x8, then the attacker would only need under $1 million dollars per hour to launch their attack. Addressing this sooner rather than later is critical to secure the network’s future.

We are confident to have addressed the two main issues above in the new proof of space. In particular, security for the network will be at least 1000x more resistant to rental attacks, so in the above scenario where H100x8 clusters become highly available and “cheap”, an attacker would require well over $1 billion dollars per hour. We are also able to introduce two smaller size k-formats, the k30 and k31 sizes for mainnet, at no additional security risks to the network, which you can see in the next section.

Overview of new plot format

Plot Sizes

Currently, the new plot format is expected to have the following k sizes on mainnet:

• K30: ~25GiB in size for default HDD friendly format.
• K31: ~50GiB in size for default HDD friendly format.
• K32: ~100GiB in size for default HDD friendly format.
• K33+: each increase in K will approximately double the previous K’s plot size.

On first release, we will have an HDD friendly format available that minimizes random disk seeks. There will also be an additional SSD format which uses a new Benes compression algorithm for up to 20% extra space savings over the entire plot. While still pending further research, this is expected to be released some months after the HDD friendly format. Due to the increased number of random seeks required for Benes compression, this format will only be usable on SSD storage. More details on this can be found in the section on Benes compression on the docs site.

Note, care should still be taken when choosing smaller plot sizes for farming on HDDs even with HDD friendly formats, as each plot imposes a certain load due to seek times. The limit for number of plots per HDD will be heavily influenced by the proof of space filter settings, which have yet to be tuned or finalized. In general, larger k sizes are better for HDD usage. If using SSDs, choosing smaller plots will have no noticeable impact on farming performance.

The plot sizes above assume default plot settings, which use a small amount of recompute to retrieve quality strings and full proofs. We expect that most farmers will use the default settings, both for convenience and economic reasons. However, we will still offer options to increase the size of the plot up to 10-15% to allow the lowest possible harvesting load, similar to the original format.

Hardware Requirements for Plotting

Memory requirements for full plotting in RAM for HDD friendly format:

• K30: less than 64GiB
• K31: less than 128GiB
• K32: less than 256GiB
• K33+: each increase in K will approximately double the previous K’s RAM requirements.

We also expect to support partial plotting in RAM for all k sizes on 32GiB RAM systems by writing to storage during plotting. This will add additional time compared to all RAM plotting, but won’t be as impactful as the previous plot format since compute takes a relatively large portion of the overall time.

Memory requirements for Benes compression used in the SSD-only format will likely require significantly more RAM and plotting time, exact details still to be determined.

GPU plotting will be strongly recommended. Plotting times for a k32 format with a 3060 Ti nVidia GPU are expected to take about 11 minutes, and times for higher-end GPUs will scale down relative to their performance in memory bandwidth and compute. For each increase or decrease in k size, expect plotting time to double or half respectively. Note that plot times are currently estimated and may still be adjusted depending on the final security and compression resistance requirements chosen at a later date.

CPU plotting will be possible but slow and expensive. CPU plotting is only recommended for small numbers of plots. Expect a modern high-end multi-threaded cpu system to take about 10x longer than a 3060 Ti GPU (~2h per k32 plot), and a single processor with DDR4 RAM to take 60-70x times longer than a 3060 Ti GPU (~11 hours per k32 plot).

Eventual support for Apple M-Series and iGPU’s acceleration. First releases will support CPU plotting and nVidia GPUs, and later additional support for other chips will be included.

Hardware Requirements for Harvesting

These are the current guidelines per harvester using the default compression for the plot format:

• <100TiB: For farmers using spare space on their machines, a modern consumer level CPU and < 1 GiB RAM should be sufficient and have negligible impact on their system.
• <1 PiB: For small farmers a modern consumer level CPU and 1GiB RAM should be sufficient.
• 1PiB – 10PiB: Medium sized farms may benefit from an integrated GPU (e.g. Apple M-Series processor or Intel with onboard graphics), or some utilization on a GPU.
• >10PiB: a dedicated GPU is required and possibly more than 1GiB of motherboard RAM depending on the number of plots.

If using a GPU, the RAM requirement is currently expected to be less than 1 GiB.

For Raspberry Pi farmers: We are still pending benchmarks to assess how many plots a Raspberry Pi may support for the default plot format settings. However, plots can be made with additional data (~10-15% more space), so that even a Raspberry Pi could support many PiB on a single harvester (see technical overview for more details).

Timeline

A lot of you are looking forward to the new plot format, so let’s take a look at our expected timeline.

• Q4 2024 – Initial draft CHIP, open for comments from the community
• Q2 2025 – First implementation for HDD friendly formats (no Benes compression yet). Hard fork locked in and starts shipping with the reference implementation (Chia 3.0). Everyone will have 6 months to upgrade to 3.0. New format proofs of space can’t yet be used to make blocks.
• *Before Q4 2025: release Benes compression improved SSD plot format.
• ~Q4 2025 Activation (6 months after release of 3.0) – Block format changes from the hard fork, non-upgraded full nodes will stop working properly. Almost nobody will have replotted at this point.
• ~Q4 2025 to ~Q4 2026 Transition (12 months) – After the hard fork activation, there will be a 12 month transition period where everyone needs to replot by the end. Initially the new proof of space format will be given less weight than the original. This is to prevent incentives for all farmers to replot immediately after the hard fork. It will gradually become more advantageous to replot over the course of the transition and each farmer will have a point where it makes economic sense for them to replot. However, replotting still won’t have to be done urgently. At the end of the transition period, the original plots will no longer be supported.

*Note that a proof of space format does not directly dictate a plot format – any kind of plot format can be chosen to best respond to the challenges for a proof of space. In the interest of getting the new proof of space out earlier, the Benes compression improvements for the SSD-only format will start development after the HDD-friendly release, and hope to be completed in time prior to the hard fork’s activation in Q4 2025.

New Proof of Space Technical Overview

This section will go over some of the novel ideas for the proof of space and reasons why they are beneficial.

The most notable changes to the proof of space are:

1. new matching algorithm: this offers security with tunable difficulty per table, yet allows instant verification of proofs. As a result, we are able to increase the difficulty of plotting without affecting proof validation time. The benefit is that we can increase resistance to rental and compression attacks without wasting energy when validating proofs for the network.

2. challenge based on x values: A challenge will no longer start lookups based on a final y bucket. Instead, we use a special kind of scan filter on x values, specifically designed to constrain attackers against re-ordering data to accommodate various bit-dropping techniques. Now, an attacker is either restricted to organize data in a very specific way which severely limits the number of potential attacks, or the attacker must re-organize data by adding extra bits to account for that restructuring and incur a large penalty.

3. default compression to drop first table: There is a small amount of compression where the first table is dropped by default. The parameters for the compression are specifically chosen to be the easiest bits to drop and recompute, with minimal cpu time needed. This creates optimal settings, and further bit dropping by an attacker will very quickly impose economic disadvantages to create an upper bound on how much compression is viable even for future hardware.

4. Benes compression:  we can compress an additional 2 bits per entry (without bit dropping) and drop a whole lookup table using a Benes network. This results in up to ~20% additional space savings depending on k-size when compared to the HDD friendly format.

In terms of impact to farmers, because we drop some data to remove the first table and optimize efficiency, a small amount of compute is required when fetching a final quality string, similar to the low C-levels of the bladebit formats. We will include an option to omit this low-level grinding if desired, so that many Petabytes could be farmed on a Raspberry Pi for instance, at the cost of adding more bits to the plot format (up to 10-15% more space).

For those of you interested in more depth, additional technical details are available on our docs site.

Conclusion

The progressively smaller plot sizes introduced in the original format were the result of years of research and development in plotting and recompute algorithms, rather than improvements in GPU efficiency. A DrPlotter plot format could have been released on day one if the technology had been available from the start. In contrast, the evolution of GPU hardware since then would not have resulted in any dramatic gains in compression.

The new proof-of-space format (and its accompanying plot format) incorporates these years of research and development into its design, and we believe it will achieve its goal of providing high security for the network and resistance to further compression. While we acknowledge that unexpected weaknesses could still emerge from the new proof of space, its design includes only small but significant adjustments to the original to address previous shortcomings. These improvements retain the core strength of its structure, making the risk of discovering new compression attacks small.

Moving forward, we expect only marginal gains in compression from GPU advancements rather than significant algorithmic breakthroughs that shift storage into recompute. Each new release of GPU technology, even with doubled efficiency, might only reduce plot size by one extra bit per entry. This would make each plot less than one percent smaller, resulting in minor ROI improvements at best. With advancements in SSD technology and reduced energy per TB in idle storage, it is likely that most farmers will overwhelmingly use idle storage to secure the network, aligning with Chia’s vision.

More information about the new plot format can be found on our docs site.
An FAQ has also been published on our docs site.

original source – Chia.net