Protocol Labs
Cloud Native Modernisation to Help Build Web3
The Story
Web3 aims to give people back control of their data, content, and other digital assets, reclaiming it from online platforms. An open source research and development organisation, Protocol Labs builds services, tools, and protocols used by millions of people around the world.
NFT.Storage is one of the two services developed by the organisation, the other being its Web3.Storage platform. It’s a long-term, decentralised storage and bandwidth service. Assets uploaded to NFT.Storage are stored using the IPFS (InterPlanetary File System) protocol, which hosts 49.55% of NFTs. It turned out that achieving Protocol Labs’ mission required a fundamental rethinking of the NFT.Storage architecture, so it would be able to deliver limitless horizontal scalability.
NearForm
“It’s exciting partnering with Protocol Labs and helping them build the next iteration of the Web. Creating the architecture to support effectively limitless scaling for storage has made this a fun and challenging project.”
Keith Madsen
Technical Director
Protocol Labs
“When we chose NearForm to collaborate with us, it was clear that their skill set would be invaluable. They had protocol engineering knowledge and a profound understanding of cloud native platforms, which resulted in a performant cloud native implementation of IPFS for Protocol Labs — not only enhancing our current capability but also providing ongoing support for running production infrastructure!”
Mikeal Rogers
Engineering Manager
Making the web more open
NFT.Storage was launched by Protocol Labs in 2021. Since then, it has supported more than 70 million uploads (images, meta data, and other assets) for off-chain NFT data of up to 31 gigabytes. The uploaded assets are stored on NFT.Storage’s hosted IPFS infrastructure.
IPFS is a protocol designed with the intention of preserving and growing humanity’s knowledge by “making the web upgradeable, resilient, and more open.” It’s supported by the Web3.Storage platform, a service that makes storing files on IPFS easier and more accessible for non-technical people.
Enhancing scalability through cloud native architecture
Usage of IPFS was increasing rapidly because the ‘Cambrian explosion of NFTs‘ generated massive load on IPFS and this was putting strain on the Protocol Labs team. There was significant manual effort going on behind the scenes to keep it running, along with attending to issues such as 5xx errors.
This meant Protocol Labs couldn’t scale NFT.Storage in the way it required to achieve its mission. Changes had to be made to the system, with the organisation needing to fundamentally re-assess its architecture in order to meet the market demands.
The solution was to use cloud native modernisation to design a system with the capability to ingest and serve an infinite amount of IPFS data.
Building technology & capability
Protocol Labs had to enhance the reliability of NFT.Storage, so its users would be able to continue benefiting from the service’s capability to easily store and receive files on IPFS as the organisation scaled.
Protocol Labs partnered with NearForm to build Elastic IPFS together, delivering the combination of open source implementation, cloud native platform, and observability required for the level of scale it demanded.
Process
NearForm helped make the Elastic IPFS service horizontally scalable with no limits. We quickly established what was needed, produced a prototype within six weeks, and rapidly delivered an open source solution that would serve the below requirements:
Handling growth: It had to be able to cope with the enormous amount of data uploaded on it, with an average of one billion blocks of data added every day.
Running on the cloud: It needed to be cloud native, so the service could benefit from the associated efficiency, infrastructure costs, and scalability.
To tackle these requirements, the team decomposed the core responsibilities of an IPFS peer into three distinct stateless and event-driven components:
The first component we built enables the uploading of many new files from users. It does so by storing and indexing the files in fast and reliable cloud storage.
The second component allows other users of the peer-to-peer network to access the files stored on Web3.Storage and NFT.Storage using IPFS protocols, whether they’re interacting with E-IPFS itself, the websites, or another peer in the network. It achieves this by publishing knowledge about newly-indexed files to other IPFS peers, along with the users of Web3.storage and NFT.storage.
The third component allows for quick retrieval of files from the network. It brings this by using a variety of interfaces and tools across the IPFS and Web3.storage ecosystem
Each stateless component can be horizontally-scaled independently, allowing the system to support spikes in read and write behaviour. And, in the face of a failure in one component, the decoupled system continues to perform its other jobs unaffected.
Our work in switching Protocol Labs to stateless architecture was a huge win for our partner, as it enables the delivery of infinite horizontal scalability.
The Results
Following the roll out of the new infrastructure in June 2022, Protocol Labs highlighted the impressive results of the project in this blog post. The organisation notes there were “immediate improvements to NFT.Storage’s performance.”
These are some of the standout results seen by Protocol Labs:
99.9% drop in 5xx errors to uploads
Huge reductions in request time
Massive improvement in performance variance
A key reason for the success of this project is how helpful the team at Protocol Labs were, both personally and technically. It was a true collaboration and a great experience.
The partnership between Protocol Labs continues to flourish, with the next chapter of our story together focused on fostering an open source community. The aim is to work together to achieve this, with NearForm acting as the stewards of the community.