Store Unlimited Data for Free on IPFS and Retrieve it with Python

"uploading" by Katerina Limpitsouni (source https://undraw.co/illustrations)

Emerging technologies, such as the InterPlanetary FileSystem (IPFS), can contribute to an ecosystem that is more verifiable and open. Since IPFS relies on content identifiers (CIDs) which are a hash of the content, you can be confident that the returned data is correct. In addition, IPFS is an open and public network. So anybody can access content on the network if they have the correct CID.

One project I am particularly excited about is web3.storage, which is a free service that reduces the friction of using decentralized storage.

Screenshot used with permission (source: https://web3.storage/pricing/)

In this post, I will …

Provide an introduction to web3.storage and how to set it up
Provide examples of basic CRUD operations with IPFS in Python
Describe my current solution to create a decentralized data repository

Introduction to web3.storage

Our aim today is to provide a user-friendly experience that massively reduces the burden for onboarding new use cases into the web3 ecosystem today – while providing an upgrade path for the future. – Web3.Storage

Web3.Storage allows users and developers to use decentralized storage provided by IPFS and the Filecoin network. Anything uploaded is duplicated across geographically distributed storage providers, ensuring the resiliency of the network. The service also handles the work of pinning your content across multiple servers.

Be mindful of what you upload, since anybody can access the content on the network. However, you can encrypt content before uploading. My rule of thumb is, only upload content you are comfortable with being permanently public.

The permanent part is important. Unlike location-based addressing (e.g. URLs), content-based addressing (e.g. CIDs) makes it challenging to remove data from the network once it has been uploaded since it can be pinned on multiple servers.

Your quota is initially capped at 1TiB but can be increased for free by submitting a request. The overhead cost is currently subsidized by Protocol Labs, this will likely switch to some form of crypto-native payment model in the near future (e.g. staking Filecoin to increase storage limits).

The backbone that holds everything together is IPFS, a hypermedia protocol designed to make the web more resilient by addressing data by its content instead of by its location. To do this, IPFS uses CIDs instead of URLs – which point to the server the data is hosted on.

There is a lot more to web3.storage and I encourage you to explore the docs if you are also excited by this project – especially if you are a developer.

Setting up web3.storage

Go to https://web3.storage to create an account

See the documentation for detailed instructions.

Create API Token

An API token is necessary to use web3.storage from the command line

Log into your web3.storage account
Click on Account at the top and then Create API token
Enter a descriptive name for your token and click Create
You can click Copy to copy your new API token to your clipboard.

Do not share your API key with anybody, it is specific to your account. You should also make a note of the Token field somewhere and securely store it.

Install the w3 Command-Line Interface

The w3 command-line interface (CLI) is node-based tool to use web3.storage from the terminal

On a Mac, you can easily install node via homebrew, this is also install the node package manager (npm).

brew install node

Use npm to install the [w3](https://github.com/web3-storage/web3.storage/tree/main/packages/w3#readme) command-line interface.

npm install -g @web3-storage/w3

Run the following command to connect w3 to web3.storage.

w3 token

You will be prompted for the API token you previously created.

The following displays information about each of the available commands.

w3 --help

Image by author (created using https://carbon.now.sh/)

Upload and Download Commands

w3 put /path/to/file(this is how we upload content to web3.storage)
w3 get CID(this is how we download content from a specific CID)

List your files on web3.storage

w3 list

Example using put

First, create a text file with a message.

echo "Hello web3.storage" > hello.txt

Let’s now use the put command to push the file to IPFS.

w3 put hello.txt --name hello

hello will be the name that appears in web3.storage, use w3 list to verify

The CID and a public gateway link are output.

If you followed the steps above exactly, then your CID should be identical to mine. The CID is a hash that uniquely identifies the content.

Use the link below to view the message through a public gateway.

https://dweb.link/ipfs/bafybeifzoby6p5khannw2wmdn3sbi3626zxatkzjhvqvp26bg7uo6gxxc4

Note: the link will be different if your message is different

Retrieve Content with Python

In the future, web3.storage will hopefully be S3 compliant, meaning we can access data stored there similar to how we access data in S3 buckets.

For now, we can use HTTP requests to read the data into Python. However, libraries like pandas allow us to directly read CSV files from a gateway URL. Also, ipfsspec allows us to read zarr data stores from IPFS with xarray.

I will demonstrate reading each of these in the following sections

Reading JSON

Here is an example of reading a .json file stored on IPFS.

Reading CSV files

If you have a CSV file, you can read it directly into a pandas DataFrame.

Zarr and IPFS

The Zarr format is new storage format which makes large datasets easily accessible to distributed computing, making it an improvement over the commonly used NetCDF – a format for storing multidimensional data.

If you starting to switch to zarr format, I encourage you check out Pangeo’s "Guide to preparing cloud-optimized data".

NetCDF files are still very common and xarray’s functions make it easy to convert these NetCDF files to zarr data stores.

import xarray as xr
ds = xr.open_dataset('/path/to/file.nc')
ds.to_zarr("./file.nc", consolidated=True)

consolidated=True creates a "hidden" dotfile along with the data store that makes reading from zarr faster with xarray. In a simple test, I found reading consolidated data stores to be 5 times faster than unconsolidated.

zarr (consolidated): 635 ms

zarr (unconsolidated): 3.19 s

If you would like to test out the code above, I uploaded NOAA Optimum Interpolation SST V2 dataset in consolidated and unconsoldiated zarr format to IPFS. This dataset provides weekly means of ocean sea surface temperature (SST) from 1990 to the present with a 1 degree spatial resolution.

The gateway URL for this data is shown below

https://dweb.link/ipfs/bafybeicocwmfbct57lt62klus2adkoq5rlpb6dfpjt7en66vo6s2lf3qmq

Upload Zarr data stores to IPFS

When uploading zarr files to IPFS you have to make sure to upload the "hidden" dotfiles. With w3 this entails adding the --hidden flag:

w3 put ./* --hidden --name filename.zarr

Read Zarr data stores from IPFS

In order to read zarr data stores from IPFS with xarray you will need the [ipfsspec](https://github.com/fsspec/ipfsspec) package (as well as xarray and zarr)

conda install -c conda-forge xarray zarr
pip install ipfsspec

ipfsspec ensures xarray can interpret the IPFS protocol.

Notice in the example below I am using the IPFS protocol instead of the gateway URL with HTTPS. However, behind the scenes the code is actually reading from a gateway.

Decentralized Data Repository

IPFS can be used as a data repository. However, you need to link the CID to something that is human-readable in order for this to be a viable solution. I am not aware of any best practices surrounding CID management. However, taking some cues from best practices for NFT data, my current approach is to store CIDs and their associated file name as a key:value pair in JSON and then use a NoSQL database, such as MongoDB, for queries.

After you upload the content to web3.storage you add a new record to the file that identifies what the dataset is and the CID of the content. This is the minimum amount of information needed.

Here is an example of CIDs stored as a JSON array.

The [pymongo](https://www.mongodb.com/docs/drivers/pymongo/) package makes it easy to work with MongoDB in Python.

Here is an example of inserting records into a collection and then querying the database for a specific CID.

Alternative approach

An alternative approach I have been considering, but have not implemented yet, is taking advantage of the w3 list output.

This displays the CID and name you supplied when you uploaded the content.

The idea is to write an awkscript to generate a JSON file from the output.

Some downsides or pitfalls to this include:

Handling CIDs that point to directories instead of pure files
Ignoring CIDs that are irrelevant to your database

The biggest hurdle I can see is dealing with directories. For that reason alone I am sticking with manually updating my database, especially since it is small and easy to manage – for now at least.

Final thoughts

IPFS could be a place for long-term storage with CIDs functioning like built-in version control.

The w3 CLI makes it easy to push data up to IPFS and any common format (JSON, CSV, Zarr) can be read from IPFS into Python.

However, one the challenges with using IPFS is CID management. My current solution to create a decentralized data repository is to store CIDs and their associated file names as key-value pairs in MongoDB. The solution is a little clunky and I am open to suggestions on how to improve this setup.

Using IPFS for web hosting

Another great application of IPFS is hosting a website. My personal webpage is hosted on IPFS. I think I came up with an elegant solution to manage a decentralized website, you can read about my solution in the article below.

Modern Way to Host a Site on Ipfs

Storj: another decentralized storage option

Finally, IPFS is not your only option. If you want an S3 compatible object storage, then I encourage you to checkout Storj. The Storj solution relies on storing shards of encrypted data across a decentralized network as opposed to duplicating your data. This is a paid service but is very affordable, especially if you are currently using AWS. I personally use Storj for my day-to-day work. Check out the article below if you are interested in learning how I use Storj.