Gathering information about data in the network as a large operator or chain index allows for some dramatic performance increases, so much so that it changes the engineering priorities for anyone operating IPFS protocols at a large enough scale to collect it.
For instance, many operations may be slow if it is only slow once and every subsequent request for that data has an improved performance profile.
Most of the examples here are information we have but aren’t even leveraging yet because we lack the ability to read by these identifiers in other services and protocols but support is starting to land.
In many cases, you can skip network content discovery altogether once you have good information. This is some of that information, and it’s key to using the cryptography already available to us in everyday tools, transports, services and protocols to dramatically improve the user experience of IPFS data.
These are mappings that not bidirectionally secure (one-to-one) but are still quite useful.
from |
to |
provides |
|---|---|---|
| Content CID | CAR CID(s), | |
| Filecoin Inclusion Proof(s) (currently commp) | verifiable single roundtrip reads over any transport that provides this data by that identifier. no graph traversals, no codec parsing, nothing, you just get the data and you have a means of verification locally. | |
| and all these reads cache perfectly in a standard LRU that can be implemented on commodity services and cheap industry standard HTTP caching infra. | ||
| Read Range (in a Content CID) | CAR CID(s), | |
| Filecoin Inclusion Proof (currently commp), | ||
| Content Hash | similar to the above, but verification is a bit trickier. | |
| CAR CID(s), | ||
| Filecoin Inclusion Proof(s) (currently commp) | Filecoin Deal Info | The time ranges basically give you a peer list you can use to read sectors directly by commp (soon) or just BitSwap and we could parallelize the read very efficiently. |
So, it’s effectively a perfect content discovery cache that scales per CAR rather than per block, and doesn’t require any graph traversal to do the read. | | Content CID | Content Hash | This allows us to leverage existing content verifications methods when we move or cache read content. |
These are 1:1 mappings that can be cryptographically verified, which means they are secure bidirectional mappings.
from/to |
to/from |
|
|---|---|---|
| CAR CID | Filecoin Inclusion Proof (currently commp) | CAR CIDs are typical for uploads because service providers already has signing features that can verify the sha2 hash in the CAR. Looking for this data in Filecoin will mean using the inclusion proof identifier for that CAR (+ padding in the current implementation). |
| Content (CID | Hash) |
first, whenever you read data by a content identifier, capture other use identifiers and proofs at that time (assuming they are low cost to compute).
as you do, you never have to do certain operations twice.
if you ever parse codecs and traverse a graph, you return a blockset (which can be serialized to a CAR) so you can map that operation statically to that identifier. all the other information we have about the secondary identifier can then be leveraged as well.
something important to know about operating these systems at scale: disc writes are not necessarily the most expensive thing you do. sometimes writing a new copy of data that is easier to read is cheaper/faster than reading it certain ways (in terms of round trips and also egress).
when you encounter expensive reads (read that require graph traversal) you want to map them to identifiers you can do faster reads on in the future.