Headjack - the base layer of cyberspace

Headjack is a blockchain that links sovereign identities to content at web-scale. Key points:

Creation is fundamentally different from transfers and exchange of value - the design space around trust & data availability for media and identity is different from finance.

Following the UNIX philosophy - in Headjack identity is simply an identifier (unique number) and anything orthogonal (KYC, profiles, privacy, finance) can be layered on top of it.
It solves single sign-on and allows for user experience similar to Web2 through hierarchical authorization management - keypairs are not required by default and even those with keys bound to their accounts may choose to not explicitly sign every interaction.
Consensus is reached on the absolute bare minimum - the history of authorizations, names, keys & off-chain content anchors (merkle roots) - the simplest mental model for developers.
Headjack can support billions of accounts and link unlimited amounts of off-chain activity to them. The entire web can be rebuilt on top of it - a claim that is easily provable.
Content addressing is with persistent & human-readable URIs (instead of hashes) - the link between identity and data is cryptographically provable even if keys & names have changed.
It doesn't deal with off-chain data storage and retrievability - those are separate problems and Headjack simply lets entities point to ways for others to retrieve addressable content.

The talk "Decentralized identity & content addressing at web-scale" at code::dive 2022 is a good high-level overview of the project and covers the most important aspects:

Book structure

What is Headjack - How the protocol technically works and how things like applications, services, DMs, social graphs, preferences, etc. could be implemented - the building blocks necessary to recreate anything from Web2 and beyond.

Why Headjack - What's broken with the web and a blueprint for what is possible - services, business models, infrastructure, algorithms, markets, metaverse, etc.
Implementation of Headjack - A detailed specification of the implementation.

What is Headjack

Named after the data port at the back of the head of synthetically-grown humans in the Matrix.

Design - guiding principles

These are the guiding principles when aiming for mass adoption of Headjack:

Customer obsession & the best possible UX

It is highly improbable that the masses (and even most crypto natives) would tolerate services that are much worse (slow, limited & cumbersome) and most of the competing attempts for decentralizing media are nowhere close. There are a few aspects to retaining the comforts and UX of Web2 that we've become so accustomed to:

Nobody wants to deal with keys, wallets & self-custody because of all the headaches & complexities that come along with that. Creation & media are different from exchange & finance and it's OK to trust by default as long as there's a fallback. We should be aiming for better trust instead of trustlessness at the cost of UX. Users shouldn't have to manage keypairs on multiple devices & explicitly sign every interaction - by default they'll be logging into identity managers (IDMs) using email & passwords or SSO ("login with Google") and would then be using these IDMs as SSO to authorize applications to post on their behalf without requiring keys & signatures - by delegating trust. This way the majority of Web2 identity & authentication plumbing can be reused with Headjack underneath as just another backend. "Sign-in with Ethereum" doesn't scale - we should aim for familiarity.

"With consumer products, simple and “wrong” beats complicated and “right.”" - @naval

"The future of mass-market crypto experiences lies within apps that provide familiar, custodial experiences with the ability to graduate into non-custodial experiences." - a16z
Users shouldn't have to think about and pay for the storage of their data & blockchain interactions by default - costs & complexity should be abstracted away & shifted to services in the background. Self-hosting is the opposite of customer obsession - let's aim for simplicity.

"People don’t want to run their own servers, and never will." - Moxie
Content addressing should be with human-friendly URIs with names & numbers instead of being full of hashes - typical for Web3. We're used to adequate URLs where the domain of the platform/website & even the user name are present when identifying content - hashes make URIs much longer & harder to remember. Contrast that to Headjack's addressing.
The applications built on top of the network must match the responsiveness of Web2 and exceed its functionality. "Latency is not an option anymore" - Amazon found that every 100ms of latency cost them 1% in sales. 16 years ago Google found an extra 500ms in search page generation time dropped traffic by 20% - our irritable nature hasn't changed. Web2 isn't going anywhere - "market dynamics and the fundamental forces of centralization" dictate that the best services will be running on huge server racks in data centers with sophisticated caches & batch processing infrastructure due to data gravity.

Web-scale, blockspace & the UNIX philosophy

People grossly underestimate the size of the web and the required infrastructure. Here are some decade old twitter, google and other statistics and a few articles about what it takes to run Twitter: 1, 2, 3, 4, 5. What was going on within a single minute of 2021 is truly mind-boggling:

Headjack follows the UNIX philosophy - it focuses only on identity (identifiers represented as numbers & name ownership) & linking data/actions to it without trying to do anything orthogonal (data storage, KYC, profiles, privacy, finance, etc.) that can be layered on top. It doesn't impose constraints on what could be built around it - separation of concerns. All kinds of systems with their own incentives, cryptoeconomics & guarantees can be implemented on top of this identity layer & addressing. The on-chain vs off-chain tradeoff and what goes into the blockspace is as follows:

Consensus should be reached on the absolute bare minimum - only identity (integers), the history of keypairs & authorizations, name ownership & anchors to off-chain activity need logical centralization and must be on-chain with guaranteed data availability.
All other activity & data is stored off-chain (IPFS & other protocols) because of the sheer volume - it's ephemeral and its relevance fades with time. Most of it won't be stored forever but any piece can be backed up through archives & IDMs. Events get cryptographically anchored with Merkle roots to the chain so that permissions, inclusion & sequence are provable.

"Developers care about risk." - Haseeb

It must be obvious & provable that the network has a credible path to handling billions of users if entrepreneurs are expected to jump on the opportunity. The easiest mental model will win over developers and users - singletons & opinionated frameworks with a concrete direction are much simpler than a fractured landscape of standards, chains & bridges.

"Consistency is incredibly important for creating a compelling user experience." - Moxie

Decentralization, neutrality & sovereignty

"The Internet has succeeded in no small part because of its purposeful avoidance of any single controlling entity." - source

Sovereignty: Users should be able to own their identity & connections with a keypair even if by default their activity is managed by an IDM - resembling a custodial service.
Credible neutrality - Anyone can permissionlessly have a pseudonymous identity, buy a name, operate an IDM, serve media through an application and publish & broadcast through Headjack using their identity. Moderation & censorship happen at the application & infrastructure level - freedom of speech is guaranteed, but not freedom of reach as that is up to the applications & services that serve media. Individuals will always be able to contact one another for private communication.
Anyone can self-host & run software locally, browse the ecosystem, and fetch content & events from entities they've subscribed to (although quite bandwidth-intensive), but their experience will be extremely limited in that they won't be able to run any sort of query/filtration/feed algorithm at scale nor aggregate the activity of billions of people in real-time.

"You can build something centralized on something decentralized but you can’t build something decentralized on top of something centralized. Decentralization is always the base layer." - @RyanSAdams

Identity & authorization

There are 3 types of roles in Headjack (although a single entity may play all 3):

Normal accounts - represented by an integer ID on-chain - keypair association is optional.
IDMs - a superset of normal accounts - required to have a keypair - can manage other accounts by submitting changes for them (name handle, updating keypairs) & acting as SSO - authorizing applications to post on behalf of users (& ability to revoke the authorization). They will also be responsible for handling DMs as discussed here.
Applications - a superset of normal accounts - required to have a keypair - they are the media presentation layer. Users can authorize them through the use of an IDM to post on their behalf without having to explicitly sign every interaction (follow, post, comment, react).

All authorizations are represented & submitted on-chain as simple integer pairs (2131 => 83253, 6331 => 14415) that get aggregated in compact blobs & signed in bulk by IDMs - achieving a very high signal-to-noise ratio (few signatures) ==> improving the throughput in the valuable block space.

With this foundation we achieve the following range of usage scenarios:

Costs for using the blockchain can be shifted to IDMs & applications with business models to support that - users won't care that there's an underlying token (they'll always be able to interact with it directly through the mempool & pay for transactions if they wish).
Users won't need wallets & keypairs - risky and cumbersome with multiple devices. Most will create accounts through IDMs & use email/pass or Web2 SSO ("login with Google") which will create on-chain integer IDs for them without associated keypairs - "owned" by the custodian. Users will be able to "log in" to applications using their IDM as SSO for Headjack which will authorize the application with a few bytes on-chain to post actions on behalf of users - all without requiring a single signature by the user - neither on-chain for the identity/authorizations (tiny bits of data - just integers & bit flags submitted by the IDM) nor for their off-chain content (posts, comments, reactions).
Users can revoke permissions to applications and even retroactively invalidate activity generated on their behalf by an application by saying "discard activity generated through application А from block X forward" through a small on-chain message published by their IDM because everything is sequenced. This is acceptable because in this blockchain such data is non-financial and fake activity has smaller consequences - it is still an enormous improvement compared to the current Web2 status quo.
At any point in time users can regain full sovereignty over their identities by binding a keypair through their IDM. Then they'll be able to cut that IDM off (revoke access) & even retroactively invalidate actions from it through another IDM or direct on-chain transactions.
Users can be completely anonymous by directly creating an identity with a keypair & paying for an on-chain transaction. They'll be able to use IDMs without having to sign with email/pass or a Web2 SSO - not revealing anything.
Applications will be usable by users that don't use an IDM but all their off-chain activity (posts, comments, reactions) will need explicit signatures.

So at the core of it all is just sequencing relations between integers & Merkle roots for content.

In practice, we expect that only cypherpunks & people that have something to lose (big audience/reputation) will go through the trouble to manage a keypair. Almost everyone will use IDMs - even most crypto natives don't want to explicitly sign every action and have their keys in hot wallets ready to get hacked. This way 99.9% of the user activity on-chain (mostly authorizations) ends up going through authorized services and gets batched in a compact way - requiring only that the service signs the aggregated payload and thus reducing the amount of signatures on-chain.

The vast majority of users will be lightweight: consumers & curators of content (through interactions & reactions) with very little creation on their part and little to no audience. At any point in time, they could shift to a more vocal role and start caring about archiving their off-chain data and not relying on the good grace of the infrastructure that sits beneath applications.

Key & session management (rotation, authorization & revocation) require ordering that is logically centralized. It is compatible with any type of DID - anything could be associated with an integer ID. The on-chain authorization has great synergy with the human-readable & persistent addressing for off-chain content.

Content addressing

The move from host-centric to data-centric addressing is a complete paradigm shift by itself but Headjack intertwines that with names, on-chain authorization and sequencing of anchors resulting in the best possible URIs in terms of human-readability & persistence - perhaps the most important aspect of the project. This chapter is broken down into a few sub-chapters:

Host vs data-centric

Let's take a look how the web works and what are the building blocks & ideas that enable Headjack:

Today's web: host-centric

Today's web revolves around hosts & unicast communication - we query DNS to get the IP of servers and open direct connections with them to retrieve the data that they host. But domains, URI paths on servers & the actual files all change & go away which leads to link rot & content drift. Guidance such as "Cool URIs don't change" is just that - guidance - and the Internet Archive is just a bandaid that can hardly keep up with the digital memory hole. In the host-certified paradigm URLs at best point to a location at which a document may have been available at some point in time - devoid of any cryptographic proofs regarding the contents, the creator, an alternative way to retrieve it or the time of publication. The implications are explored in the host-centric section in the motivation.

"It is generally recognized that the current approach of using IP address both as a locator and as an identifier was a poor design choice." - David D. Clark, Designing an Internet

Data-centric computing

Data-centric computing is an emerging concept that has relevance in information architecture and data center design - data is stored independently of the applications, which can be upgraded without costly and complicated data migration. This is a radical shift in information systems that will be needed to address organizational needs for storing, retrieving, moving, and processing exponentially growing data sets. It increases agility by prioritizing data transfer and data computation. Applications become short-lived, constantly added, updated, or removed as algorithms come and go.

"Data is the center of the universe; applications are ephemeral." - The Data-Centric Manifesto

Content-addressable storage

Content-addressable storage (CAS) is a way to store information so it can be retrieved based on its content (not its location/name) and is a key piece of the puzzle. Identifiers are based on content and any change to a data element will necessarily change its content address. The most famous example of CAS is IPFS but it suffers from non-human-friendly addresses (hashes), performance issues, and extreme latency (tens of minutes) because of the global DHT if content is not widely cached/pinned.

Self-authenticating data moves authority from hosts to users. The three components that enable it are cryptographic identifiers, CAS, and an emerging area of research called verifiable computation which is yet to be applied in any meaningful scale.

Information-centric networking

Information-centric networking (ICN) is an approach to evolving the Internet infrastructure away from a host-centric paradigm, based on perpetual connectivity and the end-to-end principle, to a network architecture in which the focal point is identified information (or content or data). Data becomes independent from location, application, storage, and means of transportation, enabling in-network caching and replication. The expected benefits are improved efficiency, better scalability with respect to information/bandwidth demand, and better robustness in challenging communication scenarios. In information-centric networking, the cache is a network-level solution, and it has rapidly changing cache states, higher request arrival rates, and smaller cache sizes.

Named Data Networking

Named Data Networking (NDN) is a Future Internet architecture that builds on top of the previous ideas (& an incarnation of ICN) and in which data is requested by name and routed by the network. However, there are many unsolved challenges with it like the need to reimplement foundational routing infrastructure to make it stateful and hierarchically structured names which require a root name authority to link them to keypairs - outside of its scope. Here's a great lecture on the topic.

Enter Headjack - the final iteration

Headjack is a weird amalgamation inspired by everything above - it provides human-readable & persistent URIs for self-authenticating data (with Merkle proofs & the blockchain) along with the means for its retrieval without forcing a specific way (IPFS is just one option). It acts as the web-scale global index used to check the authenticity of documents (requires consulting with the chain), ownership of names, key management & sequence of events throughout time. It is an addressability layer on top of the current host-centric internet technologies.

Blobs & persistent URIs

This chapter will explain how all off-chain messages (actions/events/content) get published:

Blob construction - batching of user events

Applications accumulate off-chain activity from users which they cryptographically anchor in batches with a Merkle root on-chain and they determine how often to do so (it doesn't have to be on every block) - those with little activity may submit only once per minute or even less often - the frequency is determined by applications based on the volume of activity and the on-chain publishing costs.

When enough activity has been collected it is time for the application to finalize the batch: it is packed in a blob and all the events generated since the last anchored batch are sorted & grouped by accounts in some deterministic way (perhaps accounts based on index and actions based on the type/sequence) with some schema with the following steps:

The intra-blob index (offset table) for lookup of content of specific accounts is generated.
A Merkle root that touches every event is deterministically constructed following a schema.
The IPFS CID (hash) for the blob is generated and it is pinned for others to download.

The only 2 things that are signed & submitted on-chain are thus the Merkle root and the IPFS CID for the next nonce (auto-increment counter) associated with the application account.

Stable intra-blob addressing before publishing

Applications maintain the logical order of events for the future batch in maps in order to provide intra-blob addressing even before it is fully constructed - as an example if a user posts an article and immediately after that comments on their own post - the comment should be able to refer to the post which is not yet committed on-chain. Applications will also display activity by other accounts that is not yet anchored and the interactions can still use the proper addressing when referring to the yet-to-be-anchored messages (the next nonce number is known in advance). Any type of interaction is addressable and sequenced in the blobs - including reactions (likes, etc).

Persistent & provable URIs

Each account has an associated auto-increment counter (nonce) for every time they submit an anchor for off-chain content. So if an application has submitted 2 times already, then the next submission will be with nonce == 3. The blockchain keeps a mapping in its state for each previous nonce value to the block number when it changed so that <application_id>/<nonce> can be translated to which block has the Merkle root anchor & IPFS CID for the blob that corresponds to that nonce for that account.

Once a blob is fetched through the IPFS CID (hash) we can address specific events by using the offset index in the blob header so a URI like <application_id>/<nonce>/<user_id>/<content_id> can point to a specific post, comment or even reaction (activity is grouped by users). The content ID for a specific user is usually a small single-digit number and is necessary only if there has been more than 1 interaction by that user through that application for the given nonce (maybe rare). This is what events with URIs referring to each other looks like:

The blockchain can be queried if the application was allowed to post content on behalf of the user with an on-chain authorization (which probably happened through an IDM) when that specific block was published in order to determine if the activity is authentic - the state keeps information for each account such as since what block number a given application was authorized to post on behalf of a user (and until when - all ranges). Users may avoid using IDMs and explicitly sign their actions in which case their data will be accompanied by their signatures within the data blobs and the only check required will be for the user keypair used for the specific block number.

Steps to prove the authenticity of a URI

To recap - to prove the authenticity of any event with a URI:

First check if the data is actually part of an anchored blob with a Merkle proof to a block. This requires either just the piece of data + a Merkle proof for inclusion in the blob or the entire blob in order to reconstruct the Merkle tree & proof.
Then check if the user actually submitted the event:
- Either if at that point the application was authorized to post on behalf of the user which would require a Merkle proof for a part of the blockchain state (authorization ranges).
- Or by checking for an explicit signature & the public key of that account at that time which would also require a Merkle proof for a part of the blockchain state (account key history).

URIs are persistent as long as someone hosts either the individual event + the Merkle proof or the entire blob (and can reconstruct the proof) and knows to which block it was anchored (from the <application_id>/<nonce> => <block_number> mapping). The following chapter shows how names in URIs are persistent too (even if user/application names change ownership at some point).

A few other notes

There can be many different & valid proofs for the same URI from different block heights.
Even private intranet data may be anchored but not retrievable by the public if the blob IPFS CID is never published or pinned/hosted - unified addressing for public & private.
Users should be able to see the URI of content even if created through another application and the origin should be displayed by default - acting as attribution for other applications.
Edits & updates to content come as messages with new unique URIs that reference the older message URIs and it is up to applications to properly handle this - either by showing that there have been changes and a newer version or automatically redirect to the latest. "Forks" are possible but they represent application failure to detect that an old version is being edited.
Accounts that anchor content on-chain cannot do so twice in the same block - for simplicity.

Names & paths

Headjack is also a name registry - accounts can own a handle and be identified with it. For specifics around the details (constraints, subdomains, auctions, distribution, hoarding, leasing, etc.) please refer to their dedicated page. In this chapter:

Names in URIs

Users and applications don't need a name and can operate as an integer index just fine, but the preferred case will be with handles. Names can change ownership but the blockchain will be able to translate <application_name>/<nonce>/<user_name>/<content_id> with strings into the canonical integer form discussed previously by substituting the application & user names with account IDs.

Every name has an associated auto-increment nonce (just like account IDs) for every time they submit an anchor for off-chain content and the blockchain records maps of <name>/<nonce> to <id>/<nonce> which can then be used to resolve the URI as discussed in the previous chapter.

But we need to be able to translate not just the application name but also the user name which may have changed ownership at any point - for that the blockchain keeps track of the account ID ownership of every name historically as ranges (from block X to block Y name N was owned by account A) so when we determine the block number for a given data blob we'd be able to check to which account ID does a name in a URI correspond to at that time.

And thus we're able to have URIs such as twitter.com/55212/johnny/3 to identify any event by any actor - all we'd need to do is a few lookups and we'll be able to use Merkle proofs for any piece of content to prove authenticity. Most URIs could even omit the 4th part because probably there won't be more than 1 action by a user for a given batch by an application.

Note that the canonical form (numbers instead of names) of twitter.com/55212/johnny/3 could be something like 42/783/523/3 where only the last number would be the same and the nonce would most likely be different. Also twitter.com might no longer be owned by account 42 but what matters is that the blockchain can correctly determine who owned it at nonce 55212. Multiple names can be owned by an account but their nonces for one event will probably be different.

What to ask the blockchain about a URI

To recap: we can ask the following questions about this URI: twitter.com/55212/johnny/3:

To which application account ID & nonce does twitter.com/55212 correspond?
To which block does the applicationID/nonce map correspond?
What is the IPFS CID & Merkle root of the anchored blob at that block?
What account ID does johnny correspond to in the block where this blob was anchored?
Once we download the blob or just the blob header (using the IPFS CID or any other means):
1. We can ask the offset table where within the blob is johnny's content № 3?
2. Once we fetch the actual data & depending on whether it is explicitly signed or not:
  1. either if the application was authorized to post on behalf of johnny at that time,
  2. or if the signature matches the keypair that's been bound to johnny's account at the time of the anchored block.

Web3 URIs interoperable in Web2

Application accounts can point on-chain to a host with an IP address which can be used to display content published through them. Application names can also resemble traditional domain names so it will be possible to copy-paste such URIs directly into your browser and as long as they own the same domain in the traditional DNS they should be able to serve a webpage displaying the piece of content - enabling seamless interoperability during the transition from one paradigm to the other.

Content titles in URIs

Most Web3 platforms suffer from unreadable URIs but we've done a lot better - note the brevity and lack of hashes & hexadecimal symbols (0xf56a0...) - in fact, this is as good as it gets...

Or is it?! What about headlines of articles - can we have them included as well - something like twitter.com/55212/johnny/3/how-I-went-from-vegan-to-keto-and-back-again? Absolutely! The string is not at all necessary to resolve the piece of content (just like in StackOverflow where the database key for a question is just a number (example: question 4) but the page router always changes the URL when loading the page to include the title too). Message types for posts with titles will have a dedicated field which will get included in the content hash and thus spoofing the title will be rejected by conforming applications as it would be a trivial check.

Addressing within content

Different schemas could be used for addressing within pieces of content (like a paragraph from an article or a clip from audio/video - without losing the context of the whole) and message types could have by default associated on-chain schemas (or the schema of choice could be embedded within the header of the message). For example, when medium.com/12475/elvis/0/learn-to-code/121/66 is being loaded the associated schema will be looked up depending on the type of message (in this case - an article) and used to interpret the last part (121/66) which could mean a character selection with an offset from the start and length. The embedded schema could be overridden by explicitly stating which one to use within the URI. As an example, medium.com/12475/elvis/0/learn-to-code/schema/42/121/187 could mean "use on-chain schema number 42" which could interpret the last part (121/187) as start offset and end offset instead of start & length - resulting in the same selection as before. Even individual pages & paragraphs of books should be referencable in such a manner and could be composed of multiple separate posts - and this is just scratching the surface!

For big types of content (audio/video) the message could be broken down into chunks so that users can load only the message header and then depending on the schema used and the addressing within the content - only the necessary chunks could be requested.

Messages

The terms message/event/action/data/document/content are used interchangeably in this book and refer to any type of event/content a user might have generated - post/comment/reaction/etc.

IDMs, preferences & graphs

Data storage & retrievability

In this chapter we will see the different aspects of handling unlimited amounts of off-chain data:

Ingestion and transformation of blob data

Off-chain blobs with data will be fetched, processed and stored immediately after they are published in more optimal database formats for content to be later directly served by application infrastructure. Most of the cryptography checks will be happening instantly during this process but the proofs don't need to be stored. Users will always be able to request proofs for any event at any time (& cache them locally) because they can be regenerated on the fly as necessary.

Hierarchical data blobs & partial fetches

Blobs may be in a hierarchy such that the on-chain IPFS hash points only to the "root" blob that contains the header and the actual indexed data could be in child IPFS blobs (whose IPFS CIDs are contained in the root blob or header) so entities listening for events by specific accounts on Headjack may download only these headers and determine which "leaf" blobs they need to fetch for the data they are interested in (if any). Sparse bitsets & bloom filters could be used to quickly scan for the presence of activity by specific accounts.

Direct IPFS connections & horizontal scaling

Applications can advertise the multiaddress of their IPFS nodes on-chain so that each blob of content that gets published can be downloaded by others instantly by manually connecting with IPFS’s “swarm connect” functionality - avoiding the use of the DHT for each new blob CID which may take tens of minutes. They can provide addresses to multiple IPFS nodes as a cluster for horizontal scaling and use Pinset orchestration - designed for Automated data availability and redundancy.

Applications may choose not to use IPFS at all - what they must do is anchor their blobs with a Merkle root and provide some on-chain advertised means to retrieve the data (example: REST/RPC endpoints in their on-chain account). We expect that IPFS will be the lowest common denominator and will always be used no matter what other solutions are also available.

Applications can talk to each other directly by using their on-chain advertised REST/RPC endpoints and may ask for the events & messages that are not yet published by the other applications. This way they could display "remote" events locally while they are still in the "mempool" and allow their own users to interact with those events from other applications. This is possible because URIs are stable even before publication - see Stable intra-blob addressing before publishing. High activity applications can interoperate and no longer be a slave to the block time. However:

Applications should display events that are not yet anchored in the UI differently - especially if coming from another application.
Events that refer to each other but are from different applications and have not yet been anchored on-chain could end up committed in the wrong order (if one of the applications skips a few blocks and commits at a later one) - such that an event from the past is referring to an event from the future - breaking referential integrity. However, messages have a timestamp field and could also have the current block height at the time of creation - useful for sorting.

How to retrieve data for a random URI

There are multiple options:

The entire original blob with an IPFS CID might still be retrievable from the original application account that posted it or anyone else that has pinned the data.
The user account might be using an archival service for all their activity and they can point to that archival service on-chain in their account for others to retrieve their messages.
Other well-known players without a direct on-chain connection to the application/user in a URI could be asked if they have the content:
- Infrastructure companies that do the heavy lifting for applications and store everything.
- The analog of the Internet Archive in this ecosystem that also stores everything.
IPFS can be forked & reused with the following change: instead of delivering content based on the CID hash it can deliver the data + the necessary proofs based on Headjack URIs or their hash (they are unique) - any individual off-chain message that's been anchored would be retrievable as long as someone is hosting it in this p2p network (which needs bootstrapping - could be part of Headjack nodes). However, this won't be very performant due to the granular nature of individual messages with a URI and the use of a global DHT.

Blocks, state & proofs, oh my!

Throughput & scalability

Everyone claims to be scalable, but here we'll prove that Headjack can handle billions of accounts and anchor unlimited amounts of off-chain content tied to identity with simple napkin math.

How big is a Headjack transaction

Applications post anchors to off-chain content with an IPFS CID hash and a merkle root. IDMs also anchor off-chain content (mainly user preferences & updates to social graph), but they also post authorizations to other accounts (applications) to post on behalf of users as integer pairs.

So the fields for a transaction by an application/IDM (which will be the majority) are:

version: 4 bytes
signature: 65 bytes
blob IPFS address: 32 bytes
blob merkle root: 32 bytes
nonce: 4 bytes auto-increment integer associated with the account - to prevent reordering of anchored off-chain blobs (which would mess up internal addressing based on that nonce)
value: 4 bytes amount of native token paid to validators for transaction inclusion

So far that is 141 bytes which almost every transaction by an application or IDM contains. IDMs also submit a list of authorizations (or revocations) as integer pairs. For example, 1000 accounts authorizing 15 different applications to post on their behalf would be 1000 integer pairs. Assuming 8 byte integers (up to 2^64) that would be 8 * 2 * 1000 = 16k bytes.

Naive scenario

The initial version will target block bandwidth of up to 100 kb/s. This is not a problem for ZK validiums as there are already DA solutions that offer 10 mb/s or even much more.

Assuming:

1 MB block size & 10 second block time (100 kb/s of block bandwidth)
1000 applications posting in every block
100 IDMs authorizing as much users as possible - filling the remaining block space
no on-chain actions such as keypair & name changes, account creation & direct interaction with the chain by end users

We get:

1100 actors (1000 applications + 100 IDMs) that post in every block at least 141 bytes for their transactions, which is 155100 bytes
the remaining 893476 bytes (1048576 (1MB) - 155100) can be filled with authorizations and since an authorization is 16 bytes (8 * 2) that would be 55842 authorizations/revocations every 10 seconds or 5584 authorizations/revocations per second
for 1 billion accounts that would be 0.557 authorizations/revocations per person per day which is actually quite good - people on average do way less single sign-ons per day

completely different goals - comparing the 2 protocols just to put things into perspective	Headjack	Ethereum
block size	1 MB	~80 kb
block time	10 seconds	~13 seconds
blockchain bandwidth per second	100 kb/s (x16 more than Ethereum)	~6.15 kb/s
blockchain bandwidth per day	8640 mb/d	~528 mb/d
transactions/authorizations per second	5584 APS	~14 TPS
transactions/authorizations per day	482,457,600 APS	1,209,600
transactions/authorizations per person per day for 1 billion accounts	0.482 (x400 more than Ethereum)	0.0012096

Realistic scenario

The naive scenario does not include on-chain actions for specific accounts such as:

keypair changes (new pubkey (32 bytes) + signature (65 bytes) if there is an older key)
account creation (if done by an IDM then this is just a few bytes - no pubkey)
name registration & ownership changes (see the dedicated page for more details)
updating account fields such as a URI pointing towards an off-chain account directory (which could point to archived posts) or pointing to another account index for such services
signed transactions by individual accounts that want to directly interact with the chain
- authorizing an IDM, rotating keys, or even publishing off-chain content as an application

However, the realistic scenario will not be far from the naive because:

Only a % of all accounts will have keypairs (even though 100% could) and will make just a few signed actions per year - leaving most block throughput for authorizations through IDMs.
Large % of accounts will rarely even be authorizing new applications - many people don't sign in to new services through SSO every single day. There could also be 2 types of log-ins: passive (viewing only - nothing on-chain) and authorized (allowing services to post on behalf of users).
Many applications that don't generate a lot of off-chain activity will publish less often than on every block in order to minimize on-chain block space costs.
The chain throughput can be further optimized & scaled by multiple orders of magnitude.

Optimizations & scaling

Throughput of 100 kb/s is just the start & can easily go to 1-10 mb/s as a ZK rollup.
The chain & state can be trivially sharded - there aren't problems such as fracturing liquidity or preventing composability because accounts don't care about each other - they mostly contain authorization block numbers & keypair history.
Integer indexes that only need 4 bytes can be compressed/batched together - it'll take many years to go beyond 4 billion accounts so the actual throughput is 2x of what is listed here.
A fee market can develop that tunes the cost of different actions so that actors don't just pay for on-chain bytes - the ways the system is used can be guided through incentives.
Other optimizations not listed here - this is just the starting point.

State growth

Headjack's main value proposition is keeping historical records of the sequence of authorizations, key changes & off-chain content anchors and being able to generate proofs for any specific piece of off-chain content.

TODO: finish this

https://ethereum.stackexchange.com/questions/268/ethereum-block-architecture

numbers - state - one difference from other cryptos is that this one is append-only and could be designed to be easier on memory access patterns

One difference with other blockchains is that accounts in Headjack are numbers and thus the state tree could be different.

on eth state growth: https://twitter.com/SalomonCrypto/status/1587983584471633921 https://hackmd.io/@vbuterin/state_size_management

All on-chain changes just append data to one of the few attributes of:

accounts:
- public keys: a map of keys and block height integer ranges (non-overlapping)
- authorizations: a map of indexes and arrays of block height integer ranges
- nonces: an array that maps autoincrement indexes to block numbers
  - appended only when publishing off-chain content (usually an application/IDM)
names:
- owners: a map of owner indexes and block height integer ranges (non-overlapping)
- nonces: an array that maps autoincrement indexes to account index & nonce pairs
  - appended only when publishing off-chain content (usually an application/IDM)

TODO: should IPFS hashes & merkle roots be saved in the state? - no?

TODO: light clients? in addition to merkle proofs for inclusion of content they would need merkle proofs for the state of which applications a user has authorized to post on their behalf in a given block

Off-chain content

There are no limits for off-chain content as it is all just anchored with merkle roots - it could be as high as hundreds of terabytes per second. There isn't a more minimal design that can link unbounded amounts of off-chain data to billions of identities that can change keys & names and yet still provide the guarantees & mental model simplicity of Headjack - it achieves consensus on the absolute bare minimum.

Headjack vs the competition

This chapter focuses on the disadvantages of some of the more high-profile competing solutions in the space. Most of the issues are solved in Headjack due to its guiding principles & design goals. This page doesn't list any of their positives as it would be too long (so not exhaustive by any means) but many of them have served as an inspiration for Headjack in one way or another.

Comparison table

Some of this is a subjective estimation - many of the claims lack official sources.

	Headjack	Farcaster	DSNP & Frequency	Bluesky & AT Protocol	TBD web5 slides & tweet	Ceramic & CyberConnect	Lens Protocol
blockchain-related properties
Scalability & potential scope	can handle billions of users (proof) & underpin the entire web	perhaps up to ~10 million - could move to its own rollup	perhaps up to a few million graph changes are on-chain	centralized consortium of servers	perhaps up to a few million - lots of reliance on IPFS, DHTs, hashes & keys	perhaps up to a few million - lots of reliance on IPFS, DHTs, hashes & keys	actions are on-chain as NFTs (follow, post's hash) - even a dedicated EVM chain will be futile
Users paying for TX fees & linking identity to financial accounts by default	all blockchain costs are paid for by services by default	Ethereum L1 costs initially planned for subsidy by services	all blockchain costs are paid for by services by default	centralized consortium of servers - no TXs	the anchors (on-chain Merkle roots) get batched with others	only the stream anchors to Ethereum L1 have to be paid for occasionally	yes
Blockchain TX fee stability & predictability	as scalable as necessary => no congestion	Ethereum L1 - may need to migrate to its own rollup in the future	their notion of capacity is probably good enough	centralized consortium of servers - no TXs	Bitcoin TX fees are low due to low economic activity	Ethereum L1 for stream anchors	Polygon PoS
Block time for anchoring key operations	Ethereum ZK validium with multiple blocks in one L1 slot	Ethereum	Polkadot	centralized consortium of servers	Bitcoin	Ethereum, but the anchors are occasional	Polygon PoS
Time to finality for key operations	Ethereum	Ethereum	Polkadot	centralized consortium of servers	Bitcoin	Ethereum	Polygon PoS
Contains a name registry for easy discoverability & can replace DNS	yes - & tightly integrated with addressability - URIs aren't broken even if names change ownership	yes, also works with ENS	no, but might introduce it	no - uses email-like usernames resolved with Webfinger to a DID & relies on DNS (centralized)	no	no, maybe works with ENS	no, maybe works with ENS
Decentralization for the most important parts (keys & registries)	Ethereum ZK validium with external data availability (validium) - EigenDA?	Ethereum	Polkadot - not big enough set of validators	centralized consortium of servers	Bitcoin, but DID operations are only anchored	Ethereum, but only the stream anchors go there	Polygon PoS
Incentive layer & data availability for the most important (keys & registries)	Ethereum ZK Validium	Ethereum	Polkadot	centralized consortium of servers	DID operations are stored in a network on IPFS without incentives	the actual streams are in a network w/o incentives	Polygon PoS
Data availability, storage, retrievability & addressing
Human-readable & persistent URIs for data without any hashes		URIs full of hashes (probably)	URIs full of hashes	URIs full of hashes - CIDs for IPLD objects	URIs full of hashes (probably)	URIs full of hashes	URIs full of hashes
Multiple ways to ask for a URI's document (in addition to caches/archives)	multiple ways: 1) user's IDM 2) source app identifiable from the URI 3) IPFS blob from the block 4) p2p network	1) user's Hub 2) p2p network	URIs contain only user id & content hash without user Hubs (yet) & p2p network	1) user's PDR 2) maybe p2p network with the content CID	probably 1) user's DWN 2) p2p network	only p2p network as Ceramic streams are an abstraction over IPFS	unsure - maybe the on-chain NFT post
Big reliance on a p2p network for delivering fine-grained messages	using a p2p network for specific URIs is the last resort	using a gossip-based pubsub protocol between peers & Hubs	not sure: their URIs contain only user id & content hash but they don't have an IDM/Hub/ PDR/DWN as a concept (yet)	no - talk directly to a user's PDR	not sure: perhaps could directly talk to a user's DWN	yes - IPFS, Ceramic Network & global DHTs
Push (broadcast) vs pull (polling) for fetching new content	both - event batches are broadcasted & new/individual documents can be requested	pull only - requires polling a user's Hub for anything new	both - event batches are broadcasted & new/individual documents can be requested	pull only - requires polling a user's PDR for anything new	pull only - requires polling a user's DWN for anything new	both - events are broadcasted & new/individual documents can be requested
Self-authenticating documents	proofs are validated by the blockchain	need to talk to Ethereum AND the host-certified user directory which can disappear OR change	merkle roots not present	proofs are validated by the transparency log
Ease of use for developers & users
Can leverage existing Web2 authenticating infrastructure	Can leverage all existing OAuth / SAML code
Easy to work with mental model vs high cognitive load & complexity	A bit more complexity compared to Web2
Can use "custodial" hosted services while retaining ultimate control
Ease of indexing & building responsive UI	can be as performant as Web2 and not constrained by block time

[1] [2]

1. X. ↩

2. X. ↩

What other projects get wrong

A list of problems with the contenders in the decentralized identity/media space:

No credible path to web-scale - some will hit a wall even at 1 million users. Most are vague around their scalability & data structures and don't put it front and center - obfuscating the most important bit. Instead of focusing on NFTs & developer APIs, start with the data and work up from that.

Complexity & lack of clarity - distributed systems engineers should easily figure out how they work & what the limitations are. Why build on something that others are probably having a hard time understanding as well and may not be around in the future?

"Developers care about risk." - Haseeb

"For the simplicity on this side of complexity, I wouldn't give you a fig. But for the simplicity on the other side of complexity, for that I would give you anything I have." - Oliver Wendell Holmes
Too financialized & trying to do too much - profiles & posts as NFTs, microtransactions, marketplaces, fan coins, tipping, content creator incentives.

"However, a downside I’ve observed in social networks where content is monetized is that user behavior becomes transparently driven by monetary incentives in ways that feel less genuine. This applies to influencer culture on Instagram as well, but cryptocurrency social networks bake it in from the start." - Jay Gerber

"The question remains: is the future of social media truly intrinsically linked to NFTs or is it a red herring?" - @mattigags
Users shouldn't need to use a token, use a wallet, or self-host to benefit from decentralized identity & an open social graph. Most people will always use custodial services.

"People don’t want to run their own servers, and never will." - Moxie
Linking online identity to public financial accounts on Ethereum/Solana/etc will have unintended consequences - a bad default.
Federated ones lack logical centralization which leads to fragmentation and no discoverability.
Some are solving just identity & the graph - without easy & persistent content addressing.
Social media is about aggregated views at scale - not p2p and direct comms.

"The emphasis of a social network is on "propagation" aka, propaganda." - didibus
Some use chains such as Ethereum for logical centralization & store vector commitments (Merkle roots) for events around key management (rotations, authorizations, sessions & revocations) but the data availability problem for whatever is committed is unsolved.
- The complexity is not encapsulated - there are many open questions, edge cases & failure scenarios and it would inevitably lead to assumptions & trust.
- Some anchor to Bitcoin but the time to finality matters a lot for UX - 10-minute block times with probabilistic finality is horrendous.
Some lack an economic incentive layer.

"Show me the incentive and I will show you the outcome." - Charlie Munger

Farcaster

Their architecture: link. The account registry is on a blockchain and everything else is off-chain.

Registry on Ethereum L1 - for new accounts, name/host changes & key management.
- No plans on moving to an L2 or their own chain. Also, state rent could eventually be introduced to Ethereum which would lead to further costs & complexity.
Keypairs & wallets required - harder mass adoption. Authorizations still require a signature from the root key.
Revocations invalidate all prior activity from a delegate:

"Unfortunately, this means that all messages signed by that signer will be lost since we cannot tell which ones were signed by the attacker." - source
- Root signer revocations are even more impactful.

The p2p network's ability to scale by passing around granular casts is questionable - they are already discussing possible flooding and nodes having to shadow ban and flag accounts based on behavior.

Focus is on partial views of the network as opposed to mass scale aggregation & indexing - although that could easily be implemented.
Cast URIs will look something like farcaster://id:8789213729/cast:0xf00b4r which is less readable than what Headjack will be offering with its addressing.

Overall good intuition about the concept of sufficient decentralization (putting only what is absolutely necessary on a blockchain) but the p2p node implementation takes on too much responsibility, complexity & assumptions (consensus, CRDTs, trees, ordering, flooding & replay attacks, etc.) and is lacking in other areas.

DSNP, Frequency & Project Liberty

Frequency (a Polkadot parachain) is the first implementation of DSNP (Decentralized Social Networking Protocol - whitepaper) as a standalone blockchain and has had the most influence over Headjack's design but the two have diverged in some key respects - the biggest of which are scalability, content addressability, UX & choosing Polkadot. Some of the problems with them:

No names within the project - just integer IDs for accounts. Content addressing URIs are based on hashes without connection to the batch # / service that published it - example: dsnp://78187493520/0x1234567890abcdef0123456789abcdef0123456789abcdef (source). Addressing content is much worse compared to Headjack's human-readable & persistent URIs.
Delegating applications to be able to post on behalf of users (analogous to authorization in Headjack) happens on-chain but requires a signature from the user (bulky - limiting throughput). New applications (& revocation) require the user to have access to their keys. Hierarchical delegation would allow for UX comparable to Web2 and would even allow for users without keypairs at all but DSNP doesn't have that - Headjack does.
100m$ of funding (so far) from just 1 person - Frank McCourt - no other capital & connections to reputable investors & influencers from either the crypto or tech space - generating hype & booting up the network effect might be very hard. They've been around since 2019.

TBD

Jack Dorsey's new "web5" project - slides, announcement.

Only anchors DID events to Bitcoin with vector commitments (Merkle roots) using ION & the Sidetree protocol.
- 10-minute block times with probabilistic finality. Factor in the loading times for the anchored content around key management that's on IPFS - not great at all if you want to log in/authorize a service or revoke access quickly.

Doesn't have a human-readable global name registry - lacks discoverability.
Doesn't have human-readable content addressing.
Focus is on users self-hosting their own data, running software locally & handling keypairs.
Developing their own Decentralized Web Nodes (DWN) software that would be relaying messages p2p - can't handle web-scale on such a granular level and aggregation is not even in the picture.

CyberConnect

Built on the Ceramic protocol & network.

TODO: working on incentives for pinning https://twitter.com/joelthorst/status/1588863780301156352

Requires the use of keypairs & wallets.
Every user has their own Ceramic data stream on top of IPFS - it is yet to be proven that the DHT & p2p layers can scale to hundreds of millions or billions of people.
The persistence of the social graph is handled by pinning IPFS data on nodes operated by them without any cryptoeconomic incentive for the data availability - it will grow into the tens/hundreds of terabytes for web-scale (Twitter scale: 400M users with 700 connections on average) - especially because they don't have a compact integer-based representation and everything is based on big individually signed actions. The upcoming Ceramic blockchain does not seem to be geared towards storage incentivization and will not be the solution to that.

"Long-term data retention on CyberConnect is guaranteed through Ceramic’s blockchain anchoring and a custom data pinning service." - source

DeSo

It requires wallets & users to pay for every interaction.
It puts everything on-chain and their plans to scale are with bigger blocks & sharding (see "Phase 4: Sharding") which is simply not practical for the true scale of the public web.
It financializes as much as possible (creator coins, etc.).
Their initial growth was fueled by huge sums of VC money but by now it has flatlined. It did reach 1.66$ billion market cap on the 2nd of October 2021 shortly after being listed.

Others

For details about ActivityPub, Matrix, Diaspora, Mastodon, Secure Scuttlebutt, Solid & others please refer to the excellent ecosystem review by the Bluesky project. Other good resources include:

Decentralized Social Networks - Jay Gerber
Blockchain Social Networks - Jay Gerber
There are many other projects in this space.

Implementation of Headjack

Headjack will be an Ethereum rollup - it's still not decided if it will have a custom VM (there are potential optimizations) or if it'll use the EVM.

It will make use of EigenLayer - that way it will be possible to reuse some of the staked $ETH in the Eth 2.0 consensus and to re-stake it in an opt-in way to offer economic security for the Headjack blockchain. A good talk about EigenLayer by Sreeram Kannan can be found here. Their whitepaper was released in February 2023.