Bitcoin and the immutable web

I've been thinking about what some of the next layer of apps that will be built on top of Bitcoin will be. Certainly, OpenBazaar is one of the more exciting applications and I believe has the potential to be the retail layer for ecommerce. Also, Storj has promise as a distributed Dropbox (although I believe it will possibly be replaced by a version that uses Bitcoin as the currency as opposed to it's own native token). Something that has yet to have been discussed much is what I will refer to in this blog as "The Immutable Web" (or unchangeable web). It's possible that the DATT project will turn into something like what I'm describing, but right now, they're describing the project as "A community with great content, where people get paid for their work.", which, while interesting is not exactly what I am going to describe here.

What do I mean by the immutable web? Well, to understand, lets say you had a piece of data, possibly text data, but any type of content will do, and wanted to ensure that people 1000 years from now were able to see it. How would you do it? Well, it would actually be quite hard to do reliably with today's technology. In particular, it's been very difficult to do this throughout history because during wars, the victors frequently destroyed libraries and other "content" throughout history. One of the often cited cases of this is the destruction of The Library of Alexandria. Some might argue that that doesn't happen anymore, but that's not accurate. Unfortunately, it is still the case that data is being destroyed by those that don't want information to be free. It's tempting to argue that this would not be the case when storing data with a popoular service like Facebook, Google, etc. For instance if you were to post a video on Youtube would it remain there for ever? Well, there's no reason to believe that data would be preserved and in fact, it seems unlikely to me that it would be preserved for extended periods of time. Also, it's definitely impossible if you violate any of the terms of service of Youtube, not just now, but into the future. For example, if you put bumper music in the background that someone claims to own, your video will be taken down. In particular, far into the future what if someone claims some kind of infringement after your death? What if someone doesn't like what you said or doesn't want people to know about something you said? To make a long story short, it's not something you can count on at all and I think it's highly unlikely that data on the web can be preserved, unless we create the immutable web.

Ok, I will answer my own question about what the immutable web is without giving any more examples. The Immutable Web is a decentralized distributed database of all content types which through the use of encryption and cryptocurrency allows for an incentive for storage providers to host content and protects that content by distributing it and hiding the actual location through encryption techniques (such as onion routing).

To be clear, I'm not talking about paying content creators in this case (or at least not directly). I'm talking about a distribution system for that content that offers proper incentives to host content and ensures any data that is put into this system permanently stays in the public domain regardless of whether or not anyone else finds that content to be objectionable.

So, how do we do this? Well, the main prerequisite that hadn't been around until recently was Bitcoin. Before Bitcoin, such a system would have been impossible because the incentive structure would be impossible. Now, that we have Bitcoin, we can setup such a system. So, how do we build it? It is a complex undertaking, but I see it as a merging of Bitcoin + Bittorrent + I2P. I'm not saying that it will actually necessarily use those technologies directly, but it will be an amalgamation of them. If we had a merger of these three technologies, you'd have a situation where consumers pay storage providers for the download of data in such a way where the consumer (or any observer) actually has no idea where the data is being hosted (due to the onion routing). With Bittorrent, we have this today with very popular content. For instance, if you want to download a very popular song or video, it will generally be available, because people are always downloading popular content and therefore, if at least 10 people or so are always downloading a file, it will always be available as these 10 people will be transferring chunks of the files to each other and onto the next 10 users as they intermittently join the swarm. The problem comes in with rare or less popular content. If you have a file that someone only downloads once per day, it's very likely that that file will not be available because once one downloads a file, one frequently disconnect from the network to avoid giving away free bandwidth or being discovered as participating in the network. The way that Bittorrent works is leachers download content from seeders in small chunks. Once a leacher downloads a chunk, it becomes a seeder for that chunk. Clients prefer to send chunks to other clients that have seeded recently so it's a system that incentivises cooperation. The problem is that once you have your file, there is no incentive to participate in the network further because you don't need anything. That's where Bitcoin comes in. If we could replace this preference algorithm with Bitcoin as the incentive (in the form of extremely small micropayments), there'd be an incentive to stay on the network and host content that is profitable. This would create a free market for hosts to store files to consumers. If you think about it, this is essentially what Youtube is doing, but in a decentralized way. They're hosting and serving all this data in exchange for serving ads to consumers. It's profitable for them to host even unpopular content that gets only one or two views. If we replaced this system with something like this Bitcoin + Bittorrent + I2P combo that I'm describing, we'd have a distributed version of Youtube which would be much more efficient, distributed, cheaper (since viewing ads is not free), and perpetual. Since hosts would only make money off of content that people actually download, there might be some content that eventually disappears, but since Youtube has a very long tail, I believe this same sort of long tail would occur in the system I'm describing as well. Also, unlike Youtube, such as system would be censorship resistant. There would be no way to enforce things like the DMCA and users would be free to mash up content as desired. Systems to flag duplicates and spammers would also be possible. Obviously, non video content would be able to work in this system as well.

One of the other interesting features of such a system is that all content could be timestamped into the Bitcoin blockchain so you'd know, provably, exactly when content was uploaded and you'd also know that it was not tampered with. Effectively, once you upload data into this Immutable Web, it would permanently be available in it's initial state. Data loss would be eliminated.

What would a system like this look like to users? It would probably look much like Youtube.com or Blogger.com, but it would be open source so different clients with various UIs would be available. It would be a gateway to all content available through all time. A business similar to bitcoin mining would form around efficiently storing copies of this data around the world. Incentives for low latency could be put into place to encourage decentralization of data as well.

The societal impacts of such a technology would be enormous. As they say knowledge is power, so now we'd put the power of all the world's knowledge into the hands of everyone. Some may be concerned that it would be impossible to maintain intellectual property. This may be a somewhat valid concern, but effectively, even with Bittorrent this is the case. From what we've seen it's just changed the way that content producers make money. While the music industry has lost revenue that they once made through CD sales, they can still make money off of live performances, merchandising, endorsements, etc. Bitcoin will also enhance the ability for musicians to do crowd funding and get donations. To me, it appears that the music industry has suffered greatly, but individual artists with talent have had an easier time to rise up through all the social media that is now available and it looks more like we're just cutting out the middlemen than hurting the artists. The same will apply to video as well.

So, in conclusion, the Immutable web will allow mankind to aggregate and distribute all the worlds content at extremely inexpensive commodity prices that reduce at the pace of moore's law. It's an exciting opportunity and it will be interesting to watch.

Bitcoin: Why a fee event would be good for Bitcoin

What is a fee event? As originally defined by core developer Jeff Garzik here, a fee event is an event where Bitcoin's blocks permanently fill up and thus some transactions must be rejected. As of today, there has not been a fee event because blocks are of variable size and over time, even transactions that include zero fee get into the blockchain. A fee event would be a situation where blocks remain completely full for an extended period of time (for instance 7 days in Garzik's definition). In this situation, zero fee transactions would be rejected by the network and even some fees below a particular threshold would not be included into the blockchain, ever.

According to most in the community, this sort of 'fee event' would spell doom to the Bitcoin experiment and must be avoided at all cost. In his recent medium post, Mike Hearn, in concluding bitcoin has failed stated: "Why has Bitcoin failed? It has failed because the community has failed. What was meant to be a new, decentralised form of money that lacked 'systemically important institutions' and 'too big to fail' has become something even worse: a system completely controlled by just a handful of people. Worse still, the network is on the brink of technical collapse.". He went on to say, "The block chain is full. You may wonder how it is possible for what is essentially a series of files to be 'full'. The answer is that an entirely artificial capacity cap of one megabyte per block, put in place as a temporary kludge a long time ago, has not been removed and as a result the network’s capacity is now almost completely exhausted.". There are many others in the community that share this view point, that if Bitcoin blocks fill up, Bitcoin will be destroyed and not work properly.

In this post, I will attempt to explain my view (which may be considered heretical to some) that a fee event is a good thing because it indicates that Bitcoin has succeeded. Furthermore, I will argue that it something we should encourage to happen sooner, rather than later due to the fact that it is inevitable if Bitcoin is to succeed.

First, lets start with the numbers. In a recent block that was almost (973.8 kB), a total of 2081 transactions confirmed. As an approximation, lets say that each 1mb of block can contain approximately 2000 transactions. If blocks occur every 10 minutes, that implies a limit of around 3.33 transactions per second. This number may not be exact, but it is within an order of magnitude at least. According to recent work by Jonathan Toomim, a block size of 3mb is indicated to be the largest size that is safe. This number is based on his experiments on the testnet. So, that would imply a limit of about 10 transactions per second (3 X 3.33 ~= 10). With upcoming technology (like Xtreme Thinblocks and others), we might be able to increase this number, but even then we're ultimately limited by the fact that all transactions must be broadcast to all participants in the network. This results in an O(n^2) scaling which is generally considered not acceptable. Where does this put us in terms of other payment networks? According to VISA, their peak capacity is 47,000 tps. While their annual average appears to be around 2,000 - 4,000. According to 2011 numbers, 282 million people have VISA cards (or 4% of the world's population). Even for that 4%, they don't exclusively user their VISA cards for purchases. For Bitcoin, we want to do better than that, but as these numbers illustrate, even tens to hundreds of thousands of transactions would not satisfy the world's demand for payments. The block sizes required for these amounts are unfathomable today and, even if we experience 17.7% growth in bandwidth and other technologies (as indicated as the growth rate of bandwidth by Cisco referenced by Peter Wuille in BIP103), there is little hope for Bitcoin as is to scale to the levels required to support all the world's transactions any time soon. This is not even to get into settling stocks/bonds/and other securities on the blockchain.

This means, that without question, unless Bitcoin is a failure (or just a niche success), Bitcoin blocks will eventually fill up. Therefore, a fee event indicates success of the Bitcoin network because it means that demand outstrips supply. The real questions the community should be asking is: when do we want them to fill up? and what do we do when they do fill up?

Regarding the former question "when do we want them to fill up?", it seems that many in the community want the blocks to fill up as far into the future as possible. But, in reality, that is only delaying the inevitable. Blocks will fill up, and it may be hard to accept, but we need a plan for how to handle it. When thinking about this question, one thing to keep in mind is that currently the people involved in the Bitcoin community are generally very bright and generally in a better position to understand the situation and react than the broader population. Also, very few people are dependent on Bitcoin today. Therefore, we have an opportunity to build something that we can demonstrate works before mass adoption and work out the kinks so to speak. In my view, we should take advantage of this time we have and build a sustainable system. This would lead me towards the conclusion, that we should let the fee event happen now and deal with the consequences while we're still young. Once the block size limit is reached, the block size can be increased to lower fees as long as it doesn't bring us beyond the true capacity of the system.

That brings us the question of what do we do when the blocks fill up? Fortunately, even if we do nothing at all (or practically nothing), Bitcoin could most likely be used as a new global reserve currency and serve as a more fair, secure, and verifyable gold standard. See a blog entry from 2011, in which Nick Szabo indicated (in the comment section) that he thought that "Bitcoin may be destined to become a high-powered money rather than a day-to-day payment system for the masses.". While this isn't the most ambitious of goals for the Bitcoin community, it's clearly an improvement over the present day system. In that scenario, intermediaries like Coinbase and other Bitcoin banks would use Bitcoin to settle with each other once a day and allow instant, low cost, micropayments between users without including the transactions on the block chain. The Bitcoin banks would settle with each other periodically or possibly use payment channels to do so instantly. The main benefits over the existing system is that fractional reserve banking would be impossible because consumers would demand to see proof of reserves from their bitcoin banks. Proof of reserves could also serve to reduce the costs of auditing and make it reliable as well. The controlled (and known) supply of Bitcoin is superior to the properties of Gold because we know how much Bitcoin exists at any given time and don't have to worry about a big discovery, or something similar. We'd also have the benefits of being on something much similar to a gold standard.

But more ambitious projects like Lightning Network promise to scale Bitcoin enough to bring it to the masses with more reasonable block sizes which are bigger than today, but within the realm of possibility in the near future. I'm optimistic that Lightning Network and other things (some yet discovered) will allow us to scale as we all hope, but in either case, my hope is the community will accept this reality and start planning for a realistic future for Bitcoin and allow it to reach it's true potential.

What the heck is Bitcoin Core thinking?

Yesterday, Eric Voorhees made a post on Reddit titled "IMHO BTC price will be weak until Core demonstrates competency in social consensus.". While I don't necessarily totally agree with this, I do see his point that Core (or other people that share the view of Core) can improve their communications and help build social consensus.

This blog post will be my attempt to help explain the thinking of Core. While I'm not a Bitcoin Core developer myself, I have been following this space as a Bitcoin holder for quite a while and also have 15 years experience as a software developer and software development manager. I believe I have an understanding of what the Bitcoin Core developers are thinking and will attempt to convey that.

In my experience as a developer, very frequently a product manager will come to you and say, I want you to implement X. If X is something technical, the next question is: why do you want me to implement X? In some cases, implementing X is the right solution, but in other cases it's not the best way from a development standpoint to achieve the goal that the product manager is attempting to achieve. Throughout the years, one of the frequent statements that I make to product folks is: "Just give me the requirements, don't get into implementation details". The reason that this statement is frequently useful is because it allows the product manager to focus on what they actually want and avoid dictating how to do engineering work. Engineering work is the task of the developer, not the product manager and we are trained to do it better. It's also our obligation to explain why we intend to implement the requirements the way we intend to implement them.

I believe that the Bitcoin ecosystem is focusing too much on implementation details and not on product requirements. What users are saying is: "I want a bigger block size now!". The block size is an implementation detail, not a requirement. The requirement is actually that users want to continue to have low fees. Users are also worried that Bitcoin's throughput will not keep up with demand for transactions. These are valid requirements, but the requirement is not to increase the block size, that's the implementation detail. In addition to the requirement that transaction fees stay low, there is the other requirement that Bitcoin remain decentralized. That requirement is more important than low fees because without decentralization, Bitcoin is no real different than legacy currency systems and thus has no real value.

So, given these real requirements (low fees for an increasing user base, and decentralization) we now leave it up to the engineers to come up with the solutions to these problems. That's what the two scalability conferences last year were all about. So, the engineers got together and discussed it and the solution they came up with is multi-fold:

SegWit

Lightning Network

IBLTs and Weak Blocks

A block size increase after these things are implemented A full technical discussion of the Core roadmap can be found here.

So, lets break this down. What do each of these things do? First SegWit. SegWit is a method to separate (Segregate) out part of the Bitcoin transactions called the signature (Witness). What this does is allow us to discard some of the data (the signatures) in blocks that are only needed at the time of validation over time. The benefit of this is that we can scale Bitcoin without some of the downsides of increasing the block size. The net-net of this is that we can expect somewhere between 1.7x to 2x increase in the number of transactions to fit into blocks based on the current proposal. As mentioned in the roadmap, this change is expected to be available in April, 2016. Now, this change alone does nothing, because wallets need to upgrade their code to support this change. But, several wallets have stated that they intend to roll out this change concurrently with the release of Bitcoin core's code. There is some analysis on how much of a cost reduction will be generated by SegWit that one can view here. So, once again, when we focus on what customers want (fees to remain low), all that one needs to do is upgrade to a SegWit enabled wallet (which should be available in April) and you will get somewhere near 50% reduction in cost of transactions and we're talking about a few months out.

Next, we have Lightning Network. There is a lot of technical detail to Lightning Network, but to summarize what it does is that it allows for effectively the same level of security of the Bitcoin blockchain without actually submitting most of the transactions to the blockchain. With this method, even with the current 1mb limit on block size, lightning network could allow 50 million users to do an unlimited number of transactions for very low fees. Full detail here. As seen in the link, modest increases to the block size would multiply this number such that vast number of Bitcoin users could engage in unlimited number of transactions. One of the additional benefits of Lightning Network is that payments are irreversible instantly and we no longer have to wait for multiple confirmations once you have your coins in the Lightning Network.

Next we have IBLTs and Weak Blocks. IBLT is relatively new data structure that allows for somewhat lossy data transmission. It was initially proposed for use in the Bitcoin network by Gavin Andreesen here. The basic idea is that you can compress the data into this new data structure and reconstruct the data even in the presence of partial loss of data. This is useful in the Bitcoin network because that's exactly what we have a network where some (or all) transactions may not be propagated to all nodes. Thus an IBLT allows for blocks to be transmitted much more efficiently and potentially cuts bandwidth in half. Weak blocks are a method for allowing miners to pre-send the blocks that they are working on so that when a "strong" block is discovered, the network generally knows which strong block was won and it has propagated through the network faster. These two techniques will allow for a massive improvement in bandwidth consumption of Bitcoin and when they are ready, a hard fork to increase block size could be implemented.

So, what does all of this mean? With the combination of these changes, we are looking at a MASSIVE increase in capacity of the network over the next 1-2 years which will keep transactions cheap and maintain decentralization. As I pointed out, those are the customer requirements, NOT a block size increase. That's the implementation detail only. There are also other technologies that are not included in this summary (e.g. Schnorr signatures, possibly technologies/combinations of IBLT/Weak blocks/other tech). This vision is actually much more ambitious than even a large block increase like BIP101. Even if we were able to support much larger blocks we couldn't support all the worlds transactions. Some have argued that we could come close with BIP101 (which may not be technically feasible), but I think this view is limited to something like replacing the VISA network. While replacing the VISA network sounds ambitious, if you look at all the worlds transactions (including cash payments, stock trades, etc), the VISA network is actually relatively small. With the technologies discussed in this blog entry and the Bitcoin Core roadmap, we could see a much more ambitious goal and really replace the vast majority of all transfers of value in the world. That would make the world a better place and meet the requirements of the users.