The security of a lightweight wallet can be improved by connecting to multiple full nodes or a trusted node.
-
Lisa can still censor transactions.
7. Proof of work
This chapter covers
-
Making transactions censorship-resistant by allowing multiple “Lisas”
-
Competing to produce the next block, or mining
-
Understanding miner incentives
The previous chapter made it hard for Lisa to remove transactions by introducing a blockchain in which Lisa signs all blocks. This chapter will take this a step further and make the system censorship resistant so Lisa can’t censor transactions.
To make the system censorship resistant, we’ll replace the digital signatures in the block headers with proof of work (Figure 145) to allow for any number of Lisas, or miners. These miners will compete to create the next block by trying to produce a valid proof of work. Miners can produce this proof by calculating a huge amount of cryptographic hashes. Wallets can now send their transactions to any or all of the miners to ensure that their transactions are being processed.
With the new proof-of-work system in place, miners want to make blocks as small as possible so they can upload them as quickly as possible to the shared folder. Miners have an incentive to exclude transactions, which was exactly what you wanted to avoid. To give miners an incentive to include a transaction, the transaction might pay a transaction fee that goes to the miner that produces a block that confirms the transaction.
The proof-of-work system replaces digital signatures in the block headers. But digital signatures were introduced to prevent Lisa from deleting transactions. Don’t worry—the proof-of-work system handles this too, but in a slightly different way. Instead of making it provable that Lisa cheats, you’ll make it hard and expensive to cheat.
Throughout this chapter, we’ll discuss miners’ incentives. Why would they mine? Why wouldn’t they delete transactions after being confirmed? What harm can a miner do if it controls most of the hashing computing power? We can discuss a lot of interesting dynamics regarding miner incentives.
7.1. Cloning Lisa
We discussed privacy a bit in Section 1.3.2 and Section 1.4.1 of Chapter 1. I noted that in a system with a central authority, this authority has absolute power over who gets to use the service and for what purposes.
Lisa is a central authority who can censor any transaction she wants. Suppose Lisa just read a book by a famous dietitian, in which she learned that cookies are bad for you. She feels that she must take action against the cookie orgy going on at the company. She starts to refuse to process transactions she suspects are paying for cookies—for example, by looking for transactions with a 10 CT output (Figure 146).
People wanting to pay for a cookie in the cafe will be denied service because their payments won’t go through. Lisa might also filter out other transactions that don’t have anything to do with cookies because she suspects they’re being used to pay for cookies.
Another possibility for censorship would be that Acme Insurances forces or bribes Lisa to drop suspicious cookie-buying transactions because it doesn’t want people to get ill from obesity.
What if you could have several people like Lisa so you don’t rely on a single person being honest and available all the time? Suppose you let Tom and Qi also start doing what Lisa’s doing. If wallets emailed all transactions to all three of them, the risk of a transaction being censored would decrease dramatically. But how would they produce the blocks in a controlled way so they didn’t constantly produce conflicting blocks at the same height?
7.1.1. Block collisions
Suppose the current block height is 100. Tom and Qi have just published their block-signing public keys on the billboard and on the company’s intranet. All wallets start sending transactions to all three block producers, or miners. Figure 147 illustrates what happens.
If they all just do what Lisa did, they’ll produce a block every 10 minutes, resulting in three different blocks with roughly the same transactions. The major differences between the three conflicting blocks are the coinbase transaction and the signatures. The coinbase of Tom’s blocks would pay the block reward to Tom’s cookie token address, whereas the coinbase of Lisa’s blocks would pay the block reward to Lisa’s cookie token address.
7.1.2. Drawing lucky numbers
To avoid this problem, the miners must somehow decide which one produces the next block. They could take turns, but this would be complicated because Lisa’s computer might be broken, or Tom might refuse to create a block for some reason. In such a scenario, the system would halt.
Let’s try another naive approach (Figure 148). Every second, each miner draws a random number between 0 and 999,999. If a miner happens to draw a number in the range 0 to 555, it will immediately sign and publish a block. The probability of drawing a lucky number on a single try is low—556/1,000,000, or roughly 1 in 1,800 tries. The miners draw one number per second, so each miner is expected to draw a lucky number every 30 minutes (1,800 seconds) on average. The three miners together will then produce on average 1 block every 10 minutes.
When a miner draws a lucky number, chances are low that either of the other two miners also drew a lucky number at the same time. This means, usually, only a single miner will produce the next block.
The miners save their blocks as <_last-8-hexdigits-of-blockid_>.dat in the shared folder, so multiple blocks at the same height don’t have to worry about file naming. An example of a filename is 9ce35c25.dat.
The system ticks on quite well, but once in a while, two miners draw a lucky number at the same time. They aren’t aware that another miner also drew a lucky number, so they’ll both produce a block at the same height. This situation is known as a blockchain split because the chain splits in two. Both branches are equally valid, so which one is “correct”? Which miner will “win” the block and collect the block reward of 50 CT?
You don’t know the winner yet. It’s up to the miners to decide which branch they want to extend with their next blocks. In Figure 148, both Tom and Qi have created a block at height 106. The different miners would likely think as follows:
- Tom
-
I will extend my own block, because if I win the next block, I get rewards from 2 blocks.
- Qi
-
I will extend my own block, because if I win the next block, I get rewards from 2 blocks.
- Lisa
-
I will extend either of the 2 blocks, I don’t care which. I’ll just pick the first one I successfully verified: Tom’s block. The blocks might not have landed in the shared folder at exactly the same time, so it makes sense to extend the first valid one I saw.
When the miners have picked a block at height 106 to extend, they build a new block at height 107 and start drawing numbers again. Several outcomes are possible from this situation, assuming everyone is honest: an immediate resolution, a delayed resolution, or a split of a split.
Immediate resolution
In the simplest and most common case, exactly one miner is the first to draw a lucky number. This time, it’s Lisa who’s lucky (Figure 149).
Lisa extended Tom’s block, so the branch Tom and Lisa were working on gets 1 block longer. A rule for this blockchain is that the longest chain is the correct one. This will change further along in this chapter, but for now, we follow the longest chain.
Qi, who was trying to extend her branch, notices that the other branch just got longer because Lisa published a block for that branch. Qi knows everyone else will follow the longer branch. If she stays on her short branch, she’ll probably never catch up and become longer than the other branch. She’s better off abandoning her short branch and moving over to the longer branch. Now, everyone is working on the same branch again, and the tie is resolved.
When Qi switches over to the new branch, she’ll mark all transactions of her old branch (that aren’t already in the new branch) as pending. They will be up for grabs for future blocks on the new branch. Nodes maintain a pool of pending transactions, generally called the memory pool, or mempool. To mark a transaction as pending means putting it in the mempool.
Because Qi abandoned her branch, she also abandoned her block reward. Her block will never be part of the longest chain, so she’ll never be able to spend the block reward in her block. Only blocks on the longest chain will affect the UTXO set.
Delayed resolution
But what would happen if both Lisa and Qi happened to draw a lucky number at the same second (Figure 150)? This would mean both branches would be extended by 1 block each. You still don’t know which one is the correct branch. Miners will again pick sides and try to extend their branch of choice.
Let’s say Tom is the next to draw a lucky number. He builds the next block on his branch, which now becomes 3 blocks long. It becomes longer than the other branch, which is only 2 blocks long (Figure 151).
Every miner acknowledges this by switching to Tom’s branch and moving on from there.  Again, Qi happens to be the loser in this fight.
Split of split
Say instead Tom and Lisa both draw a lucky number at the same time. They would then both extend Tom’s branch. The result would be a split of the split (Figure 152).
You now have three branches. Qi’s branch is probably abandoned because it’s shorter than the two new branches, Lisa’s branch and Tom’s branch. This new competition will be resolved in the same way as the first split. It will be resolved
-
Immediately by the next block
-
After a delay because 2 blocks appear simultaneously, one on each branch
-
When a new split is introduced on either of the two new branches
7.1.3. Probability of splits
Eventually, one branch of a split will win. The likelihood that two branches of length X happen next diminishes rapidly for increasing X:
| Branch length | Probability | Happens about every … |
|---|---|---|
1 |
5.6e-4 |
2 weeks |
2 |
2.1e-7 |
90 years |
3 |
7.6e-11 |
250,000 years |
4 |
2.8e-14 |
700,000,000 years |
A split of branch length 1 is quite likely to happen, but a branch of length 2 probably won’t happen during Lisa’s lifetime (she’s 45). No matter how long the splits are, eventually they’ll resolve with a winner. This seems like a nice scheme. But it has its issues:
-
You can cheat with lucky numbers. You can’t prove you actually drew an honest lucky number.
-
For every new miner, the system becomes more censorship resistant but also more vulnerable to private-key theft. More computers containing private keys means a higher probability that a key gets stolen. A stolen block-signing private key will let the thief create blocks by cheating with lucky numbers and collect the rewards for themselves.
-
For each new miner, the risk that one miner cheats with lucky numbers increases.
-
You can’t just add new miners to the system. You need to lower the lucky-number threshold as more miners are added, to keep the average of 10 minutes per block and the money issuance at the desired rate.
Clearly, this system won’t be able to increase the number of miners beyond a controlled group of highly trusted participants. You’ll get a flood of blocks as miners start cheating, but you can’t prove they’re cheating. It’s possible they’re just really, really lucky.
7.2. Where were we?
This chapter is about proof of work. I haven’t introduced that term properly yet, but I’ll do so in the next section.
In the Bitcoin overview in Chapter 1, Section 1.2.3, you saw that one miner takes the lead and decides which transactions go into the next block and in what order. Bitcoin uses proof of work to decide who gets to take the lead (Figure 153).
Proof of work lets you randomly select a leader among all miners without using a central authority. Pay close attention to this chapter because this is the essence of Bitcoin. It’s what makes Bitcoin truly decentralized. We want the system decentralized because this makes it censorship resistant. If the system has a central authority, then transactions can be censored.
Cloning Lisa was a first step toward decentralization, but it isn’t perfect because you trust miners to draw honest lucky numbers.
7.3. Forcing honest lucky numbers
What if you could force miners to not cheat with lucky numbers? It turns out that you can! You can make them perform huge amounts of computations with their computers and have them prove they’ve performed the work. You can make them perform so much work that it takes each of the three miners about 30 minutes on average to produce a block, which will result in a 10-minute block interval, just as before.
The trick is to replace the digital signatures in the block header with proof of work (Figure 154). Suppose Qi just published a block, and the cafe’s full node wants to verify that this block is valid. Besides verifying the usual stuff like transactions and the merkle root, the full node must verify that Qi’s block includes a valid proof of work. The proof of work is valid if the block-header hash—block ID—is less than or equal to an agreed-on target that’s written in the block header, as Figure 155 shows.
The nonce in this block header is 492781982
. Qi selects this value
using trial and error. The next section will explain how this works.
To determine whether a block’s proof of work is valid, compare the 256-bit block ID to the 256-bit target written in the block header. In Figure 155, the block ID and target are
block id: 000000003c773b99fd08c5b4d18f539d98056cf72e0a50c1b57c9bc429136e24 target:<PHONE_NUMBER>6eb900000000000000000000000000000000000000000000000000
In this example, the block ID starts with 000000003…
, whereas the
target starts with 000000009…
. The block ID is less than the target,
which means this block’s proof of work is valid.
The target is a number agreed on by all full nodes and miners. This target will change every now and then according to some common rules. Such a change is called a retarget, and I’ll describe it in a later section. For now, you can regard it as a fixed number that must be set in the block header.
7.3.1. Producing a valid proof of work
To create a new block, a miner must produce a valid proof of work for the block before it’s considered valid. To make a valid proof of work, the miner must create a block-header hash that’s less than or equal to the target in the block header.
A block ID is a double SHA256 of the block header. As you learned in Chapter 2, the only way to find a pre-image to a cryptographic hash function is to try different inputs over and over until you find one. The same goes here; the miner must try different block headers until it finds one that hashes to a value less than or equal to the target.
Let’s go back in time and look at how Qi created her block. She creates
a block, sets the target to<PHONE_NUMBER>6e…
, and sets the nonce to 0
.
She then tests whether the proof of work is valid (Figure 156).
She calculates the block ID by hashing her block header with double
SHA256. In this case, the block ID is aa9c614e7f50…
. This number
is bigger than the target:
block id: aa9c614e7f5064ef11eedc51856cc7bfcdf71a1f2d319e56d4cc65bda939be79 target:<PHONE_NUMBER>6eb900000000000000000000000000000000000000000000000000
The rule is that the block ID must be less than or equal to the target for the proof of work to be valid. Qi fails miserably.
This is where the nonce comes in. A nonce is just a silly number
that doesn’t mean anything. It can be set to any value. Qi initially
set the nonce to 0
, but she could just as well have set it to 123
or 92178237
. The nonce helps make a change in the block that will
affect the block ID without changing any real data, like transactions
or the previous block ID.
Qi will now try again to make a valid proof of work. She increases the
nonce from 0
to 1
and tests the validity again (Figure 157).
When Qi changes the block header by increasing the nonce, the block ID changes—any tiny change in the header will result in a completely different block ID. This is the same property displayed in Section 2.2 in Chapter 2, when we changed the cat picture (Figure 158).
The new block ID is 863c9bea5fd8…
This is also bigger than the target.
Qi fails again. I’m sorry, but there’s no way around it—Qi must try once
more. She increases the nonce from 1
to 2
and tests again (Figure 159).
The result is the same: miserable failure. The block ID was
005ce22db5aa…
this time, which is still bigger than the target.
She repeats this over and over. For example, Figure 160 shows her 227,299,125th try. It was close, but close doesn’t help. She has to keep trying (Figure 161). And finally she gets the result shown in Figure 162.
The nonce 492781982 results in a block ID 000000003c77…
. She
compares this to the target:
block id: 000000003c773b99fd08c5b4d18f539d98056cf72e0a50c1b57c9bc429136e24 target:<PHONE_NUMBER>6eb900000000000000000000000000000000000000000000000000
Wow—this block ID is less than the target! Qi has performed a great deal of work to find a nonce that results in a block ID less than the target. She’s created a block with a valid proof of work. Great, now she’ll publish the block to the shared folder.
It’s important to realize that all miners build their own unique blocks. For example, Tom is working on his own block concurrently with Qi (and Lisa), but his set of transactions is different than Qi’s because his coinbase transaction pays the block reward to himself, whereas Qi’s coinbase transaction pays the block reward to Qi. This difference will cause the merkle roots in their respective block headers to differ. If Tom sets Qi’s winning nonce, 492781982, on his own block, he likely won’t meet the target. Other things that probably differ between their blocks could be the timestamp or the selected list of transactions.
7.3.2. Why is this good?
Anyone can pick up the block from the shared folder and verify that the rule is met—the block ID is less than or equal to the agreed-on target. Block verification is now slightly different than before (Figure 163).
The difference from verifying a digitally signed block is that the full node verifies the block producer has provided a valid proof of work instead of a valid digital signature.
With proof of work, you don’t need anything other than the block itself to determine if the block is valid. You used to need stuff from outside the blockchain—the miner’s public key from the bulletin board. This is a major leap forward toward decentralization. No central sources for public keys are left that can be manipulated.
7.3.3. Comparing with lucky numbers
The blockchain will grow the same way as before, but the drawing of lucky numbers is replaced by hashing the block header (Figure 164). Table 20 compares the two systems.
Instead of drawing a random number each second, the miners draw a number about every 0.02 microseconds through cryptographic hashing. At the same time, the lucky number limit, or target, is set to the 256-bit number<PHONE_NUMBER>6e…=926eb9*2200 instead of just 555.
| Idea | Target | Possible values | Draw every | Average block time | Best chain in a split |
|---|---|---|---|---|---|
Lucky numbers |
|
1,000,000 |
Second |
10 minutes |
Longest chain |
Proof of work |
926eb9*2200 |
|
0.02 microseconds |
10 minutes |
Most work chain |
subtle but important difference is that with proof of work, it’s the chain with the most accumulated proof of work that’s considered the best branch to follow. In the lucky numbers case, nodes followed the longest chain. The accumulated proof of work for a blockchain is the sum of the difficulty of each individual block in the chain.
The difficulty of a block is a measurement of how many times harder it is to find a valid proof of work for that block compared to finding it for the genesis block.
More exactly, the difficulty of block B is calculated like this:
The target of the genesis block is divided by the target of B, which makes the difficulty of the genesis block exactly 1.
The gist of this is that the higher the target of a block, the lower the difficulty of that block, and the lower the target, the higher the difficulty. So, we sum all blocks’ difficulties in the blockchain to get the chain’s accumulated proof of work.
From now on, I’ll refer to the branch with the most accumulated work as the strongest branch or strongest chain. Another commonly used term is best chain. The distinction between the longest and the strongest chain will become important in Section 7.5.2 when I’ve introduced difficulty adjustments.
7.3.4. What if you run out of nonces?
The nonce is a 32-bit number. This is pretty small. If a miner has tried all 4,294,967,296 possible numbers without success, they must do something else to change the block header. Otherwise, they’ll redo the exact same tries they’ve already made. Several options exist for making a change (Figure 165):
-
Change the timestamp slightly.
-
Add, remove, or rearrange transactions.
-
Modify the coinbase transaction.
Changing the timestamp is straightforward—just add a second to the timestamp, and the header will be different. If you use one of the other two options, you’ll have to recalculate the merkle root because the transaction data has changed. When the merkle root is updated, the header changes.
Once you make any of these changes to the block, the header will change
so the nonce can be reset to 0
, and the miner can begin hashing again.
7.4. Miners have to move out
The company thinks the proof-of-work system is nice and all, but it doesn’t want to pay for the electricity needed to perform all this work. Because computers run on electricity, the more calculations a computer makes, the more electricity it needs.
The company decides that miners must run their mining software elsewhere, such as in their own homes. This is fair. After all, miners are rewarded with 50 CT for each block they find. The electricity cost for them to produce a block is less than 50 CT. The current market value of 50 CT is five cookies in the cafe, and each cookie token is currently traded at about 20 cents. Each block gives a miner about $10 worth of cookie tokens, which isn’t bad given that they each produce about 48 blocks per day.
Let’s look quickly at the hashrate of our three miners. The hashrate is a measurement of how many hashes (tries) they can perform per second:
| Miner | Hashrate (million hashes/s) | Expected blocks per day |
|---|---|---|
Lisa |
100 |
48 |
Tom |
100 |
48 |
Qi |
100 |
48 |
Total |
300 |
144 |
This system will produce about 144 blocks per day, which is 1 block per 10 minutes on average.
7.4.1. Adding more hashrate
An interesting aspect of this system is that anyone can become a miner without asking for permission. They can just set up a computer at home and start building blocks. Blocks are no longer tied to a person but to an amount of computing work:
- Lisa adds to her hashrate
-
Lisa finds this business of mining at home lucrative. She decides to add another similar computer at her house, which effectively doubles her hashrate.
- Rashid becomes a miner
-
Rashid also wants to join the mining business. He sets up a computer at home that competes for new blocks. His computer is slightly faster than the competitors’, so he expects to produce more blocks per day than, for example, Qi.
After Lisa’s and Rashid’s added hashrate, the total hashrate in the cookie token system has increased significantly:
| Miner | Hashrate (million hashes/s) | Expected blocks per day |
|---|---|---|
Lisa |
200 |
96 |
Tom |
100 |
48 |
Qi |
100 |
48 |
Rashid |
150 |
72 |
Total |
550 |
264 |
Look: we’re producing more blocks per day than we designed for! The goal is 144 blocks per day, and 264 is significantly more than this. The block rate is too high, almost double the desired rate.
7.4.2. Problems with a high block rate
A higher block rate might seem beneficial because the confirmation time of transactions will decrease, but it comes with some problems.
Too-fast money creation
Remember the planned money supply curve from Chapter 2? The plan was to issue half the money supply, 10.5 million CT, during the first four years; then, during the next four years, issue half of that, 5.25 million CT; and so on, until the issuance rounds down to 0. This whole process would take about 131 years.
Now, because Lisa beefed up her mining and Rashid added his mining computer, the issuance is too fast. With this high block rate, it will take only half the time until all the cookie tokens are created.
This means the increase rate in money supply is 264/144 = 1.8 times the desired supply increase rate.
More splits
Splits happen naturally every now and then. But when the block rate increases, the risk of natural splits increases. Imagine if 3,000 people started mining in their basements. This would increase the block rate by 1,000 times. Each and every second, several miners would find a valid proof of work and publish a block. There would be splits on almost every block height. This makes transactions in recent blocks less reliable because those blocks can more easily be split off from the main chain.
This would also be problematic from a security perspective because if two branches have about 50% of the total hashrate on each branch, individual branch security is cut in half. We’ll discuss blockchain security further in Section 7.6.
7.4.3. What’s fixed?
We’ve fixed the hard problem of forcing “honest lucky numbers” in an interesting way. Let’s see what issues from Section 7.1.3 remain:
-
You can cheat with lucky numbers. You can’t prove you actually drew an honest lucky number.
-
For every new miner, the system becomes more censorship-resistant but also more vulnerable to private key theft. A stolen block-signing private key will let the thief create blocks by cheating with lucky numbers and collect rewards for themselves.
-
For each new miner, the risk that one cheats with lucky numbers increases.
-
You can’t just add new miners to the system. You need to lower the lucky-number threshold as more miners are added, to keep the 10 minutes per block average and the money issuance at the desired rate.
There’s only one problem left in the list. We’ll fix it in the next section.
7.5. Difficulty adjustments
Now that you’ve added more miners and more hashrate to the system, the block rate has increased. This is because the miners collectively make more tries per second than before, resulting in more blocks being produced per hour.
Everyone has agreed on the target in the block header, but you need to adjust the difficulty of mining a block to cater to increased or decreased total hash rate. The target is adjusted after every 2,016 blocks. This adjustment is called a difficulty adjustment or retarget, and the 2,016-block period is called a retarget period. Remember that each block contains a coinbase transaction that creates 50 new cookie tokens. You want 1 block per 10 minutes on average, to keep the pace of newly minted cookie tokens at the desired rate. That’s two weeks for 2,016 blocks.
If the last retarget period was more than two weeks long, the target must increase to increase the probability that a block header hash will meet it. You decrease the difficulty. If the retarget period was less than two weeks long, you must decrease the target to decrease the probability of meeting it. You increase the difficulty.
The new target, \(N\), is calculated as \(N=O \times F\), where \(O\) is the old target and \(F\) is a target change factor that depends on the last retarget period, as Figure 166 shows.
Generally, we calculate the new target, \(N\), from \(O\) and the duration, \(T\), of the last retarget period as follows:
The target can’t change by more than a factor of 4 or by less than a factor of 1/4 to limit the effect of certain double-spend attacks where someone isolates a victim’s node from honest nodes to manipulate the difficulty in their favor. You can read about it at Web resource 15.
7.5.1. Rules for timestamps
The block header contains a timestamp. Timestamps are important because you want the system to automatically adjust the target without human intervention so that, on average, 1 block is produced per 10 minutes. The block-creation rate is important because you want a predictable issuance of new cookie tokens.
The miner creating a block sets the timestamp to the current time before producing a proof of work. But because different full nodes run on different computers, their clocks might not be in perfect sync.
Suppose Lisa produces a block with timestamp 2017-08-13 07:33:21 UTC and publishes it on the shared folder. Tom then produces the next block, but his clock is behind Lisa’s clock.
Tom produces a block with an earlier timestamp than the previous block. This isn’t a problem as long as the timestamps don’t differ too much (Figure 167).
The timestamp must obey a few rules. Suppose the cafe’s full node is about to verify Tom’s block:
-
The timestamp must be strictly later than the median of the previous 11 timestamps. This median is commonly referred to as the block’s median time past.
-
The timestamp must be at most two hours in the future according to the cafe’s clock.
These rules ensure that no one manipulates their blocks’ timestamps to influence the next target calculation. Imagine if the last block before the retarget had a timestamp six weeks after the current time. This would cause the next target to increase by a factor of 4, as Table 21 shows.
| Block height | Timestamp (ignoring seconds) | Elapsed timestamp time |
|---|---|---|
H |
2017-07-31 06:31 |
0 |
H+1 |
2017-07-31 06:42 |
11:17 |
… |
… |
… |
H+2013 |
2017-08-14 07:22 |
2 weeks and 51 min |
H+2014 |
2017-08-14 07:33 |
2 weeks and 1h 2 min |
H+2015 |
2017-09-25 08:51 |
8 weeks and 2h 20 min |
The last timestamp is six weeks later than when the block was actually mined. All full nodes will reject this block because it violates the timestamp rules. Someone wants to manipulate the target. If this block had been accepted, the next target would be four times bigger than the current target, making it four times easier to find a valid proof of work. This kind of misbehavior is prohibited by the timestamp rules just described. Given that you can’t lie more than two hours with your timestamp, the next target can’t be manipulated more than marginally.
7.5.2. Chain strength vs. chain length
Let’s get back to the discussion on chain strength and why it’s important not to merely look at chain length. It intuitively seems reasonable that the harder it is to rewrite the chain’s history, the better, so you should follow the strongest chain. But when do the strongest and longest chain differ?
They can differ for several reasons:
-
Natural split right before a retarget
-
Accidental splits due to incompatible software versions
-
Deliberate splits as an attack against the honest chain
We’ll look only at the first option here. Suppose a natural split occurs (Figure 168).
This is an unlikely scenario, but we need to consider it because it might happen. A split happens right before a retarget, and the 2 blocks’ timestamps differ by four hours. Next, 2 new blocks are produced at the same time, one on each branch. These new blocks have been retargeted based on different histories. The last timestamps in the respective retarget periods differ by four hours, which causes the new targets to be different. Recall the retarget formula:
Because the new targets are different, the new difficulty of the last block on each branch is different. This means the chain strength differs because the branches now have different accumulated proof of work.
7.6. What harm can miners do?
In Chapter 6, you made sure Lisa couldn’t undo transactions without revealing her fraud attempt. You did this by requiring Lisa to digitally sign blocks so anyone can verify that Lisa has approved a block. If she later signs a competing block on the same height that replaces her own transaction with a transaction paying to herself instead, everyone will notice and hold her accountable.
Now the situation is different. Lisa doesn’t sign her blocks anymore. The blocks are anonymous—nothing ties Lisa to a certain block. Doesn’t this mean she can double spend again?
Well, yes, if she’s very lucky.
7.6.1. Double spending
Suppose Lisa is about to pay for a cookie in the cafe. But at the time she pays, she also prepares a double-spend transaction (Figure 169).
C is the transaction to the cafe. L is Lisa’s double-spend transaction that she’s going to use to snatch her money back. Both transactions are perfectly valid on their own, but both can’t be valid at the same time because they both spend a common output. An output can be spent only once.
Lisa sends the honest payment, C, to all miners. While other miners try to add her honest transaction into a block and create a valid proof of work, Lisa secretly puts the double-spend transaction, L, into a secret block of her own and starts working on that block (Figure 170).
Lisa’s goal is to secretly find a valid proof of work for her fraud branch, containing L, that exceeds the honest chain’s proof of work. If she succeeds, she’ll publish all blocks in her branch, and all miners will switch over to her branch and start working to extend her branch instead. For simplicity, let’s assume this all happens without any retargets (difficulty adjustments) happening; we’re in the middle of a retarget period. This means all blocks have the same target (or difficulty), so we can strictly look at branch length instead of branch strength (accumulated proof of work).
A bunch of miners are trying to confirm Lisa’s honest transaction, C, while Lisa works to find a valid proof of work for her block with the double-spend transaction, L. The cafe is waiting for a valid transaction before it hands out the cookie.
Eventually, the honest transaction will be confirmed on the honest chain. The cafe sees that block, verifies it, and gives the cookie to Lisa. Lisa eats it. While she swallows the last crumb, her computer happens to find a valid proof of work for her block. She doesn’t publish her block yet because it won’t help her. Miners are already mining on the honest branch because that’s where they first saw a block.
The combined hashrate of all miners on the honest chain is 350 Mhash/s, whereas Lisa has only 200 Mhash/s. This means the honest chain should be able to find blocks more often than Lisa.
But everyone gets lucky once in a while. Lisa is lucky to find yet another block on her fraudulent branch. She now has 2 blocks on her branch, whereas the honest branch is only 1 block long. Lisa has more total proof of work on her chain than the honest miners have on their branch. Lisa publishes her 2 blocks to the shared folder.
Other miners will see those 2 blocks, see that Lisa’s branch has more proof of work than the honest branch, and switch over to Lisa’s branch. The miners that switch can’t see that a crime is being committed or who create the blocks; they’ll neutrally jump to the strongest valid chain.
The result is that transaction C to the cafe is effectively undone. It’s no longer part of the chain with the most proof of work. The cafe has lost the 10 CT it thought it had when it gave the cookie to Lisa.
From this point forward, new blocks will extend Lisa’s branch, and things will continue normally. The block with transaction C will become stale.
7.6.2. Protecting against double-spend attacks
Although the odds are against Lisa, she could get lucky and succeed in a double-spend attack, as in the previous example. Trying to pull off a double spend of 10 CT isn’t economically feasible from Lisa’s perspective. She risks spending lots of electricity and making her own blocks stale if she doesn’t succeed. She’d lose out on the rewards from those stale blocks.
But what if she tried to double spend a larger amount than 10 CT: say, 100,000 CT? Then it might be worth it for Lisa to try to double spend. Just imagine if she could buy the whole cafe and pull off a double-spend attack. Then she would have a cafe and still have her 100,000 CT.
The cafe owner is willing to sell the cafe to Lisa for 100,000 CT. But the cafe is, of course, aware of double-spend attacks. So, the cafe owner tells Lisa that for this much money, he’ll give her the cafe after six confirmations.
What does this mean? Lisa must pay the cafe owner 100,000 CT and then wait until the transaction is included in a block and 5 blocks have been built after that block. Only then will the owner hand the cafe over to Lisa.
To pull off a double-spend attack, Lisa must build an alternate branch in secret, just like in her previous attack, while the cafe awaits six confirmations. When the cafe owner has seen six confirmations and given the cafe to Lisa, she must at some point upload a stronger double-spend branch to the shared folder. This means Lisa must be lucky for a longer time period than in the previous example.
Let’s see how it goes (Figure 171).
The outcome is as expected. Lisa couldn’t produce more blocks than the honest chain in the long run. She gave up at 7–4.
The following table shows the sequence of events in this example:
| Event | Score (C-L) | Comment |
|---|---|---|
1, 2 |
0-0 |
Lisa starts mining on her secret branch containing her double-spend transaction. She also sends out a payment to the honest miners. |
3 |
0-1 |
Lisa finds a block but keeps it secret. She doesn’t want the cafe to notice that there’s a double-spend attack going on. |
4 |
1-1 |
The honest payment, C, gets its first confirmation. The cafe will wait for 5 more blocks before making the deal. |
5, 6, 7, 8, 9 |
5-4 |
Lisa keeps up OK, but she’s 1 block behind and must create 2 blocks more than the cafe to succeed. |
10 |
6-4 |
The honest transaction has six confirmations. Lisa gets the cafe. The deed of transfer is signed. Lisa keeps trying to catch up. |
11 |
7-4 |
Lisa thinks this stinks. The probability of creating 4 blocks more than the honest chain in the future is tiny. |
Lisa gives up for several reasons:
-
She realizes she doesn’t have enough hashrate to catch up and surpass the honest chain. At any moment, the probability that Lisa finds the next block is 200/550 = 0.36. This means the probability that the honest miners find the next block is 1 – 0.36 = 0.64. Blocks are going to be found much faster on the honest chain.
-
For each minute she keeps trying, her computer consumes electricity that costs money. If she doesn’t succeed in her double-spend attempt, the electricity cost will have been in vain.
-
For each block she mines on her own chain, she’ll lose the 50 CT block reward if she fails.
The key here is that the cafe demanded six confirmations. The more confirmations needed, the harder it is for Lisa to build a stronger branch than the honest miners. She needs more luck.
When the cafe got its six confirmations, Lisa was 2 blocks behind. She would need to grow faster than the honest chain and become 1 block longer than the honest chain. Her chances are small. The more blocks she has to catch up with, the smaller the chances, as Table 22 shows.
Catch-up blocks (z) |
Probability, \(q_z\), of the attacker catching up if they have \(q%\) of hashrate |
|||||
|---|---|---|---|---|---|---|
1% |
10% |
18% (Tom) |
36% (Lisa) |
45% |
50% |
|
1 |
0.010101 |
0.111111 |
0.219512 |
0.562500 |
0.818182 |
1.000000 |
2 |
0.000102 |
0.012346 |
0.048186 |
0.316406 |
0.669421 |
1.000000 |
3 |
1.0e-06 |
0.001372 |
0.010577 |
0.177979 |
0.547708 |
1.000000 |
4 |
1.0e-08 |
0.000152 |
0.002322 |
0.100113 |
0.448125 |
1.000000 |
5 |
1.1e-10 |
0.000017 |
0.000510 |
0.056314 |
0.366648 |
1.000000 |
6 |
1.1e-12 |
1.9e-06 |
0.000112 |
0.031676 |
0.299985 |
1.000000 |
10 |
1.1e-20 |
2.9e-10 |
2.6e-07 |
0.003171 |
0.134431 |
1.000000 |
The probability, \(q_z\), is calculated as
Look at the column for a 36% hashrate, which is what Lisa has. When she’s 3 blocks behind, she must produce 4 blocks more than the honest miners in the future. This gives her a roughly 0.10 chance of ever succeeding in this double spend—if she’s prepared to try indefinitely. She probably doesn’t want to keep trying forever, which gives her a slightly smaller probability of succeeding.
Tom tries to double spend, too
Imagine if Tom attempted a double spend instead of Lisa (Figure 172). He’s only got half of Lisa’s hashrate, 100 Mhash/s.
Tom’s chances are smaller than Lisa’s. He’s getting a bit lucky and finds 2 blocks early, but after falling 2 blocks behind the honest miners, he thinks his chances are too small and gives up. Having to produce 3 more blocks than the honest miners at a probability of about 0.011 (\(z=3\)) is a terrible thought.
Tom’s a smart guy and knows not to try this. He understands that he’s far better off securing the blockchain along with everybody else and getting his fair share of the rewards than trying to defeat it. After all, with 18% of the hashrate, he gets almost a fifth of all block rewards. That’s more than 50 CT per hour. After 2,000 hours, or 12 weeks, he’d have made 100,000 honest cookie tokens, instead of trying to steal them.
Tom and Lisa collude to double spend
Together, Tom and Lisa have 300 Mhash/s. They control more than 50% (54.5%) of the total hashrate (Figure 173).
If they cooperate on a double-spend attack, and if they’re willing to try indefinitely, their chances of succeeding are 100% (Table 22). If they’re only willing to try for, say, 50 blocks, their chances are still very close to 100%. This scary scenario means Tom and Lisa can rewrite history at will. They run faster than the combined hashrate of all the honest miners. They can create a branch from any block in the blockchain history, work their way up to the honest chain tip, and surpass it. All miners will then move over to Tom and Lisa’s branch. Note that they still can’t steal anyone’s money in the blockchain, but they can make as many double spends as they want.
Let’s play with the idea that Tom and Lisa start double spending. For example, they buy the cafe and double spend the transaction so they end up with both the cafe and 100,000 CT. Every now and then, people will notice that the blockchain history has changed. Six confirmation transactions used to be reliable, but now they can’t be trusted. What will happen to the cookie token value if the blockchain becomes less reliable? And what happens to the value of cookie tokens when people hear about the double-spend attacks going on?
Panic! People don’t want anything to do with this unreliable, insecure cookie token system anymore. Many people will sell all their cookie tokens on the cookie token marketplace outside the cafe. The problem is that there aren’t many buyers. What happens to the dollar price of cookie tokens when the demand is low and supply is high? The price tanks.
What happens when the price tanks? More panic! More people want to sell, leading to even bigger price drops.
Tom’s, Lisa’s, and all other miners’ mining business becomes less profitable because the value of their block rewards is so low that they can’t sell their cookie tokens to get enough dollars to pay their electricity bill. They need to shut down their mining business because they mine at a net loss.
Tom and Lisa should think twice before starting to attack the system, even though they can. Just the fact that two miners together control more than 50% of the total hashrate could be enough to trigger a price drop because people get nervous about mining centralization—when a few people control a large portion of the total hashrate.
Mitigating miner centralization
What can people do to counter Tom and Lisa’s power? They can start mining at home. Let’s say five more people join the mining business, and each adds a computer with 150 Mhash/s. We now have a whole new situation (Figure 174).
The total hashrate increases from 550 Mhash/s to 1,300 Mhash/s. The biggest miner, Lisa, with 200 Mhash/s, now has only about 15% of the total hashrate. At least five miners must collude to control a majority of the hashrate because the biggest four miners control 49.9%.
The incentives for people to start mining are strong. They have cookie tokens, and they want the system strong to protect their money from panic price drops due to miner centralization.
Note that as more miners join the race, the rewards per miner will decrease. At some point, some miner—probably an inefficient one—will find that mining isn’t worth it anymore and close down its mining computers. The market will push out the inefficient miners in favor of the efficient ones.
7.7. Transaction fees
The system now in place has multiple miners that produce blocks independently of each other. This is a massive gain in censorship resistance. All miners must collude to hinder transactions from entering the blockchain. A single miner or a portion of the miners will only be able to make a transaction take longer to confirm, but eventually, one of the noncensoring miners will find a valid proof of work for a block that contains the transaction and publish that block.
All good. But there are two problems:
-
Bigger blocks are slower.
-
Block size is limited.
These two properties have some implications on miners’ transaction selection. Let’s start with the first of these two problems and then discuss what effect the block-size limit will have.
7.7.1. Bigger blocks are slower
Suppose Lisa and Tom find valid proof of work for their respective blocks at the same time. Lisa’s block is 200 KB and contains 400 transactions, whereas Tom’s block is 100 KB and contains 200 transactions. They both want their own block to become part of the strongest chain, but only one of them can take that place. They start uploading their respective blocks to the shared folder at the exact same time (Figure 175).
Tom’s block is smaller than Lisa’s. This means Tom will upload his block to the shared folder faster than Lisa uploads hers. It will also be faster for Qi to download Tom’s block than it will be to download Lisa’s block. Finally, Qi has to verify blocks she downloads before building on them. A smaller block will typically be faster to verify than a big block, so Tom’s block is also faster to verify than Lisa’s block.
The result is that Qi will, at time T, select Tom’s block as the current best chain tip and start mining on top of Tom’s block. Lisa’s block doesn’t really exist for Qi at time T because Qi hasn’t verified it yet. She’s still downloading Lisa’s block from the shared folder. When Qi finally verifies Lisa’s block at time L, Qi has already decided to go for Tom’s block, and Lisa’s block will be stored in case of future chain reorganizations.
Miners have a clear incentive to keep their blocks small. For each extra transaction they add to their blocks, they lose a little competitiveness in the block race.
7.7.2. But wasn’t this about transaction fees?
This is where transaction fees come in. If the miner could get paid a little extra for each transaction it adds to its block, that would compensate for the loss of competitiveness.
People making payments are keen on having their transactions confirmed in the blockchain. Wouldn’t it be great if John could reserve a little money in his transaction for the miner who includes it? This way, the payer could compensate the miner for the loss of competitiveness.
If you use the transactions a little differently, you can offer this feature. Let’s say John wants to buy a cookie. To give miners an incentive to include his transaction, he decides to add a transaction fee. He constructs his transaction as shown in Figure 176.
When John created a similar transaction in Chapter 5, the sum of the inputs was equal to the sum of the outputs. He didn’t pay a transaction fee.
This time, John wants to add a small transaction fee to his transaction. He spends two inputs, totaling 13 CT, and adds an output of 10 CT to the cafe and a change output of 2.5 CT to himself. He then signs the transaction just as he always does and sends it to all the miners.
Lisa, the miner, receives this transaction from John. She notices that there is a transaction fee of 0.5 CT in it. She wants that fee and decides the transaction fee compensates more than enough for the small incremental risk of losing the block race due to including the transaction.
John can tune the incentive for miners to include his transaction. If it’s important to him that the transaction be confirmed in one of the next few blocks, he should pay a relatively high fee. If there’s no hurry, he can pay a low fee, but he needs to be cautious. If he pays too small a fee, no miner will be willing to confirm his transaction.
We’ll talk more about fees, and how you can change a transaction’s fee if it gets stuck pending—also known as fee bumping—in Chapter 9.
For Lisa, when she’s deciding whether to include a transaction, all that matters is how big the transaction is and what fee it pays. Basically, it’s the fee per byte she’s interested in. John’s transaction is about 400 bytes and pays a 0.5 CT fee. That’s 0.00125 CT/byte. This is a simple calculation for Lisa to do, and she does the same for all transactions. If the fee per byte is above a certain threshold, she’ll include the transaction.
She can select transactions however she wants, as described in Section <IP_ADDRESS>. For example, she can include her own transaction without any fee, or she can drop all transactions that pay for cookies no matter how high the fee is. And that’s OK. Other miners will have different strategies for selecting transactions. Most will probably make decisions based only on fee per byte.
How does Lisa collect this fee? By using her coinbase transaction (Figure 177).
Lisa sums up all transaction fees from the transactions in her block and increases the coinbase output with this amount. The amount in the coinbase output—the block reward—is the sum of the block subsidy, the 50 new cookie tokens this block creates, and all transaction fees from the transactions in the block. Note that we’ve widened the term block reward to include both the block subsidy (newly created money) and the transaction fees.
When the block is set up correctly, Lisa starts working to find a valid proof of work for this block.
7.7.3. Block size is limited
Blocks aren’t allowed to be infinitely large. Simply put, the maximum block size is 1,000,000 bytes, but we’ll discuss some nuances of this in Section 10.5.1. If more transactions are waiting to be confirmed than there is block space available, miners have to choose which transactions to include in the block and which to exclude.
The transaction fee plays an important role in this situation because a higher transaction fee gives miners more incentive to include the transaction in a block instead of other transactions. The fee is used to compete against other transactions for block space, in addition to compensating for the lost competitiveness. This situation is known as a fee market (Figure 178).
If more block space is available than there is transaction data waiting to be confirmed, transactions don’t compete with each other in the same sense (Figure 179).
In this situation, any transaction that bears the cost of lost competitiveness will be confirmed.
As of this writing, fee markets emerge from time to time during spikes of interest in Bitcoin. But there are still moments with few to no waiting transactions, in which case the fee is low, typically 1 satoshi/byte, or 0.000,000,01 BTC/byte.
7.7.4. When the block subsidy is 0
As we discussed in Chapter 2, the block subsidy will be halved roughly every four years. At some point, the block subsidy won’t be big enough on its own to give miners incentive to mine. If the value of the block reward is smaller than the electricity bill, what’s the point in mining?
Transaction fees will play a bigger and bigger role for miners as the block subsidy decreases. The typical miner wants the income from mining to at least cover their electricity bill (Figure 180).
Note that the value of the block subsidy might not always decrease over time. Table 23 shows some examples.
| Block subsidy | Value of 1 CT | Value of block subsidy |
|---|---|---|
50 CT |
$0.10 |
$5 |
25 CT |
$0.25 |
$6.25 |
This shows that the block subsidy by itself isn’t a measurement of mining income. We have to look at the value of the block subsidy and the value of the transaction fees. One thing is for sure: when the subsidy is zero, the value of the subsidy is also zero. At some point, the block subsidy isn’t incentive enough to mine.
When this happens, transaction fees will help give efficient miners revenue. If John wants his transaction confirmed, he must pay a fee big enough that one or more miners are willing to include his transaction. This is a market for block space at play.
We can only speculate about where fee levels will be in the future. Some people argue that Bitcoin’s fees are already too high for how they want to use Bitcoin today. As transaction fees go up, some current use cases for Bitcoin—for example, payments with tiny amounts—will have to find other ways to work. New systems are being developed on top of Bitcoin that enable people to lump a nearly infinite number of payments together into just one or two transactions. One such system, the Lightning Network, is of particular interest. If a million payments can be made with a single Bitcoin transaction, all those user transactions can share the cost of the transaction fee.
7.8. Recap
This chapter has solved the problem with censorship. Lisa had absolute power over what transactions to include in the blockchain. You solved this by having multiple “Lisas,” or miners. By doing so, wallets can send their transactions to any or all miners, and hopefully some of the miners will process the transactions.
The miners compete to produce the next block in the blockchain. They compete to be the first to find a valid proof of work for their block.
The miner that wins the competition will publish its block and collect the block reward, which consists of the block subsidy and the transaction fees. The reward is collected in the coinbase transaction.
The block subsidy is used to fairly get new money into circulation in the economy until all 21,000,000 new cookie tokens are minted. The sender of a transaction adds a transaction fee to incentivize miners to include the transaction in their blocks.
This competition will lead to natural splits, when two miners find a block at about the same time. These splits will eventually be resolved.
The resolution is affected by which branch miners choose to mine on. Miners usually mine on the first valid block they see.
A merchant shouldn’t trust a high-value transaction until a sufficiently high number of blocks have been mined on top of the block containing the transaction. This reduces the risk of double spends.
It can be expensive for a miner to try a double spend. If it fails, the miner will have spent a lot of electricity and lost all its block rewards. The choice of the number of required confirmations is up to the merchant and should take into account the transaction value.
7.8.1. System changes
Proof of work replaces the block signatures introduced in Chapter 6, and we can remove them from the concept mapping table (Table 24).
| Cookie tokens | Bitcoin | Covered in |
|---|---|---|
1 cookie token |
1 bitcoin |
|
Lisa |
A miner |
|
Block signature |
Proof of work |
|
The shared folder |
The Bitcoin network |
Lisa is now doing the exact same tasks as a Bitcoin miner, which is why we remove Lisa from the table as well. The shared folder will be the last bit of the cookie token system we’ll take care of. That’s for the next chapter.
It’s time to release a shiny new version of the cookie token system (Table 25).
| Version | Feature | How |
|---|---|---|
7.0 |
Censorship-resistant |
Multiple miners, “Lisas,” enabled by proof of work |
Anyone can join the mining race |
Automatic difficulty adjustments |
|
6.0 |
Prevent Lisa from deleting transactions |
Signed blocks in a blockchain |
Fully validating nodes |
Download and verify the entire blockchain |
|
Lightweight wallet saves data traffic |
Bloom filters and merkle proofs |
|
5.0 |
Spend multiple “coins” in one payment |
Multiple inputs in transactions |
Anyone can verify the spreadsheet |
Make the signatures publicly available in the transactions |
|
Sender decides criteria for spending the money |
Script programs inside transactions |
7.9. Exercises
7.9.1. Warm up
-
In what way was Lisa a central authority in Chapter 6?
-
Why would the possibility of censoring transactions decrease with multiple miners, or “Lisas”?
-
Drawing random numbers worked quite well, but we abandoned this idea. Why was the idea naive?
-
How do you check if a proof of work is valid?
-
How does a miner generate a valid proof of work?
-
What is meant by strongest chain?
-
What does it mean when a miner has the hashrate 100 Mhash/s?
-
A retarget period has just ended, and the last 2,016 blocks took 15 days to produce. Will the target increase or decrease?
-
At what percentage of the hashrate can you be certain to pull off a double spend, if you’re willing to try indefinitely?
7.9.2. Dig in
-
Suppose a big block and a small block are created at the same time. Why is the big block less likely to become part of the strongest chain compared to the small block?
-
Suppose the block rate suddenly doubles exactly in the middle of a retarget period. It goes from 6 blocks per hour to 12 blocks per hour, on average. No other changes happen during the retarget period. What will happen to the target after this period?
-
Suppose Selma has 52% of the total hashrate. She decides to change the retarget period of her software program from 2,016 blocks (two weeks) to 144 blocks (one day). No one else thinks this is a good idea, and they keep running the old software. What will happen after her next retarget period of one day when she adjusts her target? Will the rest of the miners and full nodes accept Selma’s blocks? Who will suffer from this situation?
-
Why would a miner choose not to confirm a transaction that pays a very small transaction fee?
7.10. Summary
-
Having multiple miners avoids a central authority that can censor transactions.
-
Proof of work is used to select who gets to create a block.
-
Proof of work enables anyone to start mining without asking for permission.
-
The target is automatically calibrated every 2,016 blocks to keep money creation at the predetermined rate.
-
A transaction fee gives miners incentive to include the transaction in their block.
-
To keep the risk of double spends low, the recipient of cookie tokens, or bitcoins, selects how many confirmations are needed.
-
A miner gets as much in block rewards as it deserves. The more hashrate it puts into the system, the bigger share of the rewards it gets.
-
The stronger a chain is—the more accumulated proof of work it has—the harder it is to rewrite that chain.
8. Peer-to-peer network
This chapter covers
-
Removing the last central authority: the shared folder
-
Following a transaction in the peer-to-peer network
-
Leaving behind the silly cookie tokens
-
Bootstrapping the peer-to-peer network
Let’s talk about the elephant in the room: the shared folder. All blocks the miners produce must pass through the shared folder on their way to other full nodes and miners. This chapter will remove the central shared folder and replace it with a decentralized peer-to-peer network (Figure 181). The peer-to-peer network lets full nodes (including miners) send blocks directly to each other. When nodes can talk directly to each other, we no longer need a central point of authority for communication.
Another issue we haven’t talked much about is how wallets send transactions via email to the miners. When a new miner joins the system, all wallets need to update their miner list. Not cool. With this nice peer-to-peer network of nodes, wallets can broadcast their transactions to all miners without knowing who or where they are.
We’ll follow a transaction’s path through the network, both as an unconfirmed transaction and, eventually, as part of a mined block. The transaction will start in John’s wallet and end as a confirmed transaction in the blockchain with Bob’s wallet being notified about it.
After following the transaction through the system, you’ll no longer need the cookie token system to help you understand Bitcoin. We’ll talk only about Bitcoin from that point forward. Practically no differences exist between the cookie token system and Bitcoin anymore, so it doesn’t make sense to keep talking about cookie tokens when, in fact, you want to learn about Bitcoin!
The last topic in this chapter will cover how a new node connects to and becomes part of the peer-to-peer network. This is far from trivial. How does it find nodes to connect to? How does it download the blockchain up to the latest block? We’ll sort all that out. Toward the end of the chapter, you’ll learn how to set up a full node of your own.
8.1. The shared folder
The shared folder administrator, Luke, is a central authority (Figure 182). He ultimately gets to decide which blocks can be stored in the shared folder. He also gets to decide who can read from and write to the shared folder.
So far, we’ve assumed Luke is a totally neutral good guy—but what if he isn’t, or what if he’s forced by Acme Insurances to reject certain blocks? What’s the point of proof of work if the system can be censored at the block level? Proof of work made the transactions censorship-resistant because it let users send their transactions to multiple miners. But the blocks containing the transactions can still be censored by whoever has administrator privileges over the shared folder. Simply put, the system isn’t yet censorship-resistant. As long as a single entity can decide which blocks or transactions to allow, the system isn’t censorship-resistant.
The shared folder poses yet another problem. Imagine that Rashid has created a 1 MB block and published it to the shared folder. Everyone watching the shared folder, all full nodes, will download Rashid’s block at the same time. If you have 100 full nodes, the total amount of data you need to send from the shared folder to the different nodes is 100 MB. This will cause block propagation—the transfer of a block from its creator to all other nodes—to be terribly slow. The more nodes, the slower the block propagation.
8.2. Let’s build a peer-to-peer network
What if the full nodes and miners could talk directly to each other instead of relying on the central shared folder? They could send the blocks directly to one another in a peer-to-peer network (Figure 183).
Think of the peer-to-peer network as a large number of people. One person doesn’t know everyone else, but might know three people. When something interesting happens—for example, Rashid finds a block—he tells his three friends about it, who in turn tell all their friends, and so on until everybody knows about this new block. We call such networks gossip networks for apparent reasons.
Blocks can no longer be easily stopped. A node can choose not to pass a block on, or relay it, to its peers, but the peers are connected to several other peers that will gladly relay the block to them. A single node can’t do much to censor information.
Suppose Rashid finds a block, and he wants to get this block out to all nodes. Rashid sends his block to Qi, Tom, and the cafe. For some reason, the cafe doesn’t forward the block to Lisa (Figure 184). But Lisa has several peers in this network. She’s connected to Tom and Qi. Tom will tell Lisa about this new block and send it to her. The cafe can’t hide information from Lisa as long as she’s well-connected—that is, has many different peers.
Now that you have this nice network, wallets can use it to get their transactions sent to miners. Then they won’t have to keep track of miner email addresses anymore. The transactions will be broadcast over the peer-to-peer network and reach all full nodes within seconds. This includes the miners, because they’re also full nodes. We covered this briefly in Chapter 1, as repeated in Figure 185.
The same thing goes here as for blocks: a single node can’t hinder transactions from spreading across the network. Another pleasant effect of using the peer-to-peer network for transactions is that a transaction’s recipient can be notified that the transaction is pending, or is about to be confirmed. We’ll look at how this works a bit later.
8.3. How do peers talk?
Let’s look at how the communication between two peers happens. We’ll look specifically at how Tom connects to Lisa and how they communicate across their communication channel, called a Transmission Control Protocol (TCP) connection (Figure 186).
Suppose Tom’s node knows about Lisa’s node. I’ll explain in Section 8.7 how Tom learns about other nodes. For now, let’s assume he has the IP address and port of Lisa’s node. He now wants to connect to Lisa’s node to communicate with it. All computers on the internet have an Internet Protocol (IP) address, which is how one computer can send information to another. A computer program that listens for incoming connections must listen on a specific port number of its computer’s IP address. Lisa’s computer has the IP address <IP_ADDRESS> and runs a cookie token program that listens for incoming connections on port 8333.
Tom’s node connects to Lisa’s node through the IP address <IP_ADDRESS> and TCP port 8333. His node (computer program) starts by asking its operating system (OS) to initiate a connection to Lisa (Figure 187). The OS sends a message to Lisa’s computer saying that Tom wants to talk to a computer program on Lisa’s port 8333. Her computer knows a program is listening on port 8333, so it sends back a “Sure, welcome” message. Tom’s computer acknowledges this by sending back an “OK, cool. Let’s talk …” message.
The node software on Tom’s and Lisa’s computers wasn’t involved in this exchange—it was carried out by their OSs, such as Linux, Windows, or macOS. When the message sequence is finished, the OS hands the connection over to the node software. Lisa’s and Tom’s nodes can now speak freely to each other. Tom can send data to Lisa, and Lisa can send data to Tom over this communication channel, or TCP connection.
8.4. The network protocol
Tom and Lisa can now send and receive data over a communication channel. But if Tom’s node speaks a language that Lisa’s node doesn’t understand, the communication won’t be meaningful (Figure 188). The nodes must have a common language: a protocol.
The cookie token network protocol defines a set of message types that
are allowed. A typical message in the cookie token (well, Bitcoin)
network is the inv
message (Figure 189).
A node uses the inv
—short for inventory—message to inform other
nodes about something it has. In Figure 189, Tom’s node informs
Lisa’s node that Tom has three things to offer Lisa: two transactions
and a block. The message contains an ID for each of these items.
8.4.1. John sends the transaction
Let’s follow a transaction through the network from start to end to see what network messages are being used. We’ll assume the peer-to-peer network is already set up. We’ll come back to how the network is bootstrapped later in this chapter.
In Section 6.3, we said that wallets can connect to full nodes and get information about all block headers and transactions concerning them using bloom filters and merkle proofs (Figure 190).
I didn’t go into detail then about how this communication works. It uses the same protocol the nodes use when they communicate with each other. The wallets and the full nodes (including miners) all speak the same “language.”
Suppose John wants to buy a cookie from the cafe. John’s wallet is connected to Tom’s node with a TCP connection. He scans the payment URI from the cafe’s wallet. John’s wallet creates and signs a transaction. You know the drill. Then it’s time to send the transaction to Tom’s node (Figure 191).
This happens in a three-step process. John’s wallet doesn’t just send the transaction unsolicited: it first informs Tom’s node that there’s a transaction to be fetched (Figure 192).
The first message is an inv
message, as described in the previous
section. John’s wallet sends the inv
to Tom’s full node. Tom checks if
he already has the transaction. He doesn’t, because John’s wallet just
created it and hasn’t sent it to anyone yet. Tom’s node wants to get
this transaction, so he requests it with a getdata
message that looks
just like an inv
message but with a different meaning: getdata
means “I want this stuff,” whereas inv
means “I have this stuff.”
John’s wallet receives the getdata
message and sends a tx
message
containing the entire transaction to Tom’s node. Tom will verify the
transaction and keep it. He’ll also relay this transaction to his
network neighbors.
You might ask, “Why doesn’t John’s wallet send the entire transaction
immediately? Why go through the hassle with inv
and getdata
?” This
will become clear later, but it’s because nodes might already have the
transaction; we save bandwidth by sending only transaction hashes
instead of entire transactions.
8.4.2. Tom forwards the transaction
If the transaction is valid, Tom’s node will inform his neighbors about
it (Figure 193) using an inv
message, just like John’s wallet did
when it informed Tom’s node about the transaction.
The process is the same for these three message exchanges as the one
John used when he first sent the transaction to Tom (Figure 194). Lisa,
Qi, and Rashid will get an inv
message from Tom.
When Lisa, Qi, and Rashid have received the transaction, they too will inform their peers about it after they’ve verified it. Qi’s and Rashid’s nodes are a bit slower, so it takes them a while to verify the transaction; we’ll get back to them later.
Lisa was quick to verify the transaction, so she’ll be the first of the
three to relay it. She already knows that she received the transaction
from Tom, so she won’t inform Tom’s node with an inv
message. But Lisa
doesn’t know that Qi already has the transaction, and she doesn’t know
if the cafe has it. She’ll send an inv
to those two nodes. The cafe’s
node will send back a getdata
because it hasn’t yet seen this
transaction. Qi’s node already has this transaction and won’t reply with
anything (Figure 195). She’ll remember that Lisa has it, though.
inv
to Qi’s node, but Qi’s node already has the transaction.Qi has just finished verifying the transaction. She knows that Lisa’s
node has it, so she doesn’t have to send an inv
to Lisa’s node. But
she doesn’t know if Rashid has it, so she sends an inv
to Rashid’s
node.
Rashid’s was the slowest node when verifying John’s transaction, so when
it’s time for him to send an inv
to his neighbors, he’s already
received an inv
from Qi’s node. And he also knows from earlier that
Tom already has the transaction. He’ll just send an inv
to the cafe’s
node, which will ignore the inv
because it already has the
transaction.
8.4.3. The cafe’s lightweight wallet is notified
I said earlier that a good thing about letting transactions travel the peer-to-peer network is that the recipient wallet can get a quick notification of the pending transaction. Let’s explore this now.
The cafe’s full node has received the transaction and verified it. The cafe also has a lightweight wallet on a mobile phone that it uses to send and receive money. The cafe is concerned with security, so it configured this lightweight wallet to connect only the cafe’s own full node, its trusted node (Figure 196).
This common setup gives the cafe the complete security of a full node combined with the flexibility and mobility of a lightweight wallet. I described this setup in Section 6.6.
The cafe’s full node has just verified John’s transaction. It now wants
to inform its neighbors about this new transaction. It’s connected to
Lisa’s node, Rashid’s node, and the cafe’s lightweight wallet. The full
node already knows that Lisa’s and Rashid’s nodes have this transaction,
so it doesn’t send an inv
to those two nodes. The full node doesn’t
know whether the wallet has the transaction, but it won’t immediately
send an inv
message to the wallet.
The wallet is a lightweight wallet, which uses bloom filters,
described in Section 6.3.1. The full node will
test the transaction against the bloom filter and, if it matches, send
an inv
message to the wallet. If there’s no match, it won’t send an
inv
message.
John’s transaction is for the cafe, so the bloom filter will match the
transaction, and the full node will send an inv
. The wallet will
request the actual transaction using getdata
, as Figure 197 shows.
The wallet has now received the transaction. It can show a message to the cafe owner that a transaction is pending. The cafe owner has a choice: trust that the transaction—a so-called 0-conf transaction—will be confirmed eventually, or wait until the transaction is confirmed. If the cafe accepts the 0-conf transaction, then it trusts that John has paid a high enough transaction fee and that the transaction won’t be double spent.
This time, the cafe decides that it needs to wait until the transaction is included in a valid block. This brings us to the next phase: including the transaction in a block in the blockchain.
8.4.4. Including the transaction in a block
Let’s recall some of the miners in this system. At the end of Section <IP_ADDRESS>, there were 10 different miners; but let’s go back in time and pretend Qi, Tom, Lisa, and Rashid are the only miners in this system right now.
The transaction reached all these miners during transaction propagation. John’s wallet used to send the transaction via email to all miners. Now, he sends it to any of the full nodes, and it propagates across the entire peer-to-peer network. Miners can choose to include John’s transaction in the blocks they’re mining. Suppose the transaction includes a transaction fee so that some or all miners are willing to include it, and that Rashid is the next miner to find a valid proof of work for his block, which happens to contain John’s transaction (Figure 198).
Rashid wants to get his block to the other miners as quickly as possible to minimize the risk of some other miner getting a block out before Rashid’s block.
He creates a headers
message and sends it to all his peers: Tom, the
cafe, and Qi. Rashid’s peers will send back a getdata
message, and
Rashid will reply with the actual block. The message exchange between
Rashid and Qi will look like the one in Figure 199.
The actual block is sent in a block
message containing the full block.
Let’s continue the block propagation throughout the peer-to-peer
network. Rashid has sent his block to Tom, the cafe, and Qi. Now, these
three nodes will verify the block and, if it’s valid, send out headers
messages to all their peers who might not already have it (Figure 200).
Qi and Tom happen to send their headers
messages to each other at the
same time. This isn’t a problem; because they both have the block,
they’ll ignore the headers
received from peers. Lisa will request the
block from one of her peers just like Qi requested the block from
Rashid.
headers
messages.This concludes the propagation of this block—almost. The lightweight wallets need to be informed about the block.
8.4.5. Notifying wallets
Tom’s node is connected to John’s wallet, so Tom sends a headers
message to John. Likewise, the cafe’s full node sends a headers
message to the cafe’s lightweight wallet. Tom’s and the cafe’s full
nodes won’t test the block against the bloom filters in any way. They
will send the headers
message unconditionally, but the lightweight
wallets won’t request the full blocks.
As you might recall from Chapter 6, lightweight wallets don’t download
the full blocks. Most of the time, John’s wallet is only interested in
the block headers so it can verify the blockchain’s proof of work. But
every now and then, transactions that are relevant to John’s wallet
are in the blocks, and the wallet wants proof that those transactions
are included in the block. To find out if there are any relevant
transactions, he sends a getdata
message to Tom, requesting a
merkleblock
message for the block.
John gets a merkleblock
message containing the block header and a
partial merkle tree connecting his transaction ID (txid) to the merkle
root in the block header (Figure 201).
merkleblock
containing a merkle proof that John’s transaction is in the block.Figure 202 gives a little repetition from Chapter 6.
merkleblock
message contains a block header and a partial merkle tree.John’s wallet will verify that
-
The block header is correct and has a valid proof of work.
-
The merkle root in the header can be reconstructed using the partial merkle tree.
-
The txid of John’s transaction is included in the partial merkle tree. He doesn’t care about the irrelevant transaction that’s used to obfuscate what belongs to John.
John’s wallet is now sure his transaction is contained in the new block. The wallet can display a message to John saying, “Your transaction has 1 confirmation.”
The cafe’s lightweight wallet will be notified the same way.
Because the cafe’s wallet uses a trusted node, privacy isn’t much of an
issue (Figure 203). The wallet can use a big bloom filter to reduce the
number of irrelevant transactions, which in turn will reduce mobile data
traffic. The sparser the bloom filter, the less extra obfuscation
traffic will be sent to the wallet.
The cafe’s owner feels comfortable handing the cookie over to John now. John eats his cookie. The deal is done.
8.4.6. More confirmations
As time passes, more blocks will be mined by the miners. These blocks will all propagate the network and end up on every full node. The lightweight wallets will get merkle blocks to save bandwidth.
For each new block coming in, John’s transaction will be buried under more and more proof of work (Figure 204). This makes John’s transaction harder and harder to double spend. For each new block, the transaction will get one more confirmation.
8.5. Leaving the cookie token system
I don’t think the cookie token system will add any more to your understanding of Bitcoin. It’s time to let go of the cookie tokens and start talking solely about Bitcoin. We’ve developed the cookie token system to a point where there are no differences from Bitcoin. Table 26 shows the concept mapping table.
| Cookie tokens | Bitcoin | Covered in |
|---|---|---|
1 cookie token |
1 bitcoin |
|
The shared folder |
The Bitcoin network |
The last cookie token concept that differs from Bitcoin, the shared folder, has been eliminated. Let’s look at how it all happened, in Figure 205.
We’ll keep our friends at the office a while longer. John will probably have to buy a few more cookies, but he’ll use Bitcoin to do it.
8.5.1. Bitcoin at a glance
The Bitcoin peer-to-peer network is huge. As of this writing:
-
There are about 10,000 publicly accessible full nodes.
-
Bitcoin’s money supply is about 17,400,000 BTC.
-
Each bitcoin is worth around $6,500.
-
Bitcoin processes about 250,000 transactions per day.
-
An estimate of 100,000 BTC, valued at $630 million, is moved daily.
-
The total mining hashrate is about 50 Ehash/s, or 50 × 1018 hash/s.
A typical desktop computer can do about 25 Mhash/s. -
The transaction fees paid each day total around 17 BTC. This averages to 6,800 satoshis per transaction, or about $0.40 per transaction.
-
People in all corners of the world use Bitcoin to get around problems in their day-to-day lives.
8.6. Where were we?
This chapter is about Bitcoin’s peer-to-peer network. The first half of the chapter described the network in action after it’s been set up, as illustrated by Figure 206, repeated from Chapter 1.
The second half of this chapter will look at how a new node joins the network.
8.7. Bootstrapping the network
The scenario in Section 8.4 assumed that all nodes involved were already connected to each other. But how does a new node start? How would it find other nodes to connect to? How would it download the full blockchain from the genesis block, block 0, up to the latest block? How does it know what the latest block is?
Let’s sort it out.
Suppose Selma wants to start her own full node. This is how it would typically happen (Figure 207):
-
Selma downloads, verifies, and starts the full node computer program.
-
The computer program connects to some nodes.
-
Selma’s node downloads blocks from her peers.
-
Selma’s node enters a normal mode of operation.
8.7.1. Step 1—Run the software
Selma needs a computer program to run a full node. The most commonly used such program is Bitcoin Core. Several others are available, such as libbitcoin, bcoin, bitcoinj, and btcd. We’ll focus only on Bitcoin Core, but you’re encouraged to explore the others yourself.
To download Bitcoin Core, Selma visits its web page, https://bitcoincore.org, and finds a download link there. But she encounters a potential problem: Selma isn’t sure the program she downloads is actually the version the developers behind Bitcoin Core released. Someone could have fooled Selma into downloading the program from bitconcore.org instead of bitcoincore.org, or someone might have hacked bitcoincore.org and replaced the downloadable files with alternative programs.
The Bitcoin Core team therefore signs all released versions of the program with a private key—let’s call it the Bitcoin Core key. They provide the signature in a downloadable file, named SHA256SUMS.asc. This file contains the hash value of the released Bitcoin Core software and a signature that signs the contents of the SHA256SUMS.asc file (Figure 208).
Selma has downloaded both the program, in a file called bitcoin-0.17.0-x86_64-linux-gnu.tar.gz, and the signature file, SHA256SUMS.asc. To verify that the program is in fact signed by the Bitcoin Core private key, she needs to know the corresponding public key. But how can she know what this key is?
This is a hard problem. Remember when Lisa used to sign blocks with her private key? How would the full nodes verify that the blocks were actually signed by Lisa? They used multiple sources to fetch Lisa’s public key—for example, looking at the bulletin board at the entrance of the office, checking the company’s intranet, and asking colleagues. The same applies here; you shouldn’t trust a single source, but should use at least two different sources. The key that’s currently being used to sign Bitcoin Core releases is named
Wladimir J. van der Laan (Bitcoin Core binary release signing key<EMAIL_ADDRESS>and has the following 160-bit SHA1 hash, called fingerprint:
01EA 5486 DE18 A882 D4C2 6845 90C8 019E 36C2 E964
This book can serve as one of Selma’s sources. She decides to
-
Get the fingerprint of the key from https://bitcoincore.org.
-
Verify the fingerprint with the Grokking Bitcoin book.
-
Verify the fingerprint with a friend.
The fingerprints from the three sources match, so Selma downloads the public key from a key server. A key server is a computer on the internet that provides a repository of keys. Key servers are commonly used to download keys identified by the key’s fingerprint. Selma doesn’t trust the key server, so she needs to verify that the fingerprint of the downloaded key matches the expected fingerprint, which it does.
Now, when she has the Bitcoin Core public key, she can verify the signature of the SHA256SUMS.asc file (Figure 209).
She uses the Bitcoin Core public key to verify the signature in the signature file. She must also verify that the program has the same hash value as stated in SHA256SUMS.asc. The signature is valid, and the hashes match, which means Selma can be sure the software she’s about to run is authentic.
Selma starts the program on her computer.
8.7.2. Step 2—Connect to nodes
When Selma’s full node program starts, it isn’t connected to any other nodes. She’s not part of the Bitcoin network yet. In this step, the node will try to find peers to connect to.
To connect to a peer, the full node needs the IP address and the TCP port for that peer. For example:
IP: <IP_ADDRESS> port: 8333
An IP address and port are often written as
<IP_ADDRESS>:8333
Finding initial peers
Where does Selma’s node find initial addresses of other peers? Several sources are available (Figure 210):
-
Configure the full node with custom peer addresses. Selma can get an address by asking a friend who’s running a full node.
-
Use the Domain Name System (DNS) to look up initial peer addresses to connect to.
-
Use hardcoded peer addresses in the full node program.
Selma’s node shouldn’t initially connect to just one node. If that single node is malicious, she’d have no way of knowing it. If you connect to multiple nodes initially, you can verify that they all send data consistent with each other. If not, one or more nodes are deliberately lying to you, or they themselves have been fooled.
The default way of finding initial node addresses is to look them up in the DNS system. DNS is a global name lookup system, used to look up IP numbers from computer names. For example, when you visit https://bitcoin.org with your web browser, it will use DNS to look up the IP number of the name bitcoin.org. The Bitcoin Core software does the same. Names to look up are hardcoded into Bitcoin Core, just like the hardcoded IP addresses and ports. Several DNS seeds are coded into the software. A lookup of a DNS seed can return several IP addresses, and every new lookup might return a different set of IP addresses. The final, third option is used as a last resort.
Note from Figure 210 that DNS lookups don’t return port numbers. The other two methods of finding initial peers usually include one, but the DNS response can return only IP addresses. The nodes on these IP addresses are assumed to listen on the default port that Bitcoin Core listens on, which is 8333.
Handshaking
Suppose Selma’s node chooses to connect to Qi’s node, <IP_ADDRESS>:4567, and to Rashid’s node, <IP_ADDRESS>:8333. Selma sets up a TCP connection to each of the two nodes and sends an initial message to both of them on the new TCP connections. Let’s look at how she talks to Qi’s node (Figure 211).
version
message with Qi.The exchange, called a handshake, starts with Selma, who sends a
version
message to Qi. The handshake is used to agree on which
protocol version to use and tell each other what block heights they
have. The version
message contains a lot of information not shown in
the figure, but the most essential stuff is there:
- Protocol version
-
The version of the network protocol, or “language,” that peers use to talk to each other. Selma and Qi will use version 70012 because that’s the highest version Qi will understand. Selma knows all protocol versions up to her own.
- User agent
-
This is shown as “software identification” in the figure because “user agent” is a bit cryptic. It’s used to hint to the other node what software you’re running, but it can be anything.
- Height
-
This is the height of the tip of the best chain the node has.
Other useful information in the version
message includes
- Services
-
A list of features this node supports, such as bloom filtering used by lightweight clients.
- My address
-
The IP address and port of the node sending the
version
message. Without it, Qi wouldn’t know what address to connect to if she restarts and wants to reconnect to Selma’s node.
When Qi’s node receives Selma’s version
message, Qi will reply with
her own version
message. She’ll also send a verack
message
immediately after the version
message. The verack
doesn’t contain
any information; rather, it’s used to acknowledge to Selma that Qi has
received the version
message.
As soon as Selma’s node receives Qi’s version
message, it will reply
with a verack
message back to Qi’s node. The handshake is done. Selma
also goes through the same procedure with Rashid’s node.
Finding peers’ peers
When Selma’s node is connected to Rashid’s node, it will ask that node for other peer addresses to connect to. This way, Selma will be able to expand her set of peers (Figure 212).
Selma is only connected to two peers: Qi’s node and Rashid’s node. But she thinks she needs more nodes to connect to. Being connected to only two nodes has some implications:
-
Qi and Rashid can collude to hide transactions and blocks from Selma.
-
Qi’s node may break, leaving Selma with only Rashid’s node. Rashid can then singlehandedly hide information from Selma.
-
Both Qi’s and Rashid’s nodes may break, in which case Selma will be completely disconnected from the network until she connects to some other nodes via the initial peer-lookup mechanisms.
Figure 213 shows how Selma asks Rashid for more peer addresses to connect to.
Selma sends a getaddr
message to a peer, Rashid’s node. Rashid
responds with a set of IP addresses and TCP ports that Selma can use to
connect to more peers. Rashid chooses which addresses to send to Selma,
but it’s usually the addresses to which Rashid is already connected and
possibly some that Rashid collected from his peers but didn’t use
himself.
Selma will connect to any number of the received addresses to increase her connectivity. The more peers you’re connected to, the better your connectivity. A high degree of connectivity decreases the risk of missing out on information due to misbehaving peers. Also, information propagates more quickly if nodes have higher connectivity. A typical full node in Bitcoin has about 100 active connections at the same time. Only eight (by default) of those are outbound connections, meaning connections initiated by that node. The rest are inbound connections initiated by other nodes. Consequently, a full node that isn’t reachable on port 8333 from the internet—for example, due to a firewall—won’t get more than eight connections in total.
8.7.3. Step 3—Synchronize
Now that Selma is well-connected to, and part of, the Bitcoin network, it’s time for her to download and verify the full blockchain up to the latest block available. This process is called synchronization, sync, or initial blockchain download.
Selma has only a single block: the genesis block. The genesis block is hardcoded in the Bitcoin Core software, so all nodes have this block when they start.
She needs to download all historic blocks from her peers and verify them before she can verify newly created blocks. This is because she has no idea what the current unspent transaction output (UTXO) set looks like. To build the current UTXO set, she needs to start with an empty UTXO set, go through all historic blocks from block 0, and update the UTXO set with the information in the transactions in the blocks.
The process is as follows:
-
Download all historic block headers from one peer, and verify the proof of work.
-
Download all blocks on the strongest chain from multiple peers in parallel.
Selma selects one of her peers, Tom, to download all block headers from. Figure 214 shows how Selma’s node downloads the block headers from Tom’s node.
 Tom sends back a list of
2,000 block headers; each block header is 80 bytes. Selma verifies each
header’s proof of work and requests a new batch of headers from Tom.
This process continues until Selma receives a batch of fewer than 2,000
headers from Tom, which is a signal that he has no more headers to give
her.
getheaders
message with her latest block ID.When Selma has received all the headers from Tom, she determines which branch is the strongest and starts downloading actual block data belonging to that branch from her peers. She can download block data from multiple peers at the same time to speed things up. Figure 215 shows her communication with Rashid’s node.
It starts with Selma, who sends a getdata
message to Rashid. This
message specifies which blocks she wants to download from Rashid, who
sends back the requested blocks in block
messages, one by one. Note
that Selma downloads only some of the blocks from Rashid. She also
downloads blocks from Tom in parallel, which is why there are gaps in
the sequence of requested blocks. The process repeats until Selma
doesn’t want any more blocks from Rashid.
getdata
message with a list block IDs she wants the blocks for.As Selma downloads blocks, Rashid will probably receive more fresh
blocks from his peers. Suppose he has received a new block by the time
Selma has received the first 100 blocks from Rashid. Rashid will then
send out a headers
message to his peers, including Selma, as
described in Section 8.4.4. This way, Selma
will be aware of all new blocks appearing during her initial
synchronization and can later request them from any peer.
As Selma receives blocks, she verifies them, updates her UTXO set, and adds them to her own blockchain.
Verifying early blocks
The most time-consuming part of verifying a block is verifying the transaction signatures. If you know of any block ID that’s part of a valid blockchain, you can skip verifying the signatures of all blocks prior to and including this block (Figure 216). This will greatly speed up the initial blockchain download up to that block.
Of course, other stuff, like verifying that no double spends occur or that the block rewards are correct, is still done. The syncing node must build its own UTXO set, so it must still go through all transactions to be able to update the UTXO set accordingly.
Bitcoin Core ships with a preconfigured block ID of a block from some weeks back from the release date. For Bitcoin Core 0.17.0, that block is
height: 534292 hash:<PHONE_NUMBER>000000002e63058c023a9a1de233554f28c7b21380b6c9003f36a8
This is about 10,000 blocks back in the blockchain at release date. This is, of course, a configuration parameter, and the aforementioned block is just a default reasonable value. Selma could have changed this when starting her node, or she could have verified with friends and other sources she trusts that this block is in fact representing an “all valid transactions blockchain.” She could also have disabled the feature to verify all transaction signatures since block 0.
After a while, Selma is finally on the same page as the other nodes and ready to enter the normal mode of operation.
8.7.4. Step 4—Normal operation
This step is easy because we already discussed it in Section 8.4. Selma enters the normal mode of operation. From now on, she’ll participate in block propagation and transaction propagation, and verify every transaction and block coming in (Figure 217).
Selma is now running a full-blown full node.
8.8.  It’s intended for readers comfortable with the Linux OS and command line. |
You’ve seen how a Bitcoin full node is downloaded, started, and synchronized in theory. This section will help you install your own full node.
This section requires that you
-
Have a computer with at least 2 GB of RAM running a Linux OS.
-
Have lots of available disk space. As of this writing, about 210 GB is needed.
-
Have an internet connection without a limited data plan.
-
Know how to start and use a command-line terminal.
If you don’t have a Linux OS, you can still use these instructions; but you’ll have to install the version of Bitcoin Core that’s appropriate for your system, and the commands will look different. I suggest that you visit Web resource 18 to get up-to-date instructions for your non-Linux OS.
The general process for getting your own node running is as follows:
-
Download Bitcoin Core from https://bitcoincore.org/en/download.
-
Verify the software.
-
Unpack and start.
-
Wait for the initial blockchain download to finish.
8.8.1. Downloading Bitcoin Core
To run your own full Bitcoin node, you need the software program to run. In this example, you’ll download Bitcoin Core from Web resource 19. As of this writing, the latest version of Bitcoin Core is 0.17.0. Let’s download it:
$ wget https://bitcoincore.org/bin/bitcoin-core-0.17.0/\ bitcoin-0.17.0-x86_64-linux-gnu.tar.gz
As the filename bitcoin-0.17.0-x86_64-linux-gnu.tar.gz indicates, the command downloads version 0.17.0 for 64-bit (x86_64) Linux (linux-gnu). By the time you read this, new versions of Bitcoin Core will probably have been released. Consult Web resource 19 to get the latest version of Bitcoin Core. Also, if you use another OS or computer architecture, please select the file that’s right for you.
8.8.2. Verifying the software
| This section is hard and requires a fair amount of work on the command line. If you just want to install and run the Bitcoin Core software for experimental purposes, you can skip this section and jump to Section 8.8.3. If you aren’t using it for experimental purposes, please understand the risks explained earlier in this chapter in Section 8.7.1 before skipping this step. |
This section will show you how to verify that the downloaded .tar.gz file hasn’t been tampered with in any way. This file is digitally signed by the Bitcoin Core team’s private key. The verification process involves the following steps:
-
Download the signature file.
-
Verify that the hash of the .tar.gz file matches the hash in the message part of the signature file.
-
Download the Bitcoin Core team’s public key.
-
Install the public key as trusted on your computer.
-
Verify the signature.
Let’s get started.
Downloading the signature file
To verify that your downloaded Bitcoin Core package is actually from the Bitcoin Core team, you need to download the signature file named SHA256SUMS.asc. Figure 218, repeated from Section 8.7.1, explains how the SHA256SUMS.asc file is designed.
Download the signature file SHA256SUMS.asc from the same server you downloaded the program from:
$ wget https://bitcoincore.org/bin/bitcoin-core-0.17.0/SHA256SUMS.asc
This file will be used to verify that the downloaded .tar.gz file is signed by the Bitcoin Core team. Note that this file is for version 0.17.0 only. If you use another version of Bitcoin Core, please select the correct signature file at Web resource 19.
The following listing shows what the contents of this file look like (the actual hashes have been shortened):
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 1e43...35ed bitcoin-0.17.0-aarch64-linux-gnu.tar.gz a4ff...7585 bitcoin-0.17.0-arm-linux-gnueabihf.tar.gz 967a...f1b7 bitcoin-0.17.0-i686-pc-linux-gnu.tar.gz e421...5d61 bitcoin-0.17.0-osx64.tar.gz 0aea...ac58 bitcoin-0.17.0-osx.dmg 98ef...785e bitcoin-0.17.0.tar.gz 1f40...8ee7 bitcoin-0.17.0-win32-setup.exe 402f...730d bitcoin-0.17.0-win32.zip b37f...0b1a bitcoin-0.17.0-win64-setup.exe d631...0799 bitcoin-0.17.0-win64.zip 9d6b...5a4f bitcoin-0.17.0-x86_64-linux-gnu.tar.gz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJbtIOFAAoJEJDIAZ42wulk5aQP/0tQp+EwFQPtSJgtmjYucw8L SskGHj76SviCBSfCJ0LKjBdnQ4nbrIBsSuw0oKYLVN6OIFIp6hvNSfxin1S8bipo hCLX8xB0FuG4jVFHAqo8PKmF1XeB7ulfOkYg+qF3VR/qpkrjzQJ6S/nnrgc4bZu+ lXzyUBH+NNqqlMeTRzYW92g0zGMexig/ZEMqigMckTiFDrTUGkQjJGzwlIy73fXI LZ/KtZYDUw82roZINXlp4oNHDQb8qT5R1L7ACvqmWixbq49Yqgt+MAL1NG5hvCSW jiVX4fasHUJLlvVbmCH2L42Z+W24VCWYiy691XkZ2D0+bmllz0APMSPtgVEWDFEe wcUeLXbFGkMtN1EDCLctQ6/DxYk3EM2Ffxkw3o5ehTSD6LczqNC7wG+ysPCjkV1P O4oT4AyRSm/sP/o4qxvx/cpiRcu1BQU5qgIJDO+sPmCKzPn7wEG7vBoZGOeybxCS UUPEOSGan1Elc0Jv4/bvbJ0XLVJPVC0AHk1dDE9zg/0PXof9lcFzGffzFBI+WRT3 zf1rBPKqrmQ3hHpybg34WCVmsvG94Zodp/hiJ3mGsxjqrOhCJO3PByk/F5LOyHtP wjWPoicI2pRin2Xl/YTVAyeqex519XAnYCSDEXRpe+W4BdzFoOJwm5S6eW8Q+wkN UtaRwoYjFfUsohMZ3Lbt =H8c2 -----END PGP SIGNATURE-----
The signed message in the upper part of the file lists several files along with their respective SHA256 hashes. The listed files are installation packages for all OSs and architectures for which Bitcoin Core is released. The lower part of the file is the signature of the message in the upper part. The signature commits to the entire message and thus to all the hashes and files listed in the message.
Verifying the hash of the downloaded file
The file you downloaded is named bitcoin-0.17.0-x86_64-linux-gnu.tar.gz
so you expect that the SHA256 hash of that file matches 9d6b…5a4f
exactly. Let’s check:
$ sha256sum bitcoin-0.17.0-x86_64-linux-gnu.tar.gz 9d6b472dc2aceedb1a974b93a3003a81b7e0265963bd2aa0acdcb17598215a4f bitcoin-0.17.0-x86_64-linux-gnu.tar.gz
This command calculates the SHA256 hash of your downloaded file. It does indeed match the hash in the SHA256SUMS.asc file. If they don’t match, then something is wrong, and you should halt the installation and investigate.
Getting the Bitcoin Core signing key
To verify that the signature in the signature file was done using the Bitcoin Core signing key, you need the corresponding public key. As noted in Section 8.7.1, you should convince yourself about what fingerprint the Bitcoin Core key has and then download that key from any source.
You could, for example,
-
Get the fingerprint of the Bitcoin Core team’s key from https://bitcoincore.org, the official website of the Bitcoin Core team.
-
Consult the book Grokking Bitcoin to verify the fingerprint.
-
Verify the fingerprint with a friend.
Start by finding the Bitcoin Core team’s public key fingerprint on their website. You find the following fingerprint on the downloads page:
01EA5486DE18A882D4C2684590C8019E36C2E964
Now, consult the book Grokking Bitcoin to check if the fingerprint in that book matches the fingerprint from https://bitcoincore.org. Look in the Section 8.7.1 of Chapter 8 of that book. It says
01EA 5486 DE18 A882 D4C2 6845 90C8 019E 36C2 E964
This is the same fingerprint (although formatted slightly differently). The book and the website https://bitcoincore.org both claim that this key belongs to the Bitcoin Core team. Let’s not settle for that. You’ll also call a friend you trust and have her read the fingerprint to you:
You: “Hello, Donna! What’s the fingerprint of the current Bitcoin Core signing key?”
Donna: “Hi! I verified that key myself a few months ago, and I know
the fingerprint is 01EA 5486 DE18 A882 D4C2 6845 90C8 019E 36C2 E964
.”
You: “Thank you, it matches mine. Goodbye!”
Donna: “You’re welcome. Goodbye!”
Donna’s statement further strengthens your trust in this key. You think you’ve collected enough evidence that this is, in fact, the correct key.
Let’s start downloading the key. To do this, you can use a tool called gpg, which stands for GnuPG, which in turn stands for Gnu Privacy Guard. This program conforms to a standard called OpenPGP (Pretty Good Privacy). This standard specifies how keys can be exchanged and how to do encryption and digital signatures in an interoperable way.
GnuPG is available on most Linux computers by default. To download a
public key with a certain fingerprint, you run the following gpg
command:
$ gpg --recv-keys 01EA5486DE18A882D4C2684590C8019E36C2E964 gpg: key 90C8019E36C2E964: public key "Wladimir J. van der Laan (Bitcoin Core binary release signing key<EMAIL_ADDRESS>imported gpg: no ultimately trusted keys found gpg: Total number processed: 1 gpg: imported: 1
Depending on the version of gpg you use, the output can vary. This command downloads the public key from any available key server and verifies that the downloaded public key in fact has the fingerprint that you requested. The owner of this key is “Wladimir J. van der Laan (Bitcoin Core binary release signing key).”
The prior command downloads the key into gpg and adds it to your list of known keys. But the output of this command mentions “no ultimately trusted keys found.” This means this key isn’t signed by any key that you trust. You’ve only imported the key. In gpg, keys can sign other keys to certify that the signed key is legit.
Signing the public key as trusted on your computer
You’ve verified that the key belongs to the Bitcoin Core team and installed that key onto your system using gpg.
You’ll now sign that key with a private key that you own. You do this to remember this key as trusted. The Bitcoin Core team will probably release new versions of Bitcoin Core in the future. If GnuPG remembers this public key as trusted, you won’t have to go through all these key-verification steps again when you upgrade.
The process is as follows:
-
Create a key of your own.
-
Sign the Bitcoin Core public key with your own private key.
GnuPG lets you create a key of your own with the following command:
$ gpg --gen-key gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Note: Use "gpg --full-generate-key" for a full featured key generation dialog. GnuPG needs to construct a user ID to identify your key.
GnuPG will ask for your name and email address. Answer these questions; they’ll be used to identify your key:
Real name: Kalle Rosenbaum Email address<EMAIL_ADDRESS>You selected this USER-ID: "Kalle Rosenbaum<EMAIL_ADDRESS>Change (N)ame, (E)mail, or (O)kay/(Q)uit?
Continue by pressing O (capital letter “oh”). You then need to select a password with which to encrypt your private key. Choose a password, and make sure you remember it.
Key generation might take a while, because it takes time to generate good random numbers for your key. When it’s finished, you should see output like this:
public and secret key created and signed. pub rsa2048 2018-04-27 [SC] [expires: 2020-04-26] B8C0D19BB7E17E5CEC6D69D487C0AC3FEDA7E796 B8C0D19BB7E17E5CEC6D69D487C0AC3FEDA7E796 uid Kalle Rosenbaum<EMAIL_ADDRESS>sub rsa2048 2018-04-27 [E] [expires: 2020-04-26]
You now have a key of your own that you’ll use to sign keys that you trust. Let’s sign the Bitcoin Core team key:
$ gpg --sign-key 01EA5486DE18A882D4C2684590C8019E36C2E964 pub rsa4096/90C8019E36C2E964 created: 2015-06-24 expires: 2019-02-14 usage: SC trust: unknown validity: unknown [ unknown] (1). Wladimir J. van der Laan (Bitcoin Core binary release signing key<EMAIL_ADDRESS>pub rsa4096/90C8019E36C2E964 created: 2015-06-24 expires: 2019-02-14 usage: SC trust: unknown validity: unknown Primary key fingerprint: 01EA 5486 DE18 A882 D4C2 6845 90C8 019E 36C2 E964 Wladimir J. van der Laan (Bitcoin Core binary release signing key<EMAIL_ADDRESS>This key is due to expire on 2019-02-14. Are you sure that you want to sign this key with your key "Kalle Rosenbaum<EMAIL_ADDRESS>(8DC7D3846BA6AB5E) Really sign? (y/N)
Enter y
. You’ll be prompted for your private key password. Enter it,
and press Enter. The Bitcoin Core key should now be regarded as trusted
by gpg. This will simplify the process when you upgrade your node in the
future.
Let’s look at your newly signed key:
$ gpg --list-keys 01EA5486DE18A882D4C2684590C8019E36C2E964 pub rsa4096 2015-06-24 [SC] [expires: 2019-02-14] 01EA5486DE18A882D4C2684590C8019E36C2E964 uid [ full ] Wladimir J. van der Laan (Bitcoin Core binary release signing key<EMAIL_ADDRESS>The word to look for is full
in square brackets. This means gpg, and
you, fully trust this key.
Verifying the signature
It’s time to verify the signature of the SHA256SUMS.asc file:
$ gpg --verify SHA256SUMS.asc gpg: Signature made Wed 03 Oct 2018 10:53:25 AM CEST gpg: using RSA key 90C8019E36C2E964 gpg: Good signature from "Wladimir J. van der Laan (Bitcoin Core binary release signing key<EMAIL_ADDRESS>[full]
It says that the signature is Good
and that it’s signed with a key
that you fully trust, [full]
.
To summarize, you’ve done the following:
-
Downloaded Bitcoin Core and the signature file
-
Verified that the hash of the .tar.gz file matches the stated hash in SHA256SUMS.asc
-
Downloaded a public key and verified that it belongs to Bitcoin Core
-
Signed that key with your own private key so GnuPG and you remember that the Bitcoin Core key is legit
-
Verified the signature of the SHA256SUMS.asc file
When you later upgrade the program, you can skip several of these steps. The process will then be
-
Download Bitcoin Core and the signature file.
-
Verify that the hash of the .tar.gz file matches the stated hash in SHA256SUMS.asc.
-
Verify the signature of the SHA256SUMS.asc file.
8.8.3. Unpacking and starting
Let’s unpack the software:
tar -zxvf bitcoin-0.17.0-x86_64-linux-gnu.tar.gz
This will create a directory called bitcoin-0.17.0. Go into the directory bitcoin-0.17.0/bin, and have a look:
$ cd bitcoin-0.17.0/bin $ ls bitcoin-cli bitcoind bitcoin-qt bitcoin-tx test_bitcoin
Here you have several executable programs:
-
bitcoin-cli is a program you can use to extract information about the node you’re running as well as manage a built-in wallet that’s shipped with Bitcoin Core.
-
bitcoind is the program to use if you want to run the node in the background without a graphical user interface (GUI).
-
bitcoin-qt is the program to run if you want a GUI for your node. This is mainly useful if you use the built-in wallet.
-
bitcoin-tx is a small utility program to create and modify Bitcoin transactions.
-
test_bitcoin lets you test run a test suite.
In this tutorial, you’ll run bitcoind, which stands for “Bitcoin daemon.” In UNIX systems such as Linux, the word daemon is used for computer programs that run in the background.
Let’s start the Bitcoin Core daemon in the background and see what happens:
$ ./bitcoind -daemon Bitcoin server starting
This starts your node. It will automatically begin connecting to peers and downloading the blockchain for you.
8.8.4. Initial blockchain download
This process will take time. Depending on your internet connection, processor, and disk, it can vary from several days down to a few hours.
You can use the bitcoin-cli program to query the running node about the download progress, as in the following:
$ ./bitcoin-cli getblockchaininfo { "chain": "main", "blocks": 207546, "headers": 549398, "bestblockhash": "00000000000003a6a5f2f360f02a3b8e4c214d27bd8e079a70f5fb630a0817c5", "difficulty": 3304356.392990344, "mediantime":<PHONE_NUMBER>, "verificationprogress": 0.0249296506976196, "initialblockdownload": true, "chainwork": "0000000000000000000000000000000000000000000000202ad90c17ec6ea33c", "size_on_disk":<PHONE_NUMBER>2, "pruned": false, "softforks": [ { "id": "bip34", "version": 2, "reject": { "status": false } }, { "id": "bip66", "version": 3, "reject": { "status": false } }, { "id": "bip65", "version": 4, "reject": { "status": false } } ], "bip9_softforks": { "csv": { "status": "defined", "startTime":<PHONE_NUMBER>, "timeout":<PHONE_NUMBER>, "since": 0 }, "segwit": { "status": "defined", "startTime":<PHONE_NUMBER>, "timeout":<PHONE_NUMBER>, "since": 0 } }, "warnings": "" }
This command shows a lot of information about the blockchain. Note that
blocks have been downloaded and verified up to height 207546. Bitcoin
Core will download block headers prior to the full blocks to verify
proof of work. This node has downloaded headers up to height 549398,
which are all the headers there are at this time. Another interesting
thing is the initialblockdownload
field, which will remain true
until the initial block download is finished.
Keep this daemon running. You’ll get back to it in appendix A, where I’ll give you a small tutorial on how to use bitcoin-cli to examine the blockchain and use your built-in wallet.
If you want to stop the node, issue the following command:
$ ./bitcoin-cli stop
You can start the node again whenever you like, and the node will begin where it left off.
8.9. Recap
We’ve replaced the last central point of authority, the shared folder, with a peer-to-peer network. In a peer-to-peer network, the full nodes communicate directly with each other. Each node is connected to several (potentially hundreds of) other nodes. This makes it extremely hard to prevent blocks and transactions from propagating the network.
This chapter had two main parts:
-
How transactions and blocks flow through the network
-
How new nodes join the network
8.9.1. Part 1—Following a transaction
In the first part of the chapter, we followed a transaction through the system. It started with John buying a cookie. His transaction was propagated across the peer-to-peer network and to the cafe’s wallet.
The cafe will almost immediately see that a transaction is incoming, but it’s not yet confirmed. The next stage is to mine the block. Rashid is the lucky miner who finds the next block containing John’s transaction.
Rashid sends out the block to his peers, who will relay it to their
peers and so on until the block has reached the entire network. Part of
this propagation includes sending the block to lightweight wallets.
These lightweight wallets will request merkleblock
messages from the
full node so they don’t have to download the full block.
8.9.2. Part 2—Joining the network
Starting a new node involves fours steps:
-
Download and verify, for example, the Bitcoin Core software. Then start it.
-
Connect to other nodes.
-
Download historic blocks.
-
Enter normal operation.
8.9.3. System changes
The table of concept mappings between the cookie token system and Bitcoin has become tiny (Table 27).
| Cookie tokens | Bitcoin | Covered in |
|---|---|---|
1 cookie token |
1 bitcoin |
Given that there are no longer any technical differences between the cookie token system and the Bitcoin system, we’ll drop the cookie tokens and work only with Bitcoin from now on.
This will be the final release of the cookie token system. Another, much more widely used system, Bitcoin, has taken the world by storm, and we’ve decided to ditch the cookie token project. Enjoy the last version (Table 28).
| Version | Feature | How |
|---|---|---|
8.0 |
Censorship-resistant; for real this time |
Shared folder replaced by a peer-to-peer network |
Transaction broadcasting |
Transactions broadcast to miners and others using the peer-to-peer network |
|
7.0 |
Censorship-resistant |
Multiple miners, “Lisas,” enabled by proof of work |
Anyone can join the mining race |
Automatic difficulty adjustments |
|
6.0 |
Prevent Lisa from deleting transactions |
Signed blocks in a blockchain |
Fully validating nodes |
Download and verify the entire blockchain. |
|
Lightweight wallet saves data traffic |
Bloom filters and merkle proofs |
8.10. Exercises
8.10.1. Warm up
-
Why is the shared folder a bad idea?
-
What does it mean to relay a transaction or a block?
-
What are
inv
messages used for? -
How does the full node decide what transactions to send to lightweight wallets?
-
How does a node notify a lightweight wallet about an incoming pending transaction?
-
Blocks aren’t sent in full to lightweight wallets. What part of the block is always sent to the wallet?
-
Why does the cafe send a very big bloom filter to its trusted node?
-
What would a security-conscious person do after downloading Bitcoin Core but before starting the software?
-
What types of sources for peer addresses are available to a newly started node?
-
How would a full node know if any newly created blocks are available for download when it’s finished syncing?
-
The Bitcoin peer-to-peer network consists of the following nodes:
Which node owners do you need to threaten to prevent Lisa from getting any blocks but those she creates herself?
8.10.2. Dig in
-
Suppose Qi just received two transactions with transaction IDs TXID1 and TXID2. She now wants to inform Rashid about these new transactions. She doesn’t know if Rashid already knows about them. What does she do?
-
Suppose you’re running a full node and experience a power outage for 18 minutes. When power comes back, you start your node again. During those 18 minutes, two blocks, B1 and B2, have been created. Your latest block is B0. What will your node do after reconnecting to the network? For simplicity, assume that no new blocks are found during synchronization, and that you have only one peer. Use this table of message types to fill out the following template:
Type Data Purpose block
Full block
Sends a block to a peer
getheaders
Block ID
Asks a peer for subsequent block headers after the given block ID
getdata
txids or block IDs
Requests data from a peer
headers
List of headers
Sends a list of headers to a peer
8.11. Summary
-
The peer-to-peer network makes blocks censorship-resistant.
-
A node connects to multiple peers to reduce their vulnerability for information hiding.
-
The Bitcoin network protocol is the “language” nodes speak to communicate.
-
Transactions are broadcast on the Bitcoin peer-to-peer network to reach both miners and the recipient of the money early.
-
New nodes synchronize with the Bitcoin network to get up to date with the other nodes. This takes hours or days.
-
Nodes don’t need to stay online 24/7. They can drop out and come back and sync up the latest stuff.
-
Signature verification can be skipped for older blocks to speed up initial synchronization. This is useful if you know a specific block is valid.
9. Transactions revisited
This chapter covers
-
Time-locking bitcoins
-
Swapping coins between blockchains
-
Attaching arbitrary data to transactions
-
Bumping the fee of a pending transaction
We’re now past the core chapters of the book, in which you learned the Bitcoin basics. In this chapter, we’ll dig deeper into the functionality transactions can offer.
We’ll start by exploring time locks. A time lock is a way to make a transaction invalid until some point in time. This means the transaction can’t be confirmed before that time constraint is met. Also, an output of a transaction can be programmed to prevent it from being spent until a time constraint is fulfilled. This is useful for digital contracts, such as atomic swaps, covered later in this chapter.
It’s sometimes useful to store a small amount of data in a transaction in the blockchain. For example, a car manufacturer might want to track ownership of a car by putting its chassis number into a Bitcoin transaction, effectively creating a token on the Bitcoin blockchain. The current owner can then transfer ownership of the car by sending that token to the new owner.
As mentioned in Section 1.6, several alternative cryptocurrencies are available. Sometimes, you might want to trade, for example, namecoins for bitcoins. The most obvious way to do this is to use an exchange to sell bitcoins and buy namecoins. But there are other, more decentralized ways to do it. Atomic swaps let you swap bitcoins directly with someone holding namecoins without a trusted third party, like an exchange.
If you pay a too-small transaction fee, miners might refuse to confirm the transaction within a reasonable time. In this situation, it can be helpful to replace the transaction with another one that pays a little more in fees. This is known as fee-bumping.
Finally, we’ll explore some intricate details of signatures. You can create signatures in different ways depending on your use case. You can tune what the signature should commit to: in other words, change how the signing algorithm hashes the transaction.
9.1. Time-locked transactions
When you create and sign a transaction, it’s valid and ready for inclusion in any future block. You can broadcast it immediately and have it mined. This is the normal case.
But in some cases, you may want to sign a transaction with a guarantee that it won’t be mined until after at least, say, one year has passed.
 You can create a transaction that’s time-locked (Figure 219).
What makes this transaction special are the inputs’ sequence numbers and
the transaction lock time. I briefly mentioned sequence numbers in
Chapter 5. They’re used to enable the lock time: if any input has a
sequence number less than ffffffff
—for example, fffffffe
—the lock
time set on the transaction will be effective. If all sequence numbers
are ffffffff
, the lock time won’t have any effect.
You give this transaction, Tx1, to your daughter. It’s currently invalid; your daughter stores it on her computer and prints a backup that she keeps in another place. It isn’t broadcast; no full node will accept a block containing this transaction yet. The transaction will become valid the morning of 30 April 2019. If you die before that, your daughter must wait until after the lock-time date and then claim the money by broadcasting the transaction, which will have become valid by then.
If you don’t die before that date, you want to make sure the time-locked transaction becomes useless so your daughter can’t take the money once the date has passed.
You can create, but not yet broadcast, a new transaction, Tx2, that double spends an output that Tx1 spends (Figure 220). You then create a new transaction, time-locked for yet another year, for your daugher. When she’s stored the transaction safely, you broadcast Tx2.
You need to
-
Create and sign a transaction, Tx2, that spends at least one of the outputs spent by Tx1. Tx2 is a normal, not time-locked, transaction. Don’t broadcast this transaction just yet.
-
Create a new time-locked transaction, Tx3, that spends all your outputs as if Tx2 was confirmed. Tx3 is locked for another year.
Give it to your daughter. -
Broadcast Tx2. Once Tx2 is mined, Tx1 will become forever invalid because one of the inputs of Tx1 is spent by Tx2.
Note how the order of events is important here. If Tx2 is broadcast before you give Tx3 to your daughter, there’s a chance you’ll die before giving her Tx3. Then, your daughter won’t be able to receive the funds because she has no valid transaction to claim them with. Tx1 is invalidated by Tx2 in the blockchain, and Tx3 isn’t in your daughter’s possession.
9.1.1. Time measurements
You can express a lock time in two ways. The first is by setting a date and time as in the previous example. The second is to set a block height.
Block time
The first example expressed the lock time as a date and time. This means the median time past must be greater than the lock time in the transaction. In Chapter 7, I noted that a block’s timestamp must be greater than the past 11 blocks’ median timestamp, or the median time past of the block. We use the median time past to decide whether a transaction is valid as regards the lock time. Suppose you died on 24 January 2019. Your mourning daughter wouldn’t be able to claim your money until 30 April 2019. Figure 221 illustrates this more precisely.
Your daughter’s transaction can’t be mined in any block before the last one shown. Before that block, the median time past is too early.
Her transaction won’t even propagate through the Bitcoin network until the lock time has passed. The nodes don’t want to keep time-locked transactions in their memories because there are better uses for their precious memory space than to fill them up with transactions that aren’t even valid (yet). It’s up to your daughter to broadcast the transaction after the lock time has passed.
Block height
You can also express time using block height. You can say that a transaction isn’t valid, for example, until after block height 571019. This means the transaction shown in Figure 222 can’t be mined until after block 571019 has been mined.
The earliest block in which the transaction can be included is at height 571020. It’s hard to predict exactly when that block will be mined, but thanks to the difficulty adjustments that keep the average block time at about 10 minutes, you can expect about 52,596 blocks per year.
9.1.2. Relative time locks
The earlier example showed a use case for absolute time locks on transactions. But you can also lock an input of a transaction until its spent output is old enough. This is called a relative time lock. You do this on a per-input basis (Figure 223).
The transaction’s first input has a sequence number of 004013c6
. This
says the transaction isn’t valid until 30 days have passed since the
spent output was confirmed (Figure 224).
The leftmost bit of this sequence number is 0, which means the relative
lock time is enabled. The bit at index 9 from the left is 1, which means
the rightmost 16 bits should be interpreted as “number of 512-second
intervals.” The 16 rightmost bits are 13c6
, which translates to 5,062
in decimal form; 5,062 intervals of 512 seconds is roughly 30 days.
The second input has a sequence number of 000003e8
(Figure 225). This
means the transaction is invalid until 1,000 blocks have been mined
since the spent output was mined.
The leftmost bit is 0 here, too, which means the relative lock time is
enabled for this input. The bit at index 9 from the left is 0, which
means the 16 rightmost bits should be interpreted as the number of
blocks; 03e8
is hex code for 1,000.
The transaction’s version needs to be at least 2 for relative time locks to work. If the version is 1, the sequence numbers won’t have any effect on the relative lock time, but they will affect absolute lock time and the replace-by-fee feature, which I’ll discuss later in Section 9.4.
9.2. Time-locked outputs
Time locks aren’t particularly useful in themselves. The only thing you can do with them is create a transaction that might eventually become valid.
It might be more useful to say something like, “The money in this output can’t be spent before New Year’s Eve.” This is an example of a time-locked output. An output can be locked absolutely or relatively, and locks can be time-based or height-based.
9.2.1. Absolute time-locked outputs
Suppose you want to give your daughter 1 BTC in allowance on 1 May. You can make a transaction as Figure 226 shows.
You can broadcast this transaction immediately to the Bitcoin network and have it mined. The first output is the interesting part. It says that this output can’t be spent before 1 May. For the curious, the exact pubkey script is
<may 1 2019 00:00:00> OP_CHECKLOCKTIMEVERIFY OP_DROP OP_DUP OP_HASH160 <PKHD> OP_EQUALVERIFY OP_CHECKSIG
This script will make sure the transaction spending the output is sufficiently time-locked, as Figure 227 shows.
The first two transactions will never be valid because their time locks aren’t sufficiently late. The first one isn’t locked, which is illegal according to the pubkey script. The second one is at least time-locked, but it isn’t late enough—1 second before 1 May is too early.
The third transaction is OK because the time lock is at least as high as the time in the pubkey script, 2019-05-01 00:00:00. This transaction will be valid on and after 1 May. The last transaction will be valid on New Year’s Eve, right before the fireworks. Note, however, that you can’t get both of the last two transactions confirmed—you can get at most one of them confirmed—because they spend the same output.
The result of this example is that your daughter will be able to spend the output as she pleases after 1 May.
9.2.2. Relative time-locked outputs
A relative time-locked output works similarly to an absolute time-locked output, but relative locks require a certain amount of time to pass between the block containing the spent output and the block containing the spending transaction (Figure 228).
Relative time locks are most commonly used in digital contracts. A digital contract can be regarded as a traditional contract between parties, but it’s enforced by the rules of the Bitcoin network rather than national laws. Contracts are expressed as Bitcoin pubkey scripts. We’ll illustrate the use of relative time-locked outputs with an atomic swap in the next subsection. An atomic swap means two people swap coins with each other across different cryptocurrencies.
9.2.3. Atomic swaps
A commonly mentioned digital contract is the atomic swap, where two parties want to swap coins with each other between different blockchains.
Suppose John is chatting with Fadime on a public forum on the internet. They don’t know each other and have no reason to trust one another. But they both want to trade.
They agree that John will trade 2 BTC for 100 of Fadime’s namecoins (NMC). Namecoin is an alt-coin used as a decentralized naming system, like DNS. We talked briefly about alt-coins in Chapter 1. It isn’t important what Namecoin actually is used for in this example; we only conclude that it’s another cryptocurrency on a blockchain other than Bitcoin’s.