Introduction

On the 9th of March 2020, the New York Stock Exchange halted trading for 15 minutes as the result of triggering level 1 circuit breaker caused by a 7% drop in the S&P 500. The last time the circuit breaker kicked in was back in October 1997.

What are the Circuit Breakers?

Circuit Breakers are automatic mechanisms that prevent the stock market from a free fall by halting trading for a predefined time.

The New York Stock Exchange has 3 levels of market-wide circuit breakers.

Level 1 market-wide circuit breaker is triggered if the S$P 500 falls 7% from its previous close before 3:25 pm on a trading day. It halts trading for 15 minutes.

Level 2 market-wide circuit breaker kicks in when the market plunges 13% (also before 3:25 pm). It halts trading for an additional 15 minutes.

Level 3 market-wide circuit breaker comes into effect if the market tanks 20% in a day and can be triggered at any time even after 3:25 pm. If this circuit breaker kicks in, trading is halted for the rest of the trading day.

Besides having market-wide circuit breakers based on the S&P 500 price, individual stocks also have their own circuit breakers, with the trigger level determined by the price of the stock.

Circuit breakers are not only limited to the NYSE. In fact, most of the exchanges including futures exchanges have their own circuit breakers. The noticeable exceptions are the cryptocurrency exchanges that usually do not implement circuit breakers as they are not regulated in the same way as more traditional exchanges.

Why would you stop the stock market in the first place?

The main reason is to stop panic selling. The market usually plunges by 7 or more % based on either significant, unprecedented news or a flash crash. In both cases, it might make sense to take a little break to catch a breath, absorb all the surrounding information and reassess trading strategies.

The other big reason is the Regulator. Following huge market plunges like the one on Black Monday (October 1987), the regulators came up with circuit breakers to try to cool down the stock market and prevent it from a total collapse.

On the other hand, one of the reasons against the circuit breakers is that the price discovery mechanism is significantly impacted during the halting period and can lead to abnormal trading volume and volatility when trading resumes.

A quick history of circuit breakers

First circuit breakers were introduced after Black Monday in October 1987 when the stock market tanked 23%. The initial market-wide circuit breakers were based on the Dow Jones Industrial Average and not the S&P 500. They were also point-based instead of percentage-based. The first time the circuit breaker like that kicked in was in October 1997 (mini-crash). That was the only time in the history of the NYSE when the Dow Jones Industrial Average based circuit breaker was triggered. After that, the circuit breakers evolved and started using a percentage-based mechanism based on the S&P 500 index.

On the 9th of March 2020, the New York Stock Market halted trading for 15 minutes as a result of triggering level 1 circuit breaker caused by a 7% drop in S&P500 on fears of a global pandemic. 3 days later on the 12th of March, the level 1 circuit breaker was triggered again and stopped trading for another 15 minutes.

In both cases, the circuit breakers worked as designed by calming down the stock market and preventing it from further loses.

Extra

Flash Boys by Michael Lewis (a story about high-frequency trading) ► https://amzn.to/38T2Y0N

If you have any questions about the circuit breakers, please comment down below

What is Bitcoin halving?

Bitcoin halving is an event that reduces the supply of newly produced bitcoins by half. It occurs every 4 years or to be precise every 210000 blocks and it is a part of the validation logic in the Bitcoin source code.

Bitcoin halving is important to discuss as it affects miners profitability and often has an impact on the Bitcoin price. It is also one of the main reasons why Bitcoin is perceived as sound (hard) money with the inflation rate steadily trending to zero and becoming deflationary in a long run in opposite to all the other fiat currencies.

To understand it better let’s quickly recap how miners are being rewarded for validating transactions on the Bitcoin network.

Proof of Work

Transactions on the Bitcoin network are validated and confirmed by miners using a consensus mechanism called Proof of Work. In Proof of Work, miners are constantly validating transactions and grouping them in blocks. To prove that they did all the necessary work for the Proof of Work algorithm they try to solve a simple but laborious mathematical problem. The first miner who finds the correct solution submits a newly created block to the network and is rewarded for their hard work with a block reward. The block reward consists of two elements: transaction fees and block subsidy. The block subsidy is the number of bitcoins that is the exact subject to the Bitcoin halvings.

Previous Halvings

In January 2009, when the Bitcoin network was started, the first block called genesis block was mined. According to the algorithm, the block subsidy for the first block started from 50 bitcoins which was an arbitrary number chosen by the creator of the network Satoshi Nakamoto. In November 2012, Bitcoin went through its first halving at the block height of 210000 where the block reward cut in half from 50 to 25 bitcoins. At that time there was already 10.5 millions of bitcoins in existence. The second halving happened in July 2016 at the block height of 420000 with 15.75 millions of bitcoins in the circulation. The third, highly anticipated halving will be happening in May 2020 at the block height of 630000 with 18.375 millions of bitcoins in the circulation. After that Bitcoin halvings will be occurring every 4 years until roughly the year 2140 when the Bitcoin blockchain reaches the final number of 32 halvings and stops rewarding miners with block subsidy, so they will have to rely solely on the transaction fees.

Why does Bitcoin halving matter?

There are a few main reasons why Bitcoin halving matters. Let’s dive into some of them.

Sound (hard) money theory. Bitcoin halvings create controlled issuance of Bitcoin the currency with predictable inflation and eventual deflation when all bitcoins are mined. The current inflation rate is at around 3.6% and will decrease to around 1.8% after the 3rd halving. The inflation rate will be decreasing until 2140 when there will be no new supply of bitcoins. After that Bitcoin becomes deflationary as some of the already existing bitcoins will be lost.

Price impact caused by a supply shock. Assuming the demand for Bitcoin stays the same and the supply is cut in half, it should cause the price of Bitcoin to go up (at least in theory). With the current supply, the market is able to absorb 1800 new bitcoins per day (considering the most miners sell their rewards instead of hoarding). After the halving, the amount of newly created bitcoins will drop to 900 per day. If the demand stays the same it should increase the price of each coin. Bitcoin market of course is not that easily predictable and even though previous halvings had an impact on the price there is no guarantee it will happen again. This is also broadly discussed in the Bitcoin stock to flow model which can be used to try to predict Bitcoin future price based on the current stock (coins already in existence) and flow (issuance of new coins).

Rewarding early adopters. To incentivise the early adopters and help with bootstrapping the network the inflation rate was the highest at the time of starting the network. This means that early miners were able to receive the highest rewards (50 bitcoins per block) and accumulate more bitcoins. By looking at Bitcoin Money Supply we can see that at the time of creating this video 18 out of 21 million bitcoins were already minted which again incentives early adopters.

How does the Bitcoin halving actually look like in the code?

Now let’s get a bit more technical and jump into the code that is actually responsible for the Bitcoin halving. This code snippet is taken from the Bitcoin Core Github repository from the validation.cpp file that is responsible, as you may guess, for validation. I will put a link to this piece of code in the description box below.

Bitcoin halving instead of being defined as a time constant is actually defined using the number of blocks. We can see that the number of halvings is calculated using current block height (nHeight) and subsidy halving interval (nSubsidyHalvingInterval) which is a constant equal to 210000 that represents the number of blocks after which a halving should occur.

210000*10/60/24/365.25 = 3.9926990646.

Assuming that a block is mined on average every 10 minutes we can quickly calculate that it should take roughly 4 years to mine 210000 blocks. 619894 is the current block height at the time of making this video. This means that the halvings variable is currently set to 619894/210000=2.9518761905 which in integer division is just 2 although it’s getting very close to 3.

Next two steps are also quite interesting. We have an ‘if’ check that returns 0 when halving is greater or equal to 64 which is mostly used to protect the right-shift operator that comes later. This doesn’t mean there will be 64 halvings as we will quickly realise the actual subsidy will be 0 from 33rd halving onwards. The next step is to just get the first block subsidy in satoshis. COIN constant represents 1 bitcoin in satoshis (100 000 000 satoshis).

Now we’re getting to the actual calculation which is performed by using the right-shift bit operation instead of division for performance reasons. If we start with 5 000 000 000 and begin right shifting it, we can quickly see that the maximum number of times we can do it is 32. At the 32nd attempt, our reward will be 1 satoshi if we shift one more time it will give us 0 as the result. We can quickly calculate that if one halving occurs every 4 years on average, this means that the mining reward will stop around the year 2140. We can get to that number by using Bitcoin genesis block date which is the 3rd of January 2009 + 32*4 + 4 ( last halving, all 4 years before 33rd halving). This leads us to the 3rd January 2141 which is often rounded to 2140 as the final year for Bitcoin mining block subsidy rewards.

Let’s see what happens in the code in the next halving estimated for around the 5th of May 2020. When block height reaches 630000 the halvings variable starts returning 3 (630000/210000). After that we’ll be doing right bit shifting going from 50 * 100 000 000 = 5 000 000 000 >> 3 = 625 000 000 satoshis = 6.25 BTC. And that will be the new block subsidy reward for the next 4 years.

Summary

Bitcoin halving is one of the most important mechanisms when it comes to the Bitcoin network and its monetary system. It allows for predictable, decreasing inflation rate of the currency and incentives early adopters to bootstrap and secure the network.

Extras

Mastering Bitcoin (by Andreas Antonopoulos) ► https://amzn.to/2ILqylp

If you have any questions regarding Bitcoin or Bitcoin Halving comment down below.

Pub/Sub

Before we dive into Kafka let’s start with a quick recap on what publish/subscribe (pub/sub) messaging is.

Publish/subscribe is a messaging pattern where the sender (or the publisher) doesn’t send data directly to a specific receiver. Instead, the publisher classifies the messages without knowing if there are any subscribers interested in a particular type of messages. Similarly, the receiver subscribes to receive a certain class of messages without knowing if there are any senders sending those messages.

Pub/sub systems usually have a broker, where all messages are published. This decouples publishers from subscribers and allows for greater flexibility in the type of data that subscribers want to receive. It also reduces the number of potential connections between publishers and subscribers.

Bulletin board comes handy as a good analogy to a pub/sub messaging pattern, where people can publish information in a central place without knowing who the recipients are.

What is Kafka?

Okay, so what is Kafka then?

Apache Kafka is an open-source, publish/subscribe (pub/sub) messaging system, also very often described as a distributed event log where all the new records are immutable and appended to the end of the log.

In Kafka, messages are persisted on disk for a certain period of time known as the retention policy. This is usually the main difference between Kafka and other messaging systems and makes Kafka in some way a hybrid between a messaging system and a database.

The main concepts behind Kafka are producers producing messages to different topics and consumers consuming those messages and maintaining their position in the stream of data. You can think about producers as publishers or senders of messages. Consumers, on the other hand, are analogous to the receivers or subscribers.

Kafka aims to provide a reliable and high-throughput platform for handling real-time data streams and building data pipelines. It also provides a single place for storing and distributing events that can be fed into multiple downstream systems which helps to fight the ever-growing problem of integration complexity. Besides all of that Kafka can also be easily used to build a modern and scalable ETL, CDC or big data ingest systems.

Kafka is used across multiple industries, from companies like Twitter and Netflix to Goldman Sachs and Paypal. It was originally developed by Linkedin and open sourced in 2011.

Kafka Architecture

Now let’s dive a little bit deeper into the Kafka architecture.

On a high level, usual Kafka architecture consists of a Kafka cluster, producers and consumers.

A single Kafka server within a cluster is called a broker. A Kafka cluster usually consists of at least 3 brokers to provide enough level of redundancy. The broker is responsible for receiving messages from producers, assigning offsets and committing messages to disk. It is also responsible for responding to consumers fetch requests and serving messages.

In Kafka, when messages are sent to a broker, they are sent to a particular topic. Topics provide a way of categorising data that is being sent and they can be further broken down into a number of partitions. For example, a system can have separate topics for processing new users and for processing metrics. Each partition acts as a separate commit log and the order of messages is guaranteed only across the same partition. Being able to split a topic into multiple partitions makes scaling easy as each partition can be read by a separate consumer. This allows for achieving high throughput as both partitions and consumers can be split across multiple servers.

Producers are usually other applications producing data. This can be, for example, our application producing metrics and sending them to our Kafka cluster.

Similarly, consumers are usually other applications consuming data from Kafka. As we mentioned before Kafka very often acts like a central hub for all the events in the system which means it’s a perfect place to connect to if we are interested in a particular type of data. A good example would be a database that can consume and persist messages or an elastic search cluster that can consume certain events and provide full-text search capabilities for other applications.

Messages, Batches, Schemas, Topics and Partitions

Now as we went through the general overview of Kafka, let’s jump into the nitty-gritty details.

In Kafka, a message is a single unit of data that can be sent or received. As far as Kafka is concerned, a message is just a byte array, so the data doesn’t have any special meaning to Kafka. A message can also have an optional key (also a byte array) that can be used to write data in a more controlled way to multiple partitions within the same topic.

As an example, let’s assume we want to write our data to multiple partitions as it will be easier to scale the system later. We realised that certain messages, let’s say for each user, have to be written in order. If our topic has multiple partitions, there is no guarantee which messages will be written to which partitions, most likely the new messages would be written to partitions in a round-robin fashion. To avoid that situation we can define a consistent way for choosing the same partition based on a message key. One way of doing that would as simple as using (user id) % (number of partitions) that would assign always the same partition to the same user.

Sending single messages over the network creates a lot of overhead, that’s why messages are written into Kafka in batches. A batch is a collection of messages produced for the same topic and partition. Sending messages in batches provides a trade-off between latency and throughput and can be further controlled by adjusting a few Kafka settings. Additionally, batches can be compressed which provides even more efficient data transfer.

Even though we already established that Kafka messages are just simple byte arrays, in most cases it makes sense to provide additional structure to the message content. There are multiple schema options available. The most popular ones are JSON, XML, Avro or Protobuf.

We already described what topics and partitions are, but let’s just emphasise again the importance of not having any guarantees when it comes to a message time-ordering across multiple partitions of the same topic. The only way to achieve the ordering for all messages is to have only one partition. By doing that we can be sure that events are always ordered by the time they were written into Kafka.

Another important concept when it comes to partitions is the fact that each partition can be hosted on a different server which means that a single topic can be scaled horizontally across multiple servers to improve the throughput.

Producers, Consumers, Offsets, Consumer Groups and Rebalancing

Kafka cluster wouldn’t be very useful without its clients who are the producers and consumers of the messages.

Producers create new messages and send them to a specific topic. If a partition is not specified and a topic has multiple partitions, messages would be written into multiple partitions evenly. This can be further controlled by having a consistent message key that we described earlier.

Consumers, on the other hand, read messages. They subscribe to one or multiple topics and read messages in the order they were produced. The consumer keeps track of its position in the stream of data by remembering what offset was already consumed. Offsets are created at a time a message is written to Kafka and they correspond to a specific message in a specific partition. Within the same topic, multiple partitions can have different offsets and it’s up to the consumer to remember what offset each partition is at. By storing offsets in Zookeeper or Kafka itself a consumer can stop and restart without losing its position in the stream of data.

Consumers always belong to a specific consumer group. Consumers within a consumer group work together to consume a topic. The group makes sure that each partition is only consumed by one member of a consumer group.

This way, consumers can scale horizontally to consume topics with a large number of messages. Additionally, if a single consumer fails, the remaining members of the group will rebalance the partitions to make it up for the missing member.

In case we want to consume the same messages multiple times, we have to make sure the consumers belong to different consumer groups. This can be useful if we have multiple applications that have to process the same data separately.

Clusters, Brokers and Retention

As we mentioned before, Kafka cluster consists of multiple servers called brokers. Depending on the specific hardware, a single broker can easily handle thousands of partitions and millions of messages per second.

Kafka brokers are designed to operate as part of a cluster. Within a cluster of brokers, one broker will also act as the cluster controller (elected automatically). The controller is responsible for administrative operations, including assigning partitions to brokers and monitoring for broker failures. A partition is always owned by a single broker in the cluster, who is called the leader of the partition. A partition may be assigned to multiple brokers, which will result in the partition being replicated. This provides redundancy of messages in the partition, such that another broker can take over leadership if there is a broker failure. However, all consumers and producers operating on that partition must connect to the leader.

One of the key features of Kafka is retention, which, for some period of time, provides durable storage of messages. Kafka brokers are configured with a default retention setting for topics, either retaining messages for some period of time, 7 days by default or until the topic reaches a certain size in bytes e.g. 1GB. Once these limits are reached, the oldest messages are expired and deleted so that the retention configuration is a minimum amount of data available at any time. Individual topics can also configure their own retention settings. For example, a topic for storing metrics might have very short retention of a few hours. On the other hand, a topic containing bank transfers might have a retention policy of a few months.

Reliability Guarantees

Reliability is often discussed in terms of guarantees. These are certain behaviours of a system that should be preserved under different circumstances. Understanding those guarantees is critical for anyone trying to build reliable applications on the top of Kafka.

These are the most common Kafka reliability guarantees:

• Kafka guarantees the order of messages in a partition. If message A was written before message B, using the same producer in the same partition, then Kafka guarantees that the offset of message B will be higher than message A. This means that consumers will read message A before message B
• Messages are considered “committed” when they are written to the leader and all in-sync replicas (number of in-sync replicas and number of acks can be configured)
• Committed messages won’t be lost as long as at least one replica remains alive and retention policy holds
• Consumers can only read committed messages
• Kafka provides at-least-once message delivery semantics (doesn’t prevent duplicated messages being produced)

The important thing to note is that even though Kafka provides at-least-once delivery semantics, it does not provide exactly-once semantics and to achieve that we have to either rely on an external system with some support for unique keys or use Kafka Streams (0.11.0 release onwards).

Let’s also remember that even though these basic guarantees can be used to build a reliable system, there is much more to that. In Kafka, there is a lot of trade-offs involved in building a reliable system. The usual trade-offs are reliability and consistency versus availability, high throughput and low latency.

Why Kafka? Pros and Cons

Let’s review both the pros and cons of choosing Kafka.

Pros

• Tackles integration complexity
• Great tool for ETL or CDC
• Great for big data ingestion
• High throughput
• Disk-based retention
• Supports multiple producers/consumers
• Highly scalable
• Fault-tolerant
• Fairly low-latency
• Highly configurable
• Provides backpressure

Cons

• Requires a fair amount of time to understand and do not shoot yourself in the foot by accident
• Might not be the best solution for real low-latency systems

Kafka vs other messaging systems

A lot of things in Kafka were purposely named to resemble a JMS like messaging systems. This makes people wondering what the actual differences between Kafka and standard JMS systems like RabbitMQ or ActiveMQ are.

First of all the main difference is that Kafka consumers pull messages from the brokers which allow for buffering messages for as long as the retention period holds. In most other JMS systems, messages are actually pushed to the consumers instead. Pushing messages to the consumer makes things like backpressure really hard to achieve. Kafka also makes replaying of events easy as messages are stored on disk and can be replayed anytime (again within the limits of the retention period).

Besides that Kafka guarantees the ordering of messages within one partition and it provides an easy way for building scalable and fault-tolerant systems.

Summary

Time for a quick summary. In the era of ever-growing data and integration complexity, having a reliable and high-throughput messaging system that can be easily scaled is a must. Kafka seems to be one of the best available options that meet those criteria. It has been battle-tested for years by one of the biggest companies in the world.

We have to remember that Kafka is a fairly complex messaging system and there is a lot to learn to make full potential of it without shooting ourselves in the foot.

There are also multiple libraries and frameworks that make using Kafka even easier. Some of the most notable ones are Kafka Streams and Kafka Connect.

Extras

If you want to learn even more about Kafka, I can recommend the following book: “Kafka: The Definitive Guide”, which I found very useful.