In Apache Pulsar, each message has its ID, namely the message ID, which consists of four parts: ledgerId:entryID:partition-index:batch-index.
partition-index: partition number, which is –1 for non-partition topics.
batch-index: It is set to –1 for messages not sent in batches.
The message ID generation rules are determined by the message storage mechanism of Apache Pulsar. The following figure shows the schematic diagram of message storage in Apache Pulsar.
As shown in the preceding figure, each partition of a topic corresponds to a series of ledgers in Apache Pulsar, and only one ledger is in the open state (that is, writable), and each ledger stores messages only for that partition.
When Apache Pulsar stores a message, it first finds the ledgers used by the partition and then generates an entry ID for the message. Entry IDs are incremental in a ledger. After the ledger availability duration or the number of entries stored in it exceeds the threshold, new messages are stored in the next ledger in the partition.
When messages are produced in batches, one entry may contain multiple messages.
When messages are not produced in batches, one entry contains one message. (This parameter can be configured on the producer side, and batch production is used by default.)
A ledger is just a logical dimension for data assembly and has no actual entity, and bookie nodes only write, search for, and obtain data based on the entry dimension.
Deep Dive into the Sharding Mechanisms: Ledger and Entry
In Apache Pulsar, message data is stored on bookie nodes in a BookKeeper cluster in the form of ledgers. A ledger is a data structure that only appends data, and only one writer is available. The writer is responsible for writing data to multiple bookie nodes. Ledger entries are replicated to multiple bookie nodes, and relevant data is written to ensure data consistency.
The data that BookKeeper needs to store includes:
Journals
A journal file stores BookKeeper transaction logs. Before any ledger update is made, the description information of the update is persisted to a journal file first.
BookKeeper provides a separate sync thread to roll a journal file based on its size.
EntryLogFile
The file stores actual data. Entry data from different ledgers is first cached in the memory buffer and then flushed to the entry log file in batches.
By default, data from all ledgers is aggregated and then sequentially written to the same entry log file to avoid random disk writes.
Index file
Entry data of all ledgers is written to the same entry log file. To accelerate data reading, a mapping from ledgerId + entryId to the file offset is created and cached in memory, which is called the index cache.
When the index cache capacity reaches the upper limit, entry data is flushed to the disk by the sync thread.
The following figure shows the read/write interactions of the three types of data files.
Entry data writing
1. Data is first written to the journal file (stored in the disk in real time) and memtable (read/write cache) at the same time.
2. After the data is written to the memtable, a response is given to the write request.
3. When the memtable is full, data is flushed to the entry log file and index cache. The entry log file stores the data, and the index cache stores the index information of the data.
4. The background thread persists data in the entry log file and index cache to the disk.
Entry data reading
Tailing read request: Entries are read directly from the memtable.
Catch-up read (lagging consumption) requests: Index information is read first, and then entries are read from the entry log file using the index.
Data consistency guarantee: LastLogMark
The written entry logs and indexes are cached in memory first and then flushed to the disk periodically based on certain conditions, which causes an interval between memory caching and disk storage. If the BookKeeper process crashes during this interval, data needs to be restored based on the journal file after the process is restarted. LastLogMark marks the position in the journal file from where data is restored.
LastLogMark is actually stored in memory. When data in the index cache is flushed to the disk, its value is updated. LastLogMark is also periodically persisted to a disk file for the BookKeeper process to read during startup and recover from the journal file.
Once LastLogMark is persisted to the disk, the preceding indexes and entry logs have been persisted to the disk, and the data in the journal file before LastLogMark can be cleared.
