Hot Replication among Remote Table Partition Nodes
When we update key's value of table partition on Remote Storage Node, it starts data transfer to the next remote node. Then the database does not stop, and it can process SQL statements.
In this article, I mention how to replicate and delete old data without stop.
- Architecture of Storage Engine
- Replication & Synchronizing Process on Table Partitioning
- SELECT Queries On Copying Data
This database is designed for dynamic table partitioning.
When rebuilding table partitions, it transfer data for next partitioning structure. Then it does not have to stop or lock the database.
Actual data replication and synchronizing process runs by following procedure.
After calculating the range of data to copy, and which remote nodes are source or destination, starts to copy the data.
Then it does following operation.
- Lock the table in Update Mode
- Get current commit id as data version
- Start logging update logs, and release the lock
- Read certain quantity of data with using shared lock
- Repeat until all data to copy does not remain
When data copy is done, it transfers log data from source node to destination nodes. Then start to apply the logged update to the destination remote node.
When update process with the log ends, once lock the table in Update Mode, and check no additional log does not remain. If it remained, release lock and apply log update again.
If the additional update log is not there, then update the partition key's value on the nodes. At this time, replication process finished.
After copying the data, source nodes still has records before data transfer. The vacuum operation that will be executed later erases them.
When it copying the data, the table storage has a lot of duplicated records. Even after that, source storage has them.
But SELECT queries and scanning table by UPDATE and DELETE, works correctly.
That is because table storages of each remote nodes has range of partition keys. If the record's partition key's value is out of the range, it ignores.