Design of Dynamic Partitioning of Distributed Database
Now, 2017/02/05. To make and manage partition table of database tables is essential to deal bigdata. Therefore I have to design the architecture now.
Whether dynamic maintenance of partitioning is possible or not is also up to the records format of storage engine.
Alinous Elastic DB has 2 layer table data partitioning. By using this function, you can manage cold and hot data in different cloud section.
For example, the main partition key is timestamp of the record created, and sub key is the region. The old data is managed in storages with big capacity, but not so fast. Hot data is manages in storages which are very fast, but not with so big capacity.
When we change the key of partition node's maximum value or value of Partition Key.
Each distributed table has maximum value. The table storage has records whose value of the key is less than it.
On changing partition node's maximum value, the data is transferred to other node like below.
The operation to change the partition key is below.
- Select Storage Node, Table, and set key values to change
- Transfer data
- Lock the table
- Actually change the key value and release the table lock
- Vacuum source data in last node, later
Each storage have maximum values of both Partition Key and Sub Partition Key. When we change the Partition Key, the data is transferred into other storage node like below.
The red line parts are changed stuff.
When changing the key value of the sub partition, it works same with single partition key.
When we manage database, to manage actual data structure on distributed nodes is necessary, to keep the performance of queries and updates.
Then we do not have to stop the database.
That is because each storage node has maximum and minimum value of the partition key. Therefore, if there are records whose range is out of them, they are ignored on scanning the table.