Design of Dynamic Partitioning of Distributed Database

Now, 2017/02/05. To make and manage partition table of database tables is essential to deal bigdata. Therefore I have to design the architecture now.

Whether dynamic maintenance of partitioning is possible or not is also up to the records format of storage engine.

Two Layer Partitioning of Tables

Alinous Elastic DB has 2 layer table data partitioning. By using this function, you can manage cold and hot data in different cloud section.

Two Layer Dynamic Partitioning

For example, the main partition key is timestamp of the record created, and sub key is the region. The old data is managed in storages with big capacity, but not so fast. Hot data is manages in storages which are very fast, but not with so big capacity.

Read "Design of Dynamic Table Partitioning of Distributed Database" more

Created time: 2017-02-06 Last updated time: 2017-02-06

How Dynamic Partition Table Data Grows Up

Traditional Enterprise Relational Database has Partitioning function. It has partitioning information in the schema metadata. Alinous Elastic DB also have it.

And it is designed for Dynamic Table Partition on the distributed remote nodes.

Schema Information for Dynamic Partition Table

Generally, Relational Databases set partition information on creating schema. Actually most of them use CREATE TABLE Statement to set it.

But in this way, it is hard to update the partition keys.

Remote Storage Data Node has Schema Information

Therefore, Alinous Elastic DB has the data in the remote storage node.

Read "How Dynamic Partition Table Data Grows Up" more

Created time: 2017-02-06 Last updated time: 2017-02-05

Storage Records Data Format for Partitioning

The Storage Engine of Distributed Database has records in it. The format have to resolve ACID status with the record's information.

Especially on using partitioning, the format is very important to make the performance high.

Data Format of Storage Records

There are two types of policies in database records management supporting ACID. When the records are changed, one of following operations are done.

  • Update or Delete the record
  • Add a new record with new version

MySQL uses the first method, and PostgreSQL uses second one. The Alinous Elastic DB uses the second method like PostgreSQL.

The actual format is following.

Read "Storage Records Data Format for Partitioning" more

Created time: 2017-02-05 Last updated time: 2017-02-05

Making INSERT SQL on Remote Database

Now, on 2017/02/04, I am making INSERT SQL statement works on distributed database. Now making the framework of executing distributed queries.

How INSERT SQL works on Distributed DB using shard map

The execution of INSERT SQL statement is done by following order.

Execute INSERT SQL Statement on Distributed Nodes

Execute SQL Statement on Transaction Engine

At first, SQL Engine Starts Transaction. Then it access Trx Monitor to get current node clusters and scheme version. This versions make it possible to detect the change of schema and clusters.

The INSERT SQL Statement is handled without accessing other distributed nodes, the ACID logs in the Transaction Engine are changed before commit  it.

Read "Making INSERT SQL on Remote Database" more

Created time: 2017-02-04 Last updated time: 2017-02-04

Started to create SQL on remote nodes

Today, Jan 29th 2017, I've created the CREATE TABLE SQL statement. Before that, I have been creating main framework of the distributed database.

It takes a lot of time, but finally, I come to be able to implement actual SQL functions.

Building Distributed Database Framework

The software framework is very important, especially, on making distributed database.

That is because it has a lot of bottle neck in performance.

  • Network access to synchronize node's status
  • Locking for distributed transaction
  • The role of distributed database components, and when to work

MySQL's storage engine is like this database software, but mine has region manager to support TABLE LOCK between SQL Transaction Engine and Storage Nodes.

Read "Started to create SQL on remote nodes" more

Created time: 2017-01-29 Last updated time: 2017-01-29

1 2