当ブログをご覧のみなさまこんにちは。さくらインターネット研究所の大久保です。
前回に引き続き、今回もFlareを取り上げたいと思います。前回はFlareをインストールし、とりあえず1台のサーバで動作させるところまでを説明しましたので、今回は複数台のサーバで動作させ、分散KVSの仕組みについてご紹介したいと思います。
Flareにおけるデータ分散アーキテクチャ
まず始めにFlareではどのようにデータを分散しているのか説明します。
Flareは分散KVSのソフトウェア一つで、複数のサーバにデータが分散して格納されます。Flareでのデータ分散の概念を表した図を以下に示します。
動作は以下の通りです。
- データの格納先は、キーのハッシュ値を元にサーバのグループであるパーティションという単位で分散し、いづれか1つのパーティションに格納されます。
- Flareの動作中にパーティションの追加はできますが、削除はできません。
- パーティション内にはマスターノードが1台存在し、データのレプリケーションを行う複数台のスレーブノードを置くことができ、冗長化を図ることができます。
- スレーブノードは任意のタイミングで追加、削除が可能です。
- マスターノードがダウンすると、選ばれた1台のスレーブノードがマスターに昇格します。
- データのsetはマスターノードのみで行われまが、データのgetは指定したbalance値の比率に応じてスレーブノードでも処理されます。
今回の実験サーバ構成
今回は以下のような計6台のサーバ構成で実験を行います。
| 役割 | IPアドレス | |
| インデックスサーバ | 192.168.13.21 | |
| ストレージサーバ | パーティション0 マスター | 192.168.13.22 |
| パーティション0 スレーブ | 192.168.13.23 | |
| パーティション1 マスター | 192.168.13.24 | |
| パーティション1 スレーブ | 192.168.13.25 | |
| プロキシサーバ | 192.168.13.26 | |
各役割のサーバは以下の働きをします。
- インデックスサーバ
Flareクラスタ内に1台存在し、各ノードの死活監視、各ノードへのサーバリストの通知、各ノードへの指示を行います。インデックスサーバの冗長化は、2010年5月現在のバージョン1.0.9では実装されていません。 - ストレージサーバ
データを格納します。上記の通りマスター、スレーブの2種類が存在します。 - プロキシサーバ
memcachedプロトコルをアプリケーションに提供し、リクエストをFlareクラスタへ中継します。
サーバ設定
Step1. インデックスサーバの設定と起動
以下の設定ファイルを/home/admin/flare/flarei.confとして保存します。
data-dir = /home/admin/flare log-facility = local0 server-name = 192.168.13.21 monitor-threshold=3 monitor-interval=1 monitor-read-timeout=1000
ここでは、ストレージサーバ障害発生時の切り替わりを早めるためタイマーの値を一部変更しています。なお、オプションの意味は以下URLのFlareのページに掲載されていますので、詳しくはこちらを参照ください。
http://labs.gree.jp/Top/OpenSource/Flare/Document/Reference.html
インデックスサーバ(flarei)を起動します。
% /usr/bin/flarei -f /home/admin/flare/flarei.conf --daemonize
Step2. ストレージサーバ・プロキシサーバ
ストレージサーバ・プロキシサーバの5台では、以下の設定を/home/admin/flare/flared.confとして保存します。server-nameは自身のIPアドレスを指定しますが、以下ではパーティション0 マスター(192.168.13.22)を例にした設定を示します。
data-dir = /home/admin/flare
log-facility = local0
storage-bucket-size = 16777216
index-server-name = 192.168.13.21
server-name = 192.168.13.22
※ 青文字で示した部分は各サーバのIPアドレスに変更する。
各サーバにて、flaredを起動します。
% /usr/bin/flared -f /home/admin/flare/flared.conf --daemonize
Step3. 各サーバの役割設定
インデックスサーバにtelnet接続し、各サーバの役割を設定します。設定前の初期状態では以下のように全てプロキシ状態になっているかと思います。
% telnet localhost 12120 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. stats nodes STAT 192.168.13.22:12121:role proxy STAT 192.168.13.22:12121:state active STAT 192.168.13.22:12121:partition -1 STAT 192.168.13.22:12121:balance 0 STAT 192.168.13.22:12121:thread_type 16 STAT 192.168.13.23:12121:role proxy STAT 192.168.13.23:12121:state active STAT 192.168.13.23:12121:partition -1 STAT 192.168.13.23:12121:balance 0 STAT 192.168.13.23:12121:thread_type 17 STAT 192.168.13.24:12121:role proxy STAT 192.168.13.24:12121:state active STAT 192.168.13.24:12121:partition -1 STAT 192.168.13.24:12121:balance 0 STAT 192.168.13.24:12121:thread_type 19 STAT 192.168.13.25:12121:role proxy STAT 192.168.13.25:12121:state active STAT 192.168.13.25:12121:partition -1 STAT 192.168.13.25:12121:balance 0 STAT 192.168.13.25:12121:thread_type 18 STAT 192.168.13.26:12121:role proxy STAT 192.168.13.26:12121:state active STAT 192.168.13.26:12121:partition -1 STAT 192.168.13.26:12121:balance 0 STAT 192.168.13.26:12121:thread_type 20 END
各サーバの役割を設定します。
※ 192.168.13.22をパーティション0のマスターに設定します。 node role 192.168.13.22 12121 master 1 0 OK ※ 192.168.13.23をパーティション0のスレーブに設定します。 node role 192.168.13.23 12121 slave 1 0 OK node role 192.168.13.23 12121 slave 1 0 OK ※ 192.168.13.24をパーティション1のマスターに設定します。 node role 192.168.13.24 12121 master 1 1 OK ※ 192.168.13.24をactiveにします。 node state 192.168.13.24 12121 active OK ※ 192.168.13.25をパーティション1のスレーブに設定します。 node role 192.168.13.25 12121 slave 1 1 OK node role 192.168.13.25 12121 slave 1 1 OK
ここでは、全てbalance値(getクエリがこの値で重み付けされる)はデフォルトの1に設定していますが、スレーブサーバを設定する際、すぐにgetクエリを受けないように設定直後はbalance値が0となります。そこで、もう一度同じコマンドでbalance値を変更する必要があります。
また、パーティション1のマスターを追加した後もすぐにはactiveにならず、一旦prepare状態になりますので、node stateコマンドでactive状態に設定します。
なお、コマンドの意味は以下URLのFlareのページに掲載されていますので、詳しくはこちらを参照ください。
http://labs.gree.jp/Top/OpenSource/Flare/Document/Reference/Compatibility.html
正常に設定が完了すると以下のような状態になります。赤文字のところが役割とパーティションを表していますので、これが正しく設定されたか確認します。
stats nodes STAT 192.168.13.22:12121:role master STAT 192.168.13.22:12121:state active STAT 192.168.13.22:12121:partition 0 STAT 192.168.13.22:12121:balance 1 STAT 192.168.13.22:12121:thread_type 16 STAT 192.168.13.23:12121:role slave STAT 192.168.13.23:12121:state active STAT 192.168.13.23:12121:partition 0 STAT 192.168.13.23:12121:balance 1 STAT 192.168.13.23:12121:thread_type 17 STAT 192.168.13.24:12121:role master STAT 192.168.13.24:12121:state active STAT 192.168.13.24:12121:partition 1 STAT 192.168.13.24:12121:balance 1 STAT 192.168.13.24:12121:thread_type 19 STAT 192.168.13.25:12121:role slave STAT 192.168.13.25:12121:state active STAT 192.168.13.25:12121:partition 1 STAT 192.168.13.25:12121:balance 1 STAT 192.168.13.25:12121:thread_type 18 STAT 192.168.13.26:12121:role proxy STAT 192.168.13.26:12121:state active STAT 192.168.13.26:12121:partition -1 STAT 192.168.13.26:12121:balance 0 STAT 192.168.13.26:12121:thread_type 20 END
これでサーバの準備ができました。
動作確認
プロキシサーバの12121番ポートに接続します。以下の通り正常に動作していることが確認できます。
% telnet localhost 12121 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. set test1 0 0 5 abcde STORED set test2 0 0 5 12345 STORED get test1 VALUE test1 0 5 abcde END get test2 VALUE test2 0 5 12345 END
次にプロキシサーバにて以下のperlスクリプトを走らせ、100万レコードを格納してみます。
#!/usr/bin/env perl
use Cache::Memcached;
my $mem = Cache::Memcached->new({'servers' => ["127.0.0.1:12121"]});
$value = "01234567890123456789012345678901234567890123456789";
foreach (0 .. 999999) {
$key = sprintf("test%08d", $_);
unless ($mem->set($key, $value)) {
warn "set failed $_\n";
sleep(1);
redo;
}
}
コードをご覧いただければ一目瞭然ですが、格納するキーと値は以下の通りです。
| キー | 値 |
| test00000000〜test00999999 | 01234567890123456789012345678901234567890123456789 (50bytes) |
スクリプトを実行すると以下のようにデータが格納されていることが確認できます。
% telnet localhost 12121 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. get test00000000 VALUE test00000000 0 50 01234567890123456789012345678901234567890123456789 END get test00999999 VALUE test00999999 0 50 01234567890123456789012345678901234567890123456789 END
各ストレージサーバに格納されているレコード数は、ストレージサーバに接続してstatsコマンドを入力すると、curr_itemsというフィールドで取得することができます。以下は4台のストレージサーバからnetcat(nc)コマンドを用いて情報を取得した例です。
% echo 'stats'|nc 192.168.13.22 12121|grep curr_items STAT curr_items 500001 % echo 'stats'|nc 192.168.13.23 12121|grep curr_items STAT curr_items 500001 % echo 'stats'|nc 192.168.13.24 12121|grep curr_items STAT curr_items 500001 % echo 'stats'|nc 192.168.13.25 12121|grep curr_items STAT curr_items 500001
上記のように2つのパーテションにきれいにデータが分散しており、マスターとスレーブでレコード数が等しくなっていることが確認できます。
まとめ
今回は4台のストレージサーバでレコードを分散・冗長化する方法について説明しました。次回以降はさらにサーバを増やした場合どのようになるか？パフォーマンスはどうか？という点について調べていきます。
[...] This post was mentioned on Twitter by 鷲北 賢. 鷲北 賢 said: さくらインターネット研究所ブログ Flareを使う(複数台構成) http://research.sakura.ad.jp/2010/05/12/flare-servers1/ [...]
[...] [Web]Flareを使う(複数台構成) ? さくらインターネット研究所 [...]
I have been surfing online more than three hours today, yet I never found any interesting article
like yours. It’s pretty worth enough for me. Personally, if all webmasters and bloggers made good content as you did, the net will be a lot more useful than ever before.|
I couldn’t refrain from commenting. Well written!|
I’ll right away grab your rss as I can not to find your e-mail subscription hyperlink or e-newsletter service. Do you’ve any?
Kindly allow me know so that I may subscribe.
Thanks.|
It is appropriate time to make some plans for the future and it’s time to be happy. I’ve read
this post and if I could I desire to suggest you few interesting things or
suggestions. Maybe you can write next articles referring to this article.
I desire to read even more things about it!
|
It is perfect time to make some plans for the future and it
is time to be happy. I have read this submit and if I may I wish to counsel you few attention-grabbing issues or suggestions.
Perhaps you could write next articles referring to this article.
I wish to read more things about it!|
I have been browsing on-line greater than 3 hours these days, yet I
by no means found any fascinating article like yours.
It is beautiful worth enough for me. Personally, if all webmasters and bloggers made just right content material as you did, the internet might
be much more useful than ever before.|
Ahaa, its nice conversation on the topic of this article
here at this webpage, I have read all that, so at this time me also commenting at this place.
|
I am sure this article has touched all the internet people, its really really nice piece of writing
on building up new website.|
Wow, this post is pleasant, my sister is analyzing these kinds of
things, therefore I am going to convey her.
|
bookmarked!!, I love your web site!|
Way cool! Some very valid points! I appreciate you penning this
post plus the rest of the site is also very good.
|
Hi, I do think this is an excellent web site.
I stumbledupon it ;) I will come back yet again since I saved as a favorite it.
Money and freedom is the best way to change, may you be rich and continue to help other people.
|
Woah! I’m really enjoying the template/theme of this site. It’s simple,
yet effective. A lot of times it’s very difficult to get that “perfect balance” between usability and visual appearance. I must say that you’ve done a amazing job with
this. In addition, the blog loads extremely quick
for me on Opera. Excellent Blog!|
These are genuinely impressive ideas in concerning blogging.
You have touched some nice factors here. Any way keep
up wrinting.|
Everyone loves what you guys are usually up too. This type of clever work and coverage!
Keep up the very good works guys I’ve you guys to blogroll.|
Howdy! Someone in my Facebook group shared this website with us so I came to give it a look. I’m definitely enjoying the information.
I’m bookmarking and will be tweeting this to my followers! Terrific blog and excellent design and style.|
I love what you guys are up too. This type of clever work and reporting! Keep up the excellent works guys I’ve
added you guys to my own blogroll.|
Hi there would you mind sharing which blog platform you’re using? I’m looking to start my own blog soon but I’m having a tough time making a decision between BlogEngine/Wordpress/B2evolution and Drupal. The reason I ask is because your design and style seems different then most blogs and I’m looking
for something unique. P.S Sorry for getting off-topic but I had to ask!
|
Hi would you mind letting me know which hosting company you’re utilizing? I’ve loaded your blog in 3 different web browsers and I must say this blog loads a lot quicker
then most. Can you suggest a good web hosting provider at a
fair price? Kudos, I appreciate it!|
I really like it when individuals come together and share thoughts.
Great website, keep it up!|
Thank you for the auspicious writeup. It in fact was a amusement account it.
Look advanced to more added agreeable from you!
However, how can we communicate?|
Hey just wanted to give you a quick heads up. The text in
your article seem to be running off the screen in Ie.
I’m not sure if this is a format issue or something to do with browser compatibility but I thought I’d post to let you know.
The layout look great though! Hope you get the issue
fixed soon. Thanks|
This is a topic which is close to my heart… Best
wishes! Where are your contact details though?|
It’s very effortless to find out any matter on net as compared to textbooks, as I found this paragraph at this web page.|
Does your website have a contact page? I’m having a tough time locating
it but, I’d like to shoot you an email. I’ve got some creative ideas
for your blog you might be interested in hearing. Either way, great blog and
I look forward to seeing it grow over time.|
Hi! I’ve been reading your blog for a while now and finally got the bravery to go ahead and give you a shout out from Houston Texas! Just wanted to tell you keep up the good work!|
Greetings from Colorado! I’m bored to tears at work so I decided to browse your
site on my iphone during lunch break. I really like the knowledge you present here and can’t wait to take a look when I get home. I’m
surprised at how fast your blog loaded on my mobile .
. I’m not even using WIFI, just 3G .. Anyhow, amazing site!|
Its such as you read my thoughts! You appear to grasp a lot approximately this, like you wrote the book in it or something. I think that you simply can do with some percent to force the message house a bit, but instead of that, this is great blog. An excellent read. I’ll
certainly be back.|
I visited many blogs but the audio feature for audio songs existing at this
web page is in fact fabulous.|
Hi, i read your blog occasionally and i own a similar one and i
was just wondering if you get a lot of spam feedback?
If so how do you reduce it, any plugin or anything you can advise?
I get so much lately it’s driving me crazy so any help is very much appreciated.|
Greetings! Very useful advice in this particular article! It is the little changes that produce the most important changes. Thanks for sharing!|
I seriously love your blog.. Excellent colors & theme. Did you make this amazing site yourself? Please reply back as I’m wanting to create
my own personal blog and would love to learn where you got this from or just what the theme is called.
Appreciate it!|
Hi there! This blog post could not be written much better!
Looking at this post reminds me of my previous roommate!
He continually kept talking about this. I most certainly will forward this article to him.
Fairly certain he will have a great read. Thank you for
sharing!|
Incredible! This blog looks just like my old one! It’s on a entirely different subject but it has pretty much the same page layout and design. Outstanding choice of colors!|
There’s certainly a great deal to learn about this topic.
I really like all of the points you have made.|
You’ve made some good points there. I checked on the internet to find out more about the issue and found most individuals will go along with your views on this website.|
What’s up, I check your new stuff on a regular basis. Your
writing style is witty, keep doing what you’re doing!|
I simply couldn’t leave your website prior to suggesting
that I really loved the standard information a
person supply on your guests? Is going to be again frequently
in order to check out new posts|
I want to to thank you for this excellent read!! I certainly enjoyed every little
bit of it. I have got you book-marked to check out new stuff you post厊