分布式缓存系统 Xixibase
jopen 13年前
<p>Xixibase是一个高性能，跨平台的分布式缓存系统。Xixibase server 采用 C++ 实现，底层网络库采用的是Boost Asio。</p> <h3>Xixibase 主要特点：</h3> <p>1. 实现'Local Cache'功能, 当客户端打开'Local Cache'选项, 客户端可以将数据同时存储在Server 端和本地，并且保证本地数据和Server 端的数据的一致性。当Server 端的数据被更新、删除或者过期，本地的数据会立即被删除，下次再对该数据的访问，请求会直接到达Server 。<br /> 'Local Cache'的优点：<br /> a. 极大地提高了性能：特别是对那些读操作远远大于写操作的数据，可以提供1000万以上的TPS.参考：<a href="/misc/goto?guid=4958195306494588869">http://xixibase.googlecode.com/svn/trunk/benchmark/java/benchmark2.html</a> <br /> b. 提高了应用的服务质量和用户体验：在'Local Cache'命中的情况下，请求不需要通过网络到远程Cache Server上查询，减少了总体的处理的时间，提高用户体验。<br /> c. 降低了Cache Server 的压力，原先需要10台Cache Server ，现在可能只要5台就够了。可降低成本，保护环境。<br /> d. 降低网络流量，减少网络设备的成本。<br /> e. 提高整个Cache 系统的总体吞吐量。综合a, c, d三点，系统的总体吞吐量会得到极大的提升。</p> <p>2. 支持虚拟化, 采用分Group的方法，不同的应用可以使用不同的GroupID，这样当不同的应用在访问同一个Cache Server时，不会相互干扰。<br /> a. GroupID类似与命名空间，每个应用不需要考虑：某个Key 是否会和其它应用的Key 冲突。<br /> b. 当某个应用要清空自己相关的Cache，不需要一个Key 一个Key 的去Delete ，可以直接调用Flush ，不用担心Flush 会影响其它应用的数据。<br /> c. 可以对每个Group 单独统计读写、存储的信息。</p> <p>3. 支持'MultiAPI', 如multiGet, multeUpdate, multiDelete。测试表明'MultiAPI'在提升性能方面，非常明显。参考：<a href="/misc/goto?guid=4958195307247284344">http://xixibase.googlecode.com/svn/trunk/benchmark/java/benchmark.html</a></p> <p>4. 超高性能. 无论'Local Cache'还是'MultiAPI'都在不同方面，极大地提升了性能。</p> <p>5. 跨平台, Xixibase server由C++实现，底层网络库采用的是Boost Asio。理论上，只要Boost Asio支持的平台，Xixibase server都可以支持。目前，已经测试了Linux, Windows, Mac。</p> <h4>性能：</h4> <p>读写TPS ：150,000以上</p> <p>启动Local Cache高命中情况下，读TPS：10,000,000以上。</p> <p>MultiAPI, TPS: 不同条件下，差异较大，详情参考一下链接：</p> <p>1. 和Memcached的几个Client 比较：<br /> <a href="/misc/goto?guid=4958195307247284344">http://xixibase.googlecode.com/svn/trunk/benchmark/java/benchmark.html</a></p> <p>2. Performance Benchmark：<br /> <a href="/misc/goto?guid=4958195306494588869">http://xixibase.googlecode.com/svn/trunk/benchmark/java/benchmark2.html</a></p> <p>项目：</p> <p><br /> <a href="/misc/goto?guid=4958195309325842019">http://code.google.com/p/xixibase/</a><br /> 包含：Xixibase Server 和Xixibase Java Client</p> <p>目前Client 只实现了Java Client，欢迎有兴趣的朋友可以实现其它平台的Client。<br /> 关于Xixibase Java Client 的使用，test包里有几个例子，大家可以参考一下。</p> <p>有任何问题、建议或者发现Bug可以联系本人。<br /> Java客户端示例：</p> <pre class="brush:java; toolbar: true; auto-links: false;">/* Copyright [2011] [Yao Yuan(yeaya@163.com)] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package com.xixibase.cache; import java.util.ArrayList; import java.util.List; import com.xixibase.cache.multi.MultiDeleteItem; import com.xixibase.cache.multi.MultiUpdateItem; public class CacheClientBench { int start = 1; int runs = 50000; int groupID = 0; String keyBase = "keypublic CacheClientBench(String servers, int start, int runs, int groupID,"; String object = "value"; CacheClient cc; public CacheClientBench(String servers, int start, int runs, int groupID, boolean consistentFlag, int hashingAlg, Integer[] weights) { this.start = start; this.runs = runs; this.groupID = groupID; String[] serverlist = servers.split(","); XixiWeightMap<Integer> weightMap = new XixiWeightMap<Integer>(consistentFlag, hashingAlg); CacheClientManager manager = CacheClientManager.getInstance("CacheClientBench"); manager.setInitConn(1); manager.setNoDelay(false); manager.initialize(serverlist, weights, weightMap); manager.enableLocalCache(); manager.getLocalCache().setMaxCacheSize(128 * 1024 * 1024); cc = manager.createClient(groupID); System.out.println("keybaselen=" + keyBase.length() + " objlen=" + object.length()); } public CacheClientBench(String servers, int start, int runs) { this(servers, start, runs, 0, false, XixiWeightMap.NATIVE_HASH, null); } public boolean runIt() { flush(); boolean ret = true; ret &= set(); ret &= getW(); ret &= getL(); ret &= multiGet(); ret &= delete(); ret &= multiSet(); ret &= multiGetForSet(); ret &= multiDelete(); ret &= multiSet(); ret &= flush(); CacheClientManager.getInstance("CacheClientBench").shutdown(); return ret; } boolean set() { boolean r = true; long begin = System.currentTimeMillis(); for (int i = start; i < start + runs; i++) { long ret = cc.set(keyBase + i, object); if (ret == 0) { r = false; System.out.println("set error index=" + i + " obj=" + object); } } long end = System.currentTimeMillis(); long time = end - begin; System.out.println(runs + " sets: " + time + "ms"); return r; } boolean getW() { boolean r = true; long begin = System.currentTimeMillis(); String obj = ""; for (int i = start; i < start + runs; i++) { obj = (String)cc.getW(keyBase + i); if (!object.equals(obj)) { r = false; System.out.println("getW error index=" + i + " obj=" + obj); } } long end = System.currentTimeMillis(); long time = end - begin; System.out.println(runs + " getW: " + time + "ms"); return r; } boolean getL() { boolean r = true; long begin = System.currentTimeMillis(); Object obj = ""; for (int i = start; i < start + runs; i++) { obj = (String)cc.getL(keyBase + i); if (!object.equals(obj)) { r = false; System.out.println("getL error index=" + i + " obj=" + obj); } } long end = System.currentTimeMillis(); long time = end - begin; System.out.println(runs + " getL: " + time + "ms"); return r; } boolean multiGet() { ArrayList<String> keys = new ArrayList<String>(runs); for (int i = start; i < start + runs; i++) { keys.add(keyBase + i); } long begin = System.currentTimeMillis(); List<CacheItem> m = cc.multiGet(keys); long end = System.currentTimeMillis(); long time = end - begin; int missCount = 0; for (int i = start; i < start + runs; i++) { String key = keyBase + i; if (m.get(i - start) == null) { System.out.println("can not find the key=" + key); missCount++; } } System.out.println(runs + " multiGet: " + time + "ms" + " count=" + m.size() + " missCount=" + missCount); return missCount == 0; } boolean multiSet() { ArrayList<MultiUpdateItem> items = new ArrayList<MultiUpdateItem>(runs); String value = object.toString() + 1; for (int i = start; i < start + runs; i++) { MultiUpdateItem item = new MultiUpdateItem(); items.add(item); item.key = keyBase + i; item.value = value; item.cacheID = 0; } long begin = System.currentTimeMillis(); int count = cc.multiSet(items); long end = System.currentTimeMillis(); long time = end - begin; int missCount = 0; for (int i = 0; i < items.size(); i++) { MultiUpdateItem item = items.get(i); if (item.getNewCacheID() == 0) { missCount++; } } System.out.println(runs + " multiSet: " + time + "ms" + " count=" + count + " missCount=" + missCount); return missCount == 0; } boolean multiGetForSet() { ArrayList<String> keys = new ArrayList<String>(runs); for (int i = start; i < start + runs; i++) { keys.add(keyBase + i); } long begin = System.currentTimeMillis(); List<CacheItem> items = cc.multiGet(keys); long end = System.currentTimeMillis(); long time = end - begin; int missCount = 0; String value = object.toString() + 1; for (int i = start; i < start + runs; i++) { String key = keyBase + i; CacheItem item = items.get(i - start); if (item == null || !item.getValue().equals(value)) { System.out.println("not key=" + key); missCount++; } } System.out.println(runs + " multiGetForSet: " + time + "ms" + " count=" + items.size() + " missCount=" + missCount); return missCount == 0; } boolean multiDelete() { ArrayList<MultiDeleteItem> items = new ArrayList<MultiDeleteItem>(runs); for (int i = start; i < start + runs; i++) { MultiDeleteItem item = new MultiDeleteItem(); items.add(item); item.key = keyBase + i; item.cacheID = 0; } long begin = System.currentTimeMillis(); int count = cc.multiDelete(items); long end = System.currentTimeMillis(); long time = end - begin; System.out.println(runs + " multiDelete: " + time + "ms" + " count=" + count); return true; } boolean delete() { long begin = System.currentTimeMillis(); for (int i = start; i < start + runs; i++) { cc.delete(keyBase + i); } long end = System.currentTimeMillis(); long time = end - begin; System.out.println(runs + " deletes: " + time + "ms"); return true; } boolean flush() { long begin = System.currentTimeMillis(); int count = cc.flush(); long end = System.currentTimeMillis(); long time = end - begin; System.out.println(runs + " flush: " + time + "ms" + " count=" + count); return count == runs; } public static void main(String[] args) { String servers = System.getProperty("hosts"); if (args.length >= 1) { servers = args[0]; } CacheClientBench bench = new CacheClientBench(servers, 1, 50000); bench.runIt(); } }</pre>