postgressql-9.5新特性BRIN (block range index)
jopen
9年前
PostgreSQL 9.5引入的一个全新的索引访问方法BRIN（block range index），这个索引存储了表的连续数据块区间以及对应的数据取值范围。
比如一张表有1000个数据块，我们建议一个BRIN在ID（假设这个表有ID字段）上的索引。
BRIN默认是每128个连续数据块区间存储一个字段取值的区间，所以这个索引的信息量是将1000个数据块划分为几个连续的128个块的区间，然后存储每个区间ID值的取值范围。
那么我们并不能直接从索引中精确匹配要查询的记录，但是通过索引我们可以将查询范围缩小到最小128个连续的数据块（假设我们要找的值落在这个区间）。
以上是BRIN大概的原理，那么BRIN可以用在什么场景呢？
一个非常好的场景是流式日志数据，比如用户行为，大批量的数据按时间顺序不停的插入数据表。
我们如果要按照时间来访问这样的数据，以往我们需要创建BTREE索引，可以范围查询或者精确匹配。但是BTREE索引需要存储的信息量较大，如果数据量很大，索引也很庞大。
BRIN的话，索引可以变得很小，而且因为数据是按照时间顺序插入的，所以BRIN的信息量也很大，因为每个连续的数据块区间存储的时间范围和其他连续的数据块区间独立性很好，即不会出现大量数据交叉，如果有大量较差，那么使用BRIN检索还不如全表扫描。BRIN可认为是全表扫描的切片，如果数据值分布和物理值分布的相关性很好，那么BRIN无疑是非常好的选择。
1、做好两张实验表，每张表有1000万数据量。第一张表ID字段大小顺序和物理存储顺序相关性是1，第二张表ID字段大小顺序和物理存储顺序几乎没有物理相关性。参考：http://www.postgresql.org/docs/9.5/static/view-pg-stats.html
postgres=# create table a(id int, msg text); CREATE TABLE postgres=# create table b(id int, msg text); CREATE TABLE postgres=# insert into a select generate_series(1, 10000000), md5(random()::text); INSERT 0 10000000 postgres=# insert into b select id, md5(random()::text) from generate_series(1, 10000000) as t(id) order by random(); INSERT 0 10000000 postgres=# analyze a; ANALYZE postgres=# analyze b; ANALYZE postgres=# select correlation from pg_stats where tablename = 'a' and attname = 'id'; correlation ------------- 1 (1 row) postgres=# select correlation from pg_stats where tablename = 'b' and attname = 'id'; correlation ------------- 0.00601343 (1 row) postgres=# create index idx_a_id on a using brin(id); CREATE INDEX postgres=# create index idx_b_id on b using brin(id); CREATE INDEX
2、实验两张表的访问效率。说明brin索引适合字段大小顺序和物理存储顺序相关性大的表。
postgres=# explain analyze select * from a where id >1000 and id <5000; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------ Bitmap Heap Scan on a (cost=62.53..14756.81 rows=4540 width=37) (actual time=0.662..2.952 rows=3999 loops=1) Recheck Cond: ((id > 1000) AND (id < 5000)) Rows Removed by Index Recheck: 11361 Heap Blocks: lossy=128 -> Bitmap Index Scan on idx_a_id (cost=0.00..61.40 rows=4540 width=0) (actual time=0.199..0.199 rows=1280 loops=1) Index Cond: ((id > 1000) AND (id < 5000)) Planning time: 0.050 ms Execution time: 3.581 ms (8 rows) postgres=# explain analyze select * from b where id >1000 and id <5000; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------- - Bitmap Heap Scan on b (cost=51.12..11509.18 rows=3426 width=37) (actual time=5.384..1147.752 rows=3999 loops=1) Recheck Cond: ((id > 1000) AND (id < 5000)) Rows Removed by Index Recheck: 9965281 Heap Blocks: lossy=83078 -> Bitmap Index Scan on idx_b_id (cost=0.00..50.26 rows=3426 width=0) (actual time=5.174..5.174 rows=832000 loops=1) Index Cond: ((id > 1000) AND (id < 5000)) Planning time: 0.044 ms Execution time: 1148.616 ms (8 rows)
3、实验顺序扫描和brin扫描，说明字段大小顺序和物理存储顺序相关性小的表，不如顺序扫描快。
postgres=# explain analyze select * from b where id >1000 and id <5000; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------- - Bitmap Heap Scan on b (cost=51.12..11509.18 rows=3426 width=37) (actual time=5.384..1147.752 rows=3999 loops=1) Recheck Cond: ((id > 1000) AND (id < 5000)) Rows Removed by Index Recheck: 9965281 Heap Blocks: lossy=83078 -> Bitmap Index Scan on idx_b_id (cost=0.00..50.26 rows=3426 width=0) (actual time=5.174..5.174 rows=832000 loops=1) Index Cond: ((id > 1000) AND (id < 5000)) Planning time: 0.044 ms Execution time: 1148.616 ms (8 rows) postgres=# set enable_bitmapscan = off; SET postgres=# explain analyze select * from b where id >1000 and id <5000; QUERY PLAN --------------------------------------------------------------------------------------------------------- Seq Scan on b (cost=0.00..233334.00 rows=3426 width=37) (actual time=0.196..961.569 rows=3999 loops=1) Filter: ((id > 1000) AND (id < 5000)) Rows Removed by Filter: 9996001 Planning time: 0.058 ms Execution time: 962.425 ms (5 rows)
4，比较btree索引和brin索引。brin索引不如btree索引效率高。优点是brin索引空间占用小。
postgres=# \d a Table "public.a" Column | Type | Modifiers --------+---------+----------- id | integer | msg | text | Indexes: "idx_a_id" brin (id) postgres=# explain analyze select * from a where id >1000 and id <5000; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------ Bitmap Heap Scan on a (cost=62.53..14756.81 rows=4540 width=37) (actual time=0.662..2.952 rows=3999 loops=1) Recheck Cond: ((id > 1000) AND (id < 5000)) Rows Removed by Index Recheck: 11361 Heap Blocks: lossy=128 -> Bitmap Index Scan on idx_a_id (cost=0.00..61.40 rows=4540 width=0) (actual time=0.199..0.199 rows=1280 loops=1) Index Cond: ((id > 1000) AND (id < 5000)) Planning time: 0.050 ms Execution time: 3.581 ms (8 rows) postgres=# drop index idx_a_id ; DROP INDEX postgres=# create index idx_a_id on a using btree(id); CREATE INDEX postgres=# explain analyze select * from a where id >1000 and id <5000; QUERY PLAN --------------------------------------------------------------------------------------------------------------------- Index Scan using idx_a_id on a (cost=0.43..186.45 rows=4601 width=37) (actual time=0.014..1.190 rows=3999 loops=1) Index Cond: ((id > 1000) AND (id < 5000)) Planning time: 0.174 ms Execution time: 1.742 ms (4 rows) postgres=# \di+ idx_a_id_b* List of relations Schema | Name | Type | Owner | Table | Size | Description --------+----------------+-------+----------+-------+--------+------------- public | idx_a_id_brin | index | postgres | a | 56 kB | public | idx_a_id_btree | index | postgres | a | 214 MB | (2 rows)
5、结论，brin索引适合字段大小顺序和物理存储顺序相关性大的表，如流水表的时间字段，日志表的时间字段，自增长字段。