performance_schema 诊断表锁,行锁,全局读锁,DML锁
标签: performance_schema 诊断表锁,行锁,全局读锁,DML锁 MySQL博客 51CTO博客
2023-07-01 18:24:09 210浏览
全局读锁
全局锁就是对整个数据库加锁。是数据库实例层级的锁,MySQL提供了一个加全局读锁语句:Flush tables with read lock;
如果需要让整个库处于只读状态的时候,可以使用这个语句,在其之后执行的其他线程的DML,DDL语句都会被阻塞。
create database itpuxdb1;
use itpuxdb1;
create table itpuxbak11 (id int primary key,name varchar(40));
insert into itpuxbak11 values (1,'itpux111'),(2,'itpux112'),(3,'itpux113'),(4,'itpux114'),(5,'itpux115');
commit;
select * from itpuxbak11;
谁持有全局读锁?
Waiting for global read lock < -- flush table with read lock
会话1:
mysql> flush table with read lock;
Query OK, 0 rows affected (0.00 sec)
会话2:
mysql> select * from itpuxdb1.itpuxbak11 limit 1;
+----+----------+
| id | name |
+----+----------+
| 1 | itpux111 |
+----+----------+
1 row in set (0.00 sec)
mysql> update itpuxdb1.itpuxbak11 set name='xiaomi' where id =1;
//DML被阻塞
会话3进行性能问题排查,没有任何有效信息:
mysql> select * from information_schema.innodb_trx;
+-----------------+-----------+---------------------+-----------------------+------------------+------------+---------------------+----------------------------------------------------------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| trx_id | trx_state | trx_started | trx_requested_lock_id | trx_wait_started | trx_weight | trx_mysql_thread_id | trx_query | trx_operation_state | trx_tables_in_use | trx_tables_locked | trx_lock_structs | trx_lock_memory_bytes | trx_rows_locked | trx_rows_modified | trx_concurrency_tickets | trx_isolation_level | trx_unique_checks | trx_foreign_key_checks | trx_last_foreign_key_error | trx_adaptive_hash_latched | trx_adaptive_hash_timeout | trx_is_read_only | trx_autocommit_non_locking |
+-----------------+-----------+---------------------+-----------------------+------------------+------------+---------------------+----------------------------------------------------------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| 422212121904976 | RUNNING | 2023-07-01 10:51:31 | NULL | NULL | 0 | 14 | update itpuxdb1.itpuxbak11 set name='xiaomi' where id =1 | NULL | 0 | 0 | 0 | 1136 | 0 | 0 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 0 | 0 | 0 |
+-----------------+-----------+---------------------+-----------------------+------------------+------------+---------------------+----------------------------------------------------------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
mysql> select * from sys.innodb_lock_waits;
Empty set, 3 warnings (0.01 sec)
mysql> show engine innodb status\G
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:
=====================================
2023-07-01 10:54:13 0x7fffd87f1700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 51 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 18 srv_active, 0 srv_shutdown, 190924 srv_idle
srv_master_thread log flush and writes: 190942
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 14
OS WAIT ARRAY INFO: signal count 14
RW-shared spins 0, rounds 10, OS waits 5
RW-excl spins 0, rounds 4, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 10.00 RW-shared, 4.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 3344
Purge done for trx's n:o < 3342 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 422212121906800, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422212121905888, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
会话4:
通过以上的常规手段没有任何用处,有GDB经验的人会开始使用gdb、strace、pstack等命令查看mysql的栈线程信息。
MySQL在5.7开始提供一个performance_schema.metadata_locks显示各种Server层的锁信息(包括全局读锁和DML锁信息等)。
SQL语句中, owner_thread_id != sys.ps_thread_id(connection_id())表示非本连接的其他连接。
mysql> select * from performance_schema.metadata_locks where owner_thread_id != sys.ps_thread_id(connection_id())\G
Empty set (0.00 sec)
通过performance_schema.metadata_locks排查谁有全局读锁。
全局读锁在该表中通常记录中同一个会话的OBJECT_TYPE为GLOBAL和COMMIT,LOCK_TYPE都为SHARED的两把显式锁。
看到第一行:
#使用会话4继续查询:
mysql> show processlist;
+----+------+---------------------+----------+---------+------+------------------------------+----------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+---------------------+----------+---------+------+------------------------------+----------------------------------------------------------+
| 13 | root | localhost | itpuxdb1 | Sleep | 484 | | NULL |
| 14 | root | localhost | NULL | Query | 386 | Waiting for global read lock | update itpuxdb1.itpuxbak11 set name='xiaomi' where id =1 |
| 15 | root | localhost | NULL | Query | 0 | starting | show processlist |
| 16 | root | 192.168.2.103:64123 | NULL | Sleep | 313 | | NULL |
+----+------+---------------------+----------+---------+------+------------------------------+----------------------------------------------------------+
4 rows in set (0.00 sec)
mysql> select sys.ps_thread_id(13);
+----------------------+
| sys.ps_thread_id(13) |
+----------------------+
| 40 |
+----------------------+
1 row in set (0.00 sec)
mysql> select sys.ps_thread_id(14);
+----------------------+
| sys.ps_thread_id(14) |
+----------------------+
| 41 |
+----------------------+
1 row in set (0.00 sec)
##使用
##select a.thread_id,b.processlist_id,a.SQL_text from performance_schema.events_statements_current a join performance_schema.threads b on a.thread_id=b.thread_id;
##替代select sys.ps_thread_id(14);会更有效
可以看到:
connection_id为13的连接,内部线程号为40,对应为持有全局锁的线程。
connection_id为14的连接,内部线程号为41,对应为等待锁的线程。
此时可以通过kill掉13号线程解锁。
mysql> kill 13;
Query OK, 0 rows affected (0.00 sec)
##此时 拥有全局读锁的连接被杀死
##此时 解锁了全剧读锁,所以connection_id为14的连接,内部线程号为41,可以顺利执行update,也执行完毕释放了锁。
##此时 performance_schema.metadata_locks为空。
mysql> update itpuxdb1.itpuxbak11 set name='xiaomi' where id =1;
Query OK, 1 row affected (13 min 38.28 sec)
Rows matched: 1 Changed: 1 Warnings: 0
DML锁
metadata lock即MDL,是⽤于保护MySQL内部对象的元数据,MySQL通过MDL保护DDL和DML的并发。
MDL在MySQL5.5版本引⼊,在此之前MySQL对于元数据的保护仅仅是语句级别的,引⼊MDL后,MySQL对于元数据的保护上升为事务级别的。
谁持有DML锁?
Waiting for table metadata lock
准备工作
会话1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update itpuxdb1.itpuxbak11 set name='panle' where id =3;
Query OK, 1 row affected (0.05 sec)
Rows matched: 1 Changed: 1 Warnings: 0
##执行update但是不提交
会话2:
mysql> alter table itpuxdb1.itpuxbak11 add index ind_name(name);
##被阻塞
会话3分析性能问题:
mysql> show processlist;
+----+------+---------------------+------+---------+------+---------------------------------+----------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+---------------------+------+---------+------+---------------------------------+----------------------------------------------------------+
| 16 | root | 192.168.2.103:64123 | NULL | Sleep | 1237 | | NULL |
| 17 | root | localhost | NULL | Sleep | 146 | | NULL |
| 18 | root | localhost | NULL | Query | 18 | Waiting for table metadata lock | alter table itpuxdb1.itpuxbak11 add index ind_name(name) |
| 19 | root | localhost | NULL | Query | 0 | starting | show processlist |
+----+------+---------------------+------+---------+------+---------------------------------+----------------------------------------------------------+
4 rows in set (0.00 sec)
额外提一个点,假设此时有会话4,会话5,会话6,会话7…查询这张表:
mysql> select * from itpuxdb1.itpuxbak11;
mysql> select id from itpuxdb1.itpuxbak11;
## 如果越多的人在此时执行查这张表,就会累积越多的Waiting for table metadata lock ,这会造成MySQL阻塞或最终崩溃
会话3继续分析故障问题:
方法一:
##查看执行完成但是没提交的表:
mysql> select d.trx_started ,a.thread_id,b.processlist_id,a.SQL_text from performance_schema.events_statements_current a join performance_schema.threads b on a.thread_id=b.thread_id join information_schema.processlist c on b.processlist_id=c.id join information_schema.innodb_trx d on c.id=d.trx_mysql_thread_id order by d.trx_started;
+---------------------+-----------+----------------+-----------------------------------------------+
| trx_started | thread_id | processlist_id | SQL_text |
+---------------------+-----------+----------------+-----------------------------------------------+
| 2023-07-01 11:11:32 | 44 | 17 | select * from itpuxdb1.itpuxbak11 where id =3 |
+---------------------+-----------+----------------+-----------------------------------------------+
1 row in set (0.01 sec)
可以看到17号连接执行了update并没提交或者回滚。
方法二:
在MySQL5.7之前,不能直观的看到谁持有MDL锁,(除非使用GDB),现在可以查performance_schema.metadata_locks得知MDL的消息。
select a.thread_id,b.processlist_id,a.SQL_text from performance_schema.events_statements_current a join performance_schema.threads b on a.thread_id=b.thread_id;
mysql> select a.thread_id,b.processlist_id,a.SQL_text from performance_schema.events_statements_current a join performance_schema.threads b on a.thread_id=b.thread_id;
+-----------+----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| thread_id | processlist_id | SQL_text |
+-----------+----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 43 | 16 | SELECT STATE AS `Status`, ROUND(SUM(DURATION),7) AS `Duration`, CONCAT(ROUND(SUM(DURATION)/0.001316*100,3), '') AS `Percentage` FROM INFORMATION_SCHEMA.PROFILING WHERE QUERY_ID=2 GROUP BY SEQ, STATE ORDER BY SEQ |
| 44 | 17 | select * from itpuxdb1.itpuxbak11 where id =3 |
| 45 | 18 | alter table itpuxdb1.itpuxbak11 add index ind_name(name) |
| 46 | 19 | select a.thread_id,b.processlist_id,a.SQL_text from performance_schema.events_statements_current a join performance_schema.threads b on a.thread_id=b.thread_id |
+-----------+----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
select * from performance_schema.metadata_locks where owner_thread_id != sys.ps_thread_id(connection_id());
DML锁总结
这里我认为值用方法二进行查询能够完全了解这些SQL在做什么内容的操作,并且查出来MDL的锁是被谁持有的。很好用的方法。
表级锁
表级锁是对当前操作的整张表加锁,MYISAM与INNODB都支持表级锁定。表级锁定分为表共享读锁(共享锁)与表独占写锁(排他锁)。
谁持有表级锁?
Waiting for table metadata lock
会话1:
use itpuxdb1;
lock table itpuxbak11 read;
Query OK, 0 rows affected (0.00 sec)
会话2:
mysql> update itpuxdb1.itpuxbak11 set name='jiangfei' where id =3;
##被阻塞
会话3:
mysql> show processlist;
+----+------+-----------+----------+---------+------+---------------------------------+-----------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+-----------+----------+---------+------+---------------------------------+-----------------------------------------------------------+
| 20 | root | localhost | itpuxdb1 | Sleep | 56 | | NULL |
| 22 | root | localhost | NULL | Query | 12 | Waiting for table metadata lock | update itpuxdb1.itpuxbak11 set name='jiangfei' where id =3 |
| 23 | root | localhost | NULL | Query | 0 | starting | show processlist |
+----+------+-----------+----------+---------+------+---------------------------------+-----------------------------------------------------------+
3 rows in set (0.00 sec)
可以发现,update语句在等待MDL锁。但是sleep线程的SQL语句无法确定。
既然是MDL锁,即查看performance_schema.metadata_locks
select a.thread_id,b.processlist_id,a.SQL_text from performance_schema.events_statements_current a join performance_schema.threads b on a.thread_id=b.thread_id;
+-----------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| thread_id | processlist_id | SQL_text |
+-----------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 47 | 20 | lock table itpuxbak11 read |
| 49 | 22 | update itpuxdb1.itpuxbak11 set name='jiangfei' where id =3 |
| 50 | 23 | select a.thread_id,b.processlist_id,a.SQL_text from performance_schema.events_statements_current a join performance_schema.threads b on a.thread_id=b.thread_id |
+-----------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
select * from performance_schema.metadata_locks where owner_thread_id != sys.ps_thread_id(connection_id());
查看information_schema.innodb_trx,sys.innodb_lock_waits的内容,是否有记录,发现是查不到任何有用信息的:
mysql> select * from information_schema.innodb_trx;
+-----------------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| trx_id | trx_state | trx_started | trx_requested_lock_id | trx_wait_started | trx_weight | trx_mysql_thread_id | trx_query | trx_operation_state | trx_tables_in_use | trx_tables_locked | trx_lock_structs | trx_lock_memory_bytes | trx_rows_locked | trx_rows_modified | trx_concurrency_tickets | trx_isolation_level | trx_unique_checks | trx_foreign_key_checks | trx_last_foreign_key_error | trx_adaptive_hash_latched | trx_adaptive_hash_timeout | trx_is_read_only | trx_autocommit_non_locking |
+-----------------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| 422212121904976 | RUNNING | 2023-07-01 11:28:06 | NULL | NULL | 1 | 20 | NULL | NULL | 1 | 1 | 1 | 1136 | 0 | 0 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 0 | 0 | 0 |
+-----------------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
mysql> select * from sys.innodb_lock_waits;
Empty set, 3 warnings (0.01 sec)
我们可以用过查询表级别的锁信息(performance_schema.table_handles):
mysql> select * from performance_schema.table_handles where owner_thread_id != 0 \G
*************************** 1. row ***************************
OBJECT_TYPE: TABLE
OBJECT_SCHEMA: itpuxdb1
OBJECT_NAME: itpuxbak11
OBJECT_INSTANCE_BEGIN: 140735877827488
OWNER_THREAD_ID: 47
OWNER_EVENT_ID: 17
INTERNAL_LOCK: NULL
EXTERNAL_LOCK: READ EXTERNAL
1 row in set (0.00 sec)
可以看到,内部id为47(连接号20)对 itpuxdb1.itpuxbak11 表加了表级读锁,结合processlist可以中和掉长时间处于sleep状态,所以update语句一直在等待读锁。
表级锁总结
此时可以和开发确认,如果没有什么特殊操作,可以尝试杀死这个线程,同时针对问题进行优化,避免再发生类似的情况。
行级锁
performance_schema中data_lock是MySQL8.0中新增的。
如果一个事务长时间没有提及,虽然可以从information_schema.innodb_trx,performance_schema.events_statements_current等表中查询到相应的事物信息,但是不知道这个事务的持锁信息的。虽然information_schema.innodb_locks用于记录事务的锁信息,但需要在两个不同事务发生锁等待时该表才会记录下来两个事务的锁信息。从MySQL8.0开始,在performance_schema中存在data_locks表记录任意事务的锁信息(同时废弃了information_schema.innodb_locks表),不需要有锁等待关系存在(注意,该表中只记录innodb存储引擎层的锁)。
会话1:
update itpuxdb1.itpuxbak11 set name='jiapeng' where id =3;
会话2:
mysql> select * from performance_schema.data_locks\G
*************************** 1. row ***************************
ENGINE: INNODB
ENGINE_LOCK_ID: 139970161409432:1104:139970165682704
ENGINE_TRANSACTION_ID: 3874665
THREAD_ID: 97
EVENT_ID: 24
OBJECT_SCHEMA: itpuxdb1
OBJECT_NAME: itpuxbak11
PARTITION_NAME: NULL
SUBPARTITION_NAME: NULL
INDEX_NAME: NULL
OBJECT_INSTANCE_BEGIN: 139970165682704
LOCK_TYPE: TABLE
LOCK_MODE: IX
LOCK_STATUS: GRANTED
LOCK_DATA: NULL
*************************** 2. row ***************************
ENGINE: INNODB
ENGINE_LOCK_ID: 139970161409432:47:4:2:139970165679600
ENGINE_TRANSACTION_ID: 3874665
THREAD_ID: 97
EVENT_ID: 24
OBJECT_SCHEMA: itpuxdb1
OBJECT_NAME: itpuxbak11
PARTITION_NAME: NULL
SUBPARTITION_NAME: NULL
INDEX_NAME: PRIMARY
OBJECT_INSTANCE_BEGIN: 139970165679600
LOCK_TYPE: RECORD
LOCK_MODE: X,REC_NOT_GAP
LOCK_STATUS: GRANTED
LOCK_DATA: 2
2 rows in set (0.00 sec)
##LOCK_DATA: 2被锁定的数据记录,这里的记录对应的是INDEX_NAME: PRIMARY的value
查询结果中,两行锁记录,一行是对表xpp的IX锁,状态为GRANTED,另一个锁为主键索引的X锁,REC_NOT_GAP我理解为 lock_mode x locks rec but not gap 。
如果没有创建primary key,会出现 INDEX_NAME:gen_clust_index 取代 INDEX_NAME: PRIMARY,如果表没有主键或唯一索引InnoDB内部适用,生成一个隐藏的聚集索引为合成列包含行ID值gen_clust_index。
现在,模拟两条DML发生锁等待的场景:
在会话1未提交的状态下,开启新的会话3:
update itpuxdb1.itpuxbak11 set name='jinrong' where id =3;
此时查看表:
mysql> select * from performance_schema.data_locks\G
*************************** 1. row ***************************
ENGINE: INNODB
ENGINE_LOCK_ID: 139970161410288:1104:139970165688864
ENGINE_TRANSACTION_ID: 3874666
THREAD_ID: 99
EVENT_ID: 7
OBJECT_SCHEMA: itpuxdb1
OBJECT_NAME: itpuxbak11
PARTITION_NAME: NULL
SUBPARTITION_NAME: NULL
INDEX_NAME: NULL
OBJECT_INSTANCE_BEGIN: 139970165688864
LOCK_TYPE: TABLE
LOCK_MODE: IX
LOCK_STATUS: GRANTED
LOCK_DATA: NULL
*************************** 2. row ***************************
ENGINE: INNODB
ENGINE_LOCK_ID: 139970161410288:47:4:2:139970165685760
ENGINE_TRANSACTION_ID: 3874666
THREAD_ID: 99
EVENT_ID: 7
OBJECT_SCHEMA: itpuxdb1
OBJECT_NAME: itpuxbak11
PARTITION_NAME: NULL
SUBPARTITION_NAME: NULL
INDEX_NAME: PRIMARY
OBJECT_INSTANCE_BEGIN: 139970165685760
LOCK_TYPE: RECORD
LOCK_MODE: X,REC_NOT_GAP
LOCK_STATUS: WAITING
LOCK_DATA: 2
*************************** 3. row ***************************
ENGINE: INNODB
ENGINE_LOCK_ID: 139970161409432:1104:139970165682704
ENGINE_TRANSACTION_ID: 3874665
THREAD_ID: 97
EVENT_ID: 24
OBJECT_SCHEMA: itpuxdb1
OBJECT_NAME: itpuxbak11
PARTITION_NAME: NULL
SUBPARTITION_NAME: NULL
INDEX_NAME: NULL
OBJECT_INSTANCE_BEGIN: 139970165682704
LOCK_TYPE: TABLE
LOCK_MODE: IX
LOCK_STATUS: GRANTED
LOCK_DATA: NULL
*************************** 4. row ***************************
ENGINE: INNODB
ENGINE_LOCK_ID: 139970161409432:47:4:2:139970165679600
ENGINE_TRANSACTION_ID: 3874665
THREAD_ID: 97
EVENT_ID: 24
OBJECT_SCHEMA: itpuxdb1
OBJECT_NAME: itpuxbak11
PARTITION_NAME: NULL
SUBPARTITION_NAME: NULL
INDEX_NAME: PRIMARY
OBJECT_INSTANCE_BEGIN: 139970165679600
LOCK_TYPE: RECORD
LOCK_MODE: X,REC_NOT_GAP
LOCK_STATUS: GRANTED
LOCK_DATA: 2
4 rows in set (0.00 sec)
这四行记录中,新增了两条线程id为99的记录,IX的表锁状态GRANTED,X的行数状态是WAITING。说明正在等待锁被授予。这里并不能很直观的查到锁等待关系。利用sys.innodb_lock_waits查一下:
(MySQL5.7中,可以使用sys.innodb_lock_waits查询,MySQL8.0中也可以,额外说明的是8.0中,该视图的查询表发生变化,由information_schema.innodb_locks,information_schema.innodb_locks_waits 变成 performance_schema.data_locks, performance_schema.data_locks_waits )
mysql> select * from sys.innodb_lock_waits\G
*************************** 1. row ***************************
wait_started: 2023-07-01 12:10:10
wait_age: 00:00:05
wait_age_secs: 5
locked_table: `itpuxdb1`.`itpuxbak11`
locked_index: PRIMARY
locked_type: RECORD
waiting_trx_id: 3367
waiting_trx_started: 2023-07-01 11:56:46
waiting_trx_age: 00:13:29
waiting_trx_rows_locked: 2
waiting_trx_rows_modified: 0
waiting_pid: 22
waiting_query: update itpuxdb1.itpuxbak11 set name='jinrong' where id =3
waiting_lock_id: 3367:30:3:7
waiting_lock_mode: X
blocking_trx_id: 3369
blocking_pid: 26
blocking_query: NULL
blocking_lock_id: 3369:30:3:7
blocking_lock_mode: X
blocking_trx_started: 2023-07-01 12:10:04
blocking_trx_age: 00:00:11
blocking_trx_rows_locked: 1
blocking_trx_rows_modified: 0
sql_kill_blocking_query: KILL QUERY 26
sql_kill_blocking_connection: KILL 26
1 row in set, 3 warnings (0.02 sec)
好博客就要一起分享哦!分享海报
此处可发布评论
评论(0)展开评论
展开评论
您可能感兴趣的博客
