Skip to content

Commit f6da2d5

Browse files
george-reynyainikep
george-reynya
authored and
inikep
committed
[thread_pool] Call thd_wait_* callbacks for admitting new query in thread pool plugin
Summary: 1. Change the code to admit new query through `thd_wait_begin/end` callbacks using new `THD_WAIT_ADMIT` wait type. 2. Move yield counting to the callbacks to unify the code between the server and the plugin. `yield_cond` exposes the yield condition to check once the yield counter says that yield is needed. 3. Expose all wait events through `admission_control_wait_events` var. This is especially needed for `USER_LOCK` which still controls innodb thread concurrency yield. 4. Copy and fix up admission control tests to the thread_pool suite. Only the var and table names are changed to refer to thread_pool plugin. The results have been diffed with original tests to make sure they are identical (except for var/table names). Reviewed By: lth Differential Revision: D27593084 --------------------------------------------------------------------------------------------- use lambda instead of std::bind (facebook#1243) Summary: std::bind(yield_condition, table) will generate a functor which size is larger than std::function's local buf, thus std::function needs to new/delete memory to store the functor. This PR use the lambda which just capture one pointer(table), which size can fit into std::function's local buf thus new/delete is not needed. Pull Request resolved: facebook#1243 Reviewed By: lth Differential Revision: D40858532 Pulled By: hermanlee --------------------------------------------------------------------------------------------- Add yields to hash join iterator Summary: Queries performing joins are lacking yields and monopolizing thread pool scheduler. Differential Revision: D39948376 ------------------------------------------------------------------------------ Add yield to union iterator Summary: This was detected by the stall detection (8.0.23 based callstack): ``` W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] Scheduler 12 stalled by worker 0x7f6367d5d640 W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] Stack trace for 1 thread(s) [1422783 mysqld]: W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 00000000001204d0 ppoll W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 000000000289d67f vio_io_wait(Vio*, enum_vio_io_event, timeout_t) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 0000000002952296 vio_socket_io_wait(Vio*, enum_vio_io_event) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 0000000002385fb2 net_write_packet(NET*, unsigned char const*, unsigned long) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 00000000025b2c44 SELECT_LEX_UNIT::ExecuteIteratorQuery(THD*) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 000000000258d2db Sql_cmd_dml::execute(THD*) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 000000000254f179 mysql_execute_command(THD*, bool, unsigned long long*) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 0000000002545936 dispatch_sql_command(THD*, Parser_state*, unsigned long long*) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 000000000269d931 dispatch_command(THD*, COM_DATA const*, enum_server_command) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 000000000269c3e0 do_command(THD*) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 0000000005c4ffa7 mysql::thread_pool::TpConnHandler::processEvent(void*) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 0000000005c5395f mysql::thread_pool::TpTask::execute() W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 0000000005c5c299 mysql::thread_pool::TpWorkerPool::processWorker(void*) W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 0000000004867855 pfs_spawn_thread(void*) [clone .llvm.11916604799980067416] W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 000000000009ac0e start_thread W0205 14:38:40.101111 2319640 TpScheduler.cpp:431] @ 000000000012d1db __clone3 ``` Cleaned up existing yield code to use a common default yield condition. Reviewed By: preritj24 Differential Revision: D43071441
1 parent 5bf1612 commit f6da2d5

38 files changed

+2242
-63
lines changed

‎include/mysql/service_thd_wait.h‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@
6060

6161
class THD;
6262

63-
/*
63+
/**
6464
One should only report wait events that could potentially block for a
6565
long time. A mutex wait is too short of an event to report. The reason
6666
is that an event which is reported leads to a new thread starts
@@ -73,9 +73,13 @@ class THD;
7373
holds true for global read locks, table locks and other meta data locks.
7474
Another event of interest is going to sleep for an extended time.
7575
76-
Note that user-level locks no longer use THD_WAIT_USER_LOCK wait type.
76+
@note User-level locks no longer use THD_WAIT_USER_LOCK wait type.
7777
Since their implementation relies on metadata locks manager it uses
7878
THD_WAIT_META_DATA_LOCK instead.
79+
80+
@note THD_WAIT_ADMIT is a fake wait type communicating that query has been
81+
parsed and is ready for execution. The query attributes and sql command
82+
are available at this point.
7983
*/
8084

8185
enum THD_wait_type : int {
@@ -86,15 +90,16 @@ enum THD_wait_type : int {
8690
THD_WAIT_GLOBAL_LOCK = 4,
8791
THD_WAIT_META_DATA_LOCK = 5,
8892
THD_WAIT_TABLE_LOCK = 6,
89-
THD_WAIT_USER_LOCK = 7,
93+
THD_WAIT_INNODB_CONC = 7,
9094
THD_WAIT_BINLOG = 8,
9195
THD_WAIT_GROUP_COMMIT = 9,
9296
THD_WAIT_SYNC = 10,
9397
THD_WAIT_TRX_DELAY = 11,
9498
THD_WAIT_FOR_HLC = 12,
9599
THD_WAIT_NET_IO = 13,
96100
THD_WAIT_YIELD = 14,
97-
THD_WAIT_LAST = 15
101+
THD_WAIT_ADMIT = 15,
102+
THD_WAIT_LAST = 16
98103
};
99104

100105
inline const char *THD_wait_type_str(THD_wait_type twt) {
@@ -120,8 +125,8 @@ inline const char *THD_wait_type_str(THD_wait_type twt) {
120125
case THD_WAIT_TABLE_LOCK:
121126
return "Waiting for table lock";
122127

123-
case THD_WAIT_USER_LOCK:
124-
return "Waiting for user lock";
128+
case THD_WAIT_INNODB_CONC:
129+
return "Waiting for InnoDB concurrency control";
125130

126131
case THD_WAIT_BINLOG:
127132
return "Waiting for binlog";
@@ -144,6 +149,9 @@ inline const char *THD_wait_type_str(THD_wait_type twt) {
144149
case THD_WAIT_YIELD:
145150
return "Waiting for YIELD";
146151

152+
case THD_WAIT_ADMIT:
153+
return "Waiting for WAIT ADMIT";
154+
147155
case THD_WAIT_LAST:
148156
return "<Unused LAST marker value>";
149157
} // switch

‎mysql-test/suite/sys_vars/r/admission_control_wait_events_basic.result‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,17 @@ SET GLOBAL admission_control_wait_events='YIELD';
1818
SELECT @@global.admission_control_wait_events;
1919
@@global.admission_control_wait_events
2020
YIELD
21-
SET GLOBAL admission_control_wait_events='SLEEP,NET_IO,META_DATA_LOCK,ROW_LOCK,YIELD';
21+
SET GLOBAL admission_control_wait_events='SLEEP,ROW_LOCK,META_DATA_LOCK,INNODB_CONC,NET_IO,YIELD';
2222
SELECT @@global.admission_control_wait_events;
2323
@@global.admission_control_wait_events
24-
SLEEP,ROW_LOCK,META_DATA_LOCK,NET_IO,YIELD
24+
SLEEP,ROW_LOCK,META_DATA_LOCK,INNODB_CONC,NET_IO,YIELD
25+
SET GLOBAL admission_control_wait_events='SLEEP,SLEEP';
26+
SELECT @@global.admission_control_wait_events;
27+
@@global.admission_control_wait_events
28+
SLEEP
2529
SET GLOBAL admission_control_wait_events='YIELD,NONEXISTING_BIT';
2630
ERROR 42000: Variable 'admission_control_wait_events' can't be set to the value of 'NONEXISTING_BIT'
2731
SELECT @@global.admission_control_wait_events;
2832
@@global.admission_control_wait_events
29-
SLEEP,ROW_LOCK,META_DATA_LOCK,NET_IO,YIELD
33+
SLEEP
3034
set global admission_control_wait_events = @saved_admission_control_wait_events;

‎mysql-test/suite/sys_vars/t/admission_control_wait_events_basic.test‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,11 @@ SELECT @@global.admission_control_wait_events;
1212
SET GLOBAL admission_control_wait_events='YIELD';
1313
SELECT @@global.admission_control_wait_events;
1414

15-
SET GLOBAL admission_control_wait_events='SLEEP,NET_IO,META_DATA_LOCK,ROW_LOCK,YIELD';
15+
SET GLOBAL admission_control_wait_events='SLEEP,ROW_LOCK,META_DATA_LOCK,INNODB_CONC,NET_IO,YIELD';
16+
SELECT @@global.admission_control_wait_events;
17+
18+
# check that repeating the same value is OK
19+
SET GLOBAL admission_control_wait_events='SLEEP,SLEEP';
1620
SELECT @@global.admission_control_wait_events;
1721

1822
# checking that setting variable to a non existing value raises error

‎mysql-test/suite/thread_pool/r/admission_control.result‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
create database test_db;
2+
create database test_db2;
3+
create user 'test_user'@'localhost';
4+
grant all on test_db.* to 'test_user'@'localhost';
5+
grant all on test_db2.* to 'test_user'@'localhost';
6+
grant all on test.* to 'test_user'@'localhost';
7+
use test_db2;
8+
use test_db;
9+
set @start_max_running_queries= @@global.thread_pool_max_running_queries;
10+
set @start_max_waiting_queries= @@global.thread_pool_max_waiting_queries;
11+
set @@global.thread_pool_max_running_queries=10;
12+
set @@global.thread_pool_max_waiting_queries=5;
13+
create table t1(a int) engine=InnoDB;
14+
lock table t1 write;
15+
Threads waiting for admission will have appropriate state set in processlist.
16+
Super user is exempted from admission control checks.
17+
select * from t1;
18+
a
19+
set @@global.thread_pool_admission_control_filter = 'USE';
20+
select @@global.thread_pool_admission_control_filter;
21+
@@global.thread_pool_admission_control_filter
22+
USE
23+
Maximum waiting queries reached. So this would hit an error.
24+
use test_db;
25+
select * from t1||||
26+
ERROR HY000: Maximum waiting queries 5 reached for database `test_db`
27+
Maximum waiting queries reached. So this would hit an error.
28+
use test_db2;
29+
create table t1_test(aaa int);
30+
insert into t1_test values (1);
31+
select aaa from t1_test;
32+
drop table t1_test;
33+
use test_db;
34+
select * from t1||||
35+
aaa
36+
1
37+
ERROR HY000: Maximum waiting queries 5 reached for database `test_db`
38+
use test_db;
39+
select * from t1;
40+
ERROR HY000: Maximum waiting queries 5 reached for database `test_db`
41+
set @@global.thread_pool_admission_control_filter = '';
42+
select @@global.thread_pool_admission_control_filter;
43+
@@global.thread_pool_admission_control_filter
44+
45+
Check status variables
46+
aborted_queries = 3
47+
running_queries = 10
48+
waiting_queries = 5
49+
select * from information_schema.tp_admission_control_entities where schema_name like 'test_db%' order by schema_name;
50+
SCHEMA_NAME WAITING_QUERIES RUNNING_QUERIES ABORTED_QUERIES TIMEOUT_QUERIES CONNECTIONS REJECTED_CONNECTIONS
51+
test_db 5 10 3 0 15 0
52+
test_db2 0 0 0 0 0 0
53+
Filled up queues on one db doesn't affect queries on other db.
54+
use test_db2;
55+
select * from information_schema.tp_admission_control_entities where schema_name like 'test_db%' order by schema_name;
56+
SCHEMA_NAME WAITING_QUERIES RUNNING_QUERIES ABORTED_QUERIES TIMEOUT_QUERIES CONNECTIONS REJECTED_CONNECTIONS
57+
test_db 5 10 3 0 15 0
58+
test_db2 0 1 0 0 1 0
59+
set @@global.thread_pool_max_waiting_queries=6;
60+
Kill a thread that is waiting for admission.
61+
select count(*) from t1;
62+
kill ID;
63+
use test_db;
64+
unlock tables;
65+
Verify the waiting queries received wakeup signal.
66+
select count(*) from t1;
67+
count(*)
68+
15
69+
set @save_admission_control_by_trx = @@global.thread_pool_admission_control_by_trx;
70+
select @save_admission_control_by_trx;
71+
@save_admission_control_by_trx
72+
0
73+
set @@global.thread_pool_max_running_queries=5;
74+
set @@global.thread_pool_max_waiting_queries=10;
75+
# By default, open transaction has no effect on running queries
76+
select count(*) from t1;
77+
count(*)
78+
15
79+
# Test: open transactions will take slots in running queries,
80+
# and will not be blocked
81+
set @@global.thread_pool_admission_control_filter = 'BEGIN,COMMIT,ROLLBACK';
82+
select @@global.thread_pool_admission_control_filter;
83+
@@global.thread_pool_admission_control_filter
84+
BEGIN,COMMIT,ROLLBACK
85+
set @@global.thread_pool_admission_control_by_trx = true;
86+
SELECT @@global.thread_pool_admission_control_by_trx;
87+
@@global.thread_pool_admission_control_by_trx
88+
1
89+
Open transaction is able to continue running queries
90+
connection con_max_wait;
91+
New queries will be rejected (waiting queue is full)
92+
select * from t1;
93+
ERROR HY000: Maximum waiting queries 10 reached for database `test_db`
94+
New transactions will be rejected (waiting queue is full)
95+
begin;
96+
select * from t1;
97+
ERROR HY000: Maximum waiting queries 10 reached for database `test_db`
98+
aborted_queries will increase by 2
99+
Committing a transaction will free up the running query slots
100+
The waiting queries will be unblocked
101+
Check status variables
102+
include/assert.inc [DB Admission control waiting queries should be zero]
103+
include/assert.inc [DB Admission control running queries should be zero]
104+
include/assert.inc [DB Admission control aborted queries should be five]
105+
select * from information_schema.tp_admission_control_entities where schema_name like 'test_db%' order by schema_name;
106+
SCHEMA_NAME WAITING_QUERIES RUNNING_QUERIES ABORTED_QUERIES TIMEOUT_QUERIES CONNECTIONS REJECTED_CONNECTIONS
107+
test_db 0 0 5 0 13 0
108+
test_db2 0 0 0 0 0 0
109+
set @@global.thread_pool_admission_control_by_trx = @save_admission_control_by_trx;
110+
select @@global.thread_pool_admission_control_by_trx;
111+
@@global.thread_pool_admission_control_by_trx
112+
0
113+
set @@global.thread_pool_admission_control_filter = '';
114+
select @@global.thread_pool_admission_control_filter;
115+
@@global.thread_pool_admission_control_filter
116+
117+
# End of open transaction test
118+
# Test admission_control_queue_timeout
119+
use test_db;
120+
set @@global.thread_pool_max_running_queries=1;
121+
set @@global.thread_pool_max_waiting_queries=5;
122+
set @save_admission_control_filter = @@global.thread_pool_admission_control_filter;
123+
set @save_admission_control_queue_timeout = @@global.thread_pool_admission_control_queue_timeout;
124+
set @@global.thread_pool_admission_control_queue_timeout = 100;
125+
set @@global.thread_pool_admission_control_filter = 'BEGIN,COMMIT,ROLLBACK';
126+
create table t2(a int primary key) engine=InnoDB;
127+
begin;
128+
insert into t2 values (1);
129+
begin;
130+
insert into t2 values (1);
131+
insert into t2 values (2);
132+
ERROR HY000: Got timeout while waiting on admission control queue for database `test_db`
133+
rollback;
134+
rollback;
135+
drop table t2;
136+
set @@global.thread_pool_admission_control_filter = @save_admission_control_filter;
137+
set @@global.thread_pool_admission_control_queue_timeout = @save_admission_control_queue_timeout;
138+
timeout_queries should be 1
139+
timeout_queries = 1
140+
waiting_queries should be 0
141+
waiting_queries = 0
142+
select * from information_schema.tp_admission_control_entities where schema_name like 'test_db%' order by schema_name;
143+
SCHEMA_NAME WAITING_QUERIES RUNNING_QUERIES ABORTED_QUERIES TIMEOUT_QUERIES CONNECTIONS REJECTED_CONNECTIONS
144+
test_db 0 0 5 1 15 0
145+
test_db2 0 0 0 0 0 0
146+
reset global.thread_pool_max_running_queries and global.thread_pool_max_waiting_queries
147+
set @@global.thread_pool_max_running_queries=10;
148+
set @@global.thread_pool_max_waiting_queries=5;
149+
Run parallel load and drop the database.
150+
set @@global.thread_pool_max_waiting_queries=0;
151+
Cleanup.
152+
Verify there are no waiting threads.
153+
select count(*) from information_schema.processlist where state='waiting for admission';
154+
count(*)
155+
0
156+
select * from information_schema.tp_admission_control_entities where schema_name like 'test_db%' order by schema_name;
157+
SCHEMA_NAME WAITING_QUERIES RUNNING_QUERIES ABORTED_QUERIES TIMEOUT_QUERIES CONNECTIONS REJECTED_CONNECTIONS
158+
set @@global.thread_pool_max_running_queries=@start_max_running_queries;
159+
set @@global.thread_pool_max_waiting_queries=@start_max_waiting_queries;
160+
drop user test_user@localhost;

‎mysql-test/suite/thread_pool/r/admission_control_hang.result‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
create database test_db;
2+
create user test_user@localhost;
3+
grant all on test_db.* to test_user@localhost;
4+
set @start_max_running_queries = @@global.max_running_queries;
5+
set @@global.max_running_queries = 4;
6+
set @start_innodb_lock_wait_timeout = @@global.innodb_lock_wait_timeout;
7+
set @@global.innodb_lock_wait_timeout = 10000;
8+
set @start_admission_control_filter = @@global.admission_control_filter;
9+
set @@global.admission_control_filter = 'COMMIT';
10+
create table t1 (a int) engine=innodb;
11+
insert into t1 values(1);
12+
begin;
13+
update t1 set a=2 where a=1;
14+
update t1 set a=2 where a=1;
15+
update t1 set a=2 where a=1;
16+
update t1 set a=2 where a=1;
17+
update t1 set a=2 where a=1;
18+
set @@global.admission_control_filter = 'USE';
19+
select @@global.admission_control_filter;
20+
@@global.admission_control_filter
21+
USE
22+
use test;
23+
use test_db;
24+
set @@global.admission_control_filter = 'ALTER,BEGIN,COMMIT,CREATE,DELETE,DROP,INSERT,LOAD,SELECT,SET,REPLACE,TRUNCATE,UPDATE,SHOW,ROLLBACK';
25+
select @@global.admission_control_filter;
26+
@@global.admission_control_filter
27+
ALTER,BEGIN,COMMIT,CREATE,DELETE,DROP,INSERT,LOAD,SELECT,SET,REPLACE,ROLLBACK,TRUNCATE,UPDATE,SHOW
28+
create table t2(a int) engine=innodb;
29+
begin;
30+
insert into t2 values(1);
31+
update t2 set a=2 where a=1;
32+
commit;
33+
SHOW TABLES LIKE 't2';
34+
Tables_in_test_db (t2)
35+
t2
36+
begin;
37+
alter table t2 rename t3;
38+
select * from t3;
39+
a
40+
2
41+
delete from t3;
42+
set @val = 1;
43+
truncate table t3;
44+
rollback;
45+
drop table t3;
46+
set @save_admission_control_by_trx = @@global.admission_control_by_trx;
47+
select @save_admission_control_by_trx;
48+
@save_admission_control_by_trx
49+
0
50+
# Turn on admission_control_by_trx
51+
set @@global.admission_control_by_trx = true;
52+
SELECT @@global.admission_control_by_trx;
53+
@@global.admission_control_by_trx
54+
1
55+
create table t2(a int) engine=innodb;
56+
begin;
57+
insert into t2 values(1);
58+
update t2 set a=2 where a=1;
59+
commit;
60+
SHOW TABLES LIKE 't2';
61+
Tables_in_test_db (t2)
62+
t2
63+
begin;
64+
alter table t2 rename t3;
65+
select * from t3;
66+
a
67+
2
68+
delete from t3;
69+
set @val = 1;
70+
truncate table t3;
71+
rollback;
72+
drop table t3;
73+
set @@global.admission_control_filter = default;
74+
select @@global.admission_control_filter;
75+
@@global.admission_control_filter
76+
77+
select count(*) from t1;
78+
count(*)
79+
1
80+
set @@global.admission_control_by_trx = @save_admission_control_by_trx;
81+
select @@global.admission_control_by_trx;
82+
@@global.admission_control_by_trx
83+
0
84+
set @@global.admission_control_filter = 'COMMIT';
85+
select @@global.admission_control_filter;
86+
@@global.admission_control_filter
87+
COMMIT
88+
commit;
89+
set @@global.max_running_queries = @start_max_running_queries;
90+
set @@global.innodb_lock_wait_timeout = @start_innodb_lock_wait_timeout;
91+
set @@global.admission_control_filter = @start_admission_control_filter;
92+
drop database test_db;
93+
drop user test_user@localhost;

‎mysql-test/suite/thread_pool/r/admission_control_multi_query.result‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
create database test_db;
2+
create user test_user@localhost identified with 'mysql_native_password' BY '';
3+
grant all on test_db.* to test_user@localhost;
4+
grant all on test.* to test_user@localhost;
5+
use test_db;
6+
create table t1 (a int primary key, b int) engine=InnoDB;
7+
drop database test_db;
8+
drop user test_user@localhost;

‎mysql-test/suite/thread_pool/r/admission_control_multi_query_wait_events.result‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
create database test_db;
2+
create user test_user@localhost identified with 'mysql_native_password' BY '';
3+
grant all on test_db.* to test_user@localhost;
4+
grant all on test.* to test_user@localhost;
5+
use test_db;
6+
create table t1 (a int primary key, b int) engine=InnoDB;
7+
drop database test_db;
8+
drop user test_user@localhost;

‎mysql-test/suite/thread_pool/r/admission_control_multi_query_weighted.result‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
create database test_db;
2+
create user test_user@localhost identified with 'mysql_native_password' BY '';
3+
grant all on test_db.* to test_user@localhost;
4+
grant all on test.* to test_user@localhost;
5+
use test_db;
6+
create table t1 (a int primary key, b int) engine=InnoDB;
7+
drop database test_db;
8+
drop user test_user@localhost;

0 commit comments

Comments
 (0)