osd cpu usage is bigger than 100%

[yue longguang]

hi,all
i am testing rbd performance, now there is only one vm which is
using rbd as its disk, and inside it fio is doing r/w.
the big diffenence is that i set a big iodepth other than iodepth=1.
according to my test, the bigger iodepth, the bigger cpu usage.

analyse the output of top command.
1.
12% wa, if it means disk speed is not fast enough?

2. from where we can know whether ceph's number of threads is
enough or not?


how do you think about it, which part is using up cpu? i want to find
the root cause, why big iodepth leads to high cpu usage.


---default options----
osd_op_threads": "2",
"osd_disk_threads": "1",
"osd_recovery_threads": "1",
"filestore_op_threads": "2",


thanks

----------top---------------iodepth=16-----------------
top - 15:27:34 up 2 days, 6:03, 2 users, load average: 0.49, 0.56, 0.62
Tasks: 97 total, 1 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.0%us, 8.1%sy, 0.0%ni, 59.3%id, 12.1%wa, 0.0%hi, 0.8%si, 0.7%st
Mem: 1922540k total, 1853180k used, 69360k free, 7012k buffers
Swap: 1048568k total, 76796k used, 971772k free, 1034272k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2763 root 20 0 1112m 386m 5028 S 60.8 20.6 200:43.47 ceph-osd

-------------top--------------------------------------------
top - 19:50:08 up 1 day, 10:26, 2 users, load average: 1.55, 0.97, 0.81
Tasks: 97 total, 1 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 37.6%us, 14.2%sy, 0.0%ni, 37.0%id, 9.4%wa, 0.0%hi, 1.3%si, 0.5%st
Mem: 1922540k total, 1820196k used, 102344k free, 23100k buffers
Swap: 1048568k total, 91724k used, 956844k free, 1052292k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4312 root 20 0 1100m 337m 5192 S 107.3 18.0 88:33.27 ceph-osd
1704 root 20 0 514m 272m 3648 S 0.7 14.5 3:27.19 ceph-mon



--------------iostat------------------

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 5.50 137.50 247.00 782.00 2896.00 8773.00
11.34 7.08 3.55 0.63 65.05

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 9.50 119.00 327.50 458.50 3940.00 4733.50
11.03 12.03 19.66 0.70 55.40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 15.50 10.50 324.00 559.50 3784.00 3398.00
8.13 1.98 2.22 0.81 71.25

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 4.50 253.50 273.50 803.00 3056.00 12155.00
14.13 4.70 4.32 0.55 59.55

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 10.00 6.00 294.00 488.00 3200.00 2933.50
7.84 1.10 1.49 0.70 54.85

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 10.00 14.00 333.00 645.00 3780.00 3846.00
7.80 2.13 2.15 0.90 87.55

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 11.00 240.50 259.00 579.00 3144.00 10035.50
15.73 8.51 10.18 0.84 70.20

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 10.50 17.00 318.50 707.00 3876.00 4084.50
7.76 1.32 1.30 0.61 62.65

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 4.50 208.00 233.50 918.00 2648.00 19214.50
18.99 5.43 4.71 0.55 63.20

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 7.00 1.50 306.00 212.00 3376.00 2176.50
10.72 1.03 1.83 0.96 49.70
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Chen, Xiaoxi]

1. 12% wa is quite normal, with more disks and more load ,you could even see 30%+ on random write case
2. Our BKM is set osd_op_threads to 20,

-----Original Message-----
From: ceph-devel-***@vger.kernel.org [mailto:ceph-devel-***@vger.kernel.org] On Behalf Of yue longguang
Sent: Thursday, September 11, 2014 3:44 PM
To: ceph-***@vger.kernel.org
Subject: osd cpu usage is bigger than 100%

hi,all
i am testing rbd performance, now there is only one vm which is
using rbd as its disk, and inside it fio is doing r/w.
the big diffenence is that i set a big iodepth other than iodepth=1.
according to my test, the bigger iodepth, the bigger cpu usage.

analyse the output of top command.
1.
12% wa, if it means disk speed is not fast enough?

2. from where we can know whether ceph's number of threads is enough or not?


how do you think about it, which part is using up cpu? i want to find the root cause, why big iodepth leads to high cpu usage.


---default options----
osd_op_threads": "2",
"osd_disk_threads": "1",
"osd_recovery_threads": "1",
"filestore_op_threads": "2",


thanks

----------top---------------iodepth=16-----------------
top - 15:27:34 up 2 days, 6:03, 2 users, load average: 0.49, 0.56, 0.62
Tasks: 97 total, 1 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.0%us, 8.1%sy, 0.0%ni, 59.3%id, 12.1%wa, 0.0%hi, 0.8%si, 0.7%st
Mem: 1922540k total, 1853180k used, 69360k free, 7012k buffers
Swap: 1048568k total, 76796k used, 971772k free, 1034272k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2763 root 20 0 1112m 386m 5028 S 60.8 20.6 200:43.47 ceph-osd

-------------top--------------------------------------------
top - 19:50:08 up 1 day, 10:26, 2 users, load average: 1.55, 0.97, 0.81
Tasks: 97 total, 1 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 37.6%us, 14.2%sy, 0.0%ni, 37.0%id, 9.4%wa, 0.0%hi, 1.3%si, 0.5%st
Mem: 1922540k total, 1820196k used, 102344k free, 23100k buffers
Swap: 1048568k total, 91724k used, 956844k free, 1052292k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4312 root 20 0 1100m 337m 5192 S 107.3 18.0 88:33.27 ceph-osd
1704 root 20 0 514m 272m 3648 S 0.7 14.5 3:27.19 ceph-mon



--------------iostat------------------

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 5.50 137.50 247.00 782.00 2896.00 8773.00
11.34 7.08 3.55 0.63 65.05

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 9.50 119.00 327.50 458.50 3940.00 4733.50
11.03 12.03 19.66 0.70 55.40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 15.50 10.50 324.00 559.50 3784.00 3398.00
8.13 1.98 2.22 0.81 71.25

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 4.50 253.50 273.50 803.00 3056.00 12155.00
14.13 4.70 4.32 0.55 59.55

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 10.00 6.00 294.00 488.00 3200.00 2933.50
7.84 1.10 1.49 0.70 54.85

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 10.00 14.00 333.00 645.00 3780.00 3846.00
7.80 2.13 2.15 0.90 87.55

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 11.00 240.50 259.00 579.00 3144.00 10035.50
15.73 8.51 10.18 0.84 70.20

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 10.50 17.00 318.50 707.00 3876.00 4084.50
7.76 1.32 1.30 0.61 62.65

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 4.50 208.00 233.50 918.00 2648.00 19214.50
18.99 5.43 4.71 0.55 63.20

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
vdd 7.00 1.50 306.00 212.00 3376.00 2176.50
10.72 1.03 1.83 0.96 49.70