severe librbd performance degradation in Giant

[Somnath Roy]

Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.

1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.


So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

Thanks & Regards
Somnath

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Mark Nelson]

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

Hi Somnath,

How much concurrency?

Post by Somnath Roy
Thanks & Regards
Somnath
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

Mark,
All are running with concurrency 32.

Thanks & Regards
Somnath

-----Original Message-----
From: Mark Nelson [mailto:***@inktank.com]
Sent: Wednesday, September 17, 2014 1:59 PM
To: Somnath Roy; ceph-***@vger.kernel.org
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

Hi Somnath,

How much concurrency?

Post by Somnath Roy
Thanks & Regards
Somnath
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

But, this time is ~10X degradation :-(

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
From: Stefan Priebe - Profihost AG [mailto:***@profihost.ag]
Sent: Wednesday, September 17, 2014 2:02 PM
To: Somnath Roy
Cc: ceph-***@vger.kernel.org
Subject: Re: severe librbd performance degradation in Giant

I reported the same for librbd I'm firefly after upgrading from dumpling here:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-June/040664.html

Stefan

Excuse my typo sent from my mobile phone.

Am 17.09.2014 um 22:55 schrieb Somnath Roy <***@sandisk.com>:
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.

1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K  iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K  iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K  iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.


So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

Thanks & Regards
Somnath

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

[Josh Durgin]

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

For giant the default cache settings changed to:

rbd cache = true
rbd cache writethrough until flush = true

If fio isn't sending flushes as the test is running, the cache will
stay in writethrough mode. Does the difference remain if you set rbd
cache writethrough until flush = false ?

Josh
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.

rbd_cache_writethrough_until_flush = false

But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?

Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !

So, loks like rbd_cache=true was the culprit.

Thanks Josh !

Regards
Somnath

-----Original Message-----
From: Josh Durgin [mailto:***@inktank.com]
Sent: Wednesday, September 17, 2014 2:20 PM
To: Somnath Roy; ceph-***@vger.kernel.org
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

For giant the default cache settings changed to:

rbd cache = true
rbd cache writethrough until flush = true

If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?

Josh

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Mark Nelson]

Any chance read ahead could be causing issues?

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

It's default read ahead setting. I am doing random read , so, I don't think read ahead is the issue.
Also, in the cluster side, ceph -s is reporting same iops, so, ios are hitting the cluster.
-----Original Message-----
From: Mark Nelson [mailto:***@inktank.com]
Sent: Wednesday, September 17, 2014 2:34 PM
To: Somnath Roy; Josh Durgin; ceph-***@vger.kernel.org
Subject: Re: severe librbd performance degradation in Giant

Any chance read ahead could be causing issues?

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Josh Durgin]

No, it's not merged yet. The ObjectCacher (which implements rbd and
ceph-fuse caching) has a global lock, which could be a bottleneck in
this case.

Post by Mark Nelson
Any chance read ahead could be causing issues?

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back*
the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant
over firefly release. Here is the experiment we did to isolate it as
a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd
on top of firefly based librbd/librados. For one client it is giving
~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top
of Giant based librbd/librados. For one client it is giving ~1.9K
iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant
based ceph_smaiobench on top of giant librados. For one client it is
giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I
will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will
stay in writethrough mode. Does the difference remain if you set rbd
cache writethrough until flush = false ?
Josh

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Sage Weil]

What was the io pattern? Sequential or random? For random a slowdown
makes sense (tho maybe not 10x!) but not for sequentail....

s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

Sage,
It's a 4K random read.

Thanks & Regards
Somnath

-----Original Message-----
From: Sage Weil [mailto:***@redhat.com]
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Cc: Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....

s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

Created a tracker for this.

http://tracker.ceph.com/issues/9513

Thanks & Regards
Somnath

-----Original Message-----
From: ceph-devel-***@vger.kernel.org [mailto:ceph-devel-***@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Cc: Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

Sage,
It's a 4K random read.

Thanks & Regards
Somnath

-----Original Message-----
From: Sage Weil [mailto:***@redhat.com]
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Cc: Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....

s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to ***@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.

Thanks & Regards
Somnath

-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Cc: Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

Created a tracker for this.

http://tracker.ceph.com/issues/9513

Thanks & Regards
Somnath

-----Original Message-----
From: ceph-devel-***@vger.kernel.org [mailto:ceph-devel-***@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Cc: Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

Sage,
It's a 4K random read.

Thanks & Regards
Somnath

-----Original Message-----
From: Sage Weil [mailto:***@redhat.com]
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Cc: Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....

s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to ***@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Haomai Wang]

According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

Yes Haomai...

-----Original Message-----
From: Haomai Wang [mailto:***@gmail.com]
Sent: Wednesday, September 17, 2014 7:28 PM
To: Somnath Roy
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org
Subject: Re: severe librbd performance degradation in Giant

According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

[Sage Weil]

Post by Somnath Roy
Yes Haomai...

I would love to what a profiler says about the matter. There is going
to be some overhead on the client associated with the cache for a
random io workload, but 10x is a problem!

sage

Post by Somnath Roy
-----Original Message-----
Sent: Wednesday, September 17, 2014 7:28 PM
To: Somnath Roy
Subject: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
N?????r??y??????X???v???)?{.n?????z?]z????ay?????j??f???h??????w??????j:+v???w????????????zZ+???????j"????i

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

Sage,
Any reason why the cache is by default enabled in Giant ?
Regarding profiling, I will try if I can run Vtune/mutrace on this.

Thanks & Regards
Somnath

-----Original Message-----
From: Sage Weil [mailto:***@redhat.com]
Sent: Wednesday, September 17, 2014 8:53 PM
To: Somnath Roy
Cc: Haomai Wang; Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

Post by Somnath Roy
Yes Haomai...

I would love to what a profiler says about the matter. There is going to be some overhead on the client associated with the cache for a random io workload, but 10x is a problem!

sage

Post by Somnath Roy
-----Original Message-----
Sent: Wednesday, September 17, 2014 7:28 PM
To: Somnath Roy
Subject: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
N?????r??y??????X???v???)?{.n?????z?]z????ay?????j ??f???h??????w???

???j:+v???w???????? ????zZ+???????j"????i
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Chen, Xiaoxi]

Same question as Somnath, some customer of us not feeling that comfortable with cache, they still have some consistent concern.

-----Original Message-----
From: ceph-devel-***@vger.kernel.org [mailto:ceph-devel-***@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Thursday, September 18, 2014 2:25 PM
To: Sage Weil
Cc: Haomai Wang; Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

Sage,
Any reason why the cache is by default enabled in Giant ?
Regarding profiling, I will try if I can run Vtune/mutrace on this.

Thanks & Regards
Somnath

-----Original Message-----
From: Sage Weil [mailto:***@redhat.com]
Sent: Wednesday, September 17, 2014 8:53 PM
To: Somnath Roy
Cc: Haomai Wang; Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

Post by Somnath Roy
Yes Haomai...

I would love to what a profiler says about the matter. There is going to be some overhead on the client associated with the cache for a random io workload, but 10x is a problem!

sage

Post by Somnath Roy
-----Original Message-----
Sent: Wednesday, September 17, 2014 7:28 PM
To: Somnath Roy
Subject: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
N?????r??y??????X???v???)?{.n?????z?]z????ay?????j ??f???h??????w???

???j:+v???w???????? ????zZ+???????j"????i
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to ***@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Sage Weil]

Post by Somnath Roy
Sage,
Any reason why the cache is by default enabled in Giant ?

It's recommended practice to turn it on. It improves performance in
general (especially with HDD OSDs). Do you mind comparing sequential
small IOs?

sage

Post by Somnath Roy
Regarding profiling, I will try if I can run Vtune/mutrace on this.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 8:53 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant

Post by Somnath Roy
Yes Haomai...

I would love to what a profiler says about the matter. There is going to be some overhead on the client associated with the cache for a random io workload, but 10x is a problem!
sage

Post by Somnath Roy
-----Original Message-----
Sent: Wednesday, September 17, 2014 7:28 PM
To: Somnath Roy
Subject: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Somnath Roy
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
N?????r??y??????X???v???)?{.n?????z?]z????ay?????j ??f???h??????w???

???j:+v???w???????? ????zZ+???????j"????i
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Alexandre DERUMIER]

According http://tracker.ceph.com/issues/9513, do you mean that rbd=20
cache will make 10x performance degradation for random read?=20

Hi, on my side, I don't see any degradation performance on read (seq or=
rand) with or without.

firefly : around 12000iops (with or without rbd_cache)
giant : around 12000iops (with or without rbd_cache)

(and I can reach around 20000-30000 iops on giant with disabling optrac=
ker).


rbd_cache only improve write performance for me (4k block )



----- Mail original -----=20

De: "Haomai Wang" <***@gmail.com>=20
=C3=80: "Somnath Roy" <***@sandisk.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org=20
Envoy=C3=A9: Jeudi 18 Septembre 2014 04:27:56=20
Objet: Re: severe librbd performance degradation in Giant=20

According http://tracker.ceph.com/issues/9513, do you mean that rbd=20
cache will make 10x performance degradation for random read?=20

On Thu, Sep 18, 2014 at 7:44 AM, Somnath Roy <***@sandisk.com> =
wrote:=20

Josh/Sage,=20
I should mention that even after turning off rbd cache I am getting ~=

20% degradation over Firefly.=20

=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
From: Somnath Roy=20
Sent: Wednesday, September 17, 2014 2:44 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Created a tracker for this.=20
=20
http://tracker.ceph.com/issues/9513=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20

kernel.org] On Behalf Of Somnath Roy=20

Sent: Wednesday, September 17, 2014 2:39 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Sage,=20
It's a 4K random read.=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:36 PM=20
To: Somnath Roy=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
What was the io pattern? Sequential or random? For random a slowdown =

makes sense (tho maybe not 10x!) but not for sequentail....=20

=20
s=20
=20
On Wed, 17 Sep 2014, Somnath Roy wrote:=20
=20

I set the following in the client side /etc/ceph/ceph.conf where I a=

m running fio rbd.=20

=20
rbd_cache_writethrough_until_flush =3D false=20
=20
But, no difference. BTW, I am doing Random read, not write. Still th=

is setting applies ?=20

=20
Next, I tried to tweak the rbd_cache setting to false and I *got bac=

k* the old performance. Now, it is similar to firefly throughput !=20

=20
So, loks like rbd_cache=3Dtrue was the culprit.=20
=20
Thanks Josh !=20
=20
Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:20 PM=20
Subject: Re: severe librbd performance degradation in Giant=20
=20
On 09/17/2014 01:55 PM, Somnath Roy wrote:=20

Hi Sage,=20
We are experiencing severe librbd performance degradation in Giant=

over firefly release. Here is the experiment we did to isolate it as a=
librbd problem.=20

=20
1. Single OSD is running latest Giant and client is running fio rb=

d on top of firefly based librbd/librados. For one client it is giving =
~11-12K iops (4K RR).=20

2. Single OSD is running Giant and client is running fio rbd on to=

p of Giant based librbd/librados. For one client it is giving ~1.9K iop=
s (4K RR).=20

3. Single OSD is running latest Giant and client is running Giant =

based ceph_smaiobench on top of giant librados. For one client it is gi=
ving ~11-12K iops (4K RR).=20

4. Giant RGW on top of Giant OSD is also scaling.=20
=20
=20
So, it is obvious from the above that recent librbd has issues. I =

will raise a tracker to track this.=20

=20
For giant the default cache settings changed to:=20
=20
rbd cache =3D true=20
rbd cache writethrough until flush =3D true=20
=20
If fio isn't sending flushes as the test is running, the cache will =

stay in writethrough mode. Does the difference remain if you set rbd ca=
che writethrough until flush =3D false ?=20

=20
Josh=20
=20
________________________________=20
=20
PLEASE NOTE: The information contained in this electronic mail messa=

ge is intended only for the use of the designated recipient(s) named ab=
ove. If the reader of this message is not the intended recipient, you a=
re hereby notified that you have received this message in error and tha=
t any review, dissemination, distribution, or copying of this message i=
s strictly prohibited. If you have received this communication in error=
, please notify the sender by telephone or e-mail (as shown above) imme=
diately and destroy any and all copies of this message in your possessi=
on (whether hard copies or electronically stored copies).=20

=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

"=20
=20

info at http://vger.kernel.org/majordomo-info.html=20
=20
=20

--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel"=

in the body of a message to ***@vger.kernel.org More majordomo i=
nfo at http://vger.kernel.org/majordomo-info.html=20

--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel"=

in=20

More majordomo info at http://vger.kernel.org/majordomo-info.html=20

--=20
Best Regards,=20

Wheat=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n=20
the body of a message to ***@vger.kernel.org=20
More majordomo info at http://vger.kernel.org/majordomo-info.html=20
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Mark Nelson]

Post by Somnath Roy

Post by Haomai Wang
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq =

or rand) with or without.

firefly : around 12000iops (with or without rbd_cache)
giant : around 12000iops (with or without rbd_cache)
(and I can reach around 20000-30000 iops on giant with disabling optr=

acker).

rbd_cache only improve write performance for me (4k block )

I can't do it right now since I'm in the middle of reinstalling fedora=20
on the test nodes, but I will try to replicate this as well if we=20
haven't figured it out before hand.

Mark

----- Mail original -----
Envoy=C3=A9: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting =

~20% degradation over Firefly.

Post by Somnath Roy
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----

=2Ekernel.org] On Behalf Of Somnath Roy

Post by Somnath Roy
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown=

makes sense (tho maybe not 10x!) but not for sequentail....

Post by Somnath Roy
s

Post by Haomai Wang
I set the following in the client side /etc/ceph/ceph.conf where I =

am running fio rbd.

Post by Somnath Roy

Post by Haomai Wang
rbd_cache_writethrough_until_flush =3D false
But, no difference. BTW, I am doing Random read, not write. Still t=

his setting applies ?

Post by Somnath Roy

Post by Haomai Wang
Next, I tried to tweak the rbd_cache setting to false and I *got ba=

ck* the old performance. Now, it is similar to firefly throughput !

Post by Somnath Roy

Post by Haomai Wang
So, loks like rbd_cache=3Dtrue was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant=

over firefly release. Here is the experiment we did to isolate it as a=
librbd problem.

Post by Somnath Roy

Post by Haomai Wang

Post by Somnath Roy
1. Single OSD is running latest Giant and client is running fio rb=

d on top of firefly based librbd/librados. For one client it is giving =
~11-12K iops (4K RR).

Post by Somnath Roy

Post by Haomai Wang

Post by Somnath Roy
2. Single OSD is running Giant and client is running fio rbd on to=

p of Giant based librbd/librados. For one client it is giving ~1.9K iop=
s (4K RR).

Post by Somnath Roy

Post by Haomai Wang

Post by Somnath Roy
3. Single OSD is running latest Giant and client is running Giant =

based ceph_smaiobench on top of giant librados. For one client it is gi=
ving ~11-12K iops (4K RR).

Post by Somnath Roy

Post by Haomai Wang

Post by Somnath Roy
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I =

will raise a tracker to track this.

Post by Somnath Roy

Post by Haomai Wang
rbd cache =3D true
rbd cache writethrough until flush =3D true
If fio isn't sending flushes as the test is running, the cache will=

stay in writethrough mode. Does the difference remain if you set rbd c=
ache writethrough until flush =3D false ?

Post by Somnath Roy

Post by Haomai Wang
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail mess=

age is intended only for the use of the designated recipient(s) named a=
bove. If the reader of this message is not the intended recipient, you =
are hereby notified that you have received this message in error and th=
at any review, dissemination, distribution, or copying of this message =
is strictly prohibited. If you have received this communication in erro=
r, please notify the sender by telephone or e-mail (as shown above) imm=
ediately and destroy any and all copies of this message in your possess=
ion (whether hard copies or electronically stored copies).

Post by Somnath Roy

Post by Haomai Wang
--
To unsubscribe from this list: send the line "unsubscribe ceph-deve=

l"
o

Post by Somnath Roy

Post by Haomai Wang
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

" in the body of a message to ***@vger.kernel.org More majordomo =
info at http://vger.kernel.org/majordomo-info.html

Post by Somnath Roy
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

" in

Post by Somnath Roy
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Somnath Roy]

Alexandre,
What tool are you using ? I used fio rbd.

Also, I hope you have Giant package installed in the client side as well and rbd_cache =true is set on the client conf file.
FYI, firefly librbd + librados and Giant cluster will work seamlessly and I had to make sure fio rbd is really loading giant librbd (if you have multiple copies around , which was in my case) for reproducing it.

Thanks & Regards
Somnath

-----Original Message-----
From: Alexandre DERUMIER [mailto:***@odiso.com]
Sent: Thursday, September 18, 2014 2:49 AM
To: Haomai Wang
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org; Somnath Roy
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

Post by Haomai Wang
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq or rand) with or without.

firefly : around 12000iops (with or without rbd_cache) giant : around 12000iops (with or without rbd_cache)

(and I can reach around 20000-30000 iops on giant with disabling optracker).


rbd_cache only improve write performance for me (4k block )



----- Mail original -----

De: "Haomai Wang" <***@gmail.com>
À: "Somnath Roy" <***@sandisk.com>
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.com>, ceph-***@vger.kernel.org
Envoyé: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant

According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Haomai Wang
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

[Shu, Xinxin]

I also observed performance degradation on my full SSD setup , I can got ~270K IOPS for 4KB random read with 0.80.4 , but with latest master , I only got ~12K IOPS

Cheers,
xinxin

-----Original Message-----
From: ceph-devel-***@vger.kernel.org [mailto:ceph-devel-***@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Friday, September 19, 2014 2:03 AM
To: Alexandre DERUMIER; Haomai Wang
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

Alexandre,
What tool are you using ? I used fio rbd.

Also, I hope you have Giant package installed in the client side as well and rbd_cache =true is set on the client conf file.
FYI, firefly librbd + librados and Giant cluster will work seamlessly and I had to make sure fio rbd is really loading giant librbd (if you have multiple copies around , which was in my case) for reproducing it.

Thanks & Regards
Somnath

-----Original Message-----
From: Alexandre DERUMIER [mailto:***@odiso.com]
Sent: Thursday, September 18, 2014 2:49 AM
To: Haomai Wang
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org; Somnath Roy
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

Post by Haomai Wang
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq or rand) with or without.

firefly : around 12000iops (with or without rbd_cache) giant : around 12000iops (with or without rbd_cache)

(and I can reach around 20000-30000 iops on giant with disabling optracker).


rbd_cache only improve write performance for me (4k block )



----- Mail original -----

De: "Haomai Wang" <***@gmail.com>
À: "Somnath Roy" <***@sandisk.com>
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.com>, ceph-***@vger.kernel.org
Envoyé: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant

According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Haomai Wang
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

[Shu, Xinxin]

My bad , with latest master , we got ~ 120K IOPS.

Cheers,
xinxin

-----Original Message-----
From: ceph-devel-***@vger.kernel.org [mailto:ceph-devel-***@vger.kernel.org] On Behalf Of Shu, Xinxin
Sent: Friday, September 19, 2014 9:08 AM
To: Somnath Roy; Alexandre DERUMIER; Haomai Wang
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

I also observed performance degradation on my full SSD setup , I can got ~270K IOPS for 4KB random read with 0.80.4 , but with latest master , I only got ~12K IOPS

Cheers,
xinxin

-----Original Message-----
From: ceph-devel-***@vger.kernel.org [mailto:ceph-devel-***@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Friday, September 19, 2014 2:03 AM
To: Alexandre DERUMIER; Haomai Wang
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org
Subject: RE: severe librbd performance degradation in Giant

Alexandre,
What tool are you using ? I used fio rbd.

Also, I hope you have Giant package installed in the client side as well and rbd_cache =true is set on the client conf file.
FYI, firefly librbd + librados and Giant cluster will work seamlessly and I had to make sure fio rbd is really loading giant librbd (if you have multiple copies around , which was in my case) for reproducing it.

Thanks & Regards
Somnath

-----Original Message-----
From: Alexandre DERUMIER [mailto:***@odiso.com]
Sent: Thursday, September 18, 2014 2:49 AM
To: Haomai Wang
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org; Somnath Roy
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

Post by Haomai Wang
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq or rand) with or without.

firefly : around 12000iops (with or without rbd_cache) giant : around 12000iops (with or without rbd_cache)

(and I can reach around 20000-30000 iops on giant with disabling optracker).


rbd_cache only improve write performance for me (4k block )



----- Mail original -----

De: "Haomai Wang" <***@gmail.com>
À: "Somnath Roy" <***@sandisk.com>
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.com>, ceph-***@vger.kernel.org
Envoyé: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant

According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Haomai Wang
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

[Stefan Priebe]

I also observed performance degradation on my full SSD setup , I can=

got ~270K IOPS for 4KB random read with 0.80.4 , but with latest mast=
er , I only got ~12K IOPS

This are impressive numbers. Can you tell me how many OSDs you have and=
=20
which SSDs you use?

Thanks,
Stefan

Cheers,
xinxin
-----Original Message-----

kernel.org] On Behalf Of Somnath Roy

Sent: Friday, September 19, 2014 2:03 AM
To: Alexandre DERUMIER; Haomai Wang
Subject: RE: severe librbd performance degradation in Giant
Alexandre,
What tool are you using ? I used fio rbd.
Also, I hope you have Giant package installed in the client side as w=

ell and rbd_cache =3Dtrue is set on the client conf file.

FYI, firefly librbd + librados and Giant cluster will work seamlessly=

and I had to make sure fio rbd is really loading giant librbd (if you =
have multiple copies around , which was in my case) for reproducing it.

Thanks & Regards
Somnath
-----Original Message-----
Sent: Thursday, September 18, 2014 2:49 AM
To: Haomai Wang
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

Post by Haomai Wang
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq =

or rand) with or without.

firefly : around 12000iops (with or without rbd_cache) giant : around=

12000iops (with or without rbd_cache)

(and I can reach around 20000-30000 iops on giant with disabling optr=

acker).

rbd_cache only improve write performance for me (4k block )
----- Mail original -----
Envoy=C3=A9: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd c=

ache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting =

~20% degradation over Firefly.

Post by Somnath Roy
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown=

makes sense (tho maybe not 10x!) but not for sequentail....

Post by Somnath Roy
s

Post by Haomai Wang
I set the following in the client side /etc/ceph/ceph.conf where I =

am running fio rbd.

Post by Somnath Roy

Post by Haomai Wang
rbd_cache_writethrough_until_flush =3D false
But, no difference. BTW, I am doing Random read, not write. Still t=

his setting applies ?

Post by Somnath Roy

Post by Haomai Wang
Next, I tried to tweak the rbd_cache setting to false and I *got ba=

ck* the old performance. Now, it is similar to firefly throughput !

Post by Somnath Roy

Post by Haomai Wang
So, loks like rbd_cache=3Dtrue was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant=

over firefly release. Here is the experiment we did to isolate it as a=
librbd problem.

Post by Somnath Roy

Post by Haomai Wang

Post by Somnath Roy
1. Single OSD is running latest Giant and client is running fio rb=

d on top of firefly based librbd/librados. For one client it is giving =
~11-12K iops (4K RR).

Post by Somnath Roy

Post by Haomai Wang

Post by Somnath Roy
2. Single OSD is running Giant and client is running fio rbd on to=

p of Giant based librbd/librados. For one client it is giving ~1.9K iop=
s (4K RR).

Post by Somnath Roy

Post by Haomai Wang

Post by Somnath Roy
3. Single OSD is running latest Giant and client is running Giant =

based ceph_smaiobench on top of giant librados. For one client it is gi=
ving ~11-12K iops (4K RR).

Post by Somnath Roy

Post by Haomai Wang

Post by Somnath Roy
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I =

will raise a tracker to track this.

Post by Somnath Roy

Post by Haomai Wang
rbd cache =3D true
rbd cache writethrough until flush =3D true
If fio isn't sending flushes as the test is running, the cache will=

stay in writethrough mode. Does the difference remain if you set rbd c=
ache writethrough until flush =3D false ?

Post by Somnath Roy

Post by Haomai Wang
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail mess=

age is intended only for the use of the designated recipient(s) named a=
bove. If the reader of this message is not the intended recipient, you =
are hereby notified that you have received this message in error and th=
at any review, dissemination, distribution, or copying of this message =
is strictly prohibited. If you have received this communication in erro=
r, please notify the sender by telephone or e-mail (as shown above) imm=
ediately and destroy any and all copies of this message in your possess=
ion (whether hard copies or electronically stored copies).

Post by Somnath Roy

Post by Haomai Wang
--
To unsubscribe from this list: send the line "unsubscribe ceph-deve=

l"
o

Post by Somnath Roy

Post by Haomai Wang
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

"

Post by Somnath Roy
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

"

Post by Somnath Roy
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"=

in the body of a message to ***@vger.kernel.org More majordomo i=
nfo at http://vger.kernel.org/majordomo-info.html

N r y b X =C7=A7v ^ )=DE=BA{.n + z ]z {ay =1D=CA=87=DA=99=

,j f h z =1E w j:+v w j m zZ+ =DD=A2j" ! i

N=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDr=EF=BF=BD=EF=BF=BDy=EF=

=BF=BD=EF=BF=BD=EF=BF=BDb=EF=BF=BDX=EF=BF=BD=EF=BF=BD=C7=A7v=EF=BF=BD^=EF=
=BF=BD)=DE=BA{.n=EF=BF=BD+=EF=BF=BD=EF=BF=BD=EF=BF=BDz=EF=BF=BD]z=EF=BF=
=BD=EF=BF=BD=EF=BF=BD{ay=EF=BF=BD=1D=CA=87=DA=99=EF=BF=BD,j=07=EF=BF=BD=
=EF=BF=BDf=EF=BF=BD=EF=BF=BD=EF=BF=BDh=EF=BF=BD=EF=BF=BD=EF=BF=BDz=EF=BF=
=BD=1E=EF=BF=BDw=EF=BF=BD=EF=BF=BD=EF=BF=BD=0C=EF=BF=BD=EF=BF=BD=EF=BF=BD=
j:+v=EF=BF=BD=EF=BF=BD=EF=BF=BDw=EF=BF=BDj=EF=BF=BDm=EF=BF=BD=EF=BF=BD=EF=
=BF=BD=EF=BF=BD=07=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDzZ+=EF=BF=BD=EF=BF=
=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=DD=A2j"=EF=BF=BD=EF=BF=BD!tml=3D
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Shu, Xinxin]

12 x Intel DC 3700 200GB, every SSD has two OSDs.

Cheers,
xinxin

-----Original Message-----
From: Stefan Priebe [mailto:***@profihost.ag]
Sent: Friday, September 19, 2014 2:54 PM
To: Shu, Xinxin; Somnath Roy; Alexandre DERUMIER; Haomai Wang
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org
Subject: Re: severe librbd performance degradation in Giant

Post by Shu, Xinxin
I also observed performance degradation on my full SSD setup , I can
got ~270K IOPS for 4KB random read with 0.80.4 , but with latest
master , I only got ~12K IOPS

This are impressive numbers. Can you tell me how many OSDs you have and which SSDs you use?

Thanks,
Stefan

Post by Shu, Xinxin
Cheers,
xinxin
-----Original Message-----
Sent: Friday, September 19, 2014 2:03 AM
To: Alexandre DERUMIER; Haomai Wang
Subject: RE: severe librbd performance degradation in Giant
Alexandre,
What tool are you using ? I used fio rbd.
Also, I hope you have Giant package installed in the client side as well and rbd_cache =true is set on the client conf file.
FYI, firefly librbd + librados and Giant cluster will work seamlessly and I had to make sure fio rbd is really loading giant librbd (if you have multiple copies around , which was in my case) for reproducing it.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Thursday, September 18, 2014 2:49 AM
To: Haomai Wang
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

Post by Haomai Wang
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq or rand) with or without.
firefly : around 12000iops (with or without rbd_cache) giant : around
12000iops (with or without rbd_cache)
(and I can reach around 20000-30000 iops on giant with disabling optracker).
rbd_cache only improve write performance for me (4k block )
----- Mail original -----
Envoyé: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Haomai Wang
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
--
N r y b X ǧv ^ )޺{.n + z ]z {ay ʇڙ ,j f h z  w j:+v w j m zZ+ ݢj" ! i
N r y b X ǧv ^ )޺{.n + z ]z {ay ʇڙ ,j f h z  w

j:+v w j m zZ+ ݢj" !tml=
��칻�&�~�&���+-��ݶ��w��˛���m��^��b��^n�r���z���h�����&���G���h�

[Stefan Priebe - Profihost AG]

Post by Shu, Xinxin
12 x Intel DC 3700 200GB, every SSD has two OSDs.

Crazy, I've 56 SSDs and can=C3=84t go above 20 000 iops.

Gr=C3=BC=C3=9Fe Stefan

Post by Shu, Xinxin
Cheers,
xinxin
=20
-----Original Message-----
Sent: Friday, September 19, 2014 2:54 PM
To: Shu, Xinxin; Somnath Roy; Alexandre DERUMIER; Haomai Wang
Subject: Re: severe librbd performance degradation in Giant
=20

I also observed performance degradation on my full SSD setup , I ca=

n=20

Post by Shu, Xinxin

got ~270K IOPS for 4KB random read with 0.80.4 , but with latest=20
master , I only got ~12K IOPS

=20
This are impressive numbers. Can you tell me how many OSDs you have a=

nd which SSDs you use?

Post by Shu, Xinxin
=20
Thanks,
Stefan
=20
=20

Cheers,
xinxin
-----Original Message-----
Sent: Friday, September 19, 2014 2:03 AM
To: Alexandre DERUMIER; Haomai Wang
Subject: RE: severe librbd performance degradation in Giant
Alexandre,
What tool are you using ? I used fio rbd.
Also, I hope you have Giant package installed in the client side as =

well and rbd_cache =3Dtrue is set on the client conf file.

Post by Shu, Xinxin

FYI, firefly librbd + librados and Giant cluster will work seamlessl=

y and I had to make sure fio rbd is really loading giant librbd (if you=
have multiple copies around , which was in my case) for reproducing it=
=2E

Post by Shu, Xinxin

Thanks & Regards
Somnath
-----Original Message-----
Sent: Thursday, September 18, 2014 2:49 AM
To: Haomai Wang
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

According http://tracker.ceph.com/issues/9513, do you mean that rb=

d=20

Post by Shu, Xinxin

Post by Somnath Roy

cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq=

or rand) with or without.

Post by Shu, Xinxin

firefly : around 12000iops (with or without rbd_cache) giant : aroun=

d=20

Post by Shu, Xinxin

12000iops (with or without rbd_cache)
(and I can reach around 20000-30000 iops on giant with disabling opt=

racker).

Post by Shu, Xinxin

rbd_cache only improve write performance for me (4k block )
----- Mail original -----
Envoy=C3=A9: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd =

cache will make 10x performance degradation for random read?

Post by Shu, Xinxin

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting=

~20% degradation over Firefly.

Post by Shu, Xinxin

Post by Somnath Roy
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdow=

n makes sense (tho maybe not 10x!) but not for sequentail....

Post by Shu, Xinxin

Post by Somnath Roy
s

I set the following in the client side /etc/ceph/ceph.conf where I=

am running fio rbd.

Post by Shu, Xinxin

Post by Somnath Roy

rbd_cache_writethrough_until_flush =3D false
But, no difference. BTW, I am doing Random read, not write. Still =

this setting applies ?

Post by Shu, Xinxin

Post by Somnath Roy

Next, I tried to tweak the rbd_cache setting to false and I *got b=

ack* the old performance. Now, it is similar to firefly throughput !

Post by Shu, Xinxin

Post by Somnath Roy

So, loks like rbd_cache=3Dtrue was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Gian=

t over firefly release. Here is the experiment we did to isolate it as =
a librbd problem.

Post by Shu, Xinxin

Post by Somnath Roy

Post by Somnath Roy
1. Single OSD is running latest Giant and client is running fio r=

bd on top of firefly based librbd/librados. For one client it is giving=
~11-12K iops (4K RR).

Post by Shu, Xinxin

Post by Somnath Roy

Post by Somnath Roy
2. Single OSD is running Giant and client is running fio rbd on t=

op of Giant based librbd/librados. For one client it is giving ~1.9K io=
ps (4K RR).

Post by Shu, Xinxin

Post by Somnath Roy

Post by Somnath Roy
3. Single OSD is running latest Giant and client is running Giant=

based ceph_smaiobench on top of giant librados. For one client it is g=
iving ~11-12K iops (4K RR).

Post by Shu, Xinxin

Post by Somnath Roy

Post by Somnath Roy
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I=

will raise a tracker to track this.

Post by Shu, Xinxin

Post by Somnath Roy

rbd cache =3D true
rbd cache writethrough until flush =3D true
If fio isn't sending flushes as the test is running, the cache wil=

l stay in writethrough mode. Does the difference remain if you set rbd =
cache writethrough until flush =3D false ?

Post by Shu, Xinxin

Post by Somnath Roy

Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail mes=

sage is intended only for the use of the designated recipient(s) named =
above. If the reader of this message is not the intended recipient, you=
are hereby notified that you have received this message in error and t=
hat any review, dissemination, distribution, or copying of this message=
is strictly prohibited. If you have received this communication in err=
or, please notify the sender by telephone or e-mail (as shown above) im=
mediately and destroy any and all copies of this message in your posses=
sion (whether hard copies or electronically stored copies).

Post by Shu, Xinxin

Post by Somnath Roy

--
To unsubscribe from this list: send the line "unsubscribe ceph-dev=

el"
mo=20

Post by Shu, Xinxin

Post by Somnath Roy

info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-deve=

l"
o=20

Post by Shu, Xinxin

Post by Somnath Roy
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-deve=

l"
o=20

Post by Shu, Xinxin

Post by Somnath Roy
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

" in the body of a message to ***@vger.kernel.org More majordomo =
info at http://vger.kernel.org/majordomo-info.html

Post by Shu, Xinxin

N r y b X =C7=A7v ^ )=DE=BA{.n + z ]z {ay =1D=CA=87=DA=99=

,j f h z =1E w j:+v w j m zZ+ =DD=A2j" ! i

Post by Shu, Xinxin

N r y b X =C7=A7v ^ )=DE=BA{.n + z ]z {ay =1D=CA=87=DA=99=

,j f h z =1E w =20

Post by Shu, Xinxin
j:+v w j m zZ+ =DD=A2j" !tml=3D
N=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDr=EF=BF=BD=EF=BF=BDy=EF=

=BF=BD=EF=BF=BD=EF=BF=BDb=EF=BF=BDX=EF=BF=BD=EF=BF=BD=C7=A7v=EF=BF=BD^=EF=
=BF=BD)=DE=BA{.n=EF=BF=BD+=EF=BF=BD=EF=BF=BD=EF=BF=BDz=EF=BF=BD]z=EF=BF=
=BD=EF=BF=BD=EF=BF=BD{ay=EF=BF=BD=1D=CA=87=DA=99=EF=BF=BD,j=07=EF=BF=BD=
=EF=BF=BDf=EF=BF=BD=EF=BF=BD=EF=BF=BDh=EF=BF=BD=EF=BF=BD=EF=BF=BDz=EF=BF=
=BD=1E=EF=BF=BDw=EF=BF=BD=EF=BF=BD=EF=BF=BD=0C=EF=BF=BD=EF=BF=BD=EF=BF=BD=
j:+v=EF=BF=BD=EF=BF=BD=EF=BF=BDw=EF=BF=BDj=EF=BF=BDm=EF=BF=BD=EF=BF=BD=EF=
=BF=BD=EF=BF=BD=07=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDzZ+=EF=BF=BD=EF=BF=
=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=DD=A2j"=EF=BF=BD=EF=BF=BD!tml=3D

Post by Shu, Xinxin
=20

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[David Moreau Simard]

Numbers vary a lot from brand to brand and from model to model.

Just within Intel, you'd be surprised at the large difference between DC
S3500 and DC S3700:
http://ark.intel.com/compare/75680,71914
--
David Moreau Simard


Le 2014-09-19, 9:31 AM, « Stefan Priebe - Profihost AG »

Post by Shu, Xinxin
12 x Intel DC 3700 200GB, every SSD has two OSDs.

Crazy, I've 56 SSDs and canÄt go above 20 000 iops.
Grüße Stefan

Post by Shu, Xinxin
Cheers,
xinxin
-----Original Message-----
Sent: Friday, September 19, 2014 2:54 PM
To: Shu, Xinxin; Somnath Roy; Alexandre DERUMIER; Haomai Wang
Subject: Re: severe librbd performance degradation in Giant

Post by Shu, Xinxin
I also observed performance degradation on my full SSD setup , I can
got ~270K IOPS for 4KB random read with 0.80.4 , but with latest
master , I only got ~12K IOPS

This are impressive numbers. Can you tell me how many OSDs you have and
which SSDs you use?
Thanks,
Stefan

Post by Shu, Xinxin
Cheers,
xinxin
-----Original Message-----
Sent: Friday, September 19, 2014 2:03 AM
To: Alexandre DERUMIER; Haomai Wang
Subject: RE: severe librbd performance degradation in Giant
Alexandre,
What tool are you using ? I used fio rbd.
Also, I hope you have Giant package installed in the client side as
well and rbd_cache =true is set on the client conf file.
FYI, firefly librbd + librados and Giant cluster will work seamlessly
and I had to make sure fio rbd is really loading giant librbd (if you
have multiple copies around , which was in my case) for reproducing it.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Thursday, September 18, 2014 2:49 AM
To: Haomai Wang
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

Post by Haomai Wang
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq
or rand) with or without.
firefly : around 12000iops (with or without rbd_cache) giant : around
12000iops (with or without rbd_cache)
(and I can reach around 20000-30000 iops on giant with disabling
optracker).
rbd_cache only improve write performance for me (4k block )
----- Mail original -----
Envoyé: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting
~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown
makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Haomai Wang
I set the following in the client side /etc/ceph/ceph.conf where I
am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still
this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got
back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant
over firefly release. Here is the experiment we did to isolate it as
a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd
on top of firefly based librbd/librados. For one client it is giving
~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top
of Giant based librbd/librados. For one client it is giving ~1.9K
iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant
based ceph_smaiobench on top of giant librados. For one client it is
giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I
will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will
stay in writethrough mode. Does the difference remain if you set rbd
cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail
message is intended only for the use of the designated recipient(s)
named above. If the reader of this message is not the intended
recipient, you are hereby notified that you have received this
message in error and that any review, dissemination, distribution, or
copying of this message is strictly prohibited. If you have received
this communication in error, please notify the sender by telephone or
e-mail (as shown above) immediately and destroy any and all copies of
this message in your possession (whether hard copies or
electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
N r y b X ǧv ^ )޺{.n + z ]z {ay ʇڙ ,j f h z  w
j:+v w j m zZ+ ݢj" ! i
N r y b X ǧv ^ )޺{.n + z ]z {ay ʇڙ ,j f h z  w

j:+v w j m zZ+ ݢj" !tml=
N�����r��y���b�X��ǧv�^�)޺{.n�+���z�]z���{ay�ʇڙ�,j��f���h���z��w������
j:+v���w�j�m��������zZ+�����ݢj"��!tml=

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

��칻�&�~�&���+-��ݶ��w��˛���m��^��b��^n�r���z���h�����&���G���h�

[Alexandre DERUMIER]

Post by Stefan Priebe - Profihost AG
Crazy, I've 56 SSDs and can=C3=84t go above 20 000 iops.

I just notice than my fio benchmark is cpu bound...

I can reach around 40000iops. Don't have more client machines for the m=
oment to bench


----- Mail original -----=20

De: "Stefan Priebe - Profihost AG" <***@profihost.ag>=20
=C3=80: "Xinxin Shu" <***@intel.com>, "Somnath Roy" <Somnath.Roy=
@sandisk.com>, "Alexandre DERUMIER" <***@odiso.com>, "Haomai Wang=
" <***@gmail.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org=20
Envoy=C3=A9: Vendredi 19 Septembre 2014 15:31:14=20
Objet: Re: severe librbd performance degradation in Giant=20

Am 19.09.2014 um 15:02 schrieb Shu, Xinxin:=20

12 x Intel DC 3700 200GB, every SSD has two OSDs.=20

Crazy, I've 56 SSDs and can=C3=84t go above 20 000 iops.=20

Gr=C3=BC=C3=9Fe Stefan=20

Cheers,=20
xinxin=20
=20
-----Original Message-----=20
Sent: Friday, September 19, 2014 2:54 PM=20
To: Shu, Xinxin; Somnath Roy; Alexandre DERUMIER; Haomai Wang=20
Subject: Re: severe librbd performance degradation in Giant=20
=20
Am 19.09.2014 03:08, schrieb Shu, Xinxin:=20

Post by Stefan Priebe - Profihost AG
I also observed performance degradation on my full SSD setup , I can=

=20

Post by Stefan Priebe - Profihost AG
got ~270K IOPS for 4KB random read with 0.80.4 , but with latest=20
master , I only got ~12K IOPS=20

=20
This are impressive numbers. Can you tell me how many OSDs you have a=

nd which SSDs you use?=20

=20
Thanks,=20
Stefan=20
=20
=20

Post by Stefan Priebe - Profihost AG
Cheers,=20
xinxin=20
=20
-----Original Message-----=20
Sent: Friday, September 19, 2014 2:03 AM=20
To: Alexandre DERUMIER; Haomai Wang=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Alexandre,=20
What tool are you using ? I used fio rbd.=20
=20
Also, I hope you have Giant package installed in the client side as =

well and rbd_cache =3Dtrue is set on the client conf file.=20

Post by Stefan Priebe - Profihost AG
FYI, firefly librbd + librados and Giant cluster will work seamlessl=

y and I had to make sure fio rbd is really loading giant librbd (if you=
have multiple copies around , which was in my case) for reproducing it=
=2E=20

Post by Stefan Priebe - Profihost AG
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Thursday, September 18, 2014 2:49 AM=20
To: Haomai Wang=20
Subject: Re: severe librbd performance degradation in Giant=20
=20

According http://tracker.ceph.com/issues/9513, do you mean that rb=

d=20

Post by Stefan Priebe - Profihost AG

cache will make 10x performance degradation for random read?=20

=20
Hi, on my side, I don't see any degradation performance on read (seq=

or rand) with or without.=20

Post by Stefan Priebe - Profihost AG
=20
firefly : around 12000iops (with or without rbd_cache) giant : aroun=

d=20

Post by Stefan Priebe - Profihost AG
12000iops (with or without rbd_cache)=20
=20
(and I can reach around 20000-30000 iops on giant with disabling opt=

racker).=20

Post by Stefan Priebe - Profihost AG
=20
=20
rbd_cache only improve write performance for me (4k block )=20
=20
=20
=20
----- Mail original -----=20
=20
Envoy=C3=A9: Jeudi 18 Septembre 2014 04:27:56=20
Objet: Re: severe librbd performance degradation in Giant=20
=20
According http://tracker.ceph.com/issues/9513, do you mean that rbd =

cache will make 10x performance degradation for random read?=20

Post by Stefan Priebe - Profihost AG
=20

m> wrote:=20

Post by Stefan Priebe - Profihost AG

Josh/Sage,=20
I should mention that even after turning off rbd cache I am getting=

~20% degradation over Firefly.=20

Post by Stefan Priebe - Profihost AG

=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
From: Somnath Roy=20
Sent: Wednesday, September 17, 2014 2:44 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Created a tracker for this.=20
=20
http://tracker.ceph.com/issues/9513=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:39 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Sage,=20
It's a 4K random read.=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:36 PM=20
To: Somnath Roy=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
What was the io pattern? Sequential or random? For random a slowdow=

n makes sense (tho maybe not 10x!) but not for sequentail....=20

Post by Stefan Priebe - Profihost AG

=20
s=20
=20
On Wed, 17 Sep 2014, Somnath Roy wrote:=20
=20

I set the following in the client side /etc/ceph/ceph.conf where I=

am running fio rbd.=20

Post by Stefan Priebe - Profihost AG

=20
rbd_cache_writethrough_until_flush =3D false=20
=20
But, no difference. BTW, I am doing Random read, not write. Still =

this setting applies ?=20

Post by Stefan Priebe - Profihost AG

=20
Next, I tried to tweak the rbd_cache setting to false and I *got b=

ack* the old performance. Now, it is similar to firefly throughput !=20

Post by Stefan Priebe - Profihost AG

=20
So, loks like rbd_cache=3Dtrue was the culprit.=20
=20
Thanks Josh !=20
=20
Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:20 PM=20
Subject: Re: severe librbd performance degradation in Giant=20
=20
On 09/17/2014 01:55 PM, Somnath Roy wrote:=20

Hi Sage,=20
We are experiencing severe librbd performance degradation in Gian=

t over firefly release. Here is the experiment we did to isolate it as =
a librbd problem.=20

Post by Stefan Priebe - Profihost AG

=20
1. Single OSD is running latest Giant and client is running fio r=

bd on top of firefly based librbd/librados. For one client it is giving=
~11-12K iops (4K RR).=20

Post by Stefan Priebe - Profihost AG

2. Single OSD is running Giant and client is running fio rbd on t=

op of Giant based librbd/librados. For one client it is giving ~1.9K io=
ps (4K RR).=20

Post by Stefan Priebe - Profihost AG

3. Single OSD is running latest Giant and client is running Giant=

based ceph_smaiobench on top of giant librados. For one client it is g=
iving ~11-12K iops (4K RR).=20

Post by Stefan Priebe - Profihost AG

4. Giant RGW on top of Giant OSD is also scaling.=20
=20
=20
So, it is obvious from the above that recent librbd has issues. I=

will raise a tracker to track this.=20

Post by Stefan Priebe - Profihost AG

=20
For giant the default cache settings changed to:=20
=20
rbd cache =3D true=20
rbd cache writethrough until flush =3D true=20
=20
If fio isn't sending flushes as the test is running, the cache wil=

l stay in writethrough mode. Does the difference remain if you set rbd =
cache writethrough until flush =3D false ?=20

Post by Stefan Priebe - Profihost AG

=20
Josh=20
=20
________________________________=20
=20
PLEASE NOTE: The information contained in this electronic mail mes=

sage is intended only for the use of the designated recipient(s) named =
above. If the reader of this message is not the intended recipient, you=
are hereby notified that you have received this message in error and t=
hat any review, dissemination, distribution, or copying of this message=
is strictly prohibited. If you have received this communication in err=
or, please notify the sender by telephone or e-mail (as shown above) im=
mediately and destroy any and all copies of this message in your posses=
sion (whether hard copies or electronically stored copies).=20

Post by Stefan Priebe - Profihost AG

=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-dev=

el"=20
mo=20

Post by Stefan Priebe - Profihost AG

info at http://vger.kernel.org/majordomo-info.html=20
=20
=20

--=20
To unsubscribe from this list: send the line "unsubscribe ceph-deve=

l"=20
o=20

Post by Stefan Priebe - Profihost AG

info at http://vger.kernel.org/majordomo-info.html=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-deve=

l"=20
o=20

Post by Stefan Priebe - Profihost AG

info at http://vger.kernel.org/majordomo-info.html=20

=20
=20
=20
--=20
Best Regards,=20
=20
Wheat=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

" in the body of a message to ***@vger.kernel.org More majordomo =
info at http://vger.kernel.org/majordomo-info.html=20

Post by Stefan Priebe - Profihost AG
N r y b X =C7=A7v ^ )=DE=BA{.n + z ]z {ay =CA=87=DA=99 ,j f h z w j:=

+v w j m zZ+ =DD=A2j" ! i=20

Post by Stefan Priebe - Profihost AG
N r y b X =C7=A7v ^ )=DE=BA{.n + z ]z {ay =CA=87=DA=99 ,j f h z w=20

j:+v w j m zZ+ =DD=A2j" !tml=3D=20

Post by Stefan Priebe - Profihost AG
=20

N=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDr=EF=BF=BD=EF=BF=BDy=EF=

=BF=BD=EF=BF=BD=EF=BF=BDb=EF=BF=BDX=EF=BF=BD=EF=BF=BD=C7=A7v=EF=BF=BD^=EF=
=BF=BD)=DE=BA{.n=EF=BF=BD+=EF=BF=BD=EF=BF=BD=EF=BF=BDz=EF=BF=BD]z=EF=BF=
=BD=EF=BF=BD=EF=BF=BD{ay=EF=BF=BD=CA=87=DA=99=EF=BF=BD,j=EF=BF=BD=EF=BF=
=BDf=EF=BF=BD=EF=BF=BD=EF=BF=BDh=EF=BF=BD=EF=BF=BD=EF=BF=BDz=EF=BF=BD=EF=
=BF=BDw=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDj:+v=EF=BF=
=BD=EF=BF=BD=EF=BF=BDw=EF=BF=BDj=EF=BF=BDm=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=
=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDzZ+=EF=BF=BD=EF=BF=BD=EF=BF=BD=
=EF=BF=BD=EF=BF=BD=DD=A2j"=EF=BF=BD=EF=BF=BD!tml=3D=20

=20

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Sage Weil]

Post by Alexandre DERUMIER

Post by Shu, Xinxin

Crazy, I've 56 SSDs and can?t go above 20 000 iops.

I just notice than my fio benchmark is cpu bound...
I can reach around 40000iops. Don't have more client machines for the moment to bench

A quick aside on the fio testing: Mark noticed a few weeks back that the
fio rbd driver is doing quite the right thing when you turn up the number
of threads: each one issues its own IOs but they touch the same blocks in
the image (or something like that). See

http://tracker.ceph.com/issues/9391

It would be great to get this fixed in fio...

sage

Post by Alexandre DERUMIER
----- Mail original -----
Envoy?: Vendredi 19 Septembre 2014 15:31:14
Objet: Re: severe librbd performance degradation in Giant

Post by Shu, Xinxin
12 x Intel DC 3700 200GB, every SSD has two OSDs.

Crazy, I've 56 SSDs and can?t go above 20 000 iops.
Gr??e Stefan

Post by Shu, Xinxin
Cheers,
xinxin
-----Original Message-----
Sent: Friday, September 19, 2014 2:54 PM
To: Shu, Xinxin; Somnath Roy; Alexandre DERUMIER; Haomai Wang
Subject: Re: severe librbd performance degradation in Giant

I also observed performance degradation on my full SSD setup , I can
got ~270K IOPS for 4KB random read with 0.80.4 , but with latest
master , I only got ~12K IOPS

This are impressive numbers. Can you tell me how many OSDs you have and which SSDs you use?
Thanks,
Stefan

Cheers,
xinxin
-----Original Message-----
Sent: Friday, September 19, 2014 2:03 AM
To: Alexandre DERUMIER; Haomai Wang
Subject: RE: severe librbd performance degradation in Giant
Alexandre,
What tool are you using ? I used fio rbd.
Also, I hope you have Giant package installed in the client side as well and rbd_cache =true is set on the client conf file.
FYI, firefly librbd + librados and Giant cluster will work seamlessly and I had to make sure fio rbd is really loading giant librbd (if you have multiple copies around , which was in my case) for reproducing it.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Thursday, September 18, 2014 2:49 AM
To: Haomai Wang
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

Post by Haomai Wang
According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq or rand) with or without.
firefly : around 12000iops (with or without rbd_cache) giant : around
12000iops (with or without rbd_cache)
(and I can reach around 20000-30000 iops on giant with disabling optracker).
rbd_cache only improve write performance for me (4k block )
----- Mail original -----
Envoy?: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

Post by Haomai Wang
I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
--
N r y b X ?v ^ )?{.n + z ]z {ay ?? ,j f h z w j:+v w j m zZ+ ?j" ! i
N r y b X ?v ^ )?{.n + z ]z {ay ?? ,j f h z w

j:+v w j m zZ+ ?j" !tml=
N???????????????r??????y?????????b???X???????v???^???)?{.n???+?????????z???]z?????????{ay????????,j??????f?????????h?????????z??????w??????????????????j:+v?????????w???j???m????????????????????????zZ+????????????????j"??????!tml=

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Alexandre DERUMIER]

What tool are you using ? I used fio rbd.=20

fio rbd too


[global]
ioengine=3Drbd
clientname=3Dadmin
pool=3Dtest
rbdname=3Dtest
invalidate=3D0=20
#rw=3Dread
#rw=3Drandwrite
#rw=3Dwrite
rw=3Drandread
bs=3D4k
direct=3D1
numjobs=3D2
group_reporting=3D1
size=3D10G

[rbd_iodepth32]
iodepth=3D32



I just notice something strange

with rbd_cache=3Dtrue , I got around 60000iops (and I don't see any ne=
twork traffic)

So maybe they are a bug in fio ?
maybe this is related to:


http://tracker.ceph.com/issues/9391
"fio rbd driver rewrites same blocks"

----- Mail original -----=20

De: "Somnath Roy" <***@sandisk.com>=20
=C3=80: "Alexandre DERUMIER" <***@odiso.com>, "Haomai Wang" <haom=
***@gmail.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org=20
Envoy=C3=A9: Jeudi 18 Septembre 2014 20:02:49=20
Objet: RE: severe librbd performance degradation in Giant=20

Alexandre,=20
What tool are you using ? I used fio rbd.=20

Also, I hope you have Giant package installed in the client side as wel=
l and rbd_cache =3Dtrue is set on the client conf file.=20
=46YI, firefly librbd + librados and Giant cluster will work seamlessly=
and I had to make sure fio rbd is really loading giant librbd (if you =
have multiple copies around , which was in my case) for reproducing it.=
=20

Thanks & Regards=20
Somnath=20

-----Original Message-----=20
=46rom: Alexandre DERUMIER [mailto:***@odiso.com]=20
Sent: Thursday, September 18, 2014 2:49 AM=20
To: Haomai Wang=20
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org; Somnath Roy=20
Subject: Re: severe librbd performance degradation in Giant=20

According http://tracker.ceph.com/issues/9513, do you mean that rbd=20
cache will make 10x performance degradation for random read?=20

Hi, on my side, I don't see any degradation performance on read (seq or=
rand) with or without.=20

firefly : around 12000iops (with or without rbd_cache) giant : around 1=
2000iops (with or without rbd_cache)=20

(and I can reach around 20000-30000 iops on giant with disabling optrac=
ker).=20


rbd_cache only improve write performance for me (4k block )=20



----- Mail original -----=20

De: "Haomai Wang" <***@gmail.com>=20
=C3=80: "Somnath Roy" <***@sandisk.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org=20
Envoy=C3=A9: Jeudi 18 Septembre 2014 04:27:56=20
Objet: Re: severe librbd performance degradation in Giant=20

According http://tracker.ceph.com/issues/9513, do you mean that rbd cac=
he will make 10x performance degradation for random read?=20

On Thu, Sep 18, 2014 at 7:44 AM, Somnath Roy <***@sandisk.com> =
wrote:=20

Josh/Sage,=20
I should mention that even after turning off rbd cache I am getting ~=

20% degradation over Firefly.=20

=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
From: Somnath Roy=20
Sent: Wednesday, September 17, 2014 2:44 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Created a tracker for this.=20
=20
http://tracker.ceph.com/issues/9513=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:39 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Sage,=20
It's a 4K random read.=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:36 PM=20
To: Somnath Roy=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
What was the io pattern? Sequential or random? For random a slowdown =

makes sense (tho maybe not 10x!) but not for sequentail....=20

=20
s=20
=20
On Wed, 17 Sep 2014, Somnath Roy wrote:=20
=20

I set the following in the client side /etc/ceph/ceph.conf where I a=

m running fio rbd.=20

=20
rbd_cache_writethrough_until_flush =3D false=20
=20
But, no difference. BTW, I am doing Random read, not write. Still th=

is setting applies ?=20

=20
Next, I tried to tweak the rbd_cache setting to false and I *got bac=

k* the old performance. Now, it is similar to firefly throughput !=20

=20
So, loks like rbd_cache=3Dtrue was the culprit.=20
=20
Thanks Josh !=20
=20
Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:20 PM=20
Subject: Re: severe librbd performance degradation in Giant=20
=20
On 09/17/2014 01:55 PM, Somnath Roy wrote:=20

Hi Sage,=20
We are experiencing severe librbd performance degradation in Giant=

over firefly release. Here is the experiment we did to isolate it as a=
librbd problem.=20

=20
1. Single OSD is running latest Giant and client is running fio rb=

d on top of firefly based librbd/librados. For one client it is giving =
~11-12K iops (4K RR).=20

2. Single OSD is running Giant and client is running fio rbd on to=

p of Giant based librbd/librados. For one client it is giving ~1.9K iop=
s (4K RR).=20

3. Single OSD is running latest Giant and client is running Giant =

based ceph_smaiobench on top of giant librados. For one client it is gi=
ving ~11-12K iops (4K RR).=20

4. Giant RGW on top of Giant OSD is also scaling.=20
=20
=20
So, it is obvious from the above that recent librbd has issues. I =

will raise a tracker to track this.=20

=20
For giant the default cache settings changed to:=20
=20
rbd cache =3D true=20
rbd cache writethrough until flush =3D true=20
=20
If fio isn't sending flushes as the test is running, the cache will =

stay in writethrough mode. Does the difference remain if you set rbd ca=
che writethrough until flush =3D false ?=20

=20
Josh=20
=20
________________________________=20
=20
PLEASE NOTE: The information contained in this electronic mail messa=

ge is intended only for the use of the designated recipient(s) named ab=
ove. If the reader of this message is not the intended recipient, you a=
re hereby notified that you have received this message in error and tha=
t any review, dissemination, distribution, or copying of this message i=
s strictly prohibited. If you have received this communication in error=
, please notify the sender by telephone or e-mail (as shown above) imme=
diately and destroy any and all copies of this message in your possessi=
on (whether hard copies or electronically stored copies).=20

=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

"=20
=20

info at http://vger.kernel.org/majordomo-info.html=20
=20
=20

--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel"=

=20

info at http://vger.kernel.org/majordomo-info.html=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel"=

=20

info at http://vger.kernel.org/majordomo-info.html=20

--=20
Best Regards,=20

Wheat=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n the body of a message to ***@vger.kernel.org More majordomo inf=
o at http://vger.kernel.org/majordomo-info.html=20
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Alexandre DERUMIER]

with rbd_cache=3Dtrue , I got around 60000iops (and I don't see any =

network traffic)=20

So maybe they are a bug in fio ?=20
maybe this is related to:=20

Oh, sorry, this was my fault, I didn't fill the rbd with datas before d=
oing the bench

Now the results are (for 1 osd)

firefly
------
bw=3D37460KB/s, iops=3D9364

giant
-----
bw=3D32741KB/s, iops=3D8185


So, a little regression

(the results are equals rbd_cache=3Dtrue|false)


I'll try to compare with more osds

----- Mail original -----=20

De: "Alexandre DERUMIER" <***@odiso.com>=20
=C3=80: "Somnath Roy" <***@sandisk.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org, "Haomai Wang" <***@gmail.com>=20
Envoy=C3=A9: Vendredi 19 Septembre 2014 12:09:41=20
Objet: Re: severe librbd performance degradation in Giant=20

What tool are you using ? I used fio rbd.=20

fio rbd too=20


[global]=20
ioengine=3Drbd=20
clientname=3Dadmin=20
pool=3Dtest=20
rbdname=3Dtest=20
invalidate=3D0=20
#rw=3Dread=20
#rw=3Drandwrite=20
#rw=3Dwrite=20
rw=3Drandread=20
bs=3D4k=20
direct=3D1=20
numjobs=3D2=20
group_reporting=3D1=20
size=3D10G=20

[rbd_iodepth32]=20
iodepth=3D32=20



I just notice something strange=20

with rbd_cache=3Dtrue , I got around 60000iops (and I don't see any net=
work traffic)=20

So maybe they are a bug in fio ?=20
maybe this is related to:=20


http://tracker.ceph.com/issues/9391=20
"fio rbd driver rewrites same blocks"=20

----- Mail original -----=20

De: "Somnath Roy" <***@sandisk.com>=20
=C3=80: "Alexandre DERUMIER" <***@odiso.com>, "Haomai Wang" <haom=
***@gmail.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org=20
Envoy=C3=A9: Jeudi 18 Septembre 2014 20:02:49=20
Objet: RE: severe librbd performance degradation in Giant=20

Alexandre,=20
What tool are you using ? I used fio rbd.=20

Also, I hope you have Giant package installed in the client side as wel=
l and rbd_cache =3Dtrue is set on the client conf file.=20
=46YI, firefly librbd + librados and Giant cluster will work seamlessly=
and I had to make sure fio rbd is really loading giant librbd (if you =
have multiple copies around , which was in my case) for reproducing it.=
=20

Thanks & Regards=20
Somnath=20

-----Original Message-----=20
=46rom: Alexandre DERUMIER [mailto:***@odiso.com]=20
Sent: Thursday, September 18, 2014 2:49 AM=20
To: Haomai Wang=20
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org; Somnath Roy=20
Subject: Re: severe librbd performance degradation in Giant=20

According http://tracker.ceph.com/issues/9513, do you mean that rbd=20
cache will make 10x performance degradation for random read?=20

Hi, on my side, I don't see any degradation performance on read (seq or=
rand) with or without.=20

firefly : around 12000iops (with or without rbd_cache) giant : around 1=
2000iops (with or without rbd_cache)=20

(and I can reach around 20000-30000 iops on giant with disabling optrac=
ker).=20


rbd_cache only improve write performance for me (4k block )=20



----- Mail original -----=20

De: "Haomai Wang" <***@gmail.com>=20
=C3=80: "Somnath Roy" <***@sandisk.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org=20
Envoy=C3=A9: Jeudi 18 Septembre 2014 04:27:56=20
Objet: Re: severe librbd performance degradation in Giant=20

According http://tracker.ceph.com/issues/9513, do you mean that rbd cac=
he will make 10x performance degradation for random read?=20

On Thu, Sep 18, 2014 at 7:44 AM, Somnath Roy <***@sandisk.com> =
wrote:=20

Josh/Sage,=20
I should mention that even after turning off rbd cache I am getting ~=

20% degradation over Firefly.=20

=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
From: Somnath Roy=20
Sent: Wednesday, September 17, 2014 2:44 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Created a tracker for this.=20
=20
http://tracker.ceph.com/issues/9513=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:39 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Sage,=20
It's a 4K random read.=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:36 PM=20
To: Somnath Roy=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
What was the io pattern? Sequential or random? For random a slowdown =

makes sense (tho maybe not 10x!) but not for sequentail....=20

=20
s=20
=20
On Wed, 17 Sep 2014, Somnath Roy wrote:=20
=20

I set the following in the client side /etc/ceph/ceph.conf where I a=

m running fio rbd.=20

=20
rbd_cache_writethrough_until_flush =3D false=20
=20
But, no difference. BTW, I am doing Random read, not write. Still th=

is setting applies ?=20

=20
Next, I tried to tweak the rbd_cache setting to false and I *got bac=

k* the old performance. Now, it is similar to firefly throughput !=20

=20
So, loks like rbd_cache=3Dtrue was the culprit.=20
=20
Thanks Josh !=20
=20
Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:20 PM=20
Subject: Re: severe librbd performance degradation in Giant=20
=20
On 09/17/2014 01:55 PM, Somnath Roy wrote:=20

Hi Sage,=20
We are experiencing severe librbd performance degradation in Giant=

over firefly release. Here is the experiment we did to isolate it as a=
librbd problem.=20

=20
1. Single OSD is running latest Giant and client is running fio rb=

d on top of firefly based librbd/librados. For one client it is giving =
~11-12K iops (4K RR).=20

2. Single OSD is running Giant and client is running fio rbd on to=

p of Giant based librbd/librados. For one client it is giving ~1.9K iop=
s (4K RR).=20

3. Single OSD is running latest Giant and client is running Giant =

based ceph_smaiobench on top of giant librados. For one client it is gi=
ving ~11-12K iops (4K RR).=20

4. Giant RGW on top of Giant OSD is also scaling.=20
=20
=20
So, it is obvious from the above that recent librbd has issues. I =

will raise a tracker to track this.=20

=20
For giant the default cache settings changed to:=20
=20
rbd cache =3D true=20
rbd cache writethrough until flush =3D true=20
=20
If fio isn't sending flushes as the test is running, the cache will =

stay in writethrough mode. Does the difference remain if you set rbd ca=
che writethrough until flush =3D false ?=20

=20
Josh=20
=20
________________________________=20
=20
PLEASE NOTE: The information contained in this electronic mail messa=

ge is intended only for the use of the designated recipient(s) named ab=
ove. If the reader of this message is not the intended recipient, you a=
re hereby notified that you have received this message in error and tha=
t any review, dissemination, distribution, or copying of this message i=
s strictly prohibited. If you have received this communication in error=
, please notify the sender by telephone or e-mail (as shown above) imme=
diately and destroy any and all copies of this message in your possessi=
on (whether hard copies or electronically stored copies).=20

=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

"=20
=20

info at http://vger.kernel.org/majordomo-info.html=20
=20
=20

--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel"=

=20

info at http://vger.kernel.org/majordomo-info.html=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel"=

=20

info at http://vger.kernel.org/majordomo-info.html=20

--=20
Best Regards,=20

Wheat=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n the body of a message to ***@vger.kernel.org More majordomo inf=
o at http://vger.kernel.org/majordomo-info.html=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n=20
the body of a message to ***@vger.kernel.org=20
More majordomo info at http://vger.kernel.org/majordomo-info.html=20
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Alexandre DERUMIER]

giant results with 6 osd
------------------------
bw=3D118129KB/s, iops=3D29532 : rbd_cache =3D false
bw=3D101771KB/s, iops=3D25442 : rbd_cache =3D true



fio config (note that numjobs is important, i'm going from 18000iops ->=
29000 iops for numjobs 1->4)
----------
[global]
#logging
#write_iops_log=3Dwrite_iops_log
#write_bw_log=3Dwrite_bw_log
#write_lat_log=3Dwrite_lat_log
ioengine=3Drbd
clientname=3Dadmin
pool=3Dtest
rbdname=3Dtest
invalidate=3D0 # mandatory
#rw=3Dread
#rw=3Drandwrite
#rw=3Dwrite
rw=3Drandread
bs=3D4K
direct=3D1
numjobs=3D4
group_reporting=3D1
size=3D10G

[rbd_iodepth32]
iodepth=3D32



ceph.conf
---------
debug lockdep =3D 0/0
debug context =3D 0/0
debug crush =3D 0/0
debug buffer =3D 0/0
debug timer =3D 0/0
debug journaler =3D 0/0
debug osd =3D 0/0
debug optracker =3D 0/0
debug objclass =3D 0/0
debug filestore =3D 0/0
debug journal =3D 0/0
debug ms =3D 0/0
debug monc =3D 0/0
debug tp =3D 0/0
debug auth =3D 0/0
debug finisher =3D 0/0
debug heartbeatmap =3D 0/0
debug perfcounter =3D 0/0
debug asok =3D 0/0
debug throttle =3D 0/0

osd_op_num_threads_per_shard =3D 2
osd_op_num_shards =3D 25
filestore_fd_cache_size =3D 64
filestore_fd_cache_shards =3D 32

ms_nocrc =3D true
cephx sign messages =3D false
cephx require signatures =3D false

ms_dispatch_throttle_bytes =3D 0
throttler_perf_counter =3D false

[osd]
osd_client_message_size_cap =3D 0
osd_client_message_cap =3D 0
osd_enable_op_tracker =3D false


----- Mail original -----=20

De: "Alexandre DERUMIER" <***@odiso.com>=20
=C3=80: "Somnath Roy" <***@sandisk.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org, "Haomai Wang" <***@gmail.com>=20
Envoy=C3=A9: Vendredi 19 Septembre 2014 13:30:24=20
Objet: Re: severe librbd performance degradation in Giant=20

with rbd_cache=3Dtrue , I got around 60000iops (and I don't see any =

network traffic)=20

=20
So maybe they are a bug in fio ?=20
maybe this is related to:=20

Oh, sorry, this was my fault, I didn't fill the rbd with datas before d=
oing the bench=20

Now the results are (for 1 osd)=20

firefly=20
------=20
bw=3D37460KB/s, iops=3D9364=20

giant=20
-----=20
bw=3D32741KB/s, iops=3D8185=20


So, a little regression=20

(the results are equals rbd_cache=3Dtrue|false)=20


I'll try to compare with more osds=20

----- Mail original -----=20

De: "Alexandre DERUMIER" <***@odiso.com>=20
=C3=80: "Somnath Roy" <***@sandisk.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org, "Haomai Wang" <***@gmail.com>=20
Envoy=C3=A9: Vendredi 19 Septembre 2014 12:09:41=20
Objet: Re: severe librbd performance degradation in Giant=20

What tool are you using ? I used fio rbd.=20

fio rbd too=20


[global]=20
ioengine=3Drbd=20
clientname=3Dadmin=20
pool=3Dtest=20
rbdname=3Dtest=20
invalidate=3D0=20
#rw=3Dread=20
#rw=3Drandwrite=20
#rw=3Dwrite=20
rw=3Drandread=20
bs=3D4k=20
direct=3D1=20
numjobs=3D2=20
group_reporting=3D1=20
size=3D10G=20

[rbd_iodepth32]=20
iodepth=3D32=20



I just notice something strange=20

with rbd_cache=3Dtrue , I got around 60000iops (and I don't see any net=
work traffic)=20

So maybe they are a bug in fio ?=20
maybe this is related to:=20


http://tracker.ceph.com/issues/9391=20
"fio rbd driver rewrites same blocks"=20

----- Mail original -----=20

De: "Somnath Roy" <***@sandisk.com>=20
=C3=80: "Alexandre DERUMIER" <***@odiso.com>, "Haomai Wang" <haom=
***@gmail.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org=20
Envoy=C3=A9: Jeudi 18 Septembre 2014 20:02:49=20
Objet: RE: severe librbd performance degradation in Giant=20

Alexandre,=20
What tool are you using ? I used fio rbd.=20

Also, I hope you have Giant package installed in the client side as wel=
l and rbd_cache =3Dtrue is set on the client conf file.=20
=46YI, firefly librbd + librados and Giant cluster will work seamlessly=
and I had to make sure fio rbd is really loading giant librbd (if you =
have multiple copies around , which was in my case) for reproducing it.=
=20

Thanks & Regards=20
Somnath=20

-----Original Message-----=20
=46rom: Alexandre DERUMIER [mailto:***@odiso.com]=20
Sent: Thursday, September 18, 2014 2:49 AM=20
To: Haomai Wang=20
Cc: Sage Weil; Josh Durgin; ceph-***@vger.kernel.org; Somnath Roy=20
Subject: Re: severe librbd performance degradation in Giant=20

According http://tracker.ceph.com/issues/9513, do you mean that rbd=20
cache will make 10x performance degradation for random read?=20

Hi, on my side, I don't see any degradation performance on read (seq or=
rand) with or without.=20

firefly : around 12000iops (with or without rbd_cache) giant : around 1=
2000iops (with or without rbd_cache)=20

(and I can reach around 20000-30000 iops on giant with disabling optrac=
ker).=20


rbd_cache only improve write performance for me (4k block )=20



----- Mail original -----=20

De: "Haomai Wang" <***@gmail.com>=20
=C3=80: "Somnath Roy" <***@sandisk.com>=20
Cc: "Sage Weil" <***@redhat.com>, "Josh Durgin" <***@inktank.=
com>, ceph-***@vger.kernel.org=20
Envoy=C3=A9: Jeudi 18 Septembre 2014 04:27:56=20
Objet: Re: severe librbd performance degradation in Giant=20

According http://tracker.ceph.com/issues/9513, do you mean that rbd cac=
he will make 10x performance degradation for random read?=20

On Thu, Sep 18, 2014 at 7:44 AM, Somnath Roy <***@sandisk.com> =
wrote:=20

Josh/Sage,=20
I should mention that even after turning off rbd cache I am getting ~=

20% degradation over Firefly.=20

=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
From: Somnath Roy=20
Sent: Wednesday, September 17, 2014 2:44 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Created a tracker for this.=20
=20
http://tracker.ceph.com/issues/9513=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:39 PM=20
To: Sage Weil=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
Sage,=20
It's a 4K random read.=20
=20
Thanks & Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:36 PM=20
To: Somnath Roy=20
Subject: RE: severe librbd performance degradation in Giant=20
=20
What was the io pattern? Sequential or random? For random a slowdown =

makes sense (tho maybe not 10x!) but not for sequentail....=20

=20
s=20
=20
On Wed, 17 Sep 2014, Somnath Roy wrote:=20
=20

I set the following in the client side /etc/ceph/ceph.conf where I a=

m running fio rbd.=20

=20
rbd_cache_writethrough_until_flush =3D false=20
=20
But, no difference. BTW, I am doing Random read, not write. Still th=

is setting applies ?=20

=20
Next, I tried to tweak the rbd_cache setting to false and I *got bac=

k* the old performance. Now, it is similar to firefly throughput !=20

=20
So, loks like rbd_cache=3Dtrue was the culprit.=20
=20
Thanks Josh !=20
=20
Regards=20
Somnath=20
=20
-----Original Message-----=20
Sent: Wednesday, September 17, 2014 2:20 PM=20
Subject: Re: severe librbd performance degradation in Giant=20
=20
On 09/17/2014 01:55 PM, Somnath Roy wrote:=20

Hi Sage,=20
We are experiencing severe librbd performance degradation in Giant=

over firefly release. Here is the experiment we did to isolate it as a=
librbd problem.=20

=20
1. Single OSD is running latest Giant and client is running fio rb=

d on top of firefly based librbd/librados. For one client it is giving =
~11-12K iops (4K RR).=20

2. Single OSD is running Giant and client is running fio rbd on to=

p of Giant based librbd/librados. For one client it is giving ~1.9K iop=
s (4K RR).=20

3. Single OSD is running latest Giant and client is running Giant =

based ceph_smaiobench on top of giant librados. For one client it is gi=
ving ~11-12K iops (4K RR).=20

4. Giant RGW on top of Giant OSD is also scaling.=20
=20
=20
So, it is obvious from the above that recent librbd has issues. I =

will raise a tracker to track this.=20

=20
For giant the default cache settings changed to:=20
=20
rbd cache =3D true=20
rbd cache writethrough until flush =3D true=20
=20
If fio isn't sending flushes as the test is running, the cache will =

stay in writethrough mode. Does the difference remain if you set rbd ca=
che writethrough until flush =3D false ?=20

=20
Josh=20
=20
________________________________=20
=20
PLEASE NOTE: The information contained in this electronic mail messa=

ge is intended only for the use of the designated recipient(s) named ab=
ove. If the reader of this message is not the intended recipient, you a=
re hereby notified that you have received this message in error and tha=
t any review, dissemination, distribution, or copying of this message i=
s strictly prohibited. If you have received this communication in error=
, please notify the sender by telephone or e-mail (as shown above) imme=
diately and destroy any and all copies of this message in your possessi=
on (whether hard copies or electronically stored copies).=20

=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel=

"=20
=20

info at http://vger.kernel.org/majordomo-info.html=20
=20
=20

--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel"=

=20

info at http://vger.kernel.org/majordomo-info.html=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel"=

=20

info at http://vger.kernel.org/majordomo-info.html=20

--=20
Best Regards,=20

Wheat=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n the body of a message to ***@vger.kernel.org More majordomo inf=
o at http://vger.kernel.org/majordomo-info.html=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n=20
the body of a message to ***@vger.kernel.org=20
More majordomo info at http://vger.kernel.org/majordomo-info.html=20
--=20
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n=20
the body of a message to ***@vger.kernel.org=20
More majordomo info at http://vger.kernel.org/majordomo-info.html=20
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" i=
n
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Sage Weil]

Post by Somnath Roy

with rbd_cache=true , I got around 60000iops (and I don't see any network traffic)
So maybe they are a bug in fio ?

Oh, sorry, this was my fault, I didn't fill the rbd with datas before doing the bench
Now the results are (for 1 osd)
firefly
------
bw=37460KB/s, iops=9364
giant
-----
bw=32741KB/s, iops=8185
So, a little regression
(the results are equals rbd_cache=true|false)

Do you see a difference with rados bench, or is it just librbd?

Thanks!
sage

I'll try to compare with more osds
----- Mail original -----
Envoy?: Vendredi 19 Septembre 2014 12:09:41
Objet: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

What tool are you using ? I used fio rbd.

fio rbd too
[global]
ioengine=rbd
clientname=admin
pool=test
rbdname=test
invalidate=0
#rw=read
#rw=randwrite
#rw=write
rw=randread
bs=4k
direct=1
numjobs=2
group_reporting=1
size=10G
[rbd_iodepth32]
iodepth=32
I just notice something strange
with rbd_cache=true , I got around 60000iops (and I don't see any network traffic)
So maybe they are a bug in fio ?
http://tracker.ceph.com/issues/9391
"fio rbd driver rewrites same blocks"
----- Mail original -----
Envoy?: Jeudi 18 Septembre 2014 20:02:49
Objet: RE: severe librbd performance degradation in Giant
Alexandre,
What tool are you using ? I used fio rbd.
Also, I hope you have Giant package installed in the client side as well and rbd_cache =true is set on the client conf file.
FYI, firefly librbd + librados and Giant cluster will work seamlessly and I had to make sure fio rbd is really loading giant librbd (if you have multiple copies around , which was in my case) for reproducing it.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Thursday, September 18, 2014 2:49 AM
To: Haomai Wang
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy

According http://tracker.ceph.com/issues/9513, do you mean that rbd
cache will make 10x performance degradation for random read?

Hi, on my side, I don't see any degradation performance on read (seq or rand) with or without.
firefly : around 12000iops (with or without rbd_cache) giant : around 12000iops (with or without rbd_cache)
(and I can reach around 20000-30000 iops on giant with disabling optracker).
rbd_cache only improve write performance for me (4k block )
----- Mail original -----
Envoy?: Jeudi 18 Septembre 2014 04:27:56
Objet: Re: severe librbd performance degradation in Giant
According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read?

Post by Somnath Roy
Josh/Sage,
I should mention that even after turning off rbd cache I am getting ~20% degradation over Firefly.
Thanks & Regards
Somnath
-----Original Message-----
From: Somnath Roy
Sent: Wednesday, September 17, 2014 2:44 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Created a tracker for this.
http://tracker.ceph.com/issues/9513
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:39 PM
To: Sage Weil
Subject: RE: severe librbd performance degradation in Giant
Sage,
It's a 4K random read.
Thanks & Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:36 PM
To: Somnath Roy
Subject: RE: severe librbd performance degradation in Giant
What was the io pattern? Sequential or random? For random a slowdown makes sense (tho maybe not 10x!) but not for sequentail....
s

I set the following in the client side /etc/ceph/ceph.conf where I am running fio rbd.
rbd_cache_writethrough_until_flush = false
But, no difference. BTW, I am doing Random read, not write. Still this setting applies ?
Next, I tried to tweak the rbd_cache setting to false and I *got back* the old performance. Now, it is similar to firefly throughput !
So, loks like rbd_cache=true was the culprit.
Thanks Josh !
Regards
Somnath
-----Original Message-----
Sent: Wednesday, September 17, 2014 2:20 PM
Subject: Re: severe librbd performance degradation in Giant

Post by Somnath Roy
Hi Sage,
We are experiencing severe librbd performance degradation in Giant over firefly release. Here is the experiment we did to isolate it as a librbd problem.
1. Single OSD is running latest Giant and client is running fio rbd on top of firefly based librbd/librados. For one client it is giving ~11-12K iops (4K RR).
2. Single OSD is running Giant and client is running fio rbd on top of Giant based librbd/librados. For one client it is giving ~1.9K iops (4K RR).
3. Single OSD is running latest Giant and client is running Giant based ceph_smaiobench on top of giant librados. For one client it is giving ~11-12K iops (4K RR).
4. Giant RGW on top of Giant OSD is also scaling.
So, it is obvious from the above that recent librbd has issues. I will raise a tracker to track this.

rbd cache = true
rbd cache writethrough until flush = true
If fio isn't sending flushes as the test is running, the cache will stay in writethrough mode. Does the difference remain if you set rbd cache writethrough until flush = false ?
Josh
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
info at http://vger.kernel.org/majordomo-info.html

--
Best Regards,
Wheat
--
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html