Discussion:
librados AIO problem diagnostic
Loic Dachary
2014-10-01 07:01:40 UTC
Permalink
Hi Sebastien,

In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.

Thanks in advance :-)
--
Loïc Dachary, Artisan Logiciel Libre
Sebastien Ponce
2014-10-01 07:28:57 UTC
Permalink
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.

Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Loic Dachary
2014-10-01 08:31:35 UTC
Permalink
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with

./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make

and let me know if you see a problem when running

cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio

I'm trying with your "./configure" on my end to check if this changes something.

It's going to be a fun one ;-)

Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Loïc Dachary, Artisan Logiciel Libre
Sebastien Ponce
2014-10-01 11:37:09 UTC
Permalink
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.

Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64

Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Loic Dachary
2014-10-01 13:08:27 UTC
Permalink
Post by Sebastien Ponce
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.
Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64
I'm using

ii libc-bin 2.19-0ubuntu3

and I do not have glibc installed. I'm quite sure you can reproduce the problem on a fresh Ubuntu 14.04 with the following

./autogen.sh ; ./configure --disable-static --with-debug CFLAGS="-Wall -g" CXXFLAGS="-Wall -g"

it fails consistently in my container. When I try with ./configure it does not fail. Something is going on when there is -g instead of the default flags. I checked that --disable-static and --with-debug make no difference.

Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Loïc Dachary, Artisan Logiciel Libre
Sebastien Ponce
2014-10-01 13:09:48 UTC
Permalink
Ok, we are getting closer then.
I'll try to reinstall the ubuntu machine and run with -g.

Sebastien
Post by Loic Dachary
Post by Sebastien Ponce
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.
Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64
I'm using
ii libc-bin 2.19-0ubuntu3
and I do not have glibc installed. I'm quite sure you can reproduce the problem on a fresh Ubuntu 14.04 with the following
./autogen.sh ; ./configure --disable-static --with-debug CFLAGS="-Wall -g" CXXFLAGS="-Wall -g"
it fails consistently in my container. When I try with ./configure it does not fail. Something is going on when there is -g instead of the default flags. I checked that --disable-static and --with-debug make no difference.
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sebastien Ponce
2014-10-02 09:49:35 UTC
Permalink
Hurray, I manage to seg fault ! (sounds odd this sentence...)
So indeed, fresh ubuntu, debug mode does it. What I've also changed in
the meantime is the type of machine. It's still virtual, but more
powerful (4 cores).

Let's debug it now...

Sebastien
Post by Sebastien Ponce
Ok, we are getting closer then.
I'll try to reinstall the ubuntu machine and run with -g.
Sebastien
Post by Loic Dachary
Post by Sebastien Ponce
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.
Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64
I'm using
ii libc-bin 2.19-0ubuntu3
and I do not have glibc installed. I'm quite sure you can reproduce the problem on a fresh Ubuntu 14.04 with the following
./autogen.sh ; ./configure --disable-static --with-debug CFLAGS="-Wall -g" CXXFLAGS="-Wall -g"
it fails consistently in my container. When I try with ./configure it does not fail. Something is going on when there is -g instead of the default flags. I checked that --disable-static and --with-debug make no difference.
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sebastien Ponce
2014-10-02 14:31:24 UTC
Permalink
As mentioned in the bug, I was a bit fast in concluding I could
reproduce the problem. I can reproduce the original one, but not the
remaining issue once the fix was applied.

Loic, can you confirm that you are using the wip-9356 branch of repo
***@github.com:sponce/ceph.git ?

Sebastien
Post by Sebastien Ponce
Hurray, I manage to seg fault ! (sounds odd this sentence...)
So indeed, fresh ubuntu, debug mode does it. What I've also changed in
the meantime is the type of machine. It's still virtual, but more
powerful (4 cores).
Let's debug it now...
Sebastien
Post by Sebastien Ponce
Ok, we are getting closer then.
I'll try to reinstall the ubuntu machine and run with -g.
Sebastien
Post by Loic Dachary
Post by Sebastien Ponce
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.
Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64
I'm using
ii libc-bin 2.19-0ubuntu3
and I do not have glibc installed. I'm quite sure you can reproduce the problem on a fresh Ubuntu 14.04 with the following
./autogen.sh ; ./configure --disable-static --with-debug CFLAGS="-Wall -g" CXXFLAGS="-Wall -g"
it fails consistently in my container. When I try with ./configure it does not fail. Something is going on when there is -g instead of the default flags. I checked that --disable-static and --with-debug make no difference.
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Loic Dachary
2014-10-02 14:58:53 UTC
Permalink
Post by Sebastien Ponce
As mentioned in the bug, I was a bit fast in concluding I could
reproduce the problem. I can reproduce the original one, but not the
remaining issue once the fix was applied.
Loic, can you confirm that you are using the wip-9356 branch of repo
Yes. I'm able to reproduce the problem with 0daddfbf1164d6ba3f38eee29d2f11acfa62f2b6 from your tree https://github.com/sponce/ceph.git which is the head of wip-9356

Cheers
Post by Sebastien Ponce
Sebastien
Post by Sebastien Ponce
Hurray, I manage to seg fault ! (sounds odd this sentence...)
So indeed, fresh ubuntu, debug mode does it. What I've also changed in
the meantime is the type of machine. It's still virtual, but more
powerful (4 cores).
Let's debug it now...
Sebastien
Post by Sebastien Ponce
Ok, we are getting closer then.
I'll try to reinstall the ubuntu machine and run with -g.
Sebastien
Post by Loic Dachary
Post by Sebastien Ponce
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.
Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64
I'm using
ii libc-bin 2.19-0ubuntu3
and I do not have glibc installed. I'm quite sure you can reproduce the problem on a fresh Ubuntu 14.04 with the following
./autogen.sh ; ./configure --disable-static --with-debug CFLAGS="-Wall -g" CXXFLAGS="-Wall -g"
it fails consistently in my container. When I try with ./configure it does not fail. Something is going on when there is -g instead of the default flags. I checked that --disable-static and --with-debug make no difference.
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Loïc Dachary, Artisan Logiciel Libre
Loic Dachary
2014-10-08 07:14:38 UTC
Permalink
Hi Sebastien,

Have you been able to reproduce it or did it vanish again ? This is a tricky one ;-)

Cheers
Post by Loic Dachary
Post by Sebastien Ponce
As mentioned in the bug, I was a bit fast in concluding I could
reproduce the problem. I can reproduce the original one, but not the
remaining issue once the fix was applied.
Loic, can you confirm that you are using the wip-9356 branch of repo
Yes. I'm able to reproduce the problem with 0daddfbf1164d6ba3f38eee29d2f11acfa62f2b6 from your tree https://github.com/sponce/ceph.git which is the head of wip-9356
Cheers
Post by Sebastien Ponce
Sebastien
Post by Sebastien Ponce
Hurray, I manage to seg fault ! (sounds odd this sentence...)
So indeed, fresh ubuntu, debug mode does it. What I've also changed in
the meantime is the type of machine. It's still virtual, but more
powerful (4 cores).
Let's debug it now...
Sebastien
Post by Sebastien Ponce
Ok, we are getting closer then.
I'll try to reinstall the ubuntu machine and run with -g.
Sebastien
Post by Loic Dachary
Post by Sebastien Ponce
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.
Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64
I'm using
ii libc-bin 2.19-0ubuntu3
and I do not have glibc installed. I'm quite sure you can reproduce the problem on a fresh Ubuntu 14.04 with the following
./autogen.sh ; ./configure --disable-static --with-debug CFLAGS="-Wall -g" CXXFLAGS="-Wall -g"
it fails consistently in my container. When I try with ./configure it does not fail. Something is going on when there is -g instead of the default flags. I checked that --disable-static and --with-debug make no difference.
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Loïc Dachary, Artisan Logiciel Libre
Sebastien Ponce
2014-10-08 08:00:33 UTC
Permalink
It vanished again....
I've spent quite some time analysing the helgrind output to try to find
it, but so far I got nothing but problems outside this part (BTW, it
would be interesting to run tests systematically in helgrind, even if
there is a lot of false positive).
Now I've just seen the backtrace you've sent and this already helps a
lot; I did not even have a backtrace so far.
In conclusion, still ongoing....

Sebastien
Post by Loic Dachary
Hi Sebastien,
Have you been able to reproduce it or did it vanish again ? This is a tricky one ;-)
Cheers
Post by Loic Dachary
Post by Sebastien Ponce
As mentioned in the bug, I was a bit fast in concluding I could
reproduce the problem. I can reproduce the original one, but not the
remaining issue once the fix was applied.
Loic, can you confirm that you are using the wip-9356 branch of repo
Yes. I'm able to reproduce the problem with 0daddfbf1164d6ba3f38eee29d2f11acfa62f2b6 from your tree https://github.com/sponce/ceph.git which is the head of wip-9356
Cheers
Post by Sebastien Ponce
Sebastien
Post by Sebastien Ponce
Hurray, I manage to seg fault ! (sounds odd this sentence...)
So indeed, fresh ubuntu, debug mode does it. What I've also changed in
the meantime is the type of machine. It's still virtual, but more
powerful (4 cores).
Let's debug it now...
Sebastien
Post by Sebastien Ponce
Ok, we are getting closer then.
I'll try to reinstall the ubuntu machine and run with -g.
Sebastien
Post by Loic Dachary
Post by Sebastien Ponce
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.
Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64
I'm using
ii libc-bin 2.19-0ubuntu3
and I do not have glibc installed. I'm quite sure you can reproduce the problem on a fresh Ubuntu 14.04 with the following
./autogen.sh ; ./configure --disable-static --with-debug CFLAGS="-Wall -g" CXXFLAGS="-Wall -g"
it fails consistently in my container. When I try with ./configure it does not fail. Something is going on when there is -g instead of the default flags. I checked that --disable-static and --with-debug make no difference.
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sebastien Ponce
2014-10-09 14:06:55 UTC
Permalink
Finally this should be fixed.
Loic, could you retry with the new state of my branch and confirm ?

Sebastien
Post by Sebastien Ponce
It vanished again....
I've spent quite some time analysing the helgrind output to try to find
it, but so far I got nothing but problems outside this part (BTW, it
would be interesting to run tests systematically in helgrind, even if
there is a lot of false positive).
Now I've just seen the backtrace you've sent and this already helps a
lot; I did not even have a backtrace so far.
In conclusion, still ongoing....
Sebastien
Post by Loic Dachary
Hi Sebastien,
Have you been able to reproduce it or did it vanish again ? This is a tricky one ;-)
Cheers
Post by Loic Dachary
Post by Sebastien Ponce
As mentioned in the bug, I was a bit fast in concluding I could
reproduce the problem. I can reproduce the original one, but not the
remaining issue once the fix was applied.
Loic, can you confirm that you are using the wip-9356 branch of repo
Yes. I'm able to reproduce the problem with 0daddfbf1164d6ba3f38eee29d2f11acfa62f2b6 from your tree https://github.com/sponce/ceph.git which is the head of wip-9356
Cheers
Post by Sebastien Ponce
Sebastien
Post by Sebastien Ponce
Hurray, I manage to seg fault ! (sounds odd this sentence...)
So indeed, fresh ubuntu, debug mode does it. What I've also changed in
the meantime is the type of machine. It's still virtual, but more
powerful (4 cores).
Let's debug it now...
Sebastien
Post by Sebastien Ponce
Ok, we are getting closer then.
I'll try to reinstall the ubuntu machine and run with -g.
Sebastien
Post by Loic Dachary
Post by Sebastien Ponce
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.
Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64
I'm using
ii libc-bin 2.19-0ubuntu3
and I do not have glibc installed. I'm quite sure you can reproduce the problem on a fresh Ubuntu 14.04 with the following
./autogen.sh ; ./configure --disable-static --with-debug CFLAGS="-Wall -g" CXXFLAGS="-Wall -g"
it fails consistently in my container. When I try with ./configure it does not fail. Something is going on when there is -g instead of the default flags. I checked that --disable-static and --with-debug make no difference.
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Loic Dachary
2014-10-09 14:24:04 UTC
Permalink
Post by Sebastien Ponce
Finally this should be fixed.
Loic, could you retry with the new state of my branch and confirm ?
Hi Sebastien,

I commented on the pull request and scheduled a run in the gitbuilder.

Cheers
Post by Sebastien Ponce
Sebastien
Post by Sebastien Ponce
It vanished again....
I've spent quite some time analysing the helgrind output to try to find
it, but so far I got nothing but problems outside this part (BTW, it
would be interesting to run tests systematically in helgrind, even if
there is a lot of false positive).
Now I've just seen the backtrace you've sent and this already helps a
lot; I did not even have a backtrace so far.
In conclusion, still ongoing....
Sebastien
Post by Loic Dachary
Hi Sebastien,
Have you been able to reproduce it or did it vanish again ? This is a tricky one ;-)
Cheers
Post by Loic Dachary
Post by Sebastien Ponce
As mentioned in the bug, I was a bit fast in concluding I could
reproduce the problem. I can reproduce the original one, but not the
remaining issue once the fix was applied.
Loic, can you confirm that you are using the wip-9356 branch of repo
Yes. I'm able to reproduce the problem with 0daddfbf1164d6ba3f38eee29d2f11acfa62f2b6 from your tree https://github.com/sponce/ceph.git which is the head of wip-9356
Cheers
Post by Sebastien Ponce
Sebastien
Post by Sebastien Ponce
Hurray, I manage to seg fault ! (sounds odd this sentence...)
So indeed, fresh ubuntu, debug mode does it. What I've also changed in
the meantime is the type of machine. It's still virtual, but more
powerful (4 cores).
Let's debug it now...
Sebastien
Post by Sebastien Ponce
Ok, we are getting closer then.
I'll try to reinstall the ubuntu machine and run with -g.
Sebastien
Post by Loic Dachary
Post by Sebastien Ponce
I've tried it on the RHEL6 like machine, but no luck. Or should I say I
was lucky ? Anyway no crash.
Discussing with Dan Van Der Ster, he told me that he had once the case
where the crash was there or not depending on the libc version. Just in
case, what are you using ? On the machine I've used this morning, it's
glibc-2.12-1.107.el6.x86_64
I'm using
ii libc-bin 2.19-0ubuntu3
and I do not have glibc installed. I'm quite sure you can reproduce the problem on a fresh Ubuntu 14.04 with the following
./autogen.sh ; ./configure --disable-static --with-debug CFLAGS="-Wall -g" CXXFLAGS="-Wall -g"
it fails consistently in my container. When I try with ./configure it does not fail. Something is going on when there is -g instead of the default flags. I checked that --disable-static and --with-debug make no difference.
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi,
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Same here : I'm in Paris ;-)
Post by Sebastien Ponce
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Could you try with
./autogen.sh ; ./configure --disable-static --with-debug CC='ccache gcc' CXX='ccache g++' CFLAGS="-Wall -g" CXXFLAGS="-Wall -g" ; make
and let me know if you see a problem when running
cd src
rm -fr dev out ; mkdir -p dev ; CEPH_NUM_MON=1 CEPH_NUM_OSD=3 ./vstart.sh -d -n -X -l mon osd
./ceph_test_rados_striper_api_aio
I'm trying with your "./configure" on my end to check if this changes something.
It's going to be a fun one ;-)
Cheers
Post by Sebastien Ponce
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Loïc Dachary, Artisan Logiciel Libre
Loic Dachary
2014-10-01 08:33:33 UTC
Permalink
Which kernel are you running accord to uname -a ? I have:

Linux 0a2ad46b4fb5 3.13.0-23-generic #45-Ubuntu SMP Fri Apr 4 06:58:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Loïc Dachary, Artisan Logiciel Libre
Sebastien Ponce
2014-10-01 08:50:00 UTC
Permalink
I've tried on 3 machines up to now, with different kernels :
- a RHEL6 (or close to it) with kernel 2.6.32-358.2.1.el6.x86_64
- a debian unstable box with kernel 3.12-1-amd64 #1 SMP Debian 3.12.6-2
- an ubuntu one, as you suggested, don't remember the kernel (and box is gone)

Sebastien
Post by Loic Dachary
Linux 0a2ad46b4fb5 3.13.0-23-generic #45-Ubuntu SMP Fri Apr 4 06:58:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Post by Sebastien Ponce
Thanks for this effort !
I'll look at all that this afternoon (it's mid morning here).
Concerning configure and autogen, I do the standard (naive ?)
"./autogen.sh; ./configure; make" given by the readme.
Sebastien
Post by Loic Dachary
Hi Sebastien,
In the context of http://tracker.ceph.com/issues/9356 I'm trying to collect as much information as I can in an environment that can be reconstructed to avoid interferences. Could you please let me know which ./configure line you are using ? Or which do_autogen.sh command you are using.
Thanks in advance :-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Loading...