[lttng-dev] Babeltrace2 - compilation error with intel18

[lttng-dev] Babeltrace2 - compilation error with intel18

[Rocky Dunlap]

I am trying to compile BT2 with Python bindings using Intel18.  I receive
the following error during the build:

In file included from py-common.c(31):
/apps/intel/intelpython3/include/python3.6m/Python.h(149): error #193: zero
used for undefined preprocessing identifier "_MSC_VER"
  #if _MSC_VER
      ^

In file included from py-common.c(31):
/apps/intel/intelpython3/include/python3.6m/Python.h(151): error #193: zero
used for undefined preprocessing identifier "__clang__"
  #elif __clang__ || __GNUC__
        ^

py-common.c(187): error #3179: deprecated conversion of string literal to
char* (should be const char*)
  format_exc_func_name = py_exc_tb ? "format_exception" :
                                                       ^

compilation aborted for py-common.c (code 2)
make[2]: *** [py-common.lo] Error 1
make[2]: Leaving directory
`/home/Rocky.Dunlap/babeltrace/babeltrace2-2.0.2/src/py-common'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory
`/home/Rocky.Dunlap/babeltrace/babeltrace2-2.0.2/src'


Rocky
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200320/a7177a4e/attachment.htm>

[lttng-dev] Babeltrace2 - compilation error with intel18

[Simon Marchi]

On 2020-03-20 5:10 p.m., Rocky Dunlap via lttng-dev wrote:
> I am trying to compile BT2 with Python bindings using Intel18.? I receive the following error during the build:
Hi Rocky,

I don't think we claim to support the Intel compiler, so you might be a bit
on your own here.  Although if you want to send patches to fix the build using
that compiler, I don't have anything against that.

> 
> In file included from py-common.c(31):
> /apps/intel/intelpython3/include/python3.6m/Python.h(149): error #193: zero used for undefined preprocessing identifier "_MSC_VER"
> ? #if _MSC_VER
> ? ? ? ^
> 
> In file included from py-common.c(31):
> /apps/intel/intelpython3/include/python3.6m/Python.h(151): error #193: zero used for undefined preprocessing identifier "__clang__"
> ? #elif __clang__ || __GNUC__
> ? ? ? ? ^

It took me a bit of time to understand that this is Intel's Python 3
distribution, not CPython.

I think it's technically valid to use an undefined macro in a preprocessor
condition like that, in which case it gets replaced with 0 (as the error
message mentions).  The compiler is trying to be helpful and warns you, because
relying on that behavior is a bit fragile, and often a sign of a mistake
somewhere.  But since this happens in a library you are using, I think your best
bet is just to disable this warning.

> py-common.c(187): error #3179: deprecated conversion of string literal to char* (should be const char*)
> ? format_exc_func_name = py_exc_tb ? "format_exception" :

I really don't understand this one, as format_exc_func_name is a const char *
in our code.

Simon

[lttng-dev] Babeltrace2 - compilation error with intel18

[Rocky Dunlap]

Simon,

I was able to get past the issue by passing "--enable-compile-warnings=yes"
to configure.

It get a lot further, then fails here:

gcc -pthread -shared -L/apps/intel/intelpython3/lib
-Wl,-rpath=/apps/intel/intelpython3/lib,--no-as-needed
-Wl,-z,noexecstack,-z,relro,-z,now -L../../../../src/lib/.libs -pthread
-lgmodule-2.0 -lglib-2.0 -pthread -lgmodule-2.0 -lglib-2.0 -pthread
-I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -pthread -pthread
-I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fno-strict-aliasing
-Wnested-externs -Wmissing-prototypes -Wstrict-prototypes
-Wdeclaration-after-statement -Wimplicit-function-declaration
-Wold-style-definition -Wjump-misses-init -Wall -Wextra -Wundef
-Wwrite-strings -Wpointer-arith -Wmissing-declarations -Wredundant-decls
-Wno-unused-parameter -Wno-missing-field-initializers -Wformat=2
-Wcast-align -Wformat-nonliteral -Wformat-security -Wsign-compare
-Wstrict-aliasing -Wshadow -Winline -Wpacked -Wmissing-format-attribute
-Wmissing-noreturn -Winit-self -Wmissing-include-dirs
-Wunused-but-set-variable -Warray-bounds -Wreturn-type -Wswitch-enum
-Wswitch-default -Wduplicated-cond -Wduplicated-branches -Wlogical-op
-Wrestrict -Wnull-dereference -Wdouble-promotion -Wno-sign-compare
-Wno-inline -Wno-declaration-after-statement -Wno-switch-enum
-Wno-switch-default -Wno-packed -Wno-pointer-arith -Wno-format-nonliteral
-Wno-double-promotion -Wno-cast-align -Wno-cast-function-type
-Wno-error=unused-parameter -Wno-error=missing-field-initializers
-Wno-error=sign-compare -Wno-error=inline
-Wno-error=declaration-after-statement -Wno-error=switch-enum
-Wno-error=switch-default -Wno-error=packed -Wno-error=pointer-arith
-Wno-error=format-nonliteral -Wno-error=double-promotion
-Wno-error=cast-align -Wno-error=cast-function-type -Wold-style-definition
-Werror=implicit-function-declaration -g -O2 -Wno-shadow
-Wno-null-dereference -I../../../../include -I../../../../src
-I../../../../src -include common/config.h -I./bt2
build/temp.linux-x86_64-3.6/bt2/native_bt.o
build/temp.linux-x86_64-3.6/./bt2/logging.o
../../../../src/autodisc/.libs/libbabeltrace2-autodisc.a
../../../../src/logging/.libs/libbabeltrace2-logging.a
../../../../src/common/.libs/libbabeltrace2-common.a
../../../../src/py-common/.libs/libbabeltrace2-py-common.a
../../../../src/string-format/.libs/libbabeltrace2-string-format.a
-L/apps/intel/intelpython3/lib -lbabeltrace2 -lglib-2.0 -lpython3.6m -o
build/build_lib/bt2/_native_bt.cpython-36m-x86_64-linux-gnu.so
gcc: error: unrecognized command line option ?-Wduplicated-cond?
gcc: error: unrecognized command line option ?-Wduplicated-branches?
gcc: error: unrecognized command line option ?-Wrestrict?
gcc: error: unrecognized command line option ?-Wnull-dereference?
error: command 'gcc' failed with exit status 1
make[4]: *** [build-python-bindings.stamp] Error 1

I am surprised that this is using "gcc" here, shouldn't it be using "icc"
since I have CC=icc?

Or maybe it is supposed to be using gcc?  I think if I update the GCC
compiler version then it will get past this issue as well, but I just
wanted confirmation that this indeed should be using a combination of gcc
and icc?

If it should be icc, can you point me to where in the configure/make system
I can make this change to force it to use what's sets as CC instead of
defaulting to GCC?

Rocky

On Fri, Mar 20, 2020 at 3:47 PM Simon Marchi <simark at simark.ca> wrote:

> On 2020-03-20 5:10 p.m., Rocky Dunlap via lttng-dev wrote:
> > I am trying to compile BT2 with Python bindings using Intel18.  I
> receive the following error during the build:
> Hi Rocky,
>
> I don't think we claim to support the Intel compiler, so you might be a bit
> on your own here.  Although if you want to send patches to fix the build
> using
> that compiler, I don't have anything against that.
>
> >
> > In file included from py-common.c(31):
> > /apps/intel/intelpython3/include/python3.6m/Python.h(149): error #193:
> zero used for undefined preprocessing identifier "_MSC_VER"
> >   #if _MSC_VER
> >       ^
> >
> > In file included from py-common.c(31):
> > /apps/intel/intelpython3/include/python3.6m/Python.h(151): error #193:
> zero used for undefined preprocessing identifier "__clang__"
> >   #elif __clang__ || __GNUC__
> >         ^
>
> It took me a bit of time to understand that this is Intel's Python 3
> distribution, not CPython.
>
> I think it's technically valid to use an undefined macro in a preprocessor
> condition like that, in which case it gets replaced with 0 (as the error
> message mentions).  The compiler is trying to be helpful and warns you,
> because
> relying on that behavior is a bit fragile, and often a sign of a mistake
> somewhere.  But since this happens in a library you are using, I think
> your best
> bet is just to disable this warning.
>
> > py-common.c(187): error #3179: deprecated conversion of string literal
> to char* (should be const char*)
> >   format_exc_func_name = py_exc_tb ? "format_exception" :
>
> I really don't understand this one, as format_exc_func_name is a const
> char *
> in our code.
>
> Simon
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200320/658950d6/attachment.htm>

[lttng-dev] Babeltrace2 - compilation error with intel18

[Simon Marchi]

On 2020-03-20 5:55 p.m., Rocky Dunlap wrote:
> Simon,
> 
> I was able to get past the issue by passing "--enable-compile-warnings=yes" to configure.
> 
> It get a lot further, then fails here:
> 
> gcc -pthread -shared -L/apps/intel/intelpython3/lib -Wl,-rpath=/apps/intel/intelpython3/lib,--no-as-needed -Wl,-z,noexecstack,-z,relro,-z,now -L../../../../src/lib/.libs -pthread -lgmodule-2.0 -lglib-2.0 -pthread -lgmodule-2.0 -lglib-2.0 -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -pthread -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fno-strict-aliasing -Wnested-externs -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wimplicit-function-declaration -Wold-style-definition -Wjump-misses-init -Wall -Wextra -Wundef -Wwrite-strings -Wpointer-arith -Wmissing-declarations -Wredundant-decls -Wno-unused-parameter -Wno-missing-field-initializers -Wformat=2 -Wcast-align -Wformat-nonliteral -Wformat-security -Wsign-compare -Wstrict-aliasing -Wshadow -Winline -Wpacked -Wmissing-format-attribute -Wmissing-noreturn -Winit-self -Wmissing-include-dirs -Wunused-but-set-variable -Warray-bounds -Wreturn-type -Wswitch-enum
> -Wswitch-default -Wduplicated-cond -Wduplicated-branches -Wlogical-op -Wrestrict -Wnull-dereference -Wdouble-promotion -Wno-sign-compare -Wno-inline -Wno-declaration-after-statement -Wno-switch-enum -Wno-switch-default -Wno-packed -Wno-pointer-arith -Wno-format-nonliteral -Wno-double-promotion -Wno-cast-align -Wno-cast-function-type -Wno-error=unused-parameter -Wno-error=missing-field-initializers -Wno-error=sign-compare -Wno-error=inline -Wno-error=declaration-after-statement -Wno-error=switch-enum -Wno-error=switch-default -Wno-error=packed -Wno-error=pointer-arith -Wno-error=format-nonliteral -Wno-error=double-promotion -Wno-error=cast-align -Wno-error=cast-function-type -Wold-style-definition -Werror=implicit-function-declaration -g -O2 -Wno-shadow -Wno-null-dereference -I../../../../include -I../../../../src -I../../../../src -include common/config.h -I./bt2 build/temp.linux-x86_64-3.6/bt2/native_bt.o build/temp.linux-x86_64-3.6/./bt2/logging.o
> ../../../../src/autodisc/.libs/libbabeltrace2-autodisc.a ../../../../src/logging/.libs/libbabeltrace2-logging.a ../../../../src/common/.libs/libbabeltrace2-common.a ../../../../src/py-common/.libs/libbabeltrace2-py-common.a ../../../../src/string-format/.libs/libbabeltrace2-string-format.a -L/apps/intel/intelpython3/lib -lbabeltrace2 -lglib-2.0 -lpython3.6m -o build/build_lib/bt2/_native_bt.cpython-36m-x86_64-linux-gnu.so <http://native_bt.cpython-36m-x86_64-linux-gnu.so>
> gcc: error: unrecognized command line option ?-Wduplicated-cond?
> gcc: error: unrecognized command line option ?-Wduplicated-branches?
> gcc: error: unrecognized command line option ?-Wrestrict?
> gcc: error: unrecognized command line option ?-Wnull-dereference?
> error: command 'gcc' failed with exit status 1
> make[4]: *** [build-python-bindings.stamp] Error 1
> 
> I am surprised that this is using "gcc" here, shouldn't it be using "icc" since I have CC=icc?
> 
> Or maybe it is supposed to be using gcc?? I think if I update the GCC compiler version then it will get past this issue as well, but I just wanted confirmation that this indeed should be using a combination of gcc and icc?
> 
> If it should be icc, can you point me to where in the configure/make system I can make this change to force it to use what's sets as CC instead of defaulting to GCC?
> 
> Rocky

I've hit this exact issue by trying to compile with CC=clang.  This is due to an
oddity of distutils when it builds the native module, I don't really know what to
do about it.

When it compiles native_bt.o, distutils uses the compiler specified in CC, which is
what we want.  But when it links the shared object, it:

  - uses the default system compiler (often gcc), and
  - passes the flags specified in CFLAGS

However, CFLAGS contains the warning flags that our configure script determined
$CC accepts (note, that detection might not work well with icc, we haven't tested).

So we end up passing warning flags recognized by $CC (e.g. clang) to gcc, which makes
gcc error out.

distutils allows to override the command used to link shared libraries by setting the
LDSHARED variable.  So you could try to set LDSHARED to "icc" plus the right flags
required to link a shared lib.  For reference, the default LDSHARED value for your Python
installation can be checked with:

    $ python3 -c 'import sysconfig; print(sysconfig.get_config_var("LDSHARED"))'
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro

But even worse, when building with CC=clang on Arch Linux, I noticed that distutils
passes some gcc-specific flags (which don't come from us) when compiling.  I'm not
sure what we're going to do about that.

Simon

[lttng-dev] Babeltrace2 - compilation error with intel18

[Rocky Dunlap]

Simon,

Thanks for this - so after I updated to gcc 9.2, the shared libraries
command ran correctly and the build finished. All the tests pass and I was
able to read in a CTF trace from a python script.

The issue was that the system gcc was gcc 4.X.X, so too old.  So, I had to
manually add the newer gcc to the path alongside the intel compiler.  I can
try to option of setting LDSHARED, but at this point that seems even more
confusing than just using the system gcc, which apparently has no trouble
linking intel object files into a shared library.  Maybe there are good
reasons why this is the case and this is why distutils does not use the
right compiler?

Ideally the build would just "work" with ./configure, make, make install
since these kinds of compilation issues can quickly get over people's heads
and cause them to abandon the library altogether.  I wonder if there is a
way for the configure/make to set the LDSHARED automatically so that is is
transparent to the user.  Certainly we are not the first to run into this
issue with distutils.

Rocky

On Fri, Mar 20, 2020 at 4:20 PM Simon Marchi <simark at simark.ca> wrote:

> On 2020-03-20 5:55 p.m., Rocky Dunlap wrote:
> > Simon,
> >
> > I was able to get past the issue by passing
> "--enable-compile-warnings=yes" to configure.
> >
> > It get a lot further, then fails here:
> >
> > gcc -pthread -shared -L/apps/intel/intelpython3/lib
> -Wl,-rpath=/apps/intel/intelpython3/lib,--no-as-needed
> -Wl,-z,noexecstack,-z,relro,-z,now -L../../../../src/lib/.libs -pthread
> -lgmodule-2.0 -lglib-2.0 -pthread -lgmodule-2.0 -lglib-2.0 -pthread
> -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -pthread -pthread
> -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fno-strict-aliasing
> -Wnested-externs -Wmissing-prototypes -Wstrict-prototypes
> -Wdeclaration-after-statement -Wimplicit-function-declaration
> -Wold-style-definition -Wjump-misses-init -Wall -Wextra -Wundef
> -Wwrite-strings -Wpointer-arith -Wmissing-declarations -Wredundant-decls
> -Wno-unused-parameter -Wno-missing-field-initializers -Wformat=2
> -Wcast-align -Wformat-nonliteral -Wformat-security -Wsign-compare
> -Wstrict-aliasing -Wshadow -Winline -Wpacked -Wmissing-format-attribute
> -Wmissing-noreturn -Winit-self -Wmissing-include-dirs
> -Wunused-but-set-variable -Warray-bounds -Wreturn-type -Wswitch-enum
> > -Wswitch-default -Wduplicated-cond -Wduplicated-branches -Wlogical-op
> -Wrestrict -Wnull-dereference -Wdouble-promotion -Wno-sign-compare
> -Wno-inline -Wno-declaration-after-statement -Wno-switch-enum
> -Wno-switch-default -Wno-packed -Wno-pointer-arith -Wno-format-nonliteral
> -Wno-double-promotion -Wno-cast-align -Wno-cast-function-type
> -Wno-error=unused-parameter -Wno-error=missing-field-initializers
> -Wno-error=sign-compare -Wno-error=inline
> -Wno-error=declaration-after-statement -Wno-error=switch-enum
> -Wno-error=switch-default -Wno-error=packed -Wno-error=pointer-arith
> -Wno-error=format-nonliteral -Wno-error=double-promotion
> -Wno-error=cast-align -Wno-error=cast-function-type -Wold-style-definition
> -Werror=implicit-function-declaration -g -O2 -Wno-shadow
> -Wno-null-dereference -I../../../../include -I../../../../src
> -I../../../../src -include common/config.h -I./bt2
> build/temp.linux-x86_64-3.6/bt2/native_bt.o
> build/temp.linux-x86_64-3.6/./bt2/logging.o
> > ../../../../src/autodisc/.libs/libbabeltrace2-autodisc.a
> ../../../../src/logging/.libs/libbabeltrace2-logging.a
> ../../../../src/common/.libs/libbabeltrace2-common.a
> ../../../../src/py-common/.libs/libbabeltrace2-py-common.a
> ../../../../src/string-format/.libs/libbabeltrace2-string-format.a
> -L/apps/intel/intelpython3/lib -lbabeltrace2 -lglib-2.0 -lpython3.6m -o
> build/build_lib/bt2/_native_bt.cpython-36m-x86_64-linux-gnu.so <
> http://native_bt.cpython-36m-x86_64-linux-gnu.so>
> > gcc: error: unrecognized command line option ?-Wduplicated-cond?
> > gcc: error: unrecognized command line option ?-Wduplicated-branches?
> > gcc: error: unrecognized command line option ?-Wrestrict?
> > gcc: error: unrecognized command line option ?-Wnull-dereference?
> > error: command 'gcc' failed with exit status 1
> > make[4]: *** [build-python-bindings.stamp] Error 1
> >
> > I am surprised that this is using "gcc" here, shouldn't it be using
> "icc" since I have CC=icc?
> >
> > Or maybe it is supposed to be using gcc?  I think if I update the GCC
> compiler version then it will get past this issue as well, but I just
> wanted confirmation that this indeed should be using a combination of gcc
> and icc?
> >
> > If it should be icc, can you point me to where in the configure/make
> system I can make this change to force it to use what's sets as CC instead
> of defaulting to GCC?
> >
> > Rocky
>
> I've hit this exact issue by trying to compile with CC=clang.  This is due
> to an
> oddity of distutils when it builds the native module, I don't really know
> what to
> do about it.
>
> When it compiles native_bt.o, distutils uses the compiler specified in CC,
> which is
> what we want.  But when it links the shared object, it:
>
>   - uses the default system compiler (often gcc), and
>   - passes the flags specified in CFLAGS
>
> However, CFLAGS contains the warning flags that our configure script
> determined
> $CC accepts (note, that detection might not work well with icc, we haven't
> tested).
>
> So we end up passing warning flags recognized by $CC (e.g. clang) to gcc,
> which makes
> gcc error out.
>
> distutils allows to override the command used to link shared libraries by
> setting the
> LDSHARED variable.  So you could try to set LDSHARED to "icc" plus the
> right flags
> required to link a shared lib.  For reference, the default LDSHARED value
> for your Python
> installation can be checked with:
>
>     $ python3 -c 'import sysconfig;
> print(sysconfig.get_config_var("LDSHARED"))'
>     x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> -Wl,-Bsymbolic-functions -Wl,-z,relro
>
> But even worse, when building with CC=clang on Arch Linux, I noticed that
> distutils
> passes some gcc-specific flags (which don't come from us) when compiling.
> I'm not
> sure what we're going to do about that.
>
> Simon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200320/27eae443/attachment-0001.htm>

[lttng-dev] Babeltrace2 - compilation error with intel18

[Simon Marchi]

On 2020-03-20 6:32 p.m., Rocky Dunlap via lttng-dev wrote:
> Simon,
> 
> Thanks for this - so after I updated to gcc 9.2, the shared libraries command ran correctly and the build finished. All the tests pass and I was able to read in a CTF trace from a python script.??
> 
> The issue was that the system gcc was gcc 4.X.X, so too old.? So, I had to manually add the newer gcc to the path alongside the intel compiler.? I can try to option of setting LDSHARED, but at this point that seems even more confusing?than just using the system gcc, which apparently has no trouble linking intel object files into a shared library.? Maybe there are good reasons why this is the case and this is why distutils does not use the right compiler?

Glad you got it working.

> Ideally the build would just "work" with ./configure, make, make install since these kinds of compilation issues can quickly get over people's heads and cause them to abandon the library altogether.? I wonder if there is a way for the configure/make to set the LDSHARED automatically?so that is is transparent to the user.? Certainly we are not the first to run into this issue with distutils.

I don't know if there's a good reason for them not to use $CC
for linking.  What's funny is that on Darwin/macOS they do it:

https://github.com/python/cpython/blob/6723e933c4d90a408cf3818362a0e4de6d84c932/Lib/distutils/sysconfig.py#L201-L209

Note that I don't know if Intel's Python distribution is using the
same distutils package as CPython.  But since you hit the same bug,
it must be quite close, if not the same.

There's a bug here about re-using CC for LDSHARED on Linux:

  https://bugs.python.org/issue24935

There are patches here

  https://github.com/python/cpython/pull/14950
  https://github.com/python/cpython/pull/7914

There wasn't much progress, but at least nobody spoke against them.

But even with that fixed there are other problems.  As I mentioned in my
previous message, in some cases, the CFLAGS value built into Python (which
means there's no easy way for us to override it) contains some flags
incompatible with clang (this could happen with icc too).  When trying to
build with CC=clang-9 on Arch Linux, I get:

ccache clang ... -fno-semantic-interposition ... -c bt2/native_bt.c -o build/temp.linux-x86_64-3.8/bt2/native_bt.o
clang-9: error: unknown argument: '-fno-semantic-interposition'

Arch Linux got this idea of building with `-fno-semantic-interposition` from Fedora:

  https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup

So I guess we will face the same issue on Red Hat by the end of the decade.

So since distutils really wants to compile the Python native modules using all the same
flags as the Python interpreter was built with, I presume that they really assume that
you'll be using the exact same toolchain to build your module as was used to build the
interpreter.  Maybe we could just not pass CC/CFLAGS when building the Python module,
so it will simply be built with the same compiler/linker as Python was built with, and
we'll avoid all these problems...

Simon

[lttng-dev] Babeltrace2 - compilation error with intel18

[Simon Marchi]

On 2020-03-20 11:12 p.m., Simon Marchi via lttng-dev wrote:
> So since distutils really wants to compile the Python native modules using all the same
> flags as the Python interpreter was built with, I presume that they really assume that
> you'll be using the exact same toolchain to build your module as was used to build the
> interpreter.  Maybe we could just not pass CC/CFLAGS when building the Python module,
> so it will simply be built with the same compiler/linker as Python was built with, and
> we'll avoid all these problems...

If we want to go this route, here's a patch that implements it.

  https://review.lttng.org/c/babeltrace/+/3257

This makes it so we don't override the compiler or flags (other than necessary includes
flags) when building the native module.  So when configuring with CC=clang, the Python
native module gets built with the Python distribution's default compiler, with just the
flags it wants.

Could you check if that works with CC=icc as well?

Simon

[lttng-dev] Babeltrace2 - compilation error with intel18

[Rocky Dunlap]

Simon,

Yes, I will be happy to give this a try.  What's the easiest way to get
this patch?  (Sorry, I'm less familiar with Gerrit...)

Rocky

On Mon, Mar 23, 2020 at 9:44 AM Simon Marchi <simark at simark.ca> wrote:

> On 2020-03-20 11:12 p.m., Simon Marchi via lttng-dev wrote:
> > So since distutils really wants to compile the Python native modules
> using all the same
> > flags as the Python interpreter was built with, I presume that they
> really assume that
> > you'll be using the exact same toolchain to build your module as was
> used to build the
> > interpreter.  Maybe we could just not pass CC/CFLAGS when building the
> Python module,
> > so it will simply be built with the same compiler/linker as Python was
> built with, and
> > we'll avoid all these problems...
>
> If we want to go this route, here's a patch that implements it.
>
>   https://review.lttng.org/c/babeltrace/+/3257
>
> This makes it so we don't override the compiler or flags (other than
> necessary includes
> flags) when building the native module.  So when configuring with
> CC=clang, the Python
> native module gets built with the Python distribution's default compiler,
> with just the
> flags it wants.
>
> Could you check if that works with CC=icc as well?
>
> Simon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200323/ca639269/attachment.htm>

[lttng-dev] Babeltrace2 - compilation error with intel18

[Simon Marchi]

On 2020-03-23 12:05 p.m., Rocky Dunlap wrote:
> Simon,
> 
> Yes, I will be happy to give this a try.? What's the easiest way to get this patch?? (Sorry, I'm less familiar with Gerrit...)
> 
> Rocky

You need a babeltrace git repository.  If you don't already have one, you can clone it from here:

  https://github.com/efficios/babeltrace

You can then click the "DOWNLOAD" button on the patch's review page, or press `d` (a keybinding).

Copy the "Checkout" command (the first one) and paste it in your shell (while having the
repository as the current working directory).  You'll then have the code corresponding to
the proposed patch checked out and ready to build.  Note that when you build from the git
repo, you need to run ./bootstrap first, to generate the configure script.

Simon

[lttng-dev] Babeltrace2 - compilation error with intel18

[Rocky Dunlap]

Simon,

Success!  This change worked and I was able to build and install with icc.
Thank you!

I still get a lot of warnings during the build (see below) which seems like
configure should work these out ahead of time.  They are easy enough to
ignore, but if support for intel is added, it would be nice to have these
cleaned up.  For what it's worth, adding support for additional compilers
in my experience has greatly improved the quality of library code - it
tends to snuff out those bugs that are hard to find on a single compiler.

If you decide to merge this in, a release patch 2.0.3 would certainly be
helpful as well!


Example warnings below:
icc: command line warning #10006: ignoring unknown option '-Wnested-externs'
icc: command line warning #10006: ignoring unknown option
'-Wtautological-constant-out-of-range-compare'
icc: command line warning #10006: ignoring unknown option '-Wnested-externs'
icc: command line warning #10006: ignoring unknown option
'-Wno-format-nonliteral'
icc: command line warning #10006: ignoring unknown option
'-Wno-format-nonliteral'
icc: command line warning #10006: ignoring unknown option
'-Wmissing-noreturn'
icc: command line warning #10006: ignoring unknown option
'-Wmissing-noreturn'
icc: command line warning #10006: ignoring unknown option
'-Wduplicated-cond'
icc: command line warning #10006: ignoring unknown option
'-Wduplicated-branches'
icc: command line warning #10006: ignoring unknown option
'-Wduplicated-cond'
icc: command line warning #10006: ignoring unknown option
'-Wduplicated-branches'
icc: command line warning #10006: ignoring unknown option '-Wlogical-op'
icc: command line warning #10006: ignoring unknown option '-Wlogical-op'
icc: command line warning #10006: ignoring unknown option
'-Wno-cast-function-type'
icc: command line warning #10006: ignoring unknown option
'-Wno-cast-function-type'
icc: command line warning #10006: ignoring unknown option
'-Wnull-dereference'
icc: command line warning #10006: ignoring unknown option
'-Wredundant-decls'
icc: command line warning #10006: ignoring unknown option
'-Wjump-misses-init'
icc: command line warning #10006: ignoring unknown option
'-Wsuggest-attribute=format'
icc: command line warning #10006: ignoring unknown option
'-Wtautological-constant-out-of-range-compare'
icc: command line warning #10006: ignoring unknown option '-Wnested-externs'
icc: command line warning #10006: ignoring unknown option
'-Wno-format-nonliteral'
icc: command line warning #10006: ignoring unknown option
'-Wmissing-noreturn'
icc: command line warning #10006: ignoring unknown option
'-Wduplicated-cond'
icc: command line warning #10006: ignoring unknown option
'-Wduplicated-branches'
icc: command line warning #10006: ignoring unknown option '-Wlogical-op'
icc: command line warning #10006: ignoring unknown option
'-Wno-cast-function-type'
icc: command line warning #10006: ignoring unknown option
'-Wnull-dereference'
icc: command line warning #10006: ignoring unknown option
'-Wredundant-decls'
icc: command line warning #10006: ignoring unknown option
'-Wjump-misses-init'
icc: command line warning #10006: ignoring unknown option
'-Wsuggest-attribute=format'
icc: command line warning #10006: ignoring unknown option
'-Wtautological-constant-out-of-range-compare'


On Mon, Mar 23, 2020 at 10:14 AM Simon Marchi <simark at simark.ca> wrote:

> On 2020-03-23 12:05 p.m., Rocky Dunlap wrote:
> > Simon,
> >
> > Yes, I will be happy to give this a try.  What's the easiest way to get
> this patch?  (Sorry, I'm less familiar with Gerrit...)
> >
> > Rocky
>
> You need a babeltrace git repository.  If you don't already have one, you
> can clone it from here:
>
>   https://github.com/efficios/babeltrace
>
> You can then click the "DOWNLOAD" button on the patch's review page, or
> press `d` (a keybinding).
>
> Copy the "Checkout" command (the first one) and paste it in your shell
> (while having the
> repository as the current working directory).  You'll then have the code
> corresponding to
> the proposed patch checked out and ready to build.  Note that when you
> build from the git
> repo, you need to run ./bootstrap first, to generate the configure script.
>
> Simon
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200323/3b431681/attachment.htm>

[lttng-dev] Babeltrace2 - compilation error with intel18

[Simon Marchi]

On 2020-03-23 12:56 p.m., Rocky Dunlap wrote:
> Simon,
> 
> Success!? This change worked and I was able to build and install with icc.? Thank you!

Ok, thanks.  Out of curiosity, did run a "make check"?  That would validate that
the Python bindings (and everything else) works properly.

> I still get a lot of warnings during the build (see below) which seems like configure should work these out ahead of time.? They are easy enough to ignore, but if support for intel is added, it would be nice to have these cleaned up.? For what it's worth, adding support for additional compilers in my experience has greatly improved the quality of library code - it tends to snuff out those bugs that are hard to find on a single compiler.??

I agree, that's why we want to keep babeltrace building with clang (on top of having to
support macOS, on which clang is the default compiler).

We use the AX_APPEND_COMPILE_FLAGS autoconf macro [1] to determine which warning flags
the current compiler supports.  This calls the compiler with each flag individually
to see if the compiler supports that flag.  This requires that the compiler errors
out when passed an invalid flag.  Currently, it looks like icc emits a warning, but still
returns exit code 0 when passed an invalid flag, so we end up enabling all of them.  Try
to check in config.log to see what happens currently.  For example here are the test
results for two flags, one supported and one not supported by clang:

4625 configure:17825: checking whether C compiler accepts -Winit-self
4626 configure:17844: ccache clang -c -gdwarf-5 -g3 -O2 -fsanitize=address -Werror -Winit-self  conftest.c >&5
4627 configure:17844: $? = 0
4628 configure:17853: result: yes
4629 configure:17869: : WARN_CFLAGS="$WARN_CFLAGS"
4630 configure:17872: $? = 0
4631 configure:17825: checking whether C compiler accepts -Wduplicated-cond
4632 configure:17844: ccache clang -c -gdwarf-5 -g3 -O2 -fsanitize=address -Werror -Wduplicated-cond  conftest.c >&5
4633 error: unknown warning option '-Wduplicated-cond'; did you mean '-Wduplicate-enum'? [-Werror,-Wunknown-warning-option]
4634 configure:17844: $? = 1

If you can come up with the right incantation to make that work well with icc, we
can modify how we call the macro or modify the macro itself (sending a patch to
the upstream, Autoconf Archive).

[1] https://www.gnu.org/software/autoconf-archive/ax_append_compile_flags.html

> If you decide to merge this in, a release patch 2.0.3 would certainly be helpful as well!

Things have started to diverge a bit between master and the 2.0 branch, I'm not sure if we'll
be able to backport that, we'll see.

Simon
> 

[lttng-dev] Babeltrace2 - compilation error with intel18

[Rocky Dunlap]

Simon,

See below...

On Mon, Mar 23, 2020 at 11:24 AM Simon Marchi <simark at simark.ca> wrote:

> On 2020-03-23 12:56 p.m., Rocky Dunlap wrote:
> > Simon,
> >
> > Success!  This change worked and I was able to build and install with
> icc.  Thank you!
>
> Ok, thanks.  Out of curiosity, did run a "make check"?  That would
> validate that
> the Python bindings (and everything else) works properly.
>

============================================================================
Testsuite summary for babeltrace2 2.1.0-rc1
============================================================================
# TOTAL: 21225
# PASS:  21225
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================



>
> > I still get a lot of warnings during the build (see below) which seems
> like configure should work these out ahead of time.  They are easy enough
> to ignore, but if support for intel is added, it would be nice to have
> these cleaned up.  For what it's worth, adding support for additional
> compilers in my experience has greatly improved the quality of library code
> - it tends to snuff out those bugs that are hard to find on a single
> compiler.
>
> I agree, that's why we want to keep babeltrace building with clang (on top
> of having to
> support macOS, on which clang is the default compiler).
>
> We use the AX_APPEND_COMPILE_FLAGS autoconf macro [1] to determine which
> warning flags
> the current compiler supports.  This calls the compiler with each flag
> individually
> to see if the compiler supports that flag.  This requires that the
> compiler errors
> out when passed an invalid flag.  Currently, it looks like icc emits a
> warning, but still
> returns exit code 0 when passed an invalid flag, so we end up enabling all
> of them.  Try
> to check in config.log to see what happens currently.  For example here
> are the test
> results for two flags, one supported and one not supported by clang:
>
> 4625 configure:17825: checking whether C compiler accepts -Winit-self
> 4626 configure:17844: ccache clang -c -gdwarf-5 -g3 -O2 -fsanitize=address
> -Werror -Winit-self  conftest.c >&5
> 4627 configure:17844: $? = 0
> 4628 configure:17853: result: yes
> 4629 configure:17869: : WARN_CFLAGS="$WARN_CFLAGS"
> 4630 configure:17872: $? = 0
> 4631 configure:17825: checking whether C compiler accepts -Wduplicated-cond
> 4632 configure:17844: ccache clang -c -gdwarf-5 -g3 -O2 -fsanitize=address
> -Werror -Wduplicated-cond  conftest.c >&5
> 4633 error: unknown warning option '-Wduplicated-cond'; did you mean
> '-Wduplicate-enum'? [-Werror,-Wunknown-warning-option]
> 4634 configure:17844: $? = 1
>
> If you can come up with the right incantation to make that work well with
> icc, we
> can modify how we call the macro or modify the macro itself (sending a
> patch to
> the upstream, Autoconf Archive).
>
> [1]
> https://www.gnu.org/software/autoconf-archive/ax_append_compile_flags.html


Okay, I'll try to find some time look into this.



>
>
> > If you decide to merge this in, a release patch 2.0.3 would certainly be
> helpful as well!
>
> Things have started to diverge a bit between master and the 2.0 branch,
> I'm not sure if we'll
> be able to backport that, we'll see.
>

Sure, no problem.  I'm not sure what the overall release schedule is, so
maybe it would be a 2.1 at some point soon?


>
> Simon
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200323/f2231010/attachment-0001.htm>