Build fails: opensslconf.h not found
$ git log --oneline -1 94f5cb4 Merge branch 'fix_tests' into 'master' $ python setup.py build running build running build_py copying M2Crypto/EC.py -> build/lib.linux-i686-2.7/M2Crypto [...copying other files...] running build_ext building 'M2Crypto.__m2crypto' extension swigging SWIG/_m2crypto.i to SWIG/_m2crypto_wrap.c swig -python -I/usr/include/python2.7 -I/usr/include -includeall -modern -builtin -outdir build/lib.linux-i686-2.7/M2Crypto -o SWIG/_m2crypto_wrap.c SWIG/_m2crypto.i SWIG/_evp.i:12: Error: Unable to find 'openssl/opensslconf.h' SWIG/_ec.i:7: Error: Unable to find 'openssl/opensslconf.h' error: command 'swig' failed with exit status 1
The missing OpenSSL include file,
openssl/opensslconf.h, is installed in
/usr/include/i386-linux-gnu by Ubuntu package
The error is caused by two factors:
- commit fbf27335 (made in 2006) added
#include <openssl/opensslconf.h>to file
- Ubuntu package
opensslconf.hto the architecture-dependent directory in 2013.
Any suggestions for a way to fix this?
- commit fbf27335 (made in 2006) added
It's strange. I thought that Travis CI runs on Ubuntu, and it builds there https://travis-ci.org/mcepl/M2Crypto. See https://github.com/mcepl/M2Crypto/blob/master/.travis.yml for the steps of reproduction.
Tested. Result is the same. This is expected, as the "offending" file
SWIG/_ec.iis the same in both versions. As a temporary personal workaround, I symlink
/usr/include/openssl. Do not know what the proper resolution should be.
I run into the same issue on Ubuntu 14.04 with Python 2.6.
I have also solved this by symlinking opensslconf.h just like Konstantin did. This stackoverflow article recommends this as a solution. IMO, while one can argue that libssl-dev broke its backwards compatibility by moving the file out of the standard include path, fixing this by symlinking the file does not seem like the right solution to me (except if the libssl folks agree to fix it themselves by re-establishing that file).
Instead, I suggest that M2Crypto adjusts to the change in libssl-dev and adds that architecture specific directory to the include path for swig.
The following change in an M2Crypto fork does it that way, but uses a hard coded directory name: https://github.com/eventbrite/m2crypto/commit/6e99c074de291a60cf6db203032d6d3104055cb0
I did not research how the directory name is constructed by libssl-dev, but I suppose it contains the processor architecture, so the following lines should work, instead of the two lines shown in the referenced change:
arch_include_dir = '/usr/include/%s-linux-gnu' % platform.processor() if os.path.exists(arch_include_dir): self.include_dirs.append(arch_include_dir)
Not sure whether and how the "-linux-gnu" portion of the include directory needs to be determined dynamically.
This is complete mess. We have different location of these files in the Fedora/RHEL universe as well (see the whole script https://gitlab.com/m2crypto/m2crypto/blob/master/fedora_setup.sh which I would love to eliminate asap, see #49). There must be some standard way how to deal with this in some way which at least from far resembles sanity. I will investigate this.
Could people here try the
fix_tests_cleanupbranch now (commit da23b14 or from PyPI version
0.22.6rc4), please? I tried to make the situation working without crutches of weird shell scripts.
I did not fully research the issue on Fedora, but on RHEL 6.7, the issue is that opensslconf.h depends on an architecture symbol being set, e.g. x86_64. That is set by gcc automatically, but not by swig. This is described in the issue description of this stackoverflow article. Its answer #6 comes closest to the solution. I saw another article that had the full solution but cannot find it anymore.
Anyway, what I suggest as a solution is:
-D__<arch>__to the swig options, where
<arch>is the processor architecture.
-I/usr/include/opensslto the swig options. This is needed because
opensslconf-<arch>.hwithout specifying the relative directory
- Some solutions recommend adding
-cpperraswarnto the swig options. This just hides errors and I think it should not be done.
- Some solutions recommend adding
-includeallto the swig options, but the current version of M2Crypto's setup.py already defines that.
Instead of setting the SWIG_FEATURES environment variable as recommended in some solutions, I think this should be done in M2Crypto's setup.py.
For the architecture symbol, the following worked for me:
self.swig_opts.append('-D__%s__' % platform.processor())
I also added the directory somewhere, and this works on RHEL 6.7 with swig 2.0.12 (which I had to build myself, there).
Edit: I should add, that the entire "fedora hat trick" section in setup.py can be removed if the line proposed above gets added. It did not work for RHEL, anyway, and was limited in its architecture scope.
Matej, I have commented on the commit. It does not work for RHEl 6.7.
Bottom line, I suggest that we do all of this in setup.py:
- add the changes for Ubuntu mentioned in my comment that is 5 comments up from here.
- add the changes for RHEL mentioned in my comment that is 2 comments up from here.
- do not not add
-cpperraswarnto the swig options, in order not to hide errors.
These changes have been tested on Ubuntu 14.04 and RHEl 6.7 (on x86_64), based on M2Crypto 0.22.6-rc3, and I have yet to test them on CentOS.
@andy-maier what happens on Debian/Ubuntu if I add just
/usr/include/$(DEB_HOST_MULTIARCH)/openssl) and proper
swig_opts? Concerning the latter I tend to shift towards
uname -mas a proper way how to get the right (what should be the proper value on
ppc64which uses 32bit libraries? what do I get on Mac OS X?)
uname -p(which is where
platform.processor()tends to finish seems to be wrong; but let me test it here on some more weird internal machines).
With 64-bit Ubuntu 14.04 on a x86_64 virtual machine (KVM powered), adding just
-cpperraswarn) to the swig options works fine. I don't have Debian around to test with.
uname -m: Good catch! I was confused and thought
uname -mwould translate to
platform.processor(), but that is not the case; it translates to
platform.machine(). Sorry for that.
Still, both use cases for
-D__<arch>__option that fixes the RHEL issue, and the
-I/usr/include/<arch>-linux-gnuswig option that fixes the Ubuntu issue) need to be carefully tested to verify that using
platform.machine()does the right thing. Let me see what machines I can get hold of.
Edit: I have to correct myself. I just verified that in addition, I had added
-I/usr/include/x86_64-linux-gnuto the swig options on Ubuntu. If that is not added, then opensslconf.h is not found and swig fails.
I think adding both
if 'DEB_HOST_MULTIARCH' in os.environ) should be preferable so that we have covered even the older versions of Debian/Ubuntu, and we have covered even an unlikely option that somebody goes mad and moves
$DEB_HOST_MULTIARCHsomewhere else again).
reading manifest template 'MANIFEST.in' warning: manifest_maker: MANIFEST.in, line 5: 'recursive-include' expects <dir> <pattern1> <pattern2> ... warning: manifest_maker: MANIFEST.in, line 6: 'recursive-include' expects <dir> <pattern1> <pattern2> ... warning: manifest_maker: MANIFEST.in, line 7: 'recursive-include' expects <dir> <pattern1> <pattern2> ...
This is weird. You should have cb025dcc, shouldn't you?
I had a look at how this is used in the OpenSSL project: It is not an environment variable, but is set in the debian/rules script from the result of the command:
So unless I am wrong here, I think a viable approach would be to try to execute that command in setup.py, and if it succeeds, add an include directory, using its stdout string. I suspect that the use of the architecture-specific directory only happens on Debian-based distros, so the approach above would replace using the result of
platform.machine()for this purpose.
I'll try that out.
On the use of
__<arch>__symbols needed by the RHEL version of opensslconf.h:
I found so far that at least on Fedora 22 32-bit, this does not work, because it returns
i686while the header file expects
__i386__. So that requires normalizing
I'll try that out, too.
On the error messages: No, I used the master branch, but I realize now that your merge request (WIP: Only fedora patches) also affects setup.py, so I'll rebase my proposal on that merge request.
@andy-maier that's wrong ... I don't want you to test Fedora patches on Debian. We are talking only about https://gitlab.com/mcepl/m2crypto/tree/gcc_gymnastics branch now (commit 0f9891b ).
Sorry for the delay.
The changes in the setup.py in branch https://gitlab.com/mcepl/m2crypto/tree/gcc_gymnastics (latest commit 0f9891b) include code that attempts to access the environment variable DEB_HOST_MULTIARCH in order to determine the name of the architecture-specific include directory. That code is used if os.name == 'posix', which is should be the case on all Linux distros, including Fedora and Ubuntu. I don't think there is such an environment variable (correct me if I'm wrong here, but the attached log files test that and it did not exist anywhere). Instead, the architecture-specific include directory can be determined from the output of the command
dpkg-architecture -qDEB_HOST_MULTIARCHon systems that have that command, i.e. mostly Debian and Ubuntu. Also, the change in the setup.py removes the Fedora hat trick code (which is a good thing), but it does not replace it with what is needed for 32-bit RHEL, CentOS and Fedora systems, i.e. a mapping of the architectures returned by platform.machine() to the subset that is supported by their multi-arch version of opensslconf.h. Bottom line, my review of the changes in setup.py in the gcc_gymnastics branch is negative.
The setup.py in branch https://gitlab.com/mcepl/m2crypto/tree/only_fedora_patches (last commit dafb7a0b) is exactly the same as in branch gcc_gymnastics, so my review on that one is negative for the same reasons.
I have tested the install & import for branch gcc_gymnastics, and the results are as follows:
- Ubuntu 14 64-bit py27: Failure #1
- Ubuntu 14 32-bit py27: Failure #1
- Fedora 22 64-bit py27: Ok
- Fedora 22 32-bit py27:Failure #2
- openSUSE 13 64-bit py27: Ok
- openSUSE 13 32-bit py27: Ok
- RHEL 6 64-bit py26: Ok
- I don't have a RHEL 6 32-bit or CentOS 32-bit at hand, but I assume they would also result in Failure #2.
The full logs are in gcc_gymnastics_logs.zip
Failure #1: Fails with SWIG error: "Unable to find 'openssl/opensslconf.h'". Reason is that the arch-specific directory (/usr/include/i386-linux-gnu on Ubuntu 14 32-bit and /usr/include/x86_64-linux-gnu on Ubuntu 14 64-bit) is not specified as an include directory, which in turn is caused by the code that attempts to determine that directory from a non-existing DEB_HOST_MULTIARCH env var.
Failure #2: SWIG fails with CPP error: "This openssl-devel package does not work your architecture?". Reason is that while
-D__i686__is specified to SWIG, the opensslconf.h file only supports
__i386__for 32-bit x86 systems.
I have a version of setup.py that fixes both failures, and am attaching that setup.py. The changes since the gcc_gymnastics version are marked with
Edit: Removed details about server error 500; it works again for me.
Just for info, I made this locally on my instance waiting for your patch. Works for me on ubuntu 64bits.
/src/external/m2crypto$ git diff 10625197c29c1de5c9bd26361cbae12d8912a20d 31e88b69f384f7d5b8e567609da4196dd5c254f8 diff --git a/setup.py b/setup.py index 7660e03..99474da 100644 --- a/setup.py +++ b/setup.py @@ -60,6 +60,9 @@ class _M2CryptoBuildExt(build_ext.build_ext): self.swig_opts = ['-I%s' % i for i in self.include_dirs + [openssl_include_dir]] + if platform.linux_distribution() in ['Ubuntu']: + ubuntu_dir = '/usr/include/%s-linux-gnu' % platform.processor() + self.swig_opts.append('-I%s' % ubuntu_dir) self.swig_opts.append('-includeall') self.swig_opts.append('-modern') self.swig_opts.append('-builtin')
Hi Xav. Did you test this on 32-bit systems? On my Ubuntu 14.04 32-bit, the directory is /usr/include/i386-linux-gnu, but platform.processor() returns "i686". Also, this should be done for Debian as well. See the changes in the setup.py file I attached to my previous posting in this thread. Andy
By the way:
dpkg-architecture -qDEB_HOST_MULTIARCH on systems that have that command, i.e. mostly Debian and Ubuntu.
is not a good idea. Is
baseDebian? I don't think forcing users to install developers toolkit is a good idea.
/me just installs Debian in a VM to find out how
Initial research in package lists shows that the dpkg-architecture executable is part of the dpkg-dev package, which on Debian and Ubuntu is part of the distribution:
On my Ubuntu 14.04 Desktop installation, I think it was part of the distribution. However, double checking this on a fresh installation of both Debian and Ubuntu is a good idea.
Hi. Sorry, I didn't test with 32 bits platform, I don't have one here, sorry.
Let us do something abotu this baby, we apparently needs this for GitLab CI (and merge request #22) be working. I kind of like https://gitlab.com/rodrigc/m2crypto/commit/5e12c6ace0096a48fbaea0a64f5f5f8a6cebad2d by @rodrigc but I think we need to get better understanding on how to do the thing in a minimal number of lines of code. What I really don't want to end with is a gigantic unmaintainable switch for every known platform.
My original thougths were in https://gitlab.com/mcepl/m2crypto/commit/3148c12a20734521a0fa14c6c55dbd2667709e19, but @rodrigc code is better: shorter and is less platform dependent.
Couple of notes:
os.path.join(self.ssl, 'include', sysconfig.get_config_var("MULTIARCH"))works both on Debian as well as on Fedora/RHEL (because
sysconfig.get_config_var("MULTIARCH")is empty here). Is there any particular reason for it? Can somebody find out some documentation for this variable? Where does it come from?
For a good information on platform, we may use
sysconfig.get_platform()(together with more obvious
platform.linux_distribution()). Seems to contain architecture.
Truly, Debian doesn't seem to have a clue about
pkgconfigfile for OpenSSL. Pity. It seems to be only a Fedora/RHEL thing.
Any ideas about Mac OS X, Windows?
MULTIARCH is documented here (which I added in my commit message): https://wiki.debian.org/Python/MultiArch That link has further information about what MULTIARCH is on Debian systems.
sysconfig variables (such as MULTIARCH) are compiled into the Python package when it is created.
The patch I submitted will work for all Debian-derived Linux systems: Debian, Ubuntu, Linux Mint, etc. Technically, Debian exists for non-Linux operating systems as well such as SmartOS and kFreeBSD, so it would be legal to just unconditionally check for MULTIARCH:
import sys +import sysconfig requires_list =  if sys.version_info <= (2, 6): requires_list.append("unittest2") @@ -58,6 +59,14 @@ class _M2CryptoBuildExt(build_ext.build_ext): openssl_include_dir = os.path.join(self.openssl, 'include') openssl_library_dir = os.path.join(self.openssl, 'lib') + # Debian-based systems (Debian, Ubuntu, Linux Mint, SmartOS, kFreeBSD) + # put opensslconf.h in an architecture specific directory defined + # by the MULTIARCH variable which is compiled into the Python package + # and available via sysconfig. + multiarch = sysconfig.get_config_var("MULTIARCH") + if multiarch is not None: + self.include_dirs += [os.path.join(openssl_include_dir, multiarch)] +
Whether to go with the original patch I submitted, or the above modified one, I leave up to you. However, of all the patches I have seen so far which try to fix this problem, mine is the most minimal and introduces no external dependencies.
It is worth going with this due to the huge number of Debian and Ubuntu users out there.
I'm still getting this error on Travis-CI (Ubuntu Trusty) with python 2.6 (it works fine with python 2.7) with M2Crypto-0.23.0:
SWIG/_evp.i:12: Error: Unable to find 'openssl/opensslconf.h' SWIG/_ec.i:7: Error: Unable to find 'openssl/opensslconf.h'
And why do you use old version?
pip install -U ... 'M2Crypto>=0.22.6rc4'
Just do full Monty and install the latest package. It works for me (even with python 2.6) on https://travis-ci.org/mcepl/M2Crypto (specifically, the latest release is https://travis-ci.org/mcepl/M2Crypto/builds/105552350).
The version is just a leftover from previous testing, but it's installing the latest version, see the log: "Downloading M2Crypto-0.23.0.tar.gz (183kB)".
It failed for me even when I put just M2Crypto to requirements.txt.
So I examined differences between my and your CI builds and found out that yours are running on Ubuntu Precise and mine on Trusty. So I switched to Precise and m2crypto installation now works correctly. So it seems there is some problem on Ubuntu Trusty only.
Hmm, Trusty is more recent, right? That sounds bad. Perhaps explicit setting
setup.py buildwould help?
Yes, Trusty is more recent. The path to opensslconf.h is different on these versions, so that might be a problem.
# dpkg -L libssl-dev | grep opensslconf.h /usr/include/x86_64-linux-gnu/openssl/opensslconf.h
# dpkg -L libssl-dev | grep opensslconf.h /usr/include/openssl/opensslconf.h
python setup.py build --openssl=/usr/include/x86_64-linux-gnu/openssl
Yes, that helps.
I suspect that missing sysconfig module on python2.6 is the problem. Otherwise the setup.py script can figure out openssl lib using
I know, but that location is such a mess that we would end up with
if ... then ... elseconstruct of incredible length. If somebody discovers some reasonable way how to find out the location automagically, I won't do it.
You could use
pkg-configwith something like this:
import os import subprocess import sys def get_openssl_lib_path(): libdir = None msg = '' try: libdir = subprocess.check_output(['pkg-config', '--variable=libdir', 'openssl']).strip() except (subprocess.CalledProcessError, OSError), e: msg = str(e) if not libdir or not os.path.exists(libdir): sys.stderr.write('Warning: Unable to identify the lib directory for openssl: %s\n' % msg) return None else: return libdir
That's too easy.
pkg-configAFAIK works only on Linux (is there at least a one for Mac? Surely, there is not a one for Windows). Also, AFAIK
openssl.pcis just in the patched version of OpenSSL on some Linux distros, it is not in the upstream (correct me, if I am wrong).
That's too easy.
Could be. I'm not an expert in multi-platform support. :)
pkg-config AFAIK works only on Linux (is there at least a one for Mac? Surely, there is not a one for Windows).
According to its home page (http://www.freedesktop.org/wiki/Software/pkg-config/):
pkg-config works on multiple platforms: Linux and other UNIX-like operating systems, Mac OS X and Windows.
Also, AFAIK openssl.pc is just in the patched version of OpenSSL on some Linux distros, it is not in the upstream (correct me, if I am wrong).
No, it's not in the upstream (that I can see). But it's not specific to Linux distros. It's both in my native Mac OS X installation (in /usr/lib/pkgconfig/) and in my MacPorts installation (/opt/local/lib/pkgconfig/).
I'm still still not suggesting it's the answer, but it might be among the tools used, where it (and an openssl.pc file) is installed. Of course, that creates an install dependency.