CMake build does not work on MacOS

changed the description

One minor problem in the build is that color_lib has to be manually copied, e.g.

$ cp /Users/north/src/graphviz/lib/common/color_lib /Users/north/src/graphviz/build/lib/common/

The real problem is that the graphviz/build/lib/common/CMakeFiles/common.dir/globals.c.o built via cmake does not contain the global symbol definitions needed by graphviz. You can confirm, e.g. nm -Bo | grep Concentrate. It is a mystery why there this does not work. You can run make VERBOSE=1 and see it yourself. Maybe someone that has used cmake a lot can figure this out. Graphviz builds correctly with GNU autotools so I wonder if cmake is generating the wrong code. I don't have time to figure out how to fix cmake. Stephen North

Hi Stephen, thanks for the response.

Bit of background - I've used the same source bundle to successfully build Windows and Linux binaries through CMake - it's only Mac where it doesn't work.

I'm looking to integrate graphviz into our application as an embedded library (cgraph module), which is why CMake is preferred - it gives me control over rpath and/or framework creation on the Mac. I wouldn't know how to accomplish that using autotools. I can confirm that autotools does build correctly on the Mac, but only as a system-installed application. I cannot then turn it into a relocatable library without rewriting all the rpaths afterwards - a rather labour-intensive task...

I'm using the graphviz label stable_release_2.42.2 for my graphviz sources, CMake 3.14.2, MacOS 10.14.5 (Mojave), XCode command-line tools: Apple LLVM version 10.0.1 (clang-1001.0.46.4)

Mark, if you make use of Graphviz on a Mac. Would you please have a look at https://forum.graphviz.org/t/cmake-creates-common-library-as-object-library/500/9?u=vithanco ? Maybe you could provide some input?

I'm afraid that after I got it working once, the project was put on hold and I have not investigated further. Even then, the builds were for a single platform and not targeted at further distribution. I don't really know anything about Macs to be able to give further insight into multi-platform binaries, or even MacOS frameworks.

I notice Fedora has a slightly older clang:

cmake-3.14.5-4.fc31.x86_64 clang-9.0.0-1.fc31.x86_64 llvm-9.0.0-1.fc31.x86_64

I wonder about clang's special sauce for controlling visibility of symbols. https://clang.llvm.org/docs/LTOVisibility.html

Hmm, this missing export of eg _Concentrate, do you think it refer to this?

EXTERN unsigned char Concentrate;	/* if parallel edges should be merged */

from common/globals.h?

Does the Mac build use libtool?

libtool-2.4.6-31.fc31.x86_64

In the automake builds, libcommon is only built as a "convenience" library of .o's (libcommon_C_la), and then later linked into libgvc which is the installable library. This allows for essentially independent Makefiles for lib/common and lib/gvc.

But if the scope of symbols in libcommon is being restricted prematurely then this wont work.

A possible workaround might be for cmake to not compile anything in lib/common, but instead compile lib/common/.c and lib/gvc/.c directly into libgvc

Is LTO visibility a new feature in clang_10 ?

Can anyone try on a Mac with clang_9 ?

Perhaps we need to detect mac builds, and add: attribute((visibility("...")))

like the Windows:

__declspec(dllexport)

I've had the advantage of just trying a brand new / clean install MacOS, and yes I can confirm it needs/uses libtool:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
LTDL_INCLUDE_DIR (ADVANCED)
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
LTDL_INCLUDE_DIR (ADVANCED)

used by target "gvc"

And yes, I'm getting the exact same linker error, this is with CMake 3.15.4 and AppleClang 11.0.0

The globals.c.o built by cmake does not export global symbols in the way that the automake based compile does. One can determine this just by: $ cd ~/src/graphviz $ nm lib/common/.libs/globals.o | grep N_width 0000000000002c88 S _N_width $ nm build/lib/common/CMakeFiles/common.dir/globals.c.o | grep N_width 0000000000000008 C _N_width $

I don’t think libtool is involved because the compile doesn’t generate a

Here’s how cmake compiles globals.c.o:

[ 22%] Building C object lib/common/CMakeFiles/common.dir/globals.c.o cd /Users/north/src/graphviz/build/lib/common && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DDIGCOLA -DENABLE_LTDL -DGVC_EXPORTS -DORTHO -DSFDP -D_BLD_gvc=1 -I/Users/north/src/graphviz/build -I/Users/north/src/graphviz/lib/common -I/Users/north/src/graphviz/build/lib/common -I/Users/north/src/graphviz -I/Users/north/src/graphviz/lib/cdt -I/Users/north/src/graphviz/lib/cgraph -I/Users/north/src/graphviz/lib/gvc -I/Users/north/src/graphviz/lib/label -I/Users/north/src/graphviz/lib/pack -I/Users/north/src/graphviz/lib/pathplan -I/Users/north/src/graphviz/lib/fdpgen -I/Users/north/src/graphviz/lib/xdot -fPIC -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -o CMakeFiles/common.dir/globals.c.o -c /Users/north/src/graphviz/lib/common/globals.c

It appears that the automake generated Makefile has -fno-common and adding this to build/lib/common/CMakefiles/common.dir/flags.make allows the build to progress further, but then the compilation of the pango plugin dies: [ 84%] Building C object plugin/pango/CMakeFiles/gvplugin_pango.dir/gvgetfontlist_pango.c.o In file included from /Users/north/src/graphviz/plugin/pango/gvgetfontlist_pango.c:55: In file included from /usr/local/include/pango-1.0/pango/pangocairo.h:25: In file included from /usr/local/include/pango-1.0/pango/pango.h:25: In file included from /usr/local/include/pango-1.0/pango/pango-attributes.h:25: In file included from /usr/local/include/pango-1.0/pango/pango-font.h:25: /usr/local/include/pango-1.0/pango/pango-coverage.h:28:10: fatal error: 'hb.h' file not found #include <hb.h> ^~~~~~ 1 error generated.

which means that the dependency between pango and harfbuzz is not being handled properly in CMake.

Someone with a bigger stake in the CMake build should take a turn at working on this now. Also color_lib needs to be installed (populated into the build tree) in build/lib/common which I assume is just an oversight in some config file somewhere.

Stephen

Great spot on -fno-common, looking at the clang docs this makes complete sense:

-fcommon, -fno-common¶ This flag specifies that variables without initializers get common linkage. It can be disabled with -fno-common.

With this flag added, the cgraph library builds and should hopefully be usable by my application. I shall test that further tomorrow. (I don't build or use the UI components of graphviz)

Also color_lib needs to be installed (populated into the build tree) in build/lib/common which I assume is just an oversight in some config file somewhere.

color_lib is a generated file.

Its included prebuilt in the portable sources because it requires awk to build it.

OK, I also can get the full CMake graphviz build to run, if I

manually copy lib/common/color_lib to build/lib/common/color_lib (why doesn't graphviz/cmake/generate_color_lib.cmake.in do this, or is this not being invoked?)
manually add -fno-common to C_FLAGS in build/lib/common/CMakeFiles/common.dir/flags.make wherever that really comes from.
manually add -I/usr/local/Cellar/harfbuzz/2.6.2/include/harfbuzz to C_INCLUDES in build/plugin/pango/CMakeFiles/gvplugin_pango.dir/flags.make wherever that really comes from (the value should of course come from running pkg-config --cflags harfbuzz).

Please can someone create patches for the CMake configuration to do the above?

I'm no CMake expert, but I think to fix nr 2 you would need to add:

if (APPLE)
   target_compile_options(common PRIVATE "-fno-common")
endif(APPLE)

at the bottom of lib/common/CMakeLists.txt. This should then work for all generators (I typically use Ninja rather than Make)

I tested the CMake change above, and can confirm that I can now build cgraph library and the layout plugins that my application requires. Thank you @truenorth and @ellson for your help.

Please can an expert comment on how to incorporate the proposed patch to lib/common/CMakeLists.txt, which unfortunately "does not work" in the sense that it makes CMake complain,

CMake Error at lib/common/CMakeLists.txt:39 (target_compile_options):
  Cannot specify compile options for target "common" which is not built by
  this project.

I don't think we yet have a resolution to this, but just to document some oddities I found in my travels:

cmake/generate_color_lib.cmake.in generates some intermediate files that seem unused
numerous files under cmake/ misspell "uint64_t" as "uint_64_t" when doing string replacement, so I suspect these are accidentally no-ops

I have experienced a similar undefined symbol issue on an older OSX machine (late-2009 Mac Mini, 10.11.6 El Capitan with Xcode 8.2.1/clang 800.0.42.1, CMake 3.17.3). The attached logs are based on the current graphviz source and dependencies supplied by Fink, though I've seen similar behavior with dependencies provided by brew (FWIW, brew doesn't support a platform this old and couldn't successfully build pangocairo but CMake found the minimal graphviz dependencies regardless).

I'm putting together a small MR for changes I made to get CMake to find pangocairo under Fink (glibconfig.h actually) but the only change made for the build using brew dependencies was the addition of set(CMAKE_MACOSX_RPATH TRUE) to the root CMakeLists.txt and even that should have only affected the installation paths, not compilation.

It's old and small and slow but I finally have an OSX machine for testing CMake builds. I've made separate progress building installers and I'd like to take a stab at getting CMake completely working on OSX. Guidance on where to look to sort out this library/symbol issue would be appreciated -- thanks!

cmake_configure.log

cmake_build.log

Note that applying the target_compile_options(common PRIVATE "-fno-common") change above immediately after add_library(common STATIC ...) seemed to fix the undefined symbols issue. Placing it before that line caused the Cannot specify compile options for target "common" error that @truenorth was seeing.

FWIW, this may also resolve #1456 (closed).

One further question: is this a general issue with clang or with OSX? I'm curious whether if(APPLE)... is adequate or if we should use a more specific test.

FWIW, this may also resolve #1456 (closed).

Yes, I think you're right, it may resolve the original issue there. Good spot.

One further question: is this a general issue with clang or with OSX? I'm curious whether if(APPLE)... is adequate or if we should use a more specific test.

I wondered about this too and I'm not sure. The behavior that this flag is controlling actually changed its default between GCC 9 and GCC 10 (old docs and new docs, for reference). So I wondered if this was also broken with GCC. However, experimenting on Linux I could not show this: moving back and forth between GCC 9 and 10 had no effect; the CMake build continued working.

The Clang docs don't say what is the default for this option. And additionally it's possible that the errors are caused by differences in the linkers (LLD vs GNU LD), not the compilers.

Moreover, I don't think -fno-common is a fix. It's a work around. Somewhere we are incorrectly marking a variable static or incorrectly not marking it static. Or incorrectly defining multiple copies of the same exported variable. If you can reproduce this failure, I would start looking at any definitions of the variables it's complaining about and see if you can notice discrepancies.

Thank you for working on this. I’m glad there is life for old Apple machines kept safely in the back of the closet.

I added if(APPLE) at some point because someone told me to, and it worked. I guess this held up for a while in these days where x86_64 is universal. (Though someone somewhere must be building for iOS with Xcode?) With ARM coming I’m sure there will be another big mess.

Wish I could give more light than confusion.

Does CMake's GenerateExportHeader module help any? I can't say that I understand the finer details of the shared/static symbol visibility issue but this looks like it might be relevant.

I backed out the -fno-common change to reproduce the error and extracted 103 symbols that are listed as undefined; see also undefined_global_symbols_osx.txt:

CL_type			E_tailclip		N_fontsize
CmdName			E_taillabel		N_gradientangle
Concentrate		E_visitedfillcolor	N_group
E_activefillcolor	E_visitedpencolor	N_height
E_activepencolor	E_weight		N_imagepos
E_arrowhead		E_xlabel		N_imagescale
E_arrowsz		EdgeLabelsDone		N_label
E_arrowtail		G_activefillcolor	N_layer
E_color			G_activepencolor	N_margin
E_comment		G_deletedfillcolor	N_nojustify
E_constr		G_deletedpencolor	N_ordering
E_decorate		G_gradientangle		N_orientation
E_deletedfillcolor	G_margin		N_penwidth
E_deletedpencolor	G_ordering		N_peripheries
E_dir			G_penwidth		N_selectedfillcolor
E_fillcolor		G_peripheries		N_selectedpencolor
E_fontcolor		G_visitedfillcolor	N_shape
E_fontname		G_visitedpencolor	N_showboxes
E_fontsize		GvExitOnUsage		N_sides
E_headclip		Gvfilepath		N_skew
E_headlabel		Gvimagepath		N_style
E_label			HTTPServerEnVar		N_vertices
E_label_float		Initial_dist		N_visitedfillcolor
E_labelangle		Lib			N_visitedpencolor
E_labeldistance		N_activefillcolor	N_width
E_labelfontcolor	N_activepencolor	N_xlabel
E_labelfontname		N_color			N_z
E_labelfontsize		N_comment		PSinputscale
E_layer			N_deletedfillcolor	Reduce
E_minlen		N_deletedpencolor	State
E_penwidth		N_distortion		Verbose
E_selectedfillcolor	N_fillcolor		Y_invert
E_selectedpencolor	N_fixed			graphviz_errors
E_showboxes		N_fontcolor
E_style			N_fontname

All of these symbols are defined EXTERN in lib/common/globals.h, though not all of the symbols in that file are listed as unresolved when building. Of the symbols in lib/common/globals.h that aren't in this list, only 4 are referred to by files in lib/common: MemTest, Nop, Show_cnt, and the structure fdp_parms.

How mysterious... Thank you for investigating. At this point, I'm inclined to say commit the fno-common change to get macOS builds working, but leave this issue open to continue investigating the root cause. This is basically what Stephen suggested previously, so maybe we should have gone ahead and done this but I didn't have a Mac on which to test.

mentioned in merge request apthorpe/graphviz!1

mentioned in merge request apthorpe/graphviz!2

Just added two MRs: the first adds -fno-common to address the original issue, the second fixes the unint_64_t typos found by @smattr.

Note that the type typo fix causes new warnings to be thrown: type mismatch, expected long but got uint64_t in lib/fdpgen/layout.c, lines 330, 333, lib/cgraph/id.c, line 160. I don't know the backstory on why the types are being replaced in the .cmake.in files so I marked that MR as a WIP (I'm of the ignore warnings at your peril mindset...)

I think you've posted the MRs with the target repository as your own Graphviz fork, rather than here. Are you able to edit these to point them at this repository?

To understand the uint64_t find-and-replace, we might have to do some Git archaeology. I don't know what the intention of this step is, but I suspect it's for compatibility with some old version of Bison/Flex (or maybe even Yacc/Lex). If anyone else knows the origins of this build step, please comment.

mentioned in merge request !1464 (merged)

mentioned in merge request !1465 (closed)

mentioned in merge request !1522 (closed)

mentioned in issue #1802 (closed)

Is this issue still a problem? The -fno-common change was committed in 765d2ce8. That commit along with some other recent stuff (429718cb, 3ecf2f62, b4f998c1) fixes all macOS build problems I'm aware of. There is some other disruption coming down the pipe (#1785 (closed)) but I think Graphviz should build on macOS currently.

Thanks to your work, the Mac OSX cmake build seems OK now including cairopango and quartz (I hope)

closed

mentioned in merge request !3029 (closed)

CMake build does not work on MacOS

Designs

Child items ...

Activity