Comparing C/C++ unity build with regular build on a large codebase
Intro
From the beginning, C/C++ code has been compiled into separate small code objects and later linked into one executable or library. Initially, it was done due to hardware limitations as it couldn’t process all code simultaneously. These days it is done to increase compilation speed by trying to compile just a small portion of changed code after the initial full build.
For my own smaller C projects I have been using unity build and were doing full project rebuild. I do it just for convenience and it is easier to manage and build for my purposes. There are people “on the internet” who claim that even a larger C++ project could have been built using unity build without much increase in compilation times. Even though I am a bit skeptical that this could be done on a larger scale I was interested in measuring actual timings.
Initially, I was planning to generate 500k-1m lines of C/C++ code with a bunch of functions calling each other and spread across 1000-2000 files. I was hesitating because it would be a synthetic code and it felt a bit fishy. Then I decided to take an open-source project and try to build it as a single compilation unit. I chose Inkscape as I already use it weekly for my design works and had plans to dig into its source code later to make some experiments. It felt like an ideal candidate.
Lately, I have been trying to make estimates before doing measurements and for this project my initial expectations were to get 2x-3x speedup in single-core compilation of the whole project.
What is unity build?
Unity build (unified build) is a compilation method for C/C++ projects where multiple translation units are merged into one and compiled as a single block of code. The easiest way to do it is by using the preprocessor #include directive to include multiple C/C++ files. This gives us one file which is passed to the compilator and it will compile it in one go.
In regular builds, we include header files in unity builds we include C/C++ files directly. So for a regular build, we will have multiple intermediary .o files which later will be linked into an executable while in unity builds preprocessor merges all files and the compiler creates an executable directly.
Regular builds works something like this:
And unity builds look like this:
Setup
Initially, I just cloned inkscape’s code from https://gitlab.com/inkscape/inkscape, created a ‘build’ directory inside that repo and from within the ‘build’ directory run these commands.
Regular inkscape build with cmake
cmake .. # 4.724 sec
make # 1938.98s user 144.20s system 99% cpu 34:47.83 total
So our baseline is just a little shy of 35 minutes to compile a whole inkscape from scratch using regular cmake/make combination. Running ‘cloc src’ gives us 1317 C++ files with 453842 loc, 1287 header files with 93914 loc, and 76 C files with 51776 loc.
My initial attempt was to try to figure out compilation steps from reading CMake files and later Makefiles. After some reading, I realized that I don’t know enough kung fu to untangle all the compilation graphs without errors. An easier approach for me was to get a compilation log from a regular cmake then make compilation steps and just recreate it from in a bash script. I chose to go with a bash script because I wanted to explicitly list all the steps and try not to miss anything.
Building without cmake
To begin the process of preparing a unity build I decided to make a regular build but without cmake or make. For this, I used plain bash script and compiled each translation unit separately but taking log output from cmake build and building compiling one by one.
During regular inkscape builds it builds 2 dynamic libraries lib2geom.so, libinkscape_base.so, and 10 statically linked libraries liblivarot_LIB.a, libutil_LIB.a, libuemf_LIB.a, libcroco_LIB.a, libdepixelize_LIB.a, libavoid_LIB.a, libcola_LIB.a, libvpsc_LIB.a, libautotrace_LIB.a, libgc_LIB.a. libinkscape_base.so is a main library which contains most inkscape code. In the end we just compile inkscape-main.cpp and link it with all previously mentioned libraries and get a single binary.
Here is the script used to build it: build.sh.
It is a 2300-line script file with a bit of setup and then one by one build
translation units into libraries and then into single binary. If you want to try
it you will need to put this script into the root folder of inskcape project and
just run it with ./build.sh
This one is created on linux and won’t work on windows, it will require small changes to run Mac and you might need to add missing libraries. It will create __build folder, create libraries and then an executable named inkscape inside that __build folder. Also, it will copy some data files required for running the executable. In this script, I replicated build libraries used in regular cmake build and here also measured how much time it takes to build. Here are the results:
Single build.sh
Copy data files: 2.127 seconds
Compiling lib2geom.so : 53.187 seconds | 52 C++ files
Compiling liblivarot_LIB.a : 12.551 seconds | 17 C++ files
Compiling libutil_LIB.a : 16.832 seconds | 18 C++ files
Compiling libuemf_LIB.a : 1.370 seconds | 11 C files
Compiling libcroco_LIB.a : 1.615 seconds | 27 C files
Compiling libdepixelize_LIB.a : 2.103 seconds | 1 C++ file
Compiling libavoid_LIB.a : 11.147 seconds | 23 C++ files
Compiling libcola_LIB.a : 4.903 seconds | 10 C++ files
Compiling libvpsc_LIB.a : 2.256 seconds | 7 C++ files
Compiling libautotrace_LIB.a : 0.985 seconds | 19 C files
Compiling libgc_LIB.a : 0.498 seconds | 1 C++ file
compiling libinkscape_base.so : 1501.372 seconds | 877 C++ files
Compiling inkscape : 1.912 seconds | 1 C++ file
./build.sh 1508.85s user 98.71s system 99% cpu 26:52.91 total
Regular cmake build took 34 minutes and with new build.sh we built it in just 27 minutes. I am not sure what affected this 7-minute speed-up. It should not be a file cache because two different builds were done from separate folders and I cleared the file memory cache before building. (I hope that it is related to cmake create makefiles that are less efficient and not that I missed some files)
As a result we created 1043 .o files, 10 .a statically linked files, 2 .so dynamic libraries and one executable file. During script execution we did 1431 calls to g++, 61 calls to gcc, one explicit call to ld, 10 calls to ar for static linking.
Unity build
Now that we managed to build inkscape with just inkscape we can assume that it is more or less possible to move to unity build as we see all the steps involved. Initially, I wanted to take inkscape-main.cpp and with assistance from compiler errors change the source code to be built in one go. After some time I found that it was too cumbersome to go this way so I chose a different approach. I created all.cpp file which contained only #include directives which included all files from the previous script. It looks something like this:
#include "../__generated/inkscape-version.cpp"
#include "async/async.cpp"
#include "colors/cms/profile.cpp"
#include "colors/cms/system.cpp"
#include "colors/cms/transform.cpp"
#include "colors/color.cpp"
#include "colors/document-cms.cpp"
#include "colors/dragndrop.cpp"
...
...
...
Then with the help of the compiler, I did “Monkey patching” by changing the source code just enough to make it compile. I didn’t study inkscapes architecture as it was my first time working with this code base and I didn’t want to spend a lot of time in this step. I also didn’t do a full unity build by joining all files into one big unity build. This time I tried to keep each library as it is defined by inkscape as a separate compilation unit. This was supposed to help me speed up my experiment development. So as a result of this choice, we got 12 translation units, 11 libraries, and 1 final program unit. Each library has it’s own all_*.cpp file.
Unity build timing:
Copy data files: 2.340 seconds
Compile lib2geom.so : 6.298 seconds | Merged 52 C++ files
Compile liblivarot_LIB.a : 3.072 seconds | Merged 17 C++ files
Compile libutil_LIB.a : 5.323 seconds | Merged 18 C++ files
Compile libuemf_LIB.a : 1.057 seconds | Merged 11 C files
Compile libcroco_LIB.a : 0.636 seconds | Merged 27 C files
Compile libdepixelize_LIB.a : 1.948 seconds | Merged 1 C++ file
Compile libavoid_LIB.a : 4.106 seconds | Merged 23 C++ files
Compile libcola_LIB.a : 1.693 seconds | Merged 10 C++ files
Compile libvpsc_LIB.a : 0.849 seconds | Merged 7 C++ files
Compile libautotrace_LIB.a : 0.278 seconds | Merged 19 C files
Compile libgc_LIB.a : 0.467 seconds | Merged 1 C++ file
Compile inkscape binary : 137.220 seconds | Merged 877 C++ files
~/tmp/inkscape_compile/inkscape
./build.sh 157.19s user 5.36s system 98% cpu 2:45.35 total
Total build time took about 3 minutes. Comparing it with the first cmake build it is almost a 12x increase in single-core compilation when compared with our custom build.sh build it is 9x speed up in build time. As a result we get 10 .o files, 10 .a static libraries, 1 .so dynamic library, and 1 executable. Maybe in the future, I should try to make just one compilation unit but for this case, it was not a deal breaker.
Main inkscape code is within 877 C++ files. It is built with simple: c++ $CFLAGS_MAIN $SRC/all.cpp -o inkscape $LDFLAGS. So we set some Compilation and linker flags and just compile all.cpp file. How big is it? The file itself if not very big 878 lines but if we pass -E flag to the compiler to do just preprocessing we will get a file that is a file that is over 1 million lines with just above 760k actual code lines. This is a compbination of all those 877 C++ files plus all project and external library headers.
You can look at the changed source code for unity build here: https://github.com/hereket/inkscape-unity-build
Final thoughts
Initially, I estimated 2-3x speed up for unity builds but in the end we got 9-12x speedup. Of course, this is still not enough and doing regular builds is still faster for normal development because we can use ccache and utilize all cores for the build by using make. But it was a fun little experiment and it is a good base for me to do some other experiments.
C files were very fast to build and some C++ features increased compilation speeds quite a bit. I assumed it is templates that inkscape actively uses in its codebase. To see if C files are compiled faster I preprocessed compilation units into separate files and did some LOC (lines of code) measurements. This LOC is not the number of C++ lines but rather the number of lines after merging all header/source files into one (including external library headers).
---------------------------------------------------------------------------
File Name | Time | Size | Type | LOC | Lines per second
---------------------------------------------------------------------------
lib2geom.so | 6.298 sec | 5.7M | C++ | 148_713 | 23613 l/s
all_livarot.o.cpp | 3.072 sec | 5.3M | C++ | 141_768 | 46148 l/s
all_utils.o.cpp | 5.323 sec | 12M | C++ | 281_101 | 52809 l/s
all_uemf.o.c | 1.057 sec | 1.5M | C | 35_959 | 34020 l/s
all_croco.o.c | 0.636 sec | 2M | C | 40_917 | 64335 l/s
depixelize.o.cpp | 1.948 sec | 5M | C++ | 132_714 | 68128 l/s
all_avoid.o.cpp | 4.106 sec | 3.1M | C++ | 84_444 | 20566 l/s
all_cola.o.cpp | 1.693 sec | 2.3M | C++ | 58_815 | 34740 l/s
all_vpsc.o.cpp | 0.849 sec | 1.9M | C++ | 52_152 | 61428 l/s
all_autorace.o.c | 0.278 sec | 949K | C | 21_035 | 75665 l/s
gc.o.cpp | 0.467 sec | 2.8M | C++ | 78_719 | 168563 l/s
all.o.cpp | 137.220 sec | 30M | C++ | 759_613 | 5536 l/s
At first glance, it looks like that C is not compiled faster but that creates a lot more stuff to compile. From the table, it looks like the average is compiler compiles 59k lines per second. If we remove 168k l/s then the average speed is 48k l/s. If we remove 20k versions then the average speed is 54k lines per second for unity builds.
Addtionally I decided to measure speed based not on preprocessed output of source files but on actual lines code of the C/C++ files and their headers. For this I went to the folders of the correspoing libraries and run ‘cloc’ inside those folders. After I took a sum of real lines of C/C++ code and their headers. Then only issue here is that it was too much work to figure out all “our” header files required for inkscape main executable an just calculated lines of code only of comiled C/C++ files without header files. So with this we get lines of code per second relative to actually “our” written code without counting external library headers and generated extra code by the preprocessor (macro expansion and etc). Here is the table.
File Name | Time | Type | SourceLOC | Lines per second
---------------------------------------------------------------------------
2geom | 6.298 sec | C++ | 19_518 | 3099 l/s
livarot | 3.072 sec | C++ | 14_566 | 4742 l/s
utils | 5.323 sec | C++ | 6_601 | 1240 l/s
uemf | 1.057 sec | C | 29_418 | 27832 l/s
libcroco | 0.636 sec | C | 21_768 | 34226 l/s
libdepixelize | 1.948 sec | C++ | 3_109 | 1596 l/s
libavoid | 4.106 sec | C++ | 96_202 | 23430 l/s
libcola | 1.693 sec | C++ | 16_503 | 9748 l/s
libvpsc | 0.849 sec | C++ | 4_118 | 4850 l/s
autotrace | 0.278 sec | C | 5_227 | 18802 l/s
inkgc | 0.467 sec | C++ | 603 | 1291 l/s
inkscape | 137.220 sec | C++ | (277_253) | 2020 l/s
With this table we can see that C files are “faster” to compile mostly because there are just less extra stuff included and generated which leads to increased percived speed.
This is the initial rough back on the envelope calculation. I might add more data later if I get more free time to play around.
Extra: Additional comparisons
Since there were a confusion in initial post and some people thought “Unity build rocks let’s only it” and “Make is better. Run it with more threads” I decided to add a bit more measurements. I just want to remind that I was not trying to argue that unity builds are better or worse. My goal was to make a simple research into the topic on a real project and have some baseline numbers to work on later.
Bellow is measurements of compiling inkscape from scratch using different methods. Before each build I cleared file cache from memory.
cmake .. && make | 34 min 52 sec
cmake .. && make -j 12 | 5 min 48 sec
cmake .. -GNinja && ninja | 5 min 50 sec
build.sh NON unity (-O0) | 26 min 52 sec
build.sh NON unity (-O2) | 32 min 12 sec
build.sh NON unity (-O3) | 31 min 08 sec
build.sh unity (-O0) | 2 min 57 sec
build.sh unity (-O2) | 6 min 08 sec
build.sh unity (-O3) | 6 min 17 sec
And this is what what incremental build compares to full build. In the table below I measured how much time it take to rebuild project when I change 1 lines of code in “src/io/resource.cpp” in homedir_path() to return constant string. Also I did a test for a header file where I changed one line by adding one more element to Type enum. Here are the results:
Change one line in src/io/resource.cpp
ninja | 2.7 sec
make | 7.8 sec
make -j 12 | 5 sec
Change one line in src/io/resource.h
ninja | 43 sec
make | 3 minute 30 sec
make -j 12 | 46 sec
Regular unity build will still be 2+ minutes for full rebuild for inkscape no matter what file you change. Building using make/ninja is variable. As you can see from the table above it could be very fast or it could take half the amount required for a full unity build. I guess if I changed several c++ files and headers it could take close a to time of a full build. There are claims that a project of this size written in C style without template and other features should compile in under 10 seconds but I don’t have project that I can use to verify that.