How to build cv2 in debug non-optimized mode? #424

ApproximateIdentity · Dec 3, 2020

debian 10
gcc/g++ version 8.3.0 (Debian 8.3.0-6)
Python 3.9.0

I want to build the final cv2 shared library (e.g. cv2.cpython-39-x86_64-linux-gnu.so) with debug symbols and without optimizations. I have tried various strategies like editing the opencv/CMakeLists.txt as well as running a build with something like this:

export ENABLE_HEADLESS=1
export CMAKE_ARGS='-DCMAKE_BUILD_TYPE=DEBUG -DCMAKE_C_FLAGS_DEBUG="-g -O0" -DCMAKE_CXX_FLAGS_DEBUG="-g -O0"'
TMPDIR=$PWD/build pip wheel --no-clean --verbose .

No matter what I do, the final library seems to be built in Release mode and ignores my optimizations. How do I do this?

The text was updated successfully, but these errors were encountered:

ApproximateIdentity · Dec 3, 2020

Alright I came up with an extremely hacky solution that does work. Someone more used to cmake would probably laugh at this solution, but it does work (and is faster than my trying to fight a build system I don't understand). I'll leave it here for the record so that I can move on, but I'm sure someone here can pop in and explain the right way to do this.

I ran the build as follows choosing to keep all temporary build files:

build.sh

export ENABLE_HEADLESS=1
export VERBOSE=1

mkdir build
TMPDIR=$PWD/build pip wheel --no-clean --verbose .

Then I swallowed up the output into build.log:

$ bash my_build.sh 2>&1 | tee build.log

Then I extracted and transformed all commands involving the build:

hack_build.py

#!/usr/bin/env python3

import sys

for line in sys.stdin:
    if (line.startswith('  cd') or
        line.startswith('  /usr/bin/ar') or
        line.startswith('  /usr/bin/c++') or
        line.startswith('  /usr/bin/ranlib')):
        newline = line.strip(' ').replace('-O3', '-O0 -g')
        sys.stdout.write(newline)

$ python3 hack_build.py < build.log > hacked_build.sh

Finally executing $ bash hacked_build.sh re-built the shared library without any optimizations. On my computer it produced the file:

./build/pip-req-build-ro720y4z/_skbuild/linux-x86_64-3.9/cmake-build/lib/python3/cv2.cpython-39-x86_64-linux-gnu.so

After copying that file into the directory path/to/lib/python3.9/site-packages/cv2/ I replaced the other version with mine and it works.

skvark · Dec 3, 2020

I believe scikit-build (CMake wrapper library) overwrites the -DCMAKE_BUILD_TYPE=Debug flag here: https://github.com/scikit-build/scikit-build/blob/c0741f9a1de9dd97e0a1e6f0b3b02fcd76221418/skbuild/setuptools_wrap.py#L105.

Because of the pip wheel command, the parameters to scikit-built cannot be passed directly as command line parameters as documented here (due to this there is the CMAKE_ARGS variable in this repository): https://scikit-build.readthedocs.io/en/latest/usage.html?highlight=debug#usage-scikit-build-options

However, you could try running:

pip wheel . --verbose --build-option="--build-type=Debug" (or --global-option, I did not test either one)

but then pip builds the whole dependency chain from source which is slow and not needed: pypa/pip#2677

pip wheel command docs: https://pip.pypa.io/en/stable/reference/pip_wheel/#customising-the-build

OpenCV build information: https://docs.opencv.org/master/db/d05/tutorial_config_reference.html

ApproximateIdentity · Dec 4, 2020

Thanks for the help, but when I tried pip wheel . --verbose --build-option="--build-type=Debug" I got the following error:

37361 Building wheels for collected packages: opencv-python-headless, numpy
37362   Created temporary directory: /home/user/src/opencv-python/build/pip-wheel-nljd2_ld
37363   ERROR: Cannot build wheel for opencv-python-headless using PEP 517 when --build-option is present
37364   Created temporary directory: /home/user/src/opencv-python/build/pip-wheel-mj3bd_7i
37365   ERROR: Cannot build wheel for numpy using PEP 517 when --build-option is present
37366 Failed to build opencv-python-headless numpy
37367 ERROR: Failed to build one or more wheels
37368 Exception information:
37369 Traceback (most recent call last):
37370   File "/home/user/folder/repos/vision/sandbox/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 210, in _main
37371     status = self.run(options, args)
37372   File "/home/user/folder/repos/vision/sandbox/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
37373     return func(self, options, args)
37374   File "/home/user/folder/repos/vision/sandbox/lib/python3.9/site-packages/pip/_internal/commands/wheel.py", line 184, in run
37375     raise CommandError(
37376 pip._internal.exceptions.CommandError: Failed to build one or more wheels

Given that I only want to do this for investigative/understanding work, my hack seems like the easiest approach for the moment since it already works. I'm guessing there aren't that many people who find themselves in my boat.

But thanks for the response!

skvark · Dec 5, 2020

There's one additional way of doing this via the python setup.py bdist_wheel command. I tested this on Windows and it works:

python setup.py bdist_wheel --build-type=Debug

Please note that you must install at least scikit-build and numpy before that command. Might need also something else, since the tooling does not get installed automatically via pyproject.toml. Due to that, I recommend running the build in a virtual environment which has all needed dependencies installed.

ApproximateIdentity · Dec 9, 2020

I just checked and this worked for me on Linux too. Might be worth adding something in the main readme about this (which I can try to remember to do), but other than that seems good. Thanks!

black13 · Jan 14, 2021

C:\temp\opencv-python>python setup.py bdist_wheel --build-type=Debug
C:\Program Files\Python39\lib\site-packages\setuptools\distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
warnings.warn(
[1/2] Linking CXX shared module lib\python3\cv2.cp39-win_amd64.pyd
FAILED: lib/python3/cv2.cp39-win_amd64.pyd
cmd.exe /C "cd . && "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -E vs_link_dll --intdir=modules\python3\CMakeFiles\opencv_python3.dir --rc=C:\PROGRA~~2\WI3CF2~~1\10\bin\100183~~1.0\x64\rc.exe --mt=C:\PROGRA~~2\WI3CF2~~1\10\bin\100183~~1.0\x64\mt.exe --manifests -- C:\PROGRA~~2\MICROS~~2\2019\COMMUN~~1\VC\Tools\MSVC\1428~~1.293\bin\Hostx64\x64\link.exe /nologo modules\python3\CMakeFiles\opencv_python3.dir_\src2\cv2.cpp.obj /out:lib\python3\cv2.cp39-win_amd64.pyd /implib:lib\opencv_python3.lib /pdb:lib\python3\cv2.pdb /dll /version:0.0 /machine:x64 /debug /INCREMENTAL /NODEFAULTLIB:atlthunk.lib /NODEFAULTLIB:atlsd.lib /DEBUG "C:\Program Files\Python39\libs\python39.lib" lib\opencv_core451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_ml451d.lib lib\opencv_photo451d.lib lib\opencv_dnn451d.lib lib\opencv_features2d451d.lib lib\opencv_imgcodecs451d.lib lib\opencv_videoio451d.lib lib\opencv_calib3d451d.lib lib\opencv_highgui451d.lib lib\opencv_objdetect451d.lib lib\opencv_stitching451d.lib lib\opencv_video451d.lib lib\opencv_gapi451d.lib lib\opencv_videoio451d.lib lib\opencv_imgcodecs451d.lib 3rdparty\lib\libjpeg-turbod.lib 3rdparty\lib\libwebpd.lib 3rdparty\lib\libpngd.lib 3rdparty\lib\libtiffd.lib 3rdparty\lib\libopenjp2d.lib 3rdparty\lib\IlmImfd.lib comctl32.lib gdi32.lib ole32.lib setupapi.lib 3rdparty\lib\quircd.lib lib\opencv_video451d.lib lib\opencv_dnn451d.lib 3rdparty\lib\libprotobufd.lib lib\opencv_calib3d451d.lib lib\opencv_features2d451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_core451d.lib 3rdparty\lib\zlibd.lib 3rdparty\lib\ittnotifyd.lib 3rdparty\lib\ippiwd.lib 3rdparty\ippicv\ippicv_win\icv\lib\intel64\ippicvmt.lib lib\ade.lib wsock32.lib ws2_32.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK Pass 1: command "C:\PROGRA~~2\MICROS~~2\2019\COMMUN~~1\VC\Tools\MSVC\1428~~1.293\bin\Hostx64\x64\link.exe /nologo modules\python3\CMakeFiles\opencv_python3.dir_\src2\cv2.cpp.obj /out:lib\python3\cv2.cp39-win_amd64.pyd /implib:lib\opencv_python3.lib /pdb:lib\python3\cv2.pdb /dll /version:0.0 /machine:x64 /debug /INCREMENTAL /NODEFAULTLIB:atlthunk.lib /NODEFAULTLIB:atlsd.lib /DEBUG C:\Program Files\Python39\libs\python39.lib lib\opencv_core451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_ml451d.lib lib\opencv_photo451d.lib lib\opencv_dnn451d.lib lib\opencv_features2d451d.lib lib\opencv_imgcodecs451d.lib lib\opencv_videoio451d.lib lib\opencv_calib3d451d.lib lib\opencv_highgui451d.lib lib\opencv_objdetect451d.lib lib\opencv_stitching451d.lib lib\opencv_video451d.lib lib\opencv_gapi451d.lib lib\opencv_videoio451d.lib lib\opencv_imgcodecs451d.lib 3rdparty\lib\libjpeg-turbod.lib 3rdparty\lib\libwebpd.lib 3rdparty\lib\libpngd.lib 3rdparty\lib\libtiffd.lib 3rdparty\lib\libopenjp2d.lib 3rdparty\lib\IlmImfd.lib comctl32.lib gdi32.lib ole32.lib setupapi.lib 3rdparty\lib\quircd.lib lib\opencv_video451d.lib lib\opencv_dnn451d.lib 3rdparty\lib\libprotobufd.lib lib\opencv_calib3d451d.lib lib\opencv_features2d451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_core451d.lib 3rdparty\lib\zlibd.lib 3rdparty\lib\ittnotifyd.lib 3rdparty\lib\ippiwd.lib 3rdparty\ippicv\ippicv_win\icv\lib\intel64\ippicvmt.lib lib\ade.lib wsock32.lib ws2_32.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:modules\python3\CMakeFiles\opencv_python3.dir/intermediate.manifest modules\python3\CMakeFiles\opencv_python3.dir/manifest.res" failed (exit code 1104) with the following output:
LINK : fatal error LNK1104: cannot open file 'python39_d.lib'
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "C:\Users\jjosburn\AppData\Roaming\Python\Python39\site-packages\skbuild\setuptools_wrap.py", line 589, in setup
cmkr.make(make_args, env=env)
File "C:\Users\jjosburn\AppData\Roaming\Python\Python39\site-packages\skbuild\cmaker.py", line 496, in make
raise SKBuildError(

An error occurred while building with CMake.
Command:
cmake --build . --target install --config Debug --
Source directory:
C:\temp\opencv-python
Working directory:
C:\temp\opencv-python_skbuild\win-amd64-3.9\cmake-build
Please see CMake's output for more information.

letmaik · Mar 18, 2021

Having the same issue as @black13. I just need the debug output from https://github.com/opencv/opencv/blob/1363496c1106606684d40447f5d1149b2c66a9f8/modules/videoio/src/cap_dshow.cpp#L302 but looks like to get this I need to get a debug build of python including debug builds of opencv-python dependencies. Is this true?

ApproximateIdentity · Mar 18, 2021

You should really only need a debug build of the library you're interested in, but it can be a bit tricky. The method I used in my post should work for Linux, but if you're on Windows like black13, I can't really help you with the details.

skvark · Mar 18, 2021

@letmaik @black13 The error you are seeing is caused by the missing Python debug binaries. We can see clearly from the output that the linker is trying to link against the Python debug binary (note the _d):

LINK : fatal error LNK1104: cannot open file 'python39_d.lib'

See for example this issue: opencv/opencv#7395

You must install the debug binaries. This can be done via the official Python installer as documented here (customize installation): https://docs.python.org/3/using/windows.html#installation-steps

Selecting “Customize installation” will allow you to select the features to install, the installation location and other options or post-install actions. To install debugging symbols or binaries, you will need to use this option.

Edit: You can run the installer also without UI if needed: https://docs.python.org/3/using/windows.html#installing-without-ui

letmaik · Mar 18, 2021

@skvark Thanks for the pointer. I tried to do that here by adding Include_debug=1: https://ci.appveyor.com/project/letmaik/opencv-python/builds/38298354 But I'm getting the same error (see end of log). ~~It seems _d.dll files are installed but no .lib files.~~
EDIT: The .lib files are correctly installed, e.g. python39_d.lib. I think the issue is in the CMake config of opencv somehow: -DPYTHON3_LIBRARY=C:/mypython/libs/python39.lib this should be the debug library. I'm not sure if this is really the root issue however.

skvark · Mar 19, 2021

Scikit-build picks up the wrong path, you can change it here: https://github.com/opencv/opencv-python/blob/master/setup.py#L41

letmaik · Mar 27, 2021

Scikit-build picks up the wrong path, you can change it here: https://github.com/opencv/opencv-python/blob/master/setup.py#L41

I hacked in the right path (upstream issue: scikit-build/scikit-build#533) and got a bit closer. It now fails during copying of the files from the CMake install folder:

Traceback (most recent call last):
  File "C:\projects\opencv-python\setup.py", line 458, in <module>
    main()
  File "C:\projects\opencv-python\setup.py", line 218, in main
    skbuild.setup(
  File "C:\mypython\lib\site-packages\skbuild\setuptools_wrap.py", line 621, in setup
    _classify_installed_files(cmake_manifest, package_data, package_prefixes,
  File "C:\projects\opencv-python\setup.py", line 371, in _classify_installed_files_override
    raise Exception("Not found: '%s'" % relpath_re)
Exception: Not found: 'python/cv2[^/]*\.cp39\-win_amd64\.pyd'

Here's the log: https://ci.appveyor.com/project/letmaik/opencv-python/builds/38432602

What's weird is that the "-- Installing:" line from CMake for the .pyd is missing, compared to a regular build like https://ci.appveyor.com/project/skvark/opencv-python/builds/36078624/job/vd96v0pmj7ai9bno?fullLog=true.

EDIT: Looks like it's related to this code: https://github.com/opencv/opencv/blob/fc1a15626226609babd128e043cf7c4e32f567ca/modules/python/common.cmake#L125-L129

ghost · Apr 7, 2021

Scikit-build picks up the wrong path, you can change it here: https://github.com/opencv/opencv-python/blob/master/setup.py#L41

I hacked in the right path (upstream issue: scikit-build/scikit-build#533) and got a bit closer. It now fails during copying of the files from the CMake install folder:
Traceback (most recent call last):
  File "C:\projects\opencv-python\setup.py", line 458, in <module>
    main()
  File "C:\projects\opencv-python\setup.py", line 218, in main
    skbuild.setup(
  File "C:\mypython\lib\site-packages\skbuild\setuptools_wrap.py", line 621, in setup
    _classify_installed_files(cmake_manifest, package_data, package_prefixes,
  File "C:\projects\opencv-python\setup.py", line 371, in _classify_installed_files_override
    raise Exception("Not found: '%s'" % relpath_re)
Exception: Not found: 'python/cv2[^/]*\.cp39\-win_amd64\.pyd'
Here's the log: https://ci.appveyor.com/project/letmaik/opencv-python/builds/38432602

What's weird is that the "-- Installing:" line from CMake for the .pyd is missing, compared to a regular build like https://ci.appveyor.com/project/skvark/opencv-python/builds/36078624/job/vd96v0pmj7ai9bno?fullLog=true.

EDIT: Looks like it's related to this code: https://github.com/opencv/opencv/blob/fc1a15626226609babd128e043cf7c4e32f567ca/modules/python/common.cmake#L125-L129

I faced the exact same issue when building wheel on linux.
Exception: Not found: 'python/cv2[^/]*.cpython-36m-x86_64-linux-gnu.so'
did you overcome this eventually?

DynConcepts · May 8, 2021

For those who just want a "python39_d.lib" (DLL) what is the option? [No desire to actually BUILD Python]

ApproximateIdentity mentioned this issue Dec 11, 2020

Add docs for debug build #428

Merged

skvark closed this Dec 21, 2020

opencv / opencv-python Public

How to build cv2 in debug non-optimized mode? #424

How to build cv2 in debug non-optimized mode? #424

ApproximateIdentity commented Dec 3, 2020

ApproximateIdentity commented Dec 3, 2020

skvark commented Dec 3, 2020

ApproximateIdentity commented Dec 4, 2020

skvark commented Dec 5, 2020

ApproximateIdentity commented Dec 9, 2020

black13 commented Jan 14, 2021

letmaik commented Mar 18, 2021

ApproximateIdentity commented Mar 18, 2021

skvark commented Mar 18, 2021 •

edited

letmaik commented Mar 18, 2021 •

edited

skvark commented Mar 19, 2021

letmaik commented Mar 27, 2021 •

edited

ghost commented Apr 7, 2021

DynConcepts commented May 8, 2021

Oct	NOV	Dec
	02
2020	2021	2022

opencv / opencv-python Public

How to build cv2 in debug non-optimized mode? #424

How to build cv2 in debug non-optimized mode? #424

Comments

ApproximateIdentity commented Dec 3, 2020

ApproximateIdentity commented Dec 3, 2020

skvark commented Dec 3, 2020

ApproximateIdentity commented Dec 4, 2020

skvark commented Dec 5, 2020

ApproximateIdentity commented Dec 9, 2020

black13 commented Jan 14, 2021

letmaik commented Mar 18, 2021

ApproximateIdentity commented Mar 18, 2021

skvark commented Mar 18, 2021 • edited

letmaik commented Mar 18, 2021 • edited

skvark commented Mar 19, 2021

letmaik commented Mar 27, 2021 • edited

ghost commented Apr 7, 2021

DynConcepts commented May 8, 2021

skvark commented Mar 18, 2021 •

edited

letmaik commented Mar 18, 2021 •

edited

letmaik commented Mar 27, 2021 •

edited