diff --git a/docs/build-python-wheel.md b/docs/build-python-wheel.md new file mode 100644 index 0000000..719cba3 --- /dev/null +++ b/docs/build-python-wheel.md @@ -0,0 +1,335 @@ +# Building the Python wheel + +The Python package is a compiled extension, so its wheel is built on the platform +it runs on. This page covers a native build: the build machine is the platform of +the wheel. + +A build produces one wheel per platform and architecture, not one per Python +version: the extension uses the stable ABI (`cp310-abi3`) and runs on every +CPython from 3.10 onwards. + +## Requirements + +- Python 3.10 or later, with `venv`. +- The Rust toolchain pinned in [rust-toolchain.toml](../rust-toolchain.toml); + rustup selects it inside the checkout. +- A C/C++ compiler, CMake, Clang, and libclang. The `opencv` crate generates its + bindings by running the `clang` binary during the build. +- OpenCV 4.11 or newer with its development headers. The published wheels bundle + OpenCV 4.12.0, which the steps below build. +- `maturin` for the build, `delocate` on macOS, `auditwheel` on Linux, or + `delvewheel` on Windows to bundle the native libraries, and `twine` to check the + metadata. + +For Debian 12 (bookworm), `libopencv-dev` is older than 4.11, so build the +OpenCV of the wheel as described below. The compiler, CMake, and Clang come from: + +```sh +sudo apt update +sudo apt install -y build-essential pkg-config cmake clang libclang-dev +``` + +On macOS, the Command Line Tools provide Clang and libclang. When the build does +not find libclang, point `LIBCLANG_PATH` at the directory that holds +`libclang.dylib`. + +On Windows, the build needs Visual Studio Build Tools with the C++ workload, LLVM +for the `clang` binary and `libclang.dll`, and the `x86_64-pc-windows-msvc` target +of the Rust toolchain: + +```sh +rustup target add x86_64-pc-windows-msvc +``` + +The commands of this page use the shell of Git for Windows. In PowerShell, +`export NAME=value` becomes `$env:NAME = 'value'`. + +Create a virtual environment with the build tools: + +```sh +python3 -m venv .venv +.venv/bin/pip install 'maturin>=1.9,<2' 'pytest>=8,<10' twine +.venv/bin/pip install delocate # macOS +.venv/bin/pip install auditwheel # Linux +.venv/bin/pip install delvewheel # Windows +source .venv/bin/activate +``` + +## Wheel size and the bundled OpenCV + +The extension links OpenCV dynamically. A wheel that is not repaired therefore +references the OpenCV of the build machine, and it runs only on a machine with +that OpenCV. `delocate`, `auditwheel`, and `delvewheel` copy the referenced +libraries into the wheel and rewrite the references, which makes the wheel +self-contained. + +The libraries of the build machine become part of the wheel, so the size of the +wheel is decided by the OpenCV that the build links: + +| OpenCV of the build | Bundled libraries | Wheel (macOS arm64) | Wheel (manylinux x86_64) | +| --- | --- | --- | --- | +| Minimal build from source | 3 | 8.3 MB | 11.4 MB | +| Distribution or package manager build | many | 86 MB | not measured | + +A distribution build of OpenCV enables modules and backends that this project does +not use, and each of them links libraries of its own, which the wheel then has to +carry. The next section builds an OpenCV with the three modules of the extension, +without the backends that pull in further libraries. + +A third option is to publish an unrepaired wheel. It is the smallest one, and it +is only usable on machines with a matching OpenCV, so it does not suit a package +index. + +## Build a minimal OpenCV + +The image preprocessing of the extension decodes PNG, JPEG, and WebP, resizes the +image, and encodes it as WebP through the `core`, `imgproc`, and `imgcodecs` +modules. OpenCV compiles its codecs into `imgcodecs` (`BUILD_ZLIB`, `BUILD_JPEG`, +`BUILD_PNG`, `BUILD_WEBP`), so the install holds three shared libraries and no +further codec library. + +OpenCL, TBB, OpenEXR, TIFF, FFmpeg, GTK, LAPACK, OpenJPEG, Jasper, the Python +bindings, and the Java bindings are left out: each of them links libraries of its +own. + +```sh +git clone --depth 1 --branch 4.12.0 https://github.com/opencv/opencv.git +cmake -S opencv -B opencv/build \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX="$PWD/target/opencv-min" \ + -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0 \ + -DBUILD_SHARED_LIBS=ON \ + -DBUILD_LIST=core,imgproc,imgcodecs \ + -DOPENCV_GENERATE_PKGCONFIG=ON \ + -DBUILD_ZLIB=ON -DBUILD_JPEG=ON -DBUILD_PNG=ON -DBUILD_WEBP=ON \ + -DWITH_JPEG=ON -DWITH_PNG=ON -DWITH_WEBP=ON \ + -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DBUILD_EXAMPLES=OFF \ + -DBUILD_opencv_apps=OFF -DBUILD_DOCS=OFF -DBUILD_JAVA=OFF \ + -DBUILD_opencv_python2=OFF -DBUILD_opencv_python3=OFF \ + -DBUILD_PROTOBUF=OFF \ + -DWITH_ADE=OFF -DWITH_EIGEN=OFF -DWITH_FFMPEG=OFF -DWITH_GSTREAMER=OFF \ + -DWITH_GTK=OFF -DWITH_IPP=OFF -DWITH_ITT=OFF -DWITH_JASPER=OFF \ + -DWITH_LAPACK=OFF -DWITH_OPENCL=OFF -DWITH_OPENCLAMDBLAS=OFF \ + -DWITH_OPENCLAMDFFT=OFF -DWITH_OPENEXR=OFF -DWITH_OPENJPEG=OFF \ + -DWITH_OPENMP=OFF -DWITH_QUIRC=OFF -DWITH_TBB=OFF -DWITH_TIFF=OFF \ + -DWITH_V4L=OFF -DWITH_1394=OFF +cmake --build opencv/build --parallel +cmake --install opencv/build +``` + +Two parameters of the build decide the platform tag of the wheel: + +- `CMAKE_OSX_DEPLOYMENT_TARGET` on macOS. Keep it at the oldest release the wheel + should support; 11.0 produces the tag `macosx_11_0_arm64` or + `macosx_11_0_x86_64`. +- The C library of the Linux build environment on Linux. A wheel built on a + current distribution is tagged `linux_x86_64`, which a package index rejects. + Build inside a manylinux image of the architecture of the wheel to obtain a + `manylinux_*` tag. The image carries CMake and a compiler, and the commands above + run in it unchanged: + + ```sh + docker run --rm -it -v "$PWD:/work" -w /work quay.io/pypa/manylinux_2_28_x86_64 bash + ``` + +The build installs into `target/opencv-min`, which the next steps read. On a +machine with many cores it takes about a minute. + +## Build the wheel + +The `opencv` crate reads `OPENCV_INCLUDE_PATHS`, `OPENCV_LINK_PATHS`, and +`OPENCV_LINK_LIBS` before it probes pkg-config, so an OpenCV installed elsewhere +on the build machine cannot take over: + +```sh +cd deepseek-recipe-python +export OPENCV_INCLUDE_PATHS="$PWD/../target/opencv-min/include/opencv4" +export OPENCV_LINK_PATHS="$PWD/../target/opencv-min/lib" +export OPENCV_LINK_LIBS=opencv_core,opencv_imgproc,opencv_imgcodecs +``` + +The linker resolves `-lopencv_core` in the order of its search paths, so a search +path of another OpenCV would win. When the linker flags of the build machine +already name the directory of another OpenCV, such as a global `RUSTFLAGS` with a +package manager's library directory, add this one first: + +```sh +export RUSTFLAGS="-L$PWD/../target/opencv-min/lib -C link-arg=-Wl,-rpath,$PWD/../target/opencv-min/lib ${RUSTFLAGS:-}" +unset CARGO_ENCODED_RUSTFLAGS +``` + +The extension records its OpenCV as `@rpath/libopencv_*.dylib` on macOS and as +`libopencv_*.so.412` on Linux. The rpath above makes the dynamic loader and the +repair step resolve those names against this OpenCV rather than against one from +a package manager. + +Build the wheel and the source distribution: + +```sh +maturin build --release +maturin sdist +``` + +Both land in `dist/`. The wheel is `cp310-abi3` and named after the platform, for +example `deepseek_recipe-0.1.1-cp310-abi3-macosx_11_0_arm64.whl`. + +## Bundle the native libraries + +Copy the referenced libraries into the wheel and rewrite the references: + +```sh +delocate-wheel -w dist-repaired dist/*.whl # macOS +auditwheel repair -w dist-repaired dist/*.whl # Linux +``` + +`delocate` places the libraries in `deepseek_recipe/.dylibs`, `auditwheel` places +them in `deepseek_recipe.libs`, and both rewrite the references of the extension +to load them from beside itself. Copy the source distribution, which needs no +repair, beside the repaired wheel: + +```sh +cp dist/*.tar.gz dist-repaired/ +``` + +Check that the wheel resolves every library inside itself: + +```sh +delocate-listdeps --all dist-repaired/*.whl # macOS: only /usr/lib and /System +auditwheel show dist-repaired/*.whl # Linux: the manylinux tag it claims +``` + +The check that matters is that the installed extension loads its own libraries. A +virtual environment with the wheel and the tests does this on the build machine: + +```sh +python3 -m venv /tmp/wheel-test +/tmp/wheel-test/bin/pip install dist-repaired/*.whl pytest +/tmp/wheel-test/bin/python -m pytest tests -q +/tmp/wheel-test/bin/python -c "import deepseek_recipe; print(deepseek_recipe.__file__)" +``` + +## Windows + +A Windows build differs from the steps above in the generator of the OpenCV build, +the names of the import libraries, and the tool that bundles the DLLs. The build +needs Visual Studio Build Tools with the C++ workload, and LLVM for the `clang` +binary and `libclang.dll` that the `opencv` crate runs while it generates the +bindings; point `LIBCLANG_PATH` at the directory of `libclang.dll` when the build +does not find it. + +### Build the minimal OpenCV + +The Visual Studio generator takes the architecture of the build, and it builds and +installs a configuration at a time. The arguments are the ones of the build above; +keep the two lists in step. + +```sh +git clone --depth 1 --branch 4.12.0 https://github.com/opencv/opencv.git +cmake -S opencv -B opencv/build -A x64 \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX="$PWD/target/opencv-min" \ + -DBUILD_SHARED_LIBS=ON \ + -DBUILD_LIST=core,imgproc,imgcodecs \ + -DOPENCV_GENERATE_PKGCONFIG=ON \ + -DBUILD_ZLIB=ON -DBUILD_JPEG=ON -DBUILD_PNG=ON -DBUILD_WEBP=ON \ + -DWITH_JPEG=ON -DWITH_PNG=ON -DWITH_WEBP=ON \ + -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DBUILD_EXAMPLES=OFF \ + -DBUILD_opencv_apps=OFF -DBUILD_DOCS=OFF -DBUILD_JAVA=OFF \ + -DBUILD_opencv_python2=OFF -DBUILD_opencv_python3=OFF \ + -DBUILD_PROTOBUF=OFF \ + -DWITH_ADE=OFF -DWITH_EIGEN=OFF -DWITH_FFMPEG=OFF -DWITH_GSTREAMER=OFF \ + -DWITH_GTK=OFF -DWITH_IPP=OFF -DWITH_ITT=OFF -DWITH_JASPER=OFF \ + -DWITH_LAPACK=OFF -DWITH_OPENCL=OFF -DWITH_OPENCLAMDBLAS=OFF \ + -DWITH_OPENCLAMDFFT=OFF -DWITH_OPENEXR=OFF -DWITH_OPENJPEG=OFF \ + -DWITH_OPENMP=OFF -DWITH_QUIRC=OFF -DWITH_TBB=OFF -DWITH_TIFF=OFF \ + -DWITH_V4L=OFF -DWITH_1394=OFF +cmake --build opencv/build --config Release --parallel +cmake --install opencv/build --config Release +``` + +The install places the DLLs in `bin`, the import libraries in `lib`, and the +headers directly in `include`. The file names carry the version of OpenCV, because +OpenCV appends its major, minor, and patch numbers to the module name on Windows: +`opencv_core4120.dll` and `opencv_core4120.lib` for 4.12.0. + +### Build the wheel + +The import libraries carry that version, so `OPENCV_LINK_LIBS` holds the names +with the suffix, and the include directory is `include` rather than +`include/opencv4`: + +```sh +cd deepseek-recipe-python +export OPENCV_INCLUDE_PATHS="$PWD/../target/opencv-min/include" +export OPENCV_LINK_PATHS="$PWD/../target/opencv-min/lib" +export OPENCV_LINK_LIBS=opencv_core4120,opencv_imgproc4120,opencv_imgcodecs4120 +maturin build --release +maturin sdist +``` + +A change of the OpenCV version changes the suffix; `ls ../target/opencv-min/lib` +lists the names of the current build. Windows has no rpath, so these variables are +the whole link configuration, and the repair step rewrites the import table +afterwards. + +### Bundle the DLLs + +```sh +delvewheel repair --add-path "$PWD/../target/opencv-min/bin" -w dist-repaired dist/*.whl +cp dist/*.tar.gz dist-repaired/ +``` + +`delvewheel` copies the DLLs into `deepseek_recipe.libs` and rewrites the import +table of the extension. `delvewheel show` lists the DLL dependencies that remain +outside the wheel: + +```sh +delvewheel show dist-repaired/*.whl +``` + +The wheel is `cp310-abi3-win_amd64`. A fresh environment finds no OpenCV unless one +was installed for it separately, so an import in that environment proves that the +DLLs are bundled: + +```sh +python -m venv target/wheel-test +target/wheel-test/Scripts/pip install dist-repaired/*.whl pytest +target/wheel-test/Scripts/python -m pytest tests -q +target/wheel-test/Scripts/python -c "import deepseek_recipe; print(deepseek_recipe.__file__)" +``` + +## Check the metadata + +```sh +twine check dist-repaired/* +``` + +The package declares its license files through the PEP 639 `license-files` field, +which needs `twine` 7 or newer. An older release reports the field as malformed. + +## Licenses of the bundled libraries + +The wheel contains OpenCV, which is licensed under Apache-2.0, and the codecs that +its `imgcodecs` module compiles in, which are licensed under BSD-3-Clause, the PNG +Reference Library license, and the zlib license. +[deepseek-recipe-python/THIRD_PARTY_LICENSES](../deepseek-recipe-python/THIRD_PARTY_LICENSES) +reproduces those licenses, and the package declares the file as a license file. +Collect the texts again from the OpenCV checkout when the version of OpenCV +changes. + +The CycloneDX SBOM of the wheel lists the Rust crates. OpenCV is not a Rust crate, +so that document does not cover it. + +## Upload + +Check the artifacts against TestPyPI before uploading them to PyPI: + +```sh +twine upload --repository testpypi dist-repaired/* +twine upload dist-repaired/* +``` + +An upload authenticates with an API token: `__token__` as the user name and the +token as the password, in `~/.pypirc` or in `TWINE_USERNAME` and +`TWINE_PASSWORD`. A version that reaches PyPI cannot be replaced, and a file of a +released version cannot be replaced either. diff --git a/docs/development.md b/docs/development.md index 9d6fd81..0d9cbee 100644 --- a/docs/development.md +++ b/docs/development.md @@ -24,12 +24,17 @@ sudo apt install -y build-essential pkg-config clang libclang-dev libopencv-dev For Python development on Debian, also install `python3-dev` and `python3-venv`. -Use OpenCV 4.x for this workspace. The pinned Rust `opencv` 0.93 bindings -support OpenCV 3.4 and 4.x, not OpenCV 5.x. For other systems or a custom -installation, follow the binding version's [installation guide][opencv-install] -and [troubleshooting guide][opencv-troubleshooting]. Configure -`PKG_CONFIG_PATH`, `OpenCV_DIR`, or `LIBCLANG_PATH` as appropriate for your -installation. +Use OpenCV 4.11 or newer for this workspace. OpenCV 4.11 reads an animated WebP +as its first frame, while an earlier release returns an empty matrix for the +same image. The pinned Rust `opencv` 0.93 bindings support OpenCV 3.4 and 4.x, +not OpenCV 5.x. For other systems or a custom installation, follow the binding +version's [installation guide][opencv-install] and +[troubleshooting guide][opencv-troubleshooting]. Configure `PKG_CONFIG_PATH`, +`OpenCV_DIR`, or `LIBCLANG_PATH` as appropriate for your installation. + +The `libopencv-dev` of Debian 12 predates 4.11. Build the OpenCV of the wheel as +described in [Building the Python wheel](build-python-wheel.md) to develop against +a newer release. To check an OpenCV 4 installation discovered through pkg-config: @@ -114,76 +119,13 @@ For an installation without editable development, use: python3 -m pip install ./deepseek-recipe-python ``` -## Wheel packaging +## Python wheel -The extension links OpenCV dynamically, so a wheel is only installable on a -machine with a matching OpenCV until the libraries are copied into it: -`delocate` on macOS, `auditwheel` on Linux. The libraries of the build machine -become part of the wheel, so the maintainer builds the wheel against an OpenCV -that is small enough to bundle. - -Build that OpenCV from source, with only the modules the extension uses. The -build disables OpenCL, TBB, OpenEXR, and TIFF because each of them links its -own libraries, which the wheel would then have to include. - -```sh -git clone --depth 1 --branch 4.12.0 https://github.com/opencv/opencv.git -cmake -S opencv -B opencv/build \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_INSTALL_PREFIX="$PWD/target/opencv-min" \ - -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0 \ - -DBUILD_SHARED_LIBS=ON \ - -DBUILD_LIST=core,imgproc,imgcodecs \ - -DWITH_OPENCL=OFF -DWITH_TBB=OFF -DWITH_OPENEXR=OFF -DWITH_TIFF=OFF \ - -DWITH_JPEG=ON -DWITH_PNG=ON -DWITH_WEBP=ON -cmake --build opencv/build --parallel -cmake --install opencv/build -``` - -Keep `CMAKE_OSX_DEPLOYMENT_TARGET` at the oldest macOS release that the wheel -should support: the installed libraries determine the platform tag of the wheel. - -Then build the wheel against that OpenCV, and copy its libraries into it: - -```sh -cd deepseek-recipe-python -export OPENCV_INCLUDE_PATHS="$PWD/../target/opencv-min/include/opencv4" -export OPENCV_LINK_PATHS="$PWD/../target/opencv-min/lib" -export OPENCV_LINK_LIBS=opencv_core,opencv_imgproc,opencv_imgcodecs -maturin build --release -delocate-wheel -w dist-repaired dist/*.whl # `auditwheel repair` on Linux -twine check dist-repaired/* -``` - -The `OPENCV_*` variables take precedence over pkg-config, so the build does not -use an OpenCV installed elsewhere on the build machine. The linker still -resolves `-lopencv_core` in the order of its search paths, so when the linker -flags of the build machine already name the directory of another OpenCV, add -this one first: - -```sh -export RUSTFLAGS="-L$PWD/../target/opencv-min/lib ${RUSTFLAGS:-}" -``` - -On macOS this produces a wheel of about 8 MB; bundling an OpenCV installed by a -package manager produces a much larger one, because that build links every -library of its own dependencies. - -Upload the repaired artifacts after checking them against TestPyPI: - -```sh -twine upload --repository testpypi deepseek-recipe-python/dist-repaired/* -twine upload deepseek-recipe-python/dist-repaired/* -``` - -An upload authenticates with an API token: `__token__` as the user name and the -token as the password, in `~/.pypirc` or in `TWINE_USERNAME` and -`TWINE_PASSWORD`. - -The wheel bundles OpenCV and the codecs that its `imgcodecs` module compiles in, -so it also includes their license texts: -`deepseek-recipe-python/THIRD_PARTY_LICENSES` reproduces them and the package -declares the file as a license file. +[Building the Python wheel](build-python-wheel.md) describes a native build: the +requirements of the build machine, the minimal OpenCV that keeps the wheel small, +the build with `maturin`, the bundling of the native libraries, and the checks of +the artifacts. It also covers the platform tags, the licenses of the bundled +libraries, and the upload to a package index. ## API documentation