#24919 implemented a mechanism for checking for checking for sagelib's direct and indirect dependencies at the time of running the main ./configure script. This was already done in an ad-hoc manner for a handful of packages (e.g. gcc, git, yasm, curl), but we wanted to be able to do the same for other packages, especially those that are common on Linux-like systems.

This has now been implemented for many other packages, or is in the process of being implemented. Those packages are listed below. However, there are many others yet to be done, and this ticket will attempt to list them in some rough order of priority.

For use of Python packages: See #29023 Meta-ticket: In a python 3 build, use system Python packages

Already done

The following packages already have spkg-configure.m4 checks that are working on most or all supported platforms:

• 4ti2 (Add spkg-configure.m4 for 4ti2, remove direct use of SAGE_LOCAL #30887, optional package)
• arb (spkg-configure.m4 for arb #27270)
• boost_cropped/boost (spkg-configure.m4 for boost and boost_cropped #29100) -- many systems have boost headers available
• brial (spkg-configure.m4 for brial #29369) - see also fedora-30-standard: Doctests using system brial crash #29490 - where it was disabled due to crashes on Fedora 30
• bzip2 (spkg-configure.m4 for bzip2 #27182)
• cbc (spkg-configure for cbc #28908)
• cddlib (spkg-configure for cddlib #28905)
• cliquer (spkg-configure for cliquer #28885)
• cmake (spkg-configure.m4 for cmake #27487)
• coxeter3 (coxeter3: Add spkg-configure.m4, use coxeter/ prefix for includes, move OptionalExtension options to .pyx file directives #29721, optional)
• curl (Generic mechanism for system package checks at configure time #24919)
• eclib (spkg-configure.m4 for eclib #28333)
• ecl (Support system installations of ECL and fix the kenzo SPKG build #29617)
• ecm (spkg-configure.m4 for ecm #27271)
• fflas-ffpack (Detect and use the system fflas-ffpack #29453)
• flint (spkg-configure.m4 for flint #27264) - merged by spkg-configure.m4 for arb #27270
• flintqs (spkg-configure.m4 for flintqs #29430)
• fplll (spkg-configure for fplll #29245)
• freetype (spkg-configure.m4 for freetype #27168)
• gc (spkg-configure.m4 for gc #28991)
• gcc (Generic mechanism for system package checks at configure time #24919)
• gf2x (use system's gf2x if available #27238)
• gfan (spkg-configure for gfan #28985)
• gfortran (Generic mechanism for system package checks at configure time #24919)
• giac (spkg-configure.m4 for giac #29541)
• git (Generic mechanism for system package checks at configure time #24919)
• givaro (spkg-configure.m4 for m4ri, m4rie, givaro #28342)
• glpk (spkg-configure.m4 for glpk #28459)
• gmp (spkg-configure.m4 for gmp and mpir #27212)
• gp2c (spkg-configure.m4 for pari #28242)
• gsl (spkg-configure for GSL #28879)
• igraph (igraph, python-igraph: Update to 0.8.3 #30899, optional package)
• iml (spkg-configure.m4 for iml #27272)
• isl (spkg-configure.m4 for isl #27864)
• iconv (spkg-configure.m4 for iconv #27823)
• lcalc (spkg-configure.m4 for lcalc #28224)
• libatomic (spkg-configure.m4 for libatomic #28045)
• libbraiding (spkg-configure.m4 for libbraiding #29458)
• libffi (Package libffi #25900, Further libffi fixes #27114)
• libgd (spkg-configure.m4 for libgd #27825)
• libhomfly (spkg-configure.m4 for libhomfly #29630)
• libpng (spkg-configure.m4 for libpng #27186)
• linbox (spkg-configure.m4 for linbox #29631)
• lrcalc (spkg-configure.m4 for lrcalc #28231)
• lrslib (spkg-configure.m4, system package info for lrslib #27804, optional package)
• m4ri(e) (spkg-configure.m4 for m4ri, m4rie, givaro #28342)
• maxima (spkg-configure.m4 for maxima #32867)
• mpc (spkg-configure.m4 for mpc #27259) - merged by Update configure with checks for mpfr, mpc, and ntl #27822
• mpfi (spkg-configure.m4 for mpfi #28295)
• mpfr (spkg-configure.m4 for mpfr #27258) - merged by Update configure with checks for mpfr, mpc, and ntl #27822
• mpir (spkg-configure.m4 for gmp and mpir #27212) - removed in Remove package mpir #32549, complete mpir deletion #32727
• nauty (spkg-configure.m4 for nauty #28958)
• ncurses (spkg-configure.m4 for ncurses and readline #27277)
• ninja-build (spkg-configure.m4 for ninja-build #27801)
• ntl (spkg-configure.m4 for NTL #27265) - merged by Update configure with checks for mpfr, mpc, and ntl #27822
• openblas (spkg-configure.m4 for $(BLAS) (i.e. atlas and openblas) #27870)
• openssl (Add spkg-configure.m4 for openssl #30557, {ubuntu-trusty, debian-jessie, linuxmint-17, fedora-{26,27,28}}-standard: system openssl too old for python3 #32580)
• palp (spkg-configure.m4 for palp #29672)
• pari (spkg-configure.m4 for pari #28242)
• pari's packages, apart from pari_jupyter (spkg-configure.m4 for pari #28242)
• patch (Generic mechanism for system package checks at configure time #24919)
• pcre (spkg-configure.m4 for pcre #27284)
• perl_term_readline_gnu (spkg-configure.m4 for perl_term_readline_gnu #27795)
• pkgconf (spkg-configure.m4 for pkgconf #27827)
• planarity (spkg-configure.m4 for planarity #29063)
• polymake (spkg-configure.m4 for polymake #31527)
• ppl (spkg-configure.m4 for ppl #29454)
• primecount (update and promote spkg primecount #25009)
• primesieve (update and promote spkg primecount #25009)
• pynac (spkg-configure.m4 for pynac #29542)
• python3 (spkg-configure.m4 for python3 #27824) this one is of course a very special case; see the ticket for details.
• qepcad (QEPCAD: get rid of qepcad bits in src/bin/sage-location #21783)
• qhull (build/pkgs/qhull: Update, add spkg-configure.m4 #31161)
• r (spkg-configure for R #28884) (alternative view: getting R from the system has been desirable for a long time and will require more than just an spkg-configure.m4; see Allow the use of system's R installation #25503)
• readline (spkg-configure.m4 for ncurses and readline #27277)
• rw (Configure check for system "rw" library #27814)
• singular (spkg-configure.m4 for singular #29024; dup: spkg-configure for singular #29515)
• sqlite (spkg-configure.m4 for sqlite #29002)
• suitesparse (spkg-configure for suitesparse #29502)
• symmetrica (spkg-configure.m4 for symmetrica #28208)
• sympow (spkg-configure.m4 for sympow #29673)
• tachyon (spkg-configure for tachyon #28919)
• texlive (texlive: Add spkg-configure.m4 and system package information; remove install script #31529)
• tox (spkg-configure, system package info for tox #30544)
• xz (more system package checks #26286)
• yasm (Generic mechanism for system package checks at configure time #24919)
• zeromq (more system package checks #26286)
• zlib (more system package checks #26286)
• zn_poly (spkg-configure.m4 for zn_poly #29649)

In Progress

The following packages have in-progress tickets to provide configure-time detection on the system:

• bliss (Bliss spkg config #35830)
• fricas (accept fricas from the system #35837)

Have tickets

The following packages have tickets for their conversions, but don't have working prototypes (i.e. branches) yet:

• BLAS (atlas and other non-openblas): resolving how to handle
  finding a usable BLAS should be one of the highest priorities;
  it faces many of the same issues we faced with MPIR/GMP
  in spkg-configure.m4 for gmp and mpir #27212, but perhaps with even more complexity). One can
  already use system's openblas, though (spkg-configure.m4 for $(BLAS) (i.e. atlas and openblas) #27870), and we
  removed ATLAS in Remove ATLAS #30350.
• csdp (spkg-configure and distros/ for csdp #31145, optional package)
• gap (using system GAP is desirable, but non-trivial,
  especially now that we need a working libgap) (spkg-configure.m4 for gap #29644)
• jmol (Update jmol to 14.32.35, add spkg-configure for jmol #30316)
• lie (spkg-configure.m4 for lie #32055, experimental package)
• mathjax (System package information and spkg-configure for mathjax #30296)
• perl (Script package with spkg-configure.m4 for perl #31839, script package)
• symengine (spkg-configure for symengine #32421, optional package)

No ticket yet

Use of system Python packages

This will be be handled by #29023. Please do not add Python packages here.

The following are exceptions (because it does not run in the same Python as sage):

• notebook (System package information, spkg-configure for Jupyter "notebook" package, "rst2ipynb", and dependencies #30124), rst2ipynb, and their dependencies

Testing infrastructure

• Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add tox.ini #29053: Add debian/fedora package information to build/pkgs, generate Dockerfiles and installation help
• Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages #29060: Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Meta infrastructure

• Add one file build/pkgs/*/repology.txt per spkg #31114 Add one file build/pkgs/*/repology.txt per spkg

Documentation

Unfortunately there is still not good documentation for how to implement these configure-time dependency checks.

• Add documentation for implementing spkg-configure.m4 and distros/ for SPKGs #26668: add documentation for implementing spkg-configure.m4 and distros/ for spkgs
• Improve documentation of m4/sage_spkg_configure.m4 #29018: improve documentation of m4/spkg-configure.m4
• Document spkg-configure.m4 and distros directory #30543: Document spkg-configure.m4 and distros directory

Bug-fixing

A number of bugs/deficiencies has been uncovered in spkg-configure.m4 files and related configuration, and a number of tickets opened/closed

• build/pkgs/gfortran/spkg-configure.m4 works incorrectly if CC and CXX are already there #26715: build/pkgs/gfortran/spkg-configure.m4 works incorrectly if CC and CXX are already there
• bootstrap should fail gracefully, if there is no pkg-config available #27219: have bootstrap fail gracefully if no pkg-config is available
• some dummy packages must be only re-installable via ./configure --with-... #27373: some dummy packages must be only re-installable via ./configure --with-...
• Add --with-system-<spkg> options to configure #27567: Add --with-system- options to configure
• SAGE_SPKG_CONFIGURE macro: Add new pre-check and post-check optional arguments #27641: SAGE_SPKG_CONFIGURE macro: Add new pre-check and post-check optional arguments
• Re-run configure+make after installing an SPKG with sage -i #27642: Re-run configure+make after installing an SPKG with sage -i
• spkg-configure.m4 for pcre should test for UTF support etc #27936: spkg-configure.m4 for pcre should test for UTF support etc
• R installation fails on macOS with libcurl from Anaconda #27941: R installation fails on macOS with libcurl from Anaconda
• Add --enable-OPTIONALSPKG options to configure #28095: Add --enable-OPTIONALSPKG options to configure
• Use -print-prog-name instead of -print-file-name in gcc spkg-configure.m4 #28317: Use -print-prog-name instead of -print-file-name in gcc spkg-configure.m4
• Allow doc to build with SAGE_PARI_CFG undefined #28341: Allow doc to build with SAGE_PARI_CFG undefined
• bump up the version of givaro in spkg-configure.m4 #28380: bump up givaro version in spkg-configure.m4
• pass correct --with-flint to eclib's configure #28401: pass correct --with-flint to eclib's configure
• correct the test for galpol package in pari's spkg-configure.m4 #28405: correct the test for galpol package in pari's spkg-configure.m4
• Flint configure fails when using system GMP in certain cases #28409: Flint configure fails when using system GMP in certain cases
• generate libpng.pc, zlib.pc if needed #28906: generate libpng.pc, zlib.pc if needed
• correct AC_CHECK_PROG call in spkg-configure for bzip2 #28956: correct AC_CHECK_PROG call in spkg-configure for bzip2
• verify if cblas.pc and lapack.pc should be replaced by links to openblas.pc #29071: verify if cblas.pc and lapack.pc should be replaced by links to openblas.pc
• switch the default mp library to gmp #30752: switch the default mp library to gmp
• build/pkgs/{gmp,mpfr}/spkg-configure.m4: Check pkg-config first #31348: build/pkgs/mpfr/spkg-configure.m4: Check pkg-config first
• python3 spkg-configure.m4: Do not reject python based on sysconfig LDFLAGS containing "-L." #31358: python3 spkg-configure.m4: Do not reject python based on sysconfig LDFLAGS containing "-L."
• flint spkg-configure: Reject FLINT 2.7.x #31429: flint spkg-configure: Reject FLINT 2.7.x
• Cleanup spkg-configure.m4 files that mix tabs and spaces #31528: Cleanup spkg-configure.m4 files that mix tabs and spaces
• conda-forge-standard (linux): python3 spkg-configure.m4 rejects conda's python3 #31539: conda-forge-standard (linux): python3 spkg-configure.m4 rejects conda's python3
• Fix spkg-configure.m4 for singular #32789: Fix spkg-configure.m4 for singular
• givaro spkg-configure: Reject too new versions #33042: givaro spkg-configure: Reject too new versions
• ffmpeg/imagemagick features need feature checks #33092: spkg-configure.m4 for ffmpeg/imagemagick need feature checks

CC: @sagetrac-tmonteil @orlitzky @timokau @kiwifb @isuruf @mezzarobba @tobihan @tobiasdiez

Component: build: configure

Keywords: sd111

Issue created by migration from https://trac.sagemath.org/ticket/27330