0
0
mirror of https://github.com/openwrt/packages.git synced 2025-07-08 14:55:28 +00:00
Files
Matthias Schiffer b98fb60635 perl: fix parallel build race condition in target build
We have received reports of builds of perl occasionally failing when
building with many parallel jobs, with a log like the following:

    LD_LIBRARY_PATH=[...]/perl/perl-5.40.0 ./miniperl -Ilib make_ext.pl \
        dist/constant/pm_to_blib  MAKE="make" LIBPERL_A=libperl.so
    File/Path.pm did not return a true value at [...]/hostpkg/usr/lib/perl5/5.40.0/ExtUtils/MakeMaker.pm line 13.
    BEGIN failed--compilation aborted at [...]/hostpkg/usr/lib/perl5/5.40.0/ExtUtils/MakeMaker.pm line 13.
    Compilation failed in require at Makefile.PL line 3.
    BEGIN failed--compilation aborted at Makefile.PL line 3.
    Unsuccessful Makefile.PL(dist/constant): code=65280 at make_ext.pl line 532.

The failing extension (dist/constant in the above log) would differ
between runs.

The cause of the issue is the `-Ilib` in the command line of miniperl.
In the host build, `./miniperl -I lib` will use the following include
path:

    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/AutoLoader/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/Carp/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/PathTools
    [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/PathTools/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/ExtUtils-Install/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/ExtUtils-MakeMaker/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/ExtUtils-Manifest/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/File-Path/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/ext/re
    [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/Term-ReadLine/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/Exporter/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/ext/File-Find/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/Text-Tabs/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/constant/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/version/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/Getopt-Long/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/Text-ParseWords/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/ExtUtils-PL2Bat/lib
    [..]/build_dir/hostpkg/perl/perl-5.40.0/lib
    .

Various dependencies of the extension build scripts (Makefile.PL) -
including File-Path, which failed to be loaded in the error log - are
included in the path by buildcustomize.pl, as these extensions are only
installed to `lib` as the build proceeds.

However, in a target build, miniperl is just a symlink to the previously
built host perl. As the host perl does not implicitly load
`buildcustomize.pl`, we get the following include path for
`./miniperl -Ilib`:

    lib
    [..]/staging_dir/hostpkg/usr/lib/perl5/site_perl/5.40.0/x86_64-linux
    [..]/staging_dir/hostpkg/usr/lib/perl5/site_perl/5.40.0
    [..]/staging_dir/hostpkg/usr/lib/perl5/5.40.0/x86_64-linux
    [..]/staging_dir/hostpkg/usr/lib/perl5/5.40.0

The host perl's install location is used as the default include path
which provides File-Path etc. for the target build; however, as more
and more libraries get installed into `lib` during the extension build,
they may get loaded from there instead, as `lib` is at the beginning of
the include path. When multiple extensions are built in parallel, a
Makefile.PL may attempt to load File/Path from `lib` after the file has
been created, but before its contents have been written fully, resulting
in the build to fail.

In fact, we should not load anything from `lib` during the target build,
as it is the staging directory for the target, including native
extensions built for the target architecture - with one exception: The
build scripts expect to find target information in the `Config` module,
so simply removing `lib` from the include path completely would break
the build.

Solve the issue by creating an alternative lib directory `lib_build`,
symlinking `Config.pm` and its dependencies in it, and replacing the
`-Ilib` argument with `-Ilib_build` using a wrapper script around the
host perl executable. This is similar to the approach seen in perl's own
obsolete/broken cross compile scripts (`Cross/Makefile`).

Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2025-05-05 11:20:38 +02:00
..
2017-10-24 11:40:34 -06:00
2017-10-24 11:40:34 -06:00
2017-10-24 11:40:34 -06:00
2024-07-22 18:48:26 -07:00
2017-10-24 11:40:34 -06:00
2017-10-24 11:40:34 -06:00
2017-10-24 11:40:34 -06:00
2015-09-01 09:23:24 +02:00
2022-03-02 16:30:32 -08:00
2017-10-24 11:40:34 -06:00
2023-10-22 23:29:03 -06:00
2024-06-20 14:18:35 -07:00
2017-10-24 11:40:34 -06:00

-- Perl configuration --

Perl uses a huge configuration file, normally generated via the Configure script
at build-time. This fails when cross-compiling though, so we need to supply our
own.

We're using perlconfig.pl to piece together the final configuration from a bunch
of configuration files(all ending in .config). Please refer to perlconfig.pl's
POD for information on usage and syntax.

Throughout the files, you will see a bunch of references to private symbols with
the prefix "owrt". These are used to control output in an effort to both
simplify writing configuration files, as well as to provide switchable options
to select the feature set of the resulting perl installation.

The following will be a summary/quick reference of all private symbols we're
currently using:

Passed via architecture configuration file(mipsel.config, i486.config, ...)
---------------------------------------------------------------------------
Symbol              Values              Description
owrt:bits           32/64               Target's native word length.
owrt:endian         little/big          Target's endianness.
owrt:arch           mipsel, i486, ...   Target's architecture name.
owrt:sig_count      64/128              Number of signals the target
                                        provides(NSIG - 1).
owrt:sigs           *                   Symbolic names of the first 32 signals
                                        this architecture provides, in numeric
                                        order. Separated by whitespaces.
owrt:sig_name_extra *                   Symbolic names of any additional signals
                                        this architecture provides after
                                        owrt:sig_count. Separated by
                                        whitespaces.
owrt:sig_num_extra  *                   Numeric values associated with the
                                        signal names provided in
                                        owrt:sig_name_extra. Separated by
                                        whitespaces.

Passed via command line
-----------------------
Symbol                Values              Description
owrt:libc             glibc/uclibc/musl   Which C library implementation is in
                                          use.
owrt:threads          yes/no              Whether to enable threading support.
owrt:ipv6             define/undef        Whether to enable IPv6 support.
owrt:target_cross     *                   Target architecture's host triplet.
owrt:target_cc        *                   C compiler to use.
owrt:gccversion       *                   target_cc's version number.
owrt:cflags           *                   Additional C compiler flags.
owrt:ldflags          *                   Additional linker flags.
owrt:staging_dir      *                   Same as OpenWRT buildroot's
                                          $(STAGING_DIR).
owrt:host_perl_prefix *                   host-perl installation prefix.

Passed via version.config
-------------------------
Symbol           Values              Description
owrt:perllibpath *                   Path to perl library files, from the
                                     target's point of view.