How to Fetch a Moving Target with Poudriere

The FreeBSD ports framework has a simple idea about the world: a distfile never changes. It has a fixed size and a fixed checksum, written once into distinfo and verified every time after that. It is super useful to verify that the upstream has not changed the package, or that someone changed it without authorization. Basically this is the trust contract between our FreeBSD ports system and the upstream ports. My problem is that one of my distfiles has to change - almost every build.

I build the packages for one of my projects in Poudriere, using an overlay port on top of the standard ports tree. The port fetches its source from my internal GitLab as an auto-generated archive of a branch:

https://gitlab.example.lan/firewall/src/-/archive/main/src-main.tar.bz2

GitLab generates a new tarball on every request, so it changes with every commit to the main branch. This is exactly what I need for development: a fresh source archive on every build. This allows us to have a pretty nice CI/CD, and the developers don't need to maintain a distinfo before each build - this is a total waste of time. I dug a little into the ports system to check how I can achieve this.

Failure one: the size check

The first poudriere bulk run fetched the file without any problem and then rejected it:

fetch: https://gitlab.example.lan/firewall/src/-/archive/main/src-main.tar.bz2: size of remote file is not known
src-main.tar.bz2                                 12 MB 4540 kBps    03s
=> Fetched file size mismatch (expected 0, actual 12705860)

Expected 0, actual 12 megabytes. The expected size comes from distinfo, not from the server - make makesum records the size of the file it downloaded. My hand-written distinfo had a zero there, so the framework compared my perfectly good tarball against that zero.

I grepped the ports tree for the error message, and it led me to Mk/Scripts/do-fetch.sh:

actual_size=$(stat -f %z "${file}")
if [ -n "${dp_DISABLE_SIZE}" ] || [ -z "${CKSIZE}" ] || [ "${actual_size}" -eq "${CKSIZE}" ]; then
    continue 2
else
    ${dp_ECHO_MSG} "=> Fetched file size mismatch (expected ${CKSIZE}, actual ${actual_size})"
    if [ ${sites_remaining} -gt 0 ]; then
        ${dp_ECHO_MSG} "=> Trying next site"
        rm -f "${file}"
    fi
fi

The interesting part is dp_DISABLE_SIZE. The dp_ prefix is how the framework names variables that come down from the port Makefile, so this maps back to a plain DISABLE_SIZE knob. One line in the port's Makefile, and the size comparison is gone:

DISABLE_SIZE=   yes

Failure two: the checksum

After disabling the size check, the next run got one step further and failed on the SHA256 verification. This one is at least nice enough to tell you the fix directly in the error message:

Make sure the Makefile and distinfo file (/overlays/my-overlay/opnsense-core/distinfo)
are up to date.  If you are absolutely sure you want to override this
check, type "make NO_CHECKSUM=yes [other args]".

This is exactly what I want - skip the checksum for an always-changing tarball. So into the Makefile goes:

NO_CHECKSUM=    yes

Strictly speaking, bsd.port.mk says NO_CHECKSUM is a user variable that is not supposed to be set in a port's Makefile. For a private overlay port that nobody else will ever build, I can live with that.

Failure three: yesterday's tarball

At this point fetching worked, verification was disabled, and the build turned green. It also kept building the same code, no matter how many commits landed on the branch.

The ports framework does not download a distfile it already fetched. The file name is src-main.tar.bz2 every time, so once one copy lands in the distfiles directory, every later build simply reuses it. With the size and checksum checks disabled, there is nothing left that would notice the cached file is old.

The fix is to make the port delete its own distfiles before every fetch, using a pre-fetch target:

pre-fetch:
    @${ECHO_MSG} "Deleting existing distfiles for ${PKGNAME}..."
    @cd ${.CURDIR} && ${SETENV} ${MAKE_ENV} ${MAKE} delete-distfiles \
        RESTRICTED_FILES="${_DISTFILES:Q} ${_PATCHFILES:Q}"

The framework already ships a delete-distfiles target, but there is a catch: it only removes the files listed in RESTRICTED_FILES, and for an ordinary port that list is empty. Passing the distfiles explicitly is the same trick that the framework's own distclean target uses. Every build now starts with an empty cache, fetches a fresh archive of the branch, and actually builds what is currently on main.

That's it!

These checks exist for a good reason. The size and checksum verification is the only thing standing between you and a corrupted or compromised distfile, and I just turned all of it off. That is fine for our development environment: the file comes from our own GitLab on our own network, and the thing being "verified" is code we wrote ourselves - or our colleagues, or our colleagues' AI, or our AI. For a third-party distfile fetched over the internet, do not do any of this.