Setting Up a Pkgsrc Repository
I recently found myself needing to distribute software to developers' MacOS machines. The options were limited, and not great: I have had issues with MacPorts in the past, and Homebrew reminds me of Gentoo circa 2005 (compiling everything from scratch is great!), along with a hair-raising tendency to take over /usr/local.
Enter pkgsrc: developed for NetBSD, it is intended from the ground up to maintain a clean separation between itself and base, and is dead simple to write packages for to boot. While the NetBSD handbook explains how to write packages in exhaustive detail, it is less clear about how to distribute custom binary packages to a wider audience once you have compiled them. This blog post aims to fill that gap.
Bootstrap pkgin
Obviously, you are on a Mac. Start by installing developer tools:
xcode-select --install
Go get a cup of coffee, this is going to take a while.
Get your machine ready to install pkgsrc binary packages using the pkgin frontend:
BOOTSTRAP_TAR="bootstrap-macos14-trunk-x86_64-20200716.tar.gz"
BOOTSTRAP_SHA="395be93bf6b3ca5fbe8f0b248f1f33181b8225fe"
# Download the bootstrap kit to the current directory.
curl -O https://pkgsrc.joyent.com/packages/Darwin/bootstrap/${BOOTSTRAP_TAR}
# Verify the SHA1 checksum.
echo "${BOOTSTRAP_SHA} ${BOOTSTRAP_TAR}" >check-shasum
shasum -c check-shasum
# Install bootstrap kit to /opt/pkg
sudo tar -zxpf ${BOOTSTRAP_TAR} -C /
# Reload PATH/MANPATH (pkgsrc installs /etc/paths.d/10-pkgsrc for new sessions)
eval $(/usr/libexec/path_helper)
This code is from the Joyent pkgsrc how-to, and it will change quarterly as they update their package sets, so do make sure to check out their website for fresh instructions.
Bootstrap the pkgsrc Tree
Next, install the pkgsrc tree on your machine. This is where you will develop your packages (or at least export them to after developing them somewhere else):
cd /opt
# Check to see if pkgsrc is already there. If it is, blow it away
if [[ -d ./pkgsrc ]]; then
rm -rf ./pkgsrc
fi
# Pull in a compressed archive of the pkgsrc tree and unpack it
curl -O https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.xz
tar xzf pkgsrc.tar.xz
# Only root can install things
chown -R root:wheel ./pkgsrc
I put the tree in the /opt directory created by the pkgin bootstrap step. You could just as easily put it somewhere else.
Optional: Install Binary Versions of Build Tools
This step is optional in that pkgsrc will compile the build tools for you if your custom package needs them, but necessary in that building GCC or libtool from scratch will take hours that could be better spent doing other things.
pkgin install gcc7 cmake <etc>
Develop Your Package
I'm not going to get into this (go see the handbook!), but I recommend you become intimately familiar with the functioning of the url2pkg and createbuildlink packages, both which reduce the process of packaging external software into pkgsrc format to (basically) a point-and-click operation:
pkgin install url2pkg createbuildlink
Serve Your Packages
Once you have finished building your software, you should have a directory of distributable software in /opt/pkgsrc/packages/All. Serve it up using HTTP at https://pkgdist.testdomain.com; I use nginx, but it doesn't really matter what you use.
pkgin install nginx
A simple configuration would look this:
user nginx nginx;
worker_processes 1;
events {
# After increasing this value You probably should increase limit
# of file descriptors (for example in start_precmd in startup script)
worker_connections 1024;
}
http {
default_type application/octet-stream;
gzip on;
include /opt/pkg/etc/nginx/mime.types;
keepalive_timeout 65;
sendfile on;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
# Redirect all insecure connections to HTTPS
server {
listen 80;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name pkgdist.testdomain.com;
include common.conf;
root /opt/pkgsrc/packages/;
location / {
autoindex on;
}
}
}
My SSL settings are in common.conf; you probably don't need to see them. The format of your configuration will vary slightly depending on what operating system you are serving from (I can't believe I have to say this, but yes, you can serve MacOS packages from Ubuntu. Or FreeBSD. Or anything at all, really).
Finally, index the package directory by using pkg_summary-utils to create a pkg_summary.gz:
pkgin install pkg_summary-utils
pkg_update_summary -r /opt/pkgsrc/packages/All/pkg_summary.gz \
/opt/pkgsrc/packages/All 'gzip -dc' 'gzip -c'
Installing Your Binaries
Assuming you can go to https://pkgdist.testdomain.com/All/ and see a list of archives, you are now ready to install your software on your client Mac. Start by bootstrapping pkgin on your client machine just like you did on your build machine.
Next, add your new repository to the list of places the client pkgin looks for packages:
export REPO_ADDR=https://pkgdist.testdomain.com/All
export REPO_FILE=/opt/pkg/etc/pkgin/repositories.conf
grep -q ${REPO_ADDR} ${REPO_FILE} || echo ${REPO_ADDR} | tee -a ${REPO_FILE}
I cannot stress enough how important it is to not have a trailing slash at the end of REPO_ADDR; pkgin will not read your repository if you do.
And then install your newly developed software:
pkgin install <your package>
Make sure you add /opt/pkg/bin and /opt/pkg/sbin to your $PATH so that you can access your newly installed software.
Conclusion
I started this project looking for a way to serve MacOS software to developer machines and realized part way through that I could use the same infrastructure to serve software to our production machines running Linux: pkgsrc is, after all, fully cross-platform.
It's not often that a weekend afternoon of hacking results in actionable, complexity-reducing engineering insights, but here we are: it is good to have options.