MIR mirroring-ports - how to build a mirror for ports distfiles
The OpenBSD Ports Collection offers some powerful tools to
mirror software
sources. The ports infrastructure provides a
mirror-maker target
that can be used to build a Makefile to facilitate mirroring
distfiles.
This target builds ${DISTDIR}/Makefile, which can be used on
almost any
Unix-like machine to mirror OpenBSD distfiles. The variable
RECURSIVE_FETCH_LIST can be set to `Yes' if port dependencies should also
be recorded.
A sample Makefile entry is formatted like this:
all:: audio/tracker/tracker-5.3
.PHONY: audio/tracker/tracker-5.3
audio/tracker/tracker-5.3: tracker-5.3.tgz
tracker-5.3.tgz: $F
@MAINTAINER="[email protected]"
SITES="ftp://ftp.uni-trier.de/pub/unix/audio/tracker/ "
CIPHER="sha1" CKSUM="b0973d6a9c363caebd3a71547412f42b0681f323"
exec ${FETCH} "$@"
This Makefile is usually invoked by the user from the directory where
they wish to mirror distfiles, with variables FETCH and F
set on the command
line, e.g.,
cd mirror && make -k -j 5 -f path_to_makefile
FETCH=fetch_script
Targets are set up so that each port is referenced by its
full name, and
retrieves all its distfiles. The default all target is used
to retrieve
all distfiles. Targets ftp and cdrom can be used to retrieve all distfiles
necessary to build ftp and cdrom packages, respectively. Actual
fetching is usually invoked with a parallel-make option, so
that several
retrievals through ftp can proceed simultaneously.
The F variable can be set to a dummy name, or a recent filename, to force
re-fetching of anything which is older than the filename.
Its intended
use is to force re-fetching existing files, or to checksum
all files.
The ${FETCH} script should be supplied by the user, and will
download and
verify the archive file. It must obey the following variables:
MAINTAINER Port maintainer, used to report errors,
ERROR Some ports problems can be detected while
building the Makefile,
in which case this variable will be set
to a proper
error message.
DIST_SUBDIR See ports(7) for more details. The ${FETCH}
script is responsible
for creating this subdirectory and
cd'ing to it
before performing the actual fetch.
SITES A list of sites to try for fetching the distribution file.
CIPHER The checksumming utility to use for verifying
the distribution
file. It will normally be set to sha1(1)
unless you
tinker with PREFERRED_CIPHER while building the
mirroring
Makefile.
CKSUM The corresponding checksum. If neither CIPHER
nor CKSUM nor
ERROR are set, the distribution file needs not
be checked.
A standard fetch strategy is to try all sites in order:
whenever the distribution
file is found, download it; verify the checksum;
erase the file
and try the next site if it doesn't match.
Mirroring sites should update their master Makefile fairly
often. Activities
a proper mirror should offer (in order of decreasing
importance):
Mirror new files
Use a proper fetch script to download missing
files,
Run ${PORTSDIR}/infrastructure/fetch/link-checksums
This script creates permanent hardlinks that preserve distfiles
against checksum changes.
Verify all checksums
All checksums should be verified from time to
time, and maintainers
notified of persistent discrepancies,
Check mastersites liveliness
Use a tool such as `mirror' to check that the
master sites
haven't fallen off the Earth. Even though the
first site in
the site list is the most important site, good
mirrors will
scan all sites and report all problems,
Remove old files
To gain room this, the mirror should maintain a
list of
`active' files (easy enough, just provide a fetch
script that
just lists the file names), and remove files that
are no
longer active. Since OpenBSD releases happen every six
months, this delay should be longer than that.
Sample scripts are provided in the
${PORTSDIR}/infrastructure/fetch directory.
${DISTDIR}/Makefile
Main mirroring Makefile
${PORTSDIR}/infrastructure/fetch/fetch-all
Sample script usable to retrieve distfiles.
${PORTSDIR}/infrastructure/fetch/check-all
Sample script to check all distfiles checksums.
${PORTSDIR}/infrastructure/fetch/link-checksums
Populating checksums subdirectories with links,
to guard
against shifting checksums.
ports(7)
This infrastructure was introduced for OpenBSD 2.7 by Marc
Espie, with
feedback from Bob Beck, Todd Fries, Camiel Dobbelaar, and a
few other
people.
Changing checksums is a recurring problem that is outside
the direct control
of the OpenBSD Project. Some software distributors
change distribution
files without warning, without changing the file name
proper. Once
the problem has been identified, the port maintainer should
usually contact
the software author to fix the problem, or, if the
software author
is unresponsive, the maintainer should use DIST_SUBDIR to
provide some
state to guard against shifting checksums.
However, a more robust approach is also needed, so that
ports users can
depend on distfiles mirrors to carry what they need irrespective of those
synchronization issues. The link-checksums script creates
another access
to the distfiles, indexed through the actual checksums that
the files
should match. Provided mirroring is run sufficiently often,
together
with link-checksums, two versions of the same distfile with
respective
checksums cksum1 and cksum2 will be available under the
names
${DISTFILES}/sha1/cksum1/distfile and
${DISTFILES}/sha1/cksum2/distfile.
Starting revision 1.281, if REFETCH is set to true,
bsd.port.mk will try
to retrieve files under that naming scheme as a last resort.
OpenBSD 3.6 February 19, 2000
[ Back ] |
