Unverified Commit 1be88b31 authored by Michael Dexter's avatar Michael Dexter Committed by GitHub
Browse files

Hello World!

parent 634cda31
headbanger.sh - a script to build FreeBSD 12.0 onward with many options
DISCLAIMER
The "head" in "headbanger" is misleading. FreeBSD "HEAD" is
more accurately called CURRENT or now "main", and this script also
bangs out STABLE branches as far back as FreeBSD 12.0, at which point the
architectures were separated into separate directories.
By default, this script should execute the following traditional commands and
direct the build output to /usr.obj/
'cd /usr/src ; make buildworld ; make buildkernel ; cd release ; make release'
The additional 1700+ lines help:
* Check out sources from SVN or Git
* Select a stock or custom KERNCONF
* Patch sources with a directory of patches
* Include a list of build options (see the src.conf manual page)
* Patch the sources for use with custom freebsd-update
* Build the release for use with custom freebsd-update
* Append a release with a custom name to avoid name collisions
Useful for building releases with reviews applied
* Build to tmpfs and rsync the results to another host
* Resume all steps based on .done breadcrumb files
In practice, this script allows up.bsd.lv to maintain a centralized
source/binary object server and have multiple higher-performance
hosts perform builds for multiple revisions and architectures.
USAGE EXAMPLES
Build AMD64 SVN revision 366954 from local SVN mirror /pub/FreeBSD/svn/base
The architecture defaults to the host architecture
sh headbanger.sh -s "/build/366954/src" -o "/build/366954/obj" -l \
"/build/366954/log" -r 366954 -S "file:///pub/FreeBSD/svn/base" \
-b "stable/12" -t amd64 -T amd64
Build ARM64 Git revision/hash 7ae27c2d6c4 from local Git mirror
/pub/FreeBSD/git/src with the GENERIC-NODEBUG kernel configuration file
while performing freebsd-update preparations (-u) and
rsync'ing the results to host 10.40 because FreeBSD cannot be
built on NFS for want of chflags(1) or 'cp -p' support:
sh headbanger.sh -s "/build/7ae27c2d6c4/src" -o "/build/7ae27c2d6c4/obj" \
-l "/build/7ae27c2d6c4/log" -r 7ae27c2d6c4 -g "/pub/FreeBSD/git/src" \
-t arm64 -T aarch64 -R "10.0.0.40" -k "GENERIC-NODEBUG" -u
Build SVN revision 364182 with kernel GENERIC-NODEBUG in the default /usr/src
and /usr/obj with bhyve NE2000 patches in /build/diffs/ and an appended
name "-ne2k" to distinguish it:
sh headbanger.sh -o "ne2k-build-logs" -r 364182 -b "head" -k "GENERIC-NODEBUG" \
-d "/pub/scripts/diffs" -n "-ne2k"
Build the source in /usr/src with ONLY the listed build options and an
out-of-src kernel configuration file named "OCCAM". All other build
options are excluded, resulting in a reduced-size userland.
sh headbanger.sh -O "WITHOUT_AUTO_OBJ WITHOUT_UNIFIED_OBJDIR WITHOUT_INSTALLLIB WITHOUT_LIBPTHREAD WITHOUT_LIBTHR WITHOUT_LIBCPLUSPLUS WITHOUT_CRYPT WITHOUT_DYNAMICROOT WITHOUT_BOOT WITHOUT_LOADER_LUA WITHOUT_LOCALES WITHOUT_ZONEINFO WITHOUT_VI" -K /tmp/OCCAM"
KNOWN ISSUES
* Search for DEBUG to find in-line known issues
* ARM64 disc1.iso images will fail to build with makefs on most 12.* versions
* Fix: https://svnweb.freebsd.org/base/head/usr.sbin/makefs/cd9660/cd9660_eltorito.c?revision=365847&view=co
* hs-ShellCheck would prefer more quoting
* Complex tests like "$svn_svr" -o "$git_svr" -o "$git_dir" should be verified
* eol_date should be a command line argument but hopefully freebsd-update-build will use MANIFEST files
* git(1) appears to no like output redirection - re-test this
* RPi and similar image generation would be great but the current tool is
completely independent of the standard build
* You will find additional issues
ACKNOWLEDGMENTS
Thank you Conor Beh for your extensive help with the up.bsd.lv effort.
Thank you dteske@ for your years of /bin/sh mentorship.
#!/bin/sh
#-
# SPDX-License-Identifier: BSD-2-Clause-FreeBSD
#
# Copyright 2021 Michael Dexter
# All rights reserved
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted providing that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
# VERSION: 1.0
# STYLE GUIDELINES
# This script is formatted with shfmt -l -w <script> and %s/fi #/fi #/g
# This script violates wrapping sensibilities while still in flux
# if statements are used in place of most case statements
# Every if statements is followed by # End if <variable_name> <condition>
# to allow a search for if to show the balancing of if statements
# This script avoids if, colons, apostrophes, and quotation marks in comments
# and in user messages given their tendency to confuse /bin/sh
# LOCAL VARIABLES
pub_dir="/httpd/download"
upd_dir="/usr/freebsd-update-build"
vm_img_dir="/vm-images/"
src_dir="/usr/src"
top_obj_dir="/usr/obj"
log_dir="/usr/obj"
target="$(uname -m)"
target_arch="$(uname -p)"
jobs="$(sysctl -n hw.ncpu)"
eol_date=1609401600 # Probably should not be hard-coded
# INTERNAL DEFAULTS and UNSETS
src_rev=""
src_rev_string=""
upd_rev_string=""
kernconf_string=""
build_options=""
build_options_to_keep=""
branch_override_string=""
make_flags=""
build_name=""
f_usage() {
echo By default this utility approximates a FreeBSD Release Engineering
echo weekly snapshot build using the contents of /usr/src
echo
echo "Supported Flags"
echo
echo "-s \"<path>\" Source directory path"
echo "-o \"<path>\" Object directory path"
echo "-l \"<path>\" Log and progress \".done\" bread crumb directory"
echo "-r \"<number>\" SVN or Git revision number or hash"
echo "-S \"<URL>\" SVN server base URL"
echo "-b \"/branch/\" SVN or Git branch such as \"stable/12\""
echo "-G \"<URL>\" Git server URL"
echo "-g \"<path>\" Git directory path for local clone --reference"
echo "-t \"<target>\" Architecture override such as \"arm64\""
echo "-T \"<target arch>\" Architecture arch override such as \"aarch64\""
echo "-j \"<number>\" Parallel make jobs number"
echo "-k \"<name>\" In-source KERNCONF to use such as GENERIC-NODEBUG"
echo "-K \"<path>\" External KERNCONF to replace GENERIC with"
echo "-d \"<path>\" Diffs directory - sequence is alphabetical"
echo "-O \"<list>\" Build options to keep. ALL others will be excluded"
echo " Use a src.conf if that is NOT what you want"
echo "-u freebsd-update handling"
echo "-m \"<string>\" make flags that are not handled by this monstrosity"
echo "-n \"-name\" Name to append to a release name for separation"
echo "-R \"<user@host>\" Remote user, host and path above object directory"
echo " to rsync tmpfs build to"
echo
echo "Be sure to \"quote/long/things\" like URLs"
exit 1
} # End f_usage
f_sanitize_path() {
# Remove the trailing slash and multiple slashes
echo ${1%/} | tr -s /
} # end f_sanitize_path
while getopts s:o:l:r:S:b:G:g:t:T:j:k:K:d:O:um:n:R: opts; do
case $opts in
s)
src_dir="${OPTARG}"
src_dir=$(f_sanitize_path $src_dir)
;;
o)
top_obj_dir="${OPTARG}"
top_obj_dir=$(f_sanitize_path $top_obj_dir)
echo DEBUG sanitized top_obj_dir is $top_obj_dir
;;
l)
log_dir="${OPTARG}"
;;
r)
src_rev="${OPTARG}"
# DEBUG Verify that it is set... if [ "$src_rev" ]; then...
# Either ignore this for Git or handle it in-context
# This is global for re-runs
upd_rev_string="-${src_rev}"
;;
S)
svn_svr="${OPTARG}"
;;
b)
src_branch="${OPTARG}"
;;
G)
git_svr="${OPTARG}"
;;
g)
git_dir="${OPTARG}"
[ -d "$git_dir" ] || {
echo $git_dir not found
exit 1
}
;;
t)
target="${OPTARG}"
# Verify against a list? Later?
# Is there a way to get a canonical, up to date list from src?
;;
T)
target_arch="${OPTARG}"
;;
j)
jobs="${OPTARG}"
# Verify that it is a number?
;;
k)
kernconf="${OPTARG}"
# Validate later because it may depend on a source checkout
;;
K)
custom_kernconf="${OPTARG}"
;;
d)
diff_dir="${OPTARG}"
[ -d "${diff_dir}" ] || {
echo $diff_dir is not a directory
exit 1
}
[ -r "${diff_dir}" ] || {
echo $diff_dir is not readable
exit 1
}
;;
O)
build_options=1
build_options_to_keep="${OPTARG}"
;;
u)
freebsd_update=1
;;
m)
make_flags="${OPTARG}"
;;
n)
build_name="${OPTARG}"
;;
R)
tmpfs_host="${OPTARG}"
;;
*)
f_usage
;;
esac
done # End while getopts
# It would be nice to put all of the build strings here for review
# but the update handling ones are based on version information provided
# by newvers.sh which is only obtained after source checkout
# EARLY TESTS OF THE DEFAULTS THAT MAY HAVE BEEN OVERRIDDEN WITH ARGUMENTS
if ! [ -d "${src_dir}" ]; then
echo
echo Source directory does not exist
echo
echo Making ${src_dir}
mkdir -p "${src_dir}" || {
echo Failed to create $src_dir
exit 1
}
fi # End if src_dir
if ! [ -d "${log_dir}" ]; then
echo Log directory does not exist. Creating
mkdir -p "${log_dir}" || {
echo Failed to create $log_dir
exit 1
}
fi # End if log_dir
echo -----------------------------------------
echo ---------- Validating Sources -----------
echo -----------------------------------------
echo
echo Preparing the source directory ${src_dir}
echo
# If neither SVN nor Git assume local
# DEBUG Note that this approach precludes a default for either as the default would result in decisions
if ! [ "$svn_svr" -o "$git_svr" -o "$git_dir" ]; then
# This suggested syntax from shellcheck does not work - test again
#if [ "$svn_svr" ] || [ "$git_svr" ] || [ "$git_dir" ]; then
echo Using local sources
if [ $(find $src_dir -maxdepth 0 -empty) ]; then
echo Requested $src_dir is empty
exit 1
elif [ -f "${src_dir}/sys/${target}/conf/newvers.sh" ]; then
echo ${src_dir} does not appear to contain a FreeBSD source tree
exit 1
fi # End if src_dir empty
elif [ "$svn_svr" -a "$git_svr" ]; then
#elif [ "$svn_svr" ] && [ "$git_svr" ]; then
echo Specify either -S\(vn\) or -G\(it\) but not both
exit 1
elif [ "$git_svr" -a "$git_dir" ]; then
#elif [ "$git_svr" ] && [ "$git_dir" ]; then
echo Specify either a -G\(it\) server or -g\(ig\) directory but not both
exit 1
fi # End if svn_svr git_svr git_dir preflight
if [ "$svn_svr" ]; then
if [ $(which svnlite) ]; then
svn_app=$(which svnlite)
elif [ $(which svn) ]; then
svn_app=$(which svnlite)
else
echo Neither svnlite nor svn found
exit 1
fi # End if which svn_app
# SVN strings
if [ "$src_rev" ]; then
src_rev_string=" -r $src_rev "
else
src_rev_string=""
fi # End if src_rev
# Check if src_dir is empty with find $src_dir -maxdepth 0 -empty
if [ $(find $src_dir -maxdepth 0 -empty) ]; then
echo
echo Checking out ${src_rev}/${src_branch} with $svn_app co $src_rev_string ${svn_svr}/${src_branch} ${src_dir}
echo Logging to ${log_dir}/svnco.log
if [ "$tmpfs_host" ]; then
\time -h ssh $tmpfs_host $svn_app co $src_rev_string ${svn_svr}/${src_branch} ${src_dir} >${log_dir}/svnco.log || {
echo SVN checkout failed
exit 1
}
else
\time -h $svn_app co $src_rev_string ${svn_svr}/${src_branch} ${src_dir} >${log_dir}/svnco.log || {
echo SVN checkout failed
exit 1
}
fi # End if tmpfs_host
touch ${log_dir}/svnco.done
elif [ -f ${log_dir}/svnco.done ]; then
echo Sources appear to be checked out
else
echo ${src_dir} is not empty
echo
echo Type k to keep and continue
echo Type d to delete and check out
echo Type r to revert it to the desired revision
echo Type any other key to exit
read response
if [ "$response" = "k" ]; then
response=""
true
elif [ "$response" = "d" ]; then
response=""
echo Deleting ${src_dir}
#chflags -R 0 ${src_dir}
rm -rf "${src_dir:?}"/*
rm -rf "${src_dir}"/.svn
echo
echo Checking out ${src_rev}/${src_branch} with $svn_app co $src_rev_string ${svn_svr}/${src_branch} ${src_dir}
echo Logging to ${log_dir}/svnco.log
if [ "$tmpfs_host" ]; then
\time -h ssh $tmpfs_host $svn_app co $src_rev_string ${svn_svr}/${src_branch} ${src_dir} >${log_dir}/svnco.log || {
echo SVN checkout failed
exit 1
}
else
\time -h $svn_app co $src_rev_string ${svn_svr}/${src_branch} ${src_dir} >${log_dir}/svnco.log || {
echo SVN checkout failed
exit 1
}
fi # End if tmpfs_host
touch ${log_dir}/svnco.done
elif [ "$response" = "r" ]; then
response=""
echo
echo Updating to $src_rev with svn up $src_rev_string ${src_dir}
echo Logging to ${log_dir}/svnup.log
$svn_app up $src_rev_string ${src_dir} >${log_dir}/svnup.log || {
echo SVN up failed
exit 1
}
echo
echo Reverting ${src_dir} to $src_rev with $svn_app revert -R $src_rev_string ${src_dir}
echo Logging to ${log_dir}/svnrevert.log
$svn_app revert -R ${src_dir} >${log_dir}/svnrevert.log || {
echo SVN revert failed
exit 1
}
echo
echo Cleaning up unversioned files
echo Logging to ${log_dir}/svncleanup.log
$svn_app cleanup --remove-unversioned ${src_dir} >${log_dir}/svncleanup.log || {
echo SVN cleanup failed
exit 1
}
else
exit 1
fi # End if response
fi # End if src_dir empty
fi # End if svn_svr
if [ "$git_svr" -o "$git_dir" ]; then
if ! [ $(which git) ]; then
echo git not found
exit 1
fi # End if which git_app
# Git strings
src_rev_string="$src_rev"
if [ $src_branch ]; then
src_branch_string="-b $src_branch"
fi # End if branch_string
if [ $git_svr ]; then
git_string="git clone $git_svr $src_branch_string ${src_dir}"
elif [ $git_dir ]; then
git_string="git clone --reference $git_dir $git_dir $src_branch_string ${src_dir}"
else
echo Git confusion achieved!
exit 1
fi # End if git_svr or git_dir
# LOOKS LIKE REDIRECTION CONFUSES ITS NUMBER OF ARGUMENTS
# Running git clone --reference /cgit /cgit /build/c122cf32f2a/src 2> /build/c122cf32f2a/obj/gitclone.log
#Logging to /build/c122cf32f2a/obj/gitclone.log
#fatal: Too many arguments.
if [ $(find $src_dir -maxdepth 0 -empty) ]; then
echo
echo Running $git_string
#echo Logging to ${log_dir}/gitclone.log
\time -h $git_string || {
echo Git clone failed
exit 1
}
touch ${log_dir}/gitclone.done
if [ $src_rev ]; then
echo Changing directory to ${src_dir} to checkout revision $src_rev
cd ${src_dir} || {
echo Change directory failed
exit 1
}
pwd
echo Running git checkout $src_rev
\time -h git checkout $src_rev || {
echo git checkout failed
exit 1
}
echo Running git status
git status
fi # End if src_rev
elif [ -f ${log_dir}/gitclone.done ]; then
echo Sources appear to be cloned
else
echo
echo ${src_dir} not empty
echo
echo Type k to keep and continue
echo Type d to delete it and clone
echo Type any other key to exit
read response
if [ "$response" = "k" ]; then
response=""
true
elif [ "$response" = "d" ]; then
response=""
echo Deleting ${src_dir}
chflags -R 0 ${src_dir}
rm -rf ${src_dir}
echo
echo Running $git_string
echo Logging to ${log_dir}/gitclone.log
\time -h $git_string || {
echo Git clone failed
exit 1
}
touch ${log_dir}/gitclone.done
if [ $src_rev ]; then
echo Changing directory to ${src_dir}
cd ${src_dir} || {
echo Change directory failed
exit 1
}
pwd
echo Running git checkout $src_rev
\time -h git checkout $src_rev || {
echo git checkout failed
exit 1
}
echo Running git status
git status
fi
else
response=""
exit 1
fi # End if response
fi # End if Git and empty
fi # End if git_svr or git_dir
if [ $kernconf ]; then
echo
echo -----------------------------------------
echo ---------- KERNCONF handling ------------
echo -----------------------------------------
echo Requested KERNCONF ${src_dir}/sys/${target}/conf/${kernconf}
if [ "$kernconf" -a "$custom_kernconf" ]; then
# if [ "$kernconf" ] && [ "$custom_kernconf" ]; then
echo Specify either -k or -K KERNCONF options but not both
exit 1
fi # End if kernconf
[ -f ${src_dir}/sys/${target}/conf/${kernconf} ] || {
echo ${src_dir}/sys/${target}/conf/${kernconf} not found
exit 1
}
echo Setting KERNCONF=$kernconf
kernconf_string="KERNCONF=$kernconf"
fi # End if kernconf
if [ $custom_kernconf ]; then
echo
echo -----------------------------------------
echo ------- Custom KERNCONF handling --------
echo -----------------------------------------
if [ "$kernconf" -a "$custom_kernconf" ]; then
# if [ "$kernconf" ] && [ "$custom_kernconf" ]; then
echo Specify either -k or -K KERNCONF options but not both
exit 1
fi # End if kernconf of custom_kernconf
[ -f $custom_kernconf ] || {
echo Custom KERNCONF $custom_kernconf not found
exit 1
}
[ -f ${src_dir}/sys/${target}/conf/GENERIC ] || {
echo ${src_dir}/sys/${target}/conf/GENERIC not found
exit 1
}
echo Overwriting the GENERIC KERNCONF with $custom_kernconf
cp $custom_kernconf ${src_dir}/sys/${target}/conf/GENERIC || {
echo $custom_kernconf copy failed
exit 1
}
kernconf_string="KERNCONF=GENERIC"
fi # End if custom_kerconf
if [ $diff_dir ]; then
echo
echo -----------------------------------------
echo ----- Applying diffs if requested -------
echo -----------------------------------------
# This step is "destructive" and would need a source reversion
if [ $(find $diff_dir -maxdepth 0 -empty) ]; then
echo No diffs in ${diff_dir} to apply
elif [ -f ${log_dir}/diff-patching.done ]; then
echo Sources appear to be patched
else
echo Changing directory to ${src_dir}
cd "${src_dir}"
# Moving to make -C ${src_dir}/release syntax elsewhere
# Trickier here
pwd
echo The contents of diff_dir are
echo ${diff_dir}/*
echo
echo Applying patches
for diff in $(echo ${diff_dir}/*); do
echo Running a dry run diff of $diff
# Might need svn patch?
echo patch -C \< $diff
if [ $(patch -C <$diff) ]; then
echo Diff $diff passed the dry run
diff_basename=$(basename $diff)
echo Applying and Logging to ${log_dir}/diff-${diff_basename}.log with
echo patch \< $diff >${log_dir}/diff-${diff_basename}.log
echo Apply diff $diff
patch <$diff >${log_dir}/diff-${diff_basename}.log
else
echo Diff $diff failed to apply
exit 1
fi # End diff success