etcupdate: beta version #790

tschettervictor · 2025-01-04T18:13:42Z

Add subcommand "etcupdate"

This will simply use the built in "bootstrap" command to bootstrap the "src" version of a release, then create a tarball for it ONCE. This tarball is then used to update (includes dry run) a specifie jail to a specified RELEASE version of etc.

To test.

Install a jail with 13.4-RELEASE
Bootstrap an etcupdate release using "bastille etcupdate bootstrap 14.2-RELEASE
(make sure its a higher version than the jail)
Change the RELEASE of the jail from 13.4 to 14.0 or whichever higher version you choose
Do an etcupdate by "bastille etcupdate jailname update 14.0-RELEASE" -d for dry run
Do a "bastille etcupdate jailname resolve" to resolve conflicts

Confirm the etc files updated properly and the resolve mode works as expected.
Logs are inside the jail at /var/db/etcupdate

Add subcommand "etcupdate" This will simply use the built in "bootstrap" command to bootstrap the "src" version of a release, then create a tarball for it ONCE. This tarball is then used to update (includes dry run) a specifie jail to a specified RELEASE version of etc.

bmac2 · 2025-01-05T16:10:30Z

uses built in etcupdate which is installed by default on freebsd. This version is cleaner than 660. I think this method should be adopted over 660 which kind of recreated the wheel.

@yaazkal

scoobybejesus · 2025-01-05T18:37:26Z

My jail is named bs. I tried kinda monkeypatching my pkg version, and I couldn't get past:

$ bastille etcupdate -d bs 14.2-RELEASE
[_jail]: --dry-run
Failed to extract new tree.

Considering I had to add things into common.sh, I thought maybe something hidden was going on that I couldn't see, so then I went ahead and cloned the repo:

$ git clone https://github.com/BastilleBSD/bastille.git
$ cd bastille/
$ git fetch origin pull/790/head:etcbeta
$ git checkout etcbeta

And the same thing happened.

$ ./usr/local/bin/bastille etcupdate -d bs 14.2-RELEASE
[_jail]: --dry-run
Failed to extract new tree.

Any quick way to get a more verbose output? I'm not sure if I'm doing anything wrong. All the above is run as root.

$ freebsd-version -kru
14.2-RELEASE
14.2-RELEASE
14.2-RELEASE

My only guess, which I suppose I will go ahead and try, is that perhaps the jail should be stopped first.

tschettervictor · 2025-01-05T20:18:34Z

Need to bootstrap first.

bastille etcupdate bootstrap RELEASE

scoobybejesus · 2025-01-06T04:22:24Z

I know this style of "debugging" is probably annoying, but I don't know how to proceed further. I just get this:

$ ./usr/local/bin/bastille etcupdate bootstrap 14.2-RELEASE
Failed to build tree.
Failed to build etcupdate tarball "14.2-RELEASE.tbz2"

tschettervictor · 2025-01-06T04:37:11Z

I know this style of "debugging" is probably annoying, but I don't know how to proceed further. I just get this:
$ ./usr/local/bin/bastille etcupdate bootstrap 14.2-RELEASE

Failed to build tree.

Failed to build etcupdate tarball "14.2-RELEASE.tbz2"

Can you add "set -x" to the start of the etcupdate file?

scoobybejesus · 2025-01-06T04:52:48Z

I actually don't know the intended way to run the master branch. I realized that while I was running the bastille from the git repo, it's just assuming I've got everything installed in /usr/local/, which is where the pkg version is, so it's sourcing /usr/local/share/bastille/common.sh and running usr/local/share/bastille/etcupdate.sh`.

Anyway, I added set -x to the etcupdate.sh I put in /usr/local/share/bastille/ and I don't see much useful here, but perhaps I'm wrong.

$ bastille etcupdate bootstrap 14.2-RELEASE
+ . /usr/local/share/bastille/common.sh
+ COLOR_RED=''
+ COLOR_GREEN=''
+ COLOR_YELLOW=''
+ COLOR_RESET=''
+ [ -z '' -a -t 1 ]
+ enable_color
+ . /usr/local/share/bastille/colors.pre.sh
+ COLOR_RESET='\033[0;0m'
+ COLOR_BOLD='\033[1m'
+ COLOR_UNDER='\033[4m'
+ COLOR_BLINK='\033[5m'
+ COLOR_INVERSE='\033[7m'
+ COLOR_BLACK='\033[0;30m'
+ COLOR_RED='\033[0;31m'
+ COLOR_GREEN='\033[0;32m'
+ COLOR_BROWN='\033[0;33m'
+ COLOR_BLUE='\033[0;34m'
+ COLOR_MAGENTA='\033[0;35m'
+ COLOR_CYAN='\033[0;36m'
+ COLOR_LIGHT_GRAY='\033[0;37m'
+ COLOR_DARK_GRAY='\033[1;30m'
+ COLOR_LIGHT_RED='\033[1;31m'
+ COLOR_LIGHT_GREEN='\033[1;32m'
+ COLOR_YELLOW='\033[1;33m'
+ COLOR_LIGHT_BLUE='\033[1;34m'
+ COLOR_LIGHT_MAGENTA='\033[1;35m'
+ COLOR_LIGHT_CYAN='\033[1;36m'
+ COLOR_WHITE='\033[1;37m'
+ COLOR_BG_BLACK='\033[40m'
+ COLOR_BG_RED='\033[41m'
+ COLOR_BG_GREEN='\033[42m'
+ COLOR_BG_BROWN='\033[43m'
+ COLOR_BG_BLUE='\033[44m'
+ COLOR_BG_MAGENTA='\033[45m'
+ COLOR_BG_CYAN='\033[46m'
+ COLOR_BG_LIGHT_GRAY='\033[47m'
+ . /usr/local/etc/bastille/bastille.conf
+ bastille_prefix=/usr/local/bastille
+ bastille_backupsdir=/usr/local/bastille/backups
+ bastille_cachedir=/usr/local/bastille/cache
+ bastille_jailsdir=/usr/local/bastille/jails
+ bastille_releasesdir=/usr/local/bastille/releases
+ bastille_templatesdir=/usr/local/bastille/templates
+ bastille_logsdir=/usr/local/bastille/logs
+ bastille_pf_conf=/etc/pf.conf
+ bastille_sharedir=/usr/local/share/bastille
+ bastille_bootstrap_archives=base
+ bastille_tzdata=Etc/UTC
+ bastille_resolv_conf=/etc/resolv.conf
+ bastille_url_freebsd=http://ftp.freebsd.org/pub/FreeBSD/releases/
+ bastille_url_hardenedbsd=http://installer.hardenedbsd.org/pub/hardenedbsd/
+ bastille_zfs_enable=YES
+ bastille_zfs_zpool=zroot
+ bastille_zfs_prefix=bastille
+ bastille_zfs_options='-o compress=lz4 -o atime=off'
+ bastille_compress_xz_options='-0 -v'
+ bastille_decompress_xz_options='-c -d -v'
+ bastille_compress_gz_options='-1 -v'
+ bastille_decompress_gz_options='-k -d -c -v'
+ bastille_network_loopback=bastille0
+ bastille_network_pf_ext_if=ext_if
+ bastille_network_pf_table=jails
+ bastille_network_shared=''
+ bastille_network_gateway=''
+ bastille_template_base=default/base
+ bastille_template_empty=''
+ bastille_template_thick=default/thick
+ bastille_template_clone=default/clone
+ bastille_template_thin=default/thin
+ bastille_template_vnet=default/vnet
+ [ 2 -lt 2 ]
+ [ 2 -gt 3 ]
+ [ 2 -gt 0 ]
+ [ -z 14.2-RELEASE ]
+ RELEASE=14.2-RELEASE
+ bootstrap_etc_release 14.2-RELEASE
+ local '_release=14.2-RELEASE'
+ sysrc -f /usr/local/etc/bastille/bastille.conf bastille_bootstrap_archives
+ awk '-F: ' '{print $2}'
+ local '_current=base'
+ ls -A /usr/local/bastille/releases/14.2-RELEASE/usr/src
+ bootstrap_etc_tarball 14.2-RELEASE
+ local '_release=14.2-RELEASE'
+ [ ! -f /usr/local/bastille/cache/14.2-RELEASE.tbz2 ]
+ etcupdate build -d /tmp/etcupdate -s /usr/local/bastille/releases/14.2-RELEASE/usr/src /usr/local/bastille/cache/14.2-RELEASE.tbz2
Failed to build tree.
+ error_exit 'Failed to build etcupdate tarball "14.2-RELEASE.tbz2"'
+ error_notify Failed to build etcupdate tarball '"14.2-RELEASE.tbz2"'
+ echo -e '\033[0;31mFailed to build etcupdate tarball "14.2-RELEASE.tbz2"\033[0;0m'
Failed to build etcupdate tarball "14.2-RELEASE.tbz2"
+ exit 1

Despite the above not being too interesting to me, digging further, there's this:

$ tail /tmp/etcupdate/log
>>> build command: /usr/local/bastille/cache/14.2-RELEASE.tbz2
>>> Building tree at /tmp/etcupdate/etcupdate-FXtgLZM with make  -DNO_FILEMON
make: don't know how to make _obj. Stop

make: stopped in /usr/local/bastille/releases/14.2-RELEASE/usr/src

Hopefully I'm adding this quickly enough. Sorry for the edit:

$ ls /usr/local/bastille/releases/14.2-RELEASE/usr/src

The source dir is empty. I bootstrapped 14.2-RELEASE, but I wonder if on the previous version of bastille, it didn't pull the src/ dir, but now it does?

tschettervictor · 2025-01-06T04:59:39Z

That was totally my bad. I had made a booboo. Try again.

tschettervictor · 2025-01-06T05:00:06Z

This bootstraps src on its own for you.

scoobybejesus · 2025-01-06T21:58:45Z

First, this seems to be a successful implementation, but it does raise questions.

Reading through man etcupdate, it says:

     If a file can be updated without generating a conflict, etcupdate
     will update the file automatically.  If the local changes to a file
     conflict with the changes made to a file in the source tree, then a merge
     conflict is generated.  The conflict must be resolved after the merge has
     finished.  The etcupdate utility will not perform a new merge until all
     conflicts from an earlier merge are resolved.

I mention the above because I question whether it's true. Is there a separate list of conflicts somewhere? The conflicting files appear to be unaltered.

It says the following about Default Mode:

     If a file is changed in the “current” tree, then
     etcupdate will attempt to merge the changes into the version of the file
     in the destination directory.  If the merge encounters conflicts, then a
     version of the file with conflict markers will be saved for future
     resolution.  If the merge does not encounter conflicts, then the merged
     version of the file will be saved in the destination directory.  If
     etcupdate is not able to safely merge in changes to a file other than a
     merge conflict, it will generate a warning.

I don't see file versions saved with conflict markers. Should I?

I had several files marked with a C for Conflict, namely:

  C /etc/group
  C /etc/master.passwd
  C /root/.cshrc
  C /etc/pam.d/{various I commented out opie items}

I didn't find conflict markers in any of these files. As far as I can tell, those marked with a C were skipped over completely. Is this the desired behavior?

Follow-up: should there be an option to run the command a second time in Resolve Mode (to handle conflicts)?

Separately, it could be worthwhile to specify a separate logfile for each jail. The only log I see is 4,558 lines, and that's just the bootstrap. I don't see a log for the etcupdate on the jail. All I see is what printed to the console.

tschettervictor · 2025-01-06T22:08:10Z

Logs should be in /var/db/etcupdate inside each jail.

Yes we could add a "resolve" mode.

This is just beta to see if it will satisfy a basic etcupdate.

scoobybejesus · 2025-01-06T22:27:31Z

Yep, sure enough everything was where you said. This is great. I look forward to seeing if other testers find anything noteworthy. Thank you.

usr/local/share/bastille/etcupdate.sh

bmac2 · 2025-01-09T17:30:11Z

Logs should be in /var/db/etcupdate inside each jail.

Yes we could add a "resolve" mode.

This is just beta to see if it will satisfy a basic etcupdate.

have we added a resolve mode to this? What else is missing that is in #660 ??

@scoobybejesus @michael-o @yaazkal

tschettervictor · 2025-01-09T17:51:14Z

Logs should be in /var/db/etcupdate inside each jail.

Yes we could add a "resolve" mode.

This is just beta to see if it will satisfy a basic etcupdate.

have we added a resolve mode to this? What else is missing that is in #660 ??

@scoobybejesus @michael-o @yaazkal

It hasn't been added. I'll look into it. It's just a switch inside the built in etcupdate command.

tschettervictor · 2025-01-09T18:39:41Z

@scoobybejesus I've added a resolve mode. Syntax has changed slightly.

It is now bastille etcupdate jailname [update RELEASE|resolve]

So after the jail name comes the command "update or resolve" then the RELEASE.
No RELEASE is necessary when using resolve mode.

Can you test?

Also, I've addes a [-f|--force] mode that will re-bootstrap. This will be helpful when doing updates to the patch level of a RELEASE and wanting to create a new tarball.

tschettervictor · 2025-01-09T22:13:51Z

Added a diff mode also in case someone would like to compare the current tree with what the jails tree looks like.

usr/local/share/bastille/etcupdate.sh

bmac2 · 2025-01-10T02:37:41Z

works as long as your jail name is text. I have a jail named 134 and it will not update it.
Jail named temp1 worked perfect.

@yaazkal @scoobybejesus @tschettervictor

scoobybejesus · 2025-01-10T02:49:27Z

This is working well for me. A couple comments:

bastille etcupdate -d [jail] resolve will print the usage for etcupdate

$ bastille etcupdate -d bs resolve
[_jail]: resolve --dry-run
usage: etcupdate [-npBFN] [-d workdir] [-r | -s source | -t tarball]
                 [-A patterns] [-D destdir] [-I patterns] [-L logfile]
                 [-M options] [-m make]
       etcupdate build [-BN] [-d workdir] [-s source] [-L logfile] [-M options]
                 [-m make] <tarball>
       etcupdate diff [-d workdir] [-D destdir] [-I patterns] [-L logfile]
       etcupdate extract [-BN] [-d workdir] [-s source | -t tarball]
                 [-D destdir] [-L logfile] [-M options] [-m make]
       etcupdate resolve [-p] [-d workdir] [-D destdir] [-L logfile]
       etcupdate revert [-d workdir] [-D destdir] [-L logfile] file ...
       etcupdate status [-d workdir] [-D destdir]

bastille etcupdate -d [jail] diff will print the same thing either way (I guess you don't need a dry run for a diff, but still)
Through the file, there's info "[_jail]: whatever". Should it be info "[$_jail]: whatever" (or maybe better, info "[${_jail}]: whatever") so it prints the jail name?
Similar to above, instead of info "[${_jail}]: update", is it preferable to include etcupdate in the printout, so to have info "[${_jail}]: etcupdate update" and so on for info "[${_jail}]: etcupdate resolve" and info "[${_jail}]: etcupdate diff", etc?
As a tangential question from someone who hasn't run etcupdate on their own before, is it normal for all conflicts to be resolved, yet there is still a printout from etcupdate diff? For example, the formatting in .cshrc changed, and the conflict markers didn't seem to care about anything except for the top of the file, so it was simple to resolve, but I've got a big diff from that file still. I assume it's fine for me to ignore. If bastille writes up documentation on this, perhaps this is a noteworthy point.
As of now, I have only tested this on jails with names that have letters only in their names.

Quickly, here's a quick how-to:

$ bastille etcupdate bootstrap 14.2-RELEASE
  ... wait a couple minutes ...
$ bastille etcupdate -d $jail update 14.2-RELEASE  #optional, if you want to see most results ahead of time
$ bastille etcupdate $jail update 14.2-RELEASE
$ bastille etcupdate $jail diff    #optional
$ export EDITOR=vim  #optional
$ bastille etcupdate $jail resolve
## misc other tasks, such as bastille console $jail, followed by newaliases
$ bastille etcupdate $jail update 14.2-RELEASE  #optional, to double check that there's no output

tschettervictor · 2025-01-10T07:15:37Z

-d mode might not work with "diff" and "resolve"
3 and 4 noted and updated

scoobybejesus · 2025-01-10T22:26:42Z

Forgive me for treading what might already be common ground to many.

Consider the two following scenarios:

bastille bootstrap 14.1-RELEASE
bastille etcupdate bootstrap 14.1-RELEASE
ls -alh /usr/local/bastille/cache/14.1-RELEASE

Versus

bastille bootstrap 14.1-RELEASE
bastille update 14.1-RELEASE
bastille etcupdate bootstrap 14.1-RELEASE
ls -alh /usr/local/bastille/cache/14.1-RELEASE

In both cases, the final output is identical:

/usr/local/bastille/cache$ ls -alh 14.1-RELEASE
total 413527
drwxr-xr-x   2 root wheel    5B Jan 10 17:12 .
drwxr-xr-x  11 root wheel   13B Jan 10 17:14 ..
-rw-r--r--   1 root wheel  1.0K May 31  2024 MANIFEST
-rw-r--r--   1 root wheel  198M May 31  2024 base.txz
-rw-r--r--   1 root wheel  205M May 31  2024 src.txz

Observation: the base.txz and src.txz and are identical, even after an upgrade. And they are the files from when 14.1-RELEASE was originally released (the release announcement came a few days later).

I had intended to recommend that in the bastille documentation -f should be recommended after updating a jail, i.e.,:

For a newly bootstrapped release:

bastille bootstrap [RELEASE]
bastille etcupdate bootstrap [RELEASE]
bastille etcupdate [JAIL] update [RELEASE]

Or for an existing release that has been or will be updated with patches, force a fresh bootstrap:

bastille update [RELEASE]
bastille etcupdate -f bootstrap [RELEASE]
bastille etcupdate [JAIL] update [RELEASE]

I intended to ask whether -f should not only recreate {...}/cache/${_release}.tbz2 but also replace {...}/cache/src.txz and repopulate {...}/releases/${RELEASE}/usr/src/.

Now I find myself wondering if -f is needed at all.

tschettervictor · 2025-01-10T22:43:58Z

I already answered that for myself. src.txz is indeed identical, so I kept the -f out of there at least.

I'm curious whether /usr/src changes after an update or upgrade of a release?

scoobybejesus · 2025-01-11T00:27:16Z

After deleting/destroying the directories/datasets for 14.1-RELEASE and setting `bastille_bootstrap_archives="base src", I checked:

bastille bootstrap 14.1-RELEASE
ls -alh /usr/local/bastille/releases/14.1-RELEASE/usr/src
bastille update 14.1-RELEASE
ls -alh /usr/local/bastille/releases/14.1-RELEASE/usr/src

Between steps 2 and 4, the files in /usr/src didn't change.

tschettervictor · 2025-01-11T00:29:27Z

So it seems src stays the same for each major version then.

Interesting.

scoobybejesus · 2025-01-11T01:54:51Z

It seems the original files at release time are what stick around for the duration of the release, and patches are applied on top. And it seems no patches are applied to src/, which leads me to believe the intended use would be to initialize it as a git directory and then git pull the upstream changes. I wonder the extent to which that would be doable and/or desirable.

In any event, it seems like something that could be added later. Though it would be nice to figure it out now.

tschettervictor · 2025-01-11T02:25:40Z

It's nice to at least have the option to force create a new tarball if (in any case) src gets updated.

bmac2 · 2025-01-13T05:28:41Z

@scoobybejesus @tschettervictor I have tested this as is. Is there anything additional to do on this one? Have you both tested this well??? Especially @scoobybejesus. Want to get this merged to main since it didn't make the release.

tschettervictor · 2025-01-13T15:38:32Z

No issues on my end.

tschettervictor · 2025-01-13T15:41:59Z

Only thing missing is validating the release names, but Bastille bootstrap already does that for us.

bmac2 · 2025-01-15T05:35:41Z

tested by @scoobybejesus @tschettervictor and myself. Works flawlessly doing what it is supposed to.

tschettervictor mentioned this pull request Jan 4, 2025

New 'etcupdate' sub command #660

Closed

etcupdate: fix !

b7ac062

etcupdate: add notice for building tarball

50c5e8c

Merge branch 'BastilleBSD:master' into etcupdate

bbd1de8

tschettervictor mentioned this pull request Jan 7, 2025

Debug mode option #791

Closed

scoobybejesus reviewed Jan 8, 2025

View reviewed changes

usr/local/share/bastille/etcupdate.sh Show resolved Hide resolved

etcupdate: error when RELEASE not bootstrapped

0d09ac9

etcupdate: add resolve mode

9c79f13

tschettervictor added 3 commits January 9, 2025 15:10

etcupdate: add diff mode

6ce4191

etcupdate: help message include diff mode

b90a83b

common: update set_target_single

e6e60a3

scoobybejesus reviewed Jan 9, 2025

View reviewed changes

usr/local/share/bastille/etcupdate.sh Outdated Show resolved Hide resolved

etcupdate: fix accidentally deleted error message

e4b5273

scoobybejesus reviewed Jan 9, 2025

View reviewed changes

usr/local/share/bastille/etcupdate.sh Show resolved Hide resolved

tschettervictor added 3 commits January 9, 2025 16:30

etcupdate: code optimize (usage if no RELEASE)

8882c23

etcupdate: fix “fi”

cca43cb

etcupdate: fix ;; spacing

894e5ef

etcupdate: jail var missing

cd33036

tschettervictor added 2 commits January 10, 2025 00:18

etcupdate: remove -n option from resolve mode

397b13b

etcupdate: warn on -d for diff/resolve

86c5b49

etcupdate: add “”

cab6f1a

bmac2 mentioned this pull request Jan 14, 2025

Bastille console does not work after upgrading jail to 14.1-RELEASE #704

Open

bmac2 merged commit 49a7a64 into BastilleBSD:master Jan 15, 2025
1 check passed

tschettervictor deleted the etcupdate branch January 15, 2025 14:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcupdate: beta version #790

etcupdate: beta version #790

tschettervictor commented Jan 4, 2025 •

edited by bmac2

Loading

bmac2 commented Jan 5, 2025

scoobybejesus commented Jan 5, 2025

tschettervictor commented Jan 5, 2025

scoobybejesus commented Jan 6, 2025

tschettervictor commented Jan 6, 2025

scoobybejesus commented Jan 6, 2025 •

edited

Loading

tschettervictor commented Jan 6, 2025

tschettervictor commented Jan 6, 2025

scoobybejesus commented Jan 6, 2025

tschettervictor commented Jan 6, 2025

scoobybejesus commented Jan 6, 2025

bmac2 commented Jan 9, 2025

tschettervictor commented Jan 9, 2025

tschettervictor commented Jan 9, 2025

tschettervictor commented Jan 9, 2025

bmac2 commented Jan 10, 2025 •

edited

Loading

scoobybejesus commented Jan 10, 2025

tschettervictor commented Jan 10, 2025

scoobybejesus commented Jan 10, 2025

tschettervictor commented Jan 10, 2025

scoobybejesus commented Jan 11, 2025

tschettervictor commented Jan 11, 2025

scoobybejesus commented Jan 11, 2025

tschettervictor commented Jan 11, 2025

bmac2 commented Jan 13, 2025

tschettervictor commented Jan 13, 2025

tschettervictor commented Jan 13, 2025

bmac2 commented Jan 15, 2025

etcupdate: beta version #790

etcupdate: beta version #790

Conversation

tschettervictor commented Jan 4, 2025 • edited by bmac2 Loading

bmac2 commented Jan 5, 2025

scoobybejesus commented Jan 5, 2025

tschettervictor commented Jan 5, 2025

scoobybejesus commented Jan 6, 2025

tschettervictor commented Jan 6, 2025

scoobybejesus commented Jan 6, 2025 • edited Loading

tschettervictor commented Jan 6, 2025

tschettervictor commented Jan 6, 2025

scoobybejesus commented Jan 6, 2025

tschettervictor commented Jan 6, 2025

scoobybejesus commented Jan 6, 2025

bmac2 commented Jan 9, 2025

tschettervictor commented Jan 9, 2025

tschettervictor commented Jan 9, 2025

tschettervictor commented Jan 9, 2025

bmac2 commented Jan 10, 2025 • edited Loading

scoobybejesus commented Jan 10, 2025

tschettervictor commented Jan 10, 2025

scoobybejesus commented Jan 10, 2025

tschettervictor commented Jan 10, 2025

scoobybejesus commented Jan 11, 2025

tschettervictor commented Jan 11, 2025

scoobybejesus commented Jan 11, 2025

tschettervictor commented Jan 11, 2025

bmac2 commented Jan 13, 2025

tschettervictor commented Jan 13, 2025

tschettervictor commented Jan 13, 2025

bmac2 commented Jan 15, 2025

tschettervictor commented Jan 4, 2025 •

edited by bmac2

Loading

scoobybejesus commented Jan 6, 2025 •

edited

Loading

bmac2 commented Jan 10, 2025 •

edited

Loading