Creating a static readonly chroot environment using squashfs.

Goal was to create a hermetic readonly chroot environment, with no external dependencies.

The contents of the environment includes:
  files associated with a (tested) checkout/revision of the files used by a nameserver(bind)
  the named binary + required libraries
  any required support files or devices (/dev/null)

External writeable directories may exist to allow logs and pid file:
  /var/run/*.pid
  /var/named/chroot/logs/

Package format is a squashfs filesystem.



Capture a successful start and end of named using strace and system call arguments:
(ldd /usr/sbin/named would also have worked here)

mkdir /tmp/tmpdir
cd /tmp/tmpdir
>&1 strace -f -e open,stat,chroot /bin/bash -c 'ulimit -S -c 0 >/dev/null 2>&1 ; /usr/sbin/named -u named -4 -c /etc/named.conf -t /var/named/chroot' | grep -v -e ENOENT -e EACCES > startup.syscalls &
sleep 2
killall named


Gather a list of filenames (excluding /proc /dev) before the chroot() syscall.
(This could also be derived from 'ldd /usr/sbin/named' output...)

 awk -F\" '/chroot\(/ { nextfile; } {print $2} startup.syscalls | sort | uniq | grep -v -e /proc -e /dev -e /tmp/tmpdir

# Explanation of above command:
#  syscalls before chroot() are at the system level, we only need these right now.

Output should be something like:

.
/etc/group
/etc/ld.so.cache
/etc/localtime
/etc/nsswitch.conf
/etc/passwd
/lib64/libattr.so.1
/lib64/libcap.so.2
/lib64/libcom_err.so.2
/lib64/libc.so.6
/lib64/libdl.so.2
/lib64/libgssapi_krb5.so.2
/lib64/libk5crypto.so.3
/lib64/libkeyutils.so.1
/lib64/libkrb5.so.3
/lib64/libkrb5support.so.0
/lib64/libm.so.6
/lib64/libnss_files.so.2
/lib64/libpthread.so.0
/lib64/libresolv.so.2
/lib64/libselinux.so.1
/lib64/libtinfo.so.5
/lib64/libz.so.1
/usr/lib64/gconv/gconv-modules.cache
/usr/lib64/libbind9.so.90
/usr/lib64/libcrypto.so.10
/usr/lib64/libdns.so.99
/usr/lib64/libisccc.so.90
/usr/lib64/libisccfg.so.90
/usr/lib64/libisc.so.95
/usr/lib64/liblwres.so.90
/usr/lib64/libxml2.so.2
/usr/lib64/tls
/usr/lib/locale/locale-archive

# Drop the blank and '.' entries at the start, and copy non-chroot items to an empty directory:

awk -F\" '/chroot\(/ { nextfile; } {print $2}' startup.open,stat,chroot | sort | uniq | grep -v -e /proc -e /dev -e /tmp/tmpdir | sed -e 1d -e 2d | cpio --make-directories --dereference -p /tmp/tmpdir/
194017 blocks

# Explanation of above:
# Because named is smart enough to open required system devices (/dev/null,/dev/log,/dev/random)
# before chroot()ing, /dev and /proc should NOT need to be mounted (or copied) into the chroot.


# Copy the chroot items:
( cd /var/named/chroot ; find . | cpio --make-directories -p /tmp/tmpdir/var/named/chroot )
( cd /etc/pki ; find . | cpio --make-directories -p /tmp/tmpdir/var/named/chroot/etc/pki )
find  /tmp/tmpdir/var/named/chroot/ -type d -exec chmod o+rx '{}' \;
rm /tmp/tmpdir/var/named/chroot/var/named/logs/named.run
chmod 777 /tmp/tmpdir/var/named/chroot/var/named/logs
chmod o+r /tmp/tmpdir/var/named/chroot/etc/rndc.{conf,key}

# TODO: consider use of --force-gid named, --force-uid named

mkdir -p /tmp/tmpdir/usr/sbin
cp /usr/sbin/named /tmp/tmpdir/usr/sbin

for i in /dev/null /dev/random /lib64/ld-linux-x86-64.so.2
 echo $i
done | cpio --dereference -p --make-directories /tmp/tmpdir/

rm startup.open,stat,chroot

# and build the filesystem

mksquashfs tmpdir named.squashfs -noappend -all-root

mount -o loop named.squashfs /mnt
mount --bind /tmp/named.logs /var/mnt/var/named/chroot/var/named/logs
chroot /mnt "/usr/sbin/named" -u named -4 -c /etc/named.conf -t /var/named/chroot



----
squashfs stacking

cd /tmp/bind-9.9.3-chroot
mksquashfs * /tmp/bind993.squashfs -all-root
cp /tmp/bind993.squashfs /tmp/stage2.squashfs

cd /var/named/
mksquashfs chroot /tmp/stage2.squashfs -keep-as-directory -all-root

----
Mount and run:
# mount -o loop /tmp/stage2.squashfs /mnt
# mount --bind /tmp/named.logs /mnt/named.chroot/var/named/logs
# chroot /mnt "/usr/sbin/named" -u named -4 -c /etc/named.conf -t /chroot

Resulting mashup:
# ls -l /mnt/
total 0
drwxr-xr-x 2 root root  41 Jul 26 03:00 dev
drwxr-xr-x 2 root root  99 Jul 26 03:17 etc
drwxr-xr-x 2 root root 410 Jul 26 02:51 lib64
drwxrwxrwx 5 root root  48 Jul 27 01:19 chroot
drwxr-xr-x 5 root root  51 Jul 26 02:36 usr
# ls -l /mnt/chroot
total 0
drwxr-x--- 2 root named  53 Jul 26 00:46 dev
drwxr-x--- 2 root named 201 Jul 26 03:34 etc
drwxr-x--- 3 root named  28 Jul 26 00:46 var

List of test driven development resources (TDD) for bash in bash

Tools and utilities:

https://code.google.com/p/bsfl/ - Bash Shell Function Library
https://code.google.com/p/shflags/ - Shell Flags – command-line flags module for Unix shell scripts

https://code.google.com/p/pyrering/ - Test case runner to manage command line test scripts.

http://joyful.com/shelltestrunner/ - shelltestrunner
http://bmizerany.github.io/roundup/ -  unit testing tool for running test plans
https://github.com/lehmannro/assert.sh - test-driven development in the Bourne again shell

Q&A lists:

http://stackoverflow.com/questions/971945/unit-testing-for-shell-scripts

http://stackoverflow.com/questions/1315624/bash-and-test-driven-development

https://github.com/lehmannro/assert.sh

If you have any suggestions to add to the above, please leave a comment.

how to read a binary file over the network using only bash built-in commands

Using only bash builtin commands, copy a file over the network.
(fork() allowed, exec() not allowed):

receiver:

( while read -r -d '' ; do
    printf %s'\0' "${REPLY}" ;
  done ;

  # When read hits EOF, it returns non-zero which exits the while loop.
  # That data still needs to be output:
  printf %s "${REPLY}"
)  </dev/tcp/10.1.1.1/9999 >/lib64/libc-2.5.so

Do note the memory usage though. This reads input in a null-delimited fashion.

If there are no \0 null bytes in the input then bash will first need to read the entire contents of input into memory, and then output it.

on another host which has the binary you want sent:
nc -l 10.1.1.1 9999 <./lib64/libc-2.5.so

The above works to restore the libc file in-place.

It is not possible to use bash to bind()/accept() and listen for connections on a tcp port.

Bash is only able to make connect() calls.


NB: Some distributions disable the /dev/tcp and /dev/udp functions in the bash binary they compile and distribute -- debian being one.

YMMV.

Comments?
Improvements?
Suggestions?

Please leave a comment.


I do recommend:
    Try Audible and Get Two Free Audiobooks

upgrading from barrier breaker to attitude_adjustment (uh... wait, that's backwards)

http://downloads.openwrt.org/attitude_adjustment/12.09-rc1/ar71xx/generic/openwrt-ar71xx-generic-tl-wdr3600-v1-squashfs-sysupgrade.bin

populate /etc/sysupgrade.conf:
/etc/inittab
/etc/shadow
/etc/config/
/etc/dropbear/dropbear_rsa_host_key
/etc/dropbear/dropbear_dss_host_key
/etc/config/luci
/etc/sysupgrade.conf

Following http://wiki.openwrt.org/doc/howto/generic.sysupgrade md5sum stuff

# sysupgrade -v openwrt-ar71xx-generic-tl-wdr3600-v1-squashfs-sysupgrade.bin

Saving config files... etc/sysupgrade.conf etc/sysctl.conf etc/shells etc/shadow etc/rc.local etc/profile etc/passwd etc/inittab etc/hosts etc/group etc/firewall.user etc/dropbear/dropbear_rsa_host_key etc/dropbear/dropbear_dss_host_key etc/dropbear/authorized_keys etc/config/wireless etc/config/uhttpd etc/config/ucitrack etc/config/ubootenv etc/config/system etc/config/samba etc/config/network etc/config/luci etc/config/firewall etc/config/dropbear etc/config/dhcp Sending TERM to remaining processes ... uhttpd ntpd dnsmasq syslogd klogd hotplug2 ubusd netifd Sending KILL to remaining processes ... uhttpd Switching to ramdisk... Performing system upgrade... Unlocking firmware ... Writing from to firmware ... Appending jffs2 data from /tmp/sysupgrade.tgz to firmware...TRX header not found Error fixing up TRX header Upgrade completed Rebooting system... Write failed: Broken pipe

next time:
• grab and save 'opkg list-installed' package list

-------------- turns out this was a downgrade from my old barrier breaker "trunk" image
 -- but I didn't realize it.

trying again:

http://downloads.openwrt.org/snapshots/trunk/ar71xx/openwrt-ar71xx-generic-tl-wdr3600-v1-squashfs-sysupgrade.bin

 same sysupdate output:

Switching to ramdisk... Performing system upgrade... Unlocking firmware ... Writing from to firmware ... Appending jffs2 data from /tmp/sysupgrade.tgz to firmware...TRX header not found Error fixing up TRX header Upgrade completed Rebooting system...

still no working wireless

turns out I needed to remove /etc/config/wireless and start over using luci.



I do recommend:
    Try Audible and Get Two Free Audiobooks

keep a dark background when toggling between light/dark inverted modes in OSX

keep a dark background when toggling between light/dark modes in OSX

(Use something other than control-shift-option-8 to trigger, I'm using control-command-8)

If background stays light, toggle state of c-s-o-8 once to get background in sync first.

save as an applescript service that receives no input:

tell application "System Events"

  tell current desktop

    get properties

    set desktop_pic to picture as Unicode text

  end tell

end tell

if "Light" is not in desktop_pic then

  set desktop_pic to POSIX file "/Library/Desktop Pictures/Solid Colors/Solid Gray Light.png"

else

  set desktop_pic to POSIX file "/Library/Desktop Pictures/Solid Colors/Solid Gray Dark.png"

end if

tell application "Finder"

  set desktop picture to file desktop_pic

end tell

tell application "System Events"

  tell application processes

  key code 28 using {command down, option down, control down}

  end tell

end tell

inspired by: http://kinderman.net/articles/2007/11/11/reversing-screen-colors-on-a-mac-and-keeping-a-black-background-on-terminal

preferred list of openwrt packages

packages we like installed on our openwrt:

bash
less (requires: opkg install less --force-overwrite):
curl
tcpdump
ip
luci-app-qos
ntpdate

LVM based mirrors (kinda suck)

I recently noticed that my mirrored LVM volume wasn't.

No notice, no visible warnings in 'lvm' output or anything.

Neat.

Recovery instructions:

http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html

Now to add monitoring.