Commit 2fdfe20a authored by amnesia's avatar amnesia

more improvements on boot time from CD

 * Move boot profiler execution from runlevel S to initramfs to also gather
   files used during those many `live-bottom` scripts.
 * Make the boot profiler provide a ready to use sort list, now in
   /var/log/boot-profile.
 * Implement readahead. That needs real world tests before deciding if it's
   actually an improvment.
 * Backport bootchart from Ubuntu to get accurate measurements. Binary packages
   needs to be manually copied to be used.
 * Update sort list.
 * Update documentation.
parent f29d2467
......@@ -15,6 +15,7 @@
/config/source
/config/binary_local-includes/doc/amnesia/wiki
/config/chroot_local-includes/usr/share/doc/Changelog
/config/chroot_local-includes/usr/share/amnesia/readahead-list
/config/chroot_local-includes/etc/amnesia/version
/.lock
/.stage
......
This source diff could not be displayed because it is too large. You can view the blob instead.
#! /bin/sh
### BEGIN INIT INFO
# Provides: boot-profile
# Required-Start: mountkernfs
# Required-Stop:
# X-Start-Before: udev mountdevsubfs
# Default-Start: S
# Default-Stop:
# Short-Description: profile files used during boot
### END INIT INFO
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="boot profiling system"
NAME=boot-profile
DAEMON=/usr/sbin/$NAME
DAEMON_ARGS="/lib/init/rw/boot-profile"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0
# Exit if profiling is not enable
grep -qw "profile" /proc/cmdline || exit 0
# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh
# Define LSB log_* functions.
. /lib/lsb/init-functions
do_start() {
# Raise maximum inotify watch descriptors
echo 32768 >/proc/sys/fs/inotify/max_user_watches
start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
|| return 1
start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
$DAEMON_ARGS \
|| return 2
}
do_stop() {
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
RETVAL="$?"
[ "$RETVAL" = 2 ] && return 2
start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
[ "$?" = 2 ] && return 2
rm -f $PIDFILE
return "$RETVAL"
}
case "$1" in
start)
[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
do_start
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
stop)
[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
do_stop
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
restart|force-reload)
log_daemon_msg "Restarting $DESC" "$NAME"
do_stop
case "$?" in
0|1)
do_start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
*) log_end_msg 1 ;; # Failed to start
esac
;;
*)
# Failed to stop
log_end_msg 1
;;
esac
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
exit 3
;;
esac
:
../init.d/boot-profile
\ No newline at end of file
#!/bin/sh
set -e
test -e /boot-profile.pid || exit 0
# Wait some time hoping Tor will have bootstrapped
sleep 30
iceweasel &
# Wait some time hoping Iceweasel will be started
sleep 15
sudo kill $(cat /boot-profile.pid)
#!/usr/bin/env python
from pyinotify import WatchManager, Notifier, \
ThreadedNotifier, ProcessEvent, IN_ACCESS
ThreadedNotifier, ProcessEvent, IN_OPEN, IN_ACCESS, IN_CREATE, IN_MOVED_TO
import re
import sys
import atexit
......@@ -12,44 +12,67 @@ import os.path
IGNORE_RE = "^/(tmp|sys|proc|dev|live/cow)"
class ProfileProcessor(ProcessEvent):
def __init__(self, new_profile):
def __init__(self, profile_path):
self.priority = 32767
self.files = {}
self.ignored_files = {}
self.ignore_re = re.compile(IGNORE_RE)
self.new_profile = new_profile
self.profile_path = profile_path
def add_file(self, path):
if ' ' in path:
# Skip path with white spaces: mksquashfs -sort does not
# handle them! fscanf(fd, "%s %d", ...)
return
if not self.files.has_key(path):
self.files[path] = self.priority
self.priority -= 1
def ignore_file(self, path):
self.ignored_files[path] = None
def process_IN_OPEN(self, event):
if not event.dir:
self.add_file(event.pathname)
def process_IN_ACCESS(self, event):
self.add_file(event.pathname)
def process_IN_CREATE(self, event):
self.ignore_file(event.pathname)
def process_IN_MOVED_TO(self, event):
self.ignore_file(event.pathname)
def is_excluded(self, path):
return self.ignore_re.match(path)
def end_profiling(self):
profile = open(self.profile_path, 'w')
priorities = {}
for path, priority in self.files.iteritems():
self.new_profile.write("%-68s %s \n" % (path, priority))
self.new_profile.close()
if not self.ignored_files.has_key(path):
priorities[priority] = path
keys = priorities.keys()
keys.sort(reverse=True)
for key in keys:
profile.write("%-68s %s\n" % (priorities[key][1:], key))
profile.close()
def main():
if len(sys.argv) < 2:
print >>sys.stderr, "usage: %s <new-profile>" % sys.argv[0]
sys.exit(0)
new_profile = open(sys.argv[1], 'w')
wm = WatchManager()
profiler = ProfileProcessor(new_profile)
profiler = ProfileProcessor(sys.argv[1])
atexit.register(profiler.end_profiling)
signal(SIGTERM, lambda signum, stack_frame: sys.exit(0))
notifier = Notifier(wm, profiler)
wm.add_watch('/', IN_ACCESS, rec=True, exclude_filter=profiler.is_excluded)
notifier.loop(daemonize=True)
wm.add_watch('/', IN_OPEN | IN_ACCESS | IN_CREATE | IN_MOVED_TO, rec=True, exclude_filter=profiler.is_excluded)
notifier.loop(daemonize=True, pid_file='/boot-profile.pid')
if __name__ == '__main__':
main()
#!/bin/sh
PREREQ=""
prereqs() {
echo "${PREREQ}"
}
case "${1}" in
prereqs)
prereqs
exit 0
;;
esac
if [ -n "${NOUSER}" ]; then
exit 0
fi
# Only start profiler when "profile" appears on kernel command line
grep -qw "profile" /proc/cmdline || exit 0
. /scripts/live-functions
log_begin_msg "Starting boot profiler"
# Schedule stop script
cat <<EOF >/root/home/amnesia/.config/autostart/end-profile.desktop
[Desktop Entry]
Version=1.0
Name=EndProfile
GenericName=EndProfile
Exec=/usr/local/bin/end-profile
Terminal=false
Type=Application
EOF
echo 32768 >/proc/sys/fs/inotify/max_user_watches
chroot /root /usr/sbin/boot-profile /var/log/boot-profile
# Put readahead list at the very begining
head -n 1 /root/usr/share/amnesia/readahead-list >/dev/null || true
log_end_msg
#!/bin/sh
PREREQ=""
prereqs() {
echo "${PREREQ}"
}
case "${1}" in
prereqs)
prereqs
exit 0
;;
esac
# Do not readahead when "profile" appears on kernel command line
grep -qw "profile" /proc/cmdline && exit 0
READAHEAD_LIST="/root/usr/share/amnesia/readahead-list"
BACKGROUND_AT="^sbin/splashy$"
test -e "$READAHEAD_LIST" || exit 0
. /scripts/live-functions
log_begin_msg "Starting reading ahead boot files"
(cd /root
xargs stat <"$READAHEAD_LIST" >/dev/null)
(cd /root
sed -n "\:$BACKGROUND_AT:q;p" "$READAHEAD_LIST" | xargs cat >/dev/null)
(cd /root
sed -n "\:$BACKGROUND_AT:,\$p" "$READAHEAD_LIST" | xargs cat >/dev/null) &
log_end_msg
......@@ -81,3 +81,10 @@ cp debian/changelog config/chroot_local-includes/usr/share/doc/amnesia/Changelog
# GnuPG key
cp wiki/src/amnesia.asc config/chroot_local-includes/usr/share/doc/amnesia/
# create readahead-list from squashfs.sort
if [ -e config/binary_rootfs/squashfs.sort ]; then
mkdir -p config/chroot_local-includes/usr/share/amnesia
sort -k2 -n -r config/binary_rootfs/squashfs.sort |
cut -d' ' -f1 > config/chroot_local-includes/usr/share/amnesia/readahead-list
fi
When amnesia is booted from a CD, there is a lot of seeks from one file to
another during boot. This could be improved by ordering files on the CD in
the order used during boot.
the order used during boot and reading them in the same order before those
files need to be used.
Implementation
==============
Usage
=====
An *initscript* is loaded during 'S' runlevel, just at the begining of the
boot process. If `profile` has been specified on the kernel command line, it
starts the `boot-profile` daemon which uses `inotify` to record file that are
accessed during boot.
In order to refresh the sort file, `profile` must be added to the kernel
command-line. Then, 30 seconds after `iceweasel` has been loaded, the sort
file can be retrieved from `/var/log/boot-profile`.
Once enough of the boot sequence has been profiled, `boot-profile` must be
manually `kill`'ed. Then, `/lib/init/rw/boot-profile` can be copied to
`config/binary_rootfs/squashfs.sort` in the source directory after
removing the leading slashes from every listed path.
It must then be copied to `config/binary_rootfs/squashfs.sort`. This file is
automatically used by `lh_binary_rootfs` to tell `mksquashfs` to use the
provided *sort list*.
On next build, `lh_binary_rootfs` automatically asks `mksquashfs` to
use the provided *sort list*.
Left to be done
===============
Tests
=====
* [[!taglink todo/test]] on real hardware to see if readahead is really
useful.
Further testings can be helped by using the `bootchart` package that has been
adapted from Ubuntu: [[i386|bootchart_0.90.2-5~~amnesia1_i386.deb]],
[[amd64|bootchart_0.90.2-5~~amnesia1_amd64.deb]],
[[.dsc|bootchart_0.90.2-5~~amnesia1.dsc]],
[[.tar.gz|bootchart_0.90.2-5~~amnesia1.tar.gz]]. Once the binary package has
been copied to `config/chroot_local-packages`, at the end of normal boot
(without `profile` specified), a tarball will be left in `/var/log/bootchart`,
ready to be read with `bootchart-java` or `pybootchartgui`.
(On a little bit old laptop, using a CD-RW disk.)
Implementation
==============
Boot time from pressing enter in BIOS boot device selection menu to TorK.
Two scripts have been added to the `live-bottom` stage of the `initramfs`.
* 03:54 without sort list
* 03:22 with sort list
* `00boot_profile`
Left to be done
===============
When `profile` appears on the kernel command line, it starts the
`boot-profile` program. It will also add a new *autostart* entry to the
user home directory in order to launch `/usr/local/bin/end-profile`.
* `00readahead`
When `profile` does *not* appear on the kernel command line:
1. all files listed in `/usr/share/amnesia/readahead-list` are first
stat()'ed,
2. files until `/sbin/splashy` are read,
3. the rest of the files are read in background.
The `boot-profile` daemon uses `inotify` to record file that are accessed
during boot. It writes its pid in `/boot-profile.pid`. When killed, it dumps
the sort list at the path specified as its first command line argument.
* [[!taglink todo/test]] what the effects of the now bundled
`squashfs.sort` are
* [[!tag todo/research]] See if result can further be improved by loading the beginning of the squashfs in RAM.
* [[!tag todo/code]] Implement an automated way to update the *sortlist*.
The `end-profile` script will wait and start iceweasel before automatically
killing the `boot-profile` daemon.
[[boot-profile]] is a *sortlist* generated using the aformentioned
`boot-profile` daemon.
The file list `/usr/share/amnesia/readahead-list` used by `00readahead` is
generated by `scripts/config` (called by `lh config`).
This source diff could not be displayed because it is too large. You can view the blob instead.
Format: 1.0
Source: bootchart
Binary: bootchart
Architecture: any
Version: 0.90.2-5~~amnesia1
Maintainer: Scott James Remnant <scott@ubuntu.com>
Standards-Version: 3.6.2.1
Build-Depends: debhelper (>= 7.3.15ubuntu2)
Checksums-Sha1:
babf488cb2c781542bc6ab6ec080e5d519865def 23148 bootchart_0.90.2-5~~amnesia1.tar.gz
Checksums-Sha256:
a4a014d0529847f544838d835d42a9637d85f85f6c0dda5dcec512f46ee86529 23148 bootchart_0.90.2-5~~amnesia1.tar.gz
Files:
4acf0382590014bc0d5e376e65b4631c 23148 bootchart_0.90.2-5~~amnesia1.tar.gz
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment