passwords one/two/thr/for/fiv.
The paths and ports are those of the test container (/home/brisk, 8444 for
-https): they have to be adapted elsewhere.
+https, 8082 for plain http): they have to be adapted elsewhere.
+
+
+How to run them
+---------------
+
+From this directory, straight out of the repository:
+
+ ./brisk_step.sh 150 90 3 --tls
+ ./stop.sh
+
+They used to be a copy kept in /root/load, run as root, with those paths
+written into them; the copy went stale more than once. Now every script finds
+its own directory and reads common.sh, which holds the three settings they all
+share:
+
+ BRISK_WORK where the working files go. Default /var/tmp/brisk-load.
+ NOT /tmp: that is a tmpfs here, and a restart of the container
+ wipes it in the middle of a session. Not this directory either,
+ or auth.txt and the captured streams would be left in the git
+ tree.
+ BRISK_LEGAL where the daemon keeps its log and its pid file: the path
+ given to INSTALL.sh with -l. Default /home/brisk/legal.
+ SUDO the daemon runs as another user, so restarting the service and
+ reading its open descriptors need root while the bench itself
+ does not. Set SUDO= to empty when already running as root.
+
+Nothing here needs to run as root.
The load
It shuts everything down and prints the state: running scripts, running curls,
descriptors of the daemon, disk space, load. To be run ALWAYS.
-The scripts that start background loops carry a trap that kills their own
-process group. Without it the loops outlive their parent and are left spinning
-for nothing: that happened for two days, with eighty processes alive and the
-machine load at 19.
-
-stop.sh does not use "pkill -f": the pattern would end up in the command line
-of the script itself, which would kill itself and leave the targets alive.
-
-The working files (auth.txt, tok.txt, mani.txt, *.stream) are created in the
-working directory of the scripts and can grow large: the read streams grow all
-the time. stop.sh deletes them.
+The scripts that start background loops carry a trap, bench_reap, that kills
+their own descendants on the way out. Without it the loops outlive their
+parent and are left spinning for nothing: that happened for two days, with
+eighty processes alive and the machine load at 19.
+
+It used to be "trap 'kill 0'", which signals the whole process group. That is
+almost always wider than the script: it killed the script itself, so the exit
+status was always a death by signal, and it reached whatever else shared the
+group - whoever launched it, or the other half of a pipeline, so that
+"brisk_step.sh | tail" printed nothing at all. Running everything through
+"incus exec" gave each script a session of its own and hid all of it.
+
+Two traps of the same family, both of which have already cost time here:
+
+ - stop.sh does not use "pkill -f": the pattern would end up in the command
+ line of the script itself, which would kill itself and leave the targets
+ alive.
+ - when counting leftovers by hand, the grep matches its own command line.
+ Split the pattern ("ga""me.sh") or use "[g]ame.sh", otherwise a clean
+ machine looks busy and a busy one looks clean.
+
+The working files (auth.txt, tok.txt, mani.txt, *.stream) are created in
+BRISK_WORK and can grow large: the read streams grow all the time. stop.sh
+deletes them.
# Samples the resources of daemon and frontend during a load test.
# usage: brisk_sample.sh <seconds> <interval> <label>
DUR="${1:-60}"; INT="${2:-5}"; TAG="${3:-x}"
+. "$(dirname "$0")/common.sh"
dpid="$(pgrep -f 'php \./brisk-spush\.php' | head -1)"
if [ -z "$dpid" ]; then echo "daemon not found"; exit 1; fi
while [ $(date +%s) -lt $end ]; do
sleep "$INT"
rss=$(awk '/VmRSS/{print $2}' /proc/$dpid/status 2>/dev/null || echo 0)
- fd=$(ls /proc/$dpid/fd 2>/dev/null | wc -l)
+ fd=$($SUDO ls /proc/$dpid/fd 2>/dev/null | wc -l)
conn=$(ss -x 2>/dev/null | grep -c 'brisk[0-9]*\.sock')
[ "$rss" -gt "$maxrss" ] && maxrss=$rss
[ "$fd" -gt "$maxfd" ] && maxfd=$fd
#
# usage: brisk_step.sh <clients> <duration> <write_period> [--silent] [--tls]
# Subprocess loops outlive their parent: without this they are left orphaned,
-# spinning for nothing (it happened: eighty loops alive for two days). kill 0
-# kills the process group, which setsid makes exclusive to this script.
-trap "kill 0" EXIT INT TERM
+# spinning for nothing (it happened: eighty loops alive for two days).
+#
+# bench_reap walks our descendants, so it reaches them without touching the
+# caller or the other half of a pipeline. See common.sh.
+. "$(dirname "$0")/common.sh"
+trap bench_reap EXIT INT TERM
+
N="${1:-50}"; DUR="${2:-120}"; PER="${3:-10}"; shift 3
EXTRA="$*"
PORT="${PORT:-8082}"
echo "$EXTRA" | grep -q -- "--tls" && PORT=8444
-# empty room: the users of the previous step would stay connected
-p=$(pgrep -f 'php \./brisk-spush\.php'); [ -n "$p" ] && kill $p; sleep 2
-s=$(pgrep -u www-data -x screen); [ -n "$s" ] && kill $s 2>/dev/null; sleep 1
-rm -f /tmp/brisk.log
-su -s /bin/bash www-data -c "cd /home/brisk/web/brisk/spush && screen -d -m -S brisk -L -Logfile /tmp/brisk.log ./brisk-spush.php"
+# Empty room: the users of the previous step would stay connected.
+#
+# Through the init script, never by hand. Starting the daemon beside one that
+# is already running is what used to leave orphaned instances behind: the
+# newcomer unlinks the socket files and binds its own, taking every new
+# connection, while the first one stays alive on its own shared memory and
+# nothing says so. "stop" also closes the screen sessions, which carry the
+# loop that would otherwise respawn what it just killed.
+LOG0=$(wc -l < "$BRISK_LEGAL/brisk.log" 2>/dev/null || echo 0)
+NGX0=$($SUDO grep -cE ' \[(error|crit|alert)\] ' /var/log/nginx/error.log 2>/dev/null || echo 0)
+$SUDO systemctl restart brisk
sleep 4
echo "=== $N clients, $DUR s, one write every $PER s $EXTRA ==="
-( sleep $((WARM + 3)); /root/load/brisk_sample.sh "$DUR" 5 "$N" ) > /tmp/sample.out 2>&1 &
+( sleep $((WARM + 3)); "$BENCH/brisk_sample.sh" "$DUR" 5 "$N" ) > "$BRISK_WORK/sample.out" 2>&1 &
SAMP=$!
-cd /root/load
-timeout $((WARM + DUR + 120)) python3 brisk_load.py --auth --clients "$N" \
+timeout $((WARM + DUR + 120)) python3 "$BENCH/brisk_load.py" --auth --clients "$N" \
--warmup "$WARM" --duration "$DUR" --period "$PER" --port "$PORT" $EXTRA
wait $SAMP
-cat /tmp/sample.out
-echo " daemon log: $(wc -l < /tmp/brisk.log) lines, of which unexpected: $(tr -d '#\r' < /tmp/brisk.log | grep -icE 'warning|notice|error|fatal')"
-echo " nginx errors: $(grep -cE ' \[(error|crit|alert)\] ' /var/log/nginx/error.log 2>/dev/null || echo 0)"
+cat "$BRISK_WORK/sample.out"
+# The log is cumulative, so what counts is what this step added to it
+LOG1=$(wc -l < "$BRISK_LEGAL/brisk.log" 2>/dev/null || echo 0)
+echo " daemon log: $((LOG1 - LOG0)) new lines, of which unexpected: $(tail -n +$((LOG0 + 1)) "$BRISK_LEGAL/brisk.log" 2>/dev/null | grep -icE 'warning|notice|error|fatal|crit')"
+# the nginx log is 640 www-data:adm, unreadable to the unprivileged bench
+NGX1=$($SUDO grep -cE ' \[(error|crit|alert)\] ' /var/log/nginx/error.log 2>/dev/null || echo 0)
+echo " nginx errors: $((NGX1 - NGX0)) new"
--- /dev/null
+# Shared settings of the load bench. Sourced by the scripts here, never run.
+#
+# The bench used to live in /root/load, with those paths written into it, and
+# to run as root. It runs from the repository now, as an unprivileged user, so
+# every script finds its own directory instead and keeps its working files out
+# of it: auth.txt, the captured streams and the logs would otherwise be left
+# lying in the git tree.
+
+BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Working files go here, and not in /tmp: that is a tmpfs, and a restart of the
+# container wipes it in the middle of a session, scripts included - which is
+# why they had been moved to /root/load in the first place. /var/tmp is on the
+# root filesystem.
+BRISK_WORK="${BRISK_WORK:-/var/tmp/brisk-load}"
+mkdir -p "$BRISK_WORK"
+
+# Where the daemon keeps its log and its pid file: the path handed to
+# INSTALL.sh with -l. Getting it wrong is not harmless - the init script looks
+# for brisk.pid in there, and while it cannot find it "stop" does nothing and
+# every restart leaves an orphaned daemon behind instead of replacing it.
+BRISK_LEGAL="${BRISK_LEGAL:-/home/brisk/legal}"
+
+# The daemon runs as another user: restarting the service and reading its open
+# descriptors need root, and the bench is unprivileged now.
+SUDO="${SUDO:-sudo}"
+
+# Kills the background loops this script started, and nothing else.
+#
+# The bench used to do: trap "kill 0" EXIT INT TERM. kill 0 signals the whole
+# process group, which is nearly always wider than this script. It killed the
+# script itself on the way out, so the exit status was always a death by
+# signal and buffered output could be lost; and it reached whatever else
+# shared the group - whoever launched us, or the other half of a pipeline,
+# so that "brisk_step.sh | tail" printed nothing at all. Running under
+# "incus exec" gave every script a session of its own and hid all of it.
+#
+# Walking our own descendants reaches exactly what kill 0 was there for - the
+# loops inside "( ... ) &" and the curls underneath them, which inherit us as
+# their parent - and nothing that we did not start ourselves.
+bench_descendants() {
+ local kid
+ for kid in $(pgrep -P "$1" 2>/dev/null); do
+ bench_descendants "$kid"
+ echo "$kid"
+ done
+}
+
+bench_reap() {
+ trap - EXIT INT TERM
+
+ local kids
+ kids=$(bench_descendants $$)
+ # drop them from the job table first, or the shell announces every kill
+ disown -a 2>/dev/null || true
+ if [ -n "$kids" ]; then
+ kill $kids 2>/dev/null
+ sleep 0.3
+ kids=$(bench_descendants $$)
+ [ -n "$kids" ] && kill -9 $kids 2>/dev/null
+ fi
+ return 0
+}
#
# usage: game.sh [table] [port]
#
-# It lives in /root/load and not in /tmp, which is tmpfs: a restart of the
-# container used to wipe the scripts.
# Subprocess loops outlive their parent: without this they are left orphaned,
-# spinning for nothing. kill 0 kills the group, which setsid makes exclusive.
-trap "kill 0" EXIT INT TERM
+# spinning for nothing (it happened: eighty loops alive for two days).
+#
+# bench_reap walks our descendants, so it reaches them without touching the
+# caller or the other half of a pipeline. See common.sh.
+. "$(dirname "$0")/common.sh"
+trap bench_reap EXIT INT TERM
TAB="${1:-4}"; PORTA="${2:-8444}"
B="https://127.0.0.1:${PORTA}/brisk"
CURL="curl -sSk"
-cd /root/load
+cd "$BRISK_WORK"
rm -f auth.txt r?.stream k?.stream mani.txt tok.txt
echo "== 1. login of five authenticated users =="
TAB="${1:-4}"; PORTA="${2:-8444}"
B="https://127.0.0.1:${PORTA}/brisk"
CURL="curl -sSk"
-cd /root/load
+. "$(dirname "$0")/common.sh"
+cd "$BRISK_WORK"
TK=$(cat tok.txt)
send() {
# When it is their turn they first try to pass; if after a second the turn is
# still theirs it means the auction is over, and then they play the first card
# the server accepts. That way there is no need to know which phase we are in.
-B="https://127.0.0.1:8444/brisk"; TAB="${1:-4}"; cd /tmp
+B="https://127.0.0.1:8444/brisk"; TAB="${1:-4}"
+. "$(dirname "$0")/common.sh"
CURL="curl -sSk"
+cd "$BRISK_WORK"
MAXWAIT="${2:-900}"
echo "waiting for the table to form..."
#!/bin/bash
# Measures what a client receives while 150 users come into the room.
# Subprocess loops outlive their parent: without this they are left orphaned,
-# spinning for nothing (it happened: eighty loops alive for two days). kill 0
-# kills the process group, which setsid makes exclusive to this script.
-trap "kill 0" EXIT INT TERM
+# spinning for nothing (it happened: eighty loops alive for two days).
+#
+# bench_reap walks our descendants, so it reaches them without touching the
+# caller or the other half of a pipeline. See common.sh.
+. "$(dirname "$0")/common.sh"
+trap bench_reap EXIT INT TERM
B="http://127.0.0.1:8082/brisk"
TOK=$(curl -sS -m 10 "$B/index_wr.php?mesg=getchallenge&cli_name=load303" | cut -d"|" -f2)
MP=$(printf "%s" "load303" | md5sum | cut -d" " -f1)
PRIV=$(printf "%s%s" "$TOK" "$MP" | md5sum | cut -d" " -f1)
S=$(curl -sS -m 20 "$B/index.php?name=load303&pass_private=$PRIV" | grep -oE "sess = \"[0-9a-f]+\"" | head -1 | sed "s/.*\"\(.*\)\"/\1/")
-( curl -sS -N -m 60 -b "sess=$S" "$B/index_rd.php?stat=&subst=&step=-1&from=index_php&transp=xhr" > /tmp/ramp.stream 2>/dev/null ) &
+( curl -sS -N -m 60 -b "sess=$S" "$B/index_rd.php?stat=&subst=&step=-1&from=index_php&transp=xhr" > "$BRISK_WORK/ramp.stream" 2>/dev/null ) &
sleep 3
-cd /root/load
-timeout 100 python3 brisk_load.py --auth --clients 150 --base 101 --silent --warmup 5 --duration 35 --port 8082 > /tmp/load5.log 2>&1
+timeout 100 python3 "$BENCH/brisk_load.py" --auth --clients 150 --base 101 \
+ --silent --warmup 5 --duration 35 --port 8082 > "$BRISK_WORK/load5.log" 2>&1
wait
# being with a real browser. They stay connected and keep their streams open,
# so the table forms as soon as the human sits down.
# Subprocess loops outlive their parent: without this they are left orphaned,
-# spinning for nothing (it happened: eighty loops alive for two days). kill 0
-# kills the process group, which setsid makes exclusive to this script.
-trap "kill 0" EXIT INT TERM
+# spinning for nothing (it happened: eighty loops alive for two days).
+#
+# bench_reap walks our descendants, so it reaches them without touching the
+# caller or the other half of a pipeline. See common.sh.
+. "$(dirname "$0")/common.sh"
+trap bench_reap EXIT INT TERM
-B="https://127.0.0.1:8444/brisk"; TAB="${1:-4}"; cd /tmp
+B="https://127.0.0.1:8444/brisk"; TAB="${1:-4}"
CURL="curl -sSk"
+cd "$BRISK_WORK"
rm -f auth.txt r?.stream k?.stream mani.txt tok.txt
for u in load001 load002 load003 load004; do
# this very script, which would kill itself and leave the targets alive. That
# happened several times, and it is how eighty loops survived for two days.
+. "$(dirname "$0")/common.sh"
+
MIO=$$
PAT="game\\.sh|hand\\.sh|game_pkg|game_https|game_t5|game_t9|hand_pkg|hand_https|sit4|play_human"
PAT="$PAT|join4|gjoin|ramp_probe|brisk_load|slowprobe|wsprobe|takeover"
sleep 2
# The capture files grow without bound: two days of runaway loops had filled
-# the disk, which is the same one as the host's.
-rm -f /tmp/*.stream /tmp/*.html 2>/dev/null
+# the disk, which is the same one as the host's. The streams grow the whole
+# time they are open, so they are the ones that hurt.
+rm -f "$BRISK_WORK"/*.stream "$BRISK_WORK"/*.html "$BRISK_WORK"/*.log \
+ "$BRISK_WORK"/*.out "$BRISK_WORK"/auth.txt "$BRISK_WORK"/tok.txt \
+ "$BRISK_WORK"/mani.txt 2>/dev/null
d="$(pgrep -f 'php \./brisk-spush\.php' | head -1)"
echo
printf " test scripts running: %s\n" "$(pidlist | grep -c .)"
printf " curl running: %s\n" "$(pgrep -x curl | wc -l)"
printf " daemon: %s process, %s descriptors (10 are the listening sockets)\n" \
- "$(pgrep -f 'php \./brisk-spush\.php' | wc -l)" "$(ls /proc/$d/fd 2>/dev/null | wc -l)"
+ "$(pgrep -f 'php \./brisk-spush\.php' | wc -l)" "$($SUDO ls /proc/$d/fd 2>/dev/null | wc -l)"
printf " disk space: %s\n" \
"$(df -h / | awk 'NR==2 { print $4 " free (" $5 " used)" }')"
-printf " daemon log: %s\n" "$(du -sh /tmp/brisk.log 2>/dev/null | cut -f1)"
+printf " daemon log: %s\n" "$(du -sh "$BRISK_LEGAL/brisk.log" 2>/dev/null | cut -f1)"
+printf " working files: %s in %s\n" \
+ "$(ls -1 "$BRISK_WORK" 2>/dev/null | wc -l)" "$BRISK_WORK"
printf " load: %s\n" "$(uptime | sed 's/.*average: //')"
projects / brisk.git / commitdiff