Emerge/Watch emerge progress
From Gentoo Linux Wiki
| Terminals / Shells • Network • X Window System • Portage • System • Filesystems • Kernel • Other |
Please format this article according to the guidelines and Wikification suggestions, then remove this notice {{Wikify}} from the article
Contents |
[edit] Introduction
Get an approximate status of a current running emerge job to see how far along packages have compiled.
[edit] The Make Way
This approach works by running make twice, once in a pretend run to figure out what files the package is going to try to compile and then again for real. In THEORY it should be more accurate then trying to guess what files will need compiled and faster. However some packages don't use make and a pretend run may not give a full picture of everything make will actually do. For these reasons estimating by file-counting should work better since it works for all kinds of compiled packages and new file extensions can be added easily.
- First backup /usr/lib/portage/bin/emake.
# cp /usr/lib/portage/bin/emake /root/emake.backup
- Now edit /usr/lib/portage/bin/emake and save it, so it looks like this:
| File: /usr/lib/portage/bin/emake |
#!/bin/bash
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id: emake 3483 2006-06-10 21:40:40Z genone $
#
# emake: Supplies some default parameters to GNU make. At the moment the
# only parameter supplied is -jN, where N is a number of
# parallel processes that should be ideal for the running host
# (e.g. on a single-CPU machine, N=2). The MAKEOPTS variable
# is set in /etc/make.globals. We don't source
# /etc/make.globals here because emake is only called from an
# ebuild.
#
# make ${MAKEOPTS} ${EXTRA_EMAKE} "$@"
#package name
name=$(basename $(pwd))
if [ -f /tmp/emerge.$name.build ]; then
make ${MAKEOPTS} ${EXTRA_EMAKE} "$@"
else
make -n -i -w ${MAKEOPTS} ${EXTRA_EMAKE} "$@" 2>&1 | egrep "^g?make\[?" > /tmp/emerge.$name.build
make -w ${MAKEOPTS} ${EXTRA_EMAKE} "$@" | tee /tmp/emake.$name.edone
fi
|
What this does is redirect the output of a pretend make (note the -n) with the current dirs (note the -w) of each step to a temporary file and save the output of the real make run to both a file and to the standard output. There is nothing special in the above code; the real magic is done next.
- Now create a new file called eprogress and save it so it looks like this:
| File: ~/eprogress |
#!/bin/sh
#Created by michael brown, Emerge Progress revision 2.
#Created with help from xmb(API'ed MicrosuxX Bot - http://xmb.ath.cx) from #bash
#If you don't know what it does, don't mess with it :) -- really, I mean it. ... If you
#have any problems with the code, feel free to email me '''(xxxmikey_bxxx{at}hotmail{dot}com)'''
#with a good subject like "Emerge-progress." : )
#functions:
function makeprogress {
#name="`basename $entry .etodo`";
name=$1;
A="/tmp/emake.$name.etodo"; #todo
B="/tmp/emake.$name.edone"; #Done so far
todo=""
done=""
#now for the clever bit ;)
if [ -e /tmp/emake.$name.edone ]
then
todo=$(wc -l $A)
done=$(gawk -v m="$(tail -n 1 $B)" 'BEGIN { m = gensub(/([][/])/, "\\\\\\1", "g", m) } $0 ~ m { print FNR }' $A)
done="${done##* }"
fi
if [ "$done" != "" ]
then
percent=$((100 * $done / $todo))
else
percent="-"
done="-"
fi
if [ "$todo" == "" ]
then
todo="-"
fi
name=${name:6}
len=${#name}; len=$((30-len)); i=0; sp=""
for (( i = 0; i <=len; i++ )); do sp="$sp "; done
echo "$sp $done/$todo, ($percent%)"
}
function decodetree {
action="?"
pid=""
title=$1
input=$(echo "$1-")
function=$(echo $input | grep -i -e "sandbox(")
if [ ${#function} -gt 0 ]
then
pid=${input##*sandbox\(};
pid=${pid%%\)*};
title=$(ps w --no-headers --format "%a" $pid)
title=${title##*\[};
title=${title%%\]*};
fi
#compiling
function=$(echo $input | grep -i -e "make(")
if [ ${#function} -gt 0 ]
then
action="M"
title=$(echo "$title $(makeprogress $title)")
fi
#compiling
function=$(echo $input | grep -i -e "rsync(")
if [ ${#function} -gt 0 ]
then
action="$"
fi
#Downloading...
function=$(echo $input | grep -i -e "-wget(")
if [ ${#function} -gt 0 ]
then
pid=${input##*wget\(};
pid=${pid%%\)*};
title=$(ps w --no-headers --format "%a" $pid)
title=$(basename "$title")
action="D"
fi
#Configuring...
function=$(echo $input | grep -i -e "-configure(")
if [ ${#function} -gt 0 ]
then
action="C"
fi
function=$(echo $input | grep -i -e "-prepstrip(")
if [ ${#function} -gt 0 ]
then
action="S"
fi
#final config
function=$(echo $input | grep -i -e "-ldconfig(")
if [ ${#function} -gt 0 ]
then
action="F"
fi
function=$(echo $input | grep -i -e "-ld(")
if [ ${#function} -gt 0 ]
then
action="L"
fi
function=$(echo $input | grep -i -e "-portageq(")
if [ ${#function} -gt 0 ]
then
action="P"
fi
echo "[$action] $title"
}
# Notes:
# Well, this baby does everything it should, and it's impossible to go above 100%.
# If it does, PLEASE PLEASE tell me, : ) as it's a serious problem!
res=$(pgrep -x emerge | grep -v $$)
echo "Emerge processes: (for debugging)"
for entry in $res
do
etree=$(pstree -A -l -p $entry)
eres="$(decodetree "$etree")"
if [ "$eres" == "" ]
then
echo "$etree"
else
echo " * $eres"
fi
done
exit 0;
|
- Now move eprogress to /usr/local/bin/ and make it executable.
# mv eprogress /usr/local/bin/ && chmod +x /usr/local/bin/eprogress
- Finally you can see the progress of all currently working emerge processes by running eprogress.
When you run it, you will soon discover many symbols:
[M] = Compiling the build ... [$] = It's a portage sync : ) [D] = wget is downloading it for you : ) [C] = Project is being configured [L] = Linking objects [S] = Strip command (I don't know what it does! I just found it!) [F] = This is when emerge configures your enviroment to handle the new builds. F for finished : ) [P] = This is a portage query [?] = Means it's unknown NA: [-] = Means it's doing nothing or the pid is not found (one implies the other)
Example output:
Emerge processes: (for debugging) [D] |-xterm(16781)---bash(16796)---su(20442)---bash(20553)---emerge(19264)---wget(22266) Emerge Progress (v2): (old) * (V)[D] (22266) thunderbird-1.0.2-source.tar.bz2 * (L)[-] eject-2.0.13: 31/31, (100%) * (L)[-] mozilla-thunderbird-1.0.2: -/616, (-%) * (L)[-] xmule-1.8.4-r1: 291/291, (100%)
This output is from when thunderbird-1.0.2-source was downloading.
Nota bene: The funky stuff at the top is for other people helping with this project. It's a tree of the processes being run by emerge. The numbers are the pids (To get information about a pid ... ps w 16796) Notice mozilla-thunderbird-1.0.2: -/616, (-%). This means that the last entry on the .edone could not be found. It has something to do with the pretend make. Since there are over 6,000 source-code files to compile, if someone could enlighten me as to why this happens, please email me! The log file has not yet been removed because this code is still quite testy.
[edit] Minor BugFix 1
For my revision of portage, the gawk line above simply spits a series of progressive numbers to the console, and so does not work.
I solved the problem into a "works-good-enough" stage by replacing the lines
done=$(gawk -v m="$(tail -n 1 $B)" 'BEGIN { m = gensub(/([][/])/, "\\\\\\1", "g", m) } $0 ~ m { print FNR }' $A) done="${done##* }"
with
"done=$(wc -l $B)."
Unfortunately this fix only makes it work. Because two big files are read, it produces a rather big slowdown.
I hope this is helpful for all others who encounter the same bug.
- --TheJackal 11:27, 20 October 2005 (GMT)
Got the same problem, thanks for fixing! - Max, 23 Oct 05
Bug in the BugFix this bug's not quite so bad as the one it fixes, so it's still a bugfix, but it suffers the same bug as all the C++ / C versions as it now produces more than 100% for some merges :S, so I'm currently trying to figure out what's wrong with gawk.
- --TheJackal 20:58, 23 October 2005 (GMT)
[edit] The File count Way
This approach works by counting files as such it depends on guessing what files are probably going to need to be compiled and what files have probably already been compiled. This is achieved by looking at the file extension of files. New file extensions can be added so its a little more flexiable then depending on a makefile.
I'm kind of amazed nobody mentioned this here before, but the file count way is theoretically flawed. There is no way to make a trustworthy prediction on the amount of output files from the amount of source files. The chance your build will end at 80%, 275%, or some other arbitrary number is pretty likely. The scripts presented here for you are all a display of horrid scripting habits, mainly to do with quoting issues. Can somebody *please* fix this page up?
[edit] Simple Version
- First create a new file /usr/local/bin/eprogress and save it so it looks like this:
| File: /usr/local/bin/eprogress |
#!/bin/sh
source /etc/make.globals
source /etc/make.conf
CUR=`tail -n 5 /var/log/emerge.log | grep Compiling | sed 's/\(.*\) Compiling\/\(Merging\|Packaging\) (.*\/\(.*\)::.*)/\3/'`
TOT=`find $PORTAGE_TMPDIR/portage/$CUR -iname "*.c*" | wc -l`
PRG=`find $PORTAGE_TMPDIR/portage/$CUR -iname "*.o*" | wc -l`
PER=`echo $PRG $TOT | gawk '{ sum += ($1 / $2) * 100 }; END { print sum }' -`
echo "Currently Compiled Package: " $CUR " (" $PER "%)"
|
- Next make it executable:
# chmod +x /usr/local/bin/eprogress
After that you can watch progress in %, by executing the following command:
# watch -n 5 --no-title eprogress
-n 5 sets refresh interval to 5 seconds and --no-title removes title.
If you have osd_cat (emerge xosd), you can do this:
# eprogress | osd_cat --colour=white --outline=1 --outlinecolour=black
Set that to a hotkey in your WM, and you have an OSD emerge progress hotkey :)
Newer versions of osd_cat have support for percentages. You can have it draw you a bar graph of completion instead. This next command will draw a bar of completion for the first package returned by eprogress (usually the one merging):
eprogress | grep ^\ \ \* | head -n 1 | sed 's/.*\] //' | sed 's/:[^(]*(/ /' | sed 's/%)$//' | \
awk '{system("osd_cat --colour=white --outline=1 --outlinecolour=black -b percentage -P "$2" -T "$1);}
[edit] Improved version!
As with the previous code, you can use watch, but watch doesnt support colours, so you must use the -nc option
# watch -n 5 eprogress -nc
| File: /usr/local/bin/eprogress |
#!/bin/sh
#By Michael Brown
#
#I've been working on make improvements on displaying the current emerge status,
#this is the code I've come up with so far. It's still in testing, but it works
#when it compiles java as well.. soon adding perl, py etc..
#
#Any problems or fixes please email me! xxxmikey_bxxx{AT}hotmail{dot}com
#
#If it gets any good i may rewrite it into C.
#get PORTAGE_TMPDIR <- hard coded portage location?, might want to change!
source /etc/make.globals
source /etc/make.conf > /dev/null 2>&1
notitle=0; nocol=0;
for i in $@; do
if [ "$i" == "-nt" ]; then notitle=1; fi
if [ "$i" == "-nc" ]; then nocol=1; fi
if [ "$i" == "--help" ]; then
echo -e "Emerge Progress Script V1:\nThis code was a collection of other code
with extras,\nrecoded by Michael Brown, Previous authors plasmagunman, kopfarzt
n\nDesigned to display the emerge status\nUsage: eprogress [options]\n\nOp
ons:\n -nc Enables the no colour mode\n -nt Enables the no title mode\n --help
displays this text.\n\n\nDisplay:\nStandard format of result:\n * [1] Packagena
todo/outof (percent%).\n\n[1] will either show a [C] for compiling, or [x] fo
cancelled.\nPackage is the package you are emerging\ntodo/outof is the number of
iles for that emerge left todo, and the total number."
exit 0
fi
done
if [ "$notitle" -eq "0" ]; then echo -e "Emerge Progress:"; fi
if [ "$nocol" -eq "0" ]; then
coltab1="\E[;32m"; coltab2="\033[1m * \033[0m"
else
coltab1=""; coltab2=" *"
fi
#can i read tmp folder?
folder=`find $PORTAGE_TMPDIR/portage/ -name work 2> /dev/null`
if [ $? -gt 0 ]; then
if [ "$nocol" -eq "0" ]; then echo -en '\E[;31m'; fi
echo "!! You do not have access to the portage tmp folder, unable to load stat !!"
tput sgr0
fi
#set up extentions
srcext="c cpp cc java C"
objext="o class"
for ext in $srcext; do srccmd="$srccmd -name *.$ext* -o"; done
for ext in $objext; do objcmd="$objcmd -name *.$ext* -o"; done
for WORKDIR in $folder; do
TODO=`find $WORKDIR $srccmd -false 2> /dev/null | wc -l`
DONE=`find $WORKDIR $objcmd -false 2> /dev/null | wc -l`
if [ "$((TODO))" == "0" ]; then
percent=0
else
percent=$((100 * $DONE / $TODO))
fi
name="`basename \`dirname $WORKDIR\``"
len=${#name}; len=$((30-len)); i=0; sp=""
for (( i = 0; i <=len; i++ )); do sp="$sp "; done
#is it currently compiling?
res=$(ps ax | grep -v grep | grep -i $name)
if [ ${#res} -gt 0 ]; then
action="C"
else
action="x"
fi
echo -e $coltab1 "$coltab2 [$action] $name: $sp $DONE/$TODO, ($percent%)"
done
if [ "$nocol" -eq "0" ]; then tput sgr0; fi
exit 0
|
The above shell code is bugged in so many ways it's just not funny anymore. Somebody please fix its syntax up or remove it so nobody needs to see it anymore.
[edit] C++ version
This version relies on the /proc filesystem for better speed, but currently doesn't do colors. If you don't have the /proc filesystem then this version won't work. Later Ppid identifiers may be used instead.
# g++ -o eprogress eprogress.cpp && cp eprogress /usr/local/bin/
| Code: eprogress.cpp |
// Updated by David K. (David-Dot-Kelley-At-comcast-dot-net)
// Updated by Annoymance, phox and darklama
#include <sys/types.h>
#include <dirent.h>
#include <unistd.h>
#include <fstream>
#include <string>
#include <cstdio>
using namespace std;
#define srccount (sizeof(src)/sizeof(src[0]))
#define objcount (sizeof(obj)/sizeof(obj[0]))
const char *src[] = {".c",".cc",".cpp",".c++",".C",".java",".s",".l",".y"};
const char *obj[] = {".a",".o",".class",".so",".ko",".lib"};
class path {
DIR *d,*t;
struct dirent *e;
public:
path(char *p) {d=opendir(p); t=opendir("."); if (d) chdir(p); e=NULL;}
~path() {closedir(d); if (t) fchdir(dirfd(t)); closedir(t);}
void go() {chdir(e->d_name);}
void up() {chdir("..");}
bool isdir() {return e->d_type==DT_DIR;}
operator char*() {return e ? e->d_name : 0;}
bool operator++(int a) {
while (d && (e=readdir(d))) {
if ((a=strspn(e->d_name, "."))<=2 && e->d_name[a]=='\0')
continue;
return true;
}
return false;
}
};
void countfiles(char*p,unsigned&todo,unsigned&done) {
path d(p);
char *s;
while (d++)
if (d.isdir()) {
countfiles(d,todo,done);
} else {
for (unsigned c=0; c<srccount; c++)
if ((s=strstr(d, src[c])) && *(s+strlen(src[c])) == '\0') todo++;
for (unsigned c=0; c<objcount; c++)
if ((s=strstr(d, obj[c])) && *(s+strlen(obj[c])) == '\0') done++;
}
}
int findproc(char *name) {
// Find the proc of portage :)
char buffer[256];
path p("/proc");
while (p++)
if (p.isdir()) {
p.go();
ifstream rfile("cmdline");
if (rfile.is_open()) {
rfile.getline(buffer,sizeof buffer);
if (strstr(buffer,name) != NULL)
return 0;
}
p.up();
}
return 1;
}
int main() {
unsigned int done,todo,perc;
path p("/usr/tmp/portage/");
puts("Emerge Progress v4.21 C++:");
while (p++)
if (p.isdir()) {
p.go();
todo=done=0; countfiles("/work",todo,done);
if (todo>0 && done<=todo) {
perc=(done/todo)*100;
printf("[%c] %s: %d/%d (%d%%)\n","XC"[findproc(p)],static_cast<char*>(p),done,todo,perc);
}
p.up();
}
return 0;
}
|
[edit] Minor Bugs in the c++ program
t35t0r
The c++ program works for the most part but sometimes I get errors like this:
Emerge Progress v2. C++ version: * [C] tunepimp-0.3.0: 80/49, (163%)
..and on another gentoo box I get items that aren't being emerged beginning with an [X] with completon 0% or partially completed (<100%). Now those can't be items which I've prematurely ctrl+c out of, because it doesn't occur on some boxes (like the one where I get the >100%).
Cause this program just counts object files for percentages and it's possible to have more object files than c files.
Similar, but different functionality is got from using genlop -c (emerge genlop).
[edit] checkprogress.sh (c) Sadysta
| Code: checkprogress.sh |
#!/bin/sh
# Copyright 2006 Sadysta
# Distributed under the terms of the GNU General Public License v2
if [ ! -x /usr/bin/time ]; then
echo -e "\33[1;31m!!\33[m You must have sys-process/time installed."
exit 1
fi
OLD_FULL="-"
FULL=$1
while true; do
if [[ "$1" == "" ]]; then
FULL=`tail -n 2 /var/log/emerge.log | grep Compiling | sed "s/\(.*\)::\(.*\)/\1/" | sed "s/.*(//"`
fi
if [[ "${FULL}" != "" ]]; then
if [[ "${FULL}" != "${OLD_FULL}" ]]; then
if [[ "$1" != "" ]]; then
echo -e "\33[1;33m!!\33[m Manual build directory override. Entering scanning only mode..."
fi
echo -e "\33[1;32m**\33[m Compiling ${FULL}..."
OLD_FULL=${FULL}
else
echo -e "\33M\33[1A"
fi
if [ -d /var/tmp/portage/${FULL} ]; then
SRC_COUNT=`/usr/bin/time -p -o /tmp/chkprgrs_t1.txt find /var/tmp/portage/${FULL} -regextype posix-extended -regex ".*(\.cpp|\.c|\.cc|\.asm|\.s|\.cxx|\.java|\.l|\.y)" | wc -l`
READY_COUNT=`/usr/bin/time -p -o /tmp/chkprgrs_t2.txt find /var/tmp/portage/${FULL} -regextype posix-extended -regex ".*(\.o|\.a|\.class|\.ko|\.lib|\.so)" | wc -l`
TIME1=$((`cat /tmp/chkprgrs_t1.txt | grep real | cut -d" " -f2 | sed "s/\.//" | sed "s/^0*\(.\)/\\1/"` / 10))
TIME2=$((`cat /tmp/chkprgrs_t2.txt | grep real | cut -d" " -f2 | sed "s/\.//" | sed "s/^0*\(.\)/\\1/"` / 10))
TIME=$((${TIME1} + ${TIME2}))
if [[ ${TIME} -le 5 ]]; then
TIME=5;
fi
if [[ ${SRC_COUNT} -eq 0 ]]; then
SRC_COUNT=1;
fi
PERCENT=$((${READY_COUNT}*100/${SRC_COUNT}))
echo -e "\33[1;32m**\33[m Done: ${READY_COUNT} of ${SRC_COUNT} (${PERCENT} %)..."
sleep ${TIME}
else
echo -e "\33[1;33m!!\33[m Sources haven't been unpacked yet."
sleep 5
fi
else
if [[ "${FULL}" != "${OLD_FULL}" ]]; then
echo -e "\33[1;33m**\33[m No compilation is in progress at the moment."
OLD_FULL=${FULL}
fi
sleep 5
fi
done
|
[edit] External Links
From Gentoo Forums:
