.TL Old Software Treasuries .AU markus schnalke .SS "this talk" .IT show the historical background .IT introduce, explain and demonstrate .CW ed , .CW sed , and .CW awk .IT solve real life problems .sp 1 .LP please interrupt me at any time in case of questions .LP please tell me what I should demonstrate .sp 1 .LP goal: motivate you to learn and use these programs .SS "old software treasuries" .IT older than I am .IT standardized .IT various implementations .IT freely available .IT good software .SS origin .IT classic Unix tools .IT from Bell Labs .IT representations of the Unix Philosophy .SS "historical background" 1968: .IT MULTICS fails .IT the last people working on it were Thompson, Ritchie, McIlroy, Ossanna .LP 1969: .IT they still think about operating systems .IT pushing idea: a file system (mainly Thompson) .IT in search for hardware: a rarely used PDP-7 .bp .LP 1970: .IT Kernighan suggests ``Unix'' .IT deal: a PDP-11 for a document preparation system .LP 1973: .IT rewrite of Unix in C .LP 1974: .IT first publication .SS "computer hardware back then" .IT computers are as large as closets .IT users work on terminals .IT terminals are: a keyboard and a line printer .IT line printers have a speed of 10\(en15 chars/s .sp 2 .ce 1 .SS "impact on software" .IT don't waste characters .IT be quiet .IT nothing like ``screen-orientation'' .PA ed .PP .SS ed .IT by Ken Thompson in early 70s, based on .CW qed .IT the Unix text editor .IT the .I standard text editor! .IT Unix was written with .CW ed .IT first implementation of Regular Expressions .SS "ed: old-fashioned" .IT already old-fashioned in 1984 .IT ``So why are we spending time on such a old-fashioned program?'' .IT ``Although many readers will prefer some other editor for daily use, .CW ed is universally available, efficient and effective.'' .SS "ed: reasons" .IT the standard (text editor) .IT available as .CW /bin/ed .IT line oriented \(-> works on any terminal .IT needs few bandwidth .IT good for automated editing (scripts) .IT good for presentations (?) .sp 1 .LP it's not so much about using it, but about knowing it .SS "ed: overview" .IT usage: .CW "ed [-s] [file]" .IT commands: .CW "[a1[,a2]] cmd [params]" .IT default addresses, default params .IT success \(-> no feedback .IT problems \(-> ``?'' .IT about 30 commands, half of them important .SS "ed: commands" .TS lfCW l lfCW l. i insert p print a append n print with line number c change l list characters .sp 0.5 d delete e edit file m move w write file t transfer (copy) q quit .sp .5 s substitute u undo g global command .TE .SS "ed: addresses" .TS lfCW l. 4 4th line $ last line \&. current line .sp .5 + next line (\f(CW.+1\fP) \&-- the same as \f(CW.-2\fP .sp .5 /RE/ next line matching RE ?RE? previous line matching RE .sp .5 1,$ all lines -,/foo/ from previous line to next /foo/ .TE .PA "\s-8\fR\s+8" .PP .SS "ed: derived software" .PS ED: box "ed" arrow right box "ex" arrow box "vi" arrow up right from ED.ne box "grep" arrow down right from ED.se box "sed" .PE .PA sed .PP .SS "sed" .IT by Lee McMahon in 1973 .IT ``stream editor'' .IT started as a one-night hack on .CW ed (AFAIR) .IT makes .CW ed suitable for pipeline processing .IT irony: today, people use .CW "sed -i" for in-place editing .SS "sed: differences to ed" .IT line processing cycle .IT no forward/backward references .IT hold space (\f(CWg\fR,\f(CWG\fR,\f(CWh\fR,\f(CWH\fR,\f(CWx\fR) .IT labels and branches .IT bad .CW i and .CW a syntax .PA "\s-8\fR\s+8" .PP .PA awk .PP .SS awk .IT by Aho, Weinberger, Kernighan .IT \f(CWoawk\fR in 1977, \f(CWnawk\fR in 1985 .IT name is abbreviation of surnames (but also ``awkward'') .IT the second scripting language on Unix (besides .CW sh ) .IT `` .CW sed meets C'' .SS "awk: purpose" .IT for text processing .IT avoids complex constructs in .CW sh .IT floating point arithmetic .SS "awk: usage" .IT .CW "awk 'commands' out" .IT .CW "awk -f cmdfile out" .IT program flow like in sed .IT input automatically split in records and fields .IT program is a list of blocks: .CW "cond { commands }" .SS "awk: features" .IT variables, assoc arrays .IT functions .IT pattern matching (EREs) .IT dynamic typing .IT more high-level: no pointers .SS "awk: statements" much like C .IT \f(CWif\fP, but no \f(CWswitch-case\fP .IT \f(CWwhile\fP, \f(CWdo-while\fP .IT \f(CWfor (;;) {}\fP like in C .IT \f(CWfor (i in array) {}\fP foreach .IT \f(CWbreak\fP, \f(CWcontinue\fP, \f(CWexit\fP, \f(CWreturn\fP .IT \f(CWprint\fP and \f(CWprintf\fP are statements! redirection possible with \f(CW>\fP, \f(CW>>\fP, \f(CW|\fP .SS "awk: functions" .TS l lfCW. arithmetic \fRmuch like in C\fP .sp .5 I/O \f(CWgetline()\fP, \f(CWsystem()\fP .sp .5 string \f(CWsub()\fP, \f(CWgsub()\fP \& \f(CWsubstr()\fP, \f(CWindex()\fP, \f(CWmatch()\fP, \& \f(CWsplit()\fP, \f(CWsprintf()\fP \& \f(CWtolower()\fP, \f(CWtoupper()\fP, \& \f(CWlength()\fP, \f(CWint()\fP .TE .SS "awk: variables" .TS lfCW l. RS, FS record/field separator NR number of current record NF number of fields (\f(CW$1\fP ... \f(CW$NF\fP) ARGC, ARGV like in C FILENAME name of current input file OFMT output format for numbers ORS, OFS output record/field separator SUBSEP \f(CWa[1,2]\fP equals \f(CWa[1 SUBSEP 2]\fP .TE .SS "awk: conditions" .LP examples: .IT .CW "NR == 1 {...}" .IT .CW "NR == 1, NR == 5 {...}" .IT .CW "/RE/ {...}" .IT .CW $1\ ==\ "string"\ {...} .IT .CW "$1 ~ /RE/ {...}" .IT .CW "BEGIN {...}" .IT .CW "END {...}" .PA "\s-8\fR\s+8" .PP .PA "real life examples" .PP .SS literature .IT ``\fBThe Unix Programming Environment\fP'' by Kernighan and Pike is highly recommended to support this talk .IT it's one of my favorite computer books .IT you really should read it! .sp 1 .IT ``\fBSED & AWK ge-packt\fP'' by Stephan Thesing .IT if you want a german book about .CW sed and .CW awk then get this one .IT you can get it for about 3 Euro .SS .LP this talk was prepared using tools of the Heirloom project: .CW \s-2http://heirloom.sf.net\s+2 .LP the slides macros were taken from .br .CW \s-2http://repo.cat-v.org/troff-slider/\s+2 .LP all editing was done with .CW ed , of course :-) .sp .LP the slides and examples are available on my website .CW \s-2http://marmaro.de/docs\s+2 and on .CW \s-2http://ulm.ccc.de/ChaosSeminar/2009/10_Softwareschaetze\s+2 .sp 2009-10-12