PDF shorthand

There are periods of time when I have to do a lot of article editing or book writing, and I usually do that in LaTeX. I find that writing dockets in LaTeX helps me focus on the content, maintain coherent style throughout the document, and produce better-looking results... and I lost my patience with office suites in general.

There is however, one thing that annoys me. Since I tend to just use a text editor (vim, or kate, or gedit) I usually need to go into terminal and type in the four commands to compile the document into a pdf:

pdflatex meh.tex && \
bibtex meh && \
pdflatex meh.tex && \
pdflatex meh.tex

And since I'm kinda compulsive I do that kind of often. And when it's not in the history of commands (because e.g., I'd been writing a script during a break in editing) I get annoyed.

So I figured I could do a simple script that would take care of those commands. A simple script that just takes the name of the tex file and returns a pdf. And I did just that.

And then I started fiddling with it and adding silly features: making it quiet, interactive, whatever; and then a getopt interface on top of it all. So it got kind of convoluted by the end. Feature-wise it probably lays smack in the middle between lightweight and feature-rich. I.e., it is slightly too complex but doesn't have the feature you want.

But I still like it.

If you're looking for fun stuff, the usage function is interesting, even if I say so myself. Instead of writing out a sequence of echo commands explaining what the program does, I decided to use the comments I write on the top of the scripts always. So the usage command finds and parses the script file, and then, using awk yanks the large comment, strips it of the hash character in front, and prints it out to the user. It saved time, and I should remember to sue it in the future in this context.

Some examples of usage now. First, make pdf from tex, bibliography and all:

pdfshorthand top.tex

To make it not produce all those pesky temporary documents (ask before removing each file or not):

pdfshorthand -c top.tex
pdfshorthand -cf top.tex

And if you have no bibliography in your docket, you need to use the b flag:

pdfshorthand -b top.tex
pdfshorthand -bc top.tex
pdfshorthand -bcf top.tex

The code:

 
  1 #!/bin/bash 
  2 # 
  3 # My shorthand for pdflatex 
  4 # 
  5 # Because I'm sick and tired of always putting in the same command over and  
  6 # over again just to compile a file into a PDF. This script does it for me. 
  7 # It's basically the equivalent for writing: 
  8 #   pdflatex top.tex && bibtex top && pdflatex top.tex && pdflatex top.tex and  
  9 # then cursing because I'd have much preferred it to halt on error but that is 
 10 # way too much writing. 
 11 # 
 12 # Usage: 
 13 #   shorthand [OPTION ...] FILE [FILE ...] 
 14 #    
 15 #   The FILE can but doesn't have to end with the extension 'tex'. If it isn't, 
 16 #   one will be added before passing the file to pdflatex; if it is, it will be 
 17 #   removed before passing the file to bibtex. 
 18 # 
 19 # Options: 
 20 #   -n|--no-colors    Do not use colors in output. Normally, colors are used  
 21 #                     for key messages: on success, on failure, and when  
 22 #                     compilation starts. 
 23 #    
 24 #   -b|--no-biblio    Do not attempt to generate a bibliography using bibtex. 
 25 # 
 26 #   -i|--interactive  Run pdflatex in interactive mode (halt on error). 
 27 #                     Otherwise the compilation is run in non-stop mode. 
 28 # 
 29 #   -q|--quiet        Do not show any output from the command whatsoever. 
 30 #                     Else, compilation produces a lot of output (be advised). 
 31 # 
 32 #   -c|--clean-up     Clean up after the compilation - remove aux, blg, bbl,  
 33 #                     and log files from the working directory. The files will 
 34 #                     be backed up in /tmp/ec.XXX/ just in case though, and  
 35 #                     you will be asked before the actual removal of each file 
 36 #                     takes place. 
 37 # 
 38 #   -f|--force        When used with -c or --clean-up, the script will not ask 
 39 #                     before removing each temporary file from the working  
 40 #                     directory. 
 41 # 
 42 #   -h|--help         Print usage information. 
 43 # 
 44 #   --latex-command   Specify a command to run instead of pdflatex. It will be 
 45 #                     given some arguments that pdflatex would have been given 
 46 #                     i.e., -interaction MODE and -non-stop-mode. 
 47 # 
 48 #   --bibtex-command  Specify a command to run instead of bibtex. 
 49 # 
 50 # Example: 
 51 #   shorthand top.tex 
 52 # 
 53 # Requires: 
 54 #   pdflatex 
 55 #   bibtex 
 56 # 
 57 # Author: 
 58 #   Konrad Siek <konrad.siek@gmail.com> 
 59 # 
 60 # License: 
 61 #   Copyright 2010 Konrad Siek 
 62 # 
 63 #   This program is free software: you can redistribute it and/or modify it  
 64 #   under the terms of the GNU General Public License as published by the Free  
 65 #   Software Foundation, either version 3 of the License, or (at your option)  
 66 #   any later version. See  <http://www.gnu.org/licenses/> for details. 
 67 # 
 68  
 69 texcommand=pdflatex     # name or path to command 
 70 bibcommand=bibtex       # name or path to command 
 71 quiet=$false            # true or $false 
 72 interactive=$false      # true or $false 
 73 colors=true             # true or $false 
 74 cleanup=$false          # true or $false 
 75 force=$false            # true or $false 
 76 no_biblio=$false        # true or $false 
 77  
 78 # Print usage information. 
 79 usage () { 
 80     gawk '/^#/ && NR>1 {sub(/^#[ \t]?/,""); print; next} NR>1 {exit}' "$0" 
 81 } 
 82  
 83 # Print stuff to standard error. 
 84 stderr() { 
 85     echo $@ &> 2 
 86 } 
 87  
 88 # Parse options. 
 89 options=$(\ 
 90     getopt \ 
 91     -o ciqnhfb \ 
 92     --long clean-up,interactive,quiet,latex-command:,bibtex-command:,no-colors,force,help,no-biblio \ 
 93     -n $0 -- "$@" \ 
 94 ) 
 95  
 96 # Stop if there's some sort of problem. 
 97 if [ $? != 0 ] 
 98 then 
 99     [ $quiet ] && syserr "Argh! Parsing went pear-shaped!" 
100     exit 1 
101 fi 
102  
103 # Set the parsed command options and work with the settings. 
104 eval set -- "$options" 
105 while true  
106 do 
107     case "$1" in 
108     --latex-command) texcommand="$2"; shift 2;; 
109     --bibtex-command) bibcommand="$2"; shift 2;; 
110     -c|--clean-up) cleanup=true; shift;; 
111     -n|--no-colors) colors=$false; shift;; 
112     -f|--force) force=true; shift;; 
113     -i|--interactive) interactive=true; shift;; 
114     -q|--quiet) quiet=true; shift;; 
115     -h|--help) usage; exit 1;; 
116     -b|--no-biblio) no_biblio=true; shift;; 
117     --) shift; break;; 
118     *) syserr "Ack! She cannae take it anymore, captain!"; exit 3;; 
119     esac 
120 done 
121  
122 # Various messages printed out by the script. 
123 SUCCESS="Huzzah!" 
124 FAIL="Your shipment of fail has arrived." 
125 PROCESSING="Processing file <{}>" 
126  
127 # If no params given, print usage and quit with an error. 
128 [ $# -lt '1' ] && (usage; exit 1) 
129  
130 # Specify the params for pdflatex for whatever level of interactiveness the  
131 # user expects. We do this beforehand. 
132 if [ $interactive ] 
133 then 
134     iparams="-interaction errorstopmode -halt-on-error" 
135 else 
136     iparams="-interaction nonstopmode" 
137 fi 
138  
139 clean_up () { 
140     if [ $cleanup ] 
141     then 
142         dir=`mktemp --tmpdir -d "$1.XXX"` && \
143         for ext in bbl blg log aux out nav snm
144         do 
145             f="$1.$ext"; cp -v "$f" "$dir/$f" 
146             if [ -f "$f" ] 
147             then 
148                 [ $force ] && rm -vf "$f" || rm -vi "$f" 
149             else 
150                     "Omitting file: $f" 
151             fi 
152         done 
153         cp -v "$1.tex" "$dir/" 
154     fi 
155 } 
156  
157 # If the user wants colors, the messages can be prepared beforehand. 
158 if [ $colors ] 
159 then 
160     # Add coloring control characters to the message strings: 
161     # success, green; fail, red; processing, white; all bold. 
162     SUCCESS="\033[32m\033[1m$SUCCESS\033[m" 
163     FAIL="\033[31m\033[1m$FAIL\033[m" 
164     PROCESSING="\033[37m\033[1m$PROCESSING\033[m" 
165 fi 
166  
167 # If bibtex is not supposed to be run at all, run the 'true' command instead. 
168 # The 'true' command will take whatever parameters and do absolutely nothing, 
169 # then it'll report a success. So it fits perfectly with the other commands. 
170 if [ $no_biblio ] 
171 then 
172     bibcommand=true 
173 fi 
174  
175 # The shorthand method for the pdflatex/bibtex combination. 
176 shorthand () { 
177     # Cut off the extension, if any to create the bibtex argument, 
178     dir=`dirname "$1"` 
179     base=`basename "$1" .tex` 
180     stump="$dir/$base" 
181  
182     # And now stick the extension back on to create the pdflatex argument. 
183     full="$stump.tex" 
184  
185     # Print out what file is being processed. 
186     echo -e ${PROCESSING//\{\}/$full}  
187  
188     # The pipe, where all the magic happens: 
189     $texcommand $iparams "$full" && \
190     $bibcommand "$stump" && \
191     $texcommand $iparams "$full" && \
192     $texcommand $iparams "$full" && \
193     (clean_up "$base"; echo -e $SUCCESS) || echo -e $FAIL 
194  
195     # Finally print out an information whether the processing was a success or 
196     # a failure. 
197 } 
198  
199 # Do the thing. 
200 for arg
201 do 
202     # If quiet mode is on, send all results to the black hole. 
203     [ $quiet ] && shorthand "$arg" 1> /dev/null 2>/dev/null || shorthand "$arg" 
204 done 

The code is also available at GitHub as bash/pdfshorthand.

Youtube-lst

Yeah... I had this done in Ocaml a while back... but I lost it somewhere, so I rewrote it in Bash + AWK. Good news is that it turned out to be much more simple.

I also tried to figure out if this is legal or not. I do not know this, in the end, but I figure, that all I'm doing is looping through a list, and youtube-dl is doing all the work, really.

Yeah, so it follows that youtube-dl is, of course, a dependency or a requirement.

Oh... and I didn't mention what it does yet, either... Well, if you want to download a video from Youtube to watch it later, while you're sitting in your hotel room deprived of your only link to civilization, the Internet, courtesy of your employer, or similar - in that situation you use youtube-dl, and everything's fine. If you're there for weeks on end though, you might want to get maybe 10 videos or something - then you can use this program.

First, you make a file, which lists all the files you want to look at. These files include either just lines with addresses, like this:

http://www.youtube.com/watch?v=nQmAGBbOmas
http://www.youtube.com/watch?v=-bMdTmRae6c

or with filenames like this:

-o megadeth.flv http://www.youtube.com/watch?v=1qKGZ4Ysy5M

You can put other parameters for youtube-dl in there as well - you can see in line 26 that this entire string is just inserted as-is into the command.

Here's the code:

 
 1 #!/bin/bash 
 2 #  
 3 # Batch downloader for youtube-dl 
 4 #  
 5 # Prepare a list of addresses in a file and get 
 6 # then with just one command, which does all the 
 7 # tedious copy-and-pasting for you... 
 8 # 
 9 # Parameters 
10 #   Files contaning adresses. 
11 #   If you need to rename the files, prefix the 
12 #   addresses with: -o <filename>.flv 
13 # Requires 
14 #   youtube-dl to do the actual downloading 
15 #   (http://www.arrakis.es/~rggi3/youtube-dl/) 
16 # Author 
17 #   Konrad Siek 
18  
19 # Expect each argument to be a file 
20 # and loop through all of these. 
21 for file in $@ 
22 do 
23     # Download each resource thorugh youtube-dl. 
24     cat $file | \
25     awk '/^[ \t]*$/ {next}{system("youtube-dl "$0)}' 
26 done 

Note that youtube-dl was developed on the desert planet.

The code is also available at GitHub as bash/youtube-lst.

SQL embedder

Well, have you ever been tired of making a nice little SQL script and then having to put all those damn "'s and +'s?

Well, here's a thing for you - an AWK script to parse the SQL script into those pesky things.

And here's how to use it - simple operation will just copy the string, preserving the indentation and turn it into Java-style code, like this:

"select * " +
"from " +
     "employees " ;

You need to feed a file through it though, so it's called like this:

cat select.sql | ./sql_embedder.awk

And here's a big request - it preserves the indentation in the generated code, as well as in the actual SQL script after evaluation, if prefixes the whole thing with comment tokens (for some reason) and replaces all dollar signs with the word hello and all percent signs with the word world...

cat select.sql | ./sql_embedder.awk -v indent=true -v prefix="//\t" -v replace='\$->hello,%->world'

Want to know more, read the code comments, and read the code if you want to.

And it is code:

1  #!/usr/bin/awk -f 
2  # 
3  # SQL Embedder
4  # 
5  # Wraps SQL statements so that they become string literals
6  # in various languages. You sometimes have to do that, and
7  # it's probably the most boring part of low-level database
8  # development.
9  # 
10 # Parameters
11 #   language - select the output language, takes values
12 #               'java', 'php'; Java is the default
13 #   prefix - include this string before each output line
14 #   indent - include indentation in the output strings 
15 #               (i.e. output "\tselect *\n"), takes values
16 #               'true'/'false' or 'yes'/'no'
17 #   replace - if the specified token is found, replace it 
18 #               with the specified string, it takes the
19 #               following syntax:
20 #                   "p0->s0,...,pn->sn"
21 #                   p0-pn are placeholders and s0-sn are
22 #                   substitutions. Placeholders cannot 
23 #                   contain any whitespaces and they are
24 #                   treated as POSIX regular expressions.
25 #               (i.e. to insert the string 'joe' for '$':
26 #                   "$->joe" 
27 #                it's really easier than it looks)
28 #   rule_separator - modify the rule separator of the 
29 #               'replace' clause; comma by default
30 #   implication - modify the rule implication of the 
31 #               'replace' clause; '->' by default
32 #
33 # Author
34 #   Konrad Siek
35 
36 # Do all those tedious pre-op things
37 BEGIN {
38 
39     # Select language and define tokens
40     if (language == "java" || language == "") {
41         # Java (acts as default settings)
42         STRING_TOKEN = "\""
43         CONCATENATION_TOKEN = "+"
44         INSTRUCTION_TERMINATOR = ";"
45     } else if (language == "php") {
46         # PHP
47         STRING_TOKEN = "\""
48         CONCATENATION_TOKEN = "."
49         INSTRUCTION_TERMINATOR = ";"
50     } else {
51         print "Unsupported language: " language
52         exit
53     }
54 
55     # Rename the variables
56     OUTPUT_INDENTATION = (indent ~ /(true|yes)/)
57     PREFIX = prefix
58 
59     # Initiate rule separator
60     if (rule_separator == "") {
61         RULE_SEPARATOR = ","
62     } else {
63         RULE_SEPARATOR = rule_separator
64     }
65     
66     # Initiate separators within rules
67     if (implication == "") {
68         IMPLICATION = "->"
69     } else {
70         IMPLICATION = implication
71     }
72 
73     # Initialize replacement table
74     if (length(replace) > 0) {
75         split(replace, rules, RULE_SEPARATOR)
76         for (r in rules) {
77             split(rules[r], a, IMPLICATION)
78             key = a[1]
79             value = a[2]
80             REPLACEMENTS[key] = value
81         }                 
82     }
83 }
84 
85 # Print terminator
86 END {
87     print INSTRUCTION_TERMINATOR
88 }
89 
90 # Ignore empty lines
91 /^[ \t]*$/ {
92     next
93 }
94 
95 # Concatenate previous line to this one
96 NR > 1 {
97     printf("%s\n", CONCATENATION_TOKEN)
98     indentation = ""
99     if (OUTPUT_INDENTATION) {
100        code_indentation = ""
101    }
102}
103
104# Mimic original indentation
105/^[ \t]+/ {
106    line_length = length($0)
107    for(i = 1; i < line_length; i++) {
108        char = substr($0, i, i)
109        if (char ~ /^[ \t]*$/) {
110            indentation = indentation char
111            if (OUTPUT_INDENTATION) {
112                if(indentation == "\t") {
113                    code_indentation = code_indentation "\\t"
114                } else {
115                    code_indentation = code_indentation char
116                }
117            }
118        } else {
119            break;
120        }
121    }
122}
123
124# Print a line of the statement word-by-word
125{
126    printf("%s", PREFIX)
127    printf("%s", indentation)
128    printf("%s", STRING_TOKEN)
129    if (OUTPUT_INDENTATION) {
130        printf("%s", code_indentation)    
131    }        
132    for (i = 1; i <= NF; i++) {
133        word = $i
134        for (p in REPLACEMENTS) {
135            s = REPLACEMENTS[p]
136            gsub(p, s, word)
137        }
138        printf("%s ", word)
139    }
140    if (OUTPUT_INDENTATION) {
141        printf("\\n")
142    }
143    printf("%s ", STRING_TOKEN)    
144}

The code is also available at GitHub as awk/sql_embedder.awk.

Gallerizer

This one's easy.

I needed something to put together really simple galleries of JPG files (battle reports, actually, but nevermind) so put together this glorious little bit of AWK code. It just made a list of files into a big HTML file, which basically consists of images and paragraphs underneath, which are empty and need to be filled out.

So then I though: "hey! why not fill it out on-the-fly?" So I figured this could be done in two ways, basically - with zenity or the bash read command. So why not implement them both? One for GUI-like action, and the other for a straightforward command-line thing.

Zenity, was easy (line 57), just as soon as I figured out how to make system calls from AWK... so cool! With a little bit of checking whether ok or cancel was pressed, it all came together, and you can even stop the process, if you get bored, or something.

Bash read didn't work for some reason - couldn't get the read command to actually take input from the user... oh well, 2 out of 3 ain't bad.

I will take suggestions gladly, though.

You run the thing like this, if you want to run in vanilla mode, on the current directory, and you have the script saved as gallerizer.awk:

ls | ./gallerizer.awk > gallery.html

Then just go through the gallery.html file and put all the descriptions into the paragraphs...

And if you want to run it all with zenity-powered interactive mode, go:

ls | ./gallerizer.awk -v interactive=true > gallery.html

And there you go.

The code:

1 #!/usr/bin/awk -f
2 #
3 # Gallerizer
4 #
5 # Create an extremely simple gallery, with places to insert 
6 # descriptions, or even insert descriptions on the fly with 
7 # the bash read command or zenity.
8 # 
9 # Parameters
10#   interactive: 'yes' or 'true' turns on the interctive
11#               mode with zenity input windows.
12# Requires
13#   zenity (for displaying dialogs in interactive mode)
14# Author
15#   Konrad Siek
16
17# Print HTML header
18BEGIN {
19    print "<html>"
20    print "\t<head>"
21    print "\t\t<title>"title"</title>"
22    print "\t\t<style>"
23    print "\t\t\tbody {"
24    print "\t\t\t\ttext-align: center;"
25    print "\t\t\t}"
26    print "\t\t\tp {"
27    print "\t\t\t\tmargin-bottom: 50px;"
28    print "\t\t\t}"
29    print "\t\t</style>"
30    print "\t</head>"
31    print "\t<body>"
32    if (title !~ /^[ \t\n]*$/) {
33        print "\t\t<h1>"title"</h1>"
34    }
35
36    is_interactive = interactive ~ /^(yes|true)$/
37}
38
39# Print HTML footer
40END {
41    print "\t</body>"
42    print "</html>"
43}
44
45# Ignore all temporary (files ending in '~')
46/~$/ {
47    next;
48}
49
50# Include all PNG or JPG files as images
51/\.(png|PNG|jpg|JPG)$/    {
52    print "\t\t<img id=\""$0"\" src=\""$0"\" alt=\""$0"\" />"
53    print "\t\t<p>";    
54    # Prompt for description in interactive mode
55    if (is_interactive) {
56        printf("\t\t\t")
57        r = system("zenity --entry --title=\""$0"\"")
58        if (r != 0) {
59            printf("\n")
60        }
61    }
62    print "\t\t</p>";    
63    # Kill off the script on cancel
64    if(r != 0 && is_interactive) {
65        exit
66    }
67}

The code is also available at GitHub as awk/gallerizer.awk.

Word count

Hello. Long time, no see.

Well, I had no script idead and I've been busy. Alas, here's a short script for people enjoying the challenge of NaNoWriMo.

There's of course a couple of ways to count words. If you're using some sort of office suite it's probably built in, so no problem. If you're using LaTeX, like me (because I have a LaTeX fetish) you might have it in the tool you're using too, but it's less likely.

But you want to count words anyway, so what do you do?

Well, first of all, use the Linux wc command. It does well and there are no problems. Also, to get rid of the LaTeX code from the file you can use the untex tool, which has a ton of options to choose from to have a personalized and accurate experience of removing TeX tags from the code. You just read the tex files, and save the output somewhere...

So, most of what I did was to put it all together, like so:

 
1 #!/bin/bash 
2 output=raw.txt
3 rm -f $output 
4 for file in `ls chapter-*.tex` 
5 do 
6     untex -e $file >> $output 
7 done 
8 echo -e "Word count: \n wc\t$(cat $output | wc -w ) \n awk\t$(./count.awk $output)";

I set it up, so it only reads in files, whose names start with 'chapter-' and end with '.tex', because that is just the structure I use. However, the change to any other convention can easily be applied in line number 4 by parameterizing the ls command differently.

Additionally, it produces a raw.txt file as a side effect, which contains the actual text, which got the words counted, so if you want to verify untex or any of the counting mechanisms, you can do that easily.

Also, if you look closely, you will see that in line 8 there's something extra. I call an AWK script to provide some other word count. Here's how the script looks like inside:

1 #!/usr/bin/awk -f
2 {
3     for (i = 1; i <= NF; i++) {
4         word = $i;
5         #insert punctuation here, between the square brackets.
6         n = split(word, a, /[-,.?!~`';:"'|\/@#$%^&*_-+={}\[\]<>()]+/); 
7         for (j = 1 ; j <= n; j++) {
8             if (a[j] !~ /^[ \t\n]*$/) {                
9                 words++;
10            }
11        }
12    }
13}
14
15BEGIN {
16    words = 0;
17}
18
19END {
20    print words;
21}

What it actually does, is count the words, but unlike the wc command, it tries to recognize punctuation, and split words by the punctuation as well, so that hyphenated words are split. Also, it finds out stuff like long hyphens (LaTeX: '--') and removes them, so they are no longer counted as words.

I don't know which one is more accurate, but between the two, I can always have an optimistic and a pessimistic assumption about how many words I wrote.

The code is also available at GitHub as awk/count.awk
and bash/word_count.