Mpp, a Multilingual Pretty-Printer for Noweb

Kostas N. Oikonomou oikonomou@att.com

April 8, 2002

Contents

1 Features and usage 1

2 The basic design 2

3 Language files, translation tables, and interesting tokens 7 3.1 Language specification files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.2 Translation tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4 Language-independent pretty-printing 10 4.1 Formatting a line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 4.2 Handling the interesting tokens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4.3 Comments and quotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

5 Unresolved issues 18

A Index 19

1 Features and usage

mpp is a pretty-printer, written in Icon, for the noweb system. Its main features are

• Any chunk can be written in any language, and the pretty-printer will switch among languages. The language in which a chunk is written is indicated by writing its name in parentheses at the

  end of the chunk name: e.g. <<Comparisons of reals (C)>> or <<Filtering (Icon)>> ^{1}.

• If no language is specified, a chunk inherits the language of its parent.

• A default language is specified when invoking mpp. So for a single-language noweb file, no chunk needs to specify a language.

• Languages are described by external ASCII files. Users can add their own, or modify some of the supplied language files without touching mpp’s code.

1A file legit lang names lists the strings that are considered to name languages, so when mpp sees, for example, <<Comparisons of reals (again)>> it will not attempt to switch to language “again”. Actually there is a more general mechanism: the options -d1, -d2 allow the user to specify two strings that delimit the language name; these default to parentheses.

• mpp does not touch the user’s indentation or line-breaking.

• The spec file for language L specifies L’s reserved words, which are typeset in bold, strings such as >= which are typeset in math mode and appear as ≥, and arbitrary translations of strings into TEX code.

• The strings that denote comments or quotes are customizable, read from the spec file for L. Comments are typeset in roman font, and TEX’s math mode is active in comments.

mpp is a noweb filter, invoked as

mpp -lib ⟨path⟩ -L ⟨language⟩ [ -d1 ⟨s1 ⟩ -d2 ⟨s2 ⟩]

where the (full) library path specifies where the language spec files are to be found, and the language is the initial or default one.

2 The basic design

The pretty-printer’s design is based on the following two premises:

• It should be as independent of the target language as possible, and

• We don’t want to write a full-blown scanner for the target language.

Strings of characters of the target language which we want to typeset specially are called “interesting tokens”. There are three categories of interesting tokens:

1. Reserved words of the target language: we want to typeset them in bold, say.

2. Other strings that we want to typeset specially, usually in math mode: e.g. ≤ for <=.

3. Comment and quoting tokens (characters): we want what follows them or what is enclosed by them to be typeset literally.

By reading the language spec file, a table trans is constructed that defines a translation into TEX code of every interesting token in the target language. Here is an excerpt from the language spec file for Icon; lines beginning with # are comments.

# Reserved words
+by
+break
+case
# Keywords
+&ascii
+&clock
# Translator directives
+$include
+$line
# Mathematical translations
$<=   \le
$>=   \ge
$>>   \succ

  # Arbitrary translations
  .{     \{
  .\     \\
  .~==

Entries beginning with a “+” are typeset in bold, those beginning with a “$” define math mode translations, and entries beginning with a “.” followed by a pair of strings substitute the second string for the first. We use four sets of strings to define the tokens in categories 2 and 3:

                          special, comment1, comment2, quote2.

comment1 is for unbalanced comment strings (e.g. % in Turing, # in Icon, ! in Fortran), comment2 is for balanced comment strings (e.g. /* and */ in C, or (* and *) in Mathematica), and quote2 is for literal quotes, such as ", which we assume to be balanced.

Our approach to recognizing the interesting tokens while scanning a line is to have a set of characters interesting (an Icon cset), containing all the characters by which an interesting token may begin. interesting is the union of

• the cset defining the characters which may begin a reserved word

• the cset containing the initial characters of all strings in the special, comment, and quote sets.

The basic idea is this: given a line of text, we scan up to a character in interesting, and, depending on what this character is, we may try to complete the token by further scanning. If this succeeds, we look up the token in the trans table, and if the token is found, we output its translation, otherwise we output the token itself unchanged. When comment or quote tokens are recognized, further processing of the line may stop altogether, or temporarily, until a matching (closing) token is found.

3 ⟨* 3⟩≡

  link options, strings, fullimag

⟨Data structure holding a language spec 4⟩

  global language, trans
  global res_word_chars, special, comment1, comment2, quote2, interesting
  global begin_res_word, begin_special, begin_comment1, begin_comment2, begin_quote2
  global in_comment1, in_comment2, in_quote
  global libpath, line_num

⟨Reading the language spec file 7⟩

⟨Switching languages 8c⟩

⟨Language-independent procedures 10a⟩

⟨main procedure 5⟩

⟨Utilities 16c⟩

Defines:

begin comment1, used in chunks 4, 7, 11, and 13a.

begin comment2, used in chunks 4, 7, 11, and 13a.

begin quote2, used in chunks 4, 7, 11, 13a, and 17b.

begin res word, used in chunks 4, 7, 9, 11c, and 13a.

begin special, used in chunks 4, 7, 9, 11c, and 13a.

comment1, used in chunks 4, 7, 11b, 14b, and 17b.

comment2, used in chunks 4, 7, 11b, 15a, and 17b.

in comment1, used in chunks 5, 11a, 12, and 14b.

in comment2, used in chunks 11a, 12, and 15.

in quote, used in chunks 11a, 12, 15b, and 16b.

April 8, 2002  mpp.nw  4

interesting, used in chunks 4, 7, 11c, and 13a. language, used in chunks 5 and 6c. libpath, used in chunks 6a and 7. line num, used in chunks 5 and 13b. quote2, used in chunks 4, 7, 11b, 15b, and 17b. res word chars, used in chunks 4, 7, 8b, 13c, and 17b. special, used in chunks 4, 7, 9, 14a, and 17b. trans, used in chunks 4, 7, 9, 13c, 14a, and 17b.

4 ⟨Data structure holding a language spec 4⟩≡ (3)

record lang_spec(res_word_chars, special, comment1, comment2, quote2,
  begin_res_word, begin_special, begin_comment1, begin_comment2, begin_quote2,
  interesting, trans)

global known_langs # table of lang spec indexed by language. global legit_lang_names # set of strings that can be language names

Defines:

known langs, used in chunks 5–8 and 17b. lang spec, used in chunk 7. legit lang names, used in chunks 5, 6a, and 10a.

Uses begin comment1 3, begin comment2 3, begin quote2 3, begin res word 3, begin special 3, comment1 3, comment2 3,
interesting 3, quote2 3, res word chars 3, special 3, and trans 3.

main interacts with TeXify when it comes to comments and quotes.

5 ⟨main procedure 5⟩≡ (3)

procedure main(args)
local line, chunk_name, kind, keyword, rest, L, p0
local opts, d1, d2, f

legit_lang_names := set() # strings that can be language names known_langs := table() # languages whose specs have been loaded language := table() # indexed by chunk name, gives the language used by the chunk ⟨Set the library path and the initial language 6a⟩ ⟨TEX definitions at the top of the output file 10b⟩ line_num := 0 # line no. in the input file

while line := read() do {
line_num +:= 1
line ? {
keyword := tab(upto(' ')|0) ∧
rest := if tab(match(" ")) then {p0 := &pos;  tab(0)} else &null
}
case keyword of {
"@begin" : {
rest ? kind := tab(many(&letters))
write(line)
}
"@defn" : {
if kind == "code" then {

⟨Does this chunk set a new language L? 6b⟩

  }
write(line)
}
"@use" : {

⟨Set the language of any chunk used by this chunk to L 6c⟩

write(line)
}
"@text" :
if \kind == "code" then TeXify(rest,L,p0) else write(line)
"@nl" : {

if \in_comment1 then { # unbalanced comment

  write("@literal \\endcom{}");  in_comment1 := &null
  }
write(line)
}
"@index" | "@xref" : {

⟨Suppress if in comment or quote 11a⟩

}
default : write(line)
}
}

end

Defines:

main, never used.

Uses in comment1 3, known langs 4, language 3, legit lang names 4, line num 3, and TeXify 12.

The pretty-printer must be called with two arguments:

6a ⟨Set the library path and the initial language 6a⟩≡ (5)

opts := options(args, "-lib:-L:-d1:-d2:")
libpath := (\opts["lib"] ∥ "/") |
  stop("mpp: you must specify the full path for the language spec files!")
L := \opts["L"] | stop("mpp: you must specify a language as an argument!")
f := open(libpath ∥ "legit lang names") |
  stop("mpp: can't open `", libpath ∥ "legit lang names", "'!")
while line := read(f) do
  every insert(legit_lang_names, words(line))
if member(legit_lang_names, L) then
  if /known_langs[L] then load_language_spec(L) else switch_to_language(L)

else

  stop("mpp: language `", L, "' is not in `legit lang names'!")
d1 := \opts["d1"] | "("
d2 := \opts["d2"] | ")"
Uses known langs 4, legit lang names 4, libpath 3, load language spec 7, and switch to language 8c.

To switch languages, chunk name must end with the name of a legitimate language in parentheses. See ⟨10a⟩.

6b ⟨Does this chunk set a new language L? 6b⟩≡ (5)

chunk_name := rest
if L := get_language(chunk_name,d1,d2) then {
  if /known_langs[L] then load_language_spec(L) else switch_to_language(L)
  write("@language ", map(L, &ucase, &lcase))
  }
Uses get language 10a, known langs 4, load language spec 7, and switch to language 8c.

6c ⟨Set the language of any chunk used by this chunk to L 6c⟩≡ (5)

assert(\L)
chunk_name := rest
if \language[chunk_name] ~== L then
  stop("mpp: <", chunk_name, ">'s language already set to `", language[chunk_name], "'!")

else

  language[chunk_name] := L
Uses language 3.

3 Language files, translation tables, and interesting tokens

3.1 Language specification files

Language specification files are named Icon pp spec, C pp spec, etc. They can contain comments, which are lines beginning with #, and empty lines. The file format is described in the following table^{2}. s_{i} is a string, c_{i} is a character, C_{i} is one of the standard Icon csets (character sets), such as &letters

or &digits.

res word chars:  C_{1} C_{2} … c_{1} c_{2} …

                      comment1:  s_{1} s_{2} . . .

comment2:  s_{1} s′_{1} s_{2} s′_{2} …
quote2:  c_{1} c′_{1} c_{2} c′_{2} …
⟨translation table⟩: see §3.2

7 ⟨Reading the language spec file 7⟩≡ (3)

procedure load_language_spec(L)
local name, f, line, wlist, w, w1, e, I
name := libpath ∥ L ∥ " pp spec"
f := open(name, "r") | stop("mpp: can't open file `", name, "'!")

⟨Get next line 8a⟩

w1 == "res word chars:" | stop("mpp: `res word chars:' expected!")
res_word_chars := ''

⟨Parse the res word chars line 8b⟩ ⟨Get next line 8a⟩

w1 == "comment1:" | stop("mpp: `comment1:' expected!")
comment1 := wlist       # a list

⟨Get next line 8a⟩

w1 == "comment2:" | stop("mpp: `comment2:' expected!")
comment2 := []          # a list of pairs
while put(comment2, [get(wlist),get(wlist)])

⟨Get next line 8a⟩

w1 == "quote2:" | stop("mpp: `quote2:' expected!")

quote2 := [] # a list of pairs

while put(quote2, [get(wlist),get(wlist)])

⟨Set up the translation table and special tokens for L 9⟩

close(f)

⟨Detecting the beginning of a token 11b⟩

known_langs[L] := lang_spec(res_word_chars, special, comment1, comment2, quote2,
  begin_res_word, begin_special, begin_comment1, begin_comment2, begin_quote2,
  interesting, trans)

end

Defines:

load language spec, used in chunk 6.
Uses begin comment1 3, begin comment2 3, begin quote2 3, begin res word 3, begin special 3, comment1 3, comment2 3,
interesting 3, known langs 4, lang spec 4, libpath 3, quote2 3, res word chars 3, special 3, and trans 3.

2To understand this better, look at one of the examples.

Get the next non-comment, non-empty line and its words.

8a ⟨Get next line 8a⟩≡ (7)

while line := read(f) do
if line ~== "" ∧ line[1] ~== "#" then break
wlist := get_words(line);  w1 := get(wlist)
Uses get words 16c.

Can’t use variable and name for cset keywords because they are not variables. So

8b ⟨Parse the res word chars line 8b⟩≡ (7)

every w := !wlist do {

if *w = 1 then # character

res_word_chars ∪:= w

else { # cset

res_word_chars ∪:=

case w of {

"&letters" : &letters
"&digits"  : &digits
"&lcase"   : &lcase
"&ucase"   : &ucase
"&ascii"   : &ascii
"&cset"    : &cset
default: stop("mpp: unknown cset in `res word chars' line!")
}
}
}
Uses res word chars 3.

Rather nifty code, using Icon’s variable and name constructs to avoid a lot of assignments. For every field of Lspec named n, assign its value to the global variable named n.

8c ⟨Switching languages 8c⟩≡ (3)

procedure switch_to_language(L)
local Lspec, n
Lspec := \known_langs[L] | stop("mpp : `", L, "'should be known here!")
every n := name(!Lspec) do {
n ?:= {tab(find(".")+1) ∧ tab(0)}
variable(n) := Lspec[n]
}

end

Defines:

switch to language, used in chunk 6.
Uses known langs 4.

3.2 Translation tables

The translation table is now read from the language description file. There are four kinds of translations: make bold (entry starts with a “+”), make slanted (entry starts with a “~”), turn into a mathematical symbol (entry starts with a “$”), or arbitrary substitition (line starts with a “.”, followed by a pair of strings, of which the 2nd may be empty). As translations are read, we also create the list of special tokens, and the cset begin res word.

9 ⟨Set up the translation table and special tokens for L 9⟩≡ (7)

trans := table()  # global
special := []
begin_special := begin_res_word := ''
while line := read(f) do {
if line[1] ~== "#" ∧ line ~== "" then {
wlist := get_words(line)
w1 := wlist[1]
case w1[1] of {
"+" : w1 ? {      # make bold
move(1);  w := tab(0)
trans[w] := "{\\ttb{}" ∥ w ∥ "}"
begin_res_word ∪:= w[1]
}

"~" : w1 ? { # make slanted

move(1);  w := tab(0)
trans[w] := "{\\tts{}" ∥ w ∥ "}"
begin_res_word ∪:= w[1]
}

"$" : w1 ? { # math special token

move(1);  w := tab(0);

# We don’t use ‘‘$” in the translation because ‘‘$” might be part of the language L.

trans[w] := "\\(" ∥ wlist[2] ∥ "\\)"
put(special, w)
begin_special ∪:= w[1]
}

"." : w1 ? { # arbitrary special translation, possibly empty

move(1);  w := tab(0);
trans[w] := \wlist[2] | &null
put(special, w)
begin_special ∪:= w[1]
}

default : {

}
}
}
}
special := sort_by_length(special)
Uses begin res word 3, begin special 3, get words 16c, sort by length 17a, special 3, and trans 3.

4 Language-independent pretty-printing

Find out what is the language of chunk chunk name. It must be a legitimate language name, otherwise the string in parentheses (more generally between the delimiters d1 and d2) is ignored.

10a ⟨Language-independent procedures 10a⟩≡ (3) 12 ▷

procedure get_language(chunk_name,d1,d2)
  local L,i,n
  n := *chunk_name
  chunk_name ? {

every i := find(" " ∥ d1) # get the last occurrence if \i > 0 then {

      move(i+(*d1));  L := tab(find(d2))

if &pos = n then

        if member(legit_lang_names, L) then return L
      }
    }

end

Defines:

get language, used in chunk 6b.

Uses legit lang names 4.

First we set up the typewriter bold font \ttb, corresponding to pcrb8r, and the typewriter slanted font \tts. Then we define the macros \begcom (begin comment) and \endcom. \begcom

• switches to \rmfamily,

• activates $ by changing its catcode to 3,

• makes the characters “^” and “ ” active for superscripts and subscripts,

• changes the catcode of the space character to 10. This way comments will be typeset normally, and not as if \obeyspaces were active.

10b ⟨TEX definitions at the top of the output file 10b⟩≡ (5)

write("@literal \\DeclareFontShape{OT1}{cmtt}{bx}{n}{ <-> pcrb8r }{}")
write("@nl")
write("@literal \\def\\ttb{\\bfseries}")
write("@nl")
write("@literal \\def\\tts{\\slshape}")
write("@nl")
write("@literal \\def\\begcom{\\begingroup\\rmfamily \\catcode`\\$=3
  \\catcode`\\^=7 \\catcode`\\ =8 \\catcode`\\ =10}")
write("@nl")
write("@literal \\def\\endcom{\\endgroup}")
write("@nl")

Don’t output spurious @index use or @xref lines when in a comment or quote. (@index is produced by finduses and @xref by noidx.) However, we do want to output @index defn lines. All of this works only if the language filter is run before noidx.

11a ⟨Suppress if in comment or quote 11a⟩≡ (5)

  if (/in_comment1 ∧ /in_comment2 ∧ /in_quote) | match("defn", rest) then
    write(line)
Uses in comment1 3, in comment2 3, and in quote 3.

For each interesting category define a cset containing the characters by which a token in that category may begin and set interesting to their union.

11b ⟨Detecting the beginning of a token 11b⟩≡ (7) 11c ▷

  begin_comment1 := begin_comment2 := begin_quote2 := ''
  every e := !comment1 do begin_comment1 ∪:= cset(e[1])
  every e := !comment2 do begin_comment2 ∪:= cset(e[1][1])
  every e := !quote2 do begin_quote2 ∪:= cset(e[1])
Uses begin comment1 3, begin comment2 3, begin quote2 3, comment1 3, comment2 3, and quote2 3.

The token recognition method used in procedure TeXify assumes that the various subsets of interesting are mutually disjoint. If this assumption does not hold, the results are unpredictable.

11c ⟨Detecting the beginning of a token 11b⟩+≡ (7) ◁ 11b

  I := begin_res_word ∩ begin_comment1 ∩ begin_comment2 ∩ begin_quote2 ∩ begin_special
  *I = 0 | stop("mpp: the characters in the set\n", image(I),
    "\n may begin tokens in more than one interesting category!")
  interesting := begin_res_word ∪ begin_comment1 ∪ begin_comment2 ∪
    begin_quote2 ∪ begin_special
Uses begin comment1 3, begin comment2 3, begin quote2 3, begin res word 3, begin special 3, and interesting 3.

4.1 Formatting a line

This procedure formats @text lines in the noweb file. Note that every TEXified line is a “literal” in noweb’s sense.

12 ⟨Language-independent procedures 10a⟩+≡ (3) ◁ 10a

procedure TeXify(line, L, p0)
local token, emb, c, i, q, qs, c_open, q_open, closing
static c_close, q_close, TeXspecial

initial {TeXspecial := '\\${}&#^%~'} # The cset of characters treated specially by TEX.

writes("@literal ")
while line ~== "" do
line ? {
if \in_comment1 then {

⟨Write unbalanced comment text 15c⟩

}
else if \in_comment2 then {

⟨Write balanced comment text 15d⟩

}
else if \in_quote then {

⟨Write quoted text 16a⟩

}

else { ⟨Not inside a comment or quote 13a⟩

}

line := tab(0) # There may be more on the line!

}
write()

end

Defines:

TeXify, used in chunk 5.

Uses in comment1 3, in comment2 3, and in quote 3.

13a ⟨Not inside a comment or quote 13a⟩≡ (12)

while writes(tab(upto(interesting))) do

case &pos+1 of { # The &pos+1 is because any(C, s, i) will produce i + 1 if s[i] ∈ C.

any(begin_res_word) : {

⟨Reserved word 13c⟩

}
any(begin_special) : {

⟨Possible ‘‘special” token 14a⟩

}
any(begin_comment1) : {

⟨Possible unbalanced comment 14b⟩

}
any(begin_comment2) : {

⟨Possible balanced comment 15a⟩

}
any(begin_quote2) : {

⟨Possible quote 15b⟩

}

default :

⟨Internal error! 13b⟩

}

# Now write out the (uninteresting) rest of the line:

writes(tab(0))
Uses begin comment1 3, begin comment2 3, begin quote2 3, begin res word 3, begin special 3, and interesting 3.

Well, if we got here there’s something wrong in the scanning algorithm. p0 is the position in the line of the source file where the argument line of TeXify begins. Note: the reported column is in the Emacs sense, i.e. the first character is in column 0.

13b ⟨Internal error! 13b⟩≡ (13a)

stop("\nmpp: error in procedure TeXify:\n  language = ", L, ", input line ",
line_num, ", column ", p0+&pos-2)
Uses line num 3.

4.2 Handling the interesting tokens

Check for the situation where we have an “embedded” reserved word. E.g. suppose when is a reserved word and any letter can occur in reserved words. We don’t want when matched in so when.

13c ⟨Reserved word 13c⟩≡ (13a)

emb := any(' ', &subject, &pos-1) | &null
token := tab(many(res_word_chars))
writes((/emb ∧ \trans[token]) | token)
Uses res word chars 3 and trans 3.

There are two issues here. Suppose we want = and == to be typeset specially, but not =-. So we put = and == in special. Now what happens when we encounter =? First, we have to find out if this is really the string ==. So (a) we must match the longest token in special, in case a special token is a prefix of another special token. Second, we must check that we do not have the string =-, because we do not want it to appear in the output as the translation of = followed by “-”.

(a) is easily ensured: match(!special) will match the longest token if the list special is arranged so that longest tokens come first, as specified in §3. (b) is a bigger pain. We solve it as follows: in the example given above, we do put =- in special, but don’t define a translation for it. So

14a ⟨Possible ‘‘special” token 14a⟩≡ (13a)

  if (token := tab(match(!special)) | pos(0)) then
    writes(\trans[token] | token)

else

    writes(move(1))
Uses special 3 and trans 3.

4.3 Comments and quotes

In principle, comments and quotes could be handled by Icon procedures such as bal(), or the more sophisticated ones in procs/scan.icn. What precludes this easy solution is the fact that other filters in the noweb pipeline may break up comments and quotes that begin and end on the same line into multiple lines. For example, the finduses and noidx filters are language-independent, and so can insert spurious @index and @xref lines in the middle of commented or quoted text of the target language.

This complicates greatly the handling of balanced comments, and especially of unbalanced comments and quotes. In fact, proper handling of unbalanced comments forces procedures filter and TeXify to interact, as TeXify cannot detect the end of an unbalanced comment that has been broken up into multiple lines. So filter and TeXify interact via the variables in comment and in quote when handling comments and quotes, and it is filter that detects the end of an unbalanced comment when it encounters a @nl line. If we match a token in comment1, we output it and the rest of the line as is, but in \rm font. Within a comment, characters special to TEX are active, so $x^2$ will produce x^{2}. A problem with this is that if you comment out the (C) line printf("Hi there!\n"), TEX will complain that \n is an undefined control sequence.

14b ⟨Possible unbalanced comment 14b⟩≡ (13a)

  if writes(tab(match(!comment1))) then {
    in_comment1 := "yes"
    writes("\\begcom{}" ∥ tab(0))

break # We let filter detect the end of the comment.

    }

else

writes(move(1)) # The character wasn’t the beginning of a comment token.

Uses comment1 3 and in comment1 3.

If we are at this point, it is not necessarily true that we have found a comment. For example, in Mathematica comments begin with a (, which may also appear in x+(y+z). The additional complexity comes from the fact the we have to handle comments extending over many lines.

15a ⟨Possible balanced comment 15a⟩≡ (13a)

every c := !comment2 do {
c_open := &null
writes(c_open := tab(match(c[1]))) ∧ c_close := c[2] ∧ break
}
if \c_open then {
in_comment2 := "yes";  writes("\\begcom{}")

⟨Write balanced comment text 15d⟩

}

else

writes(move(1)) # The character wasn’t the beginning of a comment after all.

Uses comment2 3 and in comment2 3.

Quoted strings may extend over multiple lines for the reasons mentioned at the beginning of §4.3. Except for the formatting, we handle them like balanced comments. The possibility of escaped quotation marks inside the quoted string makes things more difficult.

15b ⟨Possible quote 15b⟩≡ (13a)

every q := !quote2 do {
writes(q_open := tab(match(q[1]))) ∧ q_close := q[2] ∧ break
}
if \q_open then {
in_quote := "yes"

⟨Write quoted text 16a⟩

}

else

writes(move(1)) # The character wasn’t the beginning of a quoting token.

Uses in quote 3 and quote2 3.

15c ⟨Write unbalanced comment text 15c⟩≡ (12)

writes(tab(0))

15d ⟨Write balanced comment text 15d⟩≡ (12 15a)

if writes(tab(find(c_close))) then { # Comment ends here
writes("\\endcom{}" ∥ move(*c_close))
in_comment2 := &null
}

else # Comment doesn’t close on this line

writes(tab(0))
Uses in comment2 3.

After having encountered a quote we write literally, except that we precede every character special to TEX by a backslash and follow it by an empty group^{3}. Detecting the end of a quoted string is tricky: a q close character doesn’t do it if it is escaped by a backslash, unless the backslash is itself escaped by another backslash! Below, qs is the string between quotes, or a piece of it (recall the beginning of §4.3).

16a ⟨Write quoted text 16a⟩≡ (12 15b)

if qs := tab(find(q_close)) then closing := "yes" else qs := tab(0)
qs ? {
while writes(tab(upto(TeXspecial))) do writes("\\" ∥ move(1) ∥ "{}")
writes(tab(0))
}

# This took a while to get right. Is there a simpler way to express it?

if \closing then {
if \qs[-1] then {
  if qs[-1] == "\\" then {
    if \qs[-2] then

if qs[-2] == "\\" then ⟨Quote ends here 16b⟩

    }

else ⟨Quote ends here 16b⟩

  }

else ⟨Quote ends here 16b⟩

}

16b ⟨Quote ends here 16b⟩≡ (16a)

{in_quote := closing := &null;  writes(move(1))}  # q close
Uses in quote 3.

16c ⟨Utilities 16c⟩≡ (3) 17a ▷

procedure get_words(s) # Also see words in the strings library. static it local i, L initial{it := &cset \ ' \t,'} # words are separated by blanks, tabs, or commas

L := []
s ? while tab(upto(it)) do {i := tab(many(it)); put(L,i)}

return L end

Defines:

get words, used in chunks 8a and 9.

3The empty group is necessary for the characters “˜ ” and “ˆ ”.

Sort a list of strings (or other things with size) by the length of its elements, longest first.

17a ⟨Utilities 16c⟩+≡ (3) ◁ 16c 17b ▷

procedure sort_by_length(L)
local L1, L2, s, T
T := table()
every s := !L do T[s] := -(*s)
L2 := sort(T,4)
L1 := []
while put(L1, get(L2)) do get(L2)

return L1 end

Defines:

sort by length, used in chunk 9.

17b ⟨Utilities 16c⟩+≡ (3) ◁ 17a

procedure print_language_spec(L,more)
local n, tt, s
s := \known_langs[L] | stop("mpp: `", L, "' is unknown!")
write("res word chars: ", fullimage(s.res_word_chars))
write("comment1: ", fullimage(s.comment1))
write("comment2: ", fullimage(s.comment2))
write("quote2: ", fullimage(s.quote2))
write("special: ", fullimage(s.special))
if \more then {
  write("begin quote2: ", fullimage(s.begin_quote2))
}
if \more > 1 then {
  tt := sort(s.trans,1)
  every write(fullimage(!tt))
  }

end

Defines:

print language spec, never used.
Uses begin quote2 3, comment1 3, comment2 3, known langs 4, quote2 3, res word chars 3, special 3, and trans 3.

5 Unresolved issues

1. Find a good way to handle indexing and cross-referencing when there are many languages.

2. There is a niggling unresolved issue, exemplified by Icon. mpp translates the symbol “&”as “∧”, even though “&” is not in Icon’s special. This happens because “&” is in Icon’s res word chars, and a translation for it is defined in known langs[icon].trans. So when TeXify encounters it, it recognizes it as an Icon reserved word, and uses the translation defined for it. Now if this translation is not wanted, remove “&” from known langs[icon].trans and don’t bother me any more. However, if this translation is ok, we have an inconsistency, in that “&” is not in special.

While this is not a real problem, achieving consistency (which may be needed in a more general case) is not so easy. If we add “&” to special, the check in ⟨11b⟩ will fail. To fix this, we could

(a) Add a constraint to the recognition of a reserved word: it has to be a token of length > 1.

(b) Revise the case structure in ⟨13a⟩, as it will no longer work.

We could also consider having a separate translation table for special tokens.

A Index

begin comment1: \underline{3}, 4, 7, 11b, 11c, 13a begin comment2: \underline{3}, 4, 7, 11b, 11c, 13a begin quote2: \underline{3}, 4, 7, 11b, 11c, 13a, 17b begin res word: \underline{3}, 4, 7, 9, 11c, 13a begin special: \underline{3}, 4, 7, 9, 11c, 13a comment1: \underline{3}, 4, 7, 11b, 14b, 17b comment2: \underline{3}, 4, 7, 11b, 15a, 17b

get language: 6b, \underline{10a}

get words: 8a, 9, \underline{16c} in comment1: \underline{3}, 5, 11a, 12, 14b in comment2: \underline{3}, 11a, 12, 15a, 15d in quote: \underline{3}, 11a, 12, 15b, 16b interesting: \underline{3}, 4, 7, 11c, 13a known langs: \underline{4}, 5, 6a, 6b, 7, 8c, 17b

lang spec: \underline{4}, 7
language: \underline{3}, 5, 6c
legit lang names: \underline{4}, 5, 6a, 10a
libpath: \underline{3}, 6a, 7

line num: \underline{3}, 5, 13b

load language spec: 6a, 6b, \underline{7}
main: \underline{5}
print language spec: \underline{17b}

quote2: \underline{3}, 4, 7, 11b, 15b, 17b res word chars: \underline{3}, 4, 7, 8b, 13c, 17b

sort by length: 9, \underline{17a}

special: \underline{3}, 4, 7, 9, 14a, 17b

switch to language: 6a, 6b, \underline{8c}
TeXify: 5, \underline{12}

trans: \underline{3}, 4, 7, 9, 13c, 14a, 17b