--- title: Monadic Parsing --- > import Data.Char > import Data.Functor > import Control.Monad Before we continue, a word from our sponsors: **Don't Fear Monads** They are simply an (extremely versatile) abstraction, like `map` or `fold`. What is a Parser? ================= A parser is a piece of software that takes a raw `String` (or sequence of bytes) and returns some structured object, for example, a list of options, an XML tree or JSON object, a program's Abstract Syntax Tree and so on. Parsing is one of the most basic computational tasks. *Every* serious software system has a parser tucked away somewhere inside, for example - Shell Scripts (command-line options) - Web Browsers (duh!) - Games (level descriptors) - Routers (packets) - etc (Indeed I defy you to find any serious system that *does not* do some parsing somewhere!) The simplest and most accurate way to think of a parser is as a function ~~~~~{.haskell} type Parser = String -> StructuredObject ~~~~~ Composing Parsers ----------------- The usual way to build a parser is by specifying a grammar and using a parser generator (eg yacc, bison, antlr) to create the actual parsing function. While elegant, one major limitation of the grammar based approach is its lack of modularity. For example, suppose I have two kinds of primitive values `Thingy` and `Whatsit`. ~~~~~{.haskell} Thingy : rule { action } ; Whatsit : rule { action } ; ~~~~~ If you want a parser for *sequences of* `Thingy` and `Whatsit` we have to painstakingly duplicate the rules as ~~~~~{.haskell} Thingies : Thingy Thingies { ... } EmptyThingy { ... } ; Whatsits : Whatsit Whatsits { ... } EmptyWhatsit { ... } ; ~~~~~ This makes sub-parsers hard to reuse. Next, we will see how to *compose* mini-parsers for sub-values to get bigger parsers for complex values. To do so, we will generalize the above parser type a little bit, by noting that a (sub-)parser need not (indeed, will not) consume consume *all* of its input, and so we can simply have the parser return the unconsumed input ~~~~~{.haskell} type Parser = String -> (StructuredObject, String) ~~~~~ Of course, it would be silly to have different types for parsers for different kinds of objects, and so we can make it a parameterized type One last generalization: the parser could return multiple results, for example, we may want to parse the string ~~~~~{.haskell} "2 - 3 - 4" ~~~~~ either as ~~~~~{.haskell} Minus (Minus 2 3) 4 ~~~~~ or as ~~~~~{.haskell} Minus 2 (Minus 3 4) ~~~~~ So, we can have our parsers return a *list* of possible results (where the empty list corresponds to a failure to parse.) > newtype Parser a = P (String -> [(a, String)]) The above is simply the parser (*cough* action) the actual parsing is done by > doParse (P p) s = p s Lets build some parsers! Parse A Single character ------------------------- Here's a *very* simple character parser, that returns the first `Char` from a list if one exists > oneChar :: Parser Char > oneChar = P (\cs -> case cs of > c:cs' -> [(c, cs')] > _ -> []) > twoChar0 = P (\cs -> case cs of > c1:c2:cs' -> [((c1,c2), cs')] > _ -> []) > twoChar9 = oneChar `pairP` oneChar Lets run the parser ~~~~~{.haskell} ghci> doParse oneChar "hey!" [('h',"ey!")] ghci> doParse oneChar "" [] ~~~~~ Parser Composition ------------------ We can write a combinator that takes two parsers and returns a new parser that returns a pair of values ~~~~~{.haskell} pairP :: Parser a -> Parser b -> Parser (a,b) pairP p1 p2 = P (\cs -> [((x,y), cs'') | (x, cs' ) <- doParse p1 cs, (y, cs'') <- doParse p2 cs'] ) ~~~~~ Now we can write another parser that grabs a pair of `Char` values ~~~~~{.haskell} twoChar :: Parser (Char, Char) twoChar = P (\cs -> case cs of c1:c2:cs' -> [((c1, c2), cs')] _ -> []) ~~~~~ or more elegantly as > twoChar = pairP oneChar oneChar which would run like this ~~~~~{.haskell} ghci> doParse twoChar "hey!" [(('h','e'), "y!")] ghci> doParse twoChar "" [] ~~~~~ Now we could keep doing this, but often to go forward, it is helpful to step back and take a look at the bigger picture. Here's the the *type* of a parser ~~~~~{.haskell} newtype Parser a = P (String -> [(a, String)]) ~~~~~ it should remind you of something else, remember this? ~~~~~{.haskell} type ST a = State -> (a, State) ~~~~~ Parser is A Monad ================= Indeed, a parser, like a state transformer, [is a monad!][2] if you squint just the right way. We need to define the `return` and `>>=` functions. The first is very simple, we can let the types guide us ~~~~~{.haskell} :type returnP returnP :: a -> Parser a ~~~~~ which means we must ignore the input string and just return the input element > returnP x = P (\cs -> [(x, cs)]) The bind is a bit more tricky, but again, lets lean on the types ~~~~~{.haskell} :type bindP bindP :: Parser a -> (a -> Parser b) -> Parser b ~~~~~ so, we need to suck the `a` values out of the first parser and invoke the second parser with them on the remaining part of the string. > p1 `bindP` fp2 = P (\cs -> > [(y, cs'') | (x, cs') <- doParse p1 cs > , (y, cs'') <- doParse (fp2 x) cs']) Armed with those, we can officially put brand parsers as monads > instance Monad Parser where > (>>=) = bindP > return = returnP Parser Combinators ================== Since parsers are monads, we can write a bunch of high-level combinators for composing smaller parsers into bigger ones. For example, we can use our beloved `do` notation to rewrite the `pairP` as > pairP p1 p2 = do x <- p1 > y <- p2 > return (x, y) shockingly, exactly like the `pairs` function [from here]($root/lectures/monads2.html). Next, lets flex our monadic parsing muscles and write some new parsers. It will be helpful to have a a *failure* parser that always goes down in flames (returns `[]`) > failP = P $ const [] Seems a little silly to write the above, but its helpful to build up richer parsers like the following which parses a `Char` *if* it satisfies a predicate `p` > satP :: (Char -> Bool) -> Parser Char > satP p = do > c <- oneChar > if p c then return c else failP we can write some simple parsers for particular characters > lowercaseP = satP isAsciiLower ~~~~~{.haskell} ghci> doParse (satP ('h' ==)) "mugatu" [] ghci> doParse (satP ('h' ==)) "hello" [('h',"ello")] ~~~~~ The following parse alphabet and numeric characters respectively > alphaChar = satP isAlpha > digitChar = satP isDigit and this little fellow returns the first digit in a string as an `Int` > digitInt = do > c <- digitChar > return $ ord c - ord '0' which works like so ~~~~~{.haskell} ghci> doParse digitInt "92" [(9,"2")] ghci> doParse digitInt "cat" [] ~~~~~ Finally, this parser will parse only a particular `Char` passed in as input > char c = satP (== c) A Nondeterministic Choice Combinator ------------------------------------ Next, lets write a combinator that takes two sub-parsers and nondeterministically chooses between them. > chooseP :: Parser a -> Parser a -> Parser a How would we go about encoding *choice* in our parsers? Well, we want to return a succesful parse if *either* parser succeeds. Since our parsers return multiple values, we can simply return the *union* of all the results! > p1 `chooseP` p2 = P (\cs -> doParse p1 cs ++ doParse p2 cs) We can use the above combinator to build a parser that returns either an alphabet or a numeric character > alphaNumChar = alphaChar `chooseP` digitChar When we run the above we get some rather interesting results ~~~~~{.haskell} ghci> doParse alphaNumChar "cat" [('c',"at")] ghci> doParse alphaNumChar "2cat" [('2',"cat")] ghci> doParse alphaNumChar "2at" [('2',"at")] ~~~~~ What is even nicer is that if *both* parsers succeed, you end up with all the results. For example, heres a parser that grabs `n` characters from the input > grabn :: Int -> Parser String > grabn n | n <= 0 = return "" > | otherwise = do c <- oneChar > cs <- grabn (n-1) > return (c:cs) > grabn' n = mapM (\_ -> oneChar) [1..n] > grabn'' n = replicateM n oneChar can you remove the recursion from that? Now, we can use our choice combinator > grab2or4 = grabn 2 `chooseP` grabn 4 and now, we will get back *both* results if possible ~~~~~{.haskell} ghci> doParse grab2or4 "mickeymouse" [("mi","ckeymouse"),("mick","eymouse")] ~~~~~ and only one result if thats possible ~~~~~{.haskell} ghci> doParse grab2or4 "mic" [("mi","c")] ghci> doParse grab2or4 "m" [] ~~~~~ Even with the rudimentary parsers we have at our disposal, we can start doing some rather interesting things. For example, here is a little calculator. First, we parse the operation > intOp :: Parser (Int -> Int -> Int) > intOp = plus `chooseP` minus `chooseP` times `chooseP` divide > where plus = char '+' >> return (+) > minus = char '-' >> return (-) > times = char '*' >> return (*) > divide = char '/' >> return div (can you guess the type of the above parser?) And then we parse the expression > calc = do x <- digitInt > op <- intOp > y <- digitInt > return $ x `op` y which, when run, will both parse and calculate ~~~~~{.haskell} ghci> doParse calc "8/2" [(4,"")] ghci> doParse calc "8+2cat" [(10,"cat")] ghci> doParse calc "8/2cat" [(4,"cat")] ghci> doParse calc "8-2cat" [(6,"cat")] ghci> doParse calc "8*2cat" [(16,"cat")] ~~~~~ HEREHEREHEREHEREHERE Recursive Parsing ----------------- To start parsing interesting things, we need to add recursion to our combinators. For example, its all very well to parse individual characters (as in `char` above) but it would a lot more swell if we could grab particular `String` tokens. Lets try to write it! > string :: String -> Parser String ~~~~~{.haskell} string "" = return "" string (c:cs) = do char c string cs return $ c:cs ~~~~~ Ewww! Is that explicit recursion ?! Lets try again (can you spot the pattern) > string = mapM char Much better! ~~~~~{.haskell} ghci> doParse (string "mic") "mickeyMouse" [("mic","keyMouse")] ghci> doParse (string "mic") "donald duck" [] ~~~~~ Ok, I guess that wasn't really recursive then after all! Lets try again. Lets write a combinator that takes a parser `p` that returns an `a` and returns a parser that returns *many* `a` values. That is, it keeps grabbing as many `a` values as it can and returns them as a `[a]`. > manyP :: Parser a -> Parser [a] > manyP p = many1 `chooseP` many0 > where many0 = return [] > many1 = do x <- p > xs <- manyP p > return (x:xs) > maxManyP p = P $ \cs -> case doParse (manyP p) cs of > y:_ -> [y] > [] -> [] Beware, the above can yield *many* results ~~~~~{.haskell} ghci> doParse (manyP digitInt) "123a" [([],"123a"),([1],"23a"),([1,2],"3a"),([1,2,3],"a")] ~~~~~ which is simply all the possible ways to extract sequences of integers from the input string. Deterministic Maximal Parsing ----------------------------- Often we want a single result, not a set of results. For example, the more intuitive behavior of `many` would be to return the maximal sequence of elements and not *all* the prefixes. To do so, we need a *deterministic* choice combinator > (<|>) :: Parser a -> Parser a -> Parser a > p1 <|> p2 = P $ \cs -> case doParse (p1 `chooseP` p2) cs of > [] -> [] > x:xs -> [x] > The above runs choice parser but returns only the first result. Now, we can revisit the `manyP` combinator and ensure that it returns a single, maximal sequence > mmanyP :: Parser a -> Parser [a] > mmanyP p = mmany1 <|> mmany0 > where mmany0 = return [] > mmany1 = do x <- p > xs <- mmanyP p > return (x:xs) Lets use the above to write a parser that will return an entire integer (not just a single digit.) ~~~~~{.haskell} oneInt :: Parser Integer oneInt = do xs <- mmanyP digitChar return $ ((read xs) :: Integer) ~~~~~ *Aside*, can you spot the pattern above? We took the parser `mmanyP digitChar` and simply converted its output using the `read` function. This is a recurring theme, and the type of what we did gives us a clue ~~~~~{.haskell} (a -> b) -> Parser a -> Parser b ~~~~~ Aha! a lot like `map`. Indeed, there is a generalized version of `map` that we have seen before (`lift1`) and we bottle up the pattern by declaring `Parser` to be an instance of the `Functor` typeclass > instance Functor Parser where > fmap f p = do x <- p > return (f x) after which we can rewrite > oneInt :: Parser Int > oneInt = read `fmap` mmanyP digitChar Lets take it for a spin ~~~~~{.haskell} ghci> doParse oneInt "123a" [(123, "a")] ~~~~~ Parsing Arithmetic Expressions ============================== Lets use the above to build a small calculator, that parses and evaluates arithmetic expressions. In essence, an expression is either binary operand applied to two sub-expressions or an integer. We can state this as > calc1 :: Parser Int > calc1 = binExp <|> oneInt > where binExp = do x <- calc1 > o <- intOp > y <- oneInt > return $ x `o` y > calc2 = oneInt >>= grab > where grab x = kg x <|> return x > kg x = do o <- intOp > y <- oneInt > grab $ x `o` y This works pretty well! ~~~~~{.haskell} ghci> doParse calc1 "1+2+33" [(36,"")] ghci> doParse calc1 "11+22-33" [(0,"")] ~~~~~ but things get a bit strange with minus ~~~~~{.haskell} ghci> doParse calc1 "11+22-33+45" [(-45,"")] ~~~~~ Huh? Well, if you look back at the code, you'll realize the above was parsed as ~~~~~{.haskell} 11 + ( 22 - (33 + 45)) ~~~~~ because in each `binExp` we require the left operand to be an integer. In other words, we are assuming that each operator is *right associative* hence the above result. Even worse, we have no precedence, and so ~~~~~{.haskell} ghci> doParse calc1 "10*2+100" [(1020,"")] ~~~~~ as the string is parsed as ~~~~~{.haskell} 10 * (2 + 100) ~~~~~ Precedence ---------- We can add both associativity and precedence in the usual way, by stratifying the parser into different levels. Here, lets split our operations into addition- and multiplication-precedence. > addOp = plus `chooseP` minus > where plus = char '+' >> return (+) > minus = char '-' >> return (-) > > mulOp = times `chooseP` divide > where times = char '*' >> return (*) > divide = char '/' >> return div Now, we can stratify our language into (mutually recursive) sub-languages, where each top-level expression is parsed as a *sum-of-products* > sumE = addE <|> prodE > where addE = do x <- prodE > o <- addOp > y <- sumE > return $ x `o` y > > prodE = mulE <|> factorE > where mulE = do x <- factorE > o <- mulOp > y <- prodE > return $ x `o` y > > factorE = parenE <|> oneInt > where parenE = do char '(' > n <- sumE > char ')' > return n We can run this ~~~~~{.haskell} ghci> doParse sumE "10*2+100" [(120,"")] ghci> doParse sumE "10*(2+100)" [(1020,"")] ~~~~~ Do you understand why the first parse returned `120` ? What would happen if we *swapped* the order of `prodE` and `sumE` in the body of `addE` (or `factorE` and `prodE` in the body of `prodE`) ? Why? Parsing Pattern: Chaining ------------------------- There is not much point gloating about combinators if we are going to write code like the above -- the bodies of `sumE` and `prodE` are almost identical! Lets take a closer look at them. In essence, an `sumE` is of the form ~~~~~{.haskell} prodE + < prodE + < prodE + ... < prodE >>> ~~~~~ that is, we keep chaining together `prodE` values and adding them for as long as we can. Similarly a `prodE` is of the form ~~~~~{.haskell} factorE * < factorE * < factorE * ... < factorE >>> ~~~~~ where we keep chaining `factorE` values and multiplying them for as long as we can. There is something unpleasant about the above: the addition operators are right-associative ~~~~~{.haskell} ghci> doParse sumE "10-1-1" [(10,"")] ~~~~~ Ugh! I hope you understand why: its because the above was parsed as `10 - (1 - 1)` (right associative) and not `(10 - 1) - 1` (left associative). You might be tempted to fix that simply by flipping the order of `prodE` and `sumE` ~~~~~{.haskell} sumE = addE <|> prodE where addE = do x <- sumE o <- addOp y <- prodE return $ x `o` y ~~~~~ but this would prove disastrous. Can you see why? The parser for `sumE` directly (recursively) calls itself *without consuming any input!* Thus, it goes off the deep end and never comes back. Instead, we want to make sure we keep consuming `prodE` values and adding them up (rather like fold) and so we could do > sumE1 = prodE1 >>= addE1 > where addE1 x = grab x <|> return x > grab x = do o <- addOp > y <- prodE1 > addE1 $ x `o` y > > prodE1 = factorE1 >>= mulE1 > where mulE1 x = grab x <|> return x > grab x = do o <- mulOp > y <- factorE1 > mulE1 $ x `o` y > > factorE1 = parenE <|> oneInt > where parenE = do char '(' > n <- sumE1 > char ')' > return n It is easy to check that the above is indeed left associative. ~~~~~{.haskell} ghci> doParse sumE1 "10-1-1" [(8,"")] ~~~~~ and it is also very easy to spot and bottle the chaining computation pattern: the only differences are the *base* parser (`prodE1` vs `factorE1`) and the binary operation (`addOp` vs `mulOp`). We simply make those parameters to our *chain-left* combinator > p `chainl` pop = p >>= rest > where rest x = grab x <|> return x > grab x = do o <- pop > y <- p > rest $ x `o` y Similarly, we often want to parse bracketed expressions, so we can write a combinator > parenP l p r = do char l > x <- p > char r > return x after which we can rewrite the grammar in three lines > sumE2 = prodE2 `chainl` addOp > prodE2 = factorE2 `chainl` mulOp > factorE2 = parenP '(' sumE2 ')' <|> oneInt ~~~~~{.haskell} ghci> doParse sumE2 "10-1-1" [(8,"")] ghci> doParse sumE2 "10*2+1" [(21,"")] ghci> doParse sumE2 "10+2*1" [(12,"")] ~~~~~ That concludes our (in-class) exploration of monadic parsing. This is merely the tip of the iceberg. Though parsing is a very old problem, and has been studied since the dawn of computing, we saw how monads bring a fresh perspective which have recently been transferred from Haskell to [many other languages][3]. There have been several exciting [recent][4] [papers][5] on the subject, that you can explore on your own. Finally, Haskell comes with several parser combinator libraries including [Parsec][3] which you will play around with in [HW3]($root/homeworks/hw3.html). [2]: http://homepages.inf.ed.ac.uk/wadler/papers/marktoberdorf/baastad.pdf [3]: http://www.haskell.org/haskellwiki/Parsec [4]: http://www.cse.chalmers.se/~nad/publications/danielsson-parser-combinators.html [5]: http://portal.acm.org/citation.cfm?doid=1706299.1706347