Java应用中表达式解析器(Java Cup/JFlex)生成器的介绍及示例
openkk
13年前
<p>在基于Java的软件系统的构建过程中，开发人员经常会遇到词法解析、语法解析等问题，比如：在报表系统的中一般需要支持单元格计算公式（类似于 Excel的公式），在某些系统的数据转换过程要实现自定义的转换规则脚本。面对这一类问题，我们最终需要的是一个针对问题域的词法及语法解析器，通常的实现方式不外乎有以下几种：</p> <h4>1. 自己实现全部过程</h4> <p>当所涉及语法非常简单（一般只有一两句原语）时，可以选择该方式。</p> <p><strong>优点：</strong>最直接，自然而然的选择；如果开发人员理论基础扎实的话不需要额外的学习负担。</p> <p><strong>缺点：</strong>要求有一定的理论基础；不利于扩充、或设计良好利于扩充但随着未来需求的改变及新语法的不断增加使扩充的开发成本不可控；测试成本增加；因未经过应用检验而存在风险；</p> <h4>2. 使用第三方的脚本引擎</h4> <p>当所涉及的语法及其复杂（需要支持比较复杂的脚本，甚至需要实现一种新的程序设计语言时，类似于Brio Query中提供的自定义脚本功能）时，可以选择该方式。目前，在互联网上有很多第三方脚本引擎（如：针对应用有：Netscape Script Engine、JavaRhino、BeanShell、IBM BSF、Tcl-Java等，针对Delphi应用有：Innerfuse Pascal Script 3，对微软体系有Microsoft Script Engine。<em>这些引擎的具体情况不在这里讨论</em>）可以选择，只需要少量的编程工作这些脚本引擎大多都可以内嵌到宿主应用程序中，为宿主应用程序提供脚本支持功能。</p> <p><strong>优点：</strong>开发工作量少；部分引擎已经得到广泛应用的验证；具有复杂的脚本支持能力，可以提供类似于程序设计语言的处理能力。</p> <p><strong>缺点：</strong>脚本的语法由使用的引擎决定，往往比较复杂，对于处理简单问题的场合显得过于累赘；系统中需要引入第三方的类库/控件；要求最终用户学习引擎所规定的脚本语法；某些引擎在访问应用系统的运行时对象时能力有限。</p> <h4>3. 使用语法解析器生成器</h4> <p>使用过Unix的同志一般会知道在Unix中有Yacc/Lex，C语言的编译器生成器。在Java中，也有相应的工具。此方法使用JFlex /Java_Cup生成语法解析源码。本人使用这种方法成功的实现了类似Excel那样的公式，支持多种运算及函数，可以维护上下文，可以在表达式中引用系统的对象体系，可以执行动作，并且可以任意扩展而不需要修改解析源码。</p> <p>关于JFlex与Java_Cup的文档你需要自己到下面的链接中仔细看看。其中关键的部分是需要定义你的语法及词法分析文件，类似于<strong>BNF表达式</strong>。</p> <p>主要步骤如下（为了方便，这里仅仅以四则运算为例，这个例子也是编译器构造工具的通用例子，满世界都采用它。）：</p> <p>当然首先你必须下在JFlex与Java_cup，并解压到特定目录，假定二者的目录分别是${JFlex_home}、${Java_Cup_home}。</p> <h5>3.1定义JFlex词法分析文件calc.flex</h5> <p>/* <br /> This example comes from a short article series in the Linux <br /> Gazette by Richard A. Sevenich and Christopher Lopes, titled <br /> "Compiler Construction Tools". The article series starts at</p> <p><a href="/misc/goto?guid=4959500123538430726">http://www.linuxgazette.com/issue39/sevenich.html</a></p> <p> Small changes and updates to newest JFlex+Cup versions <br /> by Gerwin Klein <br /> */</p> <p>/* <br /> Commented By: Christopher Lopes <br /> File Name: lcalc.flex <br /> To Create: > jflex lcalc.flex</p> <p> and then after the parser is created <br /> > javac Lexer.java <br /> */ <br /> /* ————————–Usercode Section———————— */ <br /> import java_cup.runtime.*; <br /> %% <br /> /* —————–Options and Declarations Section—————– */ <br /> /* <br /> The name of the class JFlex will create will be Lexer. <br /> Will write the code to the file Lexer.java. <br /> */ <br /> %class Lexer</p> <p>/* <br /> The current line number can be accessed with the variable yyline <br /> and the current column number with the variable yycolumn. <br /> */ <br /> %line <br /> %column <br /> /* <br /> Will switch to a CUP compatibility mode to interface with a CUP <br /> generated parser. <br /> */ <br /> %cup <br /> /* <br /> Declarations <br /> Code between %{ and %}, both of which must be at the beginning of a <br /> line, will be copied letter to letter into the lexer class source. <br /> Here you declare member variables and functions that are used inside <br /> scanner actions. <br /> */ <br /> %{ <br /> /* To create a new java_cup.runtime.Symbol with information about <br /> the current token, the token will have no value in this <br /> case. */ <br /> private Symbol symbol(int type) { <br /> return new Symbol(type, yyline, yycolumn); <br /> } <br /> /* Also creates a new java_cup.runtime.Symbol with information <br /> about the current token, but this object has a value. */ <br /> private Symbol symbol(int type, Object value) { <br /> return new Symbol(type, yyline, yycolumn, value); <br /> } <br /> %}</p> <p>/* <br /> Macro Declarations <br /> These declarations are regular expressions that will be used latter <br /> in the Lexical Rules Section. <br /> */ <br /> /* A line terminator is a r (carriage return), n (line feed), or <br /> rn. */ <br /> LineTerminator = r|n|rn <br /> /* White space is a line terminator, space, tab, or line feed. */ <br /> WhiteSpace = {LineTerminator} | [ tf] <br /> /* A literal integer is is a number beginning with a number between <br /> one and nine followed by zero or more numbers between zero and nine <br /> or just a zero. */ <br /> dec_int_lit = 0 | [1-9][0-9]* <br /> /* A identifier integer is a word beginning a letter between A and <br /> Z, a and z, or an underscore followed by zero or more letters <br /> between A and Z, a and z, zero and nine, or an underscore. */ <br /> dec_int_id = [A-Za-z_][A-Za-z_0-9]* <br /> %% <br /> /* ————————Lexical Rules Section———————- */ <br /> /* <br /> This section contains regular expressions and actions, i.e. Java <br /> code, that will be executed when the scanner matches the associated <br /> regular expression. */ <br /> /* YYINITIAL is the state at which the lexer begins scanning. So <br /> these regular expressions will only be matched if the scanner is in <br /> the start state YYINITIAL. */ <br /> { <br /> /* Return the token SEMI declared in the class sym that was found. */ <br /> ";" { return symbol(sym.SEMI); } <br /> /* Print the token found that was declared in the class sym and then <br /> return it. */ <br /> "+" { System.out.print(" + "); return symbol(sym.PLUS); } <br /> "-" { System.out.print(" – "); return symbol(sym.MINUS); } <br /> "*" { System.out.print(" * "); return symbol(sym.TIMES); } <br /> "/" { System.out.print(" / "); return symbol(sym.DIVIDE); } <br /> "(" { System.out.print(" ( "); return symbol(sym.LPAREN); } <br /> ")" { System.out.print(" ) "); return symbol(sym.RPAREN); } <br /> /* If an integer is found print it out, return the token NUMBER <br /> that represents an integer and the value of the integer that is <br /> held in the string yytext which will get turned into an integer <br /> before returning */ <br /> {dec_int_lit} { System.out.print(yytext()); <br /> return symbol(sym.NUMBER, new Integer(yytext())); } <br /> /* If an identifier is found print it out, return the token ID <br /> that represents an identifier and the default value one that is <br /> given to all identifiers. */ <br /> {dec_int_id} { System.out.print(yytext()); <br /> return symbol(sym.ID, new Integer(1));} <br /> /* Don’t do anything if whitespace is found */ <br /> {WhiteSpace} { /* just skip what was found, do nothing */ } <br /> }</p> <p>/* No token was found for the input so through an error. Print out an <br /> Illegal character message with the illegal character that was found. */ <br /> [^] { throw new Error("Illegal character <"+yytext()+">"); }</p> <p></p> <h5>3.2生成词法解析程序</h5> <p>执行${JFlex_home}binjflex.bat，并输入calc.flex，成功后输出lex.java文件即四则运算的词法解析文件。职责主要是从本例中的四则运算字符串输入中进行词法分析，输出每一个Token，比如数字、运算符等。</p> <h5>3.3定义Java_cup语法分析文件</h5> <p>/* <br /> This example comes from a short article series in the Linux <br /> Gazette by Richard A. Sevenich and Christopher Lopes, titled <br /> "Compiler Construction Tools". The article series starts at</p> <p><a href="/misc/goto?guid=4959500123538430726">http://www.linuxgazette.com/issue39/sevenich.html</a></p> <p> Small changes and updates to newest JFlex+Cup versions <br /> by Gerwin Klein <br /> */</p> <p>/* <br /> Commented By: Christopher Lopes <br /> File Name: ycalc.cup <br /> To Create: > java java_cup.Main < ycalc.cup <br /> */ <br /> /* ———————-Preliminary Declarations Section——————–*/ <br /> /* Import the class java_cup.runtime.* */ <br /> import java_cup.runtime.*; <br /> /* Parser code to change the way the parser reports errors (include <br /> line and column number of the error). */ <br /> parser code {: <br /> /* Change the method report_error so it will display the line and <br /> column of where the error occurred in the input as well as the <br /> reason for the error which is passed into the method in the <br /> String ‘message’. */ <br /> public void report_error(String message, Object info) { <br /> /* Create a StringBuffer called ‘m’ with the string ‘Error’ in it. */ <br /> StringBuffer m = new StringBuffer("Error"); <br /> /* Check if the information passed to the method is the same <br /> type as the type java_cup.runtime.Symbol. */ <br /> if (info instanceof java_cup.runtime.Symbol) { <br /> /* Declare a java_cup.runtime.Symbol object ‘s’ with the <br /> information in the object info that is being typecasted <br /> as a java_cup.runtime.Symbol object. */ <br /> java_cup.runtime.Symbol s = ((java_cup.runtime.Symbol) info); <br /> /* Check if the line number in the input is greater or <br /> equal to zero. */ <br /> if (s.left >= 0) { <br /> /* Add to the end of the StringBuffer error message <br /> the line number of the error in the input. */ <br /> m.append(" in line "+(s.left+1)); <br /> /* Check if the column number in the input is greater <br /> or equal to zero. */ <br /> if (s.right >= 0) <br /> /* Add to the end of the StringBuffer error message <br /> the column number of the error in the input. */ <br /> m.append(", column "+(s.right+1)); <br /> } <br /> } <br /> /* Add to the end of the StringBuffer error message created in <br /> this method the message that was passed into this method. */ <br /> m.append(" : "+message); <br /> /* Print the contents of the StringBuffer ‘m’, which contains <br /> an error message, out on a line. */ <br /> System.err.println(m); <br /> } <br /> /* Change the method report_fatal_error so when it reports a fatal <br /> error it will display the line and column number of where the <br /> fatal error occurred in the input as well as the reason for the <br /> fatal error which is passed into the method in the object <br /> ‘message’ and then exit.*/ <br /> public void report_fatal_error(String message, Object info) { <br /> report_error(message, info); <br /> System.exit(1); <br /> } <br /> :};</p> <p>/* ————Declaration of Terminals and Non Terminals Section———– */ <br /> /* Terminals (tokens returned by the scanner). </p> <p> Terminals that have no value are listed first and then terminals <br /> that do have an value, in this case an integer value, are listed on <br /> the next line down. */ <br /> terminal SEMI, PLUS, MINUS, TIMES, DIVIDE, LPAREN, RPAREN; <br /> terminal Integer NUMBER, ID; <br /> /* Non terminals used in the grammar section. </p> <p> Non terminals that have an object value are listed first and then <br /> non terminals that have an integer value are listed. An object <br /> value means that it can be any type, it isn’t set to a specific <br /> type. So it could be an Integer or a String or whatever. */ <br /> non terminal Object expr_list, expr_part; <br /> non terminal Integer expr, factor, term;</p> <p>/* ————-Precedence and Associatively of Terminals Section———– */ <br /> /* <br /> Precedence of non terminals could be defined here. If you do define <br /> precedence here you won’t need to worry about precedence in the <br /> Grammar Section, i.e. that TIMES should have a higher precedence <br /> than PLUS. <br /> The precedence defined here would look something like this where the <br /> lower line always will have higher precedence than the line before it. <br /> precedence left PLUS, MINUS; <br /> precedence left TIMES, DIVIDE; <br /> */</p> <p>/* —————————-Grammar Section——————– */ <br /> /* The grammar for our parser. <br /> expr_list ::= expr_list expr_part <br /> | expr_part <br /> expr_part ::= expr SEMI <br /> expr ::= expr PLUS factor <br /> | expr MINUS factor <br /> | factor <br /> factor ::= factor TIMES term <br /> | factor DIVIDE term <br /> | term <br /> term ::= LPAREN expr RPAREN <br /> | NUMBER <br /> | ID <br /> */ <br /> /* ‘expr_list’ is the start of our grammar. It can lead to another <br /> ‘expr_list’ followed by an ‘expr_part’ or it can just lead to an <br /> ‘expr_part’. The lhs of the non terminals ‘expr_list’ and <br /> ‘expr_part’ that are in the rhs side of the production below need <br /> to be found. Then the rhs sides of those non terminals need to be <br /> followed in a similar manner, i.e. if there are any non terminals <br /> in the rhs of those productions then the productions with those non <br /> terminals need to be found and those rhs’s followed. This process <br /> keeps continuing until only terminals are found in the rhs of a <br /> production. Then we can work our way back up the grammar bringing <br /> any values that might have been assigned from a terminal. */ <br /> expr_list ::= expr_list expr_part <br /> | <br /> expr_part; <br /> /* ‘expr_part’ is an ‘expr’ followed by the terminal ‘SEMI’. The ‘:e’ <br /> after the non terminal ‘expr’ is a label an is used to access the <br /> value of ‘expr’ which will be an integer. The action for the <br /> production lies between {: and :}. This action will print out the <br /> line " = + e" where e is the value of ‘expr’. Before the action <br /> takes places we need to go deeper into the grammar since ‘expr’ is <br /> a non terminal. Whenever a non terminal is encountered on the rhs <br /> of a production we need to find the rhs of that non terminal until <br /> there are no more non terminals in the rhs. So when we finish <br /> going through the grammar and don’t encounter any more non <br /> terminals in the rhs productions will return until we get back to <br /> ‘expr’ and at that point ‘expr’ will contain an integer value. */ <br /> expr_part ::= expr:e <br /> {: System.out.println(" = " + e); :} <br /> SEMI <br /> ; <br /> /* ‘expr’ can lead to ‘expr PLUS factor’, ‘expr MINUS factor’, or <br /> ‘factor’. The ‘TIMES’ and ‘DIVIDE’ productions are not at this <br /> level. They are at a lower level in the grammar which in affect <br /> makes them have higher precedence. Actions for the rhs of the non <br /> terminal ‘expr’ return a value to ‘expr’. This value that is <br /> created is an integer and gets stored in ‘RESULT’ in the action. <br /> RESULT is the label that is assigned automatically to the rhs, in <br /> this case ‘expr’. If the rhs is just ‘factor’ then ‘f’ refers to <br /> the non terminal ‘factor’. The value of ‘f’ is retrieved with the <br /> function ‘intValue()’ and will be stored in ‘RESULT’. In the other <br /> two cases ‘f’ and ‘e’ refers to the non terminals ‘factor’ and <br /> ‘expr’ respectively with a terminal between them, either ‘PLUS’ or <br /> ‘MINUS’. The value of each is retrieved with the same function <br /> ‘intValue’. The values will be added or subtracted and then the <br /> new integer will be stored in ‘RESULT’.*/ <br /> expr ::= expr:e PLUS factor:f <br /> {: RESULT = new Integer(e.intValue() + f.intValue()); :} <br /> | <br /> expr:e MINUS factor:f <br /> {: RESULT = new Integer(e.intValue() – f.intValue()); :} <br /> | <br /> factor:f <br /> {: RESULT = new Integer(f.intValue()); :} <br /> ; <br /> /* ‘factor’ can lead to ‘factor TIMES term’, ‘factor DIVIDE term’, or <br /> ‘term’. Since the productions for TIMES and DIVIDE are lower in <br /> the grammar than ‘PLUS’ and ‘MINUS’ they will have higher <br /> precedence. The same sort of actions take place in the rhs of <br /> ‘factor’ as in ‘expr’. The only difference is the operations that <br /> takes place on the values retrieved with ‘intValue()’, ‘TIMES’ and <br /> ‘DIVIDE’ here instead of ‘PLUS’ and ‘MINUS’. */ <br /> factor ::= factor:f TIMES term:t <br /> {: RESULT = new Integer(f.intValue() * t.intValue()); :} <br /> | <br /> factor:f DIVIDE term:t <br /> {: RESULT = new Integer(f.intValue() / t.intValue()); :} <br /> | <br /> term:t <br /> {: RESULT = new Integer(t.intValue()); :} <br /> ; <br /> /* ‘term’ can lead to ‘LPAREN expr RPAREN’, ‘NUMBER’, or ‘ID’. The <br /> first production has the non terminal ‘expr’ in it so the <br /> production with its lhs side needs to be found and followed. The <br /> next rhs has no non terminals. So the grammar ends here and can go <br /> back up. When it goes back up it will bring the value that was <br /> retrieved when the scanner encounter the token ‘NUMBER’. ‘RESULT’ <br /> is assigned ‘n’, which refers to ‘NUMBER’, as the action for this <br /> production. The same action occurs for ‘ID’, except the ‘i’ is <br /> used to refer to ‘ID’. ‘ID’ is also the only thing on the rhs of <br /> the production. And since ‘ID’ is a terminal the grammar will end <br /> here and go back up. */ <br /> term ::= LPAREN expr:e RPAREN <br /> {: RESULT = e; :} <br /> | <br /> NUMBER:n <br /> {: RESULT = n; :} <br /> | <br /> ID:i <br /> {: RESULT = i; :} <br /> ;</p> <h5>3.4 生成语法解析器calc.cup</h5> <p>当前目录切换到${java_cup_home}下，执行</p> <p>java java_cup.Main options < calc.cup</p> <p>其中，Options可以自己读文档，视情况而设。</p> <p>执行的结果是生成了语法解析文件parser.java，它负责从lex.java中接受token流，构造语法规则，比如生成语法解析树。</p> <p>至此，你已经拥有了一个解析器，只要在你的代码中引入Parser.java，调用其相应parse()方法即可完成表达式解析过程。测试调用过程如下：</p> <p>/* <br /> Commented By: Christopher Lopes <br /> File Name: Main.java <br /> To Create: <br /> After the scanner, lcalc.flex, and the parser, ycalc.cup, have been created. <br /> > javac Main.java <br /> To Run: <br /> > java Main test.txt <br /> where test.txt is an test input file for the calculator. <br /> */ <br /> import java.io.*; <br /> public class Main { <br /> static public void main(String argv[]) { <br /> /* Start the parser */ <br /> try { <br /> parser p = new parser(new Lexer(new FileReader(argv[0]))); <br /> Object result = p.parse().value; <br /> } catch (Exception e) { <br /> /* do cleanup here — possibly rethrow e */ <br /> e.printStackTrace(); <br /> } <br /> } <br /> }</p> <p>不过在处理复杂表达式（涉及到上下文、系统其他对象引用等）时，你需要仔细定义语法规则文件中的action。<br /> </p> <p>在基于Java的软件系统的构建过程中，开发人员经常会遇到词法解析、语法解析等问题，比如：在报表系统的中一般需要支持单元格计算公式（类似于 Excel的公式），在某些系统的数据转换过程要实现自定义的转换规则脚本。面对这一类问题，我们最终需要的是一个针对问题域的词法及语法解析器，通常的实现方式不外乎有以下几种：</p> <h4>1. 自己实现全部过程</h4> <p>当所涉及语法非常简单（一般只有一两句原语）时，可以选择该方式。</p> <p><strong>优点：</strong>最直接，自然而然的选择；如果开发人员理论基础扎实的话不需要额外的学习负担。</p> <p><strong>缺点：</strong>要求有一定的理论基础；不利于扩充、或设计良好利于扩充但随着未来需求的改变及新语法的不断增加使扩充的开发成本不可控；测试成本增加；因未经过应用检验而存在风险；</p> <h4>2. 使用第三方的脚本引擎</h4> <p>当所涉及的语法及其复杂（需要支持比较复杂的脚本，甚至需要实现一种新的程序设计语言时，类似于Brio Query中提供的自定义脚本功能）时，可以选择该方式。目前，在互联网上有很多第三方脚本引擎（如：针对应用有：Netscape Script Engine、JavaRhino、BeanShell、IBM BSF、Tcl-Java等，针对Delphi应用有：Innerfuse Pascal Script 3，对微软体系有Microsoft Script Engine。<em>这些引擎的具体情况不在这里讨论</em>）可以选择，只需要少量的编程工作这些脚本引擎大多都可以内嵌到宿主应用程序中，为宿主应用程序提供脚本支持功能。</p> <p><strong>优点：</strong>开发工作量少；部分引擎已经得到广泛应用的验证；具有复杂的脚本支持能力，可以提供类似于程序设计语言的处理能力。</p> <p><strong>缺点：</strong>脚本的语法由使用的引擎决定，往往比较复杂，对于处理简单问题的场合显得过于累赘；系统中需要引入第三方的类库/控件；要求最终用户学习引擎所规定的脚本语法；某些引擎在访问应用系统的运行时对象时能力有限。</p> <h4>3. 使用语法解析器生成器</h4> <p>使用过Unix的同志一般会知道在Unix中有Yacc/Lex，C语言的编译器生成器。在Java中，也有相应的工具。此方法使用JFlex /Java_Cup生成语法解析源码。本人使用这种方法成功的实现了类似Excel那样的公式，支持多种运算及函数，可以维护上下文，可以在表达式中引用系统的对象体系，可以执行动作，并且可以任意扩展而不需要修改解析源码。</p> <p>关于JFlex与Java_Cup的文档你需要自己到下面的链接中仔细看看。其中关键的部分是需要定义你的语法及词法分析文件，类似于<strong>BNF表达式</strong>。</p> <p>主要步骤如下（为了方便，这里仅仅以四则运算为例，这个例子也是编译器构造工具的通用例子，满世界都采用它。）：</p> <p>当然首先你必须下在JFlex与Java_cup，并解压到特定目录，假定二者的目录分别是${JFlex_home}、${Java_Cup_home}。</p> <h5>3.1定义JFlex词法分析文件calc.flex</h5> <p>/* <br /> This example comes from a short article series in the Linux <br /> Gazette by Richard A. Sevenich and Christopher Lopes, titled <br /> "Compiler Construction Tools". The article series starts at</p> <p><a href="/misc/goto?guid=4959500123538430726">http://www.linuxgazette.com/issue39/sevenich.html</a></p> <p> Small changes and updates to newest JFlex+Cup versions <br /> by Gerwin Klein <br /> */</p> <p>/* <br /> Commented By: Christopher Lopes <br /> File Name: lcalc.flex <br /> To Create: > jflex lcalc.flex</p> <p> and then after the parser is created <br /> > javac Lexer.java <br /> */ <br /> /* ————————–Usercode Section———————— */ <br /> import java_cup.runtime.*; <br /> %% <br /> /* —————–Options and Declarations Section—————– */ <br /> /* <br /> The name of the class JFlex will create will be Lexer. <br /> Will write the code to the file Lexer.java. <br /> */ <br /> %class Lexer</p> <p>/* <br /> The current line number can be accessed with the variable yyline <br /> and the current column number with the variable yycolumn. <br /> */ <br /> %line <br /> %column <br /> /* <br /> Will switch to a CUP compatibility mode to interface with a CUP <br /> generated parser. <br /> */ <br /> %cup <br /> /* <br /> Declarations <br /> Code between %{ and %}, both of which must be at the beginning of a <br /> line, will be copied letter to letter into the lexer class source. <br /> Here you declare member variables and functions that are used inside <br /> scanner actions. <br /> */ <br /> %{ <br /> /* To create a new java_cup.runtime.Symbol with information about <br /> the current token, the token will have no value in this <br /> case. */ <br /> private Symbol symbol(int type) { <br /> return new Symbol(type, yyline, yycolumn); <br /> } <br /> /* Also creates a new java_cup.runtime.Symbol with information <br /> about the current token, but this object has a value. */ <br /> private Symbol symbol(int type, Object value) { <br /> return new Symbol(type, yyline, yycolumn, value); <br /> } <br /> %}</p> <p>/* <br /> Macro Declarations <br /> These declarations are regular expressions that will be used latter <br /> in the Lexical Rules Section. <br /> */ <br /> /* A line terminator is a r (carriage return), n (line feed), or <br /> rn. */ <br /> LineTerminator = r|n|rn <br /> /* White space is a line terminator, space, tab, or line feed. */ <br /> WhiteSpace = {LineTerminator} | [ tf] <br /> /* A literal integer is is a number beginning with a number between <br /> one and nine followed by zero or more numbers between zero and nine <br /> or just a zero. */ <br /> dec_int_lit = 0 | [1-9][0-9]* <br /> /* A identifier integer is a word beginning a letter between A and <br /> Z, a and z, or an underscore followed by zero or more letters <br /> between A and Z, a and z, zero and nine, or an underscore. */ <br /> dec_int_id = [A-Za-z_][A-Za-z_0-9]* <br /> %% <br /> /* ————————Lexical Rules Section———————- */ <br /> /* <br /> This section contains regular expressions and actions, i.e. Java <br /> code, that will be executed when the scanner matches the associated <br /> regular expression. */ <br /> /* YYINITIAL is the state at which the lexer begins scanning. So <br /> these regular expressions will only be matched if the scanner is in <br /> the start state YYINITIAL. */ <br /> { <br /> /* Return the token SEMI declared in the class sym that was found. */ <br /> ";" { return symbol(sym.SEMI); } <br /> /* Print the token found that was declared in the class sym and then <br /> return it. */ <br /> "+" { System.out.print(" + "); return symbol(sym.PLUS); } <br /> "-" { System.out.print(" – "); return symbol(sym.MINUS); } <br /> "*" { System.out.print(" * "); return symbol(sym.TIMES); } <br /> "/" { System.out.print(" / "); return symbol(sym.DIVIDE); } <br /> "(" { System.out.print(" ( "); return symbol(sym.LPAREN); } <br /> ")" { System.out.print(" ) "); return symbol(sym.RPAREN); } <br /> /* If an integer is found print it out, return the token NUMBER <br /> that represents an integer and the value of the integer that is <br /> held in the string yytext which will get turned into an integer <br /> before returning */ <br /> {dec_int_lit} { System.out.print(yytext()); <br /> return symbol(sym.NUMBER, new Integer(yytext())); } <br /> /* If an identifier is found print it out, return the token ID <br /> that represents an identifier and the default value one that is <br /> given to all identifiers. */ <br /> {dec_int_id} { System.out.print(yytext()); <br /> return symbol(sym.ID, new Integer(1));} <br /> /* Don’t do anything if whitespace is found */ <br /> {WhiteSpace} { /* just skip what was found, do nothing */ } <br /> }</p> <p>/* No token was found for the input so through an error. Print out an <br /> Illegal character message with the illegal character that was found. */ <br /> [^] { throw new Error("Illegal character <"+yytext()+">"); }</p> <p></p> <h5>3.2生成词法解析程序</h5> <p>执行${JFlex_home}binjflex.bat，并输入calc.flex，成功后输出lex.java文件即四则运算的词法解析文件。职责主要是从本例中的四则运算字符串输入中进行词法分析，输出每一个Token，比如数字、运算符等。</p> <h5>3.3定义Java_cup语法分析文件</h5> <p>/* <br /> This example comes from a short article series in the Linux <br /> Gazette by Richard A. Sevenich and Christopher Lopes, titled <br /> "Compiler Construction Tools". The article series starts at</p> <p><a href="/misc/goto?guid=4959500123538430726">http://www.linuxgazette.com/issue39/sevenich.html</a></p> <p> Small changes and updates to newest JFlex+Cup versions <br /> by Gerwin Klein <br /> */</p> <p>/* <br /> Commented By: Christopher Lopes <br /> File Name: ycalc.cup <br /> To Create: > java java_cup.Main < ycalc.cup <br /> */ <br /> /* ———————-Preliminary Declarations Section——————–*/ <br /> /* Import the class java_cup.runtime.* */ <br /> import java_cup.runtime.*; <br /> /* Parser code to change the way the parser reports errors (include <br /> line and column number of the error). */ <br /> parser code {: <br /> /* Change the method report_error so it will display the line and <br /> column of where the error occurred in the input as well as the <br /> reason for the error which is passed into the method in the <br /> String ‘message’. */ <br /> public void report_error(String message, Object info) { <br /> /* Create a StringBuffer called ‘m’ with the string ‘Error’ in it. */ <br /> StringBuffer m = new StringBuffer("Error"); <br /> /* Check if the information passed to the method is the same <br /> type as the type java_cup.runtime.Symbol. */ <br /> if (info instanceof java_cup.runtime.Symbol) { <br /> /* Declare a java_cup.runtime.Symbol object ‘s’ with the <br /> information in the object info that is being typecasted <br /> as a java_cup.runtime.Symbol object. */ <br /> java_cup.runtime.Symbol s = ((java_cup.runtime.Symbol) info); <br /> /* Check if the line number in the input is greater or <br /> equal to zero. */ <br /> if (s.left >= 0) { <br /> /* Add to the end of the StringBuffer error message <br /> the line number of the error in the input. */ <br /> m.append(" in line "+(s.left+1)); <br /> /* Check if the column number in the input is greater <br /> or equal to zero. */ <br /> if (s.right >= 0) <br /> /* Add to the end of the StringBuffer error message <br /> the column number of the error in the input. */ <br /> m.append(", column "+(s.right+1)); <br /> } <br /> } <br /> /* Add to the end of the StringBuffer error message created in <br /> this method the message that was passed into this method. */ <br /> m.append(" : "+message); <br /> /* Print the contents of the StringBuffer ‘m’, which contains <br /> an error message, out on a line. */ <br /> System.err.println(m); <br /> } <br /> /* Change the method report_fatal_error so when it reports a fatal <br /> error it will display the line and column number of where the <br /> fatal error occurred in the input as well as the reason for the <br /> fatal error which is passed into the method in the object <br /> ‘message’ and then exit.*/ <br /> public void report_fatal_error(String message, Object info) { <br /> report_error(message, info); <br /> System.exit(1); <br /> } <br /> :};</p> <p>/* ————Declaration of Terminals and Non Terminals Section———– */ <br /> /* Terminals (tokens returned by the scanner). </p> <p> Terminals that have no value are listed first and then terminals <br /> that do have an value, in this case an integer value, are listed on <br /> the next line down. */ <br /> terminal SEMI, PLUS, MINUS, TIMES, DIVIDE, LPAREN, RPAREN; <br /> terminal Integer NUMBER, ID; <br /> /* Non terminals used in the grammar section. </p> <p> Non terminals that have an object value are listed first and then <br /> non terminals that have an integer value are listed. An object <br /> value means that it can be any type, it isn’t set to a specific <br /> type. So it could be an Integer or a String or whatever. */ <br /> non terminal Object expr_list, expr_part; <br /> non terminal Integer expr, factor, term;</p> <p>/* ————-Precedence and Associatively of Terminals Section———– */ <br /> /* <br /> Precedence of non terminals could be defined here. If you do define <br /> precedence here you won’t need to worry about precedence in the <br /> Grammar Section, i.e. that TIMES should have a higher precedence <br /> than PLUS. <br /> The precedence defined here would look something like this where the <br /> lower line always will have higher precedence than the line before it. <br /> precedence left PLUS, MINUS; <br /> precedence left TIMES, DIVIDE; <br /> */</p> <p>/* —————————-Grammar Section——————– */ <br /> /* The grammar for our parser. <br /> expr_list ::= expr_list expr_part <br /> | expr_part <br /> expr_part ::= expr SEMI <br /> expr ::= expr PLUS factor <br /> | expr MINUS factor <br /> | factor <br /> factor ::= factor TIMES term <br /> | factor DIVIDE term <br /> | term <br /> term ::= LPAREN expr RPAREN <br /> | NUMBER <br /> | ID <br /> */ <br /> /* ‘expr_list’ is the start of our grammar. It can lead to another <br /> ‘expr_list’ followed by an ‘expr_part’ or it can just lead to an <br /> ‘expr_part’. The lhs of the non terminals ‘expr_list’ and <br /> ‘expr_part’ that are in the rhs side of the production below need <br /> to be found. Then the rhs sides of those non terminals need to be <br /> followed in a similar manner, i.e. if there are any non terminals <br /> in the rhs of those productions then the productions with those non <br /> terminals need to be found and those rhs’s followed. This process <br /> keeps continuing until only terminals are found in the rhs of a <br /> production. Then we can work our way back up the grammar bringing <br /> any values that might have been assigned from a terminal. */ <br /> expr_list ::= expr_list expr_part <br /> | <br /> expr_part; <br /> /* ‘expr_part’ is an ‘expr’ followed by the terminal ‘SEMI’. The ‘:e’ <br /> after the non terminal ‘expr’ is a label an is used to access the <br /> value of ‘expr’ which will be an integer. The action for the <br /> production lies between {: and :}. This action will print out the <br /> line " = + e" where e is the value of ‘expr’. Before the action <br /> takes places we need to go deeper into the grammar since ‘expr’ is <br /> a non terminal. Whenever a non terminal is encountered on the rhs <br /> of a production we need to find the rhs of that non terminal until <br /> there are no more non terminals in the rhs. So when we finish <br /> going through the grammar and don’t encounter any more non <br /> terminals in the rhs productions will return until we get back to <br /> ‘expr’ and at that point ‘expr’ will contain an integer value. */ <br /> expr_part ::= expr:e <br /> {: System.out.println(" = " + e); :} <br /> SEMI <br /> ; <br /> /* ‘expr’ can lead to ‘expr PLUS factor’, ‘expr MINUS factor’, or <br /> ‘factor’. The ‘TIMES’ and ‘DIVIDE’ productions are not at this <br /> level. They are at a lower level in the grammar which in affect <br /> makes them have higher precedence. Actions for the rhs of the non <br /> terminal ‘expr’ return a value to ‘expr’. This value that is <br /> created is an integer and gets stored in ‘RESULT’ in the action. <br /> RESULT is the label that is assigned automatically to the rhs, in <br /> this case ‘expr’. If the rhs is just ‘factor’ then ‘f’ refers to <br /> the non terminal ‘factor’. The value of ‘f’ is retrieved with the <br /> function ‘intValue()’ and will be stored in ‘RESULT’. In the other <br /> two cases ‘f’ and ‘e’ refers to the non terminals ‘factor’ and <br /> ‘expr’ respectively with a terminal between them, either ‘PLUS’ or <br /> ‘MINUS’. The value of each is retrieved with the same function <br /> ‘intValue’. The values will be added or subtracted and then the <br /> new integer will be stored in ‘RESULT’.*/ <br /> expr ::= expr:e PLUS factor:f <br /> {: RESULT = new Integer(e.intValue() + f.intValue()); :} <br /> | <br /> expr:e MINUS factor:f <br /> {: RESULT = new Integer(e.intValue() – f.intValue()); :} <br /> | <br /> factor:f <br /> {: RESULT = new Integer(f.intValue()); :} <br /> ; <br /> /* ‘factor’ can lead to ‘factor TIMES term’, ‘factor DIVIDE term’, or <br /> ‘term’. Since the productions for TIMES and DIVIDE are lower in <br /> the grammar than ‘PLUS’ and ‘MINUS’ they will have higher <br /> precedence. The same sort of actions take place in the rhs of <br /> ‘factor’ as in ‘expr’. The only difference is the operations that <br /> takes place on the values retrieved with ‘intValue()’, ‘TIMES’ and <br /> ‘DIVIDE’ here instead of ‘PLUS’ and ‘MINUS’. */ <br /> factor ::= factor:f TIMES term:t <br /> {: RESULT = new Integer(f.intValue() * t.intValue()); :} <br /> | <br /> factor:f DIVIDE term:t <br /> {: RESULT = new Integer(f.intValue() / t.intValue()); :} <br /> | <br /> term:t <br /> {: RESULT = new Integer(t.intValue()); :} <br /> ; <br /> /* ‘term’ can lead to ‘LPAREN expr RPAREN’, ‘NUMBER’, or ‘ID’. The <br /> first production has the non terminal ‘expr’ in it so the <br /> production with its lhs side needs to be found and followed. The <br /> next rhs has no non terminals. So the grammar ends here and can go <br /> back up. When it goes back up it will bring the value that was <br /> retrieved when the scanner encounter the token ‘NUMBER’. ‘RESULT’ <br /> is assigned ‘n’, which refers to ‘NUMBER’, as the action for this <br /> production. The same action occurs for ‘ID’, except the ‘i’ is <br /> used to refer to ‘ID’. ‘ID’ is also the only thing on the rhs of <br /> the production. And since ‘ID’ is a terminal the grammar will end <br /> here and go back up. */ <br /> term ::= LPAREN expr:e RPAREN <br /> {: RESULT = e; :} <br /> | <br /> NUMBER:n <br /> {: RESULT = n; :} <br /> | <br /> ID:i <br /> {: RESULT = i; :} <br /> ;</p> <h5>3.4 生成语法解析器calc.cup</h5> <p>当前目录切换到${java_cup_home}下，执行</p> <p>java java_cup.Main options < calc.cup</p> <p>其中，Options可以自己读文档，视情况而设。</p> <p>执行的结果是生成了语法解析文件parser.java，它负责从lex.java中接受token流，构造语法规则，比如生成语法解析树。</p> <p>至此，你已经拥有了一个解析器，只要在你的代码中引入Parser.java，调用其相应parse()方法即可完成表达式解析过程。测试调用过程如下：</p> <p>/* <br /> Commented By: Christopher Lopes <br /> File Name: Main.java <br /> To Create: <br /> After the scanner, lcalc.flex, and the parser, ycalc.cup, have been created. <br /> > javac Main.java <br /> To Run: <br /> > java Main test.txt <br /> where test.txt is an test input file for the calculator. <br /> */ <br /> import java.io.*; <br /> public class Main { <br /> static public void main(String argv[]) { <br /> /* Start the parser */ <br /> try { <br /> parser p = new parser(new Lexer(new FileReader(argv[0]))); <br /> Object result = p.parse().value; <br /> } catch (Exception e) { <br /> /* do cleanup here — possibly rethrow e */ <br /> e.printStackTrace(); <br /> } <br /> } <br /> }</p> <p>不过在处理复杂表达式（涉及到上下文、系统其他对象引用等）时，你需要仔细定义语法规则文件中的action。</p> <h5> <hr /> </h5> <h5>相关资源： </h5> <p>1、JFlex <a href="/misc/goto?guid=4959500123760602762">http://jflex.de/</a></p> <p>2、Java CUP <a href="/misc/goto?guid=4959500123842341720">http://www.cs.princeton.edu/~appel/modern/java/CUP/</a></p> <p>3、BeanShell <a href="/misc/goto?guid=4959500123940162744">http://www.beanshell.org</a></p> <p>4、Rhino <a href="/misc/goto?guid=4958546102843824013">http://www.mozilla.org/rhino</a></p> <p>5、Innerfuse Pascal Script <a href="/misc/goto?guid=4959500124071978908">http://www.carlo-kok.com/ifps3.php</a></p>