Lexer API

Project

The code MUST compile and work on gcc/g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16). It has to work on this version. This is to avoid conflict to where I need to submit the code.

You will be given a lexer that reads tokens from standard input. Your goal is to write, in C or C++, a program that reads all tokens from standard input by calling the lexer function getToken() and stores certain tokens in a linked list. After all tokens are read, your program will print out the content of the linked list in a specific order.

Lexer API

There are two functions that the lexer defines. These two functions compose the application programming interface (API) of our lexer. These functions are declared in lexer.h (and implemented in lexer.c). You will find the files attached to this project.

• getToken() reads the next token from standard input and returns its type as a token_type enum. If the token is of type ID, NUM, IF, WHILE, DO, THEN, or PRINT then the actual token value is stored in the global variable current_token as a null-terminated character array and the length of the string is stored in the global variable token_length. There are two special token_type values, END_OF_FILE, which is returned when the lexer encounters the end of standard input and ERROR, which is returned when the lexer encounters an unrecognized character in the input.
• ungetToken() causes the next call to getToken() to return the last token read by the previous call to getToken(). Note that this means the next call to getToken() will not read from standard input.

There are four global variables declared in lexer.h that are set when getToken() is called:

• t_type the token type is stored here (note that this will be the same value that was returned by getToken().
• current_token the token value is stored in the array current_token. If the token is of type ID, NUM, IF, WHILE, DO, THEN, or PRINT, then current_token contains the token string. For all other token types, current_token contains the empty string.
• token_length: the length of the string stored in current_token.
• line: the current line number of the input when the token was read.

Requirements

Your program should use the provided lexer and read all tokens from the input by repeatedly calling the getToken() function. Certain token strings and additional data should be stored in a linked list. Specifically, if either of the following conditions is true:

• The token is of type NUM
• The token is of type ID and the actual token is equal to one of the following values: "cse340", "programming", or "language"

Then the token string and other information needs to be stored in a node of a linked list. The information that needs to be stored about each of these tokens in the linked list is the following:

• Token type (from t_type)
• Token value (from current_token)
• Line number from the input where token was read (from line)

After reading all tokens from the input and storing information about tokens that match the criteria, your program should go over the linked list and print the information in reverse order from when that token was encountered. Each of the tokens in the linked list must be printed to standard output on a separate line with the following format:

< line> < token_type_string> < token value>

Note that < token_type_string> is the textual representation of the token type. In this case, the possible values are ID and NUM.

Evaluation Rules

You are required to store the information in a linked list that you write (either single or double linked list).The nodes in the linked list must be allocated on the heap (using malloc or other similar functions like calloc), and the allocated memory must be freed after printing the output. You are not allowed to use the STL linked list libraries.

Note that this means you are not allowed to use the C++ new operator to allocate memory (you must use the C memory allocation functions).

Example

Here is an example input with four lines:

cse340 < < + 123 *
456 programming
- cse 340 , LANGUAGE 100
. ; WHILE 200 IF

Here is your programs expected output:

4 NUM 200
3 NUM 100
3 NUM 340
2 ID programming
2 NUM 456
1 NUM 123
1 ID cse340

Testing

You can test the project using the test files.

$ ./a.out < some_input.txt > output.txt
$ diff –Bw some_input.txt.expected output.txt

If the program produces output according to the expected file, we should see no output from the diff command. Otherwise diff will show the difference between the output generated by the program and the expected output.