Writing a lisp in Golang
2. Stream
2.1. 0x22DB
- I'm quite enjoying the exploratory nature of what I'm doing in here
- was thinking of going for a definite spec but why not just build flexibly so that I can add or retract freely
- in the language design stage now that I have a basic parser going for
2.2. 0x22DB
- the next step in augmenting the lexer is testing for literals
- now the string approach might not work and I might have to pull out regexps if I want to be tidy
- so also need to start building a spec that can handle what I'm trying to do here
- the next feature I want to add is to merge tabs and spaces into one whitespace token so there's multiple stages to my lexer now
- In the first stage, I've simply read in the strings and made them accessible
- In the next stage, I can start differentiating between symbols and literals
- a literal is just that value in itself : could be a string, a number, or a boolean
- as of now my symbols are representing all that comes in.
- need to start associating a value with them.
- so the lexer handles the values of literals and the value of the variables is resolved during evaluation
- maybe I should just use special symbols to represent literals
- could have a surrounding "|" for raw strings
- think that is a fine idea
- need a string mode now.
- If I encounter a "|", anything until the next "|" get's acc'ed
- check code till this commit : https://github.com/rajp152k/lisp-in-go/commit/5539a0bb5b36b0e3886a88c86fd65b0dc06fe56b
2.3. 0x22DB
- alright, time to move from red to green
- to read a string of characters piece by piece, will just read string piece by piece
- I don't to worry about if the syntax is right in this stage (unbalanced parens for instance).
- so just read a string, store intermediates, ignore whitespace (not handling comments yet so should be fine)
- but, maybe newlines and whitespace(tabs and spaces) are important after all
- still going to make note of them in the stream and ignore them when needed
- have a recursive solution but also have an iterative one in mind so given I'm using go, will proceed with iterations.
- so I'm in the green right now with a minor feature additions (I know TDD insn't supposed to be like this, but yeah…)
- if you're following along, check code until this commit : https://github.com/rajp152k/lisp-in-go/commit/44943157128773a8c37cd322074d9232bd1f1394
- now that I've got a basic lexer, I can start thinking about how I'll represent functions and variables
- but before that, representing a grammar is going to be somewhat tricky without any external tooling
- I would like to build something fairly minimal myself rather than use a library that is bigger than the project oriented codebase.
- I'm going to give it a try: the parser simply takes in the token stream and generates an abstract synax tree based on a few rules that define a program.
- I could build a data structure for my own parse tree : just needs to be a tree
- which is really just a list
- also need to understand how to link an environment to a node
- all I need to do is an apply and an eval
- that's calling a function on a bunch of evaluated arguments
- now for a function call, the root being the function symbol and the rest being argument forms should do
- an environment is just going to be a map from symbols (string repr'd) to values or other functions
- using the same namespace for variables and functions should be fine for now : i.e. I'm going the scheme way
- stacking environments together by having a list of these maps to look for seems to be a good way to get started
- I also need to define what keywords I plan on having.
2.4. 0x22DB
- going with McCarthy's original Lisp
- there are some built in functionalities and keywords
- then there's defined functions
- one has to be able to analyse variables as well
- there are special symbols include only the "()',`"
- LPAREN
- RPAREN
- QUOTE
- UNQUOTE
- QUASIQUOTE
- other than these, everything can be dereferenced as a symbol and looked up in the environment.
- so for now, going for these special tokens + symbols sound like a good way to go about it.
- a token than is an overarching type that can either be a special token or a symbol with a repr value that will be stored as a string
- so something like (+ 1 2) or (add 1 2) should be lexically analysed as [LPAREN, SYMBOL, SYMBOL, SYMBOL, RPAREN]
- macros is something that I'll deal with later
- whether a symbol is a keyword, a variable, a function (also have to choose if i'll go the lisp or the scheme way later on),
- whitespace is something that I can ignore between symbols but might have to focus on when dealing with strings and comments
- before I move on with the red in TDD, have to frame the representation strategy in golang:
- a token stream can be a slice of token
- from what I charted about above, a token can be a struct with an id string and repr string
- checkout code until this commit if you're following along: https://github.com/rajp152k/lisp-in-go/commit/f645c41f82c02aac69feacc4c82ccb67cdb09f1b
- have added a basic failing test so that can form the basis for my work tomorrow.
- all sentinels ready, gathering some momentum
2.5. 0x22DB
- now that I can read strings into my customized tokens, I can start working on the internal representations and axioms of the language.
- going for a theoretical detour before I jump
2.6. 0x22DA
- the basics of writing a lisp:
- the lexer, the parser, the evaluator and the environment
- checkout https://github.com/rajp152k/lisp-in-go
- I'm not going to try and be write from the get go but make meaningful progress : I'm not starting out with a map (I could fetch one before I start) but am actually just going to drive in a foggy night.
- will proceed with TDD : I do like my emacs golang workflow right now
- will be starting out with McCarthy's original Lisp and building from there onwards.
- will also need to brush up some theory, all relevant nodes in this buffer regarding compilers should help out.
Tags::golang:lisp: