Writing a lisp

Any language is defined by the tooling that processes the source (they conform to a set of standards/expectations).

What makes lisp "lisp" is homoiconicity: any toolchain that I write, should expose the underlying AST in a manner that it is flexibly manipulable.

Other than that, the generic requirements to an interpreter would be:

• the Lexer : from characters to tokens
• the parser : from tokens to interpretable forms
• Evaluator : evaluating forms in a context
• Environment : the context

Crafting a basic lisp without much influence from all the existing major projects is an interesting exercise every aspiring lisp demigod might benefit from partaking in

Over the course of my life, I might consider writing a lisp in different languages.

To make the experience more uniform, I'll be starting out with McCarthy's original Lisp (courtesy: Paul Graham )

• Understand the Lisp language
  • Syntax and semantics
  • Core concepts: symbols, lists, S-expressions
• Design the interpreter structure
  • Lexer/Tokenizer
  • Parser
  • Evaluator
  • Environment representation
• Lexer/Tokenizer
  • Convert input character stream into tokens
  • Handle whitespace and comments
  • Recognize symbols, numbers, strings, parentheses
  • Example: tokenize "(+ 1 2)" into ['(', '+', '1', '2', ')']
• Parser
  • Construct Abstract Syntax Tree (AST) from tokens
  • Recognize and handle S-expressions
  • Create data structures to represent expressions
  • Example: parse ['(', '+', '1', '2', ')'] into an AST
• Evaluator
  • Traverse and interpret the AST
  • Implement core Lisp operations: addition, subtraction, etc.
  • Recursively evaluate nested expressions
  • Maintain an environment for variable bindings
• Environment Representation
  • Implement variable storage and retrieval mechanism
  • Handle lexical scoping
  • Example: store variables as key-value pairs in a dictionary
• Critiques and Considerations
  • Error handling: ensure meaningful error messages for invalid inputs
  • Performance: optimize evaluation for larger expressions
  • Extendability: consider how to add more features or functions
• Connections
  • Lexer/Tokenizer feeds tokens into the Parser
  • Parser outputs an AST for the Evaluator to process
  • Environment interacts with the Evaluator for variable management