Starting New Projects in Common Lisp (SBCL) using CLPM in VsCode in 2021

Prerequisites

If you are on Linux, Mac, or Windows WSL, and you install sbcl via this plugin for asdf-vm, you can check out the README in the plugin for simplified instructions on how to install CLPM. If you’re using Roswell, it might be worth checking this out. If you’re not already using Roswell, I recommend not using it unless you have a really good reason. Almost everything you might think you need Roswell for is actually better handled in Common Lisp once you get your bearings.

The TL;DR; version

  1. Create a folder for your new project
    1. Start a repl
      rlwrap sbcl
      
    2. Create a directory for the project called experiment
      (ensure-directories-exist "experiment/") ;; That last slash matters
      
    3. Exit the repl
      (sb-ext:exit) ;; You can also just do a ctrl-d
      
  2. Open vscode in your newly created directory
    cd experiment
    code .
    
  3. Create a new Common Lisp System using Alive.
    1. Inside of vscode, Open Command Palette on the menu at the top View/Command Palette
    2. Generate skeleton: Alive: System Skeleton
    3. The previous command should have generated the following directory structure
      • src
        • app.lisp
          (defpackage :app
            (:use :cl))
          
          (in-package :app)
          
      • test
        • all.lisp
          (defpackage :test/all
            (:use :cl
                  :app)
            (:export :test-suite))
          
          (in-package :test/all)
          
          (defun test-suite ()
            (format T "Test Suite~%"))
          
      • experiment.asd
        (in-package :asdf-user)
        
        (defsystem "experiment"
          :class :package-inferred-system
          :depends-on ("experiment/src/app")
          :description ""
          :in-order-to ((test-op (load-op "experiment/test/all")))
          :perform (test-op (o c) (symbol-call :test/all :test-suite)))
        
        (defsystem "experiment/test"
          :depends-on ("experiment/test/all"))
        
        (register-system-packages "experiment/src/app" '(:app))
        (register-system-packages "experiment/test/all" '(:test/all))
        
  4. Set up a clpm bundle in your terminal
    1. Start up your lisp repl
      rlwrap sbcl
      
    2. Init/install your clpm bundle, switch to your project context, and load it. It may drop into a debugger on some of these steps, if it does, just choose option 0 and hit enter.
      (clpm-client:bundle-init #p"/Users/me/Desktop/experiment/clpmfile" :asds `("experiment.asd"))
      (clpm-client:install :context #p"/Users/me/Desktop/experiment/clpmfile")
      (clpm-client:activate-context #p"/Users/me/Desktop/experiment/clpmfile" :activate-asdf-integration t)
      (asdf:load-system :experiment)
      
  5. Run your tests
    (asdf:test-system :experiment)
    
  6. Set up swank and hook it to vscode
    1. In your clpmfile, add a new line that looks like this
      (:system :swank)
      
    2. Your clpmfile should look something like this
      ;;; -*- Mode: common-lisp; -*-
      (:api-version "0.3")
      
      (:source "quicklisp"
               :url "https://quicklisp.common-lisp-project-index.org/"
               :type :ql-clpi)
      
      (:asd "experiment.asd")
      
      ;; Dev Dependencies
      ;;; Swank for that dank repl dev
      (:system :swank)
      
    3. Have clpm download swank, load it with asdf, and start it up
      (clpm-client:install :context #p"/Users/me/Desktop/experiment/clpmfile")
      (asdf:load-system :swank)
      (swank:create-server)
      
    4. Attach vscode to the repl from inside of vscode and get to work.
      • View/Command Palette
      • Alive: Attach to REPL

The longer version

Things To Keep in Mind

  • Read this. Yes, it is long, but having your head around this helps when things get weird.
  • I am calling this project experiment. You can call it whatever you want, but all of the examples here will use experiment.
  • I will spend as much time in the REPL as possible. In Common Lisp development, your runtime is also the compiler, which is weird at first but has some huge advantages. The largest for me, in this case, is that the repl is universal, which means that the docs should mostly work regardless of the operating system.
  • Common Lisp’s runtime is the compiler as well. You effectively have a real-time runtime/compiler. It is one of the things that make Common Lisp unique. This is one of its properties that make all of the macro stuff that lispers yammer on about so powerful. It’s also why repl development in lisp is so different than other languages and why it is important to have it well-integrated into your editor.
  • When a lisp enters the debugger, it is not the end like it is in most languages. Make sure you’re reading the messages. Get used to being there; it’s a good place to be.
  • This run-through uses ASDF. ASDF is relatively old and is included with SBCL. It doesn’t pin versions or download things for you. It doesn’t do much of what you’d expect a modern package manager to do because it was never designed to do them. It coordinates the loading of files and some other housekeeping activities. It assumes a lot of things about your environment, and it is very configurable. Common Lisp has a history of sharing dependencies across your entire machine, and things like quicklisp go along with that.
  • This run-through uses CLPM. CLPM is more like the project managers that you are used to in other languages. I want the ability to pin versions in my projects, source projects that only exist in Github, not share dependencies across my entire computer, and use libraries in quicklisp. For that, and other reasons, I use clpm. There are other tools, and at some point, I will write about them, but today they don’t matter. You don’t have to do things my way, but you will have a better time if you do for this example.
  • I use rlwrap, which gives me history and search in my repl. If you don’t have access to it, leave it off of any example commands that I provide.

Steps to create a new project

  1. Create a folder for your new project

    1. Start a repl
      rlwrap sbcl
      
    2. Create a directory for the project called experiment
      (ensure-directories-exist "experiment/") ;; That last slash matters
      
    3. Exit the repl
      (sb-ext:exit) ;; You can also just do a ctrl-d
      
  2. Open vscode in your newly created directory

    cd experiment
    code .
    
  3. Create new Common Lisp System using Alive.

    Note: You don’t have to use Alive to create a skeleton project. There are dozens of ways to do this, I am just trying to keep it simple, so do this however you want.

    1. Inside of vscode, Open Command Palette on the menu at the top View/Command Palette
    2. Generate skeleton: Alive: System Skeleton
    3. The previous command should have generated the following directory structure
      • src
        • app.lisp
          (defpackage :app
            (:use :cl))
          
          (in-package :app)
          
      • test
        • all.lisp
          (defpackage :test/all
            (:use :cl
                  :app)
            (:export :test-suite))
          
          (in-package :test/all)
          
          (defun test-suite ()
            (format T "Test Suite~%"))
          
      • experiment.asd
        (in-package :asdf-user)
        
        (defsystem "experiment"
          :class :package-inferred-system
          :depends-on ("experiment/src/app")
          :description ""
          :in-order-to ((test-op (load-op "experiment/test/all")))
          :perform (test-op (o c) (symbol-call :test/all :test-suite)))
        
        (defsystem "experiment/test"
          :depends-on ("experiment/test/all"))
        
        (register-system-packages "experiment/src/app" '(:app))
        (register-system-packages "experiment/test/all" '(:test/all))
        
  4. Set up a clpm bundle so you can have version pinning via lockfiles, git sources, etc.

    1. Initialize your project
      1. Start up your lisp repl again
        • rlwrap sbcl
          
      2. In your repl make sure clpm is running
        • Note: If the command below doesn’t return a version, you need to get clpm installed and working. Nothing else in this example will work without clpm working correctly.
        • (clpm-client:clpm-version)
          
      3. Create a bundle
        • Note: This creates your clpmfile. If you’re coming from node, this kind of like your package.lock in node, or your Cargo.toml file in rust.
        • (clpm-client:bundle-init #p"/Users/me/Desktop/experiment/clpmfile" :asds `("experiment.asd"))
          
      4. Initialize your bundle, create a lockfile, and create a context
        • Note: This will drop into the debugger; this is normal in common lisp. The program is not dead when it drops into the debugger unless you want it to be. You can do all sorts of things from there to help your program through whatever problem it has encountered. In this case, it just wants you to approve the change. Type 0 and hit <enter>. After you do that, it will return T, which means it is loaded, and you are good to go.
        • (clpm-client:install :context #p"/Users/me/Desktop/experiment/clpmfile")
          
      5. Switch to your clpm context
        • Note: This can drop you into your debugger again, this time to warn you that you’re about to change out of an active context. If it does, choose 0 to continue.
        • Note: This also activates a clpm integration with asdf. After you activate a context like this, if you try to use a library that is not installed, the debugger will provide an option to install it.
        • (clpm-client:activate-context #p"/Users/me/Desktop/experiment/clpmfile" :activate-asdf-integration t)
          
      6. Load your asdf system
        • Note: all of the previous commands have been clpm-client commands; this next one is an asdf command that loads the experiment project, the very project we just created.
        • (asdf:load-system :experiment)
          
      7. List all of your loaded asdf systems
        • Note: This is a quick way to check for what all systems are loaded. In this case, you’ll probably see asdf, uiop, asdf-package-system, clpm-client, experiment, experiment/test, and experiment/src/app.
        • (asdf:registered-systems)
          
      8. Run your tests
        • (asdf:test-system :experiment)
          
      9. You’re done! You can shut it down.
        (sb-ext:exit) ;; You can also just do a ctrl-d
        

Hooking your project up to vscode

This is where I think lisps get fun. Let’s start up our app, hook vscode to the running instance of our app, and edit it while running. We will use the SWANK server to do it, which is the same server that emacs SLIME uses for its lisp integration.

  1. Let’s add swank to our project as a dev dependency.
    • In your clpmfile, add a new line that looks like this
      (:system :swank)
      
    • Your clpmfile should look something like this
      ;;; -*- Mode: common-lisp; -*-
      (:api-version "0.3")
      
      (:source "quicklisp"
              :url "https://quicklisp.common-lisp-project-index.org/"
              :type :ql-clpi)
      
      (:asd "experiment.asd")
      
      ;; Dev Dependencies
      ;;; Swank for that dank repl dev
      (:system :swank)
      
    • Have clpm download swank and put it in the right place for this project.
      cplm bundle install
      
  2. Start up a sbcl
    • Note: You can start sbcl and auto-load this project with the asdf integration by running rlwrap clpm bundle exec -- sbcl if you want. That will make read the clpmfile and set up your context correctly. Keep in mind that your context only makes your libraries available, it doesn’t actually load them. You use ASDF to load libraries. This means that having something like swank as a dev dependency only puts swank in your runtime if you actually load it somewhere. If you were feeling REALLY wild, you could skip straight to step 7 by starting it with clpm bundle exec -- sbcl --eval '(asdf:load-system :swank)' --eval '(swank:create-server)' --eval '(asdf:load-system :experiment)' which would actually load swank and start it up, and load your project. I am putting the full instructions here to keep things moderately consistent, though. Feel free to experiment.
    rlwrap sbcl
    
  3. Activate clpm for this project
    (clpm-client:activate-context #p"/Users/me/Desktop/experiment/clpmfile" :activate-asdf-integration t)
    
  4. Load swank
    (asdf:load-system :swank)
    
  5. Start up a swank server
    (swank:create-server)
    
  6. Load up the experiment project from earlier
    (asdf:load-system :experiment)
    
  7. Attach vscode to the repl using the VsCode Command Palette.
    • View/Command Palette
    • Alive: Attach to REPL
  8. You now have vscode attached to the swank server. This attaches your editor to a running lisp compiler… in this case, a compiler with your project loaded. You can run commands in the regular CLI repl and the repl in vscode, but where it gets more interesting is when you load files directly from the project and evaluate parts of the code as you work. Check out the video above to see a really trite example of what I mean.

More To Come?

  • I’d like to add more explanation as to what each of these steps does, show how to add runtime deps using clpm, and generate standalone binaries that don’t include dev dependencies like swank or the clpm client. I’ll get to them over time.