Chicken Soup

(a bunch of stuff in a pot)

REPL

The Chicken csi REPL is appalling after using some nice REPLs, it doesn’t even have history by default. I couldn’t reliably get non-GNU readline-likes to work, so:

% chicken-install -sudo readline
% cat >~/.csirc
(use readline)
(current-input-port (make-readline-port))
(install-history-file #f "/.csi_history")
^D

So at least now it has the usual up/down/emacs-like keys.

Long fucking ways from the old Symbolics LISP Machines. Why don’t we have environments like that anymore? Why is everyone content to just use fucking emacs (I’ve never been emacsulated) or other editor, and a boring REPL? DrRacket is just a REPL that destroys its memory every time you edit code, and it’s the most graphically advanced LISP-type environment. And this is why I still just use Atom with Symbols Tree View (even though it thinks variable definitions are functions), and copy-paste into iTerm if I want to test something.

Value Unpacking

Not having nice R6RS macros for this, and unwilling to fight with classic macros, I’ve been using values to unpack lists into variables, and because I can never remember the exact syntax, I made this cheat-sheet:

(define a '(1 2 3))
(define b '(4 5 6))
;; then one of these:
(define-values (x y z) (apply values a)) (printf "~s,~s,~s\n" x y z)
(set!-values (x y z) (apply values b)) (printf "~s,~s,~s\n" x y z)
(let-values [[(x y z) (apply values a)] [(q r s) (apply values b)]] (printf "qrs:~s,~s,~s xyz:~s,~s,~s\n" q r s x y z))

Probably not efficient, but better than car, cadr, caddr, etc. Maybe I should move all my list-structures into vectors, but then I’d still have to convert them to lists half the time. Here’s where Python is the programmer’s best friend, even if it is 10,000x slower:

a = (1, 2, 3)
x, y, z = a
print(f"{x},{y},{z}")

Why Did LISP Fail?

How did a more advanced language with better tools just die off commercially, and now if you want to work in it, you have to cobble together a bunch of half-broken shit?

I think there’s 3 reasons:

  1. It’s hard and ugly. It may be logically compelling, but when you see a page of parens your brain panics and looks for a place to hide.
  2. Companies value the fake productivity of thousands of lines of C, Java, or Swift (aka C++2020) code more than having safety, security, and correct reasoning. Who cares if millions of people will suffer and possibly die from your code, as long as you can ship TODAY?
  3. A lot of LISP “hackers” are insufferable douchebags, both old beardy fuckers who’ve been doing it for 50 years and mewling children who learned it last week. Every new variant makes the older contingent more angry at even seeing a mention of it, and the sneering fetuses think whatever variant they learned is Divine Wisdom, rather than just an engineering tool that may need to be improved.

Building a Binary with Chicken Scheme

So that was fucking fun. Seems Chicken’s docs aren’t correct on how to build with modules, because those were added after the dark-ages R5RS it was modelled on.

There is an “egg” system which is used for building the libraries, but it’s difficult to use for making binaries in your own destination dir, and fills your work dir with spam temp files. Unfortunately basically useless for work.

Finally got this working:

src/somelib.scm:

(declare (unit somelib))
(module somelib
    (hello)

(import scheme)

(define (hello) (display "Hello!\n"))
)

src/somemain.scm:

(import scheme
    (chicken load)
)

(cond-expand
    (compiling (declare (uses somelib)))
    (else (load-relative "somelib.scm")))
(import somelib)

(hello)

Don’t you just love that muffin-man-muffin-man repetitious bullshit? declare is for the compiler, load is for the interpreter, then import for both. You import builtins like chicken, but use libraries (aforementioned sdl) without an import. Bizarre and contradictory.

build.zsh: [script updated 2019-06-08]

#!/bin/zsh

# EXE is binary filename
# MAIN is main script
# LIBS is space-delimited, must not include main script

function usage {
    echo "Usage: build.zsh PROJECT || MAIN.scm [LIBS] || ls || -?"
    exit 1
}

if [[ $# -eq 0 || "$1" == "-?" || "$1" == "--help" ]]; then
    usage
fi

case $1 in
    ls)
        echo "Known projects:"
        # Add known projects here
        echo "eldritch test"
        exit 0
    ;;
    eldritch)
        EXE=eldritch
        MAIN=eldritch.scm
        LIBS="marklib.scm marklib-geometry.scm marklib-ansi.scm"
    ;;
    test)
        EXE=marklib-test
        MAIN=marklib-test.scm
        LIBS="marklib.scm marklib-geometry.scm"
    ;;
    *)
        # command-line project: MAIN=$1, LIBS=$2...
        EXE=`basename $1 .scm`
        MAIN=$1
        shift
        LIBS="$*"
    ;;
esac

mkdir -p bin

cd src
for SF in ${=LIBS}; do
    SNAME=`basename $SF .scm`
    echo "Compiling $SF"
    csc -c -j $SNAME $SF -o $SNAME.o || exit 1
done
echo "Compiling $MAIN"
csc -c $MAIN -o `basename $MAIN .scm`.o || exit 1
echo "Linking..."
csc *.o -o ../bin/$EXE || exit 1
rm -f *.o
rm -f *.import.scm
cd ..

echo "Built bin/$EXE"
% ./build.zsh
Built bin/something
% ./bin/something
Hello!

Hello, working native binary! There’s a bunch more stuff about making a deployable app, but I’ll take this for now. My actual program can show a blank green graphics window!

More Fun and Swearing with Scheme

I have ideas for some little games, which aren’t suitable as giant Electron-powered applications. My preference in language for this would be Objective-C, Scheme, Pascal, C if I absolutely had to. Obj-C’s lack of portability rules it out for now, but if GNUstep gets their crap together it may go back in the running. Scheme looked promising, I’ve liked using Chez Scheme for some data processing.

So after 2 days of experimentation and abuse, and none of my tools working right in the process, I was unable to get thunderchez/sdl2 to link. I can explicitly load the dylib, and it doesn’t find SDL_Init. I wrote a little C program to prove that SDL is reachable, and it is. Chez just won’t load it.

Frustrated, I grabbed Chicken Scheme again, did a chicken-install -sudo sdl2, ran the sample code(!), and bam, it just worked. 15 minutes of effort at most, and I’m up and running.

Down side, Chicken compiles slow, and the interpreter is PAINFULLY slow; Chez “interprets” by compiling fast with low optimizations. And Chicken defaults to R5RS which is… 1990 called and wants me to watch MIT SICP lectures on VHS. It has some R7RS support, but I prefer the guarantees of portability, safety, and specified behavior in R6RS. Have to go looking thru SRFI docs to find any feature, it’s not batteries-included. Oh well, I’ll probably live just fine without ideological purity.

Programming is a Joy

“Programming is a joy. That’s why people do it. No one should spend hours in front of a computer terminal out of some dreary sense of duty, or because they have some vague notion of becoming “computer literate”. That’s not the point. Programming ought to be fun—and if you’re not having fun, you shouldn’t waste your time.”
—Michael Eisenberg, “Programming in Scheme” (1988)

Software Tools Quote

“Finally, it is a pleasure to acknowledge our debt to the Unix operating system, developed at Bell Labs by Ken Thompson and Dennis Ritchie. We wrote the text, tested the programs, and typeset the manuscript, all within Unix. Many of the tools we describe are based on Unix models. Most important, the ideas and philosophy are based on our experience as Unix users. Of all the operating systems we have used, Unix is the only one that has been a positive help in getting a job done instead of an obstacle to be overcome. The world-wide acceptance of Unix indicates that we are not the only ones who feel this way.”
“Software Tools in Pascal”, 1981, by Brian W. Kernighan, P.J. Plauger

As every neckbearded n-gate reader will now rush to well-actually at me, BWK’s experience writing this book led to Why Pascal Is Not My Favorite Programming Language, but note this rant is about “standard” ANSI Pascal, not the somewhat improved P-Code Pascal of the ’70s or the free-wheeling super-powered Turbo Pascal of the early ’80s, and nothing like modern FreePascal. Standard Pascal was a deliberately simplified pedagogical language, not a systems programming language, which the later ones are.

Anyway, the book’s interesting as a problem-solving exercise, but the Unix part amused me. And no, Linux is Not Unix. Buy a Mac or install BSD if you want UNIX®.

Text Filter

  1. I need a utility to filter some text, like awk but more modern regexp.
  2. Start writing utility.
  3. Discover I wrote this exact utility in 2006, and forgot about it. Apparently I did something with it in 2009. I may have blogged about it on my old site, too lazy to go searching.
  4. Clean it up for Python 3.7 and release.

Utility/Filter 1.1 can be downloaded, put it in your ~/bin or /usr/local/bin folder and call it with Filter.py -h.

BSD license, so do what thou wilt with it, but don’t be a dick (uploading my stuff to github as if it’s yours is dickish), OK?

Julia More Packaging & Code

I don’t just drink coffee or booze, watch movies and Internet drama, and look cool. I code sometimes, too! Who knew?!

Carrying on with my experiment in Julia, packaging has another step needed to make references. For instance, Ansi uses Geometry, so:

Geometry/Project.toml:


authors = ["Mark Damon Hughes "]
name = "Geometry"
uuid = "e3172796-a620-11e8-2cbf-612649bb77f8"
version = "0.1.0"

[deps]

Ansi/Project.toml:


authors = ["Mark Damon Hughes "]
name = "Ansi"
uuid = "72992c94-a620-11e8-3d05-55611ea0dbd0"
version = "0.1.0"

[deps]

Geometry = "e3172796-a620-11e8-2cbf-612649bb77f8"

Ansi/Manifest.toml:


[[Geometry]]
repo-rev = "master"
repo-url = "/Users/mdh/Code/CodeJulia/Geometry"
uuid = "e3172796-a620-11e8-2cbf-612649bb77f8"
version = "0.1.0"

All the boldface code is what I wrote/copy-pasted, the rest is generated by juliaMakePackage.zsh. I may go ahead and make a tool to link projects, because it’s so error-prone. In fact, I cheated, and made a single Manifest.toml which I copy to all projects so far, and can replace whenever something updates.

Anyway, this gets me to a nice state where I can write using Ansi in my project and it’ll just find it. IIUC, if I move all the libraries to a public repository, I can just change the repo-url and the packages are downloaded into ~/.julia cache somewhere.

I still haven’t followed up on making a binary application; the more I look into that, the jankier it seems, more like something to defer until there’s an official solution. Putting a real UI on it is also something to work on, but that’s much more doable.

Coding

I’ve written a lot more code, over 1000 LOC, not just screwing around with packages. Mostly this is enjoyable, it’s a nice systems programming language. The ugly parts haven’t yet driven me insane, they’re just things to work around or ignore. Far less frustrating than almost any other new language; Rusty Nail In Your Head and Go Fuck Yourself Its Google aren’t my favorites.

Strong typing really is a pain in the ass. Declare a variable or struct field foo, and it takes anything. Type it with foo::AbstractString, and you soon learn nothing is not a string; foo::Union{AbstractString,Nothing} is necessary to be nullable. Ick.

Enumerations

Enumerated types @enum are disappointing. They’re a little smarter than C enums, but not as useful as Java enums. They just represent a value; but you have to cast them to Int every time you use them for their value, so too painful to use them as array indices. Or as characters, a thing I like a lot for debugging. And they’re not easy to reflect on:

julia> @enum Terrains begin
               Ter_Floor = Int('.')
               Ter_Wall = Int('#')
       end
julia> Ter_Wall
Ter_Wall::Terrains = 35
julia> Int(Ter_Wall)
35
julia> Char(Int(Ter_Wall))
'#': ASCII/Unicode U+0023 (category Po: Punctuation, other)
julia> String(Char(Int(Ter_Wall)))
ERROR: MethodError: no method matching String(::Char)
julia> string(Char(Int(Ter_Wall)))
"#"
julia> # FFS
julia> string(Ter_Floor)
"Ter_Floor"
julia> # Surprisingly easy!
julia> instances(Terrains)
(Ter_Floor::Terrains = 46, Ter_Wall::Terrains = 35)
julia> # Shit, this is a named tuple, not a dictionary!
julia> useful_instances = Dict()
Dict{Any,Any} with 0 entries
julia> for v in values(instances(Terrains))
           useful_instances[ string(v) ] = v
           useful_instances[ string(Char(Int(v))) ] = v
       end
julia> useful_instances
Dict{Any,Any} with 4 entries:
  "Ter_Floor" => Ter_Floor
  "Ter_Wall"  => Ter_Wall
  "#"         => Ter_Wall
  "."         => Ter_Floor
julia> # JFHC

That was an annoying adventure to get a simple reverse lookup.

Julia Local Packaging

So I wanted to move all my common Julia code to a support dir. My filesystem has for 30+ years contained:

$HOME/
    Code/
        CodeC/
            foo/
                src/
                    bar.c
        CodeJava/
            foo/
                src/
                    com/
                        mdh/
                            bar/
                                Quux.java
        et fucking cetera

Build scripts for most languages expect something very like this, and it’s easy to import one package’s source into another, so I could put common code in a “Marklib” project, and get work done.

Making this happen in Julia was a lot more difficult. With a little help from Slack I made sense of the terrible package documentation for Julia and the incomprehensible errors, and wrote a script juliaMakePackage.zsh:

#!/bin/zsh
if [[ $# -ne 1 ]]; then
    echo "Usage: juliaMakePackage.zsh NAME"
    exit 1
fi
name=$1
devdir=$HOME/Code/CodeJulia
cd $devdir
julia -E "using Pkg; Pkg.activate(\".\"); Pkg.generate(\"${name}\")"
cd $name
git init
git add .
git commit -m "Initial commit"
cd ..
julia -E "using Pkg; Pkg.develop(PackageSpec(url=\"${devdir}/${name}\"))"

And then added the main dir and all packages I make to ~/.julia/config/startup.jl:

# startup.jl

push!(LOAD_PATH, pwd())
push!(LOAD_PATH, "$(homedir())/Code/CodeJulia")
push!(LOAD_PATH, "$(homedir())/Code/CodeJulia/Marklib")

println("READY $(pwd())")

Now finally I can:

% julia
READY /Users/mdh
julia> using Marklib
julia> Marklib.greet()
Hello World!
julia> 

And from there start putting in my libs. Each one needs a package and a startup entry; I may have to automate that by walking my code dir. Waste of several hours figuring that out.

Julia String Concatenation

Last time, I was uncertain about string concatenation, so I did a test:

#!/usr/bin/env julia

const kTestText = "abcdefghijklmnopqrstuvwxyz0123456789\n"
const kLoops = 10000

function stringString()
    s = ""
    for i in 1:kLoops
        s = "$s$kTestText"
    end
    return s
end

function bufferString()
    sb = IOBuffer()
    for i in 1:kLoops
        print(sb, kTestText)
    end
    return String(take!(sb))
end

function vectorString()
    sb = Vector()
    for i in 1:kLoops
        push!(sb, kTestText)
    end
    return join(sb, "")
end

function typedVectorString()
    sb = Vector{AbstractString}()
    for i in 1:kLoops
        push!(sb, kTestText)
    end
    return join(sb, "")
end

println("*** stringString")
@timev stringString()
sleep(2)

println("\n*** bufferString")
@timev bufferString()
sleep(2)

println("\n*** vectorString")
@timev vectorString()
sleep(2)

println("\n*** typedVectorString")
@timev typedVectorString()

*** stringString
  1.100197 seconds (21.24 k allocations: 1.725 GiB, 15.94% gc time)
elapsed time (ns): 1100197167
gc time (ns):      175349222
bytes allocated:   1851904041
pool allocs:       11292
non-pool GC allocs:9950
GC pauses:         79

*** bufferString
  0.006864 seconds (11.80 k allocations: 1.134 MiB)
elapsed time (ns): 6864042
bytes allocated:   1189493
pool allocs:       11794
non-pool GC allocs:3
realloc() calls:   8

*** vectorString
  0.017380 seconds (26.68 k allocations: 2.191 MiB)
elapsed time (ns): 17380237
bytes allocated:   2297091
pool allocs:       26659
non-pool GC allocs:9
realloc() calls:   8

*** typedVectorString
  0.031384 seconds (44.75 k allocations: 2.999 MiB, 10.00% gc time)
elapsed time (ns): 31384383
gc time (ns):      3137654
bytes allocated:   3144221
pool allocs:       44730
non-pool GC allocs:8
realloc() calls:   8
GC pauses:         1

Well, there’s me told off. I expected #1 typedVector, #2 vector, #3 buffer, then stringString way at the bottom. Instead the first 3 are reversed.

IOBuffer, as ugly as it is, is the clear winner. Vector did OK, but twice as much CPU & RAM loses. Amusing that typedVector is twice as slow and memory-heavy as the untyped (explained ). On larger loops, buffer gets slower, but vector remains a memory pig, and in GC that’s unacceptable. Of course stringString is terrible, and it’s almost exactly the same for string(s, kTestText).

Time to rewrite some text processing.

More Julia

Decided to take another day on Julia, write something more serious and see how that goes.

There’s an uber-juno “IDE” plugin for Atom, which at least turns on syntax highlighting and puts an interactive console in the editor. Yay. It’s not capable of linting yet, though it says it is.

So I’m rewriting a simple 1970s-style dungeon crawl game (Chorus:”As if you could make any other kind of game, Mark!” Mark:”I assure you I could, I just choose not to.”) as a test of data structures and application programming in Julia. It’s a little challenging, but not impossible.

Using Modules

The current directory is not included in the default LOAD_PATH, so you can’t import local modules right off. One solution is to put it in your startup:

% mkdir -p ~/.julia/config
% echo '@everywhere push!(LOAD_PATH, pwd())' >>~/.julia/config/startup.jl

All names are in the same namespace. This means your module and a struct or method in it can’t have the same names… My current solution has been to pluralize the module, so GridMaps contains a struct GridMap.

The export rules are a little annoying. If you use the @enum macro to make a ton of constants, they aren’t exported when you export the enum type; you can either manually export each constant name, or just use ModuleName.ConstantName in other modules. Bleh.

Debugging

Bug #1: The terminator problem is pretty bad in Julia:

% cat Foo.jl
module Foo
for i=1:10
    println(i)
#missing end
end #module

% julia Foo.jl
ERROR: LoadError: syntax: incomplete: "module" at /Users/mdh/Code/CodeJulia/Foo.jl:1 requires end
Stacktrace:
 [1] include at ./boot.jl:317 [inlined]
 [2] include_relative(::Module, ::String) at ./loading.jl:1038
 [3] include(::Module, ::String) at ./sysimg.jl:29
 [4] exec_options(::Base.JLOptions) at ./client.jl:229
 [5] _start() at ./client.jl:421
in expression starting at /Users/mdh/Code/CodeJulia/Foo.jl:1

Good luck finding that missing end if you have a 1000-line module. C used to be just as bad about semicolons and braces, but modern compilers are pretty good at guessing where you fucked up. Python’s whitespace-as-control is brilliant, because you can’t ever do that. A passable solution would be each control keyword having its own unique end keyword, but it’s too late for that. In the actual bug, I had to comment out half the code, run the module, uncomment and comment the other half, repeat until I isolated it.

Bug #2: Type annotations need to be very generic or left off entirely. As code-as-documentation, I declare a function as parseLine(line::String), and it gives me:

MethodError(Main.Foo.parseLine, ("a",), 0x00000000000061bb)

Well, thanks. Turns out I need to use AbstractString, because a prior function returns an AbstractString and not String. Or I can just leave the typing off, as was my first instinct.

String Concatenation

This is super ugly. Currently I’ve fallen back on:

sb = Vector()
push!(sb, "part of ")
push!(sb, "a string")
return join(sb, "")

Strings aren’t mutable, and there’s no StringBuffer/NSMutableString equivalent. The other option is to use an IOBuffer. I haven’t done timings yet to see which is faster/uses less memory, I just find pushing to a vector simpler.

Switch

There’s no switch statement. I could put functions in a dictionary and dispatch on that, which is not ideal for looping over a bunch of simple values, and requires passing around control flags instead of a simple break or return. Caveman solution is a chain of if/elseif, but I don’t like it. Possibly a macro could be written?

Binary

Turns out you can make binaries: Julia apps on the App Store: Building and distributing an application written in Julia

It’s an ugly process, that Nathan’s working around, but it’s a start. This should 100% be in the core libraries, and should have a cross-compiler.

Getting this working is tomorrow’s problem, I think.