Adult Engineer Over-Optimization as the Motie Problem

Looking at my Scheme code and the way I customize it, I'm starting to see the real reason evil megacorps (and wannabe evil startups) won't hire even middle-aged programmers or use your favorite weirdo language, they just want young idiots who code Java or Go.

If you think about a standard software career, there's maybe 10 years of a submissive fool badly coding crap languages ^1 like Java, Go ^3, PHP, JavaScript ^4. They just got out of college or self-trained, and can barely copy existing algorithms, let alone think of one for themselves. This is why FizzBuzzTest ^5 is such a good novice coder test: It requires following directions exactly, and slightly competent logic skills, but not much more.

Then maybe 10 years of them being project managers and "architects", running waterfall and GANTT charts; they'll say they're "agile" but then have a giant JIRA repo of "backlog" features which have to be implemented before shipping, weekly 4-hour planning "backlog grooming" meetings, and unrealistic estimates. This is sufficient to build all kinds of horrible vertical prisons of the mind like Azkaban Facebook.

Then they either retire, or are "downsized", and now what? So they work on their own code, do maintenance on old systems, or leave the industry entirely.

If they work on their own, freed of evil megacorp constraints, they're going to end up in something idiosyncratic and expressive, like Scheme, LISP, Forth, or a custom language. Make their own weirdo environment that's perfectly fit to themself, and unusable/unreadable by anyone else.

Case in point, I needed an object model. There's one I like in Gerbil, and Gerbil's blazing fast, but I can't make a full SDL2 library for it yet (Gambit's FFI is hard, I've hit some bugs, and there's a LOT of library to interface to), and I'm using a bunch of other Chickenisms anyway, so I can't really move to it yet. Instead I just made my own simple object libary, with a couple macros to hide the ugly reality behind it:

(test-group "Object"
    (test "Object" 'Object (class-name Object))
    (let [ (obj (@new Object))  (bug )  (cow )  (duck ) ]
        (test "Object-to-string" "[Object]" (@call obj 'to-string))

        (define-class Animal Object)
        (define-field Animal 'legs 0)
        (define-field Animal 'color )
        (define-method Animal 'init (self legs color)
            (set! (@field self 'legs) legs)
            (set! (@field self 'color) color) )
        (define-method Animal 'speak (self)
            (sprintf "The ~A ~A with ~A legs says " (@field self 'color) (class-name (@class self)) (@field self 'legs)) )

        (set! bug (@new Animal 6 "green"))
        (test "bug-legs" 6 (@field bug 'legs))
        (test "bug-color" "green" (@field bug 'color))
        (test "Bug speak" "The green Animal with 6 legs says " (@call bug 'speak))

        (define-class Cow Animal)
        (define-method Cow 'init (self color)
            (@super self 'init 4 color) )
        (define-method Cow 'speak (self)
            (string-append (@super self 'speak) "MOO!") )
        (set! cow (@new Cow "brown"))

        ;; second class to make sure classes don't corrupt shared superclass
        (define-class Duck Animal)
        (define-method Duck 'init (self color)
            (@super self 'init 2 color) )
        (define-method Duck 'speak (self)
            (string-append (@super self 'speak) "QUACK!") )
        (set! duck (@new Duck "black"))

        (test "Cow speak" "The brown Cow with 4 legs says MOO!" (@call cow 'speak))
        (test "Cow to string" "[Cow color:brown;legs:4]" (@call cow 'to-string))
        (test "Duck speak" "The black Duck with 2 legs says QUACK!" (@call duck 'speak))
        (test "Duck to string" "[Duck color:black;legs:2]" (@call duck 'to-string))

        (test "instance-of?"  (instance-of? cow Cow))
        (test "instance-of? parent"  (instance-of? cow Animal))
        (test "instance-of? grandparent"  (instance-of? cow Object))
        (test "instance-of? cousin-false"  (instance-of? cow Duck))
        (test "instance-of? not an obj-false"  (instance-of? "wtf" Cow))
    )
)

The implementation code's not much longer than the tests, but it's not quite done for me to show off; I need to switch my macros into non-hygeinic forms so I can get rid of the (self) in define-method, and introduce an Objective-C-like _cmd field for self-reflection, and message-not-understood handling. There's always more tinkering to do.

Which is great for me, but makes my code an undocumented (mostly) new language, unusable by anyone normal. A giant pile of crap Java program, no matter how old, can be "worked on" (more crap piled on top) by any teenage Bro Coder.

All of which brought to mind The Mote in God's Eye, where the Motie Engineers over-optimize everything into a tangled mess, and the Watchmaker vermin are even worse, wiring up everything to everything to make new devices. The threat posed by and solution to Scheme programmers, in your usual authoritarian megacorp scenario, is similar to Watchmakers.


^1 Swift is intended to fit this niche much more than weirdo expressive Smalltalk+C Objective-C was, BDSM ^2 to prevent one from writing "bad" code, but it's not there yet; the reality of low-level software dev can't be simplified as much as Apple wants, and their C++ developers weren't up to the task anyway.

^2 Bondage-Domination-Sado-Masochism; aka strict type systems and code flow analysis, that prevent one from writing "bad" code at the cost of annotating everything with types instead of doing useful work. I'm not kink-shaming people who do that for sex, only those who do it to their own software.

^3 Rob Pike has openly said they can't give a powerful language to newbie Googlers, they mostly just know Java, C, C++, which is why Go is so limited and generic.

^4 Oddly, JS is basically a LISP with really shitty syntax. It's easy to make trivial, broken junk in it, but it's also powerful and expressive if you're an old maniac who understands the Self-based object system.

^5 Oh, fine, but only so I can demonstrate something:

(define (fizzbuzz-test i n s)  (if (zero? (modulo i n))  (begin (display s) )  ) )
(define (fizzbuzz i)
    (unless (any identity (list (fizzbuzz-test i 3 'Fizz) (fizzbuzz-test i 5 'Buzz)))  (display i))
    (newline) )
(for (i 1 100) (fizzbuzz i))

Totally different structure from the usual loop-if-else repetition and hardcoding of everything, because Scheme encourages coding in small pieces. Of course I wrote my own for macro which expands to a named let loop; there's many like it but this one is mine. More Motie engineering.

A Scheme of Gerbils Runnin' in a Wheel

Chicken's mediocre performance is causing me problems, so I'm taking yet another language tour/spike.

Looking at Gerbil Scheme, as a faster Scheme and as a bonus it's closer to R7RS. It has far less library support than Chicken, but does have a FFI, and compiles to a native binary through C, so I really don't need too many libraries.

Instead of the #!key #!optional #!rest keywords used in Chicken and some others, Gerbil uses a literal template style for functions:

;; Chicken: (define (foo a #!key (b 'bar) #!optional (c "see?") #!rest rest)
;; Gerbil:
(def (foo a b: (b 'bar) (c "see?") . rest)
    (print "a=" a " b=" b " c=" c " rest=" rest "\n")
)
(foo "Mark" 1 b: "bees!" 'what) ;; note out-of-order optional & keyword
;; a=Mark b=bees! c=1 rest=what

I like this style better, but I dislike the "def" keyword (which uses the enhanced lambda that does this) instead of "define" which uses "lambda%" (traditional lambda).

Gerbil uses [] for list construction, and {} for method dispatch, so goodbye to the nicely distinguishable braces I was using in Chicken. The spec says implementations can do what they want with those symbols, but I wish they wouldn't. Ah, well. I'll add a shitload more whitespace and some comments and it'll be fine.

The struct/object system in Gerbil is pretty nice, they're slightly upgraded records, but there's an (@ obj field) syntax instead of (MyClass-field obj), and (defmethod {mymethod MyClass} ...) creates a polymorphic {mymethod obj args} which finds the right method for any object, which is especially useful for inheritance.

I tried doing some VT100 graphics just to proof-of-concept, but it's doing something to the terminal, such that the same escape codes which work in Python don't clear the screen fully in Gerbil. After a short losing battle with stty and termcap, I give up on that and I'll jump right to writing a C FFI to SDL, because in 2019 that's easier than writing to a console.

Daily reminder that everything we have made since 1984 is overcomplicated junk. On an Atari 800, this took a few seconds to type, and you could start coding a nice UI instantly:

atari-is-awesome-graphics1

Alas, we live in a fallen world, so this is going to be trouble. Here's my Gerbil FFI template so far:

package: myffi

(import
    :std/foreign
)

(export )

(begin-ffi
    ;; names of all Scheme wrappers to expose
    (chello)

(c-declare #<<CDECLEND

 <stdio.h>

int chello(void) {
    printf("Hello this is C!\n");
    return 1;
}

CDECLEND
)

    (define-c-lambda chello () int "chello")

) ;; begin-ffi


; TODO for constants: (define FOO ((c-lambda () int "___result = FOO;")))

(define (main . args)
    (chello)
)

Gerbil has a Scheme-based build system, but I'm a caveman so I make another build.zsh:

#!/bin/zsh

function usage {
    echo "Usage: build.zsh MAIN.scm [LIBS] || -?"
    exit 1
}

if [[ $# -eq 0 || "$1" == "-?" || "$1" == "--help" ]]; then
    usage
fi

mkdir -p bin

main=`basename $1 .scm`

gxc -exe -static -O -o bin/$main "$@" || exit 1
echo "Built bin/$main"

Now:

% ./build.zsh myffi.scm
Built bin/myffi
% bin/myffi
Hello this is C!

Hooray! Unconditional success! Only took all afternoon and a half-pot of coffee!

Now I "merely" have to wrap all of SDL (well, just the parts I need) and get linking working and oh sweet merciless Cthulhu. And I still won't know how much this'll help my performance until I'm days into it. But, the first step is the hardest.

Tower of Babble

Programmers almost compulsively make new languages; within just a few years of there being computers, multiple competing languages appeared:

It proliferated from there into millions; probably half of all programmers with 10+ years of experience have written one or more.

I've written several, as scripting systems or toys. I really liked my Minimal script in Hephaestus 1.0, which was like BASIC+LISP, but implemented as it was in Java the performance was shitty and I had better options to replace it. My XML game schemas in GameScroll and Aiee! were half programmer humor, but very usable if you had a good XML editor. Multiple apps have shipped with my tiny lisp interpreter Aspic, despite the fruit company's ban on such things at the time. A Brainfuck/FORTH-like Stream, working-but-incomplete tbasic, and a couple PILOT variants (I think PILOT is hilariously on the border of "almost useful").

Almost every new language is invented as marketing bullshit based on a few Ur-languages:

  • C++: Swift
  • Java: Javascript (sorta), C#, Go
  • Awk: Perl, Python, PHP, Julia
  • C: Rust
  • Smalltalk: Objective-C
  • Prolog: Erlang, Elixir
  • ALGOL: C, Pascal, PL/1, Simula, Smalltalk, Java
  • LISP: Scheme, ML, Haskell, Clojure, Racket
  • BASIC: None, other than more dialects of BASIC.
  • FORTRAN: None in decades, but is the direct ancestor of ALGOL & BASIC.
  • COBOL: None in decades.

A few of these improve on their ancestors in some useful way, often performance is better, but most do nothing new; it's plausible that ALGOL 68 is a better language than any of its descendants, it just has mediocre compiler support these days.

Certainly I've made it clear I think Swift is a major regression, less capable, stable, fast, or even readable than C++, a feat I would've called impossible except as a practical joke a decade ago. When Marzipan comes out, I'll be able to rebuild all my 15 years of Objective-C code and it'll work on 2 platforms. The Swift 1.0 app I wrote and painfully ported to 2.0 is dead as a doornail, and current Swift apps will be uncompilable in 1-2 years; and be lost when Apple abandons Swift.

When I want to move my Scheme code to a new version or any other Scheme, it's pretty simple, I made only a handful of changes other than library importing from MIT Scheme to Chez to Chicken 4 to Chicken 5. When I tested it in Racket (which I won't be using) I had to make a handful of aliases. Probably even CLISP (which is the Swift of LISPs, except it fossilized in 1994) would be 20 or 30 aliases; their broken do iterator would be hard but the rest is just naming.

Javascript is a pernicious Herpes-virus-like infection of browsers and desktops, and nothing can ever kill it, so where it fits the problem, there's no reason not to use it. But there's a lot it doesn't do well.

I was leery of using FreePascal because it has a single implementation (technically Delphi still exists, but it's $X,000 per seat on Windows) and minimal libraries, and in fact when it broke on OS X Mojave, I was disappointed but I-told-you-so.

I'm not saying we should quit making new Brainfuck and LOLCODE things, I don't think it's possible for programmers to stop without radical brain surgery. But when you're evaluating a language for a real-world problem, try moving backwards until you find the oldest and most stable thing that works and will continue to work, not piling more crap into a rickety new framework.

The Biblical reference in the title amuses me, because we know now that it requires no malevolent genocidal war deity scared of us invading Heaven to magically confuse our languages and make us work at cross purposes; anyone who can write and think splinters their thought into a unique language and then argues about it.

Eldritch World

I wrote a text adventure:

It's like 15% complete, you can reach the first of four "recursions"/other worlds but then you're stuck. And today it only has a Mac console binary (you can run it from source on other platforms, with a little effort), I'll get on the cross-platform compiling and Terminal wrappers tomorrow, but this is a playable thing under a deadline!

How Fast is My Scheme

I ran a dumb Sieve of Eratosthenes benchmark on a few Schemes. I have previously found idiomatic Swift 3,151x slower than C so surely a GC'd LISP can't do that great, right?

Chicken (compiled) and Chez do great; I would expect Chez to do better on a less numeric example, this kind of array-humping is what Chicken's perfect for, since it just compiles to C. Chicken's interpreter is ridiculous, in dev it's fine for seeing if something works, but you'd never ship that. Racket's sad, and it gets worse if you try to do anything productive. Biwascheme was impossible to even test properly, but at 10% completion it wasn't going to do anything useful.

Also, I want to complain about every Scheme having a different way to call the interpreter as a script, and invoke main with command line arguments. NOT A ONE of these are consistent, and only Chicken does what's reasonable.

# C - 100% C
mdh@Aegura:~/Code/CodeC% time ./primes 1000000 >~/tmp/primes-c.txt
./primes 1000000 > ~/tmp/primes-c.txt  0.05s user 0.00s system 92% cpu 0.054 total

# Chicken interpreter - 1.4% C
mdh@Aegura:~/Code/CodeScheme% time src/primes.scm 1000000 >~/tmp/primes-chicken.txt
src/primes.scm 1000000 > ~/tmp/primes-chicken.txt  3.74s user 0.12s system 99% cpu 3.872 total

# Chicken compiled - 26.2% C
mdh@Aegura:~/Code/CodeScheme% ./build.zsh primes
Compiling primes.scm
Linking...
./build.zsh:40: no matches found: *.import.scm
Built bin/primes
mdh@Aegura:~/Code/CodeScheme% time bin/primes 1000000 >~/tmp/primes-chicken.txt
bin/primes 1000000 > ~/tmp/primes-chicken.txt  0.19s user 0.01s system 98% cpu 0.206 total

# Chez interpreted - 23.8% C
mdh@Aegura:~/Code/CodeSchemeOld/chez% time ./primes.ss 1000000 >~/tmp/primes-chez.txt
./primes.ss 1000000 > ~/tmp/primes-chez.txt  0.20s user 0.03s system 98% cpu 0.227 total

# Chez compiled - 23.9% C
mdh@Aegura:~/Code/CodeSchemeOld/chez% chez-compile.zsh primes
compiling primes.ss with output to primes.so
()
()
mdh@Aegura:~/Code/CodeSchemeOld/chez% time bin/primes 1000000 >~/tmp/primes-chez.txt
bin/primes 1000000 > ~/tmp/primes-chez.txt  0.20s user 0.02s system 97% cpu 0.226 total

# Racket interpreter - 9.6% C
mdh@Aegura:~/Code/CodeRacket% time ./primes.rkt 1000000 >~/tmp/primes-racket.txt
./primes.rkt 1000000 > ~/tmp/primes-racket.txt  0.46s user 0.09s system 99% cpu 0.560 total

# Racket compiled - 12% C
mdh@Aegura:~/Code/CodeRacket% raco exe primes.rkt
mdh@Aegura:~/Code/CodeRacket% time ./primes 1000000 >~/tmp/primes-racket.txt
./primes 1000000 > ~/tmp/primes-racket.txt  0.37s user 0.07s system 99% cpu 0.443 total

# Biwascheme - 0.03% C
# Used stopwatch, took 13.89s for 100,000, couldn't get a result for 1,000,000

Scheme Record to RecordType

So, in "classic" Scheme (up to R5RS), there were no structs/records/classes. You could fake them with a Vector and writing all the accessor methods by hand, but it sucked.

SRFI-9 added a very minimalist record type with a lot of repetition, and no inheritance, though at least SRFI-17 fixed setters.

R6RS (as implemented in Chez Scheme) added a much more powerful system with inheritance and constructor methods, but half the Scheme community hates nice things and voted it down. There's a half-assed reimplementation of R6RS in SRFI-99 and sequels, but it still doesn't have read/write mechanisms. R7RS still only ships with SRFI-9 built in. Unbelievable mess.

Chicken has a convenient define-record macro, and read/write methods, but by default uses SRFI-9, and hides SRFI-99 in an egg; so chicken-install -s srfi-99 and then (import srfi-99) everywhere, and then write a ton of boilerplate for every type. So I just automated it with Python (doing string parsing in Scheme is more annoying):

Documentation is in the module help (or just read the source, Luke). I use it by writing the Chicken macro (define-record Point x y), then at shell:

% pbpaste|schemeRecordToRecordType.py|pbcopy

And paste back:

;; (define-record Point x y)
(define-record-type Point  
    (x)
    (y)
)
(define-reader-ctor 'Point make-Point)
(define-record-printer (Point p out)
    (format out "#,(Point ~S ~S)"
        (Point-x p) (Point-y p) 
))
; module exports:
; make-Point Point? Point-x Point-y

Note that none of this really gets me an "object-oriented" system, but records are sufficient for most programs, and inheritance works with define-record-type. There are OOP systems but I don't especially like any of them so I'm passing on them for now.

Functional Thinking Books

Advent of Code 2018

I don't know how much I'll do this year, but I'll do it in Chicken Scheme as a way of improving the text-processing and math functions in my library, and publish it. Source is now on gitlab, above.

The competitive part is still bullshit, the single starting time of midnight EST is utterly useless to most people who could participate; even in Pacific time, that's late at night. On my quiet night schedule, that's way too early to wake up; in Europe, that's 04:00 or so, long before coffee could percolate. Mid-workday for Korea & Japan. So, only for Finns and Russians?

Inline Documentation, or Lack Thereof in Scheme

I'm a big fan of inline documentation and "light" versions of Literate Programming, because docs that are more than one screen away from code are always wrong. That this is ever a revelation to anyone suggests to me that they've never written code or read API docs.

When I write Python, I write:

>>> import math
>>> def foo(x):
    "Square root of `x`"
    return math.sqrt(x)

>>> foo(5)
2.23606797749979
>>> help(foo)
Help on function foo in module __main__:
foo(x)
    Square root of `x`

Similarly in Java with Javadoc, I write:

/** Square root of {@code x} */
double foo(double x) {
    return Math.sqrt(x);
}

Javadoc isn't usable on live code or in a REPL (Java doesn't really have one), but you get nice HTML docs out of it. Javascript & Node don't have an official tool, but most code is marked up with Javadoc.

Common LISP, archaic pain in the ass though it is, has:

(defun foo (x)
    "Square root of `x`"
    (sqrt x))
> (documentation 'foo 'function)
"Square root of `x`"

Sadly and typically, the Scheme situation is much less organized.

There's a Chicken 4 egg hahn which is ugly, @() special forms and all the nested structures instead of just a string, but it's workable. Otherwise, everything seems to be external docs.

Racket has Scribble with a teeny-tiny side-note that you can put your docs in code, but no examples. There is a literate programming tool as well, but that's not quite what I'm after.

Chez Scheme has no solution, which is a little surprising given the "batteries included" philosophy.

Well, maybe I can get away with doing CLISP-type docstrings and worry about making a tool later? Does this extra junk hurt performance?

(import (chicken time))
(define (sqrt-without-docs x) (sqrt x))
(define (sqrt-with-docs x) "docs" (sqrt x))
(display "without docs\n")
(time {do [(i 0 (add1 i))] [(>= i 1000000)] (sqrt-without-docs i)})
(display "with docs\n")
(time {do [(i 0 (add1 i))] [(>= i 1000000)] (sqrt-with-docs i)})

In the interpreter, there's a 10-50% speed penalty (csi -s or in the REPL) for having that extra string created & GC'd, but compiled (csc), there's no noticeable difference. So I guess that's my solution for now.

Don't Increment Your Chickens Before They're Hatched

It hasn't shown up in MacPorts yet, so I pulled down the tarball and did:

% make PLATFORM=macosx PREFIX=/usr/local/chicken5
…
% sudo make PLATFORM=macosx PREFIX=/usr/local/chicken5 install

Then added to .profile, before the MacPorts parts of my paths:

export CHICKEN5_HOME="/usr/local/chicken5"
export MANPATH="$MANPATH:$CHICKEN5_HOME/share/man"
export PATH="$PATH:$CHICKEN5_HOME/bin"

When I want to switch back to Chicken 4, or when MacPorts gets Chicken 5, I can just comment those lines out and start a new shell.

I run csi, and it errors out because readline isn't installed. And doesn't exist as an egg anymore. Instead, I did

% chicken-install -sudo breadline

And changed my ~/.csirc based on example

(cond-expand
    (chicken-4 (begin
        (import miscmacros)
        (use readline)
        (install-history-file  "/.csi_history")
        (current-input-port (make-readline-port))
        (printf "Chicken 4 READY~%")
    ))
    (chicken-5 (begin
        (import (chicken format))
        (import (chicken process-context))
        (import miscmacros)
        (import (prefix breadline "rl-"))
        (import (prefix breadline-scheme-completion "rl-"))
        (rl-history-file (format "~A/.csi_history" (get-environment-variable "HOME")))
        (rl-completer-word-break-characters-set! "\t\n\"\'`;|()[]{}")
        (rl-completer-set! rl-scheme-completer)
        (rl-basic-quote-characters-set! "\"|")
        (rl-variable-bind! "blink-matching-paren" "on")
        (rl-paren-blink-timeout-set! 200000)
        (current-input-port (rl-make-readline-port))
        (printf "Chicken 5 READY~%")
    ))
)

UGH, semi-random API changes, but it does a little more than raw arrow-key editing now.

sdl2 isn't ported yet, so I can't really progress with that, but I can do other things, and most of my code only needs a cond-expand changing use to import and renaming a bunch of eggs.