Internet Archive Favorites

Part of my workflow with Internet Archive is to favorite things I go back to a lot. But the fav page there is nigh-unusable, it lists in order from most recent fave to oldest, including duplicates (Huh?), and even sorting by title doesn't put related things together. So I made a tool, and generated
Internet Archive Favorites which I'll update every so often.

My first attempt was simply scraping an RSS feed, but they only publish the last 50 faves! Bogus! Even if I cached them, I'd still have to check it often and reorganize things. Then I learned they have a developer interface, usable with an ia script or right from Python, which is more useful. It's slow without caching, but after first run it's very fast, mostly 1 API call.

Read the docs at the top of the script, look at the example config file (almost a Markdown outline, but I do some clever/stupid things in it). As usual license is BSD, an ye harm none, do what thou wilt shall be the whole of the law.

Now all I have to do is write a cfg file:

The stuff I've found that I like on [Internet Archive](https://archive.org), loosely sorted.

## Retrocomputing

+ Basic_Computer_Games_Microcomputer_Edition_1978_Creative_Computing
+ More_BASIC_Computer_Games
+ Basic_Computer_Adventures_1986_MS_Press
+ Best_of_Creative_Computing_Vol_1_1978_Creative_Computing_Press
+ creativecomputing
…

% archive-fav-extract.py -q mdhughes

And it makes a nice html file, tells me about any errors, and I paste the output file into a wordpress page.

Programming on Your Phone

Pythonista lets you use your pocket UNIX workstation as a workstation. I use Pythonista, if not every day, very heavily on the days I use it. As always it's crippling of Apple that there's no upgrade pricing, so I can't give him more money every year that I keep using it. The new keyboard module is an interesting script launcher, but I already wrap a bunch of utilities in a main menu program.

There should really be more of these mobile programming environments. In the early days, Apple severely restricted you from shipping one; you could kind of cheat with JavaScript, and a few games snuck in some bytecode interpreters, but scripting was right out. They loosened up eventually, but are still dicks about you saving code anywhere it could be shared, so for example I have to keep my Pythonista stuff in iCloud, not DropBox where it'd make more sense.

  • Panic's Coda and Coda for iOS (née "Code Editor" WTF) is the only other one that's really functional; I've built real web sites out of it, but I mostly use it for ssh. Sweet baby Cthulhu, I hate Panic's crooked-text "designer" sites, I hit Reader view on those instantly. Designers shouldn't be allowed access to CSS or JS.
  • Hotpaw BASIC still works (as does his Chipmunk BASIC on the Mac), but hasn't been updated in 2 years. Not that I want to program in BASIC, but it's better than no programming at all.
  • The iPad used to have a very nice "BASIC!" (with a structured BASIC and a bunch of system functionality), and a very limited "iSkeme" (scheme interpreter, R5RS-ish? with nothing but text I/O), but they were killed in the 64-bit-pocalypse. Update 2020-09: miSoft Basic! has been updated. Searching for this is utterly impossible!
  • Workflow (née Apple Shortcuts) is great for putting a few tasks in a row but you'd go insane trying to write anything complex from drag-and-drop clicky boxes.
  • Apple's Swift Playgrounds on iPad is a tutorial, not really usable for applications AIUI.
  • There's a bunch of "kids learn to code!" apps that are mostly ripoffs charging $60/year to play robot tanks. Do not buy anything like this.

I dunno if the 'droids have anything comparable, I'm sure they can root their phone and try to use vi in a busybox shell, but that's not a reasonable work environment for a thumb-sized on-screen keyboard.

REXX Primes

Just a quick sanity check on performance.

/** primes.rexx */
/* Created 2019-09-26 */
/* Copyright © 2019 by Mark Damon Hughes. All Rights Reserved. */

PARSE ARG primeCount
IF \ DATATYPE(primeCount, "W") THEN DO
    SAY "Usage: primes.rexx N"
    EXIT 1
END

CALL clearPrimes
CALL sievePrimes
CALL printPrimes
EXIT 0

clearPrimes: PROCEDURE EXPOSE primes. primeCount
    primes. = 1
    primes.0 = primeCount
    primes.1 = 0
RETURN

sievePrimes: PROCEDURE EXPOSE primes. primeCount
    DO i = 2 TO primeCount
        DO j = (i * i) TO primeCount BY i
            primes.j = 0
        END
    END
RETURN

printPrimes: PROCEDURE EXPOSE primes. primeCount
    DO i = 1 TO primeCount
        IF primes.i THEN CALL CHAROUT , i || " "
    END
RETURN
# REXX: 0.8% C
% time rexx primes.rexx 1000000 >~/tmp/primes-rexx.txt
rexx primes.rexx 1000000 > ~/tmp/primes-rexx.txt  6.43s user 0.36s system 99% cpu 6.831 total

# Regina: 1.53% C
% time regina primes.rexx 1000000 >~/tmp/primes-regina.txt
regina primes.rexx 1000000 > ~/tmp/primes-regina.txt  3.25s user 0.26s system 99% cpu 3.521 total

# Python: 4.8% C
% time ./primes.py 1000000 >~/tmp/primes-python.txt
./primes.py 1000000 > ~/tmp/primes-python.txt  0.75s user 0.02s system 68% cpu 1.123 total

# Julia: 1.4% C
% time ./primes.jl 1000000 >~/tmp/primes-julia.txt
./primes.jl 1000000 > ~/tmp/primes-julia.txt  0.45s user 0.35s system 21% cpu 3.797 total

Most of REXX's bad time can be attributed to using stem variables in a tight loop, effectively string-keyed hashtables, so I'm sure an ooRexx Array implementation would be significantly faster. But stems are what you'd use in "real code", so caveat coder. In a long real-world program I don't think it's as big an issue, but it's definitely not received the kind of optimization love that newer languages have. Aesthetically, the REXX source is a little wordier and more explicit about globals access, but not hard to write or read.

I went ahead and grabbed Regina, and it doubles the speed of ooRexx. I'm still wary of it, but that's a big win.

Python's not terrible at anything; it's within a stone's throw of a compiled language at this point. Competent mediocrity has really made Python the new dynamic Java of our time. But you still can't do multithreading in it.

Julia's slow because the startup time is just atrocious; if I timed it internally after startup it'd be as fast as a compiled native program, but as a scripting language Julia's bad news.

(I do have real work to do, but I'll keep playing with REXX more over the next few days)

Interactive Python IDLE

When using Python from a shell, the REPL is fairly awful and doesn't let you copy-paste or save, except through the shell itself. I especially find that copy-pasting functions in is error-prone. There's a nice interactive environment for Python called IDLE (after Eric ): It's probably an application in your /Applications/Python 3.7 folder, or whatever other OS's do, or can be run with IDLE3 from the shell. Other than using ^N/^P for next/previous history line, it works pretty much exactly as you'd expect a GUI REPL to work, and lets you save your session as a text file, easy to extract some functions from later or use as a doctest.

Trouble is, IDLE doesn't automatically pick up the ~/.pystartup script; I had to remember to call it with IDLE3 -s, and there's no easy way to do that from the desktop, where I'm often lazily clicking. This has been frustrating me very slightly for years.

So: open Automator, new Document, add Utilities/Run Shell Script, and paste in:

IDLE3 -s

Save as an Application, name it IdleStart. The icon's ugly and generic, so I made a half-assed icon, copy it from an image editor or Preview, and paste into the icon in Get Info on the application.

python-foot

Now I have a nice stupid foot to click on, and get a proper REPL. The running program has a hideous 16x16 or 32x32 icon scaled up, which I don't think I can easily solve; I looked at the idlelib source and while I could patch it, it's not easily configured. Maybe later. There's also no way to specify the window location, which I'd like to have mid-left of my screen, but again, maybe later.

While I'm at it, the themes are garish, and the customizer is unpleasant. So I just edited this in ~/.idlerc/config-highlight.cfg, then picked Mark's Dark theme:

[Mark's Dark]
normal-foreground = #eeeeee
normal-background = #111111
keyword-foreground = #ff8000
keyword-background = #111111
builtin-foreground = #0099ff
builtin-background = #111111
comment-foreground = #dd0000
comment-background = #111111
string-foreground = #80ffdd
string-background = #111111
definition-foreground = #80ff80
definition-background = #111111
hilite-foreground = #ffffff
hilite-background = #808080
break-foreground = #ffffff
break-background = #808000
hit-foreground = #002040
hit-background = #ffffff
error-foreground = #ffffff
error-background = #cc6666
cursor-foreground = #ffffff
stdout-foreground = #ccddff
stdout-background = #111111
stderr-foreground = #ffbbbb
stderr-background = #111111
console-foreground = #ff4444
console-background = #111111
context-foreground = #ffffff
context-background = #444444

My current ~/.pystartup is short (in python2 it was much longer, since nothing worked right), just some common imports and functions I use enough to miss:

import os, sys
import math
import random as R
import re
import time
import turtle as T

def dice(n, s):
    t = 0
    for i in range(n):
        t += R.randint(1, s)
    return t

def roundup(n):
    return math.floor(n+0.5)

def sign(i):
    if i > 0: return 1
    elif i == 0: return 0
    return -1

print("READY")

Now if I click the foot and see "READY" above >>>, it's all working.

Writing a Doorgame

So, how did I write this? Mystic BBS has a Pascal-based scripting, but I didn't really like using that; it's very powerful, but a pain in the ass to code/compile/test on, and it's just different enough from normal FreePascal to annoy me. And then tried to use Mystic's "integrated Python", which just doesn't work; I tried everything to make it find Python. And it's 2.7, so bleh. BUT: Adding a doorgame with the menu system lets you specify any executable, including a Python 3.7 script, and pass in a few command-line arguments about user and directories, so that's what I did (though the server I'm using only has Python 3.6, which hasn't been a problem yet, but be aware)…

The BBS handles all the redirection from network to stdin/stdout, otherwise I could use inetd for the same thing, but then I'd have to deal with all the login and networking state. Doing this as a door simplifies programming back to 1970s level!

I'll eventually open source it, but while I'm iterating and early on, I want to keep some of my secrets.

ANSI Color

There's ANSI color everywhere, of course. curses is fucking awful and archaic, and built to handle many incompatible terminals, which isn't a problem anymore. Mystic uses the same trick I'm using, and many others back in the day: Strings have escape codes |XX in them. Mystic uses numbers for colors, which is awful, and has hundreds of BBS-specific strings, which I have no need of. So all I did is this:

# set to 1200, 9600, etc. to simulate modem transmission rate
kModemSpeed = 0
kLinesPerScreen = 24

kAnsiEsc = "\x1b["

#      0   1   2   3   4   5   6   7   8   9   10
kAnsiBoxLine =  ["─",   "│",    "┌",    "┬",    "┐",    "├",    "┼",    "┤",    "└",    "┴",    "┘",]
kAnsiBoxDouble =["═",   "║",    "╔",    "╦",    "╗",    "╠",    "╬",    "╣",    "╚",    "╩",    "╝",]
kAnsiBoxHeavy = ["━",   "┃",    "┏",    "┳",    "┓",    "┣",    "╋",    "┫",    "┗",    "┻",    "┛",]

# ansiFilter, called by write/writeln, replaces |XX with escape codes:
kAnsiEscapeCode = {
    "CD": kAnsiEsc+"J",         # Clear Down
    "CL": kAnsiEsc+"2K",            # Clear Line
    "CS": kAnsiEsc+"2J"+kAnsiEsc+"H",   # Clear Screen
    "DI": kAnsiEsc+"2m",            # Dim
    "ES": "\x1b",               # Escape ASCII 27
    "GG": "\x07",               # Bell ASCII 7
    "HR": kAnsiBoxDouble[0] * 79,       # Horizontal Ruler
    "LI": kAnsiEsc+"1m",            # Light/Bright
    "PI": "|",              # Pipe char |
    "RE": kAnsiEsc+"0m",            # Reset Attributes
    "UN": kAnsiEsc+"4m",            # Underscore

    "FK": kAnsiEsc+"30m",           # Foreground Black
    "FR": kAnsiEsc+"31m",           # Foreground Red
    "FG": kAnsiEsc+"32m",           # Foreground Green
    "FY": kAnsiEsc+"33m",           # Foreground Yellow
    "FB": kAnsiEsc+"34m",           # Foreground Blue
    "FM": kAnsiEsc+"35m",           # Foreground Magenta
    "FC": kAnsiEsc+"36m",           # Foreground Cyan
    "FW": kAnsiEsc+"37m",           # Foreground White

    "LK": kAnsiEsc+"1;30m",         # Foreground Light Black
    "LR": kAnsiEsc+"1;31m",         # Foreground Light Red
    "LG": kAnsiEsc+"1;32m",         # Foreground Light Green
    "LY": kAnsiEsc+"1;33m",         # Foreground Light Yellow
    "LB": kAnsiEsc+"1;34m",         # Foreground Light Blue
    "LM": kAnsiEsc+"1;35m",         # Foreground Light Magenta
    "LC": kAnsiEsc+"1;36m",         # Foreground Light Cyan
    "LW": kAnsiEsc+"1;37m",         # Foreground Light White

    "BK": kAnsiEsc+"40m",           # Background Black
    "BR": kAnsiEsc+"41m",           # Background Red
    "BG": kAnsiEsc+"42m",           # Background Green
    "BY": kAnsiEsc+"43m",           # Background Yellow
    "BB": kAnsiEsc+"44m",           # Background Blue
    "BM": kAnsiEsc+"45m",           # Background Magenta
    "BC": kAnsiEsc+"46m",           # Background Cyan
    "BW": kAnsiEsc+"47m",           # Background White
}

def ansiFilter(s):
    """Replaces |XX codes with values defined in `kAnsiEscapeCode`."""
    out = ""
    slen = len(s)
    i = 0
    while i < slen:
        c = s[i]
        i += 1
        if c == "|":
            esc = s[i:i+2]
            i += 2
            out += kAnsiEscapeCode.get(esc, esc)
        else:
            out += c
    return out

def ansiGoto(pt):
    return "%s%s;%sH" % (kAnsiEsc, pt[1], pt[0])

def readln(prompt):
    """Calls input after filtering prompt."""
    return input(ansiFilter(prompt))

def write(s, end=None):
    """Filters `s` and writes to stdout, writes `end` if needed, flushes stdout."""
    if type(s) != str:
        s = str(s)
    lines = 0
    out = ansiFilter(s)
    for c in out:
        sys.stdout.write(c)
        if kModemSpeed:
            time.sleep(1.0/kModemSpeed)
        if c == '\n':
            lines += 1
        if lines >= kLinesPerScreen:
            readln("|RE|FC[MORE]|RE")
            lines = 0
    if end:
        sys.stdout.write(end)
    sys.stdout.flush()

def writeln(s):
    """Calls write with `end='\n'`."""
    write(s, end='\n')

Now if I want someone to talk in bold cyan (which I chose as my standard NPC color), I just
writeln("|RE|LC%s says, \"Mellow greetings, sir!\"|RE" % (npc["name"],))
RE before and after text is important, resetting colors from previous strings.

The one problem is I can't use | in ASCII art, so to generate my big character logos, I write a script mysticFiglet.zsh, making : the vertical char:

figlet -f small "$*"|sed "-es/|/:/g"

Flow Control

The great part about a doorgame is, no event loop. Oh, there's a REPL of sorts:

    def run(self):
        self.running = True
        try:
            #TODO: Python 3.8: while line := self.prompt():
            while self.running:
                line = self.prompt()
                if line == None:
                    break
                words = line.split()
                if len(words) == 0:
                    continue
                self.parseWords(words)
            self.save()
            self.doHelp("?", "_outro")
            return 0
        except (EOFError, KeyboardInterrupt):
            self.save()
            return 0
        except Exception:
            writeln("|GG|RE|BR|FWERROR! Something went wrong.|RE")
            logging.error("", exc_info=True)
            self.save()
            return 1

It just blocks at input (called by readln, called by prompt), and continues on until self.running = False, or an exception is thrown. Whee! The usual game loop makes it impossible to get stateful input without leaving giant memos of current state, so the event loop can keep cycling at 60fps.

Data Files

One solution would be to just make everything in classes, and use Python's pickle to archive everything. But the problem is that's pure binary, you can't easily hack on it or see what's happened, and on load you can't change any values. So instead I only use primitive types and archive everything to and from JSON. Floor maps and Players have a toData() method which produces a JSON-compatible data structure, and their initializers take a data structure and pull out or translate the values they need. It's a little more work, but means I can debug faster and add new fields faster.

The floors use a super-simple data file format, with an ASCII roguelike map and a series of command lines to create NPCs, traps, text, etc. The weapons and armor are listed in an even simpler format, with space-delimited fields.

Both the command lines and item formats are just run thru a tokenizer I wrote, that reads words, "strings", numbers, and [lists]:

type    name        cost    stat    bonus   damage  inflicts    twoh
wpn "Club"      5   Str 0   3   [blunt]     0

becomes:

{'type': 'wpn', 'name': 'Club', 'cost': 5, 'bonus': 0, 'damage': 3, 'inflicts': ['blunt'], 'twoh': 0}

Python 3.7

  • Python 3.7 released: Standard checklist:
    • Run installer
    • Delete the old 3.6 folder from /Applications
    • Run the certificate command in the new 3.7 folder (the other shits a PATH into my shell profile, don't need it)
    • Run IDLE and verify it's 3.7.0. Happily, no longer have to fight with updating Tcl/Tk.
    • Run "python3" from Terminal and verify it's 3.7.0
    • Run a random Python script to make sure nothing's broken.

Nanosecond-accurate time functions and switching more ASCII/C Locale into UTF-8 are nice improvements, but those are more patching up legacy annoyances than "must have".

I'm mostly interested in dataclasses, which makes it much easier to build little struct-type objects instead of random dicts or lists which have all sorts of problems (no equality, hashing, typo-safety).

I greatly dislike the addition of BDSM typing, but it's mostly optional, EXCEPT you have to use them in dataclasses:

from dataclasses import dataclass
@dataclass
class Point:
    x : float = 0.0
    y : float = 0.0

>>> p = Point()
>>> p
Point(x=0.0, y=0.0)
>>> q = Point(1.1, 2.2)
>>> q
Point(x=1.1, y=2.2)

If I define Point without the type annoytations[my new favorite typo!], only the default constructor works, and it doesn't print the fields as a string.

@dataclass
class Pointless:
    x = 0.0
    y = 0.0

>>> f = Pointless()
>>> f
Pointless()
>>> f.x
0.0
>>> f.y
0.0

Real examples might be a lot more complex than a point, and by then the cost of building a proper class with __init__ and everything yourself isn't such a big deal, so I can see dataclasses mostly being used for very simple struct-like containers.