Þ   briarpig  » log  » nov07

booked ¶

     I'm too busy to write: need to get up early and code more. I came this close to skipping an update. I won't return here until this weekend. However, I can always say something.

profession ¶

     For example, though I have a CS degree, I seldom think of myself as a computer scientist. Sometimes I think of myself as an applied mathematician, but I'm hard put to decide what kind of math is involved. In an earlier age that's what I'd have been, and I seem to do a lot of thinking with a kind of mathematical style to it. Is it math? Dunno, I don't think so. That's one of those category things, and it seems bullshit to me.

     Computer scientists have a bad reputation some places as inept, inexperienced, lack luster programmers — due to insufficient practice. However, degree or not, ten years of intense practice is very good for competence. Even better is twenty-five years of intense practice, in addition to a degree. You might have trouble finding theory under the common sense, but it's there.

echoes ¶

     Is it synchronicity when you seem to hear echoes? It's easy to find a thing related to what you've written, supposing a connection, when it's likely just coincidence. Yeah, that's it.

vanilla C ¶

     In my day job, I'm writing a lot of code in C. I haven't done much C the last ten years. I'm writing somewhat strange looking C, though. It's very object oriented, and I don't seem to have trouble thinking as abstractly as I do in C++. I simply indirect through function pointers in places I need virtual methods.

     I'm using an entertaining trick at the moment. To avoid hard linking system calls, I'm defining a vtable with function pointers to methods outside myself, so they can be plugged in as desired by the library user. Anything unused needn't be defined.

holiday weekend ¶

     I wrote a lot over the holiday weekend, but I'm only posting it today. Knock yourself out reading new pages. They're purposely brief, with concinnity a goal. Maybe I have enough of that out of the way now, and I can go back to coding at home. It will be interesting to get basic async dispatch support into Lathe fundamentals; then I'll be able to prototype HyperCard mechanisms.

halting state ¶

     I'm reading Charles Stross's Halting State, which is where today's juggling headline occurs. That phrase actually occurs in the book, so I couldn't let it pass.

aelig ¶

     I added new top level page aelig today, which describes the async event system I've been mentioning the last month, but only in very broad terms. (Also see companion page callstacks†, composed mainly of Wikipedia quotes on conventional runtime stack practice.) This æ subset of þ will support async flow of control in Lathe's basic runtime with as much direct native support as normal method dispatch for Scheme and Smalltalk style code.

     The async event system I write for my day job is not described now, and won't be later. Details on that particular async system version will never be aired publically, since the gnarly details are clearly proprietary. In particular, the rocket science parts of shared memory and atomic lockless coding style won't appear in any þ version of æ until a far future time (years) when a second system is due.

     Main parts in common to all my async event systems will be basic stories describing theory of operation, whose purpose is to justify rational behavior of refcounted spaghetti stacks (as per callstacks†), and to outline flow of control mechanics with callbacks and queue polling.

discrete simulation ¶

     While reading ycombinator news linking Scientists should learn to program, I ran across another of Eugene Wallingford's blog entrys on 16nov2007; here I'm quoting a side comment unrelated to the overall defense of oo practices (exceptions and garbage collection):

Bruce Sherwood said that many scientists view analytical solution as privileged over simulation, because it is exact. He then pointed out that in some domains the situation is being turned on its head: a faithful discrete simulation is a more real depiction of the world than the closed-form analytical solution -- which is, in fact, only an approximation created at a time when our tools were more limited. The best quote of this session came from John Zelle: Continuity is a hack!

     This is related to my discussion with E about asynchrony a few days ago: computing systems are actually best described as collections of async activities. But folks studying them prefer closed analytical descriptions to non-deterministic event-based simulations, which might resemble the system studied more than analytical solutions.

     How does this relate to my async event designs? I started my event designs in 1988, when a professor in school (cf simulation-engine†) asked me to redesign a Discrete event simulation system written in Fortran. I wrote a modern (but inefficient) equivalent using a mechanism I'd learned in operating systems class: send and receive mailbox queues. Send and receive primitives can be used as fully general synchronization mechanisms equivalent to semaphores or monitors. And for purposes of discrete event simulation, you can add (simulated) propagation delays to messge passing that model an actual system's signal behavior.

     Notice the word mailbox implies a possible relation to email systems, which does indeed explain why I joined the Netscape mail/news client team in 1997. I wondered whether one could find a useful point of common leverage between event handling and email processing; but the answer is: not really. There's a simple homomorphism between email mailbox and event queue processing — it's useful to illustrate event queue processing with email example stories — but otherwise there's little relation. Except you can use email as physical transport for one leg of event propagation if you wish.

     Um, let me see, what's my point? Oh yes, my point was: computing systems are best modeled by sending messages between components, with steps progressing in discrete units marked by arrival of a message event. This is like quantum mechanics for computing: forward progress ratchets in digital steps, and not continuous wave forms. Until you receive a message, you don't know what's going on. And you only hold a local viewpoint, based on messages received.

     Let's add a permalink here for mailbox ports:

mailbox ports ¶

     I tried searching for a web page describing use of mailbox ports to send and receive event messages, and I had a heck of a time finding anything not inside PDF file representing a larger academic work. I finally ran across the following material in Chapter 4 of McFadyen's Cis450 class at UMich:

     Source: http://www.engin.umd.umich.edu/CIS/course.des/ cis450/mcfadyen/chapter4.htm:

• With indirect communication, the messages are sent to and received from mailboxes (also referred to as ports)
• A mailbox can be viewed abstractly as an object into which messages can be removed.
• Each mailbox has a unique identification
• Two processes can communicate only if the processes have a shared mailbox
• The send and receive primitives are defined as followed
  • send(A,message). Send a message to mailbox A
  • receive(A,message). Receive a message from mailbox A

     I cached a local copy of that page for future reference, in case it goes offline. If you read that entire page, and think about processes in such languages like Erlang, which communicate by sending messages to each other using process IDs, you might see obvious applications of event queues and process management.

     Note this page isn't particularly better than others I've seen. It was handy. For contrast, compare to this Yale wiki page on IPC; locally: yale-ipc. Oh, here's another good 1998 page from Caltech Infospheres; locally: djinn-ipc. It caught my attention by using the fun term djinn. You might want to compare that with Xian Ming Song's 1997 RTOS Case Study: OS68 for Embedded Systems; locally: os68-ipc.

     A "send receive ipc message queues" search yields many hits on POSIX message queues, most of a highly API detail centric perspective, rather than genera explanation. So none are cited here as exemplary.

     Okay, I kept looking for more good stuff, and found Luke Gorrie's 05jan2003 diary entry at Advogato discussing exactly the right thing:

The unit of concurrency in Erlang is the process, which is the central idea in the language. An Erlang process is more like a Unix process than a thread, but is very lightweight -- creating a process takes only a few microseconds and a few hundred bytes of memory. Erlang processes share no mutable memory or variables and only communicate explicitly, using send and receive primitives. When a process receives messages, it specifies which types of message are acceptable, causing others to be queued in a mailbox -- this gives a very beautiful synchronization-through-communication programming style from CSP.

juggling notation ¶

     Yesterday I started a sequence of items on juggling so I could talk about juggling notation, to illustrate the idea some relatively simple things are hard to represent in textual notation. Here's a quote from that wiki page:

It has often been said, of many juggling patterns, that it is "easier done than said", while it might be easy to learn a given maneuver and demonstrate it for others, it is often much harder to communicate the idea accurately using speech or plain text.

     Another similar sentiment is expressed on Wikipedia's Juggler page. But this rhetoric emphasizes notation helps when everyday language otherwise helps little in conveying complex ideas, or spatial ideas involving motion:

Juggling tricks and patterns can become very complex, and hence can be very difficult to communicate using everyday language. To get around this problem, various notation systems have been developed for communication of existing patterns, as well as for investigating and discovering new patterns.

     Some of the diagrams, like this one from Wikipedia's Siteswap page, (shown

     In effect, this sort of notation introduces bureaucratic artifacts (like graph layout and annotation) that don't exist in a juggler's mental model. The same thing applies to graphical notations like UML used to express relations in coding systems. It brings bureaucratic baggage.

     Bureaucracy often looks like it brings value to the table. But when it does, it often brings as much cost as value. And those costs can then impose independent demands unrelated to the original problem. This effect of bureaucracy is endemic to software tools in general: often much code does nothing but serve demands of a towering bureaucratic ecology.

     Which brings us to ecology reduction:

ecology reduction ¶

     Part of my normal software development process involves ongoing ecology reduction, where I try to replace complex, bureaucratic ecologies with simpler ones. Who would argue against elimination of bureaucracy, anyway?

     Almost everyone, in practice. Bureaucracy is always called "standards" which carry a lot of social proof inertia. That's an understatement: by a lot of inertia, you should read immovable object. There's a tradeoff dynamic: costs and benefits come in bundles that should be counted together. (That's my year of college level accounting at age 20 intruding here.)

     If you change things to make them simpler, this often draws a permanent standing crowd of hecklers. It's a cost of simplification: boo's from the peanut gallery.

     Okay, that's enough "no good deed goes unpunished" caveats. Now let's look at bright sides.

     If you're willing to replace large standard ecologies with minimal means accomplishing your needs, cost savings can be enormous. Size and complexity reduction can be so great, you can't recognize an original in a new form. In fact, only using them for the same purpose makes them related.

     It's hard to reduce memory management means below a minimal threshold. After you strip away all middle men, you're left with gc or refcounting for complex cases. A small amount of bureaucracy remains, and you wish it could be removed without causing chaos. Is it necessary?

     Paying for refcounting bureaucracy can lose a lot of other middle men hard to dodge otherwise. Something has to cleanup resources; auditing keeps it honest.

nonsense ¶

     Today all material below is fun nonsense, clarifying none of my programming work. This sort of thing is only useful to show texture of life: it provides a little context showing what else is happening at home besides coding. (Nothing much: I hang out with my sons and read fiction; all low profile.)

     Holidays encourage retrospective views. I saw a couple late night documentaries on John F. Kennedy and relatives, which had about the same effect on me as watching soap operas does on most folks: seeing misery hit the rich and famous helps make brass rings look vacuous. Also, grand stories look a little thread bare when you dig deeply.

viral tunes ¶

     Both my sons sang odd versions of holiday music on Thanksgiving. The eldest kept singing Ding Fries Are Done (Youtube) — sung to the tune of Carol of the Bells. Warning: repeated listening might unhinge you.

     The youngest son had a weird version of phrase "This is Christmas, this is Christmas" from a song in The Nightmare Before Christmas. But it's a loopy giggle sounding like "HEH heh, HEH heh" repeated several times. It's funny and disturbing at the same time. Obviously he has to duel with his older brother's ding fries are done.

fenrir ¶

     Ironically (given this site's name and associated anti wolf mythology) my youngest son likes wolves more than just enough to make me a paper maché wolf last father's day. I noticed one of his usernames in a game was Fenrir, so I asked whether he knew what that meant. He explained it was the wolf from Norse mythology. Yep, that's right.

     I had already noticed he named one of his online villages Wolves in yet another game. I thought it might be a school mascot. Looks like more of a personal mascot. (Neither son knows about this site, so I didn't cue the interest.)

     Now the interesting thing: Briar Pig is loosely modeled on Brer Rabbit, who is a trickster character in the mold of tricksters like Loki in Norse mythology. And Loki was the father of Fenrir. Is that weird, or what? Not that I identify with Loki. The only trick I want: to avoid becoming food.

juggling metaphors ¶

     I've been able to juggle five balls in a shower, cascade, and reverse cascade since I was 20 years old. I'm so good at five that professional jugglers have asked me: how's my seven? (Nowhere yet: I still might learn someday.) I taught myself how to juggle from scratch, without watching anyone or instruction from anyone else. I figured it out. I do three balls in one hand beautifully. (Shower or reverse cascade with three balls, in one hand only.) I started at age 15.

     How did I learn? I was merely interested at age 15, and I had fabulous hand-eye coordination. Bad reflexes, but really good quick twitch muscle control. One day I went outside with tennis balls and said, "This can't be too hard to figure out." That evening, after I had three ball cascade down cold in two hands, I showed my father.

     He chuckled and asked me, "What's the trick?"

     Puzzled, I offered, "What? I'm juggling three balls."

     "Sure, sure," he demurred, "How does the trick go?"

     After a couple more minutes of this, I finally understood his viewpoint — he thought my tricking him to think I could juggle was more likely than my actually juggling. In other words, he could see me juggling, but he didn't believe it.

     I had trouble seeing why he might think such a thing. What cause had I ever given to doubt what I said? He thought me smart enough to simulate rather than do; kinda odd.

clown, magician, or juggler ¶

     I wrote the story above after I saw the quote below in a Wikipedia entry describing a Mill's Mess three ball juggling pattern, because I saw parallels in software, where you might also say we have clowns, magicians, and jugglers:

Unlike clowns, or magicians, jugglers do not typically feel a need to claim ownership of their various routines and tricks, and borrowing tricks and combinations is common practice, with skills being layered on top of each other to produce new tricks. However, it is somewhat traditional for a juggler to attribute the origin or inspiration for 'new' patterns and tricks.

     In software I'm a juggler according to this typology: I don't feel the need to claim ownership of routines and tricks. Also, I avoid claims to magic, and I try not to clown.

     Here's a local copy of the 5-ball cascade animation from Wikipedia's cascade page. It shows my favorite pattern to practice. But I can make the height rise and fall, down to 18 inches or up to several feet. I can also site swap to reverse cascade with five, so direction is reversed (upward on the outside instead of the inside).

palo alto lunch ¶

     The day before Thanksgiving I had lunch with E in Palo Alto. It was our first meeting after some years of online interaction. It was nice to finally put a face and voice to a name. E's name is one letter here because I dislike name dropping. (This is a little like the way names of Men in Black agents are reduced to one letter. Agent C already appears here elsewhere. My own name is R. That doesn't help much, does it?)

     We talked about a lot of things — but not as much about programming languages as you'd think. But I want to note one vein of discussion on information hiding and asynchrony. I'll hit those in sections below.

asynchrony ¶

     E and I only talked about a few technical ideas. Of the two, I might say less about asynchrony, so I'll treat this one first. The common thread in both discussions was the difficulty other folks have in grasping things with ambiguity and non-determinism woven into them. I think we were pondering the psychology in others making such ideas hard to use.

     My point, which E seconded, was that most people won't or can't think about multiple different viewpoints being true at the same time. It's easier to have one globally consistent universal viewpoint. Non-determinism doesn't fit well in this picture. Asynchronous code permits more than one flow of control to occur either simultaneously or interleaved in an unspecified order. To think about this very usefully, a coder must conceive code flows as somewhat independent and unpredictable. But reasoning makes folks want to nail down every part.

     Perhaps more importantly, the value of independent code flows is in the ability to make them parallel as much as possible, which only works better the less one code flow knows about another. To much inter-flow knowledge can cause sync and serialization. So managed ignorance is an essential aspect of both asynchrony and information hiding (see below). Independence in code flows enables productive behavior: parallelism, which is good for multicores.

     Many coders long for a clean, deterministic model of code to grasp what happens, but that way of knowing a system can cause a design with too few degrees of freedom (which I've mentioned before), where freedom is what makes code flows able to proceed independently. Knowing less frees code.

information hiding ¶

     E and I also talked about the use of abstraction in systems, especially with layers that hide knowledge on one side of a boundary from the other side. A lot of practitioners resist the use of information hiding. It's often treated as a temporary boot-strapping phase when not everything is known. Once all details are worked out, folks feel free to deeply lockstep all pieces, causing interdependency (big balls of mud).

     E said he'd had folks ask, "Why do I want to hide things from my colleagues?" Information hiding was seen as deprivation of intelligence. A goal to free one's colleagues from needing to know something was seldom apparent. I noted the main use of information hiding was to limit the scope of how far one must look when following chains of relationships. Once you pass through a boundary hiding the details, you should no longer need to know.

     Information hiding makes things less complex, because a graph of "who knows what?" in code is less involved. Hiding is usually unidirectional across a boundary or layer, so one side knows and another side doesn't. In other words, the typical purpose of a layer is to create ignorance on one side.

     But a coder can see both sides, and it's disturbing to have some fact F true in one place, but not true everywhere universally. Many coders don't like the non-uniformity of fact visibility, even when it simplifies reasoning across boundaries. To make matters worse, we often wear even more hats when doing global optimization, where we permit ourselves to see across all boundaries, even those that hide information.

     A system coder doing global optimization can look through all boundaries to line things up more than is guaranteed by interfaces. In fact, it's because interfaces promise less than they might that allows choices in global optimization. And when analyzing interaction of components, it's partial ignorance in pieces that makes them easier to understand.

christopher alexander ¶

     This story is here only because I mention design patterns in the next section of this blog. For reference, you might want to see a recent item I wrote on graphical language, since it's directly related. (That was 1981 — in 1982 I moved to California and lived in Richmond for a year with my fiancé, as we both burned a year to establish in-state residency, because we were almost too poor to eat.)

     In 1983, I moved to Berkeley because of Christopher Alexander, because I'd read a couple of his books and because I was interested in architecture — Berkeley had an undergraduate program in architecture.

     Alexander's Notes on the Synthesis of Form was the book that got my attention. (Hey, I'm surprised to see now on Wikipedia this book was recommended to students and colleagues by Marvin Minsky, and had an influence on programming language design in the 60's and 70's. I'll be darned — I never knew that before.)

     During my 1981 research on spatially oriented languages, one avenue of reasoning I pursued was analyzing the concept of form specifically, and I was deeply involved in contrasting ideas of analysis and synthesis in my plans, because it was clear synthesis was easier for me than analysis. That seemed central to what I did that others didn't; I'm very right-brained.

     (This reminds me of another, much stranger book named Laws of Form by George Spencer-Brown. That was my first introduction to boundary logic, which still regularly informs my thoughts on system analysis: passage through interface boundaries is often a trigger for transformations, as well as auditing for containment.)

     Anyway, Alexander's book title had both the magic words in it, and the content was also related, though it seemed in some odd direction. (Don't expect a clear book report now after 30 years, okay?) I read more of Alexander's works, and ended up interested more in architecture than computers, and planned to pursue work as an architect instead of a programmer.

     My fiancé wanted to study art history at Berkeley in grad school, and I wanted to study architecture, so that's how we ended up in Berkeley together with pooled resources. She was accepted there in grad school, and I wasn't accepted in undergrad school, leaving me stuck with a choice of leaving her to pursue my own goals, or staying to spin my wheels. I stayed and resumed (solitary) studies of computing.

     Let's get back to my Christopher Alexander story. I met Alexander in Berkeley around 1985 or so when he came into Radston's Office Supply in Berkeley, where I worked as the manager/owner's number one. I think I sold him a file cabinet. I met a lot of famous folks there in years 1983 through 1985, and often only recognized them by name when given a credit card. Alexander paid by credit card.

     I wanted to — but didn't — tell him I was only there in Berkeley to sell him furniture because of a book he'd written. Instead I only told him I was a big fan ... which didn't surprise him at all, since by then he was used to groupies of all stripes. My story had a little too much bathos to it for sharing.

     Now fast forward several years to circa 1992 at Taligent, where engineers were starting to contract a viral meme called design patterns. It was the latest rage as koolaid you had to drink, or else be condemned to an out-group of unbelievers. One of it's advocates was a designer named Robin.

     When she started reciting the design patterns gospel according to Alexander, I scratched my nose and said I'd actually read about design patterns some years ago, and frankly ... I didn't see how it contributed much to software design — at least in some way that helped one process or think differently than one would otherwise.

     Of course that was instant grounds for lesè majesté (cf Lese_majesty), and provided me with my first experience in learning to watch my mouth speaking about tech beliefs. I think the problem is: most people are bad at thinking in patterns and generalities, so they need some creed like design patterns to ever stray from thinking in concrete details only. In other words, to generalize and use abstraction, some folks need permission from religion.

     I'm good at thinking in patterns, which makes most rhetoric in this vein sound childish to me (which is still lesè majesté even today). Somehow I think design patterns are the way folks good at words try to think some other way. It's the explanation of how not to think in words when spatial thinking is better. But thinking spatially is my native mode.

crappy programmers ¶

     Damen Katz wrote a longish piece on crappy programmers who don't know they are crappy, listing a dozen signs with explanations. All my bullet below are Damen's, but each one is followed by how I do things.

     The consistent pattern here: I don't match the bulleted points. Often I'm the opposite. But I'm not sure this is the definitive criteria for being a crappy coder. I think folks are crappy coders when they don't figure out what they're doing, and rely on some crutch to replace understanding.

     I've actually never used Java. Professionally I've mostly used C++, with a little C and Python. I've used Lisp often as a scripting language with foreign function wrappers to my C++ code. (I wrote the Lisp and glue code interfaces.) I like variety in languages, but my problem is I don't like to use closed tools made by others.

     I use "enterprise" rarely, when it's one of the terms my team uses to describe a market. In the contexts where it applied to what I was doing, relevant required attributes included: 1) not too complex, 2) scalability, 3) easy admin, 4) logging for some accountability. Some folks think it means "important", but it really just means "field rugged."

     I impose few constraints, but I prefer one function fit on a screenful of display when possible. I'm apt to break functions in pieces to make them fit on a screen. Exceptions are usually large numbers or cases in giant switch statements.

     I pay absolutely no attention to patterns. I use the word "singleton" to refer to objects instantiated only one time by convention. I haven't seen anyone use patterns productively. Patterns seem drivel, but I keep that to myself. I describe abstractions without using patterns; works for me.

     What I do is best described as network "signal processing" which does need cycle optimization. But while cycles are precious, it affects little of my style except a very high priority for constant time operations, so data structures can scale into the millions. (Yes, I really do make data structures with 50 million members in RAM.)

     I do this without discussion, so it appears in neither my style nor my code descriptions, except I tell folks all the time most code is constant time. Programming language is dictated to me. (I'm arranging I can use Lisp or Smalltalk one day in control planes — at home — for parts where cycles count less.)

     In my system I write databases, i/o, and signal processing — all three — and part of the goal is to spend cycles to reduce cost of i/o and databases. It's trivial for me to change the system from cpu to i/o bound and back again, since I write the cpu intensive code suppressing more i/o. Obviously the trick is getting a balance right, and I can't spend cycles more than once, so I need to make the first time count.

     Think of it as an app kernel replacing parts of the OS kernel, because the app has enough knowledge to make better choices than one-size-fits-all Linux strategies.

     Number of return points does not matter at all, but I prefer to see returns (ie, not implicit via exceptions). In C++ I use RAII style with destructors to release resources specifically so unpredictable exit points are not a problem.

     Lately my users are smart enough to hunt me down if I ever make a mistake, and I don't want them to ever catch me making one. Disrespect for others is a chronic problem in tech geeks; they assume others are dumb to boost egos, seeing a world with themselves at pyramid peaks.

     I can and often do write a high volume of code, but it's a problem that it creates more code than others have time to read and maintain. That's a hard nut to crack. A less verbose language would help. Occasionally I write very subtle single lines of code needing a comment or three to explain consequences should it change to something similar but wrong.

     I copy and paste little these days, putting common code in single methods when I can. Cut and paste happens most when I write an abstract interface, and then clone new subclass implementations while empirically searching for better tested behavior. Cloning is usually by cut and paste.

     No I think it means "throw no exceptions at all", and normally die by assertion on conditions that should be impossible (but occur anyway). Logging is great — I emit full backtraces as well. It's best to make normal "errors" acceptable parts of state machines, to be handled except when they imply memory corruption, which is often best handled by dieing after logging. (Memory corruption is intolerable.)

     In debug code it's best to always die on any error of any kind, so you are forced to track down and remove any seen. Violent rejection of every condition you dislike will keep you more honest, and will show what a system actually does. If a condition can be survived, the way code dies (asserts) can be softened to a form that throws an exception when deployed, if it's possible to recover from the exception.

     I never model in UML. It says very little, and clarifies almost nothing better than a small amount of clear English prose. (I rarely read UML diagrams, but I peek if given no other docs.)

     I write APIs before coding, and comment parts that must make sense to someone else. (If I'm not the only consumer, then I comment more completely.) I try to make comments one line only in length for each significant point.

     Sometimes I draw monospace ascii text diagrams of memory layouts, when otherwise meaning of code is hard to grasp. They go in sources right next to the code. Often such diagrams act as informal proofs of address arithmetic.

     (You would think someone interested in graphical languages might like UML, but I find it trivial. Maybe it conveys more to folks who think less visually without props.)

     I haven't lost any user data yet as a result of my own errors. It's often been my job to ensure user data cannot be lost. (I've been slagged for designs reducing risk of user data loss while looking obtuse in another manner.) I've written database code that refused to delete files, to ensure a human being makes conscious decisions to erase data first.

     Years ago I had a coworker who lost log files worth $60K of billing to clients (they said afterward). Logs were processed by a cron job assuming (a scary word) no other consumers competed for data, and that erasure was okay post consumption. It ran so slowly it ran concurrently with another instance of itself, and they stepped on each other.

     Part of the solution was moving processed data to another location, so code or human beings could audit stages before whacking irreplaceable original data.

hypercard books ¶

     Two or three years ago I bought several (say seven or nine) different HyperCard and HyperTalk books from a Mountain View used bookstore, all priced very reasonably with essentially no market demand whatsoever.

     An owner was at the register when I took my purchases to the cashier. He said he'd wondered whether anyone would ever buy those books. It was like he'd finally won a bet with himself they'd been worth stocking in the first place, but the time carried as inventory was too high. I had the impression the books had been on the shelves for years.

     So naturally he wondered what I was up to. He had thought HyperCard was interesting, and wanted to know whether it was being released again, or something. Was I going to do some HyperCard programming?

     I said I thought I might implement HyperCard myself, and wanted books as a partial spec of needed features.

     He stared and stifled his instant candid doubt. What I just said sounded as silly as claiming I knew how to forge alien artifacts able to signal visitors in UFOs. He lived in Silicon Valley, and yet didn't believe he ever met folks who actually made things (just because they wanted to make them).

     So he finished the rest of the transaction quickly, once I'd shown I was a crank. No more friendly chit chat.

     Anyway, so I have these books, and I dug them out yesterday. I thought I might start writing APIs in Lisp syntax that might express things I wanted to be possible. That would let me see just how much I was leaving out of the original. And it would cause me to form ideas about how objects should expose interfaces in event driven runtimes.

     I'd find value in floating possible features I want, just to look for contradictions in whatever bottom-up mechanisms I first aim to code. So the interesting part would finding things I don't want to support, because this would function as negative feedback and criticism, which is better than the nothing I'd get in a vacuum of other folks also making HyperCard.

     In other words, it's a way to work through concrete use cases while considering things in other dimensions I want, like versioning in module systems. Looking for conflict is instructive, and plotting consequent problems of design choices will sharpen maps of good and bad ideas. It's a bit like taking stock of where degrees of freedom are missing, so I develop an intuition for what cannot work, to avoid pursuing it.

     (I've had several conversations with C in which we both agree design is partly thinking of as many possibilities as we can while finding and rejecting the bad ideas and ugly constructs. In other words, brute force search in a space with evolution via survival of fittest ideas is an essential part of good design technique: 1) have many ideas, 2) reject bad ones.)

crazyontap ¶

     I see from my hit logs someone copied a blog entry to crazyontap and folks mistook someone named "wild ideas" for me. I didn't post — if I had, I'd have signed myself "briarpig" — but almost anyone else could do that as well. You can tell it's me if you see a response on this site, right?

     Someone named Aaron said they'd like to know more about my recent brilliance-bane entry. It's too late to write more about it tonight, but I could expand that story later. Be warned the more I say about what I was thinking at age 22, the stranger it might sound. (Is that Aaron Swartz? I'd planned to post some treedragon material on Aaron to the archive soon.)

     It'd be easier to write about something if you send email I can publish. Just don't make the topic about me.

graphical language ¶

     When I was 21 or so, I started thinking about a graphical language, or spatial language, that was composed of graphemes instead of lexemes. I was exploring an idea that some kinds of reasoning were spatial in nature and could benefit from a more directly representative encoded form, to foster organized spatial thinking without words.

     (This was before the Mac, and before I'd ever seen a graphical user interface, or any notation to express information graphically in a manner more complex than flow charts. Some current technology we have today achieves a fraction of what I was thinking: to use graphical displays as both palette for expressive terms and space for display helping rationcination. But my direction was quite different and much more analytical. I was aiming for a tool that was half math and half speech better at teasing value out of ambiguity.)

     Within a year I was spending all my time thinking about it or related research I found in libraries while trawling for anything I could see was related. I filled notebooks with notes and ideas which aren't easily summarized — the first paragraph above doesn't cover the territory. I think I still have those notebooks, but I don't like to read them because the ideas are distracting, and use categories and taxonomies inspiring lines of thought irrelevant to my current goals.

     I was on to something interesting and original, but I came to think it wasn't very useful as a plan yielding practical real world results. But I developed a idiosyncratic toolkit of reasoning tools — rules of thumb for breaking up information and selecting parts relevant to loosely defined goals arranged in non-linear space.

     But today I've almost no reason to talk about ideas I formed which later proved useful in daily work, which involves a lot of spatial relationships in runtimes of computing systems. (The ideas aren't easy to understand or use unless you think spatially and cleverly. And frankly they put people off: linear verbal descriptions translate poorly and sound fancy if they make sense at all.)

     I might write a long web page or two someday about those ideas from almost thirty years ago. But today it would just distract from what I'm doing now on this site with programming languages, async events, and frameworks for scripting tiny servers and user programmable web apps. This site's about practical and useful things; my decades old weird ideas would win an attention contest.

scheme and event driven code ¶

     A recent update at scheme-punks (the ERR5RS alternative to Scheme R6RS) by David Rush adds commentary about async event-driven programming on the K1:Events wiki page:

David Rush: For years it has been a pet peeve of mine that nearly all programming languages and models are based entirely on synchronous and logics, when the real-world of computer hardware (and even the messy analog reality in which we live) is very fundamentally event-driven. I am thoroughly aware that a collection of communicating sequential processes (commonly referred to these days as threads) has a certain equivalence to the old-school world of Interrupt Service Routines, but over the years it has become clear that event-driven models enjoy an expressive advantage in many problem domains.

     I was feeling the same way, which is why I've been planning a more event-driven approach to Scheme-like code in Lathe. My async event daydreaming and design the last month aims to put async events in a basic programming language runtime of a Lisp/Smalltalk hybrid, so event handlers can be declared first-class in code modules in as primitive a fashion as classes and methods.

Rush: Scheme is in a unique position with respect to asynchronous programming in that it contains an operator call-with-current-continuation which reifies the control context. This is so similar to what happens in the beginning of an operating system's interrupt handlers that it would seem that Scheme is a natural semantic system for expressing asynchronous and event-driven programming styles. Any asynchronous event handler can be called with the continuation of the expression which has been interrupted due to external circumstances.

     Yes. That was one motivation for resurrecting my old stuff in my current Lathe plan. Doing it all in C++ in my day job was weirdly awkward (with baroque waste obvious to me, if not to many others at my day job).

     Continuations make resumption of control flow very natural in Scheme when an async event occurs. All you need to add is first class control over the async events themselves, also in Scheme, which is what I'm trying to do now.

     Except I'm building async events into the runtime underneath, instead of in Scheme, so the same async event system will be native to C and C++ in the same process or other processes. This will make a "Scheme" system using such async events in Scheme have native interaction with the operating system at large. That would make a Lisp layer (like Lathe) just a control plane in control of a very efficient data plane using async events, possibly with time critical parts written in C or C++.

event frameworks ¶

     I'm almost ready to code again at home, since my plans for async events no longer change a lot from day to day, so I won't write as much throw-away code. I might be able to write some unit tests for async events I don't have time to write at work, since I'll be racing the clock to complete features at the head of the critical paths everyone depends upon.

     But at work I'm writing in C now, and at home I just as soon use C++ (and comically short name schemes), so any code I write at home will need major source code massaging to use as unit tests at work. It's likely worth the effort though, since most of the time to write tests is deciding what to do and debugging what you see at runtime.

     Since I don't have a life, I work on tech stuff instead, since I'll eventually get something really interesting to happen on that front. (And maybe in a few years when that's done, I might be out of debt to the first witch, and I can afford to have a relationship again.) I can get myself to test a C++ version of single process async events, if it's related to my programming language efforts. Because tests would have value at work if I transliterate this to my day job framework, I think I can trick myself into doing tests first, instead of scratching my itch.

charisma ¶

     I'm rarely charismatic, and never so in group settings. I function best in conversations between two or three people. I've no talent at all for meetings where the game is to score the most points saying the obvious, squeezing words in edgewise when chance permits. When folks become aggressive in groups, I lack the skills to joke them back into line.

     Watching C manage review sessions is enlightening since he laughs so often in response to aggressive behavior, in an inclusive way making everyone pleased to be inside the group. He always knows exactly what is meant by pressure, and quickly says the right things in situations where I'd lose folks who object. I know it's not a skill I can learn, but I can admire it.

     C and I are near visual opposites since I'm clean shaven with a short Forrest Gump style haircut and nearly no gray, while C's long queue of braided gray hair down his back with a matching white beard says: ultimate Unix geek. I look like a gym rat who says, "Life is like a box of scorpions." (It always stings.)

     At dinner with folks visiting from Belfast last week, C mentioned he hadn't cut his hair in 25 years, when folks asked me about my habit of cutting my own hair. Someone asked how old C was, and to our surprise we learned C was younger than me by a year or two — we look years apart in age the other way. I was comfortable with C acting the senior, but since he's been treating me more equal despite my unassuming ways.

     C's the most enjoyable colleague I've had in many years. I count myself very lucky he joined us with other Cisco folks during the last year. Being around really smart people is always great, especially when it's combined with genial temperament and human spirit, without the acid edge of tech competition.

shared memory ¶

     Someday I'll go back to my purpose here on this site, but now I'm more then full time thinking about very novel problems at work. Lately I'm doing something rather weird and entertaining: C asked me Monday to put all the async event engine in shared memory, including things I meant to be single process.

     (I'm discovering exotically interesting things just thinking from first principles, but I'm not going to describe them here, since there's no reason to reveal insights giving a competitive advantage. You're welcome to the problem — just not my solutions. For example, if handles for refcounted objects are also in shared memory, and aren't in threadsafe objects because there's no pthread mutex, then how are you going to resolve races between adding a new ref and losing the last ref to a counted object from a shared handle? Or if you have event callbacks, how are you going to avoid using a pointer to code from one process in another process? Hmm. Where do you need relative pointers when shared memory gets mapped to different addresses? Hmm.)

     It's a fun problem because it broke a couple dozen basic assumptions about how the runtime works, but every time I thought of a solution keeping the engine working. Now I think it would actually work, but it involves a lot of lockless atomic memory update techniques, to allow use in severely limited contexts that cannot use pthreads. I like solving the issues because it might be useful to know the answer some day. But it's bending schedules out of shape.

     But now I can defer any more work on shared memory support, as long as the APIs now permit it without requiring it.

     Every time a runtime problem comes up that's critical, I'm asked do it even though I'm already on multiple critical paths. I'm being gently nudged toward taking charge of big views and letting others code pieces. But I'm more a doer than a planner.

comic opera ¶

     I'm in the strangest work meetings lately, full of folks guessing what others are thinking, since technical details are complex enough no one quite sees anything exactly the same way. Weird turns occur when you explain some concept abc in the plan, which someone has trouble grasping then proposes xyz instead — but it's clear xyz is the same as abc, just said differently.

     The funniest thing today was being asked if I was done with code I'd not even started, when the meeting was to redesign the prerequisite for the code in question. I just stared, and several others laughed outright, so it must have sounded absurd to them as well. I think the politics is over my head, since I can't fathom intent of such questions.

archived events ¶

     I added a new page to the treedragon archive to gather my old postings on async events and related materials (like event passing in programming languages). Following paragraphs introduce and explain the current relation to what I'm doing on this site.

     First, there's no particular reason to read that page quoting things I wrote on events from 1998 to 2002. At worst, it could severely muddy your grasp of simpler things of interest to me now. The material might only be intrinsically useful to someone who wanted to see as much on events as possible, to swim in excess detail.

     My content from five years ago favored a very high level view, and my current interests are very low level, thus strongly in contrast. Lately I want to get async events structured using least space and cycles I can spare in a server, except where spending space and cycles enables debugging visibility and resource auditing. (If I can spend more space and time on visibility, then I'm inclined to do so, since knowing your system works is more important than doing the wrong thing more efficiently.)

     Many old codenames appear in the old blog entries, such as Mithril for the programming language I planned then, which has a common runtime with the Lathe language I'm developing now in my copious free home coding time. Most of the name changes are just rebranding changes (to the simpler, focused, consistent þ branding). However, there are real substantive differences between my past and present endeavors: today I'm much more pragmatic with an intense interest in minimalism, because time is fleeting.

     When you see a detailed description of my current approach to events, you might have trouble seeing how it's in any way related to my old plans, except in spirit. My new event plans are rather programming language centric, and view events as a specialized case of subroutine implementation in high level (higher than assembler) languages. Roughly speaking, you can equate one of my new events to a stack frame for a function call in current conventional high level languages. Except event frames are heap allocated and reference counted, and do not require the receiver of request to return by replying (although this is expected when a reply-to queue is specified). My model is a bit like continuation passing where return addresses are passed as explicit arguments, and returns are effected by calling the continuation. In like manner, replying to an event by sending a new event (referencing the old) to the reply-to queue is like calling an explicit continuation. Because event frame stacks are heap based and not lifo, order of arrivals and replies is asynchronous.

     That's the gist of my new approach, which can be explained very clearly (albeit densely) in less space than that entire archive page on events from five years ago. The fancy parts of my model involve support for backtraces in event frame stacks, since only such backtraces will provide context to grasp control flows in both failure and success cases. We need backtraces for data as well as code, when data represents message passing — in async systems this visibility becomes crucial.

     I expect my approach to async events will make most sense in the context of actual syntax used in Lisp and/or Smalltalk to declare event handlers, with an explanation of control flow in practice. Right now I'm hanging up my progress on this site planning exactly how the runtime for that will work, so I can then write API samples.

past citations ¶

     Today I quoted several pieces from my old treedragon site, five years ago. The last entry written this month appears on 08nov07, but after that appear three sections I quoted today: pieces from 05oct2002, 26aug2002, and 01sep2002. There are all at least tangentially related to things I want to write about today: async event in programming languages, and use of stories (especially narrative) to structure context for useful data.

work games ¶

     My day job is very interesting lately for several reasons. I guess I'll get this out of my system, before hitting my real meat further below regarding events and stories.

     The async event work I'm doing is the most technically interesting thing I've done in over ten years: partly this is my own doing since I've been angling for it. I wanted to follow through on my event designs from ten years and five years ago. The current situation in my day job was just right to permit this pursuit.

     On one hand, more async code was exactly the right thing to try, and I pushed for it, and other folks knew it was the right thing too, creating the right political environment. Over a year I established a reputation that I could do almost anything that was well-defined: I'm brilliant at execution. I'm not good at social games though.

     And that's why it was essential someone with more clout than me also wanted async infrastructure — a brilliant coworker from Cisco (let's call him C) who has near carte blanche to do as he likes — and he wanted async events too so we could try tactics with queueing (not unlike Matt Welsh's SEDA) he's wanted to do for a while too. C and I found ourselves completely sympatico on approaches to events and threading — we both think they're orthogonal, so you can design around queues and later worry about whether to use threads and how many threads. It should not dominate architecture.

     (C's done similar things before, but with a less abstract focus than my designs. I bring more of a programming language runtime focus, with an emphasis on complete generality and low level efficiency. From experience C knows gotchas are easy to find, and my description of async events sounds like it avoids the worst.)

     Unfortunately, other folks have trouble understanding all the fuss: the phrase "over engineering" came up more than once, since fully general async events were not necessary to achieve the tasks on my plate. And that's true: a much simpler API specific to my service need not expose anything about general event machinery. But C wanted to use the same async event system to rewire

     But events are hard for lots of folks to grasp. I was asked to provide specific APIs that would map existing system behaviors onto the proposed async events. So I did, but you know I'm already on the clock. This happened more than once. I told C it was like being asked to start a new farm out in the midwest to grow a new kind of crop, pressed to run the farm myself, establish a fleet of trucks to deliver the produce everywhere it needed to go, and also create a new chain of supermarkets to retail the produce. Then of course, I'd have to do the shopping myself too, and cook the meals so the food was ready to consume. C laughed and pointed out I was also being asked to do the dishes. (You can add a lengthy restaurant estension to the metaphor too if you like overkill.) It's not enough to engineer a new genetic strain of super staple — you also have to run the farm, and the distribution network, and end user services, or otherwise your clients feel put upon. So volunteering for a new bottom of the world is taking a lot of weight on your shoulders. Well, hey, I asked for it.

     Oops, I just realized I had a pun between C's name and the C programming language going here. C wanted the async event APIs to appear in C syntax with a C runtime. This made things even more difficult than one might think. For example, my plan was that event queues would be optionally threadsafe, which would of course mean adding the right mutex and signaling (or broadcasting) support in the right spots. But on Friday then I was told the library could not link against pthreads — I could use no system calls. This adds quite an awkward degree of difficulty: it means I'll need to abstract pthread services passed in through an API by clients who are using threads (not the subsystem that C will be using). Ha ha, you can see I'm giving the clock a nervous glance now, thinking about my schedule and upcoming holidays.

     At least one of the guys who excels at games is now wandering around saying: where's this marvelous async event system I heard so much about — is it late? So he might be angling to have a no-show declared and the event plan scuttled. If it weren't for C calling the shots my async events would already be toast. Politics is beyond me; it's all I can do to track it, and try to stay out.

     My code APIs were much too long because they were inclusive, and because in C every little thing has to be visible because nothing is private, unlike private code in C++. I was asked on Friday to shuffle the APIs to exclude as much implementation detail as possible from the "public" interface, by splitting headers into "before", "core", and "after" headers, with core APIs using private types from "before" headers. So writing this async event stuff in C might be a bear to make simple, safe, and efficient all at the same time, while keeping build dependencies under control.

     My time estimates were based on blitz implementation without interference, but I can't even start yet since a negotiated API cannot be landed without more haggling, and haggling is the bane of getting things coded quickly. The only good part here so far is I really want to see the async event stuff run in real world apps.

brilliance bane ¶

     The bane of brilliant young men is distraction. Learning to fit your interests into available time is one of the lessons taking longest to learn, and it mostly involves very ruthless pruning: saying no to almost everything, then learning to say no to still more. There's only time for a slender thread through a core of interesting parts. Your plan must go through continuous pruning and culling.

     I dropped out of college the first time through — around 1981 or so — because I became fascinated with something I was studying in library stacks, and I stopped going to classes because they were drivel. I'd have made it to graduate school by that time if I'd been funded through my undergraduate program. But I had to skip quarters to work and pay my bills. I reached a takeoff point before my undergrad classes were done.

     Anyway, from an external point of view, it looked like I had crashed and burned because I blew off my classes. I was studying how knowledge was structured, and I was trying to imagine a new way to encode knowledge in language and reasoning. In other words, I sounded really crazy to folks who didn't spend enough time asking me questions to realize how brilliant I was. (And then they wondered why the system had not managed to keep me on track fitting into the way things are done.)

     However, my goal was impossible, so I was chasing a chimera. My object was just as dumb as the strong artificial intelligence goal of folks in the 80's, except I was going to revolutionize the way people thought, instead of teaching machines to think. I was trying to create a new operating system for the mind, based on different (more effective and fluid) ways of expressing ideas. In a sense, you can say I was trying to make an artificial natural language (except that would be concrete syntax and grammar to the denotational semantics actually of interest to me).

     In terminology coined by Vernor Vinge (best known for A Fire Upon the Deep) in A Deepness in the Sky, what I wanted to do is best categorized as a failed dream — something that sounds wondrous and seems achievable, but is grounded in flawed assumptions, or perhaps in uncomputable infinite regress, which amounts to the same thing. However, I learned a lot pursuing that idea, even if it took me almost a decade to finally prove to myself what I wanted must be impossible.

     So I think I taught myself very young what other brilliant folks might never learn until after committing themselves too deeply to abandon: there's no magical way to achieve infinite leverage through technology. Today you can see many technologists who still believe pursuing some one true system will suddenly scale up to address every problem, miraculously, as if things tend to convenience and goodness if you just give them half a chance. But the truth is different: the more you use one way to encode information, the bigger a pain it becomes — there's a diminishing returns effect. Progress works better by replacing large complex things with smaller simpler ones, often with some essentially new scheme. But the new better thing simply fits better in context.

     In short, things most often get better by refactoring to a smaller, more compact and rational version. But the path isn't inherently analytical — you don't get there by turning the crank of a machine that reduces things logically. That would be a kind of compression, which is complex and brittle. To make things simpler, you actually need less — less is better.

stories ¶

     Toward the end of my own failed-dream pursuit of reducing and simplying information (according to my scheme) I began to see meaning disappear in reductions, so I spent a lot of time thinking about where it went. It seemed loss of context was a big problem. I began to see similarity to research in language circles: George Lakoff and others wrote fun things about how knowledge is represented, informing approaches to natural language processing.

     For a machine to read human texts, a program must have a model of how stories are structured, because understanding natural language turns out to involve at least as much encoding in story elements as encoding in words alone. Sentences and ideas mentioned in individual stories tend to be encoded relative to well-known templates that supply missing context. Individual stories need one or more scripts informing normal expectations against which new stories express novel variations, if any.

     Abstractions of types of stories people usually tell act like canonical vocabularies for telling new stories. If you have a complex new story to tell, you are somewhat limited by the palette of existing stories understood by a population at large. A novel writer can to a certain extent write new "story subroutines" out of which larger new complex stories can be written. But the audience is still a limiting factor: subroutines can only be written using ideas palatable to an audience. If you lose folks in the small, then you also lose them in the large. You have freedom to create, but you must still stay inside the fence (if you want a sizeable readership).

     Anyway, there's a kind of social proof aspect to context in canonical stories. People only believe stories composed of elements they're willing to accept without too much suspension of disbelief. Most of the small pieces have to go down without argument in order to make a larger point that's accepted. If too many of the small pieces are strange, then you're making noise as far as convention goes.

     Without evidence in hand — preferrably evidence that cannot be ignored or misinterpreted — you cannot make people believe something very different from what they already believe. And even if most of a new story is old and accepted by an audience, if a story doesn't flatter an audience or appeal in some manner to what they wish was true, then resistance will be high.

     So your story is the most important thing you can say about technology. It needs to be pithy and appealing, like a story about a little Dutch boy who sticks his finger in a dike. Your audience needs to see what you're talking about. And it has to be acceptable in terms of what's already known. Making technical information brief enough for consumption has everything to do with telling stories, and almost nothing to do with logic and precision. A successful story causes an audience to see some of the world differently afterwards, because the story was effective at making something simpler, or more visible, or emotionally gratifying.

     (This section on Gabriel's Feyerabend project dates from October 2002 — just over five years ago. I quote it here because it seemed related to remarks I planned today on the use of stories as context-supplying devices. The piece below contains an explicit mention of story as a device along these lines.)

feyerabend project ¶ (October 2002)

     Richard Gabriel never seems to spell out his agenda in the Feyerabend project, so it's necessary to infer the general idea from all the quotes on that page. Presumably the final quote from Gerald Sussman carries a lot of water since it appears in the position where a conclusion might appear if one were offered. Let's pick out the start and end of Sussman's remarks:

Gerald Jay Sussman (1999) as quoted by Gabriel: Computer Science is in deep trouble. Structured design is a failure. Systems, as currently engineered, are brittle and fragile. [snip] This fragility and inflexibility must not be allowed to continue. The systems of the future must be both flexible and reliable. They must be tolerant of bugs and must be adaptable to new conditions. To advance beyond the existing problems we must change, in a fundamental way, the nature of the language we use to describe computational systems. We must develop languages that prescribe the computational system as cooperating combinations of redundant processes.

     I'm trying to do something about fixing brittle and fragile systems, but my approach is not based on a particular coding methodology. Perhaps the rule I'm breaking is that one should espouse one methodology as the answer.

     Coding systems and programming languages in general provide mechanisms for abstracting and controlling patterns of execution behavior. However, no matter how marvelous the abstraction mechanisms provided, once they are piled high and deep enough in composed subsystems, the result tends to be something brittle. I'm not sure this can be fixed.

     What I want to do is reduce the pile and explain what everything does in a given pile of code. Unfortunately, explanation doesn't have a methodology as such. It's the goal rather than the means.

     I think it's necessary to understand all the code in a big pile for it to have rational behavior. In fact, only because someone understands a pile that requires some overall rational behavior is imposed on a system. Systems built randomly won't automatically exhibit rational effects.

     In short, I think fragile and brittle systems are caused by too much complexity uncontrolled by coding methodologies. Adding more control (using the usual 'more is better' plan) is not going to help in particular. This is where many folks go too far in pursuit of strong typing, for example.

     The right way to bridle complexity lies along another dimension, a dimension not only of sight and sound but of mind. (Insert extended Rod Serling joke here.) For want of a better term, let's call this a dimension of of story. When code sucks, more often that not it's because there's an absence of story. If you like, you can also substitute the term design for story.

     One of the resources that human beings have at their disposal is a kind of memory called episodic memory. Briefly, this has to do with stitching together a sequence of events into a story. We don't often make use of this in our industry as a matter of standard practice. (However, use-cases directed at solving specific problems of canonical user prototypes might be a successful story in this area.)

     Developers should design software to fit the same kind of mold targeted by professional writers in certain markets: the inverted pyramid format. This format starts with the point of the pyramid, by stating the overall summary, and then slowly expands the details in successive layers.

     When this principle is applied in graphical user interfaces, it sometimes appears as progressive disclosure, which means a user can drill down into details at will if desired. This organization is also useful for developers, and not just users, but tragically we seldom design software this way.

     Good software flows from a story that makes sense at every level of detail from the point of the pyramid down to the base of hairy particulars. The presence of a rational roadmap in the mind of a developer permits the human eye and mind to see variances from the intended plan when they are encountered.

     I feel one of the makings of successful stories is simplicity of structure at the point of the pyramid. Bad software is characterized by a tech introduction like the following: "Read this twenty chapter reference manual before you start." A story without a short summary is a bad story when it comes to computing technology.

     I think the solution is literate programming without regard for which kind of methodology is used to build software. Good software must be designed as a story, and a successful developer must think in the creative ways a writer does when building a tower of concepts. Otherwise it's a tower of Babel.

     (As you can see from the date just above, all this section but this paragraph was written in August of 2002, just over five years ago. I was looking for something to quote here from treedragon, and I ran across this, apparently the first time I decided I wanted to clone HyperCard. It was the beginning of many new conversations. It signaled my intention to pursue tech I found interesting since I was unable to accumulate resources to start my own company with my wife (now ex) burning my money as fast as I earned it. Now that five years have gone by, I find it interesting that my technical interests have remained similar, even though I have a totally practical attitude these days about expecting less. None of this is a statement of my current intent; it's just a look at what I was thinking half a decade ago. I've since gotten deeper in the business of moving bytes around on the wire.)

sands of time ¶ (aka cloning HyperCard, Aug2002)

     I was asked why my current day job might not be a long term thing. Well, I don't enjoy it much, and I never did (more than marginally). But then, jobs seldom seem very interesting since the internet boom. I like basic and rigorous technology, not slapdash lego blocks crap. The boom incented folks to pursue really stupid tech for the money.

     Now, what I do in my day job is not stupid tech. It's sorta interesting. But it's missing the majority of tech features that really appeal to me. However, I don't see a lot of folks doing more appealling things lately. But I could just look harder. It's my life, right? I should enjoy it more. Currently I like my boss, I like the pay, and the tech's not too shabby.

     But I think I'd be happier if I simply enjoyed what I did for a living. I don't currently, and I don't want to go in to work. Still, I have to. I do it mainly to support my family, and not because I get any thrills. I'm much less interested in good pay lately, if it gets spent anyway. (We're working on this at home, but it's too late to be job enthused.)

     What I really need is a reason to enjoy life on a regular daily basis. I don't get that in my current work. I don't learn enough. It's not fun. Lately my boss has thought of some interesting things for me to do. So I'm not bolting out the door, but I'm still not thrilled daily either. I expect I'd be happiest working on any kind of coding environment.

     For example, if you're cloning Hypercard, then I'd like to work on it. But if you're making a new browser, that's a big pile of nasty worms. Internet apps, per se, are boring. That's just, uh, comm stuff, right? Why should I care about moving bytes around on the wire? Whoopty. Now, if end users could program the behavior, that'd be interesting.

     I like users as authors. I'm only engaged by enabling productive acts. Users as consumers is boring shit. I don't want to be an eloi enabler. Apps where users see and hear stuff, and play with widgets, are a bust. I only want to make tools someone else can use to make something. Yes I know this is too much to ask. But maybe something'll come up.

     (See the following entry dating from September 2002 for a short followup illustrating what I was thinking after August 2002.)

     (The "boom" below was a sideways reference to Bolide: a codename deriving from the name of an exploding meterorite.)

hypercard (boom!) ¶ (01 Sep 2002)

     Although I don't have time for hash tables, I'll mention Hypercard. I thought about it a while today while I was hacking away at Mithril. Then I started searching for web pages to get any recent information. I guess I'll go get an old used-book soon. This problem should be fun. I'd like to write a spec for Hypertalk features, but in a Mithril syntax.

     This would basically be a virtual machine spec for a Hypercard app. Machine semantics would be implied by the various class definitions. Hypercard was my model to open UI elements to find scripts inside. I want this universal in my app interfaces, for a good Mithril frontend. A user should be able to control everything by tweaking widget scripts.

     Recently a friend at work asked about plans for graphical interfaces. I was vague, but now I think I should focus on a Hypercard work alike. I'd like my vaporware Bolide project to be OpenDoc plus Hypercard. But a worse-is-better approach would focus on Hypercard the most. Simple expediency is often better than too much global architecture.

bad memories ¶

     I spent all my writing time tonight on the new treedragon archive page about async events, but it's not done and will take about another hour to finish.

     Reading the last eight months of 2002 is hard because I can't avoid seeing the entries showing signs my ex was cheating on me with my sons' 20 year old taekwondo instructor. It makes me almost physically ill to read old material written when she divorced me after I caught the two of them together.

     You'd think it might have less impact after five years, but since I lost everything I owned when she put me through the wringer, and the two are still together now five years later, the trauma fades slowly. Nice to be over the hysteria, though.

middle man ¶

     I'm so busy at work lately. I'm at the hub of activity between a goodly number of people, and I spend just a little too long talking each day, when I need to make more coding progress.

amphigory ¶

     I'm teaching my pet rat to cook. How long did it take you to decide that wasn't likely? Just seeing if you're awake. I wasn't in the mood to finish the archive page on events (it's not like I have a deadline), so I'll just post a few random bits and pieces instead.

     Most of this is just random things I saw recently which made me want to comment. Lately I'm thinking about event passing all the time, and to stir up ideas I read web pages I turn up from searching for keywords closest to my last train of thought.

     For example, my last search was for "lockless queueing" since that might be a useful approach for interprocess event messaging without system calls, which would be important if you wanted to send an ungodly large number of them a second, and still have a lot of cycles left over to handle them.

     I'm normally a "all threads all the time" guy, and don't often develop with multiple processes (of the native operating system sort), but a have a couple good reasons to work with processes lately, both in my day job and on home projects. Let's start with the day job reason.

     Suppose you write servers that use a lot of memory and more is always better, but you're developing on a 32-bit platform. At some point it might become feasible to install quite a bit more than four gigabytes of RAM in the platform, but less than four is usefully addressable by one server process. Using more than one process is a way to get more of the physical RAM mapped in some process's logical address space.

     Okay, now what about a home project reason for using multiple processes? Mainly it's a reason to start making server oriented infrastructure even in a local laptop environment. Might as well start by gearing everything around event passing oriented i/o with the outside world. For starters, the basic "hello world" stub should be oriented around an event handling engine which comes up and loops dispatching events in a totally context free fashion: no dependency at all on the native environment — blind, deaf, and dumb except for events-in, events-out. Then other stuff hangs off this hub.

     Then to get yourself a real server, you map system level events to the event handler hub. So on Linux, instead of making the server revolve around epoll, say, you could turn all epoll events into events for the event system. This indirection costs very little (unless you introduce too much flow of control switching — hopefully not actual context switching) and suddenly you get a new capability: you can simulate network traffic by simply sending events, because all your network traffic normally does just becoem events.

     This is a kind of testing oriented development tactic, since you can simulate in testing any kind of i/o that would normally appear in real operation. And you can record and playback events to replay system behavior without needing to have an original network context around to do so, as long as the flow consumers and producers are simulated.

     Then you'd want to write an event debugger of some kind, if there's time. I'm almost absolutely sure we'll never have time for a stepping event debugger in my day job. (We're into debugging, but not full scale development tools for new stuff.) Hmmm, that's an interesting line of thought — had it just as I began writing this paragraph. Writing does that: new things pop into mind.

paradigm warfare ¶

     In a rambling thread over at Lambda the Ultimate, Kay Schluehr said some folks make "progress through rebranding", inspiring Anton van Straaten to note:

Bouncing between progressively refined alternatives seems to be a major way that real progress occurs.

     After which Kay Schluehr replied with this:

This discussion style of quoting just an arcane, technical paper that "makes my argument" without any precise reading and patient interpretation has become pop. Programmers pretend to be language theorists and instead of using their own idioms they strive to appear like scientists. PLT is the new coolness. Instead of taming the "paradigm warfare" PLT has just become immersed.

     Yeah, it seems a variant of "social proof" abuse: observing "someone is saying X" so therefore "X must be true" and assuming "what I want is supported by X", so random papers support whatever a person wants.

     This is also a variant of confirmation bias. Programmer has an idea and looks for something related, finding something they like is positive feedback they interpret as support for the idea. But it's just random positive input while trawling with an agenda. There's no testable relation between ideas heard and ideas held dear.

applescript syntax ¶

     Applescript was roughly inspired by HyperCard's earlier HyperTalk scripting language. Both Applescript and HyperTalk had an English-like syntax, and this is often cited as essential to intended utility of Applescript. This idea is contested below by one of Applescript's main architects. I thought this was useful support for my position that HyperTalk syntax is not essential for HyperCard features either.

     In a 2004 Slashdot thread on Applescript, Kurt Piersol commented on the topic of Applescript's human readable syntax:

As one of the original designers of AppleScript, I feel compelled to mention that the syntax issue is a complete red herring. The language was originally designed to have replaceable syntax, and it could switch, in real time, between dialects with radically different syntax. The same routine would display in different language localizations, or different syntax forms, depending on user preference.

     I'll add a comment between each of Kurt's paragraphs — more to emphasize the paragraphs than because I have any important remarks. Anyway, here I kept Kurt's original emphasis that syntax was meant to be replaceable.

Apple had a Japanese dialect and a C-like dialect built during the development phase. These were never tested enough to release, primarily because of a management decision to shift a lot of the team over to work on OpenDoc after AppleScript's first release.

     Coincident to my purpose here, I also worked on OpenDoc under Kurt. I never learned much Applescript despite working with former Applescript folks. (When I'd sought a scripting language for use two years earlier at Taligent, I talked to Applescript folks about it, but they told me Applescript had little semantics, giving me nothing to leverage.)

However, that said, it can be terribly frustrating to figure out the syntax of the current system. This is because a number of language extensions have been built through a mechanism known as OSAX extensions, over the years. The language elements which come from applications work pretty well, because there's a block structure which wraps them into easily detectable blocks. OSAXen, though, tend to interact globally with the syntax of everything, and a lot of the OSAXen have really gnarly syntax that causes all sorts of unforeseen interactions. Unfortunately, the core language was missing some important features, and OSAXen were the easy way to add some of that back, so it got completely out of hand rather early on.

     In other words, despite looking very friendly, with English-like syntax, you could easily have difficulties when expecting consistency.

The original team never imagined that the current syntax would be the only one. Quite the contrary, we viewed it as a syntax that HyperCard users would find familiar, but many other folks would find rather verbose.

     So the original syntax was more a matter of market continuity with HyperCard users, rather than stating a principle that any one syntax is best.

busy week ¶

     By the end of meetings today at work we were saying, "Was that Monday of this week? Seems like a lot longer ago." This week definitely seemed like two weeks, with occasional flashes of comedy of the absurd. I'm glad it's over.

pending events ¶

     I started an events page for the archive. I might write more about events soon. How about a short gloss right now? Sure, why not. Okay, put on a programming language hat. The following compares a "function call" to passing a stack frame as a message. This perspective unifies async events with function calls if async events pass stack frames as messages.

     I'm sure I'll say this a zillion times again later. On some other occasion I'll make the presentation better. For now, think of a function call as passing a stack frame message to the receiver. If stacks are lifo, then replying pops the stack. But lifo stacks are not required in the short term. Instead of popping to reply, you could instead return from a function by pushing another frame — you could choose to make the stack deeper when you "return" from a function. To keep the stack from overflowing, the callback has to return control to the original caller who can choose to pop the stack. So it's caller stack allocation.

     Anyway, you can use this scheme for explicit event stacks and get something perfectly fluid that allows you to model whatever you want in async event terms. Sending an event just means "send a frame". The event stack always gets deeper. If the receiver doesn't want the deeper stack, it can simply stop referring to nested frames so they'll be collected.

     Do you understand the representation of closures and continuations in (say) Scheme? If you do, this might make sense. If not, then you might need to strain a little. Just remember that dynamic languages sometimes heap allocate things you think live on the stack in C. Often the "stack" in Lisp can have all or some local storage locations in the heap.

     The general idea here is stacks are really trees. One path through a stack looks like a conventional list of stack frames, but all of stack past lists together — if they never go away — form a tree with shared nodes all the places where the stack branched over time. Obviously normal stacks go away efficiently.