Þ   briarpig  » log  » oct07

too many cooks ¶

     My day job is going great, but it's very busy. Yesterday I had one of those design reviews where at least one person disagreed with each and every starting given, constraint, assumption, strategy, and tactic. By the end you feel it was all about problem definition instead of your solution — because it was.

     C said it was typical and turned out about as well as those things did. In topics where everyone feels they understand, everyone wants to take a shot, so they do. As it turns out, my bottom-up approach caused the most trouble.

     A few months ago the original architect was just joking when he kept saying they should just let me rewrite everything. He was being ironic. So now it's weird folks expect I'll rewrite his layer — in passing — while slaying dragons. Years ago I used to get squeezed from both ends when I did middleware, but now I absorb every layer next to mine. Few folks like the bottom of the stack, and more of it gets defined as bottom. Everyone wants to work up in the transcend. (SciFi joke.)

     Hey, I only have so many cycles. The load I had was fine.

mork and mindy ¶

     On Sunday son two rolled on the sofa, achieving an upside down position. "Hey," he said, "this is actually pretty comfortable." It seemed a joke: I had a suspicion.

     "Have you ever seen a tv show named Mork and Mindy?" I asked him — expecting a confirmation.

     "No," he replied, then asked curiously, "Why? What's it about?" So I told him about a 1978 comedy.

     Looking for an account of Mork's chair sitting style, I found a characterization of Mork on a sitcom website:

Mork looked human, but his strange mixture of Orkan and Earthling customs--such as wearing a suit, but putting it on backwards, or sitting in a chair, but upside down--led most people to think of him as just as some kind of nut.

     Since I'd stopped watching television a couple years before Mork debuted, a former high school English teacher told me about the show when I visited from college. "There's a guy on Mork who stole your act," she told me, "You should check it out. Robin Williams acts just like you did."

     I was a prankster in high school who often did normal every-day things slightly wrong just to mess with people. I said even stranger things, which was more fun. When you're young, eccentricity is usually forgiven — but radical conformists rarely suffer deviance gladly.

     I did check it out, and loved the show. But she was wrong: Robin Williams was much funnier. Mork had my attitude, but done at blitz speed which works better.

halloween ¶

     Earlier this month I said I'd tell a costume party story from three years ago in San Francisco. I'm less interested in personal stories now, so this version of the tale will be minimal.

     I had short hair and decided to go as a sailor. My girlfriend and I had broken up, and she was going to the same party with her old flame just as friends. The party was one of those neighborhoods of mostly straight folks — less wild.

     Realistic seamen's clothing in second hand stores was quite easy to find. So I went the navy blue and black route.

     I gave a start to a guy on the street when I walked to my car in the Mission. I suppose he was doing business. "Is the fleet in town?" he asked me with some concern.

     He was a tall black man in his late 30's wearing a nice long coat and a large brimmed hat. His manner: friendly but serious. "No, I don't think so," I replied, "This is a costume."

     He laughed in relief, explaining, "I thought I was in the wrong place. I'd be in trouble if the fleet was in town." He didn't seem to be joking. I thought a guy dressed as a sailor on Halloween would be seen as in costume.

     When I got to the party, I'm not sure I ever met someone who thought I was in costume, except my old girlfriend. Apparently I looked like some sailor on leave crashing Halloween parties as a first order of business. A couple different things contributed to my realistic appearance.

     Most costumes were just a bit precious, with glitzy toy props from stores. The typical style of costume screamed "look at me, I'm a costume." My look was very drab, ergo likely real.

     Also, I was clearly several years older than the older echelon of party goers. Folks above 35 and especially 40 are sometimes barely tolerated socially in SF, like walking dead who won't stay buried. People who don't look like they belong probably don't: so I must be a vistor to town — an old errant navy loser.

     I had a lot of fun conversations, but many of them — at least five or six — eventually saw me asked, "You're not really a sailor, are you?" Or something nearly alike. A dawning realization would crawl over their face at some point, perhaps prompted by my diction and vocabulary. I'm educated, which clashes.

     I was accosted by a gang of four dressed as members of the Village People, except two were women and two were guys. One guy was dressed like a cop, and he insisted I ought to join them. They were shy a fifth member. The cop kept yammering away so I couldn't talk to a very attractive woman in the group.

     I thought, maybe this guy mis-read me. He thinks I'm a guy dressed as a sailor on Halloween in San Francisco and ... nudge nudge, wink wink ... hey the Village People are ready to party. I started replying to everything he said by looking directly at the gorgeous woman when answering his questions.

     Eventually this was so clear she looked at the faux cop and told him very distinctly, "Straight." Well, duh. But the cop looked unhappy like she was a referee who made a bad call.

steering correction ¶

     I recently changed my near future plans for this site. In contrast with my original plan (cf about) to focus on Scheme first to teach my sons programming, I'm now spending a lot of time thinking about event-driven code.

     My long term plans are the same: create basic tech for my sons to experiment with simple web site creation, as a practical way to learn coding. But now I've specific ideas about event handling in site designs I want to work into architecture at a language level before finishing a first cut of Lathe (Lisp dialect with Smalltalk features, cf λσπ).

     My plan to first finish Scheme aimed to make progress until new ideas popped up. I expected new ideas much later, but since I have some already, I'm going with it since inspiration ages.

     I'm unsure how to explain what I'm designing. I might hint around a little in stream of consciousness fashion. I won't attempt a good presentation until (much?) later.

dreaming events ¶

     Lately I've been dreaming about event-driven code every time I go to sleep, even for a nap. Often much of it makes sense. I take it as a sign I need to cut back and relax more. It's like I'm working while asleep: a little creepy. So I bought a mountain bike today and started riding for exercise.

event specs ¶

     I spent the last four days in my day job writing specs for runtime models and C apis for async events. I prefer C++, but C is what's needed now. (Anything I add later to Lathe will be in C++.) In four days I had a hundred pages of wiki content, and I'm not done yet. It's disappointing how long a spec gets when you spell out everything.

     If you assume a reader only knows about C structs and function pointers, it takes forever to convey every nuance about just memory management, ownership, refcounting, and handles, for example. The spec is already so long I know folks won't actually read it, meaning I'll have to write a summary of the spec. What's wrong with this picture?

home project specs ¶

     This — combined with my recent experience here writing about refcounting in C++ apis for þ — tells me indirectly I don't have time to describe as much of what I'm doing as I'd like. It takes too long to explain from scratch my approaches to things that seem obvious and trivial to me.

     It's hard to find time for home projects like the one this site is about. You need enough time to do it, and you need enough time to say just enough so it's useful to you and other folks. (Otherwise it might rot in isolation.) I feel like it's going to take forever. So in the short term I'm going to say less and work on code a little more. Except right now, I'm working out some very interesting ideas about events built into a language.

blog lurkers ¶

     I'm aware I have (a few) lurkers here. It makes me want to obfuscate my best ideas just a little. I'd have found what I'm doing now very interesting in years 1995 through 2002. To the extent my younger self would've found some insights useful in designing new things, I'm assuming it's true for other folks too, and I'm wary.

     In working out clean language-friendly event-driven code, I'd hate to inspire other folks to bolster older tech in ways preventing me from getting a foothold anywhere. Having my own designs parroted back to me in someone else's work is always very grating. So I'll hold back some.

smaller specs ¶

     I'm a big fan of smaller specs. And big specs are usually what I think sucks about much existing tech. (Makes you want to blurt, "You expect me to read how many pages about your stuff? Are you insane?") Okay, so I already know I can easily write too long a spec about event handling, because I already did. How could I do better? Can I write less for my sons to read?

     Thus I want a shorter way to describe some things. Maybe a specific kind of terse notation would help. Making up new notation is a problem in itself. But if it replaces a spec too long to read, maybe that's a good enough excuse.

hypercard event handlers ¶

     This weekend I've been thinking a lot about HyperCard event handlers. They confused me a lot when I thought about them around 1990 or so, because I didn't have a physical model of async event dispatching in mind. Then I had trouble with figuring out exactly when did things happen, which is hardly surprising now I know I want everything to happen asynchronously. :-)

ycard ¶

     Back in 2002 I used "ycard" to name the HyperCard clone I was planning. Today I would call it þcard (thorncard) instead, but I'm not actually

     For example, I'll just use Lathe itself (or Scythe or Pith) for the purpose of scripting. So I won't have an English-like scripting syntax. I suppose most folks think only of Dan Winkler's HyperTalk syntax when they think of HyperCard. But I'm more interested in the runtime behavior than surface syntax.

     No, the interesting thing was how HyperTalk declared event handlers like on mouseUp in the scope of objects which would process the events. I might add a special form to Lathe using symbol "on" to declare event handlers.

     Then the interesting thing in HyperCard itself was an event hierarchy controlling order of event dispatch, so which object's event handler was used depended on hierarchy and routing between different scopes. This is similar in nature to class hierarchies in object oriented languages.

     In a subset of Lathe features called þcard, the fun thing to do would be: native event handler support similar to declaration of class methods — plus a way to define how an event hierarchy works for user defined events. Then you'd be able to define all kinds of useful event handling systems. A clone of HyperCard would be just one application.

     If you worked that into a web server, then you could figure out how to compile some handlers into javascript returned in web pages, when event handlers must be distributed between client side and server side.

programming for the masses ¶

     I find a lot of web pages I want to read when I search for "HyperCard", "event", and some other keyword of interest like "Tcl" — it's very interesting how many things were inspired by HyperCard. Previously I didn't have an outlook letting me appreciate some of the connections. (For example, I was never interested in Tcl, but now I'd like to see if I can mimic anything it does when the spec is very small.)

     In Wired (Issue 3.09 | Sep 1995) Curtis Yarvin wrote this opening paragraph on scripting languages:

The development of fancy graphical interfaces has finally made computers usable by normal humans. But computer programming remains an arcane art, understood only by members of the technical priesthood. This will have to change. A growing community of digital artists and other right-brained types are beginning to demand the power and flexibility that can only be achieved with programming.

     I think it's singularly interesting that didn't happen in the last dozen years, which is why I made realization of such a thing a goal several years ago, since I agree with Yarvin's view. I forgot about it for a while, but it's hard to think of other worthwhile pursuits (besides making money).

     Later in the same article as above, Curtis Yarvin remarks on the English-like quality of HyperCard's HyperTalk scripting language syntax:

The simplest way to put a user interface on programming is to make it prettier. Conventional programming languages look like physics formulas, with thickets of punctuation. Scripting languages are more attractive. An extreme example is Apple's HyperCard, which looks a lot like English. Instead of "a = b", a HyperCard user says "put a into b." This enchants users at first, but the extra verbiage can come to seem clumsy.

     I was never enamored of syntax like English, because I prefer highly unambiguous syntax, and languages with "do what I say" rather than "do what I mean" semantics. Also, I'll never have time to clone HyperTalk, even if I would find it amusing. So never expect HyperTalk in my future plans.

     Yarvin's last paragraph calls for a new generation of industrial strength scripting languages. Personally I want to make scripting languages industrial strength — at least in my home project pursuits.

17Nov02 agents ¶

     [All of this section is an unedited repost of one treedragon blog entry from November 2002.]

     In Thursday's brainstorm meeting, Andy Hertzfeld presented the idea of using agents in Chandler's user interface design. I'll describe this loosely below and add comments about ideas that popped into my head as a result. I was so distracted by my own emerging notions, that I might have missed crucial elements of Andy's plan.

     Andy presents the notion of agents as related to the Magic Cap interface used in the General Magic architecture. (I can't do justice to this part of Andy's background context for agents, and I hope he has time to describe some fun aspects of Magic Cap.)

     Agents are a user visible aspect of a plugin architecture, where an agent can process some event streams of content for a user tasks, like email. For example, a given email agent might have a style of handling incoming email that differs from other agents engaged in the same task. An email agent might follow spam processing scripts.

     An agent fills a role in the user interface, and more than one agent could be slotted in the same role at different times. (Can more than one agent fill the same role at one time? Can two email agents at once handle incoming email without conflict?) A user might grant control to different agents, and hot swap them depending on context.

     The existence of agents and their work is presented graphically in the user interface. An inactive agent might show an icon with desaturated color to look inactive. But an agent actively working might show animation to reflect the amount of processing involved. Logs of agent activity permit a user to audit past performance.

     Scripts for agents, probably written in Python, can be inspected when a user wants to see them, perhaps in order to author or modify agent behavior. Most users will use existing scripts, but some technical users will write new ones and share them with others.

     Script handlers for agents are a great way to add indirection to existing user interface actions, in order to introduce late binding. For example, what should happen when you select an email and click a "delete" button? Using agents, a garbageman agent can handle the delete message according to context by script.

     In fact, all user interface actions might be a message to an agent that interprets what the UI gesture means. This is even more structure than attaching scripts to every user interface widget, because now an agent is involved and not just a script. The late binding due to agent indirection allows more flexible results to be coded.

     I probably colored several of the items above with aspects I found most interesting in ideas mentioned by Andy Hertzfeld, but I think most of these were mentioned by Andy in one form or another. I became highly distracted by ideas about physical data representation.

     The undo model and transactions in Chandler will probably need an agent specific design, if undo intends to revert user caused events which are routed to agents for handling. Note undo probably doesn't undo all state change. Does it make sense to undo receiving mail? External event flows like mail might be separate storage.

sore throat ¶

     I seem to be getting a cold for the first time in many months. I almost skipped my three day deadline here on account of illness.

async events ¶

     My day job lately is very interesting, because I'm designing and coding some basic infrastructure for async event handling in our server's runtime. There's an ironic element to this I'll describe in more detail below, based on the fact I was writing about this topic a lot five years ago.

     The next page I put up under the archive from treedragon will be full of my postings on the topic of events and async dataflow using them, and how I planned to put them in a programming language I was redesigning then, in order to implement a new incarnation of HyperCard in a new runtime.

     The ironic part: five years ago I told folks at my day job (then) I wanted to pursue events for a new HyperCard and I didn't want to write an async event system for them. As a result I lost my job (laid off in a big reduction in force despite being a key developer in a startup running out of money). That was company A which was bought by company B, which was in turn bought by company C.

     Today I work for company C. :-) And I'm writing an async event system for them. So you see why this is kinda funny. It's like time travel. (And I've more than resumed my frame of mind five years ago, making headway again on the same home project.)

     My approach now is rather different than what I was thinking then, because I've become even more a low level performance wonk than I was. Either that or I'm just better at thinking about these things now. It seems more obvious and cut-n-dried.

language events ¶

     It's also clear to me how I'm going to do async event handling in Lathe, the new hybrid of Scheme and Smalltalk I'm working on at home lately.

     Now I'm glad I stopped working on the actual Lathe runtime for a couple weeks to focus on plans for async events day and night. I can go back to coding with a focus on putting events central to the lightweight process mechanisms, so Lathe is inherently concurrent.

     Basically, all inputs to Lathe will take the form of events, some of which are streams from local pipes or network connections. Having a client UI for a server backend ought to be easy, since UI events will fold into the same event flow as network messages. And any scripts that run from menu items or any place else will all get handled by event flow.

     It's obvious how I'd model the HyperCard event system when I want to do that. And as long as the event subsystem is written in C and/or C++, with refcounting instead of gc for event memory management, the performance ought to be almost commercial grade with little tinkering.

writing dialogs ¶

     I'm half planning to describe how I create dialogs. It would be interesting without ruining the effect, as explaining a joke does, because dialogs are more complex than jokes. There's a sociological aspect to it, which is always rewarding. I'd enjoy description almost as much as a dialog, because I'd actually embed a dialog in the description, probably with some self referential boundary violations.

     But that's not tech stuff, and I'm trying to focus on tech stuff. If such a description caused more folks to write dialogs about their own tech stuff, that would almost be incentive to write about dialogs.

     The main trick is inventing characters who speak in dialogs. Each takes a fair amount of work: thinking about their back stories and outlooks, then designing interpersonal strife between characters. Over time you need to evolve the personalities as you find out what makes them tick. You discover which characters are funnier and give them more lines.

     I'll write about my standard cast of characters later. My most popular character was Vex: a professional irritant who functioned as a loose cannon zanni in the tradition of Commedia dell'arte, which fascinated me 25 years ago. Vex was modeled on Michael Keaton's bio-exorcist in Betelgeuse. So actually Vex was dead, explaining why he never filtered what he said very much: he had little to lose. It gave Vex license to say any irritating thing I wanted, which made him funny. Also, he never took directions worth a damn, making him an ideal combative partner in any task.

     The next three paragraphs are excerpts from a shellgame dialog on the microcosm page, which struck me as funny enough to cite in isolation. (Maybe it was funnier new, seven years ago.)

      Vex: You make it sound like a bad thing that I want Yen to upgrade all his software. I mean c'mon. He might be using products written a couple years ago, or (shudder) even as long as five years ago. Get real! The world doesn't stop just because Yen stopped shelling out bucks for new software.

      Yen: You make it sound like you have no duty unless I pay regular software fees on a subscription model, whether or not that was our original understanding. When I pay for something I expect it to work. Now I see all software slowly decays over a few years. Does the free web app work that way, too?

      Vex: Hey, babe, I live on internet time. By now I think in terms of months instead of years. I'll guarantee your data until next quarter at the very least. The informal industry statute of limitations gets shorter all the time. It doesn't last much longer than it takes me to go find a new gig.

smartref refcounting ¶

     I put most of my handle-based refcounting code for þ C++ smartrefs on a new refcounting page in the old tech section. Nearly all the header file code is done, but I still have a bit more of the implementation to finish posting. So that's TBD (to be done) in the near future.

archive ¶

     I started reposting material from my old treedragon website under the archive page. More will appear later as suitable, when it fits what I want to say. Most of it won't be useful here since the quality and focus was all over the map. Over time treedragon became choked with detritus of wild experiments that obscured tech material I want to resurrect.

     I began treedragon in January 2000, but printed stuff from the late 90's as well, which I might repost here too. I played gleefully in the new web medium, and tried a lot of strange things which in retrospect seem weirder now than then. The weird stuff was creative and fun, but not very useful except as a learning experience. I tried new writing styles, navigation devices, and dense imagery to convey meaning.

     But the format on this site is one I prefer now, informed by what I learned in experiments. For example, I use fewer images since they tend to overwhelm text.

shellgame ¶

     Today I reposted a shellgame sub-graph of treedragon for a couple different reasons. The most obvious reason: it's visually interesting, and might inspire someone to try experiments. The second is hard to find: I said (eg) things about web apps and online storage still relevant.

So the only reason why computers seem puzzling and opaque is because they fail to show space usage issues in terms of normal daily situations. If software developers pursued the goal with any dedication, they might easily render the inner computer jungle in terms as ready to grasp as other aspects of daily life.

signal vs noise ¶

     However, in order to find fun items like the above in the shellgame section, you have to wade through a lot of drivel I wrote about the entire context of user interaction with web apps. So my shellgame experiment itself illustrates a principle of dilution: it doesn't take very much noise to drown out your signal.

     My current approach here aims for more signal, less noise. Which is why most of the time I'll stick to tech stuff without distracting fun.

character dialogs ¶

     The link above describes Galileo Galilei's use of dialogs between characters to discuss scientific theories. I first learned about Galileo's use of dialogs in high school. (Later I paid more attention to dialogs when Douglas Hofstadter imitated those of Lewis Carroll.)

     Many of my old treedragon experiments involved dialogs between characters, because I had decided over a period of years that argument and discussion between such virtual personae (cf Dramatis_personae) was a more efficient way to condense nuance of meaning in conflicting ideas and ambiguous problems. When viewpoints are opposed in writing, it efficiently contrasts competing perspectives. Word for word, the density and clarity of dialog tops that of monolog every time. First person exposition is weaker.

disk sectors ¶

     I created the image below almost exactly seven years ago to illustrate the idea of mapping content onto disk sectors (although of course this is not at all to scale). In part of my old website, this figure accompanied a group of pages named shellgame to characterize web app businesses and their

     Still unfinished today, I wrote shellgame during two weeks in October of 2000, toward the end of my long vacation after leaving Netscape. Starting tomorrow, I plan to put up some old parts of my earlier website, after I pick through pieces worth posting again.

     I once got a big kick out of making images like this with Photoshop, which came free with a scanner I bought in the 90's, and remained good to use under Mac Classic for several years. I haven't had Photoshop for some years now because I can't afford to buy a copy. Maybe I'll be able to afford it again in a year or so.

refcounting handles ¶

     Tonight I'm writing about my current best practice for refcounting, and it should be ready to post here tomorrow. On a mailing list, Darin Adler recently asked for a review of refcounting he proposes to use in WebKit. I thought I should describe what I do in þ for other folks to copy.

     At work it sounds like they'll soon let me use my refcounting techniqe in new C interfaces, which differ from the C++ smartrefs in þ, but still pursue the same approach of refcounting by handles instead of explicit refcounting method calls. It works in C too, but without RAII.

     Let's start with a couple very small jokes. If they puzzle you, just think of them as very dry. (Humor that's hard to understand is an acquired taste, anyway.)

together again ¶

     Ged and Vex vow to put the band back together [pointless and inane story truncated here].

alternate news ¶

     Another headline at slight variance with one seen recently in news channels.

     French doctors are puzzling over the case of 44-year-old civil servant who has led a quite normal life — but with an extraordinarily tiny paycheck. In a case history published in next Saturday's Lancet, doctors led by Lionel Feuillet (Hopital of Marseille) say he was admitted to hospital after suffering mild weakness in his wallet.

busted intuition ¶

     My Sunday night intuition was a bust. By Tuesday I'd replaced it with something else competely. The first idea just didn't connect to much. However, now I think I have answers I want, which is good. But they aren't simple, which is bad (and typical). The complexity gets fractically worse as you approach closure. This should sound familiar.

     The good news is I decided I can treat events with very familiar reasoning: almost the same as reasoning for code flow, so the complexity isn't new. But reasoning about code flow is notoriously difficult for most folks. That means there isn't going to be a picture simpler than for code flow. Darn.

C apis ¶

     Today I wrote C apis for event, queue, and stack objects under async control flow with full debug support to backtrace event stacks, since my past experience with async event systems were hard to debug when you can't see context for how you managed to get here if something fails.

     I won't tell you anything about these interfaces, which are a lot more complex than I'd like. Describing them is my near term task at work anyway, and how they work in detail will be proprietary. Anything I do in the future for Lathe at home will be quite different to suit gc memory management.

     But I can summarize the theoretical justification, and note how it relates to literature on threading vs events in high performance servers. The main idea is to think of stacks for event continuations as resembling stacks for code continuations. Then application is straight forward.

explicit stacks ¶

     In brainstorming conversation Tuesday a specific remark about stacks made me see a concrete way to code what I was thinking Monday night, which was that stack-based continuations for code and data looked similar to each other, and did about the same things, but folded differently. Continuations for code can use closures to hold data state for later use, and queue handlers (continuations for data) can use callbacks to hold code behavior for later use.

     How to divide data and code continuations most elegantly depends on your application. And in some languages using both in a very dove-tailed way might be best. In the Lispish mechanisms of Lathe I expect to use heap-based event stacks for data continuations (maybe new) and heap-based function stacks for code continuations (very conventional). Both together would make async control flow very fluid.

     C and C++ based servers use contiguous, monolithic (not heap based) stacks to represent continuations of a thread, so a representation of event continuations of a data task can also get away with being contiguous and monolithic. (You wouldn't want that in Lisp, but it looks barely safe in C.) Anyway, thread code stacks and event data stacks can be independent, and interlaced if you use gc or refcounting.

graph duals ¶

     By Monday night I had a new intuition: that data flow with events could be represented with data continuations that are dual to code flow and associated code continuations. For folks unfamiliar with the idea of duals, get a taste for the idea by checking out these duals:

• Duality_(electrical_engineering)
• Dual_graph
• De_Morgan_dual

     I'm basically working an idea of duality between code and data. Good programming languages let you leverage which you are more interested in emphasizing. I decided async event passing is very similar to looking at threading, but inside out, with emphasis on data instead of code.

     This is really a trivial conclusion since you can find many papers essentially saying the same thing in practical terms, when deciding whether you want to emphasize threads or events in async systems. Once I looked, I had no trouble finding quotes like Why Events Are A Bad Idea (cf Behren, Condit, Brewer) [emphasis added]:

The debate between threads and events is a very old one. Lauer and Needham attempted to end the discussion in 1978 by showing that message-passing systems and process-based systems are duals, both in terms of program structure and performance characteristics [10]. Nonetheless, in recent years many authors have declared the need for event-driven programming for highly concurrent systems [11,12,17].

     Unlike this paper, which is all about why using events is bad idea, I think using both threads and events together ought to be practical. However, there's not a lot of language support for event processing in most languages. Threads are obviously supported more directly in every language since the purpose of compilers and interpreters is to turn code into something that executes with properly managed code continuations (eg with call frames etc).

stack ripping ¶

     Although it's too much to bite off and chew right now, I can always add similar direct support for events in Lathe, by compiling descriptions of data and events into a form managed nicely in event stacks with data continuations. That's just not very helpful in the short term, since it's general work with complexity similar to compiling code.

For these high-concurrency systems, event-based programming tends to obfuscate the control flow of the application. For instance, many event systems "call" a method in another module by sending an event and expect a "return" from that method via a similar event mechanism. In order to understand the application, the programmer must mentally match these call/return pairs, even when they are in different parts of the code. Furthermore, these call/return pairs often require the programmer to manually save and restore live state. This process, referred to as "stack ripping" by Adya et al. [1], is a major burden for programmers who wish to use event systems.

     In a language with Lisp style symbolic expression syntax, where the form of code and data is the same, it should be exceptionally easy to add compilation support for events in a way preventing the drawbacks noted above. So the same kind of language support used for code can avoid stack ripping for events. You could even aim to have your debugger address data flow of events as much as code flow of threads.

async ideas ¶

     I'm thinking about a new approach to async event queues, which I'll write about if something gels. I spent most of the weekend rounding up all the usual suspects, reading pages on various existing ideas for async optimization.

     Normally I never think about problems related to my day job on weekends, because it's my time and I have my own things to do. But in this case I also need to work out some approach to async event processing for lightweight processes in Lathe. So if I have a good idea, I can kill two birds with one stone.

     The problem at work is pretty conventional: just async i/o using something more general than posix aio, so it can be used for something else besides i/o. I'd do the simplest thing that can work, but a couple folks have half an expectation I might spawn a brilliant insight. (Which I wasn't planning. :-)

     On the home front, it seems like a good idea to fold async processing into Lathe as early as possible, before I commit to conventional methods in the first working Scheme. It would be neat if the first cut of basic Lisp had native async logic smoothly fitting C++ to high level language parts.

low hanging fruit ¶

     There's obviously low hanging fruit in the async space that's been ignored judging from the mess, lack of cohesion, and lack orthogonal mechanisms. I'm just saying I feel there's something there to find, though I have no idea what it is. So I expect thinking about it will pay off enough to be worth it.

coordinate systems ¶

     Since I was a teen I've used a coordinate system metaphor to explain how I find simpler patterns in complex systems. From a very early age I loved to watch complex moving patterns until I saw unifying structure. As a teen I noticed I was trying to make the pattern stop moving. And I did this by redefining the coordinate system I used to observe.

     In other words, I was trying to find rules that didn't change in patterns, so this part was constant even though the pattern changed: the governing rules didn't. Once I had a good set of predictive rules, I was able to see a dynamic pattern as statically reflecting the rule set. (Sure, I was an odd teen.)

     I only say this to explain what I look for now in async event systems. I want an unchanging description, even though events are flowing. And, I want invariant event rules to form a picture, so events can be explained by obvious diagrams. Tonight I have something plausible half cooked, but only half.

finding simplicity ¶

     I have a knack for seeing simpler systems hiding inside more complex ones, and it works in a completely non-linear, intuitive fashion, often below my conscious attention. I don't know how the process works — just that I get results. I ask myself questions and direct my attention in a logical manner, but the answer generation is often vaguely oracular, and stumps my attempts to watch or analyze.

     So to see a simpler way to organize async event systems, I need to pile up all the ways it's done and swirl them together with half-formed conjectures on alternative approaches. It helps to hold simultaneously true alternative descriptions until some subset of ideas call attention to themselves as more dense and cogent than less useful alternatives.

     Eventually I start to consider answers to questions in a space obvious, because the answers devolve from some mental model I constructed, which is often a gnarly graph.

net link down ¶

     I know I said I'd post more yesterday, but my link was down for the first time in weeks. I was totally groggy anyway, and sleep was a better use of time.

celtic chimps ¶

     Number one son said he knows what to get for my birthday after I laughed so long at a television ad's faked riverdancing chimpanzees. I said I doubted he had time to train them, but he claimed chimps riverdance spontaneously for the right music. Logical comeback.

unmuted criticism ¶

     Today my criticism of standard technology is rather unmuted. Normally I'm quite circumspect about saying what I really think, especially at work, where criticism is rarely useful out of the right context. Negativity just impairs morale unless action can be taken. So in my day jobs you never hear me say anything remotely like my notions below.

wish list ¶

     Before I rephrase what I told my coworker at lunch, first note I have a lot of other constraints which aren't going to be mentioned in detail today. Assume I want to effect my entire wish list, which is pretty long, and can only be done in some plausible order, if at all. In other words, your idea to "just use XYZ" for some XYZ is a stupid suggestion.

     Some of my wishes are about types of apps I'd like to see users get. That's only the tip of the iceberg. A much bigger piece under the waterline matters more, and can't be addressed with today's conventional toolset (which is mostly horrible kludges: more on that below).

day job toolsets ¶

     Toolsets I use at work are lame, and have been the last fifteen years. Have you used libraries in C++? The best of them are voodoo and witch doctor props. It takes forever to get things done in excellent quality. Basically, writing software is far more expensive than it should be.

     I should be able to get the same results faster. At times I get sub tasks done quickly when I'm allowed to use my own techniques — which I've been cooking the last 15 years and upgrading one recipe at a time. Often my day job optimizations are done by applying some of them.

     Now I just want to apply all of them and flesh out a toolset I can use to jumpstart all my future jobs. In other words, I want to embody a toolset I've been nurturing which I consider superior to any I've used before. But no one will let me deploy it in day jobs without an upscale demo.

     So I won't code web apps I mention using Java or whatever kludge you use for your project, because it won't improve future day jobs when technology isn't suitable.

kludges ¶

     Today I saw a nice quote on kludges (in the blog of Jacob Harris by way of news.ycombinator):

n. A clever programming trick intended to solve a particular nasty case in an expedient, if not clear, manner. Often used to repair bugs. Often involves ad-hockery and verges on being a crock. In fact, the TMRC Dictionary defined 'kludge' as "a crock that works".

     Note "crock" is generally shorthand for crock of [vulgarism] which is exactly right in this context. It's what I'll mean when I say kludge or crock when describing software with almost polite euphemism.

     I pick jobs with pretty clean code already. But still too much time goes into sanitization. Sometimes being the local plumber is a dirty job. Making things squeaky clean for others — especially customers — is tops. I'd like this to be cheaper and easier when pursuing fast and simple code.

     I don't think you can point me at infrastructure that's tops in speed and still quite simple, and yet flexible, clean, and safe. (Some things come close. But "simple" also includes transparently obvious to me down to assembler. Your idea of simple usually isn't the same as mine.)

cost reduction ¶

     Okay, let's get back to my wish list so I can replay part of my recent lunch conversation on web apps which surprised my coworker. I want to reduce costs for three or four users in specific use cases. I'm one of the users. I already said far too much above on reducing my day job costs.

     Who are the other users? Let's bullet list all users for proverbial scanning readers. Three are devs:

• Ð: myself as dev specifically (Ðev)
• ð: server developers generally (ðev)
• Δ: scripting app devs generally (Δev)
• α: app end users (novice web devs?)

     Note everyone in this list can write code, which means I don't think web apps are good enough unless α app end users can write code if they want, which is the element surprising my coworker the most.

     Since I play all the roles, my wish list amounts to scratching my own itch. Except I also want to scratch the itches of other people like me. I see my sons starting at α and climbing the ladder up to scripting as a Δev and then hacking the underlying system as a ðev.

     I want the same path open to others, and not just my sons. In one sense, I'm against a view of end users as marks to be bled dry for the livelihood of developers. I'm a capitalist who likes to make a living, so I don't want all software to be free. But artifical scarcity from making software very complex and expensive is morally repugnant to me.

     Anyway, I'd like to see more cottage industry software development by scripting Δevs making wares that can be customized by interested α app end users. Each step up in knowledge would involve increase in expertise and costs. But I'd like all roles to have lower costs.

moi as Ðev ¶

     I want to finish growing my þ runtime library to the point I can use it to jumpstart every C++ project I develop. This amounts to using a better C++ library when coding at the sort of agonizingly detailed low level mosts folks avoid. Using þ in all the other goals is a way to pursue this.

server ðevelopers ¶

     Writing servers is hard and debugging them is harder. But still harder is versioning servers and uprading them — live if possible. Versioning and upgrading are steps we rarely have time to pursue, because everything is too expensive.

     When I tell folks about this idea, they ask what I mean. I give examples as follows. First, suppose someone reports a problem with your server. Without shutting it down, you'd like to connect in development mode and explore alternatives in a separate namespace, where you can make changes that don't affect users who are currently connected and undisturbed by your research. Examining the state of the server would just be browsing a proc style file system exposed by http.

     If you fix a problem in your namespace found during your debugging session, you should be able to mount your change/fix in the namespace visible to all users, so the fix gets deployed without restarting the server and inconveniencing users. How would you like that? If the fix shows a regression, you could back it out by undoing the namespace change you made to expose the fix. The main idea: the cost of changing your mind should be low.

     Then there's debuggers, like gdb under Linux, which suck. Standard debugging tools are crocks. The mode you need to use for debugging with such tools is exotically different from normal execution. You cannot do it on a server running live with real users (unless you don't care what happens to them). But you should be able to do it. We don't question it because we're used to crappy tools. Instead, we should be able to debug any time, without putting the state of a server at risk from a debugger.

     There's a consistent principle behind my objections: modes are expensive because they constrain your choices and it costs to work around problems. You should be able to do anything at any time, without different phases in writing, compiling, building, debugging, deploying, etc. Connecting to your live server for debugging or analysis should be just like developing it on your local workstation when you want.

     Every server should already have an HTTP server built into it from day one. The web interface you use in development should be part of the same one present when deployed in the field, except as developer you'd use more raw parts of the namespace than end users. You should be able to inspect and change a server using REST and HTTP messages. We don't do this already only because our tools are crocks.

scripting Δevelopers ¶

     The behavior of a server in places should be driven by scripts in a dynamic language. Any developer, and potentially any end user with privilege, should able to see and/or edit dynamic language scripts in a server by simply browsing the right URL. Scripts should be able to invoke any server feature as long as permissions are good.

     You should be able to debug server scripts in a sandbox so you can't whack the server by silly scripting errors. The sandbox should operate in a separate namespace that doesn't affect the rest of world until you feel like mounting new script versions with more visibility. Resource exhaustion in space or cycles should drop you in the debugger, where the debugger is just another scripted server namespace.

     Obviously another user agent client can be better than a browser for this. But in principle any web browser ought to be sufficient, even if it sucks compared to a custom client.

     Writing some kinds of scripts ought to be so easy you don't need to be a rocket scientist to succeed. Anyone who can dabble in code should be able to play. Rank novices should get training wheels to prevent shooting off a foot. If your server is popular, there should be a thriving secondary market of cottage industry scripting support for your platform.

αpp end users ¶

     Web apps today suck because end users cannot program them. This is what I told my coworker the other day. I said web apps exist where users can upload pages, pictures, contact info, and lots of other kinds of data. But users can't control the server's behavior. They can't define new dynamic behavior for their pages.

     A user's local environment in a hosted environment should be something they can control in very broad in general terms, subject only to constraints imposed by an underlying system, such as throttling of space and cycles used by end user code. If I had an Aunt Betsy, she should be able to say when email should be sent and when web pages should change and how. Aunt Betsy or her proxy must be able to write scripts.

     What am I talking about? User customizable behavior. Not just customizable data. A hosted environment should be a virtual world an end user can control like a developer. An end user need only pick up the minimum skills, or borrow the skills of someone with more knowledge.

     Servers today would already do this if it was feasible because it's a fabulous selling line for users: do whatever you want. We all know doing whatever you want is possible, just not currently feasible because our infrastructure is crap. We can't do this today and ensure any kind of safety or control, because all layers of our software stacks are crawling with gotchas. Enabling such features is too expensive.

     Now you know why my coworker was surprised. He was thinking, "Wow, what if end users could make web apps do new things unplanned by server designers?"

time off ¶

     I haven't done any coding at home the past few days. Either I'm procrastinating, or I'm doing something useful as I ponder various copy-on-write tree approaches to symbol namespaces. Since I tell myself I'm just taking a break for the first time in months, I suspect it's procrastination.

open source ¶

     At lunch with a coworker today I explained what I was doing in my home open source project. (Note I haven't told you yet either, really, so maybe I'll go into detail here very soon.)

     My coworker said he always wondered why anyone ever did that instead of just putting all their time in work projects. I gather folks at work consider it a waste I ever spend my free time coding for joy instead of grinding away at the same narrow millstone I have at work. (Nights and weekends, too?)

     So I explained the use-cases I wanted for end user scripting development, kind of like HyperCard updated for the modern age, but with web pages replacing cards, and with scripts in Lisp, Smalltalk, and Python that can do anything including send email and modify web sites. Then he started to understand.

     My plans are exciting, if too much work, but totally beyond the scope of anything we'll ever do in a work context without a new startup, which I don't intend any time soon. He could see if this was interesting to me, work wasn't enough for me and there was a purpose to my research.

     Maybe tomorrow I'll summarize here what I told him. I found his surprise at various points very interesting, and it clarified how my ideas aren't as obvious to everyone as they are to me. That helps me see what I should emphasize for others.

     Disappointingly, he asked why didn't I just use existing virtual machines to bootstrap my desired application. I'm sure he meant the JVM in particular. I explained they didn't do what I wanted, and I wasn't as interested in particular end apps as how they get done, with infrastructure easy to repurpose.

writing specs ¶

     In my day job I'm writing specs right now, which is different, since most of the previous year was code and development for something that never had a spec before. They just had an existing version of code I figured out and redesigned once I got the hang of it. I'm not sure exactly what my role is right now.

     I gather the architects want me to temporarily play architect until I define the next refactored version of the central engine with a couple different well-defined interfaces, then I'll code the new version, with help if I want help. Sure, help is great: the more folks who know stuff, the better.

     Other teams will target the API while I code it, so that's why a spec is needed early. I'm actually a big believer in reducing job security with good docs. I think the code quality ends up a lot better when anyone touching the code knows the score, and I get tied down on one less thing.

     However, I don't think writing docs during the day with parts in very high generality (entropy and noise) is helping me nurture a coding frame of mind at night after I come home. I like writing though, and I write a lot of lucid explanation quickly. It doesn't take any effort, and I make everything sound obvious.

mad skittles ¶

     I learned a new phrase from my sons yesterday outside the grocery store, after elder son rolled the cart loose to a gentle, perfect stop at the car bumper. "Mad skills!" he called out.

     "Do kids bad at math shout that?" I asked.

     "No," One replied, then appealed to son two, "tell Dad about mad skills." Son two is better at defining terms.

     Two gave it a shot. "It's like when you try something random and it works out just right by accident."

     "Yeah," One chimed in agreement, "Then you shout: Mad skills! Mad skills!" Then he noted his younger brother was lucky like that all the time: other kids noticed it often.

     "That's not lucky," I suggested. "That's just being smart."

     "No, I think I'm just lucky," Two countered modestly. "It just works when I try things." Uh huh, sure. Then he giggled and added, "Instead of 'mad skills' you can say: Mad Skittles!"

     "Yeah, Mad Skittles!" One laughed.

     I think son two underestimates his manual dexterity and takes it for granted. I started being surprised by his coordination when he was only five. I didn't see the noise I expect in the way most kids move, making little clumsy errors. He just tends to do exactly the right thing without thinking. For example, he picked up basic juggling quickly.

freedom ¶

     One reason I like my current job is I'm free to code the stuff I describe on this site — at home — without any conflict. (You did realize this site only describes my part time home project, didn't you?) There's very little overlap in my day job and my home project.

     Aspects of what I want to do with Lathe (multiple language syntaxes, lightweight processes and cow for concurrency support, etc) are just slightly grandiose. If I worked at a company with plans that also sounded slightly grandiose — especially if similar — that would encroach on my freedom. Let's consider a hypothetical example to illustrate the problem.

     Suppose I worked for a company that wanted to emit Javascript for AJAX applications the way I intend to do with Lathe. Suddenly my Lathe research would look related to my day job. That would suck: instead of it obviously belonging to me, folks might think my employer had a finger in the pie. (If they used Python too, that would just make it worse when I get Lathe to work with Python.)

     My current employer has almost zero interest in dynamic languages, which is great in terms of no conflict of interest. I'd likely enjoy working in a shop with more dynamic languages in the mix, especially in web apps. But then I'd need some kind of official permission to make Lathe as open source. Getting official permission sucks. (Had to do it once.)

reflection ¶

     I just ran into one funny consequence of my mostly visual mind: I often don't remember what I've written. Writing always seems translation of what I'm thinking. (I used to laugh every time I read some cognitive psychologist say all thought is verbal. Uh, no it's not; often it's visual or simulation.) Words I use seldom stick; what I say aloud is often gone instantly.

     Anyway, I searched for web pages matching "lisp namespaces" and read a few, until I hit a page on LtU. I really liked the first sentence of the second paragraph below. Then I was surprised to see the name of the author: it was me two years ago. Funny.

     I should probably go fetch other things I wrote on LtU (and other places) and post them someplace around here for future reference, especially when the material has a bearing on what I discuss on this site.

     Okay, I did that: I started a new page to gather ltu† quotes and added a couple first items.

namespaces ¶

     I just spent the better part of ten days reading about namespaces and file systems, revisiting some of old storage work and language work. I was trying to form a loose top-down frame for namespaces in Lathe which did not conflict with where I'll go later with lightweight processes and other tactics inspired by Erlang and Inferno.

     This game is called "don't paint yourself into a corner" and it's the reason why I put time in top-down plans instead of bottom-up code alone. Most of my energy goes into bottom-up work.

epiphanies ¶

     I had a complex but rewarding insight last night, that I doubt I can explain now. Did you ever have a brilliant idea, and the next day you wondered why you were so excited? That's partly it, but I've also been thinking long on it now, and it's getting more familiar and mundane.

     I figured out a very loosely sketched spec I'll follow for lightweight processes and some features like pipes in app framework shells. Hey, I just realized I can have a "thorn shell."

     I won't explain the big idea tonight. Instead I'll talk about some related design context, with a side excursion on top-down vs bottom-up design.

     My excitement can't be explained because I was mentally simulating effects I wanted, and seeing I could get elegant untangled results for something that normally seems complex. Simulation doesn't translate well, since you can't share it. But it caused that goofy "I found a magic key!" feeling I hope comes back later when coding.

cow paths ¶

     I kept trying to visualize cow (copy-on-write) applied to shared process namespaces mounting new paths any place they want to shadow defaults and override base code with specialized replacements.

     Space cost for namespaces would only grow in proportion to variation. Coupled with a local virtual file system, a process can shadow parts of the world any way it likes via patching.

     Right about here I started seeing Plan9-ish continuous stream-based processes as co-resident alternatives to Erlang-ish discrete message-based processes. So you could use one or the other as fit best. With everything named by a path (in addition to other names) you can make a fully reflective system with clean cow patching support.

     Just to make things more complex, I tried to see VMS style file name versioning in the mix, but it hasn't elucidated anything yet. However, I could do content negotiation (alternative format choices) with file type suffixes in the name paths.

forest trees ¶

     Several metaphors come to mind when explaining ideas with a lot of interation between figure and ground, so context plays a big and confusing role. Threads, events, and asynchrony are easily forests obscured by the trees composing them.

     Asynchrony and threads are an emergent effect. You can't easily see a thread or process per se when looking at code. The relation of processes to each other is a contextual effect, like a forest emerging from the trees.

     Now I'll spare you the iceberg metaphor, since it only illustrates separation of top and bottom layers, and doesn't say much about how the division is structured. A cave metaphor is better to shape a code support discussion:

spelunking ¶

     A cave metaphor gives us inverted top-down pyramids from a ceiling (cf Stalactite) and a upright bottom-up pyramids from a floor (cf Stalagmite) Of course stalactites and stalagmites are both too skinny and round to be effective as code models. Imagine somewhat squat triangles instead: squat pyramids are stable.

     Your systems should have both top-down and bottom-up parts, each where it does the most good. They should meet narrowly in the middle, like ends of ceiling stalactites and floor stalagmites hitting each other. This organization is efficient, making the dominant end less bulky than it would be alone.

     If you focus entirely on one or the other — top or bottom — you get something unwieldy. Too much top-down design gives you fragile and squirrelly patterns of of baroque weirdness in slavish pursuit of options that veer more wildly as you approach the bottom. It's where really bad code thrives.

     But too much bottom-up focus makes you write things you don't need. This is what inspires the "you ain't gonna need it" admonition in agile software. As you build a pyramid higher, the base gets huge. You should only aim to get as high as the bottom of the top-down stalactites. Otherwise breadth-first costs will eat all your time. So build modest pyramids.

     The metaphor stretches further: first join a few stalactites and stalagmites, favoring pyramids looking like they'll connect, since that gives work value (backend support for frontend features). Don't do all floor or all ceiling first. The breeding pillars become a forest over time.

costumes ¶

     Later this month I'll tell a story about Halloween costumes in San Francisco three years ago. I've been saving it a while to be topical around now. I learned an interesting thing about poor perception in other folks that kept me thinking.

     Here's the short version: I had quite short hair then and I dressed as a sailor in quite realistic garb (as opposed to caricature) and went to a Halloween party. I was not perceived as in costume. I looked just like a middle-aged navy loser. I had numerous chats in which the same puzzled question was eventually asked in all seriousness: "You're not really a sailor, are you?" It was fascinating.

     I'll save a couple other interesting details for later, like my encounter with the Village People. Another theme: folks' tendency to assume — incorrectly — that I was gay. Straight women eventually but slowly realized I was straight.