Þ   briarpig  » log  » dec08

     Lately I write demos under thorn with a todo list using C++ under a BriarPig mu-babel license. The run and hex demos were done on 13apr2008; crc on 14apr2008; buf on 20apr2008; in on 27apr2008; ctype on 04may2008; out on 18may2008; slice on 25may2008; quote on 31may2008; escape on 31may2008; mutex on 14jun2008; rand on 16jun2008; stat on 17jun2008; primes on 19jun2008; list on 23jun2008; heap on 29jun2008; iter on 02jul2008 and 04jul2008; atomic on 06jul2008; node on 13jul2008; page on 23jul2008; hash on 27jul2008; book on 27jul2008; pile on 03aug2008; stack on 07aug2008; mu on 12aug2008; toy on 16aug2008; weight on 23aug2008; symbol on 02sep2008; imm on 04sep2008; gc on 06sep2008; map on 14sep2008; meter on 16sep2008; thread on 22oct2008; arc on 05nov2008; this menu links all demos:

     mu: toy, peg, imm, tag, box, symbol, token, number, bigint, class, method, reader, writer, eval, env, vm, gc, world, pcode, compiler, asm, lathe, lisp, smalltalk, design, weight, jar, card, harp, debug, profile

     thorn: todo, names, iovec, assert, log, run, hex, crc, buf, in, out, quote, escape, compare, file, deck, cow, arc, blob, tree, slice, rand, time, stat, heap, node, primes, page, book, pile, stack, atomic, lock, mutex, thread, map, list, iter, ctype

security ¶

     I don't work in security because I don't want to benefit from fear, because then marketing security is basically selling fear. Over the years I was pinged several times by folks doing security software in some form or another. I always asked what users were getting out of the software, and the answer was usually the same thing: partial peace of mind and little else — partial without guarantee of safety: only lower risks with the product.

     The business of protection has low appeal when it focuses on downsides at the expense of upsides. It's more fun to make things, and provide tools to others so they can make things, or benefit more from creations. Maximizing benefit of goodness rewards more than minimizing effects of badness; security feels like the latter.

     (I interact with security software only tangentially in network optimization. Usually there's a spot in a system where SSL hooks in; I just worry about effects of handshaking latency with SSL depending on granularity of content passed in buffers, and when they dispatch to consumers. Small granularity incremental dataflow isn't helpful to a batch process doing better with more context. My part of a pipeline can be held hostage to SSL behavior, so I have mildly negative associations with it.)

encrypted stores ¶

     "Should an abstract file api worry about encryption on disk?" Eli asked, with a worried expression.

     "Eh, no, not really," Wil dismissed. "Encryption can be done transparently closer to a medium. There's no need for a file api to know whether any encryption is involved. It's fine to write code assuming blocks are stored persistently in the clear, even if they aren't, because data is in clear by the time you see it in memory."

     "Then what's the point of storing data encrypted on disk anyway?" Eli wondered.

     "Well, if you're caching stuff you see in passing on disk," explained Wil, "that could expose sensitive data more than it was before, and that might not be okay to folks sending data — especially if potential lifetime is long. Better if a stolen disk appears to contain only noise."

     "Uh, I guess I'd rather ignore that," Eli said.

     "Luckily you can, when working at a file api layer," Wil assured. "You can treat encryption as a totally independent à la carte feature appearing somewhere down in your file code, or in interaction of file with disk."

     "So I can just blow it off?" Eli asked.

     "Sure," Wil replied. "It won't change your i/o code."

birthday season ¶

     It's birthday season. My sons and I all have our birthdays within nine days of one another. So, my holiday schedule is busier than yours.

     For example, this evening I ran the big birthday grocery shopping trip: everything the boys like to eat.

stay out of trouble ¶

     Yesterday I took my youngest son to a park, after first picking up his friend Ben, who is also thirteen. After a while as I drove along, Ben suddenly said, "Wait, internet!" Then he said, "Nevermind."

     I asked what Ben meant.

     "Unsecured wifi," Ben explained. "Back there."

     "Do you pick it up on your cell phone?" I asked.

     "No, my iTouch," he corrected.

     As I thought about him snapping off his "unsecured wifi" comment, I knew I was going to write about it.

     Finally I dropped them off, and as they opened the car doors I enjoined them, "Stay out of trouble."

     "That's what my mom said," Ben marveled, like this was an incredible coincidence, or a sign.

by not minding ¶

     "How do you learn to handle boredom?" Eli asked.

     Wil looked at Eli sharply. "Are you going to start asking me personal questions?"

     "Uh," Eli tried to find a reason. "If you don't mind."

     "I suppose next you're going to ask me how to handle being lonely?" Wil asked with a smile.

     "I haven't been lonely since I started hanging out here," Eli noticed. "Zé and Finch say interesting things. Most of my peers are dull as ditchwater."

     "Let's talk about your boredom then," Wil suggested. "What kind is it? Does listening to other folks seem like watching paint dry? Damps your bit rate?"

     "Yeah," Eli nodded. "I like people, but just thinking by myself usually seems more stimulating. Is it always going to be like that? How about you?"

     "It should get better later," Wil assured. "But only because you stop having new ideas at such an alarming rate, because you finish thinking through most of the consequences of things you already know. Also, you'll get better at thinking, whether or not anyone else is demanding your attention. So you don't have to turn it off, and contrast gets smaller between just thinking and talking."

     "Were you really bored when you were my age?" Eli asked. "I bet it was worse back then."

     "Acutely bored," Wil nodded vigorously. "I felt lack of stimulus as an almost painful sensation. Queueing and scheduling stimulus was my top priority. I researched so intensely in order to keep my hopper full. Avoiding boredom was more important to me than anything. I skipped opportunities of all kinds, including good ones with interesting people, just feed my intellectual binge."

     "You make it sound a like an addiction," Eli said carefully. "Is thinking addictive?"

     "Yes, if you get endorphins from it," Wil replied. "I got pretty high when I was working on a great idea that kept dishing up new insights. I was hooked. As a side effect, I didn't care what anyone else thought."

     "Did you treat others inconsiderately?" Eli asked.

     "No," Wil frowned. "Unless skipping flattery is judged inconsiderate, and I know folks who think so. Are you trying to ask if I had sociopathic tendencies?"

     "No!" Eli barked. "No criticism implied."

     Wil sighed. "I'm actually more moral than most people, not less, and I'm constrained rather more by my concept of goodness than most. You see, I internalized mores I was taught as a child, and thought them through, making them knit consistently. As a result, I feel it's my duty to guide others away from making choices causing loss by mistake, miscalculation, or unplanned consequence. So I can't trick others, or even persuade them, to benefit myself at another's expense."

     "Wow," Eli's eyes got big. "You only get ahead in win-win games? Those pickings are kinda slim. Most people think it's okay to screw someone else as long as they say yes — each person is responsible for taking care of themselves, and then it's: finders keepers, losers weepers."

     "Yes, well my ethics aren't the best way to get ahead, but they do wonders for self respect," Wil said. "And besides, my frontal lobe whips me with regret if I let someone do themselves harm I could have stopped. I'm one of those people with natural disinclination to cheat."

     "You should have that looked at," Eli joked. "Maybe you can get permission from a shrink to screw anybody as long as it's just business."

     "Your father Ulf has more than enough of that to go around," Wil observed, watching Eli to see if this was a sensitive point.

     Eli made an easy gesture. "Yeah, well, you can't pick your father. Or your friend's nose for that matter."

     Wil winced. "Your inner 13-year-old showed through for an instant. You should have that looked at. Maybe you can get a shrink to help with your boredom, too."

     "I see you have no problem with teasing people," Eli noted, with a twinkle in his eye.

     Wil nodded gently, only half listening. "Anyway, boredom is a kind of signal telling you to use your time more productively. Maybe you fell for someone's scam — 'Just do X, and after you pay dues you'll get yours.' School is a bit like that, but you dropped out, didn't you?"

     "Yes, but I guess I was expecting more," Eli considered. "Do you think my standards are too high?"

     "Everyone's standards are too high," Wil said. "What do you think happens after folks watch highly unrealistic movies — one after the other — about adventure, thrills, love, reward, purpose, acclaim, luck, and destiny?"

     "You left out zombies," Eli noted with a grin.

     "Subcategory under thrills," Wil countered.

     "Maybe you can talk to the director about moving up the shooting schedule to the part where I have a blast," Eli suggested, with a wry attitude.

     "Talk to Zé," Wil chirped. "He'll have you in zombies up to your eyes so fast, your meter will tilt."

     "You mean like a twilight zone experience?" Eli asked. "Or a training exercise?"

     "What's the difference?" Wil shrugged.

coding again ¶

     I resumed my latest rev of file i/o using a page cache. I should have time to get mostly through it during this next week of holiday vacation. (The two weeks I'm taking now is the longest vacation I've taken in the last five years. A week was the longest before this.)

     However, the names of my C++ classes are somewhat obnoxious so far, because there's so many classes and the scheme I use — composing class names as acronyms using mainly one letter elements — generates many barely different class names. But I know my earlier revisions with long style class names were no better. You can't figure out out how the long names relate to each other, except by memorizing a class graph.

     Anyway, the file demo might look like obfuscated code when it's done. You'll be tempted to think this would not be true if I used long class names composed of concatenated words. But that would be just as obfuscated, and longer as well. I think you have to rename all code in a body of complex parts to finally feel good about the names. But it's mainly from familiarizing yourself with all parts.

file definition ¶

     "How did we get this far without defining what a file is?" Wil asked Eli. "How should we define a file?"

     "I know what a file is," Eli shrugged. "It's one of those thingies I see in the file system. I guess I think of a file as a document. Isn't that good enough?"

     "Not really," Wil shook his head. "Because we need to define file so the definition can be true for something used via a file api, and we already established such an object need not actually be a file in the file system."

     "Oh that's right," Eli remembered. "So for example, we can't include a pathname in the definition of a file — that's something in a file system, but not an attribute of a file. Are file names just keys in some index?"

     "That's right," Wil agreed. "The ypf0 api shown treats only file content as characterizing a file. So a file is mainly a sequence of bytes — one that can be written."

     "If a file is not readonly —" Eli amended. "Mutable files can be written. What defines the length of a file?"

     "The length of a file is the number of bytes in the sequence," Wil said. "And this length is the same as eof, end of file, when eof is the offset to a file's end, starting with an offset of zero at the first byte."

     "Logical or physical end of file?" Eli wondered.

     "Logical eof," Wil replied. "Since you can't read bytes beyond leof, logical end of file, that's where a file sequence ends. Physical eof, when different, is a resource hint that influences allocation, but not the sequence."

     "And leof changes... when?" Eli asked. "When you write through leof, or when you explicitly set minimum leof to a new highwater mark?"

     "Yes, using a ypf0 abstract base api," Wil confirmed, "you can only change the leof by writing through the old leof with new content, or by setting the minimum leof to a bigger offset. The starting leof is whatever was in a file when opened, however a subclass is opened."

     "Say," Eli looked thoughtful. "When you set minimum leof to a new highwater mark, what values are in the new bytes at the end of the sequence?"

     "Unspecified by ypf0," Wil explained. "But you might arrange a contract with a specific subclass to control what goes in new bytes added by increasing eof."

     "Do you want zero in new bytes?" Eli asked.

     "Zero sounds good," Wil replied. "But you might not have efficient means to provide or specify zero in new file bytes. In sample code I'll show later, I let you provide a byte sequence to use as a source when extend eof by writing to the end. That's one way."

     "Is our definition done?" Eli prompted. "What else is there to define about our files?"

     "We can also define how file content is sampled," Wil said. "Basically, you can only see bytes in a file's sequence by reading bytes into one more more contiguous buffers, where each buffer is an iovec."

     "Observational granularity is in iovecs?" Eli restated. "Does that mean we should look at a file as really a sequence of iovecs, and not a sequence of bytes?"

     "Close," Wil squinted. "A sequence of bytes is a logical model, but a sequence of iovecs is a physical model. For all you can tell, code for a file subclass might store bytes in heap allocated iovecs."

     "Except you can overlap iovecs when you read or write," Eli noticed. "I guess the ypf0 api is a copy-based model. It looks like copying content from source iovecs to destination iovecs, in both reads and writes."

     Wil pushed out his lower lip while thinking. "Sure," he said finally. "You can look at a file's internal model as a sequence of iovecs, some of which are disk blocks, while other iovecs might be cached in memory."

online sites ¶

     Other than LinkedIn, I don't network. I'd skip LinkedIn too, but it's been instrumental in routing professional contacts to me. Job feelers keep the sails taut and things running smoothly. But I've no interest in what others say about themselves and others online: dull gossip.

     If you have an online service, most likely I've never used it and never will. For example, I never buy anything online, so any site selling stuff is totally off my radar. I visit book stores in the flesh. I have never used Amazon. I'm completely uninterested in commerce.

     Someday I might buy myself an iPod, and then I might download music. But I have never downloaded music once yet, so far, because I listen to very little music. The only reason I buy compact disks is to pipe background sound into headphones to drown out background noise. I lack interest in music per se; my taste in music is tenuous.

     For years I aimed to foster computing systems to help revolutionize personal communication, because my main motive was routing around mediocre folks in power who acted like bottlenecks. It grated me that dumb folks exercised power by virtue of getting in the way. I was sure letting everyone form their own cheap alternate network pathways would be better. I hoped folks would go about more interesting business once empowered.

     I suppose things are better than they used to be. Now gross inefficiency of credentials and old boy networks isn't as big a problem as it used to be. Anyway, there's no need to remove blockages now; it's done. But most channels are gamed one way or another. And utility of what folks pursue online is very unclear to me. Now instead of the "me" generation we have something like the "us" generation: self absorption in groups and tribes.

     I still hope for more than bread and circuses. But I underestimated the role of ego in human nature — the way world views are resiliently constructed to flatter the self, with contrary evidence ignored when it undermines self image, or merely lacks strong self confirmation. Folks can't even see their own tribes clearly when the main role of a tribe is to prop up self image.

memory hierarchy ¶

     "Solid state drives are cool," Eli said out of nowhere. "Don't you think so?"

     "Uh, sure, very cool," Wil granted. "Do SSD's relate to our look at file api designs?"

     "Well, don't they?" Eli prompted. "I was hoping fancy i/o efforts would no longer be necessary with fast drives. Maybe caching and btrees no longer matter."

     "Like everything else, necessary is relative," Wil observed. "We'll still have memory hierarchy as long as drives aren't as fast as processor caches. So clever memory use matters even with really fast drives."

     "Some of my friends talk like I can stop ignoring latency of drives when data is on SSD," Eli said.

     "Really?" Wil chuckled. "Do those same friends ignore L1 and L2 caches when coding? As if main memory has zero latency? That's not good practice."

     "Then what's the big deal with SSD's?" Eli groused. "When they get cheap, what's the great benefit?"

     "The one I like most is high iops — i/o operations per second," Wil explained. "In some apps, one resource bottleneck is iops. With spinning disks, you might need a lot of drives to get available iops high enough. And disks generate a lot of heat, which is hard to dissipate; packing enough disks in a box can be awkward."

     "This sounds abstract to me," Eli chewed his lip. "Are you saying limited iops might cap some maximum used bandwidth in some system?"

     "Yeah," Wil nodded. "If your net inputs and outputs have targets you must reach, this means you can only afford to look at so much data in so many places in secondary storage before throttling throughput, if it affects your outputs. Your eyelids are drooping, Eli."

     Eli's eyes closed and he made a brief snoring sound before quickly opening his eyes to dodge Wil's poking finger with a mischievous smile.

     "Insolent pup," Wil muttered playfully.

     "Needs work before you take it on the lecture circuit," teased Eli. "Pardon me, I have trouble caring."

     "Then don't sweat it," Wil suggested. "In the meantime, I don't see how a file api is affected by using SSD or not. An api abstracts out the medium."

     "Then how does memory hierarchy matter then?" wondered Eli. "Is the api affected?"

     "On the caller side where buffers are passed, memory is going to touch cache lines," Wil explained. "So api exposes detail callers care about: cache lines touched."

     "Is that it?" Eil asked.

     "Not if you consider implementation," Wil replied. "You want a file subclass to touch memory as efficiently as possible. And the faster a medium gets — lower latency for disk — the more overhead counts in code doing i/o. So you don't want an api to require complex calculation, assuming it's always hidden by disk cost."

     "You're a grind about costs, you know?" Eil sighed.

     Wil shrugged. "You asked, I answered."

     Eli studied Wil to see if more was left on the table. "Okay, what's an especially dumb way to waste time in a file subclass? Is it just code quality?"

     "Uh," Wil paused to access an idea, which he found quickly. "Let's say you queue i/o requests to someone else to handle, and this causes a context switch."

     "How many context switches can you do a second?" Eli asked. "I smell a problem."

     "That's exactly the right line of thinking," Wil smiled. "If context switch is really slow for some reason, you can add major latency that isn't very obvious just looking at code. Polling interval can kill you."

     "Polling already has a bad reputation," Eil said. "You're going to suggest I measure latency directly, right?"

     "Good idea," Wil approved. "Never assume cost is negligible just because it's what you hope."

jingle bells ¶

     More file api material is coming, just not today. I could have made myself write a few hours. But there's little reason to rush; quality is often better when writing only if it seems the most interesting thing to do.

     In addition to sampling some of a House marathon, I started re-reading an anthology of John Varley short stories, laced with rather good contextual history about the author's past and circumstances related to stories in the volume. I read this collection before, of course, but I re-read old favorites more often than I start new material these days. This is largely because I would upgrade Sturgeon's Law (yes, I know it's actual history, and what it originally meant) to read "99% of everything is crap" and not just ninety percent.

     I really like Varley's writing, and I can spend some of my attention studying how the illusion is constructed on repeat readings. I put Varley up in the top six or seven best science fiction writers ever — although the series he started lately, beginning with Red Thunder, is only satisfactory reading. His most amazing novels are Steel Beach and Golden Globe, both of which stand up to my maximum number of re-readings — I'm probably up to about six each by now, spread out over several years. I re-read them again every time I can no longer already remember the next paragraph's content exactly.

     Reading Varley is partly for pleasure, and partly for soaking in the best quality writing style I like, hoping to eventually learn something useful in craftsmanship.

research ¶

     I seldom Google for technical information because most of the problems I solve have no extant problem definitions, that I can find. When it's my design and my code, I can't find the answer online. When it's a teammate's design or code, there's still no answer online. If only life were so simple that error messages gave online clues.

     Most error messages I see are ones I generated. These lead directly to the problem in code, if any, or a diagnosis of expected behavior.

     The hardest problem to solve is also the most common one: something doesn't happen. So there's nothing you can search for to explain this, because the symptom is that nothing happens instead of what you expected. There's no material clue to Google.

     This is partly an artifact of a deeply layered async architecture. I get asked about this symptom a lot in my day job. Folks come and tell me Z didn't happen as expected. I tell them sorry, the only thing you can do is find out what happened instead. To explain one negative you must often trace a slew of potentially irrelevant positives. To find a problem, you must find a contradiction: Z should have happened because Y did happen, and Y always leads to Z (you hope). Following the back trail of internal events is often painful because we only do just-in-time logging or tracing, added on need.

     I sometimes Google for simple versions of code I want to use as stubs pending improved replacements. For example, I recently searched online for a 128-bit crc implementation, because I really didn't want to handroll one myself. (The idea was really only to avoid a library dependency in a prototype just to get a 128-bit hash.) Oddly enough, I couldn't find one, so I will need to roll one myself (by using a couple 64-bit crc polynomials — normal and reflected forms of a 64-bit standard).

     Also, Google has gotten much less useful in the last year or two because search terms are no longer strictly honored. I would be able to find exactly what I wanted to know if Google would just honor each and every term I say must appear in a page. Instead, it seems clear Google drops exactly those terms that would increase uniqueness enough to answer my questions by narrowing to just a few hits, or to no hits. I assumed this was to make it easier to serve up ads on popular terms, which Google feels is the only thing I should be seeking. It's not in Google's interest to serve zero hits because that serves no ads, even if zero hits is exactly the correct answer I need to see. Instead, Google spams me with nonsense, missing terms. But it's hard to complain about bad free service: beggars can't be choosers. (A phrase my elders irritatingly repeated often when I was a boy.)

     When all search terms are strictly honored, I construct better searches. I learned relevant skills in college library stacks in the 70's and 80's: how to follow trails of associations, like following veins of ore in a mine. A few years ago I could use Google to quickly find relevant data on things I investigated. Folks watching me asked two things: How did I know what to search for? And how did I quickly know which pages to skip in returned results? In both cases the answer was that I had spent hundreds of hours of deliberate practice learning to recognize drivel on sight, due to some internalized Bayesian model, from reading a lot of perfectly awful academic material related to my research in knowledge representation.

     My guess is, not many folks perceive degradation in Google's search results, because they didn't see quality of signal in the first place. (Note if you like Google's results as they work now, I don't have any argument with you. Maybe you'd rather have terms removed when they prune results a lot; maybe you prefer quantity of results.)

holidays ¶

     I don't get lonely at the holidays, though I used to. I usually have my sons part of the time each holiday, and that's all I need. But I do get slightly lonely any time they're both gone for a couple days in a row. It has nothing to do with the holidays.

     I often get a certain amount of flirtation from women, which I mostly ignore because I assume poor intentions by default, and because I'm not attracted to women young enough to be my daughter, or small enough for me to pick up over my head. Small and young women are main sources of unusual attention; it's somewhat offensive to feel attractive only because you are big, or solvent.

     Any age under 35 is too young, any weight under 140 is too small, and any height under 65 inches is too short (more than five inches difference in height is too much). Older, healthier, taller women mostly ignore me because I'm too poor a prize for their standards — and they often (act as though they) see themselves in a way I find ugly in itself: precious. And thus, I only rarely see a woman who seems right: compatible and nice personality.

     I assume all such women are married. If a woman in her 40's had never been married, I would assume there was something wrong with her. So you see, I almost defined it to be impossible that a woman I was attracted to was available and not a nutcase, without having gotten to the problem of women seeking financial security.

     A friend of mine is married, and is in the right age, weight, and height range, and also has a great personality. I told her I assumed anyone near my age who had never been married was likely not okay in personality. She said she fully agreed, and then asked me with an edged inflection, "What about divorced women?"

     Of course, I didn't touch the idea she might end up in that classification. I was afraid it might be the start of a discussion of trouble at home.

     I think I said, "Maybe." That was my first thought several years ago. My last girlfriend was exactly my age and divorced, whose oldest son was older than my ex's boyfriend of the last six years. In many ways we got along great, but a height difference of nine inches was clearly a problem big enough to go on my list of things to think about. (You can't even go for walks without awkward differences in stamina.)

     Our primary problem was that I still had kids at home, and hers were at college or out of college. That my sons were my first priority created a rift we couldn't easily bridge. She had reached the stage of her life where she was her top priority, and her partner had to yield.

     I developed a theory, which I haven't tested of course, that maybe all women are unable to tolerate another woman's children. Which, if true, would mean no woman who sees me as single would actually tolerate the life I lead, working full time and spending at least half my days with my sons. (I actually spend at least one extra night a week with them.)

     In effect, I'm screwed because I didn't have my kids young enough. Another relationship won't work until my youngest finishes high school in five more years.

     This is something of a double whammy in the sense I was married for years before a first child, so I'm on the hook for child support for so many years after my ex dumped me for a boy less than half her age.

     I neither try nor think about trying to meet women. In fact, I rarely speak to anyone outside work, and this makes it almost certain I can't strike up an acquaintance with a stranger, because all women angle to be chased, because that's how it works. (I could write a pretty long essay about this, and it's never going to fit in the budget size I allow for non-professional content, so I doubt you'll ever see it.) I don't have any emotions about isolation — I simply have low expectations anyone else acts like a human being unlaced with dramatic selfishness.

     Last weekend I almost wrote a lightly fictionalized version of something that happened to me at the grocery store. Finch and Zé would be working late at Wil's place when Wil arrives with groceries picked up at 10pm, after a gym workout. Finch would note Wil's choice of groceries this time practically screams, "I'm single!" because no one who cooked at home with a partner would ever eat like that. And then Wil would report how this affected a couple of young female clerks at the store.

     When a woman speaks to the air at large about her availability, it seems slightly desparate — because men are supposed to chase — especially when proclaiming youth and boundless capacity for love. The youngest, in her 20's, said she planned to fall in love this January and February, circumstances willing, and that she fell in love quickly. Her life was going to waste so far.

     I say to the air at large because the clerks were speaking (mostly) to each other, but clearly had already heard all this from each other before. And yet each of them managed to avoid looking me in the eye before, during or after the most interesting remarks, which seemed an odd way to go about it.

     When I failed to start chatting them up, the other one ringing me up asked me whether I thought everyone deserved somebody. I agreed but not in a way that would expand the conversation. So she could only smile charmingly and watch my face to see if I had any other conversational gambits. But I only smiled.

     In my experience, no woman ever thinks of anything interesting to say to a man, after years spent deflecting lines tossed constantly in her direction by guys who chat up every woman he sees.

     One clerk was too young and too small; the other was too young and too short. And worse, there was something about the encounter that felt practiced, like they were dredging waters for every possible candidate who passed by. I felt like I had qualified for a survey by virtue of wearing gym clothes and leather jacket, paying with cash, and looking like a professional (who must not be too bright if still single at my age).

     The story seemed worth relating because it was much odder than usual. I've been avoiding that store since because I don't want a repeat.

low road stories ¶

     I pruned another dialog I wrote yesterday because it was ultimately mean-spirited. I can't tell you what it was about without implicitly making fun of the same party, and that's exactly what I decided could not stay. My headline was "repulsive codebases" which I trust conveys little.

     Let's start with a little meta commentary on dialogs.

perspectives ¶

     "Writing another dialog?" Eli asked Zé. "Which character represents you? The zany, creative one?"

     "Actually," Zé looked up from his laptop to reply, "none of them represent me closely. If you wrote more fiction, you'd see how characters never end up much like yourself, even if you had that in mind in the first place."

     "Why?" Eli puzzled. "Seems like it would be simple. Just give one character your point of view."

     "Okay," Zé granted. "What if you wrote a story with a character just like you in it — what would happen?"

     "That would be boring," Eli realized. "I'm not very interesting. At least, I don't think so."

     "There you go," Zé gestured. "Fidelity to reality is boring. So characters are tools to amplify a way of seeing things, just so you can manifest ideas as interactions."

     Eli looked dubious. "You mean the twenty-year-old clueless newbie in your last dialog wasn't intended to be me? You made him seem both bright and naive at once. You named him Eric, and your names usually mean something obvious."

     "So you assumed he was Viking?" Zé suggested with raised eyebrows. "I didn't model that character on you — as far as you know," Zé said with an odd inflection while casting his eyes upward in a parody of evasive behavior.

     "Yeah, yeah," Eli half-smiled. "So what are you writing about now? Any characters have a name starting with Z? And is it you, or just a coincidental similarity?"

     "Zelig is giving Eric's father Ulrich a hard time," Zé explained. "Ulrich just found out Zelig's secret project is raising a bunch of Adolf Hitler clones down in Brazil."

     Eli put his face in his hands. "Sometimes I find your humor in very bad taste," he mumbled into his palms.

     "I think Zelig's fake German accent is funny," Zé added. "Do you think he should lose the white lab coat?"

     "Okay," Eli looked back up again. "What's the purpose of having a character like that?"

     "Personally," Zé considered. "I think an over-the-top goofball makes other characters seem more natural, relatively speaking. Even though all of them are contrived."

     Eli mulled that over. "How about the Woody McFly character from your Back to the Future ripoff — the Irish mesomorph from Iowa. Is he not supposed to be Wil?"

     "Smart and fair, but rude guys are fun," Zé said. "I like the irony of good guys who get screwed, especially if they're gracious and bent out of shape, both at once. Are you saying you don't like plucky but flawed underdogs?"

     "I don't like how Woody never wins in your stories," Eli complained. "Despite what my father says, I don't think honest nice guys should always finish last. Why can't he have a bit of good luck? Make all his efforts worthwhile?"

     "What, and live happily ever after?" Zé looked aghast. "G rated stories are for kids. Adult fare has complexity and a lot more sadness in it than you might like. Sorry."

     Eli stood up abruptly. "I need a dose of escapism," he blurted. "Video games, yeah. See ya, Zé."

     Zé smiled slightly as Eli bolted off. "My work here is done," he said to himself sotto voce.

standard api ¶

     "Are you advocating a standard file api?" Eli asked.

     "Uh, not exactly," Wil countered. "I'm advocating an api that's almost standard, where you change nothing you plan to keep the same when comparing code alternatives."

     "Okay, I need an example," Eli begged.

     "Sure," Wil nodded. "Let's say your file needs a write() method, and you're working on Linux or MacOS."

     "What if I'm working on Windows?" Eli prodded.

     "Not my problem," Wil shrugged. "I'm sure you can get docs easily for POSIX api on Windows, too. Anyway, go to the command line and type man 2 write to get the man page for write(). You'll probably get something like this."

     "Yes, but you changed arg names a little," Eli noted.

     "Whatever," Wil waved. "Now starting from this, I'll begin an abstract file api with similar methods. But of course I'll make changes, and we can discuss any part you want."

     "Okay, what did I change?" Wil prompted. "I have a good reason for each change. Obviously, you want to know why I didn't include write() despite starting there."

     "You dropped write(), merged together pwrite() and writev(), dropped struct from the declaration of iov, and replaced standard off_t with an explicit 64-bit signed integer named i_t," Eli read from the code. "Also, you dropped the first file descriptor arg, because it's replaced by this. Did you drop write() because it's not thread safe?"

     "Bingo," Wil congratulated. "I use only signatures taking an explicit file offset, because I don't want an implicit file position which changes as a side effect of reading or writing. An implicit file position decreases potential concurrency, because there's no reason why you can't do i/o to different spots in one file at the same time."

     "As long as there's no overlap," Eli amended. "I guess users of a file must negotiate to avoid conflict in overlapping ranges somehow? How do you do that?"

     "For range mutex I pick a page size like 32K and use an array of yrwl read/write locks (cf ») covering the file's address space in a striped fashion," Wil explained. "You're sorry you asked, right? Let's talk about that some other time."

     Eli slowly unclenched his teeth and sighed. "Let's not worry about threads right now. Except ... don't we have to choose between threads and an async api?"

     "Yes," Wil paused for effect. "This api is synchronous, just like the methods on the write() man page. So if you want to do concurrent i/o instead of blocking on each read and write call, then you'll need threads and a thread pool to keep your CPU busy when file i/o is pending."

     "An async api is more complex?" Eli guessed.

     "Oh yeah," Wil agreed. "And async is mandatory when you can't have multiple threads and you can't block for i/o. But it's easier to work out a totally sync api like this one first. Then you can worry about extra trouble for async as a separate issue later. It would just swamp us now."

     "Heh, heh," Eli sighed nervously. "We squared away the reason why you dropped write(), but why did you replace off_t with a typedef for an explicitly 64-bit ypf0::i_t? What size is off_t?"

     "On Linux the size of off_t depends on what you have defined and what headers you include," Wil explained. "It might be either 64-bit or 32-bit, depending. But I hate having my api change as a side effect of code outside my headers. I used 64-bit here because lately I work with large files."

     "Does it bother you to use iovec then?" Eli asked. "It looks like you were able to drop struct because in C++ that keyword is optional? You can use iovec as a class name?"

     "That's right," Wil nodded. "Type iovec is a public class declared like in sys/uio.h."

     "But I've worked in systems where folks redefine iovec," Wil warned. "You can't stop them."

     "Ew," Eli wrinkled his nose. "And who owns the memory to which iov_base points? Unspecified?"

     "That's right," Wil agreed. "Neither pwrite() nor pwritev() care who owns the memory or how long it lives, as long as it lives as long as the duration of the call. This is one of the contracts we're borrowing from the definition."

api contracts ¶

     "Contracts?" Eli echoed. "You mean implicit or explicit expectations of how the api must behave? What's required of the caller and the callee?"

     "That's exactly right," Wil nodded. "And re-using old contracts is the main reason the api of ypf0 has methods intended to work like those in Unix man page. Otherwise I'd have to write a long and laborious description of exactly what the methods assume. Instead I can be lazy."

     "Does that mean the return values and error conditions work the same way, too?" Eli asked. "What about errno? Do you return error codes in errno? That's not thread-safe either. Why is that okay? I don't like that," Eli insisted.

     "Uh, yes, the api returns error codes in errno," Wil warned. "The idea is to act just like system calls without adding gratuitous api incompatibility. Obviously you can change the api to pass error codes as an argument, for example. But then it would be harder to swap with real system calls."

     "Okay," Eli mused slowly. "And I guess I normally never worry about races in reading errno in system calls. Maybe if I have so many errors so quickly I have races when using errno, I'll have bigger problems than losing error codes."

     "We're just trying to minimize how much api we invent," suggested Wil. "We can always make a better api later if you don't like this one."

     "Why did you feel free to merge pwrite() and writev()?" Eli asked. "And come to think of it, why doesn't Linux support native pwritev()? How interesting."

     "You can always code pwritev() as a sequence of pwrite() calls," noted Wil. "Even if this is stupidly inefficient because multiple system calls might occur. Every database vendor wants to pwritev() in the api for reasons you'd expect. I don't know why it's not already standard."

read api ¶

     "Why does ypf0 only have write methods?" Eli asked. "What about reading? Go to the man page for read()?"

     "Yes," Wil confirmed. "Execute man 2 read for this."

     "Then we extend the ypf0 api as follows," Wil said.

     "That was easy," Eli approved. "Is the ypf0 api done now? Or is something missing? Say, how do I find out how long the file is? Is there standard api for that?"

eof api ¶

     "Yes," Wil agreed. "We need something to get and set a file's eof offset. But the api I added to ypf0 distinguishes logical and physical end-of-file. This makes it easier to support a page cache later. But this might look weird."

     "Okay," Eli grimaced. "Now it looks complex. Is that really necessary? And what does it mean to set the physical eof separately from the logical eof?"

     "Setting the physical eof is like setting capacity in a C++ STL collection," Wil explained. "It allocates space but doesn't use it. Used space ends at the logical eof. When I subclass this file api to use a page cache, this allows me to make a file's size an even multiple of the page size, without forcing page aligned content."

     "Why would that matter?" asked Eli. "What would it look like in a test? Do you preserve eof semantics?"

     "I can write a pseudo random unit test performing random read and write i/o on two files," replied Wil. "If two implementations have identical behavior, if I perform the same sequence of reads and writes, then everything read from file A should match everything read from file B. And the eof of each file should remain identical at all times."

     "So you can compare cached and non-cached files?" Eli guessed. "To show caching changes only speed?"

     "That's right," Wil confirmed. "By setting physical eof in a page-cached file to a multiple of the page size, I can avoid confusing a cache about file space that exists, while still preserving a record of the highest logical offset written by explicit api calls."

     "Why do the eof setters only specify minimum size?" Eli asked. "Are you unable to truncate a file?"

     "Not with this api," Wil shook his head. "A subclass might add methods allowing truncation, but the base api doesn't insist it be possible in all subclasses."

memory mapping ¶

     "What if I wanted to memory map a file?" Eli asked. "How does this ypf0 api help? It doesn't look like it's possible to memory map — isn't that a problem?"

     "You can still do pseudo memory mapping," Wil countered. "And this ypf0 is all that need be supported on a file api to let you do it. Basically, a subclass using a page cache can let you access pages directly."

     "You mean operating system VM pages?" Eli asked, confused. "I don't get it."

     "No, buffer 'pages' — I usually use 32K or 64K pages," Wil explained. "Later I'll show you the api. Basically you can call a method that returns an iovec corresponding to a range requested, where the iovec contains pages from the cache. You need to say whether you're reading or writing so page dirty status can be maintained."

     "Huh," Eli considered. "Normal OS memory mapping gives me what looks like contiguous memory. An iovec array of pages would be inferior, wouldn't it?"

     "Not especially," Wil shook his head. "You end up having to write all your code around universal use of iovec arrays anyway. Lack of contiguity everywhere then just becomes consistency. I don't mind myself."

     "Does this approach give you some control over when paging i/o occurs?" Eli asked. "The bdflush daemon on Linux kills me when I memory map files and write a lot. Can you control when the page cache writes dirty pages back to the disk? That would be nice."

     "Yes, you can control when writes occur," Wil nodded. "But it amounts to a complex page cache feature, since scheduling iops is a big headache. Basically, you want writes to occur when the disk is not otherwise under high demand. You need either extra threads or processes to watch how busy i/o seems and balance memory pressure and traffic demands."

     "Sounds complex," Eli guessed.

     "Maybe about as complex as the rest of a page cache," chimed Wil. "Scheduling is a big, hairy feature."

     "I don't suppose you were going to do a demo for that soon," Eli picked at his teeth.

     "No, I wasn't planning to," Wil said. "I can probably do the page cache sometime — if you don't mind the synchronous api using a thread pool approach."

     "That's not how your day job does things right now, is it?" Eli wondered. "Is it okay for you to publish code like that? No trade secrets in there?"

     "Uh, we do it another way," Wil said. "I'm not at liberty to say. But one of my page caches was used to prototype once, because it showed things we could expect. Don't worry, it won't be a state of the art page cache — not quite. No trade secrets; just paged i/o that's pretty old now."

file blobs ¶

     "Isn't that how your IronDoc database did i/o?" Eli recalled. "Can you show a verison of blobs living in a page-cached file? I might be able to use that in personal app dev."

     "Yes, on all counts," Wil confirmed. "This sort of paged file infrastructure was used in IronDoc's structured storage system, and blobs were basically pseudo file streams stored inside. When I started ramping up a blob demo some time back, I reworked the ypf0 to use as the base class for blobs."

     "You mean blobs are files?" asked Eli. "That's a little confusing. I thought blobs would have their own api."

     "Well, a blob is like a file in the sense it can support the same abstract file api — one like ypf0 for example," Wil explained. "But blobs can have a little more standard api of their own. For example, blobs can also permit insert and delete in the middle of a stream, unlike files, with an expectation of O(logN) performance. So the abstract blob api is a subclass of ypf0 adding more methods."

     "But wait," Eli considered. "If blobs are not actually files, how can I use blobs as files and then use standard tools that expect files in the file system?"

actual vs api ¶

     "I think you're conflating unrelated things," Wil warned. "The ypf0 abstract file api does not promise it works on a file — it works on an api resembling a file when used in your code. Not the same thing. Having the final persistent format actually be a file in the file system is a new and separate constraint."

     "So if I need an actual file, I just make sure my file subclass writes to an actual file, okay," Eli restated. "But if I have code reading and writing to files, I can also read and write other things that pretend to be files?"

     "Yes, pretense might be a good way to look at it," encouraged Wil. "Other objects can pretend to be files to leverage common code, but it doesn't magically give such objects all attributes of actual files."

     "So when I have bugs caused by thinking all my blobs are actually files, I can submit bug reports and you'll fix it?" Eli joked with a smile.

     "Don't make me slap you," Wil laughed.

open and close api ¶

     "Okay, that helps me understand something that puzzled me," Eli announced. "I wondered why the ypf0 api said nothing about opening and closing files."

     "Ah, yes," Wil nodded. "Where file subclass instances come from originally, and where they go later, depends on what types of objects they are. For example, non-files won't have file system path names. A method to open a file is actually a directory method and not a file method."

     "For example," Eli elaborated, "I might open a blob in some store containing blobs, then pass it as a ypf0 instance for reading and writing. You mean like that?"

     "Yeah," Wil nodded. "The care, feeding, memory management, and lifecycle of objects is subclass specific, and the base ypf0 doesn't know about those parts. Doing i/o is part of the middle game needing less detail."

     "What if someone hands me a ypf0 and I need to know what it really is?" Eli wondered.

     "Well," Wil rationalized, "you should add more api to ypf0 which answers the questions you want to ask, then make all your file objects subclass that api instead of ypf0."

     "That would work," Eli nodded.

whatever you want ¶

     "What's the right api for files, Wil?" Eli asked with a wry expression. "What's your preference?"

     "The right api is whatever works for you," Wil temporized, "and whatever works in your current environment without causing trouble. I guess the latter can be hard."

     "You mean files cause a little trouble in every environment?" Eli rephrased with a puzzled look.

     "Kinda," Wil stared in mid-air while concentrating, then looked back at Eli. "Tradeoffs are present in most complex system requirements, and it's pretty hard to get ideal results on every front with files. So you should pick priorities and use them to resolve conflicts."

     Eli shook his head doubtfully, like he thought Wil was pulling a fast one. "That smells like BS. Why is that, Wil? You're just going to brush me off with platitudes?"

     "Uh no, not exactly," Wil held up a hand. "But I can't recommend any good file api style without some context. I tend to assume needs like ones I see a lot. But your situation might differ enough to make new choices. If you like, I can start with vague generalities and then head toward specifics."

     "Sure," Eli essayed a crooked smile. "And why don't you make up some new terminology while you're at it? Zé says you coin new words at the drop of a hat."

paler ¶

     "Okay, thanks," Wil laughed. "I want to talk about the path of least resistance in design and problem solving, so let's make up a couple words related to that."

     "That's okay," Eli said. "No need to be literal."

     "Too late," Wil insisted. "We might use paler as an acronym for PAth of LEast Resistance. Except that's a noun phrase, and I'd rather use paler as an adjective meaning more pale. You can say pale is a quality following a path of least resistance, and paler is even more pale: following a path of even less resistance. Are you following me?"

     "This is far too much punishment for me razzing you about new terms," Eli objected, then saw Wil might be serious. "Uh, so paler is better, when it comes to comparatively less resistance on some path?"

     "If it helps," Wil elaborated, "think of a solution space as darker in hard-to-use places and lighter in easy-to-use places; so a lighter, or paler, path through a space gives you less trouble. Does this mnemonic help any?"

     "Maybe," Eli squinted. "You don't actually think like this do you? Using synaesthesia for visual analogies?"

     "No I don't," Wil admitted. "But I do think of ideas like path of least resistance as a single idea I can wish had a short English word to express. And merely wanting a word makes one pop into mind with plausible justification."

     "How nice for you," Eli deadpanned. "Can you actually speed this up? You're boring me with a spoon-feeding pace. Cut to the chase: you see paler as less cost on some design path?"

     "Yes, I'm just talking about costs," Wil nodded. "Every choice you make has weaknesses and strengths, or costs and benefits. You can also pick problems in ways that make them harder or easier. Picking the easy path is choosing lower initial starting cost. So paler means cheaper."

     "Sure, I knew that," Eli said. "I read all about paler design style on wikipedia the other day. Old stuff."

     "Very funny," Wil granted.

starting conditions ¶

     "Here's my guess," Eli raised both hands in a box around Wil's thesis. "You're really telling me if my code already uses files, then the more I change the api I have now, the more work I make for myself. Is that close?"

     "Yes," Wil smiled. "The path of least resistance is: keep the api you have, whatever it is — unless some part of it's killing you. Change your current file api the least amount that lets you do whatever new thing you want."

     "Is there a long story behind this?" Eli asked.

     Wil tapped his cheek and twisted his mouth this way and that while considering. "The more alike each api you use, the easier it is to swap back and forth," Wil explained. "And you will swap back and forth — otherwise you can't compare competing solutions. Also, some day you'll want to use code again with your original file api, and it will be really irritating if you made porting gratuitously difficult."

     "I can see you're not done," Eli prompted. "Spit it out."

     "Every way you increase cost of using a new api is a reason to avoid it," Wil warned. "When you're in a hurry, or under pressure, or just tired, the path of least resistance is to do the simplest possible thing that gets you to a goal. If it's easier to use a plain file api, you will, instead of using a new file api if it's harder. For example, you should strongly resist changing arguments and method signatures unnecessarily. So new apis should aim to syntactically resemble old ones."

     "You're not going to suggest I use file descriptors to identify my files, are you?" Eli asked in disbelief.

     "No, of course not," Wil laughed. "A file descriptor corresponds to the this parameter of a file method, so that one changes when you go object-oriented. But all the other arguments you should keep the same."

     "Ah, you're not going to tell me to use iovecs, are you?" Eli resisted. "Those aren't object-oriented."

     "Yes, you should use iovec arguments," Wil nodded. "You should stage buffers to pass for scatter/gather i/o the same way it's done in older POSIX api. So if you want the moral equivalent of readv() and writev(), you should use roughly the same signature. Otherwise staging buffers for i/o differently is a bitch when the file api changes."

     (to be continued)

story pruning ¶

     The following fragment is one part of a dialog I wrote a few days ago, then discarded in the middle because it wasn't going to converge on file api material. It was just going to get odd when Ulf returned with coffee, and weird when Finch showed up later. It was going to turn into a longish branching reality joke about parallel universes.

     So I decided to pull the plug before the program settled in deep enough that I had follow through. A fiction binge wasn't what I had in mind; that's all it looked like I'd get.

     Incidentally, Ulf was going to bring back coffee and Christmas presents for all in such a thoughtful way Wil began wondering if he judged Ulf too harshly.

coffee run ¶

     "I'm going for coffee," Ulf stopped halfway out the doorway, leaning back in. "Anybody want espresso or something from Vintage Season?"

     Zé raised a hand, index finger up in a yea. With a manic grin Eli shook his head and held his hand out level, trembling it gently to suggest he was already over his limit.

     Wil stopped typing and pushed back from his desk to stare at Ulf in slack-jawed wonder, just slightly overdone. "Ulf? Offering to pay for coffee? Yes, thanks," Wil gushed. "Bring me three double lattes."

     "Who said I was paying, Farmboy?" Ulf sneered.

     "Don't let the door hit your ass on the way out," Wil waved, which Ulf didn't grace with a reaction. As Ulf tromped off down the hallway, Wil turned to the others and muttered, "I hate that nickname. Who said that first? Finch?"

     "Yeah," Eli agreed. "It sounds nice when she says it — sweet compared to when she calls my dad Dogface. Maybe she has a crush on you."

     Wil looked thoughtful, like Finch was too complex a topic for discussion, then finally shook his head. "It's patronizing," Wil mused, "But she looks twenty years younger than me. Want to start a pool on how old she is?"

     "Hey, Wil," Zé perked up. "Since you look ready for a break, maybe you can help me explain file class api ideas to Eli. Why didn't you write the file demo yet? Eli's been asking me about page caches and how stuff works with shared memory or memory mapping."

     "And he keeps brushing me off," Eli sighed.

     "Aren't you busy writing fiction for Zé?" Wil asked. "Or training to be a jedi or something? Why do you care about boring stuff like files? Please tell me you don't want to be a programmer. Didn't your mom warn you off?"

     Eli spun a pencil expertly through his fingers a couple times while thinking. "Every time I think about writing an application, I get wrapped up in mechanics of storage," Eli explained. "I write a prototype and the thing comes alive and tries to eat my brain. Why does that happen? Then when I make things more complex, it blows up in my face."

same bat channel ¶

     I was unable to come up with a useful plan, and I think that was long enough for a deliberate search. So I'm going to resume something resembling my past agenda, just so I have stuff to do while waiting for better ideas later. But now I don't have a specific end goal in mind, other than having bits and pieces be intrinsically useful on a small scale.

     As near as I can tell, there isn't any technical move currently which has a game changing effect on a scale worth giving serious attention. A lot of things are roughly equivalent to other choices, modulo minor details like personal familiarity. So I don't have a good top-down objective. This means any bottom-up stuff I write about might not have a unifying context for a while — if you don't see one, that's to be expected.

tabled dialog ¶

     I tabled a dialog I started in the middle of the week, so today I wrote another below to see if there was anything inherently wrong with chats between Eli and Wil. Doesn't seem so — this one reads fine.

     The dialog I'm not posting (yet anyway) aimed to introduce file api design, but it used a fictional setting in Wil's apartment in a way that was basically too interesting: it showed no sign of getting serious about code. I enjoyed it too much, so it didn't serve the purpose I had in mind, which is writing useful bits about code, or thinking about code.

     I think I gave myself too much rope. It's easier to write with constraints. Too much freedom makes it easy to play for laughs, which isn't especially more useful than listening to teenagers riff aimless jokes about nothing.

measuring performance ¶

     "Wil, how do folks at work know you're any good?" Eli asked. "Do you do anything to guide perceptions?"

     "They decide whether I'm any good themselves," Wil replied. "But all the time I try to make it clear whether I'm doing something useful. It can be tricky."

     Eli looked puzzled. "How so? Don't you just do the right thing? Isn't that enough?"

     "No, it's not," Wil shook his head. "People usually can't tell whether you've done the right thing."

     "You're kidding, I hope," Eli said. "Why not?"

     "A lot of reasons," Wil considered. "One main reason is that folks have trouble seeing relevance of any change without prior context — if you don't perceptibly change something, no one can tell the effects of what you've done."

     "Like measuring speed before and after?" Eli guessed. "Is it always about speed?"

     "Only sometimes," Wil clarified. "Many more code changes target incremental fixes and updates for stability, focus, size, control, precision, features, transparency, knobs, and options for flexible choices."

     "Transparency?" Eli scrunched his forehead.

     "That means being able to see a thing happen in addition to it just happening," Wil explained. "But let's stay on track. The problem is this: you can't just make things better — it has to be what someone wants right now. Less urgent things can wait. If you future proof things that aren't broken now, it looks like you aren't doing anything useful."

     "Who decides what's urgent?" Eli wondered.

     "That's an excellent question," Wil beamed, "whose answer is more complex than you might think. Urgency and planning is the continuous conversation in daily development. But urgency is influenced by problems blowing up in your face."

     Eli grimaced. "Does that mean you have to let things blow up before you can fix them?"

     Wil sighed. "Unfortunately yes, unless you can dramatize risk of something that hasn't blown up yet, but will."

     "Sounds like that can backfire," Eli objected.

     "Absolutely," Wil agreed. "Let's say your code has a weakness today, but it doesn't manifest yet. What are your choices? You can mess with it, showing folks you can't leave working code alone despite other 'more urgent' tasks. Or you can sell folks on the idea your code still isn't right, which undermines their confidence. Or you can wait until it blows up — if it does — which reflects badly on your reputation for perfection, but lets you fix it."

     "Ick, are those all the options?" Eli asked. "Isn't there a better way to handle it? Why not sneak fixes in under the radar? Then ask forgiveness instead of permission?"

     "Depends," Wil shrugged. "If it amounts to a rewrite of a couple hundred lines in a central, high profile piece of code, then there's no such thing as under the radar. It's politically impossible to change that many lines of core code which has not blown up, unless you want to trash talk it into a rewrite. But that works only if folks follow your description."

     "So what do you do?" Eli prompted.

     "First, you tell folks you're not quite okay with code, and why," Wil said. "But they won't remember more than a few days. Then you think of a better answer, and see if it can be done without so large a change it frightens folks. If you can't, wait for someone to measure the problem."

     "Frightened of what?" Eli puzzled.

     "You're always qualifying code to ship after a while," Wil explained. "Once it starts to be stable, any change is frightening — it's an opportunity to backslide. In fact, there's always a chance you'll screw up a fix, and make things worse instead of better, which helps give you pause."

     "Can't you see faults in a test first?" Eli asked.

     "Only if you have permission to spend time on a test," Wil said. "Otherwise you're spending time on non-urgent tasks again. So you're stuck with tests already there."

     "Until someone runs a new test showing the weakness?" Eli guessed. "How long does that take?"

     "Only until a really intense stress test is staged," Wil replied. "The only question is: does it happen early — when it's okay to have bugs — or late when it looks worse?"

     "You're right, it sounds tricky," Eli conceded. "How do you handle late exposure of bugs? Do you just take the hit? What if there's an open position for a scapegoat?"

     "It helps to know answers instantly," Wil explained, "then to fix code directly after a clear explanation of issues. But this runs a risk of looking like grand-standing. Folks might wonder if you saved a problem to be a hero."

     "But you would never do that," Eli prompted.

     "No, I shoot for consistently good code," Wil said. "But I make mistakes now and then. And I don't think of everthing. So my stock goes up and down depending on events."

     "Goes without saying," Eli said nonchalantly. "So you say measurements affect how you are seen?"

     "Yes, and lack of measurement too," Wil agreed. "Speed of code is one of the easiest things to measure. So it's easy to stay on top of efficiency and latency. But some things aren't easy to measure, so I stay away."

     "Can you give me an example?" Eli pleaded.

     Wil cleared his throat. "Say someone has an idea for a better algorithm in code: a subtle improvement that clearly ought to be better — just a bit more complex. If I can't think of a way to measure it, I say I won't do it."

     "Why not?" Eli wondered.

     "I tell folks I can't spend time on unmeasurable effects," Wil shrugged. "I say it's too easy to be judged unproductive if you write code no one can evaluate."

     "Uh oh," Eli had a thought. "Does that mean you can't write code other folks don't understand?"

     "Yes, that's exactly what it means," Wil nodded, smiling at Eli's growing look of comprehension. "Being smart is rarely as useful as being obviously effective — at least when good status in a group is the desired goal."

self censorship ¶

     I nearly decided to cut all of yesterday's material on status reports and due diligence, because it seemed like too much information. You might construct too good a model of how I see things, given random bits and pieces describing parts of real life work circumstances, which could inconvenience me. Then I thought: no way, what difference can it make? Even if folks make the classic mistake of assuming samples are representative of larger patterns, there's nothing to see.

     I thought the piece labeled due diligence was too interesting to censor, since it might give folks a constructive priority to write clearly more often. Make yourself useful: it might pay off someday.

spoke too soon ¶

     Ironically, after writing last week about how my code seldom has flaky behavior causing a fire drill at work, the next day we had a fire drill involving one of my hashmaps. I was able to eliminate any bad consequences of the odd behavior, but so far there's no explanation yet for the evidence seen — which is unacceptable because no mystery in central data structures can be tolerated. Unfortunately it doesn't reproduce in any test less than a complex, full system run.

     I started rewriting the hashmap in C++ using templates on Friday, and yesterday and today I finished it at home and ported my unit tests and solved all the minor bugs that appeared in the fit and finish. So tomorrow I can use the new version to compare to the old C version, which was somewhat more complex. (I wrote the first version in C because C++ was deprecated at the time.)

     At this point I'm sure you wonder: Why don't I just use C++ STL maps? There are actually several reasons, the least of which is easy to explain: I can't link C++ runtime libraries in the process where the map gets used. But even if that were not so, I still couldn't use an STL map and get the same desired performance.

     Why not? I'm so glad you asked. Using standard STL interfaces, it's not easy to make objects be members of several collections at once with constant time cross referencing — so that after finding a member in one collection, you already known it's exact position in another.

     Maps used as caches are classic examples, when you use LRU lists to represent which map members can be discarded first when memory pressure requires vacating cache members based on usage. Standard STL api represents object linkage using extrinsic indexing, and instrinsic links are needed to modify an LRU list in constant time after finding a map element. (I've done a lot of hashmaps like this since my first page caches for file i/o systems from the early 90's onward.)

     I finally got it squared away this evening, and then ran the code through tens of millions of iterations in pseudo random exhaustive testing, giving me reasonable confidence of correctness.

file definitions ¶

     If I hadn't been coding stuff I need at work this weekend, I'd have started the dialogs I mentioned already, featuring some discussion of file api abstractions and how you can aim to make something useful and re-usable. None of my old demos say much about how you think about api design in the first place so the result is most likely to fit what you want now, and what you might want later too.

     But I thought about rough drafts of dialogs between Wil, Zé, and Eli. (Eli is needed to ask ignorant questions, since Zé knows about as much as Wil and merely thinks differently, but Eli is a neophyte.)

     I'm going to let the discussion touch basic ideas verging on philosopy, as well as matters of taste in api design. Wil might emphasize the importance of definition, both presence and absence, since degrees of freedom you preserve derive partly from absence of rules you don't want to impose. But one of Wil's consistent memes is focusing on what you want to happen — Wil's approach to design is operational: things are defined by sets of operations to be done and little else. So for example, a file is defined by what you can do to it, and not by any other model you know about possible back stories.

status reports ¶

     We don't all write status reports where I work — my manager usually writes the closest thing to status reports, which are rarely more detailed than single line item tasks attached to engineers, with an implicit time scale. (Completion might occur during time covered by a report, which is rarely more than a few days.) It's not unusual for one third of line items not to be completed in time involved. Such reports basically describe what's on everyone's plate.

     It's tempting to say I never write status reports, but I write emails every day describing what's up with topic X, where X changes all the time. My goal is usually to write the shortest email mentioning each crucial point. Sometimes I manage to get each crucial point into one line of twelve words or so, which stand like one line paragraphs consisting of nothing but a topic sentence and conclusion rolled into one. A topic with many related consequences can take twenty different one line sentences, like an email I sent yesterday.

     Such emails take five minutes to write and only one minute to read (if you think about what it means), and only sometimes appear as one line items in status reports. Instead, my manager typically uses such treatments to summarize issue descriptions in conversation with upper management or QA engineers (because usually I write emails about something testers can see in a test run). I think he also forwards my email when someone asks, "What's up with X?"

     Each email also says what I plan to do about it, with a rough estimate of time involved (varying from an hour to two days) and a comment on what happens if I don't do it. I often add remarks to the effect my plan can change to any addressing Y, since Y is required for X, so folks can offer new plans.

     What prompts me to write these emails? Usually my manager emails a question: What does X mean?

     Other times I learn a fact requiring a change in plan, so I send out warning I might leave the current plan.

     My emails are short because people seldom read more than fills their current attention span, which for many folks is less than one minute. In particular, almost no one will stop and consciously relate multiple sentences if this is required; if you write anything requiring thought, this will be put off to later — which is another way of saying never. Everyone is busy and doing triage at some scale or another. So when you write too much, you become part of the problem.

     But a good number of folks use this as an excuse to avoid writing anything of any length at all. A resulting total void in explanation for many topics is a problem itself. I gently chide others by suggesting, "You should write a brief summary." Complex products often accrete from hundreds of choices. I wish each one had a comment along the lines of "why I did this" you could find somewhere in a log. A minimum of three sentences each would be nice, with a couple paragraphs for complex things. I can dream, can't I?

     Folks have bad memories of Big Design going back to the 90's, where lots of time was wasted writing specs that didn't subsequently help get things done. Many folks learned the wrong lesson from this: never write anything. It seems feasible in the short term. But newcomers can't play. And it's hard to upgrade old pieces without description.

     The most useful of documents is a wish list: what folks hoped was possible. A laundry list of desires is much shorter than schemes developed in their pursuit, but helps a lot for seeing intent in downstream design. Did you ever look at inscrutable code and ask: What did you hope for here? The element of why goes a long way in giving insight.

     When I write anything of larger scope or longer term significance, I write a wiki page instead of an email for future reference. Then I email an inline copy of the wiki page to folks when I'm done. This includes basic design documents or problem descriptions when appropriate. I never spend more than a few hours on one wiki page, so each one is (relatively) small, because large documents are less useful. Daily broadcast ensures other folks can interrupt as soon as mismatch appears between my plan and desired targets.

due diligence ¶

     More than a year ago, one of the folks at my company was leaving soon (to start his own company) and he stopped by to chat briefly. We hadn't interacted much before, but he was familiar with my work. He was on the team of folks doing due diligence when the company bought a startup for which I had written a web cache server for web acceleration.

     "Ah, that's interesting," I told him. "That would explain why you pinged me so many times around then."

     "Yes," he smiled. "We were reading server specs, and your name was all over them: you wrote all the docs."

     "Yes, and I wrote most of the runtime and basic server behavior, too," I confirmed. "It was fun — which is one of the reasons I came here: I hoped for more fun work."

     I had not responded to his earlier emails about my server, because at the time I was still put out by having been laid off in a reduction of staff, despite my having done the central part of the acceleration engine. I had made the mistake of saying I didn't intend to stay forever. So I had not made a penny other than salary, because I wasn't a founder. A shell company picked up my server and another, then sold them for over forty million dollars to these guys, but I didn't see any of that. Apparently the server I designed still made lots of money, but I was seldom asked any support questions. (When I asked why no one asked about the core, I was told, "It just works.")

     "Your docs were crucial to our evaluation," he explained. "We would not have been able to buy the company without the specs you wrote." Then he watched my reaction.

     "Oh," I mumbled. He was telling me what I had written was a deciding factor in a deal for tens of millions of dollars, and he thought it was interesting I had gotten nothing.

     "I guess it's a good idea to write about code you make," I told him. "Otherwise other folks have trouble seeing value."

     Around then I realized why he was talking to me: anyone starting a company might want me to write docs for my code since it would make a company sellable to a buyer.

     He didn't come right out and make any suggestion about joining his new company, since that would be recruiting; but he could see wheels were turning in my mind.

     I would have suggested I join his company myself, since it seemed a good idea, except I could not afford to give up my salary because my ex still had me in hock up to my eyeballs. In effect, I was going to live hand-to-mouth for the forseeable future, even with a good salary at BigCo. I would not be able to afford the kind of salary folks get as founders or first employees — it wouldn't cover child support.

     So he left with good wishes and keep-in-touch words. I was happy being left with a lot to think about though.

writing formats ¶

     I'm in the mood to try an experimental writing format. The premise is simple: Wil stops writing demos for a while, so Zé decides to pry more out of him. The result is a discussion about software before it's actually quite written. Instead of Wil presenting something he did last week, Zé gets him to do parts in near real time. This lets you see a little more under the covers.

     Of course, this is actually something of a shabby trick: it's like a first draft of content without a cogent plan. So Zé asks, "What was that file api going to look like anyway?" And because Wil keeps responding politely, the next thing he knows a demo is half written, but somewhat badly because there's no beginning, nor any end. Hey, I like stories that start in the middle.

     This kind of material would appear here in the log. Then later I can assemble a more polished demo on a separate page, devoted to treating something all in one place, using some of the pieces drafted here in the log at times.

     And since the format would be strange, I can break format even more and situate these conversations in Wil's living room, which is where the group in Zé's project have been meeting. I can have Finch sitting on the couch in the background making odd requests like, "Farmboy, fetch me that pitcher?"

     "No freaking way," Wil would snap. Then Zé explains to Eli that Finch is angling for as you wish, but Wil isn't cooperating. Stuff like that, constantly threatening to degrade technical discussions into chaos. You would wonder how Wil and Zé manage to keep talking about code, because it would be a lot less interesting than interruptions, especially the weird ones.

     I wish I could make Ulf interesting. The other characters cook along really well, but Ulf is such a blatant straight man for jokes he seems lifeless — he needs some covert agenda I haven't thought of yet, besides having stupid suggestions aiming to line his pockets. So far it's been too tempting to have Finch keep humiliating him for being a jerk, but that's tiresome.

work specs ¶

     Lately I've been tempted to write specs like this at work: as conversations between developers — because otherwise it's hard to show how everything in each api is just the answer to a question. Today I was drafting code api half the day, and I noticed it was going to be tough going without benefit of questions for context.

self description ¶

     I don't try to tell relatives what I do for a living. They usually get the same story I give strangers when they ask what kind of work I do. I just say, "Programmer."

     When I worked at Netscape, and places afterward, my kids wanted me to explain specifically what it was I was doing — what would it look like to them. My summary was usually, "I make things go faster." The idea was to have users wait less. Everyone gets this. "Awesome, I hate waiting," everyone says.

     The problem with this description is lack of detail: how is today any different than last year? Telling my kids the difference between a good day and a bad one is an exercise in teaching them concepts for which they have no referents. Ultimately, all my kids can grasp is whether or not coworkers have reasonable expectations lately, and whether or not something worked today.

     Actually, it's hard to tell people what I do even when they're professional software developers. The word "runtime" means almost nothing to the vast majority of developers, as near as I can tell. If I say I work on core engines, caching, optimization, and algorithms, this improves the picture some, but makes me sound like a lab coat wonk or lazy architect weasel, or some variant of high tech priest. If I say I do memory management and data structure tuning, I sound like someone who strips and cleans software bicycles — lightweight stuff. But what I really do is understand why things work, and why they'd still work with fewer moving parts, and maybe better. It's most fun to do this when really crazy stuff is involved.

     Half of what I do is exactly what everyone always does: fix bugs and tackle issues that come up. But I usually fix bugs by thinking of what could cause evidence observed. Yesterday walking back to work from lunch, I found a race condition in code that would explain an odd behavior someone reported. A lot of my bug fixes are two hour turn around deals, where the only tricky part is knowing why it happens. I get a kick out of fast turn around when I can.

     What I like to do most, and what folks enjoy when I do, is finding a simpler way to do some old thing, and do it better with fewer odd edge cases, because code that doesn't exist can't have bugs. Sometimes it means re-designing large components from scratch, where the old story is painfully baroque and the new story is elegant enough to grasp as obviously correct, modulo details. This probably would not matter much to folks, except it also tends to correlate strongly with code that runs fast and reliably.

     I'm pretty sure folks most appreciate absence of flaky behavior going forward. My code causes fewer fire drills. But it's an abstract idea meaning very little to most non-technical folks.

     A few years ago at a barbecue, when I was in my socializing phase, I tried to explain my passion for removing entropy and technical debt from code to a hip girl in a denim jacket (yes I know what this makes you think) and she had an amusing insight she shared. "Oh I get it — you do bullshit removal," she said.

     I had to admire how concisely she phrased it.

     As you can see, I moved all this to another page. But the permalinks here are still valid.

utility ¶

gullible ¶

socializing ¶

old movies ¶