|
demos are
explained here;
a menu at top column right indexes actual topic demos.
Here we demo meter.
problem
«
Wil sometimes needs out streams that measure content passing through. The writer to pretty print data in a toy language under mu uses specialized out stream subclasses to gauge total bytes written to a stream (yno) and length of the current line (yoo), both shown on this page. This meter demo might properly be considered an appendix to the out demo, since all code here is for subclasses of base out stream yo. But Wil decided to name the demo after the role of these classes in metering content written by the toy language writer. (In earlier revs of Mithril language work, Wil included meter as a substring in class names devoted to measuring output size. Here it's just the demo name.)
length fold
«
The name of class yno means thorn number out (stream). It throws away every byte written after counting them. Of N bytes written to yno, all of them are counted in one n32 integer member variable. So a stream of bytes is mapped to an integer counting the bytes. You could also call this a fold given a starting sum of zero and a map that sum each additional length written to the stream. class yno : public yo { // out to n32 (measures LENGTH only) «
//protected:
// u8* o_0; // in origin
// u8* o_p; // in cursor: must be such that o_0 <= o_p <= o_x
// u8* o_x; // one beyond end of buf
// mutable int o_e; // zero or some error status
// yb o_take; // copy buf returned by take to error check
// u16 o_tab; // current indent depth
// u16 o_pad; // net yet used
This comment shows inherited yo members. protected:
yb no_b; // buffer to hold bytes written one at a time «
// p32 no_p; // cur pos (can be less than no_n after oseek)
n32 no_n; // length written earlier not in unflushed buf «
enum { e_nobx = 96 }; // default no_b.b_x: size of no_default
u8 no_default[e_nobx]; // constructor default buf for no_b «
void no_init(const yv& v); // init inherited o_0, o_p, o_x
Member no_n counts all bytes written, and no_b describes a local byte i/o buf holding bytes transiently before they get counted. The default buffer is no_default. public: // allow explicit add to length (instead of the bytes)
void noadd(n32 more) { no_n += more; } «
yno& operator+=(n32 more) { no_n += more; return *this; } «
Method noadd() allows you to simply add more bytes instead of writing them to get them written. public:
// empty construction uses the tiny local no_default buffer
yno();
void noclear(); // clear: back to state just after construct
A just constructed yno instance has zero in count no_n, and noclear() resets it to zero (and makes the local buf empty again.) // alternatively you can supply own buf larger than e_nobx;
// v is ONLY useful if v.v_n exceeds e_nobx: more buf space
yno(const yv& v); // only if you want buf bigger than e_nobx
yno(const iovec& iov); // only for buf bigger than e_nobx
If you need a bigger buffer (and why would you?) these constructors allow you to specify the byte i/o buffer to use. // get bytes written to this outstream (no flush required);
// return count of bytes written whether or not flushed
p32 nolen() const { return no_n + (o_p - o_0); } «
operator p32() const { return no_n + (o_p - o_0); } «
Method nolen() returns the 'length' of this stream: the number of bytes written, including all counted earlier as well as any few bytes in the local buffer not yet counted. You can simply cast yno to an unsigned integer and operator p32() converts yno to the same length. protected: // visible to subclasses; public api uses oc() only
virtual void _oc(int c); // abstract fallback for yo::oc()
public: // required virtual methods for yo subclasses
virtual ~yno();
virtual p32 opos() const; // current byte position
virtual p32 olen(); // size in bytes (same as nolen())
virtual int owrite(const void* src, n32 n); // neg on err
virtual int oflush(); // neg on err
// methods needing seek: NOT YET SUPPORTED:
virtual int oseek(p32 p); // NOT supported yet
virtual int opwrite(const void* src, n32 n, p32 pos);
public: // bulk i/o
virtual int otake(yb& dest); // reserve dest.b_x buffer bytes
virtual int ogive(yb const& src);
};
inline yno& operator<<(yno& o, y1now const& ) { «
o.oflush(); return o; }
inline yno& operator<<(yno& o, y1reset const& ) { «
o.noclear(); return o; }
These operators allow you to flush and reset yno via operator <<, using ynow or yreset as the right hand side. void yno::no_init(const yv& r) { «
o_take.bclear(); o_tab = 0; o_e = 0;
if (r.v_p && r.v_n > e_nobx) {
no_b.v_p = o_0 = o_p = r.v_p;
o_x = r.v_p + r.v_n;
no_b.b_x = r.v_n;
}
else {
no_b.v_p = o_0 = o_p = no_default;
o_x = no_default + e_nobx;
no_b.b_x = e_nobx;
}
}
Initializing yno makes the (o_0, o_p, o_x) triple and the no_b buffer describe the space passed in run r. But the local default buf space is used if the passed vector is nil or smaller than the default. void yno::noclear() { «
// return to original state following construction
yv tmp(no_b.v_p, no_b.b_x); // take cur buf space as given
this->no_init(tmp);
no_n = 0;
}
Clearing yno makes the buffer empty again and resets the counted length to zero. yno::yno() : yo(), no_b(no_default,0,e_nobx), no_n(0) { «
yv vdef(no_default, e_nobx); // small default buffer
this->no_init(vdef);
}
yno::yno(const iovec& v) «
: yo(), no_b(v.iov_base,0,v.iov_len), no_n(0) {
yv r(v.iov_base, v.iov_len);
this->no_init(r);
}
Each of the constructors uses common init code. The destructor gratuitously clears — unnecessary when the inherited destructor zeroes the base yo triple. /*virtual*/ int yno::oflush() { // neg on err «
p32 some = o_p - o_0; // distance moved from origin
if (some) { // able to flush?
no_n += some; // flush
o_p = o_0; // make buffer empty
}
return 0; // no error
}
Flushing merely updates the no_n count of bytes written, if the buffer cursor has moved from the origin. Because seek is not now supported, the current position is nominally the length: the eof position in the stream. The virtual length is the same as static nolen(): total byte count written to the stream. /*virtual*/
int yno::owrite(const void* src, n32 n) { // neg on err «
p32 some = o_p - o_0; // distance moved from origin
if (some) { // able to flush?
no_n += some; // flush
o_p = o_0; // make buffer empty
}
if (src) { // able to write anything from this address?
no_n += n;
return (int) n;
}
return 0;
}
Writing N new bytes does an inline flush if the local buffer has any bytes inside, and then simply counts the new bytes. /*virtual*/ int yno::opwrite(const void* src, n32 n, p32 at) «
{
this->osayfail("yno::opwrite() not implemented yet", ENOSYS);
return -1;
}
/*virtual*/ int yno::oseek(p32 u) { «
this->osayfail("yno::oseek() not implemented yet", ENOSYS);
return -1;
}
None of the methods seeking current stream position do anything; the pretty printer in writer has no need to seek yno — it's always used with the type statically known. /*virtual*/ void yno::_oc(int c) { «
if ( o_p < o_x ) // room for more?
*o_p++ = (u8) c; // add it
else {
this->oflush();
++no_n; // count overage
}
}
Non-public method _oc() should only be called when public oc() has no more room in the buffer, so the else clause above should always run: after flushing buffered bytes (which simply counts them), the new byte passed to _oc() is counted too. int yno::otake(yb& src) { // noop / fails «
src.bclear();
this->osayfail("yno::otake() not implemented yet", ENOSYS);
return -1;
}
int yno::ogive(const yb& inBuf) { // noop fails «
this->osayfail("yno::ogive() not implemented yet", ENOSYS);
return -1;
}
Neither of the methods used for direct-to-buffer writing is current used, so Wil lazily has not written the code yet. Both otake() and ogive() are meaningful in this subclass, but Wil doesn't expect to read directly from an instream into this yno outstream. It would be simpler to call noadd() directly. If implemented, these methods would be nearly identical to the versions in class yoo, shown below. yoo.cpp « The following yoo implementations of otake() and ogive() are shown here to put them near the stubs for yno above, and to balance the two columns a little int yoo::otake(yb& dest) { // reserve dest.b_x buffer bytes «
dest.bclear();
if (!o_p) { if (!o_e) { this->ofail(ENOSR); } return -1; }
if (yunlikely(o_take.v_p)) {
yo::osay("yoo::otake() IN USE"); return -1;
}
if (o_p >= o_x) { this->oo_flush(); } // flush if buf is full
o_take.binit(oo_buf.v_p, 0, oo_buf.b_x);
dest = o_take; // o_take copies return to check in ogive()
return dest.b_x;
}
int yoo::ogive(yb const& src) { «
if (!o_p) { if (!o_e) { this->ofail(ENOSR); } return -1; }
yb take = o_take; // copy for check
o_take.bclear(); // clear despite any err
if (src.v_p != take.v_p || src.b_x != take.b_x) {
yo::osay("yoo::ogive() src buf != saved buf from otake()");
return -1;
}
n32 n = src.v_n;
if (n > src.b_x) {
yo::osayf("ogive() src.v_n=%d > src.b_x=%d",
(int) n, (int) src.b_x);
n = src.b_x; // safety
}
o_p += n; // advance buffer mark to reflect adds from src
return (int) n;
}
verbose
«
"Why the verbose commentary, Wil?" Stu asked. "Normally you don't bother to explain the inner logic of out stream methods in such detail. Are these streams special?" "Not very special," Wil replied. "But I'd like folks to easily understand how content size is measured in the pretty printer to come. Since that printing code will delegate measurement to these classes, it would obfuscate the pretty printer if these out streams seemed at all clever." "I did think the yoo::owrite() was hard to follow until your comments," Stu noted. "So maybe they helped; now I see exactly why oo_find_lf() is called or not, in each place." "Good," Wil smiled. "What did you think of yno above?" "Astonishingly trivial," Stu dismissed. "I don't even know why you bothered. I suppose triviality is the point?" "Sure," Wil agree. "Using yno is about the simplest way you can measure how long something gets when you print it to an out stream, unless you want to write separate length methods independent of actual writing." "Ugh," Stu winced. "Then I'd have to keep two sets of methods in sync; I'd rather just have one that writes. But doesn't that write big objects unnecessarily, if you only wanted to know if they were more than a new bytes in size?" "Well a pretty printer can try to give up earlier, without iterating over all members of a collection," Wil explained. "But each separate atom might best be printed — though of course yno never does anything but count."
license
«
All this code is available only under the BriarPig mu-babel license described fully on the rights page. You do not have permission to reprint this page in any way. Neither feeds nor repackaging is allowed. You can link this page if you want folks to read it. |
A submenu for demos appears below, letting you
go to the page on a topic written as a demo (as the
demos page defines it).
menu
thorn: todo, names, fd, iovec, assert, log, run, hex, crc, buf, in, out, quote, escape, compare, file, deck, cow, arc, blob, tree, slice, rand, time, stat, hash, heap, node, primes, page, book, pile, stack, atomic, lock, mutex, thread, map, meter « Þ, list, iter, ctype (mu: toy, peg, imm, tag, box, symbol, token, number, bigint, class, method, reader, writer, eval, env, vm, gc, world, pcode, compiler, asm, lathe, lisp, smalltalk, design, weight, jar, card, harp, debug, profile) Some demos are stubs: todo is a demo guide. See toy for mu updates on language pages; names introduces naming schemes.
line length
«
The name of class yoo means thorn out (to) out (stream). Basically yoo is a subclass of yo that writes to another instance of yo (presumably of another subclass) and keeps track of the current column in the current line written. In some pretty-printing algorithms, Wil wants to know the current column at which something is about to be written, allowing similar indentation to be used again later. The scheme used by yoo to track current line length is engagingly simple, and resembles other code to perform one time filters of content streams, such as string escape writing. All yoo has to do is see every newline passing by, while counting how many other bytes appear afterward. This is done in method oo_find_lf(), through which all content passes exactly one time on the way to a destination out stream. The buffer in yoo is only useful for byte-at-a-time i/o as supported by the base yo api. // yoo is like yfo but with yo sink, measuring line length
class yoo : public yo { // out to yo (measures LINE LENGTH) «
protected:
yo* oo_o; // final destination of buffered bytes written «
// oo_p is high local buffer u8 checked for newlines already
u8* oo_p; // in cursor: such that o_0<= oo_p<= o_p<= o_x «
n32 oo_ln; // current line length ("len" or Line leNgth) «
n32 oo_lines; // count times oo_ln reset by 1 or more LFs «
p32 oo_pos; // pos of 1st byte in buf: where o_0 points «
yb oo_buf; // the buffer when buffering happens «
mutable n32 oo_outflow; // number of bytes written to oo_o «
Ultimately all bytes get written to destination out stream oo_o; but before the bytes are written, they are scanned once and one time only to find newlines (aka '\n', aka 0xA, aka LF) inside, which are counted by oo_lines. A count of total bytes already written to oo_o is maintained in oo_outflow. Member oo_ln is the length of the current line, ie the number of bytes after the last newline written, ie the current column position of the next byte written in the current line. Both oo_ln and oo_lines are updated by internal method oo_find_lf(), which is the boundary through which bytes pass once and one time only on the way to destination oo_o. So oo_ln and oo_lines only reflect bytes already scanned. When a local buffer described by triple (o_0, o_p, o_x) inherited from yo has buffered bytes inside, some of these bytes might have been written since the last time oo_find_lf() was called. However, any time someone calls oocol() to find out the current column (ie line length) it's necessary to run oo_find_lf() on bytes in the buffer that have not yet been scanned the one alloted time. Pointer oo_p plays the special role of tracking how far oo_find_lf() has progressed through the local buffer — bytes between o_0 and oo_p have been scanned for newlines, and bytes between oo_p and o_p have not been scanned. So while inherited o_p tracks how much of the buffer is used, oo_p tracks what subset of that has already been scanned for newlines. enum { e_oobody = 128 }; // local u8 buf len (before oo_o)
u8 oo_body[e_oobody]; // local u8 buf before put in oo_o «
Local buffer oo_body goes in triple (o_0, o_p, o_x) to hold bytes written one at a time, and to hold small writes at times. // scan bytes for newlines and change line len accordingly;
// suffix is trailing stream edge of unchecked bytes to oo_o;
// assert: bytes going to oo_o pass oo_find_lf() exactly ONCE
void oo_find_lf(yv const& suffix); // suffix sets line length
void oo_find_lf(const u8* before, const u8* after);
bool oo_flush(); // write contents of buffer if not empty
Non-public method oo_find_lf() looks at every byte on it's way to destination oo_o, at some point in time between hitting a yoo entry and being written to oo_o. The number of newlines is counted in oo_lines, and the number of bytes after the last newline tracked in oo_ln, denoting current line length. The constructor only needs a destination out stream. // lines: the same value when no new newline has been written
n32 oolines(); // count of newlines every written to yoo
Method oolines() returns the number of newlines written; this isn't inline in case buffered bytes still need to pass through oo_find_lf() to look for more newlines. /// \brief check for newlines written via oc()
n32 oocol(); /// \return current column (line length)
operator n32() { return this->oocol(); } «
Method oocol() returns current line length: the number of bytes written after the last newline (also not inline in case buffered bytes still need to pass through oo_find_lf()). protected: // only visible to subclasses; public uses ic() only
virtual void _oc(int c); // abstract fallback for yo::oc()
Non-public method _oc() is the virtual fallback for inline oc() in the public yo api (cf «) which calls _oc() only when the local buffer is full (when o_p hits o_x). public: // required virtual methods for yo subclasses
virtual ~yoo();
virtual p32 opos() const; // current byte position
virtual p32 olen(); // size in bytes
virtual int owrite(const void* src, n32 n); // neg on err
virtual int oflush(); // neg on err
// following two methods needing seek: not yet supported
virtual int opwrite(const void* src, n32 n, p32 pos);
virtual int oseek(p32 p); // NOT supported yet
public: // bulk i/o
virtual int otake(yb& dest); // reserve dest.b_x buffer bytes
virtual int ogive(yb const& src);
};
This operator allows you to flush yoo via operator <<, using ynow as the right hand side. yoo::yoo(yo* dest) // dest is o_o: where bytes go eventually «
: yo(), oo_o(dest), oo_p(0), oo_ln(0), oo_lines(0)
, oo_pos(dest->opos()), oo_buf(oo_body, 0, e_oobody)
, oo_outflow(0)
// u8 oo_body[ e_oobody ]; // only used if necessary
{
o_0 = oo_p = o_p = oo_buf.v_p;
o_x = o_0 + oo_buf.b_x;
o_tab = 0; // no indent
}
Once the input destination outstream is saved away in oo_o, the constructor merely zeroes everything and preps the local buffer to use oo_body for bytewise i/o. Following convention in other out stream destructors, note an absence of flush: unwritten bytes are not flushed to oo_o at the last instant; if you need flush, please do this yourself explicitly. p32 yoo::opos() const { // current byte position «
if (!this->ogood()) { obad(); return 0; }
return oo_pos + (o_p - o_0);
}
p32 yoo::olen() { // size in bytes «
if (!this->ogood()) { obad(); return 0; }
return oo_pos + (o_p - o_0); // assume oo_o is end of stream
}
Just like yno, which also does not support position seeks, yoo treats both current position and length the same: just length of all bytes previously written. Since oo_pos counts all bytes written to oo_o, it's only necessary to add pending bytes in the local buffer. int yoo::oseek(p32 newPos) { // might not be supported «
yo::osayf("yoo::oseek(newPos=%d) not supported", (int)newPos);
// We can easily implement oseek() (and opwrite()) in terms of
// stream pos alone; but getting *line-length* right would be
// hard since it involves finding distance to nearest earlier
// newline, after seeking new position. Let's skip it for now.
return (int) ENOSYS;
}
int yoo::opwrite(const void* src, n32 n, p32 p) { «
// requires oseek()
yo::osayf("yoo::opwrite(n=%d, p=%d) seek not supported",
(int) n, (int) p);
return (int) ENOSYS;
}
None of the methods seeking current stream position do anything; the pretty printer in writer has no need to seek yoo. Thus Wil can simply avoid the issue of trying to figure out number of lines before the target position, and current line length there. void yoo::oo_find_lf(yv const& suffix) { «
// set line length based on suffix
// assert: bytes going to oo_o pass oo_find_lf() exactly ONCE
n32 n = suffix.v_n; // number of bytes being scanned
u8* lf = suffix.vrindex('\n'); // leftmost LF (only if found)
if (lf) { // line length is distance between lf and o_p?
oo_lines += suffix.vcount('\n');
u8* end = suffix.v_p + n; // one beyond last byte in suffix
oo_ln = end - (lf+1); // u8s between last LF & suffix end
}
else
oo_ln += n; // no LF: all new extra bytes make line longer
}
void yoo::oo_find_lf(const u8* before, const u8* after) { «
// assert: bytes going to oo_o pass oo_find_lf() exactly ONCE
n32 n = after - before; // number of bytes being scanned
yv suffix(before, n); // describe for search by yv::vrindex()
u8* lf = suffix.vrindex('\n'); // leftmost LF (only if LF)
if (lf) { // line length is distance from lf+1 to end?
u8* end = suffix.v_p + n; // 1 beyond last byte in suffix
oo_ln = end - (lf+1); // u8s between last LF & suffix end
}
else
oo_ln += n; // no LF: all new extra u8s make line longer
}
Method oo_find_lf() is the main reason for this yoo subclass of yo: to count all newline bytes (0xA) in oo_lines, and track the number of bytes following the last newline in oo_ln. First yv::vrindex() finds a rightmost newline if any. If none is found, the else clause adds all bytes to current line length oo_ln. When a rightmost newline is found, yv::vcount() counts newlines, adding the total to oo_lines; the count of bytes following the rightmost newline becomes the new line length. void yoo::_oc(int c) { // abstract fallback for yo::oc() «
if (this->oo_flush()) {
if (o_p < o_x) *o_p++ = (u8) c;
}
}
Because _oc() is called only when the buffer is full, a call to oo_flush() empties the local buffer by writing them to oo_o; afterward the new byte can be written in the next slot. bool yoo::oo_flush() { «
// write contents of the buffer if it is not empty
if (yunlikely(!this->ogood())) {
this->obad(); return false;
}
n32 size = o_p - o_0; // not neg; ogood() forces o_p>=o_0
if (size) { // anything in buffer?
if (oo_p < o_p) // need to update line length first?
this->oo_find_lf(oo_p, o_p); // scan for LF before oo_o
oo_p = o_p = o_0; // empty again (back to origin)
int actual = oo_o->owrite(o_0, size); // write them all
if (actual < 0) return false; // error
oo_pos += (p32) actual; //pos of next unflushed u8s at o_p
}
return true;
}
Flushing the local buffer must write write bytes between origin o_0 and cursor o_p to destination outstream oo_o; but if oo_p and o_p are not the same address, they bound bytes not yet run through oo_find_lf() to find and count newlines, so that happens first. Finally, the local buffer is set to fully empty and the count of bytes written to oo_o is added to running total oo_pos, which is also the logical stream position of the byte at o_0 in the buffer. n32 yoo::oocol() { /// \return current column (line length) «
if (oo_p < o_p) {
// nonzero gap exists between position and earlier vetting?
this->oo_find_lf(oo_p, o_p);
}
oo_p = o_p; // no gap now exists: all new extra u8s checked
return oo_ln;
}
n32 yoo::oolines() { // number of newlines (0xA) ever written «
if (oo_p < o_p) {
// nonzero gap exists between position and earlier vetting?
this->oo_find_lf(oo_p, o_p);
}
oo_p = o_p; // no gap now exists: all new extra u8s checked
return oo_lines;
}
Methods above merely access current line length oo_ln and total newline count oo_lines as counted by oo_find_lf(), but first call oo_find_lf() one more time if any bytes exist in the local buffer between oo_p and o_p that have not yet been scanned. int yoo::owrite(const void* psrc, n32 inSize) { // neg on err «
if (!this->ogood()) { obad(); return -1; }
const u8* src = (const u8*) psrc;
if (!inSize) { return 0; } // write nothing? harmless no-op
if (!src) { yo::onil("src.v_p"); return -1; }
n32 outSize = 0; int actual = 0;
n32 room = o_x - o_p; // space left (never neg after ogood())
if (o_p == o_0) { // at origin? empty? nothing in local buf?
When no bytes are present in the local buffer, this section writes all inSize bytes to the buffer instead of oo_o when inSize does not exceed a quarter of the local buffer size; in this case scanning for newlines is delayed until later. yassert(oo_p == o_p); // no buffered bytes to scan for LF
if (inSize > e_oobody/4) { // best written directly instead?
this->oo_find_lf(src, src + inSize); // adjust line len
if ((actual = oo_o->owrite(src, inSize)) < 0)
return -1;
outSize += (n32) actual; oo_pos += (p32) actual;
}
else { // inSize <= e_oobody/4 implies all fits in the buf
::memcpy(o_p, src, inSize); // input safely put in buf
o_p += inSize; // advance cursor
outSize += inSize; // count output
}
}
else { // already buffered bytes are present
if (inSize > room) { // more than fits in remaining space?
Here an additional inSize bytes can't fit in the local buffer with only space for room more bytes. All unscanned bytes in the local buffer and new input are scanned for newlines. Then the local buffer is packed as full as possible before writing the whole buffer to oo_o. (The packing occurs in the hope any remainder will then easily fit in the local buffer.) If remaining bytes exceed a quarter of the local buffer size, all are written directly to oo_o; otherwise these already scanned bytes are put in the local buffer, and oo_p is set to follow all of them so they won't be scanned again. if (oo_p < o_p) { // any local buf not yet scanned for LF?
this->oo_find_lf(oo_p, o_p);
oo_p = o_p; // make up-to-date
}
this->oo_find_lf(src, src + inSize); // all bytes at once
n32 plus = inSize - room; // bytes that can't fit in room
if (room) { // is there any more space left at all?
::memcpy(o_p, src, room); // capacity fill before flush
o_p += room; outSize += room; src += room; // progress
}
if ((actual = oo_o->owrite(o_0, o_p - o_0)) < 0)
return -1;
outSize += (n32) actual; oo_pos += actual;
oo_p = o_p = o_0; // buffer is empty again
if (plus > e_oobody/4) { // plus won't fit in the buffer?
if ((actual = oo_o->owrite(src, plus)) < 0)
return -1;
outSize += (n32) actual; oo_pos += actual;
}
else { // now buf is all flushed, all plus fits inside
::memcpy(o_p, src, plus); // oo_find_lf() scanned all
o_p += plus; outSize += plus;
src += plus; oo_p = o_p; // progress
}
}
else { // room >= inSize: all input bytes fit in our space
When all new bytes still fit in the local buffer despite present content, this block writes them to the local buffer and delays newline scanning for later. |
demos « Þ
+ todo + names + fd + iovec + assert + log + run + hex + crc + buf + in + out + quote + escape + compare + file + deck + cow + arc + blob + tree + slice + rand + time + stat + hash + heap + node + primes + page + book + pile + stack + atomic + lock + mutex + thread + map + meter « Þ + list + iter + ctype |