Stephen Laniel’s Unspecified Bunker

Fairly strongly dis-recommended. Even though my former coworker Tom had many good things to say about T.C. Boyle, it’s going to be hard for me to pick up another of his (Boyle’s) books. I may give A Friend of the Earth a try at some point, but I expect that what I’ll get will be similar: while World’s End is a rather ineffectual slap at hippies (with other story elements, but no believable characters), A Friend of the Earth will probably be an ineffectual slap at radical environmentalists. World’s End is the sort of novel that David Brooks would write.

It’s time to finally unsheathe, for the attention of a candid world, a hypothesis that has, over time and through the efforts of many devoted scientists, attained the level of near-unassailable fact. This is the Net Level of Attention Hypothesis, first brought to light by Josh Wretzel in 1997 or so. The Hypothesis is simply this: the amount of attention that person A pays to person B, plus the amount of attention that person B pays to person A, is constant. For this reason, perhaps it should be called the Conservation of Attention Hypothesis. Labelings aside, it is one of the most powerful tools available in the logic of human relations, and helps explain why Jenny lavishes such attention on Dexter when Dexter is so obviously preparing to dump her, Jenny should totally see that and should know that she could totally do better than Dexter, and besides he’s obviously been flirting with Andrea. The NLAH predicts exactly this behavior, and many others besides.

I will be in California this weekend, where I will in fact be meeting with Mr. Wretzel; impromptu conferences on the subject of the NLAH are likely to form over strange-colored beverages. We’ll report any additional findings here, or in our newly-minted peer-reviewed journal, He Ain’t Gon’ Return Yo’ Calls, Girlfren’, So You Best Find You A Real Man.

The Red Sox are in the middle of an 11-game winning streak, which means that the Boston press have to be gloaty (article included below the fold, because Boston.com takes stuff off the web very quickly). When the Sox are winning, it’s all “what color will our World Series bunting be?” all the time; when they’re losing, it’s  . . .  a bit more muted.

Me, I fear the Yankees, even if they’re 3.5 games back from the Sox. You can’t ever count them out. By the end of the season, I expect that it will be a much closer race for the ALCS.

P.S.: The Mets are 12 games ahead of their nearest competitors? My god. If there is, in fact, a Sox-Mets World Series, then I will have no choice but to camp out in the Maison Josh Wretzel for its duration and spend some time engaging in the talking of the trash.

(more…)

I am perhaps more excited than I ought to be, having ordered WHA-BAM and KA-CHOW from Alibris.

They’re both edited by Bernard Bailyn, whose Ideological Origins of the American Revolution is fascinating and very well written. I’m excited to read the contents of the actual debates that the Framers had. Nowadays the Anti-Federalists get virtually no coverage, as compared to their Federalist brethren; it will be good to see them on a level playing field.

Useful little discovery today: I was just now writing a little script that started off like so:

#!/usr/bin/perl use warnings; use strict; my ($dirName, $dumpFile) = (shift,shift) || die "dirName and dumpFile not supplied"; print "Dirname = $dirName\n"; 

What that should have done, if Perl adhered to Steve Laniel Syntax, is put the first and second command-line arguments into $dirName and $dumpFile, respectively. But as it happens, it reversed the order. The reason, I presume, is that Perl makes no guarantees on the order in which it processes operators that have the same precedence — like the two instances of the shift operator here. So it actually ran the second shift before the first, and all hell broke loose. Metaphorically.

I remember a similar example, lo these many years, in K&R. I went something like this: if you wanted to copy an array a into an array b, you could do something like

while( ++a* = ++b* ) {} 

Something like that. Does anyone remember the details? Anyway, I seem to remember that this didn’t quite work, because there was no guarantee that the ++b would execute before the ++a. I’m fuzzy on the details, because now that I look at it, it’s not clear that the precedence issue here would cause any problems.

In the Perl case, either

my ($dirName, $dumpFile) = @ARGV; 

or

my $dirName = shift; my $dumpFile = shift; 

would do what we want.

Suppose you’ve got a script that looks like so (in relevant part):

unless( -d $dirName ) { exit 1; } tie( my @lines, 'Tie::File', $dirName ) or warn "Could not tie \@lines to $dirName: $!"; 

Somewhere, the compiler should be smart enough to know the following:

1. The state of $dirName does not change between the unless block and the tie block; hence it still refers to a directory by the time we get to tie.
2. Tie::File cannot tie arrays to directories.
3. Hence, compilation should fail.

Is there any way to make the compiler smart enough to detect this kind of error?

If you thought Perl’s 70 operators were confusing, particularly their precedence, just you wait for user-defined operators. The user-defined ones can be any combination of Unicode characters.

Somehow this strikes me as Not A Good Idea.

Because I know you want this to be All Perl All The Time, I bring you today’s second library discovery: Regexp::Common. It contains a bunch of commonly needed regular expressions, including one that will match against numbers (even ones with separators); one that matches IP addresses; one that matches balanced parentheses and braces; one that matches comments in a number of programming languages; and one that matches profanity. Plus lots of other stuff, but those are the ones that are most useful to me. Actually, the profanity one isn’t very useful, but it’s pretty funny.

I just discovered List::Util, a set of functions for use with lists. I don’t know how I just found this thing. I’ve always felt faintly ghetto about writing my own min() and max() functions, à la;

sub min { my $min = $[0]; foreach(@) { if( $_ < $min ) { $min = $; } } return $min; } 

sub reduce (&@) { my $code = shift; return shift unless @ > 1; my $caller = caller; local({$caller.”::a”}) = \my $a; local({$caller.”::b”}) = \my $b; $a = shift; foreach (@) { $b = $; $a = &{$code}(); } $a; } 

and hence the code for min() is just

sub min (@) { reduce { $a < $b ? $a : $b } @_ } 

I wonder why this is so much faster than the ghetto way.

I’ve gotten my first taste of internationalization recently, as we import data from Microsoft Word (which apparently uses some variant of UTF-16 — UTF-16LE, possibly?), and we need to bring the content into a system that uses ISO-8859-1. This is more of a pain in the butt than one might imagine, though Perl’s Unicode::String module, and its more recent native support for Unicode, make a lot of this easier.

An acquaintance claimed recently that the Unicode people had “totally up the project.” I submit that the problem is just unbelievably hard: a smooth upgrade path from all the world’s encodings, support for left-to-right and right-to-left languages, support for right-to-left text embedded inside left-to-right text (e.g., quoting Hebrew in an English document) and vice versa (and probably so on arbitrarily deep: a Hebrew document quoted within an English document quoted within a Japanese document  . . . ), support for every language in current use and quite a number that aren’t, support for programs that are used to dealing with bytes rather than characters  . . .  there is virtually no part of computing left untouched by this. For that matter, given the number of people touched by this, I assume the politics are unbearably messy.

I don’t know Unicode well enough to say whether they “totally it up,” but my first assumption would be that there’s no perfect solution to the problem, and that even approaching something that everyone can agree to is a headache beyond my comprehension. If all of that is true, and what I’ve read about Unicode is right, then they’ve done an admirable job.

I was just looking over the 2004 Census Bureau report on poverty, and I came to page 10, where there’s a chart of real median household wages since 1967. It shows a general upward trend. But the thing is, more women are working now, so it could be that hourly wages have in fact gone down. What I want, then, is a time series of average hourly wages. Can anyone find a chart that’s as useful for this stat as the Bureau’s charts are for median income?

Another stat I’d like to see is something like “income net of basics.” That is, income net of taxes, minus at least a minimum food cost, minus childcare expenses, minus rent, and so forth. Is that net number called “discretionary income”?  . . .  I guess so. So what I want is a time series of discretionary income. But it would need to be a fairly realistic measure: if more people are required to drive to work than were required to do so in the ‘60’s, the stat should reflect that.

The Bureau of Labor Statistics’ website seems much less user-friendly than the Census Bureau’s. If anyone can help me find numerical goodness, that’d be ace.

I should note here, for the record, that my coworker Mark Pike is just hilarious and smart, and a darned good writer. He will soon be leaving to attend law school at UVA; we’re all going to miss him terribly.

Despite its dryness — or perhaps because of it, depending upon what you’re looking for — Foner’s Reconstruction is a strong, educational read. Like a lot of the books I’ve been reading recently, it’s inspired me to go read about other parts of American history. Charles Sumner will be somewhere on my queue in the upcoming weeks. For reasons near wholly unrelated, I’ll need to read about FDR’s Hundred Days also.

The major trouble with Foner’s Reconstruction book, if you’re not an historian, is that he intends to beat you over the head with the following idea: blacks were perfectly capable of self-government, despite half a century or so of historical scholarship which apparently taught that black self-rule was the worst idea in the history of American government. If Foner really was overcoming this prejudice, then by all means he should be bludgeoning it. From the perspective of someone who’s never doubted black independence, it only detracts from the book.

The evolution of the Republican party from the anti-slavery, pro-human rights party to the conservative party of moneyed interests is quite fascinating; I’ve not yet worked out all the details, nor have I figured out how Northern Democrats became the party that we know and sometimes love, but it goes something like this: Republicans ran the country after the Civil War, so they became the party of political patronage, and the party of the new industrial elite. At the same time, they had to maintain power in the South, which meant coming to some kind of understanding with the Democrats. Meanwhile the old-line Radicals, like Sumner, died off, and the Gospel of Prosperity — that economic development, not black liberation, is what the South needed — took over. Northern prejudices still abounded, and people couldn’t really get themselves to believe that blacks were the equal of any white man. Weak leadership — notably Presidents Johnson and Grant — let the South push back against laws, like the Fourteenth and Fifteenth Amendments, that protected Southern blacks, so that by the 1880’s most of those protections were gone.

That’s the part of the story that I can reconstruct with five minutes of very fatigued thought. This book raised a lot of questions for me, and I know I’ll be stewing over it for a long while.

P.S. (26 June 2006): There’s a very interesting portion of Foner’s book, in the last hundred pages or so, dealing with the battle against patronage, that dovetails in interesting ways with the Hofstadter book that I read recently. Hofstadter mentions that much of the country opposed a civil-service exam because of its supposed elitism. The idea, which gained a lot of traction at the time, was that a civil-service exam would reward those who had only book knowledge, and would block people who had plenty of relevant life experience that would make them fine civil servants. This same critique shows up in Foner’s book, in an interesting way: newly freed slaves also opposed the civil-service-exam requirement, because it really did exclude them from service. Freedmen who had never learned to read would of course be at a severe disadvantage for these jobs. Their objections seem much more plausible to me than those of any other opponent of the spoils system; for the latter, it just sounds like sour grapes.

I’ve complained for years that Perl is excessively idiomatic, and I normally get the response back either a) that the idioms just help you shorten your code, or b) that all languages have idioms, and there’s nothing particularly odd about Perl in that regard.

I think we can accept the truth of both those assertions, yet agree that use English is a good addition — e.g., $EUID to get the user’s current effective UID rather than $<.

I think we can also agree that local variables in Perl are just nasty. They are something like a global, something like a local variable. Mark-Jason Dominus does another one of his well-written Perl exposés to show us what it’s all about, and still I had to stare off into space and ask myself, “Can you really explain the difference between my and local?” I sort of can: you use local when you want to use a global variable with a temporary local value. But then you shouldn’t be using global variables, so the number of cases where you’d do this is fairly limited. In fact it looks like the special Perl punctuation variables (like $/, also known as $INPUTRECORDSEPARATOR) can in many cases not be given lexical scope via my, which means you need to use local to override their global values in particular places.

I’m not yet enough of a Perl wizard to understand the symbol table, and typeglobs and so forth. Understanding this stuff is in fact quite useful, and Dominus uses it to good effect in his Memoize module. Memoize will make a cached version of any function name you give it, and will rewrite the global symbol table so that when you call foo() it’s actually calling the memoized version of foo(). That seems pretty hot.

Anyway, I am perhaps denying myself some efficiency by avoiding local altogether. Dominus makes some light fun of those who use FileHandle to get a handle with lexical scope rather than writing a one- or two-line typeglob hack to accomplish the same thing. Me, I prefer the mindless solution: use FileHandle, and you know that you will never ever have to think of any weird namespace collisions. It Just Works.

P.S.: Reading further on that page, I discovered at least one interesting thing: Perl prototypes are not quite the same beast as C++ prototypes. Actually I didn’t even know about Perl prototypes at all until earlier today, and then I thought they were a Perl 6 development. But it turns out that if you do something like

sub arraysAreSameLen(\@\@); 

and then later on actually assign some content to arraysAreSameLen(), the Perl compiler will do some more checking for you: it will expect that arraysAreSameLen() takes two arrays as arguments. It’s a teensy bit magical, too: the \@ syntax is Perl-speak for “reference to an array”, which is well-chosen notation, because what the compiler is actually doing is turning the arguments that you pass to arraysAreSameLen() into array references. I.e., anyone calling into arraysAreSameLen() would just do

arraysAreSameLen( @array1, @array2 ); 

when in fact arraysAreSameLen() would have to do something like this in the backend:

sub arraysAreSameLen( \@\@ ) { my @array1 = @{$[0]}; my @array2 = @{$[1]}; } 

If you then were to call arraysAreSameLen(@array1), you’d get a syntax error:

(22:05) slaniel@TheloniousMonk:~$ bin/prototest.pl Not enough arguments for main::arraysAreSameLen at bin/prototest.pl line 10, near “@list1)” Execution of bin/proto_test.pl aborted due to compilation errors. 

Which is to say that this compiler instruction

1. adds one more level of safety to your code, of the sort that you’d get from use warnings and use strict;

   and it also

2. helps out your callers a little bit: you handle a bit more of the messiness, by handling referencing and dereferencing objects; they only have to worry about passing you blue-blooded, wholesome, god-fearing objects.

I’m not sure whether Perl prototypes are like their C counterparts, where you can either do

int funcName( char* varName1, int varName2 ); 

or just

int funcName( char*, int ); 

It would be nice if they were; I look for any opportunity to make Perl more literate.

This is all rather more exciting to me than it ought to be. Earlier this week I found some Perl module — I can’t find it just now — that lets you build in some argument checks, and I was all psyched to start using it. But if the compiler itself won’t even let your program run when the argument list isn’t right, then that’s obviously much better.

The next step is to use named arguments like Python uses. One gets this a bit in Perl by using hash arguments, à la

funcName( { arg1 => val1, arg2 => val2, } ); 

which is cool. But it would be nice to have built-in support for it.

P.P.S.: The next next step would be to allow higher-order syntax checks at the compiler. E.g., I want to ensure that arguments which are supposed to be probabilities are really between 0 and 1; is there any way to force the compiler to die if this condition isn’t satisfied?

I suppose the right way to do this is to create a probability class, which throws an exception or something if it’s not initialized properly. But I wonder if there’s any way to catch the error earlier than an exception — in Perl or in any other language.

Courtesy of coffee, specifically Murky Coffee:

• Running the Spanish-language description of Rayuela through Babelfish produces something that’s sublimely surreal:

  The masterpiece of Julio Cortázar is recognized corno. Of entrance, he proposes to us to choose one of accesses both: to read in the customary order and to finish in the chapter 56 (which they follow more chapters, than denominate like “prescindibles”), or, to follow the “board of direction”, that sends us to another one from a chapter, happening through varied traps or games: an apparent omission, a double and significant shipment . . .  This offers to us, in principle, two different books. Rayuela, nevertheless, is branched off as well in two physical atmospheres: the one “Of the side of there”, in Paris, with the relation of Oliveira and the Magician, the club of the serpent, the first reduction when infier to us of Horacio, et cetera; and the one “Of the side of here”, in Buenos Aires, with the encounter of Traveler and Talita, the circus, manicomio, the second reduction . . .  Style and structure, say Nabokov, make the novel. The perfection that reaches in Rayuela places to us (and this was clear since it saw the light, in 1963) before one of best novels written in our language.

• Speaking of foreign languages, my friend Mike introduced me to Vladimir Nabokov’s “The Art of Translation”. It’s really beautiful, and also ridiculously snarky; that is, it encapsulates, in a five-minute read, all of my experiences with Nabokov thus far.

• D.C. tourists have one trait that drives me nuts: they stand on the left side of escalators. On at least a few occasions, I’ve seen locals yell at tourists to stand aside, and I myself have been tempted. Along with maps of the Mall, tourists ought to be provided with “Guidelines For Peaceful Coexistence with D.C. Residents.” Among the instructions would be:

  1. if you’re not going to walk up escalators, stand to the right
  2. let everyone off the train before you get on
  3. if you’re going to stop walking to stare dumbly around and pick your nose, take a look behind you first; if there’s someone there, keep walking and pull off to the side where no one will run into you.
  4. walk at the same speed as everyone else; if you can’t, move off in some direction where no one will run into you.

• Tourists and residents alike have an annoying habit of walking into subway cars and stopping right there without moving to the center of the car. In large part I think this comes from how the city lays out those cars: right inside of each door is a floor-to-ceiling pole to hold onto. It’s quite easy just to stop and hold on, rather than to move farther in. And once even one person has held onto the pole, it’s harder for anyone else to move past the pole. I believe the Metro (first Google link for “Metro”! Nice job, WMATA) was considering removing those poles a few months ago. That would definitely be a smart idea.

Some notes on Eric Foner’s Reconstruction:

• It is alarmingly dry. At every moment when Foner could have chosen to give us a broad picture of the South, he instead descends into minute description of the particulars of life in individual Louisiana parishes. In nearly every instance where he could have said simply that “Black political power varied from place to place,” and then gone into perhaps a paragraph’s worth of detail, he instead has chosen to use one paragraph per Southern state. Perhaps this appeals to some audiences, namely historians who are looking for rigorous backing behind each claim. Given that Foner’s book was apparently a monumental shift in thinking on Reconstruction, it probably makes sense to document everything this well. However, it fails for the general reader. And if the general reader benefits most by applying historical writing to his current life and times, Foner’s book is particularly useless to him. I can’t recommend it. I hope to finish it this weekend and move on to something more engaging. Having come from Potter’s and Hofstadter’s books, which provided quite ample documentation and yet kept me turning pages, Foner is a letdown.

• I wonder if Karl Marx had anything to say about Reconstruction, specifically Radical Reconstruction. (Presidential Reconstruction was a terrible failure, as was President Andrew Johnson.) For a time, it seemed quite possible that land throughout the South would be redistributed to newly freed slaves, and that legislation meant to serve blacks would also help the poor. Marx must have cheered all of this.

• I wonder whether Foner will help me understand how the South a) descended back into white rule for the next 90 years, and b) convinced poor white folks that their interests were served by racist repression. (See “Only A Pawn In Their Game”.)

• It may be a general truth that you can see ideologies at their starkest and most ridiculous at times of great public debate over those ideologies. Hofstadter’s book includes lots of funny quotes by conservatives during the debate over universal suffrage, who were convinced that a world in which women could vote would rapidly descend into some version of hell on earth. Among the more laughable claims were that women were genetically unprepared to vote, and that by doing so they would become more like men — specifically, that they’d lose their delicacy and turn brutish. All of this was supposed to happen within a generation or two. Books were published with this as their central thesis. I wonder if it looked as ridiculous then as it does now.

  Likewise, conservatives just after the Civil War vehemently opposed land redistribution to blacks, claiming that this would make them dependent on government patronage, and that they should learn the manly virtues guaranteed by acquiring private property through honest toil. I wonder if people at the time recognized the lunacy in that assertion; whites had been profiting off unpaid slave labor for at least two hundred years. If anything, it was the plantation owners who needed to learn the how the “free market” operated. As Foner notes, some people did point out this inconsistency; indeed, it sounds like many Northerners opposed slavery in large part because it encouraged a certain weakness of character that the market would expunge. But I wonder how many people really appreciated the irony. It is, of course, worth asking what ideas we hold today that people will snicker at in 150 years.

  From a 150-year remove, talk of a “free market” during times of slavery is facially bogus. The market — including the slave trade — is free only inasmuch as the government lets it be free. Slaves who wanted to leave their plantations did not have that right, because the government backed slavery with the force of arms to return them to their masters. Without that force, slavery would not exist in the “free market.”

  So it seems to me that if we really want to understand the lunacy of various ideologies, we can do no better than to look at how people defend those ideologies when they’re most intensely under attack. Their real character emerges then.

I’m listening right now to an MP3 of Johnson’s famous “We Shall Overcome” speech before Congress. It is astonishing. I’ve read it before, but never listened to it. The most famous passage is this one:

Their cause must be our cause too. Because it is not just Negroes, but really it is all of us, who must overcome the crippling legacy of bigotry and injustice. And we shall overcome.

As a man whose roots go deeply into Southern soil I know how agonizing racial feelings are. I know how difficult it is to reshape the attitudes and the structure of our society.

But a century has passed, more than a hundred years, since the Negro was freed. And he is not fully free tonight.

It was more than a hundred years ago that Abraham Lincoln, a great President of another party, signed the Emancipation Proclamation, but emancipation is a proclamation and not a fact.

A century has passed, more than a hundred years, since equality was promised. And yet the Negro is not equal.

A century has passed since the day of promise. And the promise is unkept.

The time of justice has now come. I tell you that I believe sincerely that no force can hold it back. It is right in the eyes of man and God that it should come. And when it does, I think that day will brighten the lives of every American.

For Negroes are not the only victims. How many white children have gone uneducated, how many white families have lived in stark poverty, how many white lives have been scarred by fear, because we have wasted our energy and our substance to maintain the barriers of hatred and terror?

So I say to all of you here, and to all in the Nation tonight, that those who appeal to you to hold on to the past do so at the cost of denying you your future.

Go listen.

I’ve written a script to iterate over a large directory tree and find all the hrefs inside of all the HTML files in that tree. The trouble is that it sucks up a great deal of memory when the tree is large. That’s because its basic layout is

my @urllist = (); foreach my $file ( @largesetoffiles ) { push @urllist, urlsinfile($file); } my @slightlysmallerlist = deletejavascripturlsfrom(@urllist); undef @urllist; 

How do I avoid sucking up all the memory I have when the list is very large? This is a very general problem (which you encounter every time you populate a large list), so I assume there’s some solution to it. I want to say “closures” or “iterators”; are those the right approaches?

If you want a newspaper article that eats up every ounce of the Bush Administration’s spin, you couldn’t do better than today’s New York Times report on a secret government program to get financial records from suspected terrorists.

For some reason I was just reminded of this great line:

Guys keep hitting on my wife, which I can understand, so it doesn’t bother me. She looks pretty good, all’s fair. But please, don’t tell me I’m So Lucky or that I’m A Lucky Man. Brenna could not understand why this would make me angry when them kissing her arm or whatever would not. I let her in on a little man secret. When you tell a guy that he is a Lucky Man, you aren’t saying it because she seems like a really nice person. What you are telling him is that you would so fuck that. You would fuck that to pieces.