unofficial planet python

May 17, 2008

Shannon -jj Behrens

Python: Google App Engine: Cookie Users Beware

By default, Google App Engine Web applications runs on yourapp.appspot.com. That means that some other app, e.g. badguyz.appspot.com, can set a cookie for appspot.com, and your app will get that cookie from the user's Web browser on subsequent requests to your site.

This isn't some remarkable new exploit or anything. It's just something to keep in mind when running on subdomains like this. If you're worried about security, you should use your own domain name and cryptographically sign your cookies (here's some example source code).

by Shannon -jj Behrens (noreply@blogger.com) at May 17, 2008 12:07 AM

May 16, 2008

Doug Hellmann

October 2007 column for Python Magazine

Has your multi-threaded application grown GILs? Take a look at these packages for easy-to-use process management and interprocess communication tools.

Read more

This article was originally published by Python Magazine in October of 2007.

by Doug Hellmann (noreply@blogger.com) at May 16, 2008 09:43 PM

Shannon -jj Behrens

Python: Debugging Google App Engine Apps Locally

Python has a wonderful interactive interpreter (i.e. shell). However, sometimes you need more setup before you can start coding. Previously, I wrote Python: Coding in the Debugger for Beginners.

Google App Engine works a bit like CGI in that output to STDOUT goes to the browser. This breaks my normal "import pdb; pdb.set_trace()" trick. However, it's not hard to put STDOUT, etc. back so that you can use pdb in the way you normally would:
for attr in ('stdin', 'stdout', 'stderr'):
setattr(sys, attr, getattr(sys, '__%s__' % attr))
import pdb
pdb.set_trace()

by Shannon -jj Behrens (noreply@blogger.com) at May 16, 2008 09:32 PM

Duncan McGreggor

The Twisted Think Tank

I'm pleased to announce that we've got a new Divmod site up! We're still making tweaks, but it's ready for public viewing, and open for business.

This change currently doesn't affect our subscriber services... but it will, very shortly :-) JP's working on that now.

Anyone who knows us, knows that we know Twisted. We really know it. And how could we not, with Twisted superheroes like Glyph and JP? We've been solving very interesting problems for the past couple years, and other companies have availed themselves of this expertise. We're no longer trying to hide from our destiny as "the Twisted company."

We've found that providing specialized consulting services has not detracted from our core competency as software developers, but has rather done quite the reverse: provided a great deal of insight and clarity. The two activities have established a complementary feedback mechanism for growth and invention.

Technorati Tags: , , ,

by Duncan McGreggor (noreply@blogger.com) at May 16, 2008 07:44 PM

App Engine Haiku

I've been playing with Google App Engine a bit more tonight, and I've come to some additional conclusions: 
  1. For the best results, you gotta drink the whole pitcher of koolaide; and
  2. It's really quite tasty koolaide.
One of the things I really wanted to test was imaging. I use Flickr far more than Picasa, but Flickr's authentication API is kind of a pain in the ass (especially when used with App Engine). Google auth is (for obvious reasons) easier to work with. This, of course, led me to explore Picasa. To my surprise, it was delightful to use. I chopped up gdata for only the parts needed to support Picasa and was uploading images within mere minutes. And not only was the effort minimal, but the performance was outstanding.

One of the things that makes application development a more efficient process is having a unified platform on which to work. This is why working with the Divmod platform is so nice when I do extensive Twisted application development: all the infrastructure is already built and ready for me to use. Today, I got to really taste that same experience with the Google platform. And I liked it.

I did notice one interesting psychological side effect of working with the file count limitation: I inadvertently treated it like a game. Not unlike the rules in poetry, it provided a structure and bounds within which I was forced to operate, adjust for, and test my creativity against. I rather enjoyed this microgame, and that was a surprise :-)


Technorati Tags: , ,

by Duncan McGreggor (noreply@blogger.com) at May 16, 2008 05:29 PM

Tales of a Programming Hobo - Christopher Armstrong

Brian Jones

Social Media, The Future of News, and Data Mining

I went to a very good panel discussion yesterday hosted by the Center for Information Technology Policy at Princeton University. There has been a conference going on there that covers a lot of the overlap between technology, law, and journalism, and the panel discussion yesterday, Data Mining, Visualization, and Interactivity was even more enlightening than I had anticipated.

The panel members included Matt Hurst, of Microsoft Live Labs, Kevin Anderson, blog editor for The Guardian, and David Blei, a professor at the Computer Science Dept., Princeton University. This made for a very lively discussion, covering a wide range of perspectives about social media, “what is news?”, how technology is changing how people interact with information (including news), how the news game is changing as a result (which was far more fascinating than it sounds), and how this unfathomably enormous stream of bits, enabled by lots of open APIs, feeds, and other data streams can be managed, mined, reduced, and presented in some value-added way (part of the value being the sheer reduction in noise).

Cool Tools for Finding News

Some of the tools presented by the panelists were new to me, and aside from being great tools for bloggers and other content publishers, there are some excellent examples of how to make effective use of the data you have access to through APIs like the Digg API.

BlogPulse

This was presented by Matt Hurst. It’s is pretty neat - it’s a tool that essentially charts blog buzz of a given phrase over time, and it even lets you compare multiple phrases, which is really interesting as well. Check it out here.

I’d like to know more about how it derives the metrics, but in doing a couple of quick comparisons using the tool, it seems to line up to some degree with simple comparisons of the number of search results for different phrases on sites like technorati and bloglines. Interestingly, even though there appears to be lots more data available at Technorati, in my very limited experimenting, the percent difference between search results for any two phrases appears to be similar, indicating that bloglines may be a representative sampling of technorati data. More experimentation, of course, would be needed to lend any credibility whatsoever to that claim. It’s probably irrelevant, because you can’t ask either service for any kind of historical data regarding search results :)

Twistori

This has the potential to be really interesting. Right now, it lets you pick from several different terms, like “love”, “wish”, “think” and “feel”, and after clicking one of those, it’ll start producing a constantly updating stream of twitters that contain those words. If this experiment is successful, I would imagine they’d eventually enable the same service for arbitrary keywords, which would be really powerful, and quite a lot of fun!

Tweetwheel

Oh how boring my life according to twitter is. I’m still in the schizophrenic stage of settling on a live ‘update your friends on what you’re doing whether they care or not’ services. Facebook, myspace, twitter, jaiku… there are too many. I’m trying out the imified route now to consolidate all the cruft. According to tweetwheel, there are more places to update my status at any given moment than there are people who give a damn what my status is.

Anyway, tweetwheel shows how you’re connected to people through twitter. If you have lots of followers and follow lots of people, the wheel is really exciting to look at, as displayed by Kevin Anderson, who has a much more “robust” wheel than me — it’s actually interesting to look at. At some point I’d like to see this idea expanded to cover the other services like Facebook and even LinkedIn.

Digg Labs

You have to go to the Digg Labs site and see what people are doing with the Digg API. There are too many awesome utlities to cover them all here. It almost makes me wish I did fancy Flash UI stuff instead of back end data mining and infrastructure administration.

At a higher level…

Most of the discussion about social media seems to be about measuring buzz created by bloggers (at least where news/content publishing is concerned). However, although things have shifted dramatically in a ‘consumers are producers’ direction, causing people to start rethinking the definition of news, this shift is caused as much by consumers who are still *only* consuming as anyone else, and I didn’t see much in the way of tools that measure the interest of those people in any meaningful way. Perhaps the consensus is that the bloggers are a representative sampling of the wider internet readership? I don’t know. I would disagree with that if it were the case.

I work for AddThis.com, which seeks to provide publishers of news and all kinds of other content with statistics that help them figure out not just what pages people happen to be landing on, but which ones they have elected to take a greater interest in, either by emailing it to a friend, adding it to their favorites, or posting it to digg, delicious, or some other service. Maybe some day there will be an AddThis API that’ll let you easily do even more interesting things with social media.

by m0j0 at May 16, 2008 03:39 PM

Simon Wittber

Dear lazyweb...

I'm setting up my Python dev environment on a new macbook... and easy_install is running like treacle down a hill. It is so slow!

Anyone else experienced this, and worked out what is wrong? I'm guessing its something network related...

by Simon Wittber (noreply@blogger.com) at May 16, 2008 02:45 PM

IronPython Url's

PyCon Italia Presentation Materials Online

Menno Smit has put the materials for his presentation on IronPython to PyCon Italy online (the materials are in English). This includes the S5 presentation, sample code and an example Resolver One spreadsheet:

by Fuzzyman (noreply@blogger.com) at May 16, 2008 12:47 PM

Spyced

Quick tip for debugging with Jython

Currently, Jython ships with the pdb debugger module from Python 2.3. Unfortunately the 2.3 pdb is primitive even by command-line debugger standards. (For instance, if the program you are debugging throws an exception, it will take pdb down with it. Seriously. Did anyone actually use this thing?)

Fortunately all you have to do to get a much better experience is grab pdb.py, bdb.py, and cmd.py (for good measure) from a 2.5 CPython installation and run against that instead.

I've only tested this with Jython trunk but I think it should Just Work with the 2.2 release, too.

by Jonathan Ellis (noreply@blogger.com) at May 16, 2008 12:40 PM

Ian Bicking

pdb in the browser

People have asked me a few times about evalexception and pdb — they’d like to be able to use something like pdb through the browser, stepping through code.

The technique I used for tracebacks wouldn’t really work for pdb. For a traceback I saved all the information from the frames — mostly just the local variables — and then let the user interact with that through the browser. But with pdb you pause the application part way through waiting for user input, and the routine only completes much later.

While writing WaitForIt I played around with techniques to deal with very slow WSGI applications. Not that hard, really — you launch every request in a new thread, and you manage those requests in an application of its own. So I started thinking about pdb again, and it started seeming feasible. Whenever the app reads from stdin it goes into an interactive mode, showing you what comes out on stdout and letting you add input to stdin. It’s nothing specific to pdb really.

So, with a bit of hacking, I added it into WebError (which is an extraction of the exception handling in Paste). To give the demo a try, do:

hg clone http://knowledgetap.com/hg/weberror/
cd weberror
python setup.py develop
# You need Paste trunk:
easy_install Paste==dev
python weberror/pdbcapture.py

What you’ll see is not polished, it’s just working, but since I mostly did it to see if I could do it, that’s good enough for me.

by Ian Bicking at May 16, 2008 07:36 AM

Nick Efford

Mercurial

In the past, I’ve used Subversion for the one-person software development projects with which I’m often involved. Just recently, however, I thought I’d give Mercurial a spin, having heard good things about it and seen a lot of people start using it.

So far, I’m impressed - particularly by the ease with which you can clone a repository to create a new branch for development, and then merge changes made there back into the original. Another nice feature is the ability to view repository changelogs with a web browser, courtesy of Mercurial’s built-in web server. And then there’s the fact that large parts of it are written in Python… :)

by Nick at May 16, 2008 07:16 AM

Arc Riley

some real progress toward beta-3

With Pyrex 0.9.8 released yesterday, including a number of key changes we needed to move forward, we're back in action for PySoy belated Beta-3 release.

The new forward declarations are a bit kludgy (example), and I don't think it'll be very maintainable in the long term. I think changing a class's inheritance should require editing one file, not two (the .pxd) or more (every .pxd that uses it).

It works for now, until we can wipe out the entire include/ directory, and to be fair it's only so cumbersome due to PySoy's size and how the classes are interconnected.

The nogil allowance for cdef methods is a massive step toward stomping out the multicore race conditions. One of the developers testing just the fixes to soy.bodies noticed the examples often running longer than before, though there's a lot more cleanup to be done before this is package-wide.

One of the roadblocks I hit is the nogil fix can't be used with our Children class, which stores pointers to Python objects of the same type and uses those pointers (through a typecast) to access their cdef methods in a for loop.

While our usage is GIL-safe, the current version of Pyrex still wants to throw in an INCREF just to be extra-cautious, which makes it not. I've let Greg know about this, we'll see if it can't be fixed soon.

In the meantime, I've called for an IRC sprint this weekend for the pending cleanup, refactoring, documentation, and testing that needs doing before beta-3. Join #PySoy on irc.freenode.net if you want to pitch-in!

by Arc Riley (noreply@blogger.com) at May 16, 2008 06:42 AM

Jesse Noller

Sausage on the brain.

Mandlebraut: Rendering Sausages.

These are the ideas I get when I'm coding and thinking about food.

by jesse at May 16, 2008 01:30 AM

Getting Processing into the stdlib

I shot an email out to Python-Dev earlier this week asking for comments/questions regarding my push to get the Processing into the standard library. There's been some decent discussion about target releases and other meta-issues around getting it in.

Right now, it looks like I am going to try to target 2.7 and 3.1 - this makes sense for a few reasons.

  • First, the PEP deadline was uh, a year ago for 2.6 and 3.0
  • There's some cleanup on the module which needs to be done
  • There might be some renaming requirements
  • Need to talk to R. about a 1.0 release
  • Need to chunk out some time to convert the tests to unit test format.

That all being said - it doesn't look unfeasible to accomplish - and the response both on list and to me privately has been 95% +1 and 5% -.5 and -1 - the positive response really does make me feel that this is the right approach to take.

I am currently working on revised benchmarks for processing vs. threads vs. pp vs. other right now - I'll be publishing those as soon as I complete them to both here and the mailing list discussion as a counterpoint to some of the open questions.

I'd like to see if any of you, oh internet people, have anything else you'd like to have answered for this or anything you'd like to add to the discussion.

Note, I am not trying to solve the "distributed" problem with the inclusion of this - the remote capabilities of the processing module are a side-benefit - not the primary benefit to trying to get this in. I am taking some of the distributed stuff mentally into account - but the goal is to scratch one specific itch - not to solve everyones problem with a single addition.

Now all I have to get over is some bizarre errors with parallel python ramming into ulimit, uh, limits. Luckily I have everything from a dual core to an eight core to hack on!

by jesse at May 16, 2008 01:10 AM

Ned Batchelder

Water + laptop = ok

This past weekend, someone accidentally spilled about half a cup of water onto my laptop keyboard. I actually shouted "Agghhh!" like a cartoon character, and we all burst into action, grabbing paper towels, mopping up water, moving books, etc.

When I picked up the laptop, water came dribbling out the other side of it, making a puddle on the table. It was not the kind of thing you want to see involving your laptop. The screen started flashing and going into video conniptions, so I flipped the computer over and removed both batteries.

Max searched Google for what to do, and found a page recommending opening every part you could, and blow-drying on the coolest setting. So I got a screwdriver and started opening panels. When I got the hard drive out after 20 seconds and three screws, Max (a hardcore Apple devotee) exclaimed, "Wow, you can get it out just like that!?" I was a little pleased to have at least a small advantage go to the PC.

Max also found the service manual for my model laptop, so we could get under the keyboard and other hard-to-reach places. The scary thing was that no matter where I looked, I saw droplets of water, except on the RAM.

Over the course of two hours, we blow-dried it a few times, marveled at the insides, and discovered new places to poke paper towels to get the last of the moisture out. Finally, I put it all back together, put in the batteries, and booted it up. With a sigh of relief, I saw that it was going to be fine.

Once it was all over, my computer was cleaner, and I learned something to add to my Middle-Aged Man repertoire of minor but useful knowledge.

May 16, 2008 12:58 AM

May 15, 2008

Making It Stick (Patrick Logan)

Armin Ronacher

Mail Subjects

I found two mails in my Junk folder today that where ham. I spotted them by accident as the subjects where very stupid. The first mail arrived with “Re:” as subject, the second with “Hi”. Please make my life easier and use subjects that don’t look like spam. Thanks in advance

And no, I haven’t written a mail without subject in the first place.

by Armin Ronacher at May 15, 2008 07:01 PM

Grig Gheorghiu

Encrypting a Linux root partition with LUKS and DM-CRYPT

One of our customers needed to have his Linux laptop's root partition encrypted. We found a HOWTO on achieving this with RHEL5, and we adapted it for CentOS 5. The technique is based on LUKS and DM-CRYPT. Kudos to my colleague Chris Evans for going through the exercise of getting this to work on CentOS 5 and for producing the documentation that follows, which I'm posting here hoping that it will benefit somebody at some point.

* Boot off of a Live CD, I used Fedora Core 9 Preview
* Find out which disk is which; for me /dev/sda was the external usb, and /dev/sdb was the internal
sfdisk -d /dev/sdb | sfdisk /dev/sda
pvcreate --verbose /dev/sda2
vgextend --verbose VolGroup00 /dev/sda2
pvmove --verbose /dev/sdb2 /dev/sda2 # This takes ages
vgreduce --verbose VolGroup00 /dev/sdb2
pvremove --verbose /dev/sdb2
fdisk /dev/sdb
* Change the partition type to 83 for /dev/sdb2
* Here is when you get to choose the password that will protect your partition:
cryptsetup --verify-passphrase --key-size 256 luksFormat /dev/sdb2

cryptsetup luksOpen /dev/sdb2 cryptroot
pvcreate --verbose /dev/mapper/cryptroot
vgextend --verbose VolGroup00 /dev/mapper/cryptroot
pvmove --verbose /dev/sda2 /dev/mapper/cryptroot # This takes ages
vgreduce --verbose VolGroup00 /dev/sda2
pvremove --verbose /dev/sda2
mkdir /mnt/tmp
mount /dev/VolGroup00/LogVol00 /mnt/tmp
cp -ax /dev/* /mnt/tmp/dev # I said no to overwriting any files
chroot /mnt/tmp/
(chroot) # mount -t proc proc /proc
(chroot) # mount -t sysfs sysfs /sys
(chroot) # mount /boot
(chroot) # swapon -a
(chroot) # vgcfgbackup

For the initrd, the blog mentions /etc/sysconfig/mkinitrd as a file. CentOS had a directory, I tried doing their suggestion as a file in there, moving the directory out, and making the file as they suggested. Both failed. So I ran the following command:

(chroot) # mkinitrd -v /boot/initrd-2.6.18-53.el5.crypt.img --with=aes --with=sha256 --with=dm-crypt 2.6.18-53.el5

Now we need to modify the initrd so that it will decrypt the partition at boot time

(chroot) # cd /boot
(chroot) # mkdir /boot/initrd-2.6.18-53.el5.crypt.dir
(chroot) # cd /boot/initrd-2.6.18-53.el5.crypt.dir
(chroot) # gunzip < ../initrd-2.6.18-53.el5.crypt.img | cpio -ivd

Now, we need to modify init by adding the following lines after the line which reads “mkblkdevs” and before “echo Scanning and configuring dmraid supported devices.”:

echo Decrypting root device
cryptsetup luksOpen /dev/sda2 cryptroot
echo Scanning logical volumes
lvm vgscan --ignorelockingfailure
echo Activating logical volumes
lvm vgchange -ay --ignorelockingfailure vg00

Copy cryptsetup and lvm to be put into the initrd, the blog doesn't mention it, but I'm sure it needs it.

cp /sbin/cryptsetup bin/
cp /sbin/lvm bin/

Compress the new initrd

find ./ | cpio -H newc -o | gzip -9 > /boot/initrd-2.6.18-53.el5.crypt.img

Modify the grub.conf. Copy the grub entry for the current kernel, and change as follows

title Centos Encrypted Server (2.6.18-53.1.4.el5)
initrd /initrd-2.6.18-53.el5.crypt.img

Unmount the fs's in the chroot, and exit

cd /
umount /boot
umount /proc
umount /sys
exit

NOTE: Don't upgrade the kernel without upgrading the initrd and grub.conf.

Reboot and test :)

At this point you have an encrypted root partition. You should be prompted for a password during the boot process (the boot partition is not encrypted). If somebody steals your laptop, they won't be able to mount the root partition without knowing the password.

After you have crypto setup, you can find out information about it (such as the crypto algorithm used) via this command:

# cryptsetup luksDump /dev/sda2
LUKS header information for /dev/sda2

Version: 1
Cipher name: aes
Cipher mode: cbc-essiv:sha256
Hash spec: sha1
Payload offset: 2056
MK bits: 256
MK digest: af 2e e6 39 3e 79 60 bb 4a 2b 33 05 1c 86 3a 83 bc a0 ef c1
MK salt: 79 b2 13 53 6f 52 72 a1 b5 3d dc d3 72 cd d6 f4
e3 25 3c 6e 08 00 f3 1d 44 1e 90 47 bc 43 e7 07
MK iterations: 10
UUID: 721abe52-5122-447b-8ed0-5ca3b2b32366

Key Slot 0: ENABLED
Iterations: 247223
Salt: 86 c7 53 6a 13 a9 77 81 89 ec 90 b3 e5 6a ea 8d
da 0c 6f ad ec 3e 3c 47 2d 6e 5f 59 28 4e 7c 63
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED

by Grig Gheorghiu (noreply@blogger.com) at May 15, 2008 06:34 PM

Low Kian Seong

getting tapping to work on fc9

Currently i am running fc9 on my trusty old acer and one of the initial irritants I found was that tapping on the touchpad no longer works (In opensuse it worked just fine out of the box). This is a real annoying bug as I am used to tapping my way around using my touchpad. Scouring the fedora forums, I finally chanced upon this entry in the redhat bug list. Well, well, well it turns out the "bug" isn't really a bug after all. A long story short basically your package of synaptics from fedora has a patch that has disabled tapping on touchpads because some wiseguy thought that it was for the best since no one really likes tapping on a touchpad.

So a guy called Bob Kashani was kind enough to rebuild the packages and remove the the offending patch. All you have to do is download his rpm from :

http://www.ocf.berkeley.edu/~bobk/packages/

Run a rpm -Uvh, do a restart of the Xorg by doing the old ctrl-alt-backspace and you are good to go :)

Thanks bob for the package. It works great here !

by lowks (noreply@blogger.com) at May 15, 2008 04:39 PM

Simon Willison's Weblog

Using Git as a versioned data store in Python

Using Git as a versioned data store in Python. gitshelve supports the same interface as Python’s built-in shelve module but stores things to a versioned Git repository instead of just a pickled dictionary. I’ve been casually wondering what a Git-powered CMS would look like.

May 15, 2008 03:25 PM

Mark Rees

OSDC 2008 - Sydney - Call for Papers

The Open Source Developers' Conference 2008 is a conference run by open source developers, for developers and business people. It covers numerous programming languages across a range of operating systems, and related topics such as business processes, licensing, and strategy. Talks vary from introductory pieces through to the deeply technical. It is a great opportunity to meet, share, and learn

by Mark Rees (noreply@blogger.com) at May 15, 2008 01:10 PM

Peter Bengtsson

IronPython Url's

IronPython in Action: New chapters, website and sourcecode

Three new chapters (on WPF, ASP.NET and system administration) are now available in the IronPython in Action Early Access Program. There is now a new IronPython in Action website, where the sourcecode to accompany the book (well - the first eleven chapters so far) can be downloaded:

by Fuzzyman (noreply@blogger.com) at May 15, 2008 12:46 PM

Eric Florenzano

Revolutionary Ideas

Everyone has had the experience of hearing about something new and thinking: "That makes so much sense! Why didn't I think of that?" For programmers that keep up on open source software, new projects that fit the previous description attract not only our admiration, but we want to be a part of this new idea. We become involved and contribute and try to push that new software into any new direction that we can; learning from it and evolving it along the way.

One such idea that fits my description perfectly is Processing.js. Not to belittle John Resig's hard work in actually developing the initial codebase, but the idea is what is so much more important. Thousands of developers knew of both the Processing language and about the canvas tag which is coming to prevalence, but it was a revolutionary idea to notice that the pairing of the two was "both possible and desirable to do in the first place", as Reddit commenter MarshallBanana pointed out.

As a community we need both the revolutionary ideas and the evolutionary changes so that we get great software that solves problems in new and innovative ways, but also that doesn't have bugs and provides a polished experience. But I think that we've become too bogged down in the evolutionary. We get so wrapped up in others' ideas--so interested in polish and shine--that seldom few think outside the boundary of the incremental. I won't claim to be the exception here, and rightly can't claim to be, but it's something that's worrisome nonetheless.

I think that a big part of it is that the open source community has gotten so wary of experimentation with well-established applications. Why can't a development version of Firefox include a Python or Ruby interpreter alongside a JavaScript interpreter? Why can't CSS directives for reflections be explored, or animations be built into the rendering engine? I think that a big part of it is because we've spent so long talking about validation and standards that we forgot about that sense of wonder; that feeling of anything being possible with a bit of code and enthusiasm.

Processing.js, and projects like it, give me hope that revolutionary ideas are still out there. They rekindle that sense of wonder in me. They make me think about other things that are possible. They make me excited about open source again. Let's foster more and greater and better ideas, and just once in a while, eschew the incremental.

May 15, 2008 11:18 AM

Ted Leung on the Air

Scala liftoff

I stayed around in San Francisco for one more day after JavaOne, in order to attend the Scala liftoff. The liftoff was an open space style conference (which has a more specific meaning than “unconference”, at least to me). My friend Kaliya Hamlin did a great job of facilitating the day.

Scala liftoff 2008

Scala has steadily been gaining attention, and hasn’t yet hit (at least in my eyes) the hype part of the classic Gartner hype cycle. I’ve been poking about with Scala, mostly because of the type inferencing, the Actor library, and lift. I have great respect for the work that Martin Odersky has done over the years, which also has me interested. Couple that with what I learned about closures in Java at JavaOne, and the list of reasons to look more deeply at Scala is getting long, especially if you are determined to have a statically typed languages.

Scala liftoff 2008

I wasn’t able to make it to any of sessions on lift. It just worked out that other sessions overlapped them in a pathological way. While this is unfortunate, I am sure that I’ll be able to pick up anything that I need from the mailing lists and other documentation. I was able to attend two sessions on actors. One of the sessions had people with questions about actors, but no Scala actor experts were in that group. There was some discussion of Pi-calculus and the join calculus, but no discussion of the actual actor theory.

Steve Yen’s session on actor-d was pretty useful. Steve set out to build a version of memcached using Scala’s actors. He spent most of his slot talking about Scala/Java isms that he ran into - this was important since he was comparing to the C memcached. By the time he got to the actor related stuff, he was almost out of time. Steve found that he had to remove actors from the main loop of his server in order to get sufficient performance. He wanted to get statistics from the server in the background and discovered that he main loop actor was always processing messages and was never idle long enough to report statistics. He ended up replacing the actor with plain old Java Threads (POJT?). This was in addition to all the fact that he ran into many of the standard Java problems as well. I’m not sure what to conclude from this. I don’t recall what kind of hardware he was on, and I am not convinced that he had the right architecture for an actor based system. Some of his experience also seemed contrary to what the lift folks have been claiming. I think that we are in for a decent amount of investigation here. One of Martin’s statements about Scala is that it is possible (and better) to extend the language via libraries than via actual language constructs. For the most part, I agree with this, but there are certain extensions which have interactions with the runtime - like concurrency. In those cases, I don’t see how the library approach allows taking advantage of runtime features. The current version of Scala actors is implemented as a library.

One of the things that I am currently working on is support for Python in NetBeans, so I dropped into the session on IDE support for Scala. With the exception of IntelliJ, none of the IDE plugin principals were present, so it was hard to have a really productive discussion. Martin did attend the session and we talked about the possibiliy of getting hooks into the existing Scala compiler, particularly the parser and the type inferencer. That could yield some big dividends for people working on IDE support. One IDE feature that I would like to see is the ability to hit a key, and have the IDE “light up” all the inferred types, overlaid on the existing program code. This would allow developers to see if their intuition about the types actually matched that of the type inferencer. I’d like a feature like this for Python/Ruby/Groovy/Javascript code as well. Further discussion was deferred to the scala-tools mailing list.

Scala liftoff 2008

The other session that I participated in was the session on Scala community and governance. Several people wondered about this during Kaliya’s “What questions do you have about Scala” portion of the schedule building. When nobody else put up a session in this area, I grabbed a slot, hoping to spur some conversation - if for no other reason than my own education. Fortunately, Martin had already been thinking about the problem. He is going to adopt a Python style governance, with him (and EPFL) having the final say on language design matters. There will be Scala Enhancement Proposals (SEPs), like the Python PEPs. I’m very happy with this. I think that Python has done very well at maintaining the balance between (lots) of community input on the language design, while still retaining that “quality without a name”. One of the things that I said during the CommunityOne general session panel was that particular individuals in the right place, at the right time, matter at great deal. After watching Martin for the day, and seeing his interactions on the mailing list over the last few months, I think that the design of Scala is in very good hands.

We also talked about the evolution of the Scala libraries. The Scalax project is working to build a set of utility libraries for Scala. Martin views scalax as a place where anyone can submit a library, have it tested, vetted, reworked, etc. Eventually some code in scalax would be candidates for addition to the Scala standard libraries. This also seems like a sane approach to me. I like the idea of having a place for libraries to shakeout before going into the standard libraries. Martin also mentioned a LINQ in Scala project. I need to track that one down too.

It is good to be in a multi-language world again. There’s room for Scala, Python, Ruby, and others. Another language that I am keeping my eye on is Newspeak.

by Ted Leung at May 15, 2008 05:55 AM

May 14, 2008

S.Lott

Keeping the Customer Satisfied


One of the standard lies is the "Cone of Uncertainty". At inception, we don't know the total cost; but we'll get better at projecting the cost. At some point in the future, we'll predict with perfect accuracy. Bunk. Total Bunk. But, customers love to hear it, so we repeat it. With charts.

May 14, 2008 10:53 PM

Martijn Faassen

In Memoriam Joachim Schmitz

This is a rather unusual blog entry. I sincerely hope I don't have to write blog entries like this very often. I usually write about technology, and the community surrounding it. This community is, of course, composed of people. Yesterday I heard that Joachim Schmitz, long-standing member of the Zope and Python communities, had died suddenly last weekend. He was a regular presence at Zope-related events, and I have met him often. I feel that his passing should not go without notice for people in our community who knew him and met him.

I last saw Joachim just a week before he died, at the Grokkerdam sprint here in Rotterdam. Here is a picture of the sprint, winding down on a Saturday night, with people working and playing. Joachim is in the bottom right corner, looking pensively at the laptop in front of us, while I'm explaining some code to him. It's Joachim as I knew him, in his element, surrounded by fellow enthusiasts.

http://faassen.n--tree.net/grokkerdam_joachim.jpg

I wrote the following message to his friends and family, and I want to share it with everybody. I wrote it for myself, as someone who knew Joachim and will miss him, and also for our community.

I know Joachim as a member of the Zope community. I want to share a little about what Joachim meant to us in the Zope community.

Zope is software. I hear that he talked a lot about Zope in his private life, so you might have heard of it. He talked about it even to people that don't know much about software. I understand this perfectly, as I have the same habit. Zope, to him, like to me, is more than just a boring tool you use in your work to build web applications. It's something you can be passionate about, like a craftsman can be about his tools. It's also a community: people you know, who you work with, and who you like. It's a community of people that have known each other for years.

I remember when I met Joachim for the first time. It was at the International Python Conference in Washington DC, in the United States, in late january in the year 2000. It feels very long ago to me now; so much has happened since then. I sat next to him during some talks about Zope. I think he had come to the conference for the same reason I had: to meet other people who work with Zope. Joachim was friendly, and we started talking to each other.

We didn't know then yet, at this first meeting, that we would see each other again on many occassions afterwards, in the years following. His friends and relatives must have noticed Joachim was frequently away at one Zope related event or another. He wasn't away. It's when he was with us! He became a regular face at Zope-related events. I saw him several times each year following the first meeting. He was there so often and so reliably that we joked with him: you again! you're always there!

It isn't a proper Zope event without Joachim there.

Joachim contributed to our community in many ways. One example is his hard work for several years to make sure that people could sign up and pay for the EuroPython conference, helping to make this conference a success.

I saw Joachim for the last time only a little over a week ago as I write this. He was participating in the Grokkerdam sprint, here in Rotterdam. It was yet another Zope-related event: a proper one, as Joachim was there! I sat together with Joachim for a while, working with him and talking to him. Joachim was always interested, always learning, always participating. We were talking about how to teach Zope a few new tricks, planning for the future... See you next time, we told each other, when he left for home.

A week later he was gone, suddenly. He was one of us, and well-liked. This is why I feel so sad now, knowing he passed away. I'm glad I got to see him that one last time.

We who knew him will miss him. Zope events, and EuroPython, won't be the same without him. We will have to get used to him not being there in person with us. But we will remember him. By thinking of him, he is not entirely lost to us. This way, we will still feel his friendly presence in the future.

Martijn Faassen, Rotterdam, the Netherlands

Chairman of the Zope Foundation

May 14, 2008 05:10 PM

Sylvain Hellegouarch

Facebook goes XMPP

Via Peter Saint-André we learn that facebook is adding XMPP as a mean to connect and use its services. Independently from what usage is made of Facebook this is a great news for the Jabber community in general and the protocol itself as it demonstrates that, unlike some nay-sayers used to claim in the past, XMPP will be the the true sibling to HTTP when it comes to social networking and delivering near real-time data on a large scale. Indeed, considering that media companies like the BBC, are looking at using too (even if it's purely at a research level for now) it seems to me that XMPP has great years ahead.

I find that interesting though that Google, which has had a Jabber network for years now, has never been able to actually push much that way. I mean Orkut and Google Talk have been able to communicate for quite some time and yet it seems the Facebook prospect is more exciting for the Jabber community than Google's usage ever was. Maybe it comes down to the fact Orkut is a pale shadow of Facebook.

On a personal level, I won't complain that Google doesn't push advertisement through Google Talk of course but one may wonder why it doesn't do so from a business point of view. well probably they know having ads that way would drive their customers away, as they say (emphasis mine):

There are no ads in your chat sessions or your Quick Contacts list. Once a chat is saved, however, it becomes just like a Gmail message. And just as you may see relevant ads next to your Gmail messages, there now may be ads alongside your saved chats. Ads are only displayed when you're viewing a saved chat, and as with all ads in Gmail, they are matched entirely by computers. Only ads classified as Family-Safe are shown and we are constantly improving our technologies to prevent displaying any inappropriate ads. One of the things many Gmail users have told us is how much they appreciate the unobtrusive text ads in Gmail, as opposed to the large, irrelevant, blinking banner ads they often see in other services, and many have even cited the usefulness of the ads in Gmail.


We'll see how Facebook handles it considering the big fiasco Beacon was.

As a developer I've long felt frustrated at how limited the Google Talk standalone application is. Google Talk gadget is a bit better featured but it pains me having to use it when I prefer its big brother. Still both are very limited. To be fair though most well known IM clients speaking XMPP are limited in regards to what the protocol and its extensions offer. It's sad so little support PubSub and even those that do are somewhat basic in their support.

Let's hope we'll more and more XMPP applications out of the context of instant messaging or with a larger scope like microblogging.

by Sylvain Hellegouarch (nospam@example.com) at May 14, 2008 01:50 PM

Isotoma

Big thanks to Rapidswitch

Our server ISP is RapidSwitch. It’s unfortunate, but most ISPs are pretty poor - there aren’t enough good people to go around, margins are very tight and it’s just the kind of work where it’s hard to keep good people. Last night RapidSwitch showed that at least not all ISPs are poor.

Because of the OpenSSL issues yesterday, we chose to reboot all of our servers last night, to ensure every service was using the new SSL code. A new kernel image came down too yesterday, and a number of our machines had the updated kernel.

We rebooted a number of machines on one cluster simultaneously… and they didn’t come back. We requested a KVM session, but in the meantime one of RapidSwitch’s engineers had noticed 4 of our machines were down simultaneously, so he went and took a look. Proactivity!

He worked out what had happened, and raised a ticket for us, telling us that the new Debian kernel was incompatible with our Network Cards. We asked him to manually boot the machines into the previous kernel, and they came back up without a hitch. Clue!

He then said RapidSwitch were aware of this issue and they were offering a free PCI network card to anyone who needed them. Planning!

Frankly this is unheard of in my experience. Massively well done guys - that’s what I call service.

by Doug Winter at May 14, 2008 10:03 AM

Debian's OpenSSL Disaster

Many of you will know by now of the serious security problems revealed yesterday by Debian, the Linux distribution. We use Debian exclusively for our server platform, so we had to react very quickly to this issue, and make sure our systems were secure again. I think we’ve made all of the necessary changes now to ensure we’re safe from this particular problem.

I have also made some attempt to get to the bottom of what actually went on, and I’ll record it here for posterity. If any of the below is wrong, please let me know!

What Happened

The story, basically, is this. In April 2006 bug #363516 was raised, suggesting that openssl wasn’t clean for valgrind. Valgrind is a package that detects problems in C code, and is widely used to help ensure software is correct. Valgrind reported some errors with openssl, and the reporter wanted to be able to use valgrind with openssl.

At that bug url a change is discussed to the openssl codebase. The general feeling from the bug discussion is that making this change isn’t a good idea, but then a patch was applied on 4th May 2006. There are two instances of the specific issue in the bug, one in ssleay_rand_add and one in ssleay_rand_bytes.

In the meantime, a discussion took place on the openssl-dev list. This mentions the same two lines, and on the 1st May ulf@openssl.org says he is in favour of removing them.

The patch amends the two lines suggested.

The problem, as I understand it, was a misunderstanding by the Debian Developer who made the change. The change to ssleay_rand_bytes was fine - this added some uninitialised memory into the entropy pool, which is fine. But the software doesn’t rely on it for security, and removing it is fine.

But the other change, in ssleay_rand_add, is a complete disaster. It alters the seeding for the random number generator in key generation, a serious flaw.

This reduces the keyspace to a few hundred thousand possible keys. It’s possible to generate all these keys in a few hours, and brute force a machine that’s using public key authentication with a compromised key in a few minutes, potentially. This is a security disaster of the first water, considering the number of organisations (such as ours) that rely on public key authentication for a lot of our inter-machine security. This also affected key generation for self signed email, web certificates, private networks, anonymous proxy networks and all sorts of other things. The cleaning up is going to take some time, and cost an awful lot. Some people are going to be compromised by this, and a lot of machines may be broken into.

Some background on how distributions work

Debian has a vast number of packages under management. They produce these packages by taking source packages from “upstream” (the people who maintain the software) and modifying it to fit the rules and goals of the distribution.

Some of these changes are for compatibility - for example, using standard file locations or configuration systems. Some of them are mechanical changes to do with integration with the build process. Quite a few changes are bug fixes.

It’s recommended that bug fixes be coordinated with upstream - send patches back to them, so everyone in the community can benefit from the changes.

Whose Fault Was It

After going through the above, it’s pretty clearly the DD (Debian Developer) in question’s fault. Although he suggested making changes on the openssl-dev list, and got an affirmative from someone on the project, it was pretty clear in the response that this was “if this helps with debugging it’s a good idea” not “i’ve closely read the code in question, and I agree”.

The DD should have also submitted a patch back to the openssl guys. They’d have spotted the error and screamed blue murder. He was a bit lazy and thoughtless here, and I imagine right now he wishes he could crawl into a hole and die.

What to do about it

Debian are getting badly slammed for this but it is worth keeping some perspective. We, and many others, use Debian because of it’s long history of excellent package quality. This is a result both of their culture (which is aggressively perfectionist) and their selection criteria for developers, which weeds out many dodgy ones. We are proud to use Debian, and will continue to do so.

DD’s are generally conscientious, knowledgeable and dedicated to their work. I have no reason to believe this DD was any different. Even conscientious, knowledgeable and dedicated people make mistakes. This is what process is for, to help mitigate human error. I think there was clearly a lack of process.

Two things would have really helped. Code review internally to debian and code review by upstream. I don’t think it’s unreasonable that for security critical packages Debian should require both for non-critical changes to these packages. Even critical changes should be reviewed as soon as possible.

Internal code review is impractical for every package, since it requires a good understanding of the code in question, and would impose a huge workload - but for critical packages I think it’s a necessity.

Upstream review is potentially tricky too. Some upstreams don’t have the time or inclination to participate. There is also often a lot of friction between distributions and upstream, since they have very different goals. This isn’t a problem that can be easily resolved - these groups really do have different goals and values, and sometimes unreconcilable differences arise. But for the good of their eventual users they need to work together to help stop this sort of problem occurring.

by Doug Winter at May 14, 2008 09:39 AM

Django Weblog

Security fix released

In accordance with our security policy, a set of releases is being issued tonight to fix a security vulnerability reported to the Django project. This entry contains a description of the vulnerability, a description of the changes made to fix it, pointers to the the relevant patches for each supported version of Django and pointers to the resulting releases. A copy of this information will also be posted to the django-users and django-developers mailing lists. The Django website is being updated to reflect the new releases.

Description of vulnerability

The Django administration application will, when accessed by a user who is not sufficiently authenticated, display a login form and ask the user to provide the necessary credentials before displaying the requested page. This form will be submitted to the URL the user attempted to access, by supplying the current request path as the value of the form's "action" attribute.

The value of the request path was not being escaped, creating an opportunity for a cross-site scripting (XSS) attack by leading a user to a URL which contained URL-encoded HTML and/or JavaScript in the request path.

Affected versions

  • Django development trunk
  • Django 0.96
  • Django 0.95
  • Django 0.91

Resolution

The login form has been changed to escape the request path before use as the form's submission action.

The relevant changesets for affected versions of Django are:

The following releases have been issued based on the above changesets:

All users of affected versions of Django are strongly encouraged to apply the relevant patch or upgrade to the relevant patched release as soon as possible.

Release manager's note

If you maintain a third-party Django package and you did not receive the announcement of these releases earlier tonight, please email James Bennett (ubernostrum@gmail.com) as soon as possible.

Also, please note that potential security vulnerabilities should be reported directly to the Django project, at security@djangoproject.com, as outlined in our security policy. Following this procedure helps us to maintain high standards of response and disclosure, and makes the process of investigating and resolving security issues much easier for everyone involved.

May 14, 2008 05:33 AM

Ted Leung on the Air

JavaOne 2008: Part 2

I’ve been to so many conferences and seen so many talks that it’s hard for me to really get excited about conference presentations. I went to talks here and there, but nothing at JavaOne was really reaching out at grabbing me (in fairness, this happens at other conferences also, so it’s not just JavaOne). Or at least that was true until the last day.

Friday opened with a keynote by James Gosling, who served as the MC for a train of presenters on various cool projects.

Cool stuff

First up was Tor Norbye, who has done a lot of good work on support for editing different languages in NetBeans. Tor has been working on JavaScript support for NetBeans 6.1, and he showed off some cool features, like detecting all the exits from a function, semantic highlighting of variables, and integrated debugging between NetBeans and Firefox. All of which was cool. When I was managing the Cosmo group at OSAF, I tried a bunch of Javascript IDE’s and never really liked any of them. I haven’t done a lot with NetBeans 6.1 yet, but I will. Tor showed one feature, which was the killer one for me. NetBeans knows what Javascript will work in which browser. You can configure the IDE for the browsers that you want to support, and this affects code completion, quick fix checking and so on. Definitely useful. Here are several more references on the Javascript support in NetBeans 6.1.

The Java Platform

It’s easy for me (and others, I’d bet) to think mostly of JavaEE or perhaps JavaME when thinking about Java. That’s understandable given the worlds fixation on web applications, and looking ahead to mobile. But the majority of the talks in Gosling’s keynote session had nothing to do with Java SE, EE, or ME (at least in the phone sense).

Probably the hit (applause meter wise) of the keynote was LiveScribe’s demonstration of their Pulse Smart Pen. This is an interesting pen that records the ink strokes that it makes, and any ambient audio that it records while the writing is happening. The ink and audio can be uploaded to a computer, as long as that computer runs Windows (apparently a Mac version is in the works). Unfortunately, the pen works by sensing marks on a special paper (that would be the razor blades), so there’s a limitation on how useful this can be. The presenter said that a future version of the software would allow people to print their own special paper, but that’s still a future item for now. By reading special marks on the special paper, you get a pretty cool user interface. The pen itself can run Java programs, and there is a developer kit available for it. If they can get by the limitation of special paper, I think that this is going to be pretty interesting.

Sentilla showed off their Mote hardware, which seem like RFID chips that can run Java programs. except that these RFID chips can form mesh networks amongst themselves and can have various kinds of sensors attached. There are lots of applications for these things, going well beyond inventory tracking and such.

Sun Distinguished Engineer Greg Bollella demonstrated Blue Wonder, which is a replacement for the computers used to control factories. Blue Wonder combines off the shelf x86 hardware, Solaris, and real time Java to provide a commodity solution for factory control applications. This is far afield of Web 2.0 applications, but just as cool, in my mind.

By the end of the keynote I was reminded of the long reach of the JVM platform, something that I’d lost sight of. The latest craze in the Web 2.0 space is location data — O’Reilly has an entire conference devoted to the topic. I think that sensor fusion of various kinds (not just location sensors) is going to play a big role in the next generation of really interesting applications. The JVM looks like it’s going to be a part of that. I don’t think than any other virtual machine technology is close in this regard.

Java’s future

I also went to a talk on Maxine, a meta-circular JVM. By the twitter reactions of the JRuby and Jython committers, I’d say that Maxine is going to get some well deserved attention when it is open sourced in June. I’m particularly interested because the PI’s for Maxine worked on PJava, and MVM. Given the differences between the Erlang VM and the JVM, I think that the ability to experiment with MVM is going to be pretty interesting. Apparently, there’s already some form of MVM support in Maxine - we’ll find out for sure in June.

During the conference I had a meeting with Cay Horstmann, and at the end of the meeting Josh Bloch saw Cay and wanted to talk to him about the BGGA closures proposal for Java. Turns out that Josh has an entire slide deck which consists of a stream of examples where BGGA does the wrong thing, generates really cryptic error messages, or requires an unbelievable amount of code. The fact that BGGA depends on generics, which are already really hard, doesn’t give me much confidence about closures in Java. If you are a statically typed language fan, I think that you ought to be worried about whether Java, the language, has any headroom left.

The last session that I went to was Cliff Click and Brian Goetz’s session on concurrency. Unsurprisingly, the summary of the talk is “abandon all hope, ye who enter here”. I was glad to see a section in the talk about hardware support/changes for concurrency. The problem is that concurrency is going to introduce end-to-end problems, from the hardware all the way up to the application level, and I think that every stop along the way is going to be affected. Unlike sequential programming, where we are still largely reinventing the wheels of the past, there is no real previous history of research results to be mined for concurrency. Hotspot and other VM’s are close to implementing most of the tricks learned from Smalltalk and Lisp, but those systems were mostly used in a sequential fashion, and while there were experiments with concurrency, there was much less experience with the concurrent systems than the sequential ones. Big challenges ahead.

by Ted Leung at May 14, 2008 04:48 AM

Arc Riley

Building a better Pyrex - part 1

With our recent decision to drop the Pyrex codebase and start from scratch, we find ourselves repeating a process from 2 years ago;

When PySoy first forked off of Soya, we had a small list of things that needed to be fixed based on our negative experiences working with that project. The API needed to be cleaned up, pydocs added, Shape needed to be mutable and implemented with vertex arrays or VBOs rather than display lists, etc.

As we worked on it, that list grew longer, including rewriting large parts of the codebase, until we found ourselves at a point where starting over would be more expedient. Of course now there's little doubt that was a good move, learning from both Soya's mistakes and our own failed early attempts we now have a pretty rocking architecture.

The first and most foundational change we're making from Pyrex, as previously stated, is in the lexical scanner and parser. Both Pyrex and Cython are based on Plex, which was Greg's answer to processing a Python-like language.

In contrast, we'll be using Python's own tokenize module for our lexical scanner and an ASDL-based parser akin to the parser used in Python 3.0. We're extending Python.asdl with cdef, ctypedef, cimport, cinclude, etc.

This way we have less code to debug and a solid foundation to start with. With lexing and parsing almost "for free" we'll be able to focus on the important bits and get the platform able to support the kind of features we need quickly.

We've also reached a rough consensus on the following syntax changes from Pyrex:
  • unicode, generators, decorators, etc will be supported
  • all custom for syntaxes dropped in favor of Python's standard
  • with nogil: replaced with full with support
  • special methods will always have self as first argument (ie, both __sub__ and __rsub__ used)
  • cinclude added for direct C header parsing

There's more to come, which I'll post in additional installments as work progresses on this project.

by Arc Riley (noreply@blogger.com) at May 14, 2008 04:46 AM

Making It Stick (Patrick Logan)

A Little Knowledge

Steve Vinoski is in a bit of back and forth with another blogger about
Erlang vs. other languages, and how easily one could adopt the best
bits of the other. Steve implies, correctly, IMO, that to perform this
kind of assessment requires a fairly good understanding, or at least a
fair bit of research, into the various candidates.

http://steve.vinoski.net/blog/2008/05/09/thinking-in-language-but-not-clearly/

I've seen several attempts to build "erlang in x" where x = common
lisp, python, java, dotnet, etc. To the extent I've followed them they
implement a fraction of the message passing and then don't go much
further. Scala, etc. on the jvm and those on other conventional
runtimes have a problem. Integration with Java instances within those
runtimes is not shared-nothing. You have to remain aware of what is
and what is not following the actor paradigm.

Erlang's JInterface is probably a better integration mechanism in the
long-run than running in the same shared-memory OS process.

As for monitoring, certainly a quick glance would have shown Ted there
is SNMP, as Steve points out. But also isn't the entire
process-linking and supervisor hierarchy an important kind of
"monitoring", i.e. "monitoring for availability"? This is something
all these other platforms lack.

So the more I think about it, the less suited these other platforms
are for messaging amongst themselves *or* for monitoring!

Then I came across Avi Bryant's post on Strongtalk, a high-performance
implementation of Smalltalk, correcting an apparently common
misunderstanding that the speed has something to do with the optional
type checker...

"Why do people make this stuff up? ... Strongtalk was that much faster
whether you used the optional static type system or not. Strongtalk's
optimizing compiler completely ignored the types, and it [types -ed.]
made your program run not one iota faster to add them."

http://www.avibryant.com/2008/05/those-who-misre.html

by Patrick Logan (noreply@blogger.com) at May 14, 2008 02:30 AM

May 13, 2008

Ivan Krstic

Sic Transit Gloria Laptopi



Photo: Walter shows me improvements to the Record activity at the Lima coastline, Peru.

I’ve been displeased with the quality of community discourse surrounding the recent OLPC announcement of moving to Windows as the OS platform. I decided to withhold comment at the time, and was swayed only by the half-dozen volunteers mailing me personally to ask whether all their work had been in vain. It hadn’t. And then I left to travel for a few days.

I just caught up with my mail and RSS feeds, and what I’ve read has moved me from displeased to angry. So I’m going to comment after all, and it’ll be my last OLPC-related essay for the foreseeable future. But first, some background.

The beginning
Throughout his life, Nicholas Negroponte worked with education and technology luminaries like Alan Kay and Seymour Papert. In the early 80s, Nicholas and Seymour ran a pilot program backed by the French government that placed Apple ][ machines in a suburban computing center in Dakar, Senegal. The project was a spectacular flop due to mismanagement and personality conflicts. In '83, barely a year after the experiment started, MIT's Technology Review magazine published its damning epitaph:

Naturally, it failed. Nothing is that independent, especially an organization backed by a socialist government and staffed by highly individualistic industry visionaries from around the world. Besides, altruism has a credibility problem in an industry that thrives on intense commercial competition.

By the end of the Center's first year, Papert had quit, so had American experts Nicholas Negroponte and Bob Lawler. It had become a battlefield, scarred by clashes of management style, personality, and political conviction. It never really recovered. The new French government has done the Center a favor in closing it down.

But both Nicholas and Seymour emerged from the ashes of the Dakar pilot with their faith in the premise of children learning naturally with computers intact. Armed with the lessons from the Senegal failure, it was perhaps only a matter of time before they tried again.

Indeed, Seymour tried again only a couple of years later: the Media Lab was founded in 1985 and immediately started supporting Project Headlight, an attempt to infuse constructionist learning into the complete curriculum of the Hennigan school, a public elementary school in Boston consisting mostly of minority students.

Fast forward almost two decades, to around 2000. Former Newsweek foreign correspondent turned philanthropist, Bernie "one-man United Nations" Krisher convinced Nicholas and his wife Elaine to join Bernie's program of building schools in Cambodia. Nicholas bought used Panasonic Toughbooks for one school, and his son Dimitri taught there for a time.

"Surely," the thinking went, "there has to be a way to scale this." And the rest of the story is familiar: Nicholas wooed Mary Lou Jepsen while she was interviewing for a faculty position at the Lab, and told her about his crazy idea for an organization called One Laptop per Child. She came on board as CTO. Towards the end of 2005, the organization left stealth mode with a bang: Nicholas announced it with Kofi Annan, Nobel Peace Prize winner and then-Secretary-General of the United Nations, at a global summit in Tunis.

The part that bears repeating is that Nicholas' constructionism-based computer learning project in Senegal was a complete disaster: modulo commentary on the personalities and egos involved, it demonstrated nothing about anything. And Krisher's Cambodia project, the one evidently successful enough to motivate Nicholas to actually start OLPC, used off-the-shelf laptops running Windows without any constructivist customizations of the OS whatsoever. (They did have some constructionist tools installed as regular applications.)

What we know
The truth is, when it comes to large-scale one-to-one computing programs, we're completely in the dark about what actually works, because hey, no one has done a large-scale one-to-one computing program before. Mako Hill writes:

We know that laptop recipients will benefit from being able to fix, improve, and translate the software on their laptops into their own languages and contexts. ... We can help foster a world where technology is under the control of its users, and where learning is under the terms of its students — a world where every laptop owner has freedom through control over the technology they use to communicate, collaborate, create, and learn. It is the reason that OLPC's embrace of constructionist philosophy is so deeply important to its mission and the reason that its mission needs to continue to be executed with free and open source software. It is why OLPC needs to be uncompromising about software freedom.

This kind of bright-eyed idealism is appealing, but alas, just not backed by fact. No, we don't know that laptop recipients will benefit from fixing software on their laptops. Indeed, I bet they'd largely prefer the damn software works and doesn't need fixing. While we think and even hope that constructionist principles, as embodied in the free software culture, are helpful to education, presenting the hopes as rooted in fact is simply deceitful.

As far as I know, there is no real study anywhere that demonstrates constructionism works at scale. There is no documented moderate-scale constructionist learning pilot that has been convincingly successful; when Nicholas points to "decades of work by Seymour Papert, Alan Kay, and Jean Piaget", he's talking about theory. He likes to mention Dakar, but doesn't like to mention how that pilot ended — or that no real facts about the validity of the approach came out of it. And there sure as hell doesn't exist a peer-reviewed study (or any other kind, to my knowledge) showing free software does any better than proprietary software when it comes to aiding learning, or that children prefer the openness, or that they care about software freedom one bit.

Keeping that in mind, Richard Stallman's missive on the subject just riled me up:

Proprietary software keeps users divided and helpless. Its functioning is secret, so it is incompatible with the spirit of learning. Teaching children to use a proprietary (non-free) system such as Windows does not make the world a better place, because it puts them under the power of the system's developer — perhaps permanently. You might as well introduce the children to an addictive drug.

Oh, for fuck's sake. You really just employed a simile comparing a proprietary OS to addictive drugs? You know, ones causing actual bodily harm and possibly death? Really, Stallman? Really?

If proprietary software is half as good as free software at aiding children's learning, you're damn right it makes the world a better place to get the software out to children. Hell, if it doesn't actively inhibit learning, it makes the world a better place. The problem is that Stallman doesn't appear to actually give an acrobatic shit about learning, and sees OLPC as a vehicle for furthering his political agenda. It's shameful, the lot of it.

While we're on the subject
One of the favorite arguments of the free software and open source community for the obvious superiority of such software over proprietary alternatives is the user's supposed ability to take control and modify inadequate software to suit their wishes. Expectedly, the argument has been often repeated in relation to OLPC.

I can't possibly be the only one seeing that the emperor has no clothes.

I started using Linux in '95, before most of today's Internet-using general public knew there existed an OS outside of Windows. It took a week to configure X to work with my graphics card, and I learned serious programming because I later needed to add support for a SCSI hard drive that wasn't recognized properly. (Not knowing that C and kernel hacking are supposed to be "hard", I kept at it for three months until I learned enough to write a patch that works.) I've been primarily a UNIX user since then, alternating between Debian, FreeBSD and later Ubuntu, and recently co-writing a best-selling Linux book.

About eight months ago, when I caught myself fighting yet another battle with suspend/resume on my Linux-running laptop, I got so furious that I went to the nearest Apple store and bought a MacBook. After 12 years of almost exclusive use of free software, I switched to Mac OS X. And you know, shitty power management and many other hassles aren't Linux's fault. The fault lies with needlessly secretive vendors not releasing documentation that would make it possible for Linux to play well with their hardware. But until the day comes when hardware vendors and free software developers find themselves holding hands and spontaneously bursting into one giant orgiastic Kumbaya, that's the world we live in. So in the meantime, I switched to OS X and find it to be an overwhelmingly more enjoyable computing experience. I still have my free software UNIX shell, my free software programming language, my free software ports system, my free software editor, and I run a bunch of free software Linux virtual machines. The vast, near-total majority of computer users aren't programmers. Of the programmers, a vast, near-total majority don't dare in the Land o' Kernel tread. As one of the people who actually can hack my kernel to suit, I find that I don't miss the ability in the least. There, I said it. Hang me for treason.

My theory is that technical people, especially when younger, get a particular thrill out of dicking around with their software. Much like case modders, these folks see it as a badge of honor that they spent countless hours compiling and configuring their software to oblivion. Hey, I was there too. And the older I get, the more I want things to work out of the box. Ubuntu is getting better at delivering that experience for novice users. Serious power users seem to find that OS X is unrivaled at it.

I used to think that there was something wrong with me for thinking this. Then I started looking at the mail headers on mailing lists where I hang out, curious about what other folks I respect were using. It looks like most of the luminaries in the security community, one of the most hardcore technical communities on the planet, use OS X.

And lest you think this is some kind of Apple-paid rant, I'll mention Mitch Bradley. Have you read the story of Mel, the "real" programmer? Mitch is that guy, in 2008. Firmware superhacker, author of the IEEE Open Firmware standard, wrote the firmware that Sun shipped on its machines for a good couple of decades, and in general one of the few people I've ever had the pleasure of working with whose technical competence so inordinately exceeds mine that I feel I wouldn't even know how to start catching up. Mitch's primary laptop runs Windows.

Sleight of hand
But really, I digress. The point is that OLPC was supposed to be about learning, not free software. And the most upsetting part of the Windows announcement is not that it exposed the actual agendas of a number of project participants which had nothing to do with learning, but that Nicholas' misdirection and sleight of hand were allowed to stand.

The whole "we're investing into Sugar, it'll just run on Windows" gambit is sheer nonsense. Nicholas knows quite well that Sugar won't magically become better simply by virtue of running on Windows rather than Linux. In reality, Nicholas wants to ship plain XP desktops. He's told me so. That he might possibly fund a Sugar effort to the side and pay lip service to the notion of its "availability" as an option to purchasing countries is at best a tepid effort to avert a PR disaster.

In fact, I quit when Nicholas told me — and not just me — that learning was never part of the mission. The mission was, in his mind, always getting as many laptops as possible out there; to say anything about learning would be presumptuous, and so he doesn't want OLPC to have a software team, a hardware team, or a deployment team going forward.

Yeah, I'm not sure what that leaves either.

There are three key problems in one-to-one computer programs: choosing a suitable device, getting it to children, and using it to create sustainable learning and teaching experiences. They're listed in order of exponentially increasing difficulty.

The industry didn't want to tackle the first one because there was little profit in it. OLPC successfully made them do it in the most effective way possible: by threatening to steal their lunch. But industry laptop manufacturers still don't want to tackle deployment, because it's really, really fucking hard, isn't within a 100-mile radius of their core competency, and generally has a commercial ROI that makes baby Cthulhu cry.

Peru's first deployment module consisted of 40 thousand laptops, to be deployed in about 570 schools across jungles, mountains, plains, and with total variance in electrical availability and uniformly no existing network infrastructure. A number of the target schools are in places requiring multiple modes of transportation to reach, and that are so remote that they're not even serviced by the postal service. Laptop delivery was going to be performed by untrusted vendors who are in a position to steal the machines en masse. There is no easy way to collect manifests of what actually got delivered, where, and to whom. It's not clear how to establish a procedure for dealing with malfunctioning units, or those dead on arrival. Compared to dealing with this, the technical work I do is vacation.

Other than the incredible Carla Gomez-Monroy who worked on setting up the pilots, there was no one hired to work on deployment while I was at OLPC, with Uruguay's and Peru's combined 360,000 laptop rollout in progress. I was parachuted in as the sole OLPC person to deal with Uruguay, and sent to Peru at the last minute. And I'm really good at thinking on my feet, but what the shit do I know about deployment? Right around that time, Walter was demoted and theoretically made the "director of deployment," a position where he directed his expansive team of — himself. Then he left, and get this: now the company has half a million laptops in the wild, with no one even pretending to be officially in charge of deployment. "I quit," Walter told me on the phone after leaving, "because I can't continue to work on a lie."

It's not like OLPC was caught unawares, or somehow forgot that this was going to be an issue. I wrote in an internal memo in December:

We have multiple concurrent rollouts of differing scale in progress — Uruguay with eight thousand machines, G1G1 with potentially a quarter million — and with at least Peru and Mongolia on the horizon within a month from now. We have no real support infrastructure for these rollouts, our development process is not allocating any time for dealing with critical deployment issues that (will inevitably) come up, and we have no process for managing the crises that will ensue. I wish I could say this is the bulk of our problems, but I mention these first simply because I predict it's these deployments that will impose the heaviest burden on this organization in the coming months — a burden we're presently entirely unprepared to handle.

...

We still have not a single employee focusing on deployment, helping to plan it, working with our target countries to learn what works and what doesn't. Evidently our "deployment plan" is to send whichever hotshot superhacker we have available to each country such that he may fix any problems that arise on the spot. If that is not in fact our plan, then we have no plan at all.

That OLPC was never serious about solving deployment, and that it seems to no longer be interested in even trying, is criminal. Left uncorrected, it will turn the project into a historical information technology fuckup unparalleled in scale.

As for the last key problem, transforming laptops into learning is a non-trivial leap of logic, and one that remains inadequately explained. No, we don't know that it'll work, especially not without teachers. And that's okay — the way to find out whether it works might well be by trying. Sometimes you have to run before you can walk, yeah? But most of us who joined OLPC believed that the educational ideology behind the project is what actually set it apart from similar endeavors in the past. Learning which is open, collaborative, shared, and exploratory — we thought that's what could make OLPC work. Because people have tried plain laptop learning projects in the past, and as the New York Times noted on its front page not so long ago, they crashed and burned.

Nicholas' new OLPC is dropping those pesky education goals from the mission and turning itself into a 50-person nonprofit laptop manufacturer, competing with Lenovo, Dell, Apple, Asus, HP and Intel on their home turf, and by using the one strategy we know doesn't work. But hey, I guess they'll sell more laptops that way.

Broken windows theory
I've tried to establish already that there's no evidence that free software provides a superior learning experience when compared to a proprietary operating system. This point bears some elaboration. Bernie Innocenti, until recently the CTO for the fledgling OLPC Europe, a few days ago wrote:

I myself wouldn't oppose a Windows port of Sugar. I would never waste my time on it, or encourage anyone to waste their time on it, but it's free software and thus anyone is free to port it to anything they wish.

Stallman similarly called a Windows port of Sugar "not a good thing to do". Here's the thing: such a port is only a waste of time if free software is not the means here, but an end. At Nicholas' solicitation, I wrote an internal memo on software strategy in early March. It was co-signed by Marco Pesenti Gritti, the inimitable Sugar team lead. I am not at liberty to reproduce the entire document, but I will quote the most relevant section with minimal redactions:

... We [argue strongly that we should] decouple the Sugar UI from the Sugar technologies we’ve developed such as sharing, collaboration, the presence service, the data store, and so forth. We may then make those services run well in a regular Linux desktop environment and redefine the Sugar activity concept to simply be any Linux desktop application capable of using the Sugar services. The Sugar UI itself could, optionally and at a later date, be provided as a graphical launcher, perhaps developed by the community.

The core mistake of the present Sugar approach is that it couples phenomenally powerful ideas about learning — that it should be shared, collaborative, peer to peer, and open — with the notion that these ideas must come presented in an entirely new graphical paradigm. We reject this coupling as untenable.

Choosing to reinvent the desktop UI paradigm means we are spending our extremely overconstrained resources fighting graphical interfaces, not developing better tools for learning. … It is most important to recognize that the graphical paradigm changes are inessential both to our core mission and to the Sugar core ideas.

We gain a plethora of benefits from detaching the technologies that directly support the mode of learning we care about from the Sugar UI. Notably, it becomes far easier to spread these ideas and technologies across platforms — our UI components are the hardest parts to port. If the underlying Sugar technologies were made easily available for all major OSes, we could leverage the creativity and work of the wider development community to build applications on top of our core offerings, creating a diverse ecosystem of powerful learning tools. Those tools could then be used by learners globally and on any computer, XO or otherwise. This should have been our aim all along. Many of the technologies we’ve built would be welcomed with arms wide open into modern Linux desktops, and a large number of developers would likely get engaged with them if we provided the possibility. In contrast to the current situation, such a model must be the direction where we take things: OLPC benevolently steering development which is mostly done by the community.

Finally, with regard to the politically-sensitive question of OLPC’s commitment to open source, we think there is a simple and uncomplicated answer: OLPC’s policy should be that all OLPC-developed software is open source and uses open standards and open formats. We don’t think a stronger commitment is necessary. Our preference for open source should stem solely from the conviction that it provides a better learning environment than closed-source alternatives. As such, having an open source cross-platform set of core technologies for building collaborative learning applications makes a tremendous amount of sense. But fundamentally, requiring that a particular UI or even OS are used seems entirely superfluous; we should be satisfied with any environment where our core technologies can be used as building blocks for delivering the learning experience we care so strongly about.

At the end of the day, it just doesn’t matter to the educational mission what kernel is running Sugar. If Sugar itself remains open and free — which, thus far, has never been in question — all of the relevant functionality such as the ‘view source’ key remains operational, on Windows or not. OLPC should never take steps to willingly limit the audience for its learning software. Windows is the most widely used operating system in existence. A Windows-compatible Sugar would bring its rich learning vision to potentially tens or hundreds of millions of children all over the world whose parents already own a Windows computer, be it laptop or desktop. To suggest this is a bad course of action because it’s philosophically impure is downright evil.

And hey, maybe a Windows version of Sugar gets kids sufficiently interested in computer innards to actually want to switch to Linux. Trolltech, the company behind the Qt graphical toolkit, was recently purchased by Nokia and announced it’ll be adding platform support for the mobile version of Windows, apparently to accusations of treason in the free software community. But Trolltech’s CTO Benoit Schillings doesn’t think that’s right:

Some critics are concerned that Trolltech’s support for Windows Mobile could limit the growth of mobile and embedded Linux technologies, but Schillings sees things differently. By enabling application developers to create a single code base that can seamlessly move across platforms, he says that Trolltech is making it easier for companies that are currently using Windows Mobile to transition to Linux, which he thinks will mean more adoption of the open source operating system in the long run.

The man speaks wisely.

Now, pay close attention: while I’m unequivocally enthusiastic about Sugar being ported to every OS out there, I’m absolutely opposed to Windows as the single OS that OLPC offers for the XO. The two matters are completely orthogonal, and Nicholas’ attempt to conflate them by calling the open source community “fundamentalists” (and watching the community foam at the mouth instead of picking apart his logic) is just another bit of misdirection. Not that anyone should really feel offended, since he’s made it a habit to call his employees terrorists.

OLPC should be philosophically pure about its own machines. Being a non-profit that leverages goodwill from a tremendous number of community volunteers for its success and whose core mission is one of social betterment, it has a great deal of social responsibility. It should not become a vehicle for creating economic incentives for a particular vendor. It should not believe the nonsense about Windows being a requirement for business after the children grow up. Windows is a requirement because enough people grew up with it, not the other way around. If OLPC made a billion people grow up with Linux, Linux would be just dandy for business. And OLPC shouldn’t make its sole OS one that cripples the very hardware that supposedly set the project’s laptops apart: released versions of Windows can neither make good use of the XO power management, nor its full mesh or advanced display capabilities.

Most importantly, the OS that OLPC ships should be one that embodies the culture of learning that OLPC adheres to. The culture of open inquiry, diverse cooperative work, of freely doing and debugging — this is important. OLPC has a responsibility to spread the culture of freedom and ideas that support its educational mission; that cannot be done by only offering a proprietary operating system for the laptops.

Put differently, OLPC can’t claim to be preoccupied with learning and not with training children to be office computer drones, while at the same time being coerced by hollow office drone rhetoric to deploy the computers with office drone software. Nicholas used to say the thought of the XOs being used to teach 6-year olds Word and Excel made him cringe. Apparently, no longer so. Which is it? The vacillation needs to stop. As they say in the motherland: shit or get off the pot.

How to go forward
Here’s a paragraph from one of my last e-mails to Nicholas, sent shortly after I resigned:

I continue to think it’s a crying shame you’re not taking advantage of how OLPC is positioned. Now that it’s goaded the industry into working on low-cost laptops, OLPC could become a focus point for advocating constructionism, making educational content available, providing learning software, and keeping track of worldwide [one-to-one] deployments and the lessons arising from them. When a country chooses to do [a one-to-one computer program], OLPC could be the one-stop shop that actually works with them to make it happen, regardless of which laptop manufacturer is chosen, banking on the deployment plans it’s cultivated from experience and the readily available base of software and content it keeps. In other words, OLPC could be the IBM Global Services of one-to-one laptop programs. This, I maintain, is the right way to go forward.

I’m trying to convince Walter not to start a Sugar Foundation, but an Open Learning Foundation. For those who still care about learning in this whole clusterfuck of conflicting agendas, the charge should be to start that organization, since OLPC doesn’t want to be it. Having a company that is device-agnostic and focuses entirely on the learning ecosystem, from deployment to content to Sugar, is not only what I think is sorely needed to really take the one-to-one computer efforts to the next level, but also an approach that has a good chance of making the organization doing the work self-sustaining at some point.

So here’s to open learning, to free software, to strength of personal conviction, and to having enough damn humility to remember that the goal is bringing learning to a billion children across the globe. The billion waiting for us to put our idiotic trifles aside, end our endless yapping, and get to it already.

Let’s get to it already.

My thanks to Walter Bender and Marco P. Gritti for reading drafts of this essay.

by Ivan Krstić at May 13, 2008 09:55 PM

Davy Mitchell

Mood News SilverLight

So much to blog about... so little time... one thing... Ever since I put together Mood News, I wanted (and people suggested) having a slider to 'mix' the level of the news. ... [72 words]

May 13, 2008 08:55 PM

Sylvain Hellegouarch

XMPP, AtomPub, headstock, amplee and the BBC

Matthew Wood, from the BBC Radio Labs, posted a few days ago an exciting note regarding fun he was having with XMPP and services such as last.fm to inform user, via XMPP messages, of BBC radios broadcasting music they might like based on thier last.fm profile. I thought this was fantastic but I was even more excited when I read another note where he explained that he was using PubSub as a mean to carry and distribute metadata about BBC shows using Atom entries as the metadata format.

This evening I spent three hours expanding on the simple chat example coming with headstock to talk with the PubSub service. Then I integrated amplee as a way to offer an AtomPub interface at the same time. This means that when the demo starts both a XMPP client, connecting to Matt's service, and an AtomPub server, using amplee and served by CherryPy, are started. The XMPP client asks the server about PubSub nodes. For each node representing BBC channels I create an atompub collection within its own workspace. Simultaneously I subscribe to those nodes. I then ask the XMPP server for items belonging to those nodes and for each item, representing metadata about a show for instance, I create an atom entry that I store within the AtomPub store.

This means one can then simply subscribe with a feed reader to a given collection and/or a XMPP PubSub node. All of this happening on the fly starting from an empty AtomPub service document.

Eventually I will add support so that when an Atom entry is POSTed to a AtomPub collection, the according PubSub stanza is pushed towards the service (I doubt Matt's service accepts it though) allowing for microblogging support.

The source code of the example can be found here. If you want to understand how it works you might want to read this quick word I wrote about Kamaelia first which is at the core of headstock.

by Sylvain Hellegouarch (nospam@example.com) at May 13, 2008 08:35 PM

Armin Ronacher

Command of the Day

ssh-keygen -t dsa -b 1024 -f /etc/ssh/ssh_host_dsa_key -N '' &&
  ssh-keygen -t rsa -b 1024 -f /etc/ssh/ssh_host_rsa_key -N '' &&
  /etc/init.d/ssh restart

I really don’t think distributors should try to patch cryptographic stuff, especially not to silence debuggers.

by Armin Ronacher at May 13, 2008 03:49 PM