unofficial planet python

IronPython Url's: Silverlight 2 and Dynamic Languages

Fuzzyman — 2008-11-20T00:03:37+00:00

Well my backlog is logged so far back that I haven't even blogged about the final release of Silverlight 2.

In case you've been living under a rock for the last year, Silverlight 2 is a browser plugin from Microsoft. It is similar to Flash (aimed at games, media streaming and rich internet applications) and is cross-platform (Mac OS X and Windows - the officially blessed Linux port Moonlight by the Mono guys is making good progress though) and cross browser (IE 7+, Safari & Firefox 2+). Unlike Flash it can be programmed with a choice of languages, and through the Dynamic Language Runtime it can be programmed in Python, Ruby and Javascript.

Silverlight 2 final is now out, and according to Scott Guthrie has now been installed on over 100 million consumer computers.

Naturally Jimmy Schementi, who maintains the Dynamic Language Support for Silverlight, released an updated version of the Silverlight Dynamic Languages SDK (sucky name - more on this in a bit):

Dynamic Languages in Silverlight RTW (Release To Web)

In preparation for my talk at the PyWorks conference I updated my IronPython Web IDE (tool for experimenting with the Silverlight APIs from Python in the browser) and Try Python (interactive Python interpreter in the browser) for Silverlight 2:

Try Python and IronPython Web IDE Updated for Silverlight 2

Of course if you're interested in building internet applications with Silverlight then you will want the extended controls that come as part of the Visual Studio Tools for Silverlight 2 (this works with Visual Studio Professional or the free Visual Web Developer Express). The assemblies that come with the tools include the data grid, extra controls like the date picker and various other useful APIs.

Even better is the Silverlight Toolkit. This is by Microsoft, but Open Source (living on Codeplex and with Unit Tests). As it is a separate project it can have a separate release cycle, including experimental components and being updated more frequently than Silverlight itself.

The toolkit includes charting components plus new controls covering styling, layout, and user input.

Since these releases Jimmy Schementi has been far from idle. His latest blog entries track what he has been up to:

Adding Scripting to a C# Silverlight Application

Though, for certain scenarios, running scripts in a VB/C# application would be useful. For example, a shopping application that has a bunch of business rules, like "when someone has three items in their cart that all have to do with cooking, give them 10% off." These type of rules can change all the time, and traditionally you'd either store the rules in a database and implement a engine to understand the rules, or hand-code them yourself and have to redeploy the system every time you want to change them. Or, you could save yourself the hassle and store the rules as Python or Ruby code, and then host the DLR in your application to run the code. Want to update the rules? Just update the code, nothing more.

"console=true": REPLs in Silverlight

Embedding an IronRuby REPL (interactive interpreter) in a Silverlight application.

agdlr: Silverlight + DLR + Open Source

This is the good one! Jimmy posted this email to the IronRuby and IronPython mailing lists:

First and foremost, I want to thank anyone who has used the bits on http://codeplex.com/sdlsdk, and accepting my bullshit version of open-source. While getting monthly binaries/sources is nice, it should be about working on the project together ... not just me throwing stuff over the wall to you. That's changing, now ...

Oh, and remember me complaining about the crappy "sdlsdk" name ... well, I'm trying to get rid of that acronym ...

http://github.com/jschementi/agdlr

Above the public repository for the DLR integration in Silverlight. The following post explains what's in there, what's not, what's git, and how to contribute: http://blog.jimmy.schementi.com/2008/11/agdlr-silverlight-dlr-open-source.html

My first order of collaboration is this simple new feature, "console=true": http://blog.jimmy.schementi.com/2008/11/repls-in-silverlight.html. If you like this, please feel free to look at what's been done, and if you want to fix something that doesn't yet work correctly, I won't stop you.

Also, as I mentioned in a previous mail, I want to make the filesystem->XAP/isolatedstorage metaphor stronger, so feel free to experiment with that as well. Over the next week I'll get some website-presence/wiki/etc, and we can run this project up and running. There are still some hurdles I need to clear with getting contributed code back into our internal codebase, and shipping on Codeplex, but there are no problems with keeping things on GitHub for now.

Let me know if there are any question. I know I've been a bit silent on the Silverlight front, but take this as me making it up to you.

The resulting discussion also revealed where the Silverlight development tool Chiron got its name from:

Yep, Ag is Silver … made pretty obvious by my little logo for it.

As far as Chiron. It's a planetoid between Saturn and Uranus. The port that Chiron.exe runs on by default, 2060, is Chiron's "object" number. It was derived from the Cassini ASP.NET Web server that Dmitry Robsman wrote. Cassini was a probe mission to explore the moons of Saturn, and Chiron was initially thought to be a moon of Saturn.

Plus this from Michael Letterle:

More importantly, it's also the name of one of Jonathan Coulton's songs: "Chiron Beta Prime".

Because of this I knew how to pronounce the name :)

Peter Bengtsson: domstripper - A lxml.html test project

2008-11-19T22:00:00+00:00

I'm just playing with the impressive lxml.html package. It makes it possible to easily work with HTML trees and manipulate them.

I had this crazy idea of a "DOM stripper" that removes all but specified elements from an HTML file. For example you want to keep the contents of the <head> tag intact but you just want to keep the <div id="content">...</div> tag thus omitting <div id="banner">...</div> and <div id="nav">...</div>. domstripper now does that. This can be used for example as a naive proxy that tranforms a bloated HTML page into a more stripped down smaller version suitable for say mobile web browsers. It's more a proof of concept that anything else.

To test you just need a virtual python environment and the right system libs to needed to install lxml. This worked for me:

$ sudo apt-get install cython libxslt1-dev
$ cd /tmp
$ virtualenv --no-site-packages testenv
$ cd testenv
$ source bin/activate
$ easy_install domstripper

Now you can use it like this:

>>> from domstripper import domstripper
>>> help(domstripper)
...
>>> domstripper('bloat.html', ['#content', 'h1.header'])
<!DOCTYPE...
...

Best to just play with it and see if makes sense. I'm not saying this is an amazing package but it goes to show what can be done with lxml.html and the extremely user friendly CSS selectors.

Paulo Nuin: Creating an interface for the motif finding script, final

Paulo Nuin — 2008-11-19T21:57:24+00:00

We can say that this would be our final version of the script. There are many nice wxPython programming resources, and one is a very good book called wxPython in Action, which is co-written by Robin Dunn, the wxPython maintainer. Go check it out.

So for the last entry in this series, we just need to add a couple of changes to our interface and motif finding scripts. Basically on the interface script we need to add a line that gets the value entered (or the default one, if not changed) in the motif width input box. And we can do that by including the line below in the run_finder function.

width = self.motif_width.GetValue()

This line tells the script to get the value of the box and assign to the variable width. This method will get whatever is inside the input box and save as a string to the variable assigned. Now, we need to create the structure to actually send this value to the motif finder functions. Last version of our function calculate_motifs received two parameters, we need to add an extra one, and also change the lines that call the function that get the quorums. Basically the first lines of the function will be

def calculate_motifs(input_seqs, input_seqs2, width):

    print input_seqs, input_seqs2
    input_seqs = fasta.read_seqs(open(input_seqs).readlines())
    input_seqs2 = fasta.read_seqs(open(input_seqs2).readlines())

    foreground = get_quorums(input_seqs, width)
    background = get_quorums(input_seqs2, width)

And that’s it. Our simple interface is ready to primetime. OK, not prime primetime, we didn’t add a series of features that will make it useful by everyone. For instance, there is no error control, so someone could enter ‘ABC’ in the width input box and that value would be sent and an error will occur. Also you can click the run button without any file selected. And we could go on and on. But this is just a primer, and we can build from it.

The code is on Github, so get it there and have fun. Next time we will see … no plans yet. We’ll see …

Technorati Tags: wxPython, motifs, Python, bioinformatics

Ted Leung on the Air: Python in NetBeans

Ted Leung — 2008-11-19T16:56:00+00:00

Along with today’s launch of NetBeans 6.5, Sun, in cooperation with the NBPython community, are releasing an early access version of Python support for NetBeans. This is a result of the collaboration between Sun people and the NBPython project that I wrote about back in July. This release has been tested by folks in the NetBeans community and some folks from Sun’s NetBeans QA team, and it’s in pretty good shape for an early access release. We’re interested in getting people’s feedback. We would also love to see more people get involved with NBPython.

How to get it?

You can get NetBeans Python from the NetBeans download page.

What’s in it?

The basic feature set for the early access release consists of an editor for Python, the ability to execute Python programs (using CPython or Jython), and a debugger.

There’s a tutorial up on the NetBeans wiki.

Tor Norbye, who did most of the work on the editor, has written a series of blog posts detailing various features of the Python editor.

Code folding, semantic highlighting, and instant rename
Code completion and quick fixes
Import management
Finding unresolved symbols - Tor used this to discover a long standing bug in Jython
Support for __all__
Code completion in detail

Who did it

Allan Davis - project and platform management, interactive console.

Jean-Yves Mengant - Jean-Yves is the author of the jpydbg debugger, which he’s merged into NBPython.

Amit Saha - documentation and help sets - Amit works for Sun, but he’s doing Python on his own time.

Tor Norbye (Sun) - editing.

Tomas Zezula (Sun) - project and platform management.

Ted Leung (me) (Sun) - various behind the scenes stuff.

Frank Wierzbicki (Sun) - NBPython is using Jython’s parser and Frank worked with Tor to add support for positions and better error reporting.

Peter Lam (Sun) - Sun QA

Tony Beckham (Sun) - Sun QA

The NetBeans CAT community as well as those folks who drove by and reported bugs.

How to get involved

NBPython has become a full fledged NetBeans project, so the main project page is now on NetBeans.org, as are the issue tracker and mailing lists:

nbpython-dev@netbeans.org
nbpython-issues@netbeans.org
nbpython-commits@netbeans.org
nbpython@netbeans.org

Mike Fletcher: Now we need 3 million dollars...

Mike Fletcher — 2008-11-19T16:14:08+00:00

I chose 1 million dollars as a small number that should be possible to raise if we have a good enough proposal. The top three voted projects focus on, one way or another, performance and applicability. The winning project essentially was to improve Python's concurrency storing markedly. Basically spend the money on hiring people to really pound on multi-processing and maybe even fine-grained locking (concurrent threading) in CPython to produce far better coarse-grained concurrency support. We'd have to see who would be interested in funding this type of research... we saw concurrency as a problem with the current CPython implementation (though there was the hint that it's a problem of perception, rather than necessarily a practical problem).

The pan-mobile SDK project was more of an immediate-term goal, but it requires significant compiler technology and a lot of hard work in order to produce a commercial-quality resulting platform/SDK. There was some question as to whether $1 million would be enough to produce the requisite quality, but as it's a commercial project, the $1 million could be seed money and you might be able to draw handset manufacturers in to further funding. The project seems very doable, it's just a question of money, time, vision and effort.

The idea of fine-grained concurrency (e.g. spreading for loops across processors automatically if you can prove there's no dependencies in the loop code) was seen more as a PyPy project, as was the addition of explicit asynchronous operation syntax. We spent a while discussing what would be needed for each project, and the idea that PyPy would make the mobile-targeting SDK easier, and that concurrency/performance issues would be best worked out in PyPy came up a number of times. Effectively, while we had proposals that said "Python should be", we didn't get a lot of support for the idea of using CPython as the platform for figuring out how to do those things beyond minor changes to the current approaches.

PyPy development was, by our rough estimates, 10 person-years from being a robust platform for widespread real-world development, that's cutting it close for a mere 1 million dollars, but maybe if we partnered with other people who would have a stake in the result. Performance improvements, concurrency primitives and back-ends targeting mobile-type platforms were all identified as areas that would make sense to focus energy. Realistically, though, the core PyPy tool chain probably needs to be the first focus and then we can look at the extras.

Malthe Borch: Speedup

malthe — 2008-11-19T14:03:00+00:00

You can now have your cake and eat it, too.

Disclaimer: Don't try this at home! Or actually, do try it, but only on Plone trunk. Simply pull in the five.pt egg and load its configuration. It's a whole-sale drop-in replacement of Zope Page Templates.

If you're interesting in sponsoring this on-going effort, there's an excellent sponsorship opportunity for the upcoming performance-sprint in Bristol. Please contact me by e-mail at mborch@gmail.com.

Ned Batchelder: Pathological backtracking

2008-11-19T13:14:18+00:00

At work we've been using the well-regarded feedparser module to parse RSS feeds, and it works great for the most part, but we'd occasionally get a stuck server process. The CPU would spike to 100%, and wouldn't make any progress.

We discovered a particular feed would cause a particular regular expression in the code to spin endlessly. The regex was intended to determine if a style attribute is valid CSS:

if not re.match("^(\s*[-\w]+\s*:\s*[^:;]*(;|$))*$", style): return ''

Breaking this out into verbose regex syntax shows how it matches valid CSS:

"""(?x) # use verbose regex syntax ^( # A single CSS clause is: \s* # leading whitespace [-\w]+ # a dash-word, the property name \s*:\s* # space, colon, space [^:;]* # anything but :;, the value (;|$) # ends with a semi or the end of the string )* # Valid CSS is any number of clauses $ """

And here's the snippet discovered in the feed that spun us hard (with whitespace added for readability):

<var style="COLOR: #fffafe; coming: ; basket: ; philologist: ; gradually: ; encyclic: ; whitechapel: ; left: ; albino: ; lamelliform: ; foment: ; adjuvant: ; Room: ; Milk: ; buynow: ; wheelwork: ; unseal: ; reasons: ; socalled: ; dazed: ; Brain: ; Kaleidoscope: ; hardheaded: ; asthenic: ; preferred: ; Barbecue: ; Comet: ; Nail: ; lubberly: ; School: ; Mist: ; undercurrent: ; intwine: ; isotonic: ; Chief: ; miscellaneous: ; Book: ; Shoes: ; Chocolates: ; deuced: ; you: ; Man: ; federalize: ; Rainbow: ; Satellite: ; Printer: ; amicus: ; tautophony: ; taking: ; regrater: ; waggon: ; prescient: ; God: ; prosing: ; Bank: ; hariolation: ; patriarchs: ; Pyramid: ; Data Base: ; PaintBrush: ; ingenu: ; Rope: ; parenchyma: ; price: ; Alphabet: ; Circle: ; seeks: ; frankhearted: ; vituperate: ; dysmeromorph: ; Shop: ; firm: ; imperforation: ; lane: ; Gemstone: ; slatternly: ; Fire: ; impudence: ; Carrot: ; Fan: ; inoccupation: ; uncover: ; Liquid: ; drawee: ; Pocket: ;barbacan: ; fornicatress: ; chimes: ; Crystal: ;innovation: ; years: ; untiring: ; Freeway: ;desertful: ; unreined: ; Compass: ; Hose: ;prelusive: ; impenetrability: ; Fruit: ; direct: ; "></var>

(yes, it's garbage, and yes, spam sucks.)

It's hard to see the problem here, but this is not valid CSS because they used "Data Base" as a property name about half-way through and spaces aren't allowed in property names.

The CPU spins because when the regex encounters the failure to match "Data Base", it backtracks to reconsider previous matches in the hopes that it can still make the regex work. In fact, it isn't in an infinite loop, just a very very very long one. Eventually this regex will finish and decide that the string doesn't match.

But we don't need it to backtrack: going back to re-match previous CSS clauses isn't going to help.

Some regex libraries offer solutions to this problem. Possessive quantifiers let you use *+ to mean, match as many as possible, and once matched, don't try matching fewer during backtracking. They're called possessive because once the operator claims part of the string, it won't give it back for other operators to match later.

But Python doesn't offer possessive quantifiers (yet yet). So we have to choose a different technique than trying to match the whole string in one large regex. In this case, since we don't need the match data, we're just checking that the whole string matches, so we can use re.sub to remove matching clauses and then check that there's nothing left over:

if re.sub("\s*[-\w]+\s*:\s*[^:;]*;?\s*", '', style): return ''

Because re.sub grabs matches, performs the replacement, and moves on, there's no needless backtracking to throw a wrench in the works. Now our crazy CSS spam is speedily dispatched as invalid.

As an interesting side effect, if the string is not empty, what remains is the invalid part of the string.

Christopher Lenz: The Truth About Unicode In Python

2008-11-19T09:05:44+00:00

The unicode support in Python is generally considered to be pretty good. And in comparison to many other languages, it's good indeed.

But compared to what is provided by the International Components for Unicode (ICU) project, there's also a lot missing, including collation, special case conversions, regular expressions, text segmentation, and bidirectional text handling. Not to mention extensive support for locale-specific formatting of dates and numbers and time calculations with different calendars.

Basically what Python does provide out of the box is “only” encoding/decoding, normalization, and some other bits such as simple case conversion and splitting on whitespace. It's the absolute minimum you need to do anything useful with unicode, but often not enough to build truly internationalized applications. (Fortunately, most applications get away without true internationalization.)

In this post, I'm going to talk about a couple of the problems with unicode in Python. Please note that this is not intended as a criticism of Python's unicode support or the people who designed and implemented it. Most of those people probably know a whole lot more about unicode than I do, and the limitations discussed here are the result of a pragmatic approach to implementing unicode support, rather than due to a lack of knowledge.

read on …

Simon Willison's Weblog: Django 1.0.2 released

2008-11-19T08:46:27+00:00

Django 1.0.2 released. An update to last week’s 1.0.1 release, which I failed to link to. 1.0.2 mainly fixes some packaging issues, while 1.0.1 contains “over two hundred fixes to the original Django 1.0 codebase”. The team are holding up the promise to move to a regular release cycle after 1.0.

Shannon -jj Behrens: Misspelled Variables

Shannon -jj Behrens — 2008-11-19T05:59:51+00:00

Care to guess what happens when you execute the following PHP?

define('FOO', 'Hi');
print(FO);

It prints 'FO'.

I do believe PHP got this from Perl:

perl
print FOO . "FOO";  # Prints FOOFOO

It works even if you're strict:

perl -w
use strict;
print FOO . "FOO";  # Prints FOOFOO

Ruby behaves differently depending on whether you try to print an undefined variable/method or an undefined attribute:

irb
>> print a
NameError: undefined local variable or method `a' for main:Object
 from (irb):1
>> print @a
nil=> nil

Python raises an exception:

python
>>> print a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined

In a compiled language, these sorts of errors would be caught at compile time. However, a compiled language would never let me do something like:

python
>>> var_name = 'a'
>>> locals()[var_name] = 'yep'
>>> print a
yep

This example is a bit contrived, but I've definitely done things like it.

Personally, I like the flexibility of a scripting language. I like it even more when there's a tool like pychecker that can catch these sorts of errors. However, just because a scripting language doesn't have a compilation step that can catch stupid spelling mistakes doesn't mean it should accept them at runtime. I'd much rather deal with an exception than spend half an hour fighting a bug caused by a spelling error!

As a general rule, I think software should fail fast rather than glossing over bugs that will surely cause trouble later. I can handle the exception if I need to, but what I can't handle is a silent bug.

Django Weblog: Django 1.0.2 released

2008-11-19T05:52:39+00:00

Shortly after last week's Django 1.0.1 release, several people noted that the packaging script used to produce the release omitted several directories from the Django source tree; mostly this affected some unit tests, but at least one of the omitted directories affected the use of Django itself (specifically, of django.contrib.gis). So tonight we're issuing Django 1.0.2, which is built around an updated packaging script and should resolve these problems.

This is a recommended upgrade for anyone using or targeting Django 1.0 or Django 1.0.1; to obtain a copy, swing by the downloads page, and don't forget to read the release notes. For the security conscious, a signed file containing the package's checksums is, as always, available.

Mike Fletcher: Million Dollar Ideas Wrap-up from PyGTA

Mike Fletcher — 2008-11-19T05:44:02+00:00

Great fun at PyGTA this evening. We had an overly complex voting
scheme, basically you could vote from 0 to 5 (5 being the highest) for
each project, total count wins. We had quite a few proposals, and
laughed rather a lot during the process.

My take-aways:

Concurrency matters in multi-core systems, Python needs work here
Performance/scaling matters, we want to take Python to smaller hardware and mobile devices particularly
PyPy is seen as the vehicle for radical improvements to the language, not CPython
We want to make public services better
We want to help educate people
We'd like to stop spam
Disinterest in Python 3000 is strong, but not universal, and seems to be weakening

We also discussed the sorry state of the PyGTA web-site. We really need to get it updated. I'm thinking a simple CMS-type site, with:

a calendar
a trace of the announcements (a blog...)
the ability for a few people to post notices/topics (i.e. I shouldn't be the single point of failure)
maybe an RSS feed
pointers to
the mailing-lists

The Wiki features just don't get used enough to bother with them it seems. I've got the pygta.org domain registered with Vex, so we can redirect it wherever we want once we know where we want to direct it.
[Update] Since the PyGTA-general list is subscriber only, here's the summary reposted:

Given:
   o 1 million (CDN) dollars
   o around 5 years (maximum) for core benefits to be realized
   o must enhance Python in some way or use Python to provide some
     benefit to the world
         o you need to explain how your project will benefit the world
           and/or investors
         o you are trying to convince "us" to invest, so make it good
   o can be commercial, philanthropic, Open Source or what have you

Ideas:

   o Concurrency  45
         o Threads (coarse grained concurrency)
         o Multiprocessing module
         o Twisted constructs into the core (syntactic extensions)
         o HTTP Client Library (beyond all current C or Python versions)
         o CPython GIL Elimination

   o Mobile Applications 35
         o Frameworks for developing mobile applications for cell
           phones, smart devices etceteras
         o Unified application development framework
         o Optimization
         o Experts for devices
         o IDE Development/GUI tools
         o Money-maker

   o PyPy finishing 34
         o Performance
         o Concurrency
         o Targeting low-level hardware (mobiles)

   o Public Transportation System 33
         o Cooler signaling system for the subways
         o Timing for bus-stops and bus-stations
         o Electronic monitoring system for the public transport system
         o Central server in Python
         o Mathematicians
         o Money maker, increases ridership, no monthly-pass increases
         o On-demand signaling and routing?

   o Training for non-programmers in Python (or Programming)  29
         o Extending the utility of computers in non-technical environments
         o Mechanism for replacing spreadsheets and the like
         o Reduces effort for getting jobs done
         o Seminars, presentations, TV programs

   o Kids need to be empowered with computers 28
         o Programming education for the young (8 years)
         o Gulag method versus fun and games education
         o Game writing, game sales
         o Robots, turtles, lego mindstorms
         o Good of the world
         o No top-end to the language involved
         o It's an Empowerment Project, Not a Programming Project

   o Write a game to teach kids history/geography  25
         o Grade school or high-school level
         o Fun (constructivist) games
         o Where in the World is Carmen Sandiego
         o Commercial quality games
         o Educational input from government or the like
         o Some programmers required, user testing
         o Postgresql, Python, Web Application
         o Test driven environment, agile development methods
         o Google Software:  E-Learning

   o Seneca's Tuition 25
         o ???

   o UI Design, user feedback 22
         o Video camera showing the screen
         o Tracking and correlation of details and video of user
               + Eye tracking mechanism
         o Analytics regarding task-oriented completion (statistical
           analysis)
         o Frustration measurement instrumentation
               + Heartbeat, pulse, blood pressure monitors
         o Adjustable screen arrangements
               + UI design in-test
         o Meta queries for optimizations
               + Feedback in the middle the tests
         o Maybe money maker

   o CPython JIT 20

   o Eliminate Python 3000 19

   o Genome Study 19
         o Non-academic access to genomic research
         o Organization/body to grant access to non-professional or
           non-academic researchers
         o Open Access
         o Python as the server for the data
         o Python on a cluster to perform ad-hoc processing of genomic data
         o Sci-Py/Numpy and distributed cluster
         o Money maker

   o Voting Software 18
         o Web-base ad-hoc voting systems

and the after-the-vote proposal:

Create an AI-like system that can respond to all incoming spams in a way which requires the spammers to respond with a non-trivial response, something that convinces the spammers that they have a mark. Installing on millions upon millions of machines, the spammers are deluged with possible marks and must spend real time and effort to sort out the back-spams from the real marks. We effectively hide the marks in a forest of fake marks to target the spammer's ability to make money.

James Tauber: Discrete Cosine Transforms Part 1

James Tauber — 2008-11-19T00:30:11+00:00

I've often been intrigued by the lossy part of JPEG compression so I thought I'd explore Discrete Cosine Transforms and their use in JPEG as a short multi-part blog series.

In this part, lets just talk about what a "discrete cosine" function is and then in the next couple of posts look at how the concept can be combined with basic linear algebra to break up images into components in such a way that you can throw out some components with minimal effect on the perceived image. It's quite clever but the mathematics is fairly straightforward.

Let's start with a single cycle of a cosine function shifted up and scaled so its values range from 0-255 instead of from -1 to +1. To make it discrete, we'll divide it up into four and simply take the value of the scaled cosine function at the midpoint of each of our four sections:

Here our four discrete values are 217, 39, 39, 217 .

Now let's do the same for one and a half cycles:

which yields values of 176, 11, 245 and 80.

Now half a cycle:

which yields 245, 176, 80 and 11 and zero cycles:

which gives us 255, 255, 255, 255

We've basically calculated values for 0 thru 3 half cycles. If we wanted to split the cosine into N pieces instead of 4, we'd calculate values for 0 thru N-1 half cycles.

But, in summary, for N = 4:

k = 3: 176, 11, 245, 80
k = 2: 217, 39, 39, 217
k = 1: 245, 176, 80, 11
k = 0: 255, 255, 255, 255

Rene Dudfield: pygame reloaded

illume — 2008-11-19T00:19:28+00:00

So Marcus has been working away in the background refactoring/rewriting pygame. Hopefully his work will become part of pygame 2.0... but in the mean time we hope to do another set of 1.x pygame releases.

The main pieces I see for another pygame release are the evolution of the testing frame work into a library, examples as a library, the camera module, and many other changes are coming to pygame... including pygame coming to another implementation of python... tinypy!

Below are Marcus's notes about the pygame reloaded release.

Features
--------
* New FRect class for high precision rectangle handling using floating point
values.
FRect and Rect share the same properties and functions and can be converted
into the other rectangle type without any problems.
* New PixelFormat class that wraps up SDL_PixelFormat.
* New CDTrack class that keeps information about CD tracks.
* New physics module integrated. Zhang Fan's GSoC work on an easy to use
physics simulation is now available.
* New sdlgfx module, which wraps up the SDL_gfx library and adds features
such as fast zoom and drawing primitives or FPS management.
* Improved C API.
* Improved SDL sound and channel handling. Though not perfect, they are more
robust now.
* Completely rewritten build system, allowing a fine grained build
configuration for nearly any purpose.
* Completely rewritten documentation system.
* Installation is compatible towards pygame - it goes into a new
namespace and module hierarchy, so having pygame and pygame2 installed
won't hurt your pygame 1.x applications.

Besides those great things about which we all can get excited, there are
some downsides to it - otherwise it would not be pygame.

Changes
-------
* Module, class and package layout changed.
* Completely different API in nearly all places for both, Python and C.
* No ABI or API compatibility towards pygame 1.x (and it is not planned!)
* Feature compatibility towards pygame 1.x only in limited areas.
* Overhauled test system.

Downsides
---------
* Early development stage. Expect weird things to happen when you use it.
* Not many examples and tests yet.
* API likely to change before an official release is made.
* MacOS support not available yet.
* physics module not working yet.
* Many other issues to be revealed.

Ned Batchelder: Victoria Marcus Olds, 1911–2008

2008-11-19T00:14:08+00:00

My grandmother died this morning. She was 97 years old, a good long life. She lived in New York City most of my life, but lived in a nursing home near me in Boston for the last year, so I saw her a few times, and those visits taught me some things about her.

Brian Jones: MySQL Problem and Solution Posts: r0ck.

m0j0 — 2008-11-18T21:30:50+00:00

Taming MySQL is… challenging. Especially in very large, fast-growth, ‘always-on’ environments. It’s one of those things where you seemingly can never know all there is to know about it. That’s why I really like coming across posts like this one from FreshBooks that describes a very real problem that was affecting their users, how they dealt with it, why *that* failed, and what the final fix was. Post a link to your favorite MySQL Problem and Solution post in the comments (oh yeah, and “subscribe to comments” should be working now!)

Sean McGrath: Open Source Considered Harmful

Sean — 2008-11-18T20:02:00+00:00

"It is hot in this kitchen and getting hotter all the time. What to do? Find a way to reduce the heat or leverage the latest asbestos suits? We are mostly doing the latter. -- Open Source Considered Harmful

Paulo Nuin: Creating an interface for the motif finding script, some corrections

Paulo Nuin — 2008-11-18T19:45:31+00:00

We need to pause a bit and do some corrections on our code. First the code I posted on the last entry for the pymotif.py module is wrong. Ok, not wrong, but some of the code I use to test ended up on the blog. Ths first two lines of the calculate_motifs function contained a link to the files I use for testing and should be replaced by

input_seqs = fasta.read_seqs(open(input_seqs).readlines())
input_seqs2 = fasta.read_seqs(open(input_seqs2).readlines())

Also both variables that store the filenames and paths in pymoteGUI.py are declared in the wrong scope. The should have be declared at the pymotGUI class level, so it is accessible to all the functions in that class. This also means that every time we access the variable it should be preceded by the class name in order for the interpreter to know where the to get the value from. So both corrected files would be

#!/usr/bin/env python

import wx
import pymot
import pymotif
import fasta
import os

class pymot(wx.App):

    def __init__(self, redirect=False):
        wx.App.__init__(self, redirect)

class pymotGUI(wx.Frame):

    fore_file = ''
    back_file = ''

    def __init__(self, parent, id):
        wx.Frame.__init__(self, parent, id,  'Python Motif Finder', style=wx.DEFAULT_FRAME_STYLE)
        self.__do_layout()

    def __do_layout(self):

        #adding the panel
        panel = wx.Panel(self)

        #defines the menubar
        menubar = wx.MenuBar()

        #file menu
        filemenu = wx.Menu()
        foreground_menu = filemenu.Append(-1, 'Select foreground file')
        background_menu = filemenu.Append(-1, 'Select background file')
        sep = filemenu.AppendSeparator()
        quitmenu = filemenu.Append(-1, 'Quit')

        #appends the menu to the menubar and creates it
        menubar.Append(filemenu, 'File')
        self.SetMenuBar(menubar)

        #input box for motif width, and label
        self.one_label = wx.StaticText(panel, -1, 'Motif width', (10,50))
        self.motif_width = wx.TextCtrl(panel, -1, '10', (95, 50), (40,18))
        #result textbox
        self.results = wx.TextCtrl(panel, -1, '', (150, 50), (200, 100), wx.TE_MULTILINE | wx.TE_AUTO_SCROLL | wx.HSCROLL)

        #run bbutton
        self.run_button = wx.Button(panel, -1, 'Run', (10, 80))

        #labels
        self.fore_label = wx.StaticText(panel, -1, 'Select the foreground file', (10, 10))
        self.back_label = wx.StaticText(panel, -1, 'Select the background file', (10, 30))

        #binding the menus to functions
        self.Bind(wx.EVT_MENU, self.on_foreground, foreground_menu)
        self.Bind(wx.EVT_MENU, self.on_background, background_menu)
        self.Bind(wx.EVT_BUTTON, self.run_finder, self.run_button)

    def on_foreground(self, event):
        dialog = wx.FileDialog(self, style=wx.OPEN)
        if dialog.ShowModal() == wx.ID_OK:
            pymotGUI.fore_file = dialog.GetPath()
            self.fore_label.SetLabel(pymotGUI.fore_file)

    def on_background(self, event):
        dialog = wx.FileDialog(self, style=wx.OPEN)
        if dialog.ShowModal() == wx.ID_OK:
            pymotGUI.back_file = dialog.GetPath()
            self.back_label.SetLabel(pymotGUI.back_file)

    def run_finder(self, event):
        print pymotGUI.fore_file
        result = pymotif.calculate_motifs(pymotGUI.fore_file, pymotGUI.back_file)
        for motif in result:
            self.results.WriteText(motif + 'n')
        #wx.MessageBox('It should run, eh?')

#if __name__ == '__main__':
app = pymot()
frame = pymotGUI(parent=None, id = -1)
#frame.CentreOnScreen()
frame.Show()
app.MainLoop()

and

#!/usr/bin/env python

import fasta
import sys
from collections import defaultdict

def choose(n, k):
    if 0 <= k <= n:
        ntok = 1
        ktok = 1
        for t in xrange(1, min(k, n - k) + 1):
            ntok *= n
            ktok *= t
            n -= 1
        return ntok // ktok
    else:
        return 0

def get_quorums(seqs, mlen):
    """
    add seq id_no to a set
    use explicit counter to create seq_no
    """
    quorum = defaultdict(int)
    for seq in seqs:
        for n in range(len(seq) - mlen):
            quorum[seq[n:n + mlen]] += 1
    return quorum

def calculate_motifs(input_seqs, input_seqs2):

    print input_seqs, input_seqs2
    input_seqs = fasta.read_seqs(open(input_seqs).readlines())
    input_seqs2 = fasta.read_seqs(open(input_seqs2).readlines())

    foreground = get_quorums(input_seqs, 10)
    background = get_quorums(input_seqs2, 10)

    N = len(input_seqs) + len(input_seqs2)

    res_motifs = []
    for i in foreground:
        term1 = choose(background[i], foreground[i])
        term2 = choose((N - background[i]), len(input_seqs) - 1)
        term3 = choose(N, len(input_seqs))
        p = (float(term1) * float(term2)) / term3
        if 0 < p <= 0.0001:
            res_motifs.append(i + 't' + str(foreground[i]) + 't' + str(background[i]) + 't' + str(p))

    res_motifs.sort()
    return res_motifs

On the next post, the last in the series, we will just check how to get the value from the width input box and wrap-up everything.

Technorati Tags: wxPython, python, motifs, GUI

Brian Jones: On Remote Workers and Working Remotely

m0j0 — 2008-11-18T18:39:49+00:00

I’ve been on both sides of the remote worker relationship. On the manager side, I’ve managed some good-sized projects using an all-remote work force. Indeed, I’ve hired, managed, fired, and promoted workers without ever knowing what they look like. On the worker side, I do most of my work remotely, and I have for some time now. Judging by the amount of repeat business I get, I’d say that I’m more than acceptably productive working remotely.

In dealing with various clients, recruiters, prospective employers, business owners, and talking to friends who manage people for a living, I’ve heard pretty much every excuse/reason there is for not wanting to deal with a remote work force. I’ve heard and experienced successes with remote workers as well, and they all have a few key things in common, which are missing from the stories of failure. I’ll talk about them in a minute.

I first want to just say that I’m not some kind of fanboy who thinks remote workers are the answer to every problem. There are valid reasons for not having remote workers. For example, it’d be hard to build cars with a remote work force. Some things (some!) just require a physical presence. Whoever maintains the printers at your company really has to be around to change out ink cartridges and stuff like that.

There are certain classes of jobs, though, that are well-suited to working remotely. There are even classes of jobs that are necessarily performed remotely to some degree (field sales and support technicians for example), that could be made 100% remote with the proper tools and processes in place.

So what makes a remote worker success story different from a story of failure?

Always be prepared…

The number one difference I’ve seen between success and failure in managing a remote work force is that successful managers spent the time to prepare the managers, the team, the department, the organization, and the remote workers themselves to work remotely.

If you don’t prepare for a remote work force, you will fail miserably. As a result, I’m a big advocate of treating “Let’s go remote!” as an internal project with goals and milestones just like any other project. Preparing an organization to manage a remote work force takes a good deal of forethought, with a focus on communication and collaboration tools, reporting, accountability, scheduling, etc. In addition, you have to prepare the remote workers themselves, to insure they know what’s expected of them in terms of reporting their status, scheduling, communication, etc. They also need to know *about*, and *how to use* the tools they’ll be expected to use from home.

You have to plan this. You have to prepare, or you’re going to be like the HR manager who told me their company no longer allows for remote workers because “we tried it once and the guy made a complete mess of things”. When I asked the HR manager why he attributed that to the geographic location of the worker, he said “good point, he could just as well have made a mess here in the office”. You need good workers no matter where they’re going to work. The workers need expectations and goals from the manager, and the manager needs feedback and communication (and results!) from the worker. Tools help to facilitate these things. This is already a long post, so I’ll probably make a tools list in another post.

Communicate, and set expectations

Before the tools come other higher-level decisions and communication. For example, one problem I’ve heard more than once about remote workers is “we can’t hire a remote worker full-time, because then everyone will want to work from home”. As if they didn’t already all want to work from home! Everyone would love to have the option! Even if they didn’t take advantage of it, they’d consider it a really cool perk! They’d tell all of their friends about it, because it would make them jealous, and guess who their friends will contact first when they start to look for other opportunities?

You have to start somewhere, and you can’t just swing the barn doors open and let everyone go their own way on day 1. If you have an existing corporate structure in place with assets and services and regular meetings and the like, then you have to decide who can make the most benefit from a remote situation the soonest, make them the pilot group, and manage the expectations of the rest of the organization while the pilot group prepares to move to a remote workspace.

1, 10, 100, 1000

A common software application rollout strategy is to make it accessible to 1 user, then 10, then 100, then 1000, then… move up from there. In preparing your organization or department, you might consider a similar strategy.

I work for a client right now where I’m the “1″. If I can work effectively with the rest of the team (in the office), if I can produce results, remain accessible as-needed during working hours, manage the expectations of my team with regards to my presence (appointments happen), and overall be an asset to the team, then the management may decide that it can work on some larger scale - even if ‘larger’ means 2 instead of 1. It might also be useful to do a ‘remote rotation’ so that glitches can be caught early before making a physical presence in the office optional.

Success, of course, means getting together with the team and figuring out what tools will be used to best emulate an office working environment. We use IRC for 99% of our communication, falling back to email when we need to cc managers, we have a wiki for documentation and status updates, we have a trouble ticket system, everyone has everyone else’s phone number, blackberry PIN, or whatever. We’re a technical group doing system administration. It’s working wonderfully.

“But if the sysadmins work from home, the developers will want to work from home!” Maybe so. That’s where you have to manage expectations, and communicate with your workers to let them know that the company’s ‘office optional’ project is in an early alpha stage, that it’s being tested on the group most familiar with the technologies involved, and most capable of exploiting those technologies successfully to produce results. Once the geeks work out the shortcomings, and management is able to evaluate the effectiveness of the plan, the tests will become more widespread.

Really, it’s not a whole lot different from doing anything else that affects the whole company: changing payroll providers, healthcare options, software and desktop hardware upgrades and replacements… it just takes communication. The process has to be managed, just like every other process.

There’s more than one way to do it!

There’s no one solution out there. When I joined php|architect Magazine in 2003, it was run by Marco Tabini, and I was a remote editor. A couple of months after joining, I became editor in chief, and was in charge of remotely managing the magazine. I did it differently from Marco, but he still remained involved and engaged through good communication.

Python Magazine was created and managed by me, and for the entire lifespan of the magazine, I have not seen anyone else involved in its production in person. Ever. Design, production, web site admin, executive administration, tech editors, authors, accountants… time lines, budgets and planning documents… all remote, and mostly delegated. I started the magazine with the thought that at some point someone more engaged in the community and with Python should take charge — I was just a “temp” to get the vision off the ground. Sure enough, when I handed the magazine over to Doug Hellmann, he did things differently from me, and it’s working out wonderfully for him as well!

Everyone has their own management style. Don’t think that just because your management style is a little unique you can’t handle remote workers. Good managers are creative, and aren’t afraid to execute on creative solutions.

Simon Wittber: Fibra grows events.

Simon Wittber — 2008-11-18T15:02:11+00:00

Fibra has grown an event handling plugin.

  1 import fibra
  2 
  3 
  4 def task_a():
  5     for i in xrange(10):
  6         yield fibra.Send('AN_EVENT')
  7 
  8 def task_b():
  9     while True:
 10         msg = yield fibra.Recv('AN_EVENT')
 11         print 'Received:', msg
 12 
 13 def test():
 14     s = fibra.schedule()
 15     s.install(task_a())
 16     s.install(task_b())
 17     s.run()
 18 
 19 if __name__ == "__main__":
 20     test()

Now I need to reinstate the networking layer... and think about adding multi-core features via the new (in 2.6) multiprocessing module. It would be nice to have events magically transferred across process boundaries. FibraNet used to do this, but I haven't found a clean way of adding this to fibra, yet.

Simon Willison's Weblog: The new Lawrence.com

2008-11-18T14:25:34+00:00

The new Lawrence.com. The world’s best local entertainment website, relaunched on Django 1.0 with an accompanying substantial redesign.

Ivan Krstic: DeepSec

Ivan Krstić — 2008-11-18T13:16:50+00:00

Last Friday, I delivered my keynote at DeepSec ‘08 in Vienna. Anton Chuvakin from Qualys posted some thoughts on my talk, and ORF has a more detailed interview with me in German.

Ivan Krstic: No yin for Jerry Yang

Ivan Krstić — 2008-11-18T11:14:32+00:00

Yahoo CEO Jerry Yang announced his departure yesterday in a fluffy memo to his employees. The reader’s guide:

the board of directors and I have agreed to initiate a succession process for the ceo role of yahoo!.

The board got its head out of its ass long enough to fire me. Given their track record, I’m as shocked as you are.

once a successor is named, i will return to my previous role as chief yahoo and continue to serve as a director on the board.

Once a successor is named, I will return to complete irrelevance and hopefully crawl under a very large rock.

we have created a more open, competitive yahoo! and we believe the time is now right to transition to a new ceo who can take the company to the next level.

Zeus have mercy on the poor sod who comes in to fix the mess I’ve made.

the fact remains that yahoo! is now a significantly different company

Since, you know, I erased $30 billion in shareholder value by rejecting the Microsoft deal and then drove a hundred and twenty-five of my top executives to quit.

that is stronger in many ways than it was just 18 months ago.

And by “stronger”, I mean “more colorfully fucked”.

while this step will be an adjustment for all of us

I’ll be adjusting to golf, you’ll be adjusting to unemployment.

i will continue to do everything i can to make yahoo! fulfill its full potential.

So long, suckers.

Kelly Yancey: RFC868 UDP Time Protocol Client

Kelly Yancey — 2008-11-18T09:21:35+00:00

Here is an RFC 868 UDP Time Protocol Client implementation in less than 30 lines of python.
I know...just what you always wanted. It is a long story, but I needed a time protocol client (that would run on Windows) for testing a service we're developing at work. There are tons of implementations of the TCP version of the protocol, but I couldn't find a UDP implementation for Windows to save my life. Python to the rescue.

from socket import *
from struct import unpack
from time import ctime, sleep
from sys import argv

argv = argv[1:]
if len(argv) == 0:
   argv = [ 'time-nw.nist.gov' ]

s = socket(AF_INET, SOCK_DGRAM)
s.settimeout(5.0)

for server in argv:
   print server, ":",
   try:
       s.sendto('', 0, (server, 37))
       t = long(unpack('!L', s.recv(16)[:4])[0])
       # Convert from 1900/01/01 epoch to 1970/01/01 epoch
       t -= 2208988800
       print ctime(t)
   except timeout:
       print "TIMEOUT"
   except:
       print "ERROR"

s.close()
sleep(2)

In case you are curious, the sleep(2) at the end is there so our testers can simply click the icon on their desktop and have time to actually see the results before Windows closes the console window.

By default, it queries the time-nw.nist.gov server, but you can specify any number of servers to query on the command-line.

Shannon -jj Behrens: AI: Thankful for Bad AI

Shannon -jj Behrens — 2008-11-18T03:34:59+00:00

Imagine if the first computers man was able to create worked in pretty much the same way the human brain works. Imagine if they were pretty decent at reasoning, and terrible at calculating things quickly without error. Image that instead of having a quest for artificial intelligence, we had a quest for a "really fast, really accurate data cruncher." It'd be a different world. It definitely makes me grateful that we have humans *and* computers, each very useful in their own way.

The question of whether computers can think is like the question of whether submarines can swim -- Edsger W. Dijkstra

James Tauber: Fixing Relative Links In Entries

James Tauber — 2008-11-17T22:50:09+00:00

Occasionally I hear from someone who finds the links in my entries are broken in their feed reader, especially if they are reading a syndicated version of my entries.

The reason is that I have relative links which are not being made absolute by the reader (or syndication code).

Many readers (such as Google Reader) treat relative links as being relative to the feed itself, so /blog/ gets converted to http://jtauber.com/blog/. But even if the reader does, this, that doesn't help with syndicated entries unless the syndicator does some processing. Some do but others don't.

There's nothing wrong with relative links in Atom content, but if you use them you really should have an xml:base attribute to help a reader deal with them properly. I've now updated all my entries to include an xml:base. I'll see how many planets and other syndications pass that through when placed on the entry. I wonder if the content element would be better.

In the process of making the change, though, I noticed my entry/link[@rel="alternative"] were also relative which is, I suspect, a no no. So I've made them absolute as well.

I'm going to do a little bit more experimentation, but things should work now. I may still have goofed up somewhere but I'm fairly confident now that if links are broken then it's either a bad reader or bad syndication involved. Either way, please let me know in a comment below if you encounter any problems here on in.

UPDATE 1: The atom feed of the unofficial planet python doesn't pass through the xml:base but it doesn't need to as it has made all links absolute (and was doing so before my change)

UPDATE 2: I'm afraid that Advogato's syndication is just plain broken. In their RSS 2.0 feed (why not provide Atom!?), they make content links absolute but they do so by treating them relative to www.advogato.org !! My only suggestion is to just avoid Advogato syndication all together. I'm tempted to turn it off.

Eric Florenzano: WSGI middleware is awesome, and Django should use it more

2008-11-17T21:26:48+00:00

Most people in the Django community are deploying their apps these days with mod_wsgi. If not, then you're at least using WSGI as a communication layer with your application server, in one way or another. The great thing about WSGI is that it gives everyone a common interface through which to talk. It also has the added benefit of being a common abstraction that many people have built these great, really useful tools on top of.

Consider Repoze. If you navigate to the middleware section of their website, they have some really cool stuff available! There are utilities for logging, authentication, security, profiling, templating, etc. All of these pieces of middleware are designed to be totally pluggable, because they are designed to work solely based on what's available through WSGI.

My personal favorite of that lot is repoze.profile. It accumulates Python profiling information about whatever app is being run, and allows you to view that profile information via a web interface by visiting a special URL. There is absolutely no reason that the Pylons, TurboGears, or CherryPy guys should be able to get away with keeping this stuff for themselves, so I want to show just how easy it is to integrate this profiling module with Django.

First, though, here's a typical .wsgi file that might be used in conjunction with mod_wsgi:

import os, sys
sys.stdout = sys.stderr

os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

There's really nothing special going on here, and if you would like to learn more about how to set up this WSGI file, visit mod_wsgi's documentation on the subject. Now if you'll notice, application is simply an instance of WSGIHandler, which is simply a callable. A WSGI middleware is just a wrapper around that callable. Here's how easy it is to add the profiling middleware:

from repoze.profile.profiler import AccumulatingProfileMiddleware
application = AccumulatingProfileMiddleware(
    application,
    log_filename='/tmp/djangoprofile.log',
    discard_first_request=True,
    flush_at_shutdown=True,
    path='/__profile__')

There we go! We have imported the profiling middleware, and passed the Django WSGI application as the first argument. The rest is just setting options for the middleware. You can restart apache and the WSGI profiling middleware is already working.

Sometimes, though, you don't want all of Apache just to run some middleware. You want to be able to do the same thing, but locally. Believe it or not, Django's local development server is just a WSGI server itself, so one option would be to do the wrapping directly in django, right here. But you really don't want to be hacking inside of Django internals if you don't have to. Fortunately there are many alternative WSGI servers out there. Brian Rosner has created a custom management command to use the excellent CherryPy WSGI server with Django, on his blog.

Let's say you just want to try this out quickly after reading this blog post, though. If you're running Python 2.5 or greater, you're in luck, because a script less than 10 lines long can get you up and running:

#!/usr/bin/env python

import sys
from wsgiref.simple_server import make_server

if __name__ == "__main__":
    execfile(sys.argv[1])
    httpd = make_server('', 8000, application)
    httpd.serve_forever()

Now, to run it, simply invoke it like this:

python runserver.py my_wsgi_file.wsgi

Now, navigate around your app for a little bit and then point your browser to the profile url and see how freaking awesome middleware can be.

I'm not trying to stir up any controversy, I'm not saying we should stop making Django middleware or anything like that. But I seriously, seriously hope that someone tries this out and realizes the multitudes of great WSGI apps out there that can be taken advantage of. Mark Ramm wasn't full of hot air when he talked about this at DjangoCon or blogged about it later. He was right, and I for one wish I had listened sooner.

Ed Taekema: Mondays are for Duarte

Ed Taekema — 2008-11-17T17:36:55+00:00

Well its cold and wintery this Monday. So what better way to warm up the brain than by reviewing some great Duarte content. Nancy recently appeared at the Apple Store in San Franscisco and let the word out of the apearance through twitter. Sounds like the event was a success. Here are some good Duarte Vids to get things moving on a Monday:

S.Lott: Concealing the code base -- in Python. I think not.

2008-11-17T11:10:23+00:00

The "how do we protect our intellectual property?" question comes up from time to time because we're using Python. It also came up on Stack Overflow. Here's the approach we're taking.

Michael Sparks: Actor Model vs Denial of Service

Michael — 2008-11-17T03:40:21+00:00

Kamaelia implements something similar to, but not the same as the Actor Model. Since I'd not heard of the actor model for the majority of time whilst working on Kamaelia (kamaelia is more like occam, unix pipes, hardware, etc), I've been reading up on it. One thing that I've come across about it that suprises me is this:

Communications among actors occur asynchronously: that is, the sending actor does not wait until the message is received before proceeding with computation.
(this summary courtesy of the Wikipedia page on the Actor Model, but the literature supports this view as well)

The difference between Kamaelia and the Actor Model is in a few places, but possibly the most fundamental is this:

With the actor model (as I understand it)

You have a mailbox where you receive messages. ("recv from inbox")
communications defaults to unidirectional, though you can build bi-directional

When you send a message, you know the recipient and send it direct to the recipient. ("send to bob")
Message sending is asynchronous
ie sender knows receiver, receiver does not (necessarily) know sender

This potentially introduces coupling in ways that makes co-ordination languages harder to build

With Kamaelia's model:

You receive messages from multiple named inboxes (ala receiving data on stdin, or receiving signals)
communications defaults to unidirectional, though you can build bi-directional
You send messages to named outboxes (ala sending to stdout, stderr)
A higher level co-ordination language (effectively) joins the dots between outboxes to inboxes (generally)
Message sending defaults to asynchronous, but you can define a "pipewidth" or "max number of messages in transit" to allow synchrony, where needed (such as a producer that produces data faster than the consumer can consume it)
ie sender does NOT know receiver, receiver does not (necessarily) know sender

Much like a CPU doesn't know whether it's plugged into a motherboard or a testing harness for example.
This defaults to needing a co-ordination language - but this encourages greater reusability of components, through a dataflow model

I say "kamaelia's" model here, but this is pretty similar to hardware, unix pipes, etc.

Now the thing I can't tell by looking at literature is what the general case is for most actor systems in practice:

Do they do the literal, and solely asynchronous thing? (ie always return immediately without error when you send a message)
Or do they introduce the ability to add in synchrony where necessary? (block or return an exception when a maximum inbox size is reached)

The reason I ask is because if you don't have the ability to define pipewidths, maximum inbox sizes, maximum outbox sizes or something then you can easily cause a denial of service attack in that scenario by having producers overload consumers. Consider a frame grabber feeding a slow, experimental video codec. In that scenario, it becomes rather important to be able to have a form of blocking (or EAGAIN) behaviour available. Indeed, this is such a common issue, that it's why unix pipes & filehandles will buffer a small amount of data, but block if you send too much data (or exhibit EAGAIN behaviour :). Similarly this is what's behind the socket.listen(argument) call in a server - to allow a certain number of connections to queue up, before refusing connections...

So, the question I really have is this: if you use or implement an actor model system, do you have any ability in your implementation to be able to say "maximum number of pending incoming messages" ? If you don't, then it is remarkably easy to write code that can break an actor based system by mistake, with problems in dealing with that - making code more complex, and less reusable.

I'd really be interested in hearing both positives and negatives here...

Tales of a Programming Hobo - Christopher Armstrong: Twisted 8.2.0pre1

Christopher Armstrong — 2008-11-17T01:59:55+00:00

Please try out the first (and hopefully only) pre-release of Twisted 8.2.0, the first Twisted release since April.

Downloads

Release notes

Please try it out and report any bugs.