skip to navigation
skip to content

Planet Python

Last update: April 25, 2015 07:47 AM

April 24, 2015


PyCharm

Announcing the PyCharm 4.5 EAP build 141.832

Having announced the first Early Access Preview build of PyCharm 4.5 a couple of weeks ago, today we’re eager to let you know that the second PyCharm 4.5 EAP build 141.832 is ready for your evaluation. Please download it for your platform from our EAP page.

Just as always, this EAP build can be used for 30 days starting from its release date and it does not require any license.

This build introduces the brand-new and significantly reworked manage.py tool for Django projects with new user interface, autocompletion for manage.py tasks and their arguments:

managepy1

The manage.py console preserves the command history so you can easily fetch previous commands just hitting up/down arrows. Quick documentation for manage.py commands is also supported. Put a caret to any command and hit Ctrl+Q for the quick reference or external documentation:

managepy2

The IPython Notebook integration has been significantly improved with the new IPython Notebook console and numerous bug fixes:

ipythonnotebook

The other notable improvements in this EAP build are:

That’s not all as this build has many other improvements and bug fixes – for example, improved Django 1.8 code insight and Behave’s default parse mode for step parameters. So we urge you to check the fixed issues and compare this build to the previous one!

Please give PyCharm 4.5 EAP a try before its official release and please report any bugs and feature request to our issue tracker.

Develop with Pleasure!
-PyCharm team

April 24, 2015 06:41 PM


CubicWeb

Serving Cubicweb via WSGI with Pyramid: comparing the options

CubicWeb can now be powered by Pyramid (thank you so much Christophe) instead of Twisted.

I aim at moving all our applications to CubicWeb/Pyramid, so I wonder what will be the best way to deliver them. For now, we have a setup made of Apache + Varnish + Cubicweb/Twisted. In some applications we have two CubicWeb instances with a naive load balacing managed by Varnish.

When moving to cubicweb-pyramid, there are several options. By default, a cubicweb-pyramid instance started via the cubicweb-ctl pyramid command, is running a waitress wsgi http server. I read it is common to deliver wsgi applications with nginx + uwsgi, but I wanted to play with mongrel2 (that I already tested with Cubicweb a while ago), and give a try to the circus + chaussette stack.

I ran my tests :

Setup

In order to be able to start the application as a wsgi app, a small python script is required. I extracted a small part of the cubicweb-pyramid ccplugin.py file into a elo.py file for this:

appid = 'elo2'

cwconfig = cwcfg.config_for(appid)
application = wsgi_application_from_cwconfig(cwconfig)
repo = cwconfig.repository()
repo.start_looping_tasks()

I tested 5 configurations: twisted, pyramid, mongrel2+wsgid, uwsgi and circus+chaussette. When possible, they were tested with 1 worker and 4 workers.

Legacy Twisted mode

Using good old legacy twisted setup:

cubicwebctl start -D -l info elo

The config setting that worth noting are:

webserver-threadpool-size=6
connections-pool-size=6

Basic Pyramid mode

Using the pyramid command that uses waitress:

cubicwebctl pyramid --no-daemon -l info elo

Mongrel2 + wsgid

I have not been able to use uwsgi-mongrel2 as wsgi backend for mongrel2, since this uwsgi plugin is not provided by the uwsgi debian packages. I've used wsgid instead (sadly, the project appears to be dead).

The mongrel config is:

main = Server(
   uuid="f400bf85-4538-4f7a-8908-67e313d515c2",
   access_log="/logs/access.log",
   error_log="/logs/error.log",
   chroot="./",
   default_host="localhost",
   name="test",
   pid_file="/pid/mongrel2.pid",
   bind_addr="0.0.0.0",
   port=8083,
   hosts = [
       Host(name="localhost",
            routes={'/': Handler(send_spec='tcp://127.0.0.1:5000',
                                 send_ident='2113523d-f5ff-4571-b8da-8bddd3587475',
                                 recv_spec='tcp://127.0.0.1:5001',
                                 recv_ident='')
                   })
           ]
   )

servers = [main]

and the wsgid server is started with:

wsgid --recv tcp://127.0.0.1:5000 --send tcp://127.0.0.1:5001 --keep-alive \
--workers <N> --wsgi-app elo.application --app-path .

uwsgi

The config file used to start uwsgi is:

[uwsgi]
stats = 127.0.0.1:9191
processes = <N>
wsgi-file = elo.py
http = :8085
plugin = http,python
virtualenv = /home/david/hg/grshells/venv/jpl
enable-threads = true
lazy-apps = true

The tricky config option there is lazy-apps which must be set, otherwise the worker processes are forked after loading the cubicweb application, which this later does not support. If you omit this, only one worker will get the requests.

circus + chaussette

For the circus setup, I have used this configuration file:

[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555
pubsub_endpoint = tcp://127.0.0.1:5556
stats_endpoint = tcp://127.0.0.1:5557
statsd = True
httpd = True
httpd_host = localhost
httpd_port = 8086

[watcher:webworker]
cmd = /home/david/hg/grshells/venv/jpl/bin/chaussette --fd $(circus.sockets.webapp) elo2.app
use_sockets = True
numprocesses = 4

[env:webworker]
PATH=/home/david/hg/grshells/venv/jpl/bin:/usr/local/bin:/usr/bin:/bin
CW_INSTANCES_DIR=/home/david/hg/grshells/grshell-jpl/etc
PYTHONPATH=/home/david/hg/grshells//grshell-jpl

[socket:webapp]
host = 127.0.0.1
port = 8085

Results

The bench are very simple; 100 requests from 1 worker or 500 requests from 5 concurrent workers, getting the main index page for the application:

One ab worker

ab -n 100 -c 1 http://127.0.0.1:8085/

We get:

Synthesis (1 client)

Response times are:

Response time (1 client)

Five ab workers

ab -n 500 -c 5 http://127.0.0.1:8085/

We get:

Synthesis (5 clients)

Response times are:

Response time (5 clients)

Conclusion

As expected, the legacy (and still default) twisted-based server is the least efficient method to serve a cubicweb application.

When comparing results with only one CubicWeb worker, the pyramid+waitress solution that comes with cubicweb-pyramid is the most efficient, but mongrel2 + wsgid and circus + chaussette solutions mostly have similar performances when only one worker is activated. Surprisingly, the uwsgi solution is significantly less efficient, and especially have some requests that take significantly longer than other solutions (even the legacy twisted-based server).

The price for activating several workers is small (around 3%) but significant when only one client is requesting the application. It is still unclear why.

When there are severel workers requesting the application, it's not a surpsise that solutions with 4 workers behave significanly better (we are still far from a linear response however, roughly a 2x better for 4x the horsepower; maybe the hardware is the main reason for this unexpected non-linear response).

I am quite surprised that uwsgi behaved significantly worse than the 2 other scalable solutions.

Mongrel2 is still very efficient, but sadly the wsgid server I've used for these tests has not been developed for 2 years, and the uwsgi plugin for mongrel2 is not yet available on Debian.

On the other side, I am very pleasantly surprised by circus + chaussette. Circus also comes with some nice features like a nice web dashboard which allows to add or remove workers dynamically:

//www.cubicweb.org/file/5272071/raw//www.cubicweb.org/file/5272077/raw

April 24, 2015 06:15 PM


A. Jesse Jiryu Davis

Samantha Ritter And Me At Open Source Bridge 2015

Open source bridge 1

I'm so excited to tell you, my colleague Samantha Ritter and I were accepted to speak in Portland, Oregon this June.

Cat-herd's Crook: Enforcing Standards in 10 Programming Languages.

Samantha and I helped specify and test how MongoDB drivers behave in ten programming languages, and we persuaded dozens of open source programmers to implement these specifications the same. We created a test framework that verified compliance with YAML descriptions of how drivers should act, and we communicated updates to the specs by pushing changes to the YAML files.

We want to show you how we herded all the cats in the same direction.

How Do Python Coroutines Work?

I'll explain asyncio's new coroutine implementation in depth, including the mystical "yield from" statement. You'll know better than any of your peers how this amazing new programming model works. Plus, in a magical and entertaining feat of daring, I plan to live-code an asynchronous coroutine implementation before your very eyes!

April 24, 2015 05:01 PM


ClusterHQ

Data migration with Kubernetes and Flocker

warning

Please note: because this demo uses Powerstrip, which is only meant for prototyping Docker extensions, we do not recommend this configuration for anything approaching production usage. When Docker extensions become official, Flocker will support them. Until then, this is just a proof-of-concept.

We recently showed how you could use Docker Swarm to migrate a database container and its volume between hosts using only the native Docker Swarm CLI. Today we are going to show you how to do the same thing using only Kubernetes.

Kubernetes is great at orchestrating containers and Flocker is great at managing data volumes attached to containers.

Ideally – we want to use both systems together so we can orchestrate AND migrate containers. That is the aim of this demo, to show how using Powerstrip, we can extend Docker with tools like Flocker and still use orchestration tools like Kubernetes.

We also need to network our Kubernetes cluster – an ideal tool for this is Weave which allows us to allocate an IP address per container. In this example, we have fully integrated the Weave network into Kubernetes so each container is allocated an IP address from the weave bridge.

Scenario

This demo is the classic kubernetes guestbook app that uses PHP and Redis.

The aim is to migrate the Redis container AND it's data using nothing other than Kubernetes primitives.

We have labeled the 2 minions spinning and ssd to represent the types of disk they have. The Redis server is first allocated onto the node with the spinning disk and then migrated (along with its data) onto the node with an ssd drive.

This represents a real world migration where we realise that our database server needs a faster disk.

Before migration

before migration

fig 1. multiple PHP containers accessing Redis container on node 1

After migration

after migration

fig 2. Redis container & data volume migrated to node 2

Install

First you need to install:

We’ll use Virtualbox to supply the virtual machines that our Kubernetes cluster will run on.

We’ll use Vagrant to simulate our application stack locally. You could also run this demo on AWS or Rackspace with minimal modifications.

Demo

Step 1: Start VMs

The first step is to clone this repo and start the 3 VMs.

$ git clone https://github.com/binocarlos/powerstrip-k8s-demo
$ cd powerstrip-k8s-demo
$ vagrant up

Step 2: SSH to master

The next step is to SSH into the master node.

$ vagrant ssh master

We can now use the kubectl command to analyze our Kubernetes cluster:

master$ kubectl get nodes
NAME                LABELS              STATUS
democluster-node1   disktype=spinning   Ready
democluster-node2   disktype=ssd        Ready

Notice how we have labelled node1 with disktype=spinning and node2 with disktype=ssd. We will use these labels together with a nodeSelector for the Redis Master pod. The nodeSelector is what decides which node the redis container is scheduled onto.

Step 3: Start services

The first step is to spin up the 2 services. Services are Kubernetes way of dynamically routing around the cluster – you can read more about services here.

master$ kubectl create -f /vagrant/examples/guestbook/redis-master-service.json
master$ kubectl create -f /vagrant/examples/guestbook/frontend-service.json

We can check that those services were registered:

master$ kubectl get services

Step 4: Start redis master

The next step is to start the redis master – we use a replication controller which has a nodeSelector set to disktype=spinning.

master$ kubectl create -f /vagrant/examples/guestbook/redis-master-controller.json

Once we have done this we run kubectl get pods and wait for the redis-master to move from status Pending to status Running

NOTE: it may take a couple of minutes before all pods enter the Running state.

Step 5: Start PHP replication controller

Now we start the PHP replication controller – this will start 3 PHP containers which all link to the redis-master service:

master$ kubectl create -f /vagrant/examples/guestbook/frontend-controller.json

Once we have run this – we run kubectl get pods and wait for our PHP pods to be in the Running state.

NOTE: it may take a couple of minutes before all pods enter the Running state.

Step 6: Confirm location of redis-master

Notice how the redis-master has been allocated onto node1 (democluster-node1):

master$ kubectl get pods | grep name=redis-master
redis-master-pod            10.2.2.8            redis-master        dockerfile/redis                          democluster-node1/172.16.255.251   app=redis,name=redis-master                    Running             About an hour

Step 7: Access application

The next step is to load the app in your browser using the following address:

http://172.16.255.251:8000

This will load the guestbook application – make a couple of entries clicking Submit after each entry.

screen shot

fig 3. screenshot of the guestbook app

Step 8: Migrate database

Now it's time to tell kubernetes to move the Redis container and its data onto node2 (the one with an SSD drive).

To do this we change the nodeSelector for the pod template in the replication controller (from spinning to ssd):

master$ kubectl get rc redis-master -o yaml | sed 's/spinning/ssd/' | kubectl update -f -

Then, we delete the redis-master pod. The replication controller will spin up another redis-master and use the modified nodeSelector which means it will end up on node2 (with the ssd drive).

master$ kubectl delete pod -l name=redis-master

Once we have done this we run kubectl get pods and wait for the redis-master to move from status Pending to status Running.

Notice how the redis-master has been allocated onto node2 (democluster-node2):

master$ kubectl get pods | grep name=redis-master
redis-master-pod            10.2.3.9            redis-master        dockerfile/redis                          democluster-node2/172.16.255.252   app=redis,name=redis-master                    Running             About an hour

Step 9: Access application

Now, load the app in your browser using same address:

http://172.16.255.251:8000

It should have loaded the entries you made originally – this means that Flocker has migrated the data onto another server!

note: it sometimes take 10 seconds for the service layer to connect the PHP to the Redis – if the data does not appear wait 10 seconds and then refresh

How it works

The key part of this demonstration is the usage of Flocker to migrate data from one server to another. To make Flocker work natively with Kubernetes, we've used Powerstrip. Powerstrip is an open-source project we started to prototype Docker extensions.

This demo uses the Flocker extension prototype (powerstrip-flocker). Once the official Docker extensions mechamisn is released, Powerstrip will go away and you’ll be able to use Flocker directly with Kubernetes (or Docker Swarm, or Apache Mesos) to perform database migrations.

We have installed Powerstrip and powerstrip-flocker on each host. This means that when Kubernetes starts a container with volumes – powerstrip-flocker is able prepare / migrate the required data volumes before docker starts the container.

Kubernetes Cluster

The 2 nodes are joined by the Kubernetes master. This runs the various other parts of Kubernetes (kube-controller, kube-scheduler, kube-apiserver, etc). It also runs the flocker-control-service.

k8s

fig 4. overview of the Kubernetes cluster

Conclusion

Kubernetes is a powerful orchestration tool and we have shown that you can extend it's default behaviour using Powerstrip adapters (and soon official Docker extensions).

This demo made use of local storage for your data volumes. Local storage is fast and cheap and with Flocker, it’s also portable between servers and even clouds.

We are also working on adding support for block storage so you can use that with your application.

Notes

Restart cluster

If you vagrant halt the cluster – you will need to restart the cluster using this command:

$ make boot

This will vagrant up and then run sudo bash /vagrant/install.sh boot which spins up all the required services.

/vagrant/demo.sh

There is a script that can automate the steps of the demo:

$ vagrant ssh master
master$ sudo bash /vagrant/demo.sh up
master$ sudo bash /vagrant/demo.sh switch
master$ sudo bash /vagrant/demo.sh down

The post Data migration with Kubernetes and Flocker appeared first on ClusterHQ.

April 24, 2015 03:00 PM


PyCon Sweden

PyCon Sweden schedule published

The schedule for the upcoming PyCon Sweden in Stockholm has been published on the homepage at pycon.se. There are less than twenty tickets left, and we’re closing registration on Monday the 27th, so this is your last chance to be part of the conference!

April 24, 2015 02:20 PM


Astro Code School

Video - Functional Programming in Python

In this video our Lead Instructor Caleb Smith presents basic functional programming concepts and how to apply them in Python. Check back later for more screencasts here and on the new Astro YouTube channel.

April 24, 2015 02:05 PM

Intro to Django by PyLadies RDU

PyLadies RDU will be offering a free four hour workshop on Django here at Astro! It'll be taught by Caktus Django developer Rebecca Conley. They'll conduct it here at Astro Code School on Saturday May 30, 2015 from 4pm to 8pm. For more information and to RSVP please join the Pyladies RDU meetup group.

April 24, 2015 01:50 PM


PyCharm

Meet the winners of the Best Interactive Programming Course Contest

A couple of months ago the Best Interactive Programming Course Contest organized by JetBrains came to a close. The courses that were produced as part of this contest will surely help thousands of students around the world learn Python using our free, modern and professional tool — PyCharm Educational Edition.

pycharm_contest_banner

Dmitry Filippov seized the opportunity and interviewed the contest winners: John Zurawski, the author of “Logging in Python” (1st place) and Lisa Chung, who contributed “Introduction to Classic Ciphers” (2nd place). Read on to learn more about them, their experience participating in the contest, their interactive courses and PyCharm Educational Edition.

— Hi John and Lisa, could you tell us a bit about yourself and your day jobs?
John: I am a professional software developer using Python. I also volunteer as a co-organizer of the DFW Pythoneers, a Python user group for the Dallas/Fort-Worth metroplex.
Lisa: I’m a software developer for The Motley Fool (Washington, DC). At the moment, my team’s building a tool for managing APIs.

— What was your motivation for creating a course for the contest?
John: I know a lot of people new to programming in general are coming to Python because it’s easy to learn and has a rich ecosystem of open source libraries. In many cases these developers are more focused on solving their problems, so are not always familiar with the techniques professional software developers use to debug their programs. I felt it was important to show them better solutions exist and take almost no additional effort.
Lisa: I’m interested in coaching others to become coders. Many are capable of coding, but getting started is the hard part. Tools like PyCharm Educational Edition can be a big help for those newcomers.

helloworld

— Tell us about the courses you’ve created. What are they all about and how will they help students learn something new?
John: I created a course about the Python logging module. I am passionate about logging, because it has helped me solve many complex problems over the years. I felt if I could introduce this topic to people new to programming as an important debugging tool. I thought the interactive approach offered by PyCharm Educational, would allow them to see the benefits of using the logging module very quickly, and hopefully see the value in investing the time to learn more.

logginginpythonoverview
Lisa: “Introduction to Classic Ciphers” is a very brief overview of the null, Caesar, and Atbash ciphers. These are three of the more accessible classical ciphers. What makes it a python course is that we work on python implementations of the encipher and decipher algorithms of each cipher. The course also forces you to read chunks of code written by someone else (me). I think it’s important to practice reading and debugging code written by others. For me, it’s a good measure of how well I understand a programming language.

classic_cyphers

— In your opinion, how well does an interactive approach to learning work when combined with professional tools like PyCharm?
John: I feel the benefit of a professional tool like PyCharm is the effort to get started is very low compared to alternatives. Having everything in one cross-platform solution eliminates a lot of frustration with getting the environment set up and working, for both educators and students.
Lisa: I’m a fan, especially when it’s paired with a professional tool such as PyCharm, which I myself use. Coders of all levels can use PyCharm to write their code; newcomers and professionals both benefit from this. For newcomers, it certainly removes one hurdle of transitioning into a professional coder. Professional coders benefit because the tool is less likely to grow bloated.

cyphers_exmpl1

— How much time did you spend creating your amazing courses?
John: I spent about one entire weekend and several evenings working on the course. The first 80% came fairly easily, but the last 20% was a bit of a struggle to get done in time for the contest deadline. The contest organizers gave plenty of time, but unfortunately I didn’t get started until closer to the deadline.
Lisa: PyCharm Educational Edition is very straightforward. I watched the tutorial video and brainstormed for a few days. With respect to writing the course, I spent one day. Had I budgeted more time, I could see myself spending more days to flesh out and refine the course.

— What was your experience like creating a course with PyCharm Educational Edition?
John: I learned a lot by writing the course. There are some features in the standard library module I had no idea existed. Going through the effort to explain the course material in simple terms caused me to strengthen my own knowledge in this area.
Lisa: I had fun! Until PyCharm Educational Edition, I had never considered creating an educational course. You and your team have done a great job with the framework; for course creators like me, there is very little overhead beyond writing up the content. I appreciate the course structure, in that you have to think in terms of Lessons, each Lesson with a set of Tasks, with each Task having an html page and a python exercise. The enforced structure really helps course creators like me (no training in teaching or curriculum development) stay focused.

creation

— Would you recommend PyCharm Educational Edition to other educators?
John: I would actually recommend PyCharm Educational Edition to anyone who is interested in Python. I would recommend picking a subject area they have an interest in and attempt to create a course using PyCharm Educational Edition. It’s something that can help other users in the Python community, but will also provide an opportunity to master the subject area for their own benefit. I believe there is truth in the quote: “To teach is to learn twice.” I recommend planning out your complete course before you attempt to author your course inside PyCharm Educational edition. This will help you identify introductory topics you missed and ensure the course flows well between tasks.
Lisa: Yes, especially if you wish to develop your own courses.

— What was the most challenging part of creating your course?
John: I found it challenging to come up with fun tasks. I was not very happy with some of my examples and feel there is room for making them more interesting. I feel making interesting and fun tasks is an important part of keeping people engaged with a course.

funtask

—Do you feel anything is missing in PyCharm Educational Edition?
John: Although the emphasis is on interactive programming tasks, I would like to see the ability to organize a course into description-only tasks and quizzes. Description-only tasks would have allowed me to break up some of the longer explanatory tasks into smaller and easier to digest pieces. Quizzes would have helped reinforce key subject areas without having to implement repetitive looking tasks. I feel the courses would be easier for self-study by students with these features.
Lisa: I’m curious whether an online forum for each course would benefit participants. This could be a place for users to ask for additional support (“could you explain Task x in more detail?”) and for the course creator to collect input on how to refine the course.

— Is there anything else you would like to add?
John: I would like to thank JetBrains for being active in supporting the Python community. Not only by offering the free PyCharm Educational and Community editions, but with their support of Python events and user groups.

— Thanks for the interview, John and Lisa!

You can download PyCharm Educational Edition here and make use of the winning courses contributed by John and Lisa. To check them out, go to File / New Project / Educational and click the refresh button.

Develop and learn with pleasure!
-PyCharm team

April 24, 2015 01:33 PM


بايثون العربي

سيرفر ويب بسيط باستخدام بايثون

إذا كنت بحاجة الى تشغيل سيرفر ويب على جهازك ولا تريد ان تقوم بتثبيت سيرفر أباتشي أو غيره ، عندها يمكنك الاستعانة ببايثون حيث ياتي هذا الأخير مع دالة مدمجة بسيطة تسمى SimpleHTTPServer ومعها يمكننا من تحويل أي دليل على نظامنا الى دليل سيرفر ويب وبمعنى أخر يمكننا من تصفح اي دليل على المتصفح وكل ما نحن بحاجة اليه هو تثبيت بايثون وفقط.

وقبل أن نبدأ اريد أن أنوه الى هذه العملية مفيدة جدا على الشبكات المحلية حيث تعتبر عملية مشاركة الملفات اسهل بكثير عن طريق المتصفح .
فلنفترض انني بصدد أن أقوم بمشاركة الدليل التالي home/kader/  والعنوان IP الخاص بي هو 192.168.1.102 نقوم بفتح الطرفية ونكتب السطرين التاليين




cd /home/kader$


python -m SimpleHTTPServer$




هذا فقط!! اﻷن سيرفر HTTP الخاص بك سيعمل على المنفذ 8000 وسنحصل على الرسالة التالية:

 Serving HTTP on 0.0.0.0 port 8000 ...

 اﻷن قم بفتح المتصفح واكتب العنوان التالي :

 http://192.168.1.102:8000

أو

 http://127.0.0.1:8000

 إذا كان الدليل يحتوي على ملف اسمه index.html سيتم اعتباره ملف أو صفحة رئيسية ويقوم بعرض محتويات تلك الصفحة


 وفي حالة عدم وجود ملف بذلك الاسم سيقوم بعرض محتويات الدليل على شكل قائمة.


كان هذا درس على السريع أرجوا أن تستفيدوا منه .


April 24, 2015 10:08 AM


Python Software Foundation

Finding global voices

On the psf-members mailing list today, current Director David Mertz expressed some sentiments about increasing diversity in the governance of the foundation that I'd like to share.  Making the Python community, and the Python Software Foundation itself, more diverse, globally and across dimensions of privilege is something we have been striving for very consciously for years. Here's what he wrote (re-posted with his permission):
This year, as for the last bunch of years, I'll be the election administrator in the upcoming election. This will have some candidates for the Board of Directors of the PSF, and probably a few other issues like Sponsor Members approvals or membership resolutions.
    This year, as in past Board elections, I will use "approval voting" again. This will be explained again when you get ballots and announcements here. But the general idea is that each voter can cast as many Approve votes as they wish to for the 11 seats. A voter might vote for only the one candidate they really like to avoid diluting that vote. Or they might vote for every candidate except the one they really don't like as an "anyone but" vote. Or, in most cases, voters will vote for some number of candidates whom they feel generally comfortable with or prefer, and skip voting for any others.
    I give this preface to explain how I intend to vote. I am a white, male, middle-class, middle-aged, cis-gendered, American who has been on the Board for a long while. I may or may not run for it again (my name is on the wiki now with no candidate description, but mostly as a placeholder to get some permission issues sorted out for editing the wiki).
    But what I REALLY want is to have a PSF Board that is less American, less white, less male (and ideally represents diversity along other dimensions also: religious, sexual identities, linguistic, disability, etc). So I earnestly urge any or all PSF members, or their friends and colleagues, or other members of the Python community, or general supporters of Free Software, who might consider serving on the Board to place themselves in nomination, or allow themselves to be so placed.
    Serving on the Board is a genuine commitment of time and effort, and carries a fiduciary obligation. It's not just an item to put on a resume, and I don't want names of Directors from subaltern* groups there just as names alone. But I really do want those names as people who actively participate in making our community both more vigorous and more diverse.
    Which is to say, that for MY own vote, I can pledge to vote Approve to any candidate with a minimal indication of commitment to the selfless, volunteer tasks involved who doesn't look or sound quite so much like myself.
    Please, wonderful potential candidates, step up and let me cast these votes!
[*] In critical theory and postcolonialism, subaltern is the social group who are socially, politically and geographically outside of the hegemonic power structure of the colony and of the colonial homeland. In describing "history told from below", the term subaltern derived from Antonio Gramsci's work on cultural hegemony, which identified the social groups who are excluded from a society's established structures for political representation, the means by which people have a voice in their society.
I would love to hear from readers. Please send feedback, comments, or blog ideas to me at msushi@gnosis.cx. 

April 24, 2015 01:27 AM

April 23, 2015


Invent with Python

Celebrating the Release of "Automate the Boring Stuff with Python" with Discount Codes!

My latest book, "Automate the Boring Stuff with Python" is being released on Saturday, the 25th! This is a Python programming book for complete beginners who want to learn to code using practical projects. Automate the Boring Stuff with Python will be available online under a Creative Commons license as well, making it accessible to all. (I'm working furiously to finish formatting the HTML version right now! It will be posted to AutomateTheBoringStuff.com)

No Starch Press has the 30% discount code PYBUTLER that you can use to order the book off their site. You'll get an Early Access copy now (the first 12 chapters) and then the full book on the 25th the full ebook immediately.

In addition, I've also added 70% discount codes for my other books! These are the max discounts I can give (I don't receive royalties for these sales). Thanks so much to all my readers, I never imagined six years ago when I started writing that I would be here now. :D

Invent Your Own Computer Games with Python, 3rd Edition FAM5M2RM (Read online, download PDF or ebook)

Making Games with Python & Pygame 2A5GKQRE (Read online, download PDF or ebook)

Hacking Secret Ciphers with Python MRLEG3A7 (Read online, download PDF or ebook)

These discount codes will work for the next few days.

April 23, 2015 08:11 PM


Lightning Fast Shop

Release 0.10.1

We just released LFS 0.10.1. 

Changes

  • Adds Django 1.8 support

Information

You can find more information and help on following locations:

April 23, 2015 07:45 PM


Brett Cannon

Porting to Python 3 is like "eating your vegetables"

While at PyCon this year, someone pointed out that the hope/goal/expectation when Python 3 was released was to have over 50% of new projects using Python 3 within five years, and that had not come to fruition (while I have not personally crunched the numbers and this was told to me anecdotally, I will just assume they are right). Immediately after pointing this out they said that Python 3 is thus a failure at this point and we need to start figuring out when we call it quits on Python 3. I obviously disagree with this view.

We can’t predict the future

Just because some prediction didn’t come true does not mean we have lost hope in Python 3. All it means is that the community’s uptake of Python 3 has been slower than we initially thought, and honestly that’s fine. The extension of Python 2.7’s end-of-life to 2020 is an acknowledgment of this slower pace. There are plenty of companies and projects which have switched to Python 3 fully or are at least supporting it and seem happy with Python 3. The polling of the audience by Guido during his keynote along with some uptake numbers shared in confidence with the core developers shows that the level of use is actually too high to consider abandoning Python 3 without screwing over a sizable chunk of the community (Donald Stufft’s post on PyPI download statistics shows this).

So why are we as a community, more than 6 years after Python 3 was released, not transitioning to Python 3 faster? At the language summit, Larry Hastings said porting to Python 3 is like “eating your vegetables”; you know you should, but that doesn’t make it enjoyable. I think that is a very fair analogy as to why most people don’t finding porting fun.

Why the hell we separated text and binary data in the language

I have a sneaking suspicion that some people think we decided to clearly separate text and binary data “just because” or it seemed like the proper thing to do from some theoretical language design standpoint or something; all of those reasons are wrong. The reason we made the separation was to provide language level support/restrictions to prevent people from making subtle bugs when they mixed text and binary data. Talk to a Django core developer and ask them how much code and work it is to properly handle the separation of text and binary data in Python 2 and I’m sure you will be told it’s a lot and it’s a pain. I think Jacob Kaplan-Moss tells me almost annually at PyCon that the day Django stops supporting Python 2 they will be able to rip out a ton of code that exists purely because it was so easy to mix text and binary data and get it wrong. In other words most of us are mediocre programmers and this text/binary separation helps us not to screw up (also making sure code works with Unicode for text is also a benefit of this separation).

This means dealing with the text/binary separation in Python 3 is part of eating your vegetables. It was done to help people in the long term. People also seem to forget that the core developers have to eat the exact same vegetables as everyone else. The core developers are not some magical group of people who get to code in Python 3 all day long; plenty of us core developers still work with Python 2 as part of our job. That means decisions like the text/binary separation are not taken lightly as it impacts our lives just like everyone else’s.

How to make eating your vegetables easier

So you should eat your vegetables, but that doesn’t mean we can’t try to make it easier. I gave a talk at PyCon 2015 on making Python 2 code be 2/3 compatible. You should be able to get all the same information from the Python 2/3 porting HOWTO. The tooling and such should actually be to the point that if you run your code against Python 2.7 and Python 3.5 (when it’s released) that any porting issue shouldn’t be silent but instead raise a warning, exception, message, etc. In other words the only bit that should require a good amount of thinking is the text/binary separation for your APIs and making that separation happen.

You can also view the various steps to support Python 3 as part of your code’s health if that helps motivate you (or your manager). Since the changes in Python 3 are to make your code better, you could argue that modernizing your code and cleaning it up in Python 2 has a side-effect of supporting Python 3. And much like code maintenance, you can make porting a piecemeal process where you don’t have to do all of it at once but in pieces to gain different things (e.g., moving to iterator versions of map and other built-ins can lead to better performance and more use of iterators which just happens to align with what Python 3’s built-ins do).

But please don’t assume porting is free. It can be fairly straightforward for some projects, but harder for others. Either way there is some work involved. There have been reports of people in the community being somewhat mean to project maintainers for not porting which isn’t right. You can always ask someone to port their code, but you should always be cordial about it and be understanding if they choose not to.

Getting treats for eating your vegetables

Like many of you, when I was younger and I didn’t want to eat my vegetables my parents would tell me that if I did eat my vegetables I would get dessert. The Python development team is constantly trying to toss in more desserts into Python 3.

For some it may be asyncio that they get to look forward to when they port (and I’m willing to bet the first person to create a library which really harnesses asyncio is going to have a popular project on their hands). Others may be looking forward to type hints inlined in their code (for Python 2 you can put the type hints in stub files). Assuming things go the way I think they will, the async/await syntax for co-routines will be landing in Python 3.5 and that seems to be exciting people. For others it might be performance improvements that we have landed over the years which help with things like memory usage from strings, etc. But the point is that we are always trying to make your effort in porting to Python 3 worth that much more.

We are one community

Regardless of what version of Python you run, we are still a single community. While I think Python 3.4 is currently the best version of Python available, you’re welcome to disagree with me or agree but not think it’s worth your time and effort to use it; all I ask is you be informed in your opinion. Honestly, if some people never switch to Python 3 and make Python 2 the COBOL of dynamic programming languages I’m totally fine with it (just as long as you don’t make me maintain it past 2020 ☺). The one thing I really want to make sure never goes away is how awesome the Python community is.

April 23, 2015 07:14 PM


Import Python

Free Community Run Python Job Board

Unofficial Python Job Board is a 100% free and community run job board. To add a job vacancy/posting simply send a pull request ( yes you read that correctly ) to Python Job Repository

Submitting a Job Vacancy/Posting/Ad

The jobs board is generated automatically from the git repository, and hosted using github pages.

All adverts are held as markdown files, with an added header, under the jobs/ directory. Jobs files should look something like this:

---
title: <Job Advert Title (required)>
company: <Your Company (required)>
url: <Link to your site/a job spec (optional)>
location: <where is the job based? >
contract: permanent (or contract/temporary/part-time ..)
contact:
    name: <Your name (required)>
    email: <Email address applicants should submit to (required)>
    phone: <Phone number (optional)
    ...: ...
created: !!timestamp '2015-02-20' <- The date the job was submitted
tags:
  - london
  - python
  - sql
---

Full job description here, in Markdown format

To add your job, submit a pull request to this repo, that adds a single file to the jobs/ directory. This file should match the example above.

When each pull request is submitted, it gets validated, and then manually reviewed, before being added to the site. If the pull request fails the validation testing (Travis) then you must fix this before the pull request can proceed.

Previewing your submission

To preview your submission before creating a Review-request, there are a number of steps to follow:

  1. Install hyde - hyde.github.io <code>pip install hyde</code>
  2. Install fin - <code>pip install fin</code>
  3. clone/checkout the https://github.com/pythonjobs/template repository
  4. Within this clone, put your new file in <code>hyde/content/jobs/[job_filename].html
  5. Delete the contents of the <code>deploy</code> directory.
  6. from within <code>hyde/</code>, run <code>hyde serve</code>
  7. Open a web browser, and navigate to http://localhost:8080/

April 23, 2015 01:40 PM

Porting To Python 3 Book Campaign

TL;DR version of the post: We love Python 3. Porting to Python 3 - Edition 2 is an awesome book. There is a campaign underway to get the book updated and free for all to read / contribute to. Have a look and see if you like to support it.

2014 has seen increased adoption of Python 3

New features / changes in Python 3.x with no backporting to 2.x series. changelog output is compelling reason to adopt Python 3.x.

However Majority of in-production Python software are in 2.x . It's here the relevance of "Porting to Python 3: An in-depth guide" - written by Lennart Regebro comes into picture.

“Porting to Python 3? is in need of an update since Python 2.5 and 3.2 are in a way obsolete for Majority of Python Developers. Lennart sadly has less time. But he has decided to convert it into a community book, put it up on GitHub and give the Python Software Foundation a license to use it. However to make that a reality he has to clean up the current repository which is in a mess, and thus require efforts.

This campaign, is to get some funding to make it happen. Have a look at the campaign and see if you can contribute. You contribution will result in a Free for all "Porting to Python 3 Book" updated and available on Github.

About the Author

Lennart Regebro has been coding in Python since 1999 ( that's the last century for you ). A member of Python Software Foundation and Pykonik - Krakow Python user group. A look at Lennart Regebro's stackoverflow profile shows us he has answered 1492 Python Questions and many of his top answers are on Python 3.

April 23, 2015 01:40 PM

Python Practice Book Review

Python Practice Book is written by Anand Chitipothu.

Python Practice Book introduces one Python language concept at a time and immediately follows up with a questions or two to test your understanding of the concept.

The Book is compact and uses Programmer friendly style of showing code snippet along with an explanation. It's non verbose style makes it for a quick read.

From Pedagogy perspective the book's explanation equip the reader with requisite knowledge to answer the questions. Those who just learned Python or are in process of learning it should read Python Practice Book.

The book covers the following topics and has coding assignments/questions for each topic

Recently I recommended this Book to an Entrepreneur who incorporated it into their month long Python training for newbies/new joinees. They have nothing but good things to say about the book.

Anand's experience of taking many Python workshops over the years reflects in the quality of the content in the book.

April 23, 2015 01:40 PM

Halting Content Recommendation and Ranking

Gmail marked all emails ( Issue no 15 ) as SPAM. Many subscribers brought it to our notice.
Why ? Our guess is
Massive unique URL generation
To track which subscriber has clicked on which article/project/tweet we made every url unique. We do that so we can recommend and rank the articles for subscribers as per their interest. To track clicks each URL is unique. This results in massive URL generation. Something Gmail SPAM Filter detects and find unusual.

We decided to stop generating recommending content as of now so the emails wouldn't be marked as SPAM. What we would do instead is give each subscriber a preference page and allow them to click on topics that interest them. For e.g. a Flask Programmer would like to see more of Flask content as oppose to Django. So he could click on Flask under web framework category and we would only send him Flask related content.

The fact that gmail make for above 60% of our subscribers forces us to take this step.

Let us know if you have any alternative we could try

April 23, 2015 01:40 PM

Conversation with Albert Sweigart - Author of Automate the Boring Stuff with Python

Al Sweigart is a software developer and teaches programming to kids and adults. He has written several Python books for beginners, including Hacking Secret Ciphers with Python, Invent Your Own Computer Games with Python, and Making Games with Python & Pygame.

Having worked with TechED startups, I have witnessed first-hand the impact of Albert's book Invent with Python on students learning the ropes of programming. We are happy to have him with us for a coversation around Python, Books, Python in Education.

What got you started in the TechED space? Where did the motivation for Invent with Python come from ?

Around 2009 a friend was a nanny for a precocious 10-year-old who wanted to learn how to program. I thought this would be straight-forward web search, but the materials I found didn’t quite meet my needs. There were plenty of (costly) books for professional software engineers and plenty of computer science freshmen materials, but most people don’t kindle a joy in programming by calculating Fibonacci numbers.

So I started writing a tutorial on how to make simple games. I put the game projects in front of the concept explanations, only explaining enough about Python programming to get the reader through that particular game. Then I wrote another game, then another. Eventually this tutorial reached book-length, and some friends suggested I self-publish Invent Your Own Computer Games with Python.

People responded well enough to the book that I wrote Making Games with Python & Pygame for making games with 2D graphics and Hacking Secret Ciphers with Python for making encryption and code-breaking programs. In early 2014 I left my software developer job to write Automate the Boring Stuff with Python full time.

Congratulations on your new book Automate the Boring Stuff with Python. Who is the target audience for this book ?

There have been times when I’m chatting with non-programmer friends who incidentally tell me stories like, “Today at the office I spent three hours opening PDFs, copying a line, pasting it into a spreadsheet, and then moving on to the next PDF. Three hours.” And I used to tell them that they could have written a program to do that for them in fifteen minutes, but that news would sometimes crush them. They can’t believe how much effort they could have saved.

A computer is the primary tool for office workers, researchers, and administrators. They may have heard “everyone should learn to code” but not know how to get started or how exactly coding will practical for them. Automate is a Python book for people who have never programmed before and, while they don’t necessarily want to become software engineers, they want to know how they can make better use of their computers to save them time. It’s a programming book that skips computer science and focuses on practical application.

Which version of Python is the book written for? One FAQ is which version of Python should one learn/use? Does it make a big difference from a beginner's standpoint ?

The book uses Python 3, specifically, 3.4. There have been some backwards-incompatible changes (for the better) that were made between the 2.x and 3.x versions, but Python 3 was introduced several years ago now. The reasons to stall on upgrading, such as lack of Python 3-supporting modules, mostly no longer apply. The only excuse to use Python 2 is if you have a large existing program written in 2 that, despite all of the Python 3 migration tools that are available, can’t easily be upgraded.

But these are all technical details that don’t really apply to beginners; just use 3.

How has the experience been of writing a book for a publisher vis-a-vis self-publishing for you ?

As great as self-publishing and print-on-demand have been, it has been excellent working with a publisher like No Starch Press. I soon learned that when writing a book the writing was the easiest part. Editing, formatting, layout, and especially marketing were all hats that I had to wear when self-publishing, and at times I wore them poorly.

Self-publishing is an excellent gateway for new, unknown writers, and a good way to get feedback and experience of the entire process by frequently releasing short works. But working with a professional team let me focus on the content, and the final book is of much higher quality than the ones I produced on my own.

Having spent a large part of my career creating Visual Programming Language for Kids, I feel Visual Programming Languages like Scratch, CiMPLE, etc. are a better first step for children vis-a-vis Python. From a pedagogy standpoint, what are your thoughts on this ?

I am a huge advocate of Scratch in particular. It’s clear that they’ve done a lot of research to make their user interface on a par with Apple’s best products. Scratch is great for the 8–12 age group. It has instant, graphical feedback and the snap-together block interface makes it literally impossible to make syntax mistakes from typos. It’s incredibly frustrating for a child to slowly type out a command, only to be greeted with an obscure error message because of a typo. But from discussions with other educators, kids can quickly outgrow the limitations of Scratch. Teenagers are more than capable of putting together programs in a high-level language like Python (or even younger, if they are enthusiastic about programming).

April 23, 2015 01:40 PM


PyCharm

PyCon 2015: How It All Happened

PyCon2015-logo-lockupCircleWhat a whirlwind the past couple weeks have been! We just got back from Montreal, Canada, which hosted PyCon 2015. The conference was awesome and the people just amazing! Let me tell you all about our experience (and announce the winners of our license raffle!).

 

Five of us from the PyCharm team attended this conference:

imgo

From left to right: Anna Morozova, Dmitry Trofimov, Ekaterina Tuzova, Andrey Vlasovskikh, Dmitry Filippov

Lots of things kept us busy during the conference days.

First, we attended the Python Language Summit where the latest trends of the Python language were discussed, such as the development of the new PEP 484 for type hinting. As you may already know, at PyCharm we’re constantly improving our static code analysis to provide you with better autocompletion, navigation, code inspections, refactorings and so forth. We have our own way to infer types and the annotations format to help us better understand your code. At the summit, we shared the challenges we’ve come across with static analysis, as well as some of our ideas for improving Python to be more friendly to static analysis tools like PyCharm. For a long while we at PyCharm had been actively working on co-creation of the new PEP 484, so it was great to talk face-to-face with core Python developers like Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa. We took a pulse of the current developments and discussed further arrangements. At the end of the conference, Guido did a closing keynote about type hints and the future plans of the community. So, many exciting things to look forward to in this area!

Python Education Summit was another fantastic event. We learned a lot about how Python is being used to teach programming to kids, novice programmers and those for whom Python is a second language. Our lightning talk covering the brand-new PyCharm Educational Edition was received very well by the community. It helped us understand where we’re all heading and what the current needs are in this domain. We’re proud that PyCharm Educational Edition is evolving in the right direction. Talking to teachers and developers who are passionate about nurturing novice developers, we agreed to collaborate on getting Python-rich curriculums adopted in some local US schools, which is one of the major problems in the community right now.

The main PyCon 2015 conference was just awesome. We felt privileged to be a part of the community that is setting a standard every day in the world of Python programming. We had a booth in the expo hall, participated in development sprints and talked to hundreds of people, answering questions about our company, products, PyCharm itself and collecting valuable feedback:

booth_col

Having kicked off PyCharm 4.5 EAP prior to the conference, we were able to show off some new stuff we’d worked on lately.

It was also a great chance to meet a lot of our happy users and friends as well as new people passionate about Python and new technologies. To all the conference attendees, we say “Thank you!”. We hope you enjoyed it as much as we did!

booth3.jpg

We also held a PyCharm license raffle at the conference, with some cool swag up for grabs. Today we’re glad to give you the randomly chosen winners:

Congratulations! Your license notifications will be mailed out to you in the coming days.

If you didn’t win, you’ll still get a 15% discount for a new PyCharm personal license (if you ticked that option in the raffle application). We will get in touch with you soon with more information on how to redeem your personal discount.

With any problems, concerns or questions, ping us in comments below.

Develop with pleasure!
-PyCharm team

April 23, 2015 01:13 PM


Leonardo Giordani

Python decorators: metaprogramming with style

This post is the result of a lot of personal research on Python decorators, meta- and functional programming. I want however to thank Bruce Eckel and the people behind the open source book "Python 3 Patterns, Recipes and Idioms" for a lot of precious information on the subject. See the Resources section at the end of the post to check their work.

Is Python functional?

Well, no. Python is a strong object-oriented programming language and is not really going to mix OOP and functional like, for example, Scala (which is a very good language, by the way).

However, Python provides some features taken from functional programming. Generators and iterators are one of them, and Python is not the only non pure functional programming language to have them in their toolbox.

Perhaps the most distinguishing feature of functional languages is that functions are first-class citizens (or first-class objects). This means that functions can be passed as an argument to other functions or can be returned by them. Functions, in functional languages, are just one of the data types available (even if this is a very rough simplification).

Python has three important features that allows it to provide a functional behaviour: references, function objects and callables.

References

Python variables share a common nature: they are all references. This means that variables are not typed per se, being pure memory addresses, and that functions do not declare the incoming data type for arguments (leaving aside gradual typing). Python polymorphism is based on delegation, and incoming function arguments are expected to provide a given behaviour, not a given structure.

Python functions are thus ready to accept every type of data that can be referenced, and functions can.

Read this post to dive into delegation-based polymorphism and references in Python.

Functions objects

Since Python pushes the object-oriented paradigm to its maximum, it makes a point of always following the tenet everything is an object. So Python functions are objects as you can see from this simple example

>>> def f():
...  pass
... 
>>> type(f)
<class 'function'>
>>> type(type(f))
<class 'type'>
>>> type(f).__bases__
(<class 'object'>,)
>>>

Given that, Python does nothing special to treat functions like first-class citizens, it simply recognizes that they are objects just like any other thing.

Callables

While Python has the well-defined function class seen in the above example, it relies more on the presence of the __call__ method. That is, in Python any object can act as a function, provided that it has this method, which is invoked when the object is "called".

This will be crucial for the discussion about decorators, so be sure that you remember that we are usually more interested in callable objects and not only in functions, which, obviously, are a particular type of callable objects (or simply callables).

The fact that functions are callables can also be shown with some simple code

>>> def f():
...  pass
... 
>>> f.__call__
<method-wrapper '__call__' of function object at 0xb6709fa4>

Metaprogramming

While this is not a post on languages theory, it is worth spending a couple of words about metaprogramming. Usually "programming" can be seen as the task of applying transformations to data. Data and functions can be put together by an object-oriented approach, but they still are two different things. But you soon realize that, as you may run some code to change data, you may also run some code to change the code itself.

In low level languages this can be very simple, since at machine level everything is a sequence of bytes, and changing data or code does not make any difference. One of the most simple examples that I recall from my x86 Assembly years is the very simple self obfuscation code found is some computer viruses. The code was encrypted with a XOR cipher and the first thing the code itself did upon execution was to decrypt its own code and then run it. The purpose of such tricks was (and is) to obfuscate the code such that it would be difficult for an antivirus to find the virus code and remove it. This is a very primitive form of metaprogramming, since it recognizes that for Assembly language there is no real distinction between code and data.

In higher lever languages such as Python achieving metaprogramming is no more a matter of changing byte values. It requires the language to treat its own structures as data. Every time we are trying to alter the behaviour of a language part we are actually metaprogramming. The first example that usually comes to mind are metaclasses (probably due to the "meta" word in their name), which are actually a way to change the default behaviour of the class creation process. Classes (part of the language) are created by another part of the language (metaclasses).

Decorators

Metaclasses are often perceived as a very tricky and dangerous thing to play with, and indeed they are seldom required in Python, with the most notable exception (no pun intended) being the Abstract Base Classes provided by the collections module.

Decorators, on the other side, are a feature loved by many experienced programmers and after their introduction the community has developed a big set of very interesting use cases.

I think that the first approach to decorators is often difficult for beginners because the functional version of decorators are indeed a bit complex to understand. Luckily, Python allows us to write decorators using classes too, which make the whole thing really easy to understand and write, I think.

So I will now review Python decorators starting from their rationale, then looking at class-based decorators without arguments, class-based decorators with arguments, and finally moving to function-based decorators.

Rationale

What are decorators, and why should you learn how to define and use them?

Well, decorators are a way to change the behaviour of a function or a class, so they are actually a way of metaprogramming, but they make it a lot more accessible than metaclasses do. Decorators are, in my opinion, a very natural way of altering functions and classes.

Moreover, with the addition of some syntactic sugar, they are a very compact way to both make changes and signal that those changes have been made.

The best syntactic form of a decorator is the following

@dec
def func(*args, **kwds):
    pass

where dec is the name of the decorator and the function func is said to be decorated by it. As you can see any reader can quickly identify that the function has a special label attached, thus being altered in its behaviour.

This form, however, is just a simplification of the more generic form

def func(*args, **kwds):
    pass
func = dec(func)

But what actually ARE the changes that you may want do to functions or classes? Let us stick for the moment to a very simple task: adding attributes. This is by no means a meaningless task, since there are many practical use cases that make use of it. Let us first test how we can add attributes to functions in plain Python

>>> def func():
...  pass
... 
>>> func.attr = "a custom function attribute"
>>> func.attr
'a custom function attribute'

and to classes

>>> class SomeClass:
...  pass
... 
>>> SomeClass.attr = "a custom class attribute"
>>> SomeClass.attr
'a custom class attribute'
>>> s = SomeClass()
>>> s.attr
'a custom class attribute'

As you can see adding attributes to a class correctly results in a class attribute, which is thus shared by any instance of that class (check this post for some explanations about class attributes and sharing).

Class-based decorators without arguments

As already explained, Python allows you to call any object (as you do with functions) as long as it provides the __call__() method. So to write a class-based decorator you just need to create an object that defines such a method.

When used as a decorator, a class is instantiated at decoration time, that is when the function is defined, and called when the function is called.

class CustomAttr:
    def __init__(self, obj):
        self.attr = "a custom function attribute"
        self.obj = obj

    def __call__(self):
        self.obj()

As you can see there is already a lot of things that shall be clarified. First of all the class, when used as a decorator, is initialized with the object that is going to be decorated, here called obj (most of the time it is just called f for function, but you know that this is only a special case).

While the __init__() method is called at decoration time, the __call__() method of the decorator is called instead of the same method of the decorated object. In this case (decorator without arguments), the __call__() method of the decorator does not receive any argument. In this example we just "redirect" the call to the original function, which was stored during the initialization step.

So you see that in this case we have two different moments in which we may alter the behaviour of the decorated objects. The first is at its definition and the second is when it is actually called.

The decorator can be applied with the simple syntax shown in a previous section

@CustomAttr
def func():
    pass

When Python parses the file and defines the function func the code it executes under the hood is

def func():
    pass

func = CustomAttr(func)

according to the definition of decorator. This is why the class shall accept the decorated object as a parameter in its __init__() method.

Note that in this case the func object we obtain after the decoration is no more a function but a CustomAttr object

>>> func
<__main__.CustomAttr object at 0xb6f5ea8c>

and this is why in the __init__() method I attached the attr attribute to the class instance self and not to obj, so that now this works

>>> func.attr
'a custom function attribute'

This replacement is also the reason why you shall also redefine __call__(). When you write func() you are not executing the function but calling the instance of CustomAttr returned by the decoration.

Class-based decorators with arguments

This case is the most natural step beyond the previous one. Once you started metaprogramming, you want to do it with style, and the first thing to do is to add parametrization. Adding parameters to decorators has the only purpose of generalizing the metaprogramming code, just like when you write parametrized functions instead of hardcoding values.

There is a big caveat here. Class-based decorators with arguments behave in a slightly different way to their counterpart without arguments. Specifically, the __call__() method is run during the decoration and not during the call.

Let us first review the syntax

class CustomAttrArg:
    def __init__(self, value):
        self.value = value

    def __call__(self, obj):
        obj.attr = "a custom function attribute with value {}".format(self.value)
        return obj

@CustomAttrArg(1)
def func():
    pass

Now the __init__() method shall accept some arguments, with the standard rules of Python functions for named and default arguments. The __call__() method receives the decorated object, which in the previous case was passed to the __init__() method.

The biggest change, however is that __call__() is not run when you call the decorated object, but immediately after __init__() during the decoration phase. This results in the following difference: while in the previous case the decorated object was no more itself, but an instance of the decorator, now the decorated objects becomes the return value of the __call__() method.

Remember that, when you call the decorated object, you are now actually calling what you get from __call__() so be sure of returning something meaningful.

In the above example I stored one single argument in __init__(), and this argument is passed to the decorator when applying it to the function. Then, in __call__(), I set the attribute of the decorated object, using the stored argument, and return the object itself. This is important, since I have to return a callable object.

This means that, if you have to do something complex with the decorated object, you may just define a local function that makes use of it and return this function. Let us see a very simple example

class CustomAttrArg:
    def __init__(self, value):
        self.value = value

    def __call__(self, obj):
        def wrap():
            # Here you can do complex stuff
            obj()
            # Here you can do complex stuff
        return wrap

@CustomAttrArg(1)
def func():
    pass

Here the returned object is no more the decorated one, but a new function wrap() defined locally. It is interesting to show how Python identifies it

>>> @CustomAttrArg(1)
... def func():
...     pass
... 
>>> func
<function CustomAttrArg.__call__.<locals>.wrap at 0xb70185cc>

This pattern enables you to do every sort of things with the decorated object. Not only to change it (adding attributes, for example), but also pre- or post- filtering its results. You may start to understand that what we called metaprogramming may be very useful for everyday tasks, and not only for some obscure wizardry.

Decorators and prototypes

If you write a class-based decorator with arguments, you are in charge of returning a callable object of choice. There is no assumption on the returned object, even if the usual case is that the returned object has the same prototype as the decorated one.

This means that if the decorated object accepts zero arguments (like in my example), you usually return a callable that accepts zero arguments. This is however by no means enforced by the language, and through this you may push the metaprogramming technique a bit. I'm not providing examples of this technique in this post, however.

Function-based decorators

Function-based decorators are very simple for simple cases and a bit trickier for complex ones. The problem is that their syntax can be difficult to grasp at first sight if you never saw a decorator. They are indeed not very different from the class-based decorators with arguments case, as they define a local function that wraps the decorated object and return it.

The case without arguments is always the simplest one

def decorate(f):
    def wrap():
        f()
    return wrap

@decorate
def func():
    pass

This behaves like the equivalent case with classes. The function is passed as an argument to the decorate() function by the decoration process that calls it passing the decorated object. When you actually call the function, however, you are actually calling wrap().

As happens for class-based decorators, the parametrization changes the calling procedure. This is the code

def decorate(arg1):
    def wrap(f):
        def _wrap(arg):
            f(arg + arg1)
        return _wrap
    return wrap

@decorate(1)
def func(arg):
    pass

As you see it is not really straightforward, and this is the reason I preferred to discuss it as the last case. Recall what we learned about class-based decorators: the first call to the decorate() function happens when the decorator is called with an argument. Thus @decorate(1) calls decorate() passing 1 as arg1, and this function returns the wrap() local function.

This second function accepts another function as an argument, and indeed it is used in the actual decoration process, which can be represented by the code func = wrap(func). This wrap() function, being used to decorate func(), wants to return a compatible object, that is in this case a function that accepts a single argument. This is why, in turn, wrap() defines and returns a _wrap() local function, which eventually uses both the argument passed to func() and the argument passed to the decorator.

So the process may be summarized as follows (I will improperly call func_dec the decorated function to show what is happening)

Obviously the power of the decorator concept is that you are not dealing with func_dec() but with func() itself, and all the "magic" happens under the hood.

If you feel uncomfortable with function-based decorators don't worry, as they are indeed a bit awkward. I usually stick to function based decorators for the simple cases (setting class attributes, for example), and move to class-based ones when the code is more complex.

Example

A good example of the power of decorators which comes out of the box is functools.total_ordering. The functools module provides a lot of interesting tools to push the functional approach in Python, most notably partial() and partialmethod(), which are however out of the scope of this post.

The total_ordering decorator (documented here) wants to make an object provide a full set of comparison ordering methods starting from a small set of them. Comparison methods are those methods Python calls when two objects are compared. For example when you write a == b, Python executes a.__eq__(b) and the same happens for the other five operators > (__gt__), < (__lt__), >= (__ge__), <= (__le__) and != (__ne__).

Mathematically all those operators may be expressed using only one of them and the __eq__() method, for example __ne__ is !__eq__ and __lt__ is __le__ and !__eq__. This decorator makes use of this fact to provide the missing methods for the decorated object. A quick example

class Person:
    def __init__(self, name):
        self.name = name

    def __eq__(self, other):
        return self.name == other.name

    def __lt__(self, other):
        return self.name < other.name

This is a simple class that defines the == and < comparison methods.

>>> p1 = Person('Bob')
>>> p2 = Person('Alice')
>>> p1 == p2
False
>>> p1 < p2
False
>>> p1 >= p2
Traceback (most recent call last):
  File "/home/leo/prova.py", line 103, in <module>
    p1 >= p2
TypeError: unorderable types: Person() >= Person()

A big warning: Python doesn't complain if you try to perform the > and != comparisons but lacking the dedicated methods it does perform a "standard" comparison. This means that, as the documentation states here, "There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false."

With the total_ordering decorator, however, all six comparisons become available

import functools

@functools.total_ordering
class Person:
    def __init__(self, name):
        self.name = name

    def __eq__(self, other):
        return self.name == other.name

    def __lt__(self, other):
        return self.name < other.name
>>> p1 = Person('Bob')
>>> p2 = Person('Alice')
>>> p1 == p2
False
>>> p1 != p2
True
>>> p1 > p2
True
>>> p1 < p2
False
>>> p1 >= p2
True
>>> p1 <= p2
False

Final words

Decorators are a very powerful tool, and they are worth learning. This may be for you just the first step into the amazing world of metaprogramming or just an advanced technique you may use to simplify your code. Whatever you do with them be sure to understand the difference between the two cases (with and without arguments) and don't avoid function-based decorators just because their syntax is a bit complex.

Resources

Many thanks to Bruce Eckel for his three posts which have been (I think) the source for the page on Python 3 Patterns, Recipes and Idioms (this latter is still work in progress).

A good source of advanced decorators can be found at the Python Decorator Library, and a lot of stuff may be found on Stackoverflow under the python-decorators tag.

Feedback

Feel free to use the blog Google+ page to comment the post. The GitHub issues page is the best place to submit corrections.

April 23, 2015 12:00 PM


Intellimath blog

AXON 0.6

There is new 0.6 release of pyaxon — python library for AXON.

Read more… (3 min remaining to read)

April 23, 2015 11:33 AM


Fabio Zadrozny

Wrapping docstrings/comments in PyDev

This is a nifty trick when working with PyDev...

When you're writing docstrings it's usually pretty annoying having to manually wrap the contents of the string to fix inside a proper number of columns: say your standard is 80 columns and you've just written 5 lines of comments to fit in that, then, you decide to edit and add something in the middle: there it goes: you have to reindent all the lines below so that the block is nice again.

Well, PyDev has an utility for that: put your cursor on that block and do Ctrl+2, W (i.e.: press Ctrl+2 and type the char W later on). Doing that will fix your comment block to fit in the number of columns available (note that it works for code in docstrings and comments alike).

To give an example, say you have the code below:


Just place the cursor in the block, use Ctrl+2, W and voila, you get:


Hope PyDev users enjoy the trick (aligning those manually is definitely on the annoying group).

As a note, to configure the number of columns to be wrapped to go to preferences > general > editors > text editors and change the 'print margin column'.

To finish, Ctrl+2 has other interesting things (if you just press Ctrl+2 and wait a bit, a popup with the options will appear).

April 23, 2015 11:00 AM


Ian Ozsvald

A review of ModelInsight’s growth this last year

Early last year Chris and I founded ModelInsight, a boutique Python-focused Data Science agency in London. We’ve grown well, I figure some reflection is in order. In addition the Data Science scene has grown very well in London, I’ll put some notes on that down below too.

Through consulting, training, workshops and coaching we’ve had the pleasure of working with the likes of King.com, Intel, YouGov and ElevateDirect. Each project aimed to help our client identify and use their data more effectively to generate more business. Projects have included machine learning, natural language processing, prediction, data extraction for both prototyping and deploying live services.

I’ve particularly enjoyed the training and coaching. We’ve run courses introducing data science with Python, covering stats and scikit-learn and high performance Python (based on my book), if you want to be notified of future courses then please join our training announce list.

With the coaching I’ve had the pleasure of working with two data scientists who needed to deploy reliably-working classifiers faster, to automate several human-driven processes for scale. I’ve really enjoyed the challenges they’re posing. If your team could do with some coaching (on-site or off-site) then get in touch, we have room for one more coaching engagement.

I’ve also launched my first data-cleaning service at Annotate.io, it aims to save you time during the early data-cleaning part of a new project. I’d value your feedback and you can join an announce list if you’d like to follow the new services we have planned that’ll make data-cleaning easier.

All the above occurs because the Data Science scene here in London has grown tremendously in the last couple of years. I co-organise the PyDataLondon meetup (over 1,400 members in a year!), here’s a chart showing our month-on-month growth. At Christmas it turned up a notch and it just keeps growing:

pydatalondon_membership_growth

Each month we have 150-200 people in the room for strong Data Science talks, in a couple of months we’ll have our second conference with 300 people at Bloomberg (CfP announce list). We’re actively seeking speakers – join that list if you’d like to know when the CfP opens.

I’ve been privileged to speak as the opening keynoter on The Real Unsolved Problems in Data Science last year at PyConIreland, I’ve just spoken on data cleaning at PyDataParis and soon I’ll keynote on Data Science Deployed at PyConSE. I’m deeply grateful to the community for letting me share my experience. My goal is to help more companies utilise their data to improve their business, if you’ve got ideas on how we could help then I’d love to hear from you!


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

April 23, 2015 09:18 AM


Damyan Bogoev

Flask Series: Application Configuration

Each application needs a configuration. Flask allows developers to implement their application configuration using several different approaches: file-based; object-based; environment-variable-based; instance-folders-based; Regardless of what approach developer chooses for her/ his Flask application configuration, all configuration values are loaded on the config attribute(configuration dictionary) of the Flask application object: app = Flask(__name__) app.config['TESTING'] There is a … Continue reading Flask Series: Application Configuration

April 23, 2015 08:29 AM


Talk Python to Me

Enterprise Python and Large-Scale Projects

Mahmoud is lead developer of the Python Infrastructure team at eBay/PayPal and he has some amazing facts and studies to discuss about the truths and myths using Python for <em>real</em> projects. We discuss how eBay is using Python internally for many large-scale uses. Then we move on to discuss the 10 myths of enterprise Python, such as Python is not compiled, Python is weakly-typed, Python does not scale, and more. <more /> Links from the show: <div style="font-size: .85em;"> <b>10 Myths of Enterprise Python Article</b>: <a href='https://www.paypal-engineering.com/2014/12/10/10-myths-of-enterprise-python/' target='_blank'>https://www.paypal-engineering.com/2014/12/10/10-myths-of-enterprise-python/</a> <b>Mahmoud on Github</b>: <a href='https://github.com/mahmoud' target='_blank'>https://github.com/mahmoud</a> <b>Listen to Wikipedia</b>: <a href='http://listen.hatnote.com/' target='_blank'>http://listen.hatnote.com/</a> <b>the WEEKLYPEDIA</b>: <a href='http://weekly.hatnote.com/ ' target='_blank'>http://weekly.hatnote.com/ </a> <b>Boltons Github Project</b>: <a href='https://github.com/mahmoud/boltons' target='_blank'>https://github.com/mahmoud/boltons</a> </div>

April 23, 2015 08:00 AM