skip to navigation
skip to content

Planet Python

Last update: November 01, 2014 04:45 AM

October 31, 2014


Flavio Percoco

Non-opinionated software can't exist

Here's a thing. I don't believe there's such a thing like "non opinionated" software and I think we should all be more careful when we communicate what the goals of our projects are. The later may not be new to you, probably not even the former but yet, I keep hearing the former everywhere and I keep seeing the later being ignored.

Before I get into why I think non-opinionated software doesn't exist, I'd like to define some of the things that I'll argue about in this post. Let's start with opinions.

What's an opinion?

I'll start by emphasizing that opinions are not facts, therefore they do not represent the absolute truth and they are not verifiable. An opinion that can be verified becomes a fact, which means it's always been a fact and it was not an opinion in the first place. In fact, opinions are considered to be subjective and this prevents them from being absolute. Nonetheless, opinions can be supported by facts.

Opinions have been studied and argued about for a long time. Plato's analogy of the Divided Lines explains the difference between knowledge and belief but even before writing The Republic, Plato and other philosophers had already argued about opinions. Protagoras, for example, claimed that all men's opinions are true. The meaning of this claim and the contradiction that lies within itself were thoroughly discussed in Plato's Theaetetus dialogue. The high-level result of this dialogue is that opinions don't hold truth. I really encourage you to read the dialogue, I consider it to be enlightening.

Another common mistake with regards to opinions is that people commonly claim opinions are relative without claiming what their opinion is relative in terms of. Opinions ought to be subjective, they express something that is relative to the person providing such opinion in the context they are expressed. This subtle distinction is as important as understanding that opinions don't hold truth. The context an opinion is expressed in could affect the opinion itself.

One last thing about opinions, perhaps not so relevant for the content of this post, is that opinions have a weight. That is, depending on the source of the opinion an opinion may be more relevant than the ones coming from other less reliable sources. There are many things that can be argued about this last note. For example, if opinions don't hold truth and they are subjective, why should some opinions have more value than others? My personal opinion is that it all depends on the context where the opinion was provided. I'd probably give more value to a distributed system experts' opinion about my distributed software than I'd give to a web designer's.

What's a non-opinionated software?

Now that we've gone through some of the aspects related to opinions, lets get into what opinions mean when they're applied to software.

A quick google search returned this StackOverflow link where this same exact question was asked. Among the answers provided there, this is the one I think makes more sense:

Non-opinionated software, on the other hand, leaves lots of flexibility to the user (developer). It doesn't proscribe one method of solving a problem, but provides flexible tools that can be used to solve the problem in many ways. The downside of this can be that because the tools are so flexible, it may be relatively hard to develop any solution. Much more of the solution may have to be hand-coded by the user (developer) because the framework doesn't provide enough help. You also have to think much more about how to provide a solution and mediocre developers may end up with poorer solutions than if they had bought into some opinionated software. PERL is probably the classic example of non-opinionated software.

partial quote

Before I get into more details, I'd like to say that I'm not criticizing the answer provided on StackOverflow as such but the general misuse of the term "opinionated" in software development. That is to say that it is not, by any means, my intention to finger-point anyone.

I'm going to break the above down into several separate claims:

Non-opinionated software, on the other hand, leaves lots of flexibility to the user (developer). It doesn't proscribe one method of solving a problem, but provides flexible tools that can be used to solve the problem in many ways.

The fun thing about "non-opinionated software" is that it never claims where the opinion is not being provided by claiming it does have an opinion on something. As expressed in the StackOverflow answer, the non-opinionated software provides the necessary tools to fix a problem by leaving enough flexibility to the consumer to decide what the best thing to do is. There are 3 important things here:

  1. The software is meant to solve a specific problem, therefore it has an opinion on what the final goal is, what the problem it aims to solve is, etc.

  2. The software provides the tools to solve such problem, therefore the software has a very specific opinion about what the right tools to solve such problem are.

  3. The software leaves lots of flexibility to the developer, therefore it is of the opinion the developer knows best how to use the tools provided by itself. This could also be interpreted as the software doesn't have an opinion on how the tools should be used, therefore it leaves it up to the user.

The above describes a pretty opinionated software with regards to a specific problem it aims to solve. It tells the user what problems it is meant to solve, what tools should be used and it claims the user should know best as of how these tools should be used.

Later on in his answer, tvanfosson, describes one of the downsides of non-opinionated software:

The downside of this can be that because the tools are so flexible, it may be relatively hard to develop any solution. Much more of the solution may have to be hand-coded by the user (developer) because the framework doesn't provide enough help. You also have to think much more about how to provide a solution and mediocre developers may end up with poorer solutions than if they had bought into some opinionated software

The difficulties described above are not an effect caused by the hypothetical lack of opinion but an excessively flexible abstraction that lacks of pragmatism. The absence of opinion doesn't make an implementation any more flexible. On the contrary, it is the author's opinion itself of keeping the abstraction flexible that generates such complexity. Every software reflects its authors' opinions.

While I'm aware that the above was excessively nitpicky, I still hope to have made a point with regards to the misconception about what "non-opinionated" software is or even better about why non-opinionated software can't be.

What's opinionated software then?

For the sake of consistency, I'm going to refer to the same answer I used in the previous section. In the above section, I partially quoted tvanfosson's answer and kept the part of it that refers to non-opinionated software. I'll now do the same with the part that refers to opinionated software.

Opinionated software means that there is basically one way (the right way) to do things and trying to do it differently will be difficult and frustrating. On the other hand, doing things the right way can make it very easy to develop with the software as the number of decisions that you have to make is reduced and the ability of the software designers to concentrate on making the software work is increased. Opinionated software can be great to use, if done well, if your problem maps onto the solution nicely. It can be a real pain to solve those parts of your problem that don't map onto the tools provided. An example here would be Ruby on Rails.

I don't think there's anything wrong about this part of the answer. However, I'd like to highlight the quite sarcastic mention of right way therein. It's important to understand - I probably can't stress this enough - that opinions don't hold truth. In a software this means that opinionated software does not represent the right way of doing things - and I'm glad he used the symbol there - but one way to do them, which may or may not be a good fit for the user.

Depending on the problems a software wants to solve, the choices made by the author may be the right ones. However, this does not mean the author's opinion is right. What this means is that based on external, proven, facts the choices the author made are the right ones to solve a specific problem. Remember that opinions can be supported by facts but the opinions themselves don't hold any truth.

With all the above said, I think opinionated software exists in the context of its own goals and regardless of what the opinions of the authors are, the software will be proved to be good or bad based on external facts within specific contexts. Opinionated software that follows existing principles and standards that have been proved to be good or bad carries the opinion of its author with regard to those principles and standards. Regardless of whether those principles the software has been based on are good or bad, the author certainly thinks they are valid, hence the software is being based on them. Still, the opinions of the author hold no truth and the facts these opinions are supported by are the ones that will determine the quality of the software.

Non-opinionated software unveiled

It's not my intention to go after authors that claim their software is non-opinionated but as a member of a community that supports such a claim about its own product, I'd like to take a few minutes and unveil how opinionated such product is.

I'm of course talking about OpenStack. For a long time - and I'm guilty of this myself - we claimed to be working on a non-opinionated cloud provider. The truth is that OpenStack is very opinionated in so many different areas and ways. From it's API to the kind of technologies it sits on to of. In OpenStack we don't even consider running it in something that is not Linux - besides the obvious technical limitations of other operating systems. Moreover, all the services supported by OpenStack have strong opinions on what they provide, how they provide it and what the yet-to-be-proved Right Way of doing things is.

To make the analysis more granular, let me go deeper into one of the services that exists within OpenStack. Zaqar, for example, claimed to be a non-opinionated messaging service akin to SQS. (Un)Fortunately, this is intrinsically wrong. A messaging service can't lack of opinion because it has to provide certain guarantees to its users, therefore it has to have an opinion on what those guarantees are. The service claimed to have a lack of opinion on what storage you could bake it with and again, this is wrong. The guarantees made by the API pretty much define what kind of storage you can or should use for this service. The fact that this service allows you to create your own driver doesn't mean the service lacks of opinion. It just means it's flexible enough to allow for a custom implementation on the storage layer. However, it has a strong opinion on how the driver should be implemented, how it should behave, etc. These opinions could make the implementation of such custom driver difficult or even impossible.

The same high-level analysis can be done on every single piece of OpenStack and I don't think this is wrong. I actually think that software that claims to lack of opinion is bad. If the author of such software does not have an opinion on what the best way to reach the goal is, then I think the result of his work has very few things that can be trusted. Note that I'm not suggesting that software shouldn't be flexible, what I'm stressing here is that flexibility should be based on opinions that are supported by facts. These opinions ought to be pragmatic and simple - I encourage you to watch this talk from Rich Hickey about simplicity - to allow for specific problems to be solved.

In case it wasn't clear, the point I wanted to make with this post is that non-opinionated software can't exists because from the moment a developer chooses a problem to solve and tries to solve it in a certain way, the developer's opinion will be reflected in the software, therefore the software will have an opinion on the way things should be done. Writing software is as important as knowing how to talk about it and it's our responsibility as authors of software to express precisely what the software is about, the opinions reflected there and what we think the best way to solve a problem is. If we fail to communicate this, we'll be simply fooling ourselves and even worse, we'll be trying to fool others.

P.S: I purposely avoided talking about strong or weak opinions. I think it goes without saying that there has to be a balance between them both and that we should all keep an opened mind all times.

October 31, 2014 10:50 PM


Jorgen Schäfer

Elpy 1.6.0 released

I just released version 1.6.0 of Elpy, the Emacs Python Development Environment. This is a feature release.

Elpy is an Emacs package to bring powerful Python editing to Emacs. It combines a number of other packages, both written in Emacs Lisp as well as Python.

Quick Installation

Evaluate this:

(require 'package)
(add-to-list 'package-archives
'("elpy" .
"http://jorgenschaefer.github.io/packages/"))

Then run M-x package-install RET elpy RET.

Finally, run the following (and add them to your .emacs):

(package-initialize)
(elpy-enable)

Changes in 1.6.0

October 31, 2014 05:56 PM


Europython

EuroPython 2015 Call for Participation: On-site Teams

The EuroPython Society (EPS) is happy to announce the Call for Participation (CFP) for EuroPython 2015. The purpose of this call is to select teams willing to help organize the EuroPython conference on-site at a suitable location.

Introduction

This Call for Participation is meant to collect proposals from teams wishing to help run EuroPython 2015 in a location they are local to. As on-site team, you will be integrated into the new workgroups organizational structure the EPS is currently putting in place to run future EuroPython conferences:

EuroPython Workgroups: Call for Volunteers

The main idea behind this workgroup model is to make sure that tasks which do not need to be done by on-site team members can be implemented by distributed workgroups, that can work remotely and persist from location to location, greatly reducing the loss of institutional knowledge we have seen in recent location switches.

Another major aspect of setting up the EPS workgroups rather than having a structure centered around a local organizer, is to reduce the amount of work and financial risk for the on-site teams. As on-site team you no longer have to enter 5-6 figure Euro contracts for the venue or catering and you don’t have to bother with running a website or arranging the conference program. Please note that we still encourage local team members to participate in the workgroups for these tasks, since this simplifies coordination.

The EPS strives to keep the EuroPython conference quality as high as possible, in all of its components. We expect the on-site team to take their proposals very seriously, knowing that they will need to work hard to make the conference a real success.

Timeline for Proposals

The Call for Participation will run until the following deadline for submissions. Proposals must be submitted until midnight UTC on the deadline day, and must adhere the requirements specified in this document. Please make sure to read the whole document carefully.

  • 2014-11-28 - Deadline for submissions (announcement + 4 weeks)
  • 2014-12-05 - Deadline for EPS to review proposals (1 week)
  • 2014-12-12 - Deadline for amended proposals (1 week)
  • 2014-12-26 - Decision on the next EP host (within 2 weeks)

Proposal Workflow

  1. Send your proposal as PDF to the board list: board@europython.eu. This is a private list, so you can include confidential information such as quotes from venues and caterers.

  2. The board will review the proposals and possibly request amendments directly from the submitters. This is done in private between the submitters and the EPS board.

  3. The final version of each proposal will be released to the public after the amendment deadline (with confidential information removed). The EPS will ask to the Python community to provide comments relating to the proposals and take an active role in to discussions.

  4. The final decision for the selection of the on-site team and location will be announced by the EPS board within two weeks after the deadline for amended proposals.

What is EuroPython

EuroPython is the second largest Python conference in the world, right after PyCon US in North America. These are some statistics from EuroPython 2014, to give you an idea of what the on-site team should be ready to handle:

Note that we are open to accepting proposal which can only host fewer attendees, with 600 attendees being the minimum. These numbers are just to give you an idea of how big the EuroPython event has become over the years and how much potential there is for growth.

Please see the EPS EuroPython page for more details on past EuroPython conferences:

http://www.europython-society.org/europython

How EuroPython is run

The EuroPython conference series brand is owned by the EPS. In the past the EPS granted permission to use the brand to local organizers based on a set of requirements, and the local organizing team then had to run the event in collaboration with the EPS.

Since this model no longer scales and doesn’t encourage the community to take part in the organization process, we have started a new approach based on workgroups as explained in the document linked to in the introduction. The on-site team will integrate with the other EPS workgroups and be responsible for taking care of the tasks related to the conference organization on site.

Unlike in previous years, and to further reduce the burden on the on-site teams, we will only request the on-site teams to sign up for one year, keeping in mind, of course, that the team may want to submit a follow-up proposal for the next year. The EPS will take such prior knowledge into account when deciding on the proposals.

On-site Team Requirements

These are the requirements the on-site teams signs up to when submitting a proposal. Changes to these requirements are possible, but must be signed off by the EPS board before they can be put in place.

  1. The conference will be financially and legally run by the EPS, so the on-site team does not have to be a legal entity or enter into high-risk contracts.

  2. The on-site team should be geographically located in a specific country within Europe.

  3. The on-site team must be willing to actively coordinate with the EPS board and the other workgroups, so that all parts of the EuroPython ecosystem can work together in a productive way.

  4. The on-site team must be composed of at least 5 active people. We feel that 5 is the bare minimum for the team to successfully handle the amount of work. Please keep in mind that the team is required to grow significantly during the conference days and it’s considered an advantage, if the on-site team can show that they already have a good number of volunteers to count on during the conference days.

  5. The on-site team must provide at least 2 available venue options, together with catering options for the venues. Locations must provide room for hosting at least 600 attendees, but please keep in mind that demand for EuroPython is more in the range of 1000+ attendees.

  6. The conference must provide the following services to all attendees. Proposals will have to provide details about how these can be implemented at the proposed venues.

    1. Food and drinks for lunches and breaks on all conference days and the sprints

    2. Optionally, breakfast on all conference days (not necessarily on the sprint days)

    3. WLAN service to access the Internet during the conference and the sprints

    4. At least one social event, which can be a dinner, a show or some other form of entertainment, where food and drinks are served. The main social event should ideally be available to all attendees, but may also be limited to a lower number of people, if the selected venue cannot serve as many participants.

    5. A partner program

    6. Audio/video facilities to support speakers and talk recordings.

  7. The conference must take place within the following timeframe: May 1st - October 31th. The venues have to be available for one week plus a setup day during this timeframe. It is possible to propose more than just one possible conference date, since the costs may vary across this timeframe.

  8. The on-site team will work as on-site workgroup in the context of the EPS (see below for details) and has to follow the same rules as all other workgroups in the EPS. Members of the on-site team should also participate in other workgroups to simplify coordination, e.g. there should be on-site team members in the sponsors workgroup to help the sponsors with booth setups, shipment of goods, customs, etc.

  9. The on-site team will coordinate with the EPS board and workgroups to implement the chosen EuroPython structure. The current structure is: conference days from Monday to Sunday, in which 5 days are used for parallels talks and trainings, and 2 weekend days for sprints.

  10. The on-site team must provide a local expenses budget plan as part of the proposal. A example budget plan can be provided on request. The budget figures will then be used as basis for the conference budget maintained by the EPS and its workgroups, so care has to be taken to provide sound numbers in the budget plan.

  11. The on-site team will receive a budget for the work on site, e.g. to pay for logistics, printing, local service companies, etc. It must work together with the EPS financial workgroup to keep it updated on any spendings and changes in a timely manner.  Budget changes must be approved by the EPS board.

  12. The on-site team must be able to provide supporting letters for visa applicants wanting to attend the conference.

Proposal Structure

The proposal must cover all of the following points:

  1. A proposal introduction which describes the motivation for the proposal, underlines the willingness to commit to the needed work and acknowledges the above requirements.

  2. A description of the on-site team, its members and history in the context of their local Python community.

  3. Short biography of the key members and their previous experience in conference and events organizations.

  4. Previous conference history of the on-site team (if any). Describe which conferences were run and provide some figures on their size (registrations, tracks, total income)

  5. Assignment of key positions to on-site team members. Positions to be assigned (one member can be responsible for more than one of the following roles):

    1. Chairperson of the on-site team workgroup: usually also the conference chair and  representative of conference for that year

    2. Sponsor manager: on site contact for all sponsor activities, responsible for local sponsors

    3. Venue manager: handling all contacts with the venue and caterer

    4. Logistics manager: handling all aspects of local logistics and customs

    5. Conference desk manager: responsible for the on site help desk, on site registrations, tickets, cash refunds, attendee support, etc.

  6. List of all team members that will be present during the conference itself as volunteers (but not necessarily help before the conference begins), and/or sources from which  people can be or will be acquired if needed (eg: universities, other local associations, etc.). This list is indicative, as most of the volunteers helping during the conference are usually enrolled during the last months/weeks before the conference, usually started of through a call for volunteers.

  7. List of potential local sponsors. The EPS manages contacts with international sponsors, but local sponsors also help a lot in funding the conference. These are often smaller companies which will not move to new locations with the conference, so a local contact is highly beneficial in attracting these sponsors.

  8. Proposed dates for EuroPython 2015.

  9. Conference Events. Describe which kind of (social) events you can propose for the conference and are able to provide support for.

  10. Expected differences from previous EuroPython conferences (2013 and 2014). Please highlight what things would need to be changed in the way the conference is run on site. Examples of things we would consider worthwhile to be mentioned:

    1. “We can get additional support from the local government / city council”

    2. “We need to provide tickets to city locals at reduced prices”

    3. “We intend to serve meals in a different venue”

    4. “We have some extra space available for posters / sponsors”

    5. “We want to have our local Python conference run as satellite event”

  1. Venue proposal. Describe the following subitems:

    1. Location, reachability by car, train, plane

    2. Venue floor plans, catering floor plans (if different from venue)

    3. Total capacity for talk rooms, training rooms, plenary sessions, booth space and catering/restaurants; if possible, including room plans

    4. A/V equipment

    5. Video recording, availability of on-site support for A/V recordings, possibly even including live- or post-editing

    6. Lightning equipment, availability of on-site support for lighting

    7. Upstream internet connection (at least 100Mbit/s up and downstream)

    8. WLAN structure, service provider, their experience with similar events (WLAN has to be able to handle more than 1000 devices without problems)

    9. Available space for conference desk, sponsor booths, posters, social event, etc., planned locations of these on the floor plans

  2. Accomodation. Describe the following subitems:

    1. Available hotels/hostels near the conference venues, their distance from the conference venue, city center, airport and train station

    2. Rates for the hotels/hostels and, optionally, special rates which can be made available to attendees

    3. The above for three different categories: high end, average and low end

    4. Optionally, a “main” hotel that serves as default choice for conference attendees and serves as social hub

  3. City tourist information. Describe the following subitems:

    1. General information: touristic information, restaurants, mobility

    2. Travel information: how to get to the city and to the conference venue. This information should indicate how easily the conference city and venue are accessible from across Europe and from overseas. It should also answer questions like: Are there convenient ways to get to the conference venue from airports/trains and bus stations and without having a car/taxi?

    3. Travel costs from the previous mentioned travel hubs to the conference venue

  4. Visa: if attendees may require a visa to enter the country/city, please provide detailed information.

  5. Proposed local budget.

    1. The budget should include all expected costs for the on-site support, including the venue costs, catering, on-site rental of equipment, costs for venue services and logistics, extras like public transport passes, museum/tourist passes, etc. Optional budget positions should be indicated as such, to give more flexibility in comparing proposals.

    2. The budget must handle different scenarios related to different venues, including fixed rooms costs, catering, services, etc. Please provide cost estimates for 600, 800, 1000 and 1200 attendees (if possible with the venue).

    3. Please write to the EPS board if you need help with setting up a budget plan.

  6. Taxes/Legal. If there are special tax/legal requirements in the on-site team’s country that require the EPS to register with the local government or pay attention to special regulations, please describe the necessary steps and provide a local accounting/legal contact who can help us work out the details.

Proposal Format

Some things to consider when sending the proposal document.

This Call for Participation (CFP) is also available as PDF file.

Thank you

EuroPython Society

October 31, 2014 03:04 PM


EuroPython Society

EuroPython 2015 Call for Participation: On-site Teams

The EuroPython Society (EPS) is happy to announce the Call for Participation (CFP) for EuroPython 2015. The purpose of this call is to select teams willing to help organize the EuroPython conference on-site at a suitable location.

Introduction

This Call for Participation is meant to collect proposals from teams wishing to help run EuroPython 2015 in a location they are local to. As on-site team, you will be integrated into the new workgroups organizational structure the EPS is currently putting in place to run future EuroPython conferences:

EuroPython Workgroups: Call for Volunteers

The main idea behind this workgroup model is to make sure that tasks which do not need to be done by on-site team members can be implemented by distributed workgroups, that can work remotely and persist from location to location, greatly reducing the loss of institutional knowledge we have seen in recent location switches.

Another major aspect of setting up the EPS workgroups rather than having a structure centered around a local organizer, is to reduce the amount of work and financial risk for the on-site teams. As on-site team you no longer have to enter 5-6 figure Euro contracts for the venue or catering and you don’t have to bother with running a website or arranging the conference program. Please note that we still encourage local team members to participate in the workgroups for these tasks, since this simplifies coordination.

The EPS strives to keep the EuroPython conference quality as high as possible, in all of its components. We expect the on-site team to take their proposals very seriously, knowing that they will need to work hard to make the conference a real success.

Timeline for Proposals

The Call for Participation will run until the following deadline for submissions. Proposals must be submitted until midnight UTC on the deadline day, and must adhere the requirements specified in this document. Please make sure to read the whole document carefully.

  • 2014-11-28 - Deadline for submissions (announcement + 4 weeks)
  • 2014-12-05 - Deadline for EPS to review proposals (1 week)
  • 2014-12-12 - Deadline for amended proposals (1 week)
  • 2014-12-26 - Decision on the next EP host (within 2 weeks)

Proposal Workflow

  1. Send your proposal as PDF to the board list: board@europython.eu. This is a private list, so you can include confidential information such as quotes from venues and caterers.

  2. The board will review the proposals and possibly request amendments directly from the submitters. This is done in private between the submitters and the EPS board.

  3. The final version of each proposal will be released to the public after the amendment deadline (with confidential information removed). The EPS will ask to the Python community to provide comments relating to the proposals and take an active role in to discussions.

  4. The final decision for the selection of the on-site team and location will be announced by the EPS board within two weeks after the deadline for amended proposals.

What is EuroPython

EuroPython is the second largest Python conference in the world, right after PyCon US in North America. These are some statistics from EuroPython 2014, to give you an idea of what the on-site team should be ready to handle:

Note that we are open to accepting proposal which can only host fewer attendees, with 600 attendees being the minimum. These numbers are just to give you an idea of how big the EuroPython event has become over the years and how much potential there is for growth.

Please see the EPS EuroPython page for more details on past EuroPython conferences:

http://www.europython-society.org/europython

How EuroPython is run

The EuroPython conference series brand is owned by the EPS. In the past the EPS granted permission to use the brand to local organizers based on a set of requirements, and the local organizing team then had to run the event in collaboration with the EPS.

Since this model no longer scales and doesn’t encourage the community to take part in the organization process, we have started a new approach based on workgroups as explained in the document linked to in the introduction. The on-site team will integrate with the other EPS workgroups and be responsible for taking care of the tasks related to the conference organization on site.

Unlike in previous years, and to further reduce the burden on the on-site teams, we will only request the on-site teams to sign up for one year, keeping in mind, of course, that the team may want to submit a follow-up proposal for the next year. The EPS will take such prior knowledge into account when deciding on the proposals.

On-site Team Requirements

These are the requirements the on-site teams signs up to when submitting a proposal. Changes to these requirements are possible, but must be signed off by the EPS board before they can be put in place.

  1. The conference will be financially and legally run by the EPS, so the on-site team does not have to be a legal entity or enter into high-risk contracts.

  2. The on-site team should be geographically located in a specific country within Europe.

  3. The on-site team must be willing to actively coordinate with the EPS board and the other workgroups, so that all parts of the EuroPython ecosystem can work together in a productive way.

  4. The on-site team must be composed of at least 5 active people. We feel that 5 is the bare minimum for the team to successfully handle the amount of work. Please keep in mind that the team is required to grow significantly during the conference days and it’s considered an advantage, if the on-site team can show that they already have a good number of volunteers to count on during the conference days.

  5. The on-site team must provide at least 2 available venue options, together with catering options for the venues. Locations must provide room for hosting at least 600 attendees, but please keep in mind that demand for EuroPython is more in the range of 1000+ attendees.

  6. The conference must provide the following services to all attendees. Proposals will have to provide details about how these can be implemented at the proposed venues.

    1. Food and drinks for lunches and breaks on all conference days and the sprints

    2. Optionally, breakfast on all conference days (not necessarily on the sprint days)

    3. WLAN service to access the Internet during the conference and the sprints

    4. At least one social event, which can be a dinner, a show or some other form of entertainment, where food and drinks are served. The main social event should ideally be available to all attendees, but may also be limited to a lower number of people, if the selected venue cannot serve as many participants.

    5. A partner program

    6. Audio/video facilities to support speakers and talk recordings.

  7. The conference must take place within the following timeframe: May 1st - October 31th. The venues have to be available for one week plus a setup day during this timeframe. It is possible to propose more than just one possible conference date, since the costs may vary across this timeframe.

  8. The on-site team will work as on-site workgroup in the context of the EPS (see below for details) and has to follow the same rules as all other workgroups in the EPS. Members of the on-site team should also participate in other workgroups to simplify coordination, e.g. there should be on-site team members in the sponsors workgroup to help the sponsors with booth setups, shipment of goods, customs, etc.

  9. The on-site team will coordinate with the EPS board and workgroups to implement the chosen EuroPython structure. The current structure is: conference days from Monday to Sunday, in which 5 days are used for parallels talks and trainings, and 2 weekend days for sprints.

  10. The on-site team must provide a local expenses budget plan as part of the proposal. A example budget plan can be provided on request. The budget figures will then be used as basis for the conference budget maintained by the EPS and its workgroups, so care has to be taken to provide sound numbers in the budget plan.

  11. The on-site team will receive a budget for the work on site, e.g. to pay for logistics, printing, local service companies, etc. It must work together with the EPS financial workgroup to keep it updated on any spendings and changes in a timely manner.  Budget changes must be approved by the EPS board.

  12. The on-site team must be able to provide supporting letters for visa applicants wanting to attend the conference.

Proposal Structure

The proposal must cover all of the following points:

  1. A proposal introduction which describes the motivation for the proposal, underlines the willingness to commit to the needed work and acknowledges the above requirements.

  2. A description of the on-site team, its members and history in the context of their local Python community.

  3. Short biography of the key members and their previous experience in conference and events organizations.

  4. Previous conference history of the on-site team (if any). Describe which conferences were run and provide some figures on their size (registrations, tracks, total income)

  5. Assignment of key positions to on-site team members. Positions to be assigned (one member can be responsible for more than one of the following roles):

    1. Chairperson of the on-site team workgroup: usually also the conference chair and  representative of conference for that year

    2. Sponsor manager: on site contact for all sponsor activities, responsible for local sponsors

    3. Venue manager: handling all contacts with the venue and caterer

    4. Logistics manager: handling all aspects of local logistics and customs

    5. Conference desk manager: responsible for the on site help desk, on site registrations, tickets, cash refunds, attendee support, etc.

  6. List of all team members that will be present during the conference itself as volunteers (but not necessarily help before the conference begins), and/or sources from which  people can be or will be acquired if needed (eg: universities, other local associations, etc.). This list is indicative, as most of the volunteers helping during the conference are usually enrolled during the last months/weeks before the conference, usually started of through a call for volunteers.

  7. List of potential local sponsors. The EPS manages contacts with international sponsors, but local sponsors also help a lot in funding the conference. These are often smaller companies which will not move to new locations with the conference, so a local contact is highly beneficial in attracting these sponsors.

  8. Proposed dates for EuroPython 2015.

  9. Conference Events. Describe which kind of (social) events you can propose for the conference and are able to provide support for.

  10. Expected differences from previous EuroPython conferences (2013 and 2014). Please highlight what things would need to be changed in the way the conference is run on site. Examples of things we would consider worthwhile to be mentioned:

    1. “We can get additional support from the local government / city council”

    2. “We need to provide tickets to city locals at reduced prices”

    3. “We intend to serve meals in a different venue”

    4. “We have some extra space available for posters / sponsors”

    5. “We want to have our local Python conference run as satellite event”

  1. Venue proposal. Describe the following subitems:

    1. Location, reachability by car, train, plane

    2. Venue floor plans, catering floor plans (if different from venue)

    3. Total capacity for talk rooms, training rooms, plenary sessions, booth space and catering/restaurants; if possible, including room plans

    4. A/V equipment

    5. Video recording, availability of on-site support for A/V recordings, possibly even including live- or post-editing

    6. Lightning equipment, availability of on-site support for lighting

    7. Upstream internet connection (at least 100Mbit/s up and downstream)

    8. WLAN structure, service provider, their experience with similar events (WLAN has to be able to handle more than 1000 devices without problems)

    9. Available space for conference desk, sponsor booths, posters, social event, etc., planned locations of these on the floor plans

  2. Accomodation. Describe the following subitems:

    1. Available hotels/hostels near the conference venues, their distance from the conference venue, city center, airport and train station

    2. Rates for the hotels/hostels and, optionally, special rates which can be made available to attendees

    3. The above for three different categories: high end, average and low end

    4. Optionally, a “main” hotel that serves as default choice for conference attendees and serves as social hub

  3. City tourist information. Describe the following subitems:

    1. General information: touristic information, restaurants, mobility

    2. Travel information: how to get to the city and to the conference venue. This information should indicate how easily the conference city and venue are accessible from across Europe and from overseas. It should also answer questions like: Are there convenient ways to get to the conference venue from airports/trains and bus stations and without having a car/taxi?

    3. Travel costs from the previous mentioned travel hubs to the conference venue

  4. Visa: if attendees may require a visa to enter the country/city, please provide detailed information.

  5. Proposed local budget.

    1. The budget should include all expected costs for the on-site support, including the venue costs, catering, on-site rental of equipment, costs for venue services and logistics, extras like public transport passes, museum/tourist passes, etc. Optional budget positions should be indicated as such, to give more flexibility in comparing proposals.

    2. The budget must handle different scenarios related to different venues, including fixed rooms costs, catering, services, etc. Please provide cost estimates for 600, 800, 1000 and 1200 attendees (if possible with the venue).

    3. Please write to the EPS board if you need help with setting up a budget plan.

  6. Taxes/Legal. If there are special tax/legal requirements in the on-site team’s country that require the EPS to register with the local government or pay attention to special regulations, please describe the necessary steps and provide a local accounting/legal contact who can help us work out the details.

Proposal Format

Some things to consider when sending the proposal document.

This Call for Participation (CFP) is also available as PDF file.

Thank you

EuroPython Society

October 31, 2014 03:02 PM


Python Piedmont Triad User Group

November Project Night

Winston Salem project night


This event is also on our meetup page:
http://www.meetup.com/PYthon-Piedmont-Triad-User-Group-PYPTUG/events/213803772/


  As always, it will be the 3rd Wednesday of the month. That's the 19th of November at 6pm.


When

 

3rd Wednesday of the month, 6pm
November 19 2014


Where

Inmar

635 Vine St,
Room 1130H "Dash"
Winston-Salem, NC

This will be at the Inmar building in downtown Winston Salem.­

What

Do you have a project you want to show off? Or do you need a second set of eyes on your code? Would you like to hack on an open source project? Perhaps you need help getting a module on a specific platform? Don't have a project but would like to help others or to learn more Python?

Don Jennings will also continue our feature of our monthly project nights:
 The Newbie Corner

October 31, 2014 02:09 PM


Carl Trachte

Gtk.TreeView (grid view) with mono, gtk-sharp, and IronPython

The post immediately prior to this one was an attempt to reproduce Windows.Forms Calendar controls in Gtk for cross platform (Windows/*nix) effective rendering.

This time I am attempting to get familiar with gtk-sharp/Gtk's version of a grid view - the Gtk.TreeView object.  Some of the gtk-sharp documentation suggests the NodeView object would be easier to use.  I had some trouble instantiating the objects associated with the NodeView and went with the TreeView instead in the hopes of getting more control.

The Windows.Forms GridView I did years ago is here.  It became apparent to me shortly after embarking on this journey that I would be hard pressed to recreate all the functionality of that script in a timely manner.  I settled for a tabular view of drillhole data (fabricated, mock data) with some custom formatting.

Aside:  this is typically how mineral exploration drillhole data (core, reverse circulation drilling) is presented in tabular format - a series of from-to intervals with assay values.  Assuming the assays are all separate elements, the reported weight percents should not sum more than 100%, and never do unless someone fat fingers a decimal place.  I've projected a couple screaming hot polymetallic drill holes that end near surface (lack of funding for drilling), but show enough promise that the new mining town of Trachteville (the drill hole name CBT-BNZA stands for CBT-Bonanza) will spring up there at any moment . . . one can dream.

The data store object for the grid view Gtk.ListStore object would not instantiate in IronPython.  I was not the only person to have experienced this problem (I cannot locate the link to the mailing list thread or forum reference, but like the big fish that got away, I swear I saw it).  I didn't want to drop the effort just because of that, so I hacked and compiled some C# code:

public class storex
{
    public Gtk.ListStore drillhole =
                            // 7 columns
                            // drillhole id
          new Gtk.ListStore (typeof (string),
                            // from
                            typeof (double),
                            // to
                            typeof (double),
                            // assay1
                            typeof (double),
                            // assay2
                            typeof (double),
                            // assay3
                            typeof (double),
                            // assay4
                            typeof (double));
}


The mono command on Windows was

C:\UserPrograms\Mono-3.2.3>mcs -pkg:gtk-sharp-2.0 /target:library C:\UserPrograms\IronPythonGUI\storex.cs

Those are my file paths; locations depend on where you install things like mono and IronPython.

Anyway, I got my dll and I was off to the races.  Getting to know the Gtk and gtk-sharp object model proved challenging for me.  I'm glad I got some familiarity with it, but it would take me longer to do something in Gtk than it did with Windows.Forms.  The most fun and gratifying part of the project was getting the custom formatting to work with a Gtk.TreeCellDataFunc.  I used a function that yielded specific functions for each column - something that's really easy to do in Python.

Anyway, here are a couple screenshots and the IronPython code:





The OpenBSD one below turned out pretty good, but the Windows one had a little double line underneath the first row - it looked as though it was still trying to select that row when I told it specifically not to.  I'm not a design perfectionist Steve Jobs type, but niggling nits like that drive me batty.  For now, though it's best I publish the code and move on.

#!/usr/local/bin/mono /home/carl/IronPython-2.7.4/ipy64.exe

import clr

GTKSHARP = 'gtk-sharp'
PANGO = 'pango-sharp'

# Mock store C#
STOREX = 'storex'

clr.AddReference(GTKSHARP)
clr.AddReference(PANGO)

# C# module compiled for this project.
# Problems with Gtk.ListStore in IronPython.
clr.AddReference(STOREX)

import Gtk
import Pango

import storex

TITLE = 'Gtk.TreeView Demo (Drillholes)'
MARKUP = '<span font="Courier New" size="14" weight="bold">{:s}</span>'
MARKEDUPTITLE = MARKUP.format(TITLE)

CENTERED = 0.5
RIGHT = 1.0

WINDOWWIDTH = 350

COURFONTREGULAR = 'Courier New 12'
COURFONTBOLD = 'Courier New Bold 12'

DHNAME = 'DH_CBTBNZA-{:>02d}'
DHNAMELABEL = 'drillhole'
FROM = 'from'
TO = 'to'
ASSAY1 = 'assay1'
ASSAY2 = 'assay2'
ASSAY3 = 'assay3'
ASSAY4 = 'assay4'

FP1FMT = '{:>5.1f}'
FP2FMT = '{:>4.2f}'

DHDATAX = {(DHNAME.format(1), 0.0):{TO:8.7,
                                    ASSAY1:22.27,
                                    ASSAY2:4.93,
                                    ASSAY3:18.75,
                                    ASSAY4:35.18},
           (DHNAME.format(1), 8.7):{TO:15.3,
                                    ASSAY1:0.27,
                                    ASSAY2:0.09,
                                    ASSAY3:0.03,
                                    ASSAY4:0.22},
           (DHNAME.format(1), 15.3):{TO:25.3,
                                     ASSAY1:2.56,
                                     ASSAY2:11.34,
                                     ASSAY3:0.19,
                                     ASSAY4:13.46},
           (DHNAME.format(2), 0.0):{TO:10.0,
                                    ASSAY1:0.07,
                                    ASSAY2:1.23,
                                    ASSAY3:4.78,
                                    ASSAY4:5.13},
           (DHNAME.format(2), 10.0):{TO:20.0,
                                     ASSAY1:44.88,
                                     ASSAY2:12.97,
                                     ASSAY3:0.19,
                                     ASSAY4:0.03}}

FIELDS = [DHNAMELABEL, FROM, TO, ASSAY1, ASSAY2, ASSAY3, ASSAY4]
BOLDEDCOLUMNS = [DHNAMELABEL, FROM, TO]
NONKEYFIELDS = FIELDS[2:]

BLAZINGCUTOFF = 10.0

def genericfloatformat(floatfmt, index):
    """
    For cell formatting in Gtk.TreeView.

    Returns a function to format floats
    and to format floats' foreground color
    based on cutoff value.

    floatfmt is a format string.

    index is an int that indicates the
    column being formatted.
    """
    def setfloatfmt(treeviewcolumn, cellrenderer, treemodel, treeiter):
        cellrenderer.Text = floatfmt.format(treemodel.GetValue(treeiter, index))
        # If it is one of the assay value columns.
        # XXX - not generic.
        if index > 2:
            if treemodel.GetValue(treeiter, index) > BLAZINGCUTOFF:
                cellrenderer.Foreground = 'red'
            else:
                cellrenderer.Foreground = 'black'
    return Gtk.TreeCellDataFunc(setfloatfmt)

class TreeViewTest(object):
    def __init__(self):
        Gtk.Application.Init()
        self.window = Gtk.Window('')
        # DeleteEvent - copied from Gtk demo on internet.
        self.window.DeleteEvent += self.DeleteEvent
        # Frame property provides a frame and title.
        self.frame = Gtk.Frame(MARKEDUPTITLE)
        self.tree = Gtk.TreeView()
        self.tree.EnableGridLines = Gtk.TreeViewGridLines.Both
        self.frame.Add(self.tree)

        # Fonts for formatting.
        self.fdregular = Pango.FontDescription.FromString(COURFONTREGULAR)
        self.fdbold = Pango.FontDescription.FromString(COURFONTBOLD)

        # C# module
        self.store = storex().drillhole

        self.makecolumns()
        self.adddata()
        self.tree.Model = self.store

        self.formatcolumns()
        self.formatcells()
        self.prettyup()

        self.window.Add(self.frame)
        self.window.ShowAll()
        # Keep text viewable - size no smaller than intended.
        self.window.AllowShrink = False
        # XXX - hack to keep lack of gridlines on edges of
        #       table from showing.
        self.window.AllowGrow = False
        # Unselect everything for this demo.
        self.tree.Selection.UnselectAll()
        Gtk.Application.Run()

    def makecolumns(self):
        """
        Fill in columns for TreeView.
        """
        self.columns = {}
        for fieldx in FIELDS:
            self.columns[fieldx] = Gtk.TreeViewColumn()
            self.columns[fieldx].Title = fieldx
            self.tree.AppendColumn(self.columns[fieldx])

    def formatcolumns(self):
        """
        Make custom labels for columnn headers.

        Get each column properly justified (all
        are right justified,floating point numbers
        except for the drillhole 'number' -
        actually a string).
        """
        self.customlabels = {}

        for fieldx in FIELDS:
            # This centers the labels at the top.
            self.columns[fieldx].Alignment = CENTERED
            self.customlabels[fieldx] = Gtk.Label(self.columns[fieldx].Title)
            self.customlabels[fieldx].ModifyFont(self.fdbold)
            # 120 is about right for from, to, and assay columns.
            self.columns[fieldx].MinWidth = 120
            self.customlabels[fieldx].ShowAll()
            self.columns[fieldx].Widget = self.customlabels[fieldx]
            # ShowAll required for new label to take.
            self.columns[fieldx].Widget.ShowAll()

    def formatcells(self):
        """
        Add and format cell renderers.
        """
        self.cellrenderers = {}

        for fieldx in FIELDS:
            self.cellrenderers[fieldx] = Gtk.CellRendererText()
            self.columns[fieldx].PackStart(self.cellrenderers[fieldx], True)
            # Drillhole 'number' (string)
            if fieldx == FIELDS[0]:
                self.cellrenderers[fieldx].Xalign = CENTERED
                self.columns[fieldx].AddAttribute(self.cellrenderers[fieldx],
                        'text', 0)
            else:
                self.cellrenderers[fieldx].Xalign = RIGHT
                try:
                    self.columns[fieldx].AddAttribute(self.cellrenderers[fieldx],
                            'text', FIELDS.index(fieldx))
                except ValueError:
                    print('\n\nProblem with field definitions; field not found.\n\n')
        for fieldx in BOLDEDCOLUMNS:
            self.cellrenderers[fieldx].Font = COURFONTBOLD
        self.columns[fieldx].Widget.ShowAll()

        # XXX - not very generic, but better than doing them one by one.
        # from, to columns.
        for x in xrange(1, 3):
            self.columns[FIELDS[x]].SetCellDataFunc(self.cellrenderers[FIELDS[x]],
                    genericfloatformat(FP1FMT, x))
        # assay<x> columns.
        for x in xrange(3, 7):
            self.columns[FIELDS[x]].SetCellDataFunc(self.cellrenderers[FIELDS[x]],
                    genericfloatformat(FP2FMT, x))

    def usemarkup(self):
        """
        Refreshes UseMarkup property on widgets (labels)
        so that they display properly and without
        markup text.
        """
        # Have to refresh this property each time.
        self.frame.LabelWidget.UseMarkup = True

    def prettyup(self):
        """
        Get Gtk objects looking the way we
        intended.
        """
        # Try to get Courier New on treeview.
        self.tree.ModifyFont(self.fdregular)
        # Get rid of line.
        self.frame.Shadow = Gtk.ShadowType.None
        self.usemarkup()

    def adddata(self):
        """
        Put data into store.
        """
        # XXX - difficulty figuring out sorting
        #       function for TreeView.  Hack it
        #       with dictionary here.
        keytuples = [key for key in DHDATAX]
        keytuples.sort()
        datax = []
        for tuplex in keytuples:
            # XXX - side effect comprehension.
            #       Not great for readability,
            #       but compact.
            [datax.append(x) for x in tuplex]
            for fieldx in NONKEYFIELDS:
                datax.append(DHDATAX[tuplex][fieldx])
            self.store.AppendValues(*datax)
            # Reinitiialize data row list.
            datax = []

    def DeleteEvent(self, widget, event):
        Gtk.Application.Quit()

if __name__ == '__main__':
    TreeViewTest()


Thanks for stopping by.

October 31, 2014 07:15 AM

Mono gtk-sharp IronPython CalendarView

A number of years ago I did a post on the IronPython Cookbook site about the Windows.Forms Calendar control.  I could never get the thing to render nicely on *nix operating systems (BSD family).  It sounds as though Windows.Forms development for mono (and in general) is kind of dead, so there is not much hope that solution/example will ever render nicely on *nix.  Recently I've been playing with mono and decided to give gtk-sharp a shot with IronPython.

Quick disclaimers:

1) I suspect from the examples I've seen on the internet that PyGtk is a little easier to deal with than gtk-sharp.  That's OK; I wanted to use IronPython and have the rest of the mono/dotNet framework available, so I went through the extra trouble to forego CPython and PyGtk and go with IronPython and gtk-sharp instead.

2) The desktop is not the most cutting edge or sexy platform in 2014.  Nonetheless, where I work it is alive and well.  When I no longer see engineers hacking solutions in Excel and VBA, I'll consider the possibility of outliving the desktop.  Right now I'm not hopeful :-\

The results aren't bad, at least as far as rendering goes.  I couldn't get the Courier font to take on OpenBSD, but the Gtk Calendar control looks acceptable.  All in all, I was OK with the results on both Windows and OpenBSD.  I've heard Gtk doesn't do quite as well on Apple products, but I don't own a Mac to test with.  Here are a couple screenshots:






I run the cwm window manager on OpenBSD and have it set up to cut out borders on windows, hence the more minimalist look to the control there.

IronPython output on *nix has always come out in yellow or white - it doesn't show up on a white background, which I prefer.  In order to get around this, I run an xterm with a black background:

xterm -bg black -fg white

Here is the code for the gtk-sharp Gtk.Calendar control:

#!/usr/local/bin/mono /home/carl/IronPython-2.7.4/ipy64.exe

import clr

GTKSHARP = 'gtk-sharp'
PANGO = 'pango-sharp'

clr.AddReference(GTKSHARP)
clr.AddReference(PANGO)

import Gtk
import Pango

import datetime

TITLE = 'Gtk.Calendar Demo'
MARKUP = '<span font="Courier New" size="14" weight="bold">{:s}</span>'
MARKEDUPTITLE = MARKUP.format(TITLE)

INFOMSG = '<span font="Courier New 12">\n\n Program set to run for:\n\n '
INFOMSG += '{:%Y-%m-%d}\n\n</span>'

DATEDIFFMSG = '<span font="Courier New 12">\n\n '
DATEDIFFMSG += 'There are {0:d} days between the\n'
DATEDIFFMSG += ' beginning of the epoch and\n'
DATEDIFFMSG += ' {1:%Y-%m-%d}.\n\n</span>'

ALIGNMENTPARAMS = (0.0, 0.5, 0.0, 0.0)

WINDOWWIDTH = 350

CALENDARFONT = 'Courier New Bold 12'


class CalendarTest(object):
    inthebeginning = datetime.datetime.fromtimestamp(0)
    # Debug info - make sure beginning of epoch really
    #              is +midnight, Jan 1, 1970 GMT.
    print(inthebeginning)
    def __init__(self):
        Gtk.Application.Init()
        self.window = Gtk.Window(TITLE)
        # DeleteEvent - copied from Gtk demo on internet.
        self.window.DeleteEvent += self.DeleteEvent
        # Frame property provides a frame and title.
        self.frame = Gtk.Frame(MARKEDUPTITLE)
        self.calendar = Gtk.Calendar()
        # Handles date selection event.
        self.calendar.DaySelected += self.dateselect
        # Sets up text for labels.
        self.getcaltext()
        # Puts little box around text.
        self.datelabelframe = Gtk.Frame()
        # Try to get datelabel to align with other label.
        self.datelabelalignment = Gtk.Alignment(*ALIGNMENTPARAMS)
        self.datelabel = Gtk.Label(self.caltext)
        self.datelabelalignment.Add(self.datelabel)
        self.datelabelframe.Add(self.datelabelalignment)
        # Puts little box around text.
        self.datedifflabelframe = Gtk.Frame()
        self.datedifflabelalignment = Gtk.Alignment(*ALIGNMENTPARAMS)
        self.datedifflabel = Gtk.Label(self.timedifftext)
        self.datedifflabelalignment.Add(self.datedifflabel)
        self.datedifflabelframe.Add(self.datedifflabelalignment)
        self.vbox = Gtk.VBox()
        self.vbox.PackStart(self.datelabelframe)
        self.vbox.PackStart(self.datedifflabelframe)
        self.vbox.PackStart(self.calendar)
        self.frame.Add(self.vbox)
        self.window.Add(self.frame)
        self.prettyup()
        self.window.ShowAll()
        # Keep text viewable - size no smaller than intended.
        self.window.AllowShrink = False
        Gtk.Application.Run()

    def getcaltext(self):
        """
        Get messages for run date.
        """
        # Calendar month is 0 based.
        yearmonthday = self.calendar.Year, self.calendar.Month + 1, self.calendar.Day
        chosendate = datetime.datetime(*yearmonthday)
        self.caltext = INFOMSG.format(chosendate)
        # For reporting of number of days since beginning of epoch.
        timediff = chosendate - CalendarTest.inthebeginning
        self.timedifftext = DATEDIFFMSG.format(timediff.days, chosendate)

    def usemarkup(self):
        """
        Refreshes UseMarkup property on widgets (labels)
        so that they display properly and without
        markup text.
        """
        # Have to refresh this property each time.
        self.frame.LabelWidget.UseMarkup = True
        self.datelabel.UseMarkup = True
        self.datedifflabel.UseMarkup = True

    def prettyup(self):
        """
        Get Gtk objects looking the way we
        intended.
        """
        # Try to make frame wider.
        # XXX
        # Works nicely on Windows - try on Unix.
        # Allows bold, etc.
        self.usemarkup()
        self.frame.SetSizeRequest(WINDOWWIDTH, -1)
        # Get rid of line in middle of text on title.
        self.frame.Shadow = Gtk.ShadowType.None
        # Try to get Courier New on calendar.
        fd = Pango.FontDescription.FromString(CALENDARFONT)
        self.calendar.ModifyFont(fd)
        self.datelabel.Justify = Gtk.Justification.Left
        self.datedifflabel.Justify = Gtk.Justification.Left
        self.window.Title = ''
        self.usemarkup()

    def dateselect(self, widget, event):
        self.getcaltext()
        self.datelabel.Text = self.caltext
        self.datedifflabel.Text = self.timedifftext
        self.prettyup()

    def DeleteEvent(self, widget, event):
        Gtk.Application.Quit()

if __name__ == '__main__':
    CalendarTest()


Thanks for stopping by. 

October 31, 2014 06:25 AM


Vasudev Ram

PDF in a Net, with Netius, a pure Python network library

By Vasudev Ram


I came across Netius, a pure Python network library, recently.

Excerpt from the Netius home page:

[ Netius is a Python network library that can be used for the rapid creation of asynchronous non-blocking servers and clients. It has no dependencies, it's cross-platform, and brings some sample netius-powered servers out of the box, namely a production-ready WSGI server. ]

Note: They mention some limitations of the async feature. Check the Netius home page for more on that.

To try out netius a little (not the async features, yet), I modified their example WSGI server program to serve a PDF of some hard-coded text, generated by xtopdf, my PDF creation library / toolkit.

The server, netius_pdf_server.py, running on port 8080, generates and writes to disk, a PDF of some text, and then reads back that PDF from disk, and serves it to the client.

The client, netius_pdf_client.py, uses the requests Python HTTP library to make a request to that server, gets the PDF file in the response, and writes it to disk.

Note: this is proof-of-concept code, without much error handling or refinement. But I did run it and it worked.

Here is the code for the server:
# test_netius_wsgi_server.py 

import time
from PDFWriter import PDFWriter
import netius.servers

def get_pdf():
pw = PDFWriter('hello-from-netius.pdf')
pw.setFont('Courier', 10)
pw.setHeader('PDF generated by xtopdf, a PDF library for Python')
pw.setFooter('Using netius Python network library, at {}'.format(time.ctime()))
pw.writeLine('Hello world! This is a test PDF served by Netius, ')
pw.writeLine('a Python networking library; PDF created with the help ')
pw.writeLine('of xtopdf, a Python library for PDF creation.')
pw.close()
pdf_fil = open('hello-from-netius.pdf', 'rb')
pdf_str = pdf_fil.read()
pdf_len = len(pdf_str)
pdf_fil.close()
return pdf_len, pdf_str

def app(environ, start_response):
status = "200 OK"
content_len, contents = get_pdf()
headers = (
("Content-Length", content_len),
("Content-type", "application/pdf"),
("Connection", "keep-alive")
)
start_response(status, headers)
yield contents

server = netius.servers.WSGIServer(app = app)
server.serve(port = 8080)
In my next post, I'll show the code for the client, and the output.

You may also like to see my earlier posts on similar lines, about generating and serving PDF content using other Python web frameworks:

PDF in a Bottle , PDF in a Flask and PDF in a CherryPy.

The image at the top of this post is of Chinese fishing nets, a tourist attraction found in Kochi (formerly called Cochin), Kerala.

- Enjoy.

- Vasudev Ram - Dancing Bison Enterprises

Signup for email about new products from me.

Contact Page

October 31, 2014 05:11 AM


Montreal Python User Group

Election for MTL:PY this Monday

It's that time of the year again: the organisation team for the Montréal-Python community is going to proceed to the election of its board. People elected to the board are the legal representatives of the association, but they're also people who contribute their energy and ideas to drive the community.

Anyone who is subscribed to the mailing list can apply to one of these positions:

Please note that other positions could be added if needed.

We'd like to stress that it is not necessary to be a member of the board (or even a member of the permanent organisation team) to organise events under the banner of Montréal-Python; the board is the official organ, but it is you, the members of the community, who do great things with Python in Montreal.

When:

Monday, November 3rd, 8pm

Where:

Ajah offices, 1124 Marie-Anne Est https://goo.gl/maps/xN1ck

How:

Show of hands

Comme see us on Monday! :)

October 31, 2014 04:00 AM

October 30, 2014


PyCharm

JetBrains Debuts PyCharm Educational Edition

If you’ve ever wanted to learn Python programming, get ready to be blown away.

Today we’re launching the free and open source PyCharm Educational Edition for students and teachers. This easy-to-use yet powerful Python IDE includes special interactive educational functionality to help novice programmers learn the craft and turn professional quicker than ever before! It combines the easy learning curve of interactive coding platforms with the power of a real-world professional tool.

screenshot1@2x

Why PyCharm Educational Edition?

We all know that computer programming studies are one of today’s major global trends, driven by open-access, non-credit education. Python has long been used for educational purposes and is now the most popular language used to teach programming for beginners. We decided to create this new educational IDE, because we at JetBrains PyCharm, being a part of the Python community, are committed to providing quality, professional, seamless solutions for learning programming with Python, keeping the needs of both novice programmers and educators in mind.

What is so special about PyCharm Educational Edition?

In designing this tool we have been inspired by Guido van Rossum, the creator of the Python programming language, who said:

guido“We believe that there should be no clear-cut distinction between tools used by professionals and tools used for education—just as professional writers use the same language and alphabet as their readers!” https://www.python.org/doc/essays/cp4e/

PyCharm is a professional tool recognized among professionals all around the globe. At some point it occurred to us that, with some work, its power could also be harnessed to serve educational purposes.

We analyzed educational trends and tools on the market carefully. To understand what should be improved in PyCharm and how to make it the best educational IDE possible, we polled hundreds of instructors from different universities all around the world.

1
We found out that there are two opposite approaches to learning programming. One is based on using interactive online educational platforms and editors, which are extremely easy to start with. Despite an easy learning curve, these are not real development tools, and once you get used to them it may be difficult to switch to a real development environment and develop something real. The other approach is centered around real code editors and IDEs tools. While advanced, they are often too complex for beginners. Instead of learning programming, you must invest considerable efforts and time just into understanding how the tool works, before actually learning the essentials of programming.

PyCharm Educational Edition aims to combine both these two worlds. We’ve made it easy to get started with, not intimidating, yet powerful enough to guide you all the way through to becoming a professional developer.

All the learning you need, for FREE

freePyCharm Educational Edition is absolutely free and open-source. Novice programmers can download and use it for educational or any other purposes—for free. Instructors and course authors can use it to create, modify and share their own courses.

Included are learning essentials like an integrated Python console, Debugger and VCS, along with unique educational features such as “fill in the missing code” exercises, intelligent hints, checks, smart suggestions, code auto-completion, and much more.

So, what’s inside, besides the PyCharm Community Edition?

Possible Applications

possiblePyCharm Educational Edition can be used in MOOCs, self-studying courses or traditional programming courses. In addition to going through interactive courses, you can also use normal Python projects and the integrated Python console, as well as the debugger, VCS, and everything else that PyCharm already offers.

What to do next?

Don’t wait any longer — download PyCharm Education Edition for your platform and start learning Python programming today!

For more details and learning materials, visit the PyCharm Educational Edition website and check the Quick Start guide to get rolling. Or, for a quick visual overview, watch this introductory video:

Then, get involved:

Read our blog to stay tuned for news, updates and everything that goes on with PyCharm Educational Edition. And do give us feedback on how we’re doing.

Did you know?
JetBrains recently launched the Free Student License program. With this program any student or educator can use any JetBrains product for free!

Develop with pleasure!
JetBrains PyCharm Team

October 30, 2014 06:34 PM


Machinalis

IEPY 0.9 was released


We're happy to announce that IEPY 0.9 was released!!
It's an open source tool for Information Extraction focused on Relation Extraction.
It’s aimed at:
  • users needing to perform Information Extraction on a large dataset.
  • scientists wanting to experiment with new IE algorithms.

To give an example of Relation Extraction, if we are trying to find a birth date in:

“John von Neumann (December 28, 1903 – February 8, 1957) was a Hungarian and American pure and applied mathematician, physicist, inventor and polymath.”

Then IEPY’s task is to identify “John von Neumann” and “December 28, 1903” as the subject and object entities of the “was born in” relation.

Features

Documentation: http://iepy.readthedocs.org/en/0.9.0/
Github: https://github.com/machinalis/iepy PyPi: https://pypi.python.org/pypi/iepy twitter:@machinalis

October 30, 2014 04:28 PM


Hernan Grecco

PyVISA command-line utilities

PyVISA is a Python frontend for the VISA library that enables controlling all kinds of measurement equipment through GPIB, RS232, USB and Ethernet among others interfaces.

If you are following the development of PyVISA you might have seen that we have recently made the visa module executable to provide a few useful utilities. To try this, you need to update to the latest PyVISA:

$ pip install -U https://github.com/hgrecco/pyvisa/zipball/master

First, we now provide a simpler way to get debug information:

$ python -m visa info
Machine Details:
   Platform ID:    Darwin-10.8.0-x86_64-i386-32bit
   Processor:      i386

Python:
   Implementation: CPython
   Executable:     /Users/grecco/envs/lantz/bin/python
   Version:        3.2.3
   Compiler:       GCC 4.2.1 (Apple Inc. build 5666) (dot 3)
   Bits:           32bit
   Build:          Apr 10 2012 11:25:50 (#v3.2.3:3d0686d90f55)
   Unicode:        UCS2

PyVISA Version: 1.6.1

Backends:
   ni:
      Version: 1.6.1 (bundled with PyVISA)
      #1: /Library/Frameworks/visa.framework/visa:
         found by: auto
         bitness: 32
         Vendor: National Instruments
         Impl. Version: 5243392
         Spec. Version: 5243136
   py:
      Version: 0.1.dev0
      ASRL INSTR: Available via PySerial (10.8.0)
      TCPIP INSTR: Available
      USB INSTR: Available via PyUSB (1.0.0rc1). Backend: libusb0


Notice also that more useful information is given, including details about the different backends (in this case ni and py).

Another utility is the VISA shell which was taken from the Lantz project. It provides a way to list, open and query devices. It also allows you to get (and in the near future set) attributes. The shell has in-built help, autocomplete and

$ python -m visa shell
Welcome to the VISA shell. Type help or ? to list commands.

(visa) list
( 0) ASRL2::INSTR
( 1) ASRL1::INSTR
(visa) open ASRL1::INSTR
ASRL1::INSTR has been opened.
You can talk to the device using "write", "read" or "query".
The default end of message is added to each message
(open) attr
+-----------------------------+------------+----------------------------+-------------------------------------+
|          VISA name          |  Constant  |        Python name         |                 val                 |
+-----------------------------+------------+----------------------------+-------------------------------------+
| VI_ATTR_ASRL_ALLOW_TRANSMIT | 1073676734 |       allow_transmit       |                  1                  |
|    VI_ATTR_ASRL_AVAIL_NUM   | 1073676460 |      bytes_in_buffer       |                  0                  |
|      VI_ATTR_ASRL_BAUD      | 1073676321 |         baud_rate          |                 9600                |
|    VI_ATTR_ASRL_BREAK_LEN   | 1073676733 |        break_length        |                 250                 |
|   VI_ATTR_ASRL_BREAK_STATE  | 1073676732 |        break_state         |                  0                  |
|    VI_ATTR_ASRL_CONNECTED   | 1073676731 |                            |          VI_ERROR_NSUP_ATTR         |
|    VI_ATTR_ASRL_CTS_STATE   | 1073676462 |                            |                  0                  |
|    VI_ATTR_ASRL_DATA_BITS   | 1073676322 |         data_bits          |                  8                  |
|    VI_ATTR_ASRL_DCD_STATE   | 1073676463 |                            |                  0                  |
|  VI_ATTR_ASRL_DISCARD_NULL  | 1073676464 |        discard_null        |                  0                  |
|    VI_ATTR_ASRL_DSR_STATE   | 1073676465 |                            |                  0                  |
|    VI_ATTR_ASRL_DTR_STATE   | 1073676466 |                            |                  1                  |
|     VI_ATTR_ASRL_END_IN     | 1073676467 |         end_input          |                  2                  |
|     VI_ATTR_ASRL_END_OUT    | 1073676468 |                            |                  0                  |
|   VI_ATTR_ASRL_FLOW_CNTRL   | 1073676325 |                            |                  0                  |
|     VI_ATTR_ASRL_PARITY     | 1073676323 |           parity           |                  0                  |
|  VI_ATTR_ASRL_REPLACE_CHAR  | 1073676478 |        replace_char        |                  0                  |
|    VI_ATTR_ASRL_RI_STATE    | 1073676479 |                            |                  0                  |
|    VI_ATTR_ASRL_RTS_STATE   | 1073676480 |                            |                  1                  |
|    VI_ATTR_ASRL_STOP_BITS   | 1073676324 |         stop_bits          |                  10                 |
|    VI_ATTR_ASRL_WIRE_MODE   | 1073676735 |                            |                 128                 |
|    VI_ATTR_ASRL_XOFF_CHAR   | 1073676482 |         xoff_char          |                  19                 |
|    VI_ATTR_ASRL_XON_CHAR    | 1073676481 |          xon_char          |                  17                 |
|     VI_ATTR_DMA_ALLOW_EN    | 1073676318 |         allow_dma          |                  0                  |
|    VI_ATTR_FILE_APPEND_EN   | 1073676690 |                            |                  0                  |
|    VI_ATTR_INTF_INST_NAME   | 3221160169 |                            | ASRL1  (/dev/cu.Bluetooth-PDA-Sync) |
|       VI_ATTR_INTF_NUM      | 1073676662 |      interface_number      |                  1                  |
|      VI_ATTR_INTF_TYPE      | 1073676657 |                            |                  4                  |
|       VI_ATTR_IO_PROT       | 1073676316 |        io_protocol         |                  1                  |
|   VI_ATTR_MAX_QUEUE_LENGTH  | 1073676293 |                            |                  50                 |
|   VI_ATTR_RD_BUF_OPER_MODE  | 1073676330 |                            |                  3                  |
|     VI_ATTR_RD_BUF_SIZE     | 1073676331 |                            |                 4096                |
|      VI_ATTR_RM_SESSION     | 1073676484 |                            |               3160976               |
|      VI_ATTR_RSRC_CLASS     | 3221159937 |       resource_class       |                INSTR                |
|  VI_ATTR_RSRC_IMPL_VERSION  | 1073676291 |   implementation_version   |               5243392               |
|   VI_ATTR_RSRC_LOCK_STATE   | 1073676292 |         lock_state         |                  0                  |
|     VI_ATTR_RSRC_MANF_ID    | 1073676661 |                            |                 4086                |
|    VI_ATTR_RSRC_MANF_NAME   | 3221160308 | resource_manufacturer_name |         National Instruments        |
|      VI_ATTR_RSRC_NAME      | 3221159938 |       resource_name        |             ASRL1::INSTR            |
|  VI_ATTR_RSRC_SPEC_VERSION  | 1073676656 |        spec_version        |               5243136               |
|     VI_ATTR_SEND_END_EN     | 1073676310 |          send_end          |                  1                  |
|   VI_ATTR_SUPPRESS_END_EN   | 1073676342 |                            |                  0                  |
|       VI_ATTR_TERMCHAR      | 1073676312 |                            |                  10                 |
|     VI_ATTR_TERMCHAR_EN     | 1073676344 |                            |                  0                  |
|      VI_ATTR_TMO_VALUE      | 1073676314 |                            |                 2000                |
|       VI_ATTR_TRIG_ID       | 1073676663 |                            |                  -1                 |
|   VI_ATTR_WR_BUF_OPER_MODE  | 1073676333 |                            |                  2                  |
|     VI_ATTR_WR_BUF_SIZE     | 1073676334 |                            |                 4096                |
+-----------------------------+------------+----------------------------+-------------------------------------+
(open) close
The resource has been closed.



Again, this release is only possible thanks to the contribution of a lot of people that contributed bug reports, testing and code. Thanks to everybody!

Submit your bug reports, comments and suggestions in the Issue Tracker. We will address them promptly.

Read the development docs: https://pyvisa.readthedocs.org/en/master/
or fork the code: https:/https://github.com/hgrecco/pyvisa/

October 30, 2014 01:33 PM


Reinout van Rees

Ubuntu PPA madness

I'm going flipping insane. In ye olde days, when I was programming with the python CMS Plone, my dependencies were limited to python and PIL. Perhaps lxml. LXML was a pain to install sometimes, but there were ways around it.

Working on OSX was no problem. Server setup? Ubuntu. The only thing you really had to watch in those days was your python version. Does this old site still depends on python 2.4 or is it fine to use 2.6? Plone had its own Zope database, so you didn't even need database bindings.

Now I'm working on Django sites. No problem with Django, btw! But... the sites we build with it are pretty elaborate geographical websites with lots of dependencies. Mapnik, matplotlib, numpy, scipy, gdal, spatialite, postgis. And that's not the full list. So developing on OSX is no fun anymore, using a virtual machine (virtualbox or vmware) is a necessity. So: Ubuntu.

But... ubuntu 12.04, which we still use on most of the servers, has too-old versions of several of those packages. We need a newer gdal, for instance. And a newer spatialite. The common solution is to use a PPA for that, like ubuntugis-stable.

Now for some random things that can go wrong:

  • We absolutely need a newer gdal, so we add the ubuntugis-stable PPA. This has nice new versions for lots of geo-related packages, for instance the "proj" projection library.

  • It doesn't include the "python-pyproj" package, though, which means that the ubuntu-installed python-pyproj package is compiled against a different proj library. Which means your django site segfaults. Digging deep with strace was needed to discover the problem.

  • Of course, if you need that latest gdal for your site, you add the PPA to the server. Everything runs fine.

  • A month later, the server has to be rebooted. Now the three other sites on that same server fail to start due to the pyproj-segfault. Nobody bothered to check the other sites on the server, of course. (This happened three times on different servers. This is the sort of stuff that makes you cast a doubtful eye on our quite liberal "sudo" policy...)

  • Pinning pyproj to 1.9.3 helped, as 1.9.3 worked around the issue by bundling the proj library instead of relying on the OS-packaged one.

  • Ubuntugis-stable sounds stable, but they're of course focused on getting the latest geo packages into ubuntu. So they switched from gdal 1.9 to 1.10 somewhere around june. So /usr/lib/libgdal1.so became /usr/lib/libgdal1h.so and suddenly "apt-get update/upgrade" took down many sites.

    See this travis-ci issue for some background.

  • The solution for this PPA problem was another PPA: the postgres one. That includes gdal 1.9 instead of the too-new 1.10.

  • Possible problem: the postgres PPA also uses 1.9 on the new ubuntu 14.04. 14.04 contains gdal 1.10, so using the postgres PPA downgrades gdal. That cannot but break a lot of things for us.

  • I just discovered a site that couldn't possibly work. It needs the ubuntugis-stable PPA as it needs a recent spatialite. But it also needs the postgres PPA for the 1.9 gdal! And those two don't match.

  • It still works, though. I'm not totally sure why. On a compilation machine where we build a custom debian package for one of the components, the postgres PPA was installed manually outside of the automatic build scripts. And a jenkins server where we test it still has the ubuntugis PPA, but somehow it still has the old 1.9 gdal. Probably someone pinned it?

  • Another reason is probably that one of the components was compiled before the 1.9/1.10 gdal change and didn't need re-compilation yet. Once that must be done we're probably in deep shit.

  • If I look at some ansible scripts that are used to set up some of our servers, I see the ubuntugis PPA, the mapnik/v2.2.0 PPA and the redis PPA. Oh, how can that ever work? The software on those servers needs the 1.9 gdal, right?

  • I asked a colleague. Apparently the servers were all created before june and they haven't done an "apt-get upgrade" since. That's why they still work.

Personally, I think the best way forward is to use ubuntu 14.04 LTS with its recent versions. And to stick to the base ubuntu as much as possible. And if one or two packages are needed in more recent versions, try to somehow make a custom package for it without breaking the rest. I did something like that for mapnik, where we somehow needed the ancient 0.7 version on some servers.

If a PPA equates to never being able to do "apt-get update", I don't really think it is the best way forward for servers that really have to stay up.

Does someone have other thoughts? Other solutions? And no, I don't think docker containers are the solution as throwing around PPAs doesn't get more stable once you isolate it in a container. You don't break anything else, true, but the container itself can be broken by an update just fine.

October 30, 2014 10:10 AM


S. Lott

My First Webcast

http://www.oreilly.com/pub/e/3255

I'm a pretty good public speaker. But I've avoided webcasting and podcasting because it's kind of daunting. In a smaller venue, the audience members are right there, and you can tell if you're not making sense. In a webcast, the feedback will be indirect. In a podcast it's seems like it would be nonexistent.

Also, I find that programming is an intensely literate experience. It's about reading and writing. A podcast -- listening and watching -- seems very un-programmerly to me. Perhaps I'm just being an old "get-of-my-lawn-you-kids" fart.

But I'll see how the webcast thing goes in January, and perhaps I'll try to do some podcasts.

October 30, 2014 08:00 AM


Calvin Spealman

The Curl Pipe

If anything deserves to be called an anti-pattern it is probably the common and worry-inducing practice of documenting your installation process by asking asking users to copy and paste a line into their shell that will snag some file off the internet and pipe its contents directly into your shell to execute.

Sometimes this is even done as root.

This is something known to be awful, but which remains a cornerstone via its use by some of the most important tools in our belts. Homebrew does it. NPM does it, too. And some projects look better, but are they? Pip asks you to download get-pip.py and run it to install, which isn’t practically any different than piping from curl, just less efficient.

But worst of all, we might as well be doing this even more often, because our most depended about tooling is all just as guilty even without doing the curl pipe sh dance. What do you think happens when you pip install your favorite Python package, anyway? Pip downloads a file from the internet and executes it. Simple as that, for the purposes here. Sure, these days we have saner defaults. It has to be HTTPS and it has to be from PyPI by default, but its not like these packages are screened.

For all our concerns about security and frets over SHELLSHOCK and POODLE vulnerabilities, doesn’t it seem like the developer community does an awful lot of executing random files off the internet?

October 30, 2014 02:56 AM


Armin Ronacher

Don't Panic! The Hitchhiker's Guide to Unwinding

Rust has an awesome developer community but sometimes emotions can cloud the discussions that are taking place. One of the more interesting discussions (or should I say flamewars) evolve around the concept of stack unwinding in Rust. I consider myself very strongly on one side of this topic but I have not been aware of how hot this topic is until I accidentally tweeted by preference. Since then I spent a bit of time reading up no the issue and figured I might write about it since it is quite an interesting topic and has huge implications on how the language works.

What is this About?

As I wrote last time, there are two different error handling models in Rust these days. In this blog post I will call them result carriers and panics.

A result carrier is a type that can carry either a success value or a failure value. In Rust there are currently two very strong ones and a weak one: the strong ones are Result<T, E> which carries a T result value or an E error value and the Option<T> value which either carries a T result value or None which indicates that no value exists. By convention there is also a weak one which is bool which generally indicates success by signalling true and failure by signalling false. There is a proposal to actually formalize the carrier concept by introducing a Carrier trait that can (within reason) convert between any of those types which would aid composability.

The second way to indicate failure is a panic. Unlike value carriers which are passed through the stack explicitly in the form of return values, panics fly through the stack until they arrive at the frame of the task in which case they will terminate it. Panics are for all intents and purposes task failures. The way this works is by unwinding the stack slice by slice, invoking cleanup code at each level and finally terminate the task. Panics are intended for situations where the runtime runs out of choices about how to deal with this failure.

Why the Panic?

Currently there is definitely a case where there are too many calls in Rust that will just panic. For me one of the prime examples of something that panics in a not very nice way is the default print function. In fact, your rust Hello World example can panic if invoked the wrong way:

$ ./hello
Hello World!
$ ./hello 1< /dev/null
task '<main>' panicked at 'failed printing to stdout: Bad file descriptor'

The "task panicked" message is a task responding to a panic. It immediately stops doing what it does and prints an error message to stderr. It's a very prevalent problem unfortunately with the APIs currently as people do not want to deal with explicit error handling through value carriers and as such use the APIs that just fail the task (like println). That all the tutorials in Rust also go down this road because it's easier to read is not exactly helping.

One of my favorite examples is that the rustc compiler's pretty printing will cause an internal compiler error when piped into less and less is closed with the q key because the pipe is shutting down:

$ rustc a-long-example.rs --pretty=expanded|less
error: internal compiler error: unexpected failure
note: the compiler hit an unexpected failure path. this is a bug.
note: we would appreciate a bug report: http://doc.rust-lang.org/complement-bugreport.html
note: run with `RUST_BACKTRACE=1` for a backtrace
task '<main>' panicked at 'failed printing to stdout: broken pipe (Broken pipe)'

The answer to why the panic is that computers are hard and many things can fail. In C for instance printf returns an integer which can indicate if the command failed. Did you ever check it? In Rust the policy is to not let failure go through silently and because nobody feels like handling failures of every single print statement, that panicking behavior is in place for many common APIs.

But let's assume those APIs would not panic but require explicit error handling, why do we even need panics? Primarily the problem comes up in situations where a programming error happened that could not have been detected at compile time or the environment in which the application is executing is not providing the assumptions the application makes. Some good examples for the former are out of bound access in arrays and an example for the latter are out of memory errors.

Rust has safe and unsafe access to array members but the vast majority of array access goes through the unsafe access. Unsafe in this case does not mean that you get garbage back, but it means that the runtime will panic and terminate the task. Everything is still safe and everything but you just killed your thread of execution.

For memory errors and things of that nature it's more tricky. malloc in C returns you a null pointer when it fails to allocate. It looks a bit obvious that if you just inherit this behavior you don't need to panic. What that would allow you to do is to run a bit longer after you ran out of memory but there is very little you can actually do from this point onwards. The reason for this is that you just ran out of memory and you are at risk that any further thing you are going to do in order to recover from it, is going to run into the same issue. This is especially a problem if your error representation in itself requires memory. This is hardly a problem that is unique to Rust. Python for instance when it boots up needs to preallocate a MemoryError so that if it ever runs out of memory has an error it can use to indicate the failure as it might be impossible at that point to actually allocate enough memory to represent the out of memory failure.

You would be limited to only calling things that do not allocate anything which might be close to impossible to do. For instance there is no guarantee that just printing a message to stdout does not require an internal allocation.

What's Unwinding?

Stack unwinding is what makes panics in Rust work. To understand how it works you need to understand that Rust sticks very close to the metal and as such stack unwinding requires an agreed upon protocol to work.

When you raise an exception you need to immediately bubble up stack frame by stack frame until you hit your exception handler. In case of Rust you will hit the code that shuts down the task as you cannot setup handlers yourself. However as you blaze through the stack frames, Rust needs to execute all necessary cleanup code on each level so that no memory or resources leak.

This unwinding protocol is highly related to the calling conventions and not at all standardized. One of the big problems with stack unwinding is that it's not exactly an operation that comes natural to program execution, at least not on modern processors. When you want to fly through some stack frames you need to figure out what was the previous stack frame. On AMD64 for instance there is no guaranteed way to get a stacktrace at all without implementing DWARF. However stack unwinding does have the assumed benefit that because you are generally not going down the error path, there are less branches to take when a function returns as the calling frame does not have to check for an error result. If an error does occur, stack unwinding automatically jumps to the error branch and otherwise it's not considered.

What's the Problem with Unwinding?

Traditionally I think there are two problems with stack unwinding. The first one is that unlike function calling conventions, stack unwinding is not particularly standardized. This is especially a problem if you try to combine functions from different programing languages together. The most portable ABI is the C ABI and that one does not know anything about stack unwinding. There is some standardization on some operating systems but even then it does not guarantee that it will be used. For instance on Windows there is Structured Exception Handling (SEH) which however is not used by LLVM currently and as such not by Rust.

If the stack unwinding is not standardized between different languages it automatically limits the usefulness. For instance if you want to use a C++ library from another programming language, your best bet is actually to expose a C interface for it. This also means that any function you invoke through the C wrapper needs to catch down all exceptions and report them through an alternative mechanism out, making it more complicated for everybody. This even causes quite a bit of pain in the absence of actually going through a programming language boundary. If you ever used the PPL libraries (a framework for asynchronous task handling and parallelism) on Windows you might have seen how it internally catches down exceptions and reconstructs them in other places to make them travel between threads safely.

The second problem with stack unwinding is that it's really complex. In order to unwind a stack you need to figure out what your parent frame actually is. This is not necessarily a simple thing to do. On AMD64 for instance there is not enough information available on the stack to find higher stack frames so your only option is to implement the very complex DWARF spec or change the calling conventions so that you do have enough meta information on the stack. This might be simple for a project that has full control of all dependencies, but the moment you call into a library you did not compile, this no longer works.

It's no surprise that stack unwinding traditionally is one of the worse supported features in programming languages. It's not unheard of that a compiler does not implement exceptions for C++ and the reason for this is that stack unwinding is a complex thing. Even if they do implement it, very often exceptions are just made to work but not made to be fast.

Exceptions in a Systems Language

You don't have to be a kernel developer to not be a fan of stack unwinding. Any person that wants to develop a shared library that is used by other people will sooner or later have to think about how to prevent things from throwing exceptions. In C++ it's not hard to actually wrap all exported functions in huge try / catch blocks that will just catch down everything and report a failure code out, but in Rust it's currently actually a bit more complex.

The reason for this is that in Rust you cannot actually handle exceptions. When a function panics it terminates the task. This implies that there needs to be task in the first place that can isolate the exception or you cause issues for your users. Because tasks furthermore are actually threads the cost of encapsulating every function call in a thread does not sound very appealing.

Today you already are in the situation in Rust that if you write a library that wants to export a C ABI and is used by other people you can already not call into functions that panic unless you are in the situation where your system is generally running a thread and you dispatch messages into it.

Panicking Less and Disabling Unwinding

I wish I personally have for the language is that you can write code that is guaranteed to not panic unless it really ends up in a situation where it has no other choice. The biggest areas of concern there are traditionally memory allocations. However in the vast majority of situations failure from memory allocation is actually not something you need to be concerned with. Modern operating systems make it quite hard to end up in a situation where an allocation fails. There is virtual memory management and swapping and OOM killers. An malloc that returns null in a real world situation, other than by passing an unrealistically large size, is quite uncommon. And on embedded systems or similar situations you usually already keep an eye on if you are within your budget and you just avoid ever hitting the limit. This allocation problem is also a lot smaller if you are you a specialized context where you just avoid generic containers that allocate memory on regular operations.

Once panics are unlikely to happen, it's an option to disable the support for unwinding and to just abort the application if a panic ever happens. While this sounds pretty terrible, this is actually the right thing to do for a wide range of environments.

The best way to isolate failures is on the operating system level through separate processes. This sounds worse than it actually is for two reasons: the first is that the operating system provides good support for shipping data between processes. Especially for server applications the ability to have a watchdog processes that runs very little critical code, opens sockets and passes the file descriptors into worker processes is a very convincing concept. If you do end up crashing the worker no request is lost other than the currently handled one if it's single threaded. And if it's multi threaded you might kill a few more requests but new, incoming requests are completely oblivious that a failure happened as they will queue up in the socket held by the watchdog. This is something that systemd and launchd for instance provide out of the box.

In Rust especially a process boundary is a lot less scary than in other programming languages because the language design strongly discourages global state and encourages message passing.

Less Panic Needs Better APIs

The bigger problem than making panic a fatal thing and removing unwinding, is actually providing good APIs that make this less important. The biggest problem with coming up with replacements for panics is that any stack frame needs to deal with failure explicitly. If you end up writing a function that only ever returned a true or false for indicating success or failure, but you now need to call into something that might fail with important and valuable error information you do not have a channel to pass that information out without changing your own function's interface.

The other problem is that nobody wants to deal with failure if they can avoid doing so. The print example is a good one because it's the type of application where people really do not want to deal with it. "What can go wrong with printing". Unfortunately a lot. There are some proposals for Rust about how error propagation and handling can be made nicer but we're quite far from this reality.

Until we arrive there, I don't think disabling of stack unwinding would be a good idea. On the long run however I hope it's a goal because it would make Rust both more portable and interesting as a language to write reusable libraries in.

October 30, 2014 12:00 AM

October 29, 2014


Brett Cannon

Bringing someone into the modern age of technology

A relative just visited whose current technology consists of a Windows computer and a flip-phone. It was one of those situations where someone was coming to me as a blank slate for a technology upgrade! So I seized on the moment and made some recommendations.

For a phone I said he should get either a phone from Motorola, Google, or Apple. It would come down to price and who was going to provide technical support as to exactly which phone they should get.

For a computer we actually gave the relative an old Chromebook. With no baggage as to some set of programs they had to have access to, it made the decision easy. Even if we didn’t have a spare Chromebook to give away I would have suggested a Chromebook for its price and ease of maintenance. This probably would have also led to a suggestion of a new printer that supported Google Cloud Print to work with the Chromebook. And then the final perk was the integration with Google Drive as a way to move them into cloud backup.

For watching movies I would suggest getting a Netflix account and either a Chromecast or Roku box depending on their comfort level. I personally prefer Chromecast for its flexibility but I realize some people just prefer having a dedicated remote control.

Finally, we said to consider a music streaming service and Sonos. For ease of setup Sonos is great when someone doesn’t have pre-existing A/V equipment that they need to work with. And a streaming service like Google Play Music All-Access gives quick access to digital music to a level people who are upgrading their technological lives are not used to.

October 29, 2014 11:37 PM


Paul Everitt

Faster relevance ranking didn’t make it into PostgreSQL 9.4

Alas, the one big feature we really needed, the patch apparently got rejected.

PostgreSQL has a nice little full-text search story, especially when you combine it with other parts of our story (security-aware filtering of results, transactional integrity, etc.) Searches are very, very fast.

However, the next step — ranking the results — isn’t so fast. It requires a table scan (likely to TOAST files, meaning read a file and gunzip its contents) on every row that matched.

In our case, we’re doing prefix searches, and lots and lots of rows match. Lots. And the performance is, well, horrible. Oleg and friends had a super-fast speedup for this ready for PostgreSQL 9.4, but it apparently got rejected.

So we’re stuck. It’s too big a transition to switch to ElasticSearch or something. The customer probably should bail on prefix searching (autocomplete) but they won’t. We have an idea for doing this the right way (convert prefixes to candidate full words, as Google does, using PG’s built-in lexeme tools) but that is also too much for budget. Finally, we don’t have the option to throw SSDs at it.


October 29, 2014 05:34 PM


Andriy Kornatskyy

wheezy web: RESTful API Design

In this article we are going to explore a simple RESTful API created with wheezy.web framework. The demo implements a CRUD for tasks. Includes entity validation, content caching with dependencies and functional test cases. The source code is structured with well defined actors (you can read more about it here). Design The following convention is used with respect to operation, HTTP method (verb

October 29, 2014 05:18 PM


PyPy Development

A Field Test of Software Transactional Memory Using the RSqueak Smalltalk VM

Extending the Smalltalk RSqueakVM with STM

by Conrad Calmez, Hubert Hesse, Patrick Rein and Malte Swart supervised by Tim Felgentreff and Tobias Pape

Introduction

After pypy-stm we can announce that through the RSqueakVM (which used to be called SPyVM) a second VM implementation supports software transactional memory. RSqueakVM is a Smalltalk implementation based on the RPython toolchain. We have added STM support based on the STM tools from RPython (rstm). The benchmarks indicate that linear scale up is possible, however in some situations the STM overhead limits speedup.

The work was done as a master's project at the Software Architechture Group of Professor Robert Hirschfeld at at the Hasso Plattner Institut at the University of Potsdam. We - four students - worked about one and a half days per week for four months on the topic. The RSqueakVM was originally developped during a sprint at the University of Bern. When we started the project we were new to the topic of building VMs / interpreters.

We would like to thank Armin, Remi and the #pypy IRC channel who supported us over the course of our project. We also like to thank Toni Mattis and Eric Seckler, who have provided us with an initial code base.

Introduction to RSqueakVM

As the original Smalltalk implementation, the RSqueakVM executes a given Squeak Smalltalk image, containing the Smalltalk code and a snapshot of formerly created objects and active execution contexts. These execution contexts are scheduled inside the image (greenlets) and not mapped to OS threads. Thereby the non-STM RSqueakVM runs on only one OS thread.

Changes to RSqueakVM

The core adjustments to support STM were inside the VM and transparent from the view of a Smalltalk user. Additionally we added Smalltalk code to influence the behavior of the STM. As the RSqueakVM has run in one OS thread so far, we added the capability to start OS threads. Essentially, we added an additional way to launch a new Smalltalk execution context (thread). But in contrast to the original one this one creates a new native OS thread, not a Smalltalk internal green thread.

STM (with automatic transaction boundaries) already solves the problem of concurrent access on one value as this is protected by the STM transactions (to be more precise one instruction). But there are cases were the application relies on the fact that a bigger group of changes is executed either completely or not at all (atomic). Without further information transaction borders could be in the middle of such a set of atomic statements. rstm allows to aggregate multiple statements into one higher level transaction. To let the application mark the beginning and the end of these atomic blocks (high-level transactions), we added two more STM specific extensions to Smalltalk.

Benchmarks

RSqueak was executed in a single OS thread so far. rstm enables us to execute the VM using several OS threads. Using OS threads we expected a speed-up in benchmarks which use multiple threads. We measured this speed-up by using two benchmarks: a simple parallel summation where each thread sums up a predefined interval and an implementation of Mandelbrot where each thread computes a range of predefined lines.

To assess the speed-up, we used one RSqueakVM compiled with rstm enabled, but once running the benchmarks with OS threads and once with Smalltalk green threads. The workload always remained the same and only the number of threads increased. To assess the overhead imposed by the STM transformation we also ran the green threads version on an unmodified RSqueakVM. All VMs were translated with the JIT optimization and all benchmarks were run once before the measurement to warm up the JIT. As the JIT optimization is working it is likely to be adoped by VM creators (the baseline RSqueakVM did that) so that results with this optimization are more relevant in practice than those without it. We measured the execution time by getting the system time in Squeak. The results are:

Parallel Sum Ten Million

Benchmark Parallel Sum 10,000,000
Thread Count RSqueak green threads RSqueak/STM green threads RSqueak/STM OS threads Slow down from RSqueak green threads to RSqueak/STM green threads Speed up from RSqueak/STM green threads to RSQueak/STM OS Threads
1 168.0 ms 240.0 ms 290.9 ms 0.70 0.83
2 167.0 ms 244.0 ms 246.1 ms 0.68 0.99
4 167.8 ms 240.7 ms 366.7 ms 0.70 0.66
8 168.1 ms 241.1 ms 757.0 ms 0.70 0.32
16 168.5 ms 244.5 ms 1460.0 ms 0.69 0.17

Parallel Sum One Billion

Benchmark Parallel Sum 1,000,000,000

Thread CountRSqueak green threadsRSqueak/STM green threadsRSqueak/STM OS threadsSlow down from RSqueak green threads to RSqueak/STM green threadsSpeed up from RSqueak/STM green threads to RSQueak/STM OS Threads
1 16831.0 ms 24111.0 ms 23346.0 ms 0.70 1.03
2 17059.9 ms 24229.4 ms 16102.1 ms 0.70 1.50
4 16959.9 ms 24365.6 ms 12099.5 ms 0.70 2.01
8 16758.4 ms 24228.1 ms 14076.9 ms 0.69 1.72
16 16748.7 ms 24266.6 ms 55502.9 ms 0.69 0.44

Mandelbrot Iterative

Benchmark Mandelbrot
Thread Count RSqueak green threads RSqueak/STM green threads RSqueak/STM OS threads Slow down from RSqueak green threads to RSqueak/STM green threads Speed up from RSqueak/STM green threads to RSqueak/STM OS Threads
1 724.0 ms 983.0 ms 1565.5 ms 0.74 0.63
2 780.5 ms 973.5 ms 5555.0 ms 0.80 0.18
4 781.0 ms 982.5 ms 20107.5 ms 0.79 0.05
8 779.5 ms 980.0 ms 113067.0 ms 0.80 0.01

Discussion of benchmark results

First of all, the ParallelSum benchmarks show that the parallelism is actually paying off, at least for sufficiently large embarrassingly parallel problems. Thus RSqueak can also benefit from rstm.

On the other hand, our Mandelbrot implementation shows the limits of our current rstm integration. We implemented two versions of the algorithm one using one low-level array and one using two nested collections. In both versions, one job only calculates a distinct range of rows and both lead to a slowdown. The summary of the state of rstm transactions shows that there are a lot of inevitable transactions (transactions which must be completed). One reason might be the interactions between the VM and its low-level extensions, so called plugins. We have to investigate this further.

Limitations

Although the current VM setup is working well enough to support our benchmarks, the VM still has limitations. First of all, as it is based on rstm, it has the current limitation of only running on 64-bit Linux.

Besides this, we also have two major limitations regarding the VM itself. First, the atomic interface exposed in Smalltalk is currently not working, when the VM is compiled using the just-in-time compiler transformation. Simple examples such as concurrent parallel sum work fine while more complex benchmarks such as chameneos fail. The reasons for this are currently beyond our understanding. Second, Smalltalk supports green threads, which are threads which are managed by the VM and are not mapped to OS threads. We currently support starting new Smalltalk threads as OS threads instead of starting them as green threads. However, existing threads in a Smalltalk image are not migrated to OS threads, but remain running as green threads.

Future work for STM in RSqueak

The work we presented showed interesting problems, we propose the following problem statements for further analysis:

Details for the technically inclined

Details on the project setup

From a non-technical perspective, a problem we encountered was the huge roundtrip times (on our machines up to 600s, 900s with JIT enabled). This led to a tendency of bigger code changes ("Before we compile, let's also add this"), lost flow ("What where we doing before?") and different compiled interpreters in parallel testing ("How is this version different from the others?") As a consequence it was harder to test and correct errors. While this is not as much of a problem for other RPython VMs, RSqueakVM needs to execute the entire image, which makes running it untranslated even slower.

Summary

The benchmarks show that speed up is possible, but also that the STM overhead in some situations can eat up the speedup. The resulting STM-enabled VM still has some limitations: As rstm is currently only running on 64-bit Linux the RSqueakVM is doing so as well. Eventhough it is possible for us now to create new threads that map to OS threads within the VM, the migration of exiting Smalltalk threads keeps being problematic.

We showed that an existing VM code base can benefit of STM in terms of scaling up. Further it was relatively easy to enable STM support. This may also be valuable to VM developers considering to get STM support for their VMs.

October 29, 2014 04:55 PM


Ed Crewe

Fixing third party Django packages for Python 3

With the release of Django 1.7 it could be argued that the balance has finally tipped towards Python 3 being its preferred platform. Well given Python 2.7 is the last 2.* then its probably time we all thought about moving to Python 3 for our Django deployments.

Problem is those pesky third party package developers, because unless you are determined wheel reinventor (unlikely if you use Django!) - you are bound to have a range of third party eggs in your Django sites. As one of those pesky third party developers myself, it is about time I added Python 3 compatibility to my Django open source packages.

There are a number of resources related to porting Python from 2 to 3, including specifically for Django, but hopefully this post may still prove useful as a summarised approach for doing it for your Django projects or third party packages. Hopefully it isn't too much work and if you have been writing Python as long as me, it may also get you out of any legacy syntax  habits you have.

So lets get started, first thing is to set up Django 1.7 with Python 3
For repeatable builds we want pip and virtualenv - if they are not there.
For a linux platform such as Ubuntu you will have python3 installed as standard (although not yet the default python) so if you just add pip3 that lets you add the rest ...

Install Python 3 and Django for testing


sudo apt-get install python3-pip
(OR sudo easy_install3 pip)
sudo pip3 install virtualenv



So now you can run virtualenv with python3 in addition to the default python (2.*)

virtualenv --python=python3 myenv3
cd myenv3
bin/pip install django


Then add a src directory for putting the egg in you want to make compatible with Python 3 so an example from git (of course you can do this as one pip line if the source is in git)


mkdir src
git clone https://github.com/django-pesky src/django-pesky
bin/pip install -e src/django-pesky


Then run the django-pesky tests (assuming nobody uses an egg without any tests!)
so the command to run pesky's test may be something like the following ...

bin/django-admin.py test pesky.tests --settings=pesky.settings
One rather disconcerting thing that you will notice with tests is that the default assertEqual message is truncated in Python 3 where it wasn't in Python 2 with a count of the missing characters in square brackets, eg.

AssertionError: Lists differ: ['Failed to open file /home/jango/myenv/sr[85 chars]tem'] != []


Common Python 2 to Python 3 errors


And wait for those errors. The most common ones are:

  1. print statement without brackets
  2. except Error as err (NOT except Error, err)
  3. File open and file methods differ.
    Text files require better quality encoding - so more files default to bytes because strings in Python 3 are all stored in unicode
    (On the down side this may need more work for initial encoding clean up *,
    but on the plus side functional errors due to bad encoding are less likely to occur)
  4. There is no unicode() method in Python 3 since all strings are now unicode - ie. its become str() and hence strings no longer need the u'string' marker 
  5. Since unicode is not available as a method, it is not used for Django models default representation. Hence just using
    def __str__(self):
            return self.name
    is the future proofed method. I actually found that models with __unicode__ and __str__ methods may not return any representation, rather than the __str__ one being used, as one might assume, in Django 1.7 and Python 3
  6. dictionary has_key has gone, must use in (if key in dict)

* I found more raw strings were treated as bytes by Python 3 and these then required raw_string.decode(charset) to avoid them going into the database string (eg. varchar) fields as pseudo-bytes, ie. strings that held 'élément' as '\xc3\xa9l\xc3\xa9ment' rather than bytes, ie. b'\xc3\xa9l\xc3\xa9ment'

Ideally you will want to maintain one version but keep it compatible with Python 2 and 3,
since this is both less work and gets you into the habit of writing transitional Python :-)

Test the same code against Python 2 and 3


So to do that you want to be running your tests with builds in both Pythons.
So repeat the above but with virtualenv --python=python2 myenv2
and just symlink the src/django-pesky to the Python 2 src folder.

Now you can run the tests for both versions against the same egg code -
and make sure when you fix for 3 you don't break for 2.

For current Django 1.7 you would just need to support the latest Python 2.7
and so the above changes are all compatible except for use of unicode() and how you call open().

Version specific code


However in some cases you may need to write code that is specific to 2 or 3.
If that occurs you can either use the approach of latest or anything else (cross fingers)

try:
    latest version compatible code (e.g. Python 3 - Django 1.7)
except:
    older version compatible code (e.g. Python 2 - Django < 1.7)

Or you can use specific version targetting ...

import sys, django
django_version = django.get_version().split('.')

if sys.version_info['major'] == 3 and django_version[1] == 7:
    latest version
elif sys.version_info['major'] == 2 and django_version[1] == 6:
    older django version
else:
    older version


where ...

django.get_version() -> '1.6' or '1.7.1'
sys.version_info() -> {'major':3, 'minor':4, 'micro':0, 'releaselevel':'final', 'serial':0}

Summary

So how did I get on with my first egg, django-csvimport ? ... it actually proved quite time consuming since the csv.reader library was far more sensitive to bad character encoding in Python 3 and so a more thorough manual alternative had to be implemented for those important edge cases - which the tests are aimed to cover. After all if a CSV file is really well encoded and you already have a model for it - it hardly needs a pesky third party egg for CSV imports - just a few django shell lines using the csv library will do the job.







October 29, 2014 02:44 PM


A. Jesse Jiryu Davis

Toro 0.7 Released

Toro

I've just released version 0.7 of Toro. Toro provides semaphores, locks, events, conditions, and queues for Tornado coroutines. It enables advanced coordination among coroutines, similar to what you do in a multithreaded application. Get the latest version with "pip install --upgrade toro". Toro's documentation, with plenty of examples, is on ReadTheDocs.

There is one bugfix in this release. Semaphore.wait() is supposed to wait until the semaphore can be acquired again:

@gen.coroutine
def coro():
    sem = toro.Semaphore(1)
    assert not sem.locked()

    # A semaphore with initial value of 1 can be acquired once,
    # then it's locked.
    sem.acquire()
    assert sem.locked()

    # Wait for another coroutine to release the semaphore.
    yield sem.wait()

... however, there was a bug and the semaphore didn't mark itself "locked" when it was acquired, so "wait" always returned immediately. I'm grateful to "abing" on GitHub for noticing the bug and contributing a fix.

October 29, 2014 02:09 PM


Python Anywhere

Maintenance release: trusty + process listings

Hi All,

A maintenance release today, so nothing too exciting. Still, a couple of things you may care about:

Other than that, we've updated our client-side for our Postgres beta to 9.4, and added some of the PostGIS utilities. (Email us if you want to check out the beta). We also fixed an issue where a redirect loop would break the "reload" button on web apps, and we've added weasyprint and python-svn to the batteries included.

October 29, 2014 07:51 AM


Flavio Percoco

Hiding unnecessary complexity

This post does not represent a strong opinion but something I've been thinking about for a bit. The content could be completely wrong or it could even make some sense. Regardless, I'd like to throw it out there and hopefully gather some feedback from people interested in this topic.

Before I get into the details, I'd like to share why I care. Since I started programming, I've had the opportunity to work with experienced and non experienced folks in the field. This allowed me to learn from others the things I needed and to teach others the things they wanted to learn that I knew already. Lately, I've dedicated way more time to teaching others and welcoming new people to our field. Whether they already had some experience or not is not relevant. What is indeed relevant, though, is that there's something that needed to be taught, which required a base knowledge to exist.

As silly as it may sound, I believe the process of learning, or simply the steps we follow to acquire new knowledge, can be represented in a directed graph. We can't learn everything at once, we must follow an order. When we want to learn something, we need to start somewhere and dig into the topic of our interest one step at a time.

The thing I've been questioning lately is how deep does someone need to go to consider something as learned? When does the required knowledge to do/learn X ends? Furthermore, I'm most interested in what we - as developers or creators of these abstractions that will then be consumed - can do to define this.

Learning new things is fascinating, at least for me. When I'm reading about a topic I know nothing about, I'd probably read until I feel satisfied with what I've discovered whereas when I'm digging into something I need to know to do something else, I'd probably read until I hit that a-ha moment and I feel I know enough to complete my task. Whether I'll keep digging afterwards or not depends on how interesting I think the topic is. However, the important bit here is that I'll focus on what I need to know and I leave everything else aside.

I believe the same thing happens when we're consuming an API - regardless it's a library, a RESTFul API, RPC API, etc. We'll read the documentation - or just the API - and then we'll start using it. There's no need to read how it was implemented and, hopefully, no further reading will be necessary either. If we know enough and/or the API is simple enough - in terms of how it exposes the internal implementation, vocabulary, pattern, etc - we won't need to dig into any other topics that we may not know already.

Whenever we are writing an API, we tend to either expose too many things or too few things. Finding the right balance between the things that should be kept private and the ones that should be made public is a never-ending crusade. Moreover, keeping the implementation simple and yet flexible becomes harder as we move on writing the API. Should we expose all the underlying context? What is the feeling a consumer of this API should have?

By now, you are probably thinking that I just went nuts and this is all nonsense and you're probably right but I'll ignore that and I'll keep going. Let me try to explain what I mean by using some, hopefully more realistic, examples.

Imagine you're writing an API for a messaging system - you saw this example coming, didn't you? - that is supposed to be simple, intuitive and yet powerful in terms of features and semantics. Now, before thinking about the API you should think about the things you want this service to support. As a full featured messaging service, you probably want it to support several messaging patterns. For the sake of this post, lets make a short list:

These are the 2 messaging patterns - probably the most common ones - that you'd like to have support for in your API. Now, think about how you'd implement them.

For the Producer/Consumer case you'd probably expose endpoints that will allow your users to post messages and get messages. So far so good, it's quite simple and straightforward. To make things a little bit more complicated, lets say you'd like to support grouping for messages. That is, you'd like to provide a simple way to keep a set of messages separated from another set of messages. A very simple way to do that is by supporting the concept of queues. However, Queue is probably a more complex type of resource which implicitly brings some properties into your system. For example, by adding queues to your API you're implicitly saying that messages have an order, therefore it's possible to walk through it - pagination, if you will - and these messages cannot - or shouldn't - be accessed randomly. You probably know all this, which makes the implementation quite simple and intuitive for you but, does the consumer of the API know this? will consuming the API be as simple and intuitive as implementing it was for you? Should the consumer actually care about what queue is? Keep in mind the only thing you wanted to add is grouping for messages.

You may argue saying that you could use lightweight queues or just call it something else to avoid bringing all these properties in. You could, for example, call them topics or even just groups. The downside of doing this is that you'd be probably reinventing a concept that already exists and assigning to it a different name and custom properties. Nothing wrong with that, I guess.

You've a choice to make now. Are you going to expose queues through the API for what they are? Or are you going to expose them in a simpler way and keep them as queues internally? Again, should your users actually care? What is it that they really need to know to use your API?

As far as your user is concerned, the important bit of your API is that messages can be grouped, posting messages is a matter of sending data to your server and getting them is a matter of asking for messages. Nonetheless, many messaging services with support for queues would require the user to have a queue instance where messages should be posted but again: should users actually care?

Would it be better for your API to be something like:

MyClient.Queue('bucket').post('this is my message')

or would it be simpler and enough to be something like:

MyClient.post('this is my message', group='bucket')

See the difference? Am I finally making a point? Leave aside CS and OOP technicality, really, should the final user care?

Lets move onto the second messaging pattern we would like to have support for, publish/subscribe. At this point, you've some things already implemented that you could re-use. For instance, you already have a way to publish messages and the only thing you have to figure out for the publishing part of the message pattern is how to route the message being published to the right class. This shouldn't be hard to implement, the thing to resolve is how to expose it through the API. Should the user know this is a different message pattern? Should the user actually know that this is a publisher and that messages are going to be routed once they hit the server? Is there a way all these concepts can be hidden from the user?

What about the subscriber? The simplest form of subscription for an messaging API is the one that does not require a connection to persist. That is, you expose an API that allows users to subscribe an external endpoint - HTTP, APN, etc - that will receive messages as they're pushed by the messaging service.

You could implement the subscription model by exposing a subscribe endpoint that users would call to register the above-mentioned receivers. Again, should this subscriber concept be hidden from the user? What about asking the user where messages published to group G should be forwarded to instead of asking the users to register subscribers for the publish/subscribe pattern?

Think about how emails - I hate myself for bringing emails as a comparison - work. You've an inbox where all your emails are organized. Your inbox will normally be presented as a list. You can also send an email to some user - or group of users - and they'll receive that email as you receive other's emails. In addition to this, your email service also provides a way to forward email, filter email and re-organize it. Do you see where I'm going with this? have you ever dug into how your email service works? have you ever wondered how all these things are implemented server side? Is your email provider using a queue or just a simple database? You may have wondered all these things but, were they essential for you to understand how to use your email client? I'd bet they weren't.

Does the above make any sense? Depending on how you read the above it may seem like a silly and unnecessary way of reinventing concepts, theories and things that already exist or it may be a way to just ask the users to know what they really need to know to use your service as opposed to forcing them to dig into things they don't really need - or even care about. The more you adapt your service - or API - to what the user is expected to know, the easier it'll be for them to actually use it.

If you got to this point, I'm impressed. I'm kinda feeling I may be really going nuts but I think this post has got me to sort of a fair conclusion and probably an open question.

As much as purists may hate this, I think there's no need to force 'knowledge' into users just for the sake of it. People curious enough will dig into problems, concepts, implementations, etc. The rest of the people will do what you expect them to do, they'll use your API - for better or for worse - and they shouldn't care about the underlying implementation, theories or complexities. All these things should be hidden from the user.

Think about newcomers and how difficult it could be for a person not familiar at all with messaging system to consume a library that requires Producers and Consumers to be instantiated separately. Think about this newcomer trying to understand why there are are producers, consumers, publishers and subscribers. What if this newcomer just wanted to send a message?

As a final note, I'd probably add that the whole point here is not to use silly names for every well-known concept just to make lazy people happy. If that were the case, we wouldn't have sets and everything would be an array with random properties attached to it. The point being made here is that we tend to expose through our APIs lots of unnecessary theories and concepts that users couldn't care less about. When working on the APIs our users will consume, we should probably ask ourselves how likely it is for them to know all this already and how we can hide unnecessary concepts from them without preventing them for digging deeper into it.

Although all this may sound like "Writing APIs 101", I don't believe it is as obvious for everyone as it seems.

October 29, 2014 12:29 AM

October 28, 2014


Mikko Ohtamaa

Safe evaluation of math expressions in pure Python

This blog post discusses how to evaluate user inputted math expressions safely in Python without using any third party libraries.

E.g. the user can write

a + max(3, b) / c

… and then you run this on the serve- side with variable substitutions to get the actual result.

1. Background

There are use cases where you want to input math expressions or equations from the user as an alternative to fixed numbers. Examples include

Traditionally math expressions are done by writing your own small domain specific language using a custom BNF grammar and a library like pyparsing. You parse the equation and write some kind of evaluation loop, desperately trying cover all cases of nested functions, forward and backward looking of arguments and so on. You can see an example here.

However, Python REPL itself is pretty human-friendly calculator. Why to bother do all parsing and evaluation yourself when you just can use Python in Python?

The approach presented here is based on the example of Aleksi Torhamo and his Python code verification research. No third party libraries needed.

StackOverflow, always so helpful with moderators who understand. Luckily I got a tip towards solving the problem privately. I kindly ask you to revote to open this question, so that an answer can be posted.

StackOverflow, always so helpful with moderators who understand. Luckily I got a tip towards solving this privately. I kindly ask you to vote to reopen this question, so that an answer can be posted.

2. Protecting your service

The user input is one of the web application attack vectors. It is especially problematic when you let the user to run something, like math expressions, on the server-side  – this opens many options for a malicious actors to cause harm. Below I have listed some potential attacks and how to mitigate against them.

2. Sandbox escape

This is the traditional rationale against using eval(). You cannot simply let the user to run arbitrary code on the server, as they could simply do import os ; os.remove(“/home”). The history of Python has various harded eval and sandbox implementations coming up now and then, but the general advice is that “Python eval() can never be safe“. There are even few nice blog posts dedicated how to escape Python sandbox. Only heavyish multiprocessing approaches, like pysandbox which enforces the security on operating system process level could be considered.

In our approach, the trick is that we only do the first half of the eval(). We use the Python compile() function to prepare the Python expression to be the bytecode for evaling. Then, we actually don’t eval(). Instead we use custom opcode handlers and evaluate opcodes in a loop one after another. There cannot be a sandbox escape, because opcodes having should functionality are not implemented. Because compile() does the job of generating the microcode, we are saved from the headache of writing a custom parser. Python dis module helps us minimize the code needed for a stack-based virtual machine.

2. CPU burning attack

If you let the user to input arbitrary calculations, they can try to make the calculations so complex that your process keeps calculating the math expression forever, mounting a denial of service attack against the service. In Python’s case this means that math with really large numbers can be expensive, as Python 3 allows infinite large integers by default. Initially I was thinking the compile() approach could be easily secured against against such attacks.  One could simply limit the input value sizes in math opcodes. But unfortunately it turned out to be little trickier.

For example if you compile() Python expression 2**9999999999**9999999 (very big exponent) the Python prompt keeps pushing the bits forever. This is because Python compile() evaluates constants during the compile time, so you are stuck in the compiling step forever.

Even if we gap the power operation, which can be easily exploited to generate very large numbers, the malicious actors are quite creative of coming up with other different slow calculations. The only real mitigation for the execution time issue is having the hard limit for calculation time. We do this using Python’s multiprocess module. The calculation is run in a separate child process. This process is terminated if it doesn’t finish timely. In our case, we’ll give 100 milliseconds for the expression calculations to finish.

As a side note we threads didn’t work here because it turned out compile() does not release GIL and thus thread.start() never returns if thread.run() contains a complex compile() – the original thread does not get GIL back.

We are also protected against infinite loops. Python compile() allows only single line Python statements, thus for and while are out of the list (even they could be written on a single line).  We do not implement looping opcodes either.

2. Memory burning attack

Memory burning attack is similar to CPU burning attack. Instead of CPU cycles, the attacker tries to run make the process run out of memory. One could input a very long math expression. Alternatively, the power operation could be used to create very large numbers not fitting to the process memory.

The example code does not hard limit the memory usage of the child process. This could be done on the operating system level e.g. by capping the maximum allowed child process memory to few megabytes. The main rationale skipping it for now is that I did not (yet) find a nice portable way to do this with Python multiprocess module. Probably the child process could set its own memory cap at the beginning of the process. All tips are welcome.

Instead, a naive approach is chosen. The input size and the resulting bytecode size are checked against code size limit. If somebody knows how to make our restricted math operations program to eat a lot of memory please let me know and I’ll test against this too.

3. The implementation

Below is the sample code. The core code is ~200 lines + extras. The tests at the end of the file and having the timeout decorator consume some lines.

If you feel creative and you find a way to destroy this nice piece of code, I’d be interested to hear about the possible attack methods.

You can also find the code as Github gist.

import opcode
import dis
import sys
import multiprocessing
import time

# Python 3 required
assert sys.version_info[0] == 3, "No country for old snakes"


class UnknownSymbol(Exception):
    """ There was a function or constant in the expression we don't support. """


class BadValue(Exception):
    """ The user tried to input dangerously big value. """

    MAX_ALLOWED_VALUE = 2**63


class BadCompilingInput(Exception):
    """ The user tried to input something which might cause compiler to slow down. """


class TimeoutException(Exception):
    """ It took too long to compile and execute. """


class RunnableProcessing(multiprocessing.Process):
    """ Run a function in a child process.

    Pass back any exception received.
    """
    def __init__(self, func, *args, **kwargs):
        self.queue = multiprocessing.Queue(maxsize=1)
        args = (func,) + args
        multiprocessing.Process.__init__(self, target=self.run_func, args=args, kwargs=kwargs)

    def run_func(self, func, *args, **kwargs):
        try:
            result = func(*args, **kwargs)
            self.queue.put((True, result))
        except Exception as e:
            self.queue.put((False, e))

    def done(self):
        return self.queue.full()

    def result(self):
        return self.queue.get()


def timeout(seconds, force_kill=True):
    """ Timeout decorator using Python multiprocessing.

    Courtesy of http://code.activestate.com/recipes/577853-timeout-decorator-with-multiprocessing/
    """
    def wrapper(function):
        def inner(*args, **kwargs):
            now = time.time()
            proc = RunnableProcessing(function, *args, **kwargs)
            proc.start()
            proc.join(seconds)
            if proc.is_alive():
                if force_kill:
                    proc.terminate()
                runtime = time.time() - now
                raise TimeoutException('timed out after {0} seconds'.format(runtime))
            assert proc.done()
            success, result = proc.result()
            if success:
                return result
            else:
                raise result
        return inner
    return wrapper


def disassemble(co):
    """ Loop through Python bytecode and match instructions  with our internal opcodes.

    :param co: Python code object
    """
    code = co.co_code
    n = len(code)
    i = 0
    extended_arg = 0
    result = []
    while i < n:         op = code[i]         curi = i         i = i+1         if op >= dis.HAVE_ARGUMENT:
            # Python 2
            # oparg = ord(code[i]) + ord(code[i+1])*256 + extended_arg
            oparg = code[i] + code[i+1] * 256 + extended_arg
            extended_arg = 0
            i = i+2
            if op == dis.EXTENDED_ARG:
                # Python 2
                #extended_arg = oparg*65536L
                extended_arg = oparg*65536
        else:
            oparg = None

        # print(opcode.opname[op])

        opv = globals()[opcode.opname[op].replace('+', '_')](co, curi, i, op, oparg)

        result.append(opv)

    return result

# For the opcodes see dis.py
# (Copy-paste)
# https://docs.python.org/2/library/dis.html

class Opcode:
    """ Base class for out internal opcodes. """
    args = 0
    pops = 0
    pushes = 0
    def __init__(self, co, i, nexti, op, oparg):
        self.co = co
        self.i = i
        self.nexti = nexti
        self.op = op
        self.oparg = oparg

    def get_pops(self):
        return self.pops

    def get_pushes(self):
        return self.pushes

    def touch_value(self, stack, frame):
        assert self.pushes == 0
        for i in range(self.pops):
            stack.pop()


class OpcodeArg(Opcode):
    args = 1


class OpcodeConst(OpcodeArg):
    def get_arg(self):
        return self.co.co_consts[self.oparg]


class OpcodeName(OpcodeArg):
    def get_arg(self):
        return self.co.co_names[self.oparg]


class POP_TOP(Opcode):
    """Removes the top-of-stack (TOS) item."""
    pops = 1
    def touch_value(self, stack, frame):
        stack.pop()


class DUP_TOP(Opcode):
    """Duplicates the reference on top of the stack."""
    # XXX: +-1
    pops = 1
    pushes = 2
    def touch_value(self, stack, frame):
        stack[-1:] = 2 * stack[-1:]


class ROT_TWO(Opcode):
    """Swaps the two top-most stack items."""
    pops = 2
    pushes = 2
    def touch_value(self, stack, frame):
        stack[-2:] = stack[-2:][::-1]


class ROT_THREE(Opcode):
    """Lifts second and third stack item one position up, moves top down to position three."""
    pops = 3
    pushes = 3
    direct = True
    def touch_value(self, stack, frame):
        v3, v2, v1 = stack[-3:]
        stack[-3:] = [v1, v3, v2]


class ROT_FOUR(Opcode):
    """Lifts second, third and forth stack item one position up, moves top down to position four."""
    pops = 4
    pushes = 4
    direct = True
    def touch_value(self, stack, frame):
        v4, v3, v2, v1 = stack[-3:]
        stack[-3:] = [v1, v4, v3, v2]


class UNARY(Opcode):
    """Unary Operations take the top of the stack, apply the operation, and push the result back on the stack."""
    pops = 1
    pushes = 1


class UNARY_POSITIVE(UNARY):
    """Implements TOS = +TOS."""
    def touch_value(self, stack, frame):
        stack[-1] = +stack[-1]


class UNARY_NEGATIVE(UNARY):
    """Implements TOS = -TOS."""
    def touch_value(self, stack, frame):
        stack[-1] = -stack[-1]


class BINARY(Opcode):
    """Binary operations remove the top of the stack (TOS) and the second top-most stack item (TOS1) from the stack. They perform the operation, and put the result back on the stack."""
    pops = 2
    pushes = 1


class BINARY_POWER(BINARY):
    """Implements TOS = TOS1 ** TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        print(TOS1, TOS)
        if abs(TOS1) > BadValue.MAX_ALLOWED_VALUE or abs(TOS) > BadValue.MAX_ALLOWED_VALUE:
            raise BadValue("The value for exponent was too big")

        stack[-2:] = [TOS1 ** TOS]


class BINARY_MULTIPLY(BINARY):
    """Implements TOS = TOS1 * TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 * TOS]


class BINARY_DIVIDE(BINARY):
    """Implements TOS = TOS1 / TOS when from __future__ import division is not in effect."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 / TOS]


class BINARY_MODULO(BINARY):
    """Implements TOS = TOS1 % TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 % TOS]


class BINARY_ADD(BINARY):
    """Implements TOS = TOS1 + TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 + TOS]


class BINARY_SUBTRACT(BINARY):
    """Implements TOS = TOS1 - TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 - TOS]


class BINARY_FLOOR_DIVIDE(BINARY):
    """Implements TOS = TOS1 // TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 // TOS]


class BINARY_TRUE_DIVIDE(BINARY):
    """Implements TOS = TOS1 / TOS when from __future__ import division is in effect."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 / TOS]


class BINARY_LSHIFT(BINARY):
    """Implements TOS = TOS1 << TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 << TOS] class BINARY_RSHIFT(BINARY):     """Implements TOS = TOS1 >> TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 >> TOS]


class BINARY_AND(BINARY):
    """Implements TOS = TOS1 & TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 & TOS]


class BINARY_XOR(BINARY):
    """Implements TOS = TOS1 ^ TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 ^ TOS]


class BINARY_OR(BINARY):
    """Implements TOS = TOS1 | TOS."""
    def touch_value(self, stack, frame):
        TOS1, TOS = stack[-2:]
        stack[-2:] = [TOS1 | TOS]


class RETURN_VALUE(Opcode):
    """Returns with TOS to the caller of the function."""
    pops = 1
    final = True
    def touch_value(self, stack, frame):
        value = stack.pop()
        return value


class LOAD_CONST(OpcodeConst):
    """Pushes co_consts[consti] onto the stack.""" # consti
    pushes = 1
    def touch_value(self, stack, frame):
        # XXX moo: Validate type
        value = self.get_arg()
        assert isinstance(value, (int, float))
        stack.append(value)


class LOAD_NAME(OpcodeName):
    """Pushes the value associated with co_names[namei] onto the stack.""" # namei
    pushes = 1
    def touch_value(self, stack, frame):
        # XXX moo: Get name from dict of valid variables/functions
        name = self.get_arg()
        if name not in frame:
            raise UnknownSymbol("Does not know symbol {}".format(name))
        stack.append(frame[name])


class CALL_FUNCTION(OpcodeArg):
    """Calls a function. The low byte of argc indicates the number of positional parameters, the high byte the number of keyword parameters. On the stack, the opcode finds the keyword parameters first. For each keyword argument, the value is on top of the key. Below the keyword parameters, the positional parameters are on the stack, with the right-most parameter on top. Below the parameters, the function object to call is on the stack. Pops all function arguments, and the function itself off the stack, and pushes the return value.""" # argc
    pops = None
    pushes = 1

    def get_pops(self):
        args = self.oparg & 0xff
        kwargs = (self.oparg >> 8) & 0xff
        return 1 + args + 2 * kwargs

    def touch_value(self, stack, frame):
        argc = self.oparg & 0xff
        kwargc = (self.oparg >> 8) & 0xff
        assert kwargc == 0
        if argc > 0:
            args = stack[-argc:]
            stack[:] = stack[:-argc]
        else:
            args = []
        func = stack.pop()

        assert func in frame.values(), "Uh-oh somebody injected bad function. This does not happen."

        result = func(*args)
        stack.append(result)


def check_for_pow(expr):
    """ Python evaluates power operator during the compile time if its on constants.

    You can do CPU / memory burning attack with ``2**999999999999999999999**9999999999999``.
    We mainly care about memory now, as we catch timeoutting in any case.

    We just disable pow and do not care about it.
    """
    if "**" in expr:
        raise BadCompilingInput("Power operation is not allowed")


def _safe_eval(expr, functions_and_constants={}, check_compiling_input=True):
    """ Evaluate a Pythonic math expression and return the output as a string.

    The expr is limited to 1024 characters / 1024 operations
    to prevent CPU burning or memory stealing.

    :param functions_and_constants: Supplied "built-in" data for evaluation
    """

    # Some safety checks
    assert len(expr) < 1024

    # Check for potential bad compiler input
    if check_compiling_input:
        check_for_pow(expr)

    # Compile Python source code to Python code for eval()
    code = compile(expr, '', 'eval')

    # Dissect bytecode back to Python opcodes
    ops = disassemble(code)
    assert len(ops) < 1024

    stack = []
    for op in ops:
        value = op.touch_value(stack, functions_and_constants)

    return value


@timeout(0.1)
def safe_eval_timeout(expr, functions_and_constants={}, check_compiling_input=True):
    """ Hardered compile + eval for long running maths.

    Mitigate against CPU burning attacks.

    If some nasty user figures out a way around our limitations to make really really slow calculations.
    """
    return _safe_eval(expr, functions_and_constants, check_compiling_input)


if __name__ == "__main__":

    # Run some self testing

    def test_eval(expected_result, *args):
        result = safe_eval_timeout(*args)
        if result != expected_result:
            raise AssertionError("Got: {} expected: {}".format(result, expected_result))

    test_eval(2, "1+1")
    test_eval(2, "1 + 1")
    test_eval(3, "a + b", dict(a=1, b=2))
    test_eval(2, "max(1, 2)", dict(max=max))
    test_eval(2, "max(a, b)", dict(a=1, b=2, max=max))
    test_eval(3, "max(a, c, b)", dict(a=1, b=2, c=3, max=max))
    test_eval(3, "max(a, max(c, b))", dict(a=1, b=2, c=3, max=max))
    test_eval("2", "str(1 + 1)", dict(str=str))
    test_eval(2.5, "(a + b) / c", dict(a=4, b=1, c=2))

    try:
        test_eval(None, "max(1, 0)")
        raise AssertionError("Should not be reached")
    except UnknownSymbol:
        pass

    # CPU burning
    try:
        test_eval(None, "2**999999999999999999999**9999999999")
        raise AssertionError("Should not be reached")
    except BadCompilingInput:
        pass

    # CPU burning, see out timeoutter works
    try:
        safe_eval_timeout("2**999999999999999999999**9999999999", check_compiling_input=False)
        raise AssertionError("Should not be reached")
    except TimeoutException:
        pass

    try:
        test_eval(None, "1 / 0")
        raise AssertionError("Should not be reached")
    except ZeroDivisionError:
        pass

    try:
        test_eval(None, "(((((((((((((((()")
        raise AssertionError("Should not be reached")
    except SyntaxError:
        #    for i in range(0, 100):
        #      ^
        # SyntaxError: invalid synta
        pass

    try:
        test_eval(None, "")
        raise AssertionError("Should not be reached")
    except SyntaxError:
        # SyntaxError: unexpected EOF while parsing
        pass

    # compile() should not allow multiline stuff
    # http://stackoverflow.com/q/12698028/315168
    try:
        test_eval(None, "for i in range(0, 100):\n    pass", dict(i=-1))
        raise AssertionError("Should not be reached")
    except SyntaxError:
        #    for i in range(0, 100):
        #      ^
        # SyntaxError: invalid synta
        pass

    # No functions allowed
    try:
        test_eval(None, "lamdba x: x+1")
        raise AssertionError("Should not be reached")
    except SyntaxError:
        # SyntaxError: unexpected EOF while parsing
        pass

 

 

 

 Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

October 28, 2014 10:29 PM