skip to navigation
skip to content

Planet Python

Last update: February 23, 2018 09:46 PM

February 23, 2018


Rene Dudfield

Mu, the little #python editor that could.

Nicholas and the Mu gang have been busy with their python editor, and it now supports pgzero.
https://mu.readthedocs.io/en/latest/changes.html


Mu is a simple code editor for beginner programmers.


I'm pretty excited about this. Here's a little demo of it in action with pgzero...
.xhide-video-thumb { display:none; }

Whilst it's still in 'beta' with rough edges... it's pretty neat already. Also, they seem to be quickly moving through issues people in the community encounter with each beta release.

It's been super-lovely to see the python community get behind them and offer their support.

Stop idling, and go have a look: https://github.com/mu-editor/mu

February 23, 2018 07:00 PM


Continuum Analytics Blog

Harness the Power of Data Science at AnacondaCON 2018!

Last spring, Anaconda celebrated the inaugural AnacondaCON, where over 400 people descended upon Austin to connect with peers and thought leaders within the Python data science community. This year’s event promises to be even bigger and better! Taking place in Austin on April 8-11, AnacondaCON 2018 is shaping up to be one of the hottest …
Read more →

February 23, 2018 05:57 PM


Programiz

Writing CSV into Python

In this article, we will learn how to write data into csv files in Python of different formats.

February 23, 2018 10:25 AM


Rene Dudfield

pygame documentation updates

There are some documentation updates on https://www.pygame.org/docs/


The website documentation builder was waiting for updates from bitbucket(our previous code host). lol? oops. Robots running in the cloud doing things no one wants them to do anymore. I had to write a new github integration, so now commits to master on github/pygame/pygame will cause the website docs to be rebuilt automatically again.

And python -m pygame.docs didn't work with the wheel builds... because they don't include docs (for some reason). So now, if the docs can't be found locally, it opens the web browser instead.
There were a bunch of documentation updates from Ian Mallett and Lenard Lindstrom which are finally up on the website. Lots of editing, and improvements to the tutorials.
Additionally a bunch of old links were fixed. Mostly to point to https:// versions of pages.
The docs are being built with a new version of Sphinx, which has nicer output in a few ways. See http://www.sphinx-doc.org/en/master/changes.html#release-1-7-0-released-feb-12-2018

Since it was a major version upgrade of Sphinx, there were some small expected issues to deal with, including their python API being broken/deprecated (just call the command line program instead), and disabling the on-by-default smart quotes(which are terrible for people copy pasting code examples).

Also, the launchpad PPA is building again. It got stuck because someone snuck a gpg field in their git commit, which broke the bzr mirror code. Except the packaging for that ppa is from 2013... and needs updating. Luckily it seems a few people have done the packaging work in various flavours of Debian... but somehow none of them have gone through. But anyway... The badge is getting made again(displayed on the readme) with the github webhook, and will alert us if things break on ubuntu/i386/amd64/arm7f.

Speaking of badges... there's now a 'commits since last release' badge on the https://github.com/pygame/pygame Which links to a diff between the last release.

Finally, I updated all the old bitbucket wiki pages to point to the same link on the pygame wiki (https://www.pygame.org/wiki). eg, https://bitbucket.org/pygame/pygame/wiki/CookBook points to https://www.pygame.org/wiki/CookBook

February 23, 2018 10:20 AM


Python Bytes

#66 Wait, NoSQl with ACID and transactions?

February 23, 2018 08:00 AM


Talk Python to Me

#152 Understanding and using Python's AST

February 23, 2018 08:00 AM


Gocept Weblog

Zope – undo transactions via debug console

If you cannot access the root level in Zope 2 via the browser but you are able to use the debug console you have enough to undo transactions.

Start the debug console:

$ bin/zinstance debug

List the transaction descriptions, user names and ids of the last 10 transactions:

>>> from pprint import pprint
>>> pprint(app._p_jar.db().undoInfo(0, 10))

(0 is the youngest transaction, the first one to show)

Undo some transactions:

>>> import transaction
>>> transaction.get().note(u"Undo transactions: ...")
>>> app._p_jar.db().undoMultiple([id1, id2])
>>> transaction.commit()

(id1, id2 have to be the actual id strings from the transaction listing.)

Be aware:

 

This blog post first appeared on icemac15.wordpress.com.

February 23, 2018 06:30 AM


Codementor

Automating Simple Tasks with Scheme (Competing with Perl, Python and Ruby)

Writing Scheme/Lisp scripts to automate simple tasks that are usually automated with Perl, Python or Ruby

February 23, 2018 12:31 AM

February 22, 2018


PyCharm

PyCharm 2018.1 EAP 6

This week’s EAP version is now available: get it now from our website.

New in PyCharm 2018.1 EAP 6

Git add patch

Many people have requested it, and we’ve now added it to all JetBrains IDEs. You can now partially add files to your commit. Just use the checkboxes next to the diff to either add part of a file, or to exclude that part from the commit:

EAP 6 - Git add p

Further Improvements

You can read about further improvements in the release notes.

Interested?

Download this EAP from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date throughout the entire EAP.

If you’re on Ubuntu 16.04 or later, you can use snap to get PyCharm EAP, and stay up to date. You can find the installation instructions on our website.

PyCharm 2018.1 is in development during the EAP phase, therefore not all new features are already available. More features will be added in the coming weeks. As PyCharm 2018.1 is pre-release software, it is not as stable as the release versions. Furthermore, we may decide to change and/or drop certain features as the EAP progresses.

All EAP versions will ship with a built-in EAP license, which means that these versions are free to use for 30 days after the day that they are built. As EAPs are released weekly, you’ll be able to use PyCharm Professional Edition EAP for free for the duration of the EAP program, as long as you upgrade at least once every 30 days.

February 22, 2018 02:37 PM

February 21, 2018


Codementor

How I scraped 500k codes written by coders from the internet?

Presented at Pycon India'17. Also selected for Pycon Italy18

February 21, 2018 01:58 PM


EuroPython Society

EuroPython 2018: Getting ready

We are excited to announce the launch of the EuroPython 2018 website:

image

https://ep2018.europython.eu/

The EuroPython conference will take place in sunny Edinburgh, Scotland, this year, from July 23-29 2018.

It’s a great time of year to visit Edinburgh with 16 hours of daylight, and the festival season in full flow, so come and join us. This is just one week before the famous Edinburgh Fringe Festival and the Turing Festival, so you can extend your stay a little longer in Edinburgh, or head for the Highlands to enjoy the amazing mountains and lochs.

EuroPython 2018 - The European Python Conference

Here’s an overview of what you can expect in Edinburgh:

Overall, we will again have 7 days worth of great Python content, arranged in over 120 sessions, waiting for you.

The venue is the Edinburgh International Conference Centre, in central Edinburgh, just on the edge of the historic Old Town.

In short:

Our Sponsors

All this would not be possible without the generous help of our launch sponsors. If your company would be interested in sponsoring the 17th EuroPython please contact sponsoring@europython.eu.

Sponsoring EuroPython guarantees you highly targeted visibility and the opportunity to present yourself and your company in a professional and innovative environment. You’ll have an unique opportunity to meet many Python-enthusiastic developers, users and professionals. As a sponsor of EuroPython 2018, you will directly help to promote the work of a great open-source community and help further its development.

EuroPython 2018 is the 17th EuroPython conference. The conference tours throughout Europe. It so far has had stops in Belgium, Sweden, Lithuania, United Kingdom, Italy, Germany and Basque Country/Spain, growing from initially 240 attendees to well over 1200.

In the coming days, we will announce the start of the Call for Proposals and Early Bird Ticket sales. Please watch our EuroPython blog for updates.

https://ep2018.europython.eu/

Enjoy,

EuroPython 2018 Team
https://ep2018.europython.eu/
https://www.europython-society.org/

February 21, 2018 01:44 PM


Programiz

Python Variables, Constants and Literals

In this article, you will learn about Python variables, constants, literals and their use cases.

February 21, 2018 08:25 AM


Codementor

Self-Initializing Classes

A basic tutorial about how to use metaclasses and decorators in Python.

February 21, 2018 07:20 AM

Let’s build a Python Twitter bot (Part 1)

Read this development setup tutorial for my three-part-series about building a Twitter bot using Python.

February 21, 2018 05:16 AM

Getting started with Machine Learning using Sklearn-python

This article covers the basics of getting started with Sklearn-python and covers several classification algorithms.

February 21, 2018 04:47 AM

February 20, 2018


Codementor

Testing Celery Tasks

This post is intended to give a brief overview of how to write unit tests for celery tasks.

February 20, 2018 06:42 PM


Stack Abuse

Python Metaclasses and Metaprogramming

Imagine if you could have computer programs that wrote your code for you. It is possible, but the machines will not write all your code for you!

This technique, called metaprogramming, is popular with code framework developers. This is how you get code generation and smart features in many popular frameworks and libraries like Ruby On Rails or TensorFlow.

Functional programming languages like Elixir, Clojure, and Ruby are noted for their metaprogramming capabilities. In this guide, we show you how you can tap into the power of metaprogramming in Python. The code examples are written for Python 3, but will work for Python 2 with some adjustments.

What is a Metaclass in Python?

Python is an object-oriented language that makes working with classes easy.

Metaprogramming in Python relies on a special new type of class that is called the metaclass. This type of class, in short, holds the instructions about the behind-the-scenes code generation that you want to take place when another piece of code is being executed.

Wikipedia sums up metaclasses pretty well:

In object-oriented programming, a metaclass is a class whose instances are classes

When we define a class, the objects of that class are created using the class as the blueprint.

But what about the class itself? What is the blueprint of the class itself?

This is where a metaclass comes in. A metaclass is the blueprint of the class itself, just like a class is the blueprint for instances of that class. A metaclass is a class that defines properties of other classes.

With a metaclass, we can define properties that should be added to new classes that are defined in our code.

For example, the following metaclass code sample adds a hello property to each class which uses this metaclass as its template. This means, new classes that are instances of this metaclass will have a hello property without needing to define one themselves.

# hello_metaclass.py
# A simple metaclass
# This metaclass adds a 'hello' method to classes that use the metaclass
# meaning, those classes get a 'hello' method with no extra effort
# the metaclass takes care of the code generation for us
class HelloMeta(type):  
    # A hello method
    def hello(cls):
        print("greetings from %s, a HelloMeta type class" % (type(cls())))

    # Call the metaclass
    def __call__(self, *args, **kwargs):
        # create the new class as normal
        cls = type.__call__(self, *args)

        # define a new hello method for each of these classes
        setattr(cls, "hello", self.hello)

        # return the class
        return cls

# Try out the metaclass
class TryHello(object, metaclass=HelloMeta):  
    def greet(self):
        self.hello()

# Create an instance of the metaclass. It should automatically have a hello method
# even though one is not defined manually in the class
# in other words, it is added for us by the metaclass
greeter = TryHello()  
greeter.greet()  

The result of running this code is that the new TryHello class is able to printout a greeting that says:

greetings from <class '__main__.TryHello'>, a HelloMeta type class  

The method responsible for this printout is not declared in the declaration of the class. Rather, the metaclass, which is HelloMeta in this case, generates the code at run time that automatically affixes the method to the class.

To see it in action, feel free to copy and paste the code in a Python console. Also, read the comments to better understand what we have done in each part of the code. We have a new object, named greeter, which is an instance of the TryHello class. However, we are able to call TryHello's self.hello method even though no such method was defined in the TryHello class declaration.

Rather than get an error for calling a method that does not exist, TryHello gets such a method automatically affixed to it due to using the HelloMeta class as its metaclass.

Metaclasses give us the ability to write code that transforms, not just data, but other code, e.g. transforming a class at the time when it is instantiated. In the example above, our metaclass adds a new method automatically to new classes that we define to use our metaclass as their metaclass.

This is an example of metaprogramming. Metaprogramming is simply writing code that works with metaclasses and related techniques to do some form of code transformation in the background.

The beautiful thing about metaprogramming is that, rather than output source code, it gives us back only the execution of that code. The end user of our program is unaware of the "magic" happening in the background.

Think about software frameworks that do code generation in the background to make sure you as a programmer have to write less code for everything. Here are some great examples:

Outside Python, other popular libraries such as Ruby On Rails(Ruby) and Boost(C++) are examples of where metaprogramming is used by framework authors to generate code and take care of things in the background.

The result is simplified end-user APIs that automate a lot of work for the programmer who codes in the framework.

Taking care of making that simplicity work behind the scenes, is a lot of metaprogramming baked into the framework source code.

Theory Section: Understanding How Metaclasses Work

To understand how Python metaclasses work, you need to be very comfortable with the notion of types in Python.

A type is simply the data or object nomenclature for an object in Python.

Finding the Type of an Object

Using the Python REPL, let's create a simple string object and inspect its type, as follows:

>>> day = "Sunday"
>>> print("The type of variable day is %s" % (type(day)))
The type of variable day is <type 'str'>  

As you'd expect, we get a printout that variable day is of type str, which is a string type. You can find the type of any object just using the built-in type function with one object argument.

Finding the Type of a Class

So, a string like "Sunday" or "hello" is of type str, but what about str itself? What is the type of the str class?

Again, type in the Python console:

>>> type(str)
<type 'type'>  

This time, we get a printout that str is of type type.

Type and the Type of Type

But what about type itself? What is type's type?

>>> type(type)
<type 'type'>  

The result is, once again, "type". Thus we find that type is not only the metaclass of classes such as int, it's also its own metaclass!

Special Methods Used by Metaclasses

At this point it may help to review the theory a bit. Remember that a metaclass is a class whose instances are themselves classes, and not just simple objects.

In Python 3 you can assign a metaclass to the creation of a new class by passing in the intended masterclass to the new class definition.

The type type, as the default metaclass in Python, defines special methods that new metaclasses can override to implement unique code generation behavior. Here is a brief overview of these "magic" methods that exist on a metaclass:

These are the methods to override in your custom metaclass to give your classes behavior different from that of type, which is the default metaclass.

Metaprogramming Practice 1: Using Decorators to Transform Function Behavior

Let's take a step back before we proceed with using metaclasses metaprogramming practice. A common usage of metaprogramming in Python is the usage of decorators.

A decorator is a function that transforms the execution of a function. In other words, it takes a function as input, and returns another function.

For example, here is a decorator that takes any function, and prints out the name of the function before running the original function as normal. This could be useful for logging function calls, for example:

# decorators.py

from functools import wraps

# Create a new decorator named notifyfunc
def notifyfunc(fn):  
    """prints out the function name before executing it"""
    @wraps(fn)
    def composite(*args, **kwargs):
        print("Executing '%s'" % fn.__name__)
        # Run the original function and return the result, if any
        rt = fn(*args, **kwargs)
        return rt
    # Return our composite function
    return composite

# Apply our decorator to a normal function that prints out the result of multiplying its arguments
@notifyfunc
def multiply(a, b):  
    product = a * b
    return product

You can copy and paste the code into a Python REPL. The neat thing about using the decorator is that the composite function is executed in place of the input function. The result of the above code is that the multiply function announces it is running before its computation runs:

>>> multiply(5, 6)
Executing 'multiply'  
30  
>>>
>>> multiply(89, 5)
Executing 'multiply'  
445  

In short, decorators achieve the same code-transformation behavior of metaclasses, but are much simpler. You would want to use decorators where you need to apply common metaprogramming around your code. For example, you could write a decorator that logs all database calls.

Metaprogramming Practice 2: Using Metaclasses like a Decorator Function

Metaclasses can replace or modify attributes of classes. They have the power to hook in before a new object is created, or after the new object is created. The result is greater flexibility regarding what you can use them for.

Below, we create a metaclass that achieves the same result as the decorator from the prior example.

To compare the two, you should run both examples side by side then follow along with the annotated source code. Note that you can copy the code and paste it straight into your REPL, if your REPL preserves the code formatting.

# metaclassdecorator.py
import types

# Function that prints the name of a passed in function, and returns a new function
# encapsulating the behavior of the original function
def notify(fn, *args, **kwargs):

    def fncomposite(*args, **kwargs):
        # Normal notify functionality
        print("running %s" % fn.__name__)
        rt = fn(*args, **kwargs)
        return rt
    # Return the composite function
    return fncomposite

# A metaclass that replaces methods of its classes
# with new methods 'enhanced' by the behavior of the composite function transformer
class Notifies(type):

    def __new__(cls, name, bases, attr):
        # Replace each function with
        # a print statement of the function name
        # followed by running the computation with the provided args and returning the computation result
        for name, value in attr.items():
            if type(value) is types.FunctionType or type(value) is types.MethodType:
                attr[name] = notify(value)

        return super(Notifies, cls).__new__(cls, name, bases, attr)

# Test the metaclass
class Math(metaclass=Notifies):  
    def multiply(a, b):
        product = a * b
        print(product)
        return product

Math.multiply(5, 6)

# Running multiply():
# 30


class Shouter(metaclass=Notifies):  
    def intro(self):
        print("I shout!")

s = Shouter()  
s.intro()

# Running intro():
# I shout!

Classes that use our Notifies metaclass, for example Shouter and Math, have their methods replaced, at creation time, with enhanced versions that first notify us via a print statement of the name of the method now running. This is identical to the behavior we implemented before using a decorator function.

Metaclasses Example 1: Implementing a Class that can't be Subclassed

Common use cases for metaprogramming include controlling class instances.

For example, singletons are used in many code libraries. A singleton class controls instance creation such that there is only ever at most one instance of the class in the program.

A final class is another example of controlling class usage. With a final class, the class does not allow subclasses to be created. Final classes are used in some frameworks for security, ensuring the class retains its original attributes.

Below, we give an implementation of a final class using a metaclass to restrict the class from being inherited by another.

# final.py

# a final metaclass. Subclassing a class that has the Final metaclass should fail
class Final(type):  
    def __new__(cls, name, bases, attr):
        # Final cannot be subclassed
        # check that a Final class has not been passed as a base
        # if so, raise error, else, create the new class with Final attributes
        type_arr = [type(x) for x in bases]
        for i in type_arr:
            if i is Final:
                raise RuntimeError("You cannot subclass a Final class")
        return super(Final, cls).__new__(cls, name, bases, attr)


# Test: use the metaclass to create a Cop class that is final

class Cop(metaclass=Final):  
    def exit():
        print("Exiting...")
        quit()

# Attempt to subclass the Cop class, this should idealy raise an exception!
class FakeCop(Cop):  
    def scam():
        print("This is a hold up!")

cop1 = Cop()  
fakecop1 = FakeCop()

# More tests, another Final class
class Goat(metaclass=Final):  
    location = "Goatland"

# Subclassing a final class should fail
class BillyGoat(Goat):  
    location = "Billyland"

In the code, we've included class declarations for attempting to subclass a Final class. These declarations fail, resulting in exceptions being thrown. Using a metaclass that restricts subclassing its classes enables us to implement final classes in our codebase.

Metaclasses Example 2: Creating a Class Track Operation Execution Time

Profilers are used to take stock of resource usage in a computing system. A profiler can track things like memory usage, processing speed, and other technical metrics.

We can use a metaclass to keep track of code execution time. Our code example is not a full profiler, but is a proof of concept of how you can do the metaprogramming for profiler-like functionality.

# timermetaclass.py
import types

# A timer utility class
import time

class Timer:  
    def __init__(self, func=time.perf_counter):
        self.elapsed = 0.0
        self._func = func
        self._start = None

    def start(self):
        if self._start is not None:
            raise RuntimeError('Already started')
        self._start = self._func()

    def stop(self):
        if self._start is None:
            raise RuntimeError('Not started')
        end = self._func()
        self.elapsed += end - self._start
        self._start = None

    def reset(self):
        self.elapsed = 0.0

    @property
    def running(self):
        return self._start is not None

    def __enter__(self):
        self.start()
        return self

    def __exit__(self, *args):
        self.stop()


# Below, we create the Timed metaclass that times its classes' methods
# along with the setup functions that rewrite the class methods at
# class creation times


# Function that times execution of a passed in function, returns a new function
# encapsulating the behavior of the original function
def timefunc(fn, *args, **kwargs):

    def fncomposite(*args, **kwargs):
        timer = Timer()
        timer.start()
        rt = fn(*args, **kwargs)
        timer.stop()
        print("Executing %s took %s seconds." % (fn.__name__, timer.elapsed))
        return rt
    # return the composite function
    return fncomposite

# The 'Timed' metaclass that replaces methods of its classes
# with new methods 'timed' by the behavior of the composite function transformer
class Timed(type):

    def __new__(cls, name, bases, attr):
        # replace each function with
        # a new function that is timed
        # run the computation with the provided args and return the computation result
        for name, value in attr.items():
            if type(value) is types.FunctionType or type(value) is types.MethodType:
                attr[name] = timefunc(value)

        return super(Timed, cls).__new__(cls, name, bases, attr)

# The below code example test the metaclass
# Classes that use the Timed metaclass should be timed for us automatically
# check the result in the REPL

class Math(metaclass=Timed):

    def multiply(a, b):
        product = a * b
        print(product)
        return product

Math.multiply(5, 6)


class Shouter(metaclass=Timed):

    def intro(self):
        print("I shout!")

s = Shouter()  
s.intro()


def divide(a, b):  
    result = a / b
    print(result)
    return result

div = timefunc(divide)  
div(9, 3)  

As you can see, we were able to create a Timed metaclass that rewrites its classes on-the-fly. Whenever a new class that uses the Timed metaclass is declared, its methods are rewritten to be timed by our timer utility class. Whenever we run computations using a Timed class, we get the timing done for us automatically, without needing to do anything extra.

Metaprogramming is a great tool if you are writing code and tools to be used by other developers, such as web frameworks or debuggers. With code-generation and metaprogramming, you can make life easy for the programmers that make use of your code libraries.

Suggested Course: Mastering Python

Mastering the Power of Metaclasses

Metaclasses and metaprogramming have a lot of power. The downside is that metaprogramming can get fairly complicated. In a lot of cases, using decorators provides a simpler way to get an elegant solution. Metaclasses should be used when circumstances demand generality rather than simplicity.

To make effective use of metaclasses, we suggest reading up in the official Python 3 metaclasses documentation.

February 20, 2018 02:40 PM


Chris Moffitt

Intro to pdvega - Plotting for Pandas using Vega-Lite

Introduction

Much has been made about the multitude of options for visualizing data in python. Jake VanderPlas covered this topic in his PyCon 2017 talk and the landscape has probably gotten even more confusing in the year since this talk was presented.

Jake is also one of the creators of Altair (discussed in this post) and is back with another plotting library called pdvega. This library leverages some of the concepts introduced in Altair but seeks to tackle a smaller subset of visualization problems. This article will go through a couple examples of using pdvega and compare it to the basic capabilities present in pandas today.

pdvega

Likely sensing the inevitable questions about another plotting library, the pdvega documentation quickly gets to the point about its goals:

pdvega is a library that allows you to quickly create interactive Vega-Lite plots from Pandas dataframes, using an API that is nearly identical to Pandas’ built-in visualization tools, and designed for easy use within the Jupyter notebook.

The basic idea is that pdvega can improve on pandas plot output by adding more interactivity, improving the visual appeal and supporting the declarative Vega-Lite standard. The other nice aspect is that pdvega tries to leverage the existing pandas API so that it is relatively simple to get up and running and produce useful visualizations - especially in the Jupyter notebook environment.

plotting

For this example, I decided to use data from FiveThirtyEight’s Ultimate Halloween Candy Power Ranking post. FiveThirtyEight is gracious enough to make all of its data available here. If you are interested in finding fun data sets to analyze, I encourage you to check it out.

All of the code is meant to be run in a notebook. An example one is available on github.

Make sure the code is installed properly:

pip install pdvega
jupyter nbextension install --sys-prefix --py vega3

Get started by importing pandas and pdvega and reading the csv into a dataframe:

import pandas as pd
import pdvega
df = pd.read_csv("https://github.com/fivethirtyeight/data/blob/master/candy-power-ranking/candy-data.csv?raw=True")

Here’s what the data looks like:

competitorname chocolate fruity caramel peanutyalmondy nougat crispedricewafer hard bar pluribus sugarpercent pricepercent winpercent
0 100 Grand 1 0 1 0 0 1 0 1 0 0.732 0.860 66.971725
1 3 Musketeers 1 0 0 0 1 0 0 1 0 0.604 0.511 67.602936
2 One dime 0 0 0 0 0 0 0 0 0 0.011 0.116 32.261086
3 One quarter 0 0 0 0 0 0 0 0 0 0.011 0.511 46.116505
4 Air Heads 0 1 0 0 0 0 0 0 0 0.906 0.511 52.341465

The data includes voter results on which one of 86 candy options was their favorite. The winpercent column includes how often that candy was the vote winner. The other columns include descriptive characteristics of that candy. The good folks at FiveThirtyEight did lots of analysis, but we’ll do some quick EDA to compare stock pandas plotting vs pdvega.

First, let’s look at the distribution of winning percentages using a histogram.

In pandas:

df["winpercent"].plot.hist()
Pandas histogram

Now in pdvega:

df["winpercent"].vgplot.hist()
Pandas histogram

There are a couple of key points here:

  • The pdvega API is pretty much the same as pandas plotting. Instead of calling plot you can call vgplot .
  • The actual output looks much cleaner in pdvega
  • The png shown here does not replicated the interactivity you get in a notebook

If we want to plot multiple distributions to look at the sugar and price percentiles, it’s fairly simple:

df[["sugarpercent", "pricepercent"]].plot.hist(alpha=0.5)
Pandas histogram

In pdvega, the syntax is a little cleaner because the alpha parameter is not needed.

df[["sugarpercent", "pricepercent"]].vgplot.hist()
Pandas histogram

pdvega supports most of the standard plot types you would expect. Here’s an example of a horizontal bar chart showing the top 15 winpercentages. This fits in seamlessly with the standard pandas approach of sorting and viewing the top entries:

df.sort_values(by=['winpercent'], ascending=False).head(15).vgplot.barh(x='competitorname', y='winpercent')
Pandas histogram

The one challenge I had was figuring out how to make sure the bars were ordered by winpercent not alphabetically but the name. I’m sure there is a way but I could not figure it out.

If we’re interested in looking at a more complicated analysis, the scatter plotting functionality allows us to control the size and color of the plots based on the values in a column. For instance, if we want to look at the relationship between winning percentages, sugar percentiles, pricing percentiles and candy bar status, we can encode that all in a single vgplot.scatter call:

df.vgplot.scatter(x='winpercent', y='sugarpercent', s='pricepercent', c='bar')
Candy Scatter

Once again, the API is similar to panda’s scatter plot but it natively creates a more useful plot without additional tinkering. The ability to easily encode the size of the plot using the s argument for size and c for color is a simple enhancement that makes scatter plots much more useful.

Finally, pdvega supports statistical visualization with pdvega.plotting. A scatter matrix can be a useful tool to view multiple variable interactions in one chart:

pdvega.scatter_matrix(df[["sugarpercent", "winpercent", "pricepercent"]], "winpercent")
Candy scatter matrix

This API is slightly different in that you pass the actual dataframe to the pdvega function but the basic approach is similar to the rest of the pdvega API. The individual plots are linked together so that zooming in one, interacts with the other plot.

Closing Thoughts

There is no doubt that the python visualization landscape is crowded. However, there is a lot of activity in this space and many powerful tools available. In the end, I think the competition is good but hope that some standards eventually emerge so that new users have a simpler time figuring out which tool is best for them. It’s always a concern in the open source world when resources get spread too thin across competing projects and the barrier for new users to learn is high.

That being said, pdvega is very early in its lifecycle but it shows promise. Some of the primary benefits are that it is easy to pick up, generates very nice visualizations out of the box and is primarily developed by Jake VanderPlas who is extremely active in this space and wants to drive some convergence of solutions. In addition, the library should see improvements in functionality as more people use it and generate feedback. I look forward to seeing how it grows and develops in future releases and where it ultimately lands in the crowded visualization space.

February 20, 2018 01:05 PM


Mike C. Fletcher

Bit of PyOpenGL maintenance

So I just pushed a bunch of fixes into the PyOpenGL repository. They're mostly just minor bug-fixes people have reported. There are more bug-reports sitting in the email backlog, but I think I'm done for today.

February 20, 2018 06:32 AM

February 19, 2018


Codementor

Python Dictionary and Dictionary Methods

Dictionaries are unordered data structures that map unique keys to values. The dictionary webstersDict used strings as keys in the dictionary, but dictionary keys can be any immutable data type (numbers, strings, tuples etc). Dictionary values can be just about anything (int, lists, functions, strings, etc).

February 19, 2018 10:26 PM


Doug Hellmann

urllib.parse — Split URLs into Components — PyMOTW 3

The urllib.parse module provides functions for manipulating URLs and their component parts, to either break them down or build them up. Read more… This post is part of the Python Module of the Week series for Python 3. See PyMOTW.com for more articles from the series.

February 19, 2018 02:00 PM


Mike Driscoll

PyDev of the Week: Juan Luis Cano

This week we welcome Juan Luis Cano (@astrojuanlu) as our PyDev of the Week! He is the chair of the Python Spain non-profit and the author of the poliastro project. If you can read Spanish, then you might want to check out his website. Otherwise you can definitely take a look at his Github profile to see what he’s been working on or is interested in. Let’s take some time to learn more about Juan!

Can you tell us a little about yourself (hobbies, education, etc):

I’m an Aerospace engineer with a passion for space, programming and open source. I really enjoy solving Physics or Math problems using computers, and the more my job requires me reading scientific papers and derive mathematical equations, the happier I am. I am an open culture advocate and also chair of the Python Spain non-profit, which organizes the PyCon in our country (I have been involved one way or another since the first edition, in 2013). When I am not in front of the computer, I love to listen to all kinds of music (electronic, ’70s rock, opera, blues), go everywhere with my bicycle and travel.

Why did you start using Python?

One snowy winter morning in my first year of University, our Physics professor wanted us to derive whether it’s better to climb a frozen ramp using the reverse gear of the car or not. However, he gave us the wrong solution and I struggled the whole weekend to understand it, without knowing it was wrong. I wanted to interactively displace the center of gravity of the car and the slope of the ramp without doing the calculations every time and visualizing it in some way, so I bought a 6 month student license of Mathematica. It was totally awesome, but I could not afford buying a complete license and it worked so so on Linux, so I typed “free alternative to Mathematica” and Sagemath appeared. This was 2011, and the rest is history.

What other programming languages do you know and which is your favorite?

My first programming experience was with a game called Tzar. It’s kind of like Age of Empires, but allowed you to create your own campaigns, define events, write dialogs… I copied and pasted $myVar everywhere without knowing what I was doing, but the result was super exciting. After that, I learned ActionScript 3 and Flash to create my own games, HTML and CSS to create my own website, PHP to give it some logic… I have to admit that I somewhat miss the object oriented nature of AS3. Also, they taught us FORTRAN 90 (all caps) in University, but since I started with Python I never had to go back to FORTRAN again. As any engineer out there, I also had to do a thing or two in MATLAB, but I oppose its closed nature and pervasiveness. Overall, my favorite is Python, of course 🙂

What projects are you working on now?

I just finished a freelance project with Boeing and the European Comission to try to predict aircraft trajectories based on historical data using machine learning. In my spare time, I maintain an open source library called poliastro (more on that later) and I dedicate a lot of time to promotion, documentation, prepare talks about it for conferences… I’m also trying to gather data from the Spanish railway company, RENFE, which is a bit challenging because one has to do some ugly web scraping tricks on their website. I love challenges 🙂

Which Python libraries are your favorite (core or 3rd party)?

My favorite libraries are Astropy and SymPy, hands down. The latter constitutes what for me is one of the most successful Python libraries when we talk about new contributors, Google Summer of Code projects… And its LaTeX output mode in the Jupyter notebook is just marvelous. The former is a rare gem in which very talented astronomers have managed to put together a big, complicated library that acts as a foundation to many different projects. I use its unit handling capabilities and kinematic reference frame conversion functions a lot in poliastro.


How did the
poliastro project come about?

In 2013, when I was an Erasmus (visiting) student in Politecnico di Milano, an Italian colleague and I had to optimize an orbital trajectory from the Earth to Venus and then analyze the orbital perturbations of a Venus-orbiting spacecraft. My colleague started his part in MATLAB, and one afternoon he sent me an email with some non-working scripts and gave up. So I rewrote some parts in Python, kept some others in MATLAB (using Octave, since I didn’t have a license), and even included a FORTRAN part as well. This mess worked beautifully in my computer but was probably impossible to install on Windows, so two years later I rewrote all the algorithms in Python using numba, threw the MATLAB and FORTRAN away and became a happy man again. Now the project is more mature: we had funding from the European Space Agency last year, we presented it at the first Open Source Cubesat Workshop that took place at the European Space Operations Centre, and this year we have been accepted as mentoring organization in Google Summer of Code as part of the Open Astronomy umbrella, so I hope I get tons of applications from students!


What lessons have you learned from running this open source project?

There are other projects similar to poliastro, and I’d say that some of them have better algorithms and more features. However, I’d say that none of them have spent so much time on API design, documentation, examples and promotion (relatively to the size of the project, of course). Also, Python is not the fastest language in the world but it gets the job done, it’s easy to learn and it’s trending. My biggest lessons with poliastro have been:

  • Sometimes, language wins over performance.
  • Sometimes, documentation wins over features.
  • Sometimes, marketing wins over quality.

My favorite quote from any Python developer is from John Hunter (may he rest in peace), the creator of matplotlib: “the most important commodity an open source project needs to succeed is users”.  I’ve spent so much time making poliastro easy to install and use, that now it’s the first contact with the language for many users and they’re making insightful suggestions for new features, driving the course of the project and giving me energy to continue.

Is there anything else you’d like to say?

Python and the Open Source/Hardware/Knowledge mentality are now making its way into science, but there’s still a long road ahead in Engineering. I would like to encourage other engineers out there who are tired of paying expensive MATLAB licenses against their will, or tired of endlessly looking for papers that everybody cites and nobody can find, to embrace the Open movement, engage in open source, and make its contribution. Per Python ad astra!

Thanks for doing the interview!

February 19, 2018 01:30 PM


Patrick Kennedy

Steps for Starting a New Flask Project using Python3

Introduction

This blog post describes my current approach to starting a new Flask project. This blog post covers getting up and running with a simple “Hello World” application using the following tools:

This blog post assumes the use of Python 3.4 or greater, as these versions have the ‘venv’ module built-in for creating virtual environments and pip comes automatically installed in the virtual environment that gets created.

Structure

The easiest way to create a Flask application is to put everything into a single file. This is my preferred approach for starting a new Flask project, as it proves out the environment that is being created.

Start by creating a new directory and changing into that directory:

$ mkdir flask_simple_application
$ cd flask_simple_application/

At this point, you will likely see that you have at least two version of python available on your system:

$ python --version
Python 2.7.10
$ python3 --version
Python 3.6.3

I strongly recommend using Python3 for all new projects, as it has tons of new features and the support for Python2 (legacy Python) is ending in 2020 (countdown clock).

Virtual Environments

For even the simplest projects, creating a virtual environment is a great idea. Virtual environments create an isolated environment for each project, which means that each project can have its own dependencies (ie. version of the Python interpreter, imported python packages, etc.).

Virtual environments really became beneficial to me when I was working on two simultaneous projects:

  1. An existing project required using Python2, as a needed package only supported Python2
  2. A new project where I wanted to use Python3

Having separate virtual environments for each project allowed me to easily configure which version of Python to use.

Since this project is using Python3, there is a built-in module called venv that can be used for creating virtual environments. Note: the venv module was added in Python 3.3.

Here’s the command for creating a new virtual environment, where the directory to create the virtual environment in is specified as ‘venv’. NOTE: make sure to use ‘python3’ instead of ‘python’ to guarantee that Python3 will be used in the virtual environment.

$ python3 -m venv venv

This command created a new directory called ‘venv’ which contains the following items to keep this project isolated from the other projects on your system:

To start using the virtual environment that was created, it needs to be activated:

$ source venv/bin/activate
(venv) $

After activating the virtual environment, the virtual environment name gets displayed on your command prompt (the ‘(venv)’ at the left of the prompt).

Since the virtual environment is active, running the python interpreter will use Python3!

(venv) $ python --version
Python 3.6.3

Installing Packages with pip

After creating the virtual environment for your application, it is time to start installing the Python packages that you need using pip. If you are using Python 3.4 or greater, pip gets automatically installed in your virtual environment.

Start by installing the Flask framework, which is the ‘flask’ package:

pip install flask

This command installs the Flask package plus all of its dependencies (ie. other packages needed by Flask). Here’s a sample of all the packages that were installed:

$ pip freeze
click==6.7
Flask==0.12.2
itsdangerous==0.24
Jinja2==2.10
MarkupSafe==1.0
Werkzeug==0.14.1

It’s a good idea to save the packages (including version numbers) that are being used in the project:

$ pip freeze > requirements.txt

This can be really beneficial if you or someone else is starting with this project and wants to quickly install all these packages with ‘pip install -r requirements.txt’.

Writing Python Code

Now it’s time to actually write some Python code! In the top-level directory that was created, create a new file called app.py with the following content:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
return "Hello World!"

This file creates a new instance of the Flask application and defines a single route that returns the “Hello World!” statement. This is the most basic Flask application, but it will show that everything has been configured properly.

Running the Development Server

In order to check that app.py is working correctly, the Flask development server that comes bundled with the Flask framework is a great way to test out your application.

First, you need to specify where the Flask application is defined (via the FLASK_APP environment variable). Then, you can run your Flask application with ‘flask run’:

(venv) $ export FLASK_APP=app.py
(venv) $ flask run
* Serving Flask app "app"
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

As the comment states, you can now navigation to http://127.0.0.1:5000/ (this is mapped to ‘localhost’ on most systems, so you can alternatively navigate to ‘localhost:5000’) to view your application:

Structure of the Flask Application

At this point, the Flask application should look like this:

$ tree -L 2
.
├── app.py
├── requirements.txt
└── venv
    ├── bin
    ├── ...

Conclusion

This blog post provided the steps that I follow for creating a new Flask application using Python 3. In an upcoming blog post, I’ll describe how I like to structure a Flask application.

February 19, 2018 04:54 AM


Fabio Zadrozny

Python with PyDev on Visual Studio Code

PyDev can now be used for Python development on Visual Studio Code!

The first release already provides features such as code analysis, code completion, go to definition, symbols for the workspace and editor, code formatting, find references, quick fixes and more (see http://www.pydev.org/vscode/ for details).

All features have a strong focus on speed and have been shaped by the usage on PyDev over the last 14 years, so, I believe it's already pretty nice to use... there are still some big things to integrate (such as the PyDev debugger), but those should come on shortly.

The requisites are having java 8 (or higher) installed on the system (if it doesn't find it automatically the java home location may need to be specified in the settings -- http://www.pydev.org/vscode/ has more details) and Python 2.6 or newer.

By default it should pick the python executable available on the PATH, but it's possible to specify a different python executable through the settings on VSCode (see http://www.pydev.org/vscode/settings.html for details).

Below, I want to share some of the things that are unique in PyDev and are now available for VSCode users:
Now, the extension itself is not currently open source as PyDev... my target is making the best Python development environment around and all earnings will go towards that (as a note, all improvements done to PyDev itself will still be open source, so, most earnings from PyDev on VSCode will go toward open source development).

Enjoy!

February 19, 2018 03:10 AM


Amit Saha

Detecting RLO character in Python

At work, I learned about how Right-to-Left Override was being used to make actually malicious files to look harmless. For example, a .exe file was being made to appear as .doc files. We didn't want to allow uploading such files. This meant that I nedded to detect the presence of the RLO character in the filename.

Then, I came across this post, where I learned about unicode bidirectional class and Python's bidirectional() method.

The final solution for detection looked like this:

import unicodedata
..
filename = 'arbitrary_filename.doc'
if 'RLO' in [unicodedata.bidirectional(c) for c in unicode(filename)]:
    raise ValueError('Invalid character in one or more of the file names')
..

February 19, 2018 01:30 AM