skip to navigation
skip to content

Planet Python

Last update: October 12, 2016 01:49 PM

October 12, 2016


PyPy Development

PyPy3 5.5.0 released

We're pleased to announce the release of PyPy3 v5.5.0. Coming four months after PyPy3.3 v5.2, it improves compatibility with Python 3.3 (3.3.5). We strongly recommend updating from previous PyPy3 versions.

We would like to thank all of the people who donated to the py3k proposal for supporting the work that went into this release.

You can download the PyPy3.3 v5.5.0 release here: http://pypy.org/download.html

Towards Python 3.5

We have started to work on Python 3.5, which is a version used by many software projects. It seems to get wide adoption. We are happy to be part of the Mozilla Open Source Support (MOSS) initiative.

Nevertheless we want to give our users the chance to use PyPy in their Python 3 projects, thus we have prepared this release.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.10 and 3.3.5. It's fast due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This release supports:
Please try it out and let us know what you think. We welcome feedback, we know
you are using PyPy, please tell us about it!

Cheers

The PyPy Team

October 12, 2016 10:37 AM


Giampaolo Rodola

Virtualize OSX on Linux

Preamble

I've been a Linux user for something like 10 years now. In order to develop and maintain psutil on different platforms I've been using the excellent VirtualBox. With it, during the years, I've been able to virtualize different versions of Windows, FreeBSD, OpenBSD, NetBSD and Solaris and implement and protract support for such platforms inside psutil. Without VirtualBox there really wouldn't exist psutil as it stands nowadays.
At some point I also managed to virtualize OSX by using an hacked version of OSX called iDeneb which is based on OSX 10.5 / Leopard (note: 9 years old), and that is what I've been using up until today. Of course such an old hacked version of OSX isn't nice to deal with. It ships Python 2.5, it kernel panicks, I had to reinstall it from scratch quite often and people (rightfully) make fun of you.  Thank you iDeneb: I loved and hated you at the same time. =)

Vagrant

I'm really not sure how I could  have been missing this for all this time, but it turns out emulating OSX on Linux really is as easy as executing a one-liner:
vagrant init AndrewDryga/vagrant-box-osx; vagrant up
And that really is it! I mean... you're literally good to go and start developing! That will create a Vagrant file, download a pre-configured OSX image via internet (10GB or something) and finally run it in VirtualBox. The whole package includes:
In a couple of hours I modified the original Vagrantfile a little and managed to mount a directory which is shared between the VM and the host (my laptop) and ended up with this Vagrantfile.

Development workflow

This is the second best part, and it represents why Vagrant is such a great tool. So far in order to virtualize a platform and develop on it I had to:
  • download the platform ISO (BSD, SunOS or whatever)
  • manually install it in VirtualBox
  • install deps, configure the platform, setup SSHd
In order to automatize the system configuration I came up with a bunch of automation scripts but that really is another story.  With that done, every time I wanted to work on psutil I had to:
  • open VirtualBox (GUI)
  • "get into it" (by clicking with the mouse), open a terminal, get the IP address
  • get out of VirtualBox, open a local terminal, SSH into the VM
  • open the GUI file manager, SFTP into the VM, open the source files with my editor (Sublime), finally start working
With Vagrant the workflow becomes much more easier as no GUI is involved. Everything happens in the terminal (2 actually):
  • I install and run the VM with "vagrant up"
  • I SSH into it with "vagrant ssh" in one terminal (without knowing any IP)
  • I edit files locally in another terminal 
That's it. The best part is that Vagrant (or VirtualBox?) does the necessary magic so that a local directory (the one where you have your source files) will be mapped and visible inside the VM. With that you can edit the source files locally without ever connecting to the VM via an IP address, SFTP or whatever.
I will likely get rid of my other VMs installed "manually" and do the same: rely on Vagrant. Actually I already did for OpenBSD and NetBSD but I'm still not happy with what i have. In summary, I think this is a great improvement in terms of productivity, and I wish I had discovered this sooner.

October 12, 2016 05:39 AM

October 11, 2016


François Dion

PyData Carolinas 2016 Tutorial: Datascience on the web

PyData Carolinas 2016

Don Jennings and I presented a tutorial at PyData Carolinas 2016: Datascience on the web.

The plan was as follow:

Description

Learn to deploy your research as a web application. You have been using Jupyter and Python to do some interesting research, build models, visualize results. In this tutorial, you’ll learn how to easily go from a notebook to a Flask web application which you can share.

Abstract

Jupyter is a great notebook environment for Python based data science and exploratory data analysis. You can share the notebooks via a github repository, as html or even on the web using something like JupyterHub. How can we turn the work we have done in the notebook into a real web application?
In this tutorial, you will learn to structure your notebook for web deployment, how to create a skeleton Flask application, add a model and add a visualization. While you have some experience with Jupyter and Python, you do not have any previous web application experience.
Bring your laptop and you will be able to do all of these hands-on things:
  1. get to the virtual environment
  2. review the Jupyter notebook
  3. refactor for reuse
  4. create a basic Flask application
  5. bring in the model
  6. add the visualization
  7. profit!
Now that is has been presented, the artifacts are a github repo and a youtube video.

Github Repo


https://github.com/fdion/pydata/

After the fact

The unrefactored notebook is here while the refactored one is here.
Once you run through the whole refactored notebook, you will have train and test sets saved in data/ and a trained model in trained_models/. To make these available in the tutorial directory, you will have to run the publish.sh script. On a unix like environment (mac, linux etc):
chmod a+x publish.sh
./publish.sh

Video

The whole session is now on youtube: Francois Dion & Don Jennings Datascience on the web

Francois Dion
@f_dion

October 11, 2016 11:27 PM


Mike Driscoll

How to Create a Diff of an Image in Python

For the past couple of years, I’ve been writing automated tests for my employer. One of the many types of tests that I do is comparing how an application draws. Does it draw the same way every single time? If not, then we have a serious problem. An easy way to check that it draws the same each time is to take a screenshot and then compare it to future versions of the same drawing when the application gets updated.

The Pillow library provides a handy tool for this sort of thing that is called ImageChops. If you don’t already have Pillow, you should go install it now so you can follow along with this short tutorial.


Comparing Two Images

The first thing we need to do is find two images that are slightly different. You can create your own by using burst mode on your camera and taking a bunch of photos of animals as they move, preferably while using a tripod. Or you can take an existing photo and just add some kind of overlay, such as text. I’m going to go with the latter method. Here is my original photo of Multnomah Falls in Oregon:

multnomah_falls

And here’s the modified version where I added some text to identify the location of the photo:

multnomah_falls_text

Now let’s use ImageChops to find the difference for us!

import Image
import ImageChops
 
def compare_images(path_one, path_two, diff_save_location):
    """
    Compares to images and saves a diff image, if there
    is a difference
 
    @param: path_one: The path to the first image
    @param: path_two: The path to the second image
    """
    image_one = Image.open(path_one)
    image_two = Image.open(path_two)
 
    diff = ImageChops.difference(image_one, image_two)
 
    if diff.getbbox() is None:
        # There is no difference between the images
        return
    else:
        diff.save(diff_save_location)
 
if __name__ == '__main__':
    compare_images('/path/to/multnomah_falls.jpg',
                   '/path/to/multnomah_falls_text.jpg',
                   '/path/to/diff.jpg')

Here we have a simple function that we can use to find differences in images. All you need to do is pass it three paths! The first two paths are for the images that we want to compare. The last path is where to save the diff image, if we find a diff. For this example, we should definitely find a diff and we did. Here’s what I got when I ran this code:

diff


Wrapping Up

The Pillow package has many amazing features for working with images. I enjoy photography so it’s fun to be able to take photos and then use my favorite programming language to help me do neat things with the results. You should give this a try as well and read the Pillow documentation to see what else you can do with this clever package!


Related Reading

October 11, 2016 05:15 PM


Machinalis

Machinalis hired DRF's Tom Christie!

If you're into Python web development you've probably found yourself looking for a way to easily implement REST APIs on your backend, and I’m quite sure you found that one of the most solid and robust solutions is Django REST Framework (“DRF” from now on). DRF has a simple learning curve and nice API, integrates really well with other Django apps (both if your data comes from the ORM or elsewhere, is really fast and reliable, it's very well documented, the online community is helpful and friendly and it has a healthy ecosystem of tools, both built-in and as add-on packages). I don't want to bore you with the very long list of good things about it, but in short, it's the default solution we grab at Machinalis for many of our projects, both for internal ones and for applications we build for our clients (if you didn’t know, companies hire us to build complex web apps and APIs, knowing that our team has a high seniority in Python + Django).

DRF is an Open Source project, which for a lot of people means “it's free”, but in reality it means “someone else is paying for it”, typically with volunteered contributor time. In the case of DRF, even if there's a community of contributors, the heaviest hauler by quite a long distance is Tom Christie, who is also DRF's original author. DRF is brilliant mostly because of Tom's work, which has largely come from volunteer time (with some support from his former employers).

So now, I'm really proud to announce that Machinalis has hired Tom Christie to work on DRF. For a company like ours, ensuring the continuity of a critical piece of our software infrastructure is a must, and we know that investing on it will bring features and bug fixes that will make us more effective at providing software development services, so it's an obvious choice.

But wait... YOU should ALSO hire him!

Our agreement with him is non-exclusive. In fact, many companies and individuals have opted for different tiers of sponsorship, some of them starting as low as $15 US dollars. Tom left his engineering position as an employee in May 2016 and decided to arrange this funding program. If you use DRF in any professional capacity, you probably should join us and hire Tom Christie too. We all depend on Open Source and these support programmes are an important step in making it sustainable.

I hope the title of this post hasn't misled you. If you hire Machinalis to build your site or your API, you'll get to work with a team of amazing people and engineers which won't include Tom. However, his job will still be building an essential part of your product, and now we can count on it.

October 11, 2016 04:08 PM


Continuum Analytics News

Move Over Data Keepers: Open Source is Here to Stay

Tuesday, October 11, 2016
Michele Chambers
EVP Anaconda Business Unit & CMO
Continuum Analytics

If you walked by our booth at Strata + Hadoop 2016 this year, you may have laid witness to a larger-than-life character, the almighty data keeper. He was hard to miss as he stood seven-feet tall wearing floor length robes.

For those who weren’t in attendance, the natural thing to ask is: who is this data keeper, and what does he do? Let me explain. 

Before the creation of open source and the Anaconda platform, data was once locked away and hidden from the world—protected by the data keepers. These data keepers were responsible for maintaining data and providing access only to those who could comprehend the complicated languages such as base SAS®. 

For years, this exclusivity kept data from penetrating the outside world and allowing it to be used for good. However, as technology advanced, data began to become more accessible, taking power away from the data keepers, and giving it instead to the empowered data scientists and eventually to citizen data scientists. This technology movement is referred to as the “Open Data Science revolution”—resulting in the creation of an open source world that allows everyone and anyone to participate and interact with data. 

As the Open Data Science community began to grow, members joined together to solve complex problems by utilizing different tools and languages. This collaboration is what enabled the creation of the Anaconda platform. Anaconda is currently being used by millions of innovators from all over the world in diverse industries (from science to business to healthcare) to come up with solutions to make the world a better place. 

Thanks to open source, data keepers are no longer holding data under lock and key—data is now completely accessible, enabling those in open source communities the opportunity to utilize data for good. 

To learn more, watch our short film below. 

 

October 11, 2016 03:34 PM

Continuum Analytics Launches AnacondaCrew Partner Program to Empower Data Scientists with Superpowers

Wednesday, October 12, 2016

Leading Open Data Science platform company is working with its ecosystem to drive transformation in a data-enabled world

AUSTIN, TEXAS—October 12, 2016—Continuum Analytics, the creator and driving force behind Anaconda, the leading Open Data Science platform powered by Python, today announced the launch of the AnacondaCrew Partner Program. Following the recent momentum in revenue and Anaconda growth of over three million downloads, this program is focused on enabling partners to leverage the power of Open Data Science technology to accelerate time-to-value for enterprises by empowering modern data science teams. 

The AnacondaCrew program is designed to drive mutual business growth and financial performance for Technology, Service and OEM partners to take advantage of the fast-growing data science market. 

The AnacondaCrew Partner Program includes:

  • Technology Partners offering a hardware platform, cloud service or software that integrates with the Anaconda platform
  • Service Partners delivering Anaconda-based solutions and services to enterprises
  • OEM Partners using Anaconda, an enterprise-grade Python platform, to embed into their application, hardware or appliance

"We are extremely excited about our training partnership with Continuum Analytics,” said Jonathan Cornelissen, CEO at DataCamp. “We can now combine the best instructors in the field with DataCamp’s interactive learning environment to create the new standard for online Python for Data Science learning."

In the last year, Continuum has quickly grown the AnacondaCrew Partner Program to include a dozen of the best known modern data science partners in the ecosystems, including Cloudera, DataCamp, Intel, Microsoft, NVIDIA, Docker and others

“As a market leader, Anaconda is uniquely positioned to embrace openness through the open source community and a vast ecosystem of partners focused on helping customers solve problems that change the world,” said Michele Chambers, EVP of Anaconda and CMO at Continuum Analytics. “Our fast growing AnacondaCrew Partner Program delivers an enterprise-ready connected ecosystem that makes it easy for customers to embark on the journey to Open Data Science and realize returns on their Big Data investments.”

Learn more about the Continuum Analytics AnacondaCrew Partner Program here or simply get in touch

About Anaconda Powered by Continuum Analytics

Continuum Analytics is the creator and driving force behind Anaconda, the leading Open Data Science platform powered by Python. We put superpowers into the hands of people who are changing the world. 

With more than 3M downloads and growing, Anaconda is trusted by the world’s leading businesses across industries––financial services, government, health & life sciences, technology, retail & CPG, oil & gas––to solve the world’s most challenging problems. Anaconda does this by helping everyone in the data science team discover, analyze and collaborate by connecting their curiosity and experience with data. With Anaconda, teams manage their Open Data Science environments without any hassles to harness the power of the latest open source analytic and technology innovations. 

Continuum Analytics' founders and developers have created and contributed to some of the most popular Open Data Science technologies, including NumPy, SciPy, Matplotlib, Pandas, Jupyter/IPython, Bokeh, Numba and many others. Continuum Analytics is venture-backed by General Catalyst and BuildGroup. 

To learn more, visit http://www.continuum.io.

###

Media Contact:
Jill Rosenthal
InkHouse
continuumanalytics@inkhouse.com 

October 11, 2016 02:31 PM

Esri Selects Anaconda to Enhance GIS Applications with Open Data Science

Wednesday, October 12, 2016

Streamlined access to Python simplifies and accelerates development of deep location-based analytics for improved operations and intelligent decision-making

AUSTIN, TEXAS—October 12, 2016—Continuum Analytics, the creator and driving force behind Anaconda, the leading Open Data Science platform powered by Python, today announced a new partnership with leading geographic information system (GIS) provider, Esri. By embedding Anaconda into Esri's flagship desktop application, ArcGIS Pro, organizations now have increased accessibility to Python for developing more powerful, location-centric analytics applications. 

The integration of Anaconda into ArcGIS Pro 1.3 enables GIS professionals to build detailed maps using the most current data and perform deep analysis to apply geography to problem solving and decision making for customers in dozens of industries. Anaconda provides a strong value-added component––particularly for Esri’s scientific and governmental customers who have to coordinate code across multiple machines or deploy software through centralized IT systems. Now, developers using ArcGIS Pro can easily integrate open source libraries into projects, create projects in multiple versions of Python and accelerate the process of installing nearly all publicly available Python packages.

“Python has a rich ecosystem of pre-existing code packages that users can leverage in their own script tools from within ArcGIS. But, managing packages can prove complex and time-consuming, especially when developing for multiple projects at once or trying to share code with others,” said Debra Parish, manager of global business strategies at Esri. “Anaconda solves these challenges and lets users easily create projects in multiple versions of Python. It really makes lives easier, especially for developers who deal with complex issues and appreciate the ease and agility Anaconda adds to the Python environment.”

ArcGIS for Desktop, which includes ArcGIS Pro, boasts the most powerful mapping software in the world. Used by Fortune 500 companies, national and local governments, public utilities and tech start-ups around the world, ArcGIS Pro’s mapping platform uncovers trends, patterns and spatial connections to provide actionable insights leading to data-informed business decisions. Additionally, ArcGIS Pro is accessible to developers to create and manage geospatial apps, regardless of developer experience.

“At Continuum Analytics, we know that data science is a team sport and collaboration is critical to the success of any analytics project. Anaconda empowers Esri developers with an accelerated path to open source Python projects and deeper analytics,” said Travis Oliphant, CEO and co-founder at Continuum Analytics. “More importantly, we see this as a partnering of two great communities, both offering best-in-class technology and recognizing that Open Data Science is a powerful solution to problem solving and decision making for organizations of all sizes.”

About Esri

Since 1969, Esri has been giving customers around the world the power to think and plan geographically. As the market leader in GIS technology, Esri software is used in more than 350,000 organizations worldwide including each of the 200 largest cities in the United States, most national governments, more than two-thirds of Fortune 500 companies, and more than 7,000 colleges and universities. Esri applications, running on more than one million desktops and thousands of web and enterprise servers, provide the backbone for the world's mapping and spatial analysis. Esri is the only vendor that provides complete technical solutions for desktop, mobile, server, and Internet platforms. Visit us at esri.com/news.

Copyright © 2016 Esri. All rights reserved. Esri, the Esri globe logo, GIS by Esri, Story Map Journal, esri.com, and @esri.com are trademarks, service marks, or registered marks of Esri in the United States, the European Community, or certain other jurisdictions. Other companies and products or services mentioned herein may be trademarks, service marks, or registered marks of their respective mark owners.

About Anaconda Powered by Continuum Analytics

Continuum Analytics is the creator and driving force behind Anaconda, the leading Open Data Science platform powered by Python. We put superpowers into the hands of people who are changing the world. 

With more than 3M downloads and growing, Anaconda is trusted by the world’s leading businesses across industries––financial services, government, health & life sciences, technology, retail & CPG, oil & gas––to solve the world’s most challenging problems. Anaconda does this by helping everyone in the data science team discover, analyze and collaborate by connecting their curiosity and experience with data. With Anaconda, teams manage their Open Data Science environments without any hassles to harness the power of the latest open source analytic and technology innovations. 

Continuum Analytics' founders and developers have created and contributed to some of the most popular Open Data Science technologies, including NumPy, SciPy, Matplotlib, Pandas, Jupyter/IPython, Bokeh, Numba and many others. Continuum Analytics is venture-backed by General Catalyst and BuildGroup. 

To learn more, visit http://www.continuum.io.

###

Media Contacts:
Anaconda--
Jill Rosenthal
InkHouse
continuumanalytics@inkhouse.com

Esri--
Karen Richardson, Public Relations Manager, Esri
Mobile: +1 587-873-0157
Email: krichardson@esri.com

October 11, 2016 02:27 PM


Brett Cannon

What to look for in a new TV

I'm kind of an A/V nerd. Now I'm not hardcore enough to have a vinyl collection or have an amp for my TV, but all my headphones cost over $100 and I have a Sonos Playbar so I don't have to put up with crappy TV speakers. What I'm trying to say is that I care about the A/V equipment I use, but not to the extent that money is no object when it comes to my enjoyment of a movie (I'm not that rich and my wife would kill me if I spent that kind of money on electronics). That means I tend to research extensively before making a major A/V purchase since I don't do it very often and I want quality within reason which does not lend itself to impulse buying.

Prior to September 1, 2016, I had a 2011 Vizio television. It was 47", did 1080p, and had passive 3D. When I purchased the TV I was fresh out of UBC having just finished my Ph.D. so it wasn't top-of-the-line, but it was considered very good for the price. I was happy with the picture, but admittedly it wasn't amazing; the screen had almost a matte finish which led to horrible glare. I also rarely used the 3D in the television as 3D Blu-Ray discs always cost extra and so few movies took the time to actually film in 3D to begin with, instead choosing to do it in post-production (basically animated films and TRON: Legacy were all that we ever watched in 3D). And to top it all off, the TV took a while to turn on. I don't know what kind of LCB bulbs were in it, but they took forever to warm up and just annoyed me (yes, very much a first-world problem).

So when UHD came into existence I started to keep an eye on the technology and what television manufacturers were doing to incorporate the technology to entice people like me to upgrade. After two years of watching this space and one of the TVs I was considering having a one-day sale that knock 23% off the price, I ended up buying a 55" Samsung KS8000 yesterday. Since I spent so much time considering this purchase I figured I would try and distill what knowledge I have picked up over the years into a blog post so that when you decide to upgrade to UHD you don't have to start from zero knowledge like I did.

What to care about

First, you don't care about the resolution of the TV. All UHD televisions are 4K, so that's just taken care of for you. It also doesn't generally make a difference in the picture because most people sit too far away from their TV to make the higher resolution matter.

No, the one thing you're going to care about is HDR and everything that comes with it. And of course it can't be a simple thing to measure like size or resolution. Oh no, HDR has a bunch of parts to it that go into the quality of the picture: brightness, colour gamut, and format (yes, there's a format war; HD-DVD/Blu-Ray didn't teach the TV manufacturers a big enough lesson).

Brightness

A key part of HDR is the range of brightness to show what you frequently hear referred to as "inky blacks" and "bright whites". The way you get deep blacks and bright whites is by supporting a huge range of brightness. What you will hear about TVs is what their maximum nit is. Basically you're aiming for 1000 nits or higher for a maximum and as close to 0 as possible for a minimum.

Now of course this isn't as simple as it sounds as there's different technology being used to try and solve this problem.

LCD

Thanks to our computers I'm sure everyone reading this is familiar with LCD displays. But what you might not realize is how they exactly work. In a nutshell there are LED lightbulbs behind your screen that provides white light, and then the LCD pixels turn on and off the red/green/blue parts of themselves to filter out certain colours. So yeah, there are lightbulbs in your screen and how strong they are dictates how bright your TV screen will be.

Now the thing that comes into play here for brightness is how those LED bulbs are oriented in order to get towards that 0 nits for inky blacks. Typical screens are edge-list, which means there is basically a strip of LEDs on the edges of the TV that shine light towards the middle of the screen. This is fine and it's what screens have been working with for a while, but it does mean there's always some light behind the pixels so it's kind of hard to keep it from leaking out a little bit.

This is where local dimming comes in. Some manufacturers are now laying out the LED bulbs in an array/grid behind the screen instead of at the edges. What this allows is for the TV to switch dim an LED bulb if it isn't needed at full strength to illuminate a certain quadrant of the screen (potentially even switching off entirely). Obviously the denser the array, the more local dimming zones and thus the greater chance a picture with some black in it will be able to switch off an LED to truly get a dark black for that part of the screen. As for how often something you're watching is going to allow you to take advantage of such local dimming due to a dark area lining up within a zone is going to vary so this is going to be a personal call as to whether this makes a difference to you.

OLED

If I didn't have a budget and wanted the ultimate solution for getting the best blacks in a picture, I would probably have an OLED TV from LG. What makes these TVs so great is the fact that OLEDs are essentially pixels that provide their own light. What that means is if you want an OLED pixel to be black, you simply switch it off. Or to compare it to local dimming, it's as if every pixel was its own local dimming zone. So if you want truly dark blacks, OLED are the way to go. It also leads to better colours since the intensity of the pixel is consistent compared to an LCD where the brightness is affected by how far the pixel is from the LED bulb that's providing light to the pixel.

But the drawback is that OLED TVs only get so bright. Since each pixel has to generate its own light they can't reach really four-digit nit levels like the LCD TVs can. It's still much brighter than any HD TV, but OLED TVs don't match the maximum brightness of the higher-end LCD TVs.

So currently it's a race to see if LCDs can get their blacks down or if OLEDs can get their brightness up. But from what I have read, in 2016 your best bet is OLED if you can justify the cost to yourself (they are very expensive televisions).

Colour gamut

While having inky blacks and bright whites are nice, not everyone is waiting for Mad Max: Fury Road in black and white. That means you actually care about the rest of the rainbow, which means you care about the colour gamut of the TV for a specific colour space. TVs are currently trying to cover as much of the DCI-P3 colour space as possible right now. Maybe in a few years TVs will fully cover that colour space, at which point they will start worrying about Rec. 2020 (also called BT.2020), but there's still room in covering DCI-P3 before that's something to care about.

In the end colour gamut is probably not going to be something you explicitly shop for, but more of something to be aware of that you will possibly gain by going up in price on your television.

Formats

So you have your brightness and you have your colours, now you have to care about what format all of this information is stored in. Yes my friends, there's a new format war and it's HDR10 versus Dolby Vision. Now if you buy a TV from Vizio or LG then you don't have to care because they are supporting both formats. But if you consider any other manufacturer you need to decide on whether you care about Dolby Vision because everyone supports HDR10 these days but no one supports Dolby Vision at the moment except those two manufacturers.

There is one key reason that HRD10 is supported by all television makers: it's an open specification. By being free it doesn't cut into profits of TVs which obviously every manufacturer likes and is probably why HDR10 is the required HDR standard for Ultra Blu-Ray discs (Dolby Vision is supported on Ultra Blu-Ray, but not required). Dolby Vision, on the other hand, requires licensing fees paid to Dolby. Articles also consistently suggest that Dolby Vision requires new hardware which would also drive up costs of supporting Dolby Vision (best I can come up with is that since Dolby Vision is 12-bit and HDR10 is 10-bit that TVs typically use a separate chip for Dolby Vision processing).

Dolby Vision does currently have two things going for it over HDR10. One is that Dolby Vision is dynamic per frame while HDR10 is static. This is most likely a temporary perk, though, because HDR10 is gaining dynamic support sometime in the future.

Two is that Dolby Vision is part of an end-to-end solution from image capture to projection in the theatres. By making Dolby Vision then also work at home it allows for directors and editors to get the results they want for the cinema and then just pass those results along to your TV without extra work.

All of this is to say that Dolby Vision seems to be the better technology, but the overhead/cost of adding it to a TV along with demand will ultimately dictate whether it catches on. Luckily all TV manufacturers has agreed on the minimum standard of HDR10 so you won't be completely left out if you buy a TV from someone other than LG or Vizio.

Where to go for advice

When it comes time to buy a TV, I recommend Rtings.com for advice. They have a very nice battery of tests they put the TV through and give you nice level of detail on how they reached their scores for each test. They even provide the settings they used for their tests so you can replicate them at home.

You can also read what the Wirecutter is currently recommending. For me, though, I prefer Rtings.com and use the Wirecutter as a confirmation check if their latest TV round-up isn't too out-of-date.

Ultra HD Premium

If you want a very simple way to help choose a television, you can simply consider ones that are listed as Ultra HD Premium. That way you know the TV roughly meets a minimum set of specifications that are reasonable to want if you're spending a lot of money on a TV. The certification is new in 2016 and so there are not a ton of TVs yet that have the certification, but since TV manufacturers like having stamps on their televisions I suspect it will start to become a thing.

One thing to be aware of is that Vizio doesn't like the certification. Basically they have complained that the lack of standards around how to actually measure what the certification requires makes it somewhat of a moot point. That's a totally reasonable criticism and why using the certification as a filter for TVs consider is good, but to not blindly buy a TV just because it has Ultra HD Premium stamp of approval.

Why I chose my TV

Much like when I bought a soundbar, I had some restrictions placed upon me when considering what television I wanted. One, the TV couldn't be any larger than 55" (to prevent the TV from taking over the living room even though we should have a 65" based on the minimum distance people might sit from the TV). This immediately put certain limits on me as some model lines don't start until 65" like the Vizio Reference series. I also wasn't willing to spend CAD 4,000 on an LG, so that eliminated OLED from consideration. I also wanted HDR, so that eliminated an OLED that was only HD.

In the end it was between the 55" Samsung KS8000, 55" Vizio P-series, and the 50" Vizio P-series. The reason for the same Vizio model at different sizes is the fact that they use different display technology; the 50" has a VA display while the 55" has an IPS display. The former will have better colours but the latter has better viewing angles. Unfortunately I couldn't find either model on display here in Vancouver to see what kind of difference it made.

One other knock against the Vizio -- at least at 55" -- was that it wasn't very good in a bright room. That's a problem for us as our living room is north facing with a big window and the TV is perpendicular to those windows, so we have plenty of glare on the screen as the sun goes down. The Samsung, on the other hand, was rated to do better in a glare-heavy room. And thanks to a one-day sale it brought the price of the Samsung to within striking distance of the Vizio. So in the end with the price difference no longer a factor I decided to go with the TV that worked best with glare and maximized the size I could go with.

My only worry with my purchase is if Dolby Vision ends up taking hold and I get left in the cold somehow. But thanks to the HDR10 support being what Ultra Blu-Ray mandates I'm not terribly worried of being shut out entirely from HDR content. There's also hope that I might be able to upgrade my television in the future thanks to it using a Mini One Connect which breaks out the connections from the television. In other TVs the box is much bigger as it contains all of the smarts of the television, allowing future upgrades. There's a chance I will be able to upgrade the box to get Dolby Vision in the future, but that's just a guess at this point that it's even possible, let alone whether Samsung choose to add Do

It's been 48 hours with the TV and both Andrea and I are happy with the purchase; me because the picture is great, Andrea because I will now shut up about television technology in regards to a new TV purchase.

October 11, 2016 04:44 AM

Introducing Which Film

What I'm announcing

Today I'm happy to announce the public unveiling of Which Film! I'll discuss how the site came about and what drives it, but I thought I would first explain what it does: it's a website to help you choose what movie you and your family/friends should watch together. What you do is you go to the site, enter in the Trakt.tv usernames of everyone who wants to watch a film together (so you need at least two people and kept data like your watchlist and ratings on Trakt), and then Which Film cross-references everyone's watchlists and ratings to create a list of movies that people may want to watch together.

The list of movies is ranked based on a simple point scale. If a movie is on someone's watchlist it gets 4 points, movies rated 10 ⭐ get 3 points, 9 ⭐ get 2 points, and 8 ⭐ get 1 point. Everyone who participates contributes points and the movies are sorted from highest score to lowest. The reason for the point values is the assumption that watching a movie most people have not seen is the best, followed by a movies people rate very highly. In the case of ties, the movie seen longest ago (if ever) by anyone in the group is ranked higher than movies seen more recently by someone. That way there's a bigger chance someone will be willing to watch a movie again when everyone else wants to see it for the first time.

None of this is very fancy or revolutionary, but it's useful any time you get together with a group of friends to watch a film and you end up having a hard time choosing to watch. It can help even between spouses as it will identify movies both people want to watch, removing that particular point of contention.

The story behind Which Film

Now normally launching a new website wouldn't cause for any backstory, but this project has been under development for about six years, so there's a bit of history to it.

One fateful night ...

The inspiration for Which Film stemmed from one night when my co-creator Karl, his wife, my wife, and I got together and decided we wanted to watch a movie. This turned out to be quite an ordeal due to disparate tastes among all four of us. Karl and I thought that there had to be a better way to figure out a film we could all happily watch together. It didn't need to necessarily be something none of us had seen (although that was preferred), but it did need to be something that had a chance of making all of us happy if we chose to watch it.

This is when I realized that at least for me I had all of the relevant data to make such a decision on IMDb. I had been keeping my watchlist and ratings up-to-date on the site for years, to the point of amassing a watchlist over of 400 movies. Karl and I realized that had all four of us done that we could have cross-referenced the data and easily have found a film we all liked. Yes, it would require convincing everyone involved to keep track of what movies they wanted to see and rating movies that had seen, but we figured that wasn't an insurmountable problem. And so we decided we should code up a solution since we're both software developers.

You need an API, IMDb

But there was trouble with this project from the beginning. It turns out that while IMDb is happy for you to store your data on their servers, they don't exactly make it easy to get the data out. For instance, when I started looking into this they had two ways of getting to your data in some programmatic way: RSS and CSV files. The problem with RSS is that it was capped at (I believe) 200 entries, so I couldn't use it to access my entire data set. The issue with CSV was that you had to be logged in to download it. And the issue with both approaches was they were constantly broken for for different things simultaneously; when I looked into this last RSS was busted for one kind of list while CSV was broken for another. To top it all off the brokenness wasn't temporary, but lasted for lengths of time measured in months. That obviously doesn't work if you want to rely on the data and there's no official API (and IMDb at least used to aggressively go after anyone who use their name in a project).

Luckily I found Trakt. It has an API, it was accessible on a phone, and it wasn't ugly. The trick, though, was getting my data from IMDb to Trakt. Luckily there was a magical point when CSV exporting on IMDb worked for all of my lists, and so I downloaded the data and hacked together csv2trakt to migrate my data over (there is TraktRater for importing into Trakt as well, but at the time I had issues getting it to run on macOS).

What platform?

With my data moved over, we then had to choose what platform to have Which Film on. We toyed with the idea of doing a mobile app, but I'm an Android user and Karl is on iOS (and the same split for our wives), so that would have meant two apps. That didn't really appeal to either of us so we decided to do a website. We also consciously chose to do a single-page app to avoid maintaining a backend where would have to worry about uptime, potential server costs, etc. It also helps that there's a local company in Vancouver called Surge that does really nice static page hosting with a very reasonable free tier (when they get Let's Encrypt support I'll probably bump up to their paid tier if people actually end up using Which Film).

Choosing a programming language is never easy for me

Since we had found a website we were willing to ask people to use to store data, I had solved my data import problem, and we had decided on doing a website solution, next was what technology stack to use. The simple answer would have been Python, but for me that's somewhat boring since I obviously know Python. To make sure we both maximized our learning from this project we endeavoured to find a programming language neither of us had extensive experience in.

Eventually we settled on Dart. At the time we made this decision I worked at Google which is where Dart comes from, so I knew if I got really stuck with something I had internal resources to lean on. Karl liked the idea of using Dart because his game developer background appreciated the fact that Dart was looking into things like SIMD for performance. I also knew that Dart had been chosen by the ads product division at Google which meant it wasn't going anywhere. That also meant choosing Angular 2 was a simple decision since Google was using Dart with Angular 2 for products and so it would have solid Dart support.

But why six years?!?

As I have said, the site isn't complicated as you can tell from its source code, so you may be wondering why it took us six years before we could finish it. Well, since coming up with this idea I at least finished my Ph.D., moved five times between two countries,and worked for two different employers (if you don't count my Ph.D.). Karl had a similar busy life over the same timespan. And having me spend a majority of those six years in a different timezone didn't help facilitate discussions. At least we had plenty of time to think through various UX and design problems. ☺

If you give Which Film a try do let Karl and/or me know on Twitter (if you just want to see how the website works and you don't have a Trakt account you can use our usernames: brettcannon and kschmidt).

October 11, 2016 04:44 AM

Network protocols, sans I/O

Back in February I started taking a serious look at asynchronous I/O thanks to async/await. One of the things that led to me to looking into this area was when I couldn't find an HTTP/1.1 library that worked with async/await. A little surprised by this, I went looking for an HTTP header parser so that I could do the asynchronous I/O myself and then rely on the HTTP parsing library to at least handle the HTTP parts. But that's when I got even more shocked to find out there wasn't any such thing as an HTTP parsing library in Python!

It turns out that historically people have written libraries dealing with network protocols with the I/O parts baked in. While this has been fine up until now thanks to all I/O in Python being done in a synchronous fashion, this is going to be a problem going forward thanks to async/await and the move towards asynchronous I/O. Basically what this means is that network protocol libraries will need to be rewritten so that they can be used by both synchronous and asynchronous I/O.

If we're going to start rewriting network protocol libraries, then we might as well do it right from the beginning. This means making sure the library will work with any sort of I/O. This doesn't mean simply abstracting out the I/O so that you can plug in I/O code that can conform to your abstraction. No, to work with any sort of I/O the network protocol library needs to operate sans I/O; working directly off of the bytes or text coming off the network is the most flexible. This allows the user of the protocol library to drive the I/O in the way they deem fit instead of how the protocol library thinks it should be done. This provides the ultimate flexibility in terms of how I/O can be used with a network protocol library.

Luckily I wasn't the first to notice the lack of HTTP parsing library. Cory Benfield also noticed this and then did something about it. He created the hyper-h2 project to provide a network protocol library for HTTP/2 that does no I/O of its own. Instead, you feed hyper-h2 bytes off the network and it tells you -- through a state machine -- what needs to happen. This flexibility means that hyper-h2 has examples on how to use the library with curio, asyncio, eventlet, and Twisted (and now there's experimental support in Twisted for HTTP/2 using hyper-h2). Cory also gave a talk at PyCon US 2016 on the very topic of this blog post.

And HTTP/2 isn't the only protocol that has an implementation with no I/O. Nathaniel Smith of NumPy has created h11 which does for HTTP/1.1 what hyper-h2 does for HTTP/2. Once again, h11 does no I/O on its own and instead gets fed bytes which in turn drives a state machine to tell the user what to do.

So why am I writing this blog post? I think it's important to promote this approach to implementing network protocols, to the point that I have created a page at https://sans-io.readthedocs.io/ to act as a reference of libraries that have followed the approach I've outlined here. If you're aware of a network protocol library that performs no I/O (remember this excludes libraries that abstract out I/O), then please send a pull request to the GitHub project to have it added to the list. And if you happen to know a network protocol well, then please consider implementing a library that follows this approach of using no I/O so the community can benefit.

October 11, 2016 04:44 AM


Daniel Bader

Sublime Text plugin review: Djaneiro

Sublime Text plugin review: Djaneiro

A review of Djaneiro, a Sublime Text plugin for Django development.

I’ll admit I was skeptical at first when a friend of mine recommended Djaneiro to enhance my Django development workflow in Sublime Text.

I’d been happy with the Python development setup I built for myself over the years and I didn’t really understand what Djaneiro was going to add to that.

But when I tried out Djaneiro I was impressed how helpful it turned out to be! I decided to write another Sublime Text plugin review to share my findings.

Djaneiro’s main selling points are adding:

In this review I’ll explain how Djaneiro can make your Django development workflow more productive and I’ll go over the pros and cons of the plugin as I experienced them. After that I’ll take a look at alternatives to Djaneiro in the Sublime Text plugin landscape. At the end I’ll share my final verdict and rating.

Pros

Syntax highlighting for Django templates: In its default configuration Sublime Text doesn’t have syntax definitions for Django’s HTML templating syntax.

This means that typos and syntax errors in templates are harder to catch visually. As you can see in the screenshot below (in the editing pane on the left), the standard HTML syntax highlighting in Sublime Text 3 uses a uniform white color for Django’s template tags.

Djaneiro adds a HTML (Django) syntax that properly highlights Django’s template tags. As you can see in the right-hand editing pane in the screenshot, proper syntax highlighting makes these templates quite a bit easier to read. Also, syntax errors and typos stand out more due to the proper highlighting.

This simple change adds a lot of value – I found that I was making fewer typos in my templates with Djaneiro’s syntax highlighting. Also, templates seemed easier to read and scan quickly with Djaneiro installed.

Improved syntax highlighting for Django Python files: Djaneiro also makes some small tweaks to the default Python syntax highlighting. For example, it knows the standard Django settings constants like INSTALLED_APPS and highlights them differently so that they stand out more and typos are easier to find.

This also happens for things like field definitions when writing Django model classes, which I found handy. In summary I found that the syntax highlighting changes introduced by Djaneiro make it easier to grasp the structure of the code I’m writing.

Code completion snippets for Django templates and Python files: Another helpful feature provided by Djaneiro is a library of pre-made code completion snippets for common Django code and patterns.

For example, you can insert an {% if _____ %} {% endif %} block by typing if as an abbreviation and hitting the auto-complete key (Tab by default). You can see a quick demo of that in the screenshot below. Generally, I found the list of snippets included with Djaneiro to be comprehensive and well-chosen.

Besides snippets for Django HTML templates Djaneiro also includes a snippet library for Django Python code. These snippets let you quickly scaffold out whole view definitions or a barebones model classes, for example.

Once you’ve gotten used to these snippets they can save you a lot of typing. Be sure to check out the full list of snippets in the Djaneiro README.

Cons

Snippets might get in the way: Because Djaneiro adds quite a substantial number of new code snippets I found myself triggering some of them accidentally, especially in the beginning. I really don’t want to hold this up against Djaneiro because the snippets do add a lot of value once I learned to use them well.

It’s possible to disable individual code snippets in Sublime Text but unfortunately this process is a bit involved.

If you find that the snippets get in your way occasionally you can temporarily switch them off by selecting a different syntax highlighting definition. Just open the Sublime Text Command Palette, type Set syntax, and select the default HTML or Python syntax.

Alternatives

There are a few more Django-specific plugins available on Package Control but Djaneiro seems to be the most popular and also the most powerful of the pack.

The verdict

I’ve grown quite fond of Djaneiro since I started using it. I immediately loved the improved syntax highlighting for Django templates and I’d say Djaneiro is worth installing for that feature alone.

Once I’d gotten the hang of Djaneiro’s code snippets and their shortcodes I felt a noticeable improvement in my productivity. The snippets added by Djaneiro cover many things I encountered in day to day Django development. It’s great not having to go through the work of writing these snippets myself.

I’d recommend any Django developer using Sublime Text to at least try out Djaneiro for a few days. There aren’t any substantial downsides to it and I’m sure it will make you more productive. You can install Djaneiro via Package Control.

🐍 🐍 🐍 🐍 🐍 (5 out of 5 snakes)

October 11, 2016 12:00 AM

October 10, 2016


Wesley Chun

Migrating SQL data to Google Sheets using the new Google Sheets API

NOTE: The code covered in this post are also available in a video walkthrough.
UPDATE (Sep 2016): Removed use of argparse module & flags (effective as of Feb 2016).

Introduction

In this post, we're going to demonstrate how to use the latest generation Google Sheets API. Launched at Google I/O 2016 (full talk here), the Sheets API v4 can do much more than previous versions, bringing it to near-parity with what you can do with the Google Sheets UI (user interface) on desktop and mobile. Below, I'll walk you through a Python script that reads the rows of a relational database representing customer orders for a toy company and pushes them into a Google Sheet. Other API calls we'll make: one to create new Google Sheets with and another that reads the rows from a Sheet.

Earlier posts demonstrated the structure and "how-to" use Google APIs in general, so more recent posts, including this one, focus on solutions and use of specific APIs. Once you review the earlier material, you're ready to start with authorization scopes then see how to use the API itself.

Google Sheets API authorization & scopes

Previous versions of the Google Sheets API (formerly called the Google Spreadsheets API), were part of a group of "GData APIs" that implemented the Google Data (GData) protocol, an older, less-secure, REST-inspired technology for reading, writing, and modifying information on the web. The new API version falls under the more modern set of Google APIs requiring OAuth2 authorization and whose use is made easier with the Google APIs Client Libraries.

The current API version features a pair of authorization scopes: read-only and read-write. As usual, we always recommend you use the most restrictive scope possible that allows your app to do its work. You'll request fewer permissions from your users (which makes them happier), and it also makes your app more secure, possibly preventing modifying, destroying, or corrupting data, or perhaps inadvertently going over quotas. Since we're creating a Google Sheet and writing data into it, we must use the read-write scope:

Using the Google Sheets API

Let's look at some code that reads rows from a SQLite database and creates a Google Sheet with that data. Since we covered the authorization boilerplate fully in earlier posts and videos, we're going straight to creating a Sheets service endpoint. The API string to use is 'sheets' and the version string to use is 'v4' as we call the apiclient.discovey.build() function:

SHEETS = discovery.build('sheets', 'v4', http=creds.authorize(Http()))

With the SHEETS service endpoint in hand, the first thing to do is to create a brand new Google Sheet. Before we use it, one thing to know about the Sheets API is that most calls require a JSON payload representing the data & operations you wish to perform, and you'll see this as you become more familiar with it. For creating new Sheets, it's pretty simple, you don't have to provide anything, in which case you'd pass in an empty (dict as the) body, but a better bare minimum would be a name for the Sheet, so that's what data is for:

data = {'properties': {'title': 'Toy orders [%s]' % time.ctime()}}

Notice that a Sheet's "title" is part of its "properties," and we also happen to add the timestamp as part of its name. With the payload complete, we call the API with the command to create a new Sheet [spreadsheets().create()], passing in data in the (eventual) request body:

res = SHEETS.spreadsheets().create(body=data).execute()

Alternatively, you can use the Google Drive API (v2 or v3) to create a Sheet but would also need to pass in the Google Sheets (file) MIME type:
data = {
'name': 'Toy orders [%s]' % time.ctime(),
'mimeType': 'application/vnd.google-apps.spreadsheet',
}
res = DRIVE.files().create(body=data).execute() # insert() for v2
The general rule-of-thumb is that if you're only working with Sheets, you can do all the operations with its API, but if creating files other than Sheets or performing other Drive file or folder operations, you may want to stick with the Drive API. You can also use both or any other Google APIs for more complex applications. We'll stick with just the Sheets API for now. After creating the Sheet, grab and display some useful information to the user:
SHEET_ID = res['spreadsheetId']
print('Created "%s"' % res['properties']['title'])
You may be wondering: Why do I need to create a Sheet and then make a separate API call to add data to it? Why can't I do this all when creating the Sheet? The answer (to this likely FAQ) is you can, but you would need to construct and pass in a JSON payload representing the entire Sheet—meaning all cells and their formatting—a much larger and more complex data structure than just an array of rows. (Don't believe me? Try it yourself!) This is why we have all of the spreadsheets().values() methods... to simplify uploading or downloading of only values to or from a Sheet.

Now let's turn our attention to the simple SQLite database file (db.sqlite) available from the Google Sheets Node.js codelab. The next block of code just connects to the database with the standard library sqlite3 package, grabs all the rows, adds a header row, and filters the last two (timestamp) columns:
FIELDS = ('ID', 'Customer Name', 'Product Code', 'Units Ordered',
'Unit Price', 'Status', 'Created at', 'Updated at')
cxn = sqlite3.connect('db.sqlite')
cur = cxn.cursor()
rows = cur.execute('SELECT * FROM orders').fetchall()
cxn.close()
rows.insert(0, FIELDS)
data = {'values': [row[:6] for row in rows]}
When you have a payload (array of row data) you want to stick into a Sheet, you simply pass in those values to spreadsheets().values().update() like we do here:
SHEETS.spreadsheets().values().update(spreadsheetId=SHEET_ID,
range='A1', body=data, valueInputOption='RAW').execute()
The call requires a Sheet's ID and command body as expected, but there are two other fields: the full (or, as in our case, the "upper left" corner of the) range of cells to write to (in A1 notation), and valueInputOption indicates how the data should be interpreted, writing the raw values ("RAW") or interpreting them as if a user were entering them into the UI ("USER_ENTERED"), possibly converting strings & numbers based on the cell formatting.

Reading rows out of a Sheet is even easier, the spreadsheets().values().get() call needing only an ID and a range of cells to read:
print('Wrote data to Sheet:')
rows = SHEETS.spreadsheets().values().get(spreadsheetId=SHEET_ID,
range='Sheet1').execute().get('values', [])
for row in rows:
print(row)
The API call returns a dict which has a 'values' key if data is available, otherwise we default to an empty list so the for loop doesn't fail.

If you run the code (entire script below) and grant it permission to manage your Google Sheets (via the OAuth2 prompt that pops up in the browser), the output you get should look like this:
$ python3 sheets-toys.py # or python (2.x)
Created "Toy orders [Thu May 26 18:58:17 2016]" with this data:
['ID', 'Customer Name', 'Product Code', 'Units Ordered', 'Unit Price', 'Status']
['1', "Alice's Antiques", 'FOO-100', '25', '12.5', 'DELIVERED']
['2', "Bob's Brewery", 'FOO-200', '60', '18.75', 'SHIPPED']
['3', "Carol's Car Wash", 'FOO-100', '100', '9.25', 'SHIPPED']
['4', "David's Dog Grooming", 'FOO-250', '15', '29.95', 'PENDING']
['5', "Elizabeth's Eatery", 'FOO-100', '35', '10.95', 'PENDING']

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

'''sheets-toys.py -- Google Sheets API demo
created Jun 2016 by +Wesley Chun/@wescpy
'''
from __future__ import print_function
import sqlite3
import time

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/spreadsheets'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('client_id.json', SCOPES)
creds = tools.run_flow(flow, store)
SHEETS = discovery.build('sheets', 'v4', http=creds.authorize(Http()))

data = {'properties': {'title': 'Toy orders [%s]' % time.ctime()}}
res = SHEETS.spreadsheets().create(body=data).execute()
SHEET_ID = res['spreadsheetId']
print('Created "%s"' % res['properties']['title'])

FIELDS = ('ID', 'Customer Name', 'Product Code', 'Units Ordered',
'Unit Price', 'Status', 'Created at', 'Updated at')
cxn = sqlite3.connect('db.sqlite')
cur = cxn.cursor()
rows = cur.execute('SELECT * FROM orders').fetchall()
cxn.close()
rows.insert(0, FIELDS)
data = {'values': [row[:6] for row in rows]}

SHEETS.spreadsheets().values().update(spreadsheetId=SHEET_ID,
range='A1', body=data, valueInputOption='RAW').execute()
print('Wrote data to Sheet:')
rows = SHEETS.spreadsheets().values().get(spreadsheetId=SHEET_ID,
range='Sheet1').execute().get('values', [])
for row in rows:
print(row)
You can now customize this code for your own needs, for a mobile frontend, devops script, or a server-side backend, perhaps accessing other Google APIs. If this example is too complex, check the Python quickstart in the docs that way simpler, only reading data out of an existing Sheet. If you know JavaScript and are ready for something more serious, try the Node.js codelab where we got the SQLite database from. That's it... hope you find these code samples useful in helping you get started with the latest Sheets API!

EXTRA CREDIT: Feel free to experiment and try cell formatting or other API features. Challenge yourself as there's a lot more to Sheets than just reading and writing values! 

October 10, 2016 11:08 PM


PyCon

The PyCon 2017 site has launched — thank you to our Launch Day Sponsors

The new PyCon 2017 web site recently went live, and the conference volunteers have worked hard bring the new site up-to-date with all of the essential details about 2017’s schedule, venue, and hotels. We are very happy with the new logo and banner that Beatrix Bodó crafted to help the conference celebrate its second and final year in beautiful Portland, Oregon!

With the release of the site we have also opened up the proposal forms for Talks, Tutorials, Posters, and Education Summit presentations. Visit our “Speak at PyCon” page to read the details of our Call For Proposals (CFP) and to learn about becoming part of the 2017 conference schedule.

Our launch-day sponsors this year — the organizations who have gone ahead and pledged to support and attend PyCon 2017, helping keep the conference affordable for as wide a range of attendees as possible — are from a broad array of fields that illustrate just how widely Python is used in today’s world.

Two of our Launch Day sponsors this year are supporting the conference at the Platinum level:

Our launch-day Gold sponsors range from large Fortune 100 companies to small consultancies providing boutique consulting and programming:

And, finally, we have already signed our first Silver sponsor!

For more details about each sponsor, see the detailed sponsor descriptions on our Sponsors Page and follow the links to their web sites. We look forward to seeing every one of these sponsors in the Expo Hall on Friday and Saturday of the main conference!

Subscribe to our blog here for regular updates as the conference approaches. To get you started, here are the most important dates for the conference through the rest of the year and up to PyCon itself:

2016

2017

In Portland, Oregon

October 10, 2016 09:50 PM


Python Insider

Python 3.6.0b2 is now available

Python 3.6.0b2 is the second of four planned beta releases of Python 3.6, the next major
release of Python.

Among the major new features in Python 3.6 are:

* PEP 468 - Preserving the order of **kwargs in a function
* PEP 487 - Simpler customization of class creation
* PEP 495 - Local Time Disambiguation
* PEP 498 - Literal String Formatting
* PEP 506 - Adding A Secrets Module To The Standard Library
* PEP 509 - Add a private version to dict
* PEP 515 - Underscores in Numeric Literals
* PEP 519 - Adding a file system path protocol
* PEP 520 - Preserving Class Attribute Definition Order
* PEP 523 - Adding a frame evaluation API to CPython
* PEP 524 - Make os.urandom() blocking on Linux (during system startup)
* PEP 525 - Asynchronous Generators (provisional)
* PEP 526 - Syntax for Variable Annotations (provisional)
* PEP 528 - Change Windows console encoding to UTF-8 (provisional)
* PEP 529 - Change Windows filesystem encoding to UTF-8 (provisional)
* PEP 530 - Asynchronous Comprehensions

Please see "What’s New In Python 3.6" for more information:

https://docs.python.org/3.6/whatsnew/3.6.html

You can find Python 3.6.0b2 here:

https://www.python.org/downloads/release/python-360b2/

Beta releases are intended to give the wider community the opportunity
to test new features and bug fixes and to prepare their projects to
support the new feature release. We strongly encourage maintainers of
third-party Python projects to test with 3.6 during the beta phase and
report issues found to bugs.python.org as soon as possible. While the
release is feature complete entering the beta phase, it is possible that
features may be modified or, in rare cases, deleted up until the start
of the release candidate phase (2016-12-05). Our goal is have no changes
after rc1. To achieve that, it will be extremely important to get as
much exposure for 3.6 as possible during the beta phase. Please keep in
mind that this is a preview release and its use is not recommended for
production environments

The next planned release of Python 3.6 will be 3.6.0b3, currently
scheduled for 2016-10-31. More information about the release schedule
can be found here:

https://www.python.org/dev/peps/pep-0494/

October 10, 2016 09:18 PM


Enthought

Geophysical Tutorial: Facies Classification using Machine Learning and Python

Published in the October 2016 edition of The Leading Edge magazine by the Society of Exploration Geophysicists. Read the full article here.

By Brendon Hall, Enthought Geosciences Applications Engineer 
Coordinated by Matt Hall, Agile Geoscience

ABSTRACT

There has been much excitement recently about big data and the dire need for data scientists who possess the ability to extract meaning from it. Geoscientists, meanwhile, have been doing science with voluminous data for years, without needing to brag about how big it is. But now that large, complex data sets are widely available, there has been a proliferation of tools and techniques for analyzing them. Many free and open-source packages now exist that provide powerful additions to the geoscientist’s toolbox, much of which used to be only available in proprietary (and expensive) software platforms.

One of the best examples is scikit-learn, a collection of tools for machine learning in Python. What is machine learning? You can think of it as a set of data-analysis methods that includes classification, clustering, and regression. These algorithms can be used to discover features and trends within the data without being explicitly programmed, in essence learning from the data itself.

Well logs and facies classification results from a single well.

Well logs and facies classification results from a single well.

In this tutorial, we will demonstrate how to use a classification algorithm known as a support vector machine to identify lithofacies based on well-log measurements. A support vector machine (or SVM) is a type of supervised-learning algorithm, which needs to be supplied with training data to learn the relationships between the measurements (or features) and the classes to be assigned. In our case, the features will be well-log data from nine gas wells. These wells have already had lithofacies classes assigned based on core descriptions. Once we have trained a classifier, we will use it to assign facies to wells that have not been described.

See the tutorial in The Leading Edge here.

ADDITIONAL RESOURCES:

October 10, 2016 07:46 PM


Python Engineering at Microsoft

Python in Visual Studio “15” Preview 5

Last week, Visual Studio “15” Preview 5 was made available for download. This release has a number of exciting changes and improvements for Python developers that we are going to be covering over the next few weeks. Today’s post will look at the new install experience, covering the improvements and the new options to be aware of.

When you first run the Visual Studio “15” Preview 5 installer, rather than the traditional setup, you’ll see a collection of workloads. Rather than giving you a default install with most features, we’ve broken the product up into a much more fine-grained set of components, each of which can be individually enabled or disabled. A workload is a preconfigured set of components, designed to make it easy for you to know what to select.

Visual Studio

The two workloads our team is responsible for are Python development and Data science and analytical applications. These are designed to quickly get you started in Python for desktop, server and cloud, or to give you a data-science focused IDE that includes Python, R and F#. You can select either or both workloads, as well as any of the others, to get our recommended set of tools.

For the Python development workload, you will get support for Python editing, debugging (including remote and mixed-mode debugging), profiling, unittest tests, cookiecutter templates, as well as a copy of CPython 3.5. Some of the options available to you on the right-hand side include older or 32-bit versions of CPython, easy installation of Anaconda from Continuum Analytics, native Python development tools and support for Windows IoT Core.

The Data Science workload is designed to install the key tools and runtimes you may need for analysis, visualization, machine learning and scientific computing work. You can choose from F#, Python and R support – all languages supporting interactive, data-focused development – and we include the Microsoft R Client and Anaconda. If you are already using another version of R or Python, you can deselect these and continue to use your own.

One of the benefits of a highly-focused Visual Studio is that install time and size is significantly reduced. This current version is a preview, so we’re still working to make it even smaller and faster, but even now you can have a very functional Python development environment for only 875 MB:

Visual Studio

We still have time and work to do before Visual Studio “15” is ready for prime-time, but we hope you appreciate getting a chance to try out early functionality. Beyond installation, we also have a range of improvements to Python support, including cookiecutter templates, interactive window management, out-of-process code analysis, editor code cells, project-less browsing, and a huge amount of bug fixes. Over the next few weeks we will post more detailed blogs about some of these enhancements.

We hope you will enjoy using Python and R in this preview of Visual Studio “15”. Feel free to send feedback through the tools within Visual Studio, or post directly to our github site, which is also where you can obtain all the sources for Python support and see our full release notes.

October 10, 2016 06:00 PM


Possbility and Probability

Debugging like Elon Musk

Every time I think I’m doing pretty good with my projects, I take look over and see what Elon Musk is doing. Then I instantly feel like I’m wasting my life. That guy does such huge things and he does … Continue reading

The post Debugging like Elon Musk appeared first on Possibility and Probability.

October 10, 2016 02:25 PM


Doug Hellmann

webbrowser — Displays web pages — PyMOTW 3

The webbrowser module includes functions to open URLs in interactive browser applications. The module includes a registry of available browsers, in case multiple options are available on the system. It can also be controlled with the BROWSER environment variable. Read more… This post is part of the Python Module of the Week series for Python … Continue reading webbrowser — Displays web pages — PyMOTW 3

October 10, 2016 01:00 PM


Mike Driscoll

PyDev of the Week: Paul Logston

This week we welcome Paul Logston (@PaulLogston) as our PyDev of the Week! Paul is the new maintainer for Pyvideo.org. He took over for Will Kahn-Greene. You can read more about the transition here. He is also an organizer for his local Python Users Group. You might want to take a few moments to check out his Github profile so you can see what Paul has been up to. Let’s take some time to get to know him better!

Can you tell us a little about yourself (hobbies, education, etc):

Went to UC Davis for Chemical Engineering, studied improv in Chicago after college, studied as an EMT/Phlebotomist to work as one while doing comedy. Moved to New York, NY to continue doing comedy. Taught CPR/BLS/ACLS for a few years while doing comedy. Then decided to change careers. Decided to push for coding as I had done it all along as a hobby. After about a year of teaching myself Python, I moved from EMS education to working for 15Five as a developer. The NYC Python Meetup was integral in my Python education. Without that group, I wouldn’t be were I am today. I now help organize that group and run Saturday Office Hours along with some large talk nights and PyGotham; NYC’s regional Python conference. Other hobbies include binge watching Netflix originals, brewing kombucha, and teaching.

Why did you start using Python?

I knew PHP but I wanted a language that I could used to write a single backend for both websites and GUIs. I have yet to use Python for that purpose though.

What other programming languages do you know and which is your favorite?

Yeah, the line between “know” and “I’ve put it on my resume as a language I know” is kinda blurry, but here’s a list of languages I could be efficient with if given a week or two of ramp up time.

  • C
  • PHP
  • JavaScript

I think C is my favorite. Its not my favorite to write (that’d be Python by far), but C is just so much closer to the metal that you get that really hacker high when you write it. Or at least I do.

What projects are you working on now?

  • Pyvideo.org
  • Writing scripts to find the best deals on ticketing sites.
  • A dna-to-music player

Which Python libraries are your favorite (core or 3rd party)?

Wow. Good question. I tend to not do favorites though. Those are hard for me and not worth the cost of determining. Ones that I am really into recently though are asyncio and bokeh.

Where do you see Python going as a programming language?

I think Python is growing in a lot of different areas and I think several communities are making it their go to language. Non more so than teaching and data. I see python growing in all areas but the ground that its taking in schools is amazing. Java and C++ are being pushed out and replaced by Python. Also, as fun as R is (I’ve never used R), its hard to display the results of your work in R in an interactive way with a large number of people. I see people who want to write code for data analysis also wanting to write code for a quick web app to show their work. R is just not there (yet). In short, Python’s easy of use, gentle learning curve, and versatility make it a great language to start with and to grow with.

What is your take on the current market for Python programmers?

Yeah, it can be rough. On a micro level there tends to be 3 – 6 month spurts of hiring here in NYC, followed by slowness. I think we are in and towards the end of the spurt right now. I expect hiring to slow in about a month. Plus the holidays are right around the corner so some slowness is a given.

On a macro level, I really don’t know. I don’t know if we are on top of a hill, a mountain, or if we just took our first toddler sized step up some playground stairs. All I can feel is forward momentum. And for the record, it feels good.

Is there anything else you’d like to say?

First, thanks for having me! Secondly, I’ll use this time to shamelessly plug pyvideo.org. Pyvideo.org is a globally used (see photo below) python video index. We are always looking for new submissions and are very grateful for those that come in. If you’re looking for an easy way to break into the open source community, feel free to submit a PR for one of the issues at https://github.com/pyvideo/data. Thanks!

pyvideo

Thanks for doing the interview!

October 10, 2016 12:30 PM


Vasudev Ram

PDF cheat sheet: bin/oct/dec/hex conversion (0-255)

By Vasudev Ram



Numeral system image attribution

Hi readers,

Here is another in my series of PDF-generation applications built using xtopdf, my Python toolkit for PDF creation from various data formats.

This program generates a PDF cheat sheet for conversion of the numbers 0 to 255 between binary, octal, decimal and hexadecimal numeral systems. It can be useful for programmers, electrical / electronics engineers, scientists or anyone else who has to deal with numbers in those bases.

Here is the program, in file number_systems.py:
from __future__ import print_function
from PDFWriter import PDFWriter
import sys

'''
A program to generate a table of numbers from
0 to 255, in 4 numbering systems:
- binary
- octal
- decimal
- hexadecimal
Author: Vasudev Ram
Copyright 2016 Vasudev Ram
Web site: https://vasudevram.github.io
Blog: http://jugad2.blogspot.com
Product store on Gumroad: https://gumroad.com/vasudevram
'''

def print_and_write(s, pw):
print(s)
pw.writeLine(s)

sa, lsa = sys.argv, len(sys.argv)
if lsa == 1:
sys.stderr.write("Usage: {} out_filename.pdf\n".format(sa[0]))
sys.exit(1)

with PDFWriter(sa[1]) as pw:

pw.setFont('Courier', 12)
pw.setHeader('*** Number table: 0 to 255 in bases 2, 8, 10, 16 ***')
pw.setFooter('*** By xtopdf: https://google.com/search?q=xtopdf ***')
b = "Bin"; o = "Oct"; d = "Dec"; h = "Hex"
header = "{b:>10}{o:>10}{d:>10}{h:>10}".format(b=b, o=o, d=d, h=h)

for i in range(256):
if i % 16 == 0:
print_and_write(header, pw)
print_and_write("{b:>10}{o:>10}{d:>10}{h:>10}".format( \
b=bin(i), o=oct(i), d=str(i), h=hex(i)), pw)

print_and_write(header, pw)
And here is a screenshot of first page of the PDF output, after running the program
with the command: python number_systems.py BODH-255.pdf


Ans here is a screenshot of the last page:


You can get the cheat sheet from my Gumroad store here: gum.co/BODH-255.

(You can also get email updates about my future products.)

I named the output BODH-255.pdf (from Binary Decimal Octal Hexadecimal 255), instead of BODH-0-255.pdf, because, of course, programmers count from zero, so the 0 can be implicit :)

(The formatting of the output is a little unconventional, due to the use of the header line occurring after every 16 lines in the table, but it is on purpose, because I am experimenting with different formats, to see the effect on readability / usability).

Notice 1) the smooth and systematic progression of the numbers in the vertical columns (how the values change like a car's odometer), and also 2) the relationships between numbers in different columns in the same row, when compared across any rows where one number is a multiple of another, e.g. look at the rows for decimal 32, 64, 128). Both these effects that we see are due to the inherent properties of the numbers with those values and their representation in those number bases. Effect 1) - like a car's odometer - is most noticeable in the Bin(ary) column - because of the way all the 1 bits flip to 0 when the number bumps up to a power of 2 - e.g. when binary 111 (decimal 7) changes to binary 1000 (decimal 8 == 2 ** 3), but the effect can be seen in the other columns as well.

The image at the top of the post is from this Wikipedia page:

Numeral system

- Vasudev Ram - Online Python training and consulting

Get updates on my software products / ebooks / courses.

Jump to posts: Python   DLang   xtopdf

Subscribe to my blog by email

My ActiveState recipes

FlyWheel - Managed WordPress Hosting



October 10, 2016 04:21 AM


Michał Bultrowicz

Continuous delivery of a Python library with AngularJS commit convention

I got tired of having to manually build and upload my library (Mountepy) to PyPI, so I decided to do what any sane programmer would do - set up automation [1]. But how would my scripts know whether they need to just update the README on PyPI and when to assemble and push a new version of the library? Thanks to the AngularJS commit convention! Oh, and Snap CI will run the whole thing. Why Snap, you ask? See my previous article - Choosing a CI service for your open-source project.

October 10, 2016 12:00 AM

October 09, 2016


Weekly Python Chat

Classes in Python

Let's chat about classes in Python: what are classes useful for and how are they different from classes in other languages?

October 09, 2016 05:00 PM


Kracekumar Ramaraju

RC week 0001

This week, I made considerable progress on the BitTorrent client which I started a week back. The client is in a usable state to download the data from the swarm. The source code is available on GitHub. The project uses Python 3.5 async/await and asyncio. I presented the torrent client in RC Thursday five minute presentation evening slot. Here is the link to the slides.

Here is quick video demo recorded with asciinema.

In the demo, the client downloads a file flag.jpg of 1.3MB. Thanks a lot for Thomas Ballinger for hosting the tracker and the seeder. The tracker and the seeder are boons for developers writing torrent clients.

The downloader has two known major issue

The week had long hours of debugging blocking code in asyncio, tweaking the client to support receive data from the deluge client - as soon as the handshake is successful, the client starts sending bitfield, have messages before receiving the interested message.

The next step is to integrate seeding functionality to the torrent client and enhance UDP tracker functionality.

On one of the weekday, I witnessed the momentous incident. I decided to join other two other recursers for lunch at a nearby unvisited outlet to pick up the lunch. The shop was spacious with bar setup and we decided to eat in the building. We ordered three Panner Tika and were munching our meal. The manager/owner/bartender showed up to fill the water, said to another person, “Let me fill water for you.” She greeted him and asked, “How’s your day going? and …”. He enthusiastically replied; filled the glass; enquired about the food and moved on. After few moments, he returned with the exuberant joy; handed over her handful of candies and stated this is a gift for her good manners. Speechless moment! What a revelation and life lesson! The small act of courtesy made the person feel elated and must have made his day. All day and night, we spend time thinking of lighting smiles on family, friends, mates and others. Take a moment to spread joy among the strangers. Later, I recalled a quote,

“How you treat totally irrelevant person defines who you are.”

October 09, 2016 05:09 AM

October 08, 2016


Podcast.__init__

Episode 78 - Lorena Mesa

Summary

One of the great strengths of the Python community is the diversity of backgrounds that our practitioners come from. This week Lorena Mesa talks about how her focus on political science and civic engagement led her to a career in software engineering and data analysis. In addition to her professional career she founded the Chicago chapter of PyLadies, helps teach women and kids how to program, and was voted onto the board of the PSF.

Brief Introduction

Linode Sponsor Banner

Use the promo code podcastinit20 to get a $20 credit when you sign up!

Rollbar Logo

I’m excited to tell you about a new sponsor of the show, Rollbar.

One of the frustrating things about being a developer, is dealing with errors… (sigh)

  • Relying on users to report errors
  • Digging thru log files trying to debug issues
  • A million alerts flooding your inbox ruining your day…

With Rollbar’s full-stack error monitoring, you get the context, insights and control you need to find and fix bugs faster. It’s easy to get started tracking the errors and exceptions in your stack.You can start tracking production errors and deployments in 8 minutes - or less, and Rollbar works with all major languages and frameworks, including Ruby, Python, Javascript, PHP, Node, iOS, Android and more.You can integrate Rollbar into your existing workflow such as sending error alerts to Slack or Hipchat, or automatically create new issues in Github, JIRA, Pivotal Tracker etc.

We have a special offer for Podcast.__init__ listeners. Go to rollbar.com/podcastinit, signup, and get the Bootstrap Plan free for 90 days. That’s 300,000 errors tracked for free.Loved by developers at awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch and more. Help support Podcast.__init__ and give Rollbar a try a today. Go to rollbar.com/podcastinit

Interview with Lorena Mesa

Keep In Touch

Twitter

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Summary One of the great strengths of the Python community is the diversity of backgrounds that our practitioners come from. This week Lorena Mesa talks about how her focus on political science and civic engagement led her to a career in software engineering and data analysis. In addition to her professional career she founded the Chicago chapter of PyLadies, helps teach women and kids how to program, and was voted onto the board of the PSF.Brief IntroductionHello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable.Check out our sponsor Linode for running your awesome new Python apps. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next projectYou want to make sure your apps are error-free so give our other sponsor, Rollbar, a look. Rollbar is a service for tracking and aggregating your application errors so that you can find and fix the bugs in your application before your users notice they exist. Use the link rollbar.com/podcastinit to get 90 days and 300,000 errors for free on their bootstrap plan.Visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch.By leaving a review on iTunes, or Google Play Music it becomes easier for other people to find us.Join our community! Visit discourse.pythonpodcast.com to help us grow and connect our wonderful audience.Your host as usual is Tobias MaceyToday we're interviewing Lorena Mesa about what inspires her in her work as a software engineer and data analyst. Use the promo code podcastinit20 to get a $20 credit when you sign up! I’m excited to tell you about a new sponsor of the show, Rollbar. One of the frustrating things about being a developer, is dealing with errors… (sigh)Relying on users to report errorsDigging thru log files trying to debug issuesA million alerts flooding your inbox ruining your day...With Rollbar’s full-stack error monitoring, you get the context, insights and control you need to find and fix bugs faster. It's easy to get started tracking the errors and exceptions in your stack.You can start tracking production errors and deployments in 8 minutes - or less, and Rollbar works with all major languages and frameworks, including Ruby, Python, Javascript, PHP, Node, iOS, Android and more.You can integrate Rollbar into your existing workflow such as sending error alerts to Slack or Hipchat, or automatically create new issues in Github, JIRA, Pivotal Tracker etc. We have a special offer for Podcast.__init__ listeners. Go to rollbar.com/podcastinit, signup, and get the Bootstrap Plan free for 90 days. That's 300,000 errors tracked for free.Loved by developers at awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch and more. Help support Podcast.__init__ and give Rollbar a try a today. Go to rollbar.com/podcastinitInterview with Lorena MesaIntroductionsHow did you get introduced to Python?How did your original interests in political science and community outreach lead to your current role as a software engineer?You dedicate a lot of your time to organizations that help teach programming to women and kids. What are some of the most meaningful experiences that you have been able to facilitate?Can you talk a bit about your work getting the PyLadies chapter in Chicago off the ground and what the reaction has been like?Now that you are a member of the board for the PSF, what are your goals in that position?What is it about software development that made you want to change your career path?What are some of the most interesting projects that you have worked on, whether for your employer or for fun?Do you think that the bootcamp you attended did a good job of preparing you for a position in industry?What is your view on the concept that software development is the modern form of

October 08, 2016 04:47 PM