skip to navigation
skip to content

Planet Python

Last update: February 27, 2017 04:47 AM

February 27, 2017


William Minchin

Post Stats Plugin 1.1.0 for Pelican Released

Post Stats is a plugin for Pelican, a static site generator written in Python.

Post Stats calculates various statistics about a post and store them in an article.stats dictionary:

Installation

The easiest way to install Post Stats is through the use of pip. This will also install the required dependencies automatically.

pip install minchin.pelican.plugins.post_stats

Then, in your pelicanconf.py file, add Post Stats to your list of plugins:

PLUGINS = [
           # ...
           'minchin.pelican.plugins.post_stats',
           # ...
           ]

You may also need to configure your template to make use of the statistics generated.

Requirements

Post Stats depends on (and is really only useful with) Pelican. The plugin also requries Beautiful Soup 4 to process your content. If the plugin is installed from pip, these will automatically be installed. These can also be manually installed with pip:

pip install pelican
pip install beautifulsoup4

Configuration and Usage

This plugin calculates various statistics about a post and store them in an article.stats dictionary.

Example:

{
    'wc': 2760,
    'fi': '65.94',
    'fk': '7.65',
    'word_counts': Counter({u'to': 98, u'a': 90, u'the': 83, ...}),
    'read_mins': 12
}

This allows you to output these values in your templates, like this, for example:

<p title="~{{ article.stats['wc'] }} words">~{{ article.stats['read_mins'] }} min read</p>
<ul>
    <li>Flesch-kincaid Index/ Reading Ease: {{ article.stats['fi'] }}</li>
    <li>Flesch-kincaid Grade Level: {{ article.stats['fk'] }}</li>
</ul>

The word_counts variable is a Python Counter dictionary and looks something like this, with each unique word and it’s frequency:

Counter({u'to': 98, u'a': 90, u'the': 83, u'of': 50, u'karma': 50, .....

and can be used to create a tag/word cloud for a post.

There are no user-configurable settings.

Known Issues

An issue, as such, is that there is no formal test suite. Testing is currently limited to my in-use observations. I also run a basic check upon uploaded the package to PyPI that it can be downloaded and loaded into Python.

The package is tested in Python 3.6; compatibility with other version of Python is unknown, but there should be nothing particular keeping it from working with other “modern” versions of Python.

Credits

Original plugin by Duncan Lock (@dflock) and posted to the Pelican-Plugins repo.

License

The plugin code is assumed to be under the AGPLv3 license (this is the license of the Pelican-Plugins repo).

February 27, 2017 03:52 AM

Optimize Images Plugin 1.1.0 for Pelican Released

Optimize Images is a plugin for Pelican, a static site generator written in Python.

Optimize Images applies lossless compression on JPEG and PNG images, with no effect on image quality. It uses jpegtran and OptiPNG.

Installation

The easiest way to install Optimize Images is through the use of pip. This will also install the required Python dependencies automatically (currently none beyond Pelican itself).

pip install minchin.pelican.plugins.optimize_images

It is assumed both jpegtran and OptiPNG are installed and available on the system path.

Then, in your pelicanconf.py file, add Optimize Images to your list of plugins:

PLUGINS = [
           # ...
           'minchin.pelican.plugins.optimize_images',
           # ...
           ]

Requirements

Optimize Images depends on (and is really only useful with) Pelican. This can be manually installed with pip:

pip install pelican

It is assumed both jpegtran and OptiPNG are installed on system path. On Windows, installers are available at each respective website. On Ubuntu systems (including Travis-CI), the two can be installed via apt-get.

apt-get install optipng libjpeg-progs

Configuration and Usage

The plugin will activate and optimize images upon finalized signal of Pelican.

The plugin has no user settings.

Known Issues

Image manipulation like this can take some time to run. You may consider only adding this plugin to your publishconf.py (rather than your base pelicanconf.py), which will then only run this image optimization in preparation for site publication.

An issue, as such, is that there is no formal test suite. Testing is currently limited to my in-use observations. I also run a basic check upon uploaded the package to PyPI that it can be downloaded and loaded into Python.

The package is tested in Python 3.6; compatibility with other version of Python is unknown, but there should be nothing particular keeping it from working with other “modern” versions of Python.

Credits

Original plugin from the Pelican-Plugins repo.

License

The plugin code is assumed to be under the AGPLv3 license (this is the license of the Pelican-Plugins repo).

February 27, 2017 03:00 AM

February 26, 2017


Weekly Python Chat

Emoji: Revisited

Special guest Katie McLaughlin will answer your questions about emoji: unicode, compatibility, support, and more.

February 26, 2017 06:00 PM


Brian Okken

27: Mahmoud Hashemi : unit, integration, and system testing

What is the difference between a unit test, an integration test, and a system test? Mahmoud Hashemi helps me to define these terms, as well as discuss the role of all testing variants in software development. What is the difference between a unit test, an integration test, and a system test? TDD testing pyramid vs […]

The post 27: Mahmoud Hashemi : unit, integration, and system testing appeared first on Python Testing.

February 26, 2017 04:22 PM


Oliver Andrich

cookiecutter-flask-lambda

After my first steps with Lambda, Zappa and Flask I created some small applications to try various Zappa configuration options and play with some AWS services. Setting up the project got pretty boring after the second or third toy project. But for python developers there is a solution to automate these tasks – cookiecutter. I […]

February 26, 2017 02:16 PM


qutebrowser development blog

qutebrowser v0.10.0 released

I'm happy to annouce the release of qutebrowser v0.10.0!

qutebrowser is a keyboard driven browser with a vim-like, minimalistic interface. It's written using PyQt and cross-platform.

I haven't announced the v0.9.0 release in this blog (or any patch releases), but for v0.10.0 it definitely makes sense to do so, as it's mostly centered on QtWebEngine!

The full changelog for this release:

Added

  • Userscripts now have a new $QUTE_COMMANDLINE_TEXT environment variable, containing the current commandline contents
  • New ripbang userscript to create a searchengine from a duckduckgo bang
  • QtWebKit Reloaded (also called QtWebKit-NG) is now fully supported
  • Various new functionality with the QtWebEngine backend:
    • Printing support with Qt >= 5.8
    • Proxy support with Qt >= 5.8
    • The general -> print-element-backgrounds option with Qt >= 5.8
    • The content -> cookies-store option
    • The storage -> cache-size option
    • The colors -> webpage.bg option
    • The HTML5 fullscreen API (e.g. youtube videos) with QtWebEngine
    • :download --mhtml
  • New qute:history URL and :history command to show the browsing history
  • Open tabs are now auto-saved on each successful load and restored in case of a crash
  • :jseval now has a --file flag so you can pass a javascript file
  • :session-save now has a --only-active-window flag to only save the active window
  • OS X builds are back, and built with QtWebEngine

Changed

  • PyQt 5.7/Qt 5.7.1 is now required for the QtWebEngine backend
  • Scrolling with the scrollwheel while holding shift now scrolls sideways
  • New way of clicking hints which solves various small issues
  • When yanking a mailto: link via hints, the mailto: prefix is now stripped
  • Zoom level messages are now not stacked on top of each other anymore
  • qutebrowser now automatically uses QtWebEngine if QtWebKit is unavailable
  • :history-clear now asks for a confirmation, unless it's run with --force.
  • input -> mouse-zoom-divider can now be 0 to disable zooming by mouse wheel
  • network -> proxy can also be set to pac+file://... now to use a local proxy autoconfig file (on QtWebKit)

Fixed

  • Various bugs with Qt 5.8 and QtWebEngine:
    • Segfault when closing a window
    • Segfault when closing a tab with a search active
    • Fixed various mouse actions (like automatically entering insert mode) not working
    • Fixed hints sometimes not working
    • Segfault when opening a URL after a QtWebEngine renderer process crash
  • Other QtWebEngine fixes:
    • Insert mode now gets entered correctly with a non-100% zoom
    • Crash reports are now re-enabled when using QtWebEngine
    • Fixed crashes when closing tabs while hinting
    • Using :undo or :tab-clone with a view-source:// or chrome:// tab is now prevented, as it segfaults
  • :enter-mode now refuses to enter modes which can't be entered manually (which caused crashes)
  • :record-macro (q) now doesn't try to record macros for special keys without a text
  • Fixed PAC (proxy autoconfig) not working with QtWebKit
  • :download --mhtml now uses the new file dialog
  • Word hints are now upper-cased correctly when hints -> uppercase is true
  • Font validation is now more permissive in the config, allowing e.g. "Terminus (TTF)" as font name
  • Fixed starting on newer PyQt/sip versions with LibreSSL
  • When downloading files with QtWebKit, a User-Agent header is set when possible
  • Fixed showing of keybindings in the :help completion
  • :navigate prev/next now detects rel attributes on <a> elements, and handles multiple rel attributes correctly
  • Fixed a crash when hinting with target userscript and spawning a non-existing script
  • Lines in Jupyter notebook now trigger insert mode

February 26, 2017 10:05 AM


Import Python

ImportPython 113 - Interview with Guido aka BDFL

Worthy Read

Talkpython interview with Guido van Rossum aka BDFL.
podcast
,
BDFL

We help companies like Airbnb, Pfizer, and Artsy find great developers. Let us find your next great hire. Get started today.
sponsor

The people who introduced me to Python chose it because of the elegance of the language, and it's aesthetic qualities. Would they choose it again, I wonder? Would I?.
core-python

This book is for people with some experience in an object oriented programming language. This book will help you get better at module/class level design. Hopefully, it will teach you to identify good design from bad.
book

Python material in data science, analysis, and modeling, and optimization. Here is the youtube video channel of the site https://www.youtube.com/user/APMonitorCom
video

This time, it was different though. My distributed web crawler seemed to be slowing down over time. Adding more nodes only had a temporary performance boost; the overall crawling speed gradually declined afterwards. So simply put, it couldn't scale. But why?. In this post, you'll find out what techniques and tools I used to diagnose scaling issues - and to an extent, more general performance issues - in my Python-based web crawler.
debugging

Code reuse is a very common need. It saves you time for writing the same code multiple times, enables leveraging other smart people’s work to make new things happen. Even just for one project, it helps organize code in a modular way so you can maintain each part separately. When it comes to python, it means format your project so it can be easily packaged. This is a simple instruction on how to go from nothing to a package that you can proudly put it in your portfolio to be used by other people.
packaging

closures

Note - The video is old, but worth watching for emacs users.
emacs

deep learning

image processing

PyWren, Tfdeploy, Luigi, Kubelib, PyTorch. Note - We used luigi at my previous workplace and it's a solid library to custom pipelines for batch processing. In our case it was used to enforce database migrations.
machine learning

Type Tracing - as a program runs you trace it and record the types of variables coming in and out of functions, and being assigned to variables.
debugging

I've written a Python package called pdftabextract https://github.com/WZBSocialScienceCenter/pdftabextract that contains several helpful functions for that task and I'm explaining how to use them in that blog post.
data mining


Jobs

Pune, Maharashtra, India
iDatalabs (https://idatalabs.com/) is hiring for a junior data scientist (with 0.5-2 years of work experience).


Projects

tkui - 166 Stars, 11 Fork
A visual introspective GUI maker with live editing of the GUI and its editor at the same time

tweetfeels - 18 Stars, 1 Fork
Real-time sentiment analysis in Python using twitter's streaming api

WallpapersFromReddit - 13 Stars, 3 Fork
Download all the hot images from reddit.com/r/wallpaper subreddit every 24 hours to a local device and set an image from those local files as a wallpaper, which updates automatically every 30 minutes!

lda2vec-tf - 12 Stars, 1 Fork
Tensorflow port of the lda2vec model for unsupervised learning of document + topic + word embeddings.

ieighteen - 10 Stars, 1 Fork
Speed up your Localization/Internationalization efforts by automating translation with single script

fish-hook - 8 Stars, 1 Fork
A console tool which manages your github webhooks efficiently.

ipyaml - 8 Stars, 1 Fork
IPython notebooks with YAML file format

Scrapstagram - 3 Stars, 0 Fork
An Instagram Scrapper

kimo - 3 Stars, 0 Fork
Find OS processes of MySQL queries

February 26, 2017 06:19 AM

February 25, 2017


Catalin George Festila

Linux: OpenCV and using Lucas-Kanade Optical Flow function.

Fist I install OpenCV python module and I try using with Fedora 25.
I used python 2.7 version.

[root@localhost mythcat]# dnf install opencv-python.x86_64 
Last metadata expiration check: 0:21:12 ago on Sat Feb 25 23:26:59 2017.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
opencv x86_64 3.1.0-8.fc25 fedora 1.8 M
opencv-python x86_64 3.1.0-8.fc25 fedora 376 k
python2-nose noarch 1.3.7-11.fc25 updates 266 k
python2-numpy x86_64 1:1.11.2-1.fc25 fedora 3.2 M

Transaction Summary
================================================================================
Install 4 Packages

Total download size: 5.6 M
Installed size: 29 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): opencv-python-3.1.0-8.fc25.x86_64.rpm 855 kB/s | 376 kB 00:00
(2/4): opencv-3.1.0-8.fc25.x86_64.rpm 1.9 MB/s | 1.8 MB 00:00
(3/4): python2-nose-1.3.7-11.fc25.noarch.rpm 543 kB/s | 266 kB 00:00
(4/4): python2-numpy-1.11.2-1.fc25.x86_64.rpm 2.8 MB/s | 3.2 MB 00:01
--------------------------------------------------------------------------------
Total 1.8 MB/s | 5.6 MB 00:03
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Installing : python2-nose-1.3.7-11.fc25.noarch 1/4
Installing : python2-numpy-1:1.11.2-1.fc25.x86_64 2/4
Installing : opencv-3.1.0-8.fc25.x86_64 3/4
Installing : opencv-python-3.1.0-8.fc25.x86_64 4/4
Verifying : opencv-python-3.1.0-8.fc25.x86_64 1/4
Verifying : opencv-3.1.0-8.fc25.x86_64 2/4
Verifying : python2-numpy-1:1.11.2-1.fc25.x86_64 3/4
Verifying : python2-nose-1.3.7-11.fc25.noarch 4/4

Installed:
opencv.x86_64 3.1.0-8.fc25 opencv-python.x86_64 3.1.0-8.fc25
python2-nose.noarch 1.3.7-11.fc25 python2-numpy.x86_64 1:1.11.2-1.fc25

Complete!
[root@localhost mythcat]#
This is my test script with opencv to detect flow using Lucas-Kanade Optical Flow function.
This tracks some points in a black and white video.
First you need:
- one black and white video;
- not mp4 file type file;
- the color args need to be under 4 ( see is 3);
- I used this video:
I used cv2.goodFeaturesToTrack().
We take the first frame, detect some Shi-Tomasi corner points in it, then we iteratively track those points using Lucas-Kanade optical flow.
The function cv2.calcOpticalFlowPyrLK() we pass the previous frame, previous points and next frame.
The returns next points along with some status numbers which has a value of 1 if next point is found, else zero.
That iteratively pass these next points as previous points in next step.
See the code below:
import numpy as np
import cv2

cap = cv2.VideoCapture('candle')

# params for ShiTomasi corner detection
feature_params = dict( maxCorners = 77,
qualityLevel = 0.3,
minDistance = 7,
blockSize = 7 )

# Parameters for lucas kanade optical flow
lk_params = dict( winSize = (17,17),
maxLevel = 1,
criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

# Create some random colors
color = np.random.randint(0,255,(100,3))

# Take first frame and find corners in it
ret, old_frame = cap.read()
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
p0 = cv2.goodFeaturesToTrack(old_gray, mask = None, **feature_params)

# Create a mask image for drawing purposes
mask = np.zeros_like(old_frame)

while(1):
ret,frame = cap.read()
frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

# calculate optical flow
p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)

# Select good points
good_new = p1[st==1]
good_old = p0[st==1]

# draw the tracks
for i,(new,old) in enumerate(zip(good_new,good_old)):
a,b = new.ravel()
c,d = old.ravel()
mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2)
frame = cv2.circle(frame,(a,b),5,color[i].tolist(),-1)
img = cv2.add(frame,mask)

cv2.imshow('frame',img)
k = cv2.waitKey(30) & 0xff
if k == 27:
break

# Now update the previous frame and previous points
old_gray = frame_gray.copy()
p0 = good_new.reshape(-1,1,2)

cv2.destroyAllWindows()
cap.release()
The output of this file is:

February 25, 2017 03:04 PM


Jaime Buelta

Compendium of Wondrous Links vol XI

It has been a while since the last time. More food for though! Python Python 3 upgrade strategy. The time has come to take migrating to python 3 seriously. Another addition to Python-to-c++ compilers, in a similar way to Cython: Pythran. I tested it with code from my recent post $7.11 in four prices and the … Continue reading Compendium of Wondrous Links vol XI

February 25, 2017 01:22 PM


Coding Diet

unittest.mock small gotcha - a humbling tale of failure

Developing a small web application I recently had reason to upgrade from Python 3.4 to Python 3.6. The reason for the upgrade was regarding ordering of keyword arguments and not related to the bug in my test code that I then found. I should have been more careful writing my test code in the first place, so I am writing this down as some penance for not testing my tests robustly enough.

A Simple example program

So here I have a version of the problem reduced down to the minimum required to demonstrate the issue:

import unittest.mock as mock

class MyClass(object):
    def __init__(self):
        pass
    def my_method(self):
        pass

if __name__ == '__main__':
    with mock.patch('__main__.MyClass') as MockMyClass:
        MyClass().my_method()
        MockMyClass.my_method.assert_called_once()

Of course in reality the line MyClass().my_method() was some test code that indirectly caused the target method to be called.

Output in Python 3.4

$ python3.4 mock_example.py
$

No output, leading me to believe my assertions passed, so I was happy that my code and my tests were working. As it turned out, my code was fine but my test was faulty. Here's the output in two later versions of Python of the exact same program given above.

Output in Python 3.5

$ python3.5 mock_example.py
Traceback (most recent call last):
  File "mock_example.py", line 12, in <module>
    MockMyClass.my_method.assert_called_once()
  File "/usr/lib/python3.5/unittest/mock.py", line 583, in __getattr__
    raise AttributeError(name)
AttributeError: assert_called_once

Assertion error, test failing.

Output in Python 3.6

$ python3.6 mock_example.py
Traceback (most recent call last):
  File "mock_example.py", line 12, in <module>
    MockMyClass.my_method.assert_called_once()
  File "/usr/lib/python3.6/unittest/mock.py", line 795, in assert_called_once
    raise AssertionError(msg)
AssertionError: Expected 'my_method' to have been called once. Called 0 times.

Test also failing with a different error message. Anyone who is (pretty) familiar with the unittest.mock standard library module will know assert_called_once was introduced in version 3.6, which is my version 3.5 is failing with an attribute error.

My test was wrong

The problem was, my original test was not testing anything at all. The 3.4 version of the unittest.mock standard library module did not have a assert_called_once. The mock, just allows you to call any method on it, to see this you can try changing the line:

        MockMyClass.my_method.assert_called_once()

to

        MockMyClass.my_method.blahblah()

With python3.4, python3.5, and python3.6 this yields no error. So in the original program you can avoid the calling MyClass.my_method at all:

if __name__ == '__main__':
    with mock.patch('__main__.MyClass') as MockMyClass:
        # Missing call to `MyClass().my_method()`
        MockMyClass.my_method.assert_called_once() # In 3.4 this still passes.

This does not change the (original) results, python3.4 still raises no error, whereas python3.5 and python3.6 are raising the original errors.

So although my code turned out to be correct (at least in as much as the desired method was called), had it been faulty (or changed to be faulty) my test would not have complained.

The Actual Problem

My mock was wrong. I should instead have been patching the actual method within the class, like so:

if __name__ == '__main__':
    with mock.patch('__main__.MyClass.my_method') as mock_my_method:
        MyClass().my_method()
        mock_my_method.assert_called_once()

Now if we try this in all version 3.4, 3.5, and 3.6 of python we get:

$ python3.4 mock_example.py 
$ python3.5 mock_example.py 
Traceback (most recent call last):
  File "mock_example.py", line 12, in <module>
    mock_my_method.assert_called_once()
  File "/usr/lib/python3.5/unittest/mock.py", line 583, in __getattr__
    raise AttributeError(name)
AttributeError: assert_called_once
$ python3.6 mock_example.py 
$ 

So Python 3.4 and 3.6 pass as we expect. But Python3.5 gives an error stating that there is no assert_called_once method on the mock object, which is true since that method was not added until version 3.6. This is arguably what Python3.4 should have done.

It remains to check that the updated test fails in Python3.6, so we comment out the call to MyClass().my_method:

$ python3.6 mock_example.py 
Traceback (most recent call last):
  File "mock_example.py", line 12, in <module>
    mock_my_method.assert_called_once()
  File "/usr/lib/python3.6/unittest/mock.py", line 795, in assert_called_once
    raise AssertionError(msg)
AssertionError: Expected 'my_method' to have been called once. Called 0 times.

This is the test I should have performed with my original test. Had I done this I would have seen that the test passed in Python3.4 regardless of whether the method in question was actually called or not.

So now my test works in python3.6, fails in python3.5 because I'm using the method assert_called_once which was introduced in python3.6. Unfortunately it incorrectly passes in python3.4. So if I want my code to work properly for python versions earlier than 3.6, then I can essentially implement assert_called_once() with assert len(mock_my_method.mock_calls) == 1. If we do this then my test passes in all three version of python and fails in all three if we comment out the call MyClass().my_method().

Conclusions

I made an error in writing my original test, but my real sin was that I was a bit lazy in that I did not make sure that my tests would fail, when the code was incorrect. In this instance there was no problem with the code only the test, but that was luck. So for me, this served as a reminder to check that your tests can fail. It may be that mutation testing would have caught this error.

February 25, 2017 12:36 PM


Django Weekly

Django Weekly 27 - Advanced Django querying, Django Signals, Elasticbeanstalk and more

Worthy Read

Application of Django's Case, When queryset operators for sorting events by date.
orm
,
Query

We help companies like Airbnb, Pfizer, and Artsy find great developers. Let us find your next great hire. Get started today.
sponsor

Django Signals are extremely useful for decoupling modules. They allow a low-level Django app to send events for other apps to handle without creating a direct dependency. Signals are easy to set up, but harder to test. So In this article, I’m going to walk you through implementing a context manager for testing Django signals, step by step.
signals

Here are the release notes - https://docs.djangoproject.com/en/dev/releases/1.11/
release

Kenneth Love and Trey Hunner will answer your questions about how we use Django's forms.
forms

deployment

What to test and where to test is a common question I get asked. In this video learn about different types of tests you can write in the context of Django.
video

In case you missed the news, DjangoCon US 2017 will take place in beautiful Spokane, Washington, from August 13-18, 2017.
djangocon

In this tutorial I will cover a few strategies to create Django user sign up/registration. Usually I implement it from scratch. You will see it’s very straightforward.
signup

Run your django CMS project as a single-page application (SPA) with vue.js and vue-router.
vuejs

elastic beanstalk


Projects

drf-writable-nested - 11 Stars, 0 Fork
Writable nested model serializer for Django REST Framework.

Django reusable app that uses Celery Inspect command to monitor workers/tasks via the Django REST Framework

djeasy - 5 Stars, 3 Fork
Django simple quick setup

February 25, 2017 10:47 AM


Daniel Bader

Installing Python and Pip on Windows

Installing Python and Pip on Windows

In this tutorial you’ll learn how to set up Python and the Pip package manager on Windows 10, completely from scratch.

Step 1: Download the Python Installer

The best way to install Python on Windows is by downloading the official Python installer from the Python website at python.org.

To do so, open a browser and navigate to https://python.org/. After the page has finished loading, click Downloads.

Under Downloads → Download for Windows, click the “Python 3.X.X” (or “Python 2.X.X”) button to begin downloading the installer.

Sidebar: 64-bit Python vs 32-bit Python

If you’re wondering whether you should use a 32-bit or a 64-bit version of Python then you might want to go with the 32-bit version.

It’s sometimes still problematic to find binary extensions for 64-bit Python on Windows, which means that some third-party modules might not install correctly with a 64-bit version of Python.

My thinking is that it’s best to go with the version currently recommended on python.org. If you click the Python 3 or Python 2 button under “Download for Windows” you’ll get just that.

Remember that if you get this choice wrong and you’d like to switch to another version of Python you can just uninstall Python and then re-install it by downloading another installer from python.org.

Step 2: Run the Python Installer

Once the Python installer file has finished downloading, launch it by double-clicking on it in order to begin the installation.

Be sure to select the Add Python X.Y to PATH checkbox in the setup wizard.

Click Install Now to begin the installation process. The installation should finish quickly and then Python will be ready to go on your system. We’re going to make sure everything was set up correctly in the next step.

Step 3: Verify Python Was Installed Correctly

After the Python installer finished its work Python should be installed on your system. Let’s make sure everything went correctly by testing if Python can be accessed from the Windows Command Prompt:

  1. Open the Windows Command Prompt by launching cmd.exe
  2. Type pip and hit Return
  3. You should see the help text from Python’s “pip” package manager. If you get an error message running pip go through the Python install steps again to make sure you have a working Python installation. Most issues you will encounter here will have something to do with the PATH not being set correctly. Re-installing and making sure that the “Add Python to PATH” option is enabled in the installer should resolve this.

What Now?

Assuming everything went well and you saw the output from Pip in your command prompt window—Congratulations, you just installed Python on your system!

Wondering where to go from here? Click here to get some pointers for Python beginners.

February 25, 2017 12:00 AM

February 24, 2017


DataCamp

Matplotlib Cheat Sheet: Plotting in Python

Data visualization and storytelling with your data are essential skills that every data scientist needs to communicate insights gained from analyses effectively to any audience out there. 

For most beginners, the first package that they use to get in touch with data visualization and storytelling is, naturally, Matplotlib: it is a Python 2D plotting library that enables users to make publication-quality figures. But, what might be even more convincing is the fact that other packages, such as Pandas, intend to build more plotting integration with Matplotlib as time goes on.

However, what might slow down beginners is the fact that this package is pretty extensive. There is so much that you can do with it and it might be hard to still keep a structure when you're learning how to work with Matplotlib.   

DataCamp has created a Matplotlib cheat sheet for those who might already know how to use the package to their advantage to make beautiful plots in Python, but that still want to keep a one-page reference handy. Of course, for those who don't know how to work with Matplotlib, this might be the extra push be convinced and to finally get started with data visualization in Python. 

(By the way, if you want to get started with this Python package, you might want to consider our Matplotlib tutorial.)

You'll see that this cheat sheet presents you with the six basic steps that you can go through to make beautiful plots. 

Check out the infographic by clicking on the button below:

Python Matplotlib cheat sheet

With this handy reference, you'll familiarize yourself in no time with the basics of Matplotlib: you'll learn how you can prepare your data, create a new plot, use some basic plotting routines to your advantage, add customizations to your plots, and save, show and close the plots that you make.

What might have looked difficult before will definitely be more clear once you start using this cheat sheet! 

Also, don't miss out on our other cheat sheets for data science that cover SciPyNumpyScikit-LearnBokehPandas and the Python basics.

February 24, 2017 09:11 PM


Weekly Python StackOverflow Report

(lxii) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2017-02-24 19:53:04 GMT


  1. Why is x**4.0 faster than x**4 in Python 3? - [119/2]
  2. How to limit the size of a comprehension? - [20/4]
  3. How to properly split this list of strings? - [13/5]
  4. Why is the __dict__ of instances so small in Python 3? - [11/1]
  5. Python Asynchronous Comprehensions - how do they work? - [10/2]
  6. Immutability in Python - [10/2]
  7. Order-invariant hash in Python - [9/2]
  8. Assign a number to each unique value in a list - [8/7]
  9. How to add a shared x-label and y-label to a plot created with pandas' plot? - [8/3]
  10. What happens when I inherit from an instance instead of a class in Python? - [8/2]

February 24, 2017 07:53 PM


Will Kahn-Greene

Who uses my stuff?

Summary

I work on a lot of different things. Some are applications, are are libraries, some I started, some other people started, etc. I have way more stuff to do than I could possibly get done, so I try to spend my time on things "that matter".

For Open Source software that doesn't have an established community, this is difficult.

This post is a wandering stream of consciousness covering my journey figuring out who uses Bleach.

Read more… (4 mins to read)

February 24, 2017 04:00 PM


Rene Dudfield

setup.cfg - a solution to python config file soup? A howto guide.

Sick of config file soup cluttering up your repo? Me too. However there is a way to at least clean it up for many python tools.


Some of the tools you might use and the config files they support...
  • flake8 - .flake8, setup.cfg, tox.ini, and config/flake8 on Windows
  • pytest - pytest.ini, tox.ini, setup.cfg
  • coverage.py - .coveragerc, setup.cfg, tox.ini
  • mypy - setup.cfg, mypy.ini
  • tox - tox.ini
 Can mypy use setup.cfg as well?
OK, you've convinced me. -- Guido

With that mypy now also supports setup.cfg, and we can all remove many more config files.

The rules for precedence are easy:
  1. read --config-file option - if it's incorrect, exit
  2. read [tool].ini - if correct, stop
  3. read setup.cfg

 

How to config with setup.cfg?

Here's a list to the configuration documentation for setup.cfg.

What does a setup.cfg look like now?

Here's an example setup.cfg for you with various tools configured. (note these are nonsensical example configs, not what I suggest you use!)

## http://coverage.readthedocs.io/en/latest/config.html
#[coverage:run]
#timid = True

## http://pytest.org/latest/customize.html#adding-default-options
# [tool:pytest]
# addopts=-v --cov pygameweb pygameweb/ tests/

## http://mypy.readthedocs.io/en/latest/config_file.html
#[mypy]
#python_version = 2.7

#[flake8]
#max-line-length = 120
#max-complexity = 10
#exclude = build,dist,docs/conf.py,somepackage/migrations,*.egg-info

## Run with: pylint --rcfile=setup.cfg somepackage
#[pylint]
#disable = C0103,C0111
#ignore = migrations
#ignore-docstrings = yes
#output-format = colorized



February 24, 2017 02:01 PM


Bhishan Bhandari

Implementing Stack using List in Python – Python Programming Essentials

Intro Stack is a collection of objects inserted and removed in a last-in first-out fashion (LIFO). Objects can be inserted onto stack at any time but only the object inserted last can be accessed or removed which coins the object to be top of the stack. Realization of Stack Operations using List   Methods Realization […]

February 24, 2017 05:52 AM


Vasudev Ram

Perl-like "unless" (reverse if) feature in Python

By Vasudev Ram



Flowchart image attribution

I was mentally reviewing some topics to discuss for a Python training program I was running. Among the topics were statements, including the if statement. I recollected that some languages I knew of, such as Perl, have an unless statement, which is like a reverse if statement, in that only the first nested suite (of statements) is executed if the Boolean condition is false, whereas only the second nested suite is executed if the condition is true. See the examples below.

The term "suite" used above, follows the terminology used in Python documentation such as the Python Language Reference; see this if statement definition, for example.

That is, for the if statement:
if condition:
suite1 # nested suite 1
else:
suite2 # nested suite 2
results in suite1 being run if condition is true, and suite2 being run if condition is false, whereas, for the unless statement:
unless condition:
suite1
else:
suite2
, the reverse holds true.

Of course, there is no unless statement in Python. So I got the idea of simulating it, at least partially, with a function, just for fun and as an experiment. Here is the first version, in file unless.py:
# unless.py v1

# A simple program to partially simulate the unless statement
# (a sort of reverse if statement) available in languages like Perl.
# The simulation is done by a function, not by a statement.

# Author: Vasudev Ram
# Web site: https://vasudevram.github.io
# Blog: https://jugad2.blogspot.com
# Product store: https://gumroad.com

# Define the unless function.
def unless(cond, func_if_false, func_if_true):
if not cond:
func_if_false()
else:
func_if_true()

# Test it.
def foo(): print "this is foo"
def bar(): print "this is bar"

a = 1
# Read the call below as:
# Unless a % 2 == 0, call foo, else call bar
unless (a % 2 == 0, foo, bar)
# Results in foo() being called.

a = 2
# Read the call below as:
# Unless a % 2 == 0, call foo, else call bar
unless (a % 2 == 0, foo, bar)
# Results in bar() being called.
Here is the output:
$ python unless.py
this is foo
this is bar
This simulation of unless works because functions are objects in Python (since almost everything is an object in Python, like almost everything in Unix is a file), so functions can be passed to other functions as arguments (by passing just their names, without following the names with parentheses).

Then, inside the unless function, when you apply the parentheses to those two function names, they get called.

This approach to simulation of the unless statement has some limitations, of course. One is that you cannot pass arguments to the functions [1]. (You could still make them do different things on different calls by using global variables (not good), reading from files, or reading from a database, so that their inputs could vary on each call).

[1] You can actually pass arguments to the functions in a few ways, such as using the *args and **kwargs features of Python, as additional arguments to unless() and then forwarding those arguments to the func_if_false() and func_if_true() calls inside unless().

Another limitation is that this simulation does not support the elif clause.

However, none of the above limitations matter, of course, because you can also get the effect of the unless statement (i.e. a reverse if) by just negating the Boolean condition (with the not operator) of an if statement. As I said, I just tried this for fun.

The image at the top of the post is of a flowchart.

For something on similar lines (i.e. simulating a language feature with some other code), but for the C switch statement simulated (partially) in Python, see this post I wrote a few months ago:

Simulating the C switch statement in Python

And speaking of Python language features, etc., here is a podcast interview with Guido van Rossum (creator of the Python language), about the past, present and future of Python.


Enjoy.

- Vasudev Ram - Online Python training and consulting

Get updates (via Gumroad) on my forthcoming apps and content.

Jump to posts: Python * DLang * xtopdf

Subscribe to my blog by email

My ActiveState Code recipes

Follow me on: LinkedIn * Twitter

Managed WordPress Hosting by FlyWheel



February 24, 2017 04:33 AM

February 23, 2017


Catalin George Festila

The bad and good urllib.

This is a simple python script:

import urllib
opener = urllib.FancyURLopener({})
f = opener.open("http://www.ra___aer.ro/")
d=f.read()
fo = open('workfile.txt', 'w')
fo.write(d)
fo.close()
The really bad news come from here:
http://blog.blindspotsecurity.com/2017/02/advisory-javapython-ftp-injections.html

February 23, 2017 03:11 PM


Stories in My Pocket

What is Your Burnout Telling You?

I am glad that mental health is being discussed more often in the programming world. In particular, I would like to thank Kenneth Reitz for his transparency over the last few years and contributing his recent essay on developer burnout.

Having just recently experienced a period of burnout, I would like to share some ideas that might help you skirt by it in your own life.

And, before I go on, let me say loud and clear, there is absolutely no shame in experiencing, or reaching out for help from, anxiety, fear, anger, burnout, depression, or any other mental condition.

If you think you might be struggling with something, please reach out for help! You might have to reach out several times to multiple sources, but remember you are too valuable to waste time with this funk.




My experience is different than Kenneth's. Instead of maintaining open source projects, I work on a small team that builds web projects to meet the business needs of our clients.

Some things he wrote about didn't quite resonate with my situation, so my I hope is this article will speak to others who have similar experiences to mine.

Trapped?

I've identified that I'm burnt out when I have the feeling of being trapped in a situation with an uncertain end.

One time, I was the only programmer scheduled on a project for a period of months. Another time, I was experiencing a lot of non-constructive criticism for the graphic design work I was doing.

In these situations and others, I felt trapped in an emotional well, thinking that there was no way someone else could help, or would want to help me, and I had no idea when this situation would end.

Emotions like these serve one very crucial role: they are a great indicator of what's going on internally. Instead of allowing these emotions to fuel axiety and make things worse, I'm learning there is great freedom in realizing: 

  • Emotions like these trigger thoughts that aren't true.
  • One does not have to respond to the emotions you are experiencing at any given moment. Instead, treat them as what they are: a warning light that something is off and needs to be adjusted.

In this case, I've identified that I need to be vocal to my manager whenever I realize I need support.

In my most recent time of burnout, I waited far too long before letting my manager know. I have a "workhorse" mentality that contributed to my burnout. I thought I could shoulder the load for a period of time, to get my team through. But instead of helping my team with this behavior, I hurt it by the reduced quality of my contributions to the project. And in the long run, I made things more difficult for my team.

Cool your jets

In times of burnout, I felt pressure coming from one area of my life that was off balance, say from my team at work. As the burnout condition persisted, the pressure would spread to other areas of my life.

I believe my recent experience of burnout was made worse by this very blog, for I launched it when I was in a period of light burnout. Once my work was public, my ambition put a lot of pressure on me to create high-quality content and build an audience. That pressure was too much, and I realized I had to step back.

In this case, I had no one to delegate the work to, so progress would have to stop. I had to be okay with that.

Throughout the last few months, I found it challenging to restrain my ambition, but I had a few great insights that helped to calm it down.

One was from one of the recent three-part episodes of the Developer Tea podcast where Jonathan interviewed Kalid Azad. Kalid has built an amazing collection of writings at his website, Better Explained. He revealed in the interview that he hardly contributed to his website for one year in the middle of its life, and yet it is a very successful site. So what do I have to worry about?

The second was from a book I recently started reading from Chip Ingram on the spiritual side of ambition, in which he draws on research to segment people into four groups. He advises the group I identify with to cool your jets, to allow a deeper purpose to set in and not just check off tasks on my to-do list.

These relived a lot of pressure from my ambition, and since then I've heard similar advice from other parties. So if any of you have similar ambition, I hope you can take heart that there is a large community of successful people out there that are telling us to be patient.

With that advice in mind, I'm going to try to allow myself to write during this recovery period, but only when I'm inspired, and only while it's still fun.

Prevention and recovery via balance

People believe living a balanced life will prevent many issues like burnout, but what does this balance look like?

I like Dan Miller's approach in his book, 48 Days to the Work You Love. He describes life as a wheel with seven spokes. When I'm experiencing burnout conditions, it's a good time to reflect and see if I'm out of balance with my:

  • Career
  • Finances
  • Social Life/Media
  • Family
  • Physical Fitness
  • Personal Development
  • Spirituality

Sometimes reviewing the list will show me that I should pay more attention to my physical fitness by taking walking breaks or going to the gym. Other times, I've found that my career or personal development segments are receiving too much attention, and I need to reduce my investment in them.

Practical advice for prevention

Like other people, I highly recommend hobbies to help prevent burnout. Though unlike most of them, I have found that given certain restraints, coding outside of work can be relaxing.

I have found that the more unrelated something is from what I'm doing at work, the better I can enjoy it. For example, when I was working in a JavaScript-heavy ASP.net application, I found coding in python to be a breath of fresh air.

When I was working on a large-scale, full-stack PHP website, exploring the Arduino platform was a welcome excursion.

As I mentioned in the last section, you need to have balance, so you might need to lay off of your hobby coding and pick up a coloring book or something for a few weeks to give your mind a chance to relax. You'll have a chance to pick back up on the coding project before too long.

In Conclusion
  • Remember that burnout can happen to anyone, and that there's no shame in talking about it.
  • Try to find the warning signs your body is giving you.
  • If you are going through a more difficult time, be open and vocal about what you're experiencing—especially with trusted friends and your manager.
  • Many thoughts triggered by anxiety and stress aren't true.
  • It's okay to take a break.
  • In the end burnout, anxiety, depression and the like are signs that your body is telling you to make adjustments.


February 23, 2017 01:47 PM


Ned Batchelder

A tale of two exceptions, continued

In my last blog post, A tale of two exceptions, I laid out the long drawn-out process of trying to get a certain exception to make tests skip in my test runner. I ended on a solution I liked at the time.

But it still meant having test-specific code in the product code, even if it was only a single line to set a base class for an exception. It didn't feel right to say "SkipTest" in the product code, even once.

In that blog post, I said,

One of the reasons I write this stuff down is because I'm hoping to get feedback that will improve my solution, or advance my understanding. ... a reader might object and say, "you should blah blah blah."

Sure enough, Ionel said,

A better way is to handle this in coverage's test suite. Possible solution: wrap all your tests in a decorator that reraises with a SkipException.

I liked this idea. The need was definitely a testing need, so it should be handled in the tests. First I tried doing something with pytest to get it to do the conversion of exceptions for me. But I couldn't find a way to make it work.

So: how to decorate all my tests? The decorator itself is fairly simple. Just call the method with all the arguments, and return its value, but if it raises StopEverything, then raise SkipTest instead:

def convert_skip_exceptions(method):
    """A decorator for test methods to convert StopEverything to SkipTest."""
    def wrapper(*args, **kwargs):
        """Run the test method, and convert exceptions."""
        try:
            result = method(*args, **kwargs)
        except StopEverything:
            raise unittest.SkipTest("StopEverything!")
        return result
    return wrapper

But decorating all the test methods would mean adding a @convert_skip_exceptions line to hundreds of test methods, which I clearly was not going to do. I could use a class decorator, which meant I would only have to add a decorator line to dozens of classes. That also felt like too much to do and remember to do in the future when I write new test classes.

It's not often I say this, but: it was time for a metaclass. Metaclasses are one of the darkest magics Python has, and they can be mysterious. At heart, they are simple, but in a place you don't normally think to look. Just as a class is used to make objects, a metaclass is used to make classes. Since there's something I want to do everytime I make a new class (decorate its methods), a metaclass gives me the tools to do it.

class SkipConvertingMetaclass(type):
    """Decorate all test methods to convert StopEverything to SkipTest."""
    def __new__(mcs, name, bases, attrs):
        for attr_name, attr_value in attrs.items():
            right_name = attr_name.startswith('test_')
            right_type = isinstance(attr_value, types.FunctionType)
            if right_name and right_type:
                attrs[attr_name] = convert_skip_exceptions(attr_value)

        return super(SkipConvertingMetaclass, mcs).__new__(mcs, name, bases, attrs)

There are details here that you can skip as incantations if you like. Classes are all instances of "type", so if we want to make a new thing that makes classes, it derives from type to get those same behaviors. The method that gets called when a new class is made is __new__. It gets passed the metaclass itself (just as classmethods get cls and instance methods get self), the name of the class, the tuple of base classes, and a dict of all the names and values defining the class (the methods, attributes, and so on).

The important part of this metaclass is what happens in the __new__ method. We look at all the attributes being defined on the class. If the name starts with "test_", and it's a function, then it's a test method, and we decorate the value with our decorator. Remember that @-syntax is just a shorthand for passing the function through the decorator, which we do here the old-fashioned way.

Then we use super to let the usual class-defining mechanisms in "type" do their thing. Now all of our test methods are decorated, with no explicit @-lines in the code. There's only one thing left to do: make sure all of our test classes use the metaclass:

CoverageTestMethodsMixin = SkipConvertingMetaclass('CoverageTestMethodsMixin', (), {})

class CoverageTest(
    ... some other mixins ...
    CoverageTestMethodsMixin,
    unittest.TestCase,
):
    """The base class for all coverage.py test classes."""

Metaclasses make classes, just the way classes make instances: you call them. Here we call our with the arguments it needs (class name, base classes, and attributes) to make a class called CoverageTestMethodsMixin.

Then we use CoverageTestMethodsMixin as one of the base classes of CoverageTest, which is the class used to derive all of the actual test classes.

Pro tip: if you are using unittest-style test classes, make a single class to be the base of all of your test classes, you will be glad.

After all of this class machinations, what have we got? Our test classes all derive from a base class which uses a metaclass to decorate all the test methods. As a result, any test which raises StopEverything will instead raise SkipTest to the test runner, and the test will be skipped. There's now no mention of SkipTest in the product code at all. Better.

February 23, 2017 12:36 PM


Experienced Django

Python, Snowboarding, and time

It’s mid-snowboard season here in North America, so time for learning, researching and writing is scarce.  I wanted to post to keep the blog alive and to give a shout to a friend who’s starting up a business selling some cool python swag.

I’m not in marketing so I won’t even pretend to give a pitch, other than to say I bought a mug (from coffee import *) and love it!

If you’re looking for something unique to give to a python geek (like yourself!) check it out at NerdLettering

OK.  I promise this blog won’t turn into a sales platform AND I promise to be back in the spring with more learning in python and django.  But for now, let it SNOW!!!!

February 23, 2017 02:07 AM

February 22, 2017


Eray Özkural (examachine)

Appreciation of Python's Elegance: Metaclasses and Design Patterns

Sometimes, it amazes me how elegant Python is. A class system that is almost as powerful as CLOS, yet easy to use and comprehend. It is truly one of the most powerful programming systems in the world, if you show

February 22, 2017 10:52 PM


Python Engineering at Microsoft

Python support in Visual Studio 2017

Over the last few months, Visual Studio 2017 has been in preview and many of you have been trying it out and providing feedback. We are very appreciative of everyone who has taken the time to do this.

As many noticed, during an update in January we removed Python support from the VS 2017 Release Candidate. This was done suddenly and without warning, and for that we apologize. We will be making a preview available at launch, and Python support will return to the main VS 2017 installer in an update.

Visual Studio 2017 RC installer showing Python development workload

I want to be clear that Python support will be returning in one of the first VS 2017 updates. We removed it only because we were not going to meet product-completeness targets that are needed to be a core part of the initial Visual Studio release, and are already well on our way to completing these. Specifically, we needed to translate our user interfaces and messages into the set of languages supported by Visual Studio in time for the main release. As anyone who has attempted to provide software with multiple languages will know, this is a unique challenge that requires changes throughout the entire project. We were not confident that we could do that and also resolve any new issues in time for March 7.

In the past, we released standalone installers to add Python support into Visual Studio. The extensibility changes in VS 2017 made simply going back to a standalone installer expensive, and this work would be thrown away when Python support is integrated directly into Visual Studio.

So here’s what we’re doing: around the time of the Visual Studio 2017 release, we will release a separate preview version of VS 2017 that includes Python support. We are currently planning for simultaneous release (March 7), but the stable release is the highest priority and plans for the preview may change.

During the Visual Studio 2017 online release event, there will be more details announced. However, here is what we can tell you so far:

Currently we are expecting Python support to be in the preview release for a few months, depending on our confidence in stability and user feedback. Once we move from preview to release, there will be an update and you’ll be able to select the Python workload in the stable release of Visual Studio. If you want to keep receiving previews of Python and other VS work, you can keep the preview installed, or you can delete it.

We want to thank everyone for your patience and sticking with us as we get ready for release. In many ways, Visual Studio 2017 is going to be our best release yet, and we are looking forward to being able to let all of you use it.

February 22, 2017 08:00 PM


Catalin George Festila

The twill python module with Fedora 25.

Today I tested the twill python module with python 2.7 and Fedora 25.
This is: a scripting system for automating Web browsing. Useful for testing Web pages or grabbing data from password-protected sites automatically.
To install this python module I used pip command:
[root@localhost mythcat]# pip install twill
Collecting twill
Downloading twill-1.8.0.tar.gz (176kB)
100% |████████████████████████████████| 184kB 2.5MB/s
Installing collected packages: twill
Running setup.py install for twill ... done
Successfully installed twill-1.8.0

Let's try some tests:

[mythcat@localhost ~]$ python
Python 2.7.13 (default, Jan 12 2017, 17:59:37)
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from twill import get_browser
>>> b = get_browser()
>>>
>>> from twill.commands import *
>>> go("http://www.python.org/")
==> at https://www.python.org/
u'https://www.python.org/'
>>> b.showforms()

Form #1
## ## __Name__________________ __Type___ __ID________ __Value__________________
1 q search id-searc ...
To talk to the Web browser directly, call the get_browser function.
You can see most of the twill commands by using:
>>> import twill.shell
>>> twill.shell.main()

-= Welcome to twill! =-

current page: https://www.python.org/widgets
>> ?

Undocumented commands:
======================
add_auth fa info save_html title
add_extra_header find load_cookies setglobal url
agent follow notfind setlocal
back formaction redirect_error show
clear_cookies formclear redirect_output show_cookies
clear_extra_headers formfile reload show_extra_headers
code formvalue reset_browser showforms
config fv reset_error showhistory
debug get_browser reset_output showlinks
echo getinput run sleep
exit getpassword runfile submit
extend_with go save_cookies tidy_ok

current page: https://www.python.org/widgets
>>
Basic is used by setlocal to fill website forms and the go function.
Ban can be very good for some tasks.
Now twill also provides a simple wrapper for mechanize functionality with the API is still unstable.

February 22, 2017 03:37 PM