skip to navigation
skip to content

Planet Python

Last update: October 23, 2017 09:46 PM

October 23, 2017


Weekly Python Chat

Comprehensions Chat

What are list comprehensions and how are they different from for loops? And why are list comprehensions often so hard to read?

This week we're doing a live Q&A on list comprehensions in Python and comprehension-related topics like loops and map/filter.

October 23, 2017 06:30 PM


Doug Hellmann

pathlib — Filesystem Paths as Objects — PyMOTW 3

pathlib includes classes for managing filesystem paths formatted using either the POSIX standard or Microsoft Windows syntax. It includes so called “pure” classes, which operate on strings but do not interact with an actual filesystem, and “concrete” classes, which extend the API to include operations that reflect or modify data on the local filesystem. Read …

October 23, 2017 01:00 PM


Mike Driscoll

PyDev of the Week: Paul Kehrer

This week we welcome Paul Kehrer (@reaperhulk) as our PyDev of the Week! Paul is a core developer of the cryptography package. He also the maintainer of pyOpenSSL, bcrypt, and PyNaCl. You can get a feel for what he’s up to over on Github. Paul also has a fun blog that is worth checking out. Let’s take a few moments to get to know our fellow Pythonista better!

Can you tell us a little about yourself (hobbies, education, etc):

My name is Paul Kehrer, but many people probably only know me by my handle reaperhulk (it’s an anagram of my name). I have a degree in physics but learned to program on the side in college and stumbled into a security job directly out of school. I travel often (much of my time is spent in Asia these days), I love kiteboarding (typically in Aruba or Maui), and I also like to bake.

Why did you start using Python?

Sadly I don’t have a story about how I discovered the amazing community that is Python. At a previous job I had been doing a lot of Ruby work, but I took a new job in early 2013 and the company that hired me (Rackspace) wanted me to contribute to a security project written in Python. It was a relatively easy transition, although the two communities have wildly different opinions about the advisability of meta-programming!

What other programming languages do you know and which is your favorite?

I’ve written PHP, Basic, Objective-C, Ruby, C, Go, and probably others at various points in my career. I currently write in Python, C, and Go on a regular basis. I don’t think I have a favorite language because no language is perfect. I admire some of the tooling in Go (single binary, gofmt) but vendoring is not great. Python’s packaging story has been bad for a long time, but the heroic efforts of the PyPA are really starting to pay off at this point and we can hope that eventually reasonable distribution to end users will be possible.

What projects are you working on now?

Well I’m a maintainer on every Python Cryptographic Authority project, which, in addition to cryptography itself includes pyOpenSSL, bcrypt, and PyNaCl. My primary open source focus, however, is on cryptography. It is both the most rewarding and most complex of all the projects I participate in (from both a cryptographic and packaging standpoint).

Outside of that I am one of the creators of Frinkiac (and Morbotron, Master of All Science, and Capital Beat). Those projects take an increasing amount of time as we expand our supported shows and add features!

Which Python libraries are your favorite (core or 3rd party)?

I think Python developers might be legally required to mention requests here, but additionally I’ll mention pytest (which cryptography uses extremely heavily), cffi (the One True Way to call C ABI from Python), and magic-wormhole (Brian Warner’s fantastic cryptographically secure file sharing utility).

How did you get started with the cryptography project?

When I first started programming in Python I quickly discovered that the state of cryptographic libraries in Python left much to be desired. My boss put me in touch with some other people inside Rackspace who were interested in building a new library and had already started putting together a skeleton (along with several others from outside of Rackspace). From there I just started submitting PRs! It’s been just over 4 years since I started (here’s my first PR! https://github.com/pyca/cryptography/pull/48) and in that time we’ve gone from completely unknown to over 5 million downloads per month from PyPI.

What kinds of challenges have you faced working in open source?

I see two big issues day to day. One is the sustainability of open source projects. Businesses frequently derive enormous economic benefit from open source packages, but they do not contribute back in proportion with the advantage they derive from it. This is problematic because the vast, vast majority of open source projects are done in people’s spare time and burnout is a large issue. Even PyCA, which has some business backing (some of my time every week is dedicated to open source work and we have a large budget for our massive continuous integration setup), has only 2-4 people contributing across the entire organization. In the case of cryptography there are only two active maintainers, despite its immense popularity.

Second, parts of the open source community can be extremely toxic. I see this most often in the form of people angry about a bug or feature that is missing. This sort of entitlement is extremely discouraging to maintainers and should be avoided at all costs. Fortunately, the PyCA projects have mostly not seen this sort of behavior, but it’s hardly unusual elsewhere. Please, when something takes longer than you think it should, remember that open source maintainers are people with lives and jobs and any number of reasons why they may not do what you want/need immediately. If you want it to happen faster, get involved! Or just thank them for all the time and effort they spend to make your life easier.

Do you have any advice for new developers that would like to join an open source project?

There are many ways to get involved in a project. One of the best is to just sit down and spend a lot of time improving the documentation for a project you want to work on. Not only will the maintainers love it, it’s a great way to learn more about the API such that when you want to add a feature you can understand the philosophy behind the project.

Another avenue is to join the conversation in their mailing list, IRC channel, slack, issue tracker, etc. Lurking for a bit and learning what they’re working on and what they’re interested in solving can make it much easier for you to offer to help. Remember that saying “Hey can I help?” is great, but it does require the maintainer to spend significant effort to determine your skill level, commitment, and come up with work for you. If you say “Hey I’d like to help with and here’s my proposal based on ” that’s much easier!

That said, don’t feel like you need to be an expert on a project before you can say anything. Expressing interest and staying communicative when the maintainers explain things is a perfectly good first step.

Is there anything else you’d like to say?

You should definitely go to https://frinkiac.com, https://morbotron.com, https://masterofallscience.com, and https://capitalbeat.us and tell all your friends to do it too 😉

Also, if you’re an open source maintainer did you know that PyPI writes useful statistics to a publicly available BigQuery? You can make all sorts of maintenance decisions with actual data! More info here if you’re interested: https://langui.sh/2016/12/09/data-driven-decisions/

Thanks for doing the interview!

October 23, 2017 12:30 PM


Blue Yonder Tech

Blue Yonder at PyCon.DE 2017 and PyData Karlsruhe

  Blue Yonder uses Python throughout its technology stack. We are aware that we profit a lot from open source and it has been always our policy to give back to the community. We have open sourced libraries like turbodbc, sqlalchemy_exasol, Mesos … Read More

Der Beitrag Blue Yonder at PyCon.DE 2017 and PyData Karlsruhe erschien zuerst auf Blue Yonder Technology Blog.

October 23, 2017 11:53 AM


PyCon

PyCon Opens Financial Aid Applications

Even though PyCon prides itself on being an affordable conference, registration is one of several expenses an attendee must incur, and it’s likely the smallest one. Flying, whether halfway around the world or from a few hundred miles away, is more expensive. Staying in a hotel for a few days is also more expensive. All together, the cost of attending a conference can become prohibitively expensive. That’s where our Financial Aid program comes in. We’re opening applications for Financial Aid today, and we’ll be accepting them through February 15, 2018.

Once you have an account on the site, you can apply here or through your dashboard.

We offer need-based grants to enable people from across our community to attend PyCon. The criteria for evaluating requests takes into account several things, such as whether the applicant is a student, unemployed, or underemployed; their geographic location; and their involvement in both the conference and the greater Python community. Those presenting at the conference are additionally prioritized for grants.


For those proposing talks, tutorials, or posters, selecting the “I require a speaker grant if my proposal is accepted” box on your speaker profile serves as your request, so you won’t need to apply separately. Upon acceptance, we’ll contact those speakers to gather the appropriate information. Additionally, we do not expose grant requests to reviewers while evaluating those proposals to fill our schedule. The Program Committee evaluates proposals on the basis of their presentation, and later the Financial Aid group comes in and looks at how we can help our speakers.


Our process aims to help a large amount of people with partial grants, as opposed to covering full expenses for a small amount of people. Based on individual need, we craft grant amounts that we hope can turn PyCon from inaccessible to reality. While some direct costs—like those associated with PyCon itself—are discounted or waived, external costs such as travel are handled via reimbursement, where the attendee pays and then submits receipts to be paid back an amount based on their grant. For the full details, see our FAQ at https://us.pycon.org/2018/financial-assistance/faq/ and contact pycon-aid@python.org with further questions.


The Financial Aid program simply would not be possible without the help of the Python Software Foundation, which provides a budget for PyCon to work with. In fact, PyCon often overspends the aid budget, trading revenue for supporting an increasingly diverse and international delegation at the conference. We’ll make that move any time we can.


For more information about Financial Aid, see https://us.pycon.org/2018/financial-assistance.


Our Call for Proposals is open! Tutorial presentations are due November 24, while talk, poster, and education summit proposals are due January 3. For more information, see https://us.pycon.org/2018/speaking/.

October 23, 2017 11:20 AM


Reuven Lerner

Aha! Preview this week’s live, online Python courses

In just 48 hours, I’ll be starting my latest round of live, online courses. Wondering what it’s like to take an online course from me? Or perhaps you’re wondering what sorts of topics I’ll discuss in my “Python dictionaries” and “Python functions” courses? Well, wonder no more; here’s a short preview of my teaching style, and the sorts of things I intend to demonstrate in my courses:

Preview: Reuven’s October 2017 live, online courses about Python and Git from Reuven Lerner on Vimeo.

If you are a beginning or intermediate Python developer, then you’ll become a more effective and fluent developer — good not only for your current employer, but for your career — thanks to my courses. And we’ll have lots of fun along the way. There will be plenty of time for exercises, questions, and comments, to ensure that you understand these technologies well. You’ll return to work the next day able to do more, and more quickly, than before.

Any questions? Just send me e-mail , and I’ll be happy to answer.

I look forward to seeing you this week (for my Python courses) and next week (for my two-day “Understanding Git” course)!

Reuven

The post Aha! Preview this week’s live, online Python courses appeared first on Lerner Consulting Blog.

October 23, 2017 11:14 AM


Import Python

Issue 147: 10 common beginners mistakes, Pytest, GC in Python, DSL, Jupyter and more

Worthy Read

GoCD is a continuous delivery tool specializing in advanced workflow modeling and dependency management. It lets you track a change from commit to deploy at a glance, providing superior visibility into your workflow. It’s open source, free to use and download.
advert

Do you still make one of these?
core-python

During my work as Python developer, I have seen many different approaches to software testing. Having such developed community and tools, it may seem that this topic should not leave much to discuss in the Python world. For many developers the choice of their test framework might be simple?—?Pytest.
PyTest

This article describes garbage collection (GC) in Python 3.6. Usually, you don't need to worry about memory management when the objects are no longer needed Python automatically reclaims the memory from them. However, understanding how GC works can help you write better Python programs.
garbage collection

Making a REST API Services will help you to connect any clients of choice to the Ether Network: Chrome Extensions, Mac App, iOS App or Android App.
ethereum

Text Classification is the task of assigning the right label to a given piece of text. This text can either be a phrase, a sentence or even a paragraph. Our aim would be to take in some text as input and attach or assign a label to it. Since we will be using Tensorflow deep learning library, we can call this the Tensorflow text classification system.
text classification

World Development Indicators (WDI) are an extensive and comprehensive compilation of data by the World Bank. WDI includes 1,400 indicators for over 200 economies, and it presents the most current and accurate global development data available.
data science

Stable sorting maintains the original order if two keys are the same
code snippets

If you’re designing a REST API, auto incremented primary keys can be a threat. They expose a lot of informations about your API and the internal structure. UUIDs can help to cover these information and make your API more secure. In the following I’m going to explain what a primary key is and what problems can occur with auto incremented primary keys.
UUID

Documenting the process of building Nimoy.
testing

Embed docs directly on your website with a few lines of code. Test the API for free.
advert

Learn how to create your own Domain Specific Language with Python from scratch with this step-by-step tutorial.
DSL

Jupyter Notebook (a.k.a iPython Notebook) is brilliant coding tool. It is ideal for doing reproducible research. Here is my list of 10 tips on structuring Jupyter notebooks, I worked out over the time.
jypyter

Django 2.0 beta 1 is an opportunity for you to try out the assortment of new features in Django 2.0.
django

We help companies like Airbnb, Pfizer, and Artsy find great developers. Let us find your next great hire. Get started today.

Recent scandals around fake news have spurred an interest in programmatically gauging the journalistic quality of an article. Companies like Factmata and Full Fact have received funding from Google, and Facebook has launched its “Journalism Project” earlier this year to fight the spread of fake stories in its feed.
news

Switching to a new language is always a big step, especially when only one of your team members has prior experience with that language. Early this year, we switched Stream’s primary programming language from Python to Go. This post will explain some of the reasons why we decided to leave Python behind and make the switch to Go.
golang

Before we get elbow deep in opcodes here, let’s cover a little background. The Python standard library has a module called pickle that is used for serializing and deserializing objects. Except it’s not called serializing and deserializing, it’s pickling and unpickling.
pickles

In this post, we'll be doing a step-by-step walkthrough of a basic machine learning project, geared toward people with some knowledge of programming (preferably Python), but who don’t have much experience with machine learning. By the end of this post, you'll understand what machine learning is, how it can help you, and be able to build your own machine learning classifiers for any dataset you want.
scikit

A circular dependency occurs when two or more modules depend on each other. This is due to the fact that each module is defined in terms of the other.
core-python

PuDB is a full-screen, console-based visual debugger for Python.
debugging

This post describes a prototype project to handle continuous data sources of tabular data using Pandas and Streamz.
pandas

Python 3.7.0a2 now available for testing

code snippets

code snippets

pandas



Jobs

Trivandrum, Kerala, India
Proficiency with Python web development frameworks and mobile versions. Python, Django, C++, Linux,SQL, PHP HTML/CSS/Javascript experience,PostgreSQL,MySQL

Europe
A Senior Python Developer based anywhere in Europe is sought by DigitalMR to work on our innovative market research project.


Projects

CTEmail - 18 Stars, 1 Fork
Chart and text in Email

Static-Fire - 15 Stars, 0 Fork
Static Fire is an engine for creating static websites, primarily blogs.

UH - 14 Stars, 0 Fork
UH is a github unfollowers hunter.

FaceBoxes - 12 Stars, 3 Fork
FaceBoxes: A CPU Real-time Face Detector with High Accuracy.

flectra - 9 Stars, 8 Fork
Flectra - Open Source ERP and CRM - Forked From Odoo.

scene-baseline - 7 Stars, 2 Fork
PyTorch baseline for AI challenger Scene classification.

tuts - 7 Stars, 2 Fork
Reverse engineering tutorials.

Hackers-Challenge-Walkthroughs - 6 Stars, 7 Fork
A collection of player contributed walkthroughs for Hackers Challenge puzzles.

Tensorflow implementation of the models used in "End-to-end learning for music audio tagging at scale".

smsutil - 5 Stars, 0 Fork
Utilities for encoding, decoding and splitting SMS.

boerewors - 5 Stars, 1 Fork
Release framework based on Python.

cython-bliss - 0 Stars, 0 Fork
Fast matrix-multiplication as a self-contained Python library – no system dependencies!

trader - 0 Stars, 0 Fork
The original game was written by Dave Kaufman in BASIC and published in the "People's Computer Company" newsletter in 1974. Here is a Python version of the game (work in progress, there could still be bugs!).

darglint - 0 Stars, 0 Fork
A python documentation linter which checks that the docstring description matches the definition.

sudoku - 0 Stars, 0 Fork
This is a simple 4x4 Sudoku game I created in Python for my class.

asam mdf - 0 Stars, 0 Fork
Fast Python parser and editor for ASAM MDF (Measurement Data Format) files

October 23, 2017 06:54 AM


Glyph Lefkowitz

Careful With That PyPI

Too Many Secrets

A wise man once said, “you shouldn’t use ENV variables for secret data”. In large part, he was right, for all the reasons he gives (and you should read them). Filesystem locations are usually a better operating system interface to communicate secrets than environment variables; fewer things can intercept an open() than can read your process’s command-line or calling environment.

One might say that files are “more secure” than environment variables. To his credit, Diogo doesn’t, for good reason: one shouldn’t refer to the superiority of such a mechanism as being “more secure” in general, but rather, as better for a specific reason in some specific circumstance.

Supplying your PyPI password to tools you run on your personal machine is a very different case than providing a cryptographic key to a containerized application in a remote datacenter. In this case, based on the constraints of the software presently available, I believe an environment variable provides better security, if you use it correctly.

Popping A Shell By Any Other Name

If you upload packages to the python package index, and people use those packages, your PyPI password is an extremely high-privilege credential: effectively, it grants a time-delayed arbitrary code execution privilege on all of the systems where anyone might pip install your packages.

Unfortunately, the suggested mechanism to manage this crucial, potentially world-destroying credential is to just stick it in an unencrypted file.

The authors of this documentation know this is a problem; the authors of the tooling know too (and, given that these tools are all open source and we all could have fixed them to be better about this, we should all feel bad).

Leaving the secret lying around on the filesystem is a form of ambient authority; a permission you always have, but only sometimes want. One of the worst things about this is that you can easily forget it’s there if you don’t use these credentials very often.

The keyring is a much better place, but even it can be a slightly scary place to put such a thing, because it’s still easy to put it into a state where some random command could upload a PyPI release without prompting you. PyPI is forever, so we want to measure twice and cut once.

Luckily, even more secure places exist: password managers. If you use https://1password.com or https://www.lastpass.com, both offer command-line interfaces that integrate nicely with PyPI. If you use 1password, you’ll really want https://stedolan.github.io/jq/ (apt-get install jq, brew install jq) to slice & dice its command-line.

The way that I manage my PyPI credentials is that I never put them on my filesystem, or even into my keyring; instead, I leave them in my password manager, and very briefly toss them into the tools that need them via an environment variable.

First, I have the following shell function, to prevent any mistakes:

1
2
3
4
function twine () {
    echo "Use dev.twine or prod.twine depending on where you want to upload.";
    return 1;
}

For dev.twine, I configure twine to always only talk to my local DevPI instance:

1
2
3
4
5
6
function dev.twine () {
    env TWINE_USERNAME=root \
        TWINE_PASSWORD= \
        TWINE_REPOSITORY_URL=http://127.0.0.1:3141/root/plus/ \
        command twine "$@";
}

This way I can debug Twine, my setup.py, and various test-upload things without ever needing real credentials at all.

But, OK. Eventually, I need to actually get the credentials and do the thing. How does that work?

1Password

1password’s command line is a little tricky to log in to (you have to eval its output, it’s not just a command), so here’s a handy shell function that will do it.

1
2
3
4
5
6
function opme () {
    # Log this shell in to 1password.
    if ! env | grep -q OP_SESSION; then
        eval "$(op signin "$(jq -r '.latest_signin' ~/.op/config)")";
    fi;
}

Then, I have this little helper for slicing out a particular field from the OP JSON structure:

1
2
3
function _op_field () {
    jq -r '.details.fields[] | select(.name == "${1}") | .value';
}

And finally, I use this to grab the item I want (named, memorably enough, “PyPI”) and invoke Twine:

1
2
3
4
5
6
7
function prod.twine () {
    opme;
    local pypi_item="$(op get item PyPI)";
    env TWINE_USERNAME="$(echo ${item} | _op_field username)" \
        TWINE_PASSWORD="$(echo "${item}" | _op_field password)" \
        command twine "$@";
}

LastPass

For lastpass, you can just log in (for all shells; it’s a little less secure) via lpass login; if you’ve logged in before you often don’t even have to do that, and it will just prompt you when running command that require you to be logged in; so we don’t need the preamble that 1password’s command line did.

Its version of prod.twine looks quite similar, but its plaintext output obviates the need for jq:

1
2
3
4
5
function prod.twine () {
    env TWINE_USERNAME="$(lpass show PyPI --username)" \
        TWINE_PASSWORD="$(lpass show PyPI --password)" \
        command twine "$@";
}

In Conclusion

“Keep secrets out of your environment” is generally a good idea, and you should always do it when you can. But, better a moment in your process environment than an eternity on your filesystem. Environment-based configuration can be a very useful stopgap for limiting the lifetimes of credentials when your tools don’t support more sophisticated approaches to secret storage.1

Post Script

If you are interested in secure secret storage, my micro-project secretly might be of interest. Right now it doesn’t do a whole lot; it’s just a small wrapper around the excellent keyring module and the pinentry / pinentry-mac password prompt tools. secretly presents an interface both for prompting users for their credentials without requiring the command-line or env vars, and for saving them away in keychain, for tools that need to pull in an API key and don’t want to make the user manually edit a config file first.


  1. Really, PyPI should have API keys that last for some short amount of time, that automatically expire so you don’t have to freak out if you gave somebody a 5-year-old laptop and forgot to wipe it first. But again, if I wanted that so bad, I should have implemented it myself... 

October 23, 2017 05:10 AM


Hynek Schlawack

Sharing Your Labor of Love: PyPI Quick and Dirty

A completely incomplete guide to packaging a Python module and sharing it with the world on PyPI.

October 23, 2017 12:00 AM

October 22, 2017


Jaime Buelta

A Django project template for a RESTful Application using Docker

I used what I learn and some decisions to create a template for new projects. Part of software development is mainly plumbing. Laying bricks together and connecting parts so the important bits of software can be accessing. That’s a pretty important part of the work, but it can be quite tedious and frustrating. This is … Continue reading A Django project template for a RESTful Application using Docker

October 22, 2017 06:58 PM


Weekly Python StackOverflow Report

(xcvi) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2017-10-22 16:34:15 GMT


  1. How to find first local maximum for every group? - [8/5]
  2. How can I tell if a dataframe is of mixed type? - [8/3]
  3. Selecting single values from pandas dataframe using lists - [5/5]
  4. Numpy: Replacing values in a 2D array efficiently using a dictionary as a map - [5/4]
  5. Finding longest perfect match between two strings - [5/3]
  6. np.isreal behavior different in pandas.DataFrame and numpy.array - [5/3]
  7. Very large numpy array doesn't throw memory error. Where does it live? - [5/2]
  8. Can pip list its binary wheels? - [5/1]
  9. How does Python compiler preprocess source code via declared encoding? - [5/1]
  10. Python3 print() Vs Python2 print - [5/1]

October 22, 2017 04:34 PM


A. Jesse Jiryu Davis

Retrospective: Coaching for First-Time PyGotham Speakers

Last month I asked you to chip in enough money to hire a professional coach for every first-time PyGotham speaker. We raised $1340, which was enough not only for all the first-time speakers but for one additional speaker who wanted coaching.

These eleven people had sessions with opera singer Melissa Collom. She clarified their explanations, streamlined their stories, taught them vocal techniques and postures that improved their performances on stage, and ensured they began their speaking careers with confidence.


Here’s what PyGotham speakers said about their coaching sessions:

“Melissa helped me with content and organization, but most importantly with my presentation style.”

“Melissa gave me honest feedback. My colleagues usually pat me on the back and say ‘Good job,’ so I rarely receive constructive criticism.”

“The coaching session was everything I needed! It alerted me to the unconscious actions I wasn’t aware of.”

“I really enjoyed working with Melissa. I appreciated how she drew attention to areas that I could improve on and strategies for doing so.”

“I look forward to seeing her for a tune-up before future speaking engagements.”

“The only way to improve the sessions would be to have more of them.”

“Thanks so much to PyGotham for offering a training like this. I feel much more confident in my ability to speak in front of audiences in future presentations, and I feel more likely to do so as a result.”

“Melissa was a wonderful gift to me, and I hope that many future PyGothamists will have the opportunity to work with her.”

“Thanks Melissa!! And thank you to the people who made Melissa available to me—I know that her expertise made my talk more engaging and made me a more confident presenter.”


Image © A. Jesse Jiryu Davis

October 22, 2017 01:32 PM


EuroPython

EuroPython 2017: Videos for Thursday available online

We are pleased to announce the third batch of cut videos for EuroPython 2017.

To see the new videos, please head over to our EuroPython YouTube channel and select the “EuroPython 2017″ playlist. The new videos start at entry 96 in the playlist.

Next week we will release the last batch of videos currently marked as “private”. 

Enjoy,

EuroPython 2017 Team
EuroPython Society
EuroPython 2017 Conference 

October 22, 2017 12:02 PM


Catalin George Festila

The Google Cloud Pub/Sub python module.

This is a test of google feature from cloud.google.com/pubsub web page.
The google development team tell us about this service:
The Google Cloud Pub/Sub service allows applications to exchange messages reliably, quickly, and asynchronously. To accomplish this, a producer of data publishes a messages to a Cloud Pub/Sub topic. A subscriber client then creates a subscription to that topic and consumes messages from the subscription. Cloud Pub/Sub persists messages that could not be delivered reliably for up to seven days. This page shows you how to get started publishing messages with Cloud Pub/Sub using client libraries.
The simple idea about this is:
Publisher applications can send messages to a topic, and other applications can subscribe to that topic to receive the messages.
I start with the installation of the python module using python version 2.7 and pip tool.

C:\Python27>cd Scripts

C:\Python27\Scripts>pip install --upgrade google-cloud-pubsub
Collecting google-cloud-pubsub
Downloading google_cloud_pubsub-0.28.4-py2.py3-none-any.whl (79kB)
100% |################################| 81kB 300kB/s
...
Successfully installed google-cloud-pubsub-0.28.4 grpc-google-iam-v1-0.11.4 ply-3.8
psutil-5.4.0 pyasn1-modules-0.1.5 setuptools-36.6.0
The next steps come with some settings on google console, see this google page.
The default settings can be started and set with this command: gcloud init .
You need to edit this settings and app.yaml at: ~/src/.../appengine/flexible/pubsub$ nano app.yaml and your .
After you set all of this using the command gcloud app deploy you can see the output at https://[YOUR_PROJECT_ID].appspot.com.
The main goal of this tutorial was to start and run the Google Cloud Pub/Sub service with python and this has been achieved.

October 22, 2017 11:59 AM


Sandipan Dey

Feature Detection with Harris Corner Detector and Matching images with Feature Descriptors in Python

The following problem appeared in a project in this Computer Vision Course (CS4670/5670, Spring 2015) at Cornell. In this article, a python implementation is going to be described. The description of the problem is taken (with some modifications) from the project description. The same problem appeared in this assignment problem as well. The images used … Continue reading Feature Detection with Harris Corner Detector and Matching images with Feature Descriptors in Python

October 22, 2017 12:48 AM

October 20, 2017


Stack Abuse

Append vs Extend in Python Lists

Adding Elements to a List

Lists are one of the most useful data structures available in Python, or really any programming language, since they're used in so many different algorithms and solutions.

Once we have created a list, often times we may need to add new elements to it, whether it be at the end, beginning, or somewhere in between. Python offers us three different methods to do so. In this article I'll be showing the differences between the append, extend, and insert list methods.

Append

This method adds an element at the end of an existing list. The syntax to use it is:

a.append(x)  

Here the variable a is our list, and x is the element to add. This expression is equivalent to a[len(a):] = [x].

For example, here is how to use it to place the element "y" at the end of our list, a:

a = [1, 'x', 2]  
a.append('y')

print(a)  

Running this code will result in the following output:

$ python append.py
[1, 'x', 2, 'y']

Insert

This method inserts an item at a specified position within the given list. The syntax is:

a.insert(i, x)  

Here the argument i is the index of the element before which to insert the element x. Thus, a.insert(len(a), x) is the same thing as a.append(x). Although, the power of this method comes in using it to place items somewhere within the list and not at the end. If you only needed to add an element to the end of the list then append works just fine for that, and is faster (which matters for large lists).

For example:

a = [1, 'x', 'y']  
a.insert(2, 2)

print(a)  

This code will result in the following output:

$ python insert.py
[1, 'x', 2, 'y']

As you can see, the element given is placed anywhere in the list that we specify. This works well when you have a list in which its items are ordered, so you can't just add your element to the end, like you would do with append.

Extend

This method adds elements (notice its plural!) to a list by adding on all the elements of the iterable you pass to it. The resulting list is one containing all of the elements of both lists.

The syntax for using this method is:

a.extend(x)  

In this case a is our list and x is an iterable object, such as another list. This method is equivalent to a[len(a):] = x.

For example:

a = [1, 'x', 'y']  
b = [1, 2]  
a.extend(b)

print(a)  

Running this code results in the following output:

$ python extend.py
[1, 'x', 'y', 1, 2]

Notice how the two lists were combined together, one after the other.

In Python, you can also achieve the same results by doing a simple addition. So a + b, in this case, would result in the same exact array as our script above. This is thanks to the __add__() method being implemented in the lists, but that's out of the scope of this article.

Comparing Each Method

In order to see the different results obtained through these methods, let's do a direct comparison by running the following code:

a1 = [1, 'x', 'y']  
a2 = [1, 'x', 'y']  
a3 = [1, 'x', 'y']

b = [2, 3]

a1.append(b)  
a2.insert(3, b)  
a3.extend(b)

print(a1)  
print(a2)  
print(a3)  

In this program, we have defined three lists with exactly the same elements. We have also defined a second list, which we append, insert and extend to each of the three similar lists previously defined. The result is as follows:

$ python all.py
[1, 'x', 'y', [2, 3]]
[1, 'x', 'y', [2, 3]]
[1, 'x', 'y', 2, 3]

As we can see, both append and insert add the list b to the initial list, but as a single element, which is a list. In other words, it doesn't append each element of b individually, but instead it appends the entire object itself.

The extend method, on the other hand, actually adds the individual elements of list b, as separate and unique elements of the resulting list.

This is all in agreement with what we previously saw, that is, append and insert add only a single element. Whereas, extend, expands on the initial list by adding the elements of the second list at its end.

Another difference to consider is the measure of efficiency. Given how each operation works, we can pretty easily figure out the time complexity for each method. Or you can just cheat and check out the Time Complexity page on python.org's wiki page.

The time complexities are as follow:

Method Time Complexity
append() O(1)
insert() O(n)
extend() O(k)

Here "n" is the number of elements currently in the list, and "k" is the number of elements in the parameter object.

These points show that these three methods are complementary. We must choose which one to use, according to our needs:

Wrapping Up

Python presents several choices to add elements into a list, all of which complement each other and have their own use-cases. In this article we presented three of those choices, how to use each, and when to use each. The choice of method you choose should be based on your needs.

October 20, 2017 07:54 PM


PyCharm

PyCharm 2017.2.4 RC now available

Happy Friday everyone,

Today we’ve published PyCharm 2017.2.4 Release Candidate. It’s now available for download from Confluence.

Important bug fixes in this version:

Read full release notes

Download it now from the Confluence EAP page

We’d like to invite you to try out this version, if you run into any issues please let us know on YouTrack.

Using Ubuntu? Try installing PyCharm 2017.2.4 RC with snap packages:

1. Install PyCharm Professional Edition or PyCharm Community Edition EAP from the ‘edge’ channel:

$sudo snap install [pycharm-professional | pycharm-community] --classic --candidate

2. Run PyCharm:

$[pycharm-professional | pycharm-community]

Give us your feedback on your snap installation experience on Twitter or in comments to this blog post! Read more about how snaps work.

October 20, 2017 05:29 PM


RMOTR

How to delete multiple slack channels at once?

✨ slack.py ✨

This was possible thanks to a simple Python command we built. Do you want to learn Python? Check out our Free Python Course!

We use Drift to engage with our students in our website and we absolutely love it: it’s integrated with Slack, so every time a user makes it to our site and starts a conversation, we receive a notification and we can talk right there, from within Slack, where we spent most of our day.

Drift integrates with Slack

The only problem we have is that, when a visitor starts a conversation, a new channel is created in our Slack account. Furthermore, all those created channels are open and public to everybody 😓. The solution is to delete every channel after the conversation is over, but it’s not always possible. Sometimes we’re just in a rush and we forget, others, we’re in mobile and it’s not really convenient.

Result: Drift channels start to pile up:

We had 759 z-visitor channels created

Thinking about manually deleting all these channels was a complete bummer, so we went to check the Slack API. Slack has a really nice API that lets you interact with most parts of the application. There’s an entire section dedicated to channels with methods to archive (channels.archive) or delete channels (channels.delete). So we just built a simple command line tool to handle the task. We called it: slack.py 🐍💌. It’s a simple Python command line tool that interfaces with Slack and lets you manage channels: searching, archiving and/or deleting them.

With slack.py at hand, we were able to delete all these annoying Drift channels at once. Here’s a step by step guide in case you have the same problem:

1. Install slack.py

Just run the command: pip install slack.py

We know it’s a little bit too geeky, but you need to use Python to install it. Hopefully, you have some friend or coworker that can give you a hand with that.

2. Get your Slack Token

In order to delete channels, you’ll need to authenticate and prove that you have enough privileges to do so. Slack Auth can be complicated (specially when it involves OAuth), but there’s a workaround. You can generate a “testing” token to try things out. Go to this page and generate the correct token for your slack account: https://api.slack.com/custom-integrations/legacy-tokens

Get your token. Usually starts with the pattern xoxp-…

3. Run the command to see if everything works

We’ve added a “test mode” to our command to verify that it was installed correctly and the auth token you’ve generated in the previous step is valid. The command is auth_test and this is the how you should run it: slack -a YOUR_TOKEN auth_test

In this case, YOUR_TOKEN is the token you got in the previous step. You’ve just ran the slack command using the auth_test subcommand passing the correct token. If everything works fine, you should see something like this:

auth_test working correctly

4. Search channels, explore the ones you want to delete

Now it’s time to run the command and start exploring the channels you want to delete. You can issue this simple command to search for all the channels that start with the pattern “z-visitor”: slack -a YOUR_TOKEN channels --starts-with "z-visitor"

Just 6 channels. We deleted a bunch.

Search options include: “starts with the following word” with the --starts-with option and “contains the word” with —-contains.

5. Delete the channels

Once you’re sure about the channels you want to delete, you can use the --delete flag to perform the action. The command will be then: slack -a YOUR_TOKEN channels --starts-with "z-visitor" --delete

Deleting Channels. Dangerous!

This is obviously 🚨dangerous 🚨, as the deleted channels will be lost forever, so use it at your own risk. You can pass an optional--dry-run flag to make a “simulation” of the process without actually deleting anything, in case you want to be extra sure about that.

We hope this command makes your life a little bit easier. We’ll try to keep improving it and adding more functionality, you’re more than welcome to contribute and send your feedback: https://github.com/santiagobasulto/slack.py


How to delete multiple slack channels at once? was originally published in rmotr.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

October 20, 2017 05:25 PM

October 19, 2017


PyCharm

PyCharm 2017.3 EAP 6: Improved interpreter selection interface

The latest and greatest early access program (EAP) version of PyCharm is now available from our website:

Get PyCharm 2017.3 EAP 6

Improved Python interpreter selection interface

In the previous PyCharm 2017.3 EAP 5 build we introduced a new UI for configuring project interpreters in your existing projects. To recap:

This build has a new UI for configuring project interpreters during the project creation.
Go to File | New project…:newproject

In the New Project dialog you will get a similar experience to the one described above for the existing project interpreter selection. Here, if you’d like to reuse an existing virtualenv or system interpreter, you can select it under ‘Existing interpreter’. And if you create a new virtualenv and would like to reuse it in other projects in the future, you can check the ‘Make available to all projects’ option and it will appear in the dropdown on the project interpreter page in all of your other projects.

Install PyCharm with snap packages

Starting from this EAP build, we are offering an alternative installation method using snap packages for Ubuntu users. Snaps are quick to install, safe to run, easy to manage and they are updated automatically in the background every day, so you always have the newest builds as soon as they’re out. If you are running Ubuntu 16.04 LTS or later, snap is already preinstalled, so you can begin using snaps from the command line right away.

Installing PyCharm Professional or Community Edition 2017.3 EAP is now as easy as this simple command (please make sure you use just one option from square brackets):

$sudo snap install [pycharm-professional | pycharm-community] --classic --edge

This command will install PyCharm Professional or Community from the “Edge” channel where we store EAP builds. Please note, the snap installation method is experimental, and currently, we officially distribute only PyCharm 2017.3 EAP in the Edge channel.

Depending on which snap you’ve installed, you can run your new PyCharm 2017.3 EAP with:

$[pycharm-professional | pycharm-community]

You can now use other snap commands for managing your snaps. The most frequently used commands are:

Snap supports auto updates from the stable channel only. Since we distribute 2017.3 EAP builds in the edge channel, to update to the next EAP build, you’ll need to manually invoke:

$snap refresh [pycharm-professional | pycharm-community] --edge

Read more about how snaps work and let us know about your experience with using snap for installing and updating PyCharm 2017.3 EAP so we can consider whether snaps might be something we can utilize for our official stable releases. You can give us your feedback on Twitter or in the comments to this blog post.

Other improvements in this build:

If these features sound interesting to you, try them yourself:

Get PyCharm 2017.3 EAP 6

As a reminder, PyCharm EAP versions:

If you run into any issues with this version, or any other version of PyCharm, please let us know on our YouTrack. If you have any suggestions or remarks, you can reach us on Twitter, or by commenting on the blog.

October 19, 2017 07:07 PM


Lintel Technologies

What is milter?

Every one gets tons of email these days. This includes emails about super duper offers from amazon to princess and wealthy businessmen trying to offer their money to you from some African country that you have never heard of. In all these emails in your inbox there lies one or two valuable emails either from your friends, bank alerts, work related stuff. Spam is a problem that email service providers are battling for ages. There are a few opensource spam fighting tools available like SpamAssasin or SpamBayes.

What is milter ?

Simply put – milter is mail filtering technology. Its designed by sendmail project. Now available in other MTAs also. People historically used all kinds of solutions for filtering mails on servers using procmail or MTA specific methods. The current scene seems to be moving forward to sieve. But there is a huge difference between milter and sieve. Sieve comes in to picture when mail is already accepted by MTA and had been handed over to MDA. On the other hand milter springs into action in the mail receiving part of MTA. When a new connection is made by remote server to your MTA, your MTA will give you an opportunity to accept of reject the mail every step of the way from new connection, reception of each header, and reception of body.

simplified milter stages

" data-medium-file="http://s3.amazonaws.com/lintel-blogs-static-files/wp-content/uploads/2017/10/13131356/milter-stages-187x300.png" data-large-file="http://s3.amazonaws.com/lintel-blogs-static-files/wp-content/uploads/2017/10/13131356/milter-stages.png" class="size-full wp-image-868" src="http://s3.amazonaws.com/lintel-blogs-static-files/wp-content/uploads/2017/10/13131356/milter-stages.png" alt="milter stages " width="512" height="823" srcset="http://s3.amazonaws.com/lintel-blogs-static-files/wp-content/uploads/2017/10/13131356/milter-stages.png 512w, http://s3.amazonaws.com/lintel-blogs-static-files/wp-content/uploads/2017/10/13131356/milter-stages-187x300.png 187w" sizes="(max-width: 512px) 100vw, 512px" />
milter protocol various stages

The above picture depicts simplified version of milter protocol working. Full details of milter protocol can be found here https://github.com/avar/sendmail-pmilter/blob/master/doc/milter-protocol.txt  . Not only filtering; using milter, you can also modify message or change headers.

HOW DO I GET STARTED WITH CODING MILTER PROGRAM ?

If you want to get started in C you can use libmilter.  For Python you have couple of options:

  1. pymilter –  https://pythonhosted.org/milter/
  2. txmilter – https://github.com/flaviogrossi/txmilter

Postfix supports milter protocol. You can find every thing related to postfix’s milter support in here – http://www.postfix.org/MILTER_README.html

WHY NOT SIEVE WHY MILTER ?

I found sieve to be rather limited. It doesn’t offer too many options to implement complex logic. It was purposefully made like that. Also sieve starts at the end of mail reception process after mail is already accepted by MTA.

Coding milter program in your favorite programming language gives you full power and allows you to implement complex , creative stuff.

WATCHOUT!!!

When writing milter programs take proper care to return a reply to MTA quickly. Don’t do long running tasks in milter program when the MTA is waiting for reply. This will have crazy side effects like remote parties submitting same mail multiple time filling up your inbox.

The post What is milter? appeared first on Lintel Technologies Blog.

October 19, 2017 08:05 AM


Python Bytes

#48 Garbage collection and memory management in Python

October 19, 2017 08:00 AM


Talk Python to Me

#134 Python in Climate Science

What is the biggest challenge facing human civilization right now? Fake news, poverty, hunger? Yes, all of those are huge problems right now. Well, if climate change kicks in, you can bet it will amplify these problems and more. That's why it's critical that we get answers and fundamental models to help understand where we are, where we are going, and how we can improve things.

October 19, 2017 08:00 AM


Python Anywhere

Response times now available in webapp logs

we deployed a new version this morning. Most of the work was on a sekrit hidden feature that we can't talk about yet (oooo) but there is one small thing we've added that we hope you find useful: we've added response times to your webapp access logs:

you can see them at the end of each line, in the format response_time=N.NNN.

We hope that you'll find these useful if you ever have to chase down performance issues. Let us know what you think!

October 19, 2017 07:46 AM


Steve Holden

What's In a Namespace?

Python programmers talk about namespaces a lot. The Zen of Python* ends with

Namespaces are one honking great idea—let’s do more of those!
and if Tim Peters thinks namespaces are such a good idea, who am I to disagree?

Resolution of Unqualified Names

Python programmers learned at their mothers' knees that Python looks up unqualified names in three namespaces—first, the local namespace of the currently-executing function or method; second, the global namespace of the module containing the executing code; third and last, the built-in namespace that holds the built-in functions and exceptions. So, it makes sense to understand the various namespaces that the interpreter can use. Note that when we talk about name resolution we are talking about how a value is associated with an unadorned name in the code.

In the main module of a running program there is no local namespace. A name must be present in either the module's global namespace or, if not there, in the built-in namespace that holds functions like len, the standard exceptions, and so on. In other words, when __name__ == '__main__' the local and global namespaces are the same.

When the interpreter compiles a function it keeps track of names which are bound inside the function body (this includes the parameters, which are established in the local namespace before execution begins) and aren't declared as either global or (in Python 3) nonlocal.  Because it knows the local names the interpreter can assign them a pre-defined place in the stack frame (where local data is kept for each function call), and does not generally need to perform a lookup. This is the main reason local access is faster than global access.

Although the interpreter identifies local names by the presence of bindings within a function body, there is nothing to stop you writing code that references the names before they are bound. Under those circumstances you will see an UnboundLocalError exception raised with a message like "local variable 'b' referenced before assignment".

For non-local names, something very like a dictionary lookup takes place first in the module's global namespace and then in the built-ins. If neither search yields a result then the interpreter raises a NameError exception with a message like "name 'nosuch' is not defined."

Resolution of Qualified Names

In qualified names (those consisting of a sequence of names or expressions delimited by dots  such as os.path.join) starts by locating the first object's namespace (in this case os) in the standard way described above. Thereafter the mechanism can get complex because like many Python features you can control how it works for your own objects by defining __getattr__ and/or __getattribute__ methods, and because descriptors (primarily used in accessing properties) can cloud the picture.

In essence, though, the mechanism is that the interpreter, having located the object bound to the unqualified name, then makes a gettatr call for the second name (in this case, path) in that namespace, yielding another object, against which a further getattr call is made with the third component of the name, and so on. If at any point a getattr fails then the interpreter raises an AttributeError exception with a message such as "'module' object has no attribute 'name'."

Understanding Expression Values

Once you understand the mechanisms for looking up the values of names it becomes a little easier to understand how Python computes expression values. Once a name is resolved there may be other methods to apply such as __getitem__ for subscripting or __call__ for function calls. These operations also yield values, whose namespaces can again be used to lookup further names. So, for example, when you see an expression like

    e.orig.args[0].startswith('UNIQUE constraint failed')

you understand that the name e.orig.args is looked up by going through a sequence of namespaces and evaluates to a list object, to which a subscripting operation is applied to get the first element, in whose namespace the name startswith is resolved (hopefully to something callable) to a value that is finally called with a string argument.

Ultimately, by decomposing the expressions in this way you end up only dealing with one object at a time. Knowing how these mechanisms work in principle can help you to decipher complex Python code.

* Just type import this into a Python interpreter, or enter python -m this at the shell prompt, and hit return.

October 19, 2017 05:24 AM

October 18, 2017


Marcos Dione

dinant-0.5

I have a love and hate relantionship with regular expressions (regexps). On one side they're a very powerful tool for text processing, but on the other side of the coin, the most well known implementation is a language whose syntax is so dense, it's hard to read beyond the most basic phrases. This clashes with my intention of trying to make programs as readable as possible[1]. It's true that you can add comments and make your regexps span several lines so you can digest them more slowly, but to me it feels like eating dried up soup by the teaspoon directly from the package without adding hot water.

So I started reading regexps aloud and writing down how I describe them in natural language. This way, [a-z]+ becomes one or more of any of the letters between lowercase a and lowercase z, but of course this is way too verbose.

Then I picked up these descriptions and tried to come up with a series of names (in the Pyhton sense) that could be combined to build the same regexps. Even 'literate' programs are not really plain English, but a more condensed version, while still readable. Otherwise you end up with Perl, and not many think that's a good idea. So, that regexp becomes one_or_more(any_of('a-z')). As you can see, some regexp language can still be recognizable, but it's the lesser part.

So, dinant was born. It's a single source file module that implements that language and some other variants (any_of(['a-z'], times=[1, ]), etc). It also implements some prebuilt regexps for common constructs, like integer, a datetime() function that accepts strptime() patterns or more complex things like IPv4 or IP_port. Conforming I start using it in (more) real world examples (or issues are filed!), the language will slowly grow.

Almost accidentally, its constructive form brought along a nice feature: you can debug() your expression so you can find out the first sub expression that fails matching:

# this is a real world example!
In [1]: import dinant as d
In [2]: line = &apos&apos&apos36569.12ms (cpu 35251.71ms)\n&apos&apos&apos
# can you spot the error?
In [3]: render_time_re = ( d.bol + d.capture(d.float, name=&aposwall_time&apos) + &aposms &apos +
...:                       &apos(cpu&apos + d.capture(d.float, name=&aposcpu_time&apos) + &aposms)&apos + d.eol )

In [4]: print(render_time_re.match(line))
None

In [5]: print(render_time_re.debug(line))
# ok, this is too verbose (I hope next version will be more human readable)
# but it&aposs clear it&aposs the second capture
Out[5]: &apos^(?P<wall_time>(?:(?:\\-)?(?:(?:\\d)+)?\\.(?:\\d)+|(?:\\-)?(?:\\d)+\\.|(?:\\-)?(?:\\d)+))ms\\ \\(cpu(?P<cpu_time>(?:(?:\\-)?(?:(?:\\d)+)?\\.(?:\\d)+|(?:\\-)?(?:\\d)+\\.|(?:\\-)?(?:\\d)+))&apos
# the error is that the text &apos(cpu&apos needs a space at the end

Of course, the project is quite simple, so there is no regexp optimizer, which means that the resulting regexpes are less readable than the ones you would had written by hand. The idea is that, besides debugging, you will never have to see them again.

Two features are in the backburner, and both are related. One is to make debugging easier by simply returning a representation of the original expression instead of the internal regexp used. That means, in the previous example, something like:

bol + capture(float, name=&aposwall_time&apos) + &aposms &apos + &apos(cpu&apos + capture(float, name=&aposcpu_time&apos)

The second is that you can tell which types the different captured groups must convert to. This way, capture(float) would not return the string representing the float, but the actual float. The same for datetime() and others.

As the time of writing the project only lives on GitHub, but it will also be available in PyPI Any Time Soon®. Go grab it!


python ayrton


[1] for someone that knows how to read English, that is.

October 18, 2017 08:51 PM