skip to navigation
skip to content

Planet Python

Last update: May 30, 2020 09:48 PM UTC

May 30, 2020


Splitwise Telegram Bot

SplitwizeBot is a chat based bot to list, create and settle the expenses of Splitwise application from within Telegram 🤖

May 30, 2020 07:56 PM UTC

PSF GSoC students blogs

Weekly Blog #1

Welcome to my GSoC Blog!!!

Hello Everyone, this is Soham Biswas currently in 2nd year pursuing my Bachelor’s(B.Tech) degree in Computer Science & Engineering from Institute of Engineering & Management, Kolkata. I have been selected for GSoC' 20 at sub-org FURY under the umbrella organisation of Python Software Foundation. I will be working on building sci-fi-like 2D and 3D interfaces and provide physics engine integration under project titled "Create new UI widgets & Physics Engine Integration".

What did you do during the Community Bonding Period?

Due to the pandemic outbreak and the country wide lockdown in India, many places including my university were closed and therefore I decided to get a head start and start with the project early. During the community bonding period, we had video conference meetings with our mentors and the project's core team. We interacted with each other and discussed the implementational details and their respective deadlines for the entire event period. We will be having such meetings every week on Wednesday in order to update ourselves about the progess of our respective tasks.

I completed the remaining Pull Requests that I had pending before the GSoC students announcement. I also reviewed other issues and pull requests to make sure everything remains up-to-date.

What is comming up next week?

Currently, I am focusing on building the ComboBox2D UI element. I will try to get the skeleton model, required sub-components and their respective default callbacks done by next week.

Did you get stuck anywhere?

While working with my previous PR related to Double Click callbacks, I faced an issue where I was unable to create and implement UserEvents properly in VTK. Thankfully, my mentors helped me out I was able to implement double click callbacks for all three mouse buttons successfully.

<samp>See you next week, cheers!!</samp>


May 30, 2020 04:07 PM UTC

Weekly Python StackOverflow Report

(ccxxx) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2020-05-30 12:45:10 GMT

  1. How to concat multiple Pandas DataFrame columns with different token separator? - [13/4]
  2. Create a list including row name, column name and the value from dataframe - [10/5]
  3. Pre-populate current value of WTForms field in order to edit it - [7/1]
  4. Random number generator with conditions - Python - [6/4]
  5. Pandas Dataframe split multiple key values to different columns - [6/3]
  6. Why can a Python script file not be named - [6/2]
  7. Python Dataframe: Get alternative days based on month? - [6/2]
  8. What is the difference between string literals and string values? - [6/1]
  9. How to use regex to extract text in order? - [6/1]
  10. R regex compiler working differently for the given regex - [6/1]

May 30, 2020 12:46 PM UTC


How to handle bulk data insertion SQLite + python

SQLite Python: Inserting Data

May 30, 2020 12:04 PM UTC

PSF GSoC students blogs

GSoC Blog :  Week 1

Hey everyone!
This is my blog for this summer’s GSoC @ PSF
I am Lenix Lobo, an undergraduate student from India, and this summer I will be working with project Fury under the umbrella of the Python Software foundation. 

What did you do during the community bonding period?

Since most of the places including my university are closed due to the pandemic outbreak, I decided to get a head start and start with the project early. During the community bonding period, I had video conference meetings with my mentors scheduled every week on Wednesday. During these meetings i interacted with the mentors to have a coherent understanding of how the project design and implementation will be managed over the course of the entire period. 
Since my project involves a lot of theoretical understanding of concepts such as ray marching, I spent the period going through the theory of each topic.This week also involved going through the documentation for shaders used in VTK. 

What is coming up next ?
The next task assigned to me is to go through the theory of geometry shaders and to create a example using the same. 

Did you get stuck anywhere?
Since, some of the documentation on VTK shaders was not upto date, i had to go through example implementations to understand the nomenclature of variables and their respective usage.

May 30, 2020 07:10 AM UTC

First Blog GSoC 2020

May 30, 2020 05:46 AM UTC

Weekly Check-in #1

<tt>Hi everyone I am Abhay, I am a cse undergrad from India. </tt><tt>I got selected in TERN sub-org to work on this summer. Tern is container analysis tool.</tt>

<tt>You can read my blog post here to learn more about it.</tt>

<tt>This first of many blog posts to come. So lets get started.</tt>

<tt>What did you do this week ?</tt>

<tt>I started working on creating UI for JSON reports generated by Tern. At first I used json2html library but that gave very ugly results :p. After that I decided to make a tree view for JSON data. I started writing recursive functions to generate the tree-view HTML report. I've opened a PR and making the changes requested by the mentors. I also had my first meeting with my mentors they are very helpful and supportive.</tt>

<tt>Did you get stuck anywhere ?</tt>

<tt>While writing the recursive code I struggled as the drop-down buttons didn't worked. Turns out <ul> tag cannot have another <ul> as its child. I found this with the help of html validator tool.</tt>

What is comping up next?

<tt>Thanks for reading my blog.</tt>

<tt>Over and out.</tt>

May 30, 2020 05:22 AM UTC

May 29, 2020

Brett Cannon

The many ways to pass code to Python from the terminal

For the Python extension for VS Code, I wrote a simple script for generating our changelog (think Towncrier, but simpler, Markdown-specific, and tailored to our needs). As part of our release process we have a step where you are supposed to run python news which points Python at the news directory in our repository. A co-worker the other day asked how that worked since everyone on my team knows to use -m (see my post on using -m with pip as to why)? That made me realize that other people probably don't know the myriad of ways you can point python at code to execute, hence this blog post.

Via stdin and piping

Since how you pipe things into a process is shell-specific, I'm not going to dive too deep into this. Needless to say, you can pipe code into Python.

echo "print('hi')" | python
Piping text into python

This obviously also works if you redirect a file into Python.

python <
Redirecting a file into python

Nothing really surprising here thanks to Python's UNIX heritage.

A string via -c

If you just need to quickly check something, passing the code in as a string on the command-line works.

python -c "print('hi')"
Using the -c flag with python

I personally use this when I need to check something that's only a line or two of code instead of launching the REPL.

A path to a file

Probably the most well-known way to pass code to python is via a file path.

Specifying a file path for python

The key thing to realize about this is the directory containing the file is put at the front of sys.path. This is so that all of your imports continue to work. But this is also why you can't/shouldn't pass in the path to a module that's contained from within a package. Since sys.path won't have the directory that contains the package, all your imports will be relative to a different directory than you expect for your package.

Using -m with packages

The proper way to execute a package is by using -m and specifying the package you want to run.

python -m spam
Using -m with python

This uses runpy under the hood. To make this work with your project all you need to do is specify a inside your package and it will get executed as __main__. And the submodule can be imported like any other module, so you can test it and everything. I know some people like having a main submodule in there package and then make their be:

from . import main

if __name__ == "__main__":

Personally, I don't bother with the separate main submodule and put all the relevant code directly in as the module names feel redundant to me.

A directory

Defining a can extend to a directory as well. If you look at my example that instigated this blog post, python news works because the news directory has file. Python executes that like a file path. Now you might be asking, "why don't you just specify the file path then?" Well, it's honestly one less thing to know about a path. 😄 I could have just as easily written out instructions in our release process to run python news/, but there is no real reason to when this mechanism exists. Plus I can change the file name later on and no one would notice. Plus I knew the code was going to have ancillary files with it, so it made sense to put it in a directory versus as a single file on its own. And yes, I could have  made it a package to use -m, but there as no point as the script is so simple I knew it was going to stay a single, self-contained file (it's less than 200 lines and the test module is about the same length).

Besides, the file is extremely simple.

import runpy
# Change 'announce' to whatever module you want to run.
runpy.run_module('announce', run_name='__main__', alter_sys=True)
a for when you point python at a directory

Now obviously there's having to deal with dependencies, but if your script just  uses the stdlib or you place the dependencies next to the then you are good to go!

Executing a zip file

When you do have multiple files and/or dependencies and you  want to ship our code our as a single unit, you can place it in a zip file with a and Python will run that file on your behalf with the zip file places on sys.path.

python app.pyz
Passing a zip file to python

Now traditionally people name such zip files with a .pyz file extension, but that's purely tradition and does not affect anything; you can just as easily use the .zip file extension.

To help facilitate creating such executable zip files, the stdlib has the zipapp module. It will generate the for you and add a shebang line so you don't even need to specify python if you don't want to on UNIX. If you are wanting to move around a bunch of pure Python code it's a nice way to do it.

Unfortunately using a zip file like this only works when all the code the zip file contains is pure Python. Executing zip files as-is doesn't work for extension modules (this is why setuptools has a zip_safe flag). To load an extension module Python has to call the dlopen() function and it takes a file path which obviously doesn't work when that file path is contained within a zip file. I know at least one person who talked to the glibc team about adding support for passing in a memory buffer so Python could read an extension module into memory and pass that in, but if memory serves the glibc team didn't go for it.

But not all hope is lost! You can use a project like shiv which will bundle your code and then provide a that will handle extracting the zip file, caching it, and then executing the code for you. While not as ideal as the pure Python solution, it does work and is about as elegant as one can get in this situation.

May 29, 2020 10:44 PM UTC

Red Hat Developers

Red Hat Software Collections 3.5 brings updates for Red Hat Enterprise Linux 7

Red Hat Software Collections 3.5 and Red Hat Developer Toolset 9.1 are now available for Red Hat Enterprise Linux 7. Here’s what that means for developers.

Red Hat Software Collections (RHSCL) is how we distribute the latest stable versions of various runtimes and languages through Red Hat Enterprise Linux (RHEL) 7, with some components available in RHEL 6. RHSCL also contains the Red Hat Developer Toolset, which is the set of tools we curate for C/C++ and Fortran. These components are supported for up to five years, which helps you build apps that have a long lifecycle as well.

What changed?

Updated collections in RHSCL 3.5 include:

The last—but certainly not least—update to Red Hat Software Collections 3.5 is Red Hat Developer Toolset (DTS) version 9.1, which is the set of tools we curate for C/C++ and Fortran. For DTS, we updated the compilers, debuggers, and performance monitoring tools to ensure the best experience for software developers using these languages. At the center of DTS 9.1 is GCC 9.3, which brings a huge number of improvements including improved diagnostics and useability. The full list of tools that we updated in DTS 9.1 is available in the release notes, as always.

How do I get this great stuff?

With a Red Hat Developer Subscription, you have access to Red Hat Enterprise Linux 7, where you can update these packages. If you have already enabled Red Hat Software Collections in the subscription manager, follow the instructions below for either a specific software collection or a container image. If you haven’t already enabled RHSCLs, please follow the instructions in our online documentation.

To install a specific software collection, type the following into your command line as root:

$ yum install software_collection…

Replace software_collection with a space-separated list of the software collections you want to install. For example, to install php54 and rh-mariadb100, type as root:

$ yum install rh-php72 rh-mariadb102

Doing this installs the main meta-package for the selected software collection and a set of required packages as its dependencies. For information on how to install other packages such as additional modules, see Section 2.2.2, “Installing Optional Packages.”

Another option, of course, is to start with our container images for these packages, which make it easier to build and deploy mission-critical applications that use these components for Red Hat Enterprise Linux and Red Hat OpenShift platforms.

The full release notes for Red Hat Software Collections 3.5 and Red Hat Developer Toolset 9.1 are available in the customer portal.

What about Red Hat Enterprise Linux 8?

Software Collections are for Red Hat Enterprise Linux 7. Red Hat Enterprise Linux 8 is managed in a different way through Application Streams, and you can find updated RHEL 8 packages in the RHEL8 appstream repository. The updates for these packages might not be the same for Red Hat Enterprise Linux 8 Application Streams, so please check on the Application Streams Life Cycle page.


The post Red Hat Software Collections 3.5 brings updates for Red Hat Enterprise Linux 7 appeared first on Red Hat Developer.

May 29, 2020 06:59 PM UTC

PSF GSoC students blogs

Community Bonding Check-in

What did you do during this period?

I had an onboarding meeting with my mentors where we got to know each other a bit better. They advised me to play around with uarray and unumpy without any goal in mind which I found to be a very good advice. I played a bit with special methods by implementing a simple Vector2D class and used the code in this notebook with some print statements to understand better the protocols and how they are called. I wanted to start earlier on my project so I took over a PR from one of my mentors which adds multimethods for the linalg module.

What is coming up next?

I'm going to continue the PR that I have been working on since it still isn't finished and I will also follow the proposed timeline and start adding multimethods for other routines like checking class equality in array elements. Some mathematical constants and their aliases are also missing so I will be adding these too and probably refactoring the existing ones into classes. This week marks the end of my college classes but I still have some assignments and exams coming up in the following weeks so there's a lot of work ahead of me to proper balance both university studies and GSoC but I wouldn't have it other way.

Did you get stuck anywhere?

I consider the PR that I started working during this period to be a challenging one since some mathematical intuition is needed to translate Linear Algebra routines into proper functions. Things like decomposition of a matrix into eigenvalues and eigenvectors to calculate its n-th power is something I'm not too familiar with specially in a programming context. With that said there hasn't been a roadblock for me up until now and usually I can wrap my head around these concepts in half a day. It should be noted that mentor help plays a huge part in this as they frequently give me very good advices. Despite that I usually think a lot before doing a commit to make sure that what I'm pushing is correct I notice that I still can't avoid some mistakes, even ones that should be obvious to me. I guess these mistakes are normal and they are corrected soon after so no harm's done but I'm training myself to not do them as often.

May 29, 2020 03:21 PM UTC


EuroPython 2020: Schedule published

We are very excited to announce the first version of our EuroPython 2020 schedule:


EuroPython 2020 Schedule

More sessions than we ever dreamed of

After the 2nd CFP, we found that we had so many good talk submissions that we were able to open a fourth track. Together with the newly added slots for the Indian / Asian / Pacific and Americas time zones, we now have a fully packed program, with:

We are proud to have reached almost the size of our in-person event with the online version of EuroPython 2020.


Never miss a talk

All talks will be made available to the attendees as live Webinars, with easy switching between tracks, as well as online streams, which will allow rewinding to watch talks you may have missed during the day.

Conference Tickets

Conference tickets are available on our registration page. We have simplified and greatly reduced the prices for the EuroPython 2020 online edition.

As always, all proceeds from the conference will go into our grants budget, which we use to fund financial aid for the next EuroPython edition, special workshops and other European conferences and projects:

EuroPython Society Grants Program

We hope to see lots of you at the conference in July. Rest assured that we’ll make this a great event again — even within the limitations of running the conference online.


On Saturday and Sunday, we will have sprints/hackathons on a variety of topics. Registration of sprint topics has already started. If you would like to run a sprint, please add your sprint topic to the wiki page we have available for this:

EuroPython 2020 Sprints Listing

If registrations continue as they currently do, we will have a few hundred people waiting to participate in your sprint projects, so this is the perfect chance for you to promote your project and find new contributors.

Participation in the sprints is free, but does require registration. We will provide the necessary collaboration tools in form of dedicated Jitsi or Zoom virtual rooms and text channels on our Discord server.

EuroPython is your conference

EuroPython has always been a completely volunteer based effort. The organizers work hundreds of hours to make the event happen and will try very hard to create an inspiring and exciting event.

However, we can only provide the setting. You, as our attendees, are the ones who fill it with life and creativity.

We are very much looking forward to having you at the conference !


EuroPython 2020 Team

May 29, 2020 12:49 PM UTC

Real Python

The Real Python Podcast – Episode #11: Advice on Getting Started With Testing in Python

Have you wanted to get started with testing in Python? Maybe you feel a little nervous about diving in deeper than just confirming your code runs. What are the tools needed and what would be the next steps to level up your Python testing? This week on the show we have Anthony Shaw to discuss his article on this subject. Anthony is a member of the Real Python team and has written several articles for the site.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 29, 2020 12:00 PM UTC

Zato Blog

Backing up and restoring Zato Single Sign-On data

This article presents a procedure for backing up all of Zato Single Sign-On (SSO) data and restoring it later on.

A single Zato server with SQLite is used for simplicity reasons but the same principles hold regardless of the size of one's environment or the SQL database used.


Zato Single-Sign On overview

There are two data sources that SSO uses:

Thus, to make a backup:

Conversely, to restore a backup:

Backing up SQL data

Assuming that a Zato server is in a directory called /home/zato/sso1/server, here is how to back up an SQLite database:

$ cd /home/zato/sso1/server
$ sqlite3 zato.db ".dump zato_sso_%" > zato-sso-backup.sql

This will create a file called zato-sso-backup.sql the contents of which is the schema and rows of all the SSO objects.

To make it easier to restore it later, open the file and add the following commands right after "BEGIN TRANSACTION;"

DROP TABLE IF EXISTS zato_sso_group;
DROP TABLE IF EXISTS zato_sso_user;
DROP TABLE IF EXISTS zato_sso_user_group;
DROP TABLE IF EXISTS zato_sso_session;
DROP TABLE IF EXISTS zato_sso_attr;
DROP TABLE IF EXISTS zato_sso_linked_auth;

The idea with the DROP statements is that when you are restoring SSO from a backup, these tables, albeit empty, will already exist, so we can just drop them to silence out any SQLite warnings.

Backing up secrets.conf

Again, if the server is in /home/zato/sso1/server, the full path to secrets.conf is /home/zato/sso1/server/config/repo/secrets.conf - simply copy the whole file to a location of choice.

Just to confirm it, the contents should be akin to this:



Creating a new server

We work under assumption that a new server will be created in a directory named /home/zato/sso2/server.

Note that it should be a completely new instance in a new cluster. Do not start the server yet.

Restoring SQL data

Move the zato-sso-backup.sql file to /home/zato/sso2/server and run the commands below:

$ cd /home/zato/sso2/server
$ sqlite3 zato.db < zato-sso-backup.sql
$ echo $?
$ 0

Exit code 0 should be returned on output, indicating a successful operation.

Restoring secrets.conf

The file backed up previously needs to be saved to /home/zato/sso2/server/config/repo/secrets.conf, as below:

$ cd /home/zato/sso2/server/config/repo
$ mv ./secrets.conf ./secrets.conf.bak # Just in case
$ cp /path/to/backup/secrets.conf .

Confirming it all

Now, the server can be started and we can confirm that the SSO data can be accessed by logging it to the system as one of its users, as below - output was reformatted for clarity:

$ zato sso login /home/zato/sso2/server my.user
User logged in {
  'username': 'my.user',
  'user_id': 'zusr6htg...',
  'ust': 'gAAAAABe0M_Pf8cdBa6bimnjfVUt5CF...',
  'creation_time': '2020-05-29T09:03:11.459337',
  'expiration_time': '2020-05-29T10:03:11.459337',
  'has_w_about_to_exp': False

That concludes the process - the SSO data is now restored and the server can be fully used, just like the original one.

May 29, 2020 09:35 AM UTC

Talk Python to Me

#266 Refactoring your code, like magic with Sourcery

Refactoring your code is a fundamental step on the path to professional and maintainable software. We rarely have the perfect picture of what we need to build when we start writing code and attempts to over plan and overdesign software often lead to analysis paralysis rather than ideal outcomes. <br/> <br/> Join me as I discuss refactoring with Brendan Maginnis and Nick Thapen as well as their tool, Sourcery, to automate refactoring in the popular Python editors.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Guests</b><br/> <br/> <b>Brendan Maginnis</b>: <a href="" target="_blank" rel="noopener">@brendan_m6s</a><br/> <b>Nick Thapen</b>: <a href="" target="_blank" rel="noopener">@nthapen</a><br/> <br/> <b>Sourcery</b><br/> <b>Sourcery</b>: <a href="" target="_blank" rel="noopener"></a><br/> <b>Sourcery on Twitter</b>: <a href="" target="_blank" rel="noopener">@sourceryai</a><br/> <b>VS Code and PyCharm Plugins</b>: <a href="" target="_blank" rel="noopener"></a><br/> <b>GitHub Bot</b>: <a href="" target="_blank" rel="noopener"></a><br/> <b>For an instant demo ⭐ this repo, and Sourcery will refactor your most popular Python repo</b>: <a href="" target="_blank" rel="noopener"></a><br/> <br/> <b>Python Refactorings article</b>: <a href="" target="_blank" rel="noopener"></a><br/> <br/> <b>Nuitka</b><br/> <b>Talk Python episode</b>: <a href="" target="_blank" rel="noopener"></a><br/> <b>Nuitka site</b>: <a href="" target="_blank" rel="noopener"></a><br/> <br/> <b>Gilded Rose Kata</b>: <a href="" target="_blank" rel="noopener"></a><br/></div><br/> <strong>Sponsors</strong><br/> <br/> <a href=''>Datadog</a><br> <a href=''>Linode</a><br> <a href=''>Talk Python Training</a>

May 29, 2020 08:00 AM UTC

Python Bytes

#183 Need a beautiful database editor? Look to the Bees!

<p>Sponsored by DigitalOcean: <a href=""><strong></strong></a></p> <p>Special guest: Calvin Hendryx-Parker <a href="">@calvinhp</a></p> <hr /> <p><strong>Brian #1:</strong> <a href=""><strong>fastpages: An easy to use blogging platform, with enhanced support for Jupyter Notebooks.</strong></a> </p> <ul> <li>Uses GH actions to Jekyll blog posts on GitHub Pages.</li> <li>Create posts with code, output of code, formatted text, directory from Jupyter Notebooks.</li> <li>Altair interactive visualizations</li> <li>Collapsible code cells that can be open or closed by default.</li> <li>Metadata like title, summary, in special markdown cells.</li> <li>twitter cards and YouTube videos</li> <li>tags support</li> <li>Support for pure markdown posts </li> <li>and even MS Word docs for posts. (but really, don’t).</li> <li>Documentation and introduction written in fastpages itself, <a href=""></a></li> </ul> <hr /> <p><strong>Michael #2:</strong> <a href=""><strong>BeeKeeper Studio Open Source SQL Editor and Database Manager</strong></a></p> <ul> <li>Use Beekeeper Studio to query and manage your relational databases, like <strong>MySQL</strong>, <strong>Postgres</strong>, <strong>SQLite</strong>, and <strong>SQL Server</strong>.</li> <li>Runs on all the things (Windows, Linux, macOS)</li> <li>Features <ul> <li>Autocomplete SQL query editor with syntax highlighting</li> <li>Tabbed interface, so you can multitask</li> <li>Sort and filter table data to find just what you need</li> <li>Sensible keyboard-shortcuts</li> <li>Save queries for later</li> <li>Query run-history, so you can find that one query you got working 3 days ago</li> <li>Default dark theme</li> </ul></li> <li>Connect: Alongside normal connections you can encrypt your connection with SSL, or tunnel through SSH. Save a connection password and Beekeeper Studio will make sure to encrypt it to keep it safe.</li> <li>SQL Auto Completion: Built-in editor provides syntax highlighting and auto-complete suggestions for your tables so you can work quickly and easily.</li> <li>Open Lots of Tabs: Open dozens of tabs so you can write multiple queries and tables in tandem without having to switch windows.</li> <li>Save queries</li> <li>View Table Data: Tables get their own tabs too! Use our table view to sort and filter results by column.</li> </ul> <hr /> <p><strong>Calvin #3:</strong> <strong>2nd Annual</strong> <a href=""><strong>Python</strong></a> <a href=""><strong>Web</strong></a> <a href=""><strong>Conference</strong></a> <a href=""></a></p> <ul> <li>The most in-depth Python conference for web developers <ul> <li>Targeted at production users of Python</li> <li>Talks on Django, Flask, Twisted, Testing, SQLAlchemy, Containers, Deployment and more</li> </ul></li> <li>June 17th-19th — One day of tutorials and two days of talks in 3 tracks</li> <li>Keynote talks by <ul> <li>Lorena Mesa</li> <li>Hynek Schlawack</li> <li>Russell Keith-Magee</li> <li>Steve Flanders</li> </ul></li> <li>Fireside Chat with Carl Meyer about Instragram’s infrastructure, best practices</li> <li>Participate in 40+ presentations and 6 tutorials</li> <li>Fun will be had and connections made <ul> <li>Virtual cocktails</li> <li>Online gaming</li> <li>Board game night</li> </ul></li> <li>Tickets are $199 and $99 for Students <ul> <li>As a bonus, for every <strong>Professional</strong> ticket purchased, we'll donate a ticket to an attendee in a <a href="">developing country.</a> </li> <li>As a Python Bytes listener you can get a 20% discount with the code PB20</li> </ul></li> </ul> <hr /> <p><strong>Brian #4:</strong> <a href=""><strong>Mimesis - Fake Data Generator</strong></a></p> <ul> <li>“…helps generate big volumes of fake data for a variety of purposes in a variety of languages.”</li> <li>Custom and generic data providers</li> <li>&gt;33 locales</li> <li>Lots of locale dependent providers, like address, Food, Person, …</li> <li>Locale independent providers. </li> <li>Super fast. Benchmarking with 10k full names was like <a href="">60x faster than Faker</a>.</li> <li>Data generation by schema. Very cool</li> </ul> <pre><code> &gt;&gt;&gt; from mimesis.schema import Field, Schema &gt;&gt;&gt; _ = Field('en') &gt;&gt;&gt; description = ( ... lambda: { ... 'id': _('uuid'), ... 'name': _('text.word'), ... 'version': _('version', pre_release=True), ... 'timestamp': _('timestamp', posix=False), ... 'owner': { ... 'email': _('', domains=[''], key=str.lower), ... 'token': _('token_hex'), ... 'creator': _('full_name'), ... }, ... } ... ) &gt;&gt;&gt; schema = Schema(schema=description) &gt;&gt;&gt; schema.create(iterations=1) </code></pre> <pre><code>- Output: [ { "owner": { "email": "", "token": "cc8450298958f8b95891d90200f189ef591cf2c27e66e5c8f362f839fcc01370", "creator": "Veronika Dyer" }, "name": "widget", "version": "4.3.1-rc.5", "id": "33abf08a-77fd-1d78-86ae-04d88443d0e0", "timestamp": "2018-07-29T15:25:02Z" } ] </code></pre> <hr /> <p><strong>Michael #5:</strong> <a href=""><strong>Schemathesis</strong></a></p> <ul> <li>A tool for testing your web applications built with Open API / Swagger specifications.</li> <li><strong>Supported specification versions</strong>: <ul> <li>Swagger 2.0</li> <li>Open API 3.0.x</li> </ul></li> <li>Built with: <ul> <li><a href="">hypothesis</a></li> <li><a href="">hypothesis_jsonschema</a></li> <li><a href="">pytest</a></li> </ul></li> <li>It reads the application schema and generates test cases which will ensure that your application is compliant with its schema.</li> <li>Use: There are two basic ways to use Schemathesis: <ul> <li><a href="">Command Line Interface</a></li> <li><a href="">Writing tests in Python</a></li> </ul></li> <li>CLI supports passing options to <code>hypothesis.settings</code>.</li> <li>To speed up the testing process Schemathesis provides <code>-w/--workers</code> option for concurrent test execution</li> <li>If you'd like to test your web app (Flask or AioHTTP for example) then there is <code>--app</code> option for you</li> <li>Schemathesis CLI also available as a docker image</li> <li>Code example:</li> </ul> <pre><code> import requests import schemathesis schema = schemathesis.from_uri("") @schema.parametrize() def test_no_server_errors(case): # `requests` will make an appropriate call under the hood response = # use `call_wsgi` if you used `schemathesis.from_wsgi` # You could use built-in checks case.validate_response(response) # Or assert the response manually assert response.status_code &lt; 500 </code></pre> <hr /> <p><strong>Calvin #6:</strong> <a href="">Finding secrets by decompiling Python bytecode in public repositories</a></p> <ul> <li>Jesse’s initial research revealed that <strong>thousands of GitHub repositories contain secrets hidden inside their bytecode.</strong></li> <li>It has been common practice to store secrets in Python files that are typically ignored such as <code></code>, <code></code> or <code></code>, but this is potentially insecure</li> <li>Includes a nice crash course on Python byte code and cached source</li> <li>This post comes with a small capture-the-flag style lab for you to try out this style of attack yourself. <ul> <li>You can find it at <a href=""></a></li> </ul></li> <li>Look through your repositories for loose <code>.pyc</code> files, and delete them</li> <li>If you have <code>.pyc</code> files and they contain secrets, then revoke and rotate your secrets</li> <li>Use a standard <a href="">gitignore</a> to prevent checking in <code>.pyc</code> files</li> <li>Use JSON files or environment variables for configuration</li> </ul> <hr /> <p><strong>Extras:</strong></p> <p>Michael:</p> <ul> <li><a href="">Python 3.9.0b1 Is Now Available for Testing</a></li> <li><a href="">Python 3.8.3 Is Now Available</a></li> <li>Ventilators and Python: Some particle physicists put some of their free time to design and build a low-cost ventilator for covid-19 patients for use in hospitals. Search of the PDF for Python: <ul> <li>"Target computing platform: Raspberry Pi 4 (any memory size), chosen as a trade-off between its computing power over power consumption ratio and its wide availability on the market; • Target operating: Raspbian version 2020-02-13; • Target programming language: Python 3.5; • Target PyQt5: version 5.11.3."</li> <li>"The MVM GUI is a Python3 software, written using the PyQt5 toolkit, that allows steering and monitoring the MVM equipment."</li> </ul></li> </ul> <p>Brian: </p> <ul> <li><a href="">Call for Volunteers! Python GitHub Migration Work Group</a> <ul> <li>migration from to GitHub</li> </ul></li> </ul> <p>Calvin:</p> <ul> <li><a href="">Learn Python Humble Bundle</a> <ul> <li>Pay $15+ and get an amazing set of Python books to start learning at all levels</li> <li>Book Industry Charitable Foundation</li> <li>The No Starch Press Foundation</li> </ul></li> </ul> <hr /> <p><strong>Joke:</strong></p> <p>More O’Really book covers</p> <p><img src="" alt="" /></p> <p><img src="" alt="" /></p> <p><img src="" alt="" /></p> <p><img src="" alt="" /></p>

May 29, 2020 08:00 AM UTC

Stefan Scherfke

Attrs, Dataclasses and Pydantic

I’ve been using attrs for a long time now and I am really liking it. It is very flexible, has a nice API, is well documented and maintained, and has no runtime requirements.

The main idea behind attrs was to make writing classes with lots of data attributes (“data classes”) easier. Instead of this:

class Data:
    def __init__(self, spam, eggs):
        self.spam = spam
        self.eggs = eggs

It lets you write this:

class Data:
    spam = attr.ib()
    eggs = attr.ib()

Attrs also adds a nice string representation, comparison methods, optional validation and lots of other stuff to your classes, if you want to. You can also opt out of everything; attrs is very flexible.

Attrs became so popular, that since Python 3.7 we also have the dataclasses module in the standard library. It is predominantly inspired by attrs (the attrs team was involved in the design of data classes) but has a smaller feature set and will evolve a lot slower. But you can use it out-of-the box without adding a new requirement to your package.

Pydantic’s development roughly started during Python 3.7 development, too. Its main focus is on data validation, settings management and JSON (de)serialisation, therefore it is located at a higher level ob abstraction. Out of the box, it will recursively validate and convert all data that you pour into your model:

>>> from datetime import datetime
>>> from pydantic import BaseModel
>>> class Child(BaseModel):
...     x: int
...     y: int
...     d: datetime
>>> class Parent(BaseModel):
...     name: str
...     child: Child
>>> data = {
...     'name': 'spam',
...     'child': {
...         'x': 23,
...         'y': '42',  # sic!
...         'd': '2020-05-04T13:37:00',
...     },
... }
>>> Parent(**data)
Parent(name='spam', child=Child(x=23, y=42, d=datetime.datetime(2020, 5, 4, 13, 37)))

I only learned about pydantic when I started to work with FastAPI. FastAPI is a fast, asynchronous web framework specifically designed for building REST APIs. It uses pydantic for schema definition and data validation.

Since then, I asked myself: Why not attrs? What’s the benefit of pydantic over the widely used and mature attrs? Can or should it replace attrs?

As I begin to write this article, I still don’t know the answer to these questions. So lets explore attrs, data classes and pydantic!

Simple class definition

Originally, attrs classes were created by using the @attr.s() (or @attr.attrs() class decorator. Fields had to be created via the attr.ib() (or @attr.attrib()) factory function. By now, you can also create them nearly like data classes.

The recommended way for creating pydantic models is to subclass pydantic.BaseModel. This means that in contrast to data classes, all models inherit some “public” methods (e.g., for JSON serialization) which you need to be aware of. However, pydantic allows you to create stdlib data classes extended with validation, too.

Here are some very simple examples for data classes / models:

>>> import attr
>>> import dataclasses
>>> import pydantic
>>> # Simple data classes are supported by all libraries:
>>> @attr.dataclass
... # @dataclasses.dataclass
... # @pydantic.dataclasses.dataclass
... class Data:
...     name: str
...     value: float
>>> Data('Spam', 3.14)
Data(name='Spam', value=3.14)
>>> @attr.s
... class Data:
...     name = attr.ib()
...     value = attr.ib()
>>> Data('Spam', 3.14)
Data(name='Spam', value=3.14)
>>> class Data(pydantic.BaseModel):
...     name: str
...     value: float
>>> Data(name='Spam', value=3.14)
Data(name='Spam', value=3.14)

Pydantic models enforce keyword only arguments when creating new instances. This is a bit tedious for classes with only a few attributes but with larger models, you’re likely going to use keyword arguments anyways. The benefit of kw-only arguments is, that it doesn’t matter if you list attributes with a default before ones without a default.

Data classes support positional as well as keyword arguments. Passing values by position is very convenient for smaller classes but that also means that you must define all fields without a default value first and the ones with a default value afterwards. This may prevent you from grouping similar attributes, when only some of them have a default value.

Attrs supports both ways. The default is to allow positional and keyword arguments like data classes. You can enable kw-only arguments by passing kw_only=True to the class decorator.

Another major difference is that Pydantic always validates and converts all attribute values, but more on that later.

Class and attribute customistaion

All three libraries let you customize the created fields as well as the class itself.

In data classes and attrs, you can customize your class by passing additional arguments to the class decorator. Pydantic models can define a nested Config class for the same purpose.

Attributes can be customized via special factory functions. Instead of specifying an attribute like this: name: type [= default], you you do: name: type = field_factory(). This function is named [attr.]ib()/attrib() in attrs, field() with data classes and Field() in pydantic.

Using these functions, you can specify default values, validators, meta data and other attributes. The following tables let you compare the customisation features that each library provides:

Attribute / field settings
attrs data classes pydantic
Explicit no default NOTHING 1 MISSING 1 ... 1
Default factory yes yes yes
Validators yes 2 no no 2,3
Constraints no no const, regex, length, number range, …
Converters yes 2 no no 2,3
Exclude field from repr, eq, order, hash, init repr, compare, hash, init ø
Add arbitrary metadata yes yes yes
Additional docs no no title, description


1 Passing no default is optional in attrs and data classes, but mandatory in pydatnic.

2 Validators and converters can also be defined as decorated methods of you class.

3 Pydantic always performs basic validation and conversion for the attribute’s data type (e.g., int('42')).

Class customisation and instantiation
attrs data classes pydantic
Auto create methods for str, repr, equality, ordering, hash, init repr, equality, ordering, hash, init str, repr, equality, init
Keyword args only optional no yes
Faux immutability / Freezing yes yes yes
Slots yes no no
Safe to subclass exceptions yes no no
Dynamic creation yes yes yes
Instantiate from dict yes yes yes, recursively
Instantiate from objects no no optional, recursively
Instantiate from JSON no no yes, recursively
Instantiate from env. vars. no no yes, recursively

Generated methods

All libraries create useful “dunder” methods (like __init__() or __str__()). Attrs can generate the most methods, followed by data classes and Pydantic. Attrs and data classes also allow you to selectively disable the generation of certain methods.

Attrs is the only library that generates __slots__ and is also the only one that has explicit support for subclassing exceptions.

Default values

Without a field factory, default values for fields are simply assigned to the field name, e.g., value: int = 42. When you use a field factory, you can/need to pass a default value as argument to that function. In pydantic, the default value is always passed as first positional argument. In order to express “this attribute has no default”, you use the elipsis literal (...). Data classes use the optional keyword argument default instead. Attrs lets you choose - you can pass a default value by position or as keyword argument.

Another difference is that pydantic allows you to use mutable objects like lists or dicts as default values. Attrs and data classes prohibit this for good reason. To prevent bugs with mutable defaults, pydantic deep-copies the default value for each new instance.

You you can specify factories for default values with all libraries.

Freezing and functional programming

You can create pseudo immutable classes with all libraries. Immutable/frozen instances prevent you from changing attribute values. This helps when you aim to program in a more functional style. However, if attributes themselves are mutable (like lists or dicts), you can still change these!

In attrs and data classes, you pass frozen=True to the class decorator. In pydantic, you set allow_mutation = False in the nested Config class.

Attrs and data classes only generate dunder protocol methods, so your classes are “clean”. Having struct-like, frozen instances make it relatively easy to write purely functional code, that can be more robust and easier to test than code with a lot of side effects.

Pydantic models, on the other hand, use inheritance and always have some methods, e.g., for converting an instance from or to JSON. This facilitates a more object-orient programming style, which can be a bit more convenient in some situations.

Instantiation, validation and conversion

The main differentiating features of pydantic are its abilities to create, validate and serialize classes.

You can instantiate pydantic models not only from dicts/keyword arguments but also from other data classes (ORM mode), from environment variables, and raw JSON. Pydantic will then not only validate/convert basic data types but also more advanced types like datetimes. On top of that, it will recursively create nested model instances, as shown in the example above.

Model instances can directly be exported to dicts and JSON via the .dict()/.json() methods.

To achieve something similar in attrs or data classes, you need to install an extension package like, for example, cattrs. And even then, Pydantic has a far better user experience.

Apart from that, all libraries allow you to define custom validator and converter functions. You can either pass these functions to the field factories or define decorated methods in your class.

Metadata and schema generation

Pydantic can not only serialize model instances but also the schema of the model classes themselves. This is, for example, used by FastAPI to generate the OpenAPI spec for an API.

To aid the documentation of the generated schemas, every field can have a title and a description attribute. These are not used for docstrings, though.


In a way, the documentation of all three projects mirrors their feature sets.

It is of high quality in all cases, but technically and in terms of content very different.

The data classes documentation is part of Python’s stdlib documentation and the briefest of all candidates, but it covers everything you need to know. It contains a direct link to the source code that also has many helpful comments.

The attrs docs contain example based guides, in-depth discussions of certain features and design decisions as well as an exhaustive API reference. It uses Sphinx and, for the API reference, the autodoc extension. It provides an objects inventory which allows you to cross-reference attrs classes and functions from your own documentation via intersphinx.

The pydantic documentation is also very well written and contains many good examples that explain almost all functionality in great detail. However, it follows the unfortunate trend of using MkDocs as a documentation system. I assume that this is easier to set-up then Sphinx and allows you to use Markdown instead of ReStructuredText, but it is also lacking lots of important features and I also don’t like its UX. It has two navigation menus – one on the left for whole document’s TOC and one on the right for the current page. More serious, however, is the absence of an API reference. There is also no cross referencing (e.g., links from class and function names to their section in the API reference) and thus no objects inventory that can be used for inter-project cross referencing via Sphinx’ intersphinx extension. Even pydantic’s source code barely includes any docstrings or other comments. This can be a hindrance when you want to solve more advanced problems.


An alternative for MkDocs might be MyST, which is an extend Markdown parser that can be used with Sphinx.

MkAutoDoc allows you to add an API ref. to MkDocs. It does not (yet?) generate an objects.inv for use with intersphinx, though.

objects.inv also enables doc2dash which in turn enables Dash and Zeal.


For most use cases, the performance of a data classes library can be neglected. Performance differences only become noticeable when you create thousands or even millions of instances in a short amount of time.

However, the pydantic docs contain some benchmarks that suggest that pydantic is slightly ahead of attrs + cattrs in mean validation time. I was curious why pydantic, despite its larger feature set, was so fast, so I made my own benchmarks.

I briefly evaluate the attrs extension packages. The only one that offers reasonably convenient means for input validation and (de)serialization as well as good performance is cattrs.

I created benchmarks for three different use cases:

  1. Simple classes and no need for extensive validation
  2. Deserialization and validation of (more) complex (nested) classes
  3. Serialization of complex (nested) classes

I calculated the time and memory consumption for handling 1 million instances of different variants of attrs and data classes as well as pydantic models.

Unsurprisingly, attrs and data classes are much faster than pydantic when no validation is needed. They also use a lot less memory.

Relative time and memory consumption for basic class instantiation

I was expecting that the results would be much closer when it comes to validation/conversion and serialization, but even there, pydantic was a lot slower than attrs + cattrs.

Relative time and memory consumption for loading more complex, nested classes Relative time and memory consumption for dumping more complex, nested classes

I wondered why my benchmarks were so clearly in favor of attrs when the pydantic docs state that it is 1.4x faster than attrs + cattrs. I tried running the pydantic benchmarks myself and indeed I could reproduce these results. Wondering, why the results differed so much, I took a closer look at the benchmark’s source code. It turned out that the attrs + cattrs example used python-dateutil for parsing datetimes while pydantic uses its own implementation. I replaced datetuil.parser.parse() with the stdlib datetuil.fromisoformat() and the attrs + cattrs example suddenly became 6–7 times faster. (Note: fromisoformat() is not a general purpose parser!)

In defense of pydantic: The attrs + cattrs (de)serializers were specifically designed and implemented for this benchmark while Pydantic ships everything out-of-the box. Pydantic’s UX for these use cases is also more pleasant than that of attrs + cattrs.

You can find the source of all benchmark in the accompanying repository.


Attrs, data classes and pydantic seem very similar on a first glance, but they are very different when you take a closer look.

All three projects are of high quality, well documented and generally pleasant to use. Furthermore, they are different enough that each of them has its niche where it really shines.

The stdlib’s data classes module provides all you need for simple use cases and it does not add a new requirement to your project. Since it does not do any data validation, it is also quite fast.

When you need more features (more control over the generated class or data validation and conversion), you should use attrs. It is as fast as data classes are but its memory footprint is even smaller when you enable __slots__.

If you need extended input validation and data conversion, Pydantic is the tool of choice. The price of its many features and nice UX is a comparatively bad performance, though.

If you want to cut back on UX instead, the combination of attrs and cattrs might also be an alternative.

You can take a look at the benchmarks to get a feel for how the libraries can be used for different use cases and how they differ form each other.

I myself will stay with attrs as long as it can provide what I need. Otherwise I’ll use Pydantic.


I wish there was a library like mattrs (magic attrs) that combined Pydantic’s (de)serialization UX with attrs’ niceness and performance:

>>> from datetime import datetime
>>> from mattrs import dataclass, field, asjson
>>> @dataclass()
... class Child:
...     x: int = field(ge=0, le=100)
...     y: int = field(ge=0, le=100)
...     d: datetime
>>> @dataclass()
... class Parent:
...     name: str = field(re=r'^[a-z0-9-]+$')
...     child: Child
>>> data = {'name': 'spam', 'child': {'x': 23, 'y': '42', 'd': '2020-05-04T13:37:00'}}
>>> Parent(**data)
Parent(name='spam', child=Child(x=23, y=42, d=datetime.datetime(2020, 5, 4, 13, 37)))
>>> asjson(_)
{"name": "spam", "child": {"x": 23, "y": 42, "d": "2020-05-04T13:37:00"}}

Maybe it’s time for another side project? 🙊


Update 1: Added a few links about documentation systems

Update 2: Fixed chart for benchmark #3

May 29, 2020 07:36 AM UTC

PSF GSoC students blogs

Week 1 Check-in

Completed tasks

During the community bonding period, i am working on the first step of my proposal. I have used shlex to split the shell script into tokens, and then find the seperator(&&|;) to concatenate the commands. After the review from my mentor, we find that we can improve the code. We do not need to split into tokens at first. Instead, we can directly find the seperator(&&|;) to seperate the commands. This will save a lot of time, since we are not going through every word in the shell script.

To do

Use split_command function and add function to split branch(if and case), and loop(for and while).  We will leave the branch not to be parsed since we do not know which branch to be executed. For loop, we will futher develop a function to extract info from it. Here are 2 small steps i am going to do.

1. split command(using seperator(&&|;)), split branch and loop(finding keywords).

2. Extract loop. This is a simple version, we will just extract the commands without considering the loop.


1. Parsing command has already been implemented so this part is a key point. After the following 2 small steps finished, we should be able to parse the shell script in a Dockerfile RUN command to find what software maybe installed.

2. Try to break big issue into small independent ones.

May 29, 2020 05:02 AM UTC

May 28, 2020

Matt Layman

Bread and Butter Django - Building SaaS #58

In this episode, I worked on a views and templates. There are a number of core pages that are required to flesh out the minimal interface for the app. We’re building them. I began by showing the page that we were going to work on. I outlined the changes I planned to make, then we started. The first thing we added was data about the school year, the main model on display in the page.

May 28, 2020 12:00 AM UTC

May 27, 2020

Gaël Varoquaux

Technical discussions are hard; a few tips


This post discuss the difficulties of communicating while developing open-source projects and tries to gives some simple advice.

A large software project is above all a social exercise in which technical experts try to reach good decisions together, for instance on github pull requests. But communication is difficult, in particular between diverging points of view. It is easy to underestimate how much well-intended persons can misunderstand each-other and get hurt, in open source as elsewhere. Knowing why there are communication challenges can help, as well as applying a few simple rules.

The first challenge is to understand the other’s point of view: the different parties see the problem differently.

Maintainer’s anxiety

Open source can be anxiety-generating for the maintainers

Maintainers ensure the quality and the long-term life of an open-source project. As such, they feel responsible for any shortcoming in the product. In addition, they often do this work because they care, even though it may not bring any financial support. But they can quickly become a converging point of anxiety-generating feedback:

  • Code has bugs; the more code, the more bugs. Watching a issue tracker fill up with a long list of bugs is frightening to people who feel in charge.
  • Given that maintainers are visible and qualified, they become the target of constant requests for attention: from pleas to prioritize a specific issue to solicitations for advice.
  • A small fraction of these interactions come as plain aggressions. I have been insulted many times by unsatisfied users. Each time, it hurts me a lot. My policy is to disengage from the conversation, but I am left shaking and staring at my computer in the evening.

The more popular a project, the more weight it puts on its maintainers’ shoulders. A consequence is that maintainers are tired, and can sometimes approach discussions in a defensive way. Also, we may be plain scared of integrating a code that we do not fully comprehend.

Open-source developers may even, unconsciously, adopt a simple, but unfortunate, protection mechanism: being rude. The logic is flawless: if I am nasty to people, or I set unreasonnable expectations, people will let me alone. Alas, this strategy leads to toxic environments. It not only makes people unhappy but also harms the community dynamics that ground the excellence of open source.

The danger abusive gatekeeping

A maintainer quickly learns that every piece of code, no matter how cute it might be, will give him work in the long run, just like a puppy. This is unavoidable given that the complexity of code grows faster than its number of features [1], and, even for a company as rich as Google, project maintenance becomes intractable on huge projects [2].

A maintainer’s job is to say no often, to protect the project. But, as any gatekeeping, it can unfortunately become an excercise in unchecked power. Making objective choices for these difficult decisions is hard, and we all tend naturally to trust more people that we know.

Most often we are not aware of our shortcomings, let alone are we doing them on purpose.

Contributor’s fatigue

A new contributor starting a conversation with a group of seasoned project maintainers may easily feel an imposter. The new contributor knows less about the project. In addition, he or she is engaging with a group of people that know each-other well, and is not yet part of that inner group.

This person does not know the code base, or the conventions, and must make extra efforts, compared to the seasoned developers, to propose a contribution suitable for the project. Often, he or she does not understand fully the reasons for the project guidelines, or for the feedback given. Request for changes can easily be seen as trifles.

Integrating the contribution can often be a lengthy process –in particular in scikit-learn. Indeed, it will involve not only shaping up the contribution, but also learning the skills and discovering the process. These long cycles can undermine motivation: humans need successes to feel enthusiasm. Also, the contributor may legitimately worry: Will all these efforts be fruitful? Will the contribution make its way to the project?

Note that for these reasons, it is recommended to start contributing with very simple features, and to seek feedback on the scope of the contribution before writing the code.

Finally, contributors are seldom paid to work on the project, and there is no single line of command that makes decisions and controls incentives for all the people on the project. No one is responsible when things go astray, which means that the weight falls on the shoulder of the individuals.

The danger behind the lengthy cycle of reviews and improvements needed to contribute is death by a thousands cuts. The contributor looses motivation, and no longer finds the energy to finish the work.

How about users?

This article is focused on developers. Yet, users are also an important part of the discussion around open source.

Often communication failures with users are due to frustration. Frustration of being unable to use the software, of hitting a bug, of seeing an important issue still not addressed. This frustration stems from incorrect expectations, which can often be traced to misunderstanding of the processes and the dynamics. Managing expectations is important to improve the dialogue, via the documentation, via notes on the issue tracker.

Communication is hard

Communication is hard: messages are sometimes received differently than we would like. Overworked people discussing very technically challenging issues only makes the matter worse. I have seen people not come across well, while I know they are absolutely lovely and caring.

We are human beings; we are limited; we misunderstand things, and we have feelings.

Emotions – My most vivid memory of a communication failure was when I was a sailing instructor. Trainees that were under my responsibility had put themselves at risk, causing me a lot of worry. During the debrief, I was angry. My failure to convey the messages without emotional loading undermined my leadership on the group, putting everybody at risk for the rest of the week.

Inability to understand the others’ point of view, or to communicate ours, can bring in emotions. Emotions most often impedes technical communication.

Limited attention – We, in particular maintainers, are bombarded with email, notifications, text and code to read. As a consequence, it is easy to read things too fast, to stop in the middle, to forget.

Language barriers – Most discussions happen in English; but most of us are not native English speakers. We may hide well our difficulties, but nuances are often lost.

Clique effects – Most interactions in open source are done in writing, with low communication bandwidth. It can be much harder to convince a maintainer on the other side of the world than a colleague in the same room. Schools of thoughts naturally emerge when people work a lot together. These create bubbles, where we have the impression that everything we say is obvious and uncontroversial, and yet we fail to convince people outside of our bubble.

Little things that help

Communication can improved by continuously working on it [3]. It may be obvious to some, but it personally took me many years to learn.

[3]Training materials for managers often discuss communication, and give tricks. I am sure that there are better references than my list below. But that’s the best I can do.

Hear the other: exchange

Foster multiway discussions – The goal of a technical discussion is to come up to the best solution. Better solutions emerge via confronting different points of view: a single brilliant individual probably cannot find or recognize the best solution alone.

  • Integrate input from as many perspectives as possible.
  • Make sure everyone feels heard.

Don’t seek victory – Most important to keep in mind is that giving up on an argument and accepting the other point of view is a perfectly valid option. I naturally biased to think that my view on topics dear to me is the right one. However, I’ve learned that adopting the view of the other could bring a lot to the social dynamics of a project: we are often debating over details and the bigger benefit comes from moving forward.

In addition, if several very bright people have different conclusions than me about something that they’ve thought a lot, who am I to disagree?

Convey ideas well: pedagogy

Explain – Give the premises of your thoughts. Unroll your thought processes. People are not sitting in your head, and need to hear not only your conclusion, but how you got there.

Repeat things – Account for the fact that people can forget, and never hesitate to gently restate important points. Reformulating differently can also help explaining.

Keep it short – A typical reading speed is around 200 words a minute. People have limited time and attention span. The greatest help you can provide to your reader is to condense your ideas: let us avoid long threads that require several dozens of minutes to read and digest. There is a tension between this point and the above. My suggestion: remove every word that is not useful, move details to footnotes or postscriptums.

Cater for emotions: tone

Stay technical – Always try to get to the technical aspect of the matter, and never the human. Give specific code and wording suggestions. When explaining a decision, give technical arguments, even if they feel obvious to you.

Be positive – Being positive in general helps people feeling happy and motivated. It is well known that positive feedback leads to quicker progress than negative, as revealed eg by studies of class rooms. I am particularly guilty of this: I always forget to say something nice, although I may be super impressed by a contribution. Likewise, avoid negative words when giving feedback (stay technical).

Avoid “you” – The mere use of the pronoun “you” puts the person we are talking to in the center of message. But the message should not be about the person, it should be about the work. It’s very easy to react emotionally when it’s about us. The passive voice can be useful to avoid putting people as the topic. If the topic is indeed people, sometimes “we” is an adequate substitute for “you”.

Assume good faith – There are so many misunderstandings that can happen. People forget things, people make mistakes, people fail to convey their messages. Most often, all these failures are in good faith, and misunderstandings are legitimate. In the rare cases there might possibly be some bad faith, accounting for it will only make communication worse, not better. Along the same line, we should ignore when we feel assaulted or insulted, and avoid replying in kind.

Choose words wisely – The choice of words matter, because they convey implicit messages. In particular, avoid terms that carry judgement values: “good” or “bad”. For example “This is done wrong” (note that this sentence already avoids “you”), could be replaced by “There might be more numerically stable / efficient way of doing it” (note also the use of precise technical wording rather than the generic term “better”).

Use moderating words – Try to leave room for the other in the discussion. Statements too assertive close the door to different points of view: “this must be changed” (note the lack of “you”) should be avoided while “this should be changed” is better. For this reason, this article is riddled with words such as “tend”, “often”, “feel”, “may”, “might”.

Don’t blame someone else – If you feel that there is some pattern that you would like to change, do not point fingers, do not blame others. Rather, point yourself at the center of the story, find an example of this pattern with you, and the message should be that “it is a pattern that we should avoid. “We” is such a powerful term. It unites; it builds a team.

Give your understanding – If you feel that there is a misunderstanding, explain how you are feeling. But do it using “I”, and not “you”, and acknowledge the subjectivity: “I feel ignored” rather than “you are ignoring me”. Even better: only talk about the feeling: “I am loosing motivation, because this is not moving forward”, or “I think that am failing to convey why this numerical problem is such an important issue” (note the use of “I think”, which avoids casting the situation as necessarily true).

I hope this can be useful. I personally try to apply these rules, because I want to work better with others.


to many who gave me feedback: Adrin Jalali, Andreas Mueller, Elizabeth DuPre, Emmanuelle Gouillart, Guillaume Lemaitre, Joel Nothman, Joris Van den Bossche, Nicolas Hug.

PS: note how many times I’ve used “you” above. I can clearly get better at communication!

May 27, 2020 10:00 PM UTC

Real Python

How to Write an Installable Django App

In the Django framework, a project refers to the collection of configuration files and code for a particular website. Django groups business logic into what it calls apps, which are the modules of the Django framework. There’s plenty of documentation on how to structure your projects and the apps within them, but when it comes time to package an installable Django app, information is harder to find.

In this tutorial, you’ll learn how to take an app out of a Django project and package it so that it’s installable. Once you’ve packaged your app, you can share it on PyPI so that others can fetch it through pip install.

In this tutorial, you’ll learn:

  • What the differences are between writing stand-alone apps and writing apps inside of projects
  • How to create a setup.cfg file for publishing your Django app
  • How to bootstrap Django outside of a Django project so you can test your app
  • How to test across multiple versions of Python and Django using tox
  • How to publish your installable Django app to PyPI using Twine

Be sure to follow along with the examples by downloading the source code at the link below:

Download the sample code: Click here to get the code you'll use to learn how to write an installable Django app in this tutorial.


This tutorial requires some familiarity with Django, pip, PyPI, pyenv (or an equivalent virtual environment tool), and tox. To learn more about these topics, check out:

Starting a Sample Django App in a Project

This tutorial includes a working package to help guide you through the process of making an installable Django app. You can download the source code at the link below:

Download the sample code: Click here to get the code you'll use to learn how to write an installable Django app in this tutorial.

Even if you originally intend to make your Django app available as a package, you’re likely to start inside a project. To demonstrate the process of moving from Django project to installable Django app, I’ve made two branches available in the repo. The project branch is the starting state of an app inside of a Django project. The master branch is the finished installable app.

You can also download the finished app at the PyPI realpython-django-receipts package page. You can install the package by running pip install realpython-django-receipts.

The sample app is a short representation of the line items on a receipt. In the project branch, you’ll find a directory named sample_project that contains a working Django project. The directory looks like this:

├── receipts/
│   ├── fixtures/
│   │   └── receipts.json
│   │
│   ├── migrations/
│   │   ├──
│   │   └──
│   │
│   ├──
│   ├──
│   ├──
│   ├──
│   ├──
│   ├──
│   └──
├── sample_project/
│   ├──
│   ├──
│   ├──
│   ├──
│   └──
├── db.sqlite3

The most current version of Django at the time this tutorial was written was 3.0.4, and all testing was done with Python 3.7. None of the steps outlined in this tutorial should be incompatible with earlier versions of Django—I’ve used these techniques since Django 1.8. However, some changes are necessary if you’re using Python 2. To keep the examples simple, I’ve assumed Python 3.7 across the code base.

Creating the Django Project From Scratch

The sample project and receipts app were created using the Django admin command and some small edits. To start, run the following code inside of a clean virtual environment:

$ python -m pip install Django
$ django-admin startproject sample_project
$ cd sample_project
$ ./ startapp receipts

This creates the sample_project project directory structure and a receipts app subdirectory with template files that you’ll use to create your installable Django app.

Next, the sample_project/ file needs a few modifications:

  • Add '' to the ALLOWED_HOSTS setting so you can test locally.
  • Add 'receipts' to the INSTALLED_APPS list.

You’ll also need to register the receipts app’s URLs in the sample_project/ file. To do so, add path('receipts/', include('receipts.urls')) to the url_patterns list.

Exploring the Receipts Sample App

Read the full article at »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 27, 2020 02:00 PM UTC


Pros and Cons of Python: A Definitive Python Web Development Guide

Python web development is a popular programming language for most companies today. Read here the 5 Important pros and cons of using python for web development.

May 27, 2020 11:35 AM UTC


Return people from a list and dictionary with Python

In this article, we are going to return a list of names that show whether that person is nice or naughty based on True (Nice) or False (Naughty) value from the ‘was_nice’ key within a list of dictionaries pass into either the get_nice_names function which will only return the name of a person who is nice or get_naughty_names function which will do vise versa.

A list of dictionaries will get passed into below functions that will return either a list of nice person names or a list of naughty person names.

def get_nice_names(people):

    nice_name_list = []
    for nice_list in people:
        if nice_list["was_nice"] == True:
    return nice_name_list

def get_naughty_names(people):

    bad_name_list = []
    for bad_list in people:
        if bad_list["was_nice"] == False:
    return bad_name_list

So if you pass in a list of dictionaries into the nice person function you will receive a list of nice person names (was_nice == True) in return. For example,

nice = [{'name': 'Santa', 'was_nice': True}, {'name': 'Warrior reading this kata', 'was_nice': True}] # ['Santa', 'Warrior reading this kata']

Leave your comment below this post if you have another solution to the above problem.

May 27, 2020 06:10 AM UTC

Mike Driscoll

Python 101 – Learning About Loops

There are many times when you are writing code that you will need to find a way to iterate over something. Perhaps you’ll need to iterate over the letters in a string or the objects in a list. The process of iterating over something is done via a loop.

A loop is a programming construct that allows you to iterate over chunks. Those chunks could be the letters in the string or the lines of a file.

In Python, there are two types of loop constructs:

Besides iterating over sequences, you can use a loop to do the same thing multiple times. One example is a web server that is basically an infinite loop. A server waits, listening for a client to send it a message. When it receives the message, the loop will call a function in response.

Another example is the game loop. When you beat a game or lose a game, the game doesn’t usually exit. Instead, it will ask you if you want to play again. This is also done by wrapping the entire program in a loop.

In this chapter you will learn how to:

Let’s get started by looking at the for loop!

Creating a for Loop

The for loop is the most popular looping construct in Python. A for loop is created using the following syntax:

for x in iterable:
    # do something

Now the code above does nothing. So let’s write a for loop that iterates over a list, one item at a time:

>>> my_list = [1, 2, 3]
>>> for item in my_list:
...     print(item)

In this code, you create a list with three integers in it. Next you create a for loop that says “for each item in my list, print out the item”.

Of course, most of the time you will actually want to do something to the item. For example, you might want to double it:

>>> my_list = [1, 2, 3]
>>> for item in my_list:
...     print(f'{item * 2}')

Or you might want to only print out only the even-numbered items:

>>> my_list = [1, 2, 3]
>>> for item in my_list:
...     if item % 2 == 0:
...         print(f'{item} is even')
2 is even

Here you use the modulus operator, %, to find the remainder of the item divided by 2. If the remainder is 0, then you know that an item is an even number.

You can use loops and conditionals and any other Python construct to create complex pieces of code that are only limited by your imagination.

Let’s learn what else you can loop over besides lists.

Looping Over a String

One of the differences of the for loop in Python versus other programming languages is that you can iterate over any sequence. So you can iterate over other data types.

Let’s look at iterating over a string:

>>> my_str = 'abcdefg'
>>> for letter in my_str:
...     print(letter)

This shows you how easy it is to iterate over a string.

Now let’s try iterating over another common data type!

Looping Over a Dictionary

Python dictionaries also allow looping. By default, when you loop over a dictionary, you will loop over its keys:

>>> users = {'mdriscoll': 'password', 'guido': 'python', 'steve': 'guac'}
>>> for user in users:
...     print(user)

You can loop over both the key and the value of a dictionary if you make use of its items() method:

>>> users = {'mdriscoll': 'password', 'guido': 'python', 'steve': 'guac'}
>>> for user, password in users.items():
...     print(f"{user}'s password is {password}")
mdriscoll's password is password
guido's password is python
steve's password is guac

In this example, you specify that you want to extract the user and the password in each iteration. As you might recall, the items() method returns a view that is formatted like a list of tuples. Because of that, you can extract each key: value pair from this view and print them out.

This leads us to looping over tuples and getting out individual items from a tuple while looping!

Extracting Multiple Values in a Tuple While Looping

Sometimes you will need to loop over a list of tuples and get each item within the tuple. It sounds kind of weird, but you will find that it is a fairly common programming task.

>>> list_of_tuples = [(1, 'banana'), (2, 'apple'), (3, 'pear')]
>>> for number, fruit in list_of_tuples:
...     print(f'{number} - {fruit}')
1 - banana
2 - apple
3 - pear

To get this to work, you take advantage of the fact that you know each tuple has two items in it. Since you know the format of the list of tuples ahead of time, you know how to extract the values.

If you hadn’t extracted the items individually from the tuples, you would have ended up with this kind of output:

>>> list_of_tuples = [(1, 'banana'), (2, 'apple'), (3, 'pear')]
>>> for item in list_of_tuples:
...     print(item)
(1, 'banana')
(2, 'apple')
(3, 'pear')

This is probably not what you expected. You will usually want to extract an item from the tuple or perhaps multiple items, rather than extracting the entire tuple.

Now let’s discover another useful way to loop!

Using enumerate with Loops

Python comes with a built-in function called enumerate. This function takes in an iterator or sequence, like a string or list, and returns a tuple in the form of (position, item).

This allows you to know the position of the item in the sequence easily while looping over the sequence.

Here’s an example:

>>> my_str = 'abcdefg'
>>> for pos, letter in enumerate(my_str):
...     print(f'{pos} - {letter}')
0 - a
1 - b
2 - c
3 - d
4 - e
5 - f
6 - g

Now let’s look at the other type of loop that Python supports!

Creating a while Loop

Python has one other type of looping construct that is called the while loop. A while loop is created with the keyword while followed by an expression. In other words, while loops will run until a specific condition is met.

Let’s take a look at how these loops work:

>>> count = 0
>>> while count < 10:
...     print(count)
...     count += 1

This loop is formulated in much the same way as a conditional statement. You tell Python that you want the loop to run as long as the count is less than 10. Inside of the loop, you print out the current count and then you increment the count by one.

If you forgot to increment the count, the loop would run until you stop or terminate the Python process.

You can create an infinite loop by making that mistake or you could do something like this:

Since the expression is always True, this code will print out the string, “Program running”, forever or until you kill the process.

Breaking Out of a Loop

Sometimes you want to stop a loop early. For example, you might want to loop until you find something specific. A good use case would be looping over the lines in a text file and stopping when you find the first occurrence of a particular string.

To stop a loop early, you can use the keyword break:

>>> count = 0
>>> while count < 10:
...     if count == 4:
...         print(f'{count=}')
...         break
...     print(count)
...     count += 1

In this example, you want the loop to stop when the count reaches 4. To make that happen, you add a conditional statement that checks if count equals 4. When it does, you print out that the count equals 4 and then use the break statement to break out of the loop.

You can also use break in a for loop:

>>> list_of_tuples = [(1, 'banana'), (2, 'apple'), (3, 'pear')]
>>> for number, fruit in list_of_tuples:
...     if fruit == 'apple':
...         print('Apple found!')
...         break
...     print(f'{number} - {fruit}')
1 - banana
Apple found!

For this example, you want to break out of the loop when you find an apple. Otherwise you print out what fruit you have found. Since the apple is in the second tuple, you will never get to the third one.

When you use break, the loop will only break out of the innermost loop that the break statement is in.

You can use break to help control the flow of the program. In fact, conditional statements along with break are known as flow control statements.

Another statement you can use to control the flow of your code is continue. Let’s look at that next!

Using continue

The continue statement is used for continuing to the next iteration in the loop. You can use continue to skip over something.

Let’s write a loop that skips over even numbers:

>>> for number in range(2, 12):
...     if number % 2 == 0:
...         continue
...     print(number)

In this code, you loop over a range of numbers starting at 2 and ending at 11. For each number in this range, you use the modulus operator, %, to get the remainder of the number divided by 2. If the remainder is zero, it’s an even number and you use the continue statement to continue to the next value in the sequence. This effectively skips even numbers so that you only print out the odd ones.

You can use clever conditional statements to skip over any number of things in a sequence by using the continue statement.

Loops and the else Statement

A little known fact about Python loops is that you can add an else statement to them like you do with an if/else statement. The else statement only gets executed when no break statement occurs.

Another way to look at it is that the else statement only executes if the loop completes successfully.

The primary use case for the else statement in a loop is for searching for an item in a sequence. You would use the else statement to raise an exception if the item was not found.

Let’s look at a quick example:

>>> my_list = [1, 2, 3]
>>> for number in my_list:
...     if number == 4:
...         print('Found number 4!')
...         break
...     print(number)
... else:
...     print('Number 4 not found')
Number 4 not found

This example loops over a list of three integers. It looks for the number 4 and will break out of the loop if it is found. If that number is not found, then the else statement will execute and let you know.

Try adding the number 4 to the list and then re-run the code:

>>> my_list = [1, 2, 3, 4]
>>> for number in my_list:
...     if number == 4:
...         print('Found number 4')
...         break
...     print(number)
... else:
...     print('Number 4 not found')
Found number 4

A more proper way of doing this would be to raise an exception rather than printing a message.

Nesting Loops

Loops can also be nested inside of each other. There are many reasons to nest loops. One of the most common reasons is to unravel a nested data structure.

Let’s use a nested list for your example:

>>> nested = [['mike', 12], ['jan', 15], ['alice', 8]]
>>> for lst in nested:
...     print(f'List = {lst}')
...     for item in lst:
...         print(f'Item -> {item}')

The outer loop will extract each nested list and print it out as well. Then in the inner loop, your code will extract each item within the nested list and print it out.

If you run this code, you should see output that looks like this:

List = ['mike', 12]
Item -> mike
Item -> 12
List = ['jan', 15]
Item -> jan
Item -> 15
List = ['alice', 8]
Item -> alice
Item -> 8

This type of code is especially useful when the nested lists are of varying lengths. You may need to do extra processing on the lists that have extra data or not enough data in them, for example.

Wrapping Up

Loops are very helpful for iterating over data. In this article, you learned about Python’s two looping constructs:

You also learned about flow control using the break and continue statements. Finally you learned how to use else in your loops and why you might want to nest loops.

With a little practice, you will soon become quite adept at using loops in your own code!

The post Python 101 – Learning About Loops appeared first on The Mouse Vs. The Python.

May 27, 2020 05:05 AM UTC

May 26, 2020

PyCoder’s Weekly

Issue #422 (May 26, 2020)

#422 – MAY 26, 2020
View in Browser »

The PyCoder’s Weekly Logo

Waiting in asyncio

There are many ways to wait on the results of a coroutine in Python’s asyncio framework. They all have different properties and use cases. In this article, they’ve all been collected in one place to help you figure out which one you need.

Latency in Asynchronous Python

Where does latency in asynchronous programs come from? Explore some common mistakes in asyncio programming, the differences between bounded and unbounded queues, and a suggestion for some best practices for programming with asyncio.

Top 10 Python Tools to Make a Developer’s Life Easier


Do you ever feel like your list of sprint tasks is infinite? ActiveState has curated the top 10 Python tools to automate the most common, repetitive tasks. From web framework to code template creation, make your life easier and focus on your next big project! Check out the top 10 list →

Creating and Modifying PDF Files in Python

Explore the different ways of creating and modifying PDF files in Python. You’ll learn how to read and extract text, merge and concatenate files, crop and rotate pages, encrypt and decrypt files, and even create PDFs from scratch.

Preventing SQL Injection: A Django Author’s Perspective

How do you prevent a SQL injection? Learn about best practices and tools for checking for vulnerabilities in a codebase in this post from Django creator Jacob Kaplan-Moss.

Python Job Hunting in a Pandemic

Do you know someone in the Python community who recently was let go from their job due to the pandemic? What does the job landscape currently look like? What are skills and techniques that will help you in your job search? This week Kyle Stratis discusses how he is managing his job search after just being let go from his data engineering job.

Hypermodern Python

Opinionated guide to setting up a productive Python environment, including recommendations for testing, linting, documentation, and continuous integration.


Renaming Months From Number to Name in Pandas

In a DataFrame indexed by month and day, how do you change the month index from numbers, like 1, to a name, like Jan?

Why Is -11 % 5 Equal to 4?

How well do you know the modulo operator?

Python Jobs

Senior Python Engineer (Remote)


Senior Python Engineer (Remote - US East Coast)

Enterra Solutions

Kubernetes Engineer: Docker, Python, Linux (Remote)

Travelers Insurance

Senior Python Developer (Washington, DC)


More Python Jobs >>>

Articles & Tutorials

Concurrency in Python for Network I/O

What’s the best way to request 20,000 URLs in Python? Compare the performance of synchronous requests, multiprocessing, threading, and asyncio to find out which method is the fastest and why.

Keyword-Only Arguments In Python

Keyword-only arguments have been around for a while in Python, but you don’t see them used very frequently. Learn why keyword-only arguments are useful, and why you might want to start using them in your own code.

The Cloud Python Developers Love


DigitalOcean is the cloud provider that makes it easy for developers to deploy and scale their applications. From Flask and Django apps to JupyterHub Notebook servers, DigitalOcean enables Python developers to focus their energy on creating software →

Python Community Interview With Russell Keith-Magee

Russell Keith-Magee is a Django core developer and the founder and maintainer of the BeeWare project. In the interview, he talks about his mission to help Python become a feasible option for writing and packaging mobile applications, his PyCon 2020 talk, and more.

Debugging a Containerized Django App in VS Code

Learn how to create a VS Code run configuration to attach to a Docker container and modify to start a ptvsd (Python Tools for Visual Studio Debug Server) debug server so that you can debug a containerized Django Project in VS Code.
J-O ERIKSSON • Shared by J-O Eriksson

Profile, Understand & Optimize Python Code Performance

You can’t improve what you can’t measure. Profile and understand Python code’s behavior and performance (Wall-time, I/O, CPU, HTTP requests, SQL queries). Browse through appealing graphs. is now available as Public Beta.

A Beginner’s Guide to Pip

What is pip? In this beginner-friendly course, you’ll learn how to use pip, the standard package manager for Python, so that you can install and manage additional packages that are not part of the Python standard library.

Fil: A New Python Memory Profiler for Data Scientists and Scientists

Fil is a Python memory profiler specifically designed for batch processing applications that read in data, process it, and write out the result.

API Request Caching With Python & Redis

REDOWAN DELOWAR • Shared by Redowan Delowar

Projects & Code

einsteinpy: Community Python Package for General Relativity


filprofiler: Memory Profiler for Data Batch Processing Applications Written in Python


pdm: Modern Python Package Manager With PEP 582 Support

GITHUB.COM/FROSTMING • Shared by Frost Ming

cascadia-code: Fun, New Monospaced Font That Includes Programming Ligatures


django-grpc-gramework: gRPC for Django

GITHUB.COM/FENGSP • Shared by Shipeng Feng

Happy Pythoning!
This was PyCoder’s Weekly Issue #422.
View in Browser »


[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

May 26, 2020 07:30 PM UTC

Real Python

A Beginner's Guide to Pip

What is pip? pip is the standard package manager for Python. It allows you to install and manage additional packages that are not part of the Python standard library. This course is an introduction to pip for new Pythonistas.

In this course, you’ll learn about:

As you’ll see, the Python community is very active and has created some neat alternatives to pip that you’ll learn about later in this course.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 26, 2020 02:00 PM UTC