Planet Python

Last update: October 31, 2020 04:47 AM UTC

October 30, 2020

NumFOCUS

Public Apology to Jeremy Howard

We, the NumFOCUS Code of Conduct Enforcement Committee, issue a public apology to Jeremy Howard for our handling of the JupyterCon 2020 reports. We should have done better. We thank you for sharing your experience and we will use it to improve our policies going forward. We acknowledge that it was an extremely stressful experience, […]

The post Public Apology to Jeremy Howard appeared first on NumFOCUS.

October 30, 2020 06:51 PM UTC

PythonClub - A Brazilian collaborative blog about Python

Fazendo backup do banco de dados no Django

Apresentação

Em algum momento, durante o seu processo de desenvolvimento com Django, pode ser que surja a necessidade de criar e restaurar o banco de dados da aplicação. Pensando nisso, resolvi fazer um pequeno tutorial, básico, de como realizar essa operação.

Nesse tutorial, usaremos o django-dbbackup, um pacote desenvolvido especificamente para isso.

Configurando nosso ambiente

Primeiro, partindo do início, vamos criar uma pasta para o nosso projeto e, nela, isolar o nosso ambiente de desenvolvimento usando uma virtualenv:

mkdir projeto_db && cd projeto_db #criando a pasta do nosso projeto

virtualenv -p python3.8 env && source env/bin/activate #criando e ativando a nossa virtualenv

Depois disso e com o nosso ambiente já ativo, vamos realizar os seguintes procedimentos:

pip install -U pip #com isso, atualizamos a verão do pip instalado

Instalando as dependências

Agora, vamos instalar o Django e o pacote que usaremos para fazer nossos backups.

pip install Django==3.1.2 #instalando o Django

pip install django-dbbackup #instalando o django-dbbackup

Criando e configurando projeto

Depois de instaladas nossas dependências, vamos criar o nosso projeto e configurar o nosso pacote nas configurações do Django.

django-admin startproject django_db . #dentro da nossa pasta projeto_db, criamos um projeto Django com o nome de django_db.

Depois de criado nosso projeto, vamos criar e popular o nosso banco de dados.

python manage.py migrate #com isso, sincronizamos o estado do banco de dados com o conjunto atual de modelos e migrações.

Criado nosso banco de dados, vamos criar um superusuário para podemos o painel admin do nosso projeto.

python manage.py createsuperuser

Perfeito. Já temos tudo que precisamos para executar nosso projeto. Para execução dele, é só fazermos:

python manage.py runserver

Você terá uma imagem assim do seu projeto:

Configurando o django-dbbackup

Dentro do seu projeto, vamos acessar o arquivo settings.py, como expresso abaixo:

django_db/
├── settings.py

Dentro desse arquivos iremos, primeiro, adiconar o django-dbbackup às apps do projeto:

INSTALLED_APPS = (
    ...
    'dbbackup',  # adicionando django-dbbackup
)

Depois de adicionado às apps, vamos dizer para o Django o que vamos salvar no backup e, depois, indicar a pasta para onde será encaminhado esse arquivo. Essa inserção deve ou pode ser feita no final do arquivo settings.py:

DBBACKUP_STORAGE = 'django.core.files.storage.FileSystemStorage' #o que salvar
DBBACKUP_STORAGE_OPTIONS = {'location': 'backups/'} # onde salvar

Percebam que dissemos para o Django salvar o backup na pasta backups, mas essa pasta ainda não existe no nosso projeto. Por isso, precisamos criá-la [fora da pasta do projeto]:

mkdir backups

Criando e restaurando nosso backup

Já temos tudo pronto. Agora, vamos criar o nosso primeiro backup:

python manage.py dbbackup

Depois de exetudado, será criado um arquivo -- no nosso exemplo, esse arquivo terá uma extensão .dump --, salvo na pasta backups. Esse arquivo contem todo backup do nosso banco de dados.

Para recuperarmos nosso banco, vamos supor que migramos nosso sistema de um servidor antigo para um novo e, por algum motivo, nossa base de dados foi corrompida, inviabilizando seu uso. Ou seja, estamos com o sistema/projeto sem banco de dados -- ou seja, exlua ou mova a a sua base dados .sqlite3 para que esse exemplo seja útil --, mas temos os backups. Com isso, vamos restaurar o banco:

python manage.py dbrestore

Prontinho, restauramos nosso banco de dados. O interessante do django-dbbackup, dentre outras coisas, é que ele gera os backups com datas e horários específicos, facilitando o processo de recuperação das informações mais recentes.

Por hoje é isso, pessoal. Até a próxima. ;)

October 30, 2020 01:40 PM UTC

Real Python

The Real Python Podcast – Episode #33: Going Beyond the Basic Stuff With Python and Al Sweigart

You probably have heard of the bestselling Python book, "Automate the Boring Stuff with Python." What are the next steps after starting to dabble in the Python basics? Maybe you've completed some tutorials, created a few scripts, and automated repetitive tasks in your life. This week on the show, we have author Al Sweigart to talk about his new book, "Beyond the Basic Stuff with Python: Best Practices for Writing Clean Code."

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 30, 2020 12:00 PM UTC

Reuven Lerner

Join the data revolution with my “Intro to SQL” course!

Have you heard? Data is “the new oil” — meaning, data is the most valuable and important thing in the modern world. Which means that if you can store, retrieve, and organize your data, then you (and your company) are positioned for greater success.

This usually means working with a database — and frequently, a relational database, with which you communicate using a language called SQL.

In other words: SQL is the key to the modern data revolution. But too often, people are put off from learning SQL. It seems weird, even when compared with a programming language.

Well, I have good news: If you want to join the data revolution and work with databases, I’m offering a new course. On November 15th, I’ll be teaching a live, 4-hour online course, “Intro to SQL.” I’ll teach you the basics of what you need to work with a database.

The course includes:

• Access to the live, 4-hour online course, including numerous exercises and opportunities for Q&A
• Access to the course recording, forever
• Participation in our private forum, where you can ask me (and others) database-related questions

I’ve been using databases since 1995, and have been teaching SQL for more than 20 years. This course is based on that corporate training, and is meant to get you jump started into the world of data and relational databases. We’ll be using PostgreSQL, a powerful open-source database I’ve been using for more than two decades.

Questions? Learn more at https://store.lerner.co.il/intro-to-sql (where there’s an extensive FAQ). Or contact me on Twitter (@reuvenmlerner) or via e-mail (reuven@lerner.co.il). I’ll answer as soon as I can.

I hope to see you there!

The post Join the data revolution with my “Intro to SQL” course! appeared first on Reuven Lerner.

October 30, 2020 02:15 AM UTC

October 29, 2020

Python Morsels

Data structures contain pointers

Watch First:

• Variables are Pointers
• The 2 Types of "Change" in Python
• Equality vs Identity

Transcript

Data structures in Python don't actually contain objects. They references to objects (aka "pointers").

Referencing the same object in multiple places

Let's take a list of three zeroes:

>>> row = [0, 0, 0]

If we make a new list like this:

>>> matrix = [row, row, row]
>>> matrix
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]

We'll end up with a list of lists of zeros. We now have three lists, and each of them has three zeros inside it.

If we change one of the values in this list of lists to 1:

>>> matrix[1][1] = 1

What do you think will happen? What do you expect will change?

We're asking to change the middle item in the middle list.

So, matrix[1] is referencing index one inside the matrix, which is the second list (the middle one). Index one inside of matrix[1] (i.e. matrix[1][1]) is the second element in that list, so we should be changing the middle zero in the middle list here.

That's not quite what happens:

>>> matrix
[[0, 1, 0], [0, 1, 0], [0, 1, 0]]

Instead we changed the middle number in every list!

This happened because our matrix list doesn't actually contain 3 lists, it contains three references to the same list:

>>> matrix[0] is matrix[1]
True

We talked about the fact that all variables in Python are actually pointers. **Variables point to objects, they don't contain objects: they aren't buckets containing objects

So unlike many other programming languages, Python's varibales are not buckets containing objects. Likewise, Python's data structures are also not buckets containing objects. Python's data structures contain pointers to objects, they don't contain the objects themselves.

If we look at the row list, we'll see that it's changed too:

>>> row
[0, 1, 0]

We stored three pointers to the same list. When we "changed" one of these lists, we mutated that list (one of our two types of change in Python). And that seems to change any variable that references that list.

So matrix[0], matrix[1], and row, all are exactly the same object. We can verify this using id:

>>> id(row)
1972632707784
>>> id(matrix[0])
1972632707784
>>> id(matrix[1])
1972632707784
>>> id(matrix[2])
1972632707784

Avoiding referencing the same object

If we wanted to avoid this issue, we could manually make a list of three lists:

>>> matrix = [[0, 0, 0], [0, 0, 0], [0, 0, 0]]
>>> matrix
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]

This is not going to suffer from the same problem, because these are three independent lists.

>>> matrix[1][1] = 1
>>> matrix
[[0, 0, 0], [0, 1, 0], [0, 0, 0]]

They're different lists stored in different parts of memory:

>>> matrix[0] is matrix[1]
False
>>> matrix[0] is matrix[2]
False

An ouroboros: A list that contains itself

So data structures contain pointers, not objects.

This is the ultimate demonstration of this fact:

>>> x = []
>>> x.append(x)

The ultimate demonstration of this fact is that we can take a list and stick that list inside of itself:

At this point the first element (and only element) of this list is the list itself:

>>> x[0] is x
True

And the first element of that list is also the list itself:

>>> x[0][0] is x
True

We can index this list of lists as far down as we want because we've made an infinitely recursive data structure:

>>> x[0][0][0] is x
True
>>> x[0][0][0][0][0] is x
True

Python represents this list at the Python prompt by putting three dots inside those square brackets (it's smart enough not to show an infinite number of square brackets):

>>> x
[[...]]

We didn't stick a bucket inside itself here: we didn't stick a list inside of the same list. Instead we stuck a pointer to a list inside of itself.

Lists are allowed to store pointers to anything, even themselves.

Summary

The takeaway here is that just as variables in Python are pointers, data structures in Python contain pointers. You can't "contain" an object inside another object in Python, you can really only point to an object. You can only reference objects in Python. Lists, tuples, dictionaries, and all other data structures contain pointers.

October 29, 2020 03:00 PM UTC

Stack Abuse

How to Sort a Dictionary by Value in Python

Introduction

A dictionary in Python is a collection of items that stores data as key-value pairs. In Python 3.7 and later versions, dictionaries are sorted by the order of item insertion. In earlier versions, they were unordered.

Let's have a look at how we can sort a dictionary on basis of the values they contain.

Sort Dictionary Using a for Loop

We can sort a dictionary with the help of a for loop. First, we use the sorted() function to order the values of the dictionary. We then loop through the sorted values, finding the keys for each value. We add these keys-value pairs in the sorted order into a new dictionary.

Note: Sorting does not allow you to re-order the dictionary in-place. We are writing the ordered pairs in a completely new, empty dictionary.

dict1 = {1: 1, 2: 9, 3: 4}
sorted_values = sorted(dict1.values()) # Sort the values
sorted_dict = {}

for i in sorted_values:
    for k in dict1.keys():
        if dict1[k] == i:
            sorted_dict[k] = dict1[k]
            break

print(sorted_dict)

If you run this with the Python interpreter you would see:

{1: 1, 3: 4, 2: 9}

Now that we've seen how to sort with loops, let's look at a more popular alternative that uses the sorted() function.

Sort Dictionary Using the sorted() Function

We previously used the sorted() function to sort the values of an array. When sorting a dictionary, we can pass one more argument to the sorted() function like this: sorted(dict1, key=dict1.get).

Here, key is a function that's called on each element before the values are compared for sorting. The get() method on dictionary objects returns the value of for a dictionary's key.

The sorted(dict1, key=dict1.get) expression will return the list of keys whose values are sorted in order. From there, we can create a new, sorted dictionary:

dict1 = {1: 1, 2: 9, 3: 4}
sorted_dict = {}
sorted_keys = sorted(dict1, key=dict1.get)  # [1, 3, 2]

for w in sorted_keys:
    sorted_dict[w] = dict1[w]

print(sorted_dict) # {1: 1, 3: 4, 2: 9}

Using the sorted() function has reduced the amount of code we had to write when using for loops. However, we can further combine the sorted() function with the itemgetter() function for a more succinct solution to sorting dictionaries by values.

Sort Dictionary Using the operator Module and itemgetter()

The operator module includes the itemgetter() function. This function returns a callable object that returns an item from an object.

For example, let's use to itemgetter() to create a callable object that returns the value of any dictionary with a key that's 2:

import operator

dict1 = {1: 1, 2: 9}
get_item_with_key_2 = operator.itemgetter(2)

print(get_item_with_key_2(dict1))  # 9

Every dictionary has access to the items() method. This function returns the key-value pairs of a dictionary as a list of tuples. We can sort the list of tuples by using the itemgetter() function to pull the second value of the tuple i.e. the value of the keys in the dictionary.

Once it's sorted, we can create a dictionary based on those values:

import operator

dict1 = {1: 1, 2: 9, 3: 4}
sorted_tuples = sorted(dict1.items(), key=operator.itemgetter(1))
print(sorted_tuples)  # [(1, 1), (3, 4), (2, 9)]
sorted_dict = {k: v for k, v in sorted_tuples}

print(sorted_dict) # {1: 1, 3: 4, 2: 9}

With much less effort, we have a dictionary sorted by values!

As the key argument accepts any function, we can use lambda functions to return dictionary values so they can be sorted. Let's see how.

Sort Dictionary Using a Lambda Function

Lambda functions are anonymous, or nameless, functions in Python. We can use lamba functions to get the value of a dictionary item without having to import the operator module for itemgetter(). If you'd like to learn more about lambas, you can read about them in our guide to Lambda Functions in Python.

Let's sort a dictionary by values using a lambda function in the key argument of sorted():

dict1 = {1: 1, 2: 9, 3: 4}
sorted_tuples = sorted(dict1.items(), key=lambda item: item[1])
print(sorted_tuples)  # [(1, 1), (3, 4), (2, 9)]
sorted_dict = {k: v for k, v in sorted_tuples}

print(sorted_dict)  # {1: 1, 3: 4, 2: 9}

Note that the methods we've discussed so far only work with Python 3.7 and later. Let's see what we can do for earlier versions of Python.

Returning a New Dictionary with Sorted Values

After sorting a dictionary by values, to keep a sorted dictionary in Python versions before 3.7, you have to use the OrderedDict - available in the collections module. These objects are dictionaries that keep the order of insertion.

Here's an example of sorting and using OrderedDict:

import operator
from collections import OrderedDict

dict1 = {1: 1, 2: 9, 3: 4}
sorted_tuples = sorted(dict1.items(), key=operator.itemgetter(1))
print(sorted_tuples)  # [(1, 1), (3, 4), (2, 9)]

sorted_dict = OrderedDict()
for k, v in sorted_tuples:
    sorted_dict[k] = v

print(sorted_dict)  # {1: 1, 3: 4, 2: 9}

Conclusion

This tutorial showed how a dictionary can be sorted based on its values. We first sorted a dictionary using two for loops. We then improved our sort by using the sorted() function. We've also seen the itemgetter() function from the operator module can make our solution more succinct.

Lastly, we adapted our solution to work on Python versions lower than 3.7.

Variations of the sorted() function are the most popular and reliable to sort a dictionary by values.

October 29, 2020 12:30 PM UTC

Matt Layman

Sending Invites - Building SaaS #77

In this episode, I worked on the form that will send invites to users for the new social network app that I’m building. We built the view, the form, and the tests and wired a button to the new view. The first thing that we do was talk through the new changes since the last stream. After discussing the progress, I took some time to cover the expected budget for the application to get it to an MVP.

October 29, 2020 12:00 AM UTC

October 28, 2020

PyCharm

PyCharm 2020.3 EAP #3

The third build of PyCharm 2020.3 is now available in the Early Access Program with features and fixes for a smoother, more productive experience.

We invite you to join our EAP to try out the latest features we have coming up, test that they work properly in your environments, and help us make a better PyCharm for everyone!

DOWNLOAD PYCHARM 2020.3 EAP

Highlights

Interpreter settings

Now it is easier to create an environment for your project and set up all the dependencies at once.
When you clone a project from the repo, PyCharm checks if there is a requirements.txt, setup.py, environment.yml, or pipfile inside it. If there is, the IDE suggests per-project environment creation based on the detected files.

If you skip the environment creation at this step, autoconfiguration will still be available in the editor itself.

Inverting an “if” statement

Now you can easily invert “if” statements and switch them back in PyCharm. Kudos to Vasya Aksyonov, who contributed this feature to our open-source PyCharm Community Edition.

Go to the context menu for “if”, choose Show Context Actions, and then select “Invert ‘if’ condition”. The condition of the “if” statement will be inverted and the branches will switch places, preserving the initial semantics of the code.

When there is an “if” statement without an “else”, then after it has been inverted a “pass” will be created for the “if” that was inverted and an “else” clause will be added to the statement.

This feature works for all “if” statements without “elif” branches. The action also understands control flow, and can handle things like early return, producing sensible code.

Learn more.

VCS

We’ve added a Git tab to the Search Everywhere dialog. In it you can find commit hashes and messages, tags, and branches.

Web development

Create a React component from its usage

As you might know, PyCharm constantly checks that referenced variables and fields are valid. When they aren’t, in many cases it can suggest creating the relevant code construct for you. Now it can do this for React components, too. Place the caret at an unresolved component, press Alt+Enter, and then select the corresponding inspection. And you’re done!

Plugins enabled per project

We have taken plugin customization one step further. In Settings | Preferences / Plugins, the drop-down list next to the plugin name has been replaced with a new gear icon that has all the activation options. You can enable the plugin just for the current project or for all of them by selecting Enable for Current Project or Enable for All Projects.

Reader Mode

To make reading comments easier, we’ve implemented Reader Mode for read-only files and files from External Libraries. We’ve added a nicer display for font ligatures, code vision hints with the number of usages, and more. To configure the new mode, go to Preferences | Settings / Editor / Reader Mode.

Other updates

• PyCharm now supports the Couchbase Query service.
• The Concurrency Diagram button is now moved to the Profiler Executors group panel in the top right-hand corner of the editor.
• PyCharm now recognizes Python 3.10. Yes, we are already getting ready for it!

Notable fixes

The problem that caused copying the prompt together with the code when copying multiline commands is now fixed.

Ready to join the EAP?

Some ground rules

• EAP builds are free to use and expire 30 days after the build date.
• You can install an EAP build side by side with your stable PyCharm version.
• These builds are not fully tested and can be unstable.
• Your feedback is always welcome. Please use our issue tracker and make sure to mention your build version

How to download

Download this EAP from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date throughout the entire EAP. If you’re on Ubuntu 16.04 or later, you can use snap to get PyCharm EAP and stay up to date. You can find the installation instructions on our website.

This is all for today! For the full list of features and fixes present in this build, see our release notes. We also encourage you to stay tuned for more improvements, so come and share your feedback in the comments below, on Twitter, or via our issue tracker.

The PyCharm team

October 28, 2020 08:46 PM UTC

Stack Abuse

Change Tick Frequency in Matplotlib

Introduction

Matplotlib is one of the most widely used data visualization libraries in Python. Much of Matplotlib's popularity comes from its customization options - you can tweak just about any element from its hierarchy of objects.

In this tutorial, we'll take a look at how to change the tick frequency in Matplotlib. We'll do this on the figure-level as well as the axis-level.

How to Change Tick Frequency in Matplotlib?

Let's start off with a simple plot. We'll plot two lines, with random values:

import matplotlib.pyplot as plt
import numpy as np

fig = plt.subplots(figsize=(12, 6))

x = np.random.randint(low=0, high=50, size=100)
y = np.random.randint(low=0, high=50, size=100)

plt.plot(x, color='blue')
plt.plot(y, color='black')

plt.show()

x and y range from 0-50, and the length of these arrays is 100. This means, we'll have 100 datapoints for each of them. Then, we just plot this data onto the Axes object and show it via the PyPlot instance plt:

Now, the frequency of the ticks on the X-axis is 20. They're automatically set to a frequency that seems fitting for the dataset we provide.

Sometimes, we'd like to change this. Maybe we want to reduce or increase the frequency. What if we wanted to have a tick on every 5 steps, not 20?

The same goes for the Y-axis. What if the distinction on this axis is even more crucial, and we'd want to have each tick on evey step?

Setting Figure-Level Tick Frequency in Matplotlib

Let's change the figure-level tick frequency. This means that if we have multiple Axes, the ticks on all of these will be uniform and will have the same frequency:

import matplotlib.pyplot as plt
import numpy as np

fig = plt.subplots(figsize=(12, 6))

x = np.random.randint(low=0, high=50, size=100)
y = np.random.randint(low=0, high=50, size=100)

plt.plot(x, color='blue')
plt.plot(y, color='black')

plt.xticks(np.arange(0, len(x)+1, 5))
plt.yticks(np.arange(0, max(y), 2))

plt.show()

You can use the xticks() and yticks() functions and pass in an array. On the X-axis, this array starts on 0 and ends at the length of the x array. On the Y-axis, it starts at 0 and ends at the max value of y. You can hard code the variables in as well.

The final argument is the step. This is where we define how large each step should be. We'll have a tick at every 5 steps on the X-axis and a tick on every 2 steps on the Y-axis:

Setting Axis-Level Tick Frequency in Matplotlib

If you have multiple plots going on, you might want to change the tick frequency on the axis-level. For example, you'll want rare ticks on one graph, while you want frequent ticks on the other.

You can use the set_xticks() and set_yticks() functions on the returned Axes instance when adding subplots to a Figure. Let's create a Figure with two axes and change the tick frequency on them separately:

import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure(figsize=(12, 6))

ax = fig.add_subplot(121)
ax2 = fig.add_subplot(122)

x = np.random.randint(low=0, high=50, size=100)
y = np.random.randint(low=0, high=50, size=100)
z = np.random.randint(low=0, high=50, size=100)

ax.plot(x, color='blue')
ax.plot(y, color='black')
ax2.plot(y, color='black')
ax2.plot(z, color='green')

ax.set_xticks(np.arange(0, len(x)+1, 5))
ax.set_yticks(np.arange(0, max(y), 2))
ax2.set_xticks(np.arange(0, len(x)+1, 25))
ax2.set_yticks(np.arange(0, max(y), 25))

plt.show()

Now, this results in:

Conclusion

In this tutorial, we've gone over several ways to change the tick frequency in Matplotlib both on the figure-level as well as the axis-level.

If you're interested in Data Visualization and don't know where to start, make sure to check out our book on Data Visualization in Python.

Data Visualization in Python, a book for beginner to intermediate Python developers, will guide you through simple data manipulation with Pandas, cover core plotting libraries like Matplotlib and Seaborn, and show you how to take advantage of declarative and experimental libraries like Altair.

Data Visualization in Python

Understand your data better with visualizations! With over 275+ pages, you'll learn the ins and outs of visualizing data in Python with popular libraries like Matplotlib, Seaborn, Bokeh, and more.

October 28, 2020 07:55 PM UTC

Python Engineering at Microsoft

Python in Visual Studio Code – October 2020 Release

We are pleased to announce that the October 2020 release of the Python Extension for Visual Studio Code is now available. You can download the Python extension from the Marketplace, or install it directly from the extension gallery in Visual Studio Code. If you already have the Python extension installed, you can also get the latest update by restarting Visual Studio Code. You can learn more about Python support in Visual Studio Code in the documentation.

This was a short release where we addressed 14 issues, and it includes debugpy 1.0!

If you’re interested, you can check the full list of improvements in our changelog.

Debugpy 1.0

We’re excited to announce that we’re releasing the 1.0 version of our debugger, debugpy, that was first announced in March this year.

Debugpy offers a great number of features that can help you understand bugs, errors and unexpected behaviors in your code. You can find an extensive list on our documentation, but check below for some of our favorite ones!

Debugging Web Apps

Debugpy supports live reload of web applications, such as Django and Flask apps, when debugging. This means that when you make edits to your application, you don’t need to restart the debugger to get them applied: the web server is automatically reloaded in the same debugging session once the changes are saved.  

To try it out, open a web application and add a debug configuration (by clicking on Run > Add Configuration…, or by opening the Run view and clicking on create launch.json file).  Then select the framework used in your web application – in this example, we selected Flask. 

Now you hit F5 to start debugging, and then just watch the application reload once you make a change and save it!

You can also debug Django and Flask HTML templates. Just set up breakpoints to the relevant lines in the HTML files and watch the magic happen:

Debugging local processes

With the debugpy and the Python extension, you can get a list of processes running locally, and easily select one to attach debugpy to. Or, if you know the process ID, you can also add it directly to the “Attach using Process Id” configuration in the launch.json file:

Debugging remotely

Remote Development Extensions

You can use debugpy to debug your applications inside remote environments like Docker containers or remote machines (or even in WSL!) through the Remote Development extension. It allows VS Code to work seamlessly by running a light-weight server in the remote environment, while providing the same development experience as you get when developing locally:

This way, you can use the same configurations for debugpy as you would locally – but it will actually be installed and executed in the remote scope. No more messing around with your local environment!

You can learn more about the VS Code Remote Development extensions on the documentation.

Remote attach

You can also configure the debugger to attach to a debugpy server running on a remote machine. All you need to provide is the host name and the port number the debugpy server is listening to in the remote environment:

You can learn more about remote debugging in the documentation.

Other changes and enhancements

We have also added small enhancements and fixed issues requested by users that should improve your experience working with Python in Visual Studio Code. Some notable changes include:

• Fix exporting from the interactive window. (#14210)
• Do not opt users out of the insiders program if they have a stable version installed. (#14090)

We’re constantly A/B testing new features. If you see something different that was not announced by the team, you may be part of the experiment! To see if you are part of an experiment, you can check the first lines in the Python extension output channel. If you wish to opt-out of A/B testing, you can open the user settings.json file (View > Command Palette… and run Preferences: Open Settings (JSON)) and set the “python.experiments.enabled” setting to false.

Be sure to download the Python extension for Visual Studio Code now to try out the above improvements. If you run into any problems or have suggestions, please file an issue on the Python VS Code GitHub page.

The post Python in Visual Studio Code – October 2020 Release appeared first on Python.

October 28, 2020 07:03 PM UTC

Python Software Foundation

Key generation and signing ceremony for PyPI

On Friday October 30th at 11:15 AM EDT the Python Software Foundation will be live streaming a remote key generation and signing ceremony to bootstrap The Update Framework for The Python Package Index. You can click here to see what time this is in your local timezone.

This ceremony is one of the first practical steps in deploying The Update Framework to PyPI per PEP 458.

The Python Software Foundation Director of Infrastructure, Ernest W. Durbin III, and Trail of Bits Senior Security Engineer, William Woodruff, will be executing the runbook developed at https://github.com/psf/psf-tuf-runbook.

For transparency purposes a live stream will be hosted from the Python Software Foundation's YouTube channel. Please subscribe to the channel to be notified when the stream is live if you'd like to follow along.

Additionally the recording will be archived on the Python Software Foundation's YouTube channel.

This work is being funded by Facebook Research and was originally announced in late 2018 and a portion of it commenced in 2019 while awaiting PEP 458's acceptance. With PEP 458 in place we announced that work would commence in March.

We appreciate the patience and contributions of the community, Facebook Research, and Trail of Bits in seeing through the implementation of PEP 458.

Additionally volunteers from The Secure Systems Lab at NYU, Datadog, and VMWare have helped to develop the implementation for PyPI but have begun work on client implementations to verify the results in pip.

October 28, 2020 04:38 PM UTC

Real Python

Get Started With Django Part 3: Django View Authorization

$ python -m pip install django==3.0.7
$ django-admin startproject Blog
$ cd Blog
$ python manage.py startapp core
$ mkdir templates
$ python manage.py migrate
$ python manage.py createsuperuser
Username: superuser
Email address: superuser@example.com
Password:
Password (again):

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "core",
]

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [os.path.join(BASE_DIR, "templates")],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
            ],
        },
    },
]

from django.db import models

class Blog(models.Model):
    title = models.CharField(max_length=50)
    content = models.TextField()

from django.http import HttpResponse
from django.shortcuts import render, get_object_or_404
from core.models import Blog

def listing(request):
    data = {
        "blogs": Blog.objects.all(),
    }

    return render(request, "listing.html", data)

def view_blog(request, blog_id):
    blog = get_object_or_404(Blog, id=blog_id)
    data = {
        "blog": blog,
    }

    return render(request, "view_blog.html", data)

Read the full article at https://realpython.com/django-view-authorization/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 28, 2020 02:00 PM UTC

Will Kahn-Greene

Everett v1.0.3 released!

October 28, 2020 01:00 PM UTC

Peter Bengtsson

Generating random avatar images in Django/Python

tl;dr; <img src="/avatar.random.png" alt="Random avataaar"> generates this image:

(try reloading to get a random new one. funny aren't they?)

When you use Gravatar you can convert people's email addresses to their mugshot.
It works like this:

<img src="https://www.gravatar.com/avatar/$(md5(user.email))">

But most people don't have their mugshot on Gravatar.com unfortunately. But you still want to display an avatar that is distinct per user. Your best option is to generate one and just use the user's name or email as a seed (so it's always random but always deterministic for the same user). And you can also supply a fallback image to Gravatar that they use if the email doesn't match any email they have. That's where this blog post comes in.

I needed that so I shopped around and found avataaars generator which is available as a React component. But I need it to be server-side and in Python. And thankfully there's a great port called: py-avataaars.

It depends on CairoSVG to convert an SVG to a PNG but it's easy to install. Anyway, here's my hack to generate random "avataaars" from Django:

import io
import random

import py_avataaars
from django import http
from django.utils.cache import add_never_cache_headers, patch_cache_control


def avatar_image(request, seed=None):
    if not seed:
        seed = request.GET.get("seed") or "random"

    if seed != "random":
        random.seed(seed)

    bytes = io.BytesIO()

    def r(enum_):
        return random.choice(list(enum_))

    avatar = py_avataaars.PyAvataaar(
        style=py_avataaars.AvatarStyle.CIRCLE,
        # style=py_avataaars.AvatarStyle.TRANSPARENT,
        skin_color=r(py_avataaars.SkinColor),
        hair_color=r(py_avataaars.HairColor),
        facial_hair_type=r(py_avataaars.FacialHairType),
        facial_hair_color=r(py_avataaars.FacialHairColor),
        top_type=r(py_avataaars.TopType),
        hat_color=r(py_avataaars.ClotheColor),
        mouth_type=r(py_avataaars.MouthType),
        eye_type=r(py_avataaars.EyesType),
        eyebrow_type=r(py_avataaars.EyebrowType),
        nose_type=r(py_avataaars.NoseType),
        accessories_type=r(py_avataaars.AccessoriesType),
        clothe_type=r(py_avataaars.ClotheType),
        clothe_color=r(py_avataaars.ClotheColor),
        clothe_graphic_type=r(py_avataaars.ClotheGraphicType),
    )
    avatar.render_png_file(bytes)

    response = http.HttpResponse(bytes.getvalue())
    response["content-type"] = "image/png"
    if seed == "random":
        add_never_cache_headers(response)
    else:
        patch_cache_control(response, max_age=60, public=True)

    return response

It's not perfect but it works. The URL to this endpoint is /avatar.<seed>.png and if you make the seed parameter random the response is always different.

To make the image not random, you replace the <seed> with any string. For example (use your imagination):

{% for comment in comments %}
  <img src="/avatar.{{ comment.user.id }}.png" alt="{{ comment.user.name }}">
  <blockquote>{{ comment.text }}</blockquote>
  <i>{{ comment.date }}</i>
{% endfor %}

I've put together this test page if you want to see more funny avatar combinations instead of doing work :)

October 28, 2020 12:10 PM UTC

Codementor

Dissecting a Web stack

A layer-by-layer review of the components of a web stack and the reasons behind them

October 28, 2020 11:20 AM UTC

Introducing AutoScraper: A Smart, Fast and Lightweight Web Scraper For Python

Scraping the web just got a lot more automated

October 28, 2020 11:04 AM UTC

Stefan Scherfke

Raise … from … in Python

When you recently upgraded to pylint 2.6.0, you may have stumbled across a new warning:

src/mylib/core.py:74:20: W0707:
  Consider explicitly re-raising using the
  'from' keyword (raise-missing-from)

The reason for this message is an exception that you raised from within an except block like this:

>>> class MyLibError(Exception):
...     """Base class for all errors raised by mylib"""
...
>>> def do_stuff("onoes"):
...     try:
...         int(text)
...     except ValueError as e:
...         raise MyLibError(e)

When you run do_stuff(), you’ll get the following traceback:

>>> do_stuff(text)
Traceback (most recent call last):
  File "<stdin>", line 3, in do_stuff
ValueError: invalid literal for int() with base 10: 'onoes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in do_stuff
__main__.MyLibError: invalid literal for int() with base 10: 'onoes'

The important line here is:

During handling of the above exception, another exception occurred

This means that while you were handling the ValueError, another (unexpected) exception occurred: a MyLibError.

But this is not what we wanted to do – we wanted to replace the ValueError with MyLibError, so that our uses only have to handle a single exception type!

Enter raise … from …

To express I want to modify and forward an existing exception, you can use the raise NewException from cause syntax:

>>> def do_stuff(text):
...     try:
...         int(text)
...     except ValueError as e:
...         raise MyLibError(e) from e

When we run this now, we’ll get a different traceback printed:

>>> do_stuff("onoes")
Traceback (most recent call last):
  File "<stdin>", line 3, in do_stuff
ValueError: invalid literal for int() with base 10: 'onoes'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in do_stuff
__main__.MyLibError: invalid literal for int() with base 10: 'onoes'

Your users will now receive a MyLibError with the attached information, that the cause of this error was a ValueError somewhere in your code.

When the underlying cause is not important

If your users shouldn’t care about the underlying cause, because the new exception contains all the relevant information (i.e., that the provided input cannot be parsed), you may also omit the cause:

>>> def do_stuff(text):
...     try:
...         int(text)
...     except ValueError as e:
...         raise MyLibError(e) from None

When you run this now, you’ll get a nice an clean traceback:

>>> do_stuff("onoes")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in do_stuff
__main__.MyLibError: invalid literal for int() with base 10: 'onoes'

Summary

When you raise an exception from within an except block in Python, you have three options:

• If a new/unexpected exception occurs in the code handling the original exception, raise NewException.
• If you want to wrap the original exception(s) (e.g., with a common base exception to reduce complexity for your users), raise NewException from cause.
• If you want to hide the original exception because it is irrelevant for your users, raise NewException from None.

October 28, 2020 10:23 AM UTC

Reuven Lerner

Now playing on YouTube: Answers to your Python questions

Over the last year, I’ve gotten increasingly active on my YouTube channel, https://YouTube.com/reuvenlerner. Each week, I upload 1-2 new videos, typically answering questions that I’ve gotten in my corporate training classes or from people online — via e-mail, or on Twitter (@reuvenmlerner).

So if you’re looking to learn about Jupyter shortcuts, or inner classes in Python, or the differences between “modules” and “packages,” then head on over to https://YouTube.com/reuvenlerner, and subscribe! And if there are Python topics you would like me to address, don’t hesitate to contact me. You might see your question answered in a video!

The post Now playing on YouTube: Answers to your Python questions appeared first on Reuven Lerner.

October 28, 2020 07:28 AM UTC

October 27, 2020

Exxact Corp

PyTorch 1.7.0 Now Available

PyTorch 1.7.0

PyTorch is a widely used, open source deep learning platform used for easily writing neural network layers in Python enabling a seamless workflow from research to production. Based on Torch, PyTorch has become a powerful machine learning framework favored by esteemed researchers around the world.

The newest stable release of PyTorch, version 1.7.0, has a number of new highlights including  CUDA 11, New APIs for FFTs, Windows support for Distributed training and more.

PyTorch 1.7.0 Release Notes

• Highlights
• Backwards Incompatible Change
• New Features
• Improvements
• Performance
• Documentation

Interested in a deep learning solution?
Learn more about Exxact AI workstations starting at $3,700

Highlights

The PyTorch 1.7 release includes a number of new APIs including support for NumPy-Compatible FFT operations, profiling tools and major updates to both distributed data parallel (DDP) and remote procedure call (RPC) based distributed training. In addition, several features moved to stable including custom C++ Classes, the memory profiler, the creation of custom tensor-like objects, user async functions in RPC and a number of other features in torch.distributed such as Per-RPC timeout, DDP dynamic bucketing and RRef helper.

A few of the highlights include:

• CUDA 11 is now officially supported with binaries available at PyTorch.org
• Updates and additions to profiling and performance for RPC, TorchScript and Stack traces in the autograd profiler
• (Beta) Support for NumPy compatible Fast Fourier transforms (FFT) via torch.fft
• (Prototype) Support for Nvidia A100 generation GPUs and native TF32 format
• (Prototype) Distributed training on Windows now supported

To reiterate, starting PyTorch 1.6, features are now classified as stable, beta and prototype. You can see the detailed announcement here. Note that the prototype features listed in this blog are available as part of this release.

Front End APIs

[Beta] NumPy Compatible torch.fft module

FFT-related functionality is commonly used in a variety of scientific fields like signal processing. While PyTorch has historically supported a few FFT-related functions, the 1.7 release adds a new torch.fft module that implements FFT-related functions with the same API as NumPy.

This new module must be imported to be used in the 1.7 release, since its name conflicts with the historic (and now deprecated) torch.fft function.

Example usage:

>>> import torch.fft>>> t = torch.arange(4)>>> ttensor([0, 1, 2, 3]) >>> torch.fft.fft(t)tensor([ 6.+0.j, -2.+2.j, -2.+0.j, -2.-2.j]) >>> t = tensor([0.+1.j, 2.+3.j, 4.+5.j, 6.+7.j])>>> torch.fft.fft(t)tensor([12.+16.j, -8.+0.j, -4.-4.j,  0.-8.j])

• Documentation | Link

[Beta] C++ Support for Transformer NN Modules

Since PyTorch 1.5, we’ve continued to maintain parity between the python and C++ frontend APIs. This update allows developers to use the nn.transformer module abstraction from the C++ Frontend. And moreover, developers no longer need to save a module from python/JIT and load into C++ as it can now be used it in C++ directly.

• Documentation | Link

[Beta] torch.set_deterministic

Reproducibility (bit-for-bit determinism) may help identify errors when debugging or testing a program. To facilitate reproducibility, PyTorch 1.7 adds the torch.set_deterministic(bool) function that can direct PyTorch operators to select deterministic algorithms when available, and to throw a runtime error if an operation may result in nondeterministic behavior. By default, the flag this function controls is false and there is no change in behavior, meaning PyTorch may implement its operations nondeterministically by default.

More precisely, when this flag is true:

• Operations known to not have a deterministic implementation throw a runtime error;
• Operations with deterministic variants use those variants (usually with a performance penalty versus the non-deterministic version); and
• torch.backends.cudnn.deterministic = True is set.

Note that this is necessary, but not sufficient, for determinism within a single run of a PyTorch program. Other sources of randomness like random number generators, unknown operations, or asynchronous or distributed computation may still cause nondeterministic behavior.

See the documentation for torch.set_deterministic(bool) for the list of affected operations.

• RFC | Link
• Documentation | Link

Performance & Profiling

[Beta] Stack traces added to profiler

Users can now see not only operator name/inputs in the profiler output table but also where the operator is in the code. The workflow requires very little change to take advantage of this capability. The user uses the autograd profiler as before but with optional new parameters: with_stack and group_by_stack_n. Caution: regular profiling runs should not use this feature as it adds significant overhead.

• Details | Link
• Documentation | Link

Distributed Training & RPC

[Stable] TorchElastic now bundled into PyTorch docker image

Torchelastic offers a strict superset of the current torch.distributed.launch CLI with the added features for fault-tolerance and elasticity. If the user is not be interested in fault-tolerance, they can get the exact functionality/behavior parity by setting max_restarts=0 with the added convenience of auto-assigned RANK and MASTER_ADDR|PORT (versus manually specified in torch.distributed.launch).

By bundling torchelastic in the same docker image as PyTorch, users can start experimenting with torchelastic right-away without having to separately install torchelastic. In addition to convenience, this work is a nice-to-have when adding support for elastic parameters in the existing Kubeflow’s distributed PyTorch operators.

• Usage examples and how to get started | Link

[Beta] Support for uneven dataset inputs in DDP

PyTorch 1.7 introduces a new context manager to be used in conjunction with models trained using torch.nn.parallel.DistributedDataParallel to enable training with uneven dataset size across different processes. This feature enables greater flexibility when using DDP and prevents the user from having to manually ensure dataset sizes are the same across different process. With this context manager, DDP will handle uneven dataset sizes automatically, which can prevent errors or hangs at the end of training.

• RFC | Link
• Documentation | Link

[Beta] NCCL Reliability – Async Error/Timeout Handling

In the past, NCCL training runs would hang indefinitely due to stuck collectives, leading to a very unpleasant experience for users. This feature will abort stuck collectives and throw an exception/crash the process if a potential hang is detected. When used with something like torchelastic (which can recover the training process from the last checkpoint), users can have much greater reliability for distributed training. This feature is completely opt-in and sits behind an environment variable that needs to be explicitly set in order to enable this functionality (otherwise users will see the same behavior as before).

• Documentation | Link
• RFC | Link

[Beta] TorchScript remote and rpc_sync

torch.distributed.rpc.rpc_async has been available in TorchScript in prior releases. For PyTorch 1.7, this functionality will be extended the remaining two core RPC APIs, torch.distributed.rpc.rpc_sync and torch.distributed.rpc.remote. This will complete the major RPC APIs targeted for support in TorchScript, it allows users to use the existing python RPC APIs within TorchScript (in a script function or script method, which releases the python Global Interpreter Lock) and could possibly improve application performance in multithreaded environment.

• Documentation | Link
• Usage examples | Link

[Beta] Distributed optimizer with TorchScript support

PyTorch provides a broad set of optimizers for training algorithms, and these have been used repeatedly as part of the python API. However, users often want to use multithreaded training instead of multiprocess training as it provides better resource utilization and efficiency in the context of large scale distributed training (e.g. Distributed Model Parallel) or any RPC-based training application). Users couldn’t do this with with distributed optimizer before because we need to get rid of the python Global Interpreter Lock (GIL) limitation to achieve this.

In PyTorch 1.7, we are enabling the TorchScript support in distributed optimizer to remove the GIL, and make it possible to run optimizer in multithreaded applications. The new distributed optimizer has the exact same interface as before but it automatically converts optimizers within each worker into TorchScript to make each GIL free. This is done by leveraging a functional optimizer concept and allowing the distributed optimizer to convert the computational portion of the optimizer into TorchScript. This will help use cases like distributed model parallel training and improve performance using multithreading.

Currently, the only optimizer that supports automatic conversion with TorchScript is Adagrad and all other optimizers will still work as before without TorchScript support. We are working on expanding the coverage to all PyTorch optimizers and expect more to come in future releases. The usage to enable TorchScript support is automatic and exactly the same with existing python APIs, here is an example of how to use this:

import torch.distributed.autograd as dist_autogradimport torch.distributed.rpc as rpcfrom torch import optimfrom torch.distributed.optim import DistributedOptimizer with dist_autograd.context() as context_id:  # Forward pass.  rref1 = rpc.remote(“worker1”, torch.add, args=(torch.ones(2), 3))  rref2 = rpc.remote(“worker1”, torch.add, args=(torch.ones(2), 1))  loss = rref1.to_here() + rref2.to_here()   # Backward pass.  dist_autograd.backward(context_id, [loss.sum()])   # Optimizer, pass in optim.Adagrad, DistributedOptimizer will  # automatically convert/compile it to TorchScript (GIL-free)  dist_optim = DistributedOptimizer(     optim.Adagrad,     [rref1, rref2],     lr=0.05,  )  dist_optim.step(context_id)

• Documentation | Link
• RFC | Link

[Beta] Enhancements to RPC-based Profiling

Support for using the PyTorch profiler in conjunction with the RPC framework was first introduced in PyTorch 1.6. In PyTorch 1.7, the following enhancements have been made:

• Implemented better support for profiling TorchScript functions over RPC
• Achieved parity in terms of profiler features that work with RPC
• Added support for asynchronous RPC functions on the server-side (functions decorated with rpc.functions.async_execution).

User are now able to use familiar profiling tools such as with torch.autograd.profiler.profile() and with torch.autograd.profiler.record_function, and this works transparently with the RPC framework with full feature support, profiles asynchronous functions, and TorchScript functions.

• Design doc | Link
• Usage examples | Link

[Prototype] Windows support for Distributed Training

PyTorch 1.7 brings prototype support for DistributedDataParallel and collective communications on the Windows platform. In this release, the support only covers Gloo-based ProcessGroup and FileStore.
To use this feature across multiple machines, please provide a file from a shared file system in init_process_group.

# initialize the process groupdist.init_process_group(    “gloo”,    # multi-machine example:    # Shared files need six “/”    # init_method = `”file://////{machine}/{share_folder}/file”`    # Local file need three “/”    init_method=”file:///{your local file path}”,    rank=rank,    world_size=world_size) model = DistributedDataParallel(local_model, device_ids=[rank])

• Design doc | Link
• Documentation | Link
• Acknowledgement | gunandrose4u

Mobile

PyTorch Mobile supports both iOS and Android with binary packages available in Cocoapods and JCenter respectively. You can learn more about PyTorch-Mobile here.

[Beta] PyTorch Mobile Caching allocator for performance improvements

On some mobile platforms, such as Pixel, we observed that memory is returned to the system more aggressively. This results in frequent page faults as PyTorch being a functional framework does not maintain state for the operators. Thus outputs are allocated dynamically on each execution of the op, for the most ops. To ameliorate performance penalties due to this, PyTorch 1.7 provides a simple caching allocator for CPU. The allocator caches allocations by tensor sizes and, is currently, available only via the PyTorch C++ API. The caching allocator itself is owned by client and thus the lifetime of the allocator is also maintained by client code. Such a client owned caching allocator can then be used with scoped guard, c10::WithCPUCachingAllocatorGuard, to enable the use of cached allocation within that scope.

Example usage:

#include <c10/mobile/CPUCachingAllocator.h>…..c10::CPUCachingAllocator caching_allocator;  // Owned by client code. Can be a member of some client class so as to tie the  // the lifetime of caching allocator to that of the class……{  c10::optional<c10::WithCPUCachingAllocatorGuard> caching_allocator_guard;  if (FLAGS_use_caching_allocator) {    caching_allocator_guard.emplace(&caching_allocator);  }  ….  model.forward(..);}…..

NOTE: Caching allocator is only available on mobile builds, thus the use of caching allocator outside of mobile builds won’t be effective.

• Documentation | Link
• Usage examples | Link

Backwards Incompatible changes

Python API

torch.conj now returns the input as-is for real Tensors (#43270)

Previously, torch.conj and Tensor.conj were making a clone for Tensors of real dtype. It now returns the Tensor as-is to improve performance.
You can recover the original behavior by adding a .clone() for real Tensors.
Note that this behavior is different from numpy for which np.conj returns a new ndarray and ndarray.conj returns the ndarray as-is.

torch.tensor, torch.as_tensor, and torch.sparse_coo_tensor now use the input Tensor’s device when it is not specified (#41984)

This will change the device on which the Tensor is created and so the user can start seeing device mismatch errors.
It also means for sparse Tensors that both of the provided Tensors must be on the same device if the device is not specified.
You can recover the original behavior by passing the device argument.

Improve torch.norm handling of keepdim=True (#41956)

Before this change, when calling torch.norm with keepdim=True and p=’fro’ or p=number, leaving all other optional arguments as their default values, the keepdim argument would be ignored. It is now properly respected.
Also, any time torch.norm was called with p=’nuc’ and keepdim=True, the result would have one fewer dimension than the input, and the dimensions could be out of order depending on which dimensions were being reduced. It is now properly keeping all the dimensions.
You can recover the original behavior by setting keepdim=False.
NOTE: this function is now deprecated (see below) and we recommend you use torch.linalg.norm, which follows NumPy’s conventions.

torch.split and torch.chunk: Fix view tracking for the autograd (#41567)

The autograd system is able to correctly handle modifications through views of Tensors by explicitly tracking known view operations. In prior releases, torch.split and torch.chunk were not marked as known view operations, which could lead to silently wrong gradients.

Note that since v1.5, inplace modification of views created by functions that return multiple views is deprecated. Such case is not properly handled by the autograd and can lead to internal errors or wrong gradients. So, as a side effect of this view fix, inplace modifications of the outputs of torch.split and torch.chunk will now raise a warning and can lead to internal errors or wrong gradients while they were previously silently computing wrong gradients.
If you see such a warning, you should replace the inplace operation with an out of place one.
You can recover the original behavior by using the new torch.unsafe_split and torch.unsafe_chunk. Note that these functions are only here to ease the transition and will also be removed in a future version.

torch.{argmin,argmax} now always return the first min/max index (#42004)

torch.argmin (torch.argmax) now always returns the index of the first minimum (maximum) element. This choice is consistent with NumPy. Previously if there were multiple minima (maxima) the index returned could be the index of any of them.
You cannot recover the original behavior as it was platform dependent and not guaranteed. If your code was relying on a specific index for your specific platform, you should update it to work with the first index and this new code will work on all platforms.

torch.{min,max,median}: Update backward formula when doing full reduction (dim argument not provided) (#43519)

When no dimension is specified, full reduction is performed and the gradient will now flow back evenly towards all the input that realized the output value. The old behavior was to propagate the gradient only for one of such input selected arbitrarily.
This should improve stability of training by gradient descent.
To recover the previous behavior, you can perform the reduction with the dim= argument. It will ensure that the gradient only flows back for the input whose index was returned.

nn.BCELoss size mismatch warning is now an error (#41426)

This is the end of the deprecation cycle for this op to make sure it does not have different broadcasting semantic compared to numpy’s broadcasting semantic used everywhere else in PyTorch’s codebase.
You need to make sure all inputs are the same size to avoid the error.

Custom autograd.Function stop materializing None output Tensors (#41490)

To improve performance, the custom autograd.Function will not create a Tensor full of zeros when an input is differentiable but the user’s backward function returns None for it. This means that code for which the .backward() or autograd.grad() final result will now be None while it used to be a Tensor full of zeros.
You can recover the previous behavior by having your custom autograd.Function materialize the zero Tensor with torch.zeros_like(input) to replace the None output for the backward method.

import torch # Custom Function that returns None for the gradientclass GetTwos(torch.autograd.Function):    @staticmethod    def forward(ctx, inp):        return inp.clone().fill_(2)     @staticmethod    def backward(ctx, grad_out):        # To recover the 1.6 behavior, replace the line below with `return torch.zeros_like(grad_out)`        return None a = torch.rand(10, requires_grad=True)b = GetTwos.apply(a)b.sum().backward() print(a.grad)# In PyTorch 1.6 this will print# tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])# In PyTorch 1.7 this will print# None

Fix inplace detection for non-differentiable outputs (#41269)

We fixed a bug in the inplace detection code that was preventing the detection of some inplace operations for output that are not differentiable (like integer type Tensors).
This can lead to code that used to run fine to throw the error “a Tensor that was needed for backward was modified in an inplace operation”.
Such failure is true and the user code must be fixed to compute proper gradients. In general, this involves cloning the Tensor before modifying it inplace to make sure the backward pass can happen safely.

import torch a = torch.rand(10, requires_grad=True)with torch.no_grad():    a[2] = 10 b, ind = a.max(dim=0)# ind is 2 here with torch.no_grad():    t = torch.rand(10)    t[4] = 10    res = torch.max(t, dim=0, out=(torch.Tensor(), ind))    # ind becomes 4 here # This backward runs in 1.6 but will fail in 1.7b.sum().backward()print(a.grad)# tensor([0., 0., 0., 0., 1., 0., 0., 0., 0., 0.])# The value is wrong is at index 4 while it should be at index 2 # The issue is avoided by not modifying ind inplace by replacing the line# above with:# res = torch.max(t, dim=0, out=(torch.Tensor(), ind.clone()))

Add __torch_functions__ for methods (#37091)

Functions, slicing and Tensor methods will now properly preserve the subclass type when possible.

>>> class SubTensor(torch.Tensor):…     pass>>> type(torch.add(SubTensor([0]), SubTensor([1]))).__name__’SubTensor’>>> type(torch.add(SubTensor([0]), torch.Tensor([1]))).__name__’SubTensor’

The old behavior of “any operations on your subclass produces a torch.Tensor instead of the subclass” can be recovered by doing:

from torch._C import _disabled_torch_function_impl    class SubTensor(torch.Tensor):    __torch_function__ = _disabled_torch_function_impl

tensor.__iter__: Use torch.unbind instead of a for loop (#40884)

This improves performances significantly but it changes the behavior of in-place operations on the value returned by the iterator. This happens only if either the input Tensor or any argument of the in-place operation is a Tensor that requires gradients. And it will fail with “Output X of UnbindBackward is a view and is being modified inplace”.
You can recover the previous behavior by manually slicing the Tensor: [t[i] for i in range(t.size(0))] as shown in the example below.

Updated most function that take zero, one or two Tensor arguments and indexing op to check for memory overlap in the Tensor being worked on (#43418, #43419, #43420, #43421, #43423, #43422)

It fixes silent correctness errors: something that used to be silently incorrect now errors out. Code that raises this error must be updated to avoid doing such op that was returning wrong results as shown in the example below:

>>> x = torch.randn(1, 3)>>> # Create a tensor that has internal memory overlap>>> y = x.expand(2, 3) # In 1.6, this would not error out, but in 1.7, this errors out>>> torch.nn.functional.elu(y, inplace=True)RuntimeError: unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation. # Here is the fix in 1.7>>> torch.nn.functional.elu(y, inplace=False)

c++ API: Any external users of TensorIterator now always get the memory overlap check. The previous behavior can be recovered by setting set_check_mem_overlap(false) when creating the iterator.

TorchScript

TorchScript now correctly supports various exception type and custom exception message (#41907)

• Exceptions raised in TorchScript was traditionally replaced with a generic runtime error that doesn’t carry exception type or message, leading to crashes that are difficult to pin-point and debug. We improved TorchScript to correctly parse exception types and messages and surface them to users.
• This change is backward incompatible because TorchScript now attempts to compile user code that creates custom exception messages instead of ignoring them. Any TorchScript-incompatible Python features used in those code snippets would lead to failures.
• There is no fixed formula to fix this backward incompatibility failure other than updating code that generates exceptions to be TorchScript-able.

TorchScript now supports properties of TorchScript classes and ScriptModules (#42389, #42390)

• TorchScript added support for @property of TorchScript classes and ScriptModules. Custom setters and getters are also supported. Custom deleters are not supported.
• This improvement is backward incompatible because TorchScript now attempts to script properties of existing classes and Modules. If these properties use Python or Pytorch features that are not supported in Torchscript, scripting will fail.
• There are two ways of fixing backward incompatibility failures introduced by this change. One is using @torch.jit.unused to annotate problematic properties, the other is to update the implementation of the property so that the getter and setter are scriptable.

Quantization

The convolution parameters now support versioning.

• This change means that any quantized convolution module saved using PyTorch 1.7+ cannot be loaded in v1.6 and lower.
• But this change is backward compatible: if the model (with conv layers) is saved in version 1.6, it can be safely loaded in version 1.7.

Some undocumented functions that were mistakenly made public have been removed

• torch.absolute_ has been removed, the Tensor method (Tensor.absolute_) should be used instead just like all other inplace ops.
• torch.ExtraFilesMap is an internal jit construct and should not be used.

TorchScript Compiler Update

In 1.7, we are enabling a Profiling Executor and a new Tensor-Expressions-based (TE) Fuser. All compilations will now go through one (an adjustable setting) profiling run and one optimization run. For the profiling run, complete tensor shapes are recorded and used by the new Fuser. For the optimization run, the focus is on finding (in torch.jit.ScriptModules) and fusing element-wise operations over CUDA tensors into a single CUDA kernel.

The TE fuser is expected to deliver performance similar to the old fuser used in 1.6. It however unlocks more opportunities for performance improvements in future releases. In rare cases, performance of some models may degrade 5-10%. If you experience any regressions please report it on Github, so we can address them as soon as possible! For 1.7, we are providing an option for our users to revert back to the old fuser by calling torch._C._jit_set_profiling_executor(False) in Python and torch::jit::getExecutorMode()“ = false; in C++. For more information, please see “Graph Executor” section in our documentation.

Deprecations

Python API

torch.norm and torch.functional.norm are deprecated in favor of torch.linalg.norm (#44321)

The new torch.linalg.norm has the same behavior as numpy.linalg.norm
Both deprecated functions had odd behaviors for matrix and vector norms. You should refer to the doc here to find the exact behavior they had and how to replicate it with the new API.

Deprecate fft functions in torch. namespace in favor of torch.fft. namespace (#44876)

Please use torch.fft.foo as a drop-in replacement for torch.foo for the following functions: fft, ifft, rfft and irfft.

Warns when some out= functions need to resize an output which is not 0-size (#42079)

This behavior is dangerous and leads to an API that is hard to use. It is being deprecated to be able to fix that API in future versions.
You should resize the output before-hand to avoid any issue in the future:

a = torch.rand(5)b = torch.rand(25) # This is deprecatedtorch.add(a, a, out=b) # This has the same behavior but will work in future versionstorch.add(a, a, out=b.resize_(0))

torch.optim: Warn for duplicate params in param group (#41597)

Providing multiple times the same Parameter in a single param group is most likely due to user error and is being deprecated.
Please open an issue if you have a valid use case that require this feature.

torch.linspace and torch.logspace: Not giving the step argument is deprecated (#43860)

The default steps argument that has been used historically in PyTorch is not consistent with other libraries and so is being removed to avoid confusion.
For both functions, passing steps=100 keyword argument can be used to recover the original behavior.

Distributed

• Make TensorPipe the default backend for RPC (#43246)
• Infer RPC backend type to preserve backward compatibility as we make TensorPipe the default (#45065)
• Add deprecation warning to ProcessGroup backend and make TensorPipe backend stable. (#45356)
• Add warnings on ProcessGroup and ProcessGroup::Work APIs which will be retired soon. (#46366)

New features

Python API

New namespaces:

• torch.fft added (#41911)
• torch.linalg added (#42664)
• torch.optim.functional added (#44715)

New operators:

• torch.count_nonzero added (#39992)
• nn.SiLU activation added (#41034)
• torch.logit added (#41062)
• torch.gcd, torch.lcm added (#40651, #41552, #42254)
• torch.functional.atleast_{1d/2d/3d} added (#41317)
• torch.isreal added (#41298)
• nn.Unflatten added (#41564)
• torch.movedim added (#41480)
• torch.isposinf, torch.isneginf added (#41588)
• torch.signbit added (#41589)
• torch.absolute added (#42586)
• torch.clip alias added (#42770)
• torch.quantile added (#42755)
• torch.linalg.det and torch.outer alias added (#42802)
• torch.nansum added (#38628)
• torch.hypot added (#42291)
• torch.nextafter added (#42580)
• torch.hstack, torch.vstack, torch.dstack added (#42799)
• torch.arccosh alias added (#43107)
• Tensor.movedim as a method added (#43122)
• torch.matrix_exp added (#40161)
• torch.fix alias added (#43326)
• torch.arccos, torch.arcsin, torch.arctan aliases added (#43319)
• torch.negative alias added (#43400)
• torch.maximum, torch.minimum added (#42579)
• torch.arctanh, torch.arcsinh aliases added (#43762)
• torch.linalg.norm added (#42749, #43907)
• torch.amax, torch.amin added (#43819)
• torch.heaviside added (#42523)
• torch.i0 added (#43132)
• torch.not_equal, torch.greater, torch.greater_equal, torch.less, torch.less_equal aliases added (#43870)
• torch.exp2 added (#44184)
• torch.kaiser_window added (#44271)
• torch.nanquantile added (#44393)
• torch.multiply, torch.divide aliases added (#44463)
• nn.TripletMarginWithDistanceLoss added (#43680)
• torch.fft.fft, torch.fft.ifft, torch.fft.rfft, torch.fft.irfft, torch.fft.hfft, torch.fft.ihfft added (#43011)
• torch.fft.fftn, torch.fft.ifftn, torch.fft.rfftn, torch.fft.irfftn added (#44550)
• optim.functional.adagrad added (#44715)
• optim.functional.adam added (#44791)
• torch.complex, torch.polar added (#39617)
• Tensor.__complex__ added (#43844)
• torch.vdot added (#43004)

API extension:

• torch.full added support for bool and integer dtypes (#41912)
• torch.lt and torch.masked_select added support for half dtype (#43704)
• torch.div, torch.true_divide, torch.atan2 added support for integer to float type promotion in (#42359)
• unflatten added support for non-named dimensions (#42563)
• torch.polygamma added support for n >= 2 (#42499)
• torch.qr added backward support for wide input matrices (#42216)
• nn.Linear for MKLDNN added support for no-bias (#43703)
• torch.lerp added support for half dtype (#43541)
• Updates torch.div to perform true division (end of deprecation cycle) (#42907)
• torch.scatter added support for reductions on CUDA (#41977)
• BFloat16 support type promotion (#41698, #43324)
• BFloat16 support on CUDA for torch.pow (#44760), unary ops and activations (#44813, #44824, #44834), torch.i0 (#44750), softmax (#44837), div, addcdiv, addcmul, mean, var (#44758), layernorm (#45002),all pooling layers (#44836, #45151)), torch.logspace (CPU and CUDA) (#44675), random kernels on Windows (#44918), torch.addmm, torch.addmv (#44986), loss functions (#45011), batched gemm (#45167), nccl path (#38515), binary logical operators (#42485), torch.neg (#45240), Conv (non-cuDNN) (#45007), torch.abs (#44804), torch.erfinv (#43399), comparison ops (#44748)
• torch.asin, torch.neg added support for sparse Tensors (#44028)
• torch.softmax added support for CUDA (#42307)
• Tensor.{real,imag} added setter for these attributes (#39860)
• torch.{addmm,bmm} added support for complex on CUDA (#40431, #42383, #43827)
• torch.dot added support for complex (#42745)
• torch.stft, torch.istft added support for complex (#43886)
• torch.cholesky added support for complex (#44895, #45267)
• torch.sgn added (to support complex) (#39955)
• Binary ops added support for complex (#43174)
• Add allowlist for complex backward (#45461)

Autograd

• Don’t automatically materialize output grads with zeros for autograd.Function (#41821)
• Benchmark tool for autograd.functional API (#43428)
• Added reset_grad API to remove gradient instead of setting them to zero (#44423)
• Allow Tensor-like objects in torch.autograd.gradcheck (#43877)
• Added support for nested call for @torch.no_grad() decorator (#44633)
• Added support for torch.lobpcg backward (#43002)

CUDA

• Added TF32 support (#41498)
• CUDA RTX30 series support (#45489, #45130)
  • **Note: **At the time of the 1.7 release, the currently available and stable Nvidia CUDA libraries are not fully tuned for the RTX 3080 and 3090 so users might see performance regressions.
• torch.cuda.amp.GradScaler now supports sparse gradients (#36786)
• Autocast support for cudnn RNNs (#42385)
• Support AMP in nn.parallel (#43102)
• Support for tf32 in cudnn and backends.cudnn.allow_tf32 flag to control it (#40737)
• Added torch.cuda.memory.list_gpu_processes to list running processes on a give GPU (#44616)
• Add env variable to bypass CUDACachingAllocator for debugging (#45294)
• Add non-deterministic alert to CUDA operations that use atomicAdd() (#41538)

C++ API

• nn::TransformerEncoderLayer added (#42633)
• nn::TransformerDecoderLayer added (#42717)
• nn::TransformerEncoder added (#43187)
• nn::TransformerDecoder added (#42886)
• nn::Transformer added (#44333)
• nn::Unflatten added (#42613)
• nn.ParameterList added (#41259)
• torch::cuda::manual_seed and torch::cuda::manual_seed_all added (#42638)

Mobile

• Support Tensor MemoryFormat in java wrappers (#40785)
• Add mobile_optimized boolean flag to optimized model. (#45479)

Vulkan

• Backend added (#36491, #43076)
• Add many operators adaptive_avg_pool2d (#41220), mm (#41221), reshape (#41223), max_pool2d (#41379), add_ and relu_ (#41380), cat (#41434), add and mul (#42674) and avg_pool2d (#42675).
• Model preparation via torch.utils.optimize_for_vulkan (#44903)
• Add to Java API option to load on Vulkan and test app (#44896, #44897)

Distributed

• Support alltoall collective in ProcessGroupGloo (#41424, #41690)
• Add a DDP Communication Hook providing the flexibility to completely override DDP gradient communication (#40848)
• Examples on how to use the DDP communication hook (#43310)
• Add NCCL Alltoall to NCCL process group (#42514)
• Support allgather and gather APIs for Python Objects (#42189)
• Join-based API to support uneven inputs in DDP (#42577)
• broadcast_object API for c10d (#43887)
• Async Error Handling support for ProcessGroupNCCL (#41050, #41051, #41052, #41053, #41054, #44163)
• Add a “gradient_as_bucket_view” parameter to DDP to reduce memory overhead (#44344)
• Add getNumKeys API to c10d TCPStore (#43962)
• Add DeleteKey API for c10d TCP Store (#45401)

Quantization

• New quantized ops
  • Adaptive average pooling (#40271)
  • Max pooling (#45152)
  • Embedding and EmbeddingBag quantization (8-bit + partial support for 4-bit): (#40076, #41293, #41612, #42924, #42762, #42881, #43077, #43088, #43090, #43176, #43296, #43433, #43989, #44008, #44207, #44208, #44217, #45149, #44845, #44048, #42690, #42612)
  • QNNPACK Transposed convolution2D and 3D (#39714, #40351, #40360, #40370, #40371, #44844, #45078, #45081)
  • Operations on quantized tensors
    • aten::repeat (#40644)
    • aten::apend (#40743)
    • stack (#42187)
    • fill_ (#43303)
    • clone for per channel affine quantized tensor (#44573)
    • append (graphmode) (#44641)
  • 1D batch normalization support (#42491)
  • N-Dimensional constant padding (#43304)
  • CELU operator (#39199)
• Support for FP16 quantization (#40708, #40709, #40710, #42147, #42221, #42222, #42348, #41049)
• Add Quantizer support to IValue (#42438)
• Custom module support (#44835)
• Preserving pre and post forward hooks (#37233)

Misc

• torch.set_deterministic and torch.is_deterministic: Raise error when the flag is set and a non-deterministic operation is used (#15359, #41377)
• Add CUDA 11 to nightly binaries (#44086, #43366)
• Dev Tool: Nightly checkout tool and doc in CONTRIBUTING.md (#42635, #43294)
• Website: Add docs for tagged version (include rc) on the general website (#45204)
• Build: Added BUILD_CAFFE2 flag to be able to disable caffe2 compilation (#43673)
• Dataloader: Add prefetch_factor argument to control the number of batch loaded ahead of time(#41130)
• Dataloader: Allow handling of np.memmap objects (#39847)
• ROCm: Add support torch utils.cpp_extension (#41257, #43528)
• ROCm: Enable complex BLAS (#43744)
• docker: Add torchelastic to docker image (#45438)
• docker: Add CUDA 11 support (#45071)
• docker: Use python 3.8 in pytorch docker image (#45466)

Improvements

Python API

• Use tree-based sum for floats to avoid numerical instability (#39516)
• nn.ReflectionPad: Add support for 0-dim batch sizes. (#39231)
• torch.scatter: Add reductions for CPU (#36447)
• Allow any valid ASCII python identifiers as dimnames (#40871)
• Improve Python warning prints when there is also an error (#41116)
• torch.iinfo, torch.finfo: Improve printing (#40488)
• torch.where: Add support for scalar input (#40336)
• torch.nonzero: Remove deprecation warning for as_tuple argument (#45413)
• torch.distributions.Categorical: Clamp logit to avoid -inf when calculating entropy (#41002)
• torch.futures.Future: Add done function to query the status of the future (#42013)
• torch.add: Add support for complex backward (#45839)

torch.nn

• nn.EmbeddingBag: Add support for incude_last_offset=True when reduction is mean or max (#42215)
• nn.AvgPooling{1,2,3}d: Ensure all cells are valid in ceil mode to avoid division by 0 (#41368)
• nn,[Adaptive]MaxPool{1,2,3}d: Handle edge case when input is filled with -inf (#40665)
• nn.Hardsigmoid, nn.Hardswish: Add inplace option (#42346)
• nn.MSELoss, nn.L1Loss, nn.SmoothL1Loss: Add support for target that requires gradients. (#44437, #44471, #44486)
• nn.Parameter{List,Dict}: Add warning when improperly used (with DataParallel or weight_norm) (#44405)
• nn.functional.smooth_l1: Add beta parameter (#44433)

Build

• Report error when ATEN_THEADING is OMP and USE_OPENMP is turned off. (#40146)
• Raise nice error when trying to build PyTorch on 32-bit Windows system (#40321)
• Make setup.py Python-2 syntactically correct and work for version >= 3.9 (#41960, #46388)
• Don’t proceed into setup.py too far if Python version is unsupported (#42870)

Distributed

• Support profiling rpc_async in TorchScript (#40652)
• Allow RPC to be initialized again after shutdown. (#42723)
• Support rpc_sync, rpc.remote in TorchScript (#43043, #43046)
• Make async_execution compatible with RRef helpers (#44666)
• Extend RPC profiling to support async function execution over RPC. (#44664)
• Support record_shapes in RPC profiling (#44419)
• Add variants for cuda.comm.broadcast/gather/scatter which store the result in a provided “out” parameter (#39681)
• Explicitly abort NCCL Communicators on ProcessGroupNCCL Destruction (#40585)
• Helper function to print out all DDP-relevant env vars (#41297)
• Add timeout to ProcessGroup Work Wait (#40944)
• Support Wait Timeout in ProcessGroupNCCL (#40946)
• Support work-level timeouts in ProcessGroupGloo (#40948)
• Support for torch.bool in ProcessGroupNCCL (#41959)
• DDP.train() returns self to stay consistent with nn.Module (#42131)
• Add a drop_last option in DistributedSampler to drop tail of the data to ensure data is even across ranks (#41171)
• Additional error checking for torch.cuda.nccl APIs. (#43247)
• Support work.result() to get result tensors for allreduce for Gloo, NCCL backends (#43970)
• Add a device parameter to RemoteModule (#44254)
• Add remote_parameters() API for RemoteModule. (#43906)
• Add a warning log when there is high skew of uneven inputs in DDP training (#45238)

TorchScript

• Support string concatenation (cc29c19)
• Support using Python Enum in TorchScript (#41390,#41965,#42085,#42623,#42661,#42661,#42874,#43460,#43188,#44243,#44891)
• Support sorting list of strings (#42398)
• Support boolean key in dictionary (#42833)
• Support @torch.no_grad (#41371)
• Support del to TorchScript classes (#44352)
• Speed up saving modules in case of having many classes (#44589)
• Support Python Slice class in TorchScript (#44335)
• Support sorting a list of tuples (#43448)
• Enable @torch.jit.unused syntax for ignoring properties (#45261)
• Enable ProfilingExecutor + TensorExpression (#45546) (#45546)
• Support @torch.jit.unused on a @torch.no_grad decorated function (#41496)
• Improve ModuleList indexing error msg (#43361)
• Better match behavior of loaded `ScriptModule“s vs. freshly created ones (#43298)
• Support backend-lowered submodules (#41146)
• Allow freezing of modules containing interface attribute (#41860)
• to_backend API now accepts wrapped modules (#43612)
• Allow submodule methods inference rules to be different (#43872)
• Support default values for arguments of class type methods (#45098)
• Improve sugared value’s error message when closing over global variables (#42889)
• Support backend-lowered submodules (#40841)
• Turn on non-ASCII string literals serialization (#40719)
• Better printing of Tensor stride information (#45156)

Mobile

• Allow specifying PYTHON executable to build_android (#41927)
• Include all overloads for OSS custom build (a01e91e)

Quantization

• Change the whitelist to allowlist (#41771, #41802)
• dequantize now supports list and tuple of tensors (#41079)
• User now has a way to add a activation post process hook using register_activation_post_process_hook function (#42342)
• add/mul now support different variants (#42769)
• Fake quantizer now has more info when printed (#43031)
• OP_LIST_TO_FUSER_METHOD is exposed to the user (#43286)
• quantize_jit can handle new upsample overloads (#43407)
• Setter/getter method for quantization and fusion mappings (#43990)
• fake_quant and observer can be disabled in scriptmodule (#44773)
• convert_jit can now take preserved_attrs argument (#44490)
• SyncBN: preserve qconfig if it exists (#45317)
• Add quant APIs to save/load observer state_dict (#44846)
• Add version support for the conv parameters (#43524, #43086, #43651, #44671)

ONNX

In PyTorch 1.7, we have continued to add and improve PyTorch operator export to ONNX. We have enabled export of 10 new operators, and further enhanced and optimized export of 10+ torch operators to ONNX. We have also focused on improving export of TorchScript modules, in particular laying some groundwork required for better support in near future. We have also created an API (torch.onnx.utils._find_missing_ops_onnx_export) as a diagnostic tool (preview only) to get a list of operators in a model that are not supported or implemented by ONNX exporter. Support for export of torch.quantization.FakeQuantize has also been added to help enable some QAT workflows.

• Add support to export more torch ops torch.view_as (#40496), fake quantize functions (#39738), embedding_bag (#41234, #44693), torch.eye (#41357), Tensor.as_strided (#41569), torch.tensor (#41872), addition between list of tensors (#41888), Tensor.__floordiv__ (#43022), torch.nn.KLDivLoss (#41858), Tensor.new_empty and Tensor.new_zeros (#43506)
• Improves existing export logic and optimizing exported ONNX graph
  • Add warning in ONNX export when constant folding is on in training-amenable mode (#40546)
  • Fix export of torch.full_like (#40063)
  • Add pass that fuses Conv and BatchNormalization (#40547)
  • torch.where export, add support for ByteTensor (#42264)
  • Fix scalar type cast for comparison ops (#37787)
  • torch.scatter export, add support for src being scalar or different dtype (#42765, #43440)
  • Fix Squeeze operator when applied to a dimension with shape > 1 (#38476)
  • Extend support for torch.where (#41544)
  • Update ops torch.slice (#42935), torch.split (#43670), torch.repeat (#43430), torch.arange (#43777), len (#43824), torch.narrow (#44039), flatten (#40418), adaptive_pool (#46100)
• Update export to follow pytorch changes
  • Update div export to perform true divide (#44831)
  • Enable true_divide scripting export with ONNX shape inference (#43991)

Misc

• torch.utils.collect_env: Collect more informations (python 32/64bit, clang version, CPU architecture, ROCm version) (#42887, #42961, #44106)
• torch.hub.load_local: Allow to load models from any local directory (#44204)
• Add warning if import torch is called from the source root (#39995)
• Improve Dynamic Library loading for Windows (#40365)
• serialization: validate sparse tensors after loading (#34059)
• Add –continue-through-error option to run_test.sh script (#41136)
• Tensorboard: Support custom run_name and “hparam_domain_discreteinadd_hparams` (#40660, #40720)
• MKLDNN: Enable conv3d, batchnorm3d, max_pool3d and avg_pool3d (#40691, #40995, #40996)
• Profiler: Do not record zero duration kernel events (#41540)
• Profiler: Improve cuda time counting (#45209)
• Profiler: Adding with_source parameter to enable tracking source code (#43898)
• Optim: Add verbose param for all schedulers (#41580)
• Pruning: check attributes before deleting (#41913)
• Autograd: In zero_grad, avoid using inpalce detach when it is not required (#41283)
• Autograd: Update the torch.div backward formula to improve numerical stability (#43627)
• Autograd: Print all traceback for higher order backwards in detect_anomaly (#43626)
• Autograd: Stop saving input of torch.repeat as only input.dim() is needed in backward (#40766)
• CUDA: Improve cuDNN error messages to include call parameters (#45023)
• CUDA: Improve device_count and cuda init error detection and messages (#42249)
• Improve Tensor layout propagation for pointwise ops to follow input layout more closely (#42922)
• Remove blacklist/whitelist references (#41447, #41644, #41636, #41777, #41822, #41691, #41789, #41979, #41627, #42011, #41796, #42067, #42091, #42097, #42071, #42089, #42279, #42047, #42088, #45260)

Python Type Annotations

• Update some types in top level torch/*.py (#40235, #40873)
• Added typing for Tensor attributes and methods: T and grad_fn (#40879), Tensor._version (#41125), ndim (#42909), nonzero (#43053), #40499)
• Added typing for torch.serialization (#40862)
• Added typing for torch.tensor (#45077)
• Added typing for torch.Size (#40879)
• Added typing for torch.futures (#41675)
• Added typing for torch.random (#42234)
• Added typing for torch.hub (#42252)
• Added typing for collect_env.py (#43062)
• Added typing for torch.utils (#39392, #42647, #42711, #42960, #43806, #44136, #44216)
• Added typing for torch.nn (#43044, #44093, #43080, #42231, #40669)
• Added typing for torch.sparse (#43108)
• Added typing for torch.cuda.nvtx (#43443)
• Added typing for torch.cuda.memory (#43444)
• Added typing for torch.functional (#43446)
• Added typing for torch.autograd (#44451, #46206)
• Added typing for torch.quantization.fuse_modules (#43786)
• Added typing for torch.nn.quantized (#43186, #44154, #43110)
• Added typing for torch.testing._internal submodules (#44575, #44805, #44832, #44911, #44927, #44985, #44971, #45107, #45368, #45375)
• Added typing for torch.backends.quantized (#44794)
• Added typing for torch.backends.cuda (#44916)
• Added typing for torch.cuda.{comm,nccl,amp} (#45350, #45344, #45480)
• Added typing for torch.quasirandom (#45434)
• Fix typing for jit.trace and onnx.export (#41093)
• Fix typing for torch/optim/lr_scheduler.pyi (#41775, #41866)

Bug fixes

Python API

• torch.linspace: Fix step computation for large integral types (#40132)
• torch.pca_lowrank: Fix un-expected memory consumption (#40853)
• torch.linspace: Fix behavior for non-contiguous inputs on CPU (#41286)
• torch.div: Fix division by low precision scalar (#41446)
• torch.expm1: disable mkl as it produces wrong values in some cases (#41654)
• torch.utils.data.RandomSampler: Stop generating samples one at a time when replacement=True (#41682)
• torch.nn.functional.grid_sample: Fix 64-bit indexing (#41923)
• torch.nn.functional.grid_sample: Fix crash when grid has NaNs (#42703)
• torch.det: Fix on CPU (#35136)
• torch.interpolate: Avoid zero division in cubic mode (#42093)
• torch.fmod: Fix to work with zero divisors consistently (#41948)
• torch.masked_select: Fix for discontiguous outputs (#41841)
• torch.cummin, torch.cummax: Fix for discontiguous inputs/outputs (#42507)
• torch.einsum: Fix for discontiguous inputs (#42425)
• torch.orgqr: Fix input size conditions (#42825)
• torch.manual_seed: Fix argument unpacking (#42206)
• torch.searchsorted: Properly mark output as non differentiable (#42933)
• torch.bucketize: Properly mark output as non differentiable (#44102)
• torch.addmm: Properly raise error on device mismatch (#43505)
• torch.chain_matmul: Properly handle empty args (#43553)
• torch.multinomial: Properly handle 0 size dim (#43775)
• torch.cholesky_solve: Fix broadcast and error checking (#43137)
• torch.movedim: Fix uniqueness check (#44307)
• torch.min, torch.max, torch.mean: Properly throw error if dim is repeated (#44281)
• torch.lerp: Fix for discontiguous outputs on CUDA (#44559)
• torch.addmv, torch.mv: Fix beta=0 case in slow path (#44681)
• torch.triangular_solve: Fix error check on CPU (#44720)
• torch.empty_like, torch.zeros_like: Properly raise error if any memory format is provided with sparse input (#44058)
• torch.atan2: Fix type promotion (#43466)
• torch.repeat: Fix backward for 0 size repeats (#45212)
• torch.min, torch.max, torch.median: Fix handling of nan in backward (#45280)
• torch.rdiv: Properly make it consistent with div (#45407)
• torch.std: Fix hanling of nan in backward (#45468)
• torch.distributions.Binomial: Fix CUDA sampling at extreme points (#42702)
• torch.dot, torch.vdot: Add complex support (#45074)
• torch.pow: Fix when scalar base is complex (#45259)
• torch.round, torch.abs_: Disable complex inputs (#45330)
• torch.svd: Fix memory corruption for complex inputs (#45486)
• torch.view_as_complex: Fix zero dimensional input (#44175)
• torch.kthvalue: Fix for non-contiguous input (#46177)
• torch.save: Fix python binding that could lead to out of bound read (#46207)

Torch.nn

• nn.ModuleDict: Fix input dict key ordering (#40905)
• nn.LayerNorm: Fix handling of gamma in the backward when create_graph=True (#41595)
• nn.functional.{max,avg}_pool{1,2,3}d: Raise RuntimeError for zero stride (#41819)
• nn.Module: Fix missing attribute when loading model from older version (#42290)
• nn.Embedding: Raise proper error for 0-D weight (#42550)
• nn.SyncBatchNorm: Fix forward pass for non-default process group (#43861)
• nn.functional.embedding_bag: Fix for non-contiguous weight (#44032)
• nn.functional.upsample: Add nondeterministic checks (df6ea62)
• nn.GroupNorm: Fix bug when input does not require_grad on CUDA (#44863)
• functional.{l1_loss,smoothl1_loss,mse_loss}: Properly check that reduction strings are valid (#43527)
• functional.smoothl1_loss: Properly raise error for negative beta values (#45759)
• functional.pad: Fix extra memory allocation and invalid result for negative or zero pad when using circular padding (#39273)

C++ API

• nn::MultiheadAttention: Ensure all parameters are properly registered (#42037)
• Tensor::grad: Fix the thread safety issues (#40887)
• Tensor::var: Ensure that var(0) does not call the var(bool keepdim) overload but var(int dim) (#40451)

Distributed

• Fix RPC and ProcessGroup GIL deadlock (#45088)
• Relax size check in flatten_for_scatter_gather (#40573)
• BAND, BOR and BXOR for NCCL all_reduce should throw runtime errors (#42669)
• Disallow creation of ProcessGroupNCCL without GPUs (#45642)
• Fix read/write of bulk data (#42504)
• Fix thread safety issue with distributed optimizers and TorchScript (#46071)

TorchScript

• Fix type annotations in select assignments (#40528)
• Fix compilation issues with GCC-5.4 (#41055, #41063, #43223)
• Fix JIT not round to even if constant is folded (#40897)
• Fix torch.jit.freeze import (#42319)
• Fix List[str].index (#40348)
• Fix torch.jit.is_tracing() so that it is correctly called rather than returning the method itself (#42486)
• Fix Str -> Device implicit conversions (#43213)
• Fix NaN propagation in fuser’s min/max implementation (#43590)
• Cast return values of functions returning Any (#42259)
• Fix NaN propagation in TensorExpression fuser’s min/max implementation (#43609)
• Fix segfault in attribute lookup on loaded ScriptModules (#43284)
• Fix casting of unsigned char, and abs(int) (#44157)
• Fix frac in CUDA fuser (#44152)
• Fix model_name not logged properly issue. (#45488)
• Fix len, contains, getitem inherited from interface class derived from nn container (#40789)
• Fix support for FP16 in CudaCodgen (#44209)
• Fix torch.tensor for empty multidimensional-typed lists (#44652)
• Fix freeze_module pass for sharedtype (#42457)
• Correctly clone schema in insert_observers (#40624)
• Fix value association with dictionaries in the tracer (#40885)
• Fix preserve submodule attribute in freezing (#45143)
• Fix Half conversion of immediates in NNC Cuda backend (#45213)
• Fix a bug in SplitWithMask when splitting multiple times (#45141)
• Fix inlining interface call in fork subgraph (#43790)
• Fix operator order in combineMultilane in TensorExpr fuser(#45157)
• Correctly mark Tensor types inferred from empty annotation as inferred=True (#45360)
• Fix some bugs in Round+Mod simplification in NNC (#42934)
• Fix set_grad_enabled scripted version (#46060)
• Fix for dict.update() scripted version (#46105)
• Fix segfault when scripting nested classes (#46422)
• Fix memory leak in Profiling Mode (#46621)

Quantization

• Resolved namespace conflict in qnnpack for init_win symbol (a7e09b8)
• Fix linking of qnnpack params on windows. (#40920)
• Adding zero point type check for per channel quantization (#40811)
• Remove activation_post_process in qat modules (#42343) (#43015)
• qlinear_dynamic: Fix ASAN error in QNNPACK’s integration. (#41967)
• Change quantizer to account for input tensor’s memory format. (#42178)
• Fixing the output shape for the linear (#44513)
• Ensure observers and fq modules are scriptable (#44749)
• histogram observer: ensure buffer shape consistency (#44956)
• Attach qconfig to all modules (#42576)
• Fix qnnpack quantized activations for NHWC memory format (#46217)

ONNX

• Fix crash when exporting a model with nn.Sequential (#19227)
• Fix default ignore_index for nll loss (#44816)
• Rename Black to Block for various files (#42913)
• Fix bug in onnx::SsaRewrite (#42148)

Misc

• Fix torch.hub for new zipfile format. (#42333)
• Preserve python backtrace in autograd engine errors. (#43684)
• optim.SparseAdam: Fix check that params are dense on init (#43668)
• Fix clang build (#44934)
• nn::MultiheadAttention: Fix parameter registration (#42037)
• MaxPool2D: Fix memory leak for XNNPACK (#41874)
• Numpy scalar detection for bool and complex types fixed (#43644)
• Add missing file to BUILD.bazel (#40536)
• autograd.gradcheck: Add support for complex (#43208)
• Fix bug in mobile-specific CPU caching allocator (#43719)

Performance

Python API

• torch.{view_as_complex,view_as_real}: Remove unnecessary temporary Tensor (#44908)
• tensorboard.SummaryWriter.add_audio: Remove unnecessary for loops (#44201)
• Conv2d and Conv3d: bypass the im2col for 1×1 conv (#40324)
• Fix max_pool2d perf regression (#41174)
• Disable the mkldnn for conv2d in some special cases (#40610)
• addmm: Reduce constant time overhead (#41374)
• cumsum, cumprod: Enable non-synchronizing cub scan for cum* operations (#42036)
• max_pool2d: CUDA NCHW performance improvement (#42182)
• arenge: Vectorize CPU implementation (#38697)
• istft: optimize by using col2im (#42826)
• LayerNorm: improved performance on CPU both forward and backward (#35750)
• silu: improved performance (#42976)
• addmv: improved performance for zero sized input cases (#41824)
• Mobile: Simple caching allocator for CPU (#42006)
• MaxPool1d: improved performance for cases without indices (#43745)
• adaptive_avg_pool2d: optimized code path for cases when output size is (1, 1) (#44211)
• Vectorized complex copy (#44722)
• cat: optimized cuda kernel (#44833)
• Vectorized int8_t on CPU (#44759)
• Vectorized bitwise_not (#45103)
• Added stateful XNNPack deconvolution2d operator to torch (#43233)
• Enabled mkldnn dilation convolution (#40483)

Distributed

• Skip allreducing local_used_maps_dev_ when find_unused_param=False in DDP to improve performance (#40407)
• Remove unnecessary copies in ProcessGroupGloo for multiple inputs allreduce (#43543)
• Add option to run NCCL operations on high priority cuda stream (#43796)
• Enhance DistributedOptimizer to be functional and torchscriptable to avoid GIL and global lock (#45221)

TorchScript

• JIT pass for add relu fusion. (#39343)
• Optimize autodiff subgraph slicing (#41437)
• Don’t re-run CSE on every block (#41479)
• Add loop unroll optimization in NNC (#42465)
• Speed up CUDA kernel launch when block/thread extents are statically known (#42899)
• Support merging adjacent fusion groups in TensorExpression Fuser. (#43671)
• Add passes to profiling executor pipeline (#43636)
• Improve performance of KernelSumMultipleAxes (#43905)
• Latency improvements for pointwise + reduction fusion (#45218)
• Add simplification of Loop + Condition patterns in NNC (#44764)
• Fix fallback graph in specialize autogradzero (#44654)
• Fix masking for all block and thread dimensions in CudaCodeGen (#44733)
• Improve performance of simple reduction and softmax in nvFuser (#40864)
• Add a new optimization pass, the Registerizer, which looks for common Stores and Loads to a single item in a buffer and replaces them with a local temporary scalar which is cheaper to write. (#42606)
• Fuse identical conditions in NNC simplifier (#44886)
• Add _out variants and reuse memory in static runtime(#44128)

Mobile

• Add add_relu fusion pass to optimize_for_mobile. (#40252)
• optimize_for_mobile: bring packed params to root module (#42740)
• Apply selective build on RNN operators (#44132)
• Add neon backend for vectorization (#39341)

Quantization

• Use the _min_max function instead of two separate calls for min and max(#41570, #42957, #44537)
• Improve performance of the QNNPACK kernels (#41342, #42007, #42008)
• Speed up HistogramObserver by vectorizing critical path (#41041)
• Speed up AdaptivePool3d by checking if input is ChannelsLast or ChannelsLast3d (#42780)
• observers: use clamp instead of min/max in calculate_qparams (#43150)
• observers: use torch.all to check for valid min and max values (#43151)
• Avoid resizing in MinMaxObserver (#43789)
• observers: make eps a buffer (#43149)

Misc

• ROCm: Fix performance issues with torch.cat (#46323)

Documentation

Python API

• Numerous typo and grammatical improvements (#39854, #40217, #40285, #40544, #40692, #40617, #41025, #41031, #40984, #41066, #41203, #41263, #41384, #41526, #41563, #41632, #41643, #41599, #41799, #41679, #41835, #41851, #41963, #42016, #42076, #41946, #42046, #42065, #42236, #42184, #42734, #42923, #42891, #43063, #43131, #43395, #43588, #43583, #43697, #43779, #43569, #43893, #43695, #43973, #44667, #44753, #44740, #45045, #45192, #43308, #40334)
• Remove use of term “blacklist” (#41450)
• Add overflow notice for cuFFT on half precision (#40551)
• Add complex Note (#41012, #41252, #40450)
• Add documentation about data sharing for Tensors during serialization (#40412)
• Add nn.Module.training to docs (#40923)
• nn.CrossEntropyLoss: Clarify that the mean argument is weighted (#40991)
• torch.scatter_: Update doc with support for reduction methods. (#40962)
• Fix HTTP links in documentation to HTTPS (#40878)
• Fix warnings when building docs (#41068, #41334, #41335, #44686)
• Add PyTorch Glossary (#40639)
• Fix documentation references following page split (#39086)
• Update serialization note to explain versioned symbols and dynamic versioning (#41395)
• Make elementwise comparison docs more consistent (#41626)
• Update CONTRIBUTING.md to explain how to use ccache (#41619)
• Add doc warning for LSTM non-deterministic behavior (#40893)
• Document default dim for cross being None (#41850)
• Clarify Python 3.6 is the minimum supported version in the installation section. (#41937)
• Split quantization subsection into smaller pages (#41321)
• Documentation for torch.optim.swa_utils (#41228)
• Improve the documentation of DistributedDataParallel (#42471)
• Update docs about CUDA stream priority (#41364)
• Update the documentation for torch.scatter to include streams parameter. (#42814)
• Update Tensor.clone doc (#42931, #43098)
• Update external links in the README.md (#43100)
• Update torch.Tensor.is_set_to documentation (#43052)
• Polish the nightly pull docs in CONTRIBUTING (#43494)
• Update the torch.qr documentation to include a warning about when the QR.backward is well-defined. (#43547)
• Update the instructions to build from source on windows (#43479, #45553)
• Document the beta=0 behavior of BLAS functions (#43823)
• Fix docs for kwargs-only functions (#43586, #43589)
• Documents torch.sub properly, adds torch.subtract alias (#43850)
• Update determinism documentation (#41692)
• Update instructions to build (#42850)
• Clarify nn.Batchnorm track_running_stats docs (#44445)
• Fix latex error in torch.heaviside docs (#44481)
• Update torch.median doc to explain returned value for even-sized input (#44562)
• Fix the nn.ELU formula in the docs (#43764)
• torch.min, torch.max: remove incorrect warning from docs (#44615)
• Reference torch.cuda.amp tutorial from core amp docs (#44725)
• Mention TF32 on related docs (#44690)
• Clarify that 5-D ‘bilinear’ grid_sample is actually trilinear (#45090)
• Update linalg warning + docs (#45415)
• Update torch.floor_divide documentation to clarify it’s actually torch.trunc_divide (#45411)
• Update torch.fft doc and make warning clearer (#45409)
• Update for complex autograd (#45270, #46281)
• Update nn.Flatten docs (#42084)

Distributed

• Add a CONTRIBUTING.md for the distributed package. (#44224)
• Added docs for Store API (#45543)
• Add all_gather_object and gather_object documentation (#43772)

TorchScript

• Fix torch.jit.trace_module documentation (#40248)
• Fix the docs for the inputs arg of torch.jit.trace_module (#41586)
• Add documentation for PYTORCH_JIT_TYPE_VERBOSITY (#42241)
• Grammatical corrections in JIT overview (#43473)
• Update docs for recently added JIT features, including Enum Support, torch.no_grad etc. (#45232)
• Add function signature for pixel_shuffle (#45661)
• Fix signature for torch.poisson in documentation (#45656)

Mobile

• Aar native linking add fbjni (#40578)
• fix scripts (#44464)
• [PyTorch Mobile] Move some string ops to register_prim_ops.cpp and make them selective (#44500)

Quantization

• Fix several quantization documentation typos (#40567, #43693)
• API summary section (#45848)
• Documentation for dynamically quantized RNN cells (#40896)

Misc

• Update ONNX docs for release (#45086)

This release has 2 assets:

• Source code (zip)
• Source code (tar.gz)

Visit the release page to download them.

Interested in learning more about best practices and how to avoid common pitfalls when implementing deep learning?
Download our eBook: Getting Started in Deep Learning

The post PyTorch 1.7.0 Now Available appeared first on Exxact.

October 27, 2020 09:30 PM UTC

PyCoder’s Weekly

Issue #444 (Oct. 27, 2020)

#444 – OCTOBER 27, 2020
View in Browser »

Pinpoint Hard-To-Reproduce Problems in Your Production Code Without Affecting Your App Performance.

Datadog’s Continuous Profiler is an always-on, production code profiler that enables you to analyze code-level performance across your entire environment, with minimal overhead. Profiles reveal which functions (or lines of code) consume the most resources, such as CPU and memory. Try it yourself →
DATADOG sponsor

Discussions

Python Jobs

Senior Full Stack Developer (Chicago, IL, USA)

Panopta

Senior Software Engineer (Remote)

Silicon Therapeutics

Senior Research Programmer (Remote)

Silicon Therapeutics

Business Analyst/Data Analyst (South San Francisco, CA, USA)

Projas Technologies, LLC

More Python Jobs >>>

Articles & Tutorials

Python Developers Are in Demand on Vettery

Get discovered by top companies using Vettery to actively grow their tech teams with Python developers (like you). Here’s how it works: create a profile, name your salary, and connect with hiring managers at startups to Fortune 500 companies. Sign up today - it’s completely free for job-seekers →
VETTERY sponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #444.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

October 27, 2020 07:30 PM UTC

Python Morsels

Equality vs Identity

Watch First:

• Variables are Pointers
• The 2 Types of "Change" in Python

Transcript

You're probably already familiar with equality: that's the == operator. identity uses the is operator.

Equality

Let's say we have two variables, x and y pointing to two lists:

>>> x = [1, 2, 3]
>>> y = [1, 2, 3]

If we say x == y we're asking about equality:

>>> x == y
True

Equality is about whether these two objects represent the same value, whether they're essentially the same thing.

The == operator delegates to one of these objects and asks it, "do you represent the same value as the other object?" That's up to the actual objects to answer.

Identity

The is operator asks about identity.

>>> x is y
False

Unlike ==, the is operator doesn't even look at the objects that x and y point to. Instead, is asks if the variables x and y are pointing to the same object.

The is operator answers the question do two object references actually point to the same object? The expression x is y checks the memory location that x and y are pointing to (by their id) and checks to see if those locations are the same.

>>> id(x)
139957046343296
>>> id(y)
139957046343488

If x and y have the same id in memory, that means we're referencing the same object in two places: x and y are actually referring to the same exact object.

In Python, assignment points a variable to an object.

If we assign x to y:

>>> x = y

We're now pointing the variable x to the same object that y is currently pointing to.

>>> id(x)
139957046343488
>>> id(y)
139957046343488

If we call the append method on the list that x points to

x.append(4)

We've mutated that list object, but we've also mutated the object that y points to because both x and y point to exactly the same object.

>>> x
[1, 2, 3, 4]
>>> y
[1, 2, 3, 4]

These two objects are equal:

>>> x == y
True

But they're also identical (which means they point to the same object):

>>> x is y
True

Inequality and Non-identity

Just as we have equality (==) and inequality (!=):

>>> x == y
True
>>> x != y
False

we also have is for identity (is) and unidentity (is not)... or is it inidentity. How about non-identity.

>>> x is y
True
>>> x is not y
False

The is not operator is one of the few operators in Python that actually has a space inside it.

When is identity used?

You really don't see identity used very often. This is really the most important takeaway about identity and equality: you'll use == all the time, but you'll almost never use is. When comparing two objects, you'll almost always want to check for equality instead of identity.

The place you'll most commonly see is usedis with None:

>>> x is None
False
>>> x is not None
True

, there are other places you might see it used, the one place you'll most commonly see it used

There's only one None value in memory in Python. We're asking the question "is x pointing to the one and only None value".

You'll only see is used with special values where there's only one of them in memory. I call these sentinel values. Sentinel objects are considered completely unique. There's only one of them floating in the memory.

Sentinel values are pretty much the one place you'll see identity used and None is by far the most common sentinel value in Python.

PEP 8, the Python style guide, says you should use identity to compare with None. So you should never x == None, but instead type x is None. That's the convention that we use with None in Python.

Summary

When you want to ask the question "does one object represent the same data as another object", you pretty much always want to use equality (with the == or != operators).

You'll almost never need to ask the question "is one pointer referencing literally the same object as another pointer". If you do want to ask that question though, you'll check identity (with the is and is not operators).

The one time that you really should rely on identity is when comparing to None (checking x is None).

October 27, 2020 03:00 PM UTC

Real Python

Creating a Binary Search in Python

Binary search is a classic algorithm in computer science. It often comes up in programming contests and technical interviews. Implementing binary search turns out to be a challenging task, even when you understand the concept. Unless you’re curious or have a specific assignment, you should always leverage existing libraries to do a binary search in Python or any other language.

In this course, you’ll learn how to:

• Use the bisect module to do a binary search in Python
• Implement a binary search in Python both recursively and iteratively
• Recognize and fix defects in a binary search Python implementation
• Analyze the time-space complexity of the binary search algorithm
• Search even faster than binary search

This course assumes you’re a student or an intermediate programmer with an interest in algorithms and data structures. At the very least, you should be familiar with Python’s built-in data types, such as lists and tuples. In addition, some familiarity with recursion, classes, data classes, and lambdas will help you better understand the concepts you’ll see in this course.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 27, 2020 02:00 PM UTC

Stack Abuse

How to Set Axis Range (xlim, ylim) in Matplotlib

Introduction

Matplotlib is one of the most widely used data visualization libraries in Python. Much of Matplotlib's popularity comes from its customization options - you can tweak just about any element from its hierarchy of objects.

In this tutorial, we'll take a look at how to set the axis range (xlim, ylim) in Matplotlib, to truncate or expand the view to specific limits.

Creating a Plot

Let's first create a simple plot:

import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(figsize=(12, 6))

x = np.arange(0, 10, 0.1)
y = np.sin(x)
z = np.cos(x)

ax.plot(y, color='blue', label='Sine wave')
ax.plot(z, color='black', label='Cosine wave')

plt.show()

Here, we've plotted two sine functions, starting at 0 and ending at 100 with a step of 0.1. Running this code yields:

Now, we can tweak the range of this axis, which currently goes from 0 to 100.

Setting Axis Range in Matplotlib

Now, if we'd like to truncate that view, into a smaller one or even a larger one, we can tweak the X and Y limits. These can be accessed either through the PyPlot instance, or the Axes instance.

How to Set X-Limit (xlim) in Matplotlib

Let's first set the X-limit, using both the PyPlot and Axes instances. Both of these methods accept a tuple - the left and right limits. So, for example, if we wanted to truncate the view to only show the data in the range of 25-50 on the X-axis, we'd use xlim([25, 50]):

fig, ax = plt.subplots(figsize=(12, 6))

x = np.arange(0, 10, 0.1)
y = np.sin(x)
z = np.cos(x)

ax.plot(y, color='blue', label='Sine wave')
ax.plot(z, color='black', label='Cosine wave')

plt.xlim([25, 50])

This limits the view on the X-axis to the data between 25 and 50 and results in:

This same effect can be achieved by setting these via the ax object. This way, if we have multiple Axes, we can set the limit for them separately:

import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure(figsize=(12, 6))

x = np.arange(0, 10, 0.1)
y = np.sin(x)
z = np.cos(x)

ax = fig.add_subplot(121)
ax2 = fig.add_subplot(122)

ax.set_title('Full view')
ax.plot(y, color='blue', label='Sine wave')
ax.plot(z, color='black', label='Cosine wave')

ax2.set_title('Truncated view')
ax2.plot(y, color='blue', label='Sine wave')
ax2.plot(z, color='black', label='Cosine wave')

ax2.set_xlim([25, 50])

plt.show()

How to Set Y-Limit (ylim) in Matplotlib

Now, let's set the Y-limit. This can be achieved with the same two approaches:

ax.plot(y, color='blue', label='Sine wave')
ax.plot(z, color='black', label='Cosine wave')

plt.ylim([-1, 0])

Or:

ax.plot(y, color='blue', label='Sine wave')
ax.plot(z, color='black', label='Cosine wave')

ax.set_ylim([-1, 0])

Both of which result in:

Conclusion

In this tutorial, we've gone over how to set the axis range (i.e. the X and Y limits) using Matplotlib in Python.

If you're interested in Data Visualization and don't know where to start, make sure to check out our book on Data Visualization in Python.

Data Visualization in Python, a book for beginner to intermediate Python developers, will guide you through simple data manipulation with Pandas, cover core plotting libraries like Matplotlib and Seaborn, and show you how to take advantage of declarative and experimental libraries like Altair.

Data Visualization in Python

Understand your data better with visualizations! With over 275+ pages, you'll learn the ins and outs of visualizing data in Python with popular libraries like Matplotlib, Seaborn, Bokeh, and more.

October 27, 2020 01:15 PM UTC

Artem Golubin

On code isolation in Python

I started learning Python in 2009, and I had a pretty challenging task and somewhat unusual use of Python. I was working on a desktop application that used PyQT for GUI and Python as the main language.

To hide the code, I embedded Python interpreter into a standalone Windows executable. There are a lot of solutions to do so (e.g. pyinstaller, pyexe), and they all work similarly. They compile your Python scripts to bytecode files and bundle them with an interpreter into an executable. Compiling scripts down to bytecode makes it harder for people with bad intentions to get the source code and crack or hack your software. Bytecode has to be extracted from the executable and decompiled. It can also produce obfuscated code that is much harder to understand.

October 27, 2020 12:54 PM UTC

Python Software Foundation

Python Software Foundation Fellow Members for Q3 2020

It's that time of year! Let us welcome the new PSF Fellows for Q3! The following people continue to do amazing things for the Python community:

Débora Azevedo

Twitter, Website

Ines Montani

Twitter, GitHub, Website

John Roa

Karolina Ladino

Website, Twitter

Katia Lira

Twitter

Mariatta Wijaya

Twitter, GitHub Sponsor, GitHub, LinkedIn

Melissa Weber Mendonça

GitHub, Twitter

Ng Swee Meng

LinkedIn, GitHub, Twitter, Instagram

Nilo Ney Coutinho Menezes

GitHub, Blog, Twitter, Website

Park Hyun-woo

GitHub, Twitter

Ram Rachum

GitHub, Blog

Sebastian Vetter

LinkedIn, Website

Thank you for your continued contributions. We have added you to our Fellow roster online.

The above members help support the Python ecosystem by contributing to CPython, contributing to the PyLadies community, maintaining Python libraries, creating educational material, translating courses, organizing Python events and conferences, starting Python communities in local regions, and overall being great mentors in our community. Each of them continues to help make Python more accessible around the world. To learn more about the new Fellow members, check out their links above.

Let's continue to recognize Pythonistas all over the world for their impact on our community. The criteria for Fellow members is available online: https://www.python.org/psf/fellows/. If you would like to nominate someone to be a PSF Fellow, please send a description of their Python accomplishments and their email address to psf-fellow at python.org. We are accepting nominations for quarter 4 through November 20, 2020.

Work Group Needs Members

The Fellow Work Group is looking for more members from all around the world! If you are a PSF Fellow and would like to help review nominations, please email us at psf-fellow at python.org. More information is available at: https://www.python.org/psf/fellows/.

October 27, 2020 10:47 AM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Japan
  • Planet Python Brasil
  • Planet Python Indonesia
  • Planet Python Poland
• Python Libraries
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Coder's Cat"
  • "CodersLegacy"
  • "Control F'd"
  • "Fredrik Håård's Blaag"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • 4teamwork
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • AWeber Engineering
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Ahmed Bouchefra
  • Al-Ahmadgaid Asaad
  • Albert Hopkins
  • Alec Munro
  • Alex Clark
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amit Saha
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anastasios Hatzis
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Ankit Chandawala
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • AppNeta Blog
  • Armin Ronacher
  • Arnav Khare
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Atul Varma -- Toolness
  • Audrey Roy Greenfeld
  • Automating OSINT
  • Awesome Python Applications
  • Baiju Muthukadan
  • Bajusz Tamás
  • Balthazar Rouberol
  • BangPypers
  • BeDjango
  • Ben Bass
  • Ben Rousch
  • Ben Tappin
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • Bill Ward / AdminTome
  • BioPython News
  • Bit of Cheese
  • Blue Yonder Tech
  • BlueBream
  • Bojan Mihelac
  • Brad Lucas
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Harring
  • Brian Okken
  • Bruno Oliveira
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Carlos de la Guardia
  • Casey Duncan
  • Catalin George Festila
  • Catherine Devlin
  • Checking and Sharing
  • Ching-Hwa Yu
  • Chris Hager
  • Chris Leary
  • Chris McAvoy
  • Chris McDonough
  • Chris Miles
  • Chris Miller
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Barra
  • Christian Heimes
  • Christian Scholz
  • Christoph Zwerschke
  • Christopher Lenz
  • Chuck Thier
  • ClusterHQ
  • Cobe.io
  • CodeGrades
  • CodeSnipers
  • Codementor
  • Coding Diet
  • Collin Winter
  • Continuum Analytics Blog
  • Continuum Analytics News
  • Continuum Blog
  • Core Software
  • Corey Gallon
  • Corey Goldberg
  • Corey Oordt
  • Cormoran Project
  • Craig Kerstiens
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dalius Dobravolskas
  • Dallas Fort Worth Pythoneers
  • Damián Avila
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Dariusz Suchojad
  • Data Community DC
  • Data School
  • DataCamp
  • Dataquest
  • Dave Beazley
  • David Caron
  • David Goodger
  • David Grant
  • David J C Beach
  • David MacIver
  • David Malcolm
  • David Marte
  • David Stanek
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • DeGizmo
  • Denis Kurov
  • Derrick Petzold
  • Dethe Elza
  • Diego Garcia
  • Django Weblog
  • Django Weekly
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Doug Latornell
  • Dougal Matthews
  • Douglas Napoleone
  • DreamHost
  • Duncan McGreggor
  • Dusty Phillips
  • EasyGUI
  • Ed Crewe
  • Ed Taekema
  • Edward K. Ream
  • Eko S. Wibowo
  • Eli Bendersky
  • Eniram Ltd.
  • Enthought
  • Eray Özkural (examachine)
  • Eric Holscher
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Even Wiik Thomassen
  • Evennia
  • Everyday Superpowers
  • Experienced Django
  • Exxact Corp
  • Fabio Zadrozny
  • Fazal Majid
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Francisco Souza
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • Fredrik Lundh
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Gabriel Pettier
  • Galvanize
  • Gary Bernhardt
  • Gary Wilson
  • Geek Scrap
  • Georg Brandl
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • GoDjango
  • Gocept Weblog
  • Godson Gera
  • Gonçalo Margalho
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Gridarrow
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Guilherme Toti
  • Gustavo Narea
  • Gustavo Niemeyer
  • Guyon Moree
  • Gökhan Sever
  • Hany Fahim
  • Hector Garcia
  • Heikki Toivonen
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Humberto Rocha
  • Hynek Schlawack
  • IO Digital Sec
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Intellimath blog
  • Ionel Cristian Maries
  • Iraj Jelodari
  • IronPython-URLs
  • Ishan Chattopadhyaya
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Krstic
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • James Mills
  • Jan-Carel Brand
  • Janusworx
  • Jarrod Millman
  • Jason Meyers
  • Jason Webb
  • Jaysinh Shukla
  • Jean-Louis Fuchs
  • Jean-Paul Calderone
  • Jeethu Rao
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Knupp
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Jim Hughes
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Anderson
  • John Burns
  • John Cook
  • John Jacobsen
  • John Paulett
  • Jon Parise
  • Jonathan Dobson
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan LaCour
  • Jonathan Street
  • Jorge Niedbalski
  • Jorge Puente Sarrín
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Juan Rodríguez Monti
  • Juho Vepsäläinen
  • Julien Danjou
  • Julien Palard
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kate Editor
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kenneth Reitz
  • Kevin Dahlhausen
  • Kevin Dangoor
  • Kiwi PyCon
  • Kodnito
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kracekumar Ramaraju
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • LAAC Technology
  • Laurence Rowe
  • Laurent Luce
  • Laurent Szyster
  • Learn PyQt
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lightning Fast Shop
  • Lintel Technologies
  • Lion Kimbro
  • Lionel Tan
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Lucas Cimon
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Macken
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Manuel de la Pena Saenz
  • Marc Garcia
  • Marc Kerins
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Mark Mruss
  • Mark Ramm
  • Marko Samastur
  • Martijn Faassen
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mathieu Virbel
  • Mats Kindahl
  • Matt Harrison
  • Matt Layman
  • Matt Reid
  • Matt Wilkes
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Mattias Brändström
  • Mauveweb
  • Max Ischenko
  • Michael Bayer
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Hudson
  • Michael Nelson
  • Michael Sparks
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikeal Rogers
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mitch Chapman
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Morten W Petersen
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Nadia Alramli
  • Naomi Ceder
  • Natan Zabkar
  • Nathan Lemoine
  • Nathan Piccini Data Science Dojo Blog
  • Ned Batchelder
  • Neil Schemenauer
  • Ng Pheng Siong
  • Nicholas Amorim
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • NumFOCUS
  • Obey the Testing Goat
  • Ofosos
  • Oliver Andrich
  • Omaha Python Users Group
  • Ondřej Čertík
  • OpenHatch Python posts
  • PSF GSoC students blogs
  • Paolo Amoroso
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patricio Paez
  • Patrick Kennedy
  • Patrick Stinson
  • Paul Bissex
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • PayPal Engineering Blog
  • Pedro Lima
  • Petecode
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philip Semanchuk
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Pierre Carbonnelle
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programando Ciência
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyATL Bytecode
  • PyBites
  • PyCarolinas
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyCon Australia
  • PyCon Pune
  • PyCon.DE & PyData Karlsruhe
  • PyPy Development
  • PyTennessee
  • PyTexas
  • Pypix
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python Insider
  • Python Morsels
  • Python Open Mike
  • Python Piedmont Triad User Group
  • Python Software Foundation
  • Python Sweetness
  • Python Testing Cookbook
  • Python User Groups
  • Python for Beginners
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • PythonThreads
  • PythonTrainingHQ
  • Pythonicity
  • Pythonistas.dev
  • Pythonology
  • Python⇒Speed
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ralph Heinkel
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Reinout van Rees
  • Rene Dudfield
  • Reuven Lerner
  • Richard Gomes
  • Richard Jones
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rob Galanakis
  • Rob Golding
  • Rob Miller
  • Robert Brewer
  • Robert Collins
  • Robert Zaremba
  • Robin Parmar
  • Robin Wilson
  • Roche Compaan
  • Rodrigo Araúj
  • Roman Imankulov
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Salman Haq
  • Sandipan Dey
  • Sandro Tosi
  • Sayan Chowdhury
  • Scripting the web with Python
  • Sean McGrath
  • Sebastian Pölsterl
  • Sebastian Witowski
  • Selena Deckelmann
  • Semaphore Community
  • Senthil Kumaran
  • Server Density
  • Shannon -jj Behrens
  • ShiningPanda
  • Shopkick Tech Blog
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stacks
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Petrea
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Dower
  • Steve Holden
  • Steven Klass
  • Stijn Debrouwere
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Sumith - Blog about SymPy/Python
  • Swaroop C H
  • Swisscom ICT
  • Sylvain Hellegouarch
  • Talk Python to Me
  • Tarek Ziade
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Ted Leung
  • Tennessee Leeuwenburg
  • Terri Oda
  • Terry Jones
  • Terry Peppers
  • Test and Code
  • TestDriven.io
  • The Artificial Intelligence Cookbook
  • The Code Bits
  • The Data Scientist
  • The Digital Cat
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Parcon Blog
  • The Python Papers
  • The R Trader
  • The Three of Wands
  • TheAutomatic.net
  • Thibauld Nion
  • Thomas Guest
  • Thomas Vander Stichele
  • Thomi Richards
  • Tiago Montes
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Golden
  • Tim Knapp
  • Tim Lesher
  • Tim van der Linden
  • Tobias Ivarsson
  • Toes to the Edge
  • Tom Christie
  • Tomasz Ducin
  • Tomasz Früboes
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Torsten Engelbrecht
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Troy Melhase
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • Two Bit Arcade
  • Uche Ogbuji
  • United Coders
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinayak Mehta
  • Vinta Software
  • Virgil Dupras
  • Vladimir Iakolev
  • Vladimir Perić
  • Wallaroo Labs
  • Wallix
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Wichert Akkerman
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • William Thompson
  • Wing Tips
  • Wingware
  • Wraithan
  • Wyatt Baldwin
  • Yaco
  • Yaniv Aknin
  • Yann Larrivée
  • Yasoob Khalid
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero to Mastery
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • codingdirectional
  • eGenix.com
  • eRambler
  • gamingdirectional
  • hypothesis.works articles
  • kdev-python
  • leftmouseclickin
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • spikelantern
  • testmon
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal:
    Open an issue on github
    or e-mail planet at python.org
    (note, responses can take up to a few days)