skip to navigation
skip to content

Planet Python

Last update: October 17, 2019 01:48 PM UTC

October 17, 2019

Abhijeet Pal

Using PostgreSQL with Django

Django is a high level full-stack open-source web framework written in Python, that encourages rapid development and clean, pragmatic design.

Django, in its ‘out-of-the-box’ state, is set up to communicate with SQLite – a lightweight relational database included with the Python distribution. So by default, Django automatically creates an SQLite database for your project.

In addition to SQLite, Django also has support for other popular databases that include PostgreSQL, MySQL, and Oracle.

However, PostgreSQL has a number of features that are not shared by the other databases Django supports, which makes it an idle choice for a Django app in production.

In this article, we will go through the integration of PostgreSQL with a Django Application.


We are assuming you already have Django installed on your machine and one Django project up and running, if not then read the following article – Starting A Django Project

Installing PostgreSQL

Windows and macOS X users can download PostgreSQL from the official site and simply install it.

Linux User

sudo apt-get install postgresql postgresql-contrib

Also, Linux users need to install some dependencies for PostgreSQL to work with Python.

sudo apt-get install libpq-dev python-dev

Install psycopg2

Next, we need to install the PostgreSQL database adapter to communicate to the database with Python to install it run the following command in the shell.

pip install psycopg2

Create A PostgreSQL User and Database

As the default configuration of Postgres is, a user called Postgres is made on, and the user Postgres has full super admin access to entire PostgreSQL instance running on your OS.

sudo -u postgres psql

Now the terminal should be prefixed with postgres=# , The above command gets you the psql command-line interface in full admin mode.

Now let’s create a user and database.

Creating Database


This will create a database named mydb, note that every SQL statement must end with a semicolon.

Creating User


Here we are creating a user named myuser with password mypass. You can use any username and password you wish.

Modifying Connection Parameters

 ALTER ROLE myuser SET client_encoding TO 'utf8';
 ALTER ROLE myuser SET default_transaction_isolation TO 'read committed';
 ALTER ROLE myuser SET timezone TO 'UTC';

We are setting the default encoding to UTF-8, which Django expects.

We are also setting the default transaction isolation scheme to “read committed”, which blocks reads from uncommitted transactions.

Lastly, we are setting the timezone by default, our Django projects will be set to use UTC.These are essential parameters recommended by the official Django team.

Granting Permission To The User


Now our user has administrative access to the database.

Now exit the SQL prompt.


Integrating PostgreSQL With Django

Open the file of your project and scroll straight to the database section, which should look like this.

    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),

We need to update these settings to integrate our PostgreSQL with the project.

    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'mydb',
        'USER': 'myuser',
        'PASSWORD': 'mypass',
        'HOST': 'localhost',
        'PORT': '',

Let’s quickly go over the settings,

DATABASES – This constant is a dictionary of database connection information and is required by Django. You can have multiple connections to different databases, but most of the time, you will just need an entry called default.

default – This is the default database connection configuration. You should always have a default set of connections settings.

'ENGINE': 'django.db.backends.postgresql_psycopg2' – This tells Django to use the Postgres backend. This, in turn uses psycopg2, Python’s Postgres library which we installed earlier.

'NAME': 'mydb' – The name of the database you want to connect to.

'USER': 'myuser' – The User with access to the database.

'PASSWORD': 'mypass' – The password for your database user.

'HOST': 'localhost' – The address of the database server you want to connect to.

'PORT': '' – The port you want to connect to, which by default is ‘5432’

Test The Database Connection

After updating the database configurations, it’s time to test the connection. The Django database migration process ensures all Django project logic associated with a database is reflected in the database itself.

During the first migration against the database, there are a series of migrations Django requires that create tables to keep track of administrators and sessions.

In the directory where script exists, run the following command.

python migrate

If everything went right you should see an output like this.

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying sessions.0001_initial... OK

Furthermore, you can now create a superuser and login to the admin dashboard.

The post Using PostgreSQL with Django appeared first on Django Central.

October 17, 2019 11:51 AM UTC

Creating A Super User In Django

Django’s prominent feature is the admin interface, which makes it stand out from the competition. It is a built-in app that automatically generates a user interface to add and modify a site’s content.

The built-in administration interface that is very useful for editing content. The Django admin site is built dynamically by reading your model metadata and providing a production-ready interface for editing content. You can use it out of the box, configuring how you want your models to be displayed in it.

In order to use the Django’s admin app which is already included in django.contrib.admin inside INSTALLED_APPS setting, first we have to create a superuser.

Creating A Super User In Django

In the directory where script exists, execute the following command.

python createsuperuser

Now Django will prompt you to enter the details, enter your desired details and hit enter.

Username (leave blank to use 'admin'): admin
Email address:
Password: ********
Password (again): ********
Superuser created successfully.

Now that the superuser is created, run the development server.

python runserver

Open the browser and navigate to You should see a login page, enter the details you provided for the superuser.

Django Admin login

After you log in you should see  the Django’s admin panel with Groups and Users models which come from Django authentication framework located in django.contrib.auth.

Django admin dashboard

The post Creating A Super User In Django appeared first on Django Central.

October 17, 2019 07:10 AM UTC


Multiple Slugs for Multiple Webpages in Nikola

Here’s a delightful little thing I discovered, using Nikola today.

As part of a little housecleaning, I wanted to have a single page have two web urls.
As in Books I’ve Read and Books in 2020.
Don’t ask why, my nerdy brain just wants it that way.

And I was thinking of Nginx redirects and other things I could try.
Until I looked up and saw the slug section in the metadata section of my post (the section that I normally turn a blind eye to).
This is what I assume, Nikola uses to generate links.
And I thought hmm, what if?
And at the same time I remembered what Roberto Alsina, the original author of Nikola had encouraged me to do, when I had just started using it to blog.
That he wrote Nikola, so that it would mostly do the logical thing; that I wouldn’t really need to use the documentation (excellent as it is)

So that is what I did then. I put in two slugs each with the unique url I wanted, and sure enough, Nikola generated two web pages from that single source file.

It’s the little things like this, that make me so happy, I use Nikola for my writing.

October 17, 2019 04:09 AM UTC

October 16, 2019

Roberto Alsina

Episodio 12: Buen Gusto

Enfrentado a un problema, uno de los criterios para buscar la solución es el misterioso "que sea pitónico". ¿Qué es eso? ¿Vale la pena usar algo más lindo pero 12 veces más lento? ¿Y cómo sé eso? ¿Puedo mejorarlo? ¿Y si es 2 veces más lento nomás?

October 16, 2019 11:00 PM UTC

Full Stack Python

Basic Data Types in Python 3: Strings

There is a lot to learn on your Python journey when you are new to the programming language. Once you are comfortable writing and executing code, your first stop becomes understanding how to represent data in your code. No matter the language, there are a few basic data types you'll use all the time - strings, numbers, booleans, lists, and dictionaries.

Those data types, and how to use them in Python 3, are the topic of this blog post series. Today, we're starting with strings.

If you're learning Python, you might also want to check out TwilioQuest 3. You'll learn about basic data types and much more about Python programming.

Ready to learn how to use strings in Python 3? Let's get started!

Strings in Python 3

One of the most common data types in any programming language is a string. A string represents a series of characters, which you would use to represent usernames, blog posts, tweets, or any text content in your code. You can create a string and assign it to a variable like this.

my_name = "Jonathan Joestar"

Strings are "immutable"

In Python, strings are considered immutable - once you create them, they can't be changed. You can, however, use a variety of methods to create new strings from existing strings. This type of work in programming is called string manipulation. Some web developers joke that at the end of the day, their job is just mashing strings together - and this isn't far from the truth!

Here are some common tasks you might undertake when using strings in your code.

Common task - combining strings together

Combining strings together - concatenating them - is a very common task. In Python 3, you can use the + operator for this purpose. You can use the + operator multiple times to concatenate multiple strings.

first_name = "Jonathan"
last_name = "Joestar"

full_name = first_name + " " + last_name

Common task - inserting data into strings

Another common task with strings is inserting data into a specific place within a string. In programming, we call this string interpolation. Python 3 provides a handy tool for doing this called "f" strings. The "f" in "f strings" stands for format - you can insert other data from your program into a string when you define it rather than doing complex string concatenation as demonstrated previously.

Here is an example of creating a formatted string - note the letter f is included just before the first double quote when defining the message variable. When you want to insert data from your program into the string, you can include it between two "curly braces" - the { and } characters.

first_name = "Jonathan"
last_name = "Joestar"
age = 24

message = f"My name is {first_name} {last_name}, and I am {age} years old."

Common task - using built-in string methods to manipulate strings

String objects have a number of methods to perform common tasks, like changing the case of strings or trimming their content. Below, you'll find a few examples. In two of these examples, we are creating a string variable, and then assigning the same variable a new value, which is the result of calling a method on a string object.

Example 1: Convert a string to all caps using the upper method.

example_string = "am I stoked enough yet?"
example_string = example_string.upper()
print(example_string) # prints "AM I STOKED ENOUGH YET?"

Example 2: Replace all instances of the word kale with tacos.

example_string = "We're having kale for dinner! Yay kale!"
example_string = example_string.replace("kale", "tacos")
print(example_string) # prints "We're having tacos for dinner! Yay tacos!"

Example 3: Split a comma-delimited string into a list of strings.

example_string = "Apples,Oranges,Pears"
groceries = example_string.split(',')

# Code below prints:
# Apples
# Oranges
# Pears
for item in groceries:

Check our more strings can do in the Python 3 docs!

Type casting

Frequently, you will want to convert data from one type into another. In programming, we call this process type casting. There are a number of functions built in to Python which allow us to do these type conversions on basic data types.

Example 1: Convert a number into a string using the str function.

example_number = 42
converted = str(example_number)
message = "The meaning of life is " + converted

Example 2: Convert a string into a whole number (integer) using int.

example_string = "2"
converted = int(example_string)
message = f"Two plus two equals { converted + 2 }"

Wrapping up

Strings of text are one of the most common pieces of data you will work with in programming. Hopefully, you've learned a bit about how to work with strings in Python 3! Stay tuned for more blog posts in this series to learn more about basic data types like strings, numbers, booleans, lists, and dictionaries.

Also, be sure to download and play TwilioQuest 3 to learn even more about Python!

October 16, 2019 09:40 PM UTC


Theory: average bus factor = 1

Two articles recently made me realize that all my free software projects basically have a bus factor of one. I am the sole maintainer of every piece of software I have ever written that I still maintain. There are projects that I have been the maintainer of which have other maintainers now (most notably AlternC, Aegir and Linkchecker), but I am not the original author of any of those projects.

Now that I have a full time job, I feel the pain. Projects like Gameclock, Monkeysign, Stressant, and (to a lesser extent) Wallabako all need urgent work: the first three need to be ported to Python 3, the first two to GTK 3, and the latter will probably die because I am getting a new e-reader. (For the record, more recent projects like undertime and feed2exec are doing okay, mostly because they were written in Python 3 from the start, and the latter has extensive unit tests. But they do suffer from the occasional bitrot (the latter in particular) and need constant upkeep.)

Now that I barely have time to keep up with just the upkeep, I can't help but think all of my projects will just die if I stop working on them. I have the same feeling about the packages I maintain in Debian.

What does that mean? Does that mean those packages are useless? That no one cares enough to get involved? That I'm not doing a good job at including contributors?

I don't think so. I think I'm a friendly person online, and I try my best at doing good documentation and followup on my projects. What I have come to understand is even more depressing and scary that this being a personal failure: that is the situation with everyone, everywhere. The LWN article is not talking about silly things like a chess clock or a feed reader: we're talking about the Linux input drivers. A very deep, core component of the vast majority of computers running on the planet, that depend on that single maintainer. And I'm not talking about whether those people are paid or not, that's related, but not directly the question here. The same realization occured with OpenSSL and NTP, GnuPG is in a similar situation, the list just goes on and on.

A single guy maintains those projects! Is that a fluke? A statistical anomaly? Everything I feel, and read, and know in my decades of experience with free software show me a reality that I've been trying to deny for all that time: it's the average.

My theory is this: our average bus factor is one. I don't have any hard evidence to back this up, no hard research to rely on. I'd love to be proven wrong. I'd love for this to change.

But unless economics of technology production change significantly in the coming decades, this problem will remain, and probably worsen, as we keep on scaffolding an entire civilization on shoulders of hobbyists that are barely aware their work is being used to power phones, cars, airplanes and hospitals. A lot has been written on this, but nothing seems to be moving.

And if that doesn't scare you, it damn well should. As a user, one thing you can do is, instead of wondering if you should buy a bit of proprietary software, consider using free software and donating that money to free software projects instead. Lobby governments and research institutions to sponsor only free software projects. Otherwise this civilization will collapse in a crash of spaghetti code before it even has time to get flooded over.

October 16, 2019 07:21 PM UTC

Stories in My Pocket

PyCon 2019: Open Spaces

There have been a few thoughts about PyCon US 2019 that have been bouncing around my head, wanting to come out. Today, I want to talk about PyCon's Open Spaces, and why you should start planning on attending next year.


October 16, 2019 04:48 PM UTC

Real Python

Python sleep(): How to Add Time Delays to Your Code

Have you ever needed to make your Python program wait for something? Most of the time, you’d want your code to execute as quickly as possible. But there are times when letting your code sleep for a while is actually in your best interest.

For example, you might use a Python sleep() call to simulate a delay in your program. Perhaps you need to wait for a file to upload or download, or for a graphic to load or be drawn to the screen. You might even need to pause between calls to a web API, or between queries to a database. Adding Python sleep() calls to your program can help in each of these cases, and many more!

In this tutorial, you’ll learn how to add Python sleep() calls with:

This article is intended for intermediate developers who are looking to grow their knowledge of Python. If that sounds like you, then let’s get started!

Free Bonus: Get our free "The Power of Python Decorators" guide that shows you 3 advanced decorator patterns and techniques you can use to write to cleaner and more Pythonic programs.

Adding a Python sleep() Call With time.sleep()

Python has built-in support for putting your program to sleep. The time module has a function sleep() that you can use to suspend execution of the calling thread for however many seconds you specify.

Here’s an example of how to use time.sleep():

>>> import time
>>> time.sleep(3) # Sleep for 3 seconds

If you run this code in your console, then you should experience a delay before you can enter a new statement in the REPL.

Note: In Python 3.5, the core developers changed the behavior of time.sleep() slightly. The new Python sleep() system call will last at least the number of seconds you’ve specified, even if the sleep is interrupted by a signal. This does not apply if the signal itself raises an exception, however.

You can test how long the sleep lasts by using Python’s timeit module:

$ python3 -m timeit -n 3 "import time; time.sleep(3)"
3 loops, best of 3: 3 sec per loop

Here, you run the timeit module with the -n parameter, which tells timeit how many times to run the statement that follows. You can see that timeit ran the statement 3 times and that the best run time was 3 seconds, which is what was expected.

The default number of times that timeit will run your code is one million. If you were to run the above code with the default -n, then at 3 seconds per iteration, your terminal would hang for approximately 34 days! The timeit module has several other command line options that you can check out in its documentation.

Let’s create something a bit more realistic. A system administrator needs to know when one of their websites goes down. You want to be able to check the website’s status code regularly, but you can’t query the web server constantly or it will affect performance. One way to do this check is to use a Python sleep() system call:

import time
import urllib.request
import urllib.error

def uptime_bot(url):
    while True:
            conn = urllib.request.urlopen(url)
        except urllib.error.HTTPError as e:
            # Email admin / log
            print(f'HTTPError: {e.code} for {url}')
        except urllib.error.URLError as e:
            # Email admin / log
            print(f'URLError: {e.code} for {url}')
            # Website is up
            print(f'{url} is up')

if __name__ == '__main__':
    url = ''

Here you create uptime_bot(), which takes a URL as its argument. The function then attempts to open that URL with urllib. If there’s an HTTPError or URLError, then the program catches it and prints out the error. (In a live environment, you would log the error and probably send out an email to the webmaster or system administrator.)

If no errors occur, then your code prints out that all is well. Regardless of what happens, your program will sleep for 60 seconds. This means that you only access the website once every minute. The URL used in this example is bad, so it will output the following to your console once every minute:

HTTPError: 404 for

Go ahead and update the code to use a known good URL, like Then you can re-run it to see it work successfully. You can also try to update the code to send an email or log the errors. For more information on how to do this, check out Sending Emails With Python and Logging in Python.

Adding a Python sleep() Call With Decorators

There are times when you need to retry a function that has failed. One popular use case for this is when you need to retry a file download because the server was busy. You usually won’t want to make a request to the server too often, so adding a Python sleep() call between each request is desirable.

Another use case that I’ve personally experienced is where I need to check the state of a user interface during an automated test. The user interface might load faster or slower than usual, depending on the computer I’m running the test on. This can change what’s on the screen at the moment my program is verifying something.

In this case, I can tell the program to sleep for a moment and then recheck things a second or two later. This can mean the difference between a passing and failing test.

You can use a decorator to add a Python sleep() system call in either of these cases. If you’re not familiar with decorators, or if you’d like to brush up on them, then check out Primer on Python Decorators. Let’s look at an example:

import time
import urllib.request
import urllib.error

def sleep(timeout, retry=3):
    def the_real_decorator(function):
        def wrapper(*args, **kwargs):
            retries = 0
            while retries < retry:
                    value = function(*args, **kwargs)
                    if value is None:
                    print(f'Sleeping for {timeout} seconds')
                    retries += 1
        return wrapper
    return the_real_decorator

sleep() is your decorator. It accepts a timeout value and the number of times it should retry, which defaults to 3. Inside sleep() is another function, the_real_decorator(), which accepts the decorated function.

Finally, the innermost function wrapper() accepts the arguments and keyword arguments that you pass to the decorated function. This is where the magic happens! You use a while loop to retry calling the function. If there’s an exception, then you call time.sleep(), increment the retries counter, and try running the function again.

Now rewrite uptime_bot() to use your new decorator:

def uptime_bot(url):
        conn = urllib.request.urlopen(url)
    except urllib.error.HTTPError as e:
        # Email admin / log
        print(f'HTTPError: {e.code} for {url}')
        # Re-raise the exception for the decorator
        raise urllib.error.HTTPError
    except urllib.error.URLError as e:
        # Email admin / log
        print(f'URLError: {e.code} for {url}')
        # Re-raise the exception for the decorator
        raise urllib.error.URLError
        # Website is up
        print(f'{url} is up')

if __name__ == '__main__':
    url = ''

Here, you decorate uptime_bot() with a sleep() of 3 seconds. You’ve also removed the original while loop, as well as the old call to sleep(60). The decorator now takes care of this.

One other change you’ve made is to add a raise inside of the exception handling blocks. This is so that the decorator will work properly. You could write the decorator to handle these errors, but since these exceptions only apply to urllib, you might be better off keeping the decorator the way it is. That way, it will work with a wider variety of functions.

Note: If you’d like to brush up on exception handling in Python, then check out Python Exceptions: An Introduction.

There are a few improvements that you could make to your decorator. If it runs out of retries and still fails, then you could have it re-raise the last error. The decorator will also wait 3 seconds after the last failure, which might be something you don’t want to happen. Feel free to try these out as an exercise!

Adding a Python sleep() Call With Threads

There are also times when you might want to add a Python sleep() call to a thread. Perhaps you’re running a migration script against a database with millions of records in production. You don’t want to cause any downtime, but you also don’t want to wait longer than necessary to finish the migration, so you decide to use threads.

Note: Threads are a method of doing concurrency in Python. You can run multiple threads at once to increase your application’s throughput. If you’re not familiar with threads in Python, then check out An Intro to Threading in Python.

To prevent customers from noticing any kind of slowdown, each thread needs to run for a short period and then sleep. There are two ways to do this:

  1. Use time.sleep() as before.
  2. Use Event.wait() from the threading module.

Let’s start by looking at time.sleep().

Using time.sleep()

The Python Logging Cookbook shows a nice example that uses time.sleep(). Python’s logging module is thread-safe, so it’s a bit more useful than print() statements for this exercise. The following code is based on this example:

import logging
import threading
import time

def worker(arg):
    while not arg["stop"]:
        logging.debug("worker thread checking in")

def main():
        format="%(relativeCreated)6d %(threadName)s %(message)s"
    info = {"stop": False}
    thread = threading.Thread(target=worker, args=(info,))
    thread_two = threading.Thread(target=worker, args=(info,))

    while True:
            logging.debug("Checking in from main thread")
        except KeyboardInterrupt:
            info["stop"] = True

if __name__ == "__main__":

Here, you use Python’s threading module to create two threads. You also create a logging object that will log the threadName to stdout. Next, you start both threads and initiate a loop to log from the main thread every so often. You use KeyboardInterrupt to catch the user pressing Ctrl+C.

Try running the code above in your terminal. You should see output similar to the following:

 0 Thread-1 worker thread checking in
 1 Thread-2 worker thread checking in
 1 MainThread Checking in from main thread
752 MainThread Checking in from main thread
1001 Thread-1 worker thread checking in
1001 Thread-2 worker thread checking in
1502 MainThread Checking in from main thread
2003 Thread-1 worker thread checking in
2003 Thread-2 worker thread checking in
2253 MainThread Checking in from main thread
3005 Thread-1 worker thread checking in
3005 MainThread Checking in from main thread
3005 Thread-2 worker thread checking in

As each thread runs and then sleeps, the logging output is printed to the console. Now that you’ve tried an example, you’ll be able to use these concepts in your own code.

Using Event.wait()

The threading module provides an Event() that you can use like time.sleep(). However, Event() has the added benefit of being more responsive. The reason for this is that when the event is set, the program will break out of the loop immediately. With time.sleep(), your code will need to wait for the Python sleep() call to finish before the thread can exit.

The reason you’d want to use wait() here is because wait() is non-blocking, whereas time.sleep() is blocking. What this means is that when you use time.sleep(), you’ll block the main thread from continuing to run while it waits for the sleep() call to end. wait() solves this problem. You can read more about how all this works in Python’s threading documentation.

Here’s how you add a Python sleep() call with Event.wait():

import logging
import threading

def worker(event):
    while not event.isSet():
        logging.debug("worker thread checking in")

def main():
        format="%(relativeCreated)6d %(threadName)s %(message)s"
    event = threading.Event()

    thread = threading.Thread(target=worker, args=(event,))
    thread_two = threading.Thread(target=worker, args=(event,))

    while not event.isSet():
            logging.debug("Checking in from main thread")
        except KeyboardInterrupt:

if __name__ == "__main__":

In this example, you create threading.Event() and pass it to worker(). (Recall that in the previous example, you instead passed a dictionary.) Next, you set up your loops to check whether or not event is set. If it’s not, then your code prints a message and waits a bit before checking again. To set the event, you can press Ctrl+C. Once the event is set, worker() will return and the loop will break, ending the program.

Note: If you’d like to learn more about dictionaries, then check out Dictionaries in Python.

Take a closer look at the code block above. How would you pass in a different sleep time to each worker thread? Can you figure it out? Feel free to tackle this exercise on your own!

Adding a Python sleep() Call With Async IO

Asynchronous capabilities were added to Python in the 3.4 release, and this feature set has been aggressively expanding ever since. Asynchronous programming is a type of parallel programming that allows you to run multiple tasks at once. When a task finishes, it will notify the main thread.

asyncio is a module that lets you add a Python sleep() call asynchronously. If you’re unfamiliar with Python’s implementation of asynchronous programming, then check out Async IO in Python: A Complete Walkthrough and Python Concurrency & Parallel Programming.

Here’s an example from Python’s own documentation:

import asyncio

async def main():
    print('Hello ...')
    await asyncio.sleep(1)
    print('... World!')

# Python 3.7+

In this example, you run main() and have it sleep for one second between two print() calls.

Here’s a more compelling example from the Coroutines and Tasks portion of the asyncio documentation:

import asyncio
import time

async def output(sleep, text):
    await asyncio.sleep(sleep)

async def main():
    print(f"Started: {time.strftime('%X')}")
    await output(1, 'First')
    await output(2, 'Second')
    await output(3, 'Third')
    print(f"Ended: {time.strftime('%X')}")

# Python 3.7+

In this code, you create a worker called output() that takes in the number of seconds to sleep and the text to print out. Then, you use Python’s await keyword to wait for the output() code to run. await is required here because output() has been marked as an async function, so you can’t call it like you would a normal function.

When you run this code, your program will execute await 3 times. The code will wait for 1, 2, and 3 seconds, for a total wait time of 6 seconds. You can also rewrite the code so that the tasks run in parallel:

import asyncio
import time

async def output(text, sleep):
    while sleep > 0:
        await asyncio.sleep(1)
        print(f'{text} counter: {sleep} seconds')
        sleep -= 1

async def main():
    task_1 = asyncio.create_task(output('First', 1))
    task_2 = asyncio.create_task(output('Second', 2))
    task_3 = asyncio.create_task(output('Third', 3))
    print(f"Started: {time.strftime('%X')}")
    await task_1
    await task_2
    await task_3                                 
    print(f"Ended: {time.strftime('%X')}")

if __name__ == '__main__':

Now you’re using the concept of tasks, which you can make with create_task(). When you use tasks in asyncio, Python will run the tasks asynchronously. So, when you run the code above, it should finish in 3 seconds total instead of 6.

Adding a Python sleep() Call With GUIs

Command-line applications aren’t the only place where you might need to add Python sleep() calls. When you create a Graphical User Interface (GUI), you’ll occasionally need to add delays. For example, you might create an FTP application to download millions of files, but you need to add a sleep() call between batches so you don’t bog down the server.

GUI code will run all its processing and drawing in a main thread called the event loop. If you use time.sleep() inside of GUI code, then you’ll block its event loop. From the user’s perspective, the application could appear to freeze. The user won’t be able to interact with your application while it’s sleeping with this method. (On Windows, you might even get an alert about how your application is now unresponsive.)

Fortunately, there are other methods you can use besides time.sleep(). In the next few sections, you’ll learn how to add Python sleep() calls in both Tkinter and wxPython.

Sleeping in Tkinter

tkinter is a part of the Python standard library. It may not be available to you if you’re using a pre-installed version of Python on Linux or Mac. If you get an ImportError, then you’ll need to look into how to add it to your system. But if you install Python yourself, then tkinter should already be available.

You’ll start by looking at an example that uses time.sleep(). Run this code to see what happens when you add a Python sleep() call the wrong way:

import tkinter
import time

class MyApp:
    def __init__(self, parent):
        self.root = parent
        self.frame = tkinter.Frame(parent)
        b = tkinter.Button(text="click me", command=self.delayed)

    def delayed(self):

if __name__ == "__main__":
    root = tkinter.Tk()
    app = MyApp(root)

Once you’ve run the code, press the button in your GUI. The button will stick down for three seconds as it waits for sleep() to finish. If the application had other buttons, then you wouldn’t be able to click them. You can’t close the application while it’s sleeping, either, since it can’t respond to the close event.

To get tkinter to sleep properly, you’ll need to use after():

import tkinter

class MyApp:
    def __init__(self, parent):
        self.root = parent
        self.frame = tkinter.Frame(parent)
        self.root.after(3000, self.delayed)

    def delayed(self):
        print('I was delayed')

if __name__ == "__main__":
    root = tkinter.Tk()
    app = MyApp(root)

Here you create an application that is 400 pixels wide by 400 pixels tall. It has no widgets on it. All it will do is show a frame. Then, you call self.root.after() where self.root is a reference to the Tk() object. after() takes two arguments:

  1. The number of milliseconds to sleep
  2. The method to call when the sleep is finished

In this case, your application will print a string to stdout after 3 seconds. You can think of after() as the tkinter version of time.sleep(), but it also adds the ability to call a function after the sleep has finished.

You could use this functionality to improve user experience. By adding a Python sleep() call, you can make the application appear to load faster and then start some longer-running process after it’s up. That way, the user won’t have to wait for the application to open.

Sleeping in wxPython

There are two major differences between wxPython and Tkinter:

  1. wxPython has many more widgets.
  2. wxPython aims to look and feel native on all platforms.

The wxPython framework is not included with Python, so you’ll need to install it yourself. If you’re not familiar with wxPython, then check out How to Build a Python GUI Application With wxPython.

In wxPython, you can use wx.CallLater() to add a Python sleep() call:

import wx

class MyFrame(wx.Frame):
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        wx.CallLater(4000, self.delayed)

    def delayed(self):
        print('I was delayed')

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()

Here, you subclass wx.Frame directly and then call wx.CallLater(). This function takes the same parameters as Tkinter’s after():

  1. The number of milliseconds to sleep
  2. The method to call when the sleep is finished

When you run this code, you should see a small blank window appear without any widgets. After 4 seconds, you’ll see the string 'I was delayed' printed to stdout.

One of the benefits of using wx.CallLater() is that it’s thread-safe. You can use this method from within a thread to call a function that’s in the main wxPython application.


With this tutorial, you’ve gained a valuable new technique to add to your Python toolbox! You know how to add delays to pace your applications and prevent them from using up system resources. You can even use Python sleep() calls to help your GUI code redraw more effectively. This will make the user experience much better for your customers!

To recap, you’ve learned how to add Python sleep() calls with the following tools:

Now you can take what you’ve learned and start putting your code to sleep!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 16, 2019 02:00 PM UTC

Continuum Analytics Blog

What Can AI Teach Us about Bias and Fairness?

By: Peter Wang & Natalie Parra-Novosad As researchers, journalists, and many others have discovered, machine learning algorithms can deliver biased results. One notorious example is ProPublica’s discovery of bias in a software called COMPAS used…

The post What Can AI Teach Us about Bias and Fairness? appeared first on Anaconda.

October 16, 2019 01:30 PM UTC

Catalin George Festila

Python 3.7.4 : Test the DHCP handshakes.

First, the DHCP is based on the earlier BOOTP protocol which uses well-known port numbers for both server and client instead of an ephemeral port. The server and the client communicate via broadcast and the server broadcasts the offered IP address to the client on UDP port 68. This python example has a learning purpose and does not harm anyone. import subprocess as sub import re def find_word(w

October 16, 2019 12:03 PM UTC

Test and Code

91: Python 3.8 - there's a lot more new than most people are talking about

Python 3.8.0 final is live and ready to download.

On todays episode, we're going to run through what's new, picking out the bits that I think are the most interesting and affect the most people, including

Not just the big stuff everyone's already talking about. But also some little things that will make programming Python even more fun and easy.

I'm excited about Python 3.8. And really, this episode is to my way to try to get you excited about it too.

Sponsored By:

Support Test & Code


<p>Python 3.8.0 final is live and ready to download.</p> <p>On todays episode, we&#39;re going to run through what&#39;s new, picking out the bits that I think are the most interesting and affect the most people, including</p> <ul> <li>new language features</li> <li>standard library changes</li> <li>optimizations in 3.8</li> </ul> <p>Not just the big stuff everyone&#39;s already talking about. But also some little things that will make programming Python even more fun and easy.</p> <p>I&#39;m excited about Python 3.8. And really, this episode is to my way to try to get you excited about it too.</p><p>Sponsored By:</p><ul><li><a href="" rel="nofollow">PyCharm Professional</a>: <a href="" rel="nofollow">Try PyCharm Pro before Oct 29 with a 4 month free trial. </a> Promo Code: TESTNCODE2019</li></ul><p><a href="" rel="payment">Support Test & Code</a></p><p>Links:</p><ul><li><a href="" title="What’s New In Python 3.8 - at" rel="nofollow">What’s New In Python 3.8 - at</a></li><li><a href="" title="Download Python 3.8 at" rel="nofollow">Download Python 3.8 at</a></li></ul>

October 16, 2019 07:15 AM UTC

Hynek Schlawack

Sharing Your Labor of Love: PyPI Quick and Dirty

A completely incomplete guide to packaging a Python module and sharing it with the world on PyPI.

October 16, 2019 12:00 AM UTC

October 15, 2019

PyCoder’s Weekly

Issue #390 (Oct. 15, 2019)

#390 – OCTOBER 15, 2019
View in Browser »

The PyCoder’s Weekly Logo

CPython 3.8.0 Released

Assignment expressions, positional-only arguments, = specifier in f-strings, and more. Click the link for the full changelog and download links.

Cool New Features in Python 3.8

What does Python 3.8 bring to the table? Learn about some of the biggest changes and see you how you can best make use of them.

Join the Slack Team at the Spec Annual Developer Conference


Hear the latest product and tooling enhancements for builders of Slack apps. Learn best practices and apply them in hands-on workshops. Troubleshoot with Slack engineers, ask questions and give feedback to our product team. Enjoy or share 50% off registration with the code S19_ADMIN →
SLACK sponsor

Thousands of Scientific Papers May Be Invalid Due to Misunderstanding Python

Developers assumed that glob.glob() returns a sorted list of files, but it doesn’t. And this led to a number of statistical errors and data discrepancies in several published scientific papers.

The Python range() Function

In this step-by-step course, you’ll master the Python range() function, learn how its implementation differs in Python 3 vs 2, and see how you can use it to write faster and more Pythonic code.

Meta-Programming in Python

“Meta-programming is an act of building functions and classes who can manipulate code by modifying, wrapping existing code or generating code.”

Django 3.0 Beta 1 Released

See the release notes for a list of changes and new features.

PyPy V7.2 Released

ARM aarch64 support, new JSON decoder, sandboxing makes a return, …


PyPI Just Crossed the 200,000 Packages Threshold! 🥳


Python Jobs

Full Stack Developer (Toronto, ON, Canada)

Beanfield Metroconnect

Backend Developer (Kfar Saba, Israel)


More Python Jobs >>>

Articles & Tutorials

Binning Data With Pandas qcut And cut

Pandas qcut and cut are both used to bin continuous values into discrete buckets or bins. This article explains the differences between the two commands and how to use each.

Emacs: The Best Python Editor?

Learn about using Emacs for Python development. You’ll install and configure Emacs on your selected platform, then write Python code to explore its capabilities. Finally, you’ll run, test, and debug Python code in the Emacs environment.

Python Developers Are in Demand on Vettery


Vettery is an online hiring marketplace that’s changing the way people hire and get hired. Ready for a bold career move? Make a free profile, name your salary, and connect with hiring managers from top employers today →
VETTERY sponsor

How to Read SAS Files in Python With Pandas

Learn how to read SAS (.sas7bdat) files in Python, and how to write a SAS file to CSV using Pandas and pyreadstat.

Python Is Not a Great Programming Language

Related discussion/rebuttal on Hacker News.

Top Three Mistakes With K-Means Clustering During Data Analysis


VS Code Adds Native Editing of Jupyter Notebooks


Big-O Notation With Python Examples


Projects & Code

systemd-logging: Simplifies Logging for systemd


PyperCard: HyperCard-Inspired GUI Framework


inspectortiger: Python Code Review Tool / Framework


Utility for Converting Curl Syntax to Python


daudin: A Python Command-Line Shell


django-admin-autocomplete-list-filter: Ajax Autocomplete List Filter for Django Admin



PyCode Conference 2019

October 14 to October 17, 2019

PyData Bristol Meetup

October 17, 2019

Python Northwest

October 17, 2019

PyLadies Dublin

October 17, 2019

PyCon China 2019 Beijing Branch

October 19 to October 20, 2019

Python Brasil 2019

October 23 to October 29, 2019

Happy Pythoning!
This was PyCoder’s Weekly Issue #390.
View in Browser »


[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

October 15, 2019 07:30 PM UTC

Python Insider

Python 3.7.5 is now available

Python 3.7.5 is now available, the next maintenance release of Python 3.7.  You can find the release files, a link to the changelog, and more information here:

Note that the next feature release of Python 3, Python 3.8.0, is also now available.  Python 3.8 contains many new features and optimizations. You should consider upgrading to it. We plan to continue regular bugfix releases of Python 3.7.x through mid-year 2020 and provide security fixes for it until mid-year 2023.  More details are available in PEP 537, the Python 3.7 Release Schedule (

Thanks to all of the many volunteers who help make Python Development and these releases possible!  Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation.

October 15, 2019 04:05 PM UTC

Roberto Alsina

Episodio 11: Type Hints

¿Qué son los Type Hints en Python? ¿Para qué sirven? ¿Cómo se usan?

October 15, 2019 03:00 PM UTC

Red Hat Developers

What’s new in Red Hat Dependency Analytics

We are excited to announce a new release of Red Hat Dependency Analytics, a solution that enables developers to create better applications by evaluating and adding high-quality open source components, directly from their IDE.

Red Hat Dependency Analytics helps your development team avoid security and licensing issues when building your applications. It plugs into the developer’s IDE, automatically analyzes your software composition, and provides recommendations to address security holes and licensing problems that your team may be missing.

Without further ado, let’s jump into the new capabilities offered in this release. This release includes a new version of the IDE plugin and the server-side analysis service hosted by Red Hat.

Support for Python applications

Along with Java (maven) and JavaScript (npm), Dependency Analytics now offers its full set of capabilities for Python (PyPI) applications. From your IDE, you can perform the vulnerability and license analysis of the “requirements.txt” file of your Python application, incorporate the recommended fixes, and generate the stack analysis report for more details.

Software composition analysis based on current vulnerability data

An estimated 15,000 open source packages get updated every day. On average, three new vulnerabilities get posted every day across JavaScript (npm) and Python (PyPi) packages. With this new release, the server-side analysis service hosted by Red Hat automatically processes the daily updates to open source packages that it is tracking. The hosted service also automatically ingests new vulnerability data posted to National Vulnerability Database (NVD) for JavaScript and Python packages. This allows the IDE plugin and API calls to provide source code analysis based on current vulnerability and release data.

Analyze transitive dependencies

In addition to the direct dependencies included in your application, Dependency Analytics now leverages the package managers to discover and add the dependencies of those dependencies, called “transitive” dependencies, to the dependency graph of your application. Analysis of your application is performed across the whole graph model and recommendations for fixes are provided across the entire set of dependencies.

Recommendations about complementary open source libraries

With this release, Dependency Analytics looks to recommend high-quality open source libraries that are complementary to the dependencies included in your application. The machine learning technology of the hosted service collects and analyzes various statistics on GitHub to curate a list of high-quality open source libraries that can be added to the current set of dependencies to augment your application. You can provide your feedback about the add-on libraries by clicking on the “thumbs-up” or “thumbs-down” icons shown for each recommendation. Your feedback is automatically processed to improve the quality of the recommendations.

IDE plugin support

The Dependency Analytics IDE plugin is now available for VS Code, Eclipse Che, and any JetBrains IDE, including IntelliJ and PyCharm.

We will continuously release new updates to our Dependency Analytics solution so you can minimize the delays in delivery of your applications due to last-minute security and licensing related issues.

Stay tuned for further updates; we look forward to your feedback about Dependency Analytics.


The post What’s new in Red Hat Dependency Analytics appeared first on Red Hat Developer.

October 15, 2019 02:52 PM UTC

Abhijeet Pal

Creating Comments System With Django

In this tutorial, we will build a basic commenting system for a Django 2.X app, which lets readers add comments on posts.

Here is a preview of what we are going to build by the end of this tutorial.

creating comment system with django


Before diving into the tutorial I am assuming that you already have a blog or some similar Django 2.X project up and running. I am using Bootstrap 4 for the styling part you can ignore it if you are using any other framework.

I am gonna use my previous blog project you can grab the repo from here,

Since Python 3 is the current version in active development and addressed as the future of Python, Django rolled out a significant update, and now all the releases after Django 2.0 are only compatible with Python 3.x. Therefore this tutorial is strictly for Python 3.x.

Roadmap To Build A Comment System

1. Create a model to save the comments.
2. Create a form to submit comments and validate the input data.
3. Add a view that processes the form and saves the new comment to the database.
4. Edit the post detail template to display the list of comments and the form to add a new comment.

Building Comment Model

Open The file of blog application and below the Post model create the Comment model.

class Comment(models.Model):
    post = models.ForeignKey(Post,on_delete=models.CASCADE,related_name='comments')
    name = models.CharField(max_length=80)
    email = models.EmailField()
    body = models.TextField()
    created_on = models.DateTimeField(auto_now_add=True)
    active = models.BooleanField(default=False)

    class Meta:
        ordering = ['created_on']

    def __str__(self):
        return 'Comment {} by {}'.format(self.body,

In this comment model first, we have a Foreign key relation that establishes a many-to-one relationship with the Post model, since every comment will be made on a post and each post will have multiple comments.

The related_name attribute allows us to name the attribute that we use for the relation from the related object back to this one. After defining this, we can retrieve the post of a comment object using and retrieve all comments of a post using post.comments.all() . If you don’t define the related_name attribute, Django will use the name of the model in lowercase, followed by _set (that is, comment_set ) to name the manager of the related object back to this one.

As a traditional comment system, we are accepting the commenter’s name, email and comment body as inputs. Then we have an active boolean field that is set to False to prevent spam we will manually allow all the comments posted.

The Meta class inside the model contains metadata. We tell Django to sort results in the created_on field in descending order by default when we query the database. We specify descending order using the negative prefix. By doing so, comments made recently will appear first.

The __str__() method is the default human-readable representation of the object. Django will use it in many places, such as the administration site.

Next, we need to synchronize this comment model into the database by running migrations to reflects the changes in the database.

(django) $ python makemigrations 
(django) $ python migrate

We are done with the models, now let’s include the Comment model in our Admin dashboard.

Adding Comments Model To The Administration Site

Open file and write the following code.

from django.contrib import admin
from .models import Post, Comment
class CommentAdmin(admin.ModelAdmin):
    list_display = ('name', 'body', 'post', 'created_on', 'active')
    list_filter = ('active', 'created_on')
    search_fields = ('name', 'email', 'body')
    actions = ['approve_comments']

    def approve_comments(self, request, queryset):

Going over the code @admin.register(Comment) registers the comment into the Admin area. Below the CommentAdmin class to customizes the representation of data on the screen.

The list_display attribute does what its name suggests display the properties mentioned in the tuple in the comments list for each comment.

The list_filter method will filter the comments based on the creation date and their active status and search_fields will simply search the database for the parameters provided in the tuple.

Finally, we have the actions method this will help us for approving many comment objects at once, the approve_comments method is a simple function that takes a queryset and updates the active boolean field to True.

Now create a superuser if you haven’t already and log in to the dashboard you should see the comment model there.

django comment admin dashboard

Now click on comments and create your comments.

In case you struck an error like no such table: blog_comment you might wanna delete the SQLite file and run migrations again for a quick fix.

Creating forms from models

Django offers a very rich and secure API to handle forms. Since the form input will be saved in the database models we are gonna use the Django’s ModelForm.

A common practice is to create a file inside your app directory for all the forms of an app. So create a file in your app and write the following code.

from .models import Comment
from django import forms

class CommentForm(forms.ModelForm):
    class Meta:
        model = Comment
        fields = ('name', 'email', 'body')

In the model form, we just need to provide the model name in the Meta class of the form Django will handle the form processing and validation on the basis of fields of the model.

By default, Django will generate a form dynamically from all fields of the model but we can explicitly define the fields we want the forms to have, that is what fields attribute is doing here.

Building Views

We will modify the post detail view for form processing using function based view.

from .models import Post
from .forms import CommentForm
from django.shortcuts import render, get_object_or_404

def post_detail(request, slug):
    template_name = 'post_detail.html'
    post = get_object_or_404(Post, slug=slug)
    comments = post.comments.filter(active=True)
    new_comment = None
    # Comment posted
    if request.method == 'POST':
        comment_form = CommentForm(data=request.POST)
        if comment_form.is_valid():

            # Create Comment object but don't save to database yet
            new_comment =
            # Assign the current post to the comment
   = post
            # Save the comment to the database
        comment_form = CommentForm()

    return render(request, template_name, {'post': post,
                                           'comments': comments,
                                           'new_comment': new_comment,
                                           'comment_form': comment_form})

This post detail view will show the post and all its comments, let’s break it down to see what’s happening.

First, we assigned the HTML template to a variable name template_name for future reference and we are assigning the Post object inside the post variable.

This comments = post.comments.filter(active=True) queryset retrieves all the approved comments from the database.

Since this is the same page where users will create new comments we initialized the new_comment variable by setting it to none.

Next, we have a conditional statement if a POST request is made, the comment_form variable will hold the data of user input next Django will validate the data using the is_valid() method.

If the form is valid the following actions take place.

  1. We create a new Comment object by calling the form’s save() method and assign it to the new_comment variable, but with commit=Flase which will prevent it from saving into the database right away because we still have to link it the post object
  2. We assigned the comment object to the current post
  3. Finally, save the object into the database

Else if it is a GET request we initialize the form object and pass it to the template.

Adding URL patterns for Views

Open the file of your app and map the view.

path('/', views.post_detail, name='post_detail')

Creating Templates For The Views

Let’s see what we will do in the templates.

   {% for comment in comments %}
        <div class="comments" style="padding: 10px;">
          <p class="font-weight-bold">
            {{ }}
            <span class=" text-muted font-weight-normal">
              {{ comment.created_on }}
          {{ comment.body | linebreaks }}
        {% endfor %}

Here we are using Django’s {% for %} template tag for looping over comments, then for each comment object, we are displaying the user’s name,creation date and the comment body.

 <div class="card-body">
        {% if new_comment %}
        <div class="alert alert-success" role="alert">
          Your comment is awaiting moderation
        {% else %}
        <h3>Leave a comment</h3>
        <form method="post" style="margin-top: 1.3em;">
          {{ comment_form.as_p }}
          {% csrf_token %}
          <button type="submit" class="btn btn-primary  btn-lg">Submit</button>
        {% endif %}

When a user makes a new comment we show them a message saying, “Your comment is awaiting moderation” else we render the form.

So putting the entire template all together we have this,

{% extends 'base.html' %} {% block content %}

<div class="container">
  <div class="row">
    <div class="col-md-8 card mb-4  mt-3 left  top">
      <div class="card-body">
        <h1>{% block title %} {{ post.title }} {% endblock title %}</h1>
        <p class=" text-muted">{{ }} | {{ post.created_on }}</p>
        <p class="card-text ">{{ post.content | safe }}</p>
    {% block sidebar %} {% include 'sidebar.html' %} {% endblock sidebar %}

    <div class="col-md-8 card mb-4  mt-3 ">
      <div class="card-body">
        <!-- comments -->
        <h2>{{ comments.count }} comments</h2>

        {% for comment in comments %}
        <div class="comments" style="padding: 10px;">
          <p class="font-weight-bold">
            {{ }}
            <span class=" text-muted font-weight-normal">
              {{ comment.created_on }}
          {{ comment.body | linebreaks }}
        {% endfor %}
    <div class="col-md-8 card mb-4  mt-3 ">
      <div class="card-body">
        {% if new_comment %}
        <div class="alert alert-success" role="alert">
          Your comment is awaiting moderation
        {% else %}
        <h3>Leave a comment</h3>
        <form method="post" style="margin-top: 1.3em;">
          {{ comment_form.as_p }}
          {% csrf_token %}
          <button type="submit" class="btn btn-primary  btn-lg">Submit</button>
        {% endif %}
{% endblock content %}

If you are working on a completely different project with some other CSS framework then ignore the styling.

Testing The Comment System

Save all the files and run the server and visit and visit a post detail page.

Django comment model form

Now create your comment and hit the submit button you should see the message. Now login to the Admin dashboard and approve the comment.

Django comment moderation system

Making Comment Form Crispy

Although our form works as expected, yet we can make the form look better without much changing the template using the Django crispy form library. It’s a very popular library for form managing you can check it out here –

Install it using

pip install django-crispy-forms

Add it to the installed apps list.



If you using Bootstrap 4 for styling add this in file.


Now in your template, you just need to load the crispy tag and use the crispy tag beside the form, as follows.

{% load crispy_forms_tags %}
<form method="post" style="margin-top: 1.3em;">
          {{ comment_form | crispy }}
          {% csrf_token %}
          <button type="submit" class="btn btn-primary  btn-lg">Submit</button>

Save the files and run the server. Django comment system

If you are stuck at any step, refer to this GitHub repo

The post Creating Comments System With Django appeared first on Django Central.

October 15, 2019 02:35 PM UTC

Catalin George Festila

Python 3.8.0 : New release of python development.

Good news from the python development area with the new release of python development: Python 3.7.5 Oct. 15, 2019 and Python 3.8.0 Oct. 14, 2019 Now you can use the new python version 3.8.0 from the official webpage. Major new features of the 3.8 series, compared to 3.7 - release Date: Oct. 14, 2019: PEP 572, Assignment expressions PEP 570, Positional-only arguments PEP 587, Python

October 15, 2019 02:04 PM UTC

Real Python

The Python range() Function

Python’s built-in range function is handy when you need to perform an action a specific number of times. As an experienced Pythonista, you’ve most likely used it before. But what does it do?

By the end of this course, you’ll:

Let’s get cracking!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 15, 2019 02:00 PM UTC

Stack Abuse

Uploading Files to AWS S3 with Python and Django


In the quest to build more interactive websites, we don't only relay information to users but also allow them to upload data of their own. This opens up more opportunities and more ways that our websites can serve the end-users.

By allowing users to upload files, we can allow them to share photographs, videos, or music with others or back them up for safekeeping. We can also provide the functionality to manage files and convert them into other formats through websites instead of installing native apps.

The rise of social media globally can be attributed to the ability of users to upload their files, mostly in the form of images and videos for other users to see and also as a means of communication. By enabling users to upload files to websites and platforms, means of communication have been enhanced and information can now be spread in very many different formats.

In this post, we will explore how Django handles file uploading and how we can tap into and extend this functionality with cloud storage to suit our needs.

How Django Handles File Storage

Django not only allows us to turn concepts into web applications but also provides functionality for us to handle files and allow users to upload files to our web applications for further interaction. Through forms, users can attach files to their requests and have their files uploaded and stored in our backend servers.

Before a file is saved, it is temporarily stored somewhere before being processed and stored in the intended final location. For instance, if the uploaded file is less than 2.5MB, the contents of that file will be stored in memory, then written to disk once all the operations are completed while processing it.

This makes the process fast for small files. For files larger than 2.5MB, they are first written to a temporary location as the data is being received, then once processing is complete, the file is moved to its final destination.

File behavior in Django can be customized through various settings, such as FILE_UPLOAD_MAX_MEMORY_SIZE, which allows us to modify the 2.5MB upload limit size for the files that are written to memory first and not to a temporary location. We can also configure the default permissions for the uploaded files through the FILE_UPLOAD_PERMISSIONS.

Other settings can be found in this section of the official Django documentation.

Where can we store our files?

In a Django-powered web application, we can store the uploaded files in various different locations. We can store them on our own servers where the Django code is deployed, or we can send them over to other servers which may have set up elsewhere for storage purposes.

In a bid to cut server maintenance costs and enhance performance, we can also choose not to store the uploaded files on our own servers. In this case, we can hand them over to other hosted storage providers such as AWS, Azure, or OneDrive, among others.

There are several packages that allow us to interact with the APIs provided by the various service providers that we have mentioned. They include:

For this post, we will use the Django-s3direct package to store our files on AWS's S3.

Our Application - Django Drive

We will use Django to build a web application in which we will upload content for end-users to view. This will be achieved by using the Django administration interface, which comes with the framework.

Our site will be used to sell cars and on it, we will display details and add images or videos of the cars on sale.

The images or videos of the cars on sale will be stored on S3. We will not implement user registration or login at this time for brevity.


We will use Pipenv to set up and manage our isolated environment in which we will build our Django application by running the following command to set it up using Python3:

$ pipenv install --three

With the environment set up, we can now install Django and Django-s3direct to handle our file uploads to S3:

$ pipenv install django django-s3direct

Django provides a set of commands to bootstrap our project before we start implementing the core functionality of our application. Our Django drive project will have a single application that will be the focus of this post. To achieve this, we run the following commands:

$ django-admin startproject django_drive && cd django_drive
$ django-admin startapp django_drive_app

The django-admin startproject ... command creates the project, and the django-admin startapp ... command creates the application.

The last step of our setup is to create database tables by running the migrate command:

$ python migrate

When we start our project by running the command python runserver, we are welcomed by the following page, which confirms that our setup was successful:


Since we will be uploading our files to AWS S3, we will need to set up a free-tier AWS account for demo purposes. After setting up, we can navigate to the S3 dashboard and create a new bucket that will contain our uploads.

For Django-s3direct to interact with our AWS setup, we need to provide the following credentials AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and the AWS_STORAGE_BUCKET_NAME.

Next, we will add the following to our django_drive/ file:

AWS_ACCESS_KEY_ID = 'aws-access-key-id'
AWS_SECRET_ACCESS_KEY = 'secret-access-key'
AWS_STORAGE_BUCKET_NAME = 'name-of-the-bucket'
AWS_S3_REGION_NAME = 'name-of-the-region'

    'primary_destination': {
        'key': 'uploads/',
        'allowed': ['image/jpg', 'image/jpeg', 'image/png', 'video/mp4'],

Django-s3direct allows us to specify more than one destination for our uploads, this way we can direct different files to separate S3 buckets. For this project, we will put all the uploads in one bucket. Another nifty feature is that we can also limit the file types that can be uploaded to our website. In our case, we have limited it to MP4 videos, JPEG, and PNG images only.

Note: More details about setting up Django-s3direct, such as CORS and Access Setup, can be found here.

We also need to add the following entries in the django_drive/ file:

from django.urls import path, include

urlpatterns = [
    path('', include('django_drive_app.urls')),
    path('s3direct/', include('s3direct.urls')),


We will start by creating the model for our car data, which will be displayed to the end-users. This model will also define the information we will input in our admin dashboard when adding cars to our platform. The car model will be as follows:

from django.db import models
from s3direct.fields import S3DirectField

class Car(models.Model):
    name = models.CharField(max_length=255, blank=False, null=False)
    year_of_manufacture = models.CharField(max_length=255, blank=False, null=False)
    price = models.CharField(max_length=255, blank=False, null=False)
    image = S3DirectField(dest='primary_destination', blank=True)
    video = S3DirectField(dest='primary_destination', blank=True)

    def __str__(self):
        return f"{} ({self.year_of_manufacture}) - {self.price}"

For each car, we will store its name, year of manufacture, price, and an image or video. After creating the model, let us make migrations to create the table in the database that will hold our data by running:

$ python makemigrations
$ python migrate

Since we will be using the Django admin dashboard to manage the cars on our platform, we need to register our model in the django_drive_app/

from django.contrib import admin
from.models import Car

Then we need to create the superuser who will be in charge of adding the cars by running the following command and following the prompts:

$ python createsuperuser
$ python runserver

The python runserver command simply restarts our application.

After restarting our server, we can now navigate to the administration dashboard at and log in with the credentials we specified earlier. Under site administration, we can see our DJANGO_DRIVE_APP with the option to add or change existing cars.

This is the form we use to add a car and its details:


Once we save our car, we can find the image we have uploaded in our S3 bucket on the AWS console. This means that our file has been uploaded to AWS.

Now we will create a view to display the cars and their data to the end-users of our website and also display the images or videos associated with each car. We will start by creating a view in the django_drive_app/

from django.shortcuts import render
from django.views.generic import TemplateView
from .models import Car

class CarView(TemplateView):
    template_name = 'django_drive_app/cars.html'

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['cars'] = Car.objects.all()
        return context

In this view, we use a class-based Django view to render the HTML file to display our cars. In our view, we run a query to fetch all the cars as stored in our database.

Next, let us create django_drive_app/templates/django_drive_app/cars.html to render our cars:

<!DOCTYPE html>
    <title>Django Drive</title>
    <h3>Welcome to Django Drive. </h3>
    <p>Here are the current cars available for sale: </p>
    <div class="cars-container">
      {% for car in cars %}
        <div class="car">
            <b> {{ }} ({{ car.year_of_manufacture }}) </b> <br>
            Price: {{ car.price }}
          <!-- if the car has an image attached -->
          {% if car.image %}
          <img src="{{ car.image }}" height="200" width="400"/>
          {% endif %}
          <!-- If the car has a video -->
          {% if %}
            <video width="320" height="240" controls>
                <source src="{{ }}" type="video/mp4">
              Your browser does not support the video tag.
          {% endif %}
      {% endfor %}

With the view and the template in place, let us add the endpoint that will be used to display the list of cars to the end-users by creating the django_drive_app/

from django.conf.urls import url
from .views import CarView

urlpatterns = [
  url(r'^cars/$', CarView.as_view(), name="cars"),

We import our view and add a URL entry to map the endpoint to the view that will render the cars. When we restart our server and navigate to, we encounter the following:


As we can see, we created cars with attached images and videos and had them uploaded to AWS's S3 service. The same images and videos have been rendered in our web application after being fetched from AWS.


In this article, we have created a simple Django application that allows administrators to upload files to AWS S3 through the Django administration dashboard. We rendered the uploaded files as hosted on S3 on our landing page, including videos and images of the cars that users would wish to purchase or view.

We used Django-s3direct library to handle the interaction between our Django application and AWS S3 where our files are stored. Through the Django administration application, we were able to upload files that were eventually rendered to the end users on our landing page. We were able to upload and render both images and videos.

The source code for this project is available here on GitHub.

October 15, 2019 12:24 PM UTC

Catalin George Festila

Python 3.7.4 : Testing python source code with streamlit tool.

The official webpage for this python package can be found at Let's install it with pip3 tool: [mythcat@desk proiecte_github]$ mkdir streamlit_examples [mythcat@desk proiecte_github]$ cd streamlit_examples/ [mythcat@desk streamlit_examples]$ pip3 install streamlit --user Let's try some examples. Create a file named This simple example will show a map with randoms spots: import

October 15, 2019 11:25 AM UTC

Julien Danjou

Sending Emails in Python — Tutorial with Code Examples

Sending Emails in Python — Tutorial with Code Examples

What do you need to send an email with Python? Some basic programming and web knowledge along with the elementary Python skills. I assume you’ve already had a web app built with this language and now you need to extend its functionality with notifications or other emails sending. This tutorial will guide you through the most essential steps of sending emails via an SMTP server:

  1. Configuring a server for testing (do you know why it’s important?)
  2. Local SMTP server
  3. Mailtrap test SMTP server
  4. Different types of emails: HTML, with images, and attachments
  5. Sending multiple personalized emails (Python is just invaluable for email automation)
  6. Some popular email sending options like Gmail and transactional email services

Served with numerous code examples written and tested on Python 3.7!

Sending an email using an SMTP

The first good news about Python is that it has a built-in module for sending emails via SMTP in its standard library. No extra installations or tricks are required. You can import the module using the following statement:

import smtplib

To make sure that the module has been imported properly and get the full description of its classes and arguments, type in an interactive Python session:


At our next step, we will talk a bit about servers: choosing the right option and configuring it.

An SMTP server for testing emails in Python

When creating a new app or adding any functionality, especially when doing it for the first time, it’s essential to experiment on a test server. Here is a brief list of reasons:

  1. You won’t hit your friends’ and customers’ inboxes. This is vital when you test bulk email sending or work with an email database.
  2. You won’t flood your own inbox with testing emails.
  3. Your domain won’t be blacklisted for spam.

Local SMTP server

If you prefer working in the local environment, the local SMTP debugging server might be an option. For this purpose, Python offers an smtpd module. It has a DebuggingServer feature, which will discard messages you are sending out and will print them to stdout. It is compatible with all operations systems.

Set your SMTP server to localhost:1025

python -m smtpd -n -c DebuggingServer localhost:1025

In order to run SMTP server on port 25, you’ll need root permissions:

sudo python -m smtpd -n -c DebuggingServer localhost:25

It will help you verify whether your code is working and point out the possible problems if there are any. However, it won’t give you the opportunity to check how your HTML email template is rendered.

Fake SMTP server

Fake SMTP server imitates the work of a real 3rd party web server. In further examples in this post, we will use Mailtrap. Beyond testing email sending, it will let us check how the email will  be rendered and displayed, review the message raw data as well as will provide us with a spam report. Mailtrap is very easy to set up: you will need just copy the credentials generated by the app and paste them into your code.

Sending Emails in Python — Tutorial with Code Examples

Here is how it looks in practice:

import smtplib

port = 2525
smtp_server = ""
login = "1a2b3c4d5e6f7g" # your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # your password generated by Mailtrap

Mailtrap makes things even easier. Go to the Integrations section in the SMTP settings tab and get the ready-to-use template of the simple message, with your Mailtrap credentials in it. It is the most basic option of instructing your Python script on who sends what to who is the sendmail() instance method:

Sending Emails in Python — Tutorial with Code Examples

The code looks pretty straightforward, right? Let’s take a closer look at it and add some error handling (see the comments in between). To catch errors, we use the try and except blocks.

# The first step is always the same: import all necessary components:
import smtplib
from socket import gaierror

# Now you can play with your code. Let’s define the SMTP server separately here:
port = 2525
smtp_server = ""
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

# Specify the sender’s and receiver’s email addresses:
sender = ""
receiver = ""

# Type your message: use two newlines (\n) to separate the subject from the message body, and use 'f' to  automatically insert variables in the text
message = f"""\
Subject: Hi Mailtrap
To: {receiver}
From: {sender}
This is my first message with Python."""

  # Send your message with credentials specified above
  with smtplib.SMTP(smtp_server, port) as server:
    server.login(login, password)
    server.sendmail(sender, receiver, message)
except (gaierror, ConnectionRefusedError):
  # tell the script to report if your message was sent or which errors need to be fixed
  print('Failed to connect to the server. Bad connection settings?')
except smtplib.SMTPServerDisconnected:
  print('Failed to connect to the server. Wrong user/password?')
except smtplib.SMTPException as e:
  print('SMTP error occurred: ' + str(e))

Once you get the Sent result in Shell, you should see your message in your Mailtrap inbox:

Sending Emails in Python — Tutorial with Code Examples

Sending emails with HTML content

In most cases, you need to add some formatting, links, or images to your email notifications. We can simply put all of these with the HTML content. For this purpose, Python has an email package.

We will deal with the MIME message type, which is able to combine HTML and plain text. In Python, it is handled by the email.mime module.

It is better to write a text version and an HTML version separately, and then merge them with the MIMEMultipart("alternative") instance. It means that such a message has two rendering options accordingly. In case an HTML isn’t be rendered successfully for some reason, a text version will still be available.

import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

port = 2525
smtp_server = ""
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

sender_email = ""
receiver_email = ""

message = MIMEMultipart("alternative")
message["Subject"] = "multipart test"
message["From"] = sender_email
message["To"] = receiver_email
# Write the plain text part
text = """\ Hi, Check out the new post on the Mailtrap blog: SMTP Server for Testing: Cloud-based or Local? Feel free to let us know what content would be useful for you!"""

# write the HTML part
html = """\ <html> <body> <p>Hi,<br> Check out the new post on the Mailtrap blog:</p> <p><a href="">SMTP Server for Testing: Cloud-based or Local?</a></p> <p> Feel free to <strong>let us</strong> know what content would be useful for you!</p> </body> </html> """

# convert both parts to MIMEText objects and add them to the MIMEMultipart message
part1 = MIMEText(text, "plain")
part2 = MIMEText(html, "html")

# send your email
with smtplib.SMTP("", 2525) as server:
  server.login(login, password)
  server.sendmail( sender_email, receiver_email, message.as_string() )

Sending Emails in Python — Tutorial with Code ExamplesThe resulting output

Sending Emails with Attachments in Python

The next step in mastering sending emails with Python is attaching files. Attachments are still the MIME objects but we need to encode them with the base64 module. A couple of important points about the attachments:

  1. Python lets you attach text files, images, audio files, and even applications. You just need to use the appropriate email class like or email.mime.image.MIMEImage. For the full information, refer to this section of the Python documentation.
  2. Remember about the file size: sending files over 20MB is a bad practice.

In transactional emails, the PDF files are the most frequently used: we usually get receipts, tickets, boarding passes, order confirmations, etc. So let’s review how to send a boarding pass as a PDF file.

import smtplib
from email import encoders
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

port = 2525
smtp_server = ""
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

subject = "An example of boarding pass"
sender_email = ""
receiver_email = ""

message = MIMEMultipart()
message["From"] = sender_email
message["To"] = receiver_email
message["Subject"] = subject

# Add body to email
body = "This is an example of how you can send a boarding pass in attachment with Python"
message.attach(MIMEText(body, "plain"))

filename = "yourBP.pdf"
# Open PDF file in binary mode
# We assume that the file is in the directory where you run your Python script from
with open(filename, "rb") as attachment:
# The content type "application/octet-stream" means that a MIME attachment is a binary file
part = MIMEBase("application", "octet-stream")
# Encode to base64
# Add header
part.add_header("Content-Disposition", f"attachment; filename= {filename}")
# Add attachment to your message and convert it to string

text = message.as_string()
# send your email
with smtplib.SMTP("", 2525) as server:
  server.login(login, password)
  server.sendmail(sender_email, receiver_email, text)

Sending Emails in Python — Tutorial with Code ExamplesThe received email with your PDF

To attach several files, you can call the message.attach() method several times.

How to send an email with image attachment

Images, even if they are a part of the message body, are attachments as well. There are three types of them: CID attachments (embedded as a MIME object), base64 images (inline embedding), and linked images.

For adding a CID attachment, we will create a MIME multipart message with MIMEImage component:

import smtplib
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart

port = 2525
smtp_server = ""
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

sender_email = ""
receiver_email = ""

message = MIMEMultipart("alternative")
message["Subject"] = "CID image test"
message["From"] = sender_email
message["To"] = receiver_email

# write the HTML part
html = """\
<img src="cid:myimage">
part = MIMEText(html, "html")

# We assume that the image file is in the same directory that you run your Python script from
with open('mailtrap.jpg', 'rb') as img:
  image = MIMEImage(
# Specify the  ID according to the img src in the HTML part
image.add_header('Content-ID', '<myimage>')

# send your email
with smtplib.SMTP("", 2525) as server:
  server.login(login, password)
  server.sendmail(sender_email, receiver_email, message.as_string())

Sending Emails in Python — Tutorial with Code ExamplesThe received email with CID image

The CID image is shown both as a part of the HTML message and as an attachment. Messages with this image type are often considered spam: check the Analytics tab in Mailtrap to see the spam rate and recommendations on its improvement. Many email clients — Gmail in particular — don’t display CID images in most cases. So let’s review how to embed a base64 encoded image instead.

Here we will use base64 module and experiment with the same image file:

import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
import base64

port = 2525
smtp_server = ""
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap
sender_email = ""
receiver_email = ""

message = MIMEMultipart("alternative")
message["Subject"] = "inline embedding"
message["From"] = sender_email
message["To"] = receiver_email

# We assume that the image file is in the same directory that you run your Python script from
with open("image.jpg", "rb") as image:
  encoded = base64.b64encode(

html = f"""\
<img src="data:image/jpg;base64,{encoded}">
part = MIMEText(html, "html")

# send your email
with smtplib.SMTP("", 2525) as server:
  server.login(login, password)
  server.sendmail(sender_email, receiver_email, message.as_string())

Sending Emails in Python — Tutorial with Code ExamplesA base64 encoded image

Now the image is embedded into the HTML message and is not available as an attached file. Python has encoded our JPEG image, and if we go to the HTML Source tab, we will see the long image data string in the img src attribute.

How to Send Multiple Emails

Sending multiple emails to different recipients and making them personal is the special thing about emails in Python.

To add several more recipients, you can just type their addresses in separated by a comma, add Cc and Bcc. But if you work with a bulk email sending, Python will save you with loops.

One of the options is to create a database in a CSV format (we assume it is saved to the same folder as your Python script).

We often see our names in transactional or even promotional examples. Here is how we can make it with Python.

Let’s organize the list in a simple table with just two columns: name and email address. It should look like the following example:

John Johnson,
Peter Peterson,

The code below will open the file and loop over its rows line by line, replacing the {name} with the value from the “name” column.

import csv
import smtplib

port = 2525
smtp_server = ""
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

message = """Subject: Order confirmation
To: {recipient}
From: {sender}
Hi {name}, thanks for your order! We are processing it now and will contact you soon"""
sender = ""
with smtplib.SMTP("", 2525) as server:
  server.login(login, password)
  with open("contacts.csv") as file:
  reader = csv.reader(file)
  next(reader)  # it skips the header row
  for name, email in reader:
      message.format(name=name, recipient=email, sender=sender),
    print(f'Sent to {name}')

In our Mailtrap inbox, we see two messages: one for John Johnson and another for Peter Peterson, delivered simultaneously:

Sending Emails in Python — Tutorial with Code Examples

Sending emails with Python via Gmail

When you are ready for sending emails to real recipients, you can configure your production server. It also depends on your needs, goals, and preferences: your localhost or any external SMTP.

One of the most popular options is Gmail so let’s take a closer look at it.

We can often see titles like “How to set up a Gmail account for development”. In fact, it means that you will create a new Gmail account and will use it for a particular purpose.

To be able to send emails via your Gmail account, you need to provide access to it for your application. You can Allow less secure apps or take advantage of the OAuth2 authorization protocol. It’s a way more difficult but recommended due to the security reasons.

Further, to use a Gmail server, you need to know:

import smtplib
import ssl

port = 465
password = input("your password")
context = ssl.create_default_context()

with smtplib.SMTP_SSL("", port, context=context) as server:
  server.login("", password)

If you tend to simplicity, then you can use Yagmail, the dedicated Gmail/SMTP. It makes email sending really easy. Just compare the above examples with these several lines of code:

import yagmail

yag = yagmail.SMTP()
contents = [
"This is the body, and here is just text http://somedomain/image.png",
"You can find an audio file attached.", '/local/path/to/song.mp3'
yag.send('', 'subject', contents)

Next steps with Python

Those are just basic options of sending emails with Python. To get great results, review the Python documentation and experiment with your own code!

There are a bunch of various Python frameworks and libraries, which make creating apps more elegant and dedicated. In particular, some of them can help improve your experience with building emails sending functionality:

The most popular frameworks are:

  1. Flask, which offers a simple interface for email sending: Flask Mail.
  2. Django, which can be a great option for building HTML templates.
  3. Zope comes in handy for a website development.
  4. Marrow Mailer is a dedicated mail delivery framework adding various helpful configurations.
  5. Plotly and its Dash can help with mailing graphs and reports.

Also, here is a handy list of Python resources sorted by their functionality.

Good luck and don’t forget to stay on the safe side when sending your emails!

This article was originally published at Mailtrap’s blog: Sending emails with Python

October 15, 2019 10:33 AM UTC

S. Lott

Apple's Numbers and the All-in-One CSV export

Author F. L. Stevens has a hellishly complex (and irregular) spreadsheet with agents, agencies, and query status. (This is how fiction gets marketed: querying agents.) The spreadsheet has become unmanageably complex, with multiple pages. Each page has multiple tables. Buried in this are three "interesting" tables with agent query information.

Can we talk about drama? There is the dark night of the soul for anyone interested in regular, normalized data.

We have some fundamental choices for working with this mess:

  1. Export each relevant table to separate files. Lots of manual pointy-clicky and opportunities for making mistakes.
  2. Export the whole thing to separate files. Less pointy-clicky.
  3. Export the whole thing to one file. About the same pointy-clicky and error vulnerability as #2. But. Simpler still because there's one file to take care of. Something a fiction author should be able to handle.
The all-in-one CSV export is (initially) exasperating. Each table is wrapped in a prefix and suffix.
The prefix is a line with "Sheet: Table" Yes. There's a ": " (colon space) separator. The suffix is a simple blank line, essentially indistinguishable from a blank line within a table.

If the table was originally in strict first normal form (1NF) each row would have the same number of commas. If cells are merged, however, the number of commas can be fewer. This makes it potentially difficult to distinguish blank rows in a table from blank lines between tables.

It's generally easiest to ignore the blank lines entirely. We can distinguish table headers because they're a single cell with a sheet: table format. We are left hoping there aren't any tables that have values that have this format.

We have two ways to walk through the values:
The hierarchical form requires a number of generator functions for Sheet-from-CSV, Table-from-CSV, and Row-from-CSV. Each of these works with a single underlying iterator over the source file and a fairly complex hand-off of state. If we only use the sheet iterator, the tables and rows are skipped. If we use the table within a sheet, the first table name comes from the header that started a sheet; the table names come from distinct headers until the sheet name changes. 

The table-within-sheet iteration is very tricky. The first table is a simple yield of information gathered by the sheet iterator. Any subsequent tables, however, may be based one one of two conditions: either no rows have been consumed, in which case the table iterator consumes (and ignores) rows; or, all the rows of the table have been consumed and the current row is another "sheet: table" header. 

The code sample below involves a fair amount of repetition. It's not appealing to refactor this because it's ungainly in its complexity, and doesn't create any tangible value. (I haven't even tried to get the type hints right.)

class SheetTable:
def __init__(self, source_path: Path) -> None:
self.path: Path = source_path
self.csv_source = None
self.rdr = None
self.header = None
self.row = None

def __enter__(self) -> None:
self.csv_source =
self.rdr = csv.reader(self.csv_source)
self.header = None
self.row = next(self.rdr)
return self

def __exit__(self, *args) -> None:

def _sheet_header(self) -> bool:
return len(self.row) == 1 and ': ' in self.row[0]

def sheet_iter(self):
while True:
while not (self._sheet_header()):
self.row = next(self.rdr)
except StopIteration:
self.sheet, _, self.table = self.row[0].partition(": ")
self.header = next(self.rdr)
self.row = next(self.rdr)
yield self.sheet, self.table_iter()

def table_iter(self):
yield self.table, self.row_iter()
while not (self._sheet_header()):
self.row = next(self.rdr)
except StopIteration:
next_sheet, _, next_table = self.row[0].partition(": ")
while next_sheet == self.sheet:
self.table = next_table
self.header = next(self.rdr)
self.row = next(self.rdr)
yield self.table, self.row_iter()
while not (self._sheet_header()):
self.row = next(self.rdr)
except StopIteration:
next_sheet, _, next_table = self.row[0].partition(": ")

def row_iter(self):
while not self._sheet_header():
yield dict(zip(self.header, self.row))
self.row = next(self.rdr)
except StopIteration:

Clearly, this is craziness.

Flattening is much nicer.

def sheet_table_iter(source_path: Path) -> Iterator[Tuple[str, str, Dict[str, Any]]]:
with as csv_source:
rdr = csv.reader(csv_source)
header = None
for row in rdr:
if len(row) == 0:
elif len(row) == 1 and ": " in row[0]:
sheet, table = row[0].split(": ", maxsplit=1)
header = next(rdr)
# Inject headers to create dict from row
yield sheet, table, dict(zip(header, row))

This provides a relatively simple way to find the relevant tables and sheets. We can use something as simple as the following to locate the relevant data.

    for sheet, table, row in sheet_table_iter(source_path):
if sheet == 'AgentQuery' and table == 'agent_query':
agent = agent_query_row(database, row)
elif sheet == 'AAR-2019-03' and table == 'Table 1':
agent = aar_2019_row(database, row)

This lets us write pleasant functions that handle exactly one row from the source table. We'll have one of these for each target table. In the above example, we've only shown two, you get the idea. Each new source table, with its unique headers can be accommodated.

October 15, 2019 08:00 AM UTC

Talk Python to Me

#234 Awesome Python Applications

Have you heard of awesome lists? They are well, pretty awesome! Gathering up the most loved libraries and packages for a given topic.

October 15, 2019 08:00 AM UTC

Python Bytes

#152 You have 35 million lines of Python 2, now what?

October 15, 2019 08:00 AM UTC