Planet Python

Last update: October 17, 2019 01:48 PM UTC

October 17, 2019

Abhijeet Pal

Using PostgreSQL with Django

Django is a high level full-stack open-source web framework written in Python, that encourages rapid development and clean, pragmatic design.

Django, in its ‘out-of-the-box’ state, is set up to communicate with SQLite – a lightweight relational database included with the Python distribution. So by default, Django automatically creates an SQLite database for your project.

In addition to SQLite, Django also has support for other popular databases that include PostgreSQL, MySQL, and Oracle.

However, PostgreSQL has a number of features that are not shared by the other databases Django supports, which makes it an idle choice for a Django app in production.

In this article, we will go through the integration of PostgreSQL with a Django Application.

Pre-Requirements

We are assuming you already have Django installed on your machine and one Django project up and running, if not then read the following article – Starting A Django Project

Installing PostgreSQL

Windows and macOS X users can download PostgreSQL from the official site https://www.postgresql.org/download/ and simply install it.

Linux User

sudo apt-get install postgresql postgresql-contrib

Also, Linux users need to install some dependencies for PostgreSQL to work with Python.

sudo apt-get install libpq-dev python-dev

Install psycopg2

Next, we need to install the PostgreSQL database adapter to communicate to the database with Python to install it run the following command in the shell.

pip install psycopg2

Create A PostgreSQL User and Database

As the default configuration of Postgres is, a user called Postgres is made on, and the user Postgres has full super admin access to entire PostgreSQL instance running on your OS.

sudo -u postgres psql

Now the terminal should be prefixed with postgres=# , The above command gets you the psql command-line interface in full admin mode.

Now let’s create a user and database.

Creating Database

 CREATE DATABASE mydb;

This will create a database named mydb, note that every SQL statement must end with a semicolon.

Creating User

 CREATE USER myuser WITH ENCRYPTED PASSWORD 'mypass';

Here we are creating a user named myuser with password mypass. You can use any username and password you wish.

Modifying Connection Parameters

 ALTER ROLE myuser SET client_encoding TO 'utf8';
 ALTER ROLE myuser SET default_transaction_isolation TO 'read committed';
 ALTER ROLE myuser SET timezone TO 'UTC';

We are setting the default encoding to UTF-8, which Django expects.

We are also setting the default transaction isolation scheme to “read committed”, which blocks reads from uncommitted transactions.

Lastly, we are setting the timezone by default, our Django projects will be set to use UTC.These are essential parameters recommended by the official Django team.

Granting Permission To The User

 GRANT ALL PRIVILEGES ON DATABASE mydb TO myuser;

Now our user has administrative access to the database.

Now exit the SQL prompt.

\q

Integrating PostgreSQL With Django

Open the settings.py file of your project and scroll straight to the database section, which should look like this.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

We need to update these settings to integrate our PostgreSQL with the project.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'mydb',
        'USER': 'myuser',
        'PASSWORD': 'mypass',
        'HOST': 'localhost',
        'PORT': '',
    }
}

Let’s quickly go over the settings,

DATABASES – This constant is a dictionary of database connection information and is required by Django. You can have multiple connections to different databases, but most of the time, you will just need an entry called default.

default – This is the default database connection configuration. You should always have a default set of connections settings.

'ENGINE': 'django.db.backends.postgresql_psycopg2' – This tells Django to use the Postgres backend. This, in turn uses psycopg2, Python’s Postgres library which we installed earlier.

'NAME': 'mydb' – The name of the database you want to connect to.

'USER': 'myuser' – The User with access to the database.

'PASSWORD': 'mypass' – The password for your database user.

'HOST': 'localhost' – The address of the database server you want to connect to.

'PORT': '' – The port you want to connect to, which by default is ‘5432’

Test The Database Connection

After updating the database configurations, it’s time to test the connection. The Django database migration process ensures all Django project logic associated with a database is reflected in the database itself.

During the first migration against the database, there are a series of migrations Django requires that create tables to keep track of administrators and sessions.

In the directory where manage.py script exists, run the following command.

python manage.py migrate

If everything went right you should see an output like this.

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying sessions.0001_initial... OK

Furthermore, you can now create a superuser and login to the admin dashboard.

The post Using PostgreSQL with Django appeared first on Django Central.

October 17, 2019 11:51 AM UTC

Creating A Super User In Django

Django’s prominent feature is the admin interface, which makes it stand out from the competition. It is a built-in app that automatically generates a user interface to add and modify a site’s content.

The built-in administration interface that is very useful for editing content. The Django admin site is built dynamically by reading your model metadata and providing a production-ready interface for editing content. You can use it out of the box, configuring how you want your models to be displayed in it.

In order to use the Django’s admin app which is already included in django.contrib.admin inside INSTALLED_APPS setting, first we have to create a superuser.

Creating A Super User In Django

In the directory where manage.py script exists, execute the following command.

python manage.py createsuperuser

Now Django will prompt you to enter the details, enter your desired details and hit enter.

Username (leave blank to use 'admin'): admin
Email address: admin@xyz.com
Password: ********
Password (again): ********
Superuser created successfully.

Now that the superuser is created, run the development server.

python manange.py runserver

Open the browser and navigate to http://127.0.0.1:8000/admin/ You should see a login page, enter the details you provided for the superuser.

After you log in you should see  the Django’s admin panel with Groups and Users models which come from Django authentication framework located in django.contrib.auth.

The post Creating A Super User In Django appeared first on Django Central.

October 17, 2019 07:10 AM UTC

Janusworx

Multiple Slugs for Multiple Webpages in Nikola

October 17, 2019 04:09 AM UTC

October 16, 2019

Roberto Alsina

Episodio 12: Buen Gusto

October 16, 2019 11:00 PM UTC

Full Stack Python

Basic Data Types in Python 3: Strings

There is a lot to learn on your Python journey when you are new to the programming language. Once you are comfortable writing and executing code, your first stop becomes understanding how to represent data in your code. No matter the language, there are a few basic data types you'll use all the time - strings, numbers, booleans, lists, and dictionaries.

Those data types, and how to use them in Python 3, are the topic of this blog post series. Today, we're starting with strings.

If you're learning Python, you might also want to check out TwilioQuest 3. You'll learn about basic data types and much more about Python programming.

Ready to learn how to use strings in Python 3? Let's get started!

Strings in Python 3

One of the most common data types in any programming language is a string. A string represents a series of characters, which you would use to represent usernames, blog posts, tweets, or any text content in your code. You can create a string and assign it to a variable like this.

my_name = "Jonathan Joestar"

Strings are "immutable"

In Python, strings are considered immutable - once you create them, they can't be changed. You can, however, use a variety of methods to create new strings from existing strings. This type of work in programming is called string manipulation. Some web developers joke that at the end of the day, their job is just mashing strings together - and this isn't far from the truth!

Here are some common tasks you might undertake when using strings in your code.

Common task - combining strings together

Combining strings together - concatenating them - is a very common task. In Python 3, you can use the + operator for this purpose. You can use the + operator multiple times to concatenate multiple strings.

first_name = "Jonathan"
last_name = "Joestar"

full_name = first_name + " " + last_name

Common task - inserting data into strings

Another common task with strings is inserting data into a specific place within a string. In programming, we call this string interpolation. Python 3 provides a handy tool for doing this called "f" strings. The "f" in "f strings" stands for format - you can insert other data from your program into a string when you define it rather than doing complex string concatenation as demonstrated previously.

Here is an example of creating a formatted string - note the letter f is included just before the first double quote when defining the message variable. When you want to insert data from your program into the string, you can include it between two "curly braces" - the { and } characters.

first_name = "Jonathan"
last_name = "Joestar"
age = 24

message = f"My name is {first_name} {last_name}, and I am {age} years old."
print(message)

Common task - using built-in string methods to manipulate strings

String objects have a number of methods to perform common tasks, like changing the case of strings or trimming their content. Below, you'll find a few examples. In two of these examples, we are creating a string variable, and then assigning the same variable a new value, which is the result of calling a method on a string object.

Example 1: Convert a string to all caps using the upper method.

example_string = "am I stoked enough yet?"
example_string = example_string.upper()
print(example_string) # prints "AM I STOKED ENOUGH YET?"

Example 2: Replace all instances of the word kale with tacos.

example_string = "We're having kale for dinner! Yay kale!"
example_string = example_string.replace("kale", "tacos")
print(example_string) # prints "We're having tacos for dinner! Yay tacos!"

Example 3: Split a comma-delimited string into a list of strings.

example_string = "Apples,Oranges,Pears"
groceries = example_string.split(',')

# Code below prints:
# Apples
# Oranges
# Pears
for item in groceries:
    print(item)

Check our more strings can do in the Python 3 docs!

Type casting

Frequently, you will want to convert data from one type into another. In programming, we call this process type casting. There are a number of functions built in to Python which allow us to do these type conversions on basic data types.

Example 1: Convert a number into a string using the str function.

example_number = 42
converted = str(example_number)
message = "The meaning of life is " + converted

Example 2: Convert a string into a whole number (integer) using int.

example_string = "2"
converted = int(example_string)
message = f"Two plus two equals { converted + 2 }"

Wrapping up

Strings of text are one of the most common pieces of data you will work with in programming. Hopefully, you've learned a bit about how to work with strings in Python 3! Stay tuned for more blog posts in this series to learn more about basic data types like strings, numbers, booleans, lists, and dictionaries.

Also, be sure to download and play TwilioQuest 3 to learn even more about Python!

October 16, 2019 09:40 PM UTC

Anarcat

Theory: average bus factor = 1

Two articles recently made me realize that all my free software projects basically have a bus factor of one. I am the sole maintainer of every piece of software I have ever written that I still maintain. There are projects that I have been the maintainer of which have other maintainers now (most notably AlternC, Aegir and Linkchecker), but I am not the original author of any of those projects.

Now that I have a full time job, I feel the pain. Projects like Gameclock, Monkeysign, Stressant, and (to a lesser extent) Wallabako all need urgent work: the first three need to be ported to Python 3, the first two to GTK 3, and the latter will probably die because I am getting a new e-reader. (For the record, more recent projects like undertime and feed2exec are doing okay, mostly because they were written in Python 3 from the start, and the latter has extensive unit tests. But they do suffer from the occasional bitrot (the latter in particular) and need constant upkeep.)

Now that I barely have time to keep up with just the upkeep, I can't help but think all of my projects will just die if I stop working on them. I have the same feeling about the packages I maintain in Debian.

What does that mean? Does that mean those packages are useless? That no one cares enough to get involved? That I'm not doing a good job at including contributors?

I don't think so. I think I'm a friendly person online, and I try my best at doing good documentation and followup on my projects. What I have come to understand is even more depressing and scary that this being a personal failure: that is the situation with everyone, everywhere. The LWN article is not talking about silly things like a chess clock or a feed reader: we're talking about the Linux input drivers. A very deep, core component of the vast majority of computers running on the planet, that depend on that single maintainer. And I'm not talking about whether those people are paid or not, that's related, but not directly the question here. The same realization occured with OpenSSL and NTP, GnuPG is in a similar situation, the list just goes on and on.

A single guy maintains those projects! Is that a fluke? A statistical anomaly? Everything I feel, and read, and know in my decades of experience with free software show me a reality that I've been trying to deny for all that time: it's the average.

My theory is this: our average bus factor is one. I don't have any hard evidence to back this up, no hard research to rely on. I'd love to be proven wrong. I'd love for this to change.

But unless economics of technology production change significantly in the coming decades, this problem will remain, and probably worsen, as we keep on scaffolding an entire civilization on shoulders of hobbyists that are barely aware their work is being used to power phones, cars, airplanes and hospitals. A lot has been written on this, but nothing seems to be moving.

And if that doesn't scare you, it damn well should. As a user, one thing you can do is, instead of wondering if you should buy a bit of proprietary software, consider using free software and donating that money to free software projects instead. Lobby governments and research institutions to sponsor only free software projects. Otherwise this civilization will collapse in a crash of spaghetti code before it even has time to get flooded over.

October 16, 2019 07:21 PM UTC

Stories in My Pocket

PyCon 2019: Open Spaces

There have been a few thoughts about PyCon US 2019 that have been bouncing around my head, wanting to come out. Today, I want to talk about PyCon's Open Spaces, and why you should start planning on attending next year.

October 16, 2019 04:48 PM UTC

Real Python

Python sleep(): How to Add Time Delays to Your Code

Have you ever needed to make your Python program wait for something? Most of the time, you’d want your code to execute as quickly as possible. But there are times when letting your code sleep for a while is actually in your best interest.

For example, you might use a Python sleep() call to simulate a delay in your program. Perhaps you need to wait for a file to upload or download, or for a graphic to load or be drawn to the screen. You might even need to pause between calls to a web API, or between queries to a database. Adding Python sleep() calls to your program can help in each of these cases, and many more!

In this tutorial, you’ll learn how to add Python sleep() calls with:

• time.sleep()
• Decorators
• Threads
• Async IO
• Graphical User Interfaces

This article is intended for intermediate developers who are looking to grow their knowledge of Python. If that sounds like you, then let’s get started!

Adding a Python sleep() Call With time.sleep()

Python has built-in support for putting your program to sleep. The time module has a function sleep() that you can use to suspend execution of the calling thread for however many seconds you specify.

Here’s an example of how to use time.sleep():

>>> import time
>>> time.sleep(3) # Sleep for 3 seconds

If you run this code in your console, then you should experience a delay before you can enter a new statement in the REPL.

You can test how long the sleep lasts by using Python’s timeit module:

$ python3 -m timeit -n 3 "import time; time.sleep(3)"
3 loops, best of 3: 3 sec per loop

Here, you run the timeit module with the -n parameter, which tells timeit how many times to run the statement that follows. You can see that timeit ran the statement 3 times and that the best run time was 3 seconds, which is what was expected.

The default number of times that timeit will run your code is one million. If you were to run the above code with the default -n, then at 3 seconds per iteration, your terminal would hang for approximately 34 days! The timeit module has several other command line options that you can check out in its documentation.

Let’s create something a bit more realistic. A system administrator needs to know when one of their websites goes down. You want to be able to check the website’s status code regularly, but you can’t query the web server constantly or it will affect performance. One way to do this check is to use a Python sleep() system call:

import time
import urllib.request
import urllib.error

def uptime_bot(url):
    while True:
        try:
            conn = urllib.request.urlopen(url)
        except urllib.error.HTTPError as e:
            # Email admin / log
            print(f'HTTPError: {e.code} for {url}')
        except urllib.error.URLError as e:
            # Email admin / log
            print(f'URLError: {e.code} for {url}')
        else:
            # Website is up
            print(f'{url} is up')
        time.sleep(60)

if __name__ == '__main__':
    url = 'http://www.google.com/py'
    uptime_bot(url)

Here you create uptime_bot(), which takes a URL as its argument. The function then attempts to open that URL with urllib. If there’s an HTTPError or URLError, then the program catches it and prints out the error. (In a live environment, you would log the error and probably send out an email to the webmaster or system administrator.)

If no errors occur, then your code prints out that all is well. Regardless of what happens, your program will sleep for 60 seconds. This means that you only access the website once every minute. The URL used in this example is bad, so it will output the following to your console once every minute:

HTTPError: 404 for http://www.google.com/py

Go ahead and update the code to use a known good URL, like http://www.google.com. Then you can re-run it to see it work successfully. You can also try to update the code to send an email or log the errors. For more information on how to do this, check out Sending Emails With Python and Logging in Python.

Adding a Python sleep() Call With Decorators

There are times when you need to retry a function that has failed. One popular use case for this is when you need to retry a file download because the server was busy. You usually won’t want to make a request to the server too often, so adding a Python sleep() call between each request is desirable.

Another use case that I’ve personally experienced is where I need to check the state of a user interface during an automated test. The user interface might load faster or slower than usual, depending on the computer I’m running the test on. This can change what’s on the screen at the moment my program is verifying something.

In this case, I can tell the program to sleep for a moment and then recheck things a second or two later. This can mean the difference between a passing and failing test.

You can use a decorator to add a Python sleep() system call in either of these cases. If you’re not familiar with decorators, or if you’d like to brush up on them, then check out Primer on Python Decorators. Let’s look at an example:

import time
import urllib.request
import urllib.error

def sleep(timeout, retry=3):
    def the_real_decorator(function):
        def wrapper(*args, **kwargs):
            retries = 0
            while retries < retry:
                try:
                    value = function(*args, **kwargs)
                    if value is None:
                        return
                except:
                    print(f'Sleeping for {timeout} seconds')
                    time.sleep(timeout)
                    retries += 1
        return wrapper
    return the_real_decorator

sleep() is your decorator. It accepts a timeout value and the number of times it should retry, which defaults to 3. Inside sleep() is another function, the_real_decorator(), which accepts the decorated function.

Finally, the innermost function wrapper() accepts the arguments and keyword arguments that you pass to the decorated function. This is where the magic happens! You use a while loop to retry calling the function. If there’s an exception, then you call time.sleep(), increment the retries counter, and try running the function again.

Now rewrite uptime_bot() to use your new decorator:

@sleep(3)
def uptime_bot(url):
    try:
        conn = urllib.request.urlopen(url)
    except urllib.error.HTTPError as e:
        # Email admin / log
        print(f'HTTPError: {e.code} for {url}')
        # Re-raise the exception for the decorator
        raise urllib.error.HTTPError
    except urllib.error.URLError as e:
        # Email admin / log
        print(f'URLError: {e.code} for {url}')
        # Re-raise the exception for the decorator
        raise urllib.error.URLError
    else:
        # Website is up
        print(f'{url} is up')

if __name__ == '__main__':
    url = 'http://www.google.com/py'
    uptime_bot(url)

Here, you decorate uptime_bot() with a sleep() of 3 seconds. You’ve also removed the original while loop, as well as the old call to sleep(60). The decorator now takes care of this.

One other change you’ve made is to add a raise inside of the exception handling blocks. This is so that the decorator will work properly. You could write the decorator to handle these errors, but since these exceptions only apply to urllib, you might be better off keeping the decorator the way it is. That way, it will work with a wider variety of functions.

There are a few improvements that you could make to your decorator. If it runs out of retries and still fails, then you could have it re-raise the last error. The decorator will also wait 3 seconds after the last failure, which might be something you don’t want to happen. Feel free to try these out as an exercise!

Adding a Python sleep() Call With Threads

There are also times when you might want to add a Python sleep() call to a thread. Perhaps you’re running a migration script against a database with millions of records in production. You don’t want to cause any downtime, but you also don’t want to wait longer than necessary to finish the migration, so you decide to use threads.

To prevent customers from noticing any kind of slowdown, each thread needs to run for a short period and then sleep. There are two ways to do this:

1. Use time.sleep() as before.
2. Use Event.wait() from the threading module.

Let’s start by looking at time.sleep().

Using time.sleep()

The Python Logging Cookbook shows a nice example that uses time.sleep(). Python’s logging module is thread-safe, so it’s a bit more useful than print() statements for this exercise. The following code is based on this example:

import logging
import threading
import time

def worker(arg):
    while not arg["stop"]:
        logging.debug("worker thread checking in")
        time.sleep(1)

def main():
    logging.basicConfig(
        level=logging.DEBUG,
        format="%(relativeCreated)6d %(threadName)s %(message)s"
    )
    info = {"stop": False}
    thread = threading.Thread(target=worker, args=(info,))
    thread_two = threading.Thread(target=worker, args=(info,))
    thread.start()
    thread_two.start()

    while True:
        try:
            logging.debug("Checking in from main thread")
            time.sleep(0.75)
        except KeyboardInterrupt:
            info["stop"] = True
            logging.debug('Stopping')
            break
    thread.join()
    thread_two.join()

if __name__ == "__main__":
    main()

Here, you use Python’s threading module to create two threads. You also create a logging object that will log the threadName to stdout. Next, you start both threads and initiate a loop to log from the main thread every so often. You use KeyboardInterrupt to catch the user pressing Ctrl+C.

Try running the code above in your terminal. You should see output similar to the following:

 0 Thread-1 worker thread checking in
 1 Thread-2 worker thread checking in
 1 MainThread Checking in from main thread
752 MainThread Checking in from main thread
1001 Thread-1 worker thread checking in
1001 Thread-2 worker thread checking in
1502 MainThread Checking in from main thread
2003 Thread-1 worker thread checking in
2003 Thread-2 worker thread checking in
2253 MainThread Checking in from main thread
3005 Thread-1 worker thread checking in
3005 MainThread Checking in from main thread
3005 Thread-2 worker thread checking in

As each thread runs and then sleeps, the logging output is printed to the console. Now that you’ve tried an example, you’ll be able to use these concepts in your own code.

Using Event.wait()

The threading module provides an Event() that you can use like time.sleep(). However, Event() has the added benefit of being more responsive. The reason for this is that when the event is set, the program will break out of the loop immediately. With time.sleep(), your code will need to wait for the Python sleep() call to finish before the thread can exit.

The reason you’d want to use wait() here is because wait() is non-blocking, whereas time.sleep() is blocking. What this means is that when you use time.sleep(), you’ll block the main thread from continuing to run while it waits for the sleep() call to end. wait() solves this problem. You can read more about how all this works in Python’s threading documentation.

Here’s how you add a Python sleep() call with Event.wait():

import logging
import threading

def worker(event):
    while not event.isSet():
        logging.debug("worker thread checking in")
        event.wait(1)

def main():
    logging.basicConfig(
        level=logging.DEBUG,
        format="%(relativeCreated)6d %(threadName)s %(message)s"
    )
    event = threading.Event()

    thread = threading.Thread(target=worker, args=(event,))
    thread_two = threading.Thread(target=worker, args=(event,))
    thread.start()
    thread_two.start()

    while not event.isSet():
        try:
            logging.debug("Checking in from main thread")
            event.wait(0.75)
        except KeyboardInterrupt:
            event.set()
            break

if __name__ == "__main__":
    main()

In this example, you create threading.Event() and pass it to worker(). (Recall that in the previous example, you instead passed a dictionary.) Next, you set up your loops to check whether or not event is set. If it’s not, then your code prints a message and waits a bit before checking again. To set the event, you can press Ctrl+C. Once the event is set, worker() will return and the loop will break, ending the program.

Take a closer look at the code block above. How would you pass in a different sleep time to each worker thread? Can you figure it out? Feel free to tackle this exercise on your own!

Adding a Python sleep() Call With Async IO

Asynchronous capabilities were added to Python in the 3.4 release, and this feature set has been aggressively expanding ever since. Asynchronous programming is a type of parallel programming that allows you to run multiple tasks at once. When a task finishes, it will notify the main thread.

asyncio is a module that lets you add a Python sleep() call asynchronously. If you’re unfamiliar with Python’s implementation of asynchronous programming, then check out Async IO in Python: A Complete Walkthrough and Python Concurrency & Parallel Programming.

Here’s an example from Python’s own documentation:

import asyncio

async def main():
    print('Hello ...')
    await asyncio.sleep(1)
    print('... World!')

# Python 3.7+
asyncio.run(main())

In this example, you run main() and have it sleep for one second between two print() calls.

Here’s a more compelling example from the Coroutines and Tasks portion of the asyncio documentation:

import asyncio
import time

async def output(sleep, text):
    await asyncio.sleep(sleep)
    print(text)

async def main():
    print(f"Started: {time.strftime('%X')}")
    await output(1, 'First')
    await output(2, 'Second')
    await output(3, 'Third')
    print(f"Ended: {time.strftime('%X')}")

# Python 3.7+
asyncio.run(main())

In this code, you create a worker called output() that takes in the number of seconds to sleep and the text to print out. Then, you use Python’s await keyword to wait for the output() code to run. await is required here because output() has been marked as an async function, so you can’t call it like you would a normal function.

When you run this code, your program will execute await 3 times. The code will wait for 1, 2, and 3 seconds, for a total wait time of 6 seconds. You can also rewrite the code so that the tasks run in parallel:

import asyncio
import time

async def output(text, sleep):
    while sleep > 0:
        await asyncio.sleep(1)
        print(f'{text} counter: {sleep} seconds')
        sleep -= 1

async def main():
    task_1 = asyncio.create_task(output('First', 1))
    task_2 = asyncio.create_task(output('Second', 2))
    task_3 = asyncio.create_task(output('Third', 3))
    print(f"Started: {time.strftime('%X')}")
    await task_1
    await task_2
    await task_3                                 
    print(f"Ended: {time.strftime('%X')}")

if __name__ == '__main__':
    asyncio.run(main())

Now you’re using the concept of tasks, which you can make with create_task(). When you use tasks in asyncio, Python will run the tasks asynchronously. So, when you run the code above, it should finish in 3 seconds total instead of 6.

Adding a Python sleep() Call With GUIs

Command-line applications aren’t the only place where you might need to add Python sleep() calls. When you create a Graphical User Interface (GUI), you’ll occasionally need to add delays. For example, you might create an FTP application to download millions of files, but you need to add a sleep() call between batches so you don’t bog down the server.

GUI code will run all its processing and drawing in a main thread called the event loop. If you use time.sleep() inside of GUI code, then you’ll block its event loop. From the user’s perspective, the application could appear to freeze. The user won’t be able to interact with your application while it’s sleeping with this method. (On Windows, you might even get an alert about how your application is now unresponsive.)

Fortunately, there are other methods you can use besides time.sleep(). In the next few sections, you’ll learn how to add Python sleep() calls in both Tkinter and wxPython.

Sleeping in Tkinter

tkinter is a part of the Python standard library. It may not be available to you if you’re using a pre-installed version of Python on Linux or Mac. If you get an ImportError, then you’ll need to look into how to add it to your system. But if you install Python yourself, then tkinter should already be available.

You’ll start by looking at an example that uses time.sleep(). Run this code to see what happens when you add a Python sleep() call the wrong way:

import tkinter
import time

class MyApp:
    def __init__(self, parent):
        self.root = parent
        self.root.geometry("400x400")
        self.frame = tkinter.Frame(parent)
        self.frame.pack()
        b = tkinter.Button(text="click me", command=self.delayed)
        b.pack()

    def delayed(self):
        time.sleep(3)

if __name__ == "__main__":
    root = tkinter.Tk()
    app = MyApp(root)
    root.mainloop()

Once you’ve run the code, press the button in your GUI. The button will stick down for three seconds as it waits for sleep() to finish. If the application had other buttons, then you wouldn’t be able to click them. You can’t close the application while it’s sleeping, either, since it can’t respond to the close event.

To get tkinter to sleep properly, you’ll need to use after():

import tkinter

class MyApp:
    def __init__(self, parent):
        self.root = parent
        self.root.geometry("400x400")
        self.frame = tkinter.Frame(parent)
        self.frame.pack()
        self.root.after(3000, self.delayed)

    def delayed(self):
        print('I was delayed')

if __name__ == "__main__":
    root = tkinter.Tk()
    app = MyApp(root)
    root.mainloop()

Here you create an application that is 400 pixels wide by 400 pixels tall. It has no widgets on it. All it will do is show a frame. Then, you call self.root.after() where self.root is a reference to the Tk() object. after() takes two arguments:

1. The number of milliseconds to sleep
2. The method to call when the sleep is finished

In this case, your application will print a string to stdout after 3 seconds. You can think of after() as the tkinter version of time.sleep(), but it also adds the ability to call a function after the sleep has finished.

You could use this functionality to improve user experience. By adding a Python sleep() call, you can make the application appear to load faster and then start some longer-running process after it’s up. That way, the user won’t have to wait for the application to open.

Sleeping in wxPython

There are two major differences between wxPython and Tkinter:

1. wxPython has many more widgets.
2. wxPython aims to look and feel native on all platforms.

The wxPython framework is not included with Python, so you’ll need to install it yourself. If you’re not familiar with wxPython, then check out How to Build a Python GUI Application With wxPython.

In wxPython, you can use wx.CallLater() to add a Python sleep() call:

import wx

class MyFrame(wx.Frame):
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        wx.CallLater(4000, self.delayed)
        self.Show()

    def delayed(self):
        print('I was delayed')

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()
    app.MainLoop()

Here, you subclass wx.Frame directly and then call wx.CallLater(). This function takes the same parameters as Tkinter’s after():

1. The number of milliseconds to sleep
2. The method to call when the sleep is finished

When you run this code, you should see a small blank window appear without any widgets. After 4 seconds, you’ll see the string 'I was delayed' printed to stdout.

One of the benefits of using wx.CallLater() is that it’s thread-safe. You can use this method from within a thread to call a function that’s in the main wxPython application.

Conclusion

With this tutorial, you’ve gained a valuable new technique to add to your Python toolbox! You know how to add delays to pace your applications and prevent them from using up system resources. You can even use Python sleep() calls to help your GUI code redraw more effectively. This will make the user experience much better for your customers!

To recap, you’ve learned how to add Python sleep() calls with the following tools:

• time.sleep()
• Decorators
• Threads
• asyncio
• Tkinter
• wxPython

Now you can take what you’ve learned and start putting your code to sleep!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 16, 2019 02:00 PM UTC

Continuum Analytics Blog

What Can AI Teach Us about Bias and Fairness?

By: Peter Wang & Natalie Parra-Novosad As researchers, journalists, and many others have discovered, machine learning algorithms can deliver biased results. One notorious example is ProPublica’s discovery of bias in a software called COMPAS used…

The post What Can AI Teach Us about Bias and Fairness? appeared first on Anaconda.

October 16, 2019 01:30 PM UTC

Catalin George Festila

Python 3.7.4 : Test the DHCP handshakes.

First, the DHCP is based on the earlier BOOTP protocol which uses well-known port numbers for both server and client instead of an ephemeral port. The server and the client communicate via broadcast and the server broadcasts the offered IP address to the client on UDP port 68. This python example has a learning purpose and does not harm anyone. import subprocess as sub import re def find_word(w

October 16, 2019 12:03 PM UTC

Test and Code

91: Python 3.8 - there's a lot more new than most people are talking about

Python 3.8.0 final is live and ready to download.

On todays episode, we're going to run through what's new, picking out the bits that I think are the most interesting and affect the most people, including

• new language features
• standard library changes
• optimizations in 3.8

Not just the big stuff everyone's already talking about. But also some little things that will make programming Python even more fun and easy.

I'm excited about Python 3.8. And really, this episode is to my way to try to get you excited about it too.

Sponsored By:

• PyCharm Professional: Try PyCharm Pro before Oct 29 with a 4 month free trial. Promo Code: TESTNCODE2019

Support Test & Code

Links:

• What’s New In Python 3.8 - at docs.python.org
• Download Python 3.8 at Python.org

October 16, 2019 07:15 AM UTC

Hynek Schlawack

Sharing Your Labor of Love: PyPI Quick and Dirty

A completely incomplete guide to packaging a Python module and sharing it with the world on PyPI.

October 16, 2019 12:00 AM UTC

October 15, 2019

PyCoder’s Weekly

Issue #390 (Oct. 15, 2019)

#390 – OCTOBER 15, 2019
View in Browser »

Join the Slack Team at the Spec Annual Developer Conference

Hear the latest product and tooling enhancements for builders of Slack apps. Learn best practices and apply them in hands-on workshops. Troubleshoot with Slack engineers, ask questions and give feedback to our product team. Enjoy or share 50% off registration with the code S19_ADMIN →
SLACK sponsor

Discussions

Python Jobs

Full Stack Developer (Toronto, ON, Canada)

Beanfield Metroconnect

Backend Developer (Kfar Saba, Israel)

3DSignals

More Python Jobs >>>

Articles & Tutorials

Python Developers Are in Demand on Vettery

Vettery is an online hiring marketplace that’s changing the way people hire and get hired. Ready for a bold career move? Make a free profile, name your salary, and connect with hiring managers from top employers today →
VETTERY sponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #390.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

October 15, 2019 07:30 PM UTC

Python Insider

Python 3.7.5 is now available

October 15, 2019 04:05 PM UTC

Roberto Alsina

Episodio 11: Type Hints

October 15, 2019 03:00 PM UTC

Red Hat Developers

What’s new in Red Hat Dependency Analytics

We are excited to announce a new release of Red Hat Dependency Analytics, a solution that enables developers to create better applications by evaluating and adding high-quality open source components, directly from their IDE.

Red Hat Dependency Analytics helps your development team avoid security and licensing issues when building your applications. It plugs into the developer’s IDE, automatically analyzes your software composition, and provides recommendations to address security holes and licensing problems that your team may be missing.

Without further ado, let’s jump into the new capabilities offered in this release. This release includes a new version of the IDE plugin and the server-side analysis service hosted by Red Hat.

Support for Python applications

Along with Java (maven) and JavaScript (npm), Dependency Analytics now offers its full set of capabilities for Python (PyPI) applications. From your IDE, you can perform the vulnerability and license analysis of the “requirements.txt” file of your Python application, incorporate the recommended fixes, and generate the stack analysis report for more details.

Software composition analysis based on current vulnerability data

An estimated 15,000 open source packages get updated every day. On average, three new vulnerabilities get posted every day across JavaScript (npm) and Python (PyPi) packages. With this new release, the server-side analysis service hosted by Red Hat automatically processes the daily updates to open source packages that it is tracking. The hosted service also automatically ingests new vulnerability data posted to National Vulnerability Database (NVD) for JavaScript and Python packages. This allows the IDE plugin and API calls to provide source code analysis based on current vulnerability and release data.

Analyze transitive dependencies

In addition to the direct dependencies included in your application, Dependency Analytics now leverages the package managers to discover and add the dependencies of those dependencies, called “transitive” dependencies, to the dependency graph of your application. Analysis of your application is performed across the whole graph model and recommendations for fixes are provided across the entire set of dependencies.

Recommendations about complementary open source libraries

With this release, Dependency Analytics looks to recommend high-quality open source libraries that are complementary to the dependencies included in your application. The machine learning technology of the hosted service collects and analyzes various statistics on GitHub to curate a list of high-quality open source libraries that can be added to the current set of dependencies to augment your application. You can provide your feedback about the add-on libraries by clicking on the “thumbs-up” or “thumbs-down” icons shown for each recommendation. Your feedback is automatically processed to improve the quality of the recommendations.

IDE plugin support

The Dependency Analytics IDE plugin is now available for VS Code, Eclipse Che, and any JetBrains IDE, including IntelliJ and PyCharm.

We will continuously release new updates to our Dependency Analytics solution so you can minimize the delays in delivery of your applications due to last-minute security and licensing related issues.

Stay tuned for further updates; we look forward to your feedback about Dependency Analytics.

The post What’s new in Red Hat Dependency Analytics appeared first on Red Hat Developer.

October 15, 2019 02:52 PM UTC

Abhijeet Pal

Creating Comments System With Django

In this tutorial, we will build a basic commenting system for a Django 2.X app, which lets readers add comments on posts.

Here is a preview of what we are going to build by the end of this tutorial.

Pre-Requirements

Before diving into the tutorial I am assuming that you already have a blog or some similar Django 2.X project up and running. I am using Bootstrap 4 for the styling part you can ignore it if you are using any other framework.

I am gonna use my previous blog project you can grab the repo from here, https://github.com/TheAbhijeet/Django_blog/releases/tag/1

Since Python 3 is the current version in active development and addressed as the future of Python, Django rolled out a significant update, and now all the releases after Django 2.0 are only compatible with Python 3.x. Therefore this tutorial is strictly for Python 3.x.

Roadmap To Build A Comment System

1. Create a model to save the comments.
2. Create a form to submit comments and validate the input data.
3. Add a view that processes the form and saves the new comment to the database.
4. Edit the post detail template to display the list of comments and the form to add a new comment.

Building Comment Model

Open The models.py file of blog application and below the Post model create the Comment model.

class Comment(models.Model):
    post = models.ForeignKey(Post,on_delete=models.CASCADE,related_name='comments')
    name = models.CharField(max_length=80)
    email = models.EmailField()
    body = models.TextField()
    created_on = models.DateTimeField(auto_now_add=True)
    active = models.BooleanField(default=False)

    class Meta:
        ordering = ['created_on']

    def __str__(self):
        return 'Comment {} by {}'.format(self.body, self.name)

In this comment model first, we have a Foreign key relation that establishes a many-to-one relationship with the Post model, since every comment will be made on a post and each post will have multiple comments.

The related_name attribute allows us to name the attribute that we use for the relation from the related object back to this one. After defining this, we can retrieve the post of a comment object using comment.post and retrieve all comments of a post using post.comments.all() . If you don’t define the related_name attribute, Django will use the name of the model in lowercase, followed by _set (that is, comment_set ) to name the manager of the related object back to this one.

As a traditional comment system, we are accepting the commenter’s name, email and comment body as inputs. Then we have an active boolean field that is set to False to prevent spam we will manually allow all the comments posted.

The Meta class inside the model contains metadata. We tell Django to sort results in the created_on field in descending order by default when we query the database. We specify descending order using the negative prefix. By doing so, comments made recently will appear first.

The __str__() method is the default human-readable representation of the object. Django will use it in many places, such as the administration site.

Next, we need to synchronize this comment model into the database by running migrations to reflects the changes in the database.

(django) $ python manage.py makemigrations 
(django) $ python manage.py migrate

We are done with the models, now let’s include the Comment model in our Admin dashboard.

Adding Comments Model To The Administration Site

Open admins.py file and write the following code.

from django.contrib import admin
from .models import Post, Comment
class CommentAdmin(admin.ModelAdmin):
    list_display = ('name', 'body', 'post', 'created_on', 'active')
    list_filter = ('active', 'created_on')
    search_fields = ('name', 'email', 'body')
    actions = ['approve_comments']

    def approve_comments(self, request, queryset):
        queryset.update(active=True)

Going over the code @admin.register(Comment) registers the comment into the Admin area. Below the CommentAdmin class to customizes the representation of data on the screen.

The list_display attribute does what its name suggests display the properties mentioned in the tuple in the comments list for each comment.

The list_filter method will filter the comments based on the creation date and their active status and search_fields will simply search the database for the parameters provided in the tuple.

Finally, we have the actions method this will help us for approving many comment objects at once, the approve_comments method is a simple function that takes a queryset and updates the active boolean field to True.

Now create a superuser if you haven’t already and log in to the dashboard you should see the comment model there.

Now click on comments and create your comments.

In case you struck an error like no such table: blog_comment you might wanna delete the SQLite file and run migrations again for a quick fix.

Creating forms from models

Django offers a very rich and secure API to handle forms. Since the form input will be saved in the database models we are gonna use the Django’s ModelForm.

A common practice is to create a forms.py file inside your app directory for all the forms of an app. So create a forms.py file in your app and write the following code.

from .models import Comment
from django import forms


class CommentForm(forms.ModelForm):
    class Meta:
        model = Comment
        fields = ('name', 'email', 'body')

In the model form, we just need to provide the model name in the Meta class of the form Django will handle the form processing and validation on the basis of fields of the model.

By default, Django will generate a form dynamically from all fields of the model but we can explicitly define the fields we want the forms to have, that is what fields attribute is doing here.

Building Views

We will modify the post detail view for form processing using function based view.

from .models import Post
from .forms import CommentForm
from django.shortcuts import render, get_object_or_404

def post_detail(request, slug):
    template_name = 'post_detail.html'
    post = get_object_or_404(Post, slug=slug)
    comments = post.comments.filter(active=True)
    new_comment = None
    # Comment posted
    if request.method == 'POST':
        comment_form = CommentForm(data=request.POST)
        if comment_form.is_valid():

            # Create Comment object but don't save to database yet
            new_comment = comment_form.save(commit=False)
            # Assign the current post to the comment
            new_comment.post = post
            # Save the comment to the database
            new_comment.save()
    else:
        comment_form = CommentForm()

    return render(request, template_name, {'post': post,
                                           'comments': comments,
                                           'new_comment': new_comment,
                                           'comment_form': comment_form})

This post detail view will show the post and all its comments, let’s break it down to see what’s happening.

First, we assigned the HTML template to a variable name template_name for future reference and we are assigning the Post object inside the post variable.

This comments = post.comments.filter(active=True) queryset retrieves all the approved comments from the database.

Since this is the same page where users will create new comments we initialized the new_comment variable by setting it to none.

Next, we have a conditional statement if a POST request is made, the comment_form variable will hold the data of user input next Django will validate the data using the is_valid() method.

If the form is valid the following actions take place.

1. We create a new Comment object by calling the form’s save() method and assign it to the new_comment variable, but with commit=Flase which will prevent it from saving into the database right away because we still have to link it the post object
2. We assigned the comment object to the current post
3. Finally, save the object into the database

Else if it is a GET request we initialize the form object and pass it to the template.

Adding URL patterns for Views

Open the urls.py file of your app and map the view.

path('/', views.post_detail, name='post_detail')

Creating Templates For The Views

Let’s see what we will do in the templates.

   {% for comment in comments %}
        <div class="comments" style="padding: 10px;">
          <p class="font-weight-bold">
            {{ comment.name }}
            <span class=" text-muted font-weight-normal">
              {{ comment.created_on }}
            </span>
          </p>
          {{ comment.body | linebreaks }}
        </div>
        {% endfor %}

Here we are using Django’s {% for %} template tag for looping over comments, then for each comment object, we are displaying the user’s name,creation date and the comment body.

 <div class="card-body">
        {% if new_comment %}
        <div class="alert alert-success" role="alert">
          Your comment is awaiting moderation
        </div>
        {% else %}
        <h3>Leave a comment</h3>
        <form method="post" style="margin-top: 1.3em;">
          {{ comment_form.as_p }}
          {% csrf_token %}
          <button type="submit" class="btn btn-primary  btn-lg">Submit</button>
        </form>
        {% endif %}
      </div>

When a user makes a new comment we show them a message saying, “Your comment is awaiting moderation” else we render the form.

So putting the entire template all together we have this,

{% extends 'base.html' %} {% block content %}

<div class="container">
  <div class="row">
    <div class="col-md-8 card mb-4  mt-3 left  top">
      <div class="card-body">
        <h1>{% block title %} {{ post.title }} {% endblock title %}</h1>
        <p class=" text-muted">{{ post.author }} | {{ post.created_on }}</p>
        <p class="card-text ">{{ post.content | safe }}</p>
      </div>
    </div>
    {% block sidebar %} {% include 'sidebar.html' %} {% endblock sidebar %}

    <div class="col-md-8 card mb-4  mt-3 ">
      <div class="card-body">
        <!-- comments -->
        <h2>{{ comments.count }} comments</h2>

        {% for comment in comments %}
        <div class="comments" style="padding: 10px;">
          <p class="font-weight-bold">
            {{ comment.name }}
            <span class=" text-muted font-weight-normal">
              {{ comment.created_on }}
            </span>
          </p>
          {{ comment.body | linebreaks }}
        </div>
        {% endfor %}
      </div>
    </div>
    <div class="col-md-8 card mb-4  mt-3 ">
      <div class="card-body">
        {% if new_comment %}
        <div class="alert alert-success" role="alert">
          Your comment is awaiting moderation
        </div>
        {% else %}
        <h3>Leave a comment</h3>
        <form method="post" style="margin-top: 1.3em;">
          {{ comment_form.as_p }}
          {% csrf_token %}
          <button type="submit" class="btn btn-primary  btn-lg">Submit</button>
        </form>
        {% endif %}
      </div>
    </div>
  </div>
</div>
{% endblock content %}

If you are working on a completely different project with some other CSS framework then ignore the styling.

Testing The Comment System

Save all the files and run the server and visit http://127.0.0.1:8000/ and visit a post detail page.

Now create your comment and hit the submit button you should see the message. Now login to the Admin dashboard and approve the comment.

Making Comment Form Crispy

Although our form works as expected, yet we can make the form look better without much changing the template using the Django crispy form library. It’s a very popular library for form managing you can check it out here – https://github.com/django-crispy-forms/django-crispy-forms

Install it using

pip install django-crispy-forms

Add it to the installed apps list.

INSTALLED_APPS = [
    ...

    'crispy_forms',
]

If you using Bootstrap 4 for styling add this in settings.py file.

CRISPY_TEMPLATE_PACK = 'bootstrap4'

Now in your template, you just need to load the crispy tag and use the crispy tag beside the form, as follows.

{% load crispy_forms_tags %}
...
<form method="post" style="margin-top: 1.3em;">
          {{ comment_form | crispy }}
          {% csrf_token %}
          <button type="submit" class="btn btn-primary  btn-lg">Submit</button>
 </form>

Save the files and run the server. 

If you are stuck at any step, refer to this GitHub repo

The post Creating Comments System With Django appeared first on Django Central.

October 15, 2019 02:35 PM UTC

Catalin George Festila

Python 3.8.0 : New release of python development.

Good news from the python development area with the new release of python development: Python 3.7.5 Oct. 15, 2019 and Python 3.8.0 Oct. 14, 2019 Now you can use the new python version 3.8.0 from the official webpage. Major new features of the 3.8 series, compared to 3.7 - release Date: Oct. 14, 2019: PEP 572, Assignment expressions PEP 570, Positional-only arguments PEP 587, Python

October 15, 2019 02:04 PM UTC

Real Python

The Python range() Function

Python’s built-in range function is handy when you need to perform an action a specific number of times. As an experienced Pythonista, you’ve most likely used it before. But what does it do?

By the end of this course, you’ll:

• Understand how the Python range function works
• Know how the implementations differ in Python 2 and Python 3
• Have seen a number of hands-on range() examples
• Be equipped to work around some of its limitations

Let’s get cracking!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 15, 2019 02:00 PM UTC

Stack Abuse

Uploading Files to AWS S3 with Python and Django

Introduction

In the quest to build more interactive websites, we don't only relay information to users but also allow them to upload data of their own. This opens up more opportunities and more ways that our websites can serve the end-users.

By allowing users to upload files, we can allow them to share photographs, videos, or music with others or back them up for safekeeping. We can also provide the functionality to manage files and convert them into other formats through websites instead of installing native apps.

The rise of social media globally can be attributed to the ability of users to upload their files, mostly in the form of images and videos for other users to see and also as a means of communication. By enabling users to upload files to websites and platforms, means of communication have been enhanced and information can now be spread in very many different formats.

In this post, we will explore how Django handles file uploading and how we can tap into and extend this functionality with cloud storage to suit our needs.

How Django Handles File Storage

Django not only allows us to turn concepts into web applications but also provides functionality for us to handle files and allow users to upload files to our web applications for further interaction. Through forms, users can attach files to their requests and have their files uploaded and stored in our backend servers.

Before a file is saved, it is temporarily stored somewhere before being processed and stored in the intended final location. For instance, if the uploaded file is less than 2.5MB, the contents of that file will be stored in memory, then written to disk once all the operations are completed while processing it.

This makes the process fast for small files. For files larger than 2.5MB, they are first written to a temporary location as the data is being received, then once processing is complete, the file is moved to its final destination.

File behavior in Django can be customized through various settings, such as FILE_UPLOAD_MAX_MEMORY_SIZE, which allows us to modify the 2.5MB upload limit size for the files that are written to memory first and not to a temporary location. We can also configure the default permissions for the uploaded files through the FILE_UPLOAD_PERMISSIONS.

Other settings can be found in this section of the official Django documentation.

Where can we store our files?

In a Django-powered web application, we can store the uploaded files in various different locations. We can store them on our own servers where the Django code is deployed, or we can send them over to other servers which may have set up elsewhere for storage purposes.

In a bid to cut server maintenance costs and enhance performance, we can also choose not to store the uploaded files on our own servers. In this case, we can hand them over to other hosted storage providers such as AWS, Azure, or OneDrive, among others.

There are several packages that allow us to interact with the APIs provided by the various service providers that we have mentioned. They include:

• Django-Storages enables us to store our uploaded files to AWS Simple Storage Service (S3), Dropbox, OneDrive, Azure and Digital Ocean.
• Django-S3-Storage through which we can upload files directly to Amazon S3
• Django-Cumulus, which allows us to interact with Rackspace for our storage need
• Others include Django-Dropbox, Django-Storage-Swift, and Django-Cloudinary-Storage. More can be found here

For this post, we will use the Django-s3direct package to store our files on AWS's S3.

Our Application - Django Drive

We will use Django to build a web application in which we will upload content for end-users to view. This will be achieved by using the Django administration interface, which comes with the framework.

Our site will be used to sell cars and on it, we will display details and add images or videos of the cars on sale.

The images or videos of the cars on sale will be stored on S3. We will not implement user registration or login at this time for brevity.

Setup

We will use Pipenv to set up and manage our isolated environment in which we will build our Django application by running the following command to set it up using Python3:

$ pipenv install --three

With the environment set up, we can now install Django and Django-s3direct to handle our file uploads to S3:

$ pipenv install django django-s3direct

Django provides a set of commands to bootstrap our project before we start implementing the core functionality of our application. Our Django drive project will have a single application that will be the focus of this post. To achieve this, we run the following commands:

$ django-admin startproject django_drive && cd django_drive
$ django-admin startapp django_drive_app

The django-admin startproject ... command creates the project, and the django-admin startapp ... command creates the application.

The last step of our setup is to create database tables by running the migrate command:

$ python manage.py migrate

When we start our project by running the command python manage.py runserver, we are welcomed by the following page, which confirms that our setup was successful:

Since we will be uploading our files to AWS S3, we will need to set up a free-tier AWS account for demo purposes. After setting up, we can navigate to the S3 dashboard and create a new bucket that will contain our uploads.

For Django-s3direct to interact with our AWS setup, we need to provide the following credentials AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and the AWS_STORAGE_BUCKET_NAME.

Next, we will add the following to our django_drive/settings.py file:

AWS_ACCESS_KEY_ID = 'aws-access-key-id'
AWS_SECRET_ACCESS_KEY = 'secret-access-key'
AWS_STORAGE_BUCKET_NAME = 'name-of-the-bucket'
AWS_S3_REGION_NAME = 'name-of-the-region'
AWS_S3_ENDPOINT_URL = 'https://s3.amazonaws.com'

S3DIRECT_DESTINATIONS = {
    'primary_destination': {
        'key': 'uploads/',
        'allowed': ['image/jpg', 'image/jpeg', 'image/png', 'video/mp4'],
    },
}

Django-s3direct allows us to specify more than one destination for our uploads, this way we can direct different files to separate S3 buckets. For this project, we will put all the uploads in one bucket. Another nifty feature is that we can also limit the file types that can be uploaded to our website. In our case, we have limited it to MP4 videos, JPEG, and PNG images only.

Note: More details about setting up Django-s3direct, such as CORS and Access Setup, can be found here.

We also need to add the following entries in the django_drive/urls.py file:

from django.urls import path, include

urlpatterns = [
    ...
    path('', include('django_drive_app.urls')),
    path('s3direct/', include('s3direct.urls')),
    ...
]

Implementation

We will start by creating the model for our car data, which will be displayed to the end-users. This model will also define the information we will input in our admin dashboard when adding cars to our platform. The car model will be as follows:

from django.db import models
from s3direct.fields import S3DirectField

class Car(models.Model):
    name = models.CharField(max_length=255, blank=False, null=False)
    year_of_manufacture = models.CharField(max_length=255, blank=False, null=False)
    price = models.CharField(max_length=255, blank=False, null=False)
    image = S3DirectField(dest='primary_destination', blank=True)
    video = S3DirectField(dest='primary_destination', blank=True)

    def __str__(self):
        return f"{self.name} ({self.year_of_manufacture}) - {self.price}"

For each car, we will store its name, year of manufacture, price, and an image or video. After creating the model, let us make migrations to create the table in the database that will hold our data by running:

$ python manage.py makemigrations
$ python manage.py migrate

Since we will be using the Django admin dashboard to manage the cars on our platform, we need to register our model in the django_drive_app/admin.py:

from django.contrib import admin
from.models import Car

admin.site.register(Car)

Then we need to create the superuser who will be in charge of adding the cars by running the following command and following the prompts:

$ python manage.py createsuperuser
$ python manage.py runserver

The python manage.py runserver command simply restarts our application.

After restarting our server, we can now navigate to the administration dashboard at http://127.0.0.1:8000/admin and log in with the credentials we specified earlier. Under site administration, we can see our DJANGO_DRIVE_APP with the option to add or change existing cars.

This is the form we use to add a car and its details:

Once we save our car, we can find the image we have uploaded in our S3 bucket on the AWS console. This means that our file has been uploaded to AWS.

Now we will create a view to display the cars and their data to the end-users of our website and also display the images or videos associated with each car. We will start by creating a view in the django_drive_app/views.py:

from django.shortcuts import render
from django.views.generic import TemplateView
from .models import Car

class CarView(TemplateView):
    template_name = 'django_drive_app/cars.html'

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['cars'] = Car.objects.all()
        return context

In this view, we use a class-based Django view to render the HTML file to display our cars. In our view, we run a query to fetch all the cars as stored in our database.

Next, let us create django_drive_app/templates/django_drive_app/cars.html to render our cars:

<!DOCTYPE html>
<html>
  <head>
    <title>Django Drive</title>
  </head>
  <body>
    <h3>Welcome to Django Drive. </h3>
    <p>Here are the current cars available for sale: </p>
    <div class="cars-container">
      {% for car in cars %}
        <div class="car">
          <p>
            <b> {{ car.name }} ({{ car.year_of_manufacture }}) </b> <br>
            Price: {{ car.price }}
          </p>
          <!-- if the car has an image attached -->
          {% if car.image %}
          <img src="{{ car.image }}" height="200" width="400"/>
          {% endif %}
          <!-- If the car has a video -->
          {% if car.video %}
            <video width="320" height="240" controls>
                <source src="{{ car.video }}" type="video/mp4">
              Your browser does not support the video tag.
            </video>
          {% endif %}
        </div>
        <hr>
      {% endfor %}
    </div>
  </body>
</html>

With the view and the template in place, let us add the endpoint that will be used to display the list of cars to the end-users by creating the django_drive_app/urls.py:

from django.conf.urls import url
from .views import CarView

urlpatterns = [
  url(r'^cars/$', CarView.as_view(), name="cars"),
]

We import our view and add a URL entry to map the endpoint to the view that will render the cars. When we restart our server and navigate to 127.0.0.1:8000/cars/, we encounter the following:

As we can see, we created cars with attached images and videos and had them uploaded to AWS's S3 service. The same images and videos have been rendered in our web application after being fetched from AWS.

Conclusion

In this article, we have created a simple Django application that allows administrators to upload files to AWS S3 through the Django administration dashboard. We rendered the uploaded files as hosted on S3 on our landing page, including videos and images of the cars that users would wish to purchase or view.

We used Django-s3direct library to handle the interaction between our Django application and AWS S3 where our files are stored. Through the Django administration application, we were able to upload files that were eventually rendered to the end users on our landing page. We were able to upload and render both images and videos.

The source code for this project is available here on GitHub.

October 15, 2019 12:24 PM UTC

Catalin George Festila

Python 3.7.4 : Testing python source code with streamlit tool.

The official webpage for this python package can be found at streamlit.io. Let's install it with pip3 tool: [mythcat@desk proiecte_github]$ mkdir streamlit_examples [mythcat@desk proiecte_github]$ cd streamlit_examples/ [mythcat@desk streamlit_examples]$ pip3 install streamlit --user Let's try some examples. Create a file named 001.py This simple example will show a map with randoms spots: import

October 15, 2019 11:25 AM UTC

Julien Danjou

Sending Emails in Python — Tutorial with Code Examples

What do you need to send an email with Python? Some basic programming and web knowledge along with the elementary Python skills. I assume you’ve already had a web app built with this language and now you need to extend its functionality with notifications or other emails sending. This tutorial will guide you through the most essential steps of sending emails via an SMTP server:

1. Configuring a server for testing (do you know why it’s important?)
2. Local SMTP server
3. Mailtrap test SMTP server
4. Different types of emails: HTML, with images, and attachments
5. Sending multiple personalized emails (Python is just invaluable for email automation)
6. Some popular email sending options like Gmail and transactional email services

Served with numerous code examples written and tested on Python 3.7!

Sending an email using an SMTP

The first good news about Python is that it has a built-in module for sending emails via SMTP in its standard library. No extra installations or tricks are required. You can import the module using the following statement:

import smtplib

To make sure that the module has been imported properly and get the full description of its classes and arguments, type in an interactive Python session:

help(smtplib)

At our next step, we will talk a bit about servers: choosing the right option and configuring it.

An SMTP server for testing emails in Python

When creating a new app or adding any functionality, especially when doing it for the first time, it’s essential to experiment on a test server. Here is a brief list of reasons:

1. You won’t hit your friends’ and customers’ inboxes. This is vital when you test bulk email sending or work with an email database.
2. You won’t flood your own inbox with testing emails.
3. Your domain won’t be blacklisted for spam.

Local SMTP server

If you prefer working in the local environment, the local SMTP debugging server might be an option. For this purpose, Python offers an smtpd module. It has a DebuggingServer feature, which will discard messages you are sending out and will print them to stdout. It is compatible with all operations systems.

Set your SMTP server to localhost:1025

python -m smtpd -n -c DebuggingServer localhost:1025

In order to run SMTP server on port 25, you’ll need root permissions:

sudo python -m smtpd -n -c DebuggingServer localhost:25

It will help you verify whether your code is working and point out the possible problems if there are any. However, it won’t give you the opportunity to check how your HTML email template is rendered.

Fake SMTP server

Fake SMTP server imitates the work of a real 3rd party web server. In further examples in this post, we will use Mailtrap. Beyond testing email sending, it will let us check how the email will  be rendered and displayed, review the message raw data as well as will provide us with a spam report. Mailtrap is very easy to set up: you will need just copy the credentials generated by the app and paste them into your code.

Here is how it looks in practice:

import smtplib

port = 2525
smtp_server = "smtp.mailtrap.io"
login = "1a2b3c4d5e6f7g" # your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # your password generated by Mailtrap

Mailtrap makes things even easier. Go to the Integrations section in the SMTP settings tab and get the ready-to-use template of the simple message, with your Mailtrap credentials in it. It is the most basic option of instructing your Python script on who sends what to who is the sendmail() instance method:

The code looks pretty straightforward, right? Let’s take a closer look at it and add some error handling (see the comments in between). To catch errors, we use the try and except blocks.

# The first step is always the same: import all necessary components:
import smtplib
from socket import gaierror

# Now you can play with your code. Let’s define the SMTP server separately here:
port = 2525
smtp_server = "smtp.mailtrap.io"
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

# Specify the sender’s and receiver’s email addresses:
sender = "from@example.com"
receiver = "mailtrap@example.com"

# Type your message: use two newlines (\n) to separate the subject from the message body, and use 'f' to  automatically insert variables in the text
message = f"""\
Subject: Hi Mailtrap
To: {receiver}
From: {sender}
This is my first message with Python."""

try:
  # Send your message with credentials specified above
  with smtplib.SMTP(smtp_server, port) as server:
    server.login(login, password)
    server.sendmail(sender, receiver, message)
except (gaierror, ConnectionRefusedError):
  # tell the script to report if your message was sent or which errors need to be fixed
  print('Failed to connect to the server. Bad connection settings?')
except smtplib.SMTPServerDisconnected:
  print('Failed to connect to the server. Wrong user/password?')
except smtplib.SMTPException as e:
  print('SMTP error occurred: ' + str(e))
else:
  print('Sent')

Once you get the Sent result in Shell, you should see your message in your Mailtrap inbox:

Sending emails with HTML content

In most cases, you need to add some formatting, links, or images to your email notifications. We can simply put all of these with the HTML content. For this purpose, Python has an email package.

We will deal with the MIME message type, which is able to combine HTML and plain text. In Python, it is handled by the email.mime module.

It is better to write a text version and an HTML version separately, and then merge them with the MIMEMultipart("alternative") instance. It means that such a message has two rendering options accordingly. In case an HTML isn’t be rendered successfully for some reason, a text version will still be available.

import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

port = 2525
smtp_server = "smtp.mailtrap.io"
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

sender_email = "mailtrap@example.com"
receiver_email = "new@example.com"

message = MIMEMultipart("alternative")
message["Subject"] = "multipart test"
message["From"] = sender_email
message["To"] = receiver_email
# Write the plain text part
text = """\ Hi, Check out the new post on the Mailtrap blog: SMTP Server for Testing: Cloud-based or Local? https://blog.mailtrap.io/2018/09/27/cloud-or-local-smtp-server/ Feel free to let us know what content would be useful for you!"""

# write the HTML part
html = """\ <html> <body> <p>Hi,<br> Check out the new post on the Mailtrap blog:</p> <p><a href="https://blog.mailtrap.io/2018/09/27/cloud-or-local-smtp-server">SMTP Server for Testing: Cloud-based or Local?</a></p> <p> Feel free to <strong>let us</strong> know what content would be useful for you!</p> </body> </html> """

# convert both parts to MIMEText objects and add them to the MIMEMultipart message
part1 = MIMEText(text, "plain")
part2 = MIMEText(html, "html")
message.attach(part1)
message.attach(part2)

# send your email
with smtplib.SMTP("smtp.mailtrap.io", 2525) as server:
  server.login(login, password)
  server.sendmail( sender_email, receiver_email, message.as_string() )

print('Sent')

Sending Emails with Attachments in Python

The next step in mastering sending emails with Python is attaching files. Attachments are still the MIME objects but we need to encode them with the base64 module. A couple of important points about the attachments:

1. Python lets you attach text files, images, audio files, and even applications. You just need to use the appropriate email class like email.mime.audio.MIMEAudio or email.mime.image.MIMEImage. For the full information, refer to this section of the Python documentation.
2. Remember about the file size: sending files over 20MB is a bad practice.

In transactional emails, the PDF files are the most frequently used: we usually get receipts, tickets, boarding passes, order confirmations, etc. So let’s review how to send a boarding pass as a PDF file.

import smtplib
from email import encoders
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

port = 2525
smtp_server = "smtp.mailtrap.io"
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

subject = "An example of boarding pass"
sender_email = "mailtrap@example.com"
receiver_email = "new@example.com"

message = MIMEMultipart()
message["From"] = sender_email
message["To"] = receiver_email
message["Subject"] = subject

# Add body to email
body = "This is an example of how you can send a boarding pass in attachment with Python"
message.attach(MIMEText(body, "plain"))

filename = "yourBP.pdf"
# Open PDF file in binary mode
# We assume that the file is in the directory where you run your Python script from
with open(filename, "rb") as attachment:
# The content type "application/octet-stream" means that a MIME attachment is a binary file
part = MIMEBase("application", "octet-stream")
part.set_payload(attachment.read())
# Encode to base64
encoders.encode_base64(part)
# Add header
part.add_header("Content-Disposition", f"attachment; filename= {filename}")
# Add attachment to your message and convert it to string
message.attach(part)

text = message.as_string()
# send your email
with smtplib.SMTP("smtp.mailtrap.io", 2525) as server:
  server.login(login, password)
  server.sendmail(sender_email, receiver_email, text)

print('Sent')

To attach several files, you can call the message.attach() method several times.

How to send an email with image attachment

Images, even if they are a part of the message body, are attachments as well. There are three types of them: CID attachments (embedded as a MIME object), base64 images (inline embedding), and linked images.

For adding a CID attachment, we will create a MIME multipart message with MIMEImage component:

import smtplib
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart

port = 2525
smtp_server = "smtp.mailtrap.io"
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

sender_email = "mailtrap@example.com"
receiver_email = "new@example.com"

message = MIMEMultipart("alternative")
message["Subject"] = "CID image test"
message["From"] = sender_email
message["To"] = receiver_email

# write the HTML part
html = """\
<html>
<body>
<img src="cid:myimage">
</body>
</html>
"""
part = MIMEText(html, "html")
message.attach(part)

# We assume that the image file is in the same directory that you run your Python script from
with open('mailtrap.jpg', 'rb') as img:
  image = MIMEImage(img.read())
# Specify the  ID according to the img src in the HTML part
image.add_header('Content-ID', '<myimage>')
message.attach(image)

# send your email
with smtplib.SMTP("smtp.mailtrap.io", 2525) as server:
  server.login(login, password)
  server.sendmail(sender_email, receiver_email, message.as_string())

print('Sent')

The CID image is shown both as a part of the HTML message and as an attachment. Messages with this image type are often considered spam: check the Analytics tab in Mailtrap to see the spam rate and recommendations on its improvement. Many email clients — Gmail in particular — don’t display CID images in most cases. So let’s review how to embed a base64 encoded image instead.

Here we will use base64 module and experiment with the same image file:

import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
import base64

port = 2525
smtp_server = "smtp.mailtrap.io"
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap
sender_email = "mailtrap@example.com"
receiver_email = "new@example.com"

message = MIMEMultipart("alternative")
message["Subject"] = "inline embedding"
message["From"] = sender_email
message["To"] = receiver_email

# We assume that the image file is in the same directory that you run your Python script from
with open("image.jpg", "rb") as image:
  encoded = base64.b64encode(image.read()).decode()

html = f"""\
<html>
<body>
<img src="data:image/jpg;base64,{encoded}">
</body>
</html>
"""
part = MIMEText(html, "html")
message.attach(part)

# send your email
with smtplib.SMTP("smtp.mailtrap.io", 2525) as server:
  server.login(login, password)
  server.sendmail(sender_email, receiver_email, message.as_string())

print('Sent')

Now the image is embedded into the HTML message and is not available as an attached file. Python has encoded our JPEG image, and if we go to the HTML Source tab, we will see the long image data string in the img src attribute.

How to Send Multiple Emails

Sending multiple emails to different recipients and making them personal is the special thing about emails in Python.

To add several more recipients, you can just type their addresses in separated by a comma, add Cc and Bcc. But if you work with a bulk email sending, Python will save you with loops.

One of the options is to create a database in a CSV format (we assume it is saved to the same folder as your Python script).

We often see our names in transactional or even promotional examples. Here is how we can make it with Python.

Let’s organize the list in a simple table with just two columns: name and email address. It should look like the following example:

#name,email
John Johnson,john@johnson.com
Peter Peterson,peter@peterson.com

The code below will open the file and loop over its rows line by line, replacing the {name} with the value from the “name” column.

import csv
import smtplib

port = 2525
smtp_server = "smtp.mailtrap.io"
login = "1a2b3c4d5e6f7g" # paste your login generated by Mailtrap
password = "1a2b3c4d5e6f7g" # paste your password generated by Mailtrap

message = """Subject: Order confirmation
To: {recipient}
From: {sender}
Hi {name}, thanks for your order! We are processing it now and will contact you soon"""
sender = "new@example.com"
with smtplib.SMTP("smtp.mailtrap.io", 2525) as server:
  server.login(login, password)
  with open("contacts.csv") as file:
  reader = csv.reader(file)
  next(reader)  # it skips the header row
  for name, email in reader:
    server.sendmail(
      sender,
      email,
      message.format(name=name, recipient=email, sender=sender),
    )
    print(f'Sent to {name}')

In our Mailtrap inbox, we see two messages: one for John Johnson and another for Peter Peterson, delivered simultaneously:

Sending emails with Python via Gmail

When you are ready for sending emails to real recipients, you can configure your production server. It also depends on your needs, goals, and preferences: your localhost or any external SMTP.

One of the most popular options is Gmail so let’s take a closer look at it.

We can often see titles like “How to set up a Gmail account for development”. In fact, it means that you will create a new Gmail account and will use it for a particular purpose.

To be able to send emails via your Gmail account, you need to provide access to it for your application. You can Allow less secure apps or take advantage of the OAuth2 authorization protocol. It’s a way more difficult but recommended due to the security reasons.

Further, to use a Gmail server, you need to know:

• the server name = smtp.gmail.com
• port = 465 for SSL/TLS connection (preferred)
• or port = 587 for STARTTLS connection
• username = your Gmail email address
• password = your password

import smtplib
import ssl

port = 465
password = input("your password")
context = ssl.create_default_context()

with smtplib.SMTP_SSL("smtp.gmail.com", port, context=context) as server:
  server.login("my@gmail.com", password)

If you tend to simplicity, then you can use Yagmail, the dedicated Gmail/SMTP. It makes email sending really easy. Just compare the above examples with these several lines of code:

import yagmail

yag = yagmail.SMTP()
contents = [
"This is the body, and here is just text http://somedomain/image.png",
"You can find an audio file attached.", '/local/path/to/song.mp3'
]
yag.send('to@someone.com', 'subject', contents)

Next steps with Python

Those are just basic options of sending emails with Python. To get great results, review the Python documentation and experiment with your own code!

There are a bunch of various Python frameworks and libraries, which make creating apps more elegant and dedicated. In particular, some of them can help improve your experience with building emails sending functionality:

The most popular frameworks are:

1. Flask, which offers a simple interface for email sending: Flask Mail.
2. Django, which can be a great option for building HTML templates.
3. Zope comes in handy for a website development.
4. Marrow Mailer is a dedicated mail delivery framework adding various helpful configurations.
5. Plotly and its Dash can help with mailing graphs and reports.

Also, here is a handy list of Python resources sorted by their functionality.

Good luck and don’t forget to stay on the safe side when sending your emails!

This article was originally published at Mailtrap’s blog: Sending emails with Python

October 15, 2019 10:33 AM UTC

S. Lott

Apple's Numbers and the All-in-One CSV export

Author F. L. Stevens has a hellishly complex (and irregular) spreadsheet with agents, agencies, and query status. (This is how fiction gets marketed: querying agents.) The spreadsheet has become unmanageably complex, with multiple pages. Each page has multiple tables. Buried in this are three "interesting" tables with agent query information.

Can we talk about drama? There is the dark night of the soul for anyone interested in regular, normalized data.

We have some fundamental choices for working with this mess:

1. Export each relevant table to separate files. Lots of manual pointy-clicky and opportunities for making mistakes.
2. Export the whole thing to separate files. Less pointy-clicky.
3. Export the whole thing to one file. About the same pointy-clicky and error vulnerability as #2. But. Simpler still because there's one file to take care of. Something a fiction author should be able to handle.

• Preserving the Sheet, Table, Row hierarchy. We can think of this as the for s in sheet: for t in table: for r in rows structure. The sheet iterator is Iterator[Tuple[str, Table_Iterator]]. The Table_Iterator is similar: Iterator[Tuple[str, Row_Iterator]]. The Row_Iterator, is the most granular Iterator[Dict[str, Any]].
• Flattening this into a sequence of "(Sheet name, Table Name, Row)" triples. Since a sheet and table have no other attributes beyond a name, this seems advantageous to me.

class SheetTable:
    def __init__(self, source_path: Path) -> None:
        self.path: Path = source_path
        self.csv_source = None
        self.rdr = None
        self.header = None
        self.row = None

    def __enter__(self) -> None:
        self.csv_source = self.path.open()
        self.rdr = csv.reader(self.csv_source)
        self.header = None
        self.row = next(self.rdr)
        return self

    def __exit__(self, *args) -> None:
        self.csv_source.close()

    def _sheet_header(self) -> bool:
        return len(self.row) == 1 and ': ' in self.row[0]

    def sheet_iter(self):
        while True:
            while not (self._sheet_header()):
                try:
                    self.row = next(self.rdr)
                except StopIteration:
                    return
            self.sheet, _, self.table = self.row[0].partition(": ")
            self.header = next(self.rdr)
            self.row = next(self.rdr)
            yield self.sheet, self.table_iter()

    def table_iter(self):
        yield self.table, self.row_iter()
        while not (self._sheet_header()):
            try:
                self.row = next(self.rdr)
            except StopIteration:
                return
        next_sheet, _, next_table = self.row[0].partition(": ")
        while next_sheet == self.sheet:
            self.table = next_table
            self.header = next(self.rdr)
            self.row = next(self.rdr)
            yield self.table, self.row_iter()
            while not (self._sheet_header()):
                try:
                    self.row = next(self.rdr)
                except StopIteration:
                    return
            next_sheet, _, next_table = self.row[0].partition(": ")

    def row_iter(self):
        while not self._sheet_header():
            yield dict(zip(self.header, self.row))
            try:
                self.row = next(self.rdr)
            except StopIteration:
                return

def sheet_table_iter(source_path: Path) -> Iterator[Tuple[str, str, Dict[str, Any]]]:
    with source_path.open() as csv_source:
        rdr = csv.reader(csv_source)
        header = None
        for row in rdr:
            if len(row) == 0:
                continue
            elif len(row) == 1 and ": " in row[0]:
                sheet, table = row[0].split(": ", maxsplit=1)
                header = next(rdr)
                continue
            else:
                # Inject headers to create dict from row
                yield sheet, table, dict(zip(header, row))

    for sheet, table, row in sheet_table_iter(source_path):
        if sheet == 'AgentQuery' and table == 'agent_query':
            agent = agent_query_row(database, row)
        elif sheet == 'AAR-2019-03' and table == 'Table 1':
            agent = aar_2019_row(database, row)

October 15, 2019 08:00 AM UTC

Talk Python to Me

#234 Awesome Python Applications

Have you heard of awesome lists? They are well, pretty awesome! Gathering up the most loved libraries and packages for a given topic.

October 15, 2019 08:00 AM UTC

Python Bytes

#152 You have 35 million lines of Python 2, now what?

October 15, 2019 08:00 AM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Japan
  • Planet Python Brasil
  • Planet Python Indonesia
  • Planet Python Poland
• Python Libraries
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Control F'd"
  • "Fredrik Håård's Blaag"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • 4teamwork
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • AWeber Engineering
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Al-Ahmadgaid Asaad
  • Albert Hopkins
  • Alec Munro
  • Alex Clark
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amit Saha
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anastasios Hatzis
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Ankit Chandawala
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • AppNeta Blog
  • Armin Ronacher
  • Arnav Khare
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Atul Varma -- Toolness
  • Audrey Roy Greenfeld
  • Automating OSINT
  • Baiju Muthukadan
  • Bajusz Tamás
  • Balthazar Rouberol
  • BangPypers
  • BeDjango
  • Ben Bass
  • Ben Rousch
  • Ben Tappin
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • Bill Ward / AdminTome
  • BioPython News
  • Bit of Cheese
  • Blue Yonder Tech
  • BlueBream
  • Bojan Mihelac
  • Brad Lucas
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Harring
  • Brian Okken
  • Bruno Oliveira
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Carlos de la Guardia
  • Casey Duncan
  • Catalin George Festila
  • Catherine Devlin
  • Checking and Sharing
  • Chris Hager
  • Chris Leary
  • Chris McAvoy
  • Chris McDonough
  • Chris Miles
  • Chris Miller
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Barra
  • Christian Heimes
  • Christian Scholz
  • Christoph Zwerschke
  • Christopher Lenz
  • Chuck Thier
  • ClusterHQ
  • Cobe.io
  • CodeGrades
  • CodeSnipers
  • Codementor
  • Coding Diet
  • Collin Winter
  • Continuum Analytics Blog
  • Continuum Analytics News
  • Continuum Blog
  • Core Software
  • Corey Goldberg
  • Corey Oordt
  • Cormoran Project
  • Craig Kerstiens
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dalius Dobravolskas
  • Dallas Fort Worth Pythoneers
  • Damián Avila
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Dariusz Suchojad
  • Data Community DC
  • Data School
  • DataCamp
  • Dataquest
  • Dave Beazley
  • David Goodger
  • David Grant
  • David J C Beach
  • David MacIver
  • David Malcolm
  • David Marte
  • David Stanek
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • DeGizmo
  • Denis Kurov
  • Derrick Petzold
  • Dethe Elza
  • Diego Garcia
  • Django Weblog
  • Django Weekly
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Doug Latornell
  • Dougal Matthews
  • Douglas Napoleone
  • DreamHost
  • Duncan McGreggor
  • Dusty Phillips
  • EasyGUI
  • Ed Crewe
  • Ed Taekema
  • Edward K. Ream
  • Eko S. Wibowo
  • Eli Bendersky
  • Eniram Ltd.
  • Enthought
  • Eray Özkural (examachine)
  • Eric Holscher
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Even Wiik Thomassen
  • Evennia
  • Experienced Django
  • Fabio Zadrozny
  • Fazal Majid
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Francisco Souza
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • Fredrik Lundh
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Gabriel Pettier
  • Galvanize
  • Gary Bernhardt
  • Gary Wilson
  • Geek Scrap
  • Georg Brandl
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • GoDjango
  • Gocept Weblog
  • Godson Gera
  • Gonçalo Margalho
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Gridarrow
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Guilherme Toti
  • Gustavo Narea
  • Gustavo Niemeyer
  • Guyon Moree
  • Gökhan Sever
  • Hany Fahim
  • Hector Garcia
  • Heikki Toivonen
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Humberto Rocha
  • Hynek Schlawack
  • IO Digital Sec
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Intellimath blog
  • Ionel Cristian Maries
  • Iraj Jelodari
  • IronPython-URLs
  • Ishan Chattopadhyaya
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Krstic
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • James Mills
  • Jan-Carel Brand
  • Janusworx
  • Jarrod Millman
  • Jason Meyers
  • Jason Webb
  • Jaysinh Shukla
  • Jean-Louis Fuchs
  • Jean-Paul Calderone
  • Jeethu Rao
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Knupp
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Jim Hughes
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Anderson
  • John Burns
  • John Cook
  • John Jacobsen
  • John Paulett
  • Jon Parise
  • Jonathan Dobson
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan LaCour
  • Jonathan Street
  • Jorge Niedbalski
  • Jorge Puente Sarrín
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Juan Rodríguez Monti
  • Juho Vepsäläinen
  • Julien Danjou
  • Julien Palard
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kate Editor
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kenneth Reitz
  • Kevin Dahlhausen
  • Kevin Dangoor
  • Kiwi PyCon
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kracekumar Ramaraju
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • Laurence Rowe
  • Laurent Luce
  • Laurent Szyster
  • Learn PyQt
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lightning Fast Shop
  • Lintel Technologies
  • Lion Kimbro
  • Lionel Tan
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Macken
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Manuel de la Pena Saenz
  • Marc Garcia
  • Marc Kerins
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Mark Mruss
  • Mark Ramm
  • Marko Samastur
  • Martijn Faassen
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mathieu Virbel
  • Mats Kindahl
  • Matt Harrison
  • Matt Layman
  • Matt Reid
  • Matt Wilkes
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Mattias Brändström
  • Mauveweb
  • Max Ischenko
  • Michael Bayer
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Hudson
  • Michael Nelson
  • Michael Sparks
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikeal Rogers
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mitch Chapman
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Morten W Petersen
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Nadia Alramli
  • Naomi Ceder
  • Natan Zabkar
  • Nathan Lemoine
  • Nathan Piccini Data Science Dojo Blog
  • Ned Batchelder
  • Neil Schemenauer
  • Ng Pheng Siong
  • Nicholas Amorim
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • NumFOCUS
  • Obey the Testing Goat
  • Ofosos
  • Oliver Andrich
  • Omaha Python Users Group
  • Ondřej Čertík
  • OpenHatch Python posts
  • PSF GSoC students blogs
  • Paolo Amoroso
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patricio Paez
  • Patrick Kennedy
  • Patrick Stinson
  • Paul Bissex
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • PayPal Engineering Blog
  • Pedro Lima
  • Petecode
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philip Semanchuk
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Pierre Carbonnelle
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programando Ciência
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyBites
  • PyCarolinas
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyCon Australia
  • PyCon Pune
  • PyCon.DE & PyData Karlsruhe
  • PyPy Development
  • PyTennessee
  • PyTexas
  • Pypix
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python Insider
  • Python Open Mike
  • Python Piedmont Triad User Group
  • Python Software Foundation
  • Python Sweetness
  • Python Testing Cookbook
  • Python User Groups
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • PythonThreads
  • Pythonicity
  • Pythonology
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ralph Heinkel
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Reinout van Rees
  • Rene Dudfield
  • Reuven Lerner
  • Richard Gomes
  • Richard Jones
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rob Galanakis
  • Rob Golding
  • Rob Miller
  • Robert Brewer
  • Robert Collins
  • Robert Zaremba
  • Roberto Alsina
  • Robin Parmar
  • Robin Wilson
  • Roche Compaan
  • Rodrigo Araúj
  • Roman Imankulov
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • Salim Fadhley
  • Salman Haq
  • Samuel Sutch
  • Sandipan Dey
  • Sandro Tosi
  • Sayan Chowdhury
  • Scripting the web with Python
  • Sean McGrath
  • Selena Deckelmann
  • Semaphore Community
  • Senthil Kumaran
  • Server Density
  • Shannon -jj Behrens
  • ShiningPanda
  • Shopkick Tech Blog
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stacks
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Petrea
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Dower
  • Steve Holden
  • Steven Klass
  • Stijn Debrouwere
  • Stories in My Pocket
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Sumith - Blog about SymPy/Python
  • Swaroop C H
  • Swisscom ICT
  • Sylvain Hellegouarch
  • Talk Python to Me
  • Tarek Ziade
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Ted Leung
  • Tennessee Leeuwenburg
  • Terri Oda
  • Terry Jones
  • Terry Peppers
  • Test and Code
  • The Artificial Intelligence Cookbook
  • The Code Bits
  • The Data Scientist
  • The Digital Cat
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Parcon Blog
  • The Python Papers
  • The Three of Wands
  • TheAutomatic.net
  • Thibauld Nion
  • Thomas Guest
  • Thomas Vander Stichele
  • Thomi Richards
  • Tiago Montes
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Golden
  • Tim Knapp
  • Tim Lesher
  • Tim van der Linden
  • Tobias Ivarsson
  • Toes to the Edge
  • Tom Christie
  • Tomasz Ducin
  • Tomasz Früboes
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Torsten Engelbrecht
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Troy Melhase
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • Two Bit Arcade
  • Uche Ogbuji
  • United Coders
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinta Software
  • Virgil Dupras
  • Vladimir Iakolev
  • Vladimir Perić
  • Wallaroo Labs
  • Wallix
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Wichert Akkerman
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • William Thompson
  • Wingware Blog
  • Wingware News
  • Wraithan
  • Yaco
  • Yaniv Aknin
  • Yann Larrivée
  • Yasoob Khalid
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • codingdirectional
  • eGenix.com
  • eRambler
  • gamingdirectional
  • hypothesis.works articles
  • kdev-python
  • leftmouseclickin
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • بايثون العربي
  • To request addition or removal:
    Open an issue on github
    or e-mail planet at python.org
    (note, responses can take up to a few days)