Folium is a Python library that lets you create interactive maps using the Leaflet JavaScript library. With Folium, you can visualize geospatial data on a map that you can share as a website. This tutorial guides you through creating an interactive choropleth map with Folium, showcasing how to bind data to GeoJSON layers and style it for intuitive viewing.

By the end of this tutorial, you’ll understand that:

• Folium allows you to create interactive maps and save them as HTML files.
• You can choose from different web map tiles to customize the map’s appearance.
• Anchoring your map to a specific geolocation focuses the view on that area.
• You can bind data to a GeoJSON layer, which enables for example the creation of a choropleth map.
• You can customize colors and style elements of a map to enhance data visualization.

This tutorial helps you leverage Folium to visualize data with a geographical component, enhancing insights and creating shareable reports.

You’ll build the web map shown below, which displays the ecological footprint per capita of many countries and is based on a similar map on Wikipedia. Along the way, you’ll learn the basics of using Folium for data visualization.

If you work through the tutorial, then your interactive map will look like this in the end:

In this tutorial, you’ll create HTML files that you can serve online at a static web hosting service.

An alternative workflow is to use Folium inside of a Jupyter notebook. In that case, the Folium library will render your maps directly in the Jupyter notebook, which gives you a good opportunity to visually explore a geographical dataset or include a map in your data science report.

If you click below to download the associated materials to this tutorial, then you’ll also get a Jupyter notebook set up with the code of this tutorial. Run the notebook to see how well Folium and Jupyter can play together:

Take the Quiz: Test your knowledge with our interactive “Python Folium: Create Web Maps From Your Data” quiz. You’ll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

Python Folium: Create Web Maps From Your Data

Python’s Folium library gives you access to the mapping strengths of the Leaflet JavaScript library through a Python API. It allows you to create interactive geographic visualizations that you can share as a website.

Install Folium

To get started, create and activate a virtual environment and install folium and pandas. You can use the platform switcher below to see the relevant commands for your operating system:

• Windows
• Linux + macOS

PS> python -m venv venv
PS> venv\Scripts\activate
(venv) PS> python -m pip install folium pandas

$ python -m venv venv
$ source venv/bin/activate
(venv) $ python -m pip install folium pandas

You can use many features of Folium without pandas. However, in this tutorial you’ll eventually create a choropleth map using folium.Choropleth, which takes a pandas DataFrame or Series object as one of its inputs.

Create and Style a Map

A useful and beginner-friendly feature of Folium is that you can create a map with only three lines of code. The map looks good by default because the underlying Leaflet JavaScript library works well with a number of different tile providers, which provide high-quality map backgrounds for your web maps.

Additionally, the library boasts attractive default styles for map features and gives you many options to customize the map to fit your needs exactly.

Display Your Web Map Tiles—in Style!

You want to show data on a world map, so you don’t even need to worry about providing any specific geolocation yet. Open up a new Python file in your favorite text editor and create a tiled web map with three lines of code:

import folium

m = folium.Map()
m.save("footprint.html")

When you run your script, Python creates a new HTML file in your working directory that displays an empty world map with the default settings provided by Folium. Open the file in your browser by double-clicking on it and take a look:

Python lists and tuples are sequence data types that store ordered collections of items. While lists are mutable and ideal for dynamic, homogeneous data, tuples are immutable, making them suitable for fixed, heterogeneous data. Read on to compare tuples vs. lists.

By the end of this tutorial, you’ll understand that:

• Lists are mutable, allowing you to modify their content, while tuples are immutable, meaning you can’t change them after creation.
• You should prefer tuples when you need an immutable sequence, such as function return values or constant data.
• You can create a list from a tuple using the list() constructor, which converts the tuple into a mutable list.
• Tuples are immutable, and this characteristic supports their use in scenarios where data should remain unchanged.

In this tutorial, you’ll learn to define, manipulate, and choose between these two data structures. To get the most out of this tutorial, you should know the basics of Python programming, including how to define variables.

Take the Quiz: Test your knowledge with our interactive “Lists vs Tuples in Python” quiz. You’ll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

Lists vs Tuples in Python

Challenge yourself with this quiz to evaluate and deepen your understanding of Python lists and tuples. You'll explore key concepts, such as how to create, access, and manipulate these data types, while also learning best practices for using them efficiently in your code.

Getting Started With Python Lists and Tuples

In Python, a list is a collection of arbitrary objects, somewhat akin to an array in many other programming languages but more flexible. To define a list, you typically enclose a comma-separated sequence of objects in square brackets ([]), as shown below:

>>> colors = ["red", "green", "blue", "yellow"]

>>> colors
['red', 'green', 'blue', 'yellow']

In this code snippet, you define a list of colors using string objects separated by commas and enclose them in square brackets.

Similarly, tuples are also collections of arbitrary objects. To define a tuple, you’ll enclose a comma-separated sequence of objects in parentheses (()), as shown below:

>>> person = ("Jane Doe", 25, "Python Developer", "Canada")

>>> person
('Jane Doe', 25, 'Python Developer', 'Canada')

In this example, you define a tuple with data for a given person, including their name, age, job, and base country.

Up to this point, it may seem that lists and tuples are mostly the same. However, there’s an important difference:

Both lists and tuples are sequence data types, which means they can contain objects arranged in order. You can access those objects using an integer index that represents their position in the sequence.

Even though both data types can contain arbitrary and heterogeneous objects, you’ll commonly use lists to store homogeneous objects and tuples to store heterogeneous objects.

You can perform indexing and slicing operations on both lists and tuples. You can also have nested lists and nested tuples or a combination of them, like a list of tuples.

The most notable difference between lists and tuples is that lists are mutable, while tuples are immutable. This feature distinguishes them and drives their specific use cases.

Essentially, a list doesn’t have a fixed length since it’s mutable. Therefore, it’s natural to use homogeneous elements to have some structure in the list. A tuple, on the other hand, has a fixed length so the position of elements can have meaning, supporting heterogeneous data.

Creating Lists in Python

In many situations, you’ll define a list object using a literal. A list literal is a comma-separated sequence of objects enclosed in square brackets:

>>> countries = ["United States", "Canada", "Poland", "Germany", "Austria"]

>>> countries
['United States', 'Canada', 'Poland', 'Germany', 'Austria']

In this example, you create a list of countries represented by string objects. Because lists are ordered sequences, the values retain the insertion order.

Python’s in and not in operators allow you to quickly check if a given value is or isn’t part of a collection of values. This type of check is generally known as a membership test in Python. Therefore, these operators are known as membership operators.

By the end of this tutorial, you’ll understand that:

• The in operator in Python is a membership operator used to check if a value is part of a collection.
• You can write not in in Python to check if a value is absent from a collection.
• Python’s membership operators work with several data types like lists, tuples, ranges, and dictionaries.
• You can use operator.contains() as a function equivalent to the in operator for membership testing.
• You can support in and not in in custom classes by implementing methods like .__contains__(), .__iter__(), or .__getitem__().

To get the most out of this tutorial, you’ll need basic knowledge of Python, including built-in data types such as lists, tuples, ranges, strings, sets, and dictionaries. You’ll also need to know about Python generators, comprehensions, and classes.

Getting Started With Membership Tests in Python

Sometimes you need to find out whether a value is present in a collection of values or not. In other words, you need to check if a given value is or is not a member of a collection of values. This kind of check is commonly known as a membership test.

Arguably, the natural way to perform this kind of check is to iterate over the values and compare them with the target value. You can do this with the help of a for loop and a conditional statement.

Consider the following is_member() function:

>>> def is_member(value, iterable):
...     for item in iterable:
...         if value is item or value == item:
...             return True
...     return False
...

This function takes two arguments, the target value and a collection of values, which is generically called iterable. The loop iterates over iterable while the conditional statement checks if the target value is equal to the current value. Note that the condition checks for object identity with is or for value equality with the equality operator (==). These are slightly different but complementary tests.

If the condition is true, then the function returns True, breaking out of the loop. This early return short-circuits the loop operation. If the loop finishes without any match, then the function returns False:

>>> is_member(5, [2, 3, 5, 9, 7])
True

>>> is_member(8, [2, 3, 5, 9, 7])
False

The first call to is_member() returns True because the target value, 5, is a member of the list at hand, [2, 3, 5, 9, 7]. The second call to the function returns False because 8 isn’t present in the input list of values.

Membership tests like the ones above are so common and useful in programming that Python has dedicated operators to perform these types of checks. You can get to know the membership operators in the following table:

As with Boolean operators, Python favors readability by using common English words instead of potentially confusing symbols as operators.

Like many other operators, in and not in are binary operators. That means you can create expressions by connecting two operands. In this case, those are:

1. Left operand: The target value that you want to look for in a collection of values
2. Right operand: The collection of values where the target value may be found

The syntax of a membership test looks something like this:

value in collection

value not in collection

In these expressions, value can be any Python object. Meanwhile, collection can be any data type that can hold collections of values, including lists, tuples, strings, sets, and dictionaries. It can also be a class that implements the .__contains__() method or a user-defined class that explicitly supports membership tests or iteration.

If you use the in and not in operators correctly, then the expressions that you build with them will always evaluate to a Boolean value. In other words, those expressions will always return either True or False. On the other hand, if you try and find a value in something that doesn’t support membership tests, then you’ll get a TypeError. Later, you’ll learn more about the Python data types that support membership tests.

Now that you know what membership operators are, it’s time to learn the basics of how they work.

Python’s mutable objects, such as lists and dictionaries, allow you to change their value or data directly without affecting their identity. In contrast, immutable objects, like tuples and strings, don’t allow in-place modifications. Instead, you’ll need to create new objects of the same type with different values.

By the end of this tutorial, you’ll understand that:

• The difference between mutable and immutable objects is that mutable objects can be modified, while immutable objects can’t be altered once created.
• Python lists are mutable, allowing you to change, add, or remove elements.
• Strings in Python are immutable, meaning you can’t change their content after creation.
• To check if an object is mutable, you can try altering its contents. If you succeed without an error, it’s mutable.

In Python, mutability is a characteristic that may profoundly influence your decision when choosing which data type to use for solving a programming problem. Therefore, you need to know how mutable and immutable objects work in Python.

To dive smoothly into this fundamental Python topic, you should be familiar with how variables work in Python. You should also know the basics of Python’s built-in data types, such as numbers, strings, tuples, lists, dictionaries, sets, and others. Finally, knowing how object-oriented programming works in Python is also a good starting point.

Mutability vs Immutability

In programming, you have an immutable object if you can’t change the object’s state after you’ve created it. In contrast, a mutable object allows you to modify its internal state after creation. In short, whether you’re able to change an object’s state or contained data is what defines if that object is mutable or immutable.

Immutable objects are common in functional programming, while mutable objects are widely used in object-oriented programming. Because Python is a multiparadigm programming language, it provides mutable and immutable objects for you to choose from when solving a problem.

To understand how mutable and immutable objects work in Python, you first need to understand a few related concepts. To kick things off, you’ll take a look at variables and objects.

Variables and Objects

In Python, variables don’t have an associated type or size, as they’re labels attached to objects in memory. They point to the memory position where concrete objects live. In other words, a Python variable is a name that refers to or holds a reference to a concrete object. In contrast, Python objects are concrete pieces of information that live in specific memory positions on your computer.

The main takeaway here is that variables and objects are two different animals in Python:

• Variables hold references to objects.
• Objects live in concrete memory positions.

Both concepts are independent of each other. However, they’re closely related. Once you’ve created a variable with an assignment statement, then you can access the referenced object throughout your code by using the variable name. If the referenced object is mutable, then you can also perform mutations on it through the variable. Mutability or immutability is intrinsic to objects rather than to variables.

However, if the referenced object is immutable, then you won’t be able to change its internal state or contained data. You’ll just be able to make your variable reference a different object that, in Python, may or may not be of the same type as your original object.

If you don’t have a reference (variable) to an object, then you can’t access that object in your code. If you lose or remove all the references to a given object, then Python will garbage-collect that object, freeing the memory for later use.

Now that you know that there are differences between variables and objects, you need to learn that all Python objects have three core properties: identity, type, and value.

Objects, Value, Identity, and Type

In Python, everything is an object. For example, numbers, strings, functions, classes, and modules are all objects. Every Python object has three core characteristics that define it at a foundational level. These characteristics are:

1. Value
2. Identity
3. Type

Arguably, the value is probably the most familiar object characteristic that you’ve dealt with. An object’s value consists of the concrete piece or pieces of data contained in the object itself. A classic example is a numeric value like an integer or floating-point number:

>>> 42
42
>>> isinstance(42, int)
True

>>> 3.14
3.14
>>> isinstance(3.14, float)
True

These numeric values, 42 and 3.14, are both objects. The first number is an instance of the built-in int class, while the second is an instance of float. In both examples, you confirm the object’s type using the built-in isinstance() function.

>>> obj = object()
>>> dir(obj)
[
    '__class__',
    '__delattr__',
    ...
    '__str__',
    '__subclasshook__'
]

Python’s zipfile module allows you to efficiently manipulate ZIP files, a standard format for compressing and archiving data. With this module, you can create, read, write, extract, and list files within ZIP archives.

By the end of this tutorial, you’ll understand that:

• Python’s zipfile module lets you create, read, and modify ZIP files.
• You can extract specific files or all contents from a ZIP archive using zipfile.
• Reading metadata about ZIP contents is straightforward with zipfile.
• You can compress files in ZIP archives using different algorithms.
• PyZipFile allows you to bundle Python modules and packages into ZIP files for distribution.

This tutorial teaches you how to handle ZIP files using Python’s zipfile, making file management and data exchange over networks more efficient. Understanding these concepts will enhance your ability to work with compressed files in Python, optimizing both data storage and transfer.

To get the most out of this tutorial, you should know the basics of working with files, using the with statement, handling file system paths with pathlib, and working with classes and object-oriented programming.

To get the files and archives that you’ll use to code the examples in this tutorial, click the link below:

Getting Started With ZIP Files

ZIP files are a well-known and popular tool in today’s digital world. These files are fairly popular and widely used for cross-platform data exchange over computer networks, notably the Internet.

You can use ZIP files for bundling regular files together into a single archive, compressing your data to save some disk space, distributing your digital products, and more. In this tutorial, you’ll learn how to manipulate ZIP files using Python’s zipfile module.

Because the terminology around ZIP files can be confusing at times, this tutorial will stick to the following conventions regarding terminology:

Having these terms clear in your mind will help you avoid confusion while you read through the upcoming sections. Now you’re ready to continue learning how to manipulate ZIP files efficiently in your Python code!

What Is a ZIP File?

You’ve probably already encountered and worked with ZIP files. Yes, those with the .zip file extension are everywhere! ZIP files, also known as ZIP archives, are files that use the ZIP file format.

PKWARE is the company that created and first implemented this file format. The company put together and maintains the current format specification, which is publicly available and allows the creation of products, programs, and processes that read and write files using the ZIP file format.

The ZIP file format is a cross-platform, interoperable file storage and transfer format. It combines lossless data compression, file management, and data encryption.

Data compression isn’t a requirement for an archive to be considered a ZIP file. So you can have compressed or uncompressed member files in your ZIP archives. The ZIP file format supports several compression algorithms, though Deflate is the most common. The format also supports information integrity checks with CRC32.

Even though there are other similar archiving formats, such as RAR and TAR files, the ZIP file format has quickly become a common standard for efficient data storage and for data exchange over computer networks.

ZIP files are everywhere. For example, office suites such as Microsoft Office and Libre Office rely on the ZIP file format as their document container file. This means that .docx, .xlsx, .pptx, .odt, .ods, and .odp files are actually ZIP archives containing several files and folders that make up each document. Other common files that use the ZIP format include .jar, .war, and .epub files.

You may be familiar with GitHub, which provides web hosting for software development and version control using Git. GitHub uses ZIP files to package software projects when you download them to your local computer. For example, you can download the exercise solutions for Python Basics: A Practical Introduction to Python 3 book in a ZIP file, or you can download any other project of your choice.

ZIP files allow you to aggregate, compress, and encrypt files into a single interoperable and portable container. You can stream ZIP files, split them into segments, make them self-extracting, and more.

Why Use ZIP Files?

Knowing how to create, read, write, and extract ZIP files can be a useful skill for developers and professionals who work with computers and digital information. Among other benefits, ZIP files allow you to:

• Reduce the size of files and their storage requirements without losing information
• Improve transfer speed over the network due to reduced size and single-file transfer
• Pack several related files together into a single archive for efficient management
• Bundle your code into a single archive for distribution purposes
• Secure your data by using encryption, which is a common requirement nowadays
• Guarantee the integrity of your information to avoid accidental and malicious changes to your data

When you use the raise statement in Python to raise (or throw) an exception, you signal an error or an unusual condition in your program. With raise, you can trigger both built-in and custom exceptions. You can also include custom messages for more clarity or even re-raise exceptions to add context or handle further processing.

By the end of this tutorial, you’ll understand that:

• Raising an exception in Python signals an error condition, halting normal program flow.
• You use raise to initiate exceptions for error handling or to propagate existing exceptions.
• You can raise custom exceptions by defining new exception classes derived from Exception.
• The difference between raise and assert lies in their use. You use assert for debugging, while raise is used to signal runtime errors.
• You can re-raise an exception by using a bare raise within an except block to preserve the original traceback.

Learning about the raise statement will allow you to handle errors and exceptional situations effectively in your code. By knowing how to use it, you’ll be able to develop more robust programs and improve the quality of your code.

To get the most out of this tutorial, you should understand the fundamentals of Python, including variables, data types, conditional statements, exception handling, and classes.

Take the Quiz: Test your knowledge with our interactive “Python's raise: Effectively Raising Exceptions in Your Code” quiz. You’ll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

Python's raise: Effectively Raising Exceptions in Your Code

In this quiz, you'll test your understanding of how to raise exceptions in Python using the raise statement. This knowledge will help you handle errors and exceptional situations in your code, leading to more robust programs and higher-quality code.

Handling Exceptional Situations in Python

Exceptions play a fundamental role in Python. They allow you to handle errors and exceptional situations in your code. But what is an exception? An exception represents an error or indicates that something is going wrong. Some programming languages, such as C, and Go, encourage you to return error codes, which you check. In contrast, Python encourages you to raise exceptions, which you handle.

When a problem occurs in a program, Python automatically raises an exception. For example, watch what happens if you try to access a nonexistent index in a list object:

>>> colors = [
...     "red",
...     "orange",
...     "yellow",
...     "green",
...     "blue",
...     "indigo",
...     "violet"
... ]

>>> colors[10]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range

In this example, your colors list doesn’t have a 10 index. Its indices go from 0 to 6, covering your seven colors. So, if you try to get index number 10, then you get an IndexError exception telling you that your target index is out of range.

Every raised exception has a traceback, also known as a stack trace, stack traceback, or backtrace, among other names. A traceback is a report containing the sequence of calls and operations that traces down to the current exception.

In Python, the traceback header is Traceback (most recent call last) in most situations. Then you’ll have the actual call stack and the exception name followed by its error message.

Exceptions will cause your program to terminate unless you handle them using a try … except block:

>>> try:
...     colors[10]
... except IndexError:
...     print("Your list doesn't have that index :-(")
...
Your list doesn't have that index :-(

The first step in handling an exception is to predict which exceptions can happen. If you don’t do that, then you can’t handle the exceptions, and your program will crash. In that situation, Python will print the exception traceback so that you can figure out how to fix the problem. Sometimes, you must let the program fail in order to discover the exceptions that it raises.

In the above example, you know beforehand that indexing a list with an index beyond its range will raise an IndexError exception. So, you’re ready to catch and handle that specific exception. The try block takes care of catching exceptions. The except clause specifies the exception that you’re predicting, and the except code block allows you to take action accordingly. The whole process is known as exception handling.

If your code raises an exception in a function but doesn’t handle it there, then the exception propagates to where you called function. If your code doesn’t handle it there either, then it continues propagating until it reaches the main program. If there’s no exception handler there, then the program halts with an exception traceback.

Exceptions are everywhere in Python. Virtually every module in the standard library uses them. Python will raise exceptions in many different circumstances. The Python documentation states that:

Exceptions are a means of breaking out of the normal flow of control of a code block in order to handle errors or other exceptional conditions. An exception is raised at the point where the error is detected; it may be handled by the surrounding code block or by any code block that directly or indirectly invoked the code block where the error occurred. (Source)

In short, Python automatically raises exceptions when an error occurs during a program’s execution. Python also allows you to raise exceptions on demand using the raise keyword. This keyword lets you handle your program’s errors in a more controlled manner.

In shell scripts, the shebang line (#!) specifies the path to the interpreter that should execute the file. You can place it at the top of your Python file to tell the shell how to run your script, allowing you to execute the script directly without typing python before the script name. The shebang is essential for Unix-like systems but ignored on Windows unless using specific compatibility layers.

By the end of this tutorial, you’ll understand that:

• A shebang specifies the path to the Python interpreter in scripts, allowing direct execution.
• You should include a shebang when a script needs direct execution, but not in import-only modules.
• Best practices for shebangs include using /usr/bin/env for portability and ensuring the script is executable.
• Shebangs have limitations, such as being ignored on Windows without compatibility layers like WSL.

To proceed, you should have basic familiarity with the command line and know how to run Python scripts from it. You can also download the supporting materials for this tutorial to follow along with the code examples:

What’s a Shebang, and When Should You Use It?

In short, a shebang is a special kind of comment that you may include in your source code to tell the operating system’s shell where to find the interpreter for the rest of the file:

#!/usr/bin/python3

print("Hello, World!")

If you’re using a shebang, it must appear on the first line in your script, and it has to start with a hash sign (#) followed by an exclamation mark (!), colloquially known as the bang, hence the name shebang. The choice of the hash sign to begin this special sequence of characters wasn’t accidental, as many scripting languages use it for inline comments.

You should make sure you don’t put any other comments before the shebang line if you want it to work correctly, or else it won’t be recognized! After the exclamation mark, specify an absolute path to the relevant code interpreter, such as Python. Providing a relative path will have no effect, unfortunately.

It’s not uncommon to combine a shebang with the name-main idiom, which prevents the main block of code from running when someone imports the file from another module:

#!/usr/bin/python3

if __name__ == "__main__":
    print("Hello, World!")

With this conditional statement, Python will call the print() function only when you run this module directly as a script—for example, by providing its path to the Python interpreter:

$ python3 /path/to/your/script.py
Hello, World!

As long as the script’s content starts with a correctly defined shebang line and your system user has permission to execute the corresponding file, you can omit the python3 command to run that script:

$ /path/to/your/script.py
Hello, World!

A shebang is only relevant to runnable scripts that you wish to execute without explicitly specifying the program to run them through. You wouldn’t typically put a shebang in a Python module that only contains function and class definitions meant for importing from other modules. Therefore, use the shebang when you don’t want to prefix the command that runs your Python script with python or python3.

#!/usr/bin/python3
# -*- coding: utf-8 -*-

if __name__ == "__main__":
    print("Grüß Gott")

>>> "Grüß Gott".encode("unicode_escape")
b'Gr\\xfc\\xdf Gott'

Now that you have a high-level understanding of what a shebang is and when to use it, you’re ready to explore it in more detail. In the next section, you’ll take a closer look at how it works.

How Does a Shebang Work?

Normally, to run a program in the terminal, you must provide the full path to a particular binary executable or the name of a command present in one of the directories listed on the PATH environment variable. One or more command-line arguments may follow this path or command:

$ /usr/bin/python3 -c 'print("Hello, World!")'
Hello, World!

$ python3 -c 'print("Hello, World!")'
Hello, World!

Here, you run the Python interpreter in a non-interactive mode against a one-liner program passed through the -c option. In the first case, you provide an absolute path to python3, while in the second case, you rely on the fact that the parent folder, /usr/bin/, is included on the search path by default. Your shell can find the Python executable, even if you don’t provide the full path, by looking through the directories on the PATH variable.

Python’s flush parameter in the print() function allows you to control when to empty the output data buffer, ensuring your output appears immediately. This is useful when building visual progress indicators or when piping logs to another application in real-time.

By default, print() output is line-buffered in interactive environments and block-buffered otherwise. You can override this behavior by using the flush=True parameter to force the buffer to clear immediately.

By the end of this tutorial, you’ll understand that:

• Flush in coding refers to emptying the data buffer to ensure immediate output.
• flush=True in print() forces the buffer to clear immediately.
• Clearing output involves managing buffer behavior, typically with newlines.
• Turning off print buffering can be achieved using the -u option or PYTHONUNBUFFERED.

By repeatedly running a short code snippet that you change only slightly, you’ll see that if you run print() with its default arguments, then its execution is line-buffered in interactive mode, and block-buffered otherwise.

You’ll get a feel for what all of that means by exploring the code practically. But before you dive into changing output stream buffering in Python, it’s helpful to revisit how it happens by default, and understand why you might want to change it.

Understand How Python Buffers Output

When you make a write call to a file-like object, Python buffers the call by default—and that’s a good idea! Disk write and read operations are slow in comparison to random-access memory (RAM) access. When your script makes fewer system calls for write operations by batching characters in a RAM data buffer and writing them all at once to disk with a single system call, then you can save a lot of time.

To put the use case for buffering into a real-world context, think of traffic lights as buffers for car traffic. If every car crossed an intersection immediately upon arrival, it would end in gridlock. That’s why the traffic lights buffer traffic from one direction while the other direction flushes.

However, there are situations when you don’t want to wait for a data buffer to fill up before it flushes. Imagine that there’s an ambulance that needs to get past the crossroads as quickly as possible. You don’t want it to wait at the traffic lights until there’s a certain number of cars queued up.

In your program, you usually want to flush the data buffer right away when you need real-time feedback on code that has executed. Here are a couple of use cases for immediate flushing:

• Instant feedback: In an interactive environment, such as a Python REPL or a situation where your Python script writes to a terminal

• File monitoring: In a situation where you’re writing to a file-like object, and the output of the write operation gets read by another program while your script is still executing—for example, when you’re monitoring a log file

In both cases, you need to read the generated output as soon as it generates, and not only when enough output has assembled to flush the data buffer.

There are many situations where buffering is helpful, and there are some situations where too much buffering can be a disadvantage. Therefore, there are different types of data buffering that you can implement where they fit best:

• Unbuffered means that there’s no data buffer. Every byte creates a new system call and gets written independently.

• Line-buffered means that there’s a data buffer that collects information in memory, and once it encounters a newline character (\n), the data buffer flushes and writes the whole line in one system call.

• Fully-buffered (block-buffered) means that there’s a data buffer of a specific size, which collects all the information that you want to write. Once it’s full, it flushes and sends all its contents onward in a single system call.

Python uses block buffering as a default when writing to file-like objects. However, it executes line-buffered if you’re writing to an interactive environment.

To better understand what that means, write a Python script that simulates a countdown:

from time import sleep

for second in range(3, 0, -1):
    print(second)
    sleep(1)
print("Go!")

By default, each number shows up right when print() is called in the script. But as you develop and tweak your countdown timer, you might run into a situation where all your output gets buffered. Buffering the whole countdown and printing it all at once when the script finishes would lead to a lot of confusion for the athletes waiting at the start line!

So how can you make sure that you won’t run into data buffering issues as you develop your Python script?

Add a Newline for Python to Flush Print Output

If you’re running a code snippet in a Python REPL or executing it as a script directly with your Python interpreter, then you won’t run into any issues with the script shown above.

In an interactive environment, the standard output stream is line-buffered. This is the output stream that print() writes to by default. You’re working with an interactive environment any time that your output will display in a terminal. In this case, the data buffer flushes automatically when it encounters a newline character ("\n"):

Python makes it straightforward to download files from a URL with its robust set of libraries. For quick tasks, you can use the built-in urllib module or the requests library to fetch and save files. When working with large files, streaming data in chunks can help save memory and improve performance.

You can also perform parallel file downloads using ThreadPoolExecutor for multithreading or the aiohttp library for asynchronous tasks. These approaches allow you to handle multiple downloads concurrently, significantly reducing the total download time if you’re handling many files.

By the end of this tutorial, you’ll understand that:

• You can use Python to download files with libraries like urllib and requests.
• To download a file using a URL in Python, you can use urlretrieve() or requests.get().
• To extract data from a URL in Python, you use the response object from requests.
• To download a CSV file from a URL in Python, you may need to specify the format in the URL or query parameters.

In this tutorial, you’ll be downloading a range of economic data from the World Bank Open Data platform. To get started on this example project, go ahead and grab the sample code below:

Facilitating File Downloads With Python

While it’s possible to download files from URLs using traditional command-line tools, Python provides several libraries that facilitate file retrieval. Using Python to download files offers several advantages.

One advantage is flexibility, as Python has a rich ecosystem of libraries, including ones that offer efficient ways to handle different file formats, protocols, and authentication methods. You can choose the most suitable Python tools to accomplish the task at hand and fulfill your specific requirements, whether you’re downloading from a plain-text CSV file or a complex binary file.

Another reason is portability. You may encounter situations where you’re working on cross-platform applications. In such cases, using Python is a good choice because it’s a cross-platform programming language. This means that Python code can run consistently across different operating systems, such as Windows, Linux, and macOS.

Using Python also offers the possibility of automating your processes, saving you time and effort. Some examples include automating retries if a download fails, retrieving and saving multiple files from URLs, and processing and storing your data in designated locations.

These are just a few reasons why downloading files using Python is better than using traditional command-line tools. Depending on your project requirements, you can choose the approach and library that best suits your needs. In this tutorial, you’ll learn approaches to some common scenarios requiring file retrievals.

Downloading a File From a URL in Python

In this section, you’ll learn the basics of downloading a ZIP file containing gross domestic product (GDP) data from the World Bank Open Data platform. You’ll use two common tools in Python, urllib and requests, to download GDP by country.

While the urllib package comes with Python in its standard library, it has some limitations. So, you’ll also learn to use a popular third-party library, requests, that offers more features for making HTTP requests. Later in the tutorial, you’ll see additional functionalities and use cases.

Using urllib From the Standard Library

Python ships with a package called urllib, which provides a convenient way to interact with web resources. It has a straightforward and user-friendly interface, making it suitable for quick prototyping and smaller projects. With urllib, you can perform different tasks dealing with network communication, such as parsing URLs, sending HTTP requests, downloading files, and handling errors related to network operations.

As a standard library package, urllib has no external dependencies and doesn’t require installing additional packages, making it a convenient choice. For the same reason, it’s readily accessible for development and deployment. It’s also cross-platform compatible, meaning you can write and run code seamlessly using the urllib package across different operating systems without additional dependencies or configuration.

The urllib package is also very versatile. It integrates well with other modules in the Python standard library, such as re for building and manipulating regular expressions, as well as json for working with JSON data. The latter is particularly handy when you need to consume JSON APIs.

In addition, you can extend the urllib package and use it with other third-party libraries, like requests, BeautifulSoup, and Scrapy. This offers the possibility for more advanced operations in web scraping and interacting with web APIs.

To download a file from a URL using the urllib package, you can call urlretrieve() from the urllib.request module. This function fetches a web resource from the specified URL and then saves the response to a local file. To start, import urlretrieve() from urlllib.request:

>>> from urllib.request import urlretrieve

Next, define the URL that you want to retrieve data from. If you don’t specify a path to a local file where you want to save the data, then the function will create a temporary file for you. Since you know that you’ll be downloading a ZIP file from that URL, go ahead and provide an optional path to the target file:

>>> url = (
...     "https://api.worldbank.org/v2/en/indicator/"
...     "NY.GDP.MKTP.CD?downloadformat=csv"
... )
>>> filename = "gdp_by_country.zip"

Because your URL is quite long, you rely on Python’s implicit concatenation by splitting the string literal over multiple lines inside parentheses. The Python interpreter will automatically join the separate strings on different lines into a single string. You also define the location where you wish to save the file. When you only provide a filename without a path, Python will save the resulting file in your current working directory.

TOML stands for Tom’s Obvious Minimal Language. Its human-readable syntax makes TOML convenient to parse into data structures across various programming languages. In Python, you can use the built-in tomllib module to work with TOML files. TOML plays an essential role in the Python ecosystem. Many of your favorite tools rely on TOML for configuration, and you’ll use pyproject.toml when you build and distribute your own packages.

By the end of this tutorial, you’ll understand that:

• TOML in Python refers to a minimal configuration file format that’s convenient to read and parse.
• TOML files are primarily used for configuration, separating code from settings for flexibility.
• pyproject.toml is crucial for package configuration and specifies the build system and dependencies.
• Loading a TOML file in Python involves using tomli or tomllib to parse it into a dictionary.
• tomli and tomllib differ mainly in origin, with tomllib being a standard library module in modern Python.

If you want to know more about why tomllib was added to Python, then have a look at the companion tutorial, Python 3.11 Preview: TOML and tomllib.

Use TOML as a Configuration Format

TOML is short for Tom’s Obvious Minimal Language and is humbly named after its creator, Tom Preston-Werner. It was designed expressly to be a configuration file format that should be “easy to parse into data structures in a wide variety of languages” (Source).

In this section, you’ll start thinking about configuration files and look at what TOML brings to the table.

Configurations and Configuration Files

A configuration is an important part of almost any application or system. It’ll allow you to change settings or behavior without changing the source code. Sometimes you’ll use a configuration to specify information needed to connect to another service like a database or cloud storage. Other times you’ll use configuration settings to allow your users to customize their experience with your project.

Using a configuration file for your project is a good way to separate your code from its settings. It also encourages you to be conscious about which parts of your system are genuinely configurable, giving you a tool to name magic values in your source code. For now, consider this configuration file for a hypothetical tic-tac-toe game:

player_x_color = blue
player_o_color = green
board_size     = 3
server_url     = https://tictactoe.example.com/

You could potentially code this directly in your source code. However, by moving the settings into a separate file, you achieve a few things:

• You give explicit names to values.
• You provide these values more visibility.
• You make it simpler to change the values.

Look more closely at your hypothetical configuration file. Those values are conceptually different. The colors are values that your framework probably supports changing. In other words, if you replaced blue with red, that would be honored without any special handling in your code. You could even consider if it’s worth exposing this configuration to your end users through your front end.

However, the board size may or may not be configurable. A tic-tac-toe game is played on a three-by-three grid. It’s not certain that your logic would still work for other board sizes. It may still make sense to keep the value in your configuration file, both to give a name to the value and to make it visible.

Finally, the project URL is usually essential when deploying your application. It’s not something that a typical user will change, but a power user may want to redeploy your game to a different server.

To be more explicit about these different use cases, you may want to add some organization to your configuration. One popular option is to separate your configuration into additional files, each dealing with a different concern. Another option is to group your configuration values somehow. For example, you can organize your hypothetical configuration file as follows:

[user]
player_x_color = blue
player_o_color = green

[constant]
board_size = 3

[server]
url = https://tictactoe.example.com

The organization of the file makes the role of each configuration item clearer. You can also add comments to the configuration file with instructions to anyone thinking about making changes to it.

There are many ways for you to specify a configuration. Windows has traditionally used INI files, which resemble your configuration file from above. Unix systems have also relied on plain-text, human-readable configuration files, although the actual format varies between different services.

Over time, more and more applications have come to use well-defined formats like XML, JSON, or YAML for their configuration needs. These formats were designed as data interchange or serialization formats, usually meant for computer communication.

On the other hand, configuration files are often written or edited by humans. Many developers have gotten frustrated with JSON’s strict comma rules when updating their Visual Studio Code settings or with YAML’s nested indentations when setting up a cloud service. Despite their ubiquity, these file formats aren’t the easiest to write by hand.

TOML: Tom’s Obvious Minimal Language

Building Dictionary Comprehensions in Python

In this video course, you’ll learn how to write dictionary comprehensions in Python. You’ll also explore the most common use cases for dictionary comprehensions and learn about some bad practices that you should avoid when using them in your code.
REAL PYTHON course

PyViz: Python Tools for Data Visualization

This site contains an overview of all the different visualization libraries in the Python ecosystem. If you’re trying to pick a tool, this is a great place to better understand the pros and cons of each.
PYVIZ.ORG

Building AI Agents With Your Enterprise Data: A Developer’s Guide

Building AI agents over enterprise data is hard—legacy systems, scattered data, complex queries. MindsDB, a leading open-source enterprise AI platform, makes it much easier: turn your data into AI-driven ‘skills’ and build smarter, more customized agents →
MINDSDB sponsor

Catching Memory Leaks With Your Test Suite

“If you have a good test suite, you may be able to use pytest fixtures to identify memory and other resource leaks.”
ITAMAR TURNER-TRAURING

Djangonaut Space Session 4 Applications Open

DJANGONAUT.SPACE

Python 3.14.0 Alpha 4 Released

PYTHON.ORG

Django 5.2 Alpha 1 Released

DJANGO SOFTWARE FOUNDATION

Django Security Releases Issued: 5.1.5, 5.0.11, and 4.2.18

DJANGO SOFTWARE FOUNDATION

EuroPython Prague 2025 Call for Proposals

EUROPYTHON.EU

10 Ways to Work With Large Files in Python

“Handling large text files in Python can feel overwhelming. When files grow into gigabytes, attempting to load them into memory all at once can crash your program.” This article covers different ways of dealing with this challenge.
ALEKSEI ALEINIKOV

Investigating the Popularity of Python Build Backends

This post analyzes the popularity of build backends used in pyproject.toml files over time. Over half of the projects analyzed use setuptools, with Poetry coming in second around 30%.
BASTIAN VENTHUR

Testing Your Python Package Releases

Upon creating the release for the Django Debug Toolbar, Tim discovered a problem when uploading to PyPI. This article talks about how he could have caught it earlier with better tests.
TIM SCHILLING

How I Automated Data Cleaning in Python

Learn how to automate data cleaning in Python using reusable functions and pipelines. Improve efficiency by handling missing values, duplicates, and data types.
SATYAM SAHU

The “Active Enum” Pattern

This article talks about the challenges associated with enum declarations being separate from their associated business logic, and proposes some alternatives.
GLYPH LEFKOWITZ

A Guide to JAX for PyTorch Developers

“PyTorch users can learn about JAX in this tutorial that connects JAX concepts to the PyTorch building blocks that they’re already familiar with.”
ANFAL SIDDIQUI

Use a Python venv in a Container Like Docker

Bite code talks about why you should use a virtual environment inside of Docker containers, even though that is containment within a container.
BITE CODE!

Creating a Fitness Tracker App With Python Reflex

A detailed guide to building a simple Fitness Tracker app in Python using Reflex that lets you log the number of workouts completed per week.
BOB BELDERBOS • Shared by Bob Belderbos

Database Indexing in Django

This article explores the basics of database indexing, its advantages and disadvantages, and how to apply it in a Django application.
OLUWOLE MAJIYAGBE • Shared by Michael Herman

Python’s range() Function

The range() function can be used for counting upward, countdown downward, or performing an operation a number of times.
TREY HUNNER

Quickly Visualizing an SBOM Document

Quick instructions on how to chart a Software Bill-Of-Materials (SBOM) document.
SETH LARSON

Zasper: A Supercharged IDE for Data Science

N/A
ZASPER.IO • Shared by Poruri Sai Rahul

pyper: Concurrent Python Made Simple

GITHUB.COM/PYPER-DEV

freezegun: Mocks the datetime Module for Testing

GITHUB.COM/SPULEC

guppy3: Guppy/Heapy Ported to Python3

GITHUB.COM/ZHUYIFEI1999

VocabCLI: Word Insights TUI

GITHUB.COM/HIGHNESSATHARVA • Shared by Atharva Shah

Weekly Real Python Office Hours Q&A (Virtual)

January 22, 2025
REALPYTHON.COM

PyCon+Web 2025

January 24 to January 26, 2025
PYCONWEB.COM

PyDelhi User Group Meetup

January 25, 2025
MEETUP.COM

PythOnRio Meetup

January 25, 2025
PYTHON.ORG.BR

Python Leiden User Group

January 27, 2025
MEETUP.COM

Python Sheffield

January 28, 2025
GOOGLE.COM