Planet Python

Last update: May 14, 2025 09:42 PM UTC

May 14, 2025

Hugo van Kemenade

PEPs & Co.

PEPs #

Here’s Barry Warsaw on the origin of PEPs, or Python Enhancement Proposals (edited from PyBay 2017):

I like backronyms. For those who don’t know: a backronym is where you come up with the acronym first and then you come up with the thing that the acronym stands for. And I like funny sounding words, like FLUFL was one of those. When we were working for CNRI, they also ran the IETF conferences. The IETF is the Internet Engineering Task Force, and they’re the ones who come up with the RFCs. If you look at RFC 822, it defines what an email message looks like.

We got to a point, because we were at CNRI we were more intimately involved in the IETF and how they do standards and things, we observed at the time that there were so many interesting ideas coming in being proposed for Python that Guido really just didn’t have time to dive into the details of everything.

So I thought: well, we have this RFC process, let’s try to mirror some of that so that we can capture the essence of an idea in a document that would serve as a point of discussion, and that Guido could let people discuss and then come in and read the summary of the discussion.

And I was just kind of thinking: well, PEPs, that’s kind of peppy, it’s kind of a funny sounding word. I came up with the word and then I backronymed it into Python Enhancement Proposal. And then I wrote PEP 0 and PEP 1. PEP 0 was originally handwritten, and so I was the first PEP author because I came up with the name PEP.

But the really interesting thing is that you see the E.P. part used in a lot of other places, like Debian has DEPs now. There’s a lot of other communities that have these enhancement proposals so it’s kind of interesting. And then the format of the PEP was directly from that idea of the RFC’s standard.

& Co. #

Here’s a collection of enhancement proposals from different communities.

Are there more? Let me know!

Header photo: Grand Grocery Co., Lincoln, Nebraska, USA (1942) by The Library of Congress, with no known copyright restrictions.

May 14, 2025 03:45 PM UTC

Real Python

How to Get the Most Out of PyCon US

Read the full article at https://realpython.com/pycon-guide/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 14, 2025 02:00 PM UTC

Django Weblog

DSF member of the month - Simon Charette

For May 2025, we welcome Simon Charette as our DSF member of the month! ⭐

Simon Charette is a longtime Django contributor and community member. He served on the Django 5.x Steering Council and is part of the Security team and the Triage and Review team. He has been a DSF member since November 2014.
You can learn more about Simon by visiting Simon's GitHub Profile.

Let’s spend some time getting to know Simon better!

Can you tell us a little about yourself (hobbies, education, etc)

My name is Simon Charette and I'm based in Montréal. I've been contributing to Django for over a decade mainly to the ORM and I have a background in software engineering and mathematics. I work as a principal backend engineer at Zapier where we use Python and Django to power many of our backend services. Outside of Django and work I like to spend time cycling around the world, traveling with my partner, and playing ultimate frisbee.

Out of curiosity, your GitHub profile picture appears to be a Frisbee, is it correct? If so, have you been playing for a long time?

I've been playing ultimate frisbee since college which is around the time I started contributing to Django. It has been a huge part of my life since then as I made many friends and met my partner playing through the years. My commitment to ultimate frisbee can be reflected in my volume of contributions over the past decade as it requires more of my time during certain periods of the year. It also explains why I wasn't able to attend most DjangoCon in spring and fall as this is usually a pretty busy time for me. I took part in the world championships twice and I played in the UFA for about 5 years before retiring three years ago. Nowadays I still play but at a lower intensity level and I am focused on giving back to the community through coaching.

How did you start using Django?

Back in college I was working part time for a web agency that had an in house PHP framework and was trying to determine which tech stack and framework they should migrate to in order to ease onboarding of their developers and reduce their maintenance costs. I was tasked, with another member of the team, to identify potential candidates and despite my lack of familiarity with Python at the time we ended up choosing Django over PHP's Symphony mainly because of its spectacular documentation and third-party app ecosystem.

What other framework do you know and if there is anything you would like to have in Django if you had magical powers?

If I had magical powers I'd invent Python ergonomics to elegantly address the function coloring problem so it's easier for Django to be adapted to an async-ready world. I'm hopeful that the recent development on the GIL removal in Python 3.13+ will result a renewed interest in the usage of threading, which Django is well equipped to take advantage of, over the systematic usage of an event loop to deal with web serving workloads as the async world comes with a lot of often overlooked drawbacks.

What projects are you working on now?

I have a few Django related projects I'm working on mainly relating to ORM improvements (deprecating extra, better usage of RETURNING when available) but the main one has been a tool to keep track of the SQL generated by the Django test suite over time to more easily identity unintended changes that still pass the test suite. My goal with this project is to have a CI invokable command that would run the full Django test suite and provide a set of tests that generated different SQL compared to the target branch so its much easier to identify unintended side effects when making invasive changes to the ORM.

Which Django libraries are your favorite (core or 3rd party)?

• DRF
• django-filter
• django-seal (shameless plug)

What are the top three things in Django that you like?

• The people
• The ORM, unsurprisingly
• The many entry points the framework provides to allow very powerful third-party apps to be used together

You've contributed significantly to improving the Django ORM. What do you believe is the next big challenge for Django ORM, and how do you envision it evolving in the coming years?

The ORM's expression interface is already very powerful but there are effectively some remaining rough edges. I believe that adding generalized support for composite virtual fields (a field composed of other fields) could solve many problems we currently face with how relationships are expressed between models as we currently lack a way to describe an expression that can return tuples of values internally. If we had this building block, adding a way to express and compose table expressions (CTE, subquery pushdown, aggregation through subqueries) would be much easier to implement without denaturing the ORM by turning it into a low level query builder. Many of these things are possible today (e.g. django-cte) but they require a lot of SQL compilation and ORM knowledge and can hardly be composed together.

How did you start to contribute to the ORM? What would be the advice you have for someone interested to contribute to this field?

I started small by fixing a few issues that I cared about and by taking the time to read through Trac, mailing lists, and git-blame for changes in the area that were breaking tests as attempted to make changes. One thing that greatly helps in onboarding on the ORM is to at least have some good SQL fundamentals. When I first started I already had written a MSSQL ORM in PHP which helped me at least understand the idea behind the generation of SQL from a higher level abstraction. Nowadays there are tons of resources out there to help you get started on understand how things are organized but I would suggest this particular video where I attempt to walk through the different phases of SQL generation.

Is there anything else you’d like to say?

It has been a pleasure to be able to be part of this community for so long and I'd like to personally thank Claude Paroz for initially getting me interested in contributing seriously to the project.

Thank you for doing the interview, Simon !

May 14, 2025 12:00 PM UTC

eGenix.com

eGenix Antispam Bot for Telegram 0.7.1 GA

Introduction

eGenix has long been running a local user group meeting in Düsseldorf called Python Meeting Düsseldorf and we are using a Telegram group for most of our communication.

In the early days, the group worked well and we only had few spammers joining it, which we could well handle manually.

More recently, this has changed dramatically. We are seeing between 2-5 spam signups per day, often at night. Furthermore, the signups accounts are not always easy to spot as spammers, since they often come with profile images, descriptions, etc.

With the bot, we now have a more flexible way of dealing with the problem.

Please see our project page for details and download links.

Features

• Low impact mode of operation: the bot tries to keep noise in the group to a minimum
• Several challenge mechanisms to choose from, more can be added as needed
• Flexible and easy to use configuration
• Only needs a few MB of RAM, so can easily be put into a container or run on a Raspberry Pi
• Can handle quite a bit of load due to the async implementation
• Works with Python 3.9+
• MIT open source licensed
  

News

The 0.7.1 release fixes a few bugs and adds more features:

• Added missing dependency on emoji package to setup (bug introduced in 0.7.0, fixed in 0.7.1)
  
• Added user name check for number of emojis, since these are being used a lot by spammers
• Added wheel as requirement, since this is no longer included per default
• Updated copyright year

It has been battle-tested in production for several years already and is proving to be a really useful tool to help with Telegram group administration.

More Information

For more information on the eGenix.com Python products, licensing and download instructions, please write to sales@egenix.com.

Enjoy !

Marc-Andre Lemburg, eGenix.com

May 14, 2025 08:00 AM UTC

May 13, 2025

PyCoder’s Weekly

Issue #681: Loguru, GeoDjango, flexicache, and More (May 13, 2025)

#681 – MAY 13, 2025
View in Browser »

From try/except to Production Monitoring: Learn Python Error Handling the Right Way

This guide starts with the basics—errors vs. exceptions, how try/except works—and builds up to real-world advice on monitoring and debugging Python apps in production with Sentry. It’s everything you need to go from “I think it broke?” to “ai autofixed my python bug before it hit my users.” →
SENTRY sponsor

Python Jobs

Senior Software Engineer – Quant Investment Platform (LA or Dallas) (Los Angeles, CA, USA)

Causeway Capital Management LLC

More Python Jobs >>>

Articles & Tutorials

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #681.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

May 13, 2025 07:30 PM UTC

PyCharm

We’re excited to launch the second edition of our User Experience Survey for DataGrip and the Database Tools & SQL Plugin!

Your feedback from the previous survey helped us better understand your needs and prioritize the features and improvements that matter most to you.

Thanks to your input, we’ve already delivered a first set of enhancements focused on improving your experience:

• Faster introspection for MySQL and MariaDB (with more DBMS support coming soon!)
• New Quick Start Guide with sample database
• Non-modal Create and Modify dialogs
• AI-powered Fix and Explain SQL errors
• Better database context integration within the AI Assistant
• And much more: What’s New in DataGrip

Now, we’d love to hear from you again! Have these improvements made a difference for you? What should we focus on next to better meet your needs? 

The survey takes approximately 10 minutes to complete.

As a thank you, everyone who provides meaningful feedback will be entered to win:

• A $100 Amazon Gift Card
• A 1-year JetBrains All Products Pack (individual license)

Take the survey

Thank you for helping us build the best database tools!

May 13, 2025 04:40 PM UTC

DataGrip and Database Tools UX Survey #2

May 13, 2025 03:05 PM UTC

Real Python

Working With Missing Data in Polars

Efficiently handling missing data in Polars is essential for keeping your datasets clean during analysis. Polars provides powerful tools to identify, replace, and remove null values, ensuring seamless data processing.

This video course covers practical techniques for managing missing data and highlights Polars’ capabilities to enhance your data analysis workflow. By following along, you’ll gain hands-on experience with these techniques and learn how to ensure your datasets are accurate and reliable.

By the end of this video course, you’ll understand that:

• Polars allows you to handle missing data using LazyFrames and DataFrames.
• You can check for null values in Polars using the .null_count() method.
• NaN represents non-numeric values while null indicates missing data.
• You can replace NaN in Polars by converting them to nulls and using .fill_null().
• You can fix missing data by identifying, replacing, or removing null values.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 13, 2025 02:00 PM UTC

Daniel Roy Greenfeld

Exploring flexicache

An exploration of using flexicache for caching in Python.

May 13, 2025 01:35 PM UTC

Luke Plant

Knowledge creates technical debt

The term technical debt, now used widely in software circles, was coined to explain a deliberate process where you write software quickly to gain knowledge, and then you have to use that knowledge gained to improve your software.

This perspective is still helpful today when people speak of technical debt as only a negative, or only as a result of bad decisions. Martin Fowler’s Tech Debt Quadrant is a useful antidote to that.

A consequence of this perspective is that technical debt can appear at any time, apparently from nowhere, if you are unfortunate enough to gain some knowledge.

If you discover a better way to do things, the old way of doing it that is embedded in your code base is now “debt”:

• you can either live with the debt, “paying interest” in the form of all the ways that it makes your code harder to work with;

• or you can “pay down” the debt by fixing all the code in light of your new knowledge, which takes up front resources which could have been spent on something else, but hopefully will make sense in the long term.

This “better way” might be a different language, library, tool or pattern. In some cases, the better way has only recently been invented. It might be your own personal discovery, or something industry wide. It might be knowledge gained through the actual work of doing the current project (which was Ward Cunningham’s usage of the tem), or from somewhere else. But the end result is the same – you know more than you did, and now you have a debt.

The problem is that this doesn’t sound like a good thing. You learn something, and now you have a problem you didn’t have before, and it’s difficult to put a good spin on “I discovered a debt”.

But from another angle, maybe this perspective gives us different language to use when communicating with others and explaining why we need to address technical debt. Rather than say “we have a liability”, the knowledge we have gained can be framed as an opportunity. Failure to take the opportunity is an opportunity cost.

The “pile of technical debt” is essentially a pile of knowledge – everything we now think is bad about the code represents what we’ve learned about how to do software better. The gap between what it is and what it should be is the gap between what we used to know and what we now know.

And fixing that code is not “a debt we have to pay off”, but an investment opportunity that will reap rewards. You can refuse to take that opportunity if you want, but it’s a tragic waste of your hard-earned knowledge – a waste of the investment you previously made in learning – and eventually you’ll be losing money, and losing out to competitors who will be making the most of their knowledge.

Finally, I think phrasing it in terms of knowledge can help tame some of our more rash instincts to call everything we don’t like “tech debt”. Can I really say “we now know” that the existing code is inferior? Is it true that fixing the code is “investing my knowledge”? If it’s just a hunch, or a personal preference, or the latest fashion, maybe I can both resist the urge for unnecessary rewrites, and feel happier about it at the same time.

May 13, 2025 08:08 AM UTC

Talk Python to Me

#505: t-strings in Python (PEP 750)

Python has many string formatting styles which have been added to the language over the years. Early Python used the % operator to injected formatted values into strings. And we have string.format() which offers several powerful styles. Both were verbose and indirect, so f-strings were added in Python 3.6. But these f-strings lacked security features (think little bobby tables) and they manifested as fully-formed strings to runtime code. Today we talk about the next evolution of Python string formatting for advanced use-cases (SQL, HTML, DSLs, etc): t-strings. We have Paul Everitt, David Peck, and Jim Baker on the show to introduce this upcoming new language feature.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/connect'>Posit</a><br> <a href='https://talkpython.fm/auth0'>Auth0</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <h2 class="links-heading">Links from the show</h2> <div><strong>Guests:</strong><br/> <strong>Paul on X</strong>: <a href="https://x.com/paulweveritt?featured_on=talkpython" target="_blank" >@paulweveritt</a><br/> <strong>Paul on Mastodon</strong>: <a href="https://fosstodon.org/@pauleveritt" target="_blank" >@pauleveritt@fosstodon.org</a><br/> <strong>Dave Peck on Github</strong>: <a href="https://github.com/davepeck/?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>Jim Baker</strong>: <a href="https://github.com/jimbaker?featured_on=talkpython" target="_blank" >github.com</a><br/> <br/> <strong>PEP 750 – Template Strings</strong>: <a href="https://peps.python.org/pep-0750/?featured_on=talkpython" target="_blank" >peps.python.org</a><br/> <strong>tdom - Placeholder for future library on PyPI using PEP 750 t-strings</strong>: <a href="https://github.com/t-strings/tdom?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>PEP 750: Tag Strings For Writing Domain-Specific Languages</strong>: <a href="https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408?featured_on=talkpython" target="_blank" >discuss.python.org</a><br/> <strong>How To Teach This</strong>: <a href="https://peps.python.org/pep-0750/#how-to-teach-this" target="_blank" >peps.python.org</a><br/> <strong>PEP 501 – General purpose template literal strings</strong>: <a href="https://peps.python.org/pep-0501/?featured_on=talkpython" target="_blank" >peps.python.org</a><br/> <strong>Python's new t-strings</strong>: <a href="https://davepeck.org/2025/04/11/pythons-new-t-strings/?featured_on=talkpython" target="_blank" >davepeck.org</a><br/> <strong>PyFormat: Using % and .format() for great good!</strong>: <a href="https://pyformat.info?featured_on=talkpython" target="_blank" >pyformat.info</a><br/> <strong>flynt: A tool to automatically convert old string literal formatting to f-strings</strong>: <a href="https://github.com/ikamensh/flynt?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>Examples of using t-strings as defined in PEP 750</strong>: <a href="https://github.com/davepeck/pep750-examples/?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>htm.py issue</strong>: <a href="https://github.com/jviide/htm.py/issues/11?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>Exploits of a Mom</strong>: <a href="https://xkcd.com/327/?featured_on=talkpython" target="_blank" >xkcd.com</a><br/> <strong>pyparsing</strong>: <a href="https://github.com/pyparsing/pyparsing?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>Watch this episode on YouTube</strong>: <a href="https://www.youtube.com/watch?v=WCWNeZ_rE68" target="_blank" >youtube.com</a><br/> <strong>Episode transcripts</strong>: <a href="https://talkpython.fm/episodes/transcript/505/t-strings-in-python-pep-750" target="_blank" >talkpython.fm</a><br/> <br/> <strong>--- Stay in touch with us ---</strong><br/> <strong>Subscribe to Talk Python on YouTube</strong>: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a><br/> <strong>Talk Python on Bluesky</strong>: <a href="https://bsky.app/profile/talkpython.fm" target="_blank" >@talkpython.fm at bsky.app</a><br/> <strong>Talk Python on Mastodon</strong>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <strong>Michael on Bluesky</strong>: <a href="https://bsky.app/profile/mkennedy.codes?featured_on=talkpython" target="_blank" >@mkennedy.codes at bsky.app</a><br/> <strong>Michael on Mastodon</strong>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

May 13, 2025 08:00 AM UTC

Quansight Labs Blog

The first year of free-threaded Python

A recap of the first year of work on enabling support for the free-threaded build of CPython in community packages.

May 13, 2025 12:00 AM UTC

May 12, 2025

Paolo Melchiorre

My DjangoCon Europe 2025

A summary of my experience at DjangoCon Europe 2025 told through the posts I published on Mastodon during the conference.

May 12, 2025 10:00 PM UTC

Real Python

Python's T-Strings Coming Soon and Other Python News for May 2025

>>> template = t"Howdy, {input('Enter your name: ')}!"
Enter your name: Stephen

>>> template
Template(
    strings=('Howdy, ', '!'),
    interpolations=(
        Interpolation('Stephen', "input('Enter your name: ')", None, ''),
    )
)

>>> for item in template:
...     print(item)
...
Howdy,
Interpolation('Stephen', "input('Enter your name: ')", None, '')
!

Read the full article at https://realpython.com/python-news-may-2025/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 12, 2025 02:00 PM UTC

Python GUIs

What does @Slot() do? — Is the Slot decorator even necessary?

When working with Qt slots and signals in PySide6 you will discover the @Slot decorator. This decorator is used to mark a Python function or method as a slot to which a Qt signal can be connected. However, as you can see in our signals and slots tutorials you don't have to use this. Any Python function or method can be used, normally, as a slot for a Qt signals. But elsewhere, in our threading tutorials we do use it.

What's going on here?

• Why do you sometimes use @Slot but usually not?
• What happens when you omit it?
• Are there times when it is required?

What does the documentation say?

The PyQt6 documentation has a good explanation:

Although PyQt6 allows any Python callable to be used as a slot when connecting signals, it is sometimes necessary to explicitly mark a Python method as being a Qt slot and to provide a C++ signature for it. PyQt6 provides the pyqtSlot() function decorator to do this.

Connecting a signal to a decorated Python method has the advantage of reducing the amount of memory used and is slightly faster.

In PySide6 the decorator is named simply Slot() but is otherwise functionally compatible:

PySide6 adopts PyQt's new signal and slot syntax as-is. The PySide6 implementation is functionally compatible with the PyQt one [...].

From the above we see that:

• Any Python callable can be used as a slot when connecting signals.
• It is sometimes necessary to explicitly mark a Python method as being a Qt slot and to provide a C++ signature for it.
• There is a side-benefit in that marking a function or method with Slot() reduces the amount of memory used, and makes the slot faster.

When is it necessary?

Sometimes necessary is a bit vague. In practice the only situation where you need to use Slot decorators is when working with threads. This is because of a difference in how signal connections are handled in decorated vs. undecorated slots.

1. If you decorate a method with @Slot then that slot is created as a native Qt slot, and behaves identically
2. If you don't decorate the method then PySide6 will create a "proxy" object wrapper which provides a native slot to Qt

In normal use this is fine, aside from the performance impact (see below). But when working with threads, there is a complication: is the proxy object created on the GUI thread or on the runner thread. If it ends up on the wrong thread, this can lead to segmentation faults. Using the Slot decorator side-steps this issue, because no proxy is created.

When updating my PySide6 book I wondered -- is this still necessary?! -- and tested removing it from the examples. Many examples continue to work, but some failed. To be safe, use Slot decorators on your QRunnable.run methods.

What about performance?

The PyQt6 documentation notes that using native slots "has the advantage of reducing the amount of memory used and is slightly faster". But how much faster is it really, and does decorating slots actually save much memory?

We can test this directly by using this script from Oliver L Schoenborn. Updating for PySide6 (replace PyQt5 with PySide6 and pyqtSlot with Slot and it will work as-is) and running this we get the following results:

See the original results for PyQt5 for comparison.

First the results for the speed of emitting signals when connected to a decorated slot, vs non-decorated.

Raw slot mean, stddev:  1.608 0.066
Pyqt slot mean, stddev: 1.587 0.045
Percent gain with Slot: 1 %

The result shows Slot as 1% faster, but this is negligible (the original data on PyQt5 also showed no difference). So, using Slot will have no noticeable impact on the speed of signal handling in your applications.

Next are the results for establishing connections. This shows the speed, and memory usage of connecting to decorated vs. non-decorated slots.

Comparing mem and time required to create 10000000 connections, 1000 times

Measuring for 100000 connections
              # connects   mem (bytes)       time (sec)
Raw         :   100000     38670336 (36MB)    0.381
PySide Slot :   100000     17858560 (17MB)    0.426
Ratios      :                     2               1

The results show that decorated slots are marginally faster to connect to, but the difference is negligible. Based on these numbers, when connecting 100 signals the total execution time difference would be 0.03 ms vs 0.04 ms. This is negligible, not to mention imperceptible.

Perhaps more significant is that using raw connections uses 2x the memory of decorated connections. Again though, bear in mind that for a more realistic upper limit of connections (100) the actual difference here is 3.6KB vs 1.7KB.

The bottom line: don't expect any dramatic improvements in performance or memory usage from using slot decorators, unless you're working with insanely large numbers of signals or making regular connections you won't see any difference at all. That said, decorating your slots is an easy win if you need it.

Are there any other reasons to decorate a slot?

In Qt signals can be used to transmit more than one type of data by overloading signals and slots with different types.

For example, with the following code the my_slot_fn will only receive signals which match the signature of two int values.

@Slot(int, int)
def my_slot_fn(a, b):
    pass

This is a legacy of Qt5 and not recommended in new code. In Qt6 all of these signals have been replaced with separate signals with distinct names for different types. I recommend you follow the same approach in your own code for the sake of simplicity.

Conclusion

The Slot decorator can be used to mark Python functions or methods as Qt slots. This decorator is only required on slots which may be connected to across threads, for example the run method of QRunnable objects. For all other slots it can be omitted. There is a very small performance benefit to using it, which you may want to consider when your application makes a large number of signal/slot connections.

May 12, 2025 06:00 AM UTC

May 09, 2025

Real Python

The Real Python Podcast – Episode #248: Experiments With Gen AI, Knowledge Graphs, Workflows, and Python

Are you looking for some projects where you can practice your Python skills? Would you like to experiment with building a generative AI app or an automated knowledge graph sentiment analysis tool? This week on the show, we speak with Raymond Camden about his journey into Python, his work in developer relations, and the Python projects featured on his blog.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 09, 2025 12:00 PM UTC

Python Insider

Python 3.14.0 beta 1 is here!

Only one day late, welcome to the first beta!

https://www.python.org/downloads/release/python-3140b1/

This is a beta preview of Python 3.14

Python 3.14 is still in development. This release, 3.14.0b1, is the first of four planned beta releases.

Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release.

We strongly encourage maintainers of third-party Python projects to test with 3.14 during the beta phase and report issues found to the Python bug tracker as soon as possible. While the release is planned to be feature-complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Tuesday 2025-07-22). Our goal is to have no ABI changes after beta 4 and as few code changes as possible after the first release candidate. To achieve that, it will be extremely important to get as much exposure for 3.14 as possible during the beta phase.

Please keep in mind that this is a preview release and its use is not recommended for production environments.

Major new features of the 3.14 series, compared to 3.13

Some of the major new features and changes in Python 3.14 are:

New features

• PEP 649: The evaluation of type annotations is now deferred, improving the semantics of using annotations.
• PEP 750: Template string literals (t-strings) for custom string processing, using the familiar syntax of f-strings.
• PEP 784: A new module compression.zstd providing support for the Zstandard compression algorithm.
• PEP 758: except and except* expressions may now omit the brackets.
• Syntax highlighting in PyREPL, and support for color in unittest, argparse, json and calendar CLIs.
• PEP 768: A zero-overhead external debugger interface for CPython.
• UUID versions 6-8 are now supported by the uuid module, and generation of versions 3-5 and 8 are up to 40% faster.
• PEP 765: Disallow return/break/continue that exit a finally block.
• PEP 741: An improved C API for configuring Python.
• A new type of interpreter. For certain newer compilers, this interpreter provides significantly better performance. Opt-in for now, requires building from source.
• Improved error messages.
• Builtin implementation of HMAC with formally verified code from the HACL* project.

(Hey, fellow core developer, if a feature you find important is missing from this list, let Hugo know.)

For more details on the changes to Python 3.14, see What’s new in Python 3.14. The next pre-release of Python 3.14 will be 3.14.0b2, scheduled for 2025-05-27.

Build changes

• PEP 761: Python 3.14 and onwards no longer provides PGP signatures for release artifacts. Instead, Sigstore is recommended for verifiers.
• Official macOS and Windows release binaries include an experimental JIT compiler.

Incompatible changes, removals and new deprecations

• Incompatible changes
• Python removals and deprecations
• C API removals and deprecations
• Overview of all pending deprecations

Python install manager

The installer we offer for Windows is being replaced by our new install manager, which can be installed from the Windows Store or our FTP page. See our documentation for more information. The JSON file available for download contains the list of all the installable packages available as part of this release, including file URLs and hashes, but is not required to install the latest release. The traditional installer will remain available throughout the 3.14 and 3.15 releases.

More resources

• Online documentation
• PEP 745, 3.14 Release Schedule
• Report bugs at github.com/python/cpython/issues
• Help fund Python and its community

Note

During the release process, we discovered a test that only failed when run sequentially and only when run after a certain number of other tests. This appears to be a problem with the test itself, and we will make it more robust for beta 2. For details, see python/cpython#133532.

And now for something completely different

The mathematical constant pi is represented by the Greek letter π and represents the ratio of a circle’s circumference to its diameter. The first person to use π as a symbol for this ratio was Welsh self-taught mathematician William Jones in 1706. He was a farmer’s son born in Llanfihangel Tre’r Beirdd on Angelsy (Ynys Môn) in 1675 and only received a basic education at a local charity school. However, the owner of his parents’ farm noticed his mathematical ability and arranged for him to move to London to work in a bank.

By age 20, he served at sea in the Royal Navy, teaching sailors mathematics and helping with the ship’s navigation. On return to London seven years later, he became a maths teacher in coffee houses and a private tutor. In 1706, Jones published Synopsis Palmariorum Matheseos which used the symbol π for the ratio of a circle’s circumference to diameter (hunt for it on pages 243 and 263 or here). Jones was also the first person to realise π is an irrational number, meaning it can be written as decimal number that goes on forever, but cannot be written as a fraction of two integers.

But why π? It’s thought Jones used the Greek letter π because it’s the first letter in perimetron or perimeter. Jones was the first to use π as our familiar ratio but wasn’t the first to use it in as part of the ratio. William Oughtred, in his 1631 Clavis Mathematicae (The Key of Mathematics), used π/δ to represent what we now call pi. His π was the circumference, not the ratio of circumference to diameter. James Gregory, in his 1668 Geometriae Pars Universalis (The Universal Part of Geometry) used π/ρ instead, where ρ is the radius, making the ratio 6.28… or τ. After Jones, Leonhard Euler had used π for 6.28…, and also p for 3.14…, before settling on and popularising π for the famous ratio.

Enjoy the new release

Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organisation contributions to the Python Software Foundation.

Regards from Helsinki as the leaves begin to appear on the trees,

Your release team,

Hugo van Kemenade
Ned Deily
Steve Dower
Łukasz Langa

May 09, 2025 05:08 AM UTC

May 08, 2025

Python Engineering at Microsoft

Python in Visual Studio Code – May 2025 Release

We’re excited to announce the May 2025 release of the Python, Pylance and Jupyter extensions for Visual Studio Code!

This release includes the following announcements:

• Python Environments Quick Create command
• Python Environments chat tools
• Automatic environment activation with Python Environments (Experimental)
• Color picker with Pylance
• AI Code Actions: Convert Format String (Experimental)
• PyConUS 2025

If you’re interested, you can check the full list of improvements in our changelogs for the Python, Jupyter and Pylance extensions.

Python Environments Quick Create command

The Python Environments extension (preview) has added support for Quick Create, making the environment creation process more seamless. Quick Create minimizes the input needed from you to create new virtual environments by detecting the latest Python version on your machine to create an environment and install any workspace dependencies with a single click. This will create a .venv in your workspace for venv based environments, and .conda conda for conda based environments.

You can access Quick Create via the Python: Create Environment command in the Command Palette.

Python Environments chat tools

The Python Environments extension now includes two chat tools: “Get Python Environment Information” and “Install Python Package”. To use these tools, you can either directly reference them in your prompt by adding #pythonGetEnvironmentInfo and #pythonInstallPackage, or agent mode will automatically call the tool as applicable.

“Get Python Environment Information” seamlessly detects the appropriate environment information based on your file or workspace context. This includes the Python version, installed packages, and their versions.

“Install Python Package” automatically installs packages in the correct environment based on your workspace context. This means you can easily install packages without worrying about which environment you’re in or which package manager to use.

Automatic environment activation with Python Environments (Experimental)

The Python Environments extension introduced a new mechanism to auto activate your terminals with the correct environment information. The new "python-envs.terminal.autoActivationType" setting can be set to command (default), shellStartup, or off.

When set to command, the Python environments extension sends the appropriate activation command directly to the terminal resulting in activation.

Alternatively, with shell startup activation (shellStartup), the extension updates your shell’s startup script (such as .bashrc, .zshrc, or PowerShell profile) so that whenever you open a new terminal in VS Code, your chosen Python environment is automatically activated. This is only enabled for the zsh, fsh, pwsh, bash, and cmd. Once changes are written to your shell profile, your terminals will need to be refreshed in order for activation to occur.

If you want to undo these changes, simply run the Python Envs: Revert Shell Startup Script Changes command from the Command Palette. This will restore your shell profile and switch back to the previous activation mode.

Color picker with Pylance

Pylance can now display an interactive color swatch directly in the editor for recognized color values in Python files, making it easier to visualize and pick colors on the fly. To try it out, you can enable it by adding setting(python.analysis.enableColorPicker:true) to your settings.json file. Supported formats include #RGB (like “#001122”) and #RGBA (like “#001122FF”).

AI Code Actions: Convert Format String (Experimental)

When using Pylance, there’s a new experimental AI Code Action for converting string concatenations to f-string or format() enabled via "python.analysis.aiCodeActions": {"convertFormatString": true} setting. To try it out, select the Convert to f-string with Copilot or the Convert to format() call with Copilot Code Actions via the light bulb when selecting a symbol in the string you wish to convert, or through Ctrl + . / Cmd + ..

Then once you define a new symbol, for example, a class or a function, you can select the Generate Symbol with Copilot Code Action and let AI handle the implementation! If you want, you can then leverage Pylance’s Move Symbol Code Actions to move it to a different file.

PyConUS 2025

We will be attending PyCon US 2025 in Pittsburgh, PA, May 14-22, and cannot wait to connect with you all! Stop by booth #301 to say hello, learn along with us, and grab some swag!

There are a number of amazing earned and sponsored talks our team will be giving, so make sure to add them to your schedule:

Other Changes and Enhancements

We have also added small enhancements and fixed issues requested by users that should improve your experience working with Python and Jupyter Notebooks in Visual Studio Code. Some notable changes include:

• Branch coverage for Python is now supported in the Testing Explorer! Note that your coveragepy version must be >= 7.7 for this feature. You can upgrade coverage by running pip install coverage==7.7.

We would also like to extend special thanks to this month’s contributors:

• @hutch3232 Prevent native REPL from caching state in between sessions in vscode-python#24857
• @danila-grobov Django tests run with coverage in vscode-python#24927
• @kinto0 Disable language services if Pyrefly extension is installed and active in vscode-python#24987

Try out these new improvements by downloading the Python extension and the Jupyter extension from the Marketplace, or install them directly from the extensions view in Visual Studio Code (Ctrl + Shift + X or ⌘ + ⇧ + X). You can learn more about Python support in Visual Studio Code in the documentation. If you run into any problems or have suggestions, please file an issue on the Python VS Code GitHub page.

The post Python in Visual Studio Code – May 2025 Release appeared first on Microsoft for Python Developers Blog.

May 08, 2025 10:39 PM UTC

eGenix.com

PyDDF Python Spring Sprint 2025

The following text is in German, since we're announcing a Python sprint in Düsseldorf, Germany.

Ankündigung

Python Meeting Spring Sprint 2025 in
Düsseldorf

Samstag, 24.05.2025, 10:00-18:00 Uhr
Sonntag, 25.05.2025. 10:00-18:00 Uhr

Eviden / Atos Information Technology GmbH, Am Seestern 1, 40547 Düsseldorf

Informationen

• Sprint Ort in Google Maps
  

• Mobile Apps: App Entwicklung mit Python und Flet
• Mobile Apps: App Entwicklung mit Python und BeeWare
• LaTeX: Tikz Support für Python und matplotlib
• Deutsche Cloud Stacks: Managed Stackable Data Platform als Alternative zu US Clouds
• Django: Schnelles Deployen von Django Apps mit django-simple-deploy
  

Anmeldung, Kosten und weitere Infos

Alles weitere und die Anmeldung findet Ihr auf der Meetup Sprint Seite:

• Python Spring Sprint & Hackathon in Düsseldorf
  

• PyDDF Telegram Gruppe

Über das Python Meeting Düsseldorf

Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python-Begeisterte aus der Region wendet.

Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf.

Marc-André Lemburg, eGenix.com

May 08, 2025 09:00 AM UTC

Test and Code

pytest-metadata - provides access to test session metadata

pytest-metadata is described as a plugin for pytest that provides access to test session metadata. 
That is such a humble description for such a massively useful plugin. 
If you're already using pytest-html, you have pytest-metadata already installed, as pytest-metadata is one of the dependencies for pytest-html.
However, pytest-metadata is very useful even on its own.

Links:

• pytest-metadata - The plugin we're talking about in this episode
• pytest-base-url - Adds the base URL to the metadata.
• pytest-html - Displays the metadata at the start of each report. 
  • See S2:E6: pytest-html - a plugin that generates HTML reports for test results
• pytest-reporter-html1 - Presents metadata as part of the report.
• pytest-selenium - Adds the driver, capabilities, and remote server to the metadata.

If you've got other plugins that work well with pytest-metadata, please let me know.

Sponsored by: 

• Porkbun -- named the #1 domain registrar by USA Today from 2023 to 2025!
• Get a .app or.dev domain name for only $5.99 first year.

Help support the show AND learn pytest: 

• The Complete pytest course is now a bundle, with each part available separately.
  • pytest Primary Power teaches the super powers of pytest that you need to learn to use pytest effectively.
  • Using pytest with Projects has lots of "when you need it" sections like debugging failed tests, mocking, testing strategy, and CI
  • Then pytest Booster Rockets can help with advanced parametrization and building plugins.
• Whether you need to get started with pytest today, or want to power up your pytest skills, PythonTest has a course for you.

May 08, 2025 05:57 AM UTC

Seth Michael Larson

A(nimal Cros)SCII

What is the character encoding for Animal Crossing? This page details all the characters that are allowed for player names, town names, and passwords in Animal Crossing for the GameCube. A much larger character set was used for writing mail and communication.

Each character was internally represented as a value between 0x00 and 0xFF which is the same size as ASCII, thus me naming this encoding “Animal CrosSCII”. The characters between 0xD5 and 0xDE are only used in the EU version. 0xDF to 0xFF are unused.

The names of the characters (and descriptions when ambiguous) were sourced from the Animal Crossing decompilation project. I've mapped each character to the best of my ability back to Unicode and collected them in a table below:

And here's the Unicode characters laid out into a “square”:

¡¿ÄÀÁÂÃȦÇÈÉÊËÌÍÎ
ÏĐÑÒÓÔÕÖØÙÚÛÜβ?à
 !"áâ%&'()~♥,-.𝅘𝅥𝅮
0123456789:🌢<=>?
@ABCDEFGHIJKLMNO
PQRSTUVWXYZã💢äȧ_
çabcdefghijklmno
pqrstuvwxyzèéêë 
 ìíîï·?ñòóôõöøùú
-ûüýÿþÝ|§ªº∥ᵧ³²¹
¯¬Ææ„»«☀☁☂꩜☃⚞⚟/∞
⭕❌☐△+⚡♂♀⚘★☠😯😄😞😠😃
×÷🔨🎀✉💰🐾🐶🐱🐰🐦🐮🐷␤🐟🪲
;#  🔑“”‘’Œœᵉ  \\

I wasn't able to find Unicode equivalents to some of the characters and a few others (marked with ??? in the table). If you're able to find Unicode characters for the missing ones please send me an email or pull request. Time to brush up on Animalese!

May 08, 2025 12:00 AM UTC

May 07, 2025

The Python Coding Stack

A Story About Parameters and Arguments in Python Functions • "AI Coffee" Grand Opening This Monday

Alex had one last look around. You could almost see a faint smile emerge from the deep sigh—part exhaustion and part satisfaction. He was as ready as he could be. His new shop was as ready as it could be. There was nothing left to set up. He locked up and headed home. The grand opening was only seven hours away, and he'd better get some sleep.

Grand Opening sounds grand—too grand. Alex had regretted putting it on the sign outside the shop's window the previous week. This wasn't a vanity project. He didn't send out invitations to friends, journalists, or local politicians. He didn't hire musicians or waiters to serve customers. Grand Opening simply meant opening for the first time.

Alex didn't really know what to expect on the first day. Or maybe he did—he wasn't expecting too many customers. Another coffee shop on the high street? He may need some time to build a loyal client base.

• • •

He had arrived early on Monday. He'd been checking the lights, the machines, the labels, the chairs, the fridge. And then checking them all again. He glanced at the clock—ten minutes until opening time. But he saw two people standing outside. Surely they were just having a chat, only standing there by chance. He looked again. They weren't chatting. They were waiting.

Waiting for his coffee shop to open? Surely not?

But rather than check for the sixth time that the labels on the juice bottles were facing outwards, he decided to open the door a bit early. And those people outside walked in. They were AI Coffee's first customers.

Today's article is an overview of the parameters and arguments in Python's functions. It takes you through some of the key principles and discusses the various types of parameters you can define and arguments you can pass to a Python function. There are five numbered sections interspersed within the story in today's article:

1. Parameters and Arguments

2. Positional and Keyword Arguments

3. Args and Kwargs

4. Optional Arguments with Default Values

5. Positional-Only and Keyword-Only Arguments

Espressix ProtoSip v0.1 (AlphaBrew v0.1.3.7)

Introducing the Espressix ProtoSip, a revolutionary coffee-making machine designed to elevate the art of brewing for modern coffee shops. With its sleek, futuristic design and cutting-edge technology, this prototype blends precision engineering with intuitive controls to craft barista-quality espresso, cappuccino, and more. Tailored for innovators, the Espressix delivers unparalleled flavour extraction and consistency, setting a new standard for coffee excellence while hinting at the bold future of café culture.

Alex had taken a gamble with the choice of coffee machine for his shop. His cousin set up a startup some time earlier that developed an innovative coffee machine for restaurants and coffee shops. The company had just released its first prototype, and they offered Alex one at a significantly reduced cost since it was still a work in progress—and he was the founder's cousin!

The paragraph you read above is the spiel the startup has on its website and on the front cover of the slim booklet that came with the machine. There was little else in the booklet. But an engineer from the startup company had spent some time explaining to Alex how to use the machine.

The Espressix didn't have a user interface yet—it was still a rather basic prototype. Alex connected the machine to a laptop. He was fine calling functions from the AlphaBrew Python API directly from a terminal window—AlphaBrew is the software that came with the Espressix.

What the Espressix did have, despite being an early prototype, is a sleek and futuristic look. One of the startup's cofounders was a product design graduate, so she went all out with style and looks.

1. Parameters and Arguments

"You're AI Coffee's first ever customer", Alex told the first person to walk in. "What can I get you?"

"Wow! I'm honoured. Could I have a strong Cappuccino, please, but with a bit less milk?"

"Sure", and Alex tapped at his laptop:

All code blocks are available in text format at the end of this article • #1 • The code images used in this article are created using Snappify. [Affiliate link]

And the Espressix started whizzing. A few seconds later, the perfect brew poured into a cup.

Here's the signature for the brew_coffee() function Alex used:

#2

Alex was a programmer before deciding to open a coffee shop. He was comfortable with this rudimentary API to use the machine, even though it wasn't ideal. But then, he wasn't paying much to lease the machine, so he couldn't complain!

The coffee_type parameter accepts a string, which must match one of the available coffee types. Alex is already planning to replace this with enums to prevent typos, but that's not a priority for now.

The strength parameter accepts integers between 1 and 5. And milk also accepts integers up to 5, but the range starts from 0 to cater for coffees with no milk.

Terminology can be confusing, and functions come with plenty of terms. Parameter and argument are terms that many confuse. And it doesn't matter too much if you use one instead of the other. But, if you prefer to be precise, then:

• Use parameter for the name you choose to refer to values you pass into a function. The parameter is the name you place within parentheses when you define a function. This is the variable name you use within the function definition. The parameters in the above example are coffee_type, strength, and milk_amount.

• Use argument for the object you pass to the function when you call it. An argument is the value you pass to a function. In the example above, the arguments are "Cappuccino", 4, and 2.

When you call a function, you pass arguments. These arguments are assigned to the parameter names within the function.

To confuse matters further, some people use formal parameters to refer to parameters and actual parameters to refer to arguments. But the terms parameters and arguments as described in the bullet points above are more common in Python, and they're the ones I use here and in all my writing.

Alex's first day went better than he thought it would. He had a steady stream of customers throughout the day. And they all seemed to like the coffee.

But let's see what happened on Alex's second day!

2. Positional and Keyword Arguments

Chloezz @chloesipslife • 7m

Just visited the new AI Coffee shop on my high street, and OMG, it’s like stepping into the future! The coffee machine is a total sci-fi vibe—sleek, futuristic, and honestly, I have no clue how it works, but it’s powered by AI and makes a mean latte! The coffee? Absolutely delish. If this is what AI can do for my morning brew, I’m here for it! Who’s tried it? #AICoffee #CoffeeLovers #TechMeetsTaste

— from social media

Alex hadn't been on social media after closing the coffee shop on the first day. Even if he had, he probably wouldn't have seen Chloezz's post. He didn't know who she was. But whoever she is, she has a massive following.

Alex was still unaware his coffee shop had been in the spotlight when he opened up on Tuesday. There was a queue outside. By mid-morning, he was no longer coping. Tables needed cleaning, fridge shelves needed replenishing, but there had been no gaps in the queue of customers waiting to be served.

And then Alex's sister popped in to have a look.

"Great timing. Here, I'll show you how this works." Alex didn't hesitate. His sister didn't have a choice. She was now serving coffees while Alex dealt with everything else.

• • •

But a few minutes later, she had a problem. A take-away customer came back in to complain about his coffee. He had asked for a strong Americano with a dash of milk. Instead, he got what seemed like the weakest latte in the world.

Alex's sister had typed the following code to serve this customer:

#3

But the function's signature is:

#4

I dropped the type hints, and I won't use them further in this article to focus on other characteristics of the function signature.

Let's write a demo version of this function to identify what went wrong:

#5

The first argument, "Americano", is assigned to the first parameter, coffee_type. So far, so good…

But the second argument, 1, is assigned to strength, which is the second parameter. Python can only determine which argument is assigned to which parameter based on the position of the argument in the function call. Python is a great programming language, but it still can't read the user's mind!

And then, the final argument, 4, is assigned to the final parameter, milk_amount.

Alex's sister had swapped the two integers. An easy mistake to make. Instead of a strong coffee with a little milk, she had input the call for a cup of hot milk with just a touch of coffee. Oops!

Here's the output from our demo code to confirm this error:

Coffee type: Americano
Strength: 1
Milk Amount: 4

Alex apologised to the customer, and he made him a new coffee.

"You can do this instead to make sure you get the numbers right," he showed his sister as he prepared the customer's replacement drink:

#6

Note how the second and third arguments now also include the names of the parameters.

"This way, it doesn't matter what order you input the numbers since you're naming them", he explained.

Here's the output now:

Coffee type: Americano
Strength: 4
Milk Amount: 1

Even though the integer 1 is still passed as the second of the three arguments, Python now knows it needs to assign this value to milk_amount since the parameter is named in the function call.

When you call a function such as brew_coffee(), you have the choice to use either positional arguments or keyword arguments.

Arguments are positional when you pass the values directly without using the parameter names, as you do in the following call:

brew_coffee("Americano", 1, 4)

You don't use the parameter names. You only include the values within the parentheses. These arguments are assigned to parameter names depending on their order.

Keyword arguments are the arguments you pass using the parameter names, such as the following call:

brew_coffee(coffee_type="Americano", milk_amount=1, strength=4)

In this example, all three arguments are keyword arguments. You pass each argument matched to its corresponding parameter name. The order in which you pass keyword arguments no longer matters.

Keyword arguments can also be called named arguments.

Positional and keyword arguments: Mixing and matching

But look again at the code Alex used when preparing the customer's replacement drink:

#7

The first argument doesn't have the parameter name. The first argument is a positional argument and, therefore, it's assigned to the first parameter, coffee_type.

However, the remaining arguments are keyword arguments. The order of the second and third arguments no longer matters.

Therefore, you can mix and match positional and keyword arguments.

But there are some rules! Try the following call:

#8

You try to pass the first and third arguments as positional and the second as a keyword argument, but…

  File "...", line 8
    brew_coffee("Americano", milk_amount=1, 4)
                                             ^
SyntaxError: positional argument follows
    keyword argument

Any keyword arguments must come after all the positional arguments. Once you include a keyword argument, all the remaining arguments must also be passed as keyword arguments.

And this rule makes sense. Python can figure out which argument goes to which parameter if they're in order. But the moment you include a keyword argument, Python can no longer assume the order of arguments. To avoid ambiguity—we don't like ambiguity in programming—Python doesn't allow any more positional arguments once you include a keyword argument.

3. Args and Kwargs

Last week, AI Coffee, a futuristic new coffee shop, opened its doors on High Street, drawing crowds with its sleek, Star Trek-esque coffee machine. This reporter visited to sample the buzzworthy brews and was wowed by the rich, perfectly crafted cappuccino, churned out by the shop’s mysterious AI-powered machine. Eager to learn more about the technology behind the magic, I tried to chat with the owner, but the bustling shop kept him too busy for a moment to spare. While the AI’s secrets remain under wraps for now, AI Coffee is already a local hit, blending cutting-edge tech with top-notch coffee.

— from The Herald, a local paper

Alex had started to catch up with the hype around his coffee shop—social media frenzy, articles in local newspapers, and lots of word-of-mouth. He wasn't complaining, but he was perplexed at why his humble coffee shop had gained so much attention and popularity within its first few days. Sure, his coffee was great, but was it so much better than others? And his prices weren't the highest on the high street, but they weren't too cheap, either.

However, with the increased popularity, Alex also started getting increasingly complex coffee requests. Vanilla syrup, cinnamon powder, caramel drizzle, and lots more.

Luckily, the Espressix ProtoSip was designed with the demanding urban coffee aficionado in mind.

Args

Alex made some tweaks to his brew_coffee() function:

#9

There's a new parameter in brew_coffee(). This is the *args parameter, which has a leading * in front of the parameter name. This function can now accept any number of positional arguments following the first three. We'll explore what the variable name args refers to shortly. But first, let's test this new function:

#10

You call the function with five arguments. And here's the output from this function call:

Coffee type: Latte
Strength: 3
Milk Amount: 2
Add-ons: cinnamon, hazelnut syrup

1. The first argument, "Latte", is assigned to the first parameter, coffee_type.

2. The second argument, 3, is assigned to the second parameter, strength.

3. The third argument, 2, is assigned to the third parameter, milk_amount.

4. The remaining two arguments, "cinnamon" and "hazelnut syrup", are assigned to args, which is a tuple.

You can confirm that args is a tuple with a small addition to the function:

#11

The first two lines of the output from this code are shown below:

args=('cinnamon', 'hazelnut syrup')
<class 'tuple'>

The parameter name args is a tuple containing the remaining positional arguments in the function call once the function deals with the first three.

There's nothing special about the name args

What gives *args its features? It's not the name args. Instead, it's the leading asterisk, *, that makes this parameter one that can accept any number of positional arguments. The parameter name args is often used in this case, but you can also use a name that's more descriptive to make your code more readable:

#12

Alex uses the name add_ons instead of args. This parameter name still has the leading * in the function signature. Colloquially, many Python programmers will still call a parameter with a leading * the args parameter, even though the parameter name is different.

Therefore, you can now call this function with three or more arguments. You can add as many arguments as you wish after the third one, including none at all:

#13

The output confirms that add_ons is now an empty tuple:

add_ons=()
<class 'tuple'>
Coffee type: Latte
Strength: 3
Milk Amount: 2
Add-ons: 

This coffee doesn't have any add-ons.

We have a problem

However, Alex's sister, who was now working in the coffee shop full time, could no longer use her preferred way of calling the brew_coffee() function:

#14

This raises an error:

  File "...", line 9
    brew_coffee("Latte", strength=3,
        milk_amount=2, "vanilla syrup")
                                      ^
SyntaxError: positional argument follows
    keyword argument

This is a problem you've seen already. Positional arguments must come before keyword arguments in a function call. And *add_ons in the function signature indicates that Python will collect all remaining positional arguments from this point in the parameter list. Therefore, none of the parameters defined before *add_ons can be assigned a keyword argument if you also include args as arguments. They must all be assigned positional arguments.

All arguments preceding the args arguments in a function call must be positional arguments.

Alex refactored the code:

#15

The *add_ons parameter is now right after coffee_type. The remaining parameters, strength and milk_amount, come next. Unfortunately, this affects how Alex and his growing team can use brew_coffee() in other situations, too. The strength and milk_amount arguments must now come after any add-ons, and they must be used as keyword arguments.

See what happens if you try to pass positional arguments for strength and milk_amount:

#16

This raises an error:

Traceback (most recent call last):
  File "...", line 9, in <module>
    brew_coffee("Latte", "vanilla syrup", 3, 2)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: brew_coffee() missing
    2 required keyword-only arguments:
    'strength' and 'milk_amount'

The args parameter, which is *add_ons in this example, marks the end of the positional arguments in a function. Therefore, strength and milk_amount must be assigned arguments using keywords.

Alex instructed his team on these two changes:

1. Any add-ons must go after the coffee type.

2. They must use keyword arguments for strength and milk_amount.

It's a bit annoying that they have to change how to call the function but they're all still learning and Alex feels this is a safer option.

Kwargs

But Alex's customers also had other requests. Some wanted their coffee extra hot, others needed oat milk, and others wanted their small coffee served in a large cup.

Alex included this in brew_coffee() by adding another parameter:

#17

The new parameter Alex added at the end of the signature, **kwargs, has two leading asterisks, **. This parameter indicates that the function can accept any number of optional keyword arguments after all the other arguments.

Whereas *args creates a tuple called args within the function, the double asterisk in **kwargs creates a dictionary called kwargs. The best way to see this is to call this function with additional keyword arguments:

#18

The final two arguments use the keywords milk_type and temperature. These are not parameters in the function definition.

Let's explore these six arguments:

• The first argument, "Latte", is the required argument assigned to coffee_type.

• The second argument, "vanilla syrup", is also a positional argument. Therefore, Python assigns this to add_ons. There's only one additional positional argument in this call but, in general, you can have more.

• Next, you have the two required keyword arguments, strength=3 and milk_amount=2.

• But there are also two more keyword arguments, milk_type="oat" and temperature="extra hot". These are the additional optional keyword arguments, and they're assigned to the dictionary kwargs.

Here is the first part of the output from this call:

kwargs={
    'milk_type': 'oat',
    'temperature': 'extra hot'
}
<class 'dict'>

This confirms that kwargs is a dictionary. The keywords are the keys, and the argument values are the dictionary values.

The rest of the output shows the additional special instructions in the printout:

Coffee type: Latte
Strength: 3
Milk Amount: 2
Add-ons: vanilla syrup
Instructions:
	milk type: oat
	temperature: extra hot

There's nothing special about the name kwargs

You've seen this when we talked about args. There's nothing special about the parameter name kwargs. It's the leading double asterisk that does the trick. So, you can use any descriptive name you wish in your code:

#19

Warning: the following paragraph is dense with terminology!

So, in its current form, this function needs a required argument assigned to coffee_type and two required keyword arguments assigned to strength and milk_amount. And you can also have any number of optional positional arguments, which you add after the first positional argument but before the required keyword arguments. These are the add-ons a customer wants in their coffee.

But you can also add any number of keyword arguments at the end of the function call. These are the special instructions from the customer.

Both args and kwargs are optional. So, you can still call the function with only the required arguments:

#20

The output shows that this gives a strong espresso with no milk, no add-ons, and no special instructions:

instructions={}
<class 'dict'>
Coffee type: Espresso
Strength: 4
Milk Amount: 0
Add-ons: 
Instructions:

Note that in this case, since there are no args, you can also pass the first argument as a keyword argument:

#21

But this is only possible when there are no add-ons—no args. We'll revisit this case in a later section of this article.

A quick recap before we move on.

Args and kwargs are informal terms used for parameters with a leading single and double asterisk.

The term args refers to a parameter with a leading asterisk in the function's signature, such as *args. This parameter indicates that the function can accept any number of optional positional arguments following any required positional arguments. The term args stands for arguments, but you've already figured that out!

And kwargs refers to a parameter with two leading asterisks, such as **kwargs, which indicates that the function can accept any number of optional keyword arguments following any required keyword arguments. The 'kw' in kwargs stands for keyword.

Coffee features often when talking about programming. Here's another coffee-themed article, also about functions: What Can A Coffee Machine Teach You About Python's Functions?

4. Optional Arguments with Default Values

Alex's team grew rapidly. The coffee shop now had many regular customers and a constant stream of coffee lovers throughout the day.

Debra, one of the staff members, had some ideas to share in a team meeting:

"Alex, many customers don't care about the coffee strength. They just want a normal coffee. I usually type in 3 for the strength argument for these customers. But it's time-consuming to have to write strength=3 for all of them, especially when it's busy."

"We can easily fix that", Alex was quick to respond:

#22

The parameter strength now has a default value. This makes the argument corresponding to strength an optional argument since it has a default value of 3. The default value is used by the function only if you don't pass the corresponding argument.

Alex's staff can now leave this argument out if they want to brew a "normal strength" coffee:

#23

This gives a medium strength espresso with no add-ons or special instructions:

Coffee type: Espresso
Strength: 3
Milk Amount: 0
Add-ons: 
Instructions:

The output confirms that the coffee strength has a value of 3, which is the default value. And here's a coffee with some add-ons that also uses the default coffee strength:

#24

Here's the output confirming this normal-strength caramel-drizzle latte:

Coffee type: Latte
Strength: 3
Milk Amount: 2
Add-ons: caramel drizzle
Instructions:

Ambiguity, again

Let's look at the function's signature again:

#25

The coffee_type parameter can accept a positional argument. Then, *add_ons collects all remaining positional arguments, if there are any, that the user passes when calling the function. Any argument after this must be a keyword argument. Therefore, when calling the function, there's no ambiguity whether strength, which is optional, is included or not, since all the arguments after the add-ons are named.

Why am I mentioning this? Consider a version of this function that doesn't have the args parameter *add_ons:

#26

I commented out the lines with *add_ons to highlight they've been removed temporarily in this function version. When you run this code, Python raises an error. Note that the error is raised in the function definition before the function call itself:

  File "...", line 5
    milk_amount,
    ^^^^^^^^^^^
SyntaxError: parameter without a default follows
    parameter with a default

Python doesn't allow this function signature since this format introduces ambiguity. To see this ambiguity, let's use a positional argument for the amount of milk, since this would now be possible as *add_ons is no longer there. Recall that in the main version of the function with the parameter *add_ons, all the arguments that follow the args must be named:

#27

As mentioned above, note that the error is raised by the function definition and not the function call. I'm showing these calls to help with the discussion.

Is the value 0 meant for strength, or is your intention to use the default value for strength and assign the value 0 to milk_amount? To avoid this ambiguity, Python function definitions don't allow parameters without a default value to follow a parameter with a default value. Once you add a default value, all the following parameters must also have a default value.

Of course, there would be no ambiguity if you use a keyword argument. However, this would lead to the situation where the function call is ambiguous with a positional argument, but not when using a keyword argument, even though both positional and keyword arguments are possible. Python doesn't allow this to be on the safe side!

This wasn't an issue when you had *add_ons as part of the signature. Let’s put *add_ons back in:

#28

There's no ambiguity in this case since strength and milk_amount must both have keyword arguments.

However, even though this signature is permitted in Python, it's rather unconventional. Normally, you don't see many parameters without default values after ones with default values, even when you're already in the keyword-only region of the function (after the args).

In this case, Debra's follow-up suggestion fixes this unconventional function signature:

"And we also have to input milk_amount=0 for black coffees, which are quite common. Can we do a similar trick for coffees with no milk?"

"Sure we can"

#29

Now, there's also a default value for milk_amount. The default is a black coffee.

In this version of the function, there's only one required argument—the first one that's assigned to coffee_type. All the other arguments are optional either because they're not needed to make a coffee, such as the add-ons and special instructions, or because the function has default values for them, such as strength and milk_amount.

A parameter can have a default value defined in the function's signature. Therefore, the argument assigned to a parameter with a default value is an optional argument.

And let's confirm you can still include add-ons and special instructions:

#30

Here's the output from this function call:

Coffee type: Cappuccino
Strength: 3
Milk Amount: 2
Add-ons: chocolate sprinkles, vanilla syrup
Instructions:
	temperature: extra hot
	cup size: large cup

Note that you rely on the default value for strength in this example since the argument assigned to strength is not included in the call.

A common pitfall with default values in function definitions is the mutable default value trap. You can read more about this in section 2, The Teleportation Trick, in this article: Python Quirks? Party Tricks? Peculiarities Revealed…

Support The Python Coding Stack

5. Positional-Only and Keyword-Only Arguments

Let's summarise the requirements for all the arguments in Alex's current version of the brew_coffee() function. Here's the current function signature:

#31

1. The first parameter is coffee_type, and the argument you assign to this parameter can be either a positional argument or a keyword argument. But—and this is important—you can only use it as a keyword argument if you don't pass any arguments assigned to *add_ons. Remember that positional arguments must come before keyword arguments in function calls. Therefore, you can only use a keyword argument for the first parameter if you don't have args. We'll focus on this point soon.

2. As long as the first argument, the one assigned to coffee_type, is positional, any further positional arguments are assigned to the tuple add_ons.

3. Next, you can add named arguments (which is another term used for keyword arguments) for strength and milk_amount. Both of these arguments are optional, and the order in which you use them in a function call is not important.

4. Finally, you can add more keyword arguments using keywords that aren't parameters in the function definition. You can include as many keyword arguments as you wish.

Read point 1 above again. Alex thinks that allowing the first argument to be either positional or named is not a good idea, as it can lead to confusion. You can only use the first argument as a keyword argument if you don't have add-ons. Here's proof:

#32

The first argument is a keyword argument, coffee_type="Cappuccino". But then you attempt to pass two positional arguments, chocolate sprinkles and vanilla syrup. This call raises an error:

File "...", line 25
    )
    ^
SyntaxError: positional argument follows
    keyword argument

You can't have positional arguments following keyword arguments.

Alex decides to remove this source of confusion by ensuring that the argument assigned to coffee_type is always a positional argument. He only needs to make a small addition to the function's signature:

#33

The rogue forward slash, /, in place of a parameter is not a typo. It indicates that all parameters before the forward slash must be assigned positional arguments. Therefore, the object assigned to coffee_type can no longer be a keyword argument:

#34

The first argument is a keyword argument. But this call raises an error:

Traceback (most recent call last):
  File "...", line 19, in <module>
    brew_coffee(
    ~~~~~~~~~~~^
        coffee_type="Cappuccino",
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        cup_size="large cup",
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
TypeError: brew_coffee() missing 1 required
    positional argument: 'coffee_type'

The function has a required positional argument, the one assigned to coffee_type. The forward slash, /, makes the first argument a positional-only argument. It can no longer be a keyword argument:

#35

This version works fine since the first argument is positional:

Coffee type: Cappuccino
Strength: 3
Milk Amount: 2
Add-ons: 
Instructions:
	temperature: extra hot
	cup size: large cup

Alex feels that this function's signature is neater and clearer now, avoiding ambiguity.

• • •

The R&D team at the startup that's developing the Espressix ProtoSip were keen to see how Alex was using the prototype and inspect the changes he made to suit his needs. They implemented many of Alex's changes.

However, they were planning to offer a more basic version of the Espressix that didn't have the option to include add-ons in the coffee.

The easiest option is to remove the *add-ons parameter from the function's signature:

#36

No *add_ons parameter, no add-ons in the coffee.

Sorted? Sort of.

The *add_ons parameter enabled you to pass optional positional arguments. However, *add_ons served a second purpose in the earlier version. All parameters after the args parameter, which is *add_ons in this example, must be assigned keyword arguments. The args parameter, *add_ons, forces all remaining parameters to be assigned keyword-only arguments.

Removing the *add_ons parameter changes the rules for the remaining arguments.

But you can still implement the same rules even when you're not using args. All you need to do is keep the leading asterisk but drop the parameter name:

#37

Remember to remove the line printing out the add-ons, too. That’s the second of the highlighted lines in the code above.

Notice how there's a lone asterisk in one of the parameter slots in the function signature. You can confirm that strength and milk_amount still need to be assigned keyword arguments:

#38

When you try to pass positional arguments to strength and milk_amount, the code raises an error:

Traceback (most recent call last):
  brew_coffee(
    ~~~~~~~~~~~^
        "Espresso",
        ^^^^^^^^^^^
        3,
        ^^
        0,
        ^^
    )
    ^
TypeError: brew_coffee() takes 1 positional argument
    but 3 were given

The error message tells you that brew_coffee() only takes one positional argument. All the arguments after the * are keyword-only. Therefore, only the arguments preceding it may be positional. And there's only one parameter before the rogue asterisk, *.

A lone forward slash, /, among the function's parameters indicates that all parameters before the forward slash must be assigned positional-only arguments.

A lone asterisk, *, among the function's parameters indicates that all parameters after the asterisk must be assigned keyword-only arguments.

If you re-read the statements above carefully, you'll conclude that when you use both / and * in a function definition, the / must come before the *. Recall that positional arguments must come before keyword arguments.

It's also possible to have parameters between the / and *:

#39

You add a new parameter, another_param, in between / and * in the function's signature. Since this parameter is sandwiched between / and *, you can choose to assign either a positional or a keyword argument to it.

Here's a function call with the second argument as a positional argument:

#40

The second positional argument is assigned to another_param.

But you can also use a keyword argument:

#41

Both of these versions give the same output:

Coffee type: Espresso
another_param='testing another parameter'
Strength: 4
Milk Amount: 0
Instructions:

Any parameter between / and * in the function definition can have either positional or keyword arguments. So, in summary:

• Arguments assigned to parameters before a forward slash, /, are positional-only.

• Arguments assigned to parameters between a forward slash, /, and an asterisk, *, can be either positional or keyword.

• Arguments assigned to parameters after an asterisk, *, are keyword-only.

Remember that the * serves a similar purpose as the asterisk in *args since both * and *args force any parameters that come after them to require keyword-only arguments. Remember this similarity if you find yourself struggling to remember what / and * do!

Why use positional-only or keyword-only arguments? Positional-only arguments (using /) ensure clarity and prevent misuse in APIs where parameter names are irrelevant to the user. Keyword-only arguments (using *) improve readability and avoid errors in functions with many parameters, as names make the intent clear. For Alex, making coffee_type positional-only and strength and milk_amount keyword-only simplifies the API by enforcing a consistent calling style, reducing confusion for his team.

Using positional-only arguments may also be beneficial in performance-critical code since the overhead to deal with keyword arguments is not negligible in these cases.

Do you want to join a forum to discuss Python further with other Pythonistas? Upgrade to a paid subscription here on The Python Coding Stack to get exclusive access to The Python Coding Place's members' forum. More Python. More discussions. More fun.

Subscribe now

And you'll also be supporting this publication. I put plenty of time and effort into crafting each article. Your support will help me keep this content coming regularly and, importantly, will help keep it free for everyone.

Final Words

The reporter from The Herald did manage to chat to Alex eventually. She had become a regular at AI Coffee, and ever since Alex employed more staff, he's been able to chat to customers a bit more.

"There's a question I'm curious about", she asked. "How does the Artificial Intelligence software work to make the coffee just perfect for each customer?"

"I beg your pardon?" Alex looked confused.

"I get it. It's a trade secret, and you don't want to tell me. This Artificial Intelligence stuff is everywhere these days."

"What do you mean by Artificial Intelligence?" Alex asked, more perplexed.

"The machine uses AI to optimise the coffee it makes, right?"

"Er, no. It does not."

"But…But the name of the coffee shop, AI Coffee…?"

"Ah, that's silly, I know. I couldn't think of a name for the shop. So I just used my initials. I'm Alex Inverness."

• • •

Python functions offer lots of flexibility in how to define and use them. But function signatures can look cryptic with all the *args and **kwargs, rogue / and *, some parameters with default values and others without. And the rules on when and how to use arguments may not be intuitive at first.

Hopefully, Alex's story helped you grasp all the minutiae of the various types of parameters and arguments you can use in Python functions.

Now, I need to make myself a cup of coffee…

#42

Photo by Viktoria Alipatova: https://www.pexels.com/photo/person-sitting-near-table-with-teacups-and-plates-2074130/

Code in this article uses Python 3.13

The code images used in this article are created using Snappify. [Affiliate link]

You can also support this publication by making a one-off contribution of any amount you wish.

Support The Python Coding Stack

For more Python resources, you can also visit Real Python—you may even stumble on one of my own articles or courses there!

Also, are you interested in technical writing? You’d like to make your own writing more narrative, more engaging, more memorable? Have a look at Breaking the Rules.

And you can find out more about me at stephengruppetta.com

Further reading related to this article’s topic:

• Coffee features often when talking about programming. Here's another coffee-themed article, also about functions: What Can A Coffee Machine Teach You About Python's Functions?

• Python Quirks? Party Tricks? Peculiarities Revealed… [in particular see section 2, The Teleportation Trick]

Appendix: Code Blocks

Code Block #1

brew_coffee("Cappuccino", 4, 2)

Code Block #2

brew_coffee(coffee_type: str, strength: int, milk_amount: int)

Code Block #3

brew_coffee("Americano", 1, 4)

Code Block #4

brew_coffee(coffee_type, strength, milk_amount)

Code Block #5

def brew_coffee(coffee_type: str, strength: int, milk_amount: int):
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
    )

brew_coffee("Americano", 1, 4)

Code Block #6

brew_coffee("Americano", milk_amount=1, strength=4)

Code Block #7

brew_coffee("Americano", milk_amount=1, strength=4)

Code Block #8

brew_coffee("Americano", milk_amount=1, 4)

Code Block #9

def brew_coffee(coffee_type, strength, milk_amount, *args):
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        f"Add-ons: {', '.join(args)}\n"
    )

Code Block #10

brew_coffee("Latte", 3, 2, "cinnamon", "hazelnut syrup")

Code Block #11

def brew_coffee(coffee_type, strength, milk_amount, *args):
    print(f"{args=}")
    print(type(args))
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        f"Add-ons: {', '.join(args)}\n"
    )

brew_coffee("Latte", 3, 2, "cinnamon", "hazelnut syrup")

Code Block #12

def brew_coffee(coffee_type, strength, milk_amount, *add_ons):
    print(f"{add_ons=}")
    print(type(add_ons))
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        f"Add-ons: {', '.join(add_ons)}\n"
    )

brew_coffee("Latte", 3, 2, "cinnamon", "hazelnut syrup")

Code Block #13

brew_coffee("Latte", 3, 2)

Code Block #14

def brew_coffee(coffee_type, strength, milk_amount, *add_ons):
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        f"Add-ons: {', '.join(add_ons)}\n"
    )

brew_coffee("Latte", strength=3, milk_amount=2, "vanilla syrup")

Code Block #15

def brew_coffee(coffee_type, *add_ons, strength, milk_amount):
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        f"Add-ons: {', '.join(add_ons)}\n"
    )

brew_coffee("Latte", "vanilla syrup", strength=3, milk_amount=2)

Code Block #16

brew_coffee("Latte", "vanilla syrup", 3, 2)

Code Block #17

def brew_coffee(
        coffee_type,
        *add_ons, 
        strength, 
        milk_amount, 
        **kwargs,
):
    print(f"{kwargs=}")
    print(type(kwargs))
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        f"Add-ons: {', '.join(add_ons)}\n"
        f"Instructions:"
    )
    for key, value in kwargs.items():
        print(f"\t{key.replace('_', ' ')}: {value}")

Code Block #18

brew_coffee(
    "Latte",
    "vanilla syrup",
    strength=3,
    milk_amount=2,
    milk_type="oat",
    temperature="extra hot",
)

Code Block #19

def brew_coffee(
        coffee_type,
        *add_ons,
        strength,
        milk_amount,
        **instructions,
):
    print(f"{instructions=}")
    print(type(instructions))
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        f"Add-ons: {', '.join(add_ons)}\n"
        f"Instructions:"
    )
    for key, value in instructions.items():
        print(f"\t{key.replace('_', ' ')}: {value}")

Code Block #20

brew_coffee("Espresso", strength=4, milk_amount=0)

Code Block #21

brew_coffee(coffee_type="Espresso", strength=4, milk_amount=0)

Code Block #22

def brew_coffee(
        coffee_type,
        *add_ons,
        strength=3,
        milk_amount,
        **instructions,
):
    # ...

Code Block #23

brew_coffee("Espresso", milk_amount=0)

Code Block #24

brew_coffee("Latte", "caramel drizzle", milk_amount=2)

Code Block #25

def brew_coffee(
        coffee_type,
        *add_ons,
        strength=3,
        milk_amount,
        **instructions,
):
    # ...

Code Block #26

def brew_coffee_variant(
        coffee_type,
        # *add_ons,
        strength=3,
        milk_amount,
        **instructions,
):
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        # f"Add-ons: {', '.join(add_ons)}\n"
        f"Instructions:"
    )
    for key, value in instructions.items():
        print(f"\t{key.replace('_', ' ')}: {value}")

brew_coffee_variant("Espresso", milk_amount=0)

Code Block #27

brew_coffee_variant("Espresso", 0)

Code Block #28

def brew_coffee(
        coffee_type,
        *add_ons,
        strength=3,
        milk_amount,
        **instructions,
):
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        f"Add-ons: {', '.join(add_ons)}\n"
        f"Instructions:"
    )
    for key, value in instructions.items():
        print(f"\t{key.replace('_', ' ')}: {value}")

brew_coffee("Espresso", milk_amount=0)

Code Block #29

def brew_coffee(
        coffee_type,
        *add_ons,
        strength=3,
        milk_amount=0,
        **instructions,
):
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        f"Add-ons: {', '.join(add_ons)}\n"
        f"Instructions:"
    )
    for key, value in instructions.items():
        print(f"\t{key.replace('_', ' ')}: {value}")

brew_coffee("Espresso")

Code Block #30

brew_coffee(
    "Cappuccino",
    "chocolate sprinkles",
    "vanilla syrup",
    milk_amount=2,
    temperature="extra hot",
    cup_size="large cup",
)

Code Block #31

def brew_coffee(
        coffee_type,
        *add_ons,
        strength=3,
        milk_amount=0,
        **instructions,
):
    # ...

Code Block #32

brew_coffee(
    coffee_type="Cappuccino",
    "chocolate sprinkles",
    "vanilla syrup",
    milk_amount=2,
    temperature="extra hot",
    cup_size="large cup",
)

Code Block #33

def brew_coffee(
        coffee_type,
        /,
        *add_ons,
        strength=3,
        milk_amount=0,
        **instructions,
):
    # ...

Code Block #34

brew_coffee(
    coffee_type="Cappuccino",
    milk_amount=2,
    temperature="extra hot",
    cup_size="large cup",
)

Code Block #35

brew_coffee(
    "Cappuccino",
    milk_amount=2,
    temperature="extra hot",
    cup_size="large cup",
)

Code Block #36

def brew_coffee(
        coffee_type,
        /,
        # *add_ons,
        strength=3,
        milk_amount=0,
        **instructions,
):

Code Block #37

def brew_coffee(
        coffee_type,
        /,
        *,
        strength=3,
        milk_amount=0,
        **instructions,
):
    print(
        f"Coffee type: {coffee_type}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        # f"Add-ons: {', '.join(add_ons)}\n"
        f"Instructions:"
    )
    for key, value in instructions.items():
        print(f"\t{key.replace('_', ' ')}: {value}")

brew_coffee(
    "Cappuccino",
    milk_amount=2,
    temperature="extra hot",
    cup_size="large cup",
)

Code Block #38

brew_coffee(
    "Espresso",
    3,
    0,
)

Code Block #39

def brew_coffee(
        coffee_type,
        /,
        another_param,
        *,
        strength=3,
        milk_amount=0,
        **instructions,
):
    print(
        f"Coffee type: {coffee_type}\n"
        f"{another_param=}\n"
        f"Strength: {strength}\n"
        f"Milk Amount: {milk_amount}\n"
        # f"Add-ons: {', '.join(add_ons)}\n"
        f"Instructions:"
    )
    for key, value in instructions.items():
        print(f"\t{key.replace('_', ' ')}: {value}")

Code Block #40

brew_coffee(
    "Espresso",
    "testing another parameter",
    strength=4,
)

Code Block #41

brew_coffee(
    "Espresso",
    another_param="testing another parameter",
    strength=4,
)

Code Block #42

brew_coffee(
    "Macchiato",
    strength=4,
    milk_amount=1,
    cup="Stephen's espresso cup",
)

For more Python resources, you can also visit Real Python—you may even stumble on one of my own articles or courses there!

Also, are you interested in technical writing? You’d like to make your own writing more narrative, more engaging, more memorable? Have a look at Breaking the Rules.

And you can find out more about me at stephengruppetta.com

May 07, 2025 08:19 PM UTC

death and gravity

Process​Thread​Pool​Executor: when I‍/‍O becomes CPU-bound

So, you're doing some I‍/‍O bound stuff, in parallel.

Maybe you're scraping some websites – a lot of websites.

Maybe you're updating or deleting millions of DynamoDB items.

You've got your ThreadPoolExecutor, you've increased the number of threads and tuned connection limits... but after some point, it's just not getting any faster. You look at your Python process, and you see CPU utilization hovers above 100%.

You could split the work into batches and have a ProcessPoolExecutor run your original code in separate processes. But that requires yet more code, and a bunch of changes, which is no fun. And maybe your input is not that easy to split into batches.

If only we had an executor that worked seamlessly across processes and threads.

Well, you're in luck, since that's exactly what we're building today!

And even better, in a couple years you won't even need it anymore.

• Establishing a baseline
  • Threads
  • Problem: CPU becomes a bottleneck
  • Processes?
  • Problem: more processes, more memory
• Why not both?
  • A minimal plausible solution
  • Getting results
  • Fine, we'll make our own futures
  • Death becomes a problem
• Bonus: free threading

Establishing a baseline #

To measure things, we'll use a mock that pretends to do mostly I‍/‍O, with a sprinkling of CPU-bound work thrown in – a stand-in for something like a database connection, a Requests session, or a DynamoDB client.

class Client:
    io_time = 0.02
    cpu_time = 0.0008

    def method(self, arg):
        # simulate I/O
        time.sleep(self.io_time)

        # simulate CPU-bound work
        start = time.perf_counter()
        while time.perf_counter() - start < self.cpu_time:
            for i in range(100): i ** i

        return arg

We sleep() for the I‍/‍O, and do some math in a loop for the CPU stuff; it doesn't matter exactly how long each takes, as long I‍/‍O time dominates.

Real multi-threaded clients are usually backed by a shared connection pool, which allows for connection reuse (so you don't pay the cost of a new connection on each request) and multiplexing (so you can use the same connection for multiple concurrent requests, possible with protocols like HTTP/2 or newer). We could simulate this with a semaphore, but limiting connections is not relevant here – we're assuming the connection pool is effectively unbounded.

Since we'll use our client from multiple processes, we write an initializer function to set up a global, per-process client instance (remember, we want to share potential connection pools between threads); we can then pass the initializer to the executor constructor, along with any arguments we want to pass to the client. Similarly, we do the work through a function that uses this global client.

# this code runs in each worker process

client = None

def init_client(*args):
    global client
    client = Client(*args)

def do_stuff(*args):
    return client.method(*args)

Finally, we make a simple timing context manager:

@contextmanager
def timer():
    start = time.perf_counter()
    yield
    end = time.perf_counter()
    print(f"elapsed: {end-start:1.3f}")

...and put everything together in a function that measures how long it takes to do a bunch of work using a concurrent.futures executor:

def benchmark(executor, n=10_000, timer=timer, chunksize=10):
    with executor:
        # make sure all the workers are started,
        # so we don't measure their startup time
        list(executor.map(time.sleep, [0] * 200))

        with timer():
            values = list(executor.map(do_stuff, range(n), chunksize=chunksize))

        assert values == list(range(n)), values

Threads #

So, a ThreadPoolExecutor should suffice here, since we're mostly doing I‍/‍O, right?

>>> from concurrent.futures import *
>>> from bench import *
>>> init_client()
>>> benchmark(ThreadPoolExecutor(10))
elapsed: 24.693

More threads!

>>> benchmark(ThreadPoolExecutor(20))
elapsed: 12.405

Twice the threads, twice as fast. More!

>>> benchmark(ThreadPoolExecutor(30))
elapsed: 8.718

Good, it's still scaling linearly. MORE!

>>> benchmark(ThreadPoolExecutor(40))
elapsed: 8.638

...more?

>>> benchmark(ThreadPoolExecutor(50))
elapsed: 8.458
>>> benchmark(ThreadPoolExecutor(60))
elapsed: 8.430
>>> benchmark(ThreadPoolExecutor(70))
elapsed: 8.428

Problem: CPU becomes a bottleneck #

It's time we take a closer look at what our process is doing. I'd normally use the top command for this, but since the flags and output vary with the operating system, we'll implement our own using the excellent psutil library.

@contextmanager
def top():
    """Print information about current and child processes.

    RES is the resident set size. USS is the unique set size.
    %CPU is the CPU utilization. nTH is the number of threads.

    """
    process = psutil.Process()
    processes = [process] + process.children(True)
    for p in processes: p.cpu_percent()

    yield

    print(f"{'PID':>7} {'RES':>7} {'USS':>7} {'%CPU':>7} {'nTH':>7}")
    for p in processes:
        try:
            m = p.memory_full_info()
        except psutil.AccessDenied:
            m = p.memory_info()
        rss = m.rss / 2**20
        uss = getattr(m, 'uss', 0) / 2**20
        cpu = p.cpu_percent()
        nth = p.num_threads()
        print(f"{p.pid:>7} {rss:6.1f}m {uss:6.1f}m {cpu:7.1f} {nth:>7}")

And because it's a context manager, we can use it as a timer:

>>> init_client()
>>> benchmark(ThreadPoolExecutor(10), timer=top)
    PID     RES     USS    %CPU     nTH
  51395   35.2m   28.5m    38.7      11

So, what happens if we increase the number of threads?

>>> benchmark(ThreadPoolExecutor(20), timer=top)
    PID     RES     USS    %CPU     nTH
  13912   16.8m   13.2m    70.7      21
>>> benchmark(ThreadPoolExecutor(30), timer=top)
    PID     RES     USS    %CPU     nTH
  13912   17.0m   13.4m    99.1      31
>>> benchmark(ThreadPoolExecutor(40), timer=top)
    PID     RES     USS    %CPU     nTH
  13912   17.3m   13.7m   100.9      41

With more threads, the compute part of our I‍/‍O bound workload increases, eventually becoming high enough to saturate one CPU – and due to the global interpreter lock, one CPU is all we can use, regardless of the number of threads.¹

Processes? #

I know, let's use a ProcessPoolExecutor instead!

>>> benchmark(ProcessPoolExecutor(20, initializer=init_client))
elapsed: 12.374
>>> benchmark(ProcessPoolExecutor(30, initializer=init_client))
elapsed: 8.330
>>> benchmark(ProcessPoolExecutor(40, initializer=init_client))
elapsed: 6.273

Hmmm... I guess it is a little bit better.

More? More!

>>> benchmark(ProcessPoolExecutor(60, initializer=init_client))
elapsed: 4.751
>>> benchmark(ProcessPoolExecutor(80, initializer=init_client))
elapsed: 3.785
>>> benchmark(ProcessPoolExecutor(100, initializer=init_client))
elapsed: 3.824

OK, it's better, but with diminishing returns – there's no improvement after 80 processes, and even then, it's only 2.2x faster than the best time with threads, when, in theory, it should be able to make full use of all 4 CPUs.

Also, we're not making best use of connection pools (since we now have 80 of them, one per process), nor multiplexing (since we now have 80 connections, one per pool).

Problem: more processes, more memory #

But it gets worse!

>>> benchmark(ProcessPoolExecutor(80, initializer=init_client), timer=top)
    PID     RES     USS    %CPU     nTH
   2479   21.2m   15.4m    15.0       3
   2480   11.2m    6.3m     0.0       1
   2481   13.8m    8.5m     3.4       1
  ... 78 more lines ...
   2560   13.8m    8.5m     4.4       1

13.8 MiB * 80 ~= 1 GiB ... that is a lot of memory.

Now, there's some nuance to be had here.

First, on most operating systems that have virtual memory, code segment pages are shared between processes – there's no point in having 80 copies of libc or the Python interpreter in memory.

The unique set size is probably a better measurement than the resident set size, since it excludes memory shared between processes.² So, for the macOS output above,³ the actual usage is more like 8.5 MiB * 80 = 680 MiB.

Second, if you use the fork or forkserver start methods, processes also share memory allocated before the fork() via copy-on-write; for Python, this includes module code and variables. On Linux, the actual usage is 1.7 MiB * 80 = 136 MiB:

>>> benchmark(ProcessPoolExecutor(80, initializer=init_client), timer=top)
    PID     RES     USS    %CPU     nTH
 329801   17.0m    6.6m     5.1       3
 329802   13.3m    1.6m     2.1       1
  ... 78 more lines ...
 329881   13.3m    1.7m     2.0       1

However, it's important to note that's just a lower bound; memory allocated after fork() is not shared, and most real work will unavoidably allocate more memory.

Why not both? #

One reasonable way of dealing with this would be to split the input into batches, one per CPU, and pass them to a ProcessPoolExecutor, which in turn runs the batch items using a ThreadPoolExecutor.⁴

But that would mean we need to change our code, and that's no fun.

If only we had an executor that worked seamlessly across processes and threads.

A minimal plausible solution #

In keeping with what has become tradition by now, we'll take an iterative, problem-solution approach; since we're not sure what to do yet, we start with the simplest thing that could possibly work.

We know we want a process pool executor that starts one thread pool executor per process, so let's deal with that first.

class ProcessThreadPoolExecutor(concurrent.futures.ProcessPoolExecutor):

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        super().__init__(
            initializer=_init_process,
            initargs=(max_threads, initializer, initargs)
        )

By subclassing ProcessPoolExecutor, we get the map() implementation for free, since the original is implemented in terms of submit().⁵ By going with the default max_workers, we get one process per CPU (which is what we want); we can add more arguments later if needed.

In a custom process initializer, we set up a global thread pool executor,⁶ and then call the process initializer provided by the user:

# this code runs in each worker process

_executor = None

def _init_process(max_threads, initializer, initargs):
    global _executor

    _executor = concurrent.futures.ThreadPoolExecutor(max_threads)

    if initializer:
        initializer(*initargs)

Likewise, submit() passes the work along to the thread pool executor:

class ProcessThreadPoolExecutor(concurrent.futures.ProcessPoolExecutor):
    # ...
    def submit(self, fn, *args, **kwargs):
        return super().submit(_submit, fn, *args, **kwargs)

# this code runs in each worker process
# ...
def _submit(fn, *args, **kwargs):
    return _executor.submit(fn, *args, **kwargs).result()

OK, that looks good enough; let's use it and see if it works:

def _do_stuff(n):
    print(f"doing: {n}")
    return n ** 2

if __name__ == '__main__':
    with ProcessThreadPoolExecutor() as e:
        print(list(e.map(_do_stuff, [0, 1, 2])))

 $ python ptpe.py
doing: 0
doing: 1
doing: 2
[0, 1, 4]

Wait, we got it on the first try?!

Let's measure that:

>>> from bench import *
>>> from ptpe import *
>>> benchmark(ProcessThreadPoolExecutor(30, initializer=init_client), n=1000)
elapsed: 6.161

Hmmm... that's unexpectedly slow... almost as if:

>>> multiprocessing.cpu_count()
4
>>> benchmark(ProcessPoolExecutor(4, initializer=init_client), n=1000)
elapsed: 6.067

Ah, because _submit() waits for the result() in the main thread of the worker process, this is just a ProcessPoolExecutor with extra steps.

But what if we send back the future object instead?

    def submit(self, fn, *args, **kwargs):
        return super().submit(_submit, fn, *args, **kwargs).result()

def _submit(fn, *args, **kwargs):
    return _executor.submit(fn, *args, **kwargs)

Alas:

$ python ptpe.py
doing: 0
doing: 1
doing: 2
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "concurrent/futures/process.py", line 210, in _sendback_result
    result_queue.put(_ResultItem(work_id, result=result,
  File "multiprocessing/queues.py", line 391, in put
    obj = _ForkingPickler.dumps(obj)
  File "multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "ptpe.py", line 42, in <module>
    print(list(e.map(_do_stuff, [0, 1, 2])))
  ...
TypeError: cannot pickle '_thread.RLock' object

The immediate cause of the error is that the future has a condition that has a lock that can't be pickled, because threading locks only make sense within the same process.

The deeper cause is that the future is not just data, but encapsulates state owned by the thread pool executor, and sharing state between processes requires extra work.

It may not seem like it, but this is a partial success: the work happens, we just can't get the results back. Not surprising, to be honest, it couldn't have been that easy.

Getting results #

If you look carefully at the traceback, you'll find a hint of how ProcessPoolExecutor gets its own results back from workers – a queue; the module docstring even has a neat data-flow diagram:

|======================= In-process =====================|== Out-of-process ==|

+----------+     +----------+       +--------+     +-----------+    +---------+
|          |  => | Work Ids |       |        |     | Call Q    |    | Process |
|          |     +----------+       |        |     +-----------+    |  Pool   |
|          |     | ...      |       |        |     | ...       |    +---------+
|          |     | 6        |    => |        |  => | 5, call() | => |         |
|          |     | 7        |       |        |     | ...       |    |         |
| Process  |     | ...      |       | Local  |     +-----------+    | Process |
|  Pool    |     +----------+       | Worker |                      |  #1..n  |
| Executor |                        | Thread |                      |         |
|          |     +----------- +     |        |     +-----------+    |         |
|          | <=> | Work Items | <=> |        | <=  | Result Q  | <= |         |
|          |     +------------+     |        |     +-----------+    |         |
|          |     | 6: call()  |     |        |     | ...       |    |         |
|          |     |    future  |     |        |     | 4, result |    |         |
|          |     | ...        |     |        |     | 3, except |    |         |
+----------+     +------------+     +--------+     +-----------+    +---------+

Now, we could probably use the same queue somehow, but it would involve touching a lot of (private) internals.⁷ Instead, let's use a separate queue:

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        self.__result_queue = multiprocessing.Queue()
        super().__init__(
            initializer=_init_process,
            initargs=(self.__result_queue, max_threads, initializer, initargs)
        )

On the worker side, we make it globally accessible:

# this code runs in each worker process

_executor = None
_result_queue = None

def _init_process(queue, max_threads, initializer, initargs):
    global _executor, _result_queue

    _executor = concurrent.futures.ThreadPoolExecutor(max_threads)
    _result_queue = queue

    if initializer:
        initializer(*initargs)

...so we can use it from a task callback registered by _submit():

def _submit(fn, *args, **kwargs):
    task = _executor.submit(fn, *args, **kwargs)
    task.add_done_callback(_put_result)

def _put_result(task):
    if exception := task.exception():
        _result_queue.put((False, exception))
    else:
        _result_queue.put((True, task.result()))

Back in the main process, we handle the results in a thread:

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        # ...
        self.__result_handler = threading.Thread(target=self.__handle_results)
        self.__result_handler.start()

    def __handle_results(self):
        for ok, result in iter(self.__result_queue.get, None):
            print(f"{'ok' if ok else 'error'}: {result}")

Finally, to stop the handler, we use None as a sentinel on executor shutdown:

    def shutdown(self, wait=True):
        super().shutdown(wait=wait)
        if self.__result_queue:
            self.__result_queue.put(None)
            if wait:
                self.__result_handler.join()
            self.__result_queue.close()
            self.__result_queue = None

Let's see if it works:

$ python ptpe.py
doing: 0
ok: [0]
doing: 1
ok: [1]
doing: 2
ok: [4]
Traceback (most recent call last):
  File "concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
AttributeError: 'NoneType' object has no attribute 'result'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
AttributeError: 'NoneType' object has no attribute 'cancel'

Yay, the results are making it to the handler!

The error happens because instead of returning a Future, our submit() returns the result of _submit(), which is always None.

Fine, we'll make our own futures #

But submit() must return a future, so we make our own:

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        # ...
        self.__tasks = {}
        # ...

    def submit(self, fn, *args, **kwargs):
        outer = concurrent.futures.Future()
        task_id = id(outer)
        self.__tasks[task_id] = outer

        outer.set_running_or_notify_cancel()
        inner = super().submit(_submit, task_id, fn, *args, **kwargs)

        return outer

In order to map results to their futures, we can use a unique identifier; the id() of the outer future should do, since it is unique for the object's lifetime.

We pass the id to _submit(), then to _put_result() as an attribute on the future, and finally back in the queue with the result:

def _submit(task_id, fn, *args, **kwargs):
    task = _executor.submit(fn, *args, **kwargs)
    task.task_id = task_id
    task.add_done_callback(_put_result)

def _put_result(task):
    if exception := task.exception():
        _result_queue.put((task.task_id, False, exception))
    else:
        _result_queue.put((task.task_id, True, task.result()))

Back in the result handler, we find the maching future, and set the result accordingly:

    def __handle_results(self):
        for task_id, ok, result in iter(self.__result_queue.get, None):
            outer = self.__tasks.pop(task_id)
            if ok:
                outer.set_result(result)
            else:
                outer.set_exception(result)

And it works:

$ python ptpe.py
doing: 0
doing: 1
doing: 2
[0, 1, 4]

I mean, it really works:

>>> benchmark(ProcessThreadPoolExecutor(10, initializer=init_client))
elapsed: 6.220
>>> benchmark(ProcessThreadPoolExecutor(20, initializer=init_client))
elapsed: 3.397
>>> benchmark(ProcessThreadPoolExecutor(30, initializer=init_client))
elapsed: 2.575
>>> benchmark(ProcessThreadPoolExecutor(40, initializer=init_client))
elapsed: 2.664

3.3x is not quite the 4 CPUs my laptop has, but it's pretty close, and much better than the 2.2x we got from processes alone.

Death becomes a problem #

I wonder what happens when a worker process dies.

For example, the initializer can fail:

>>> executor = ProcessPoolExecutor(initializer=divmod, initargs=(0, 0))
>>> executor.submit(int).result()
Exception in initializer:
Traceback (most recent call last):
  ...
ZeroDivisionError: integer division or modulo by zero
Traceback (most recent call last):
  ...
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

...or a worker can die some time later, which we can help along with a custom timer:⁸

@contextmanager
def terminate_child(interval=1):
    threading.Timer(interval, psutil.Process().children()[-1].terminate).start()
    yield

>>> executor = ProcessPoolExecutor(initializer=init_client)
>>> benchmark(executor, timer=terminate_child)
[ one second later ]
Traceback (most recent call last):
  ...
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Now let's see our executor:

>>> executor = ProcessThreadPoolExecutor(30, initializer=init_client)
>>> benchmark(executor, timer=terminate_child)
[ one second later ]
[ ... ]
[ still waiting ]
[ ... ]
[ hello? ]

If the dead worker is not around to send back results, its futures never get completed, and map() keeps waiting until the end of time, when the expected behavior is to detect when this happens, and fail all pending tasks with BrokenProcessPool.

Before we do that, though, let's address a more specific issue.

If map() hasn't finished submitting tasks when the worker dies, inner fails with BrokenProcessPool, which right now we're ignoring entirely. While we don't need to do anything about it in particular because it gets covered by handling the general case, we should still propagate all errors to the outer task anyway.

    def submit(self, fn, *args, **kwargs):
        # ...
        inner = super().submit(_submit, task_id, fn, *args, **kwargs)
        inner.task_id = task_id
        inner.add_done_callback(self.__handle_inner)

        return outer

    def __handle_inner(self, inner):
        task_id = inner.task_id
        if exception := inner.exception():
            if outer := self.__tasks.pop(task_id, None):
                outer.set_exception(exception)

This fixes the case where a worker dies almost instantly:

>>> executor = ProcessThreadPoolExecutor(30, initializer=init_client)
>>> benchmark(executor, timer=lambda: terminate_child(0))
Traceback (most recent call last):
  ...
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

For the general case, we need to check if the executor is broken – but how? We've already decided we don't want to depend on internals, so we can't use Process​Pool​Executor.​​_broken. Maybe we can submit a dummy task and see if it fails instead:

    def __check_broken(self):
        try:
            super().submit(int).cancel()
        except concurrent.futures.BrokenExecutor as e:
            return type(e)(str(e))
        except RuntimeError as e:
            if 'shutdown' not in str(e):
                raise
        return None

Using it is a bit involved, but not completely awful:

    def __handle_results(self):
        last_broken_check = time.monotonic()

        while True:
            now = time.monotonic()
            if now - last_broken_check >= .1:
                if exc := self.__check_broken():
                    break
                last_broken_check = now

            try:
                value = self.__result_queue.get(timeout=.1)
            except queue.Empty:
                continue

            if not value:
                return

            task_id, ok, result = value
            if outer := self.__tasks.pop(task_id, None):
                if ok:
                    outer.set_result(result)
                else:
                    outer.set_exception(result)

        while self.__tasks:
            try:
                _, outer = self.__tasks.popitem()
            except KeyError:
                break
            outer.set_exception(exc)

When there's a steady stream of results coming in, we don't want to check too often, so we enforce a minimum delay between checks. When there are no results coming in, we want to check regularly, so we use the Queue.get() timeout to avoid waiting forever. If the check fails, we break out of the loop and fail the pending tasks. Like so:

>>> executor = ProcessThreadPoolExecutor(30, initializer=init_client)
>>> benchmark(executor, timer=terminate_child)
Traceback (most recent call last):
  ...
concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore

So, yeah, I think we're done. Here's the final executor and benchmark code.

Some features left as an exercise for the reader:

• providing a ThreadPoolExecutor initializer
• using other start methods
• shutdown()'s cancel_futures

Learned something new today? Share this with others, it really helps! PyCoder's Weekly HN Reddit linkedin Twitter

Want to know when new articles come out? Subscribe here to get new stuff straight to your inbox!

Bonus: free threading #

You may have heard people being excited about the experimental free threading support added in Python 3.13, which allows running Python code on multiple CPUs.

And for good reason:

$ python3.13t
Python 3.13.2 experimental free-threading build
>>> from concurrent.futures import *
>>> from bench import *
>>> init_client()
>>> benchmark(ThreadPoolExecutor(30))
elapsed: 8.224
>>> benchmark(ThreadPoolExecutor(40))
elapsed: 6.193
>>> benchmark(ThreadPoolExecutor(120))
elapsed: 2.323

3.6x over to the GIL version, with none of the shenanigans in this article!

Alas, packages with extensions need to be updated to support it:

>>> import psutil
zsh: segmentation fault  python3.13t

...but the ecosystem is slowly catching up.

1. At least, all we can use for pure-Python code. I‍/‍O always releases the global interpreter lock, and so do some extension modules. ^[return]

2. The psutil documentation for memory_full_info() explains the difference quite nicely and links to further resources, because good libraries educate. ^[return]

3. You may have to run Python as root to get the USS of child processes. ^[return]

4. And no, asyncio is not a solution, since the event loop runs in a single thread, so you'd still need to run one event loop per CPU in dedicated processes. ^[return]

5. We could have used composition instead, but then we'd have to implement the full Executor interface, defining each method explicitly to delegate to the inner process pool executor, and keep things up to date when the interface gets new methods (and we'd have no way to trick the inner executor's map() to use our submit(), so we'd have to implement it from scratch).

   Yet another option would be to use both inheritance and composition – inherit the Executor base class directly for the common methods (assuming they're defined there and not in subclasses), and delegate to the inner executor only where needed (likely just map() and shutdown()). But, the only difference from the current code would be that it'd say self._inner instead of super() in a few places, so it's not really worth it, in my opinion. ^[return]

6. A previous version of this code attempted to shutdown() the thread pool executor using atexit, but since atexit functions run after non-daemon threads finish, it wasn't actually doing anything. Not shutting it down seems to work for now, but we may still need do it to support shutdown(​cancel_futures=​True) properly. ^[return]

7. Check out nilp0inter/threadedprocess for an idea of what that looks like. ^[return]

8. pkill -fn '[Pp]ython' would've done it too, but it gets tedious if you do it a lot, and it's a different command on Windows. ^[return]

May 07, 2025 06:00 PM UTC

Django Weblog

Django security releases issued: 5.2.1, 5.1.9 and 4.2.21

In accordance with our security release policy, the Django team is issuing releases for Django 5.2.1, Django 5.1.9 and Django 4.2.21. These releases address the security issues detailed below. We encourage all users of Django to upgrade as soon as possible.

May 07, 2025 02:00 PM UTC

Real Python

How to Use Loguru for Simpler Python Logging

In Python, logging is a vital programming practice that helps you track, understand, and debug your application’s behavior. Loguru is a Python library that provides simpler, more intuitive logging compared to Python’s built-in logging module.

Good logging gives you insights into your program’s execution, helps you diagnose issues, and provides valuable information about your application’s health in production. Without proper logging, you risk missing critical errors, spending countless hours debugging blind spots, and potentially undermining your project’s overall stability.

By the end of this tutorial, you’ll understand that:

• Logging in Python can be simple and intuitive with the right tools.
• Using Loguru lets you start logging immediately without complex configuration.
• You can customize log formats and send logs to multiple destinations like files, the standard error stream, or external services.
• You can implement automatic log rotation and retention policies to manage log files effectively.
• Loguru provides powerful debugging capabilities that make troubleshooting easier.
• Loguru supports structured logging with JSON formatting for modern applications.

After reading this tutorial, you’ll be able to quickly implement better logging in your Python applications. You’ll spend less time wrestling with logging configuration and more time using logs effectively to debug issues. This will help you build production-ready applications that are easier to troubleshoot when problems occur.

To get the most from this tutorial, you should be familiar with Python concepts like functions, decorators, and context managers. You might also find it helpful to have some experience with Python’s built-in logging module, though this isn’t required.

Don’t worry if you’re new to logging in Python. This tutorial will guide you through everything you need to know to get started with Loguru and implement effective logging in your applications.

You’ll do parts of the coding for this tutorial in the Python standard REPL, and some other parts with Python scripts. You’ll find full script examples in the materials of this tutorial. You can download these scripts by clicking the link below:

Get Your Code: Click here to download the free sample code that shows you how to use Loguru for simpler Python logging.

Take the Quiz: Test your knowledge with our interactive “Python Logging With the Loguru Library” quiz. You’ll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

Python Logging With the Loguru Library

Think you know Python logging? Take this quick Loguru quiz to test your knowledge of formatting, sinks, rotation, and more!

Installing Loguru

Loguru is available on PyPI, and you can install it with pip. Open a terminal or command prompt, create a new virtual environment, and then install the library:

• Windows
• Linux + macOS

Windows PowerShell

PS> python -m venv venv
PS> venv\Scripts\activate
(venv) PS> python -m pip install loguru

Copied!

Shell

$ python -m venv venv/
$ source venv/bin/activate
(venv) $ python -m pip install loguru

Copied!

This command will install the latest version of Loguru from Python Package Index (PyPI) onto your machine.

Verifying the Installation

To verify that the installation was successful, start a Python REPL:

Shell

(venv) $ python

Copied!

Next, import Loguru:

Python

>>> import loguru

Copied!

If the import runs without error, then you’ve successfully installed Loguru and can now use it to log messages in your Python programs and applications.

Understanding Basic Setup Considerations

Before diving into Loguru’s features, there are a few key points to keep in mind:

1. Single Logger Instance: Unlike Python’s built-in logging module, Loguru uses a single logger instance. You don’t need to create multiple loggers, just import the pre-configured logger object:

   Python

   from loguru import logger
   

   Copied!

2. Default Configuration: Out of the box, Loguru logs to stderr with a reasonable default format. This means you can start logging immediately without any setup.

3. Python Version Compatibility: Loguru supports Python 3.5 and above.

Now that you understand these basic considerations, you’re ready to start logging with Loguru. In the next section, you’ll learn about basic logging operations and how to customize them to suit your needs.

Read the full article at https://realpython.com/python-loguru/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 07, 2025 02:00 PM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Brasil
  • Planet Python Poland
• Python Libraries
  • OLPC
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Mathspp Pydon'ts"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Michael Kennedy's Thoughts on Technology"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Adrarsh Divakaran
  • Agendaless Consulting
  • Ahmed Bouchefra
  • Al-Ahmadgaid Asaad
  • Alec Munro
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • Ari Lamstein
  • Armin Ronacher
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Automating OSINT
  • Awesome Python Applications
  • Baiju Muthukadan
  • Bajusz Tamás
  • BeDjango
  • Ben Bass
  • Ben Cook
  • Ben Rousch
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • BioPython News
  • Bit of Cheese
  • Bojan Mihelac
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Okken
  • Bruno Oliveira
  • Bruno Ponne / Coding The Past
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Düvel
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Casey Duncan
  • Catherine Devlin
  • Ching-Hwa Yu
  • Chris Hager
  • Chris Miles
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Heimes
  • Christian Ledermann
  • Christian Scholz
  • Christoph Schiessl
  • Christoph Zwerschke
  • CodeGrades
  • CodeSnipers
  • CodersLegacy
  • Coding Diet
  • Corey Gallon
  • Corey Goldberg
  • Cormoran Project
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dallas Fort Worth Pythoneers
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Data Community DC
  • Data School
  • DataWars.io
  • Dave Beazley
  • David Amos
  • David Caron
  • David Goodger
  • David Lindelof
  • David MacIver
  • David Malcolm
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • Declassed Art
  • Denis Kurov
  • Django Weblog
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Dougal Matthews
  • Duncan McGreggor
  • Ed Crewe
  • Edward K. Ream
  • Eli Bendersky
  • Eniram Ltd.
  • Eray Özkural (examachine)
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Evennia
  • Everyday Superpowers
  • Fabio Zadrozny
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • Gocept Weblog
  • Godson Gera
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Gustavo Narea
  • Gustavo Niemeyer
  • Gökhan Sever
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • HoloViz
  • Hugo van Kemenade
  • Humberto Rocha
  • Hynek Schlawack
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Inspired Python
  • Ionel Cristian Maries
  • IronPython-URLs
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Velichko
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • Janusworx
  • Jarrod Millman
  • Jean-Paul Calderone
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Burns
  • John Cook
  • John Jacobsen
  • John Ludhi/nbshare.io
  • Jon Parise
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan Street
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Julien Danjou
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kodnito
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • LAAC Technology
  • Laurent Szyster
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lintel Technologies
  • Linux Stans
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Lucas Cimon
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Mariatta
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mats Kindahl
  • Matt Layman
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Matthew Wright
  • Mattias Brändström
  • Mauveweb
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Nelson
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mirek Długosz
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Naomi Ceder
  • Natan Zabkar
  • Ned Batchelder
  • Neil Schemenauer
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • Nsukami Patrick
  • Obey the Testing Goat
  • Ofosos
  • Omaha Python Users Group
  • Ondřej Čertík
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patrick Kennedy
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • Pedro Lima
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyATL Bytecode
  • PyBites
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyPodcats
  • PyPy
  • PyTennessee
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Docs Editorial Board
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python GUIs
  • Python Insider
  • Python Morsels
  • Python People
  • Python Piedmont Triad User Group
  • Python Pool
  • Python Software Foundation
  • Python Sweetness
  • Python User Groups
  • Python for Beginners
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • Pythonicity
  • Pythonology
  • Python⇒Speed
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Rene Dudfield
  • Reuven Lerner
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rishi Maker
  • Rob Galanakis
  • Robert Collins
  • Robert Zaremba
  • Robin Parmar
  • Robin Wilson
  • Rodrigo Araúj
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Sandipan Dey
  • Sandro Tosi
  • Sean McGrath
  • Sebastian Pölsterl
  • Sebastian Witowski
  • Selena Deckelmann
  • Senthil Kumaran
  • Seth Michael Larson
  • Shannon -jj Behrens
  • ShiningPanda
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Scherfke
  • Stefanie Molin
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Holden
  • Steven Klass
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Suresh Dasari/Tutlane.com
  • Swisscom ICT
  • Talk Python Blog
  • Talk Python to Me
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Terri Oda
  • Terry Jones
  • Test and Code
  • TestDriven.io
  • The Data Scientist
  • The Digital Cat
  • The Lunar Cowboy
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Open Sourcerer
  • The Parcon Blog
  • The Python Coding Blog
  • The Python Coding Stack
  • The Python Papers
  • The Python Show
  • The Three of Wands
  • Thibauld Nion
  • ThisHosting.Rocks
  • Thomas Guest
  • Thomas Vander Stichele
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Knapp
  • Tim Lesher
  • Tobias Ivarsson
  • Tom Christie
  • Tomasz Ducin
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • TypeThePipe
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinayak Mehta
  • Virgil Dupras
  • Vladimir Perić
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • Wing Tips
  • Wingware
  • Wyatt Baldwin
  • Yaniv Aknin
  • Yann Larrivée
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero to Mastery
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • death and gravity
  • eGenix.com
  • hypothesis.works articles
  • kdev-python
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pygame
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • scikit-learn
  • testmon
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal:
    Open an issue on github