Planet Python

Last update: November 12, 2026 04:47 PM UTC

November 12, 2026

Luke Plant

Why I’m calling your contribution “slop”

You have probably arrived at this web page because you have attempted to contribute to an Open Source software project, probably via a Pull Request and I, or perhaps someone else, has called your efforts “slop” and promptly closed/ignored it.

I might also use this word to describe emails or other communication received which are of a similar nature.

This post is addressed to you, and is my attempt to explain why I’ve called it “slop”. I don’t want to repeat myself on this subject, and I also don’t want to leave you totally puzzled, upset and embittered, so I’m writing it once, and just once.

By “slop”, I’m referring especially to AI slop. It is, however, entirely possible that you didn’t use AI, but I can’t tell that, and your code or communication seems to share the same quality problems that AI slop has.

The old way

It would be nice if I could give more detailed feedback about why exactly I think your contribution matches the term “slop”. In the past, when I got a pull request that wasn’t up to scratch from a new contributor, I would try to take the time to give detailed feedback, to effectively train them.

This approach was based on a few assumptions:

• First, there was a person on the other end of the interaction, who deserved to be treated like a person.

• Second, that person had already demonstrated a significant amount of effort just by the fact that they bothered to write the code, and then prepare a pull request.

• Third, there was a good chance that, equipped with some of the feedback I gave, the new contributor would get better in the future, to the point where they would be an asset to the project, and more of an asset to the world in general.

Unfortunately, use of AI in software has massively eroded all of the above. The human involvement in creating the PR may have been almost zero. Writing is no longer proof of work. Many people will not benefit at all from the detailed feedback I give, because often they just give up. Basically, the balance that Open Source relied has been destroyed by the slopocalypse, which means that I have to allocate my time in a very different way to the past.

If you are a budding Open Source contributor trying to do the right thing, trying to learn, maybe using AI or maybe not using it all but still getting lumped with those who are, this sucks for you. I’m sorry for that, and for the world that has been lost, but I’m afraid I can’t fix it, and this post is the best I can do.

I’m also sorry for the curtness of my interactions with you, and the cynical tone you might pick up from this post, but again it is a matter of necessity to avoid the burn-out that many others have suffered, and to ensure I can give a good account for how I’ve used my life (Romans 14:12).

What’s wrong with your contribution

It might be that you have used an AI to generate something, or you’ve worked hard on it yourself, and you think it looks pretty good, but it is still labelled slop. Below are some clues as to why that might be. You will have to work through these yourself and see which apply to yours.

The best summary I can give for what I’m looking for in a good quality code contribution is the following:

Every single line, word, token and character should have been written with the understanding that another human being will need to read it, understand it and potentially maintain it indefinitely. This covers:

• implementation code

• test code

• documentation

• commit messages

• descriptions and comments in pull requests

• basically, everything.

Of course, the code must actually work (that’s step zero, the computer has to understand it too), and in addition, it must be for the benefit of all current and future users, which means it mustn’t recklessly break anything, for example, and must be designed with long term needs in mind.

In short, the contribution should treat the recipients (reviewers and maintainer) as human beings whose time is extremely valuable, and the project as something valuable that needs to be protected.

Here are some of the most common problems I see with AI generated stuff:

• General “low quality”:

  • Hacky code that fixes the author’s problem, without caring about what it does to the code structure and maintainability.

  • Buggy code of all kind.

• Simply too much stuff. This includes things like:

  • Silly comments that simply repeat what you can see immediately from the code.

  • Unnecessary tests, which are not adding anything useful, and seem to demonstrate a failure to understand what the specific feature/bug/edge cases are. You should write tests like you are betting on your life.

  • a PR description or commit message which includes a load of irrelevant stuff. e.g.:

    • “100% of the test suite passes” – this is too obvious to be stated, of course it wasn’t going to be merged if it added failing tests or broke existing things, or broke other documented coding/contribution standards.

    • “3 new tests added, 2 altered” – I can see that from the diff.

    • etc. etc.

  With my own code (in common with many of my blog posts and sermons), I often spend as much time deleting things as I do adding them.

• Not enough detail and background (yes, I know this is the opposite of the above).

  If the contribution breaks compatibility or has some negative behaviour or limitations, I need to know about that. You might not care about it, and the AI certainly doesn’t, but I do.

  If something looks like it might break something, but in fact doesn’t for some good reason, I want to know that.

  Have you considered the edge cases? Have you considered why it was the way it was? Are there more obvious approaches which can’t be used? Have you considered the issues raised in the bug report or background? And have you then demonstrated that consideration in how you prepared your contribution (without overloading the reviewer with unnecessary volume)?

• A failure to think about “what”.

  It is very easy to tell an AI to do a certain task or implement something, without really digging into whether that was what you should be implementing, and what the consequences of that would be.

At the end of this, you might think something like this:

Wow, this is really hard! It would probably be faster to start from scratch and write every word myself. That way, I’m only going to write the necessary stuff, because it will take too much effort to write unnecessary things. It will also force me to think about whether something really is necessary.

Then, if something looks unnecessary but actually isn’t I might even need to justify why it is needed!

If I end up actually needing a lot of text, I’m going to have to structure it extremely well and design my whole approach to this work to ensure that people’s eyes don’t just glaze over. But, I really need to avoid writing a lot of text if I possible can…

Yep, exactly, all that is true. It was always true of good coding. Doing the above is simply what is necessary when you treat the recipient as a human. A high quality PR was always a lot of work – you have to do all the work needed to make the maintainer’s job easy, because you are asking them to do something for you.

The only thing that has got harder about contributing to an Open Source project is that I’m now not telling you exactly what is wrong, you’ll have to figure that bit out for yourself, and I’m now much more likely to simply put something in the slop bin.

Your own use of AI may well have made things harder for you:

• Since it is cheap to write tests, you don’t write them like you are betting on your life. But while the cost of writing them has gone down, they still have to be read by someone else, executed in the test suite, and maintained by someone else who has to figure out whether they are just repetition or actually add value.

• An AI has no sense of boredom, so won’t care enough about repetitive edits that should have been turned into neater code (reducing repetition) or manual tasks that should have been an automated process.

• If you delete or modify code, you’ll almost certainly stop and think “hmm, I wonder why this was there in the first place, and written like that?”

To gain empathy with the recipient, you are indeed better off starting from scratch and literally typing everything in, even if you use some of the insights from the AI.

In saying this, don’t think I’m 100% against AI. I have successfully used AI to help debug things that I might never have been able to work out otherwise, and I’ve used agents in very controlled ways where I’ve been able to judge their output and the trade-offs they involve. But, for many of the reasons above, I virtually never use them to write actual code or text if I need another human to read it.

Well, that’s the end. You better believe that I typed every letter myself.

November 12, 2026 11:58 AM UTC

LernerPython blog, from Reuven Lerner

Announcing: Vibe to valid

Vibe coding is fun: You can go from an idea to a working app in a weekend.

But if you don’t have a background in software, how can you be sure your app is secure? Tested? Backed up? Ready for real customers, or for an investor’s technical review?

Today I’m launching Vibe to Valid, an audit service for vibe-coded Python apps.

Here’s how it works:

1. You share your repo.
2. I review it and send you a plain-English report: what’s wrong, what’s right, and what to fix first.
3. If you want, I fix the problems, and set up your AI coding tools so you can keep vibe coding with the AI on a short leash.

There’s also a due diligence audit for investors and acquirers, and a walkthrough session for your team. I’ve been a Python trainer and consultant since 1995, and Claude Code is a core part of my workflow. But every finding is reviewed and explained by me. You’re getting an experienced human, not an automated scan.

Learn more at https://VibeToValid.com, where you can also book a free intro call.

The post Announcing: Vibe to valid appeared first on LernerPython.

November 12, 2026 10:01 AM UTC

PyPy

20 Years of Pyrolog

On November 12, 2006 I did the first surviving commit of my RPython based Prolog implementation Pyrolog into the PyPy repository. I had been working on it since spring 2006 already. In that year we were finishing up the first PyPy EU research project and I was finishing up my Bachelor studies at Uni Düsseldorf. Since it's now 20 years later I wanted to write a personal retrospective post on this project. Pyrolog never had any real users, but it still led to some interesting research and influenced a number of other RPython projects.

Contents

• Context

• 2006/2007 The first year

• 2008/2009 JIT developments on the PyPy side

• 2009-2010 New Pyrolog Architecture: More Continuations

  • JIT support

• 2011/2012 Widening Prolog support

• 2012/2013 Compressed Terms

• 2013 Unipycation

• 2013 Pycket

• 2022-2024 RPython implementation of Effekt

• 2026 Git Conversion

• Conclusion

• Bibliography

Context

As part of the EU project we had promised lots of things, including implementing some constraint programming in PyPy. For this strand of the work I learned Prolog by reading Clocksin and Mellish's Programming Prolog [CM2003] and later the Art of Prolog [SS1994].

In March 2006, the PyPy project partner Logilab organized a sprint in UCLouvain in Louvain-la-Neuve (Belgium), with Peter van Roy and some of the researchers working on Gecode. The plan was to add some kind of unification and constraint solving to PyPy. The Logilab people were quite inspired by the Van Roy/Haridi book [VH2004] and I later studied his Prolog implementation survey paper [VanRoy1994].

In 2006 I also attended my advisor Michael Leuschel's Prolog lectures in Düsseldorf. During that I started working on a Prolog interpreter in RPython, later called Pyrolog. Much of the early commit history is lost unfortunately, it didn't make the Subversion->Mercurial jump correctly and I don't know where the codespeak archives are now.

2006/2007 The first year

Pyrolog's first incarnation had a fairly straightforward implementation of unification in object-oriented RPython code. Logical variables stored an index, with their bindings stored in a global array at that index.

Pyrolog Memory Layout from [Bolz2007]

That version represented choice points by having exception handlers on the interpreter level, and backtracking via throwing exceptions, catching them further up. It used a continuation-based architecture for storing the calls left to do. Those success continuations were stored on the heap and continuously allocated and consumed. This approach was taken from Paul Tarau's BinProlog ([TB1990], [Tarau1992]) and also based on some other Prolog papers [Lindgren1994] [Neumerkel1995].

To deal with deep recursion that Prolog programs typically do, that version of Pyrolog relied on RPython's stackless transformation [BR2007D071], which would unwind the host stack (i.e. the C stack) and store its information on the host heap if host stack overflow was getting close. When storing C stack frames on the heap in this way, we also did some amount of tail call optimization. We were super into this idea at the time. However, it turned out not to be too practical, making the binary twice as large and slower, so eventually we removed it again.

I also experimented with using the first RPython JIT on that version of Pyrolog. That JIT was a partial-evaluation-based method JIT and managed to speed up some small Python programs. I don't think it did very much for Pyrolog though.

The performance of that version of Pyrolog was quite disappointing. In my B.Sc. thesis [Bolz2007] I did some measurements against SWI and Sicstus Prolog. Sicstus was often an order of magnitude faster than Pyrolog at this point.

In the thesis I also implemented a way to call into Prolog from Python code and had a case study where I implemented a Tic-Tac-Toe GUI in Python, with the game logic in Prolog.

Shortly after finishing my thesis, I changed the variable representation: instead of indexing into a global binding array, each variable object stored its own binding. The exception-based backtracking architecture stayed in place for another two years.

2008/2009 JIT developments on the PyPy side

After the EU project finished we started working on various new architectural ideas for the RPython meta-JIT. Armin Rigo, me and some others wrote two new prototypes. We weren't happy with them, so in 2008 Armin and I started experimenting with a prototype for a new trace-based JIT written in Prolog. We used Prolog because its pattern matching and ease of meta-programming allowed us to experiment more easily with different designs.

Towards the end of 2008 and throughout 2009 Armin, Maciej Fijałkowski, me and others started porting the design back to RPython and called it pyjitpl5 (the "pl" in the name stands for "Prolog"). This is still the base of the current PyPy meta-JIT compiler and became the subject of the second (smaller) PyPy EU project in the Eurostars program.

We eventually published a paper about the Prolog version of the JIT [BLR2010] at LOPSTR 2009.

2009-2010 New Pyrolog Architecture: More Continuations

After working mostly on PyPy and the new JIT in 2008 and 2009 I started working on Pyrolog again over the 2009 Christmas break. Since the stackless transformation wasn't compatible with the new JIT I had to switch to a different architecture, still used today. The first version of the Pyrolog architecture had already stored success continuations on the heap. The goal of this refactoring was to also store the failure continuations as reified RPython objects.

# the simplified main interpreter loop

def interpret(sc, fc, trail):
    while not sc.is_done():
        try:
            sc, fc, trail = sc.activate(fc, trail)
        except UnificationFailed:
            sc, fc, trail = fc.fail(trail)

This had the effect that the host stack use is completely flat when interpreting Prolog calls. The state needed to continue with Prolog execution is always stored in continuation objects on the heap. The interpreter is therefore dealing with two continuations, one activated upon success, and one on failure.

Additionally, there is a (segmented) trail object which stores the variable bindings that need to be undone when backtracking. Failure continuations have a reference to the part of the trail they need to backtrack to, when activated.

The downside of this approach is that the interpreter continuously has to allocate heap data for all the success and failure continuations. This puts a lot of pressure on the RPython GC. Also, term unification still uses the host stack, so unifying very deep terms can still be a problem and lead to implementation-level recursion errors.

JIT support

The real goal of the refactoring was to make the interpreter more JIT-friendly. Since the interpreter doesn't use recursion for Prolog calls any more, the loops at the Prolog level (expressed as recursion) can be traced by the JIT. All the continuous allocations of continuations can be optimized by the allocation removal of the RPython JIT. If all this works correctly, the performance of Pyrolog becomes very good, often beating Sicstus by a significant factor on suitable microbenchmarks.

Pyrolog VM Generation vs. Runtime from [BLS2010]

Unfortunately this is mainly the case for code with very predictable control flow, ideally with nice loops. Since the main appeal of Prolog is to do backtracking search, control flow is typically not very predictable for interesting more Prolog-like programs. As soon as the control flow becomes too complicated and unpredictable, the continuations survive beyond the region that the JIT optimizes and have to be allocated on the heap again. Therefore Pyrolog has quite extreme performance cliffs.

We wrote a paper about this version of Pyrolog for PPDP 2010 [BLS2010] (together with Michael Leuschel and my later colleague David Schneider). That conference was super interesting and important for me. I met Paul Tarau (who invented BinProlog, one of the inspirations of Pyrolog. He later gave an excellent Prolog talk at the virtual machine summer school in 2016), Olivier Danvy (whose paper [BD2004] I should have cited. Genuinely one of the nicest people), Kenichi Asai (who had worked on partial evaluation and reflection) and several others. At that point during my PhD I was feeling relatively demotivated, and my chats with these researchers were incredibly encouraging.

Group photo at PPDP 2010

On the bus trip to Gmunden, where the conference dinner took place, somebody explained how to implement unification of cyclic terms (i.e. rational trees) to me. Unfortunately I am not quite sure who it was any more, I think it might have been Thierry Martinez. This led to me implementing support for cyclic term unification. I still like that trick a lot and tried to explain it to Phil Zucker earlier this year. I hope to blog about it eventually. For a fun use case see my blog post on rational tree unification for dfa minimization.

2011/2012 Widening Prolog support

During 2011 and 2012 I supervised several Bachelor/Master theses that worked on adding features to Pyrolog. In the process we added support for Prolog modules, DCGs, attributed variables, freeze and when. All these were done by my excellent students Sven Hager and René Bartelmus. This left the fundamental execution model intact (with some small additions). I also spent some time improving the error messages of Pyrolog to for example show backtraces on Prolog exceptions.

Pyrolog also served as an early test case for David Schneider's ARM JIT backend. In January 2011, he could already run Pyrolog microbenchmarks with the JIT on a BeagleBoard, while the backend still needed debugging before running more complex Python benchmarks.

In 2012 I mostly wrote my PhD thesis [Bolz2012]. It contains a chapter on Pyrolog based on the PPDP 2010 paper. That chapter argues that the fact that Pyrolog works relatively well demonstrates that the RPython meta-JIT can be used for quite a wide variety of languages.

There was also another chapter that compared how a meta-tracing JIT works with more classical partial evaluation approaches. The code of that was again written in Prolog and originally presented here on the blog.

At this point I left Düsseldorf. In the winter semester 2012/13 I spent half a year in Berlin at Hasso-Plattner-Institute with my PhD co-advisor Robert Hirschfeld as a postdoc. In 2013 I started working with Laurie Tratt at King's College.

At this point the main Pyrolog story kind of ends. There were only some small improvements happening in the succeeding years. However, Pyrolog ideas led to followup research in other projects, and I want to discuss some of them.

2012/2013 Compressed Terms

One experiment that I had tried to get working in Pyrolog in 2010 and 2011, but ended up never merging, was compressed terms. The idea comes from the observation that certain patterns of terms occur repeatedly. For example an association list has terms of the form [A/B|C] repeated. Instead of allocating separate list and pair objects for every entry, the plan was to share a description of that structure, the "shape", and store each entry's keys and values in a flat storage data structure. This would make every entry have less indirection and use fewer words of memory.

However, I had never managed to get that working in Prolog, because variable bindings can create these patterns after the fact. A field that initially contained an unbound variable could later refer to a whole structured term, so the opportunities for compression changed after construction, and again on backtracking.

When I was at HPI for six months in winter 2012/2013 I worked with Tobias Pape to implement the same compression idea for a simpler functional language called lamb, later Theseus. Tobias pushed this approach much further in the following years [PBH2015], [PBH2017]. My hope was to eventually understand the problem well enough to be able to port it back to Pyrolog. But that part never happened. We did eventually try those ideas in Pycket though.

2013 Unipycation

In 2013 I started working in London. The goal of our research project there was "language composition". Edd Barrett, Laurie Tratt and I implemented combined RPython implementations of PyPy plus another language, with a way to integrate the two involved languages. The first pair we tried was combining PyPy and Pyrolog in a project called Unipycation [BBT2013]. The project made it possible to call into Prolog from Python and vice versa. The RPython meta-JIT can optimize across this language boundary and even perform cross-language inlining. Unfortunately we never found super compelling use-cases for this combination.

2013 Pycket

Also in 2013 I started working on Pycket, an RPython implementation of Racket, together with Sam Tobin-Hochstadt and Jeremy Siek. Later Tobias Pape and Spenser Bauman also joined. The Pycket architecture was indirectly inspired by the Pyrolog continuation approach. We implemented the CEK machine relatively directly in RPython. This also used a flat host stack and it made implementing things like call/cc relatively straightforward [Pycket2015].

One of the great new ideas we adopted in Pycket was "false loop filtering", originally from IBM Research [HWISN2011]. This takes calling context into account when tracing function calls and prevents the tracing JIT from recognizing "false loops", which would never run for more than one iteration. I always wanted to port this idea back to Pyrolog and implemented a prototype, but never really polished it enough to be able to merge it.

Another implementation technique in Pycket was to make continuation objects much smaller, to make the case of them escaping to the heap less bad. This approach did make it back to Pyrolog.

2022-2024 RPython implementation of Effekt

I left King's College in 2016 and have been back at Uni Düsseldorf (half-time) since 2017. In 2022 I started working with Marcial Gaißert and Jonathan Brachthäuser on an RPython implementation of Effekt and related languages. This was an implementation of a JIT for effect handlers in RPython [GBB2025]. Again, it uses the approach of having reified stack frames/continuations which can then be manipulated by the interpreter, adding effect handlers on top of it.

2026 Git Conversion

This fall, I got an email by a researcher about Pyrolog and realized that it was 20 years old this year. Subsequently I merged various strands of the surviving mercurial history into a new Github repo, fixed some bugs and implemented some convenience features.

Screenshot of Pyrolog

Conclusion

commits per month on the main branch, by author

Pyrolog has always been a much more researchy project than PyPy itself. While PyPy tries to be an open source project and "product", Pyrolog never had any users and even I never used it for anything beyond small demos, some research prototypes, and benchmarking. While some quite interesting research came out of Pyrolog, I think having some real applications running on top of it would have given me a much more concrete goal to improve it towards.

Implementation wise, Pyrolog inherited some of the problems of PyPy. In PyPy we were quite strongly focused on peak performance in the first few years, leading to performance cliffs. Pyrolog also had those in even stronger form, due to the constant allocations of the interpreter.

Some very interesting interaction between Pyrolog, various Prolog JIT prototypes and the PyPy/RPython meta-JIT emerged. Pyrolog forced the RPython JIT to support quite different execution models which made it applicable to other languages than just object-oriented imperative ones.

I still have no concrete use case for Pyrolog so it's kind of unlikely that I will work much more on it right now. But my development on this project was always quite bursty, so I am bound to return to it at some point in the future.

I would like to end with thanking all the contributors, my other collaborators and the long line of Prolog researchers that inspired and encouraged me.

Bibliography

November 12, 2026 08:16 AM UTC

Tryton News

Release 0.3.0 of tryton-filestore-cellar

We are proud to announce the release of tryton-filestore-cellar version 0.3.0.

tryton-filestore-cellar is Tryton’s filestore implementation based on the Cellar storage service.

In addition to bug fixes, this release contains the following improvements:

• Implement delete
• Upgrade dependency to boto3

The package is available at tryton-filestore-cellar · PyPI

1 post - 1 participant

Read full topic

November 12, 2026 07:00 AM UTC

Graham Dumpleton

Muru: releases, not restarts

With the previous post I finished with development mode. Something I kept coming back to in those posts was that when muructl run reloads after a change, it isn't restarting anything. It starts a new release beside the one serving, and only moves traffic across once the new release is ready. In how it fits together I said the same mechanism is what the daemon uses for deploying an application to a server. Time to show that.

This is where Muru departs most from how a Python web application is usually deployed. With gunicorn, deploying new code typically means putting the new code in place and sending the master process a HUP signal. Going by its own description, that reloads the configuration, starts new worker processes and gracefully shuts down the old ones, with the new workers loading the application code afresh unless preload_app is on, in which case the old code stays. A USR2 signal goes further, starting a second master and set of workers from the same command line beside the first, with the old master left around so a change can be backed out by sending more signals. Either way, there is one copy of the code on disk, both sets of workers share the one listening socket while they overlap, and anything more than that, such as keeping the previous version running so you can switch straight back to it, is left to a load balancer and a deploy script you write yourself. I can hardly point the finger, as mod_wsgi was no better, with a deploy there meaning a restart of Apache, or a similar mix of signals and gymnastics to get a daemon process to reload against different code.

Under the Muru daemon, each deploy is a release of its own, started beside the one serving, given traffic only once it is ready, and kept warm afterwards in case you want it back.

Starting a daemon

The daemon runs in the background as your own user, or under systemd as a service. For trying things out, starting it as your own user is enough:

muructl daemon start

muru: daemon started (pid 31018), serving on http://127.0.0.1:8080, log /Users/graham/Library/Logs/muru/daemon.log

muructl status --processes shows the processes the daemon is looking after, which for now is only the edge it has started, listening on port 8080. The daemon's own row says whose daemon it is, here one run as my own user, and where it keeps its state:

muructl status --processes

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE      WHAT                                                    USER     UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
31018   daemon    user daemon, state ~/Library/Application Support/muru   graham   1s   0.0%   32.5MB
31019   └─ edge   listening on 127.0.0.1:8080                             graham   1s   0.0%   26.0MB       0

Nothing has been deployed yet, so the edge answers every request with a 503 and says why:

503 Service Unavailable
reason: no_workers
request id: 01M6TDFVFPFWF4JKE4TJ1G86E2

No worker is available to handle the request.

A release is a directory

What gets deployed is a source directory. For this post the application is a small shop, kept in a directory per version under releases, as a deploy script might lay out checkouts of each tag. Each version is its own uv project with a pyproject.toml and a uv.lock, and the first version has no dependencies at all:

[project]
name = "shop"
version = "1.0"
requires-python = ">=3.15"
dependencies = []

[tool.muru]
entrypoint = "app:application"

[tool.muru.commands.migrate]
description = "Apply database migrations"
run = ["python", "migrate.py"]
exclusive = true

The application in app.py reports how many orders are in an SQLite database, and which release answered, using muru.context():

import os
import sqlite3

import muru

DATABASE = os.environ["SHOP_DATABASE"]


def application(environ, start_response):
    where = muru.context()
    with sqlite3.connect(DATABASE) as db:
        (count,) = db.execute("SELECT COUNT(*) FROM orders").fetchone()
    body = f"{count} orders, served by {where.app}@{where.release}\n".encode()
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [body]

The migrate command declared in pyproject.toml runs a small script which applies any SQL files in a migrations directory that haven't been applied yet. I will come back to commands later, as they matter once two releases share a database.

The first deploy names the application and the source directory, and gives the settings the release needs:

muructl deploy shop releases/v1 --processes 2 --env SHOP_DATABASE=/Users/graham/shop/data/shop.db

muru: release shop@r1 is ready and serving

The database starts out empty, so before anything else its tables need creating, by running the declared command with the release:

muructl command run shop migrate

applied 0001_orders.sql

after which the shop answers:

3 orders, served by shop@r1

There is a fair bit hidden behind that first deploy, and muructl events tells the story:

     2 07:10:12 info     release.created      shop@r1                  from /Users/graham/shop/releases/v1
     3 07:10:12 info     release.building     shop@r1                  from /Users/graham/shop/releases/v1/uv.lock
     4 07:10:12 info     release.built        shop@r1                  /Users/graham/Library/Application Support/muru/envs/shop/r1 in 0.1s from uv.lock 8c788a3a641a
     5 07:10:12 info     release.state_changed shop@r1                  fetched the runtime library muru-1.0.0-cp315-cp315t-macosx_11_0_arm64.whl from PyPI, once
     6 07:10:13 info     release.ready        shop@r1                  2 workers ready in 0.7 s
     7 07:10:13 notice   traffic.switched     shop@r1                  traffic moved to the release
     8 07:10:13 info     command.started      shop@r1                  migrate: started (pid 31050)
     9 07:10:13 info     command.finished     shop@r1                  migrate: exited with status 0 after 60ms

Because the source directory has a uv.lock and no .venv, the daemon built a virtual environment for the release from the lock file, in a directory belonging to that release alone. The application doesn't need to depend on Muru, as the daemon adds its own runtime library to the environment. Only once both workers were ready did traffic move to the release.

Asking for the processes again shows what the deploy added:

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE           WHAT                                                    USER     UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
31018   daemon         user daemon, state ~/Library/Application Support/muru   graham   2s   0.0%   38.9MB
31019   ├─ edge        listening on 127.0.0.1:8080                             graham   2s   0.0%   26.5MB       0
31045   └─ host        shop@r1 serving                                         graham   0s   0.0%   54.5MB            0/8       0
31044      ├─ relay    shop@r1 output to apps/shop/r1.log                      graham   0s   0.0%   17.5MB
31047      ├─ worker   group default, worker 0                                 graham   0s   0.0%   20.3MB            0/4
31048      └─ worker   group default, worker 1                                 graham   0s   0.0%   19.5MB            0/4

The release is a host process, which imported the application, with the two workers forked from it, each with four threads. The host's row also says what part the release has in routing, which for now is that it is serving. Beside the host is a log relay, which takes what the host and its workers write and puts it in the release's own log, so that a log which is slow to write never holds up a request. The relay is shown under the host it serves, although it is the daemon which starts it. The columns on the right, for requests per second, busy threads, queueing, response times and time spent waiting on the GIL, stay mostly empty until there is some traffic.

That each release gets an environment of its own is more important than it might look. If releases shared one virtual environment, installing a new dependency for the next release would change what the running release sees underneath it. Giving each release its own means a release really is fixed once started, which is what makes the rest of this work.

A release that can't start

The second version of the shop renders its page with the markdown package, so where the first version returned plain text, it returns the count in bold as a line of HTML. Whoever made the change imported the package but forgot to add it to the project's dependencies, so it isn't in uv.lock. Deploying it:

muructl deploy shop releases/v2

Error: the host exited with exit status: 1 before its workers were ready: cannot load app:application: ModuleNotFoundError: No module named 'markdown'

The site carries on as if nothing happened, still answering from the first release:

3 orders, served by shop@r1

muructl release list shows the new release as failed beside the one still serving:

RELEASE                  STATE      SERVING  WORKERS               DISK SOURCE
shop@r2                  failed              - (2x4)              4.7MB /Users/graham/shop/releases/v2
                         the host exited with exit status: 1 before its workers were ready: cannot load app:application: ModuleNotFoundError: No module named 'markdown'
shop@r1                  ready      yes      2 (2x4)              4.7MB /Users/graham/shop/releases/v1

and the release's own log, from muructl logs shop@r2, has the same reason in context, after the lines from building its environment:

2026-11-07T07:10:13.431Z [build] Creating virtual environment at: /Users/graham/Library/Application Support/muru/envs/shop/r2
2026-11-07T07:10:13.478Z [build] Bytecode compiled 0 files in 42ms
2026-11-07T07:10:13.886Z muru.host[31069]: cannot load app:application: ModuleNotFoundError: No module named 'markdown'

Nor did the failed release leave anything running, with muructl status --processes showing the same processes as before the deploy. Had the new code been put in place over the old and the server reloaded, the new workers would have been trying to import the broken code from where the working code used to be. Here, the release that couldn't start cost nothing but the attempt.

Moving traffic without refusing requests

The third version adds markdown to the project properly, with uv add markdown, so it ends up in the lock file.

Each deploy after the first takes what it isn't told from the release before it, so the two processes and the SHOP_DATABASE variable given to the first deploy carry through. A deploy with --dry-run shows what the deploy would do, with where each setting would come from, and changes nothing. Trimmed down a bit, that looks like:

muructl deploy shop releases/v3 --dry-run

muru: plan (nothing changed): release shop@r3, app:application 2x4 from /Users/graham/shop/releases/v3
  entrypoint = app:application          ([tool.muru])
  python = /Users/graham/.local/bin/python3.15t  (the previous release)
  build.when = auto                     (default)
  ...
  processes = 2                         (the previous release)
  threads = 4                           (the previous release)
  ...
  env SHOP_DATABASE (the previous release)
  ...
  settings nothing else gives come from shop@r1; --fresh takes none
  an environment would be built from the source's uv.lock
  the release would take over from shop@r1, which would stay warm for 1800s with 1 worker to a group

Environment variables carry across by name, which is why SHOP_DATABASE doesn't need to be given again. Where you would rather a deploy be described entirely by its own command line, --fresh ignores the previous release.

To see whether the switch refuses anything, I ran a small script which requests the page every 10 milliseconds, prints a line whenever the response changes, and counts the status codes it got back. With it running, I deployed the third version:

muructl deploy shop releases/v3

muru: release shop@r3 is ready and serving

The output of the script was:

18:10:14 200 3 orders, served by shop@r1
18:10:16 200 <p><strong>3 orders</strong>, served by shop@r3</p>
{200: 695}

Every one of the 695 requests got a 200, with the responses changing over from one release to the other without a gap. The events show the order things happened in:

    14 07:10:15 info     release.created      shop@r3                  from /Users/graham/shop/releases/v3
    15 07:10:15 info     release.building     shop@r3                  from /Users/graham/shop/releases/v3/uv.lock
    16 07:10:16 info     release.built        shop@r3                  /Users/graham/Library/Application Support/muru/envs/shop/r3 in 0.2s from uv.lock 62c319f2cb1f
    17 07:10:16 info     release.ready        shop@r3                  2 workers ready in 0.6 s
    18 07:10:16 notice   traffic.switched     shop@r3                  traffic moved to the release
    19 07:10:16 info     release.draining     shop@r1                  draining
    20 07:10:16 info     release.retained     shop@r1                  kept warm for 1800s with 1 worker per group
    21 07:10:17 info     worker.exited        shop@r1                  worker 1 (pid 31048) stopped: scaled in

The new release was built and started while the old one carried on serving. Traffic moved only once the new release was ready, and the old release then drained, finishing the requests it already had. What happened next to the old release is the interesting part. It wasn't stopped, but scaled in to a single worker and kept warm for 30 minutes, which muructl release list notes against it:

RELEASE                  STATE      SERVING  WORKERS               DISK SOURCE
shop@r3                  ready      yes      2 (2x4)              6.4MB /Users/graham/shop/releases/v3
shop@r2                  failed              - (2x4)              4.7MB /Users/graham/shop/releases/v2
                         the host exited with exit status: 1 before its workers were ready: cannot load app:application: ModuleNotFoundError: No module named 'markdown'
shop@r1                  retained            1 (2x4)              4.7MB /Users/graham/shop/releases/v1
                         warm until 07:40 UTC; a rollback to it is instant

The switch can also be watched as it happens, by leaving muructl status --processes --watch running in another terminal during the deploy. The third release's host, log relay and workers appear beside those of the first, the requests per second move across from one to the other, and the first then drops to a single worker.

Rolling back is a routing change

Say the new page turns out to be a mistake. With the script running again:

muructl rollback shop

muru: rolled back to shop@r1 (ready)

18:10:24 200 <p><strong>3 orders</strong>, served by shop@r3</p>
18:10:26 200 3 orders, served by shop@r1
{200: 363}

The command took a fifth of a second. Nothing was deployed, built or imported. The retained release still had its host and a worker running, with the application already loaded, so rolling back was only a matter of telling the edge to send requests there instead. The events show what followed:

    22 07:10:26 notice   traffic.switched     shop@r1                  traffic moved to the release
    23 07:10:26 notice   app.rolled-back      shop                     rolled back to r1
    24 07:10:26 info     release.draining     shop@r3                  draining
    25 07:10:26 info     release.retained     shop@r3                  kept warm for 1800s with 1 worker per group
    26 07:10:26 info     worker.started       shop@r1                  worker 1 started as pid 31226
    27 07:10:26 info     worker.ready         shop@r1                  worker 1 (pid 31226) is ready
    28 07:10:27 info     worker.exited        shop@r3                  worker 1 (pid 31122) stopped: scaled in

Once it had traffic again, the first release grew back to its two workers by forking from its host, which still had the application imported, as described in import once, fork many. The processes show this, with the first release's second worker only a few seconds old under a host which has been running all along, and the third release, now the retained one, down to the one worker it is kept warm with:

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE           WHAT                                                    USER      UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
31018   daemon         user daemon, state ~/Library/Application Support/muru   graham   17s   0.1%   40.7MB
31019   ├─ edge        listening on 127.0.0.1:8080                             graham   17s   0.0%   29.7MB       0
31045   ├─ host        shop@r1 serving                                         graham   15s   0.0%   54.5MB       0    0/8       0
31044   │  ├─ relay    shop@r1 output to apps/shop/r1.log                      graham   15s   0.0%   17.6MB
31047   │  ├─ worker   group default, worker 0                                 graham   15s   0.0%   48.8MB       0    0/4
31226   │  └─ worker   group default, worker 1                                 graham    3s   0.1%   29.4MB       0    0/4
31117   └─ host        shop@r3 retained until 18:40                            graham   13s   0.1%   56.2MB       0
31116      ├─ relay    shop@r3 output to apps/shop/r3.log                      graham   13s   0.0%   17.5MB
31121      └─ worker   group default, worker 0                                 graham   12s   0.0%   58.1MB

The release rolled back from is now the retained one, so running muructl rollback shop a second time goes forward to it again. Having decided the new page was fine after all, that is what I did, putting the third release back in front of the first:

muructl rollback shop

muru: rolled back to shop@r3 (ready)

The third release grew back to two workers in the same way, and the first went back to being the retained release with one.

Rolling back to a release which has already been stopped, because its 30 minutes are up, also works. The release then has to be started again first, with traffic moving once it is ready. When I tried that with this application, it took about a second rather than a fifth of one, which for a larger application would be however long its import takes.

Restarting the daemon

The other thing which has to happen to a server from time to time is the hosting software itself being restarted or upgraded. As I explained in how it fits together, the daemon is never on the path a request takes, which means it can go away while requests carry on. With the script running once more, I stopped the daemon:

muructl daemon stop

muru: daemon detached; hosts and the edge keep serving

With the daemon gone there is nothing for muructl status --processes to ask, so this time it has to be ps. Each of Muru's processes sets a title saying what it is, so ps is still readable, and shows the edge, the release hosts and their log relays are all still there, now with process 1 as their parent, and the workers still children of their hosts:

31019     1 muru edge user
31044     1 muru relay shop@r1
31045     1 muru host shop@r1
31047 31045 muru worker shop@r1 default/0
31116     1 muru relay shop@r3
31117     1 muru host shop@r3
31121 31117 muru worker shop@r3 default/0
31266 31117 muru worker shop@r3 default/1

Starting the daemon again, it finds them and takes them over:

    36 07:10:31 info     daemon.detached      daemon                   detached; hosts and the edge keep serving
    37 07:10:34 info     daemon.started       daemon                   serving on http://127.0.0.1:8080, admin socket /tmp/muru-501/admin.sock
    38 07:10:34 info     release.adopted      shop@r3                  re-adopted the host (pid 31117)
    39 07:10:34 info     release.adopted      shop@r1                  re-adopted the host (pid 31045)
    40 07:10:34 notice   traffic.switched     shop@r3                  traffic moved to the release

and muructl status --processes has them back in the tree under the new daemon, noting against each process it re-adopted that the operating system now has process 1 as its parent:

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE           WHAT                                                           USER      UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
31301   daemon         user daemon, state ~/Library/Application Support/muru          graham    1s   0.0%   35.8MB
31019   ├─ edge        listening on 127.0.0.1:8080 (re-adopted, OS parent 1)          graham   25s   8.4%   30.2MB       0
31045   ├─ host        shop@r1 retained until 18:40 (re-adopted, OS parent 1)         graham   23s   0.0%   54.5MB
31044   │  ├─ relay    shop@r1 output to apps/shop/r1.log (re-adopted, OS parent 1)   graham   23s   0.0%   17.6MB
31047   │  └─ worker   group default, worker 0                                        graham   23s   0.0%   48.9MB
31117   └─ host        shop@r3 serving (re-adopted, OS parent 1)                      graham   21s   0.0%   56.2MB            0/8       0          0.1 ms
31116      ├─ relay    shop@r3 output to apps/shop/r3.log (re-adopted, OS parent 1)   graham   21s   0.0%   17.5MB
31121      ├─ worker   group default, worker 0                                        graham   20s   5.8%   68.5MB            0/4
31266      └─ worker   group default, worker 1                                        graham    7s   5.3%   63.1MB            0/4

The daemon is a second old, while everything under it has been running since before it was stopped. Meanwhile the script didn't notice anything had happened:

18:10:29 200 <p><strong>3 orders</strong>, served by shop@r3</p>
{200: 703}

Upgrading Muru works along similar lines. muructl daemon upgrade re-executes the daemon from the newly installed muructl and replaces the edge without refusing connections, while the releases keep the runtime library they started with until the next deploy replaces them.

Two releases, one database

Being able to run two releases side by side has a consequence which is easy to overlook. While the new release is starting, and for as long as the old one is retained for a rollback, both are running against the same database. A schema change made for the new release therefore has to keep working with the code of the old one.

The usual answer is to split such a change into steps. The first step only adds things, such as a new table or a new column with a default, which the old code won't notice. Then the release which uses the new schema is deployed. Only once no release that relies on the old schema is serving or retained does a later release remove what is no longer needed. That is general advice, whatever you host with. What Muru adds is a way of running each step with the release it belongs to.

The fourth version of the shop adds a quantity column, in a new file migrations/0002_quantity.sql:

ALTER TABLE orders ADD COLUMN quantity INTEGER NOT NULL DEFAULT 1;

and changes the page to report the number of items as well. If this were deployed like the others, the release would start, be ready, take all the traffic, and then fail every request because the column doesn't exist yet. Instead, it is deployed with --hold, which starts the release but leaves traffic where it is:

muructl deploy shop releases/v4 --hold

muru: release shop@r4 is ready and held: not serving

With the script from earlier still sending requests, the processes show the fourth release fully started and held, with its host, log relay and two workers, while every request is still going to the workers of the third:

edge  37 req/s  0 err/s  p50 1.8 ms  p95 2.7 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 1.9KB/s

PID     ROLE           WHAT                                                           USER      UP    CPU      RSS   REQ/S   BUSY   QUEUE      P95   GIL/LAG
31301   daemon         user daemon, state ~/Library/Application Support/muru          graham    4s   0.0%   38.4MB
31019   ├─ edge        listening on 127.0.0.1:8080 (re-adopted, OS parent 1)          graham   28s   0.0%   30.3MB      37
31045   ├─ host        shop@r1 retained until 18:40 (re-adopted, OS parent 1)         graham   27s   0.0%   54.5MB       0
31044   │  ├─ relay    shop@r1 output to apps/shop/r1.log (re-adopted, OS parent 1)   graham   27s   0.0%   17.6MB
31047   │  └─ worker   group default, worker 0                                        graham   26s   0.0%   48.9MB
31117   ├─ host        shop@r3 serving (re-adopted, OS parent 1)                      graham   24s   0.0%   56.2MB      37    0/8       0   2.7 ms    0.1 ms
31116   │  ├─ relay    shop@r3 output to apps/shop/r3.log (re-adopted, OS parent 1)   graham   24s   0.0%   17.5MB
31121   │  ├─ worker   group default, worker 0                                        graham   24s   0.0%   70.1MB      19    0/4           3.3 ms
31266   │  └─ worker   group default, worker 1                                        graham   11s   0.0%   63.5MB      19    0/4           2.5 ms
31345   └─ host        shop@r4 held                                                   graham    0s   0.0%   56.9MB            0/8       0
31344      ├─ relay    shop@r4 output to apps/shop/r4.log                             graham    0s   0.0%   17.4MB
31346      ├─ worker   group default, worker 0                                        graham    0s   0.0%   20.7MB            0/4
31347      └─ worker   group default, worker 1                                        graham    0s   0.0%   19.8MB            0/4

A held release can still be reached by naming it in a Muru-Preview header. Doing that now shows the problem the migration needs to fix, while ordinary requests are still served by the third release:

curl -H 'Muru-Preview: r4' http://127.0.0.1:8080/

500 Internal Server Error
reason: app_error
request id: 01M6TDGPENFWF4JKE4TJ1G86E2

The application raised an exception.

with muructl logs shop@r4 showing why:

2026-11-07T07:10:40.086Z muru.worker[0]:     (count, items) = db.execute("SELECT COUNT(*), SUM(quantity) FROM orders").fetchone()
2026-11-07T07:10:40.086Z muru.worker[0]:                      ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-11-07T07:10:40.086Z muru.worker[0]: sqlite3.OperationalError: no such column: quantity

The migration is then run with the held release:

muructl command run shop@r4 migrate

applied 0002_quantity.sql

Naming the release matters here. A command runs with the release's own source directory, interpreter, environment variables and secrets, so this ran the fourth release's migrate.py against its own migrations directory, which is the only one with the new file in it. Commands are declared in the project and are part of each release, so the same command name always runs the version of the command which goes with that release's code. With exclusive = true, a second run of the command for the application is refused while one is under way, and muructl command run exits with the command's status, so a deploy script can stop at a failed migration before any traffic moves.

With the migration applied, the held release works, and the third release, which knows nothing about the new column, carries on serving ordinary requests:

<p><strong>3 orders</strong> for 3 items, served by shop@r4</p>
<p><strong>3 orders</strong>, served by shop@r3</p>

All that remains is to move the traffic:

muructl rollout bluegreen shop@r4

muru: release shop@r4 is ready and serving

The third release then becomes the one kept warm for a rollback, and the first, no longer needed for one, is stopped. Because the migration only added a column, rolling back to the third release afterwards still works against the new schema, which I checked by rolling back and forward again. The rollback doesn't touch the schema, which is the reason for doing changes like this in steps.

What's next

With gunicorn and similar servers, all of this is something you build yourself around the server. Here it is what a deploy does.

That said, a deploy only protects you from a release which can't start. The fourth release, deployed without first holding it, would have been ready as far as Muru was concerned, and would then have failed every request it was given. The same goes for a release which starts fine but has a bug in only some of its pages. In the next post I will look at canaries, where a new release is given a small share of the traffic first and is judged against the release already serving, using what the workers report about the requests they handle, before it gets any more.

November 12, 2026 12:00 AM UTC

November 11, 2026

The Python Coding Stack

Use Your Initiative

Subscribe now

stephengruppetta.com

November 11, 2026 04:31 PM UTC

LernerPython blog, from Reuven Lerner

Pandas option_context: Temporarily change display settings

Want to set an option in Python Pandas, but just temporarily? You can use option_context:

with pd.option_context('display.precision', 2):
    print(s/3)

Within the “with” block, the display precision is 2. But outside — before or after — it is the default.

The post Pandas option_context: Temporarily change display settings appeared first on LernerPython.

November 11, 2026 07:00 AM UTC

Python GUIs

Keeping Your Main Window Open After Closing a Dialog in PyQt6 — Why your app closes when you dismiss a dialog, and how to fix it

I'm working with dialogs in PyQt6, and when I press either Accept or Cancel, the entire application closes. How can I prevent this?

When you create and display a dialog (for example, using QDialog.exec()), the dialog runs its own local event loop. Once you click OK or Cancel, the dialog closes and exec() returns a result. At that point, your code continues executing whatever comes after the dialog call.

If there's no main window being shown and no main event loop running, the application simply has nothing left to do — so it exits.

In other words, the dialog is your entire application. Once it's gone, everything shuts down.

The Solution: Run a Main Window with an Event Loop

To keep your application alive after the dialog closes, you need a main window that stays open and a properly running event loop. The dialog should be launched from that main window, so that when the dialog is dismissed, control returns to the main window which is still visible and active.

Here's a minimal complete example that demonstrates the correct setup:

import sys

from PyQt6.QtWidgets import (
    QApplication, QDialog, QDialogButtonBox,
    QMainWindow, QPushButton, QVBoxLayout,
)


class CustomDialog(QDialog):
    def __init__(self, parent=None):
        super().__init__(parent)
        self.setWindowTitle("My Dialog")

        buttons = QDialogButtonBox.StandardButton.Ok | QDialogButtonBox.StandardButton.Cancel
        self.button_box = QDialogButtonBox(buttons)
        self.button_box.accepted.connect(self.accept)
        self.button_box.rejected.connect(self.reject)

        layout = QVBoxLayout()
        layout.addWidget(self.button_box)
        self.setLayout(layout)


class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Main Window")

        button = QPushButton("Open Dialog")
        button.clicked.connect(self.open_dialog)
        self.setCentralWidget(button)

    def open_dialog(self):
        dialog = CustomDialog(self)
        result = dialog.exec()

        if result == QDialog.DialogCode.Accepted:
            print("User clicked OK")
        else:
            print("User clicked Cancel")


if __name__ == "__main__":
    app = QApplication(sys.argv)
    window = MainWindow()
    window.show()
    sys.exit(app.exec())

Let's walk through what's happening here.

The Main Window

The MainWindow class creates a simple window with a single button. When you click that button, it calls open_dialog(), which creates an instance of CustomDialog and displays it using exec().

Once you accept or cancel the dialog, exec() returns and you're back inside the open_dialog method. The main window is still visible and responsive — you can click the button again to reopen the dialog as many times as you like. If you're new to building windows in PyQt6, see our tutorial on creating your first window.

The Event Loop

The block at the bottom of the script is what keeps everything running:

if __name__ == "__main__":
    app = QApplication(sys.argv)
    window = MainWindow()
    window.show()
    sys.exit(app.exec())

QApplication manages the main event loop for your application. Calling app.exec() starts that loop, which keeps the application alive, processing user input, repainting widgets, and handling signals and slots. The application will only exit when you close the main window (or explicitly call app.quit()).

Without this event loop, your script runs top-to-bottom and exits as soon as it reaches the end — which is exactly the behavior you'd see if a dialog was the only thing in your script.

Passing self as Parent

You might have noticed that we pass self when creating the dialog:

dialog = CustomDialog(self)

This sets the main window as the dialog's parent. It means the dialog will appear centered over the main window and will be properly tied to it in terms of window management. It's a good habit to set a parent for your dialogs whenever you can.

Summary

If your app is closing when you dismiss a dialog, make sure you have:

1. A QApplication instance with its event loop running via app.exec().
2. A main window that is shown using .show().
3. Your dialog launched from within that main window, so control returns to it after the dialog is dismissed.

With this structure in place, your dialogs will open and close without affecting the rest of your application. If you want to open additional windows beyond dialogs, take a look at our guide on creating multiple windows in PyQt6.

For an in-depth guide to building Python GUIs with PyQt6 see my book, Create GUI Applications with Python & Qt6.

November 11, 2026 06:00 AM UTC

Graham Dumpleton

AI as a lifestyle choice

When I wrote about twenty years of mod_wsgi, I mentioned that I was on sabbatical, that AI coding tools had changed what I was prepared to take on, and that I was hoping people would tell me what they think is missing from the current options for hosting Python web applications. Each of those only got a sentence or two in a post which was mostly about Apache, and I want to expand on them here.

I should start by clarifying my situation, or at least try to. I would like to call it retirement. My better half would like to see me still working. Nothing is official either way, and negotiations continue. Some people in my position would buy a caravan and head off around Australia, but I already travelled the outback a lot when I was much younger. What I seem to have chosen to do instead is spend my days building software with AI, and I am having about as much fun creating software as I have ever had.

What the lifestyle looks like

It is only a month since I decided I would start on Muru. There were a few days of planning and thinking before any code got written, and since then I have been working on it seven days a week, days and evenings. Nobody is making me do this. I can't remember the last time something was engaging, fulfilling and fun all at the same time.

Before that sounds too alarming, it doesn't mean I am chained to a desk. AI coding tools have turned out to fit quite well around the rest of life. I can set something going, then head out for a walk, do some gardening, or deal with whatever else needs doing around the house, and come back to see how it went. When I was writing all the code myself, stepping away meant the work stopped. My better half may even be prepared to count that in my favour during the negotiations.

That is what I mean by a lifestyle choice. I said in that earlier post that the coding is the part I find a chore these days, which might seem odd coming from someone who has never really stopped creating software. The making of things was always the point though, not the typing. AI gets rid of the grunt work that can be so frustrating, and leaves me with the ideas and the satisfaction of seeing them work.

Getting away from the meter

It took me a while to realise how much the price of these tools was affecting how I used them. When I was paying $20 a month I was always worried I would run out, and as a result I actually held back from relying on it. Moving up to $200 a month produced the opposite problem. Paying that much, I felt I had to make the most of it, and so would end up using it for a crazy number of hours just to get my money's worth. Neither is a great way to go about something you are meant to be doing for fun.

Right now I don't have that problem, because I have been accepted into the Claude for Open Source Program for a second round. I very much appreciate that, and it has made a huge difference. More than anything it has released me from worrying about the cost, and I feel free to do whatever I want without feeling constrained.

Obviously there is the question of what happens when I go back to having to pay for it, and I don't have a good answer to that yet. For now I am taking as much advantage of it as I can.

Everything starts out as mud

I will acknowledge that AI may not always generate the best code out of the box. To be frank though, neither did I when I started on mod_wsgi. The early versions had plenty wrong with them. I made lots of mistakes and had to go back and do things differently, and it took years of fixing and rethinking to get it to where it is now.

All developers are like this when they start out on something. People seem to forget that as soon as AI is involved, and expect instant perfection of a kind they would never have been able to achieve themselves.

Writing software is an incremental process of improvement, and in some respects can be likened to making a dorodango. This is a Japanese craft where you take ordinary dirt and water and form it into a ball. It starts out as a crude lump of mud. You shape it, let it dry, add layers of fine dust and keep working at the surface, over and over, until you end up with a sphere with a shine on it that looks nothing like what it was made from.

I wrote recently about how calling something "AI slop" is often just a way of saying you didn't read it. Well, slop is more or less what a dorodango starts out as. The insult only sticks if you stop there. The AI may be doing a lot of the work on the surface, but deciding what isn't good enough yet is still down to me, and twenty years of getting things wrong with mod_wsgi turns out to be useful for that. I am prepared to put in the hard work to keep polishing Muru until it is something I would trust, and something you would too.

Tinkering in the back room

I have been programming for over 40 years now, and have been involved in free or open source software for nearly all of that time. Even as I head towards retirement, or whatever we end up calling it, I still want to do more. The Python community has always been the most welcoming community I have ever interacted with, and Muru is my way of giving something back to it.

There are many ways of contributing to a community like that. A lot of people give their time to the PSF and to running Python conferences, and we are all better off for the work they do. It just wouldn't suit me, as bureaucracy and organising things are not where my strengths lie. Back room tinkering and creating software has always been my thing, and I expect always will be.

I also know how badly AI written code being handed to open source projects can go, with maintainers left to deal with what lands on them. The difference here, I hope, is that I am not throwing anything over the fence for someone else to look after. If Muru has problems, they are my problems.

The things you never had time to build

I know how a month old project written largely by AI sounds. Muru is not a weekend of vibe coding that will be forgotten about by Christmas though. The code may be new, but the thinking behind it has been accumulating for twenty years. Muru is my thing for the foreseeable future, and I intend to commit to it and see how far I can take it.

I am not doing this only to see what I can achieve based on my own ideas. Part of the reason for putting it out there is so people can see those ideas and come back with their own, about what they would like to see in a system for hosting Python web applications.

A few people have already left comments on my social media posts about the series, showing an interest in specific things Muru is doing. That has been encouraging, and suggests I am at least on the right track. I would like a lot more of that feedback though, especially about what isn't there yet.

If you work with Python web applications for a living, I suspect you have a list. There will be points of friction in your day to day development work that you would love to see gone. There will be things you need to do in production which are much harder than they should be. Some of it you have wanted to fix for years, but you never had the time, or your employer wasn't interested. I know that list well, as I had one of my own for most of my working life.

That is where my situation is now different. Not having a job, I have the time. Open source has always said "patches welcome", which isn't much use when lack of time is the whole problem. So don't send me a patch, just describe the problem.

If you aren't sure where to start, think about the scripts you have written around your deployments that you wish you didn't need, or the thing you end up explaining to every new person who joins the team. Maybe there is something that keeps getting pushed down the backlog, or a production task you put off because of how painful it is. I can't promise everything will make it in, as an idea needs to fit within the overall system I am trying to develop, but if it does then I am happy to look at it.

In that earlier post I said I wasn't going to point to where Muru lives just yet. It is time to change that. To see what Muru can already do, which is a lot more than the posts so far have covered, have a look through the documentation. The place to tell me what you would like to see is the ideas section of the Muru discussions on GitHub. I am not a huge fan of online discussion forums like this, but there don't seem to be any other good choices these days. If they aren't your thing either, then just leave a comment on one of my social media posts when something sounds like it is what you would use or need. Even that much helps me work out where to put the effort.

Polishing in public

You will have to put up with me posting about Muru every day for a while yet. Sorry about that. The posts are actually part of the polishing process, as are the online interactive workshops I have been creating for it using the workshop extension I wrote for JupyterLab. If I can't write a post which explains something easily, or create a workshop which makes it usable for the people who may want to use it, then its design and implementation were bad to start with. Having to explain it is what shows me where it needs more work.

The ball is still pretty muddy in places. I have the time though, I am having fun, and whatever my better half and I end up deciding to call this, I plan to keep polishing. Fingers crossed it ends up with a shine.

November 11, 2026 12:00 AM UTC

Muru: WSGI and ASGI together

The previous post was about serving HTTPS from a development run. This time I want to get back to something I mentioned in passing when I first talked about Muru, which is that it isn't limited to WSGI, and that WSGI and ASGI can be mixed in the one application.

The case where this matters most is an existing WSGI application which now needs websockets. WSGI has no way of doing websockets, so the usual answer is to stand up a separate ASGI server beside the WSGI one, put a proxy in front to route the websocket paths to it, and then work out how the two halves are going to share things like sessions and configuration. The other answer is to port the whole application to ASGI, which for a large application is rarely something anyone wants to do just to get one feature.

With Muru, neither is necessary. The WSGI application can hand a request over to an ASGI application running in the same worker process.

An existing WSGI application

To show how this works I will use a small Flask application, standing in for an application which has been around for years. The project has Flask installed with uv add flask, and the pyproject.toml file names the entrypoint:

[tool.muru]
entrypoint = "shop.app:app"

The application itself has the existing page, plus a new /chat view where the websocket is to live:

from flask import Flask, request

from shop.chat import chat

app = Flask(__name__)


@app.route("/")
def index():
    return "Welcome to the shop\n"


@app.route("/chat")
def chat_view():
    bridge = request.environ["muru.wsgi_to_asgi"]
    if not bridge.is_websocket:
        return "websocket required\n", 426
    return bridge.handoff(chat)

Every WSGI request Muru passes to an application has a muru.wsgi_to_asgi object in the WSGI environ. This is the bridge across to the ASGI side. It can say whether the request is a websocket upgrade, and its handoff() method takes an ASGI application, not a WSGI one. Calling handoff() marks the request so that, once the WSGI side has returned, Muru carries on with the request by passing it to that ASGI application.

What handoff() returns is there for convenience, and is designed to suit both raw WSGI applications and web frameworks. It is an empty iterable, which is what a WSGI application returns for a response with no body, so a raw WSGI application can return it straight away. The same object can also be called as if it were a WSGI application, in which case it responds with a 200 status and returns itself as an empty body. Between them, these cover what web frameworks will accept from a view. Current versions of Flask take an iterator returned from a view as the body of a response, while older versions instead run a callable returned from a view as a WSGI application. Either way, a Flask view can return what handoff() gives back, as the example above does.

For any other framework, how you return from the handler doesn't matter. Once a request has been handed off, Muru ignores whatever response the WSGI side produces, so all that is needed is to call handoff() and then return whatever response the framework expects.

Websockets as callbacks

What the request is handed off to, chat, has to be an ASGI application. You could write that as a raw ASGI coroutine, but if your application is WSGI, the people working on it are probably more at home with synchronous code. For that Muru provides muru.asgi.websocket.Handler, an ASGI application which turns the events of a websocket into method calls on an object of your own. Here is a minimal chat room in shop/chat.py, where every message received is sent on to everyone connected:

from muru.asgi import websocket


class Chat:
    def __init__(self):
        self.sockets = set()

    def on_connect(self, ws):
        self.sockets.add(ws)

    def on_receive(self, ws, message):
        for peer in list(self.sockets):
            try:
                peer.send(message)
            except websocket.ConnectionClosed:
                self.sockets.discard(peer)

    def on_disconnect(self, ws, code, reason):
        self.sockets.discard(ws)


chat = websocket.Handler(Chat())

The methods are called on the worker's application threads, the same threads the WSGI application uses. Calls for any one socket happen one at a time and in order, while different sockets can be handled at the same time. The important part, which I will come back to, is that a socket sitting there waiting for its next message isn't holding on to a thread while it waits.

Turning handoff on

Start this with muructl run and the ordinary page works, but a websocket client connecting to /chat is turned away. The access log shows why, with the view having answered with a 426 status:

426 GET /chat 19B 0.5ms queue=0.0ms app=0.5ms worker=0 id=01M6JWP0E8W9FQ721K81F9AJ9H

Handing requests to ASGI has to be enabled, as it means the worker needs to run an event loop beside its threads. Without the setting, a websocket upgrade is just an ordinary request, so is_websocket is false and the view turns it away. Enabling it is one more line in pyproject.toml:

[tool.muru]
entrypoint = "shop.app:app"
handoff = "asgi"

Run it again and the start-up output confirms the worker now has an event loop as well as its threads:

muru: serving shop.app:app on http://127.0.0.1:8080 with 1 worker of 4 threads and an event loop of concurrency 1000

To check the chat room really works, I used a small client written with the websockets package, installed into a separate virtual environment, which connects as two users and has each of them say something:

import asyncio

from websockets.asyncio.client import connect

URL = "ws://127.0.0.1:8080/chat"


async def main():
    async with connect(URL) as alice, connect(URL) as bob:
        await alice.send("hello from alice")
        print("alice got:", await alice.recv())
        print("bob got:", await bob.recv())
        await bob.send("hi alice, bob here")
        print("alice got:", await alice.recv())
        print("bob got:", await bob.recv())


asyncio.run(main())

Running this the output is:

alice got: hello from alice
bob got: hello from alice
alice got: hi alice, bob here
bob got: hi alice, bob here

The access log has a line for each websocket when it is accepted, and another when it closes, recording who closed it, how, and how many messages went each way. Both are marked with handoff=asgi so you can tell the request was handed across:

101 GET /chat 0B 1.8ms queue=0.0ms app=0.5ms worker=0 handoff=asgi id=01M6JWPC691F8KMJB3TQH8CKSK
101 GET /chat 0B 0.3ms queue=0.1ms app=0.2ms worker=0 handoff=asgi id=01M6JWPC6B1F8KMJB3TQH8CKSK
101 GET /chat 34B 1.7ms websocket=closed by=client code=1000 received=1 sent=2 worker=0 handoff=asgi id=01M6JWPC6B1F8KMJB3TQH8CKSK
101 GET /chat 34B 2.9ms websocket=closed by=client code=1000 received=1 sent=2 worker=0 handoff=asgi id=01M6JWPC691F8KMJB3TQH8CKSK

Sockets don't hold threads

The reason this isn't just a party trick is what happens to the thread the WSGI view ran on. Once the view hands the request off and returns, the thread goes back to the pool to handle other requests. The websocket now lives on the worker's event loop, and only borrows a thread for the brief time it takes to run one of the callback methods.

That matters, because this worker only has four threads. If each open websocket tied up a thread, the fifth person to open the chat would stop the rest of the site from responding. To check, I changed the client to open fifty websockets to /chat, leave them all open, and then make an ordinary request to the home page:

50 websockets open
GET / answered b'Welcome to the shop\n' in 6.2ms

Fifty open websockets on a worker with four threads, and the rest of the site carries on as normal.

Side by side by path

Handing off in code is the right tool when the decision about which side a request goes to belongs in the application, such as wanting the Flask view to check the user is logged in before accepting the websocket. When the decision is simply about the path, Muru can instead mount an ASGI application beside the WSGI one.

Say newer parts of the shop are being written as async code under /api. Here is a stand-in for that in shop/api.py, where the sleep pretends to be waiting on some other service:

import asyncio
import json


async def app(scope, receive, send):
    if scope["type"] != "http":
        return

    await asyncio.sleep(1)  # stand in for waiting on another service

    body = json.dumps({"root_path": scope["root_path"], "path": scope["path"]}).encode()
    await send({"type": "http.response.start", "status": 200, "headers": [(b"content-type", b"application/json")]})
    await send({"type": "http.response.body", "body": body})

Mounting it at /api is a table in pyproject.toml:

[tool.muru.mounts.api]
path = "/api"
entrypoint = "shop.api:app"

On start-up, Muru says what each entrypoint is being served as, and where each mount is:

muru.host[89570]: shop.app:app is served as WSGI
muru.host[89570]: mount api: shop.api:app is served as ASGI
...
muru: mount api serves /api from shop.api:app

Anything under /api now goes to the ASGI application, and everything else to Flask. The ASGI application sees the mount path as its root_path, as the ASGI specification asks:

curl http://127.0.0.1:8080/api/orders

{"root_path": "/api", "path": "/api/orders"}

Both applications run in the same worker processes, imported by the same preload, from the one project.

Requests to the ASGI application run on the worker's event loop, so they aren't held back by the four threads the WSGI application has. Firing off twenty requests to /api at once, each awaiting that one second sleep, they all finished together in 1.06 seconds, rather than queuing up four at a time behind the threads.

Left alone, a mount shares the worker's event loop, and so its concurrency, with everything else running on it, such as the websockets handed off from Flask. If one part of the site shouldn't be able to take all of that, a mount can be given a concurrency of its own:

[tool.muru.mounts.api]
path = "/api"
entrypoint = "shop.api:app"
concurrency = 500

Muru confirms this when it starts:

muru: mount api serves /api from shop.api:app with concurrency 500 of its own

There is a similar max_connections setting to put a limit on how many websockets an ASGI mount can hold open.

And back the other way

Handing off works in the other direction too. An ASGI application can hand a request to WSGI code, which is useful when moving an application to ASGI a piece at a time, leaving the parts not yet converted as they are. In the scope given to an ASGI application, Muru's extensions include an asgi_to_wsgi bridge, whose handoff() runs a WSGI application for the rest of the request:

def legacy_report(environ, start_response):
    body = f"report generated by WSGI code for {environ['SCRIPT_NAME']}{environ['PATH_INFO']}\n".encode()
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [body]


async def app(scope, receive, send):
    if scope["type"] != "http":
        return

    if scope["path"].startswith("/api/reports/"):
        return scope["extensions"]["muru"]["asgi_to_wsgi"].handoff(legacy_report)

    ...

As before, this has to be enabled, and if you forget, the error says exactly what to add:

muru.HandoffNotEnabled: mount api does not enable handing requests to WSGI; set handoff = "wsgi" on the mount

WSGI code needs threads to run on, so as well as setting handoff, the mount here is given two threads of its own:

[tool.muru.mounts.api]
path = "/api"
entrypoint = "shop.api:app"
concurrency = 500
handoff = "wsgi"
threads = 2

The start-up output now shows the mount with its own threads as well as its own concurrency:

muru: mount api serves /api from shop.api:app on 2 threads of its own with concurrency 500 of its own

Now a request for a report goes to the old WSGI code, while everything else under /api stays with the ASGI application:

curl http://127.0.0.1:8080/api/reports/daily

report generated by WSGI code for /api/reports/daily

with the access log marking it with handoff=wsgi:

200 GET /api/reports/daily 53B 0.4ms queue=0.0ms app=0.4ms mount=api worker=0 handoff=wsgi id=01M6JWRJ0DQ38194C2WQX2KDCJ

gRPC as well

The same mount mechanism also takes care of the third kind of service I mentioned in the first post, which is gRPC. A gRPC mount doesn't have a path. Instead, calls are recognised by their content type and service name, so gRPC and ordinary HTTP share the same port. The entrypoint is a function which registers the servicers generated by grpcio-tools, just as you would with a grpcio server:

class Greeter(greeter_pb2_grpc.GreeterServicer):
    def SayHello(self, request, context):
        return greeter_pb2.HelloReply(message=f"Hello {request.name}")


def register(server):
    greeter_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)

[tool.muru.mounts.api]
interface = "grpc"
entrypoint = "svc.app:register"

muru.host[91221]: mount api: svc.app:register is served as gRPC
...
muru: mount api serves the gRPC services of svc.app:register

With that, a grpcio client calling SayHello and a browser fetching the home page can talk to the same port of the same development run, and both show up in the one access log:

200 GET / 30B 0.2ms queue=0.0ms app=0.2ms worker=0 id=01M6JWS8HV7NK43EP3VYEA225P
200 POST /greet.Greeter/SayHello 14B 0.9ms queue=0.0ms app=0.8ms worker=0 id=01M6JWS8KT7NK43EP3VYEA225P

One catch is that grpcio doesn't provide wheels for free-threaded Python, so for gRPC I had to use a regular build of Python. There is a lot more to gRPC in Muru, such as streaming calls and gRPC-Web, which I may come back to in a post of its own.

What's next

What I like about all of this is that you don't have to make a big decision up front. An application can start as WSGI, take websockets when it needs them, grow async parts under a path, and move across to ASGI piece by piece if that ever makes sense, all without becoming more than one deployment.

That wraps up what I wanted to show of Muru in development mode. Next it is time to move on to the daemon, starting with deploying an application to a server, where the releases that a reload creates in development mode become deploys that can run side by side, take a share of the traffic, and be rolled back.

November 11, 2026 12:00 AM UTC

Seth Michael Larson

Creating for yourself

Today marks the 7th anniversary of starting this blog. If you've been following along for a while: thank you! 😊

After observing Volunteer Responsibility Amnesty Day in 2025 and stopping most of my volunteer OSS contributions, I really leaned hard into using my creative leisure time "for me". This has helped protect my flame from burn-out and focus on areas I'm excited about, like writing, video-games (retro, modding, ...developing? 👀), and keeping my volunteer OSS focus on mostly a single project: truststore. This year was the first time since 2016 that I was uninvolved in the latest release of urllib3, which felt both deeply rewarding and strange as I read the v2.8.0 release notes.

Writing

This year I published 54 articles to my blog, an average of once per week across a year. This wasn't a specific goal I had in mind, looking through the past year of articles has me thinking "oh, that was this year??". Writing continued to be a focus of my role at the PSF, too, where I wrote 14 articles for Python Insider, 5 for the PSF blog, and 2 for the Python Package Index blog (that's a lot of yapping!)

The piece I'm most proud of from this year is “Food JPEGs in Super Smash Bros. and Kirby Air Riders”. This took tons of research, buying the Japanese Kirby Air Ride, writing some custom software, and I think this work really showed through in the end. This was my most-visited article of the year (mostly from the Hacker News front page).

My under-dog favorite is “Adding Animal Crossing events to your digital calendar” where I got to deep-dive into how Animal Crossing events work, differences between regions and games, the Animal Crossing decompilation project, and bought a physical Animal Crossing calendar too. I learned about the ICS calendar format and which features are supported across different platforms like iOS and Android.

Software

I took a much more fast-and-loose approach to sharing software I've worked on this year, opting to use this blog and Gists instead of each project needing a forge, issue tracker, and to be published to PyPI. I found that this approach let me spend more time on what actually mattered to me: being able to tell the story!

• LAN Party calculator for Mario Kart, Kirby Air Riders, F-Zero (Source)
• Adding Animal Crossing events to your digital calendar (Source)
• Blind Carbon Copy (BCC) for SMS (Source)
• sugarcookie: a sprite-sheet cutting tool (Source)

I continued building tools allowing you to extract and archive ROMs from legally owned physical copies of games:

• Nintendo e-Reader cards
• GC-to-GBA Joyboot, such as Billy Hatcher "Joycarry" games
• Ocarina of Time: Master Quest Bonus Disc
• GBA BIOS

Finally, I created tools that track my play activity across all the platforms I care about. Can't improve what you don't measure, this data comes up in the next section:

• Nintendo Play Activity (OCR) (Source)
• Retroachievements Play Activity (Source)
• GameCube Play Activity (Memcard Pro GC) (Source)

Gaming

Ever since 2023 I've been “in my feels” about gaming. I considered myself a gamer, but I wasn't actually playing games. I set a goal to get back into the hobby more seriously, and figure out how I can integrate play back into my week. 2025 was successful already, and 2026 was even more so.

I ended up really experimenting this year by writing about a slice of my experience for four of the games I completed. I don't think I'll want to write about every game I play, but thinking deeply about my feelings while playing the game and especially the time between play sessions has been an interesting lens to look through. Maybe more to come here?

• “The Legend of Zelda: Link’s Awakening” respects your time
• The Frog for Whom the Bell Tolls
• Bowser’s Fury outshines Sunshine, SA2B, and Billy Hatcher
• Mario Kart World and “seamless” media

These are all the games that are "completable" that I started and mostly finished this year:

Using the three play activity tracking projects I have data about play activity from the Nintendo Switch, GameCube, and the Delta emulator. This data showed I played 83 hours of games this year compared to 78 in 2025.

I'm ending this year playing “Ocarina of Time” for the first time and oh my is this game incredible! I cannot stop thinking about it having immersed myself completely in the design, discussion, and themes prior to starting. I'm part-way through the Forest Temple and all I can think is that there are 4 more temples to complete, what a game!

Thanks for reading ♥ I would love to hear your thoughts! Contact me via Mastodon, Bluesky, or email. Browse the blog archive. Check out my blogroll.

November 11, 2026 12:00 AM UTC

November 10, 2026

Python Docs Editorial Board

Meeting Minutes: Nov 10, 2026

Meeting Minutes from Python Docs Editorial Board: Nov 10, 2026

November 10, 2026 08:00 PM UTC

PyCoder’s Weekly

Issue #760: Type Checkers, Claude Code Hooks, MarkItDown, and More (2026-11-10)

#760 – NOVEMBER 10, 2026
View in Browser »

Why Your Python Agent Returns Outdated Memories

A user changes a preference, but vector search still surfaces the old one alongside the new. Link the new memory to the one it replaces and mark the old one historical, so your agent retrieves what’s current and keeps the history. Explore the walkthrough and runnable notebook.
ORACLE sponsor

Discussions

Articles & Tutorials

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #760.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

November 10, 2026 07:30 PM UTC

Python Bytes

#504 Who turned the GIL back on?

<strong>Topics covered in this episode:</strong><br> <ul> <li><strong><a href="https://blog.pypi.org/posts/2026-11-06-ui-updates-complete/?featured_on=pythonbytes">PyPI's redesigned project pages</a> now warn when provenance changes</strong></li> <li><strong><a href="https://mkennedy.codes/posts/catch-the-import-name-the-culprit-keep-threads-free-introducing-gilguard/?featured_on=pythonbytes">gilguard catches the import that turns the GIL back on</a></strong></li> <li><strong><a href="https://pip.pypa.io/en/stable/news/#v26-3">pip 26.3:</a> pylock extras, and pip says plainly that untrusted wheels aren't safe</strong></li> <li><strong><a href="https://heyutils.com/tools/?featured_on=pythonbytes">All developer tools</a></strong></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=rPczSyhinpc' style='font-weight: bold;' data-umami-event="Livestream-Past" data-umami-event-episode="504">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by <a href="https://pythonbytes.fm/longlink"><strong>LongLink</strong></a>: Open-source platform for building and running custom business applications with Python and FastAPI <strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy">Mastodon</a> / <a href="https://bsky.app/profile/mkennedy.codes?featured_on=pythonbytes">BlueSky</a> / <a href="https://x.com/mkennedy?featured_on=pythonbytes">X</a> / <a href="https://www.linkedin.com/in/mkennedy/?featured_on=pythonbytes">LinkedIn</a></li> <li>Calvin: <a href="https://sixfeetup.social/@calvin?featured_on=pythonbytes">Mastodon</a> / <a href="https://bsky.app/profile/calvinhp.com?featured_on=pythonbytes">BlueSky</a> / <a href="https://x.com/calvinhp?featured_on=pythonbytes">X</a> / <a href="https://www.linkedin.com/in/calvinhp/?featured_on=pythonbytes">LinkedIn</a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes">Mastodon</a> / <a href="https://bsky.app/profile/pythonbytes.fm">BlueSky</a> / <a href="https://x.com/PythonBytes?featured_on=pythonbytes">X</a> Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Tuesday at 7am PT</strong>. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it.</li> </ul> <p><strong>Calvin #1: <a href="https://blog.pypi.org/posts/2026-11-06-ui-updates-complete/?featured_on=pythonbytes">PyPI's redesigned project pages</a> now warn when provenance changes</strong></p> <ul> <li>Live now on <a href="http://pypi.org/?featured_on=pythonbytes">pypi.org</a> (Nov 6). Announced in July, shaped by user research, funded by OpenSSF and built by Kabu Creative.</li> <li>New Security tab on every project page shows each release's provenance state and per-file attestations, with a link to the transparency log.</li> <li>Two attestation types: PyPI Publish (uploaded by a Trusted Publisher) and SLSA Build Provenance (ties a file to its source commit, build platform and instructions).</li> <li>"No provenance" isn't a malware flag; it means the file's origin can't be cryptographically verified. That's typical for older releases or projects without Trusted Publishing.</li> <li>Four new provenance warnings: Incomplete (e.g. wheels via Trusted Publishing but sdist via API token), Inconsistent (mixed build systems or repos), Change (different repo or workflow than last release), Loss (stopped publishing attestations).</li> <li>Warnings are meant to catch files that may not have come from the maintainers, such as after an account or publishing-workflow takeover. PyPI says a Change of Provenance "can also indicate a compromised pipeline or account takeover," and an unexpected Loss of Provenance should be treated "with skepticism until verified.”</li> <li>Warnings now sit at the top of the page with yanked, archived and pre-release banners. A quarantine banner overrides all of them. Sidebar metadata is labeled "PyPI data" or "Data verified by PyPI."</li> <li>Release files tab gets search, filters, a per-file provenance column and orange icons for "late" files (uploaded more than 14 days after release). Those only appear on releases from before July 2026, when PyPI started rejecting new files after 14 days.</li> </ul> <p><strong>Michael #2: <a href="https://mkennedy.codes/posts/catch-the-import-name-the-culprit-keep-threads-free-introducing-gilguard/?featured_on=pythonbytes">gilguard catches the import that turns the GIL back on</a></strong></p> <p>gilguard catches a nasty free-threading surprise: importing a C extension without declared free-threading support can turn the GIL back on for the rest of the process. Import gilguard first and it raises a detailed error naming the extension, distribution/version, import chain, and the line of your code responsible.</p> <ul> <li>A transitive dependency can undo your move to free-threaded Python. Fail at the offending import instead of discovering lost parallelism in a benchmark.</li> <li>Uses import hooks plus CPython's warning path to catch GIL activation; GILEnabledError inherits BaseException, so even except Exception won't swallow it.</li> <li>python -m gilguard audits installed top-level packages in fresh interpreters. Supports targeted modules and JSON output; exits 1 when it finds an offender for CI.</li> <li>The audit misses extensions loaded lazily on first use. Exercise those paths in tests with gilguard installed early; one process can identify only its first GIL-flipping import.</li> <li>GILGUARD_ACTION=warn reports without stopping; off permits standard builds. Default mode raises on standard Python or an already-enabled GIL. PYTHON_GIL=0 prevents activation, so the guard cannot detect offenders there.</li> <li>Published on <a href="https://pypi.org/project/gilguard/?featured_on=pythonbytes">PyPI</a> as 0.1.0. Install with uv pip install gilguard. Python 3.13+, MIT; no runtime dependencies.</li> </ul> <p><strong>Calvin #3: <a href="https://pip.pypa.io/en/stable/news/#v26-3">pip 26.3:</a> pylock extras, and pip says plainly that untrusted wheels aren't safe</strong></p> <ul> <li>pip 26.3 released Nov 2. Its security policy now says pip "has never considered installing untrusted wheels safe," because Python's import system is dynamic, so pip can't guarantee installing a wheel won't execute third-party code (#14412).</li> <li>The change: pip blocks unexpected imports during wheel installation "to avoid loading code from packages being replaced." It loads what it needs for installing and error reporting up front (PR #14247, merged Nov 2: https://github.com/pypa/pip/pull/14247).</li> <li>CLI vs. library use: from the command line, imports stay blocked until pip exits. Code that calls pip programmatically gets warnings after control returns.</li> <li>Rolled out in steps: pip 26.1 started warning ("Unexpected import of '...' after pip install started. pip 26.3 will enforce this behavior change"), and 26.3 turns it into an error. Tracking issue #13842 collected reports before enforcement: https://github.com/pypa/pip/issues/13842</li> <li>What tripped it in the wild: a <code>.pth</code> startup hook at Pradyun Gedam's workplace that registers an atexit handler to report interpreter usage metrics over HTTP (https://github.com/pypa/pip/issues/13842#issuecomment-4613463933), plus warnings involving <code>lark_oapi</code> and <code>rich._emoji_codes</code>.</li> <li>Escape-hatch debate: the author, Damian Shaw (notatallshaw), suggested appending module names to <code>pip._internal.commands.install._EAGER_IMPORTS</code>. Pradyun objected that this requires importing from <code>pip._internal</code>, that importing pip's install command is expensive (about 689 ms with no .pyc files and a cold filesystem cache), and that pip isn't necessarily installed in every environment. Shaw's reply: it "just moves where the import cost is," and turning the warning into an error is "crucial."</li> <li>Interrupted installs: pip now writes recovery metadata and reports recovery failures and where it kept backups. An interrupted uninstall may need manual recovery, and an interrupted --target install can leave a partial target directory.</li> <li>Also in 26.3: <code>-r pylock.toml[extra1,extra2]</code> for extras, <code>--group pylock.toml:docs</code> for non-default groups, <code>--no-binary</code> with pylock files, and pip refuses to follow an HTTP redirect to a file: URL.</li> </ul> <p><strong>Michael #4: <a href="https://heyutils.com/tools/?featured_on=pythonbytes">All developer tools</a></strong></p> <p>HeyUtils is a collection of 306 free developer utilities in 41 categories, with no account or installation required. The full catalog covers data conversion, APIs, networking, DevOps, frontend work and binary protocols. Python developers get model generation and packaging helpers alongside the everyday formatters and decoders worth bookmarking.</p> <ul> <li>One bookmark for the tiny jobs that interrupt coding: inspect a payload, compare configs, decode a token or figure out a cron schedule.</li> <li>Data tools include JSON/YAML/TOML conversion, CSV joins, JSONPath and JMESPath testers, a jq playground, and a SQLite viewer with read-only queries.</li> <li>Python-specific options go beyond regex: generate dataclasses or Pydantic models from JSON, check PEP 440 versions, and inspect requirements.txt or pyproject.toml dependencies.</li> <li>The collection also covers JWT/OAuth debugging, DNS and TLS checks, Docker Compose, Kubernetes YAML, GitHub Actions, and frontend helpers such as color contrast checks.</li> <li>Most tools process input locally, according to the site. Network tools make requests after you start them and explain what they send; don't assume every page is offline.</li> <li>Generated models need review: sample data cannot describe a complete validation contract, and dataclass type hints do not validate incoming values.</li> </ul> <p><strong>Extras</strong></p> <p>Calvin:</p> <ul> <li><a href="https://github.com/hedgerow-dev/coldframe?featured_on=pythonbytes">Coldframe</a>: a stdlib-only honeypot that poses as a supermarket fridge controller** Michael:</li> <li><a href="https://pyfound.blogspot.com/2026/10/announcing-new-security-developer-2026.html?featured_on=pythonbytes">Announcing new Security Developer at the PSF</a> <ul> <li>Brett Smith is joining the PSF as a Security Developer, a new role funded by Anthropic. He'll work with Seth Larson and Mike Fiedler on vulnerability response and PyPI malware removal, with more time for proactive work as the team's workload grows. <ul> <li>More paid security capacity for the Python and PyPI infrastructure we depend on every day.</li> <li>The PSF reports 3-4x the volume of reports and CVEs for Python and pip compared with a year ago.</li> <li>Malware published and removed from PyPI is up 60% year over year. That's a workload measure, not a claim that 60% more packages are malicious.</li> <li>Planned work includes scaling PyPI's autonomous quarantine program and streamlining how the Python Security Response Team handles reports.</li> <li>Smith brings software development and Linux administration experience, including work at W3C and Software Freedom Conservancy.</li> </ul></li> </ul></li> </ul> <p><strong>Joke: <a href="https://x.com/PR0GRAMMERHUM0R/status/2016768305395040692?featured_on=pythonbytes">No papa</a></strong></p>

November 10, 2026 07:04 PM UTC

eGenix.com

PyDDF Python Herbst Sprint 2026

The following text is in German, since we're announcing a Python sprint in Düsseldorf, Germany.

Ankündigung

Python Meeting Herbst Sprint 2026 in
Düsseldorf

Samstag, 14.11.2026, 10:00-18:00 Uhr
Sonntag, 15.11.2026. 10:00-18:00 Uhr

Atos Information Technology GmbH, Am Seestern 1, 40547 Düsseldorf

Informationen

• Sprint Ort in Google Maps
  

• Exo-dus – Wenn Kollegen gehen, bleibt die Rechenleistung
• Schattenparker – Wie meine Vibe-Coded-App aus der Shadow AI ins Licht der Compliance fuhr bzw hoffentlich bald fährt
• pyinfra – Umstellung von Ansible auf pyinfra
• remarkable (e-ink tablet) in andere Infrastruktur wie Obsidian / etc integrieren
• Entenhausen – ein S3 System auf DuckDB/DuckLake Basis
  
• Weitere Themen können folgen
  

Anmeldung, Kosten und weitere Infos

Alles weitere und die Anmeldung findet Ihr auf unserer Blog Seite:

• Python Herbst Sprint & Hackathon in Düsseldorf
  

• PyDDF Telegram Gruppe

Über das Python Meeting Düsseldorf

Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python-Begeisterte aus der Region wendet.

Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf.

Marc-André Lemburg, eGenix.com

November 10, 2026 09:00 AM UTC

The Python Coding Stack

Generalist vs Specialist in the After AI Era

I always struggled to identify whether I was a specialist or a generalist.

During my studies, I narrowed my focus at every step. I studied maths, physics, and chemistry at A level, then maths and physics during my undergrad, and finally physics during my PhD. And the PhD wasn't in all of physics. It was in optics, and specifically in a narrow field within optics. I then spent over a decade working in this specialised field.

That settles it, then.

But was I a true specialist? I got quite good at the branch of optics I needed for my work, but others around me were better. I got reasonably proficient with coding to model physical processes and analyse data, but I had plenty of gaps in my programming knowledge. I quickly learnt how to become an effective lecturer and communicate with students as I designed and delivered undergraduate courses. Being an expert in a subject doesn't automatically make you good at teaching it.

When I left academia and started teaching Python, I wasn't an expert programmer and I was relatively new to Python. Sure, I got a lot better relatively quickly. But my strength was in teaching and communication, not in my programming expertise.

To run my business, I had to know or learn a bit of lots of stuff: accounting, marketing, sales, customer service, and managing staff. But I lacked a lot in all of these areas, too.

I'm a scientist by training and a technical educator by profession. But I enjoy languages, history, geography, sports, economics. The list goes on. I like to know stuff about any topic.

Perhaps I am a generalist and I've always been, even though I thought I was a specialist. Maybe it's time to embrace my generalist nature.

In the last few months, I've found a good reason to do just that. AI is helping me do more with knowledge and skills I already had, in several different fields. Things I knew something about, but not enough to carry through on my own, are becoming things I can actually do.

And I'm loving it.

Subscribe now

Generalist vs Specialist in the Before AI Era

The debate is not new. We need people who know a lot about a narrow field, and we need people who know enough about several fields to connect them and put that knowledge to use.

Knowledge in fields such as science and medicine has expanded so much that it's impossible to master a field without dedicating large amounts of time and effort to it. There's good reason for specialisation.

But depth takes time. And time spent going deeper into one field is time you can't spend learning about another.

That's not a criticism. It's a constraint.

A generalist makes a different trade-off. They have a broader base, but they run into the limits of their knowledge in more places. Knowing something about marketing isn't the same as being able to put together an effective marketing campaign. Understanding programming doesn't mean you're equally comfortable building every kind of software.

You can learn more, of course. Or work with someone who has the expertise you lack. But both require resources, whether that's time, money, or access to the right people.

So you can have interests, ideas, and useful knowledge that you never quite manage to turn into anything.

AI is changing that.

Generalists and Specialists During the Transition

To get from Before AI to After AI, we need to go through a transition. A period when things change. And boy oh boy we're in turbulent, changing, uncertain, unstable times right now.

When the situation changes and what worked yesterday no longer works today, it helps to have more than one way forward. A generalist has a large toolbox and more possibilities for using the tools differently. They're typically also more used to making connections across different fields.

A specialist often sees things through the lens of their speciality, and their skillset is optimised to go deeper and further within their speciality. Change can be challenging.

That doesn't mean specialists can't adapt. And being interested in lots of things doesn't automatically make you good at dealing with change.

But having several useful starting points gives you options. AI makes some of those options easier to act on.

That's the part I find most exciting. Not just having somewhere else to turn when one kind of work changes, but being able to do more in areas I already care about.

Generalist vs Specialist in the After AI Era

Before we go further, let's tear down a simplification I've been using in this post so far.

No one is really just a specialist or a generalist.

A pure specialist could be represented by this:

Don't worry about the maths-y bits there if you're not familiar with delta functions. An "ideal" specialist, or a 100% specialist, has expertise in one narrow topic and no knowledge of anything else. A generalist could look like this:

The 100% generalist has a constant amount of knowledge about everything.

You can tell I'm a physicist by nature. Physicists love to start from idealised assumptions. You've probably heard the “let's assume we have a spherical cow” joke.

A person who knows everything about one narrow topic and absolutely nothing about anything else would be a useful starting point for a model. But not a person you'll ever meet.

And a generalist doesn't know exactly the same amount about every topic, either.

There aren't any 100% specialists or 100% generalists.

Here's a typical sketch comparing specialists and generalists. The bars represent different skill areas, and their heights represent depth of expertise:

But there's an infinite number of variants. Some people have two or more high bars among the low ones. Others have a speciality but a good generalist level across many other skills. And not all generalists are the same. Some can have high bars across the board, even if not a single bar is at specialist level.

You don't have to choose between depth and breadth as though you can only have one. You can have both, in different proportions and in different fields.

So, rather than asking which of two boxes a person belongs in, we can look at the shape of their knowledge and skills. The labels are shorthand. The profile is more interesting.

AI helps fill gaps where you're not an expert. But I think the best use of AI is when a person and an AI agent collaborate (and I'm writing about this in a future post, so stay tuned), rather than when the agent just gets an instruction and works without input from the human.

If you know something about a topic, you can bring more to that collaboration. You have a better idea of what you want, you can ask more useful questions, and you have some basis for questioning the answers.

Let me give you a personal example that has nothing to do with programming.

From coaching theory to a training programme

I'm a newly qualified athletics (track and field) coach. I have plenty of theoretical knowledge, because I've been interested in this field for a long time, but limited coaching experience.

I worked with my AI agent to create a full training programme for myself. I didn't ask it to choose everything for me. I knew the philosophy I wanted to follow: build speed first, with a focus on sprinting form (running technique), then extend to speed endurance.

The focus on form mattered to me. I wanted to prioritise maintaining it, even if that meant delaying speed endurance. I also had views on how to organise the training programme across various phases.

AI helped me translate that direction into individual sessions, including the details of how the different exercises should fit together. It helped me plan the progression across the season and balance the different training demands.

I brought knowledge and priorities. AI helped me work through the detail needed to turn them into a programme.

Does that make me an experienced specialist coach? No. Creating a programme isn't the same as gaining years of experience coaching athletes, observing how they respond, and adjusting what you do.

But I could put more of what I knew into practice than I could have done alone. And I learnt more about the details of season and session planning along the way.

That's the opportunity I'm talking about.

More than one place to start

The same opportunity exists in other fields where I have some knowledge but lack the skills or experience to do everything I'd like to do. But it's not limited just to those fields. AI also helps me take some of my strongest skills further.

In some areas, it helps me bridge gaps in expertise. In others, it helps me carry through ideas I wouldn't otherwise have the time or resources to pursue.

I'll share more about some of the work my AI agent and I have been doing recently, spanning many of my strong and not-so-strong skills, in the bonus section at the end of this post.

Sometimes, several of those skills come together in one project. But they don't have to. My coaching knowledge is useful when I'm planning athletics training, whether or not I use my programming or writing skills alongside it.

A broad base gives me several places where I can work with AI to take things further. Here's a sketch of what that could look like:

The lighter parts represent what working with AI might add to what a person can do. They're not extra years of experience magically appearing.

Nor should we read the drawing as a promise that AI adds the same amount in every field or for every person. It's a simple illustration of the opportunity. The specialist can benefit, too. The generalist has several areas with a useful foundation to build on.

And the starting point matters. Knowing nothing about a subject doesn't stop you from using AI. It can help you learn. But that's different from using knowledge you already have to direct the work and question what comes back.

Nor does knowing a little guarantee you'll spot every mistake. AI can be wrong, and so can I. The need to check the work doesn't disappear because we produced something impressive together.

My first post in the Agent & Co. section pokes fun at this idea:

Read “Thoroughly Checked”

Still, the gap between “I know something about this” and “I can do something useful with it” is becoming easier to cross.

Subscribe now

So, Is This the Age of the Generalist?

AAI, After AI, is the Age of the Generalist. I think so. But not because specialists are doomed.

Specialists can use AI to go further in their own fields, and to explore others. Deep knowledge still matters. My years in physics didn't become worthless when I started teaching Python, and they haven't become worthless now that I work with AI. They help shape how I think and how I approach problem-solving.

What excites me is that breadth can now take me further, too.

Before, I could look at some of my interests and see mostly what I lacked. I knew a bit, but there was a lot more I'd need to learn before I could do what I wanted.

There still is plenty to learn. But I don't always have to fill every gap myself before I can get started.

A generalist with strong skills across several fields has several opportunities to do more with those skills. In some tasks, working with AI may bring the result closer to what previously required specialist help. That doesn't mean the person has become a specialist in all those fields.

But it does change what's possible.

And AI can also help the generalist with bespoke learning to push those skills further.

Perhaps being interested in lots of things wasn't such a distraction after all.

Bonus: How I'm Enjoying Being a Generalist in This AAI Era

I always liked the idea of being a specialist. That's probably what led me to scientific research in my first career. But over the years, I gradually realised I like lots of things and I'm not too bad at several of them.

I didn't realise until recently that I'm more of a generalist than a specialist.

That's a shame, because I could have capitalised more on my strengths as a generalist, perhaps. But never mind, because in the last few months, I got it.

Here are a few places where I've been putting that breadth to use. Not just what I asked AI to do, but what I brought to the work and how we worked through it together.

These aren't examples of handing over a task and coming back when it's finished. The conversation is part of the work.

The bonus section is exclusively available to premium subscribers.

stephengruppetta.com

November 10, 2026 08:05 AM UTC

LernerPython blog, from Reuven Lerner

Pandas options: Setting display precision with pd.set_option

There are tons of config options in Python Pandas. Set them with pd.set_option:

pd.set_option('display.precision', 2)

Now floats in a series or data frame will have 2 digits after the decimal.

Retrieve the value with “get_option”:

pd.get_option('display.precision')

The post Pandas options: Setting display precision with pd.set_option appeared first on LernerPython.

November 10, 2026 07:00 AM UTC

Python Insider

Python 3.16.0 alpha 2 is now available

The second developer preview of Python 3.16 is ready for testing.

November 10, 2026 12:00 AM UTC

Graham Dumpleton

Muru: HTTPS in development

Everything I have shown of Muru so far has been served over plain HTTP on port 8080. That is how almost everyone runs a Python web application while developing it, and most of the time it is fine. The application in production is going to be behind HTTPS though, and there are some things which behave differently when it is, which you only find out about once it is deployed.

The usual way of testing those things locally is to put a proxy such as nginx or Caddy in front of the development server, generate a certificate for it, and then configure the application to trust the headers the proxy adds to say the original request was HTTPS. It is enough of a hassle that most people never bother. With a Muru development run it is one option.

Turning it on

Given a WSGI application in shop/wsgi.py, add --https when starting it:

muructl run shop.wsgi:application --https

muru.host[61235]: preloaded shop.wsgi:application in 1 ms
muru.host[61235]: shop.wsgi:application is served as WSGI
muru.host[61235]: worker 0 started as pid 61241
muru.host[61235]: worker 0 (pid 61241) is ready
muru: serving shop.wsgi:application on http://127.0.0.1:8080 and https://127.0.0.1:8443 with 1 worker of 4 threads
...
muru: certificate for localhost, 127.0.0.1, ::1 issued by the mkcert authority in /Users/graham/Library/Application Support/mkcert (created, /Users/graham/Library/Caches/muru/certs/dev.pem)

The run now listens for HTTPS on port 8443 as well as HTTP on 8080, and it has created a certificate to use.

How that certificate was made depends on what you have installed. If you use mkcert and have run mkcert -install at some point, there is a local certificate authority on your machine which your browsers already trust. Muru finds it and issues the certificate from it, so the browser accepts the certificate without any warnings. If mkcert isn't installed, Muru falls back to creating a self-signed certificate, and prints out how to trust it, or how to install mkcert instead.

Either way, the certificate is kept in your user cache directory rather than in the project, so there is no risk of a private key ending up committed. It is reused on later runs, which the same line says by showing reused rather than created, and is only replaced when it nears expiry or is asked to cover a name it doesn't already. If you already have a certificate of your own you would prefer to use, --tls-cert and --tls-key will take that instead.

Why bother

To see what difference HTTPS makes to the application, here is a WSGI application which reports back the URL scheme and protocol of the request, and the host it was sent to:

def application(environ, start_response):
    body = f"{environ['wsgi.url_scheme']} {environ['SERVER_PROTOCOL']} {environ['HTTP_HOST']}\n".encode()
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [body]

Making a request to each of the two ports:

curl http://127.0.0.1:8080/
curl https://127.0.0.1:8443/

the result being:

http HTTP/1.1 127.0.0.1:8080
https HTTP/2.0 127.0.0.1:8443

Over HTTPS, the application sees wsgi.url_scheme set to https, and the request arrived as HTTP/2. Neither of these can be tested over plain HTTP. Browsers only use HTTP/2 over TLS, so if you want to know how your application behaves when a page pulls in dozens of assets over the one connection, this is the only way to find out locally. Any code which uses wsgi.url_scheme to construct absolute URLs, for redirects or links in emails, will get them wrong unless the request really did arrive over HTTPS. And cookies marked as Secure, which any session cookie in production should be, aren't sent back by the browser over plain HTTP at all. An application which sets SESSION_COOKIE_SECURE in its production settings can behave quite differently to the one you have been developing against.

What makes this simple to provide is how Muru is put together, which I went through in how it fits together. TLS and HTTP/2 are handled by the edge, which is written in Rust and runs no Python code. By the time a request reaches the worker it has already been decrypted and parsed, and the application gets an ordinary WSGI environ with the scheme already set correctly. There is no proxy to configure, and no need to tell the application to trust an X-Forwarded-Proto header. It is also the same code which terminates TLS when running under the daemon in production, rather than something which only exists for development.

Names other than localhost

Using localhost for everything gets awkward once you work on more than one project, since cookies are tied to the host name and not the port. Log in to one project on localhost and its session cookie gets sent to every other project you run on localhost too.

The --host option adds names to the certificate:

muructl run shop.wsgi:application --https --host shop.localhost

muru: certificate for localhost, 127.0.0.1, ::1, shop.localhost issued by the mkcert authority in /Users/graham/Library/Application Support/mkcert (created, /Users/graham/Library/Caches/muru/certs/dev.pem)

Modern browsers such as Chrome and Firefox, as well as curl, resolve any name ending in .localhost to the loopback address, so there is no need to edit /etc/hosts to use them. The project can then be reached at https://shop.localhost:8443/, with its cookies kept apart from those of any other project, and with a certificate the browser trusts for that name.

HTTPS only

With --https, both HTTP and HTTPS are served, which can let a problem slip by if some page or redirect is quietly sending you back to plain HTTP. If you want to be sure everything is going over HTTPS, as it will be in production, use --https-only instead:

muructl run shop.wsgi:application --https-only --host shop.localhost

muru: serving shop.wsgi:application on https://127.0.0.1:8443 (http://127.0.0.1:8080 redirects to it) with 1 worker of 4 threads

Port 8080 is still listened on, but now answers every request with a redirect to the same path on the HTTPS port:

curl -i 'http://shop.localhost:8080/cart?item=42'

HTTP/1.1 307 Temporary Redirect
location: https://shop.localhost:8443/cart?item=42
server: muru
...

The redirect is a 307 rather than a 301 for two reasons. A 307 keeps the method, so a form posted to the HTTP port is posted again to the HTTPS port rather than turning into a GET. And browsers don't cache a 307, so if you later go back to running without HTTPS on the same port, the browser won't keep sending you off to a port where nothing is listening any more.

So you don't have to remember to type it each time, the same can be turned on for the project in pyproject.toml:

[tool.muru.run]
https_only = true
hosts = ["shop.localhost"]

These settings only apply to muructl run. They have no effect on how the application is served by the daemon in production, where certificates are a different matter altogether.

What's next

None of this is especially complicated, but it is the sort of thing which only gets tested if it is easy, and getting it for free is a consequence of the edge doing all the work of facing clients. Next up I will look at what else Muru lets a single application do, which is to mix WSGI and ASGI, including handing a request from one to the other so an existing WSGI application can take on websockets.

November 10, 2026 12:00 AM UTC

November 09, 2026

LernerPython blog, from Reuven Lerner

Pandas 3.1 CSV loading: Multi-threaded, but PyArrow is still faster

Reading a CSV into a Python Pandas data frame?

Before version 3.1, loading CSV was single-threaded and slow. My test file took 1.03s.

In 3.1, it’s now multi-threaded and much faster, taking 252 ms.

However, it’s still far slower than PyArrow, which took 25.5 ms.

The post Pandas 3.1 CSV loading: Multi-threaded, but PyArrow is still faster appeared first on LernerPython.

November 09, 2026 07:00 AM UTC

Graham Dumpleton

Muru: import once, fork many

Every time Muru has started an application in this series, starting with a first look at Muru, one of the first lines of output has been something like:

muru.host[88644]: preloaded app:application in 1 ms

In how it fits together I described the host as the Python process which imports the application once and forks the workers from it, but skipped over what that involves. It is time to come back to it, because how an application gets loaded into worker processes is one of those things that matters a lot for Python, and is easy to get wrong.

Why import once

To see what is going on, here is an application which records the process ID at the point the module was imported, and then reports that along with the process ID handling each request:

import os

IMPORTED_IN = os.getpid()

def application(environ, start_response):
    body = f"imported in {IMPORTED_IN}, serving from {os.getpid()}\n".encode()
    start_response("200 OK", [("Content-Type", "text/plain"), ("Content-Length", str(len(body)))])
    return [body]

Running it with two worker processes:

muructl run app:application --processes 2

muru.host[88644]: preloaded app:application in 1 ms
muru.host[88644]: app:application is served as WSGI
muru.host[88644]: worker 0 started as pid 88645
muru.host[88644]: worker 1 started as pid 88646
muru.host[88644]: worker 0 (pid 88645) is ready
muru.host[88644]: worker 1 (pid 88646) is ready
muru: serving app:application on http://127.0.0.1:8080 with 2 workers of 4 threads

and making a few requests, the result being:

imported in 88644, serving from 88646
imported in 88644, serving from 88645
imported in 88644, serving from 88646
imported in 88644, serving from 88645

The application was imported once, in the host process, and both workers were then forked from the host. Neither worker ever imported the application itself. Each started out as a copy of the host with every module already loaded.

For a toy application this makes no difference. For a large Django application, where importing everything can take several seconds and a lot of memory, it makes a big difference. Forked workers share the memory pages holding those modules with the host, and with each other, until something writes to them. Starting a new worker, whether to replace one that crashed, to recycle one that has grown too large, or to add capacity, costs a fork rather than a full import. Muru also takes care to freeze the garbage collector's view of the preloaded objects before forking, so that collections in the workers don't go touching, and therefore copying, all those shared pages.

This was one of the things I always wanted for mod_wsgi daemon mode but couldn't have. The daemon processes in mod_wsgi are forked by the Apache parent process, and that isn't a place where it is safe to load application code. Each daemon process therefore has to import the application itself, and restarting one means paying for that import again.

What doesn't survive a fork

Preloading isn't a new idea, and other Python WSGI servers have had an option for it for a long time. The reason it is usually an option, rather than the default, is that it can break applications in subtle ways.

Forking copies a process's memory, but some things live outside of memory. A socket is a kernel object, so after a fork both the parent and child hold the same connection, each thinking it is theirs alone. A thread exists only in the process which started it, so a background thread started during import simply doesn't exist in the workers, even though all its data structures do. A child process has one parent, so the workers can't manage a helper process the host launched.

So if importing the application opens a database connection, starts a thread to flush metrics, or launches a helper process, preloading produces workers that are quietly broken. Worse, these problems tend to be intermittent, and show up in production under load rather than on the first request you make in development.

The import check

What Muru does differently is that it doesn't just trust you to know whether your application is safe to preload. When the host imports the application, it records any sockets, pipes, threads and child processes that were left behind by the import.

To show this, here is a module standing in for a database client which connects when it is imported, as some do:

import socket

# Pretend this is a database client connecting when the module is imported.
connection = socket.create_connection(("127.0.0.1", 9340))

The WSGI application in shop/wsgi.py imports it with from shop import db. Running it gives:

muru.host[88997]: preloaded shop.wsgi:application in 2 ms
muru.host[88997]: shop.wsgi:application is served as WSGI
muru.host[88997]: preload: the import of shop.db left a connected socket (fd 3, to 127.0.0.1:9340); close it in a pre_fork hook and reopen it in post_fork, or open it lazily on first use
muru.host[88997]: preload: falling back to per-worker imports, which costs slower worker replacement and no shared memory; set preload to false to silence this
muru.host[88997]: worker 0 started as pid 89031
muru.host[88997]: worker 1 started as pid 89032
muru.worker[0]: worker 0 imported shop.wsgi:application in 2 ms
muru.worker[0]: worker 0 serves shop.wsgi:application as WSGI
muru.host[88997]: worker 0 (pid 89031) is ready
muru.worker[1]: worker 1 imported shop.wsgi:application in 2 ms
muru.worker[1]: worker 1 serves shop.wsgi:application as WSGI
muru.host[88997]: worker 1 (pid 89032) is ready

Muru found the connected socket, said where it goes, and decided not to fork from the host. Instead it fell back to having each worker import the application itself, which is always safe. The application still works, it just loses the benefits of preloading, and the message tells you what you would need to change to get them back.

Threads are caught the same way. Here is a module which starts a background thread when imported, to periodically report how many requests have been handled:

import threading
import time

counts = {"requests": 0}

def flush():
    while True:
        time.sleep(5)
        print(f"flushed {counts['requests']} requests", flush=True)

threading.Thread(target=flush, daemon=True).start()

Under a naive preload this thread would run in the host process, which never handles a request, and so would forever report zero. The workers doing the actual work would have no thread at all. Muru spots it:

muru.host[89680]: preload: the import of stats left a thread running; start it from a post_fork hook instead
muru.host[89680]: preload: falling back to per-worker imports, which costs slower worker replacement and no shared memory; set preload to false to silence this

Doing things at the right time

Most things which are wrong to do at import time are perfectly fine to do a moment later, once the worker exists. For that Muru supports a hooks module, named with the --hooks option or with hooks in the [tool.muru] table of pyproject.toml. The module can define functions to be called at different points in the life of the host and its workers.

The one you will need most is post_fork, which is called in each worker immediately after it is forked. For the metrics thread, the module is changed to provide a function to start the thread rather than starting it on import:

def start():
    threading.Thread(target=flush, daemon=True).start()

and a hooks.py calls it:

import stats

def post_fork(context):
    print(f"worker {context.worker_id} starting the stats thread", flush=True)
    stats.start()

Each hook is passed a context object which says where it is running, including the index of the worker. Running with the hooks module and two processes:

muructl run app:application --hooks hooks --processes 2

muru.host[90139]: preloaded app:application in 1 ms
muru.host[90139]: app:application is served as WSGI
muru.host[90139]: worker 0 started as pid 90140
muru.host[90139]: worker 1 started as pid 90141
muru.worker[0].stdout: worker 0 starting the stats thread
muru.host[90139]: worker 0 (pid 90140) is ready
muru.worker[1].stdout: worker 1 starting the stats thread
muru.host[90139]: worker 1 (pid 90141) is ready
...
muru.worker[0].stdout: flushed 3 requests
muru.worker[1].stdout: flushed 3 requests

The import check passes, the host preloads, and each worker gets its own thread. Note that each worker reports its own count. The counts dictionary started out shared from the host, but the moment a worker writes to it, it has its own copy. Anything that needs to be shared across workers has to live somewhere else, such as a database or a metrics system.

The database connection is dealt with in the same way. The module provides a connect() function rather than connecting on import, and a post_fork hook calls it, so that each worker opens its own connection. Alternatively, the connection could be opened lazily on first use, which many database libraries already do.

Beyond post_fork, there is on_load for things to run once in the host after the import, such as warming caches that are safe to share, and pre_fork which runs in the host before the first fork. In the workers, on_ready runs just before a worker starts taking requests, on_drain when it has been told to stop, and on_exit after its last request. When the fallback to per-worker imports is in effect, on_load and pre_fork are run in each worker before post_fork, so the same hooks module works either way.

Choosing how strict to be

What happens when the import check fails is controlled by the preload setting, which can be given as an option or set in [tool.muru]. The default is auto, which is what you have seen so far, preloading when the check passes and falling back with a warning when it doesn't.

Setting it to true makes a failed check an error. Going back to the version of the shop which connected on import:

muructl run shop.wsgi:application --preload true

muru.host[89241]: preload: the import of shop.db left a connected socket (fd 3, to 127.0.0.1:9340); close it in a pre_fork hook and reopen it in post_fork, or open it lazily on first use
muru.host[89241]: preload is true, so the import check failing stops the release
Error: the host exited with exit status: 1 before its workers were ready

This is what you want in production. Once an application has been made safe to preload, you don't want a change by someone that breaks that to quietly fall back to per-worker imports and go unnoticed until someone wonders why memory use went up.

Setting it to false skips preloading and the check altogether, with each worker importing the application:

muru.host[89381]: worker 0 started as pid 89382
muru.host[89381]: worker 1 started as pid 89384
muru.worker[1]: worker 1 imported shop.wsgi:application in 5 ms
muru.worker[1]: worker 1 serves shop.wsgi:application as WSGI
muru.worker[0]: worker 0 imported shop.wsgi:application in 13 ms
muru.worker[0]: worker 0 serves shop.wsgi:application as WSGI

That is for applications which can't be made fork safe, or where you have decided it isn't worth the effort.

Processes and threads

Preloading is about how workers come to exist. The other half of the process model is what runs inside each worker. By default a WSGI worker has four threads, each of which handles one request at a time, and you choose how many worker processes to have and how many threads each gets.

For most web applications, which spend their time waiting on databases and other services, threads work well. On a regular Python interpreter though, only one thread can be running Python code at a time because of the global interpreter lock (GIL). Waiting on a database releases the GIL so other threads can get on with things, but code that is busy computing holds it.

To show the effect, here is an application which does some pure Python computation for each request:

import sys

def work(n):
    total = 0
    for i in range(n):
        total += i * i
    return total

def application(environ, start_response):
    work(10_000_000)
    gil = "enabled" if sys._is_gil_enabled() else "disabled"
    body = f"done, GIL {gil}\n".encode()
    start_response("200 OK", [("Content-Type", "text/plain"), ("Content-Length", str(len(body)))])
    return [body]

Running this on a regular build of Python 3.15, with the default of one worker and four threads, and firing off four requests at the same time, all four had finished after about 1.3 seconds. The access log shows each request taking most of that:

200 GET / 18B 1.19s queue=0.0ms app=1.19s worker=0 id=01M6JWV35JYRMR8K10KBMQ1H5N
200 GET / 18B 1.21s queue=0.0ms app=1.21s worker=0 id=01M6JWV35JYRMR8K10KBMQ1H5J
200 GET / 18B 1.27s queue=0.1ms app=1.27s worker=0 id=01M6JWV35JYRMR8K10KBMQ1H5K
200 GET / 18B 1.28s queue=0.1ms app=1.28s worker=0 id=01M6JWV35JYRMR8K10KBMQ1H5M

There were enough threads, so nothing waited in the queue, but the four threads had to take turns holding the GIL, and so each request took about four times as long as it would on its own. This is a good illustration of why the queue and app times aren't the whole story. Here the time is being charged to the application, yet the code isn't any slower, it is waiting for the GIL. Muru does keep track of time spent waiting for the GIL separately, which I will come back to in a later post.

The traditional answer for CPU bound work is to use processes rather than threads. Running the same thing with --processes 4 --threads 1, so each request gets a process of its own, all four requests finished in about 0.5 seconds.

Free-threaded Python

Python 3.14 brought official support for free-threaded builds of the interpreter, where the GIL is removed and threads can run Python code in parallel. The uv tool knows them by names like 3.15t, so a project only needs a .python-version file containing 3.15t for uv venv to create its virtual environment from a free-threaded build of Python 3.15.

Muru supports these builds from the start. Its runtime library was written for them rather than adapted later, and it fetches the right build of the library for whichever kind of interpreter the project uses. Running the same application on the free-threaded build, again with one worker and four threads, all four requests were done in about 0.36 seconds:

200 GET / 19B 331ms queue=0.1ms app=331ms worker=0 id=01M6JWV8A36WZSBQEFEM4EZ7W5
200 GET / 19B 331ms queue=0.1ms app=331ms worker=0 id=01M6JWV8A36WZSBQEFEM4EZ7W7
200 GET / 19B 334ms queue=0.1ms app=334ms worker=0 id=01M6JWV8A36WZSBQEFEM4EZ7W6
200 GET / 19B 336ms queue=0.0ms app=336ms worker=0 id=01M6JWV8A36WZSBQEFEM4EZ7W4

The four threads ran in parallel within the one process, each request taking only as long as it would have on its own. These are rough numbers from my laptop rather than a proper benchmark, but the difference is clear enough.

This changes the trade-off between processes and threads. Processes still give you isolation, so that a crash, a memory leak or a stuck request takes down one worker rather than all of them. But you no longer need lots of processes just to make use of every CPU core, and fewer processes with more threads means sharing caches and connection pools rather than duplicating them.

One thing to watch for is that importing an extension module which hasn't declared support for free-threading causes the interpreter to turn the GIL back on. That is why the example reports whether the GIL is enabled. Your application can end up running on a free-threaded build with the GIL on and not realise it, which is something Muru keeps an eye on as well.

What's next

Having an application imported once, with the check to make sure that is safe, and with workers forked from it, is the foundation for much of what Muru does later, such as starting new releases quickly and scaling the number of workers up and down. The next post will be a shorter one, on something a development run does which most development servers don't, which is serve HTTPS with a certificate your browser will trust.

November 09, 2026 12:00 AM UTC

Armin Ronacher

AI and Irontown

For our last movie night we watched Princess Mononoke. I remembered loving that movie when I was younger, but watching it as an adult made me once again realize how Miyazaki has a way of expressing diverging sides without picking one. The film shows people, animals and spirits with legitimate and largely incompatible needs, and each faction is further engaged in infighting.

A central arc of the story is Lady Eboshi’s Irontown, which gives people who were largely rejected by society work, protection and dignity. She gives shelter and work to those who have leprosy and to former sex workers. The mine and the weapon factory provide them with a livelihood, but they also destroy the forest around them. The forest’s inhabitants, in particular the boars, response choose war against the humans to defend themselves. Eboshi is not portrayed as a villain. She does real good for the people of Irontown, but also is an arms manufacturer who is destroying the forest around it.

The film anchors around a character called Ashitaka, who has been condemned to death by an unfortunate encounter with a god that was turned into a demon after being fatally wounded by one of Eboshi’s guns. He doesn’t take sides, but he wants to understand and find the truth. Another character is San (whom they call Mononoke), a girl who was raised by wolves and, despite being human, considers herself part of the forest.

This movie’s backdrop is human progress, and it reminded me so much of the challenges we are facing today with AI. AI is useful, empowering and consequential beyond the people who benefit from it. It’s similar to the iron in the story. Irontown gives the outcast purpose and power. There are people today who see themselves in an environment just like that: people who were never let in before and who can now build things that were out of reach. They are wielding AI to their benefit. But there are negative externalities. In Mononoke it’s not just the spirits and animals in the forest who bear the consequences, it’s also other people. Irontown produces guns, and its people are engaged in fights with other humans. There is conflict and power seeking everywhere.

Another, unseen character is the power-hungry emperor, who wants the head of the forest spirit, a god, because he believes it will make him immortal. Eboshi is willing to support that cause, not because she thinks it’s right, but because she is at war with the forest’s gods who are preventing her expansion, so they have a common interest. But there really is no need for anyone to kill the forest spirit, in the same way there is no need for any one country to “win at AI”. What drives it is the fear of what happens if someone else wins first. And the emperor is willing to have a god killed for it. The people who help him with that desire, Eboshi included, don’t understand the chaos it will unleash.

There it also shows the dangers of seeking overwhelming power without comprehension of how to control its consequences. The pursuit of power itself becomes destructive. In the end, the gods die, and what grows back is not the old forest anymore. Irontown is destroyed. Eboshi wants to rebuild it, and to build a better town this time. Ashitaka stays with the people of Irontown after his curse is lifted by the dying forest spirit. San cannot forgive the humans and stays with the wolves. In the end, nothing is truly resolved, but there is a truce that came at great cost.

Ashitaka’s role in the movie is one of a neutral actor. He was wronged both by a spirit turned demon in the forest, who put a curse on him, and, indirectly, by Eboshi, who caused the spirit to turn evil. Yet he harbors no ill will towards either side. He sets out to “see with eyes unclouded by hate,” and in the course of the movie he is seen as an unaligned actor. He helps people on both sides, and stands in their way to prevent further conflict. He doesn’t feel like a pacifist, but he’s the closest to one there is. He’s a truth seeker in a changing world. Because he engages he’s seen as suspicious by both sides, and ends up shot by the people of Irontown when he tries to rescue San.

I use AI every day and find it genuinely empowering. But I also recognize the pain it causes some, and the moral problems it poses. It’s genuine progress for humanity, but it comes at significant cost. And many of those costs will be carried by people who aren’t going to benefit from it.

So it’s no surprise that AI feels like a threat to many today. Communities currently strongly align themselves with one side and fight against the other. The forest spirit did not have to die and the boars did not have to die in a senseless war.

The question is not whether there will be iron, but what kind of Irontown we build with it. I want our conversation about AI to make room for nuance and genuine understanding, for us to look at it and each other with eyes unclouded by hate. We should take advantage of its real benefits, but also recognize the damage it can do, and the choices we still have about what we build and what we’re willing to sacrifice for it. I like to think there is a path for people with differing opinions can find a way to live alongside before losing something we can’t get back.

November 09, 2026 12:00 AM UTC

November 08, 2026

LernerPython blog, from Reuven Lerner

New course: Introduction to agentic coding with Claude Code

I’m excited to announce my newest course, Introduction to agentic coding with Claude Code (https://lernerpython.com/courses/intro-agentic-coding-claude-code/).

If you want to go beyond giving simple prompts or using Claude Code in trivial ways, this is the course for you. I’ve distilled what I’ve learned from using Claude Code every day for about a year to run my business, create and improve open-source projects, and create new applications, and I’ve put them into this four-hour course.

Of course, it includes hands-on exercises. And it includes advice on using “uv” (for Python projects) and Git (for all projects).

The course is included in a LernerPython membership, which has a two-week free trial. Give it a shot at https://LernerPython.com !

The post New course: Introduction to agentic coding with Claude Code appeared first on LernerPython.

November 08, 2026 06:12 PM UTC

Pandas 3.1 filter: Selecting rows with a pd.col condition

Starting with Python Pandas 3.1, use “filter” to apply a condition to rows. So:

(
  df
  .loc[ pd.col('passenger_count') > 1 , 'trip_distance']
  .mean()
)

can now be (the much slower):

(
  df
  .filter(cond=pd.col('passenger_count') > 1)
  ['trip_distance']
  .mean()
)

The post Pandas 3.1 filter: Selecting rows with a pd.col condition appeared first on LernerPython.

November 08, 2026 07:00 AM UTC

Graham Dumpleton

Muru: how it fits together

In a first look at Muru and looking inside the workers I have been showing what Muru can do in development mode, without saying much about how it is put together. Words like host, worker, edge and release have kept turning up in the output, and I have mostly waved my hands at what they mean. Before going any further, it is time to explain what those parts are and how they fit together, since a lot of what makes Muru different follows from that design.

The processes in a development run

When you run muructl run with two worker processes and look at the process tree, this is what you find:

59435     1 muructl run app:application --processes 2
59440 59435 .venv/bin/python -m muru.host --app app:application --release r1 --processes 2 --threads 4 ...
59443 59440 .venv/bin/python -m muru.host --app app:application --release r1 --processes 2 --threads 4 ...
59444 59440 .venv/bin/python -m muru.host --app app:application --release r1 --processes 2 --threads 4 ...

The muructl process started a Python process running muru.host. That host then forked the two workers, which is why they show the same command line as the host. There is no Python in the muructl process at all. It is a Rust program, and in development mode it contains two of Muru's main parts, the edge and the controller.

The edge is the part which faces clients. It accepts connections, handles TLS, speaks HTTP/1.1 and HTTP/2, works out where each request should go, and passes it on to a worker. It also deals with the things that don't need Python at all, such as serving static files, compressing responses and producing error pages. Because it is written in Rust and never runs any Python code, a slow request in your application can't hold it up.

The host is a Python process which imports your application once and forks workers from it, which I will look at more closely in the next post. It never serves requests itself. Its job is to make workers, and to replace them if they die.

The workers are where your application runs. Each one is a Python interpreter with your code already loaded, plus a small layer written in Rust which talks to the edge and hands requests over to your code, either on a thread for WSGI or on the event loop for ASGI.

The controller decides what should be running. It starts and stops hosts, tells the edge where to send traffic, and keeps track of releases. Importantly, it is never on the path a request takes.

The path a request takes

Following a single request through the system shows how the work is divided up.

The edge reads the request and decides whether it is something it can answer itself, such as a static file or a redirect, or whether it needs to go to the application. If it does, the edge picks which release should handle it according to the traffic policy, then which worker within that release, choosing the one with the fewest requests already in flight. If every worker is busy, the request waits in a queue at the edge. That time spent waiting is the queue figure in the access log, which I made a point of back in a first look at Muru.

The request then crosses to the worker over what Muru calls the worker link, a connection over a Unix socket. What travels over it isn't HTTP. The edge has already parsed the request, so the method, path, headers and details about the connection are sent as small typed messages, followed by the request body as it arrives. Nothing gets turned back into HTTP text, only to be parsed all over again on the other side.

In the worker, your application sees exactly what it would under any other server. A WSGI application gets an ordinary environ dictionary, an ASGI application gets an ordinary scope. The response then flows back over the same link, with the worker gathering up small pieces of a response before sending them, so an application which yields a thousand tiny strings doesn't cause a thousand separate writes. The edge writes the response to the client at whatever pace the client can take it. If the client is slow, that slowness is passed back through the edge and the worker to the application, which waits with the interpreter released, so nothing ends up being buffered without limit along the way.

Request bodies are handled on demand in the same way. The edge only reads more of the body from the client as the application reads it, so an application which rejects a request without looking at the body doesn't cause a large upload to be read in for nothing. Where the application responds with a file, the edge sends it to the client straight from the file, rather than the contents being copied through the worker.

What the extra hop costs

Anyone who has looked at how mod_wsgi daemon mode works will recognise the general shape here. There, the Apache child processes accept the connection and pass the request over a Unix socket to the daemon processes where the Python application runs. Muru does much the same, with a separate front end proxying requests to Python processes. The obvious question is what that costs compared with a server where Python reads the HTTP request directly off the socket.

The answer is one more copy of each byte, and some tens of microseconds of work per request, when both processes are on the same machine. Next to the time a typical Python web application spends handling a request, that is lost in the noise, and in return you get the edge doing all the client-facing work in Rust.

The bigger payoff though isn't performance. It is that the edge and the Python processes can come and go independently of each other. The edge can keep accepting connections while the application behind it is replaced. Workers can be recycled or restarted without dropping connections at the edge. More than one version of an application can be running at the same time, each on a different version of Python if need be. None of that is possible if the process handling the client connection is also the one running your Python code.

A release is a unit you can swap

That brings things back to releases. A release is a fixed combination of application code, Python version and settings, together with the host and workers running it. Releases are never changed once started. If something needs to change, a new release is started.

That is what is happening when development mode reloads after you save a file. The controller doesn't restart anything. It starts a new release beside the current one, and only moves traffic across once the new release is ready.

If release r2 never becomes ready, because the code fails to import for example, the step that changes the routing never happens, and r1 keeps on serving as if nothing happened. That is what was going on in the first post when a broken import left the previous release serving.

Because moving traffic is just a change to the edge's routing, it doesn't have to be all or nothing either. The edge could just as easily send most requests to one release and a small share to another, or send particular clients to a preview of a new release. A reload in development mode is a straight switch, but the mechanism underneath is the same one used for blue/green deploys, canaries and rolling back.

The same parts under the daemon

In development mode, the edge and the controller both live inside the one muructl run process, and there is only ever one application. Running on a server under the daemon, the same parts are arranged a bit differently.

The controller now lives in a long running daemon process, along with the admin API that muructl commands and the dashboard talk to, and a store which keeps track of what should be running. The edge runs as a separate process of its own. There can be many applications, and several releases of each running side by side, here with a new release of an application on a newer version of Python running beside the old one.

Keeping the daemon off the request path has a useful consequence. If the daemon is restarted or upgraded, the edge carries on serving requests using the routing it last received, and the hosts keep running. Replacing the edge is also handled without refusing connections, as the new edge process takes over the listening sockets from the old one. Traffic keeps flowing while the parts managing it are swapped out.

This is the part I most wanted to get right, given my history. Under Apache, the parent process owns the lifecycle of every child process, including the mod_wsgi daemon processes. Their number is fixed by the configuration, and changing anything significant means a restart of Apache, with everything that comes with it. With Muru, the thing that decides what should run is separate from the thing serving traffic, and both are separate from the Python processes. Each can be changed without disturbing the others.

The same model also applies when running in a container with muructl serve. There the edge, the controller, the host and its workers all run as the container's main process, much as in development mode, since a container is meant to run a single application.

What's next

That covers how the parts fit together. The daemon, and deploying an application to a server with releases and traffic moving between them, will come once I have finished with development mode. Before that, in the next post I will go back to the host, and how Muru imports an application once and forks its workers from it, along with what it does to make sure that is safe.

November 08, 2026 12:00 AM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Brasil
  • Planet Python Poland
• Python Libraries
  • OLPC
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Michael Kennedy's Thoughts on Technology"
  • "Michiel's Blog"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Adrarsh Divakaran
  • Agendaless Consulting
  • Ahmed Bouchefra
  • Al-Ahmadgaid Asaad
  • Alec Munro
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Antonio Cuni
  • Anwesha Das
  • Ari Lamstein
  • Armin Ronacher
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Audrey M. Roy Greenfeld
  • Automating OSINT
  • Awesome Python Applications
  • Baiju Muthukadan
  • Bajusz Tamás
  • BeDjango
  • Ben Bass
  • Ben Cook
  • Ben Rousch
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • BioPython News
  • Bit of Cheese
  • Bob Belderbos
  • Bojan Mihelac
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Okken
  • Bruno Oliveira
  • Bruno Ponne / Coding The Past
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Düvel
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Casey Duncan
  • Catherine Devlin
  • Ching-Hwa Yu
  • Chris Hager
  • Chris Miles
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Heimes
  • Christian Ledermann
  • Christian Scholz
  • Christoph Schiessl
  • Christoph Zwerschke
  • CodeGrades
  • CodeSnipers
  • CodersLegacy
  • Coding Diet
  • Core Dispatch
  • Corey Gallon
  • Corey Goldberg
  • Cormoran Project
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dallas Fort Worth Pythoneers
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Data Community DC
  • Data School
  • DataWars.io
  • Dave Beazley
  • David Amos
  • David Caron
  • David Goodger
  • David Lindelof
  • David MacIver
  • David Malcolm
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • Declassed Art
  • Denis Kurov
  • Django Weblog
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Dougal Matthews
  • Duncan McGreggor
  • Ed Crewe
  • Edward K. Ream
  • Eli Bendersky
  • Eniram Ltd.
  • Eray Özkural (examachine)
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Evennia
  • Everyday Superpowers
  • Fabio Zadrozny
  • Filip Wasilewski
  • Filipe Saraiva
  • First Institute of Reliable Software
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • Gocept Weblog
  • Godson Gera
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Gustavo Narea
  • Gustavo Niemeyer
  • Gökhan Sever
  • Hernan Grecco
  • Hilary Mason
  • Holger Peters
  • HoloViz
  • Hugo van Kemenade
  • Humberto Rocha
  • Hynek Schlawack
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Inspired Python
  • Ionel Cristian Maries
  • IronPython-URLs
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Velichko
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • Janusworx
  • Jarrod Millman
  • Jean-Paul Calderone
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Burns
  • John Cook
  • John Jacobsen
  • John Ludhi/nbshare.io
  • Jon Parise
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan Street
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Julien Danjou
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kevin Renskers
  • Kodnito
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • LAAC Technology
  • Laurent Szyster
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • LernerPython blog, from Reuven Lerner
  • Lintel Technologies
  • Linux Stans
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Lucas Cimon
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Mariatta
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mats Kindahl
  • Matt Layman
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Matthew Wright
  • Mattias Brändström
  • Mauveweb
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Nelson
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mirek Długosz
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Naomi Ceder
  • Natan Zabkar
  • Ned Batchelder
  • Neil Schemenauer
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • Nsukami Patrick
  • Obey the Testing Goat
  • Ofosos
  • Omaha Python Users Group
  • Ondřej Čertík
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patrick Altman
  • Patrick Kennedy
  • Patrick Müller
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • Pedro Lima
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyATL Bytecode
  • PyBites
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyCon Ireland
  • PyPodcats
  • PyPy
  • PyTennessee
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Docs Editorial Board
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python GUIs
  • Python Insider
  • Python Koans
  • Python Morsels
  • Python People
  • Python Piedmont Triad User Group
  • Python Pool
  • Python Software Foundation
  • Python Sweetness
  • Python User Groups
  • Python for Beginners
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • Pythonicity
  • Pythonology
  • Python⇒Speed
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Rene Dudfield
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rishi Maker
  • Rob Galanakis
  • Robert Collins
  • Robert Zaremba
  • Robin Parmar
  • Robin Wilson
  • Rodrigo Araúj
  • Rodrigo Girão Serrão
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Sandipan Dey
  • Sandro Tosi
  • Sean McGrath
  • Sebastian Pölsterl
  • Sebastian Witowski
  • Selena Deckelmann
  • Senthil Kumaran
  • Seth Michael Larson
  • Shannon -jj Behrens
  • ShiningPanda
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Scherfke
  • Stefanie Molin
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Holden
  • Steven Klass
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Suresh Dasari/Tutlane.com
  • Swisscom ICT
  • Talk Python Blog
  • Talk Python to Me
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Terri Oda
  • Terry Jones
  • Test and Code
  • TestDriven.io
  • The Data Scientist
  • The Digital Cat
  • The Lunar Cowboy
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Open Sourcerer
  • The Parcon Blog
  • The Python Coding Blog
  • The Python Coding Stack
  • The Python Papers
  • The Python Show
  • The Three of Wands
  • Thibauld Nion
  • ThisHosting.Rocks
  • Thomas Guest
  • Thomas Vander Stichele
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Knapp
  • Tim Lesher
  • Tobias Ivarsson
  • Tom Christie
  • Tomasz Ducin
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • TypeThePipe
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinayak Mehta
  • Virgil Dupras
  • Vladimir Perić
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • Wing Tips
  • Wingware
  • Wyatt Baldwin
  • Yair Lenga
  • Yaniv Aknin
  • Yann Larrivée
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero to Mastery
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • death and gravity
  • eGenix.com
  • hypothesis.works articles
  • kdev-python
  • meejah.ca
  • nl-project
  • pgcli
  • psutil
  • py.CheckIO
  • pygame
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • scikit-learn
  • testmon
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal, open a PR or issue