skip to navigation
skip to content

Planet Python

Last update: November 07, 2026 01:47 PM UTC

November 07, 2026

Graham Dumpleton

Muru: looking inside the workers

In a first look at Muru I made a point of the access log separating how long a request waited for a worker from how long it spent in the application. That tells you where a slow request lost its time, but not why. Sooner or later the question you end up asking is what the workers are actually doing, usually because a request is taking far longer than it should.

The traditional answers to that are to guess, to sprinkle print statements and timing code through the application, or to bring in a separate monitoring product. Having spent twenty years on mod_wsgi seeing what people struggle with when hosting Python web applications, and five years at New Relic working on exactly this problem for Python, I have some fairly strong opinions about how much of this a hosting system should be able to tell you itself. Quite a lot, as it turns out, is the answer Muru gives.

A slow report

To have something to look into, here is a small WSGI application with a deliberately slow report. Fetching the rows for the report pretends to wait on a database for a tenth of a second per month, and summarising them is pure Python arithmetic over eight million rows per month.

import time
from urllib.parse import parse_qs


def fetch_rows(months):
    time.sleep(0.1 * months)  # as if waiting on a slow database query
    return range(8_000_000 * months)


def summarise(rows):
    total = 0

    for row in rows:
        total += row * row % 7

    return total


def build_report(months):
    rows = fetch_rows(months)
    return f"{months} months: {summarise(rows)}\n"


def application(environ, start_response):
    query = parse_qs(environ["QUERY_STRING"])
    months = int(query.get("months", ["1"])[0])

    if environ["PATH_INFO"] == "/report":
        body = build_report(months)
    else:
        body = "ok\n"

    start_response("200 OK", [("Content-Type", "text/plain")])
    return [body.encode()]

Rather than giving everything as options to muructl run, Muru can read its settings from a [tool.muru] table in the project's pyproject.toml. Here they name the entrypoint, ask for two worker processes, and also set a soft timeout of three seconds, which I will come back to shortly:

[tool.muru]
entrypoint = "report:application"
processes = 2

[tool.muru.timeouts]
soft_timeout = "3s"

Start it with muructl run and ask for reports covering one month and then three months. The access log shows what we would expect:

200 GET /report?months=1 19B 382ms queue=0.0ms app=382ms worker=0 id=01M6T02DPVG7XRY5V2P2V9S7WG
200 GET /report?months=3 19B 1.15s queue=0.0ms app=1.15s worker=1 id=01M6T02E30G7XRY5V2P2V9S7WG

The queue time is zero and the application time is all of it, so we know straight away the problem is in our code and not a lack of capacity. Three months taking over a second also means a year is going to be painful.

Catching the worst case

By default a request in Muru can run for as long as it likes, up to a deadline at the edge of five minutes. Even when the edge gives up and answers the client, though, the request keeps going in the worker, tying up a thread for no benefit to anyone. That is what the soft timeout is for, and why I turned it on for this example.

Ask for twelve months, using JSON since that is easier to read here:

curl -H 'Accept: application/json' 'http://127.0.0.1:8080/report?months=12'

Three seconds later the response comes back:

{"type":"about:blank","title":"Gateway Timeout","status":504,"detail":"The request ran past the soft timeout of 3 s.","reason":"soft_timeout","request_id":"01M6T02F76G7XRY5V2P2V9S7WG","traceback":"..."}

When the request passed three seconds, a watchdog in the worker noted where the request was, then raised a muru.RequestTimeout exception in the thread running it. The exception unwound the report just like any other exception would, and Muru answered with a 504. The terminal says exactly where the request was when it was stopped:

muru.host[74087]: request.soft_timeout: worker 0 (pid 74088): request 01M6T02F76G7XRY5V2P2V9S7WG GET /report?months=12 ran past the soft timeout of 3 s at report.py line 14 in summarise, called from report.py line 21 in build_report, called from report.py line 29 in application
muru.worker[0]: Traceback (most recent call last):
muru.worker[0]:   File "/Users/graham/reports/report.py", line 29, in application
muru.worker[0]:     body = build_report(months)
muru.worker[0]:   File "/Users/graham/reports/report.py", line 21, in build_report
muru.worker[0]:     return f"{months} months: {summarise(rows)}\n"
muru.worker[0]:                                ~~~~~~~~~^^^^^^
muru.worker[0]:   File "/Users/graham/reports/report.py", line 14, in summarise
muru.worker[0]:     total += row * row % 7
muru.worker[0]:     ^^^^^
muru.worker[0]: muru.RequestTimeout
504 GET /report?months=12 638B 3.07s queue=0.0ms app=3.07s reason=soft_timeout worker=0 id=01M6T02F76G7XRY5V2P2V9S7WG

Only the one request was stopped, and the worker carried on serving others.

One detail worth knowing is that muru.RequestTimeout derives from BaseException and not Exception. A lot of application code catches Exception to log an error and carry on, and if the timeout were an Exception that code would swallow it and keep going. An application which does want to respond in its own way, such as telling the user to narrow down the date range, can still catch muru.RequestTimeout by name.

The soft timeout can only interrupt Python code. A thread stuck inside a C extension won't see the exception until the call returns. For that case there is also a hard timeout, which gives up on the whole worker and replaces it, but that is a topic for another time.

Which requests are in flight

A timeout tells you about a request after the fact. Often what you want is to know what is running right now, and with more than one worker, the first thing to work out is which worker a slow request is running in. A six month report takes about two seconds, so start one in the background while a few one month reports are also being requested, and ask Muru what requests are in flight:

curl -s 'http://127.0.0.1:8080/report?months=6' &
muructl requests

RELEASE              GROUP        QUEUED  QUEUED-OLDEST  THREADS  THREAD-OLDEST
report@r1            default      0       -              5/8      1.2s

Under the daemon you would need to say which application to look at, but a development run only ever serves the one application. The muructl commands find the run by its admin socket when you run them from the project directory, so there is nothing more to type.

This first view counts the requests for each release. Nothing is waiting in the queue, five of the eight threads across the two workers are busy, and the oldest request has been running for 1.2 seconds. Adding --list shows each request, oldest first:

muructl requests --list

AGE      STATE    KIND            WORKER  REQUEST                                  CLIENT           REQUEST-ID
1.2s     running  wsgi            1       GET /report?months=6                     127.0.0.1        01M6T02SA1G7XRY5V2P2V9S7WH
0.5s     running  wsgi            0       GET /report?months=1                     127.0.0.1        01M6T02T18G7XRY5V2P2V9S7WJ
0.4s     running  wsgi            0       GET /report?months=1                     127.0.0.1        01M6T02T56G7XRY5V2P2V9S7WJ
0.3s     running  wsgi            0       GET /report?months=1                     127.0.0.1        01M6T02T94G7XRY5V2P2V9S7WJ
0.1s     running  wsgi            1       GET /report?months=1                     127.0.0.1        01M6T02TD2G7XRY5V2P2V9S7WJ

The six month report is the oldest, and it is running in worker 1.

Strictly speaking, this isn't looking inside the workers at all. The answer comes from the edge, which knows about every request it has passed to a worker, along with how long it has been running and which worker it went to. Nothing has to be asked of the workers to get it, so it costs them nothing, and it still answers even when a worker is too wedged to respond. That is why it is the first place to look. Once you know which worker a request is in, the more expensive tools can be pointed at just that worker.

There is an equivalent muructl websockets command for WebSockets, which aren't being used here. It counts the WebSockets open for each endpoint, and with --list shows each one, the longest open first.

Seeing a request in progress

Knowing the six month report is in worker 1, the next step is to see what it is doing. Asking for the stacks of that worker:

muructl debug stacks --worker 1

Among the threads shown is the one serving the report:

muru-app-1 in worker 1:
  serving GET /report?months=6 for 1.3 s (request 01M6T02SA1G7XRY5V2P2V9S7WH)
  File "/Users/graham/Library/Caches/muru/runtime/0.1.0a18/cp315t/muru/_pool.py", line 30, in serve_jobs
    serve()
  File "/Users/graham/reports/report.py", line 29, in application
    body = build_report(months)
  File "/Users/graham/reports/report.py", line 21, in build_report
    return f"{months} months: {summarise(rows)}\n"
  File "/Users/graham/reports/report.py", line 14, in summarise
    total += row * row % 7

It shows which request the thread is handling, how long that request has been running, and where in the code it is at that moment. Leave off --worker and you get every thread in every worker instead, all without stopping anything.

How the stacks are obtained depends on the platform. On Linux with Python 3.15 or later, Muru reads the stacks from outside the worker process using the remote debugging support added by PEP 768, so it works even if a worker is wedged in C code while holding the interpreter. On macOS, where reading another process needs root, the worker is asked to report its own stacks instead, which is what happened here.

Where the time goes

A stack is a single moment in time. To know where the time goes overall, you want a profile, which samples the stacks repeatedly over a period and counts where they were. To get some load to profile, keep reports coming in another terminal:

while true; do curl -s 'http://127.0.0.1:8080/report?months=2' > /dev/null; done

and then profile the workers for ten seconds:

muructl debug profile --for 10s --output report.folded

808 samples of running Python code in 1632 rounds across 2 workers
   73.4%  summarise (report.py:10)
   26.6%  fetch_rows (report.py:5)
muru: collapsed stacks written to report.folded

Three quarters of the time is going on summarising the rows and a quarter on waiting for them, so a faster summary would be worth a lot more than a faster query. Being able to say that with evidence, rather than a hunch, is the whole point.

The file written out holds the collapsed stacks with their counts, which is the format flame graph tools expect, so flamegraph.pl can turn it into an SVG or you can open it in speedscope in a browser. If the application has many endpoints and only one of them is a problem, the profile can also be restricted to requests for a particular path.

Sampling every so often tells you proportions. If you want exact numbers instead, muructl debug trace counts every call of every Python function over a period, along with the time spent in each:

muructl debug trace --for 3s

worker 0 (pid 74088)
    total ms     calls  function
    1483.823         2  application (report.py:24)
    1483.674         2  build_report (report.py:19)
    1073.508         2  summarise (report.py:10)
     410.103         2  fetch_rows (report.py:5)
       0.095         2  parse_qs (parse.py:859)
       0.056         2  parse_qsl (parse.py:902)
       0.042         4  parse_qsl.<locals>._unquote (parse.py:937)
       0.031         4  unquote_plus (parse.py:983)
       0.014         4  unquote (parse.py:836)

worker 1 (pid 74089)
    total ms     calls  function
     736.969         2  application (report.py:24)
     736.857         2  build_report (report.py:19)
     531.764         2  summarise (report.py:10)
     410.168         2  fetch_rows (report.py:5)
       0.104         2  parse_qs (parse.py:859)
       0.064         2  parse_qsl (parse.py:902)
       0.048         4  parse_qsl.<locals>._unquote (parse.py:937)
       0.036         4  unquote_plus (parse.py:983)
       0.018         4  unquote (parse.py:836)

Tracing every call is a lot more expensive than sampling, so Muru will refuse to trace a worker which already has more than half of its threads busy with requests unless forced to with --force. The worker does keep serving while the trace runs though, since the requests it takes in that time are what the trace is measuring. Those requests are slowed down by the tracing, so this isn't something to leave running for long against an application people are using.

Memory and running code

Slow requests aren't the only thing that goes wrong. Python web applications also have a habit of slowly growing in memory until something gives. Running muructl debug memory reports for each worker its memory use, the garbage collector's counts and the object types it holds most of. Adding --trace turns on tracemalloc for a period and reports the places in the code which allocated the most memory during that time, and there are options for taking a snapshot at one point and comparing against it later.

When all else fails, muructl debug exec will run a snippet of code inside the workers and print the result, which is handy for checking the state of some cache or other in a running process:

muructl debug exec -- 'import sys; sys.version'

worker 0 (pid 74088):
'3.15.1 free-threading build (main, Oct 21 2026, 17:44:52) [Clang 23.1.3 ]'
worker 1 (pid 74089):
'3.15.1 free-threading build (main, Oct 21 2026, 17:44:52) [Clang 23.1.3 ]'

Under the daemon this one is turned off unless explicitly allowed in the daemon's configuration, since it lets someone run arbitrary code inside a production application. In a development run it is always on, since the person at the terminal is the one who started the workers in the first place.

Stopping at a breakpoint

All of the above is about looking at a worker from the outside. Sometimes though you really do want to stop in the middle of a request and poke around, and the usual way to do that in Python is a call to breakpoint().

The catch is that with a normal development server, breakpoint() drops you into pdb on whatever terminal the server's output is going to, interleaved with log output from everything else. In Muru, a worker which hits breakpoint() instead waits for a debugger to attach, and the terminal tells you how to do that. Add a breakpoint() call to build_report() after the rows are fetched, save the file so it reloads, make a request, and the terminal shows:

muru.host[75915]: debug.breakpoint: worker 0 (pid 75918): breakpoint() at /Users/graham/reports/report.py:21 waits 60 s for muructl debug attach report --worker 0

The end of that line gives the command to run, which is muructl debug attach report --worker 0. Running it from another terminal opens a pdb session in the worker, at the point the request stopped:

muructl debug attach report --worker 0

muru: attached to worker 0 at /Users/graham/reports/report.py:21; the end of input lets the code go on
> /Users/graham/reports/report.py(21)build_report()
-> breakpoint()
(Pdb) p months
1
(Pdb) continue
muru: the session in worker 0 has ended

Typing continue lets the request carry on, and it completes as normal. While the session was open, the worker was taken out of routing so it wouldn't be given any new requests, leaving those to worker 1. The terminal records that, along with the worker being put back once the session ended:

muru.host[75915]: worker 0 (pid 75918) leaves routing while it is debugged
muru.host[75915]: debug.detached: worker 0 (pid 75918): the debugger session at /Users/graham/reports/report.py:21 ended
muru.host[75915]: worker 0 (pid 75918) serves again: no debugger is attached
200 GET /report?months=1 19B 2.28s queue=0.0ms app=2.28s worker=0 id=01M6T043M4G7XRY5V2P2V9S7WJ

The request was also exempt from the soft timeout, so it wasn't killed off while I was looking at it. If nobody attaches within the timeout, the request just carries on by itself.

For those who prefer to debug from an IDE, muructl debug attach --dap speaks the Debug Adapter Protocol instead, so something like VS Code can attach to the worker through muructl. That needs debugpy installed in the project's environment, and I will leave the details of that to the Muru documentation.

The dashboard

Everything so far has been done from the command line. When you just want to keep an eye on things, the run also serves a dashboard, by default at http://127.0.0.1:8081/.

The Muru development dashboard showing the last minute's requests, latency and errors, the two workers and how busy they are, charts over time, and the most recent requests

The top of the page shows the last minute at a glance, with requests per second, the 50th and 95th percentile latency, the count of server errors, how many requests are in flight and how many WebSockets are open. Below that is a table of the workers and how many of their threads are busy, charts of the same figures over time which can be switched to cover the last hour or the last day, the most recent requests with the same queue and application times as the access log, and the latest output from the run. A link from the workers table opens a page for each worker with its status, and the same stacks, tasks and memory figures the muructl debug commands give. In a development run there is no sign-in, and the dashboard is read-only, so nothing on it can change what the run is doing.

The same address also serves a /metrics endpoint in the Prometheus format, so if you do want to point some other monitoring system at a development run, you can.

What's next

What I like about all this is that none of it required changing the application, installing an agent, or restarting anything, and all of it works the same way against an application running under the daemon in production. Along the way I have been talking about hosts, workers, the edge and admin sockets without really explaining them, so in the next post I will step back and look at how the parts of Muru fit together.

November 07, 2026 12:00 AM UTC

Peter Hoffmann

Stable URLs for 42,000 Huts and Peaks

biwak.info is my map of Alpine huts, bivouacs and peaks. For a long time the only way to link to a hut was a #Name hash that did an exact name match and fell back to the search when the name wasn't unique. I wanted real URLs I can share and that show a proper preview card in a chat: biwak.info/p/matterhorn.

The data comes from OpenStreetMap: 9,429 huts and bivouacs and 33,437 named peaks, 42,866 places in total. Turning a name into a URL is easy. The hard part is that many places share a name, and that a URL, once shared, should keep pointing to the same place, even when the data changes underneath.

The preview card for https://biwak.info/p/bivouac-des-periades

From a name to a slug

The first step is a plain slugify: German umlauts are written out, everything else is folded to ASCII, lowercased, and every run of other characters becomes a single dash.

GERMAN_TRANSLIT = str.maketrans({"ä": "ae", "ö": "oe", "ü": "ue", "Ä": "Ae", "Ö": "Oe", "Ü": "Ue",
                                 "ß": "ss", "ẞ": "ss"})


def slugify(text):
    """German transliteration for umlauts and ß, then ASCII-fold, lowercase, everything else to single dashes."""
    text = (text or "").translate(GERMAN_TRANSLIT)
    text = unicodedata.normalize("NFKD", text).encode("ascii", "ignore").decode("ascii").lower()
    return re.sub(r"[^a-z0-9]+", "-", text).strip("-")

So Oberaletschhütte becomes oberaletschhuette, Štuhecov dom becomes stuhecov-dom and Rifugio Città di Busto becomes rifugio-citta-di-busto. None of the 42,866 names ends up empty.

Names are not unique

Alpine names repeat a lot. Every valley has its Jagdhütte and its Rosskopf. With the name alone, the most common slugs are:

Slug Places
jagdhuette 70
schwarzkogel 37
buchberg 32
rosskopf 32
mitterberg 31

Before choosing a scheme, I measured how many places would still share a slug with different suffixes:

Slug made of Shared slugs Places affected Worst
name 2,181 6,690 (15.6 %) 70
name + country 1,993 5,936 (13.8 %) 64
name + region 1,516 3,932 (9.2 %) 27
name + region + elevation 118 394 (0.9 %) 21

Adding the country barely helps: most duplicates are in the same country. The region roughly halves them, but there are still 27 places called Jagdhütte in Carinthia alone. Elevation fixes most of the rest. It can't fix everything, because 4,311 of the huts have no elevation in OpenStreetMap. The worst case in the last row is casella in Liguria: 21 small huts with the same name and no elevation.

So there is no single suffix that always works, and adding all of them to every URL would make /p/matterhorn look like /p/matterhorn-wallis-4478.

Add a suffix only while the name is shared

The rule I settled on: a place gets the shortest slug that is unique, trying these in order:

  1. the plain name: /p/matterhorn
  2. name and region: /p/oberaletschhuette stays plain, but a shared name becomes something like /p/rosskopf-tirol
  3. name, region and elevation: /p/rosskopf-tirol-2257
  4. name, region and OpenStreetMap id: /p/jagdhuette-kaernten-w1202594655

In code this is a generator that yields the candidates for one place, given all current places with the same name slug:

def candidates(p, group):
    """Slugs for POI p in order of preference; `group` = all current POIs with the same name slug."""
    base, region = p["base"], slugify(p["region"])
    others = [o for o in group if o["id"] != p["id"]]
    if not others:
        yield base
    if not any(slugify(o["region"]) == region for o in others):
        yield f"{base}-{region}"
    if p["ele"] is not None and not any(slugify(o["region"]) == region and o["ele"] == p["ele"] for o in others):
        yield f"{base}-{region}-{p['ele']}"
    yield f"{base}-{region}-{p['id']}"

The OpenStreetMap id is the last resort because it's always unique, but nobody can read it. On today's data the 42,866 places split like this:

Slug Places
plain name 36,176
+ region 2,758
+ elevation 3,474
+ OpenStreetMap id 458

Regions from Natural Earth

The region has to be something people recognise. I cut the Natural Earth admin-1 boundaries down to the units that make sense in the Alps: the Bundesland in Austria and Germany, the Kanton in Switzerland, the département in France (Savoie and Haute-Savoie are better known than the big régions), the regione in Italy with South Tyrol and Trentino split, and the statistical regions in Slovenia. Names use the German form where that is common (Wallis, Tessin, Aostatal), otherwise the local one. The places on the map fall into 63 of these regions.

Each place gets its region by a point-in-polygon lookup with Shapely. The boundaries are simplified, so a hut right on a border or a lake shore can fall outside every polygon. Then the nearest polygon wins:

def lookup(self, lat, lon):
    """(cc, region name); points just outside every polygon (coast, simplified border) take the nearest."""
    pt = Point(lon, lat)
    for i in self.tree.query(pt):
        if self.geoms[i].contains(pt):
            break
    else:
        i = self.tree.nearest(pt)
    return self.props[i]["cc"], self.props[i]["name"]

The region is also shown on the preview card and in the detail view, next to the type: "Biwak · Haute-Savoie, Frankreich".

Stable: a registry that never forgets

The scheme above depends on the data. If someone adds a second Matterhorn to OpenStreetMap tomorrow, the plain rule would rename the real one to matterhorn-wallis, and every link shared so far would break.

So the build keeps a slug registry, data/slug-registry.json, which is committed to git. It maps every slug ever handed out to the OpenStreetMap id and the name:

"matterhorn": ["n26863664", "Matterhorn"],
"oberaletschhuette": ["n11262553552", "Oberaletschhütte"],

The assignment then follows three rules:

primary = {}
for p in pois:
    fitting = [s for s in by_id.get(p["id"], []) if s == p["base"] or s.startswith(p["base"] + "-")]
    if fitting:
        primary[p["id"]] = min(fitting, key=lambda s: (len(s), s))

for base in sorted(groups):
    for p in groups[base]:
        if p["id"] in primary:
            continue
        slug = next(s for s in candidates(p, groups[base]) if s not in registry)
        registry[slug] = [p["id"], p["name"]]
        primary[p["id"]] = slug

A place that disappears from OpenStreetMap keeps its slug reserved. Opening it starts a search for the old name instead of showing someone else's hut.

Because the registry is a sorted JSON file with one slug per line, a build that changes slugs shows up as a readable git diff. That made one mistake easy to spot: the first version folded ü to u, so the Oberaletschhütte was /p/oberaletschhutte. Two weeks after launch I switched to ue, which changed 7,182 slugs. I rebuilt the registry from scratch, so the old slugs are gone instead of redirecting.

Two fallbacks

A plain name that is shared, like /p/rosskopf, doesn't return an error. It opens the search with exactly the places of that name, with their type, elevation and region, and each of them links to its own slug. The same works one level down for /p/rosskopf-tirol.

A slug that ends in an OpenStreetMap id always resolves, even if the name part is wrong or outdated. /p/old-name-n11262553552 answers with a 301 to /p/oberaletschhuette.

Limits

The same /p/ URLs also let the map open peaks directly, as described in Loading 33,000 Peaks on Demand with a z10 Tile Grid.

November 07, 2026 12:00 AM UTC

November 06, 2026

"Michael Kennedy's Thoughts on Technology"

Catch the import. Name the culprit. Keep threads free: Introducing gilguard

Catch the import. Name the culprit. Keep threads free: gilguard

TL;DR: gilguard is a tiny, zero-dependency package for free-threaded Python (3.13t, 3.14t, 3.15t). Import it first. If any later import quietly turns the GIL back on, you get an error naming the extension module, the package that shipped it, and the line of your code that pulled it in. python -m gilguard audits an entire environment in one go. It’s on PyPI now, and it’s the second piece of a larger free-threaded Python effort I’ve been working on.

The quiet failure mode of free-threaded Python

Free-threaded Python is officially supported as of 3.14, and I’m all in. Threads that actually run Python code in parallel? Yes please. But the migration has a failure mode that I think is going to bite a lot of teams, and there isn’t much written about it.

Here’s the scenario. You move a service to 3.14t. You give the hot path a thread pool. You deploy, pull up the dashboard, and the throughput graph does… nothing. It’s the exact same shape as last week, as if the whole migration had been a dream.

An hour later you find the reason, one line in a startup log that scrolled past in half a second:

RuntimeWarning: The global interpreter lock (GIL) has 
been enabled to load module 'somepackage._speedups', 
which has not declared that it can run safely without the GIL.

That line is CPython being careful. Every C extension has to declare that it’s safe to run without the GIL (via the Py_mod_gil slot). When the interpreter loads one that hasn’t, it turns the GIL back on for the rest of the process and carries on. Your threads still run, they just take turns again. And here’s the kicker: once the GIL is on, nothing can turn it back off for the life of that process.

The warning names a private module you’ve never heard of. It doesn’t tell you which package you installed shipped it, or which of your imports dragged it in. So now you’re bisecting imports.

What is gilguard?

gilguard turns that one line into an exception, raised at the import statement that caused it. Make it the first import in your program and you’re done. You don’t call anything or configure anything:

import gilguard  # must be first

import fastapi
import pandas  # <- raises here if anything in pandas' import tree needs the GIL

When something in that tree forces the GIL back on, this is what you see instead of a warning you’d miss:

gilguard.GILEnabledError: Importing 'gilneeder._speedups' re-enabled the GIL.

  Module:       gilneeder._speedups
  File:         .../site-packages/gilneeder/_speedups.cpython-314t-x86_64-linux-gnu.so
  Distribution: gilneeder 1.2.0
  Import chain: gilneeder -> gilneeder._speedups
  Your code:    /home/me/app.py:5
  Imported by:  .../site-packages/gilneeder/__init__.py:1

This extension module does not declare free-threading 
support (Py_mod_gil), so CPython turned the GIL back 
on for the rest of the process. It cannot be turned off again.

Options:
  - Upgrade gilneeder to a release that ships free-threaded (cp314t) wheels.
  - Drop or replace gilneeder.
  - If you have verified it is thread-safe, run with PYTHON_GIL=0 to keep the GIL off (at your own risk).
  - Run `python -m gilguard` to list every installed package that does this.

(gilneeder is one of gilguard’s test fixtures, a tiny C extension built to misbehave so the real ones don’t have to.)

Everything you’d otherwise be asking a debugger for is right there: which module flipped it, what file it lives in, and which distribution and version shipped it (so you know what to upgrade or report). You also get the import chain from your code down to the culprit, which matters when the offender is a dependency of a dependency of something you installed for one function, and the line in your code where it all started.

You might notice that GILEnabledError derives from BaseException rather than Exception, putting it alongside KeyboardInterrupt. Here’s why. Lots of packages have an optional speedup import like try: import fastthing / except ImportError: import slowthing. If gilguard raised a normal exception, that except would eat it, the package would quietly fall back to its pure-Python path, and your app would carry on with the GIL already back on. The report would end up inside someone else’s except block, where nobody would ever see it. With BaseException, it sails right past those handlers. To catch it, you have to ask for GILEnabledError by name (or use a bare except:, and you know better).

Not ready to fail hard? Set GILGUARD_ACTION=warn and you get the same report as a warning while the program keeps running. Set it to off and the same code runs on a standard build.

How do I find every offender in an environment?

A single process can only tell you about the first module that flips the GIL. After that it’s already on, and the rest have nothing left to flip. So gilguard also ships a scanner:

python -m gilguard

GIL    gilneeder                via gilneeder._speedups  [gilneeder 1.2.0]

8 checked: 7 ok, 1 re-enable the GIL, 0 failed to import.

It gives every installed package its own fresh interpreter (in parallel, across your CPUs), so none of them can hide behind the one that went first. It exits 1 when it finds anything, so it drops into CI as a single step without any extra scripting. There’s --json for tooling and --strict if a package that fails to import should fail the build too. Pass package names to check just those, which is how you re-check one after an upgrade to see if the fix took.

The scan has one blind spot: extensions a package loads lazily, on first use of a feature, don’t show up at import time. For those, run your test suite with gilguard loaded as a plugin:

pytest -p gilguard

pytest loads -p plugins before it touches conftest.py or any test module, so the guard is watching before your code imports anything. The two checks cover different gaps. The scanner sees every installed package but only at import time, and the test suite sees lazy loads but only on the code paths your tests run. Use both. Tests and CI has a GitHub Actions job that does exactly that.

What does a clean run actually mean?

“The GIL stayed off” is easy to over-read. Here are the limits. A clean run means none of the imports that actually ran turned the GIL back on. It does not mean your code or your dependencies are thread-safe, and it does not mean your program gets faster with more threads. Those are separate problems with their own tests. gilguard catches one specific failure, and that’s all it tries to do.

Once you have a name, the docs have a whole page on what to do with it: upgrade, report upstream (with exactly which lines to paste into the issue), replace, or, if the extension is yours, the one slot that declares support, with the one-line equivalents for Cython, pybind11, and PyO3.

Why I built this

When I announced chameleon-rs ten days ago, I said free-threaded Python means we need to start expecting more from our web apps. What I didn’t say is that chameleon-rs was the first piece of a larger free-threaded effort, and just the first one I am able to talk about publicly. gilguard is the second.

It’s a much smaller piece, a few hundred lines of plain Python with no dependencies. But it’s the kind of building block everything else depends on. There’s no point making a template engine or a web framework thread-safe and fast if one dependency turned the lock back on at startup and nobody noticed for a month. Before you can make free-threaded Python fast, you have to be able to trust that you’re actually running free-threaded.

My goal with all of this work is simple to state: free-threaded Python faster, safer, better, and adopted sooner. gilguard is my contribution to the “safer” and “sooner” parts. I’ll write about the rest as soon as I’m able to.

Try it

uv pip install gilguard

Then import gilguard at the top of your entry point, or run python -m gilguard against your current environment and see what’s lurking. I’d love to hear what it finds. The docs are short and the code is on GitHub. If you use an AI coding agent, the docs also ship an agent skill that teaches it the rules (first import, don’t catch the error, what a pass means), so it can wire gilguard into your project and CI without you explaining any of this.

November 06, 2026 11:29 PM UTC

Mike Driscoll

An Intro to Python Logging Handlers (Video)

Learn about Python’s log handlers, part of the built-in logging module, in this video.

This video is a part of my Python Logging Video Course. Check it out today!

The post An Intro to Python Logging Handlers (Video) appeared first on Mouse Vs Python.

November 06, 2026 07:08 PM UTC

Tryton News

Security Release for issue 15206

Clément Mercier using Claude has discovered that it is possible to retrieve the ephemeral data from the wizard execution of another user.

Impact

CVSS v3.0 Base Score: 2.6

Workaround

There is no known workaround.

Resolution

All affected users should upgrade trytond to the latest version.

Affected versions per series:

Not affected versions per series:

Reference

Concerns?

Any security concerns should be reported on the bug-tracker at https://bugs.tryton.org/ with the confidential checkbox checked.

1 post - 1 participant

Read full topic

November 06, 2026 01:00 PM UTC

Security Release for issue 15205

Clément Mercier using Claude has discovered that there is no restriction on running wizards without any model or group.

Impact

CVSS v3.0 Base Score: 8.1

Workaround

Add at least one group to the listed wizards in the issue via “Administration / User Interface / Actions / Wizards”.
This must also be done after the update in order to allow users to execute the said wizards.

Resolution

All affected users should upgrade trytond to the latest version and ensure that no unwanted users have been created on the system.

Affected versions per series:

Not affected versions per series:

Reference

Concerns?

Any security concerns should be reported on the bug-tracker at https://bugs.tryton.org/ with the confidential checkbox checked.

1 post - 1 participant

Read full topic

November 06, 2026 01:00 PM UTC

LernerPython blog, from Reuven Lerner

Pandas 3.1 `select` method: Retrieving specific columns

Pandas 3.1 `select` method: Retrieving specific columns

Want to retrieve specific columns from your Python Pandas data frame? In the past, you would say

df[['x', 'y']]

But now, with version 3.1, you can use the “select” method:

df.select('x', 'y')

The post Pandas 3.1 `select` method: Retrieving specific columns appeared first on LernerPython.

November 06, 2026 07:00 AM UTC

Core Dispatch

Core Dispatch #12

Welcome back to Core Dispatch! This edition covers October 9 through November 6, 2026. Python 3.16.0 alpha 1 is out, with alpha 2 due this Tuesday, November 10. October kept the release team busy: we shipped 16 Python releases, though only five were planned. Ten of the eleven extras were security releases, some of which we cut during the core team sprint. Huge thanks to everyone who helped get them out. The October 26 releases block the compromised eMudhra intermediate CA keys in ssl and include additional security fixes.

On the PEP front, two new proposals cover wheel-variant providers (PEP 850) and list.discard() (PEP 851). The pretty print protocol (PEP 813) is back for discussion after a substantial revision at the core team sprint. The proposed roadmap for a supported JIT (PEP 836) and the proposal to drop LLVM as a JIT build requirement (PEP 774) have both been withdrawn. The Steering Council has made no further decision about the JIT.

On CPython's main branch, notable merges speed up UUID operations, numeric strptime() parsing, and buffered asyncio reads. Free-threaded builds get safer memoryview operations and a fix for a crash in asyncio.

Community updates include proposals for better performance tooling, ecosystem testing with snake_pit, and the updated AI-policy triage process.

Please test your projects with Python 3.16.0 alpha 1 and file any CPython issues you find.

Upcoming Releases

Official News

PEP Updates

Steering Council Updates

Merged PRs

Discussion

Core Dev Musings

Upcoming CFPs & Conferences

Community

One More Thing

"We are not going to establish a social credit system on discuss.python.org!"

— Emma Smith

"Yeah we messed up and left your time machine in Monday by accident. I’m asking research for a loaner so we can go back and get it."

— Brandt Bucher

Credits

November 06, 2026 12:00 AM UTC

November 05, 2026

Glyph Lefkowitz

Why You Should Care About The Political Project of Twisted

Twisted was originally the network component of a text-based role-playing game engine. In the late 90s when it was first developed, the idea of a video game being able to send you actual emails and instant messages1 seemed pretty neat, rather than the valence that it would have today (nightmare). We never ended up making an actual game, so we eventually removed the “game engine” piece, and just kept the protocol implementations of the email and the AOL Instant Messenger, and the DNS and eventually SSH and so on. The residual pieces of that project have not seen maintenance for a decade.

So it’s not really a way to make a text-based game any more.

Then, Twisted was the premier longest-lived and best-maintained event-driven I/O library for Python. If you wanted to make high-performance servers, it was the best and most obvious way to do it. Even if it still acquits itself relatively well in this regard, its claim to fame, and to indispensability, has been somewhat obviated by the inclusion of asyncio in the standard library. Even if it has some stuff to recommend it — better-than-average testing facilities, for example — you can get better-documented and better-integrated low-level asynchronous networking tools without even installing any packages.

So it’s not really the premier choice for an event-loop these days.

Twisted does still contain an SSH server and client, an email server and client, a DNS server and client, a web server and client (now including websockets). But you can cobble these together by installing a variety of packages from other sources.

So... it’s a bunch of protocols? And I guess it doesn’t use AI? But... why would anyone care about that? Why contribute to such a thing?

What is Twisted doing, and who is it doing it for? What is the political project of Twisted, nigh on 30 years into its existence?

Most of all, why should you care about it, or want to contribute?

If you’re asking what you could use Twisted for today, it’s an excellent choice if you’re implementing a new network service and you need more than one protocol. If you need to do HTTP, DNS, packet capture, and a custom SSH server, it would be very hard to arrange a collage of other Python packages to manage this. It’s still much better at deterministic testing than most of the standard library. It’s a bit easier to hook into a GUI.

But that’s not what Twisted is doing.

What I want Twisted to do for the world is still an unfinished project. It’s honestly kind of bad at doing what I want it to, but it could be great. The overarching goal of having and maintaining all this stuff is to, as much as possible, demystify networked software operation and let you build Internet facing systems at a human scale.

It’s all in a memory-safe language with good error handling, so you can avoid the scariest security pitfalls. It’s all integrated into a single paradigm with built-in tools for testing. It’s designed to scale down to be runnable on the average laptop, and not require large elaborate server farms or distributed clusters to run. Via the twist command line utility, it provides a Swiss army knife of simple tools that you can use out of the box.

In a world where all of our compute is in massive distant opaque datacenters, where a tiny cartel of gigantic tech companies control nearly the whole supply chain, one of the few things that you still can have agency over is your own server on the Internet.

It’s still possible to get a domain name for a few bucks a year, a free-tier or very low-cost VPS, then stick a glue record in and run your own DNS. From that point forward, you can run your own email, your own web server, your own terminal server. But doing this requires deep expertise, and that expertise requirement is massively inflated by the operational concerns that go with running twenty different pieces of software from twenty different projects, probably in twenty different languages, with completely inconsistent aesthetics.

The reason that Twisted is event-driven is because as much as it can be a pain to wrap your head around async/await (or worse yet, Deferred directly), it is much easier for a single person to understand the concurrency requirements of an async I/O multiplexed system than to understand what is going on with an application that requires multiple thread pools. It is possible to understand concurrent code that manages parallel tasks in that idiom. The reason that Twisted integrates so many different protocol implementations and attempts2 to unify them with a common authentication layer is to reduce the overhead required to manage a comprehensive suite of network services.

Even the not-using-AI thing is not an aside due to some accidental personal preference about tools — it’s a core part of building an engine that you can personally understand if you dive into it, that’s built with a consistent style and layered in a way that you can take apart if you need to. It’s not built with AI because we’re not building for AI, we’re building for people and the entire point is that people are supposed to be able to understand it.

We used to have a tag line, which is somewhat less emphasized these days because its meaning is not immediately clear without this background, but was once quite prominently featured on the home page: “Twisted: Building the Engine of Your Internet”. That’s what I want to bring to you: an engine for you to build your Internet, that is small enough that you can make it personal, that doesn’t require massive investment in management tooling just to get started.

Now, for various reasons, I no longer use twisted.names to host my own DNS (although I did for many years!), and we never wrote enough of a twisted.mail suite for it to be usable for email hosting on its own. But, if you’re reading this article, it was probably delivered to you directly via Twisted’s own twisted.web HTTP server, using twisted.protocols.tls to configure our own OpenSSL TLS termination. I have run with extremely high reliability for years and next to zero management overhead, in a single simple docker container. Which I then administer and upload files to via Twisted’s own twisted.conch SSH server.

If that seems like an appealing project and you’d like to volunteer to help out, drop me a line and maybe I can point you at some tasks to get started. Twisted falls far short of this aspiration despite 30 years of work, and we could use all the help we can get.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor!

  1. Yes, it was the 90s, we didn’t have “texts” yet, it was all “IMs”. ↩

  2. — with varying degrees of success — ↩

November 05, 2026 10:51 PM UTC

PyCharm

What’s New in PyCharm 2026.3

At the center of this release is JetBrains Air – a new layer for agentic development. It lets you run, track, and verify multi-agent workflows directly inside your IDE, with code intelligence keeping agents on track and complete review tools in your hands. This release also brings workspaces that adapt terminals and consoles to individual subprojects, initial support for Python 3.15 lazy imports, interactive FastAPI dependency graphs, and deeper Pydantic inspections, and so much more.

Download PyCharm 2026.3 now

Check out an overview video by our developer advocate William Vincent:

Now, let’s take a closer look at the key highlights.

JetBrains Air in PyCharm – One place to run your agents and verify their work 

As of this release, JetBrains IDEs ship with Air, a new layer for agentic development. Work with the coding agents of your choice, and run, track, and verify multi-agent workflows inside your IDE, where code intelligence helps agents make fewer wrong turns and gives you the tools to review every change before it ships.

Your agents, your models, your subscriptions – with Junie Lite for free

Air works with Claude Agent, Codex, Gemini, GitHub Copilot, Junie, OpenCode, and other ACP-compatible agents. It detects the agents already installed on your machine so you can connect them in a click, and you can add more from the agent registry.

Your existing setup carries over, too: provider subscriptions, API keys, and even a custom base URL all work, with no JetBrains AI subscription required. Run Claude Agent on your Claude subscription in the terminal or in a full-screen IDE tab, use other agent subscriptions in the chat or terminal, and switch between subscriptions at any moment, even mid-session.

Air includes free AI with the Junie Lite configuration, so you can use an agent without a subscription. You only need a JetBrains Account.

A central workspace to review changes and guide the agent

Review an agent’s changes with the surrounding code and your project tools in view, and send feedback straight back to the agent. Anchor a note to a specific line in the diff, or comment directly from the code, so the agent gets concrete context about the change you want. Work in the chat or the terminal, whichever suits you.

Still using AI Assistant? A notification will offer you a trial of Air. Moving to Air hides AI Assistant, if you wish to return to AI Assistant follow the steps outlined here. Air automatically migrates your IDE skills, MCP configuration, agents installed through ACP, authentication, and chat history. Chat mode history isn’t migrated.

Workspace support

A console and terminal for each subproject

Each subproject has its own Python console and terminal running in its own environment, so you run code against the dependencies that subproject actually uses. In uv workspaces, the console uses the workspace environment and adds the subproject root to sys.path.

Subproject creation and conversion

You can add a new subproject or convert an existing folder from the Project tool window context menu. The subproject is initialized using uv, Hatch, or Poetry, and automatically added as a workspace member to your root pyproject.toml in uv workspaces.

Python 3.15 lazy imports

PyCharm now supports the lazy import syntax introduced in Python 3.15. A lazy import defers loading a module until it’s first used, which can noticeably cut startup time in applications and scripts with heavy dependencies. PyCharm parses and resolves lazy imports like regular ones, so you can adopt the syntax without giving up code insight on the names you import.

Learn more about lazy imports in our blog post.

FastAPI support [Pro]

An endpoint can depend on a chain of Depends() and Security() calls spread across many files. PyCharm draws that chain as a graph for any endpoint. Each dependency is marked as yield or return, and circular dependencies, which raise a RuntimeError at startup, are flagged in the editor. In the other direction, a dependency shows which endpoints rely on it, directly or through other dependencies.

You can open the graph from the gutter icon next to an endpoint or click the endpoint count above a dependency to see its usages.

Python interpreter widget

The redesigned Python interpreter widget in the status bar is built around the tools that manage your environments, such as uv, Poetry, Hatch, and conda. It detects the ones installed on your machine and lists the environments each one manages for your project.

Environments created from the command line also appear under their tool. To start working in one, open the widget, select the tool, and pick the environment. To recreate it with a different Python version, choose Recreate Environment under Current Environment. If the Python version you choose isn’t installed, you’ll be prompted to download it.

Pydantic support

PyCharm now provides deeper editor support for Pydantic models. Field constraints, validators, and configuration are read from the model’s declaration and used across the editor.

What you see here only scratches the surface of what is included in PyCharm 2026.3. From editing standalone files in the new Home window to full SQLAlchemy ORM support, automatic project configuration copy in Git worktrees, and remote development directly from the IDE via SSH – there is plenty more to explore across code insight, testing, environments, and version control.

Head over to the What’s new in PyCharm 2026.3 page to see the complete list of features, performance improvements, and fixes.

Read the full What’s New

We would love to hear how JetBrains Air and the new workspace tooling fit into your day-to-day work – share your thoughts in the comments below or submit feedback directly in our issue tracker.

November 05, 2026 03:03 PM UTC

Python Software Foundation

How the Python ecosystem responded to the eMudhra intermediate compromise

The news of the eMudhra Intermediate CA compromise was published on October 20th, 2026. Responding to an incident like a Certificate Authority (CA) compromise at an ecosystem-scale is far from “routine”; the assumption that HTTPS is secure is baked into many aspects of Python applications and packaging. This meant it was critical to mount a response that was both comprehensive and customized to each affected project and service, ranging from the Python language, pip, and the Python Package Index (PyPI).

This is where security staffing at the Python Software Foundation made a difference for Python. Staffing meant that Seth Larson, the Security Developer-in-Residence, was on standby able to respond and begin work on the issue immediately. Pre-existing relationships with affected project maintainers afforded plenty of trust, and the work was completed in a way that respected the time of volunteer maintainers for affected packages by reaching out proactively with clear next steps. Thanks to Alpha-Omega for funding the Security Developer-in-Residence position at the Python Software Foundation.

The incident was mitigated across multiple components within two days. Thanks to the work of their respective maintainers: fixes were made available for certifi, Python, pip, uv, sigstore, virtualenv, conda, pixi, manylinux, gh-action-pypi-publish, and many other projects. Vulnerable versions of certifi, pip, and sigstore were patched across all projects and services that the Python Software Foundation stewards, including PyPI, Python’s release process and build bots, and python.org. Containers have been rebuilt with the correct set of certificates.

Advisories, CVE records, banners, and discussion threads were created to alert the community about the work necessary to mitigate this incident and to be a location where relevant discussions and questions were answered by experts. Subscribe to the security-announce mailing list for future advisories about Python, pip, and other exceptional circumstances like this incident. Information about which projects are affected is available on discuss.python.org and around the web for more projects.

How did this incident affect Python as an ecosystem?

This section details the current situation of multiple Python projects. An aspect of positive security culture is “blameless post-portems”, we can only improve the future and the best way to do this is to show up and support the projects and maintainers that you depend on. Please note that all decisions below have historical context and sound reasons for being: Python is about the same age as TLS and is much older than the expectation that your system has up-to-date and secure CA certificate bundles.

The key component requiring a prompt response to the incident from the Python ecosystem is “certifi” and the advisory GHSA-6q48-6c7w-9gw2. Certifi is a repackaging of Mozilla’s CA bundle as a Python package and used by many popular projects including pip and requests. The package only includes root CAs, not intermediates, and therefore needed to remove the root CA from the CA bundle (“emSign Root TLS CA - G3”) to distrust all the impacted intermediate certificates.

Keeping a long story short: certifi is a critical dependency for being able to bootstrap a Python virtual environment into being able to install packages from PyPI. pip bundles its dependencies which includes “Requests” which depends on certifi. Python bundles pip so new virtual environments always contain pip through a mechanism called “ensurepip”.

This arrangement put all three of these components, Python, pip, and certifi, in the critical path for permanently remediating this incident as a user through software updates. There also isn’t an easy answer for older versions of Python which would permanently contain an older version of pip due to compatibility concerns. To address this concern, Python has added a “last-resort” denylist to reject the compromised intermediate certificates during a TLS handshake, fixed versions and more information tracked via CVE-2026-115557 which has been backported to older versions of Python.

What happens next?

We’re continuing our efforts to alert users through advisories and CVE and using banners on the PyPI and discuss.python.org and will be closely monitoring the uptake of patched versions of “certifi” across the ecosystem. The immediate, synchronous response has concluded, now we turn to what should be prioritized so a future incident like a CA compromise might not require as urgent a response from volunteer maintainers.

Right from the beginning of the incident it was identified that TLS handshakes made by browsers were secure, but tools like pip, git, uv weren’t secure. Why is this? Browsers were able to secure their certificate stores quickly and without requiring a software update by using Certificate Revocation Lists (CRL). CRLs allow providers of CAs to provide information to verifiers outside of the software update process, such as when certain certificates should no longer be used due to revocation or compromise.

These technologies may be integrated into Python in the future to provide a quicker and robust response to future incidents involving CA certificates and allow distributions to configure these facilities system-wide. One potential project to watch is upki, which has already been adopted by Ubuntu, provides certificate revocation functionality similar to that of browsers on platforms which previously did not offer such facilities.

November 05, 2026 09:05 AM UTC

LernerPython blog, from Reuven Lerner

Python 3.15 comprehension unpacking: Why you can’t transform items

Python 3.15 comprehension unpacking: Why you can't transform items

The comprehension-unpacking syntax in Python 3.15 is great, but you cannot transform the unpacked elements.

Given:

[[10, 20, 30], [40, 50, 60], [70, 80]]

We can get the individual numbers, but cannot run str on each. For that, we still need a nested comprehension.

The post Python 3.15 comprehension unpacking: Why you can’t transform items appeared first on LernerPython.

November 05, 2026 07:00 AM UTC

Wing Tips

Getting Started with Autonomous Development in Wing Pro 12.1

This Wing Tip describes how to get started with autonomous development in Wing Pro 12.1. You provide a high-level specification for a new application or a major feature. Wing's Tasks Manager fleshes it out, plans the work, splits it into tasks, and keeps a team of AI agents working on them for hours or days, as needed. The agents write the code, unit tests, and documentation, and commit their work in logical units. You answer questions, review results, and ask for revisions in the Tasks tool, which summarizes work done and decisions made.

Autonomous development shifts your focus from coding details to product design and oversight. You explain your vision. Your team of AI agents takes care of the implementation, end to end, far faster than you could develop the code on your own.

This Wing Tip is part of a series of Wing Tips on working with AI agents in Wing. For more detail on each step, see the How-To Autonomous Development with Wing Pro.

Getting Started

We recommend using a reasonably isolated environment, such as a virtual machine, when using AI agents. At the very least, prevent agents from accessing the keys to your server infrastructure, reading sensitive data, or pushing to your main repository.

(1) Install an AI agent and set up an account for it that covers the amount of work you expect to do. For sustained use, a subscription is generally much cheaper than paying for usage on an API key. Claude Code and Codex are best for autonomous development, since they check the safety of the commands they run. Antigravity, Copilot, and Cursor also work but don't provide the same level of safety. See Setting Up an AI Agent and AI Agent Safety for details.

(2) Install git or Mercurial, if you don't already have one of them. Wing uses it to track, review, and commit each task's changes. You don't need to learn how to use it, since Wing and the agents do that for you.

(3) Create a new project with New Project in the Project menu. Choose New directory and enter a Name, then choose your agent under AI Agent and a Python environment under Python. Leave Create a Git repository checked and click Create Project. Setting up the project for an agent turns on the Tasks Manager and lets it commit autonomously:

Setting Up an AI Agent in the New Project Dialog

Shown above: Setting up a new project for an AI agent.

(4) Check the Tasks Manager's settings with the settings icon at the right of the Manager's bar, at the top of the Execute tab in the Tasks tool. The defaults are a good place to start. We recommend leaving automatic commits on, so work can go on while you are away. You can still look over every committed task on the Review tab and ask for changes.

The Tasks Manager settings dialog

Shown above: The Tasks Manager's settings.

(5) Write a specification or tasks on the Plan tab of the Tasks tool, and mark them Ready by clicking their Draft buttons. A task can be anything from a bug report to a high-level specification for a whole application. Describe what the code should do and any technical constraints that matter from the start. The Manager researches the rest, asks you questions, and writes out the tasks:

A specification on the Plan tab, marked Ready

Shown above: A specification on the Plan tab, marked Ready.

You can watch progress on the Execute tab, and click a queue's name to see its session in the AI Agent tool. Problems that agents notice along the way, and follow-up tasks the Manager proposes, appear in the Incoming view on the Plan tab for you to make Ready or dismiss.

(6) Answer questions on the Questions tab when the Manager or a task needs your input. Other tasks keep running while it waits. You can also talk to the Manager directly on the Execute tab, to ask about progress or give it instructions such as "ask me first about anything that changes a file format."

Questions about the specification on the Questions tab

Shown above: Questions about the specification, waiting on the Questions tab.

(7) Review finished work on the Review tab. Each task reports a summary of its changes and the decisions made in it. Use Revise to ask for changes to any decision or the task as a whole. If you are committing work yourself, use Start Review to view and comment on the code, or Commit to commit it without further review.

The Review tab with a task report and the Start Review and Commit links

Shown above: A finished task's report on the Review tab, with automatic commits turned off.

Usage Tips

For better results, build tests and cross-checks into your process:

  • Ask the Manager early on to set up a way for agents to run your application as a user would, so they can see the results of their work.
  • Add tasks at key points that review the code for security flaws, gaps in test coverage, or anything else that matters to you. Problems they find automatically become new tasks.
  • Let the Manager commit on a separate branch, and review that branch before merging it into your main branch.
  • Use the application yourself, and add tasks for the bugs and improvements you find.

If you're unfamiliar with any of this, just tell the Manager what you need, in the Execute tab of the Tasks tool. It can help you set up your development environment and get started with autonomous software development.

Further Reading

See The Tasks Manager Agent in the Wing Pro manual for details. For examples of complete projects built this way, see the Autonomous Development Demos.



That's it for now! We'll be back soon with more Wing Tips for Wing Pro.

As always, please don't hesitate to email support@wingware.com if you run into problems or have any questions.

November 05, 2026 01:00 AM UTC

Graham Dumpleton

A first look at Muru

In my last post I talked about where Muru came from and what I want it to be, but didn't show it doing anything. Time to fix that.

Muru has three ways of running an application. There is a development mode for when you are working on an application on your own machine, a daemon for hosting applications on a server, and a mode for running an application in a container. The same process model sits underneath all three, so what you see while developing is what you get in production. That makes development mode a good place to start, since it is the simplest of the three and still shows off some of what makes Muru different.

Installing it

Muru comes as two pieces. The muructl command is what you install and interact with. It is a single binary written in Rust, with no Python dependencies of its own. The second piece is muru, a runtime library which runs inside your application's Python interpreter. You don't install that one yourself. The first time muructl runs an application on a particular Python version, it fetches the matching runtime library into your user cache directory. Nothing gets installed into your project.

The easiest way to install muructl is as a tool with uv:

uv tool install muructl

Using pipx install muructl, or pip install muructl into some environment, works just as well. Muru is still in alpha, with only pre-releases available, and supports Linux and macOS on CPython 3.11 or later, including the free-threaded builds.

For a project to try it with, create a directory with a virtual environment:

mkdir hello && cd hello
uv init --bare
uv venv

A first application

Not wanting to complicate things by bringing a web framework into it just yet, we will start with a plain WSGI application. Save this as app.py:

def application(environ, start_response):
    body = f"Hello from {environ['PATH_INFO']}\n".encode()
    start_response("200 OK", [("Content-Type", "text/plain"), ("Content-Length", str(len(body)))])
    return [body]

To run it, give muructl run the module and the name of the WSGI callable within it:

muructl run app:application

There is no need to activate the virtual environment first, as muructl will find the .venv directory in the project by itself. Running this the output is:

muru.host[13737]: preloaded app:application in 1 ms
muru.host[13737]: app:application is served as WSGI
muru.host[13737]: worker 0 started as pid 13773
muru.host[13737]: worker 0 (pid 13773) is ready
muru: serving app:application on http://127.0.0.1:8080 with 1 worker of 4 threads
muru: development mode: error pages show tracebacks, nothing is persisted, the run reloads on change and only loopback is served; production runs under the daemon (muructl daemon start)
muru: dashboard on http://127.0.0.1:8081/
muru: admin socket /tmp/muru-501/runs/8080/admin.sock
muru: watching /Users/graham/hello for changes to **/*.py

Even with an application this small, there is more going on than you might expect. A host process was started which imported the application once. A worker process was then forked from the host, meaning that it starts out as a copy of the host with the application already loaded. Finally, the edge, which is the part of Muru that accepts HTTP connections, started listening for requests, passing them through to the worker.

Importing the application once and forking workers from it isn't a new idea, but doing it safely is harder than it looks. If importing the application opens a database connection or starts a background thread, that doesn't survive being forked, and what you get are mysterious problems later on. Muru checks for this when it does the import, before any workers are forked. How that check works deserves a post of its own, so I will come back to it.

Note too that development mode only listens on the loopback interface. It is not meant to be exposed to the outside world, which is what the daemon is for.

What the access log tells you

Open http://127.0.0.1:8080/hello in a browser, or use curl, and a line is added to the terminal for the request:

07:50:49.268 200 GET /hello 18B 0.2ms queue=0.0ms app=0.1ms worker=0 id=01M6JRM25MKE6AS8RXBZ8AS1MG

Much of this is what you would expect from any access log, with the status, the method and path, the size of the response, and the total time. The parts which are a bit different are queue and app. The first is how long the request waited for a worker thread to become free to handle it, and the second is how long your code took.

On a laptop with one user these are tiny numbers and not very interesting. In production they are anything but. A request that takes a long time because your code is slow, and one that takes a long time because every thread was busy and it had to wait its turn, are very different problems. The first means you have code to fix, the second means you don't have enough capacity. If you can't tell the two apart, you will often end up fixing the wrong thing. Being able to see that distinction, right down to individual requests, is a small example of what I meant in the last post about Muru knowing what Python web applications need.

The line also says which worker handled the request, and gives the request a unique id. The same id is returned to the client in an X-Request-ID response header, so a response can be matched up to the log line for it. Anything the application prints is likewise tagged with the worker that wrote it, so when there are multiple workers you can tell which output came from where.

When the application fails

Applications have bugs, so let's add one. Change app.py to:

def application(environ, start_response):
    raise RuntimeError("not today")

Make a request and, because this is development mode, the error page sent back includes the traceback:

500 Internal Server Error
reason: app_error
request id: 01M6JRM54RKE6AS8RXBZ8AS1MG

The application raised an exception.

Traceback (most recent call last):
  File "/Users/graham/hello/app.py", line 2, in application
    raise RuntimeError("not today")
RuntimeError: not today

The same traceback appears in the terminal, tagged with the worker, along with an access log line showing the 500 status and reason=app_error.

If the client says it wants JSON, as an API client would, it instead gets back an application/problem+json response with the same details:

curl -H 'Accept: application/json' http://127.0.0.1:8080/

{"type":"about:blank","title":"Internal Server Error","status":500,"detail":"The application raised an exception.","reason":"app_error","request_id":"01M6JRM553KE6AS8RXBZ8AS1MG","traceback":"Traceback (most recent call last):\n ..."}

Either way, the worker itself is fine. An exception raised by your code results in a failed request, but never a dead process.

Reloading is a release

You may have noticed that I didn't restart anything after changing app.py. That is because development mode watches the project directory, and when a .py file changes it reloads automatically. This is on by default.

Reloading on code changes is something every development server does, so on its own it isn't remarkable. What is different is how Muru does it. Rather than restarting the worker in place, a reload starts a whole new host and worker beside the existing ones. Only once the new worker is ready is traffic switched across to it, after which the old one is shut down. The terminal shows it happening:

muru: reloading, app.py changed
muru.host[13786]: preloaded app:application in 2 ms
muru.host[13786]: app:application is served as WSGI
muru.host[13786]: worker 0 started as pid 13787
muru.host[13786]: worker 0 (pid 13787) is ready
muru: switched to release r2
muru.host[13737]: received SIGTERM, stopping

Each set of host and workers is a release, numbered as they are created. Where this gets interesting is when the change you make breaks things badly enough that the application can't even be imported. Add import nosuchmodule to the top of app.py and save it:

muru: reloading, app.py changed
muru.host[13858]: cannot load app:application: ModuleNotFoundError: No module named 'nosuchmodule'
muru: release r3 failed to start: the host exited with exit status: 1 before its workers were ready; release r2 keeps serving

The new release failed to start, so traffic never moved to it, and the previous release carried on serving requests the whole time. Fix the code and save again, and the next release starts and takes over. The failed attempt still counts as a release, so this will be r4.

In development mode this mostly saves you from staring at a dead server after a typo. The reason I find it worth dwelling on is that it is exactly the same mechanism the daemon uses when deploying a new version of an application to a server. There, a new release starts beside the old one, traffic moves across once it is ready, and if it never becomes ready the old release just keeps serving. That is the basis of zero-downtime deploys in Muru, and of things like blue/green deploys, canaries and rolling back to a previous release. Development mode is giving you a small taste of that every time you save a file.

Real frameworks

Nothing about this is specific to hand-written WSGI applications. For a Flask application you point muructl run at the Flask application object:

muructl run app:app

and for Django at the WSGI application it already generates for you:

DJANGO_SETTINGS_MODULE=mysite.settings muructl run mysite.wsgi:application

Neither framework needs to know anything about Muru.

What's next

That covers the basics of running an application in development mode. Rather than walk through every option, the posts which follow will concentrate on the places where Muru does things differently to the other ways of hosting a Python web application, still in development mode for now, before moving on to the daemon.

November 05, 2026 12:00 AM UTC

November 04, 2026

Python Morsels

Unnecessary syntax in Python

Unnecessary syntax in Python isn't always unhelpful. An explicit return None, an else, a keyword argument, or a trailing comma can make your code more readable.

Table of contents

  1. Return None
  2. An else-return
  3. An else on a try
  4. Keyword arguments
  5. Unnecessary default values
  6. The dictionary keys method
  7. Explicit string concatenation
  8. Trailing commas
  9. Unnecessary variables
  10. Parentheses
  11. Whitespace
  12. Consider unnecessary syntax

Return None

The None in the return statement below isn't necessary:

def item_or_none(sequence, index):
    try:
        return sequence[index]
    except IndexError:
        return None

Using a bare return statement would have worked:

def item_or_none(sequence, index):
    try:
        return sequence[index]
    except IndexError:
        return

In fact, even no return at all would have worked:

def item_or_none(sequence, index):
    try:
        return sequence[index]
    except IndexError:
        pass

Functions return None by default in Python.

But I find return None more explicit and PEP 8 recommends return None in this case:

Be consistent in return statements. Either all return statements in a function should return an expression, or none of them should. If any return statement returns an expression, any return statements where no value is returned should explicitly state this as return None, and an explicit return statement should be present at the end of the function (if reachable).

Explicitly returning None makes it clear that we're returning None deliberately. The function's return value is significant, unlike most functions that simply return None by default.

An else-return

This else statement is unnecessary:

…

Read the full article: https://www.pythonmorsels.com/unnecessary-syntax/

November 04, 2026 06:00 PM UTC

The Python Coding Stack

The Second Opinion

Subscribe now

stephengruppetta.com

November 04, 2026 02:02 PM UTC

death and gravity

pyinfra – simple infrastructure automation in Python

Hi, I'm Adrian, and today I will live demo pyinfra, an infrastructure automation tool.

We will deploy two websites – a landing page and a Python web app – first on a local virtual machine, and then on two remote servers, using only Python code.

Note

This is the edited transcript of a talk I gave at a local DevOps meetup. While part of the "live" aspect is lost in writing, I think it's a pretty decent pyinfra tutorial, so you are welcome to follow along at home.

Contents

About me #

Before we talk about pyinfra, I should introduce myself.

I worked at a Fortune 10 company for most of my career, where I did site reliability engineering for the reverse proxy behind some really big websites and two thousand smaller ones, and where I built a data-lake-plus-feature-store for an ML organization.

In my spare time, I write about Python on this website, and I maintain the creatively named reader, a Python feed reader library and web app. Lately I've been working on a hosted version of reader, which is how I ended up using pyinfra.

What is pyinfra? #

pyinfra is a mostly command-line tool that turns Python code into shell commands and runs them on many servers over SSH, in parallel. Think Ansible, but simpler.

deploy.py
for user in ['admin', 'alice', 'bob']:
    server.user(user)

files.put('files/hello.txt', '/etc/motd')

if host.data.get('install_postgres'):
    apt.install('postgresql')

Why use pyinfra? #

I chose pyinfra because, like Ansible, it is declarative – you describe what a server should look like, and pyinfra makes it so – and idempotent – nothing happens if the server is already in the desired state.

Unlike Ansible, there's no YAML – deploys (playbooks) are Python code, which means you can do anything Python code can do: you can have fors and ifs, you can have code completion and type checking, and you can even import libraries.

Of interest to people using Ansible at larger scales, pyinfra is also faster; in my limited, small-scale experience, it was faster than Ansible with the Mitogen plugin.

In fairness to Ansible, while pyinfra has been around for 10 years, it has fewer built-in operations (modules) and a smaller ecosystem, although that's partially offset by how easy it is to write custom operations.

When to use pyinfra? #

pyinfra use cases follow from what it does – configuring servers over SSH.

As such, pyinfra is good for boring, cloud-agnostic architectures, from lots of servers with different roles all the way down one-computer programming, where it still makes deploys reproducible and prevents drift.1

It is pretty easy to learn too, so it's a good fit for self-hosting and home servers, and even if you don't use it for configuration management, pyinfra gives you a convenient way of running ad-hoc commands on fleets of servers.2

Simplicity comes with limitations: fully-automated pipelines are extra work, there's no built-in pull mode, and care is needed if multiple people run deploys at the same time. Nevertheless, you can get quite far with well-structured deploys, and there's a Python API for building stuff on top if you really need to.

The demo #

I'll try to focus on the interesting parts, but you can find the entire code on GitHub.

At the end of the demo, we'll have two websites, a landing page and a Python web app, deployed in two stages – a local virtual machine for beta:

.--- pyinfra.local ---.- app.pyinfra.local -.
|                   nginx                   |
|   static website    :   Python web app    |
'-------------------------------------------'

...and two separate servers for prod:

.--- pyinfra.a.c ----. .- app.pyinfra.a.c --.
|       nginx        | |       nginx        |
|   static website   | |   Python web app   |
'--------------------' '--------------------'
Here's a sneak peek of the final deploy:
infra/deploy.py
apt.update(name="apt update", cache_time=24 * 3600)

apt.packages(['nginx', 'python3-certbot-nginx'], name="install nginx")

disable_default = files.link(
    name="disable default site",
    path='/etc/nginx/sites-enabled/default',
    present=False,
)
server.service(
    name="reload nginx",
    service='nginx',
    reloaded=True,
    _if=disable_default.did_change,
)

if host.data.get('home_domain'):
    local.include('tasks/home.py')
if host.data.get('app_domain'):
    local.include('tasks/app.py')

Operations and facts #

First, we should get familiar with two basic pyinfra concepts.

Operations change the server in some way; most are declarative and idempotent, and you use them to ensure state. A Python file containing operations is called a deploy.

Here's a simple deploy with one operation that says "I want hello.txt to exist":

examples/create_file.py
from pyinfra.operations import files

files.file('hello.txt', present=True)

If we run this deploy (below, on my @local machine), we see a list of all the operations and on which hosts they will produce changes, and are asked to approve them:

$ pyinfra @local examples/create_file.py
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
    [@local] Connected

--> Preparing operation files...
    Loading: examples/create_file.py
    [@local] Ready: examples/create_file.py

--> Detected changes:
    Operation                              Change       Conditional Change
    files.file (hello.txt, present=True)   1 (@local)   -

    ...
    Press enter to execute...

If we do, we see the operations running and their outcomes:

    Press enter to execute... [Enter]
--> Beginning operation run...
--> Starting operation: files.file (hello.txt, present=True)
    [@local] Success

--> Results:
    Operation                              Hosts   Success   Error   No Change
    files.file (hello.txt, present=True)   1       1         -       -
    Grand total                            1       1         -       -

--> Disconnecting from hosts...

And now hello.txt exists:

$ ls hello.txt
hello.txt

Since files.file is idempotent, nothing happens if we run the deploy again:

$ pyinfra @local examples/create_file.py
...
--> Detected changes:
    Operation                              Change   Conditional Change
    files.file (hello.txt, present=True)   -        -
...
--> Beginning operation run...
--> Starting operation: files.file (hello.txt, present=True)
    [@local] No changes
...

Facts gather data about the target host; you use them to control operations.

Here's a deploy that installs the Apache web server:

examples/install_apache.py
from pyinfra import host
from pyinfra.facts.server import LinuxName
from pyinfra.operations import server

if host.get_fact(LinuxName) in ['Ubuntu', 'Debian']:
    server.packages('apache2')
else:
    server.packages('httpd')

The package is usually called httpd, but Debian-based distributions call it apache2, so we use the LinuxName fact to detect where we are and pass the right name to the server.packages operation. My machine is a Mac, so the httpd branch is chosen:3

$ pyinfra @local examples/install_apache.py --dry
...
--> Detected changes:
    Operation                 Change       Conditional Change
    server.packages (httpd)   1 (@local)   -
...

Where are the commands? #

Before we look at debug output, let's see how pyinfra operations are implemented.

Here's a simplified version of the files.file implementation:

@operation()
def file(path: str, present: bool = True):
    info = host.get_fact(File, path=path)

    if info is None and present:
        yield f"touch {quote(path)}"

    elif info and not present:
        yield f"rm -f {quote(path)}"

The structure is similar for all operations: read the host state using facts, diff it against the desired state, and return commands to reach the desired state if they don't match.

Here's another deploy that says "I want hello.txt to not exist":

examples/delete_file.py
from pyinfra.operations import files

files.file('hello.txt', present=False)

With --debug, we can see the commands being run during each execution phase.

During the prepare phase (before approval), pyinfra runs the deploy code once per host to load operations, and then runs each operation to see if it returns commands, without running the commands themselves (this is how "detected changes" works).

Running operations runs facts, which results in read-only commands being run:

$ pyinfra @local examples/delete_file.py --debug
...
--> Preparing operation files...
    Loading: examples/delete_file.py
    [pyinfra.api.operation] Adding operation, {'files.file'}, ...
    [pyinfra.api.facts] Getting fact: files.File (path=hello.txt)
[pyinfra.connectors.local] --> Running command on localhost: sh -c '! (test -e hello.txt || test -L hello.txt ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' hello.txt 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' hello.txt || ls -ld hello.txt )'
    [pyinfra.api.facts] [@local] Loaded fact files.File (path=hello.txt)
    [@local] Ready: examples/delete_file.py

--> Detected changes:
    Operation                               Change       Conditional Change
    files.file (hello.txt, present=False)   1 (@local)   -

    ...
    Press enter to execute...

During the execute phase, pyinfra runs the previously loaded operations again, but this time it runs the commands returned by them too:

    Press enter to execute... [Enter]
--> Beginning operation run...
--> Starting operation: files.file (hello.txt, present=False)
    [pyinfra.api.facts] Getting fact: files.File (path=hello.txt)
[pyinfra.connectors.local] --> Running command on localhost: sh -c '! (test -e hello.txt || test -L hello.txt ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' hello.txt 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' hello.txt || ls -ld hello.txt )'
    [pyinfra.api.facts] [@local] Loaded fact files.File (path=hello.txt)
[pyinfra.connectors.local] --> Running command on localhost: sh -c 'rm -f hello.txt'
    [@local] Success
...

Inventory + nginx setup #

So far we've been using my local machine as target host, but we want beta to be as similar to prod as possible, so I've set up an Ubuntu virtual machine with Vagrant.

Also, you will typically use an inventory file containing groups of hosts instead of passing individual hosts directly to the CLI. For our demo, we need just one group:

infra/inventories/beta.py
# group_name = [host1, host2, ...]

my_hosts = ['pyinfra.local']

# you can have more groups:
#
# db_servers = ['db-1.net', 'db-2.net']
# web_servers = [i.hostname for i in aws.ec2.list_instances(asg='my-asg')]

Since the inventory is a Python file, groups don't have to be list literals – you could, for example, import a library and list instances in an autoscaling group dynamically.

Here's how you use an inventory to run arbitrary commands:

$ pyinfra inventories/beta.py exec -- lsb_release -d
...
--> Beginning operation run...
--> Starting operation: server.shell (lsb_release -d)
[pyinfra.local] Description: Ubuntu 26.04.1 LTS
    [pyinfra.local] Success
...

We'll use nginx for both of our websites, which means we need to install it first.

Here's what the main deploy looks like (minus imports):

infra/deploy.py
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
apt.update(name="apt update", cache_time=24 * 3600)

apt.packages('nginx', name="install nginx")

disable_default = files.link(
    name="disable default site",
    path='/etc/nginx/sites-enabled/default',
    present=False,
)
server.service(
    name="reload nginx",
    service='nginx',
    reloaded=True,
    _if=disable_default.did_change,
)

We update the package cache, we install nginx, we disable the default "Welcome to nginx!" site, and then we reload the nginx config. To avoid unnecessary reloads, we use the _if global argument to reload only if the previous operation actually ran.

In theory, we could use a fact to gate the last two operations instead:

apt.packages(...)
if host.get_fact(Link, '/etc/nginx/sites-enabled/default'):
    files.link(...)
    server.service(...)

...but that if happens in the prepare phase, when nginx (and the symlink) hasn't been installed yet, so they would be skipped on the first run.

If we run the deploy, no change is detected for "disable default site" (no symlink yet):

$ pyinfra inventories/beta.py deploy.py
...
--> Detected changes:
    Operation              Change              Conditional Change
    apt update             1 (pyinfra.local)   -
    install nginx          1 (pyinfra.local)   -
    disable default site   -                   -
    reload nginx           -                   1 (pyinfra.local)

    Detected changes may not include every change pyinfra will execute.
    Hidden side effects of operations may alter behaviour of future operations,
    this will be shown in the results. The remote state will always be updated
    to reflect the state defined by the input operations.

    Press enter to execute... [Enter]
...
--> Results:
    Operation              Hosts   Success   Error   No Change
    apt update             1       1         -       -
    install nginx          1       1         -       -
    disable default site   1       1         -       -
    reload nginx           1       1         -       -
...

...but the operation does run, as the prophecy foretold ("side effects of operations may alter behaviour of future operations").

Liking this so far? Here's another article you might like:

Write an SQL query builder in 150 lines of Python!

Static site + code reuse #

With nginx installed, we can proceed to the static home page.

We'll put the operations in a separate file that we include in the main deploy, which in addition to keeping code tidy, allows specific tasks to be conditional:

infra/deploy.py
23
24
if host.data.get('home_domain'):
    local.include('tasks/home.py')

Host data is arbitrary data associated with hosts in the inventory file:

infra/inventories/beta.py
# group_name = [(host1, {'some custom': 'host data'}), ...]

my_hosts = [
    ('pyinfra.local', {
        'home_domain': 'pyinfra.local',
    }),
]  # fmt: skip

The home page deploy itself is pretty straightforward:

infra/tasks/home.py
 4
 5
 6
 7
 8
 9
10
11
12
13
from modules import nginx

sync_files = files.sync(
    name="sync files",
    src='../home',
    dest='/opt/home',
    delete=True,
)

nginx.site('templates/nginx/home.conf', host.data.home_domain)

There's one operation that synchronizes files from a local directory to a remote one, and another operation that sets up an nginx site using the domain from host data.

Except nginx.site isn't actually an operation, but a packaged deploy; besides the function definition that makes it reusable, it looks like any other deploy:

infra/modules/nginx.py
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
@deploy("nginx site")
def site(template, domain, **context):

    write_site = files.template(
        name="write",
        src=template,
        dest=f'/etc/nginx/sites-available/{domain}',
        domain=domain,
        **context,
    )
    enable_site = files.link(
        name="enable",
        path=f'/etc/nginx/sites-enabled/{domain}',
        target=f'/etc/nginx/sites-available/{domain}',
    )

    server.service(
        name="reload",
        service='nginx',
        reloaded=True,
        _if=any_changed(write_site, enable_site),
    )

files.template uses Jinja, so we can have a base template for all HTTP sites:

infra/templates/nginx/http.conf
server {
    listen 80;
    listen [::]:80;
    server_name {{ domain }};

    {% block server %}{% endblock %}
}

...that we then extend with site-specific configuration:

infra/templates/nginx/home.conf
{% extends 'templates/nginx/http.conf' %}

{% block server -%}

    location / {
        root /opt/home;
    }

{%- endblock %}

Passing --diff shows the rendered template, which is pretty neat:

$ pyinfra inventories/beta.py deploy.py --diff
...
--> Preparing operation files...
    Loading: deploy.py
    [pyinfra.local] Will create /etc/nginx/sites-available/pyinfra.local
      @@ -0,0 +1,9 @@
      + server {
      +     listen 80;
      +     listen [::]:80;
      +     server_name pyinfra.local;
      +
      +     location / {
      +         root /opt/home;
      +     }
      + }

    [pyinfra.local] Ready: deploy.py

--> Detected changes:
    Operation                             Change              Conditional Change
    apt update                            -                   -
    install nginx                         -                   -
    disable default site                  -                   -
    reload nginx                          -                   1 (pyinfra.local)
    tasks/home.py | sync files            1 (pyinfra.local)   -
    tasks/home.py | nginx site | write    1 (pyinfra.local)   -
    tasks/home.py | nginx site | enable   1 (pyinfra.local)   -
    tasks/home.py | nginx site | reload   -                   1 (pyinfra.local)
...

And now we have a static site!

browser window showing http://pyinfra.local:2080

Web app + secrets #

The web application is a simple Flask app that we can install with pip:

$ tree app
app
├── app
│   ├── __init__.py
│   ├── static
│   │   └── style.css
│   └── templates
│       └── index.html
└── pyproject.toml

We serve it using a traditional Gunicorn being nginx on a Unix socket setup:

   :80, :443
       |
.-------------.
|    nginx    |  groups: app-socket, www-data
'-------------'
       |
    app.sock     groups: app-socket
       |
.- Gunicorn --.
|   web app   |  groups: app-socket, app
'-------------'

The deploy for it is slightly bigger, but hopefully still straightforward.

We have operations to create users and groups:

infra/tasks/app.py
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
server.group('app', name="create app group")
server.group('app-socket', name="create socket group")
server.user(
    name="create user",
    user='app',
    group='app',
    groups=['app-socket'],
    home="/var/lib/app",
    shell="/usr/sbin/nologin",
    system=True,
)
server.user(
    name="nginx socket access",
    user='www-data',
    groups=['app-socket'],
    append=True,
)

...to create a virtual environment with our app and its dependencies:

infra/tasks/app.py
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apt.packages('python3-venv', name="install venv")
pip.venv('/opt/app', 'python3', name="create venv")
pip.packages(['gunicorn'], virtualenv='/opt/app', name="install deps")

sync_files = files.sync(
    name="sync app files",
    src='../app',
    dest='/opt/app/src',
    delete=True,
    exclude_dir=['**/__pycache__'],
)
install_app = server.shell(
    "/opt/app/bin/pip install /opt/app/src",
    name="install app",
    _if=sync_files.did_change,
)

...to create and start a systemd service for Gunicorn:

infra/tasks/app.py
56
57
58
59
60
61
62
63
64
65
66
67
68
write_service = files.template(
    name="write service",
    src='templates/systemd/app.service',
    dest='/etc/systemd/system/app.service',
)
systemd.service(
    name="enable service",
    service='app.service',
    enabled=True,
    restarted=True,
    daemon_reload=True,
    _if=any_changed(install_app, write_config, write_service),
)

...and finally to configure nginx:

infra/tasks/app.py
70
nginx.site('templates/nginx/app.conf', host.data.app_domain)

In addition, Flask needs a secret key, which usually comes from a config file:

infra/tasks/app.py
47
48
49
50
51
52
53
54
write_config = files.template(
    name="write config",
    src='templates/config.py.jinja',
    dest='/etc/opt/app/config.py',
    group='app',
    mode=640,
    get_secret=secrets.get_secret,
)

The question is, where do we store the secret key before writing the config?

The simple (and wrong) solution is to hardcode it in the deploy or in the template; unencrypted-secrets-in-version-control issue aside, we'd also like the secrets to be different between beta and prod.

As I said, a deploy is a Python file, so we can do anything: we can get them from a password manager, a security key, or even an encrypted file. To illustrate the point, here's a simple function that reads named secrets from an unencrypted JSON file:4

infra/modules/secrets.py
import json
from pyinfra import host

secrets_path = 'secrets.json'

def get_secret(name, *, stage=None):
    if not stage:
        stage = host.data.stage

    with open(secrets_path) as f:
        data = json.load(f)

    if secret := data.get(stage, {}).get(name):
        return secret

    raise RuntimeError(f"secret {name!r} for stage {stage!r} not found")

The stage comes from host data, this time not in the inventory, but in a group data file that applies to all hosts in a group (beta, the inventory name, is an implicit group):

infra/group_data/beta.py
stage = 'beta'

The config template looks like this:

infra/templates/config.py.jinja
SECRET_KEY = {{ '{!r}'.format(get_secret('app_secret_key')) }}

Deploying looks the same, only with way more operations:

$ pyinfra inventories/beta.py deploy.py
...
--> Results:
    Operation                             Hosts   Success   Error   No Change
    ...
    Grand total                           23      15        -       8
...

And now we have a web app:

browser window showing http://app.pyinfra.local:2080

HTTPS + prod #

That's a nice looking web app, but it would be even nicer if it were not "not secure". This requires HTTPS, which requires a TLS certificate.

Fortunately, those are easy to get these days – Let's Encrypt provides free certificates, and Certbot will request, install, and renew them automatically.

Unfortunately, there's no built-in Certbot operation, so we have to write a custom one:

infra/modules/certbot.py
 8
 9
10
11
12
13
14
@operation()
def certificate(domain):

    if domain in host.get_fact(Certificates):
        return

    yield f"certbot certonly -d {quote(domain)} -n --nginx --agree-tos"

Like in the initial example, we read the host state using a fact and then return commands if it doesn't match the desired state. The fact is custom too:

infra/modules/certbot.py
17
18
19
20
21
22
23
24
25
class Certificates(FactBase):
    default = dict
    requires_command = "certbot"

    def command(self):
        return "certbot certificates"

    def process(self, output):
        return {c['name']: c for c in _parse_certbot_certificates(output)}

And we already have a convenient place where we can use the operation:

infra/modules/nginx.py
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
@deploy("nginx site")
def site(template, domain, https=None, **context):
    reload_ops = []

    if https is None:
        https = host.data.get('nginx_https')
    if https:
        reload_ops.append(certbot.certificate(domain, name="cert"))

    write_site = files.template(
        name="write",
        src=template,
        dest=f'/etc/nginx/sites-available/{domain}',
        base=f"templates/nginx/http{'s' if https else ''}.conf",
        domain=domain,
        **context,
    )

Also unfortunately, Certbot needs a resolvable domain name to work, so we can't test this using the beta virtual machine... but we can always test in prod. ( ≖‿  ≖ )

Here's a new inventory with two hosts, each with its own domain and HTTPS enabled:

infra/inventories/prod.py
my_hosts = [
    ('pyinfra.andgravity.com', {
        'home_domain': 'pyinfra.andgravity.com',
    }),
    ('app.pyinfra.andgravity.com', {
        'app_domain': 'app.pyinfra.andgravity.com',
    }),
]  # fmt: skip
infra/group_data/prod.py
stage = 'prod'
nginx_https = True

I say we just go for it:

$ pyinfra inventories/prod.py deploy.py --yes
... stuff happens ...

One minute later, there are ✨ two new websites ✨ on the internet:5

browser window showing https://pyinfra.andgravity.com
browser window showing https://app.pyinfra.andgravity.com

Questions? #

Can I use pyinfra from an interactive Python shell? #

Yes, although it doesn't look exactly like a deploy and there's a bit of boilerplate, which you can reduce by using (undocumented) CLI code to e.g. load inventories.

Can I use pyinfra to provision cloud resources? #

No, you'd use something like Terraform for that.

Based on this 2020 comment from the author and on pyinfra's evolution since, it's unlikely it will ever grow beyond "commands over SSH", which is a feature IMHO.

If I may editorialize further, maybe if you had fewer resources you wouldn't need an automated way to provision them; here's some propaganda for you:

Can you use pyinfra with cloud inventories? #

Yes, kinda.

As mentioned when introducing inventories, you can use your cloud's Python SDK to list hosts directly in the inventory file (shouldn't take more than twenty lines of code).

If you use Terraform, there's also a @terraform connector6 that allows using an output variable as inventory (pyinfra @terraform/name ...).

Anyway, that's it for now.

Learned something new today? Share it with others, it really helps!

Want to know when new articles come out? Subscribe via email to get new stuff straight to your inbox!

  1. And you're bound to have extra dev and pre-prod machines anyway. [return]

  2. Say you have a memory leak during a high traffic period and need to reload stuff once in a while until you can find the root cause. [return]

  3. Deploy deploy code runs once per host, so you could have different branches on different hosts within the same run. [return]

  4. Which you absolutely should not do, use keyrings.cryptfile instead. [return]

  5. Well, not anymore, there's no point in paying to host this :) [return]

  6. I didn't talk about them in the article, but connectors execute commands and/or generate inventories; we've used the default @ssh connector and @local. [return]

November 04, 2026 01:45 PM UTC

Python Software Foundation

Joshua Mcshannon Joins the PSF as Infrastructure Engineer

The Python Software Foundation (PSF) is excited to announce our newest staff member and Infrastructure Engineer, Joshua Mcshannon, reporting to Director of Engineering Jacob Coffee. Joshua brings over 16 years of technical expertise from various backgrounds including bootstrapped startups and big tech. Most recently he built an AI-integrated learning platform for cybersecurity students and taught secure web application development along with DevOps/platform engineering.

Joshua at an outdoor fall festival with a stroller

Joshua will focus on reducing operational complexity and toil of the PSF infrastructure, optimizing to better serve the community at large and internal PSF platforms that CPython core developers and staff members rely on daily; this includes services like python.org, PyPI, and infrastructure behind PyCon US. With Joshua on the team, the PSF will gain much needed redundancy for on-call rotations as well as capacity for new infrastructure projects.

“We put people on the moon with a Ti-82, I’m sure we can figure something out.” 

Joshua has already shown great initiative in the work that the PSF has, diving deep into our Kubernetes infrastructure and preparing improvements to keep costs low. Away from the keyboard you’ll find him at a local show, on a nature trail, or at a flea market with his family. Say hello when you see Joshua in the repos, issue trackers, mailing lists, and on discuss.python.org!

About the Python Software Foundation

The Python Software Foundation (PSF) is a non-profit whose mission is to promote, protect, and advance the Python programming language, and to support and facilitate the growth of a diverse and international community of Python programmers. The PSF supports the Python community using corporate sponsorships, grants, and donations. Are you interested in sponsoring or donating to the PSF so it can continue supporting Python and its community? Check out our sponsorship program, donate directly here, or contact our team!

November 04, 2026 08:59 AM UTC

LernerPython blog, from Reuven Lerner

Python set comprehension: Unique items from nested iterables

Python set comprehension: Unique items from nested iterables

In Python 3.15, get the unique elements in a list of iterables with the new in-comprehension unpacking syntax to create a set comprehension:

x = [[10, 20], [20, 30, 40], [40, 50, 60]]

{*n for n in x} # returns {10, 20, 30, 40, 50, 60}

The post Python set comprehension: Unique items from nested iterables appeared first on LernerPython.

November 04, 2026 07:00 AM UTC

Python GUIs

Change color in QProgressBar delegate — How to dynamically change QProgressBar colors in a QTableView delegate

How can I change the color of a QProgressBar delegate in a QTableView when a process starts or stops? I'm using a delegate to show an indeterminate progress bar (with setRange(0, 0)), and I want it to be green when running and red when stopped, but replacing the delegate doesn't seem to redraw the color.

The issue here comes from how delegates and persistent editors interact. When you call openPersistentEditor, Qt creates the editor widget and keeps it alive. Replacing the delegate on a row doesn't automatically close and recreate those existing editors — so even though you're assigning a new ProgressBarDelegate with a different color, the old progress bar widget is still sitting there unchanged.

The solution is to store the color information in the model data and have a single delegate read that color when it creates or updates the editor. This way, when the data changes, the delegate can update the editor's stylesheet accordingly.

Let's walk through how to set this up properly.

Store state in the model, not the delegate

Rather than creating a new delegate every time a button is clicked, store the running/stopped state in the model itself. You can use a custom Qt.ItemDataRole (or just Qt.UserRole) to hold the color or state for each row. The delegate then reads this value when painting or configuring the editor.

This keeps your architecture clean: the model holds the data, and the delegate is responsible only for presentation.

Use setEditorData to update the progress bar

The setEditorData method on your delegate is called whenever the model data changes for an index that has a persistent editor. This is where you update the progress bar's stylesheet based on the current state.

Here's how to write a ProgressBarDelegate that reads a color from the model:

python 
class ProgressBarDelegate(QtWidgets.QStyledItemDelegate):
    def paint(self, painter, option, index):
        if index.column() == 2:
            if (
                isinstance(self.parent(), QtWidgets.QAbstractItemView)
                and self.parent().model() is index.model()
            ):
                self.parent().openPersistentEditor(index)
        QtWidgets.QStyledItemDelegate.paint(self, painter, option, index)

    def createEditor(self, parent, option, index):
        editor = QtWidgets.QProgressBar(parent)
        editor.setRange(0, 0)
        editor.setTextVisible(False)
        return editor

    def setEditorData(self, editor, index):
        color = index.data(QtCore.Qt.ItemDataRole.UserRole)
        if color:
            editor.setStyleSheet(
                f"QProgressBar::chunk {{background-color: {color}; "
                f"width: 20px; margin: 0.5px;}}"
            )

    def updateEditorGeometry(self, editor, option, index):
        editor.setGeometry(option.rect)

The delegate no longer stores any color itself. Instead, setEditorData looks up the color from Qt.UserRole on the model index and applies it to the stylesheet. Every time you call setData on the model with Qt.UserRole, Qt will automatically call setEditorData on the persistent editor for that index.

Update the model when the button is clicked

In your button click handler, instead of creating a new delegate, just update the model data:

python 
def on_button_clicked(self, index):
    model = self.table_view.model()
    progress_index = model.index(index.row(), 2)

    current_state = progress_index.data(QtCore.Qt.ItemDataRole.DisplayRole)

    if current_state is None or current_state == 0:
        model.setData(progress_index, "green", QtCore.Qt.ItemDataRole.UserRole)
        model.setData(progress_index, 1, QtCore.Qt.ItemDataRole.DisplayRole)
    else:
        model.setData(progress_index, "red", QtCore.Qt.ItemDataRole.UserRole)
        model.setData(progress_index, 0, QtCore.Qt.ItemDataRole.DisplayRole)

The DisplayRole tracks the state (1 for running, 0 for stopped), and UserRole holds the color string. When setData is called, the model emits dataChanged, which triggers setEditorData on the persistent editor — and the progress bar updates its color.

Complete working example

Here's the full example putting everything together. The QTableView shows a list of paths, each with a Start/Stop button and an indeterminate progress bar. Clicking the button toggles the progress bar between green and red using a single delegate that reads color from the model.

python 
import sys

from PySide6 import QtCore, QtGui, QtWidgets


class ProgressBarDelegate(QtWidgets.QStyledItemDelegate):
    def paint(self, painter, option, index):
        if index.column() == 2:
            if (
                isinstance(self.parent(), QtWidgets.QAbstractItemView)
                and self.parent().model() is index.model()
            ):
                self.parent().openPersistentEditor(index)
        QtWidgets.QStyledItemDelegate.paint(self, painter, option, index)

    def createEditor(self, parent, option, index):
        editor = QtWidgets.QProgressBar(parent)
        editor.setRange(0, 0)
        editor.setTextVisible(False)
        return editor

    def setEditorData(self, editor, index):
        color = index.data(QtCore.Qt.ItemDataRole.UserRole)
        if color:
            editor.setStyleSheet(
                f"QProgressBar::chunk {{background-color: {color}; "
                f"width: 20px; margin: 0.5px;}}"
            )

    def updateEditorGeometry(self, editor, option, index):
        editor.setGeometry(option.rect)


class PushButtonDelegate(QtWidgets.QStyledItemDelegate):
    clicked = QtCore.Signal(QtCore.QModelIndex)

    def paint(self, painter, option, index):
        if (
            isinstance(self.parent(), QtWidgets.QAbstractItemView)
            and self.parent().model() is index.model()
        ):
            self.parent().openPersistentEditor(index)
        QtWidgets.QStyledItemDelegate.paint(self, painter, option, index)

    def createEditor(self, parent, option, index):
        button = QtWidgets.QPushButton(parent)
        button.clicked.connect(lambda *args, ix=index: self.clicked.emit(ix))
        return button

    def setEditorData(self, editor, index):
        editor.setText("Start/Stop")

    def updateEditorGeometry(self, editor, option, index):
        editor.setGeometry(option.rect)


class MainWindow(QtWidgets.QMainWindow):
    def __init__(self, parent=None):
        super().__init__(parent)
        self.setFixedSize(600, 300)

        self.table_view = QtWidgets.QTableView()
        layout = QtWidgets.QHBoxLayout()
        layout.addWidget(self.table_view)
        cw = QtWidgets.QWidget()
        cw.setLayout(layout)
        self.setCentralWidget(cw)

        self.init_model()

        self.button_delegate.clicked.connect(self.on_button_clicked)

    def init_model(self):
        headers = ["Path", "Control", "Progress"]

        self.button_delegate = PushButtonDelegate(self.table_view)
        self.progress_delegate = ProgressBarDelegate(self.table_view)

        self.model = QtGui.QStandardItemModel()
        self.model.setHorizontalHeaderLabels(headers)

        for row in range(5):
            self.model.setItem(
                row, 0, QtGui.QStandardItem(f"some_path{row}")
            )
            self.model.setItem(row, 1, QtGui.QStandardItem())
            self.model.setItem(row, 2, QtGui.QStandardItem())

        self.table_view.setModel(self.model)
        self.table_view.setItemDelegateForColumn(1, self.button_delegate)
        self.table_view.setItemDelegateForColumn(2, self.progress_delegate)
        self.table_view.resizeColumnsToContents()
        self.table_view.horizontalHeader().setSectionResizeMode(
            QtWidgets.QHeaderView.ResizeMode.Stretch
        )

    def on_button_clicked(self, index):
        model = self.table_view.model()
        progress_index = model.index(index.row(), 2)

        current_state = progress_index.data(QtCore.Qt.ItemDataRole.DisplayRole)

        if current_state is None or current_state == 0:
            model.setData(
                progress_index, "green", QtCore.Qt.ItemDataRole.UserRole
            )
            model.setData(
                progress_index, 1, QtCore.Qt.ItemDataRole.DisplayRole
            )
        else:
            model.setData(
                progress_index, "red", QtCore.Qt.ItemDataRole.UserRole
            )
            model.setData(
                progress_index, 0, QtCore.Qt.ItemDataRole.DisplayRole
            )


if __name__ == "__main__":
    app = QtWidgets.QApplication(sys.argv)
    window = MainWindow()
    window.show()
    sys.exit(app.exec())

When you run this, each row has a "Start/Stop" button. Clicking it the first time turns the progress bar green (running). Clicking again turns it red (stopped). Each subsequent click toggles between the two states.

For an in-depth guide to building Python GUIs with PySide6 see my book, Create GUI Applications with Python & Qt6.

November 04, 2026 06:00 AM UTC

Graham Dumpleton

Twenty years of mod_wsgi, and what comes next

Twenty years ago, around this time, the idea for mod_wsgi was taking shape in my head. The coding started in December 2006. I had no idea back then that it would end up being something I would still be maintaining two decades later, or that it would become the thing many people in the Python community know me for.

People tend to either love mod_wsgi or hate it. Those who love it value that it is solid and has been around forever, and that it just keeps running once you have it set up. Those who hate it usually aren't really complaining about mod_wsgi itself, but about Apache httpd. Configuring Apache was never anyone's idea of fun, and mod_wsgi inherits all of that baggage simply by being an Apache module.

Why Apache in the first place

It is easy to forget what the landscape looked like back then. The WSGI specification existed, but there was no standard, well trodden way of hosting a WSGI application in production. The options were things like CGI, FastCGI or SCGI bridges, or mod_python, which wasn't built around WSGI and came with its own set of problems. Nginx wasn't on the radar of most people outside of Russia, and Python WSGI servers like gunicorn were still years away.

What did exist was Apache, and it was everywhere. Pretty much any hosting environment you could get your hands on had Apache installed. Building on top of Apache meant that mod_wsgi could provide a standardised way of hosting Python web applications on infrastructure people already had, and that is the hole it ended up filling.

Living within Apache's rules

The catch is that Apache httpd is even older than mod_wsgi, and its architecture was never designed with something like mod_wsgi in mind. This is particularly the case with daemon mode, where the Python web application runs in separate processes from the Apache worker processes handling incoming requests.

Apache owns the lifecycle of all its child processes. The daemon processes mod_wsgi uses are created by the Apache parent process, and so are bound by how Apache decides to start, stop and restart processes. The number of daemon processes is fixed by the configuration, and changing anything means restarting Apache. If you wanted a process manager that could make smarter decisions about the Python application, such as scaling the number of processes up or down based on load, or rolling out a new version of the application without disrupting requests in flight, you were fighting Apache to do it.

So mod_wsgi did the best it could within those constraints. Over the years there was a long list of things I would have loved to add, but which were never going to be practical while sitting inside Apache. Some of them I tried to approximate, others I simply had to let go.

Why revisit this now

Being on sabbatical, and with full retirement looking more likely, I have the time to come back to the problem and think about how I would do it if I wasn't bound by Apache.

To be frank though, if I had to sit down and write such a system from scratch myself, I wouldn't even try. The scale of what is involved is large, and the coding is the part I find a chore these days. What makes the difference is AI coding tools. They have become good enough at taking an idea and turning it into a working implementation that the coding is no longer my problem. Instead I get to focus on the parts I actually enjoy, which is the architecture, the design, and working out which features are worth having and how they should behave.

Working at that level has turned out to be a lot of fun. I can go from an idea to seeing it work far quicker than I ever could before, and that gratification of seeing an idea come to life is, I suspect, what most of us got into this for in the first place.

So what is Muru

With the free time to think the problem through properly, and AI tools able to take on the heavy lifting of writing the code, I decided to embark on creating a new and more modern system for hosting Python web applications. That system is called Muru.

Muru isn't an Apache module, and it isn't just another WSGI server to run behind something else. The aim is to take everything I learned from twenty years of mod_wsgi, and from watching how people actually deploy Python web applications, and build something that covers the whole job of running a Python web application, from your laptop through to production.

Despite where it comes from, Muru isn't limited to WSGI either. It can also host ASGI applications, as well as gRPC services. More than that, these can be mixed together in the one application, rather than needing to be run as separate deployments. Extensions are available to hand off requests from WSGI to ASGI and vice versa, which makes it easy to bring things like websockets to an existing WSGI application without having to rewrite it.

Beyond what it can host, part of what Muru sets out to do is simply bring in good practice. There are a lot of things that every production deployment needs, but which are left as an exercise for the user to get right. Things like how a new version of an application gets rolled out, how requests get routed to it, how it reports whether it is healthy, and how it restarts without dropping requests. Most people end up reinventing these badly, or not doing them at all. Muru tries to make them the default rather than something you have to bolt on.

The other part, and the bit I think makes Muru different, is that it is built with an intimate knowledge of what Python web applications need. A generic process manager or container platform treats your application as a black box. It can see whether the process is running and how much CPU and memory it is using, but not much more. Python web applications have their own quirks though, whether that is the GIL and how it shapes the choice between processes and threads, how long an application can take to import and become ready, or how memory use can creep up over time. Muru is designed around those realities rather than ignoring them.

That knowledge also extends to metrics. A Python web application can tell you a lot about how it is coping, such as how busy its request handler threads are, how long requests are waiting before being handled, and whether a process is getting into trouble. The idea with Muru is that these aren't just numbers to put on a dashboard after the fact, but signals that feed back into how the application is hosted, driving decisions about capacity, process management and routing.

Muru is still very much in development. I am not going to point to where it lives just yet. If you go looking and find it, that is fine, but at this stage I would rather it was only people with a real need for something like it who were poking at it.

Where these posts are heading

Rather than try and describe everything in one go, I intend to write a series of posts working through what Muru can do. Part of the reason for doing this is to get my own ideas out there about what hosting a Python web application should involve. I am also hoping it might prompt others to share what they think is missing from the current options, because hearing about the problems other people are running into would help me work out where Muru should go next.

As for what the posts will cover, I plan to start with local development, since the same tool you use to host the application in production is also there to help while you are developing it. From there I will look at deploying to a host or virtual machine, including how Muru manages deployments of new versions and handles routing of requests. Further along there is running Muru in containers and on Kubernetes, and all the bits those platforms need from an application for it to behave well, such as health probes, metrics, and draining requests on shutdown.

A word about AI

As has become usual for me of late, a small disclaimer. Muru is being built with heavy use of AI coding tools. I am the one deciding what it should do and how it should be structured, and I am reviewing and steering the result, but most of the code was written by an AI.

I know some people are firmly opposed to AI-written software, and I am not going to argue with that position. If AI involvement rules Muru out for you, then Muru is not for you, and that is okay.

What comes next

Twenty years is a long time to have been working on the one problem, and it seems I am still not done with it. I am not abandoning mod_wsgi, but Muru is my chance to finally build the things I always wanted to and never could. Fingers crossed it turns out to be as useful to others as I hope it will be.

November 04, 2026 12:00 AM UTC

November 03, 2026

TestDriven.io

Taking Website Screenshots with Django, Celery, and Playwright

This tutorial looks at how to build a Django app that takes website screenshots using Playwright and Celery.

November 03, 2026 10:28 PM UTC

Talk Python to Me

#568: Cloudflare Python Workers

Where does your Python code actually run? A server in Virginia? A container in Frankfurt? Pyodide put Python into the browser. Cloudflare put the browser into the server. Combine the two and you get Python Workers, now generally available. My guests are Hood Chatham, Gyeongjae Choi, and Dominik Picheta from Cloudflare. Hood and Gyeongjae maintain Pyodide. Dominik got Python running in Cloudflare Workers in the first place. Your FastAPI app ships as plain Python files. It runs on CPython compiled to WebAssembly, inside V8, at over 350 locations worldwide.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/talk-python-membership-inner-circle'>Talk Python Membership</a><br> <a href='https://talkpython.fm/talk-python-membership-continue-conversation'>Talk Python Membership</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <h2 class="links-heading mb-4">Links from the show</h2> <div><strong>Guests</strong><br/> <strong>Gyeongjae Choi</strong>: <a href="https://www.linkedin.com/in/gyeongjae-choi-b259b0163?featured_on=talkpython" target="_blank" >linkedin.com</a><br/> <strong>Hood Chatham</strong><br/> <strong>Dominik Picheta</strong><br/> <br/> <strong>Pyodide</strong>: <a href="https://pyodide.com?featured_on=talkpython" target="_blank" >pyodide.com</a><br/> <strong>Python Everywhere: The State of Python on WebAssembly - Hood Chatham, Gyeongjae Choi</strong>: <a href="https://www.youtube.com/watch?v=HklunTc9giA" target="_blank" >www.youtube.com</a><br/> <strong>PEP 783</strong>: <a href="https://peps.python.org/pep-0783/?featured_on=talkpython" target="_blank" >peps.python.org</a><br/> <strong>D1 Databases</strong>: <a href="https://www.cloudflare.com/products/d1/?featured_on=talkpython" target="_blank" >www.cloudflare.com</a><br/> <strong>Hyperdrive</strong>: <a href="https://www.cloudflare.com/products/hyperdrive/?featured_on=talkpython" target="_blank" >www.cloudflare.com</a><br/> <strong>R2 Object Storage</strong>: <a href="https://www.cloudflare.com/products/r2/?featured_on=talkpython" target="_blank" >www.cloudflare.com</a><br/> <strong>Queues</strong>: <a href="https://developers.cloudflare.com/queues/?featured_on=talkpython" target="_blank" >developers.cloudflare.com</a><br/> <strong>AI</strong>: <a href="https://www.cloudflare.com/products/workers-ai/?featured_on=talkpython" target="_blank" >www.cloudflare.com</a><br/> <strong>now generally available</strong>: <a href="https://blog.cloudflare.com/python-workers-ga/?featured_on=talkpython" target="_blank" >blog.cloudflare.com</a><br/> <strong>Memory Snapshots for CPython (Python Language Summit 2026\)</strong>: <a href="https://blog.python.org/2026/09/language-summit-2026-memory-snapshots/?featured_on=talkpython" target="_blank" >blog.python.org</a><br/> <strong>Cloudflare Worker Limits</strong>: <a href="https://developers.cloudflare.com/workers/platform/limits/?featured_on=talkpython" target="_blank" >developers.cloudflare.com</a><br/> <strong>Litestream</strong>: <a href="https://litestream.io?featured_on=talkpython" target="_blank" >litestream.io</a><br/> <strong>workerd</strong>: <a href="https://github.com/cloudflare/workerd?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>Are we PEP 783 yet?</strong>: <a href="https://pyodide.github.io/are-we-pep783-yet/?featured_on=talkpython" target="_blank" >pyodide.github.io</a><br/> <strong>The Birth &amp; Death of JavaScript</strong>: <a href="https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript?featured_on=talkpython" target="_blank" >www.destroyallsoftware.com</a><br/> <br/> <strong>Watch this episode on YouTube</strong>: <a href="https://www.youtube.com/watch?v=Kgh9-p7yb9A" target="_blank" >youtube.com</a><br/> <strong>Episode #568 deep-dive</strong>: <a href="https://talkpython.fm/episodes/show/568/cloudflare-python-workers#takeaways-anchor" target="_blank" >talkpython.fm/568</a><br/> <strong>Episode transcripts</strong>: <a href="https://talkpython.fm/episodes/transcript/568/cloudflare-python-workers" target="_blank" >talkpython.fm</a><br/> <br/> <strong>Theme Song: Developer Rap</strong><br/> <strong>🥁 Served in a Flask 🎸</strong>: <a href="https://talkpython.fm/flasksong" target="_blank" >talkpython.fm/flasksong</a><br/> <br/> <strong>---== Don't be a stranger ==---</strong><br/> <strong>YouTube</strong>: <a href="https://talkpython.fm/youtube" target="_blank" ><i class="fa-brands fa-youtube"></i> youtube.com/@talkpython</a><br/> <br/> <strong>Bluesky</strong>: <a href="https://bsky.app/profile/talkpython.fm" target="_blank" >@talkpython.fm</a><br/> <strong>Mastodon</strong>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i> @talkpython@fosstodon.org</a><br/> <strong>X.com</strong>: <a href="https://x.com/talkpython" target="_blank" ><i class="fa-brands fa-twitter"></i> @talkpython</a><br/> <br/> <strong>Michael on Bluesky</strong>: <a href="https://bsky.app/profile/mkennedy.codes?featured_on=talkpython" target="_blank" >@mkennedy.codes</a><br/> <strong>Michael on Mastodon</strong>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i> @mkennedy@fosstodon.org</a><br/> <strong>Michael on X.com</strong>: <a href="https://x.com/mkennedy?featured_on=talkpython" target="_blank" ><i class="fa-brands fa-twitter"></i> @mkennedy</a><br/></div>

November 03, 2026 08:33 PM UTC

PyCoder’s Weekly

Issue #759: Pillow Flaw, JIT Removed, Security, and More (2026-11-03)

#759 – NOVEMBER 3, 2026
View in Browser »

The PyCoder’s Weekly Logo

Critical Flaw in Pillow Allows Code Execution

A critical flaw was found in Pillow, the popular image library. A specially crafted EPS file could result in code execution. Pillow 13.0.0 fixed this and several other issues. This post describes the release and exploit.
SWATI KHANDELWAI

Python’s Experimental JIT to Be Removed

PEP 836 has been withdrawn and the experimental JIT is to be removed from CPython’s main branch. This article talks about the JIT’s journey and what still may be in store.
JOE BROCKMEIER

CodeAF: 78% on DeepSWE with an Open Model

alt

With DeepSeek V4.1 Flash, CodeAF fixed 88 of 113 GitHub issues on DeepSWE. Against Codex and eight other harnesses on one open model, it ranked #1 at the lowest cost per fix. CodeAF is an open-source coding harness built for open models, and works with closed models too. Star on GitHub
AGENTFIELD sponsor

Supply Chain Security for the Age of AI-Speed Exploits

Learn Python supply chain security as AI speeds up attacks: what the LiteLLM backdoor teaches, how to check your setup, and habits that last.
REAL PYTHON

Quiz: Supply Chain Security for the Age of AI-Speed Exploits

REAL PYTHON

Python 3.15.2, 3.14.10, 3.13.18, 3.12.16 and 3.11.18 Released

PYTHON.ORG

Move Past the Chat Window With Codex CLI

If you’re still pasting code and tracebacks into a chat, let an agent work inside your project instead. Codex CLI runs in your terminal, reads your files, runs your tests, and follows the project rules you write in an AGENTS.md file. Over two live days on November 14–15, you’ll build a full Python CLI app this way, from an empty directory to GitHub.
REAL PYTHON

Articles & Tutorials

Google Adopts Pyrefly for Typechecking

Python is one of the primary languages at Google, powering everything from machine learning research to developer infrastructure and data pipelines. Replacing previous type-checking infrastructure with Pyrefly provided significant speedups for engineers and agents alike, while substantially reducing build-system compute resources.
GOOGLEBLOG.COM • Shared by anonymous

Polars 2.0: Streaming by Default and Stricter Types

Polars 2.0 makes lazy queries faster without code changes, but its new defaults can affect existing workflows. This article compares Polars 1.x and 2.0, covering streaming performance, spill-to-disk, changes to join ordering, and stricter type checking that catches silent precision loss.
OPENTEAMS.COM • Shared by Khuyen Tran

Django as a Simple Static Site Builder

Combining three libraries: nanodjango, django-nanopages, and django-distill, you can create a static site generator in just a few lines of code. Why learn another tool when you already know Django?
RADIAC

GPT-6.1 Terra Draws a Python Reading a Book

Explore GPT-6.1 Terra’s Python code, turtle drawing, and responses to small tasks that check modern syntax, release knowledge, focused edits, and made-up functions.
REAL PYTHON

Python vs Go: A Practical Comparison for Developers

This article looks at how Python and Go differ, what surveys say about the devs who use them, and what interviews say about the trade-offs between them.
DOMINIKA STANKIEWICZ

Python Typing Survey 2026 Results

This post summarizes the results of the 2026 Python Typing survey. Type hint use changed somewhat, but the choice of type checker has changed sharply.
FACEBOOK

13 Project Ideas for Intermediate Python Developers

Try 13 Python project ideas for intermediate developers. Build web, GUI, and command-line apps that challenge you and make you a better Pythonista.
REAL PYTHON

Three Ways of Storing and Accessing Lots of Images in Python

Compare three ways of storing images in Python, as PNG files on disk, in LMDB, and in HDF5, with timings for reading and writing lots of images.
REAL PYTHON

NumPy Tutorial: Your First Steps Into Data Science in Python

Follow this NumPy tutorial to learn how to create and manipulate arrays in Python and take your first steps into data science.
REAL PYTHON

Python on My Wrist

This blog post talks about reviving PineTime smartwatch running up-to-date MicroPython and wasp-os.
MICHIEL BEIJEN

Tachyon: Sampling Profiling in Python

Learn how to use Tachyon, Python’s sampling profiler.
RODRIGO GIRÃO SERRÃO

Projects & Code

borb: Read, Create and Edit PDF Files in Python

GITHUB.COM/BORB-PDF

f1-race-replay: Interactive Formula 1 Race Visualisation

GITHUB.COM/IAMTOMSHAW

quarantine: Dead Letter Queue and Replay Debugger

GITHUB.COM/HALCYON-PAST • Shared by Aritro Saha

django-cachalot: Cache Queries With Automatic Invalidation

GITHUB.COM/NORIPYT

fastpybase64: Fast Base64 Encoding and Decoding

GITHUB.COM/SIMDUTF

Events

Weekly Real Python Office Hours Q&A (Virtual)

November 4, 2026
REALPYTHON.COM

Canberra Python Meetup

November 5, 2026
MEETUP.COM

Sydney Python User Group (SyPy)

November 5, 2026
SYPY.ORG

PyCon ES 2026

November 6 to November 9, 2026
PYCON.ORG

Python El Salvador Vuelve

November 7, 2026
PYTHON.ORG.SV

PyCon Chile 2026

November 14 to November 16, 2026
PYCON.CL

Happy Pythoning!
This was PyCoder’s Weekly Issue #759.
View in Browser »

alt

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

November 03, 2026 07:30 PM UTC

Django Weblog

Django security releases issued: 6.1.3, 6.0.10, and 5.2.19

In accordance with our security release policy, the Django team is issuing releases for Django 6.1.3, Django 6.0.10, and Django 5.2.19. These releases address the security issues detailed below. We encourage all users of Django to upgrade as soon as possible.

CVE-2026-108147: Potential memory exhaustion in MultiPartParser via base64-encoded file upload

When using django.http.multipartparser.MultiPartParser, multipart uploads with Content-Transfer-Encoding: base64 could be buffered entirely in memory, regardless of FILE_UPLOAD_MAX_MEMORY_SIZE, before being passed to upload handlers, potentially consuming excessive memory.

This issue has severity "low" according to the Django Security Policy.

Thanks to Ignacy Kujawa for the report.

CVE-2026-111211: Potential log injection via unescaped geometry form values

django.contrib.gis.forms.GeometryField and its widgets logged invalid submitted values directly, allowing control characters (e.g. newlines or ANSI escape sequences) to be written unescaped into logs. This could enable log injection or forgery, letting attackers manipulate log appearance or structure, especially in logs processed by external systems or viewed in terminals.

This issue has severity "low" according to the Django Security Policy.

Thanks to Aaron chen for the report.

CVE-2026-111362: Server-side request forgery via crs member of GeoJSON geometries

Earlier versions of the GeoJSON spec allowed specifying a coordinate reference system (CRS) by URL. Since GDAL still resolves these URLs, any validation, save, or query path where a geometry is instantiated from user input allowed an attacker to submit a malicious CRS that would make a network request as the Django process user, or read a file, or both.

Lookups against spatial fields, the GeometryField form field, and GEOSGeometry now reject GeoJSON geometries with CRS references to external resources. The DataSource and LayerMapping utilities for loading trusted data continue to accept these values.

This issue has severity "moderate" according to the Django Security Policy.

Thanks to Submersion Security Research Team for the report.

CVE-2026-113593: Potential exposure of private data via empty and missing Vary headers

UpdateCacheMiddleware and cache_page() generated the same cache key for a request that omitted a header listed in the response's Vary header and a request that sent that header with an empty value. As a result, a response cached for one of these requests could be served to the other, which could lead to the exposure of private data.

Cache keys now distinguish between absent and empty header values, as required by RFC 9111#section-4.1.

Responses previously cached for requests that omitted a header listed in Vary will no longer be matched, and will be cached again on the next request.

This issue has severity "low" according to the Django security policy.

Thanks to Andrew Gonzalez for the report.

CVE-2026-113733: Password hash disclosure to "view only" admin users with custom user models

Admin users with the view (but not change) permission to a user model were displayed the entire password hash when the model was not AUTH_USER_MODEL, or when AUTH_USER_MODEL inherited its password field from a concrete parent model through multi-table inheritance.

This issue has severity "low" according to the Django Security Policy.

Thanks to Aaron chen for the report.

CVE-2026-114582: Server-side request forgery via SpatialReference string input

SpatialReference passed string input to GDAL, which by default resolves URLs and reads files. This behavior was not documented by Django. Other affected methods included GEOSGeometry.transform(), OGRGeometry.transform(), GDALRaster.transform(), the srs setter of OGRGeometry, and the srs and srid setters of GDALRaster. A URL or file path reaching any of these would issue an outbound request or read a file as the Django process user. The attack vectors could be combined to reveal local file contents by observing outbound requests.

SpatialReference now disables network and file access when parsing string input on GDAL 3.9 and later. As a consequence, file paths and URLs are no longer accepted as SpatialReference inputs. Dictionary references using the DICT: prefix, including those prefixed with ESRI::, are rejected on all GDAL versions.

Full protection requires GDAL 3.9 or later. Integer SRIDs are looked up directly in the EPSG database, so prefer them when the input is untrusted. Or, if strings must be used on lower GDAL versions, compare to an allow list. The new gis.W001 system check warns when the installed version of GDAL is not at least 3.9.

This issue has severity "low" according to the Django security policy.

Thanks to Adraca for the report.

Affected supported versions

Resolution

Patches to resolve the issue have been applied to Django's main, 6.1, 6.0, and 5.2 branches. The patches may be obtained from the following changesets.

CVE-2026-108147: Potential memory exhaustion in MultiPartParser via base64-encoded file upload

CVE-2026-111211: Potential log injection via unescaped geometry form values

CVE-2026-111362: Server-side request forgery via crs member of GeoJSON geometries

CVE-2026-113593: Potential exposure of private data via empty and missing Vary headers

CVE-2026-113733: Password hash disclosure to "view only" admin users with custom user models

CVE-2026-114582: Server-side request forgery via SpatialReference string input

The following releases have been issued

The PGP key ID used for this release is Jacob Walls: 131403F4D16D8DC7

General notes regarding security reporting

As always, we ask that potential security issues be reported via private email to security@djangoproject.com, and not via Django's Trac instance, nor via the Django Forum. Please see our security policies for further information.

November 03, 2026 03:30 PM UTC