skip to navigation
skip to content

Planet Python

Last update: November 18, 2026 07:48 AM UTC

November 18, 2026


Python GUIs

How to Change the Decimal Separator in QDoubleValidator — Use QLocale to control the decimal point format in PyQt6 QLineEdit validation

I'm using QDoubleValidator with a QLineEdit in PyQt6, but the validator allows a comma as the decimal separator instead of a point (dot). How can I force the decimal separator to be a point?

When you use QDoubleValidator in PyQt6, the character it treats as the decimal separator — a comma (,) or a point (.) — is determined by the locale the validator uses. A locale defines how numbers, dates, and currencies are formatted for a particular language and region. In many European countries, for example, the comma is the standard decimal separator (3,14), while in the United States and United Kingdom, the point is used (3.14).

By default, QDoubleValidator picks up your system's locale settings. If your operating system is configured for a locale that uses commas, the validator will accept commas as the decimal separator. To override this and force a point separator, you can create a QLocale and apply it directly to the validator.

Setting a Custom Locale on the Validator

The QLocale class lets you specify a language and country combination that determines number formatting rules. To get a point as the decimal separator, use a locale like English, UnitedStates:

python
from PyQt6.QtCore import QLocale

locale = QLocale(QLocale.English, QLocale.UnitedStates)
validator.setLocale(locale)

Once this locale is set on the validator, the point will be accepted as the decimal separator in your QLineEdit.

One thing to be aware of: even with this locale, the comma may still be accepted as input because the comma serves as the thousands grouping separator in English locales (e.g., 1,000.50). This is expected behavior and generally won't cause problems.

Setting Standard Notation

You can also control whether the validator accepts scientific notation (like 1.5e10) by setting the notation mode. If you only want standard decimal numbers, use:

python
validator.setNotation(QDoubleValidator.StandardNotation)

This prevents users from entering numbers in scientific notation format.

Constructor Arguments

When creating a QDoubleValidator, the three main arguments are:

For example:

python
validator = QDoubleValidator(0.1, 9990, 2)

This creates a validator that accepts numbers between 0.1 and 9990, with up to 2 decimal places.

Be careful with Python syntax here — something like 99.9.0 is not a valid Python number and will raise a SyntaxError.

Complete Working Example

Here's a full example that creates a QLineEdit with a QDoubleValidator configured to use a point as the decimal separator:

python
from PyQt6.QtWidgets import QApplication, QLineEdit
from PyQt6.QtGui import QDoubleValidator
from PyQt6.QtCore import QLocale

app = QApplication([])

lineedit = QLineEdit()
lineedit.setPlaceholderText("Enter a number (e.g. 3.14)")

# Create a validator: min 0.1, max 9990, up to 2 decimal places.
validator = QDoubleValidator(0.1, 9990, 2)

# Create a locale that uses a point for the decimal separator.
locale = QLocale(QLocale.English, QLocale.UnitedStates)
validator.setLocale(locale)

# Only allow standard notation (no scientific notation).
validator.setNotation(QDoubleValidator.StandardNotation)

# Apply the validator to the line edit.
lineedit.setValidator(validator)

lineedit.show()

app.exec()

Run this, and you'll have a QLineEdit that only accepts numbers formatted with a point as the decimal separator. Try typing 3.14 — it will be accepted. The comma is still permitted as a grouping separator (e.g., 1,000), but it won't function as a decimal point.

If you're looking for more ways to validate user input, you might also be interested in input validation in Tkinter for a comparison of how other Python GUI frameworks handle similar problems. For a broader overview of the widgets available in PyQt6, including QLineEdit, see the basic widgets tutorial.

Summary

The decimal separator used by QDoubleValidator comes from the locale. To change it:

  1. Create a QLocale with the language and country that matches your desired formatting.
  2. Call setLocale() on the validator to apply it.
  3. Optionally, call setNotation(QDoubleValidator.StandardNotation) to restrict input to standard decimal notation.

This gives you full control over how numbers are entered in your QLineEdit fields, regardless of the user's system settings. To learn more about checking whether a QLineEdit is empty or contains valid data, see how to check if a QLineEdit is empty.

For an in-depth guide to building Python GUIs with PyQt6 see my book, Create GUI Applications with Python & Qt6.

November 18, 2026 06:00 AM UTC


Graham Dumpleton

Muru: overlays on a schedule

The last post was about the Muru daemon taking an idle site down to nothing. This one is about something every site has to deal with eventually, which is a change that is only meant to last for a while.

The database is backed up at 2am, and the site shouldn't take orders while that happens. A schema change on Saturday night needs the site down for an hour, with a page saying when it will be back. A sale starts at 9am on Monday, and the new version of the shop should be serving it at exactly that time. None of these is a permanent change to how the site is served, but each needs the configuration to be different for a while, and then to go back to how it was.

With gunicorn or uvicorn behind nginx, the usual answer is to edit something and remember to undo it. A maintenance page is often an if (-f /srv/shop/maintenance.html) test in the nginx configuration, with someone touching and removing the file, or a cron job doing it for them. A read-only period is a flag the application checks, set by hand or from another cron job. A timed switch to a new version is someone awake at 9am to run the deploy. Each of these lives somewhere different, and nothing in the server knows any of them is temporary.

Muru has one mechanism for all of these, which it calls an overlay.

An overlay is laid over the configuration

An overlay is a named piece of configuration laid over routes and applications while it is active. The configuration underneath never changes. When the overlay ends, the routes are as they were, including any change made to them while it was active, so ending an overlay can't undo something someone else did in the meantime.

Overlays are declared beside the routes they change, in the files that muructl apply reads. For the shop in this post, that starts with a virtual host and a route which has a name an overlay can refer to:

[host.shop]
names = ["shop.localhost"]
default = true

[[route]]
name = "shop-root"
host = "shop"
app = "shop"

[overlay.nightly-backup]
apps.shop = { mode = "read-only", deny = ["POST", "PUT", "PATCH", "DELETE"] }
trigger = { schedule = { cron = "0 2 * * *", duration = "1h", timezone = "Australia/Sydney" } }

The overlay makes the shop read-only for an hour from 2am Sydney time each night. While it is active the edge refuses the methods in deny without troubling the workers, and the application is passed the mode as muru.mode in the WSGI environ, so it can stop offering what the mode rules out:

def application(environ, start_response):
    time.sleep(0.02)  # as if rendering from a database

    if environ["REQUEST_METHOD"] == "POST":
        start_response("303 See Other", [("Location", "/")])
        return [b""]

    read_only = environ.get("muru.mode") == "read-only"
    page = PAGE.format(actions=PAUSED if read_only else CART)

    start_response("200 OK", [("Content-Type", "text/html")])
    return [page.encode()]

The mode is any short word, so an application can have modes of its own. Muru doesn't interpret it beyond passing it on.

Once applied, muructl overlay list shows the overlay, its trigger, what it changes and when it is next due:

$ muructl overlay list
OVERLAY                  STATE                          TRIGGER                                  CHANGES
nightly-backup           idle from 2026-11-17T15:00:00Z cron '0 2 * * *' for 1h Australia/Sydney app shop

The time shown is UTC, which is 2am in Sydney. The schedule follows daylight saving in the zone it names, so a window doesn't drift by an hour twice a year, and because a cron schedule depends on the system clock, the daemon checks every ten minutes that the clock is synchronised and raises a warning event when it isn't.

muructl overlay preview says what activating the overlay now would change, field by field:

$ muructl overlay preview nightly-backup
app shop: mode (none) -> read-only
app shop: methods.deny (none) -> POST, PUT, PATCH, DELETE

Turning one on by hand

A schedule is only one way to activate an overlay. Backups don't always run when they are meant to, and sometimes the read-only period is needed right now. Any overlay can be activated by hand, until it is ended, for a duration, or until a given time:

$ muructl overlay activate nightly-backup --for 10m
muru: overlay nightly-backup activated by hand, until 2026-11-17T06:38:03Z

The page now says orders are paused instead of offering the cart:

$ curl -s http://127.0.0.1:8080/
<!doctype html>
<title>Shop</title>
<h1>Today's offers</h1>
<p>Orders are paused for a few minutes, but you can still browse.</p>

and anything that would change the site is refused by the edge:

$ curl -si -X POST http://127.0.0.1:8080/cart
HTTP/1.1 503 Service Unavailable
content-type: text/plain; charset=utf-8
content-length: 147
cache-control: no-store
x-request-id: 01M7M31W21TJAZMK3QZF9FZD7M
date: Tue, 17 Nov 2026 06:28:03 GMT

503 Service Unavailable
reason: read_only
request id: 01M7M31W21TJAZMK3QZF9FZD7M

The site is read-only for now, and this request would change it.

With answer = "method_not_allowed" beside deny, the edge would answer 405 instead, with the methods still allowed in Allow.

The overlay list says how the overlay was activated and when it ends:

$ muructl overlay list
OVERLAY                  STATE                                      TRIGGER                                  CHANGES
nightly-backup           active (manual) until 2026-11-17T06:38:03Z cron '0 2 * * *' for 1h Australia/Sydney app shop

When something isn't behaving as expected, the question is usually what configuration is actually in force right now. muructl config show --effective answers that, headed by what each active overlay sets:

$ muructl config show shop --effective
# Muru effective state: muructl 1.0.0 on grumpys-mac-mini, 2026-11-17T06:28:03Z, state version 9.
# The active overlays set, and the routes below include:
#   app shop: methods.deny = POST, PUT, PATCH, DELETE (overlay nightly-backup)
#   app shop: mode = read-only (overlay nightly-backup)
...

An active overlay is part of the daemon's state rather than something held in memory. Stopping the daemon in the middle of the overlay, and starting it again:

$ muructl daemon stop
muru: daemon detached; hosts and the edge keep serving
$ muructl daemon start
muru: daemon started (pid 73330), serving on http://127.0.0.1:8080, log ...
$ muructl overlay list
OVERLAY                  STATE                                      TRIGGER                                  CHANGES
nightly-backup           active (manual) until 2026-11-17T06:38:03Z cron '0 2 * * *' for 1h Australia/Sydney app shop

the overlay was still active, and a POST was still refused. The daemon works out each second which overlays should be active from the time and what it has recorded, so a daemon restarted in the middle of a scheduled window carries on with it as well. Ending the overlay by hand put things back:

$ muructl overlay end nightly-backup
muru: overlay nightly-backup ended by hand

The POST was answered by the application again, and the overlay went back to waiting for 2am.

A maintenance window with its own page

Some work needs the site down entirely. An overlay can turn on maintenance mode for an application, and choose the page people see while it is on:

[overlay.db-upgrade]
apps.shop = { maintenance = true }
routes.shop-root.error_pages = { "503:maintenance" = { file = "pages/maintenance.html" } }
trigger = { schedule = { start = "2026-11-17T17:31", end = "2026-11-17T17:33", timezone = "Australia/Sydney" } }

A start and an end give one window rather than a recurring one.

What maintenance mode lets through is part of the application's own settings, in its pyproject.toml, since it is something the application knows about itself:

[tool.muru.maintenance]
exempt_paths = ["/health"]
exempt_addresses = ["10.0.0.0/8"]
bypass_cookie = "muru_bypass=letmein"
retry_after = "15m"

When the window began, the overlay activated, and every request routed to the shop was answered by the edge with the page from the overlay and a Retry-After of fifteen minutes:

$ curl -si http://127.0.0.1:8080/
HTTP/1.1 503 Service Unavailable
content-type: text/html; charset=utf-8
content-length: 126
cache-control: no-store
retry-after: 900
x-request-id: 01M7M37BXBTJAZMK3QZF9FZD7S
date: Tue, 17 Nov 2026 06:31:03 GMT

<!doctype html>
<title>Back soon</title>
<h1>We're upgrading the shop</h1>
<p>It will be back by 10pm. Your cart is safe.</p>

muructl app list marks the application as in maintenance, and names the overlay which put it there:

$ muructl app list
APP                  CURRENT      PREVIOUS     USER
shop                 r1           -            - [maintenance by overlay db-upgrade]

The health check path was still served, so a load balancer in front doesn't decide the whole node is gone. With the bypass cookie, the real site came back:

$ curl -s -b muru_bypass=letmein http://127.0.0.1:8080/
<!doctype html>
<title>Shop</title>
<h1>Today's offers</h1>
<form method="post" action="/cart"><button>Add to cart</button></form>

That cookie is the part I find most useful. Whoever is doing the upgrade can check that the site works before anyone else sees it, and then end the maintenance early if it is done before the window is over:

$ muructl overlay end db-upgrade
muru: overlay db-upgrade ended by hand
$ muructl overlay list
OVERLAY                  STATE                            TRIGGER                                                       CHANGES
db-upgrade               ended until 2026-11-17T06:33:00Z from 2026-11-17T17:31 until 2026-11-17T17:33 Australia/Sydney route shop-root, app shop
...

A scheduled overlay ended by hand stays ended until its window is over, rather than the schedule turning it straight back on a second later. The events record what happened and why:

$ muructl events
...
    27 17:31:00 notice   overlay.activated    overlay db-upgrade       activated: its schedule's window began
    28 17:31:20 notice   overlay.ended        overlay db-upgrade       ended by hand

Trying a schedule before letting it act

A schedule which acts on its own is only as good as the cron expression and time zone it was given, and getting either wrong means something happens at the wrong time with nobody watching. With advisory = true, a schedule only raises an event as each window begins, and the overlay has to be activated by hand:

[overlay.stocktake]
routes.shop-root = { mode = "read-only", deny = ["POST", "PUT", "PATCH", "DELETE"] }
trigger = { schedule = { cron = "32 17 * * *", duration = "1m", timezone = "Australia/Sydney" } }
advisory = true

When its window came around, the overlay list said it had been advised rather than activated:

stocktake                advised until 2026-11-17T06:33:00Z cron '32 17 * * *' for 1m Australia/Sydney, advisory          route shop-root

and the event says what would have happened:

    29 17:32:00 notice   overlay.would_activate overlay stocktake        would activate: its schedule's window began; it is advisory

The shop kept taking orders. Once the events show a schedule firing when it should, removing advisory lets it act.

A timed cutover

The sale starting at 9am on Monday is a different kind of temporary change. The overlay pins the route's traffic to a new release at a given time:

[overlay.summer-sale]
routes.shop-root.traffic = { pin = "r2" }
trigger = { schedule = { start = "2026-11-17T17:35", timezone = "Australia/Sydney" } }
warm_up = { start_release = "r2", lead = "1m" }
preconditions = { release_ready = "r2", probe = "/healthz" }

The release is deployed ahead of time with --hold, so it starts but takes no traffic, as covered in the releases, not restarts post. The warm_up starts the release ahead of the window if it has been stopped, so the switch lands on warm workers. The preconditions are what stop a timed switch moving traffic to something broken. The release has to be ready, and it has to answer the probe path with a 2xx or 3xx response, asked through the edge as a preview of that release.

In my case, r2 had a bug, and its health check couldn't reach its database:

$ muructl deploy shop v2 --hold
muru: release shop@r2 is ready and held: not serving
$ curl -s -H "Muru-Preview: r2" http://127.0.0.1:8080/healthz
database unreachable

Trying the overlay out by hand ahead of time showed this straight away:

$ muructl overlay activate summer-sale
muru: error: overlay summer-sale was not activated: its probe of shop@r2 failed: /healthz (as a preview, at host shop) answered 503 Service Unavailable

Had nobody checked, the scheduled activation would have been skipped in the same way, with an overlay.skipped warning event saying why, and the old release would have gone on serving. That is a much better outcome at 9am on a Monday than the sale page answering with errors.

With the bug fixed and deployed as r3, also held, I changed the overlay to name r3 and applied it again:

$ muructl overlay preview summer-sale
route shop-root: traffic.split (none) -> r3 100%

A minute before the window began, the warm-up started, and at the start of the window the overlay activated:

    30 17:34:00 info     overlay.warming      overlay summer-sale      warming up: its window begins in 59s; starting shop@r3
    31 17:35:00 notice   overlay.activated    overlay summer-sale      activated: its schedule's window began

A client asking for the page every fifty milliseconds through the switch saw the old page, then the new one, and nothing else:

17:34:03 200 Today's offers
17:35:01 200 Summer sale
{"200 Today's offers": 739, '200 Summer sale': 424}

A timed switch like this is usually meant to be permanent. muructl overlay promote folds an active overlay's changes into the base configuration and ends the overlay. For a pin, that makes the pinned release the application's serving release, just as a deploy would:

$ muructl overlay promote summer-sale
muru: overlay summer-sale promoted: its changes are now the base configuration; shop@r3 serves in place of shop@r1, which is retained
muru: the next apply undoes this unless the applied files say the same: put this in them, and remove overlay summer-sale

# shop: where the files name its serving release
[app.shop]
serving = "r3"

The release which was serving before is kept warm for a rollback, the same as after a deploy, and the next deploy replaces r3 as usual:

$ muructl release list
RELEASE                  STATE      SERVING  WORKERS               DISK SOURCE
shop@r3                  ready      yes      2 (2x4)              4.9MB /Users/graham/shop/v3
shop@r2                  ready               2 (2x4)              4.9MB /Users/graham/shop/v2
                         held: not serving
shop@r1                  retained            1 (2x4)              4.9MB /Users/graham/shop/v1
                         warm until 07:05 UTC; a rollback to it is instant

The applied files still say what they said before, so promote prints what to put in them, here the serving release, for the next muructl apply to keep the switch rather than undo it. The promoted overlay should then be taken out of the files.

Known busy periods

Overlays can also set capacity, and an application can declare schedules for load that is known in advance. A schedule on the application is an overlay by another name, written beside the application:

[[app.shop.schedule]]
name = "black-friday"
start = "2026-11-27T00:00"
end = "2026-11-30T23:59"
timezone = "America/New_York"
min_workers = 4
scale_in = "hold"
freeze_deploys = true

For the four days of the sale the shop keeps at least four workers, never scales in, and refuses deploys. It shows up in the overlay list as shop.black-friday:

shop.black-friday        idle from 2026-11-27T05:00:00Z from 2026-11-27T00:00 until 2026-11-30T23:59 America/New_York app shop

and since it is an overlay, it can be tried out by hand like any other:

$ muructl overlay preview shop.black-friday
app shop: group default: at least 4 workers, no scale-in
app shop: deploys and rollouts frozen
$ muructl overlay activate shop.black-friday --for 10m
muru: overlay shop.black-friday activated by hand, until 2026-11-17T06:48:17Z

The worker group rose to the floor, and muructl scale status names the overlay which set it:

$ muructl scale status shop
shop@r3
GROUP        WORKERS      SLOTS      IDLE    QUEUE-P95     REAPS      POLICY
default      4            0/16       4.00    under 1 ms    -          static
  scheduled: 4-4 workers, no scale-in from shop.black-friday
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity
  last: scaled out to 4: raising toward the floor of 4 that shop.black-friday sets (3s ago)

The shop has a static count of two workers, so the floor raises that count for the window. With an autoscaling policy, as in the scaling from what the workers report post, the floor sets the least the autoscaler may go down to, and a max_workers in the schedule can raise its ceiling too.

The deploy freeze is there for the person who decides on the Friday that a small fix can't wait:

$ muructl deploy shop v1
muru: error: deploys of shop are frozen by overlay shop.black-friday; give --override-freeze to deploy anyway

When it really can't wait, --override-freeze lets the deploy through, and records that it did:

    74 17:38:23 warning  deploy.freeze_overridden shop                     deploy despite the freeze set by overlay shop.black-friday

Rolling back is never frozen, since a freeze that stopped someone undoing a bad deploy would be worse than no freeze at all.

What's next

What these overlays have in common is that someone knew ahead of time when they were needed, even if the answer was "now". The backup runs at 2am, the upgrade is on Saturday night, and the sale starts at 9am. Muru holds the change, applies it at the right time, refuses to apply it when the release it depends on is broken, and takes it away again afterwards, and the base configuration is never touched in the process.

Not everything can be planned for, though. Crawlers turn up when they turn up, and a payment provider goes down without telling anyone first. The next post, noticing and responding, is about overlays which are activated by what Muru sees happening rather than by the time.

November 18, 2026 12:00 AM UTC

November 17, 2026


Python Bytes

#505 Introducing diskcache-oxide

<strong>Topics covered in this episode:</strong><br> <ul> <li><strong><a href="https://github.com/Piergiuseppe/tinyjobs?featured_on=pythonbytes">tinyjobs: a SQLite job queue goes free-threaded</a></strong></li> <li><strong><a href="https://alexwlchan.net/2026/funkycache/?featured_on=pythonbytes">funkycache</a>: cache Python functions in SQLite</strong></li> <li><strong><a href="https://github.com/4yk5/Ghostman?featured_on=pythonbytes">Ghostman</a> - a keyboard-driven terminal API client</strong></li> <li><strong><a href="https://mkennedy.codes/docs/diskcache-oxide/?featured_on=pythonbytes">diskcache-oxide: A shared, durable disk cache</a></strong></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=SM5OwEmfHVA' style='font-weight: bold;' data-umami-event="Livestream-Past" data-umami-event-episode="505">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p><strong>Sponsored by</strong> LongLink: <a href="https://pythonbytes.fm/longlink">pythonbytes.fm/longlink</a></p> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy">Mastodon</a> / <a href="https://bsky.app/profile/mkennedy.codes?featured_on=pythonbytes">BlueSky</a> / <a href="https://x.com/mkennedy?featured_on=pythonbytes">X</a> / <a href="https://www.linkedin.com/in/mkennedy/?featured_on=pythonbytes">LinkedIn</a></li> <li>Calvin: <a href="https://sixfeetup.social/@calvin?featured_on=pythonbytes">Mastodon</a> / <a href="https://bsky.app/profile/calvinhp.com?featured_on=pythonbytes">BlueSky</a> / <a href="https://x.com/calvinhp?featured_on=pythonbytes">X</a> / <a href="https://www.linkedin.com/in/calvinhp/?featured_on=pythonbytes">LinkedIn</a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes">Mastodon</a> / <a href="https://bsky.app/profile/pythonbytes.fm">BlueSky</a> / <a href="https://x.com/PythonBytes?featured_on=pythonbytes">X</a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Tuesday at 7am PT</strong>. Older video versions available there too.</p> <p>Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it.</p> <p><strong>Michael #1: <a href="https://github.com/Piergiuseppe/tinyjobs?featured_on=pythonbytes">tinyjobs: a SQLite job queue goes free-threaded</a></strong></p> <p>tinyjobs is a Python job queue and cron scheduler that starts with a SQLite file and no dependencies. Switch the URL to Postgres or Redis without rewriting tasks. It runs async tasks natively, embeds workers in FastAPI, and plugs into Django 6's tasks framework. The creator's free-threading audit found four races hidden by the GIL.</p> <ul> <li>Background jobs without deploying a broker, plus a concrete case study in what removing the GIL exposes.</li> <li>The creator reports races in the task registry, codec cache, cross-thread flag and thread pool. hiredis can re-enable the GIL; the worker warns when it detects that.</li> <li>On 3.14t, --loops N runs event loops in separate threads. The author's laptop benchmark hit 42,000 async jobs/s with eight loops and two Redis instances, close to eight processes. SQLite rejects loops above one.</li> <li>Clock-skew lesson: two hosts 20 minutes apart duplicated 492 of 3,000 jobs on 0.7.0. Using Redis server time fixed that test, not the general exactly-once problem.</li> <li>Author's SQLite measurements: about 572 jobs/s end-to-end; a 20,000-job burst drained in four seconds. Synthetic workloads and different harnesses, not interchangeable throughput claims.</li> <li>At-least-once delivery still requires idempotent tasks. Python 3.12+; young project, with the README now listing v0.11.0 versus the email's v0.9.0. Docker setup:</li> </ul> <div class="codehilite"> <pre><span></span><code><span class="nt">services</span><span class="p">:</span> <span class="w"> </span><span class="nt">web</span><span class="p">:</span> <span class="w"> </span><span class="nt">build</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">.</span> <span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">uvicorn main:api --host 0.0.0.0</span> <span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">jobs</span><span class="p p-Indicator">:</span><span class="nv">/data</span><span class="p p-Indicator">]</span> <span class="w"> </span><span class="nt">worker</span><span class="p">:</span> <span class="w"> </span><span class="nt">build</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">.</span> <span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">tinyjobs worker --app tasks:app --shutdown-timeout 25</span> <span class="w"> </span><span class="nt">stop_grace_period</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">40s</span> <span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">jobs</span><span class="p p-Indicator">:</span><span class="nv">/data</span><span class="p p-Indicator">]</span> <span class="nt">volumes</span><span class="p">:</span> <span class="w"> </span><span class="nt">jobs</span><span class="p">:</span> <span class="w"> </span><span class="nt">external</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span> </code></pre> </div> <p><strong>Calvin #2: <a href="https://alexwlchan.net/2026/funkycache/?featured_on=pythonbytes">funkycache</a>: cache Python functions in SQLite</strong></p> <ul> <li>Alex Chan's <code>functools.cache-style</code> decorator stores results in SQLite, so they survive after the process exits. He built it to speed up his static site build.</li> <li>It's released on PyPI: <code>funkycache</code> 1.0.4, from November 15. The first release, 1.0.0, came out November 7. It needs Python 3.14+ and depends on <code>pydantic</code> and <code>xxhash</code>.</li> <li>Usage: <code>cache = FunkyCache(storage="sqlite")</code>, then <code>@cache.register(files=[...], values=[...])</code>. A cache entry is invalidated when the function's source file changes or when any listed file or value changes, such as a library version.</li> <li>Return values go through Pydantic's <code>TypeAdapter</code>, based on the return type hint, so it only works on typed code.</li> <li>Generators get special handling. <code>functools.cache</code> hands every caller the same generator object. funkycache saves the output as a list and gives each caller a fresh iterator, up to a soft memory limit.</li> <li>Cache keys are JSON hashed with XXH64. They're deliberately lossy, and that's fine because nothing reads the original arguments back out of a key.</li> <li>The design notes are a good segment on their own: the author had used MD5 because they "conflated weak with fast," and in their limited testing JSON was faster than pickle and easier to debug.</li> <li>With a warm cache the site builds in about half a second, and the library is only a few hundred lines including tests. It doesn't watch files on purpose, to stay small.</li> </ul> <p><strong>Calvin #3: <a href="https://github.com/4yk5/Ghostman?featured_on=pythonbytes">Ghostman</a> - a keyboard-driven terminal API client</strong></p> <ul> <li>A local-first Postman alternative that runs in the terminal, with no cloud account required. It's MIT licensed and was featured in PyCoder's Weekly #760.</li> <li>It's released on PyPI: pip install ghostman, then run ghostman. The current version is 0.1.8, from November 3. The first release, 0.1.0, came out October 31. It needs Python 3.10+.</li> <li>It's built on Textual, httpx and Rich, so it's pure Python under the hood.</li> <li>It sends GET, POST, PUT, DELETE and PATCH requests, shows DNS and time-to-first-byte timing live, and colors JSON responses automatically.</li> <li>A built-in webhook inspector: Ctrl+W starts a listener on port 9090. Captured payloads show headers and timestamps, and you can load one straight into the request editor.</li> <li>Collections are plain JSON/YAML files you can commit to Git. The README gives two different folders for them (.ghostman/ and ~/.ghostman/).</li> <li>It's fully keyboard-driven: Ctrl+S sends, Ctrl+K saves, Ctrl+N starts a new request, Ctrl+Q quits.</li> <li>The author claims about 30MB of RAM and a cold start of about 150ms; I haven't checked either number. It's a young project, with 158 stars and 15 commits.</li> </ul> <p>Michael #4: <a href="https://mkennedy.codes/docs/diskcache-oxide/?featured_on=pythonbytes">diskcache-oxide: A shared, durable disk cache</a></p> <p>Michael's diskcache-oxide is a Rust-backed, SQLite-based cache for Python workers sharing one directory on the same machine. Entries survive process restarts without running a cache server. It keeps the familiar diskcache API, adds fork and free-threading safety, and uses msgpack or JSON by default instead of pickle.</p> <ul> <li>Michael's item: a shared cache for multi-worker apps without deploying Redis or keeping a separate dict warm in every process.</li> <li>Cache, FanoutCache sharding and transactions are available now. NullCache switches caching off without changing call sites; Caches manages named caches for an app.</li> <li>Author's bench/gate.py, Linux container on Apple Silicon, CPython 3.14, 99:1 reads/writes: about 750k ops/s with one thread and 723k with eight; diskcache 5.6.3 falls to 44k. Not independently tested here.</li> <li>Free-threading support is explicit: the Rust extension declares gil_used = false. A process-wide fork barrier protects SQLite state when workers fork.</li> <li>Pickle is opt-in. Tagged serialization preserves types such as tuples, sets, datetimes and Decimals, with registration for custom types.</li> <li>The docs come to about <strong>70,000 words</strong>, the length of a typical novel. Of that, about 53,000 are the written story (guide, recipes and the home page), and the rest is API reference, the agent skill and code samples.</li> <li>Early release, not full diskcache parity: string keys only; AsyncCache, memoize, tags, Deque and Index are still planned. The network cache server is a roadmap item too.</li> <li>Python 3.14+; macOS and Linux wheels include SQLite. Windows support is planned. Migration changes the import and uses a new directory, not the old cache.db.</li> </ul> <p><strong>Extras</strong></p> <p>Calvin:</p> <ul> <li>Django Software Foundation <a href="https://www.djangoproject.com/weblog/2026/nov/15/2027-dsf-board-nominations/?featured_on=pythonbytes">Board Nominations</a> open until December 2nd Michael:</li> <li><a href="https://www.startpage.com/bill/plan-selection?sc=gQhKyhqUvCKn0oNP9jyRwvCbbJR1aKQqd3wHub8JxnqtNC93sPzZGl3WhHVfkT88mpDQILTiYK6yFx93LyoDt9e3eQz9MuHuH0H1MmSe9rtc9vd1P&t=device&segment=startpage.udog&lui=english&language=english&cat=web&abd=1&abe=1&query=&featured_on=pythonbytes">Startpage paid plans</a></li> <li>Test driving <a href="https://helium.computer/?featured_on=pythonbytes">Helium Browser</a></li> <li><a href="https://github.com/mikeckennedy/tallyman/releases/tag/v0.4.3?featured_on=pythonbytes">Tallyman</a> got a big update, give it another try <ul> <li>Tallyman is my command-line tool for seeing the shape of a codebase, with line counts grouped by language and categories such as Code, Docs, and Specs. Version 0.4.3 bundles new controls and a long list of fixes. <ul> <li>I made a bunch of massive updates this week. If you tried Tallyman before and it didn't work for your project, give it another try.</li> <li>Use --no-save to try setup choices without writing a config, and --exclude PATTERN for one-run file exclusions.</li> <li>Lots of fixes to gitignore handling, nested configs, the setup TUI, encodings, and image export. First runs in CI or pipes no longer hang waiting for setup.</li> <li>Optional exclusion of inline Rust test code, plus MATLAB and Solidity support, bringing the language count to 44.</li> <li>Want another feature? Submit an idea through a GitHub issue.</li> </ul></li> </ul></li> <li><a href="http://Bunny.net?featured_on=pythonbytes">Bunny.net</a> fonts disappoint. <ul> <li><strong>Root cause:</strong> Bunny Fonts began serving its font files from <code>cdn.jsdelivr.net</code>, which our <strong>CSP blocks</strong> - production had silently fallen back to system fonts.</li> </ul></li> <li><a href="https://talkpython.fm/meetups/sqlite-might-actually-be-all-you-need?featured_on=pythonbytes">Talk Python Meetup: SQLite Might <em>Actually</em> Be All You Need</a></li> </ul> <p><strong>Jokes</strong>:</p> <ul> <li><a href="https://x.com/pr0grammerhum0r/status/2121573926568395183?a=0VHlGMnpqO9U8FjKPKw&s=46&featured_on=pythonbytes">Disaster recovery</a></li> <li><a href="https://www.reddit.com/r/iiiiiiitttttttttttt/s/THQxhz3hfz?featured_on=pythonbytes">Prophecy fulfilled</a></li> </ul>

November 17, 2026 10:23 PM UTC


PyCoder’s Weekly

Issue #761: Goose Typing, Gemini 4 Carbon, HTTPX2, and More (2026-11-17)

#761 – NOVEMBER 17, 2026
View in Browser »

The PyCoder’s Weekly Logo


Goose Typing

In Python you can practice type checking while also practicing duck typing. This article shows you how to use the __instancecheck__() method to change the behaviour of isinstance().
TREY HUNNER

Gemini 4 Carbon: Google’s Model That Thinks Longer

Gemini 4 Carbon is Google’s new reasoning model. Learn what it is, what it costs, and how to call it from Python with the google-genai SDK.
REAL PYTHON

Positron: The Data Science IDE from Posit PBC

alt

Tired of duct-taping your Python workflow together? Positron is a free, with interactive data frames, Jupyter notebooks, AI assistance, and one-click app deployment, all in one editor. Download it free. Explore Positron
POSIT sponsor

Python HTTPX: Making HTTP Requests With HTTPX2

Learn how to use Python HTTPX (now HTTPX2) to make HTTP requests, send async calls, set safe timeouts, stream large downloads, and enable HTTP/2.
REAL PYTHON

Quiz: Python HTTPX: Making HTTP Requests With HTTPX2

REAL PYTHON

Python 3.16.0 Alpha 2 Is Now Available

PYTHON.ORG

Discussions

In Memoriam: Tim Peters 1952–2026

PYTHON.ORG

Articles & Tutorials

Mentoring Developers Beyond the Code With Tim Schilling

What’s it take to be a mentor to developers? Where in an engineer’s career can a mentor make an impact? This week we speak with Tim Schilling, Django Steering Council member and Djangonaut Space administrator, about mentoring Python developers.
REAL PYTHON podcast

Keep Threads Free: Introducing gilguard

On free-threaded Python, one import can quietly turn the GIL back on for good. gilguard turns that moment into an error that names the module, the package, and the line of your code that pulled it in.
MICHAEL KENNEDY

Adopting Python 3.15’s Lazy Imports

This post describes one company’s experience with Python 3.15’s lazy import mechanism. It resulted in a 4x speed-up for one of their CLIs, but also describes the places where it didn’t help.
NAMEGARAGE

PyPI’s Redesigned Project Pages Are Now Live

In July, PyPI announced a series of updates to the PyPI UI. The work is now complete and includes a new “Security” tab which gives details on where files are sourced from.
NICOLE HARRIS

TDD With AI Agents

Learn how to use test-driven development (TDD) with AI agents in Python. Design the tests first, then let a coding agent write code that passes them.
REAL PYTHON

Get Started With Meta Muse in Python: Spark and Glimmer

Get started with Meta Muse in Python. Make your first Muse Spark call, then add streaming, images, and tool calls, and try the open-weights Glimmer.
REAL PYTHON

Memory Management in Python

Take a deep dive into Python memory management and learn how CPython allocates and frees memory for your objects with arenas, pools, and blocks.
REAL PYTHON

Making Django Admin Easier to Use Every Day

This post describes a few improvements to the Django Admin: a command palette, a search shortcut, and tabs for long forms.
RUST SAIARGALIEV

Migrating to Python 3.15

This practical guide tells you what to pay attention to when migrating to Python 3.15.
RODRIGO GIRÃO SERRÃO

Could a Modern Chess Engine on a Pentium Have Defeated Kasparov in 1996?

Sometimes the right algorithm matters more than the beefiest hardware. This post shows how better approaches to chess in software mean that old hardware could have beat the supercomputers used to challenge grand masters at the time.
NICHOLAS CARLINI

Projects & Code

Wheelgap: Which Dependencies Will Compile From Source?

GITHUB.COM/PATRICKBALLARD • Shared by Patrick Ballard

Waymark: Resumable Python Workflows Backed by Postgres

GITHUB.COM/PIERCEFREEMAN • Shared by Pierce Freeman

sqlty: Type-Check Raw SQL in Python Without Database

KUNCAR.DEV • Shared by Jiri Kuncar

izaya: HTMX-Powered Autocomplete Widget for Django Forms

GITHUB.COM/METRON-PROJECT

django-cmdbar: A Navigation Palette for Django Admin

GITHUB.COM/AMUREKI

Events

Weekly Real Python Office Hours Q&A (Virtual)

November 18, 2026
REALPYTHON.COM

Wagtail Space 2026

November 18 to November 21, 2026
WAGTAIL.ORG

PyData Bristol Meetup

November 19, 2026
MEETUP.COM

PyLadies Dublin

November 19, 2026
PYLADIES.COM

PyCon Wroclaw 2026

November 20 to November 21, 2026
PYCONWROCLAW.COM

PyCon Ireland 2026

November 21 to November 22, 2026
PYCON.IE


Happy Pythoning!
This was PyCoder’s Weekly Issue #761.
View in Browser »

alt

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

November 17, 2026 07:30 PM UTC


PyCon

PyCon US 2027 - Call for Proposals Now Open!

PyCon US 2027 is heading back to California for a second year in our sunny host city of Long Beach, CA!


We’re also excited to announce the launch of our conference website, along with our Call for Proposals and our sponsorship program!

Welcome Back to Long Beach!

PyCon US is returning to Long Beach, CA, located in the heart of sunny Southern California, right along the Pacific Ocean.

PyCon US 2027 will be held at the Long Beach Convention and Entertainment Center, in downtown Long Beach, CA, on the following dates:
We will continue to return in person, with Health and Safety Guidelines in place.

PyCon US 2027 Website is Live

You can now head over to the PyCon US 2027 website for all the conference details and more information about our sponsorship program.

This year’s PyCon US design leans into the storied silver screen history of Southern California, giving a Pythonic twist to classic Hollywood glamour.

We’re excited to collaborate with designer Anastasiia, whose illustrations and designs brought the PyCon US 2027 website to life. The design was brought together with the coordination of Georgi K and implemented by Mirakuru Europa OÜ.

PyCon US Call for Proposals is Now Open

PyCon US 2027’s Call for Proposals is officially open for Talks, Tutorials, Posters, and Charlas! The deadline to submit for all tracks is December 19, 2026.

We need beginner, intermediate, and advanced proposals on all sorts of topics—and beginner, intermediate, and advanced speakers to give said presentations. You don’t need to be a 20-year veteran who has spoken at dozens of conferences! On all fronts, we need all types of people. Our community is composed of a diverse set of people with unique skill sets, and we want our conference program to be a true reflection of that diversity.

PyCon US is bringing back two dedicated Talk tracks to the schedule: “the Future of AI with Python” and “Trailblazing Python Security”. For more information and how to submit your proposal, visit the speaker guidelines page.

For new and first-time speakers, be sure to take advantage of the speaker mentorship program where you can be matched with experienced speakers who can help you with crafting your proposal–deadline November 27, 2026. Check out the info on our Proposal Mentorship Program page.

For more information on where and how to submit your proposal, visit the Proposal Guidelines page on the PyCon US 2027 website.

Sponsorship Has Tremendous Impact

Sponsors are what make PyCon US and the Python Software Foundation (PSF) possible. PyCon US is the main source of revenue for the PSF, the non-profit behind the Python language and the Python Packaging Index (PyPI), and the hub for the Python community.

PyCon US is the largest and longest-running Python gathering globally, with a diverse group of highly engaged attendees, many of whom you won’t find at other conferences. We’re excited to be able to provide our sponsors with opportunities to connect with and support the Python community. You’ll be face-to-face with talented developers, qualified recruits, and potential customers, access a large and diverse audience, as well as elevate your visibility and corporate identity within the Python community.

Check out our prospectus and our full menu of benefits to learn more about the 2027 PSF sponsorship program.

What you can expect when you sponsor PyCon US and the PSF:
  • Reach - Access to 2000+ attendees interested in your products and services and generate qualified leads.
  • Brand strength - Be part of the biggest and most prestigious Python conference in the world and support the nonprofit organization behind the Python language.
  • Connections - Networking with attendees in person to create connections and provide detailed information about your products and services.
  • Recruiting - Access to qualified job candidates. If you’re hiring, there’s no better place to find Python developers than PyCon US.
  • 12 months of benefits - Reach the Python community during PyCon US and beyond, with options for recognition on Python.org, PyPI.org, and more.
Because of the generosity of our sponsors, the PSF is able to continually improve our support of Python and its community. Read more about the PSF on the PyCon US website and in the most recent PSF Annual Impact Report. You can help us ensure the Python ecosystem thrives for years to come.

Apply to sponsor PyCon US and the PSF today! Head over to our prospectus to learn more about the 2027 PSF sponsorship program or go directly to the Sponsorship Application page to sign up to sponsor.

If you have any questions about sponsoring PyCon US and the PSF, please contact us at sponsors@python.org.

Stay in the Loop

As we get closer to the event, the conference website is where you’ll find details for our call for proposals, registration launch, venue information, and everything PyCon US-related!

Be sure to subscribe here to the PyCon US Blog, follow @PyCon on X, @PyCon.US on Bluesky, @pycon@fosstodon.org and @ThePSF@fosstodon.org on Mastodon, the PSF on LinkedIn, and subscribe to PyCon US 2027 News so you won’t miss a thing. Our official hashtag is #PyConUS.

November 17, 2026 03:31 PM UTC


Mike Driscoll

Python Black Friday 2026 Sale

Python is still a great language to start learning even in this age of AI. I am starting my Black Friday day sales early so everyone can get a jump start on their learning this holiday season or prepare for 2027!

You can use the following coupon code to get 35% off anything in my Gumroad store which includes video courses and eBooks: Black2026

Here is a listing of what my books covers:

The post Python Black Friday 2026 Sale appeared first on Mouse Vs Python.

November 17, 2026 01:54 PM UTC


Graham Dumpleton

Muru: scaling to zero

The last post was about a busy release, and what the Muru daemon does when requests start to wait. The opposite problem is just as common. A server hosting a dozen small sites, a few preview deployments and a staging copy or two will spend most of its time with nearly all of them idle, and each one sitting there with its workers started is holding memory nobody is using.

Most Python web servers don't do anything about this. Gunicorn, uvicorn and granian keep the workers they started for as long as they run. mod_wsgi has long had an inactivity-timeout option for its daemon processes, which restarts a process once it has had no requests for a while, so its memory drops back to what it was before the application was loaded, and the next request waits while the application is loaded again. uWSGI has a cheap mode, where workers aren't spawned until the first request arrives, and an idle option to go back into cheap mode after a period of inactivity. Platforms such as Knative or Cloud Run scale whole containers to zero, and the next request waits for a new container to start and the application to be imported in it.

Muru does this in two stages, and the first is cheap because of how the daemon starts workers.

No workers, but the host stays

Every release under the Muru daemon has a host process which imports the application once, and workers which are forked from it, as covered in the import once, fork many post. With a scaling policy whose min_workers is zero, the daemon will drain the last worker once nothing has needed it for a while, but leave the host running.

To make the difference between the stages obvious, the application for this post sleeps for three seconds when it is imported, standing in for a large application whose import is slow:

[tool.muru]
entrypoint = "app:application"
processes = 1
threads = 4

[tool.muru.scaling]
mode = "spare"
min_workers = 0
max_workers = 4
idle_reap_after = "20s"
scale_in_window = "20s"
hold_after_growth = "10s"
reap_interval = "5s"
min_lifetime = "5s"

As in the last post, the windows are shortened so this happens in seconds. By default idle_reap_after is ten minutes. Twenty seconds after the last request, the daemon took the worker away:

default: scale in from 1 to 0 workers: no request for 20s; diagnosis: idle

The release still shows as ready and serving, with no workers:

docs@r1
GROUP        WORKERS      SLOTS      IDLE    QUEUE-P95     REAPS      POLICY
default      0            0/0        0.00    -             1/6        spare 0-4 workers, 1-3 idle
  workers: cpu -, gc -, gil wait p95 -, loop lag p95 -
  diagnosis: idle
  last: scaled in to 0: no request for 20s (8s ago)

and of its processes, only the host and its log relay were left. Asking for the release's processes with muructl status --processes --app docs while it was serving gave:

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE           WHAT                                                    USER     UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
55119   daemon         user daemon, state ~/Library/Application Support/muru   graham   4s   0.0%   38.1MB
55130   └─ host        docs@r1 serving                                         graham   4s   0.0%   54.4MB            0/4       0
55129      ├─ relay    docs@r1 output to apps/docs/r1.log                      graham   4s   0.0%   17.8MB
55141      └─ worker   group default, worker 0                                 graham   0s   0.0%   20.9MB            0/4

and once it was at zero workers:

edge  0 req/s  0 err/s  p50 0.0 ms  p95 0.0 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 0B/s

PID     ROLE          WHAT                                                    USER      UP    CPU      RSS   REQ/S   BUSY   QUEUE   P95   GIL/LAG
55119   daemon        user daemon, state ~/Library/Application Support/muru   graham   26s   0.7%   39.5MB
55130   └─ host       docs@r1 serving                                         graham   25s   0.0%   54.4MB       0              0
55129      └─ relay   docs@r1 output to apps/docs/r1.log                      graham   25s   0.0%   17.8MB

That saving is modest for one worker, and the host still holds the imported application. The point of this stage is what the next request costs. The edge holds the request as queued, the daemon forks a worker from the host, and the request is served by it:

default: scale out from 0 to 1 workers: a request is waiting for a group scaled to zero
docs from pid 55198
0.005145s

Five milliseconds, compared with a fraction of a millisecond once the worker was there. Nobody visiting a site will notice that, and it is only possible because the application was already imported in the host. A server which has to load the application again, as mod_wsgi does once its inactivity timeout has restarted the process, makes this first request wait for the import.

No host either

For a site which might not see a request for hours, the host's memory can be given back as well. Adding host_idle_timeout to the release's scaling policy has the host stop once every worker group of the release has been at zero for that long:

host_idle_timeout = "30s"

Thirty seconds after the last worker went, the daemon stopped the host:

release.idle         docs@r2                  every worker group has been at zero for 30s: its host stopped, and the next request starts it again

The release has no processes at all now, but stays in the routing as ready, and muructl release list says what state it is in:

RELEASE                  STATE      SERVING  WORKERS               DISK SOURCE
docs@r2                  ready      yes      - (1x4)              4.7MB ...
                         idle: its host has stopped, and the next request starts it

This time the next request has to wait for the host to start and import the application:

docs from pid 55280
200 3.351380s
docs from pid 55280
200 0.000661s
release.woken        docs@r2                  a request is waiting: starting its host again
release.ready        docs@r2                  1 worker ready in 3.3 s

So there is a choice to make per release. Scaling to zero workers and keeping the host is nearly free for the next visitor. Stopping the host as well gives back everything, but the next visitor waits through the import. For a preview deployment nobody has looked at in a day, that is probably a fair trade. For a small site with occasional visitors, probably not.

The queue timeout still applies

While a request waits for a worker to be forked, or for a host to start, it is in the edge's queue like any other request, and the queue timeout still applies. The default is five seconds. I set this release's to two seconds, less than its import takes:

[tool.muru]
entrypoint = "app:application"
processes = 1
threads = 4
timeouts = { queue_timeout = "2s" }

The request which woke the idle release got a 503 after exactly the queue timeout:

HTTP/1.1 503 Service Unavailable
content-type: text/plain; charset=utf-8
content-length: 155
cache-control: no-store
retry-after: 1
x-request-id: 01M6ZMF2Y40K8S7W1FVPQARCZD

503 Service Unavailable
reason: cold_start_timeout
request id: 01M6ZMF2Y40K8S7W1FVPQARCZD

No worker had started for the request within the queue timeout.
2.003713s

The host went on starting regardless, and a request a second and a half later was served in about a millisecond. The Retry-After header tells a well behaved client to come back after a second, which is the right answer. The body says no worker had started in time, and the access log records the reason cold_start_timeout, so this can be told apart from an overloaded release.

The lesson is that once a release's host can stop, its queue timeout has to cover its import, or the first visitor after a quiet spell gets an error.

Many sites on one node

Scaling to zero is mostly about many applications sharing a server, and the daemon has a budget which caps what all the releases on the node hold between them. It goes in the daemon's configuration file:

[daemon.budget]
workers = 6

Memory and CPU can be capped the same way, and on Linux, when the daemon itself runs under a limit, as under a systemd unit with MemoryMax= or CPUQuota=, those two default to that limit. I deployed four applications, each behind its own virtual host. One was a shop with the spare mode policy from the last post, and the other three were sites like the one above, two of them previews. With the three small sites idle, the budget showed only the shop's worker in use:

budget   1 of 6 workers

With sixteen clients asking for the shop's slow page, its scale-out stopped at the budget:

default      6            12/12      -1.50   under 250 ms  0/6        spare 1-12 workers, 1-3 idle
  ...
  last: held at 6: the node's worker budget (6 of 6 workers) (3s ago)
scaling.denied_by_budget shop@r1                  default: 1 of the 4 more workers asked for fit the node's worker budget (5 of 6 workers), at priority 0

That is the budget doing its job. Workers already running are never stopped to make room for another application, and when several releases want more at the same time, an application's priority decides which goes first. What happens to the sites scaled to zero while the budget is full is less obvious.

Neither kind of wake is held back by the workers limit. One preview had been idle long enough for its host to stop. A request for it woke it in the usual three seconds or so, with the event saying it went over the budget:

release.woken        preview-b@r1             a request is waiting: starting its host again, over the node's budget of 6 workers
budget   7 of 6 workers

The other preview still had its host, with zero workers, so it needed only a fork, and was served in six milliseconds:

scaling.decision     preview-a@r1             default: scale out from 0 to 1 workers: a request is waiting for a group scaled to zero, over the node's budget of 6 workers

A site which can't answer its first request is a worse outcome than a busy site not getting one more worker, so the node goes over its limit for the moment. Retained releases holding workers are stopped, oldest first, to bring it back under, and until then other releases' scale-out is refused. The memory limit still applies to both kinds of wake.

Deploys fit in with this too. When a new release replaces an old one, the old one is kept for half an hour so that a rollback can go back to it. For a group whose min_workers is zero it is kept without workers, so it holds none of the node's budget, and muructl release list says what a rollback to it would cost:

RELEASE                  STATE      SERVING  WORKERS               DISK SOURCE
docs@r2                  ready      yes      1 (1x4)              4.7MB ...
docs@r1                  retained            0 (1x4)              4.7MB ...
                         host kept until 08:16 UTC; a rollback to it forks a worker

An explicit --retain-workers on the deploy keeps that many workers in every group instead.

There is one more case where scaling to zero doesn't fit. Muru matches gRPC calls by the services a worker has registered, so a release with a gRPC mount needs at least one worker running for its calls to be routed at all.

What's next

The first stage is what makes this worthwhile. Because the host has already imported the application, getting from no workers back to one costs a fork, so a release can drop to zero workers whenever it is idle without its next visitor really paying for it. Stopping the host as well is there for when memory matters more than the first request.

Scaling is something the daemon works out for itself from moment to moment, but some changes to how a site is served are planned ahead, such as a read-only period while the database is backed up, a maintenance window, or a switch to a new release at a set time. The next post, overlays on a schedule, is about how Muru lays changes like these over a site's configuration for a while, and then takes them away again.

November 17, 2026 12:00 AM UTC

November 16, 2026


Talk Python to Me

#569: Connecting Agents to Data via MCP, CLI, and More

Should your agent call an MCP server or just run a CLI? Pamela Fox joins us to share how she thinks about connecting AI and her data. She’s taught MCP workshops from PyCon to WeAreDevelopers. And, every week, an agent combs her Gmail, Outlook, and X history to build her news roundup. She uses MCP all day. Yet, there are still times you when to skip it. You'll hear how her colleague pitted the GitHub CLI against the GitHub MCP server on a real bug fix. The CLI used fewer tokens. It succeeded more often. So do we choose CLIs, MCPs, or even just APIs for connecting our agents to our data and services? Let's find out.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/talk-python-membership-continue-conversation'>Talk Python Membership</a><br> <a href='https://talkpython.fm/talk-python-membership-inner-circle'>Talk Python Membership</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <h2 class="links-heading mb-4">Links from the show</h2> <div><strong>Guest</strong><br/> <strong>Pamela Fox</strong>: <a href="https://pamelafox.org?featured_on=talkpython" target="_blank" >pamelafox.org</a><br/> <br/> <strong>Meetup: SQLite Might *Actually* Be All You Need</strong>: <a href="https://talkpython.fm/meetups/sqlite-might-actually-be-all-you-need" target="_blank" >talkpython.fm</a><br/> <br/> <strong>MCP</strong>: <a href="https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro?featured_on=talkpython" target="_blank" >modelcontextprotocol.io</a><br/> <strong>WeAreDevelopers workshop</strong>: <a href="https://pamelafox.github.io/github-copilot-mcp-skills-workshop/?featured_on=talkpython" target="_blank" >pamelafox.github.io</a><br/> <strong>slide 16</strong>: <a href="https://pamelafox.github.io/github-copilot-mcp-skills-workshop/#/16" target="_blank" >pamelafox.github.io</a><br/> <strong>To MCP or not to MCP</strong>: <a href="http://blog.pamelafox.org/2025/07/to-mcp-or-not-to-mcp.html?featured_on=talkpython" target="_blank" >blog.pamelafox.org</a><br/> <strong>MCP: Bringing mashups back</strong>: <a href="http://blog.pamelafox.org/2025/07/mcp-bringing-mashups-back.html?featured_on=talkpython" target="_blank" >blog.pamelafox.org</a><br/> <strong>MCP auth session</strong>: <a href="https://www.youtube.com/watch?v=_Redi3ChzFA" target="_blank" >www.youtube.com</a><br/> <strong>Pydantic AI \+ Playwright</strong>: <a href="http://blog.pamelafox.org/2026/08/browser-automation-with-pydantic-ai.html?featured_on=talkpython" target="_blank" >blog.pamelafox.org</a><br/> <strong>post</strong>: <a href="http://blog.pamelafox.org/2026/03/do-stricter-mcp-tool-schemas-increase.html?featured_on=talkpython" target="_blank" >blog.pamelafox.org</a><br/> <strong>PyAI talk</strong>: <a href="https://www.youtube.com/watch?v=l0McGgmngeU" target="_blank" >www.youtube.com</a><br/> <strong>post</strong>: <a href="http://blog.pamelafox.org/2026/08/building-safe-mcp-servers-for-your.html?featured_on=talkpython" target="_blank" >blog.pamelafox.org</a><br/> <strong>POSETTE talk</strong>: <a href="https://www.youtube.com/watch?v=3_JPHuXgDyQ" target="_blank" >www.youtube.com</a><br/> <strong>post</strong>: <a href="http://blog.pamelafox.org/2025/09/filter-tools-from-mcp-servers.html?featured_on=talkpython" target="_blank" >blog.pamelafox.org</a><br/> <strong>Can MCP choose my outfit</strong>: <a href="http://blog.pamelafox.org/2026/03/can-mcp-choose-my-outfit.html?featured_on=talkpython" target="_blank" >blog.pamelafox.org</a><br/> <strong>Talk Python now has a CLI</strong>: <a href="https://talkpython.fm/blog/posts/talk-python-now-has-a-cli/" target="_blank" >talkpython.fm</a><br/> <strong>FastMCP</strong>: <a href="https://gofastmcp.com?featured_on=talkpython" target="_blank" >gofastmcp.com</a><br/> <strong>Typer</strong>: <a href="https://typer.tiangolo.com?featured_on=talkpython" target="_blank" >typer.tiangolo.com</a><br/> <strong>Click CLI</strong>: <a href="https://click.palletsprojects.com/en/stable/?featured_on=talkpython" target="_blank" >click.palletsprojects.com</a><br/> <strong>Boldin</strong>: <a href="https://www.boldin.com?featured_on=talkpython" target="_blank" >www.boldin.com</a><br/> <strong>Sentry MCP</strong>: <a href="https://mcp.sentry.dev?utm_source=talkpython" target="_blank" >mcp.sentry.dev</a><br/> <strong>X.com MCP</strong>: <a href="https://docs.x.com/tools/mcp?featured_on=talkpython" target="_blank" >docs.x.com</a><br/> <br/> <strong>Watch this episode on YouTube</strong>: <a href="https://www.youtube.com/watch?v=F5-DtOl-9oM" target="_blank" >youtube.com</a><br/> <strong>Episode #569 deep-dive</strong>: <a href="https://talkpython.fm/episodes/show/569/connecting-agents-to-data-via-mcp-cli-and-more#takeaways-anchor" target="_blank" >talkpython.fm/569</a><br/> <strong>Episode transcripts</strong>: <a href="https://talkpython.fm/episodes/show/569/connecting-agents-to-data-via-mcp-cli-and-more#transcript-section" target="_blank" >talkpython.fm</a><br/> <br/> <strong>Theme Song: Developer Rap</strong><br/> <strong>🥁 Served in a Flask 🎸</strong>: <a href="https://talkpython.fm/flasksong" target="_blank" >talkpython.fm/flasksong</a><br/> <br/> <strong>---== Don't be a stranger ==---</strong><br/> <strong>YouTube</strong>: <a href="https://talkpython.fm/youtube" target="_blank" ><i class="fa-brands fa-youtube"></i> youtube.com/@talkpython</a><br/> <br/> <strong>Bluesky</strong>: <a href="https://bsky.app/profile/talkpython.fm" target="_blank" >@talkpython.fm</a><br/> <strong>Mastodon</strong>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i> @talkpython@fosstodon.org</a><br/> <strong>X.com</strong>: <a href="https://x.com/talkpython" target="_blank" ><i class="fa-brands fa-twitter"></i> @talkpython</a><br/> <br/> <strong>Michael on Bluesky</strong>: <a href="https://bsky.app/profile/mkennedy.codes?featured_on=talkpython" target="_blank" >@mkennedy.codes</a><br/> <strong>Michael on Mastodon</strong>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i> @mkennedy@fosstodon.org</a><br/> <strong>Michael on X.com</strong>: <a href="https://x.com/mkennedy?featured_on=talkpython" target="_blank" ><i class="fa-brands fa-twitter"></i> @mkennedy</a><br/></div>

November 16, 2026 08:21 PM UTC


Ed Crewe

From Routing Checks to Trajectory Testing: Evaluating an Agentic Chatbot

Which Agentic Chatbot?

I have been working on a Python based AI test framework for a chatbot interface for my company's product, Postgres AI Hybrid Manager. The manager allows the setup of Postgres clusters across cloud or on-prem and attaching various AI tools such as Langflow.  So a combination of more traditional Postgres backup, migration, telemetry and analytics features along with LLM workflows leveraging the data it holds.

The product already has a control plane UI for managing Postgres estates. It also has full help for the product, all Postgres versions, analytics, AI and add ons. The chatbot brings all these things together: ask a question, get the relevant help, or ask it to do something such as migrate a cluster, or evaluate telemetry that would otherwise require clicking through the UI.

That makes it a pretty handy interface, especially for the less technical. However it is not simple to test and ensure good quality responses.

A normal deterministic API test is simple. Send a request, check the status code, check the JSON body, perhaps check the database state. An LLM-backed agent does not pass or fail so clearly. It can route to the wrong capability and still return fluent text. It can pick a plausible but wrong tool. It can miss half the task and still sound confident. It can complete the first turn of a conversation and lose the plot on the second. It could get malformed or missing data from tooling that leads it to deliver a misleading conclusion. It might only provide help to something that should be from tool data or was a request for an action such as create a cluster.

So the testing problem was not “does the chatbot return a reasonable response?” It was “how do we test the whole chat path is doing the right thing?”

This is the story of how our agent-eval test framework evolved as we worked to see that our chatbot was not only getting the right answer,  42 , but whether it was asking all the right questions of the right tools to get that answer. Known as trajectory testing ...


 You're Golden 

Before we can tell our story we need to define some terms.

A Golden is an example of a perfect desired output from a test input. They often refer to more complex outputs that may need saving as separate files, but a simple assertable output such as 42, is a golden too!
Whilst complex goldens may be used and marked for semantic similarity against the test output. It is more common for complex outputs to be described by a rubric. A rubric is a checklist of qualitative properties a good answer must exhibit, written in plain English as opposed to a golden example of an answer.

For AI testing the tests are termed evals, ie they evaluate the tool, but not by strict assertions, because one thing you can be sure of with an LLM is that given the same input, you usually get subtly different output, ie they are non-deterministic. Which means for LLM outputs the only way to test them is to use an LLM-as-judge,  ie give that LLM the test output and a rubric or golden and let it mark it against that. Then you set a pass threshold for that mark, to translate your complex output into a pass or fail.

You can also total up all the passes to give you a Task Completion Rate, TCR. So with complex AI agentic LLM interactions a 100% pass of all evals is often not realistic. Hence you set a TCR below 100% for the whole test suite of evals to pass. Start with the smallest useful test. The core principle of evals is not complicated, you want the input to give you the expected output.

But for an Agentic application this may require a sequence of LLM calls and tools: Making the final output dependent on the route that should be chosen, the tool(s) that should be called, the actions to be taken, further LLM calls that may be necessary and finally the core data that the response to the user should contain.

Our first version did not try to solve every part of that. It started with routing, simple and deterministic.

Routing is the starting point

The chatbot originally had an agent per tool. The tool being the code and API calls that performed actions or returned data or help.

Different specialist agents owned different parts of the product surface: Control-plane actions, Postgres database operations, schema design, roles and permissions, cluster reporting, migration, and so on.

Before any specialist can help, something has to choose the right specialist.

So the first eval suite asked a narrow question:

Given this user prompt, did the chatbot route to the expected tool?

That gave us a fast health check. We could keep a corpus of prompts, map each one to an expected destination, run them through either a direct model path or against the real deployment and its tools, and score whether the selected destination tool matched the golden.

 A golden here is just the name of the tool:

- id: "core-iam-001"
    prompt: "List all my projects"
    expected_tool: "control-plane"
    tags: ["core", "control-plane", "project"]

And the check on the other end is deliberately dumb — an equality test, not a semantic one:

self.success = tool_match(predicted_tool, expected_tool)

Agents became skills, but routing remained 

The design moved away from “one agent per tool family” toward a more consolidated orchestrating agent with skills that could better compose different tool use for common tasks.

That is a better fit for how modern agent systems are evolving. A skill = instructions, constraints, and a subset of tools that are relevant for a task. It is a form of progressive disclosure. Give the model the minium it needs at each step to save tokens.

But this did not make routing irrelevant.

Instead of asking “did we transfer to the right sub-agent?”, the eval asks “was the right skill made visible and selected for this task?” The labels changed but a skill could still use the wrong tool.

Routing evals stayed valuable because they were fast, explainable, and easy to run in CI. But they are limited, routing should always be correct but it doesn't mean that the final agent response is too.

TCR jumps to the endpoint, the response

Task Completion Rate, or TCR, was the next step.

The user asked for a cluster comparison, or a schema recommendation, or help diagnosing a database issue. We need to know whether the full response actually completed these tasks.

Responses are complex goldens so they need the LLM-as-a-judge pattern: run the chatbot, take the actual response, and ask a judge model to score it against expected sections.

The eval has a rubric here for judging the output:

- id: "tcr-core-014"
  prompt: "Compare CPU usage between these two clusters"
  expected_sections:
    - "identifies which cluster has higher CPU usage"
    - "cites at least one supporting metric"
    - "suggests a plausible next step"

The judge gets one simple instruction: score each expected_sections between 0.0–1.0 A metric class then just thresholds it for pass / fail:

self.success = score >= 0.7

The judge must be calibrated and a consistent model used for comparing runs over time. Consistent judging enables skill and/or prompt tuning from metric trends. The rubric must be specific enough to avoid marking waffle as success. But it turns a non-deterministic complex output into a simple pass and fail. It also separated two different levels of QA:

That led to two execution modes.

Direct mode calls the model with simulated context. It is faster and useful for prompt and rubric development.

Proxy mode calls the real chatbot. It is slower, but it exercises the production path: routing, skill selection, tool calls, guardrails, streaming responses, conversation state, and the actual service wiring.

Both matter. Direct mode tells you whether the model is capable of the answer. Proxy mode tells you whether your product is capable of really delivering it via agents running your deployment's tools. 

This is the major difference from standard AI LLM testing, the model is only a small pluggable engine for the full agentic skill set that requires the actual deployment domain of data, actions and tools. Direct mode testing of only the model, is occasionally useful but E2E testing of the Chatbot deployment is required for agentic AI Chatbot QA, tuning and validation.   

Multi-step conversations changed the unit of testing

Single-turn TCR is still too small for many real chatbot tasks.

Users do not always provide all required information in one message. They ask to create a cluster, then pick a project, then choose a size, then confirm. They ask for a schema review, then refine the problem, then ask for a migration path. They troubleshoot by adding information over time.

So the framework has to exercise test cases that are conversations, not just single prompts.

That sounds like a minor data-model change. It was not. Once a test has steps, the eval runner has to preserve conversation state. In proxy mode, that means carrying the real conversation_id returned by the chatbot and sending each follow-up as part of the same server-side conversation. In direct mode, it means building a synthetic conversation history so the model sees the prior turns.

In code that split is about as literal as it sounds. Proxy mode threads a real id through each call:

response = client.send_message(prompt=msg, conversation_id=conversation_id)
conversation_id = response.conversation_id  # captured on turn 1, reused after

Direct mode has no server-side conversation to lean on, so it fakes one by re-rendering the transcript into the prompt itself, every turn:

full_prompt = f"## Conversation History\n{render(history)}\n\n{next_prompt}"

Same test case, same expected outcome, but a different code path depending on which half of the system is actually holding the conversation state. That's impacts multi-turn evals because conversation memory is part of the harness code for the actual deployment not just a model issue.

The scoring also becomes more interesting. You want per-step checks, because the assistant should ask the right clarifying question at the right time. You also want an overall score, because a conversation can have reasonable individual turns and still fail to complete the user's goal.

Coding it yourself: deepeval underneath

Everything above sits on top of deepeval, the open-source LLM eval library. We add a Synthesize → Execute → Evaluate pipeline, a plugin system, YAML goldens, CI wiring, and Langfuse push on top of it But the core library underneath is plain deepeval, and you do not need any of the surrounding machinery we used. Here are routing, TCR and multi-step just built directly on deepeval (simplified deepeval 3.6.9)
NB: Once tests have stabilised, the synthesize output should be retained and the step skipped and just rerun as part of test modification PR merges.

A test case is just an input/output pair. LLMTestCase is the base unit everything else scores:

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="List all my projects",
    actual_output=chatbot_response_text,       # what the system under test said
    expected_output="control-plane",                  # the golden - a skill label here, not prose
    additional_metadata={"predicted_skill": predicted_skill},
)

Routing is a custom metric, not a built-in one. deepeval ships plenty of semantic metrics, but “did it route to the right skill” is an exact-match business rule, so you write your own BaseMetric. This is a simplified version of the same shape our real AgentMatch metric takes:

from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase

class AgentMatch(BaseMetric):
    def __init__(self, threshold: float = 1.0):
        self.threshold = threshold
        self.async_mode = False  # routing checks are cheap; no need for async here

    def measure(self, test_case: LLMTestCase) -> float:
        predicted = test_case.additional_metadata["predicted_skill"]
        expected = test_case.expected_output
        self.score = 1.0 if tool_match(predicted, expected) else 0.0
        self.success = self.score >= self.threshold
        return self.score

    async def a_measure(self, test_case: LLMTestCase) -> float:
        return self.measure(test_case)

    def is_successful(self) -> bool:
        return bool(self.success)

    @property
    def __name__(self):
        return "Agent Match"

tool_match is the check from earlier. Run it with deepeval's own runner rather than hand-rolled assertions, and you get retries, pretty output, and a result object for free:

from deepeval import evaluate

evaluate(test_cases=[test_case], metrics=[AgentMatch()])

TCR is where deepeval's built-in GEval earns its keep. GEval is deepeval's off-the-shelf LLM-as-judge metric, you give it criteria (or explicit evaluation steps) and it handles the judge prompt, the JSON parsing, and the scoring for you. Our rubric-per-line expected_sections maps onto evaluation_steps almost directly:

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

task_completion = GEval(
    name="TaskCompletion",
    evaluation_steps=[
        "Check whether the response identifies which cluster has higher CPU usage",
        "Check whether the response cites at least one supporting metric",
        "Check whether the response suggests a plausible next step",
    ],
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.7,
)

test_case = LLMTestCase(
    input="Compare CPU usage between these two clusters",
    actual_output=chatbot_response_text,
)

evaluate(test_cases=[test_case], metrics=[task_completion])

Multi-step conversations get their own test case type. ConversationalTestCase takes a list of Turns instead of a single input/output pair, and pairs with a BaseConversationalMetric instead of BaseMetric:

from deepeval.test_case import ConversationalTestCase, Turn

convo = ConversationalTestCase(
    turns=[
        Turn(role="user", content="Create a new cluster"),
        Turn(role="assistant", content="Sure - which project should it go in?"),
        Turn(role="user", content="acme-prod"),
        Turn(role="assistant", content=final_response_text),
    ],
    expected_outcome="A cluster is created in acme-prod after resolving the missing project name",
)

deepeval has a conversational counterpart to GEval too (ConversationalGEval), scored against the whole turn sequence rather than a single response which is the natural fit for “did the assistant ask the right clarifying question at the right time”, the per-step-plus-overall shape TCR needed once prompts became conversations.

Put together, that is the whole starting kit: LLMTestCase plus a hand-written BaseMetric for hard business rules like routing, GEval for rubric-style task completion, ConversationalTestCase plus ConversationalGEval once a prompt becomes a conversation, and evaluate() to run the lot and get a result object back.

Everything else we built, the YAML goldens, the plugin architecture, the CI wiring, the Langfuse push exists to run more of these at scale and make the failures easy to find. But none of it is required to get started. If you are testing your own agentic chatbot, this is how to begin.

This is where instrumentation started to matter much more.

For a single-turn answer, a markdown report with pass/fail rows is often enough to start debugging. For multi-step conversations, that is thin. You need to know which turn failed, whether the route changed, whether the wrong tool was called, whether the tool call used correct arguments, whether the model forgot earlier context, or whether the final answer simply missed a required section.

That is why we added span-level telemetry and pushed eval traces into Langfuse.

Langfuse made the failures inspectable

The useful thing about Langfuse is not just having another pretty dashboard. Although that is important for spotting quality regressions over time via regular CI/CD automated runs.

The vital thing was being able to treat an eval run as a set of traces. A run becomes a session. Each test case becomes a trace. The trace carries the prompt, response, scores, tags, model, mode, scenario, and the spans emitted by the proxy.

For a chatbot path, those spans are where the debugging starts. You can see routing, tool execution, LLM calls, latency, and token usage where it is available. You can filter by scenario and model. You can compare runs. You can look at a failing conversation and see whether the problem began at route selection, tool selection, tool arguments, or final synthesis.

That changes the tuning loop.



Without traces, an eval failure says “this case failed”. With traces, it can say why it failed.

That distinction matters because the fix lands in different places...
Is it a routing rule?
Is it a skill description?
Is it a tool schema?
Is it the judge rubric?
Is it that the eval has has an expectation that the product has never actually promised?

 Trajectory testing ->  knitted the pieces together

Routing and TCR started as separate signals.

Routing asked whether the right capability was selected. TCR asked whether the final task was completed. Multi-step testing asked whether that held across a conversation. Instrumentation showed what happened between those points.

Trajectory testing is the next natural step: score the path itself.

For an agentic product, the fully correct path is essential to response quality.
So trajectory tests add expectations about intermediate actions:

The label-based routing tests are still useful as fast canaries. They tell us whether the classifier shape has drifted and distinguish tiers - see the next section.
But full trajectory tests judge the route by consequence: did the system actually follow the tool path that would satisfy the user?

So retain the fast determisitc routing tests, but move more user-visible behavioural coverage into trajectory and TCR.

Sovereign AI makes the eval problem tiered S/M/L/XL

There is one more constraint that makes this more than a generic chatbot-testing story.

Our chatbot has to work for sovereign and air-gapped deployments. In those environments, prompts, tool results, schema details, and operational data cannot be sent to a hosted frontier model outside the customer's trust boundary. The inference model may run inside the customer's environment.

That usually means a smaller model.

Smaller models are not just cheaper versions of larger ones. They have different context limits, weaker tool-selection behaviour, and less tolerance for an over-wide capability surface. If you show a smaller model every possible tool and skill, you have increased the chance that it chooses a bad one.

So the architecture becomes tiered. Models are effectively T-shirt sized. A small self-hosted model sees a curated subset of reliable skills. A larger model can be allowed to see more. Some experimental or complex skills only make sense for the highest tiers.

That changes the meaning of a routing eval again.

The correct visible skill set is no longer universal. It depends on the model tier. A prompt that should route to an advanced skill for an XL model may need to be dropped, refused, or handled differently for a smaller model that should not see that skill at all.

This is why trajectory testing and routing need to be tier-aware. We are not only asking whether the chatbot can complete a task. We are asking whether it can complete the task through the capability surface that a deployment's LLM size allows.

What I would keep from the journey

The final shape was not obvious at the start.

We began with routing because it was the first integration failure point and the cheapest one to isolate. We added TCR because correct routing did not prove task completion. We added multi-step cases because real users have conversations, not isolated prompts. We added telemetry because multi-step failures are otherwise too hard to debug. We moved toward trajectory testing because the route, tools, arguments, and answer need to be judged as one path.

If I were starting another agentic product eval framework, I would keep that order.

Do not start by trying to build a grand universal benchmark. Start with the smallest failure point that would embarrass the product if it regressed. Then move the signal closer to the user's actual goal.

For a chatbot wired into a real control plane, that means testing more than the output text. It means testing the route, the skill, the tool call, the arguments, the conversation state, the final answer, and the model tier that made those options visible in the first place.

That is the difference between checking that an AI system said something vaguely relevant and checking that it actually did all the things the user asked of it.

November 16, 2026 09:13 AM UTC


Django Weblog

2027 DSF Board Nominations

The Django Software Foundation is inviting nominations for its 2027 Board of Directors. Five of the seven Board positions are open for election this year.

The Board of Directors play a central role in the Foundation's work, overseeing its governance, fundraising, marketing, and outreach, and supporting Django events and initiatives across the globe. Directors serve staggered two-year terms.

Nominations are open to all. You do not need to be a DSF member or have previous Board experience to apply. We welcome applications from anyone interested in contributing to the Foundation and the future of Django.

An interesting time to join

This is a particularly interesting year to join the Board. For the first time in 21 years, the Foundation is hiring an Executive Director. Joining the Board means being part of this important governance transition, alongside the opportunity to shape and contribute to the DSF’s initiatives.

We look forward to receiving applications from a diverse range of backgrounds and experiences.

Thank you

A special thank you to Jacob Kaplan-Moss for his guidance and contributions during his time on the Board. Jacob has decided to step down after completing one year of his term.

We also thank Jeff Triplett, Abigail Gbadago, Paolo Melchiorre, and Tom Carrick for the time, dedication, and effort they have invested in the Foundation as they complete their two-year terms.

Note on Board terms: Four of the five positions open for election will carry two-year terms and one position will be a one-year term (completing the term originally held by Jacob Kaplan-Moss) to maintain the Board’s staggered election cycle.

How to apply

If you are interested in serving on the Board, please fill out the 2027 DSF Board Nomination Form by 23:59 on December 2, 2026 Anywhere on Earth (AoE).

For questions about the role or nomination process, feel free to contact foundation@djangoproject.com.

We look forward to your applications!

~ The 2026 DSF Board of Directors

November 16, 2026 05:33 AM UTC


Python Insider

XTM Transifex sponsors Python’s documentation translations

XTM Transifex is supporting Python’s documentation translation teams with a year of free access to its Growth plan.

November 16, 2026 12:00 AM UTC


Graham Dumpleton

Muru: scaling from what the workers report

In the Kubernetes post the scaling was done from outside. Muru published how busy its worker slots were, and a HorizontalPodAutoscaler added pods when the number went over a target. That works, but it can only ever add pods, and it has no idea why the slots were busy. Back on a host running the Muru daemon, Muru decides for itself how many worker processes each release has, and because it sits between the requests and the workers, it can say rather more about why requests are waiting than a single number. The same scaling policy also applies under muructl serve in a container, where another worker is only added if it fits under the container's memory limit, but here I am back with the daemon.

Most Python web servers don't scale the number of processes at all. Gunicorn starts the number of workers it is told to, and the only way to change that while it runs is to send it SIGTTIN or SIGTTOU, which add or remove one worker. Uvicorn and granian both take a fixed --workers count, and mod_wsgi's daemon mode has a fixed number of processes set in the Apache configuration. The exception is uWSGI, whose cheaper subsystem can spawn and stop workers based on how many are busy, or on the backlog of connections waiting to be accepted.

So adding workers when they are all busy isn't new, and if that were all Muru did it wouldn't be worth a post. What is different is what Muru does with everything else it knows about the workers, and whether more workers is even the right answer.

Growing and shrinking

The application for this post has a few pages which each misbehave in a different way. The first, /orders, sleeps for a fifth of a second as if waiting on a database. The release starts with one worker of two threads, and has a scaling policy in spare mode, which tries to keep between one and three idle workers in hand, much as Apache httpd does with spare processes or threads:

[tool.muru]
entrypoint = "app:application"
processes = 1
threads = 2

[tool.muru.scaling]
mode = "spare"
min_workers = 1
max_workers = 12
min_spare = 1
max_spare = 3
scale_in_window = "30s"
hold_after_growth = "30s"
reap_interval = "5s"
min_lifetime = "10s"
idle_reap_after = "45s"

The last five settings shorten windows which default to minutes, so it all happens quickly enough to watch. I deployed it to a daemon and set sixteen clients asking for /orders over and over for a minute. Three seconds in, muructl scale status showed the group to be well short:

shop@r1
GROUP        WORKERS      SLOTS      IDLE    QUEUE-P95     REAPS      POLICY
default      1            2/2        -7.00   under 2.5 s   0/6        spare 1-12 workers, 1-3 idle
  workers: cpu -, gc -, gil wait p95 -, loop lag p95 -
  diagnosis: waiting for the workers' first report

Both slots were busy, the requests waiting at the edge were counted as seven workers' worth of slots wanted, and the 95th percentile of the time requests waited for a slot was in the seconds. The workers hadn't yet sent their first report, so the diagnosis had to wait for it. Once the shortage had held for the five second scale-out window, the daemon forked more workers, at most four at a time:

default: scale out from 1 to 5 workers: short by 7.00 workers for 5999ms; diagnosis: waiting for the workers' first report
default: scale out from 5 to 9 workers: short by 3.00 workers for 5999ms; diagnosis: I/O bound with exhausted slots; ...

Each new worker is a fork of the host which already imported the application, as covered in the import once, fork many post, so the second step came only seven seconds after the first. From then on the group sat at nine workers, sixteen of eighteen slots busy and one worker in hand, and the queue wait was under a millisecond.

After the load stopped, the workers were taken away again, but slowly, one at a time:

default: scale in from 9 to 8 workers: at least 9.00 idle workers over the last 30s, over max_spare 3; diagnosis: idle
default: scale in from 8 to 7 workers: at least 8.00 idle workers over the last 30s, over max_spare 3; diagnosis: idle
...
default: scale in from 4 to 3 workers: at least 4.00 idle workers over the last 30s, over max_spare 3; diagnosis: idle

and then stopped at three, with more than it needed:

default      3            0/6        3.00    -             6/6 halt   spare 1-12 workers, 1-3 idle
  ...
  last: suspended: 6 workers reaped in the last hour, the churn budget; scale-in is suspended and the scale-in window doubled until it recovers (1s ago)

Each release has a churn budget of six reaps an hour. I had shortened every window but left that alone, so it was spent and scale-in was suspended. It is there so that a badly tuned policy can't keep a group swinging up and down all day.

When the answer is more threads

Once the workers had reported, the status had more to say. This is from while the group was still growing:

default      5            10/10      -1.00   under 2.5 s   0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: I/O bound with exhausted slots; for the next release, more threads per worker, or more concurrency for an ASGI application

The group's slots were all busy, but its workers were using no CPU at all. That is what waiting on a database looks like, and nine processes is a costly way to hold sixteen requests that are each just waiting. The suggestion was more threads per worker, and that can't be done by the autoscaler. A release's processes and threads are fixed when it starts, so the change goes into the next release.

I deployed the same application with threads = 8 and ran the same load. Instead of nine workers it settled at three:

default      3            16/24      1.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 1%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity

Over the minute it served 4440 requests compared with 4007, the 99th percentile response time was 0.409 seconds where before it was 1.623, as it needed only one step to scale out rather than two, and each of the three workers had 12MB of memory of its own:

WORKER  PID      GROUP      UP       REQUESTS   THREADS  THREAD-OLDEST  RSS     USS     ROUTED  STATE
0       67546    default    38s      1090       6/8      0.2s           18MB    12MB    yes     ready
1       67729    default    31s      813        5/8      0.0s           17MB    12MB    yes     ready
2       67730    default    31s      790        5/8      0.2s           17MB    12MB    yes     ready

When the answer is fewer threads

More threads isn't always the answer. In the import once, fork many post I showed four threads of CPU bound code running four times slower than one, because they had to take turns holding the global interpreter lock, and said Muru kept track of time spent waiting for the GIL. This is where that comes in.

The /prices page does about 13 milliseconds of pure Python arithmetic. With the eight thread release still serving, and running on CPython 3.15 with the GIL, eight clients asking for it gave:

default      3            8/24       2.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 100%, gc 0%, gil wait p95 under 50 ms, loop lag p95 -
  diagnosis: GIL bound; for the next release, more processes with fewer threads each

By the measure the autoscaler uses, there was nothing to do. Only eight of the twenty-four slots were busy and there were two idle workers in hand. Yet the three workers were each using all of the one core the GIL lets them have, and the 95th percentile of the time threads with work to run waited for their turn was put at under 50 milliseconds. A request which takes 13 milliseconds on its own had a median time of 20 milliseconds.

An autoscaler working from busy slots or a connection backlog, Muru's own or uWSGI's, can't see this. Adding threads would make it worse, and adding workers wouldn't happen because the slots aren't full. The diagnosis says what would help, which is more processes with fewer threads each. I deployed the release again with one thread per worker:

default      9            8/9        1.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 86%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: CPU bound; for the next release, more processes, up to the node's cores; more threads will not help

Now each request occupied a whole worker, the slots filled, and the autoscaler took the group to nine workers on this ten core machine. It served 24863 requests in the 45 seconds compared with 18626, and the median time was 13 milliseconds.

The other way around the GIL is to not have one. I deployed the eight thread release again, with nothing changed but a .python-version of 3.15t, so uv built its environment with the free-threaded build of Python 3.15:

default      2            7/16       1.12    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 48%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity

Two workers of eight threads served 26040 requests in the same 45 seconds, a little more than the nine single threaded processes, with nothing waiting for a GIL. The CPU figure is now a share of a core per thread, as threads on a free-threaded build really can run in parallel.

When the answer is less garbage

The /search page makes short lived objects, 40,000 dictionaries per request, in pairs which refer to each other. With a release of four threads per worker and four clients:

default      2            4/8        1.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 100%, gc 73%, gil wait p95 under 5 ms, loop lag p95 -
  diagnosis: paused by garbage collection of short-lived reference cycles; for the next release, break the cycles, or use weakref for the references back

Close to three quarters of the workers' time was going into garbage collection. A collection stops every thread in the process, so that time is lost to every request in flight.

The diagnosis says why. A dictionary only needs the garbage collector, rather than being freed as soon as nothing refers to it, when it is part of a reference cycle, and muructl debug memory showed the youngest generation's collections freeing hundreds of millions of objects. Long lived objects being scanned over and over don't get freed, so this was garbage being made and collected. Removing the reference back was a one line change:

default      2            4/8        1.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 100%, gc 0%, gil wait p95 under 1 ms, loop lag p95 -
  diagnosis: CPU bound; for the next release, more processes, up to the node's cores; more threads will not help

The garbage collection share went to zero, and in 40 seconds the release served 73637 requests where before it had served 17977, with the median time down from 8 milliseconds to 2. The diagnosis then moved on to the next thing holding the group back, which was now plain CPU.

When the event loop is blocked

An ASGI application has one event loop per worker rather than a pool of threads, and one blocking call in a coroutine holds up everything else on that loop. This release had an /invoice page calling a function which blocks for a fifth of a second, as a synchronous client library would, directly from the coroutine. With eight clients on a page which awaits for 50 milliseconds, and one client on /invoice:

default      2            9/2000     1.99    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 2%, gc 0%, gil wait p95 under 0.5 ms, loop lag p95 under 500 ms
  diagnosis: event loop held by blocking code; for the next release, await blocking calls, or run them in a thread, so the loop keeps serving

Each worker's event loop has a heartbeat twice a second, and the lag is how late it runs. Here it was hundreds of milliseconds late. The slots barely registered, nine busy of two thousand, which is the concurrency limit of two event loops, so nothing based on slots would ever have noticed. The page which should have taken 50 milliseconds had a 95th percentile of 0.410 seconds, as it kept getting stuck behind an invoice.

Changing the one line to body = await asyncio.to_thread(render_invoice), so the blocking call ran in a thread, brought the loop lag down to under 5 milliseconds and the 95th percentile back to 0.054 seconds. The eight clients got through 6064 requests in 40 seconds instead of 2960.

When the answer is a worker group

The last case is one slow endpoint making everything else wait. Here /reports takes two seconds and is used in bursts, ten requests at once every six seconds, alongside two clients steadily asking for /orders. The release had two workers of four threads and no scaling policy, but even a static release gets a diagnosis. Every few seconds it read:

default      2            2/8        1.50    under 2.5 s   -          static
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: a few heavy requests; for the next release, a worker group of their own for the slow paths

Most of the time the slots were mostly free, but the queue wait was in the seconds, because each burst of reports took every slot and /orders requests arriving then waited behind them. For /orders the median response time was fine at 0.208 seconds, but one in a hundred requests took over two seconds.

Scaling out would help for a moment, until the next burst took all the extra slots as well. What the diagnosis suggests is separating the slow paths, so I gave them a worker group of their own, forked from the same host:

[tool.muru]
entrypoint = "app:application"
processes = 2
threads = 4

[tool.muru.groups.reports]
paths = ["/reports"]
processes = 1
threads = 12

Each group then gets its own line, and its own diagnosis:

default      2            2/8        1.50    under 1 ms    -          static
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity
reports      1            10/12      0.17    -             -          static
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: I/O bound with exhausted slots; for the next release, more threads per worker, or more concurrency for an ASGI application

The reports now only compete with each other. For /orders, the slowest of the 389 requests took 0.212 seconds, where before the 99th percentile was 2.093. Each group can have its own scaling policy too, so a group like reports could be allowed to grow on its own when a burst arrives.

Scaling to a target

Spare mode is about keeping idle workers in hand. The other mode, target mode, keeps the share of busy slots near a target, and can also scale on how long requests wait:

[tool.muru.scaling]
mode = "target"
min_workers = 1
max_workers = 8
slot_utilization = 0.6
queue_wait_p95 = "100ms"

With eight threads per worker and the same sixteen clients on /orders, it went straight to the count which would carry the load at the target:

default      4            16/32      2.00    under 1 ms    0/6        target 1-8 workers, 60% of slots busy, queue wait p95 100 ms
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity
  last: scaled out to 4: 100% of slots busy, over the target of 60% for 6s (19s ago)

Sixteen busy slots at 60 percent needs about 27 slots, which is four workers of eight threads. This is much the same rule the HorizontalPodAutoscaler applied in the Kubernetes post, except that adding a worker here costs a fork rather than a new pod, and the diagnosis comes along with it.

What's next

What the daemon does about the worker count is fairly conventional. Grow quickly when requests wait, shrink slowly, and don't thrash. The part I find more useful is the diagnosis, because in every case above except the first, adding workers was either not going to happen or not going to help. Whether the answer was more threads, fewer threads, a free-threaded interpreter, less garbage, a thread for a blocking call or a group of its own for a slow path, it was a change to the next release, and you only know which by measuring what the workers are actually doing.

Scaling also goes the other way. A server with many small sites, previews or staging copies doesn't want each of them holding workers while nobody is using it. The next post, scaling to zero, is about taking a release all the way down to no workers, and what it then costs for the next request to wake it up again.

November 16, 2026 12:00 AM UTC

November 15, 2026


Speed Matters

zar-rs

ZAR: A New Archive Format with a Python API

Have you ever been bored because of slow compression/decompression tools? Because speed matters, I’ve developed a new archive format with very fast tools.

ZAR is a new archive format, not just another Python wrapper around ZIP or TAR. It is designed to combine fast parallel Zstandard compression with features that the familiar formats don’t offer together: content deduplication, versioned commits, historical restores, and recovery checks.

The format is implemented in Rust and comes with a native Python API. Create archives quickly, keep earlier file versions in the same archive, and restore part of the archive without unpacking everything.

And Python is just one way to use it: the same Rust library also powers a command-line tool, a desktop archive manager, and bindings for many other languages.

What makes the format different?

Most archive formats are great at packaging files. ZAR is built to manage changes to those files over time, too. Each update can add a commit while preserving earlier states, so you can inspect or restore history without keeping a separate full archive for every version.

Capability ZAR ZIP tar.gz 7z
Parallel, block-based Zstandard compression Yes No (Deflate) No (single-threaded gzip) No (typical LZMA workflows)
Commit-based updates and historical states Yes Appendable, not versioned No No
Within-commit content deduplication Yes No No Solid blocks only
Random access to a file Yes Yes No Limited by solid blocks
Per-block checksums and recovery scan past damaged regions Yes CRC; no recovery scan No Limited
Authenticated encryption with per-commit passwords Yes Varies No (requires a wrapper) AES-CBC

These are not just checkboxes: commits let you update an archive without discarding the old state, deduplication avoids storing duplicate content within a create or add operation, and block framing lets ZAR compress in parallel and verify individual blocks. The result is a format for people who want both quick archive operations and useful history.

Why Python developers should take a look

🚀 Fast compression, without leaving Python

The Python package, pyzarlib, is a PyO3 extension for the Rust archive library. ZAR splits data into blocks and compresses them in parallel with Zstandard. Long-running Rust operations release Python’s GIL, so other Python threads can keep working while an archive operation is running.

In one unencrypted, hot-cache benchmark on a Linux Core i7-10700, ZAR compressed a 525 MiB Linux source tree in 0.86 seconds. In the same run, ZIP took 11.01 seconds and 7z took 24.91 seconds. ZAR’s archive was 115.5 MiB—smaller than ZIP’s 151.1 MiB, though not as small as 7z’s 81.1 MiB. If you need a much smaller archive, 7z may be the better fit; if you want a compelling balance of compression time and size, ZAR is worth a try.

ZAR archive benchmark results on a Linux Core i7-10700 One hot-cache run on a Linux Core i7-10700; results vary by machine, storage, and data.

The full benchmark report includes extraction times, cold-cache runs, and details about the test setup.

🐍 Python API — and a CLI for Python workflows

Create, update, inspect, verify, and extract archives directly from Python. Inputs accept pathlib.Path as well as strings:

from pathlib import Path

import pyzarlib

archive = Path("backup.zar")

created = pyzarlib.create_archive(
    archive,
    [Path("src"), Path("README.md")],
)
print(f"Created with {len(created['warnings'])} warning(s)")

# Add a new commit. Earlier file versions remain in the archive history.
pyzarlib.add_archive(archive, [Path("README.md")])

# Store and read custom archive-level metadata.
pyzarlib.update_custom_metadata(
    archive,
    set={"project": "example", "revision": 1},
)
metadata = pyzarlib.read_custom_metadata(archive)
print(metadata["project"], metadata["revision"])

for entry in pyzarlib.list_archive(archive):
    print(entry["path"], entry["type"])

check = pyzarlib.check_archive(archive)
print(f"Verified {check['commits']} commit(s)")

# Preview a restore before writing files.
plan = pyzarlib.extract_archive(
    archive,
    "restored",
    dry_run=True,
)
print(f"Would extract {len(plan['files'])} files")

Custom metadata values can be strings, signed integers, or finite floats. Updating metadata appends a metadata-only commit; keys beginning with zar. are reserved. Metadata updates require an unencrypted archive.

To append an encrypted commit to the archive above and read its file back:

from getpass import getpass

password = getpass("Commit password: ").encode("utf-8")
pyzarlib.add_archive(
    archive,
    [Path("secrets.txt")],
    password=password,
    encryption_scope="header-and-data",
)

contents = pyzarlib.extract_archive_to_memory(archive, password=password)
print(f"Read {len(contents['secrets.txt'])} bytes")

Only the new commit is encrypted; earlier commits remain unchanged. Without the password, commits with encrypted headers are skipped, so a listing or extraction can show only part of the archive. For commits with different passwords, pass a passwords mapping from commit IDs to password bytes.

Need a terminal workflow? The package also provides zarc-py, a Python CLI for creating, adding, removing, joining, undoing, checking, listing, inspecting, and extracting archives.

The package supports Python 3.11 and newer, with CI coverage for CPython 3.11–3.15 and free-threaded CPython 3.14t and 3.15t. Native work releases the GIL, and the extension is tested with concurrent Python calls.

🗜️ Compression controls and familiar archive inputs

Choose a Zstandard compression level from -7 to 22 and a block size from 64 KiB to 32 MiB, or let ZAR select a block size automatically. Default compression level is 3. Files with identical content can be deduplicated within a create or add operation. Some already-compressed formats and media are stored without an extra compression pass to improve compression speed.

You don’t have to start with a ZAR archive. Python create_archive and add_archive can import entries from ZIP, 7z, RAR, ISO, TAR, TAR.GZ, TGZ, and GZ files, preserving their internal paths. Encrypted ZIP/7z/RAR inputs and some special archive entries are not supported; see the project documentation for the precise limits.

More than a compressed file

ZAR archives are built around commits:

That makes ZAR useful for more than a one-off download bundle: it can serve as a compact, inspectable history of project snapshots or application data.

One Rust core, many ways to use it

The shared archive library is available through:

Language or runtime Integration
Rust Native zarlib crate
Python PyO3 pyzarlib extension
C and C++ czarlib C ABI and cxxzarlib
Go gozarlib through cgo
Java FFM and JNI bindings
C# P/Invoke binding
WASI WebAssembly Component; Node.js integration is available

You can use the zarc command-line tool, manage archives in the zarg desktop application, or connect an AI client to the local zarmcp server. The companion zcopy tool transfers file trees over a network using compressed, optionally encrypted ZAR batches. Try it out. You’ll be impressed how fast you can transfer big file trees over the network without creating (temporary) archives.

Installation

Build the Python extension from the project source with Maturin:

python -m pip install maturin
cd pyzarlib
maturin develop --release

Then import pyzarlib in your Python code. The repository includes Python installation and API documentation, examples, and tests.

Get started

If you’re a Python developer who wants quicker archives without giving up a comfortable Python API, try ZAR on a representative dataset. The benchmark includes machine and method details so you can compare it against your own workload—not just trust a headline number.

Explore the source, complete documentation, and benchmark reports:

https://github.com/brmmm3/zar-rs

Licensed under the MIT or Apache-2.0 License.

November 15, 2026 07:00 AM UTC


Graham Dumpleton

Muru: working with Kubernetes, not around it

The last post was about muructl serve as the main process of a container: running the workers it is told to fit the container, answering probes on a port of its own, shutting down cleanly on SIGTERM, and running with a read-only root under any user ID. None of that cares what runs the container. Most of these containers end up on Kubernetes though, and Kubernetes has its own way of doing several things a Python web server would otherwise need help with. Rolling out new versions, spreading traffic, scaling, running one-off tasks, all of that is done by the platform already.

So the aim with Muru on Kubernetes is to fit in with what Kubernetes provides rather than duplicate it, and to fill the gaps where what Kubernetes provides doesn't quite reach the application.

The Deployment

The image is built the same way as in the last post, with a small WSGI application which has a page that waits as if on a database. The Deployment follows the one in the Muru documentation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
spec:
  replicas: 2
  selector:
    matchLabels: {app: shop}
  template:
    metadata:
      labels: {app: shop}
    spec:
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: shop
          image: muru-blog-k8s:1
          imagePullPolicy: IfNotPresent
          env:
            - {name: MURU_SERVE_PROCESSES, value: "2"}
            - {name: MURU_SERVE_THREADS, value: "4"}
            - {name: MURU_SERVE_SHUTDOWN_DELAY, value: "5s"}
            - {name: MURU_SERVE_SHUTDOWN_BUDGET, value: "25s"}
            - {name: MURU_SERVE_LOG_FORMAT, value: "json"}
            - {name: SHOP_DATABASE, value: "postgresql://db.shop/shop"}
            - name: POD_NAME
              valueFrom: {fieldRef: {fieldPath: metadata.name}}
            - name: POD_NAMESPACE
              valueFrom: {fieldRef: {fieldPath: metadata.namespace}}
            - name: NODE_NAME
              valueFrom: {fieldRef: {fieldPath: spec.nodeName}}
          ports:
            - {name: http, containerPort: 8080}
            - {name: admin, containerPort: 9090}
          startupProbe:
            httpGet: {path: /startupz, port: admin}
            periodSeconds: 2
            failureThreshold: 60
          livenessProbe:
            httpGet: {path: /livez, port: admin}
          readinessProbe:
            httpGet: {path: /readyz, port: admin}
            periodSeconds: 2
            failureThreshold: 1
          resources:
            requests: {cpu: 500m, memory: 512Mi}
            limits: {cpu: "2", memory: 1Gi}
          resizePolicy:
            - {resourceName: cpu, restartPolicy: NotRequired}
            - {resourceName: memory, restartPolicy: NotRequired}
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities: {drop: ["ALL"]}
          volumeMounts:
            - {name: run, mountPath: /run/muru}
            - {name: tmp, mountPath: /tmp}
      volumes:
        - name: run
          emptyDir: {sizeLimit: 2Gi}
        - name: tmp
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: shop
spec:
  selector: {app: shop}
  ports:
    - {name: http, port: 80, targetPort: http}

Much of this is what the last post covered, now expressed as Kubernetes configuration. All the settings come from environment variables. The probes go to the admin port, which the Service doesn't expose. The pod, namespace and node names come in through the Downward API so they end up in every JSON log record. I labelled the namespace to enforce the restricted Pod Security Standard and the pods were admitted without any warnings, as Muru only needs /run/muru and /tmp to be writable.

As in the last post, the number of processes and the threads in each are set explicitly, chosen to go with the pod's resources. With a CPU limit of 2 and a memory limit of 1Gi, that is two workers of four threads:

container limits: 2 CPUs from cpu.max 200000 100000, memory.max 1.0GB with 128MB of headroom
...
serving app:application on http://0.0.0.0:8080 with 2 workers of 4 threads

If the limits change, so should these. The memory limit is also what the memory guard from the last post works from, so a worker which grows is replaced before the pod is out of memory, with nothing more to set. The parts which are particular to Kubernetes are the shutdown delay, the readiness probe's failureThreshold and the absence of any preStop hook, and those are what the rest of this post is about, along with autoscaling and migrations.

Stopping without a preStop hook

When a pod is deleted, as happens to each old pod during a rolling update, two things start at the same time. The kubelet on the node begins shutting the pod down, which means sending SIGTERM to process 1 in each container. Separately, the control plane marks the pod's endpoint in the Service's EndpointSlices as terminating. Every kube-proxy, and any ingress controller, then has to notice that and stop sending new connections to the pod. Nothing makes the second wait for the first, so for a short while after SIGTERM a pod can still be handed new connections.

A server which closes its listening socket as soon as it gets SIGTERM, which is what gunicorn and uvicorn both do, will refuse those connections. The usual fix is a preStop hook which sleeps for a few seconds. The kubelet runs the hook before sending SIGTERM, so the server goes on serving during the sleep while the endpoint changes spread through the cluster. For a long time that meant an exec hook running sleep, which needs a sleep binary in the image. There is now also a sleep action, stable since Kubernetes 1.34, which the kubelet runs itself. Either way, it is something you have to know to add, and during the sleep the server has no idea it is about to be stopped. Neither gunicorn nor uvicorn has an option for this. Gunicorn's --graceful-timeout and uvicorn's --timeout-graceful-shutdown only govern what happens after the signal, by which time the listener is already closed.

Muru does this itself, with --shutdown-delay, set here through MURU_SERVE_SHUTDOWN_DELAY. On SIGTERM, /readyz starts failing at once. For the length of the delay the listener stays open and new connections are still served, but each response carries Connection: close, and idle connections are closed, so clients reconnect and land on another pod. After the delay, the listener closes, the requests in flight finish and the container exits, all within the budget, which is set below terminationGracePeriodSeconds.

To see what difference it makes, I ran a small Python client in a pod in the same namespace. It used eight threads, each on its own keep-alive connection to the Service, asking for a page which takes 50 milliseconds, for 40 seconds. Five seconds in I ran kubectl rollout restart deploy/shop. With the five second delay:

responses: {200: 6106}
errors: {} first at (s): {}
responses with Connection: close, by pod: {'shop-7f8f5b489f-bbnt5': 13, 'shop-7f8f5b489f-hl9zb': 5}

Every request succeeded, and the 18 responses which carried Connection: close were the client's connections being moved off the two old pods as each was stopped. Watching the EndpointSlice at the same time showed the first old pod being marked terminating:

shop-64df649bb5-x4hwq ready=true serving=true terminating=false
shop-7f8f5b489f-bbnt5 ready=false serving=true terminating=true
shop-7f8f5b489f-hl9zb ready=true serving=true terminating=false

That happened at 09:03:43.350. The last request that pod's access log recorded was at 09:03:43.455, and its host process didn't log receiving SIGTERM, which is when the workers start stopping, until 09:03:48.353, five seconds later. On a single node cluster, the endpoint changes reached kube-proxy within a fraction of a second.

So is the delay needed at all on a cluster this small? I set it to zero and ran the same test:

responses: {200: 6100}
errors: {'ConnectionRefusedError': 4} first at (s): {'ConnectionRefusedError': 11.4}
responses with Connection: close, by pod: {'shop-55885b469d-rxk2k': 2, 'shop-55885b469d-hftl9': 7}

Four requests failed because their connections were refused. I repeated both tests: four refused again with no delay, none again with the delay. Even when the endpoints catch up in a fraction of a second, there is a window where kube-proxy still routes new connections to a pod whose listener has closed. On a larger cluster, with an ingress controller in front, that window only gets longer, which is why the Muru documentation starts with five seconds.

Under Docker or Compose the delay defaults to zero, since a stopped container stops getting connections straight away. Under Kubernetes it is a single environment variable in the Deployment, and the pod spec has no preStop hook.

Resizing and restarting in place

As of Kubernetes 1.35 it is a stable feature that a running container's CPU and memory can be changed in place, through the pod's resize subresource, which a VerticalPodAutoscaler can also use. With the resizePolicy in the Deployment above, restartPolicy: NotRequired for both, no restart is needed, as the kubelet just changes the container's cgroup limits. Muru reads them again at its next memory check, which a pod I gave three CPUs and 1.5Gi showed in its log:

container limits changed: cpu.max from 200000 100000 to 300000 100000, memory.max from 1.0GB to 1.5GB

The memory guard then works from the new limit, and the warnings Muru gives at start are checked again, so a resize which leaves the workers short of CPUs or memory says so. What doesn't change is the number of processes and threads, which are configuration. Giving a pod more CPU without more workers to use it gains little, and giving it less memory only has the memory guard replace workers sooner. A change to the resources which ought to change the number of workers is better made to the Deployment, with the MURU_SERVE_* settings changed alongside the limits, so it rolls out as new pods, which the shutdown delay above already covers.

A resize did turn out to be a handy way of showing what happens when a container is restarted without its pod going anywhere, as also happens after a failed liveness probe. Setting restartPolicy: RestartContainer for memory in the resizePolicy instead has the kubelet restart the container for a resize. I did that and, with the client opening a new connection for each request, changed the memory limit of one pod. The container restarted in the same pod, on the same node with the same address.

The endpoint went to not ready 1.5 seconds after the resize, and nothing failed:

responses: {200: 4448}
errors: {} first at (s): {}
responses with Connection: close, by pod: {'shop-567fc9d9c7-46lpz': 59}

The reason nothing failed is the failureThreshold: 1 on the readiness probe. With a rolling update the pod is deleted, so its endpoint is marked terminating at once. A container restarting in a pod which isn't going anywhere doesn't get that. The only thing taking the pod out of the Service is the readiness probe, and by default a readiness probe has to fail three times in a row. Every two seconds, three failures take up to six seconds, which is longer than the five second delay. Taking the setting out and repeating the same resize under the same load shows what happens:

responses: {200: 4448}
errors: {'ConnectionRefusedError': 10} first at (s): {'ConnectionRefusedError': 10.0}
responses with Connection: close, by pod: {'shop-5dfdbd5847-nz5bh': 180}

The resize went in at 09:28:23.665. Muru started its shutdown delay, the kubelet's readiness probe got a 503 from 09:28:25, and Muru sent Connection: close on 180 responses. But the EndpointSlice still listed the pod as ready until 09:28:29.062, by which time the old container had exited.

A threshold of one would be a risky setting for a server which answers readiness from the same threads that serve requests, since one slow probe under load would take the pod out of the Service. With Muru, /readyz is answered by Muru itself and not by a worker, and as the last post showed it answers in a couple of milliseconds with every worker busy, so a failure means the pod really is stopping or has no ready workers.

Autoscaling on busy slots, not CPU

A HorizontalPodAutoscaler usually scales on CPU. For a Python web application which spends most of its time waiting on a database, that is close to useless. Every thread can be waiting, with requests queuing, while the CPU sits almost idle.

Muru publishes the measure that actually matters, muru_capacity_utilisation, the share of worker slots in use averaged over 30 seconds, on the admin port's /metrics. To get that to an autoscaler takes Prometheus to scrape it and the Prometheus adapter to present it as a pods metric through the custom metrics API. The pods got the usual scrape annotations:

template:
  metadata:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9090"

and the adapter got a rule for the one metric:

rules:
  default: false
  custom:
    - seriesQuery: 'muru_capacity_utilisation{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: namespace}
          pod: {resource: pod}
      metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

The autoscaler aims to keep the average across pods at 70 percent:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shop
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: shop}
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric: {name: muru_capacity_utilisation}
        target: {type: AverageValue, averageValue: 700m}

For load, the client ran 24 threads for four minutes, each asking over and over for a page which waits half a second as if on a database. Each request was on a new connection, so the Service spread them across however many pods there were, much as an ingress controller balances each request. Two pods have 16 slots between them, two workers of four threads each, so 24 requests at a time can't all be served at once. Checking every ten seconds:

NAME   REFERENCE         TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
shop   Deployment/shop   0/700m      2         10        2          40s
shop   Deployment/shop   391m/700m   2         10        2          50s
shop   Deployment/shop   853m/700m   2         10        2          61s
shop   Deployment/shop   839m/700m   2         10        3          81s
shop   Deployment/shop   754m/700m   2         10        4          91s
shop   Deployment/shop   732m/700m   2         10        4          2m1s
shop   Deployment/shop   726m/700m   2         10        4          3m2s

The autoscaler's events:

New size: 3; reason: pods metric muru_capacity_utilisation above target
New size: 4; reason: pods metric muru_capacity_utilisation above target

Within a minute of the load starting, the Deployment had gone from two pods to four. It then stayed at four, because 24 requests over 32 slots is 0.75, and the autoscaler ignores anything within its default tolerance of 10 percent of the target. While all that was happening, kubectl top showed the pods using next to no CPU:

shop-5dfdbd5847-d5ds9   10m   39Mi
shop-5dfdbd5847-nz5bh   10m   39Mi
shop-5dfdbd5847-pjmzj   11m   40Mi
shop-5dfdbd5847-pmfdt   10m   39Mi

That is about 2 percent of the 500m each pod requests. An autoscaler targeting 70 percent CPU would have left the Deployment at two pods, with a third of the requests waiting for a slot. When the load stopped, the metric dropped to zero, and the autoscaler took the Deployment back to two pods about five minutes later, its default stabilisation window for scaling in.

The other figure worth scaling on is how long requests wait for a slot, which is the queue phase of Muru's request duration histogram. Asking Prometheus for the 95th percentile ninety seconds into the run, with its two minute window covering the time at two pods:

histogram_quantile(0.95, sum by (le) (rate(muru_request_duration_seconds_bucket{namespace="muru-blog",phase="queue"}[2m])))

gave 0.457 seconds. Unlike the utilisation, this is a single figure for the whole Deployment rather than one for each pod, so it can't be given to the autoscaler in the same way. It can instead be set up as an external metric, through a separate rule in the Prometheus adapter, or by using KEDA in place of the adapter. KEDA is a project which manages a HorizontalPodAutoscaler for you and feeds it from sources such as a Prometheus query, so a query like this one can be given to it directly, with a threshold of, say, 100 milliseconds. Either way, the Deployment would then scale on how long users wait rather than on how busy the workers are.

Neither gunicorn nor uvicorn exposes anything like this. You can get at request latency through a Prometheus client in the application, but how many slots are busy and how long a request waited for one are things only the server knows.

Migrations as a Job

In the releases post, migrations ran from the new release, with muructl command run shop@r4 migrate, while the old release kept serving. On Kubernetes the new release is a new image, and the natural place to run something once from it before the Deployment changes is a Job. The project declares the command as before:

[tool.muru.commands.migrate]
description = "Apply database migrations"
run = ["python", "migrate.py"]
timeout = "10m"

and the Job runs it from the new image:

apiVersion: batch/v1
kind: Job
metadata:
  name: shop-migrate-2-0
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: migrate
          image: muru-blog-k8s:2
          imagePullPolicy: IfNotPresent
          command: ["muructl", "command", "run", "migrate"]
          env:
            - {name: SHOP_DATABASE, value: "postgresql://db.shop/shop"}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: {drop: ["ALL"]}

With no muructl serve running in the Job's pod, muructl command run runs the command itself, using the image's interpreter, the project's environment and the command's timeout. The Deployment is then only updated once the Job has completed:

kubectl apply -f migrate-job-2.yaml
kubectl wait --for=condition=Complete job/shop-migrate-2-0 --timeout=120s
kubectl set image deploy/shop shop=muru-blog-k8s:2
job.batch/shop-migrate-2-0 created
job.batch/shop-migrate-2-0 condition met
deployment.apps/shop image updated

The migration's output is in the Job's logs:

connecting to postgresql://db.shop/shop
applied 0007_order_notes
applied 0008_order_notes_index

A failed migration fails the Job, since muructl exits with the command's status. Running the migration in a Job of its own without the database setting:

SHOP_DATABASE is not set
muru: the command exited with status 1 after 8ms
NAME             STATUS   COMPLETIONS   DURATION   AGE
shop-migrate-2   Failed   0/1           3s         3s

A kubectl wait for that Job would never see it complete, so a deploy script running the three commands above would stop there rather than update the Deployment. With backoffLimit: 0 the Job doesn't try again, which for a migration that has failed halfway is usually what you want.

You could of course write command: ["python", "migrate.py"] in the Job and get much the same result with gunicorn in the image. What the declared command buys is that the Job, a kubectl exec into a running pod and the daemon from the releases post all run the migration the same way, from the one definition in the project, with its timeout.

Looking inside a pod

Everything the last post did with docker exec works through kubectl exec, since muructl in the container finds muructl serve through its run directory. With a request stuck in a page which sleeps for eight seconds:

kubectl exec deploy/shop -- muructl requests
RELEASE              GROUP        QUEUED  QUEUED-OLDEST  THREADS  THREAD-OLDEST
shop@r1              default      0       -              1/8      1.9s
kubectl exec deploy/shop -- muructl debug stacks
muru-app-2 in worker 1:
  serving GET /slow?seconds=8 for 1.9 s (request 01M6TNH4EM7EES4KX4129GEBM8)
  File "/opt/muru/muru/_pool.py", line 30, in serve_jobs
    serve()
  File "/app/app.py", line 18, in application
    time.sleep(seconds)

Who is allowed to do this is then up to RBAC, like any other kubectl exec.

What's next

Very little in this post needed Muru to do anything Kubernetes-specific. The shutdown delay is one setting, the workers are set to go with the pod's resources, the autoscaler reads a metric Muru already publishes, and the migration is a Job running a command the project already declares.

The autoscaling here works from outside, with Kubernetes adding whole pods once the workers are busy. The next post, scaling from what the workers report, goes back to the daemon, where Muru decides itself how many workers a release needs, as a scaling policy also has it do within a pod, and where, because it knows why requests are waiting, it can tell whether more workers will help or whether something in the release needs changing.

November 15, 2026 12:00 AM UTC


Bob Belderbos

Give Your AI Reviewer the Spec, Not Just the Diff

I review everything Claude Code writes, but at the current speed increase, it's easier to miss things. So after my review, a second, independent reviewer checks the work, and it gets more than the diff.

Here's how I've set it up.

Claude Opus generates, I review, Codex does a second review

The idea here is to use a different model/tool (in my case Codex) to keep bias out. But instead of a pure diff, which is already helpful, the reviewer also needs to know what the aim was.

For serious work (one-off scripts can get away with less), I start each feature with a grillme session to come up with a concise and well-scoped spec. I updated my variant of this skill to write the resulting spec to a local specs folder (goal, decisions (including rejected options), in/out of scope, acceptance). This way the reviewer has much richer context about the diff it's reviewing.

This is the spec template my grillme skill writes to specs/<slug>.md, kept under ~40 lines:

# <Feature>

## Goal
## Decisions
## Scope
## Acceptance
## Out of scope

If you're new to skills, check out how I use Claude Code skills for structured AI collaboration.

This was also a succinct comment from somebody on X when I posted about this last week:

"a reviewer with zero context from the writing session, otherwise it's grading its own homework. Do you give the reviewer the spec, or only the diff?"

So the answer is both!

A tight CLAUDE.md (touch only what the task needs, grep callers before changing a signature, ruff → ty → pytest) stops the simple mistakes before review.

So my flow for more serious work: grillme → spec → Opus implements → I review (!), then Codex for a second, independent and well-scoped review.

Recent example: Codex raised one P2. Each input passed validation on its own, but their product overflowed the DB column. Unlikely in practice for that app, but cross-field constraints like this are easy to miss (I've seen it in 1:1 coaching more than once).

The diff tells the reviewer what changed. The spec tells it whether that was the point.

November 15, 2026 12:00 AM UTC

November 14, 2026


Graham Dumpleton

Muru: a better tenant in a container

The last two posts, on releases and canaries, were all about the Muru daemon running more than one release of an application side by side and moving traffic between them. When the application runs in a container, Muru deliberately does none of that. A container image is already a fixed release of an application, and starting new containers, moving traffic to them and putting the old ones back is the job of whatever runs the containers. Doing the same thing again inside the container would only get in its way.

So in a container Muru runs as muructl serve, which serves the single application held in the image. The edge, the controller, the host and its workers are all still there, as in development mode, but releases and traffic policies aren't. Instead, it concentrates on being a good main process for a container, which mostly means handling things people otherwise end up doing by hand when using gunicorn or uvicorn.

An image

The image for this post holds a small WSGI application with a few pages which misbehave in different ways, which I will get to later. The Dockerfile follows the pattern from the Muru documentation:

FROM python:3.13-slim AS build
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
RUN UV_TOOL_BIN_DIR=/usr/local/bin uv tool install muructl
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN muructl build --deployment --deps-only --no-runtime --target /app/.venv
COPY . .
RUN muructl build --deployment --target /app/.venv --runtime-dir /opt/muru

FROM python:3.13-slim
COPY --from=build /usr/local/bin/muructl /usr/local/bin/muructl
COPY --from=build /app /app
COPY --from=build /opt/muru /opt/muru
WORKDIR /app
RUN useradd --uid 10001 app && install -d -o 10001 -g 0 -m 0770 /run/muru
USER 10001
EXPOSE 8080 9090
CMD ["muructl", "serve"]

muructl is a single executable, so the final image copies just the binary. The two muructl build steps install the application's environment from uv.lock. The first installs only the dependencies, so that its layer can be reused until they change. Each says what it is running:

muru: building /app/.venv with uv sync --frozen --no-dev --no-build --compile-bytecode --no-install-project
...
muru: building /app/.venv with uv sync --frozen --no-dev --no-build --compile-bytecode
muru: built /app/.venv on Python 3.13.18 (cp313)
muru: fetched the runtime library muru-1.0.0-cp313-cp313-manylinux_2_28_aarch64.whl from PyPI into /opt/muru

The last line is the part which is specific to Muru. The runtime library which runs inside the workers has to match both the version of muructl and the interpreter, so it is fetched when the image is built. It is kept outside the application's environment, so the application doesn't need to list Muru as a dependency.

The important thing is that this is the only point at which anything is fetched or installed. When the container starts, muructl serve never builds an environment, downloads a Python or fetches the runtime library, and anything it runs gets UV_OFFLINE=1 and UV_PYTHON_DOWNLOADS=never in its environment. If something is missing, it stops with a message naming the build step which would have provided it. For example, pointing it at the wrong runtime directory gives:

muru: error: no runtime library for /app/.venv/bin/python (cp313): build the image with muructl build --runtime-dir /nowhere, or add muru 1.0.0 to the environment

That matters more than it might seem. It isn't unusual to find images where an entrypoint script pip installs something, or where a missing package is quietly downloaded at start. That works until the network is unavailable or the package index changes, at which point two containers started from the same image are no longer the same.

Telling it how many workers

Starting the image with limits of two CPUs and a gigabyte of memory, with no swap, on a Docker host with ten CPUs, I gave it the number of worker processes and the threads in each:

docker run --name shop --cpus 2 --memory 1g --memory-swap 1g \
  -e MURU_SERVE_PROCESSES=2 -e MURU_SERVE_THREADS=4 -p 8080:8080 -p 9090:9090 shop

The output says what it found and what it is running:

muru: container limits: 2 CPUs from cpu.max 200000 100000, memory.max 1.0GB with 128MB of headroom
...
muru.host[12]: preloaded app:application in 1 ms
muru.host[12]: app:application is served as WSGI
muru.host[12]: worker 0 started as pid 15
muru.host[12]: worker 1 started as pid 16
...
muru: serving app:application on http://0.0.0.0:8080 with 2 workers of 4 threads
...
muru: the host holds 34MB once the application is imported

These settings should always be supplied, chosen to suit the resources given to the container. If they are left out, Muru runs a single worker. Two processes for two CPUs suits an application whose requests mostly use the CPU, while one whose requests spend most of their time waiting on a database can use more threads in each. The memory has to fit as well. Two workers, along with the host which imported the application, leave plenty of room in a gigabyte, but an application whose workers each need 750MB would have to run a single process in a container this size, or be given a bigger container. The other examples in this post start the container with the same settings, which I leave out to keep the commands short.

Muru doesn't try to choose these numbers itself, because they can't be reliably worked out from inside the container. Asking Python how many CPUs it has, in the same container, gives:

docker run --rm --cpus 2 shop python3 -c "import os; print(os.cpu_count(), os.process_cpu_count())"
10 10

A CPU limit set with --cpus, or resources.limits.cpu in Kubernetes, is a CFS quota in the cgroup's cpu.max file, which neither of those functions reads. Gunicorn's default is a single worker unless WEB_CONCURRENCY says otherwise, and uvicorn's is the same. Gunicorn's documentation suggests (2 x CPU cores) + 1, and the image the Gunicorn project publishes uses that formula by default, working out the count with nproc, which also says 10 here, so it would start 21 workers in a container allowed two CPUs. On a large Kubernetes node, with a pod limited to a couple of CPUs, it is far worse. Even a count which got the CPUs right would say nothing about how much memory each worker needs, or whether the requests use much CPU at all.

Muru does read the limits from the cgroup, as the first line of its output showed, but only to check the settings against them. Asking for four workers in the same container still starts four, but with a warning:

muru: warning: 4 workers is more than the 2 CPUs cpu.max allows; see https://docs.muruhq.dev/en/latest/howto/containers.html#container-limits

It also warns if the workers wouldn't fit in memory once each holds its own copy of what the host imported. A forked Python process tends to end up with such a copy over time, as it gradually stops sharing memory with the process it was forked from.

Running short of memory

A Python process tends to hang on to memory it has used, whether through a cache, a leak or one unusually large request. In a container, the usual outcome of this is that the container runs out of memory and the kernel kills a process outright.

When the container has a memory limit, Muru gets in first. Every few seconds it checks how much memory is left under the limit. If that drops below a margin it keeps in reserve, which it calls the headroom, it takes the largest worker out of service and replaces it with a fresh one. The old worker finishes the requests it already has before it goes, so nobody sees an error, and its memory is given back. Nothing needs to be set for this beyond the container's memory limit.

To see it in action, I used a page of the application which holds on to another 10MB each time it is requested, and asked for it once a second:

muru.host[12]: worker 0 (pid 15) the container was short of memory, 102MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 424MB; replacing it
muru.host[12]: worker 0 (pid 15) stops before its replacement starts: 101MB is left under the memory limit, less than its size
muru.host[12]: worker 0 started as pid 38
muru.host[12]: worker.recycled: worker 0 (pid 15) replaced by pid 38: the container was short of memory, 102MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 424MB; most objects: cell 31, tuple 20, builtin_function_or_method 16, function 11, list 11
muru.host[12]: worker 0 (pid 38) is ready
...
muru.host[12]: worker 1 (pid 16) the container was short of memory, 100MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 645MB; replacing it
muru.host[12]: worker 1 (pid 16) stops before its replacement starts: 100MB is left under the memory limit, less than its size
muru.host[12]: worker 1 started as pid 49
muru.host[12]: worker.recycled: worker 1 (pid 16) replaced by pid 49: the container was short of memory, 100MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 645MB; most objects: cell 31, tuple 20, builtin_function_or_method 16, function 11, list 11
muru.host[12]: worker 1 (pid 49) is ready

Both workers had been growing at the same rate, so when the container first ran short, worker 0 was only just the larger of the two, at 424MB. Replacing it gave that memory back, which left room for worker 1 to keep growing. By the time the container ran short again, worker 1 was at 645MB, and it was replaced in turn.

Normally Muru starts the new worker before the old one leaves, so there is never a gap. Here less memory was left in the container than the old worker was using, so rather than risk two large workers at once, the old worker stopped first, finishing the requests it already had, and its replacement started once it had gone. The other worker carried on serving in the meantime, and all 150 requests succeeded.

Without this, the first anyone knows of a worker growing is when the kernel's out-of-memory killer picks a process in the container to kill, and that can just as easily be one in the middle of a perfectly good request. Neither gunicorn nor uvicorn keeps any watch on memory. The best either offers is restarting a worker after a set number of requests, however much memory it is using.

Muru calls this its memory guard. Running muructl status inside the container shows the limits, how much memory is used and left, and what the guard last did:

docker exec shop muructl status
...
cpu      2 CPUs from cpu.max
memory   memory.max 1.0GB, 402MB used, 622MB left, headroom 128MB
         workers 329MB, host 34MB
guard    watching; it last replaced shop@r1 worker 1 (pid 16) at 07:31:18 UTC

The guard looks at the container as a whole. If a single worker should never grow beyond a certain size, however much room is left, it can also be given a hard memory limit, such as MURU_SERVE_MEMORY=hard_limit=300MB. A worker which goes over that limit is replaced in the same way. The value is chosen to suit the application, and should be low enough that all the workers together still fit in the container.

Memory is only checked every five seconds, so if it grows by more than the headroom between checks, the out-of-memory killer can still strike first. A request which allocated a gigabyte all at once showed what happens then:

502 Bad Gateway
reason: worker_crashed
request id: 01M6TF11TNSGS7R9AW4G704V4X

The worker sent no response head.
muru.host[12]: worker.oom_killed: worker 0 (pid 15) was killed by the kernel for memory, restarting
muru.host[12]: worker 0 started as pid 47
muru.host[12]: worker 0 (pid 47) is ready

Each worker raises its own oom_score_adj as it starts, which makes the workers the processes the kernel picks first. So it was the worker which was killed, rather than the host or muructl, and the host started a new one. The kill is also reported for what it was, as a worker.oom_killed event and in the muru_worker_exits_total metric with the reason oom_killed, rather than leaving a worker which mysteriously went away.

Probes on their own port

The output when it started also included this line:

muru: admin port on http://0.0.0.0:9090/

Port 9090 is separate from the application's port and answers the questions a container platform asks. /livez says whether the process is alive, /readyz whether enough workers are ready to take requests, and /startupz whether the application has finished loading. /metrics has the metrics in Prometheus format.

Gunicorn and uvicorn have nothing like this. The usual answer is to add a /health view to the application, which means the health check is served by the same workers as everything else. When the application is slow, perhaps because its database is, every worker is busy. The health check then queues behind real requests and times out, and the platform restarts containers which were doing nothing wrong, just when they are needed most.

To check that Muru doesn't suffer from this, I sent twelve requests to a page which takes six seconds to answer, more than the two workers with four threads each could handle at once, and then queried the admin port:

livez 200 0.001336s
readyz 200 0.001337s

The metrics confirmed every slot was busy, with requests queuing:

muru_queue_depth{app="shop",release="r1",group="default"} 4
muru_slots_busy{app="shop",release="r1",group="default",side="threads"} 8
muru_slots_total{app="shop",release="r1",group="default",side="threads"} 8

The probes are answered by muructl itself, from what it knows about the host and its workers, without a request having to go through a worker. A container whose workers are all busy is still alive, so it isn't killed for being busy. Having a port of their own also keeps the probes and metrics out of reach of the application's Ingress or load balancer.

An image without curl, such as one based on the slim Python images, can use muructl itself for a health check:

HEALTHCHECK --interval=5s CMD ["muructl", "probe", "ready"]

Process 1

Looking at the processes inside the container (by reading /proc, since the slim image has no ps), each of Muru's processes has set a title saying what it is:

    1     0 S  muructl serve
    7     1 S  muru serve shop
   11     7 S  muru relay shop@r1
   12     7 S  muru host shop@r1
   15    12 S  muru worker shop@r1 default/0
   16    12 S  muru worker shop@r1 default/1

Process 1 is muructl acting as an init, in the way tini or docker run --init would. It forwards signals to the real muructl serve, which is its child, and collects any process in the container which exits, so nothing is left as a zombie. Process 1 in a container has to do this, because orphaned processes are handed to it, and if it never collects them, zombies fill up the process table. Below muructl serve are the host and its two workers, and a log relay which passes on what they write.

The application has a page which starts a report by running a shell command, which leaves a process running in the background. That happens in real applications more often than anyone would like. Just after I requested the page three times, muructl status --processes --all, where --all adds the processes Muru didn't start itself, showed them:

docker exec shop muructl status --processes --all
edge  3.0 req/s  0 err/s  p50 2.1 ms  p95 4.6 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 53B/s

PID   ROLE           WHAT                         UP    CPU      RSS   REQ/S   BUSY   QUEUE      P95   GIL/LAG
7     serve          controller and edge, :8080   5s   2.3%   19.5MB
12    └─ host        app:application (r1)         5s   0.0%   34.7MB     3.0    0/8       0   4.6 ms
11       ├─ relay    output to muructl            5s   0.0%   11.4MB
15       ├─ worker   worker 0                     5s   0.0%   28.7MB     2.0    0/4           4.8 ms
16       ├─ worker   worker 1                     5s   0.0%   28.9MB     1.0    0/4           2.4 ms
54       ├─ other    sleep 1                      1s           1.5MB
56       ├─ other    sleep 1                      1s           1.5MB
58       └─ other    sleep 1                      1s           1.5MB

The orphaned sleep processes were taken over by the host, which collects any process its workers leave behind, and a couple of seconds later they were gone, with no zombies. Gunicorn does collect any child that exits, including ones it didn't start. Uvicorn doesn't, so when run as process 1 it leaves a zombie behind for each one. Neither project's documentation says anything about tini or --init. An image which uses its own init can turn off Muru's by setting MURU_NO_INIT=1.

Stopping in order

When docker stop or Kubernetes wants a container gone, it sends SIGTERM, waits, and then sends SIGKILL. Docker waits ten seconds and Kubernetes thirty by default. For a server, what happens in between decides whether anyone sees an error.

I started four requests to the page which takes three seconds, and half a second later ran docker stop, while checking /readyz every 300 milliseconds:

readyz 200
readyz 503
readyz 503
readyz 503
readyz 503

The four requests all completed:

 200 3.004430s
 200 3.004411s
 200 3.004890s
 200 3.004973s

and docker stop returned after 2.76 seconds, with the container's exit status 0. On SIGTERM, readiness starts failing at once, so nothing new is sent to the container. The listener then closes, the requests in flight finish, and the workers stop. All of that has to happen within a budget, eight seconds unless --shutdown-budget changes it. That is less than the ten seconds docker stop allows, so anything still running at the end is stopped by Muru rather than killed by Docker.

Compare that with the defaults elsewhere. Gunicorn's graceful_timeout is 30 seconds, so under docker stop a slow request is cut off by SIGKILL at ten seconds and gunicorn never gets to finish shutting down. Uvicorn has no graceful shutdown timeout by default, so it waits as long as it takes, which under Docker means until the SIGKILL arrives. With either, the fix is to remember to set a timeout which fits within the platform's.

Kubernetes adds a wrinkle, as a pod can keep receiving new connections for a few seconds after SIGTERM, while the endpoints catch up. That is what --shutdown-delay is for, and I will come back to it in the next post.

A read-only root and any user

A security conscious platform will run the container with a read-only root filesystem, and OpenShift will run it as an arbitrary user ID in group 0, with no entry for it in /etc/passwd. Everything muructl serve writes at run time goes under one directory, /run/muru. With a read-only root and nothing mounted there, it stops at once and says what it needs:

docker run --rm --read-only --user 123456:0 shop
muru: error: the run directory /run/muru cannot be written (Read-only file system (os error 30)); with a read-only root filesystem give it a writable volume, a tmpfs or an emptyDir mounted at /run/muru, or name another with --run-dir

Giving it what it asks for:

docker run --read-only --tmpfs /run/muru --tmpfs /tmp --user 123456:0 ... -p 8080:8080 -p 9090:9090 shop

it runs as normal under a user ID which doesn't exist in the image:

hello from pid 15

Gunicorn in the same situation fails at start, with No usable temporary directory found, as it writes a heartbeat file for each worker to the temporary directory. That can be fixed with a tmpfs on /tmp, or by pointing --worker-tmp-dir at /dev/shm, but you have to know to do it.

Settings in layers

Every setting of muructl serve can be given as a command line flag. An image can also set defaults in the project's pyproject.toml, under [tool.muru.serve], and a deployment can override any of them with an environment variable named after the flag. --print-config shows the settings in force and where each came from, and works inside a running container too:

docker run --name shop -e MURU_SERVE_PROCESSES=3 -e MURU_SERVE_LOG_FORMAT=json ... shop
docker exec shop muructl serve --print-config
processes = 3                     # MURU_SERVE_PROCESSES
...
log_format = "json"              # MURU_SERVE_LOG_FORMAT

--config, or MURU_SERVE_CONFIG, names a settings file to read instead of the one in the image, such as a file mounted from a ConfigMap.

Logs a collector can use

With the JSON log format, every line is a JSON object. When the environment has POD_NAME, POD_NAMESPACE and NODE_NAME, which the Kubernetes Downward API can provide, each line also says where it came from. I set those by hand for this run. More useful is what happens to a traceback. The application has a page which raises an exception, and the error shows up as a single record, with the whole traceback in its exception field, alongside the request, release and worker it happened in:

{"time":"2026-11-07T07:37:52.296Z","kind":"app","source":"muru.worker[0]","app":"shop","exception":"Traceback (most recent call last):\n  File \"/app/app.py\", line 25, in application\n    raise RuntimeError(\"something went wrong\")\nRuntimeError: something went wrong","group":"default","level":"ERROR","logger":"muru","message":"the application raised an exception","release":"r1","request_id":"01M6TF2DXFEHAEQZN1C1JEXGBE","worker":0,"pod":"shop-7d4b9c-x2k8p","namespace":"shop","node":"worker-2"}

Anyone who has had to piece a Python traceback back together from a log system which treated each line as a separate entry will appreciate that.

Inside the container

Finally, the commands from looking inside the workers work inside the container through docker exec, talking to muructl serve over its admin socket. With a slow request under way:

docker exec shop muructl debug stacks
muru-app-0 in worker 0:
  serving GET /slow?seconds=4 for 1.2 s (request 01M6TF2SRXPXTHSAPPDG6MVNJQ)
  File "/opt/muru/muru/_pool.py", line 30, in serve_jobs
    serve()
  File "/app/app.py", line 15, in application
    time.sleep(seconds)

What's next

None of this needs anything outside the image. There is no init to add, no health check view to write, no timeouts to line up with the platform's and no temporary directory to remember. What it does need to be told is how many workers of how many threads to run, chosen to fit the resources the container is given.

Most containers end up on Kubernetes though, and there are a few more things there where Muru can work with what the platform provides rather than around it, such as the shutdown delay in place of the usual preStop hook, autoscaling on how busy the workers really are rather than on CPU, and running migrations as a Job from the new image. That is the subject of the next post, working with Kubernetes, not around it.

November 14, 2026 12:00 AM UTC

November 13, 2026


James Bennett

Boring Python: dependency management

This post has been updated since it was originally written. The most recent update was in November 2026; a summary of what changed is at the end.

This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible.

So before I go any further, I want to be absolutely clear on what I mean by “boring”: I don’t mean “reliable” or “bug-free” or “no incidents”. While there is some overlap, and some of the things I’ll be recommending can help to reduce bugs, I also want to be clear: there will be bugs. There will be incidents where a feature or maybe an entire service is down. “Boring”, to me, is about the sources of those incidents. It’s difficult enough to manage your own code and the bugs and other problems that will inevitably pop up in it from time to time; you don’t want to compound that by having bugs or other surprises coming from the tools and processes you use to build, manage, and deploy it. So when I call something “boring”, I mean it’s unlikely to add another source of bugs and nasty surprises that cause a pager to go off at 2AM; the pager will, of course, eventually go off at 2AM, but when it does you’ll be able to feel reasonably confident (if that’s the right word for such a situation) that the source of it was something in your own code that you can diagnose and fix.

And so I’m planning several posts exploring different aspects of making Python development boring (in this sense). But for this first installment, I’ll be talking about one of the internet’s favorite topics: managing dependencies.

If you’re just interested in the end result, the process I recommend is pretty simple and can be found in the section titled “Putting it all together”. The rest of this post exists to explain, in as much detail as I can manage, all the things going on behind those simple-looking recommendations, and why I make those specific recommendations.

Python packaging: a brief overview

I’ve written about this before, but to make sure it’s clear I want to do a quick refresher.

As I see it, there are three basic aspects to “packaging”, and one of the difficult things about “Python packaging” discussions is lack of clarity around which are being discussed. They are:

  1. Given some working code, define and produce from it a distributable artifact.
  2. Given that someone has produced a distributable artifact, use it to cause the corresponding code to appear, in working form, somewhere else.
  3. Given the existence of potentially many separate projects or versions of projects each with their own independent and potentially-conflicting sets of dependencies, make it possible to work on and run more than one at a time, on the same system.

Most of the actual remaining complexity in “Python packaging” today is concentrated in item (1), and even then only really kicks in if you’re packaging extensions written in non-Python languages.

A lot of the perceived complexity of “Python packaging” is due to its long history, evolving expectations over the course of that history, and the resulting pile of strata of different things people have come up with to try to suit the expectations of different eras.

Much of the public grumbling and complaining about “Python packaging” these days is due to the lack of a single, standard first-party, all-in-one high-level tool to handle all aspects of packaging, and the proliferation of third-party tools to try to fill that niche.

Finally, most of the meaningful work in Python packaging, for quite a while now, has been the unglamorous, underappreciated slog of people on mailing lists and discussion forums patiently working out what Python packaging really should look like and thoroughly specifying/standardizing it, but with the added thrill of all that having to occur in a way that doesn’t break things for the vast number of people relying on Python and its huge package ecosystem for their day-to-day work.

To the extent that we now have a bunch of shiny fancy flashy third-party all-in-one packaging tools, it’s because of that heroic unsung work that’s been quietly going on for years. The fancy new tools get to be so fancy and flashy in large part because modern Python packaging has such strong foundations (and also because, unlike the pre-existing tools, the new ones don’t have to maintain compatibility with literal decades of historical features and configuration options, all of which have been relied on, at some point, by someone).

If I went into detail on everything I’d wind up with several posts’ worth of material, so let me just pick out a few things that I think are the most crucial underpinnings of modern Python packaging:

Boring Python packaging: a philosophy

When I wrote the first version of this post in 2022, I focused on defining a workflow that relied as much as possible on standard first-party (as in, included with Python itself) tooling. I recommended exactly one third-party tool, and only as an optional convenience; the functionality it provided could also be obtained from chaining together invocations of first-party tooling.

That focus on first-party tooling was deliberate. As I wrote at the time:

[The] core default tools I’ve mentioned have all been around and stable for a long time, in software terms, at least — they’re all over a decade old, and their end-user interfaces evolve incredibly slowly, when they evolve at all. They’re reliable. They’re well-understood. That’s exactly what I want and exactly what I recommend.

This is in line with what I said at the start of this post about wanting things to be boring. And my stance hasn’t changed, but advances in the foundational standards of Python packaging do let me be a bit more flexible. So although I’m still going to be focused on first-party tools I’ll now point out a couple of places where I think it’s safe to use some of the hip and trendy new third-party tools if you want the added conveniences they offer.

Also, I should note here that my use case for Python is deploying networked (primarily web) services, on servers. If you primarily use Python for, say, machine learning or data science, you are likely to already be a happy user of a completely different world of tools optimized for those use cases (in particular, for the much greater amount of non-Python code wrapped by/accessed through Python in those fields and for their different patterns of code reuse and sharing). Please continue using them! They suit your use case really well, and replacing them with my preferred tools and workflow would probably be a regression for you.

Now, let’s get started.

Lock it up

The standard Python package manager is pip. And my eventual goal is to invoke pip install to, well, do what it says: install a bunch of Python packages. But how to tell pip which packages to install? There are a couple of options:

Historically, the second option was referred to as a “requirements file”, and by convention was generally named requirements.txt. Now that PEP 751 has defined a standard Python lock file format, and now that pip supports producing and consuming that standard lock file format, that’s the format I recommend.

This prompts a question, though: since dependencies could already be specified as part of the metadata in pyproject.toml, why did we need another file format to record them? By way of an answer, consider a web application written using the Django web framework. You might think it’s simple enough to just declare Django as a dependency in your pyproject.toml file, like so:

[project]
dependencies = ["Django"]

But if I create a new virtual environment and run pip install Django inside it, three packages get installed:

$ pip install Django
Collecting Django
  Using cached django-6.1-py3-none-any.whl.metadata (3.9 kB)
Collecting asgiref>=3.9.1 (from Django)
  Using cached asgiref-3.12.1-py3-none-any.whl.metadata (9.4 kB)
Collecting sqlparse>=0.5.0 (from Django)
  Using cached sqlparse-0.6.0-py3-none-any.whl.metadata (6.0 kB)
Using cached django-6.1-py3-none-any.whl (8.4 MB)
Using cached asgiref-3.12.1-py3-none-any.whl (25 kB)
Using cached sqlparse-0.6.0-py3-none-any.whl (50 kB)
Installing collected packages: sqlparse, asgiref, Django
Successfully installed Django-6.1 asgiref-3.12.1 sqlparse-0.6.0

And which versions of those packages I get will depend on when I run pip install. Right now I get Django 6.1, but Django does frequent bug fix releases, so if I re-run that pip install in the future I’ll get Django 6.1.1, and further in the future I’ll get Django 6.1.2, and so on. If I test my application locally with the set of packages I installed today, and then deploy it to a server later using pip install to fetch dependencies, the deployment will very likely get a different set of packages than I used during development, which is a potential source of problems.

So the key difference is that the pyproject.toml file typically specifies a list of direct dependencies, often with broad version constraints or none at all, while a lock file specifies an entire environment, with all direct and transitive dependencies resolved and pinned to exact versions along with the expected checksums of the packages, to be reproduced as exactly as possible.

There are two main approaches to generating a lock file:

Managing your local environment and producing your lock file is one of the areas where I think it’s generally OK to use a third-party tool. If something goes wrong with it, it might temporarily mess with your ability to get work done on your own computer, but that won’t break your actual deployed application.

The hip and trendy third-party all-in-one tool these days is uv, which has its own tool-specific lock file format (uv.lock), but also has a uv export command which can export to the standard pylock.toml format.

My own personal preference, and the tool I use on my own projects, is PDM, which has a similar feature set to uv and can export to pylock.toml from its own lock format, but also can just manage a project directly in a pylock.toml file.

As long as the tool you choose is one you and your team are all happy with, I think you can pick anything that can produce the standard lock file format and be OK. Also, you should probably automate the process of exporting the lock file, to ensure developers don’t have to remember how to manually do it. I personally like to include a Makefile in most projects, with a target specifically to refresh the lock file (make lock, for example), but the idea is portable to a lot of task-automation tools, so it’s easy to make it work with whatever your team likes to use.

I also want to point out here that if you’re deploying services written in Python, you can build them as distributable packages and then install those packages to automatically pull their dependencies, rather than using a lock file. I just don’t recommend this, for a few reasons:

Getting testy

In addition to direct dependencies to run an application, it’s fairly common to have additional dependencies needed to run a test suite, linters, or other development and quality-control tasks. But there’s no need for a production deployment to have all those extra dependencies installed, and in general the less stuff you put in your production environment the fewer things you have that can go wrong, so I generally like to install those only when the test suite (or other task requiring extra dependencies) is actually going to be run.

The standard solution for this is dependency groups, which were first defined for library-style package dependencies but are also part of the lock file specification. And as of pip version 26.3, selecting groups when installing from a lock file is now supported. Popular third-party package-manager frontends also seem to have broad support now, in late 2026, for specifying dependency groups and exporting them to a standard lock file, so you can manage multiple logical groups of dependencies with ease.

The general consensus among the third-party tools seems to be to automatically support a group named dev for dependencies used only in development/testing, so my recommendation is to go with that convention rather than manually creating your own group with a different name.

Here’s an example with PDM of how you could set up a project to use Django as the web framework in production, and the pytest-django library for testing:

pdm add Django
pdm add --dev pytest-django

And then you can export to a standard lock file like so:

pdm export --format pylock -o pylock.toml

The lock file format is pretty readable, so you can look inside and see that it specifies multiple groups (one named default and one named dev) and tracks which dependencies belong to which group(s). The exact commands for adding packages to and exporting a “dev” group with other tools will vary, but should generally look pretty similar to what’s above.

Using the right invocations

Before putting all this together, it’s worth covering one more detail: how to invoke the tools. This may seem a bit silly, since they all provide executable entry points. Just run pip install, right?

But there’s a potential issue here: in a moment I’m going to recommend creating a Python virtual environment, which opens up the possibility of multiple Python environments coexisting on the same machine. This is certainly a useful feature, but it comes with a new concern, which is how to ensure you’re using and running things in the environment you expect to be using.

For example, one way to run into trouble with multiple Python environments is to have one environment’s package directory be first on your $PYTHONPATH (which controls Python import locations) while a different one’s bin/ directory is first on your general $PATH (which is where executable scripts will be found). If you just run pip install you’ll get the second one’s instance of pip, which may not be at all what you want.

This is why the official Python packaging guides, and official documentation for tools like pip, always use a different approach: they tell you to run python -m pip instead of pip, and python -m venv instead of a standalone script like virtualenv. The -m flag allows a Python module to be run directly like a script (as long as it’s been written to provide an entry point for this, which pip and venv both have), and can prevent a lot of potential hard-to-debug issues that can accidentally result from things like manually hacking around with paths, by ensuring you’re getting the version of pip or venv that actually goes with the Python environment you invoked.

And to guarantee that you get the Python environment you actually want, you can specify the full path. For example, many base Python container images will put the Python interpreter in /usr/local/bin, so invoking /usr/local/bin/python instead of just python ensures you get that interpreter.

There’s also one more thing I like to do that isn’t (currently) in some of the popular guides, and that’s invoking Python with the -I flag. This runs Python in “isolated” mode, which removes some of the automatic implicit directories from the import path and also ignores environment variables like PYTHONPATH. Once again, this reduces the number of things that can go wrong (for example, with -I the current working directory won’t be implicitly added to the import path, so you can’t accidentally depend on it being importable). So whenever you can invoke Python in isolated mode, I generally recommend that you do.

Putting it all together

That was a lot of explanation for what ends up being, ultimately, a pretty simple process to actually use. So now let’s finally take a look at it.

First things first: always work in, and always deploy in, a virtual environment. Even if you think you don’t need one. In fact, especially if you don’t think you need one. This may seem like strange advice if you’re already using a container or other virtual machine, since you’re probably thinking that provides all the isolation you’ll need. But virtual environments don’t cost you anything to create, and if you ever do end up with multiple Python interpreters—which is easy to accidentally do, if you use a base system with a purpose-built Python and then install a system package that turns out to depend on the distro’s own Python, for example—using one from the start will help to save you from potentially having a pager go off one night when suddenly the wrong Python is being invoked.

Virtual environments also provide a useful “Python environment” artifact that can be copied between stages of a container build, and many tools automatically recognize and can work with them. And ever since PEP 668 started to be adopted by operating-system vendors, many “system” Python installations will require you to use a virtual environment in order to install packages with pip (and only allow you to interact with the “system” Python environment through the system’s own package manager). So use a virtual environment, even if you’re working in a container or other VM.

When you’re working in a local directory on your own computer, you can invoke the correct Python version with -m venv to create a virtual environment and install things into it, but again this is an area where third-party package tools can be useful, because they’ll manage this for you automatically. If you do create your local virtual environment manually, you should almost certainly put it inside your project in a subdirectory named .venv, because that’s already the common unofficial convention supported by a lot of IDEs and other tools, and likely to become the official standard convention once PEP 832 finalizes.

In a container, I think the choice that’s most in line with Linux filesystem hierarchy standards is to put a Python virtual environment somewhere under /opt. Generally I like to create an /opt/venvs in a base “build” stage, which can populate multiple virtual environments with different package sets to be copied into later stages for tasks like CI or deployment.

So, taking as an example current Debian stable (as I write this, Debian 13 “trixie”) and the most recent Python (as I write this, Python 3.15), let’s see what this actually looks like. This example assumes you’re producing a standard Python lock file as described above:

# syntax=docker/dockerfile:1

ARG DISTRO="slim-trixie"
ARG PYTHON_VERSION="3.15"

FROM python:${PYTHON_VERSION}-${DISTRO} AS dependencies

# Turn off pip upgrade reminders, since we're about to upgrade it
# anyway, and also specify the root cache directory pip under which
# will create its package cache.
ENV PIP_DISABLE_VERSION_CHECK=1 \
    XDG_CACHE_HOME=/var/cache

RUN <<END
mkdir -p /opt/venvs/deploy
mkdir -p /opt/venvs/tests
# Ensure the pip cache directory exists.
mkdir -p ${XDG_CACHE_HOME}/pip
/usr/local/bin/python -Im venv /opt/venvs/deploy
/usr/local/bin/python -Im venv /opt/venvs/tests
END

COPY pylock.toml  /opt/venvs

The next step is to ensure pip is present at the latest version in each virtual environment, and invoke it to install packages.

At this point you might be wondering: if I was willing to recommend using a third-party packaging frontend earlier for local development use and producing lock files, why am I insisting on pip here? And the answer is this is one of the places where I think using boring standard default tools really matters. As I said above, if something goes wrong with your fancy third-party package tool in a local development setup, it just causes a problem for that local development setup. If it goes wrong in your production build/deploy process, it breaks all your builds and deployments until you resolve the issue. I want to avoid that, so I stick to pip here, because even if it’s not as fancy as some of the newer third-party tools, in my experience it’s much less likely to be a source of unexpected issues than the newer third-party alternatives.

Also, I’m not going to go into a full explanation of an ideal Python application Dockerfile here, but the example below does at least use a Docker cache mount to store the downloaded packages, so that subsequent rebuilds only have to download new or changed packages, and can pull everything else from cache.

The pip invocation adds a few flags, mostly to ensure maximum safety and reproducibility:

If you’re using GitHub Actions as your CI/CD, Python core developer Brett Cannon has written a reusable action which automatically invokes pip with these flags.

But continuing with the example of a Dockerfile containing a “dependencies” stage, here’s what it looks like. Note the difference between the two pip install invocations, though. Asking pip to install from a lock file with -r pylock.toml will only install the default dependency group, and for the deployment dependencies, that’s the right behavior, but for the test dependencies it isn’t, so the test dependencies use the --group flag to explicitly select the set of groups to install (both default and dev, in this case).

# Install test dependencies first since they're likely to be a
# superset of the deployment dependencies and will populate the
# package cache.
RUN --mount=type=cache,sharing=locked,target=${XDG_CACHE_HOME}/pip,id=pip <<END
/opt/venvs/tests/bin/python -Im pip install --upgrade pip
/opt/venvs/tests/bin/python -Im pip install \
    --no-deps \
    --only-binary :all: \
    --require-hashes \
    --group /opt/venvs/pylock.toml:default \
    --group /opt/venvs/pylock.toml:dev
END

RUN --mount=type=cache,sharing=locked,target=${XDG_CACHE_HOME}/pip,id=pip <<END
/opt/venvs/deploy/bin/python -Im pip install --upgrade pip
/opt/venvs/deploy/bin/python -Im pip install \
    --no-deps \
    --only-binary :all: \
    --require-hashes \
    -r /opt/venvs/pylock.toml
END

Now you can pull the installed packages into later stages of your build by copying the virtual environment. For example, in a deployment stage:

FROM python:${PYTHON_VERSION}-${DISTRO} AS deploy

COPY --from=dependencies /opt/venvs/deploy /opt/venvs/deploy

# Do the rest of your deployment stage setup here: copying in
# your application source code, setting the entry point, etc.

Once again, I’m not going to go into a ton of detail here on other things that make a good Python Dockerfile. If you want to learn more about that, I’d recommend reading Itamar Turner-Trauring’s articles on Python and Docker or Hynek Schlawack’s guide to Python and Docker. Both are regularly updated, and they’ll both give you a solid education in how to containerize Python.

Be up-to-date, but stay cool

The only thing still missing here is how to handle updates as new versions of your dependencies are released. For security updates this is crucial, but it’s also important as a general practice. If you make dependency updates a regular, routine part of your development process that’s easy for developers to do, then it’ll also be routine and easy to apply critical updates when they appear. And by applying updates as they come, a couple dependencies at a time, you avoid building up a huge backlog of deferred updates that will make a critical issue even more difficult to address when one inevitably occurs.

And luckily, all of the popular third-party package management tools have straightforward commands you can run to identify and update outdated packages. But there is one wrinkle: you probably don’t want to always eagerly accept most routine dependency updates as soon as they’re released, since that can expose you to potential security issues if a dependency has been compromised but nobody’s noticed it yet. The solution to this is dependency cooldowns: when upgrading, only consider packages which have had at least a bit of time for security researchers and scanners to take a look. The typical recommended window is three days; anything newer than that should be excluded from your package updates unless it’s a critical security patch.

At this point, most popular tools in the Python ecosystem support dependency cooldowns. For example:

You also can configure third-party dependency update bots to apply updates for you as they appear. GitHub’s Dependabot automatically applies a three-day cooldown for everything except critical security updates, and so does the Renovate update bot.

Whether to have a dependency update be a periodic manual task for a developer or something to automate via a tool like Dependabot is up to you (though I personally prefer automation). The important thing is you do it and commit to it, on a cadence that lets you take in updates only one or a few at a time, rather than falling behind and needing to update dozens of packages all in one go.

And that’s a wrap

As promised, that was a lot of words for what’s really a pretty simple set of recommendations. Unfortunately there’s a lot of complexity—necessary complexity, of a sort that pops up in any software packaging ecosystem—lurking in this topic, and explaining all that is what drives up the word count.

But hopefully you now understand how to do “boring” Python dependency management, relying primarily on standard first-party tooling. Even if you don’t want to adopt my recommendations, I’d like to think that learning what they are and why I make them is helpful to you. For me, these recommendations are the result of over a decade of work, across multiple employers, to develop a dependency management workflow that keeps things up-to-date with minimal risk of causing pagers to go off.

Meanwhile I’ve got some ideas for further “boring Python” articles, but those will have to wait for another day.

Changelog

The first version of this post was written and published in May 2022. A major revision was written during late August 2026 and published in early September 2026. The key changes between those versions were:

A second update was written and published in November 2026. This was a smaller change, in response to pip 26.3 adding support for selecting and installing specific dependency groups from a lock file. Previously, the lack of this support led me to recommend the use of multiple lock files to track separate deployment/test/etc. dependencies; now, with dependency groups supported in pip, I can recommend the simpler approach of a single lock file containing multiple dependency groups, and instruct pip to install the correct group(s) for a given use case.

November 13, 2026 08:23 PM UTC


Robin Wilson

I released Cloud-Native Environment Agency DEM data!

At the FOSS4G UK 2026 conference in October, I gave a talk announcing that I’d re-released the Environment Agency DEM data for England in cloud-native formats. This was covering some work I did for my client Rebalance Earth and that we’ve decided to release openly.

file

If you already know what I’m talking about then you can find the data on Source Cooperative as COGs with STAC indexes. The README on that page will explain how to access the data, and you can access an interactive webmap of the data too. If that didn’t make full sense to you, then read on…

I’m not going to go into full details here, as sometime soon the video recording of my talk at the conference should be released, and I can link to that here. In the meantime, my slides are available and may help with understanding some of what I explain below.

So, basically, the Environment Agency (EA) collect LiDAR data across England and process it to produce Digital Elevation Model (DEMs). DEMs are rasters where every pixel contains a height value, and the EA produce two types: a Digital Terrain Model which is just the height of the terrain, and a Digital Surface Model which includes the height of everything on top of the ground – like buildings, trees and so on.

The EA provide access to this data under the Open Government License, but their web interface for getting hold of the data is a bit of a pain – you can’t download much data at once. You can ask for bulk access to the data, but even then you have to work out exactly what OS 5x5km grid squares you want, and the whole thing is a pain.

As the data is released under the OGL, people are allowed to re-release processed versions of it under the same license. So, I converted the files to modern cloud-native formats, specifically Cloud Optimized GeoTIFF with a Spatio-Temporal Asset Catalog. These formats store the data efficiently, allowing small parts of the image to be easily downloaded, as well as ‘zoomed-out’ lower-resolution versions of the data.

The upshot of all of this is that you can use a few lines of Python or a GDAL command-line to download and mosaic the DEM over any area you choose, and you can also display the data easily on a web map. The latter is quite impressive – you can see an example webmap here, which does live rescaling in the browser, along with hillshading and colour blending, producing a far nicer output than the EA’s viewer – the example below is the DSM over Bristol (the location of next year’s FOSS4G international conference!)

I’ll leave the detailed documentation on how to use this to the README file in the data repository, but a few quick examples will show how easy it is to use.

You can use a command like this to download and mosaic all of the DSM data in a particular bounding box:

gdal raster clip \
  --bbox 438000,108000,446000,116000 \
  GTI:/vsicurl/https://data.source.coop/rebalance-earth/ea-dem/ea-lidar-fz-dsm-1m-items.parquet output.tif

You can also do similarly in Python using the lazycogs library:

import lazycogs

da = lazycogs.open(
            “items.parquet”,
            bbox=lonlat_to_bng_bbox(bbox_lonlat),
            crs="EPSG:27700", 
            resolution=1.0, 
            chunks=chunks)

da = da.squeeze(["band", "time"], drop=True)

which gives you a nice xarray DataArray to play with.

The talk went down very well, and was described by some people as the "highlight of the conference" (including Alastair Graham on the Geomob Podcast). The data has also been downloaded quite a lot since the conference – there have been over 1 million individual downloads, with over 4TB of data transferred. Given that each tile of DEM data is only around 80MB in size then that’s the equivalent of half a million full tiles (the whole of England is only around 5,500 tiles).

I’m quite chuffed with this – it’s something I’ve wanted to do for ages but never quite got round to, until we found that my client Rebalance Earth needed something like this, and decided we could release it openly. I want to thank Rebalance Earth for paying me to work on this (alongside lots of other stuff) and covering my costs of attending the conference to present it.

November 13, 2026 07:13 PM UTC


Mike C. Fletcher

Tim Peters will be missed

Tim spent a very long time in the early days of Python List teaching me all sorts of things I didn't know from my father's scattershot teaching as a child. He was generous with his time, full of humour and willing to teach some random kid on the internet when he had lots of other important things he could be doing.

Spending ages with him and Christian trying to optimise some random function to the Nth degree to work around Python's quirks is one of my fondest memories from that era of my life.

November 13, 2026 05:28 PM UTC

PyOpenGL 4.0.0b1 Released

So we seem to have reached mostly-stable state for the PyOpenGL 4.0.0 release. There's no other planned major changes to the core library, so unless we see significant regressions we'll likely release 4.0.0 final over the next few weeks.

Changes since the 4.0.0a5 release last mentioned here:

Bug/crash fixes:

* OSMesa -- a dropped OffscreenContext cleans up and prevents further writes
* Buffers -- a memoryview upload sent the wrapper instead of the data; this is fixed
* Buffers -- use-after-free in the buffers format handler is fixed
* Offsets -- an integer where a const void* is wanted is treated as a buffer offset (regression)
* Arrays -- bare numbers are accepted as input types again (regression)
* Arrays -- an input array is checked against the length count
* Dispatch -- a function derived from a C entry point keeps its own customisations
* Dispatch -- define() applies the annotation table to every command
* Dispatch -- context handles now compares equal to themselves
* Bindings -- glGetUniformIndices takes names and returns indices
* Bindings -- GL 3.0 pixel formats (GL_RG and similar) are sized without importing their extension module
* Bindings -- A wrapper that names a parameter its entry point lacks is handled
* OpenGL.GLES2 imports with error checking switched off (regression)
* Errors -- error messages are bounded size for all wrappers
* Errors -- NullFunctionError says which of three things went wrong
* Errors -- a platform plugin or format handler that will not import now reports which one and why
* Errors -- a config flag set too late says so rather than silently failing to activate
* Python 3.15 -- no longer reads ctypes.__version__ and 3.15 is added to CI/tox

VBO Helper

* References and aliases -- VBO attributes hold their own references
* VBO and VBOOffset are accepted by every array alias
* vbo.VBO is a class to type checkers
* Context sharing -- dispatch.share_context, gives contexts that share objects one identity

Platform Support

* Tk GLFrame for macOS (Aqua), working on Apple silicon (only tested in CI however)
* Tk Contexts take a swapInterval
* OSMesa now runs on Windows
* EGL -- the gbm render node is opened read-write
* Windows -- a GL library beside the interpreter is preferred to the system one (ANGLE support)
* Windows -- user-installed freeglut is found by its own name
* Windows -- opengl32 is loaded from Windows rather than from PATH
* Windows -- freeglut and GLE binaries left the main wheel. pip install PyOpenGL[glut] brings them back
* Namespaces -- a platform-specific API says which platform serves it

Packaging and CI

* Wheels -- arm64, aarch64 and Intel Mac wheels are built
* PyOpenGL_accelerate now has the same version as PyOpenGL, run-time checks it
* sdist no longer ships Cython output
* Test and CI setup -- the suite runs on OSMesa, and CI runs it there
* macOS test suite that never ran is not counted as a failing suite.

Tests, tooling and docs

* Tests -- a large batch of added tests. Many are tickets from the github tracker
* Lint -- ruff cleanups, typing cleanups
* Docs -- lots of updates, particularly around how we're generating them

November 13, 2026 04:11 PM UTC


PyCharm

What’s Fixed and Improved in PyCharm 2026.3

PyCharm 2026.3 is here! Alongside major features like JetBrains Air, workspaces, Python 3.15 lazy imports, and the redesigned interpreter widget, this release includes 303 fixes and improvements to everyday Python development.

From smarter code completion to more reliable environments and debugging, here are some of the changes worth knowing about—including fixes for issues reported more than a decade ago.

Smarter code insight and type inference

Better support for functools.partial

functools.partial lets you create a new callable by supplying some of a function’s arguments in advance. Previously, PyCharm could struggle to infer the resulting signature, leading to missing parameter information and incorrect warnings.

PyCharm now understands the remaining parameters, checks their types, and recognizes partial objects as valid Callable instances. In the example below, PyCharm correctly identifies the remaining parameter and flags an incompatible argument.

PyCharm understands the remaining parameters of a functools.partial object.

This resolves several long-standing issues with completion, parameter information, and type checking (PY-35363, PY-33618, PY-25698, PY-36385).

Fewer false positives for generics and collections

PyCharm now provides more accurate type inference for generic classes, bounded and constrained type variables, overloaded constructors, and generic __getattr__ methods. For example, calling a method on a bounded type variable now preserves its return type instead of falling back to Any (PY-89863).

Collection inference has improved, too. Nested lists, optional values, dictionary literals, and tuple unpacking produce fewer false warnings (PY-89564, PY-91999). There are also improvements to Literal, Enum, and TypedDict, along with support for PEP 747’s TypeForm[T] and intersection types through ty_extensions.Intersection (PY-90287, PY-62524, PY-89043, PY-87024).

Clearer type warnings and suppression comments

When PyCharm finds a type mismatch, its tooltips now provide clearer explanations, improved formatting, and links to the relevant types. Call and control-flow analysis also handles more cases involving optional callables, bound methods, and paths that never return, reducing false warnings (PY-90555, PY-83644).

When a warning is intentional, you have more control over suppressing it. PyCharm now supports shorter codes in # type: ignore[...] comments and introduces # pycharm: ignore[...] for individual PyCharm inspections. There’s also a Suppress with ‘type: ignore’ comment action, and unknown suppression codes are flagged to catch typos (PY-90787, PY-92114).

PyCharm 2026.3 introduces a new action to suppress individual type warnings using # type: ignore[...] comments.

Testing and developer tools

Better completion for pytest and mocks

Mocks and fixtures can be tricky for an IDE to understand. PyCharm now provides completion for the pytest-mock mocker fixture, navigation for mocker.patch() targets, and completion based on classes passed to Mock(spec=...) or Mock(spec_set=...).

from unittest.mock import Mock

class Client:
    def fetch(self, url: str) -> str:
        return "OK"

client = Mock(spec=Client)
client.fetch("https://example.com")

PyCharm 2026.3 provides completion for methods defined in classes passed to Mock(spec=...).

PyCharm also flags invalid unittest.mock.patch() targets and improves completion and documentation for pytest fixtures (PY-36260, PY-32282).

For projects with custom test infrastructure, you can configure a unittest runner script, such as Django’s tests/runtests.py. If it uses unittest.TextTestRunner, test results appear in PyCharm’s test tree (PY-26068).

More capable type-checker and Ruff integrations

Zuban, a mypy-compatible type checker and language server, joins ty, Pyrefly, and basedpyright as a supported external tool. Its integration provides diagnostics, completion, hover information, inlay hints, and quick fixes. Pyrefly also receives fixes for custom import roots and missing-package installation (PY-85009, PY-89805).

With Ruff, Optimize Imports now removes unused imports and organizes them. Ruff fixes can run automatically on save or before a commit, and the new Apply Ruff Fixes action works across a selected scope, including an entire project. Configuration discovery is also more reliable in multi-root, network, and WSL projects (PY-85534, PY-86352).

Apply Ruff fixes to an individual file, custom scope, or entire project directly from PyCharm.

Django, FastAPI, and SQLAlchemy

Django: You can create run configurations for Django 6.0 tasks, and TextChoices members defined with value-and-label tuples no longer trigger false type errors (PY-86368, PY-92177).

In Django REST framework, rest_framework.reverse.reverse() now offers URL-name completion and navigation, just like Django’s built-in reverse() (PY-78900).

PyCharm 2026.3 provides URL-name completion for Django REST framework’s reverse() function.

FastAPI: Alongside the new interactive dependency graphs, path parameters supplied through dependencies no longer cause incorrect warnings about missing endpoint parameters (PY-88935).

SQLAlchemy: Expanded ORM support improves completion, navigation, and type inference for declarative models. Fixes also cover attributes added through relationship() and columns that use SQLAlchemy’s ENUM type (PY-79736, PY-29212).

Editing and refactoring

Several small fixes make editing more predictable. Pressing Enter after a comment introducing a block now inserts the correct indentation, while moving lines in tab-indented files no longer causes false Unexpected indent warnings (PY-15693, PY-78251).

A new Align consecutive assignments formatter option lines up assignment operators. You can enable it under Settings | Editor | Code Style | Python | Wrapping and Braces. The formatter also does a better job preserving configured blank lines (PY-34006).

Refactoring is safer, too. Inline Function preserves fallback behavior for omitted optional arguments, while Implement abstract methods generates valid implementations of asynchronous abstract methods (PY-81983, PY-53074).

More reliable environments and faster startup

conda: Environment variables configured using conda env config vars set or activate.d scripts are now available when you run scripts, tests, and the Python console from PyCharm. Previously, variables could work in a terminal but be missing inside the IDE—particularly frustrating for projects using GDAL, QGIS, PyMC, or compiler toolchains (PY-55405, PY-38751).

uv: Workspaces using a shared root .venv retain their interpreter after upgrading from PyCharm 2026.1. Projects that use uv pip without a pyproject.toml now use the correct installation command, creating a uv environment no longer requires uv init, and new environments can optionally access system site-packages (PY-91108, PY-85589).

Performance: Python scripts start faster thanks to reduced helper overhead. In particular, simply having matplotlib installed no longer adds approximately 100 milliseconds to the startup of an otherwise empty script (PY-92211).

Debugging improvements

The debugpy debugger mode now supports Set Value and Jump to Cursor. Other improvements include sorting variables by type, displaying plots in the Plots tool window, accepting interactive input with terminal emulation, and using Drop into debugger for failed tests.

In the example below, we’ve paused execution and changed discount from 0.10 to 0.25, allowing us to test a different result without restarting the program.

Changing a variable’s value while debugging in debugpy mode.

Debugging also works more reliably for tests in WSL, applications launched with fastapi dev, and long-running IDE sessions on macOS (PY-90157, PY-90281, PY-90588).

Elsewhere in the debugger, dictionary keys appear in insertion order, Python 3.15’s frozendict can be expanded, and the debug console supports a configurable starting script (PY-75697, PY-21419).

Easier Jupyter notebook workflows

Find now searches cell outputs as well as source code, so you can locate a result without scrolling through the notebook. You can also move several selected cells together or use Extract to Python File to copy them into a regular Python file. A bug that caused scripts embedded in HTML outputs to execute twice has been fixed, too (PY-84634, PY-89385).

Thank you for your feedback

Not every meaningful improvement is a headline feature. Sometimes it’s a false warning disappearing, completion working where it didn’t before, or a debugger behaving as expected. Some fixes in this release trace back to issues reported more than a decade ago.

Download PyCharm 2026.3 and try these improvements in your projects. And if PyCharm gets something wrong, report it in YouTrack. Your reports and votes help us decide what to improve next.

Thanks for using PyCharm, and happy coding!

November 13, 2026 02:28 PM UTC


Python Software Foundation

Python is for everyone: Join in the PSF year-end fundraiser & membership drive!

The Python Software Foundation (PSF) is the charitable organization behind Python, dedicated to advancing, supporting, and protecting the Python programming language and the community that sustains it. That mission is more than words to us; it’s the community we show up for every day. Our tiny but mighty team works hard to deliver the projects, services, and programs that allow Python to be the thriving, independent, community-driven language it is today. Some of what the PSF does includes producing PyCon US, hosting the Python Package Index (PyPI), supporting Developers in Residence, maintaining critical community infrastructure, and more.

Python is for gathering, teaching, learning, playing, researching, exploring, creating, building, working, sustaining. The list goes on and on and on! Support this year’s fundraiser with your donations and memberships to help the PSF, the Python community, and the language stay strong and sustainable. Because Python is for everyone, thanks to you. 

 

There are two direct ways to join through donate.python.org: 

>>> Donate or Become a Member Today! <<<

 
  

If you already donated and/or you’re already a member, you can:

 

Your donations and support:

 

PSF highlights from 2026:

November 13, 2026 08:41 AM UTC


Graham Dumpleton

Muru: canaries without a service mesh

In releases, not restarts I showed the Muru daemon deploying a new release beside the one serving, and only moving traffic across once the new release was ready. That protects you from a release which can't start. It does nothing for a release which starts perfectly well and then fails some of the requests it is given, which in my experience is the more common sort of bad deploy. A health check passes, the deploy reports success, and the first you hear about it is from the people using the site.

The usual answer is a canary. The new release is given a small share of the traffic first, is compared with the release already serving, and only gets more if it holds up. Automating that is normally something added around the server rather than something the server does. On Kubernetes it means something like Argo Rollouts or Flagger, typically with a service mesh or an ingress controller doing the traffic split and Prometheus queries deciding whether the new version is healthy. Off Kubernetes it tends to mean load balancer weights and someone watching a dashboard. Gunicorn, uvicorn and the other Python servers have nothing to say about it, since none of them has any notion of two versions of an application being compared.

Under the Muru daemon, the edge already splits traffic between releases and already counts what each release does with the requests it is given, so the daemon can run the canary itself.

A release with a bug

Since the last post the shop has gained a page for each order, deployed as release r5 the way that post showed. The next change adds prices to those pages:

PRICES = {"teapot": 45.00, "kettle": 60.00}


def order(db, order_id):
    row = db.execute("SELECT item, quantity FROM orders WHERE id = ?", (order_id,)).fetchone()
    if row is None:
        return None
    item, quantity = row
    total = quantity * PRICES[item]
    return f"Order {order_id}: **{quantity} x {item}** for ${total:.2f}"

Someone forgot about mugs. The order pages for teapots and kettles work, the home page works, and the page for the one order of a mug raises a KeyError. The release starts without a problem, so a plain deploy would put it in front of everyone.

Instead it is deployed with --hold, which starts the release but leaves it out of traffic:

muructl deploy shop releases/v6 --hold
muru: release shop@r6 is ready and held: not serving

In the last post I used the Muru-Preview header to reach a held release, and doing that here would find the bug for anyone who thought to check the right page:

curl -H 'Muru-Preview: r6' http://127.0.0.1:8080/orders/1
curl -H 'Muru-Preview: r6' http://127.0.0.1:8080/orders/3
<p>Order 1: <strong>1 x teapot</strong> for $45.00, served by shop@r6</p>
500 Internal Server Error
reason: app_error
request id: 01M6TE1RN0DZ6YEXWKNV4W5AKN

The application raised an exception.

The point of a canary is not having to know which page to check.

Running a canary

A canary is judged on real requests, so to stand in for people using the site I ran a small script with four clients, each requesting one of the home page and the three order pages at random, as fast as the shop would answer them, for 25 seconds. The script doesn't keep cookies, which matters below. With it running:

muructl rollout canary shop@r6 --steps 10,50 --hold 30s
muructl rollout status shop --wait
muru: rollout 1 of shop@r6 started against r5: step 1 of 2: 10% for 30s
muru: rollout 1 running step 1 of 2: 10% for 30s
muru: rollout 1 rolled_back: gate failed at 10%: 5xx rate 21.95% over the limit of 1.00% (baseline 0.00%)

The plan was to give the new release 10% of the requests for 30 seconds, then 50% for another 30, and then all of them. It never got past the first step. The events show how quickly it was decided:

    44 18:20:02 notice   rollout.started      shop@r6                  rollout 1 against r5: 10% for 30s, 50% for 30s
    45 18:20:07 warning  rollout.gate_failed  shop@r6                  step 1 at 10%: 5xx rate 21.95% over the limit of 1.00% (baseline 0.00%)
    46 18:20:07 warning  rollout.rolled_back  shop@r6                  rollout 1: gate failed at 10%: 5xx rate 21.95% over the limit of 1.00% (baseline 0.00%); the candidate is kept for inspection

Five seconds after the canary started, the candidate was out of traffic again. Over the run the script got this back:

{200: 6618, 500: 27}

so 27 requests failed out of 6,645. A plain deploy would have failed a quarter of the order page requests for everyone until somebody noticed and rolled back.

What decided it is what Muru calls a gate. The edge keeps counters for every release, of the requests it answered, those answered with a 5xx status, and histograms of how long requests took overall and how long they waited for a worker. During a step the daemon reads those for the candidate and for the baseline, the release serving everything else, over the same window since the step began. Once the candidate has served 100 requests in the step, the gates are checked every second, and a failing gate rolls back straight away rather than waiting for the hold to end.

Importantly, a gate only fails if the candidate is both over its limit and worse than the baseline. If the database goes away and both releases start failing, that isn't the candidate's fault, and it isn't rolled back for it. The candidate is also kept running out of traffic, so its log can still be looked at, which muructl logs shop@r6 shows has the traceback:

2026-11-07T07:20:07.756Z muru.worker[0]:   File "/Users/graham/shop/releases/v6/app.py", line 22, in order
2026-11-07T07:20:07.756Z muru.worker[0]:     total = quantity * PRICES[item]
2026-11-07T07:20:07.756Z muru.worker[0]:                        ~~~~~~^^^^^^
2026-11-07T07:20:07.756Z muru.worker[0]: KeyError: 'mug'

A release that passes

The next release adds the price of a mug. This time I deployed it held as before, but started the canary with shorter steps, and asked for it to wait for my approval before each step after the first:

muructl deploy shop releases/v7 --hold
muructl rollout canary shop@r7 --steps 10,50 --hold 20s --approve-each-step
muru: rollout 2 of shop@r7 started against r5: step 1 of 2: 10% for 20s

While the candidate had its 10%, I wanted to see who was getting it. The edge doesn't pick a release at random for each request. It gives each new client an anonymous id in a muru_id cookie, and hashes that to decide which release the client gets. Because my load script keeps no cookies, each of its requests counts as a new client, which is why it saw both releases. A browser keeps the cookie, so to see what that means I used curl with a cookie jar as 20 separate clients, each making five requests to the home page:

client 1: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
client 2: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
...
client 7: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
client 8: shop@r7 shop@r7 shop@r7 shop@r7 shop@r7
client 9: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
...
client 16: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5
client 17: shop@r7 shop@r7 shop@r7 shop@r7 shop@r7
client 18: shop@r7 shop@r7 shop@r7 shop@r7 shop@r7
client 19: shop@r7 shop@r7 shop@r7 shop@r7 shop@r7
client 20: shop@r5 shop@r5 shop@r5 shop@r5 shop@r5

Four of the 20 clients landed on the new release, and each of them stayed on it for every request, as did everyone else on the old one. Someone clicking around the site doesn't flip between versions from one page to the next. The buckets are also laid out so that a client in the canary at 10% is still in it at 50%.

After 20 seconds with the gates passing, the rollout stopped and waited:

muructl rollout status shop
APP              ROLLOUT  CANDIDATE    BASELINE     STATE              TRAFFIC  MESSAGE
shop             1        r6           r5           rolled_back        0%       gate failed at 10%: 5xx rate 21.95% over the limit of 1.00% (baseline 0.00%)
shop             2        r7           r5           awaiting_approval  10%      step 2 of 2: 50% for 20s awaits approval: muructl rollout resume shop

Approving the next step and following the rollout to the end:

muructl rollout resume shop
muructl rollout status shop --wait
muru: rollout 2 running: step 2 of 2: 50% for 20s
muru: rollout 2 running step 2 of 2: 50% for 20s
muru: rollout 2 promoted: every step passed its gates

The last step passing promoted the new release to take all of the traffic, with the old one drained and kept warm, so muructl rollback shop would still put things back at once. This time the load script got nothing but 200 responses, 20,421 of them.

When a worker gets stuck

Errors are the easy case, as they show up as 5xx responses. Some bugs never produce a response at all. Here is a release where the order page takes a lock it already holds, but only for kettles:

prices_lock = threading.Lock()


def order(db, order_id):
    ...
    with prices_lock:
        if item == "kettle":
            prices_lock.acquire()  # a bug: takes the lock it already holds
        total = quantity * PRICES[item]

The first request for a kettle blocks forever, and every request after it which wants the lock blocks behind it. Under gunicorn's sync workers the master would eventually kill the stuck worker for not answering its heartbeat, but a threaded worker keeps answering its heartbeat while its request threads are stuck, and nothing outside the server would connect any of that to the deploy which caused it.

Muru has a hard timeout for this, which replaces a worker when one of its requests has run too long. It is off by default, so I gave this release one of three seconds when deploying it, then started a canary as before:

muructl deploy shop releases/v8 --hold --timeout hard=3s
muructl rollout canary shop@r8 --steps 10,50 --hold 30s
muru: rollout 3 of shop@r8 started against r7: step 1 of 2: 10% for 30s
muru: rollout 3 running step 1 of 2: 10% for 30s
muru: rollout 3 rolled_back: gate failed at 10%: 2 worker crashes and timeout kills, over the limit of 0 (baseline 0)

This gate doesn't wait for 100 requests. A worker which crashes or is killed by a timeout counts against the candidate as soon as it happens, and by default one more than the baseline had in the step is enough to roll it back. The events say which workers were killed, and where they were stuck, which came from the stacks the host gathered from inside the worker before replacing it:

    70 20:53:23 warning  worker.timeout_killed shop@r8                  worker 0 (pid 82673) replaced by pid 83393: a request has run for 3.8 s, past the hard timeout of 3 s; at app.py line 27 in order, called from app.py line 39 in application, called from _pool.py line 30 in serve_jobs
    72 20:53:23 warning  worker.timeout_killed shop@r8                  worker 1 (pid 82674) replaced by pid 83395: a request has run for 3.6 s, past the hard timeout of 3 s; at app.py line 27 in order, called from app.py line 39 in application, called from _pool.py line 30 in serve_jobs
    75 20:53:24 warning  rollout.gate_failed  shop@r8                  step 1 at 10%: 2 worker crashes and timeout kills, over the limit of 0 (baseline 0)

Line 27 of app.py is the second acquire(). The release log has the full stack of every thread in each worker as well, much like the muructl debug stacks output from looking inside the workers.

Looking at the timestamps, the rollback was about as quick as with the errors. The hard timeout fired about four seconds into the canary, and the decision to replace a stuck worker is what counted against the candidate, so it was out of traffic within a second. The stuck workers themselves were given their drain grace and went about 15 seconds later, by which time the candidate had no traffic left to lose. The load script ended up with two 504 responses and three requests it gave up on after five seconds, out of about 9,500. That is a long way short of every kettle order hanging for everyone.

There are more gates than the two I have shown. --max-p95 and --max-p99 judge how long the candidate's requests take, --max-queue-wait judges how long they wait for a worker thread, and --probe names URLs the daemon requests from both releases itself every five seconds, with one failing on the candidate alone enough to roll it back.

Sticky clients and previews

The canary above uses the anonymous muru_id cookie to keep each client on one release. That can be changed per route, to hash a header such as a user id set by a proxy in front, a cookie of the application's own, or the client's address. When the canary ends, though, a client on the losing side is moved across on its next request. That is usually fine, but not for an application where a session started on one release shouldn't be continued on another. For that a route can be made sticky, so the edge sets a cookie naming the release a client was given and keeps honouring it for a grace period after that release leaves the route, before it drains.

The preview header used earlier also has a cookie form, muru_preview=r7, which makes it easy to try a held release in a browser, and a route can give a release a share of zero, so it is reachable only by preview, for a dark launch.

What's next

Everything in this post and the last relied on the daemon being able to run more than one release of an application and move traffic between them. When the application is in a container, Muru deliberately does none of that, since there the container is the release, and starting, replacing and rolling back containers is the orchestrator's job. Next I will look at what muructl serve does instead as a container's main process, and why that makes it a better tenant than the usual gunicorn or uvicorn command.

November 13, 2026 12:00 AM UTC

November 12, 2026


Luke Plant

Why I’m calling your contribution “slop”

You have probably arrived at this web page because you have attempted to contribute to an Open Source software project, probably via a Pull Request and I, or perhaps someone else, has called your efforts “slop” and promptly closed/ignored it.

I might also use this word to describe emails or other communication received which are of a similar nature.

This post is addressed to you, and is my attempt to explain why I’ve called it “slop”. I don’t want to repeat myself on this subject, and I also don’t want to leave you totally puzzled, upset and embittered, so I’m writing it once, and just once.

By “slop”, I’m referring especially to AI slop. It is, however, entirely possible that you didn’t use AI, but I can’t tell that, and your code or communication seems to share the same quality problems that AI slop has.

The old way

It would be nice if I could give more detailed feedback about why exactly I think your contribution matches the term “slop”. In the past, when I got a pull request that wasn’t up to scratch from a new contributor, I would try to take the time to give detailed feedback, to effectively train them.

This approach was based on a few assumptions:

Unfortunately, use of AI in software has massively eroded all of the above. The human involvement in creating the PR may have been almost zero. Writing is no longer proof of work. Many people will not benefit at all from the detailed feedback I give, because often they just give up. Basically, the balance that Open Source relied has been destroyed by the slopocalypse, which means that I have to allocate my time in a very different way to the past.

If you are a budding Open Source contributor trying to do the right thing, trying to learn, maybe using AI or maybe not using it all but still getting lumped with those who are, this sucks for you. I’m sorry for that, and for the world that has been lost, but I’m afraid I can’t fix it, and this post is the best I can do.

I’m also sorry for the curtness of my interactions with you, and the cynical tone you might pick up from this post, but again it is a matter of necessity to avoid the burn-out that many others have suffered, and to ensure I can give a good account for how I’ve used my life (Romans 14:12).

What’s wrong with your contribution

It might be that you have used an AI to generate something, or you’ve worked hard on it yourself, and you think it looks pretty good, but it is still labelled slop. Below are some clues as to why that might be. You will have to work through these yourself and see which apply to yours.

The best summary I can give for what I’m looking for in a good quality code contribution is the following:

Every single line, word, token and character should have been written with the understanding that another human being will need to read it, understand it and potentially maintain it indefinitely. This covers:

Of course, the code must actually work (that’s step zero, the computer has to understand it too), and in addition, it must be for the benefit of all current and future users, which means it mustn’t recklessly break anything, for example, and must be designed with long term needs in mind.

In short, the contribution should treat the recipients (reviewers and maintainers) as human beings whose time is extremely valuable, and the project as something valuable that needs to be protected.

Here are some of the most common problems I see with AI generated stuff:

At the end of this, you might think something like this:

Wow, this is really hard! It would probably be faster to start from scratch and write every word myself. That way, I’m only going to write the necessary stuff, because it will take too much effort to write unnecessary things. It will also force me to think about whether something really is necessary, and whether I understand it.

Yep, exactly, all that is true. It was always true of good coding. Doing the above is simply what is necessary when you treat the recipient as a human. A high quality PR was always a lot of work – you have to do all the work needed to make the maintainer’s job easy, because you are asking them to do something for you.

The only thing that has got harder about contributing to an Open Source project is that I’m now not telling you exactly what is wrong, you’ll have to figure that bit out for yourself, and I’m now much more likely to simply put something in the slop bin.

Your own use of AI may well have made things harder for you:

To gain empathy with the recipient, you are indeed better off starting from scratch and literally typing everything in, even if you use AI to gain some insights.

In saying this, don’t think I’m 100% against AI. I have successfully used AI to help debug things that I might never have been able to work out otherwise, and I’ve used agents in very controlled ways where I’ve been able to judge their output and the trade-offs they involve. But, for many of the reasons above, I virtually never use them to write actual code or text if I need another human to read it.

Well, that’s the end. You better believe that I typed every letter myself.

November 12, 2026 11:58 AM UTC


LernerPython blog, from Reuven Lerner

Announcing: Vibe to valid

Vibe coding is fun: You can go from an idea to a working app in a weekend.

But if you don’t have a background in software, how can you be sure your app is secure? Tested? Backed up? Ready for real customers, or for an investor’s technical review?

Today I’m launching Vibe to Valid, an audit service for vibe-coded Python apps.

Here’s how it works:

  1. You share your repo.
  2. I review it and send you a plain-English report: what’s wrong, what’s right, and what to fix first.
  3. If you want, I fix the problems, and set up your AI coding tools so you can keep vibe coding with the AI on a short leash.

There’s also a due diligence audit for investors and acquirers, and a walkthrough session for your team. I’ve been a Python trainer and consultant since 1995, and Claude Code is a core part of my workflow. But every finding is reviewed and explained by me. You’re getting an experienced human, not an automated scan.

Learn more at https://VibeToValid.com, where you can also book a free intro call.

The post Announcing: Vibe to valid appeared first on LernerPython.

November 12, 2026 10:01 AM UTC