skip to navigation
skip to content

Planet Python

Last update: October 22, 2024 04:43 PM UTC

October 22, 2024

Real Python

Understanding Python's Global Interpreter Lock (GIL)

The Python Global Interpreter Lock or GIL, in simple words, is a mutex (or a lock) that allows only one thread to hold the control of the Python interpreter.

This means that only one thread can be in a state of execution at any point in time. The impact of the GIL isn’t visible to developers who execute single-threaded programs, but it can be a performance bottleneck in CPU-bound and multi-threaded code.

Since the GIL allows only one thread to execute at a time even in a multi-threaded architecture with more than one CPU core, the GIL has gained a reputation as an “infamous” feature of Python.

In this video course you’ll learn how the GIL affects the performance of your Python programs, and how you can mitigate the impact it might have on your code.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 22, 2024 02:00 PM UTC

Python Anywhere

Improving PythonAnywhere's File Storage System

PythonAnywhere has been around for over 10 years, and as our platform continues to grow with thousands of users, we’re committed to keeping it in top shape. Part of this involves upgrading some of the older parts of our infrastructure, with a special focus on our file storage servers—some of the oldest systems we have.

October 22, 2024 08:00 AM UTC

Julien Tayon

Tune your guitar with python

Today's exercice is just about turning a very nice example of the python soundevice module into something that works for me© to help me tune my bass.

Long story short, I suck at tuning my instrument and just lost my tuner...

This will require the python module soundevice and matplotlib.

So in order to tune my guitar I indeed need a spectrosonogram that displays the frequencies captured in real time by an audio device with an output readable enough I can actually know if I am nearing a legit frequency called a Note.

The frequencies for the notes are pretty arbitrary and I chose to only show the frequency for E, A , D, G, B since I have a 5 strings bass.
I chose the frequency between 100 and 2000 knowing that anyway any frequency below will trigger harmonics and above will trigger reasonance in the right frequency frame.

Plotting a spectrogram is done by tweaking the eponym matplotlib grapher with values chosen to fit my need and show me a laser thin beam around the right frequency. 
#!/usr/bin/env python3
"""Show a text-mode spectrogram using live microphone data."""
import argparse
import math
import shutil
import matplotlib.pyplot as plt
from multiprocessing import Process, Queue
import matplotlib.animation as animation

import numpy as np
import sounddevice as sd

usage_line = ' press enter to quit,'

def int_or_str(text):
    """Helper function for argument parsing."""
    try:
        return int(text)
    except ValueError:
        return text

try:
    columns, _ = shutil.get_terminal_size()
except AttributeError:
    columns = 80

parser = argparse.ArgumentParser(add_help=False)
parser.add_argument(
    '-l', '--list-devices', action='store_true',
    help='show list of audio devices and exit')
args, remaining = parser.parse_known_args()
if args.list_devices:
    print(sd.query_devices())
    parser.exit(0)
parser = argparse.ArgumentParser(
    description=__doc__ + '\n\nSupported keys:' + usage_line,
    formatter_class=argparse.RawDescriptionHelpFormatter,
    parents=[parser])
parser.add_argument(
    '-b', '--block-duration', type=float, metavar='DURATION', default=50,
    help='block size (default %(default)s milliseconds)')
parser.add_argument(
    '-d', '--device', type=int_or_str,
    help='input device (numeric ID or substring)')
parser.add_argument(
    '-g', '--gain', type=float, default=10,
    help='initial gain factor (default %(default)s)')
parser.add_argument(
    '-r', '--range', type=float, nargs=2,
    metavar=('LOW', 'HIGH'), default=[50, 4000],
    help='frequency range (default %(default)s Hz)')
args = parser.parse_args(remaining)
low, high = args.range
if high <= low:
    parser.error('HIGH must be greater than LOW')
q = Queue()
try:
    samplerate = sd.query_devices(args.device, 'input')['default_samplerate']
    def plot(q):
        global samplerate
        fig, ( ax,axs) = plt.subplots(nrows=2)
        plt.ioff()
        def animate(i,q):
            data = q.get()
            ax.clear()
            axs.clear()
            axs.plot(data)
            ax.set_yticks([
                41.20,	82.41,	164.8,	329.6,	659.3,  # E
                55.00, 	110.0, 	220.0, 	440.0, 	880.0,  # A
                73.42,	146.8,	293.7,	587.3,          # D
                49.00, 	98.00, 	196.0, 	392.0, 	784.0,  #G 
                61.74, 	123.5, 	246.9, 	493.9, 	987.8 ])#B 
            ax.specgram(data[:,-1],mode="magnitude", Fs=samplerate*2, scale="linear",NFFT=9002)
            ax.set_ylim(150,1000)
        ani = animation.FuncAnimation(fig, animate,fargs=(q,), interval=500)
        plt.show()

    plotrt = Process(target=plot, args=(q,))
    plotrt.start()

    def callback(indata, frames, time, status):
        if any(indata):
            q.put(indata)
        else:
            print('no input')

    with sd.InputStream(device=args.device, channels=1, callback=callback,
                        blocksize=int(samplerate * args.block_duration /50 ),
                        samplerate=samplerate) as sound:
        while True:
            response = input()
            if response in ('', 'q', 'Q'):
                break
            for ch in response:
                if ch == '+':
                    args.gain *= 2
                elif ch == '-':
                    args.gain /= 2
                else:
                    print('\x1b[31;40m', usage_line.center(args.columns, '#'),
                          '\x1b[0m', sep='')
                    break
except KeyboardInterrupt:
    parser.exit('Interrupted by user')
except Exception as e:
    parser.exit(type(e).__name__ + ': ' + str(e))

October 22, 2024 05:49 AM UTC

October 21, 2024

Real Python

Python's property(): Add Managed Attributes to Your Classes

With Python’s property(), you can create managed attributes in your classes. You can use managed attributes when you need to modify an attribute’s internal implementation and don’t want to change the class’s public API. Providing stable APIs will prevent you from breaking your users’ code when they rely on your code.

Properties are arguably the most popular way to create managed attributes quickly and in the purest Pythonic style.

In this tutorial, you’ll learn how to:

  • Create managed attributes or properties in your classes
  • Perform lazy attribute evaluation and provide computed attributes
  • Make your classes Pythonic using properties instead of setter and getter methods
  • Create read-only and read-write properties
  • Create consistent and backward-compatible APIs for your classes

You’ll also write practical examples that use property() for validating input data, computing attribute values dynamically, logging your code, and more. To get the most out of this tutorial, you should know the basics of object-oriented programming, classes, and decorators in Python.

Get Your Code: Click here to download the free sample code that shows you how to use Python’s property() to add managed attributes to your classes.

Take the Quiz: Test your knowledge with our interactive “Python's property(): Add Managed Attributes to Your Classes” quiz. You’ll receive a score upon completion to help you track your learning progress:

Python's property(): Add Managed Attributes to Your Classes

Interactive Quiz

Python's property(): Add Managed Attributes to Your Classes

In this quiz, you'll test your understanding of Python's property(). With this knowledge, you'll be able to create managed attributes in your classes, perform lazy attribute evaluation, provide computed attributes, and more.

Managing Attributes in Your Classes

When you define a class in an object-oriented programming language, you’ll probably end up with some instance and class attributes. In other words, you’ll end up with variables that are accessible through the instance, class, or even both, depending on the language. Attributes represent and hold the internal state of a given object, which you’ll often need to access and mutate.

Typically, you have at least two ways to access and mutate an attribute. Either you can access and mutate the attribute directly or you can use methods. Methods are functions attached to a given class. They provide the behaviors and actions that an object can perform with its internal data and attributes.

If you expose attributes to the user, then they become part of the class’s public API. This means that your users will access and mutate them directly in their code. The problem comes when you need to change the internal implementation of a given attribute.

Say you’re working on a Circle class and add an attribute called .radius, making it public. You finish coding the class and ship it to your end users. They start using Circle in their code to create a lot of awesome projects and applications. Good job!

Now suppose that you have an important user that comes to you with a new requirement. They don’t want Circle to store the radius any longer. Instead, they want a public .diameter attribute.

At this point, removing .radius to start using .diameter could break the code of some of your other users. You need to manage this situation in a way other than removing .radius.

Programming languages such as Java and C++ encourage you to never expose your attributes to avoid this kind of problem. Instead, you should provide getter and setter methods, also known as accessors and mutators, respectively. These methods offer a way to change the internal implementation of your attributes without changing your public API.

Note: Getter and setter methods are often considered an anti-pattern and a signal of poor object-oriented design. The main argument behind this proposition is that these methods break encapsulation. They allow you to access and mutate the components of your objects from the outside.

These programming languages need getter and setter methods because they don’t have a suitable way to change an attribute’s internal implementation when a given requirement changes. Changing the internal implementation would require an API modification, which can break your end users’ code.

The Getter and Setter Approach in Python

Technically, there’s nothing that stops you from using getter and setter methods in Python. Here’s a quick example that shows how this approach would look:

Python point_v1.py 
class Point:
    def __init__(self, x, y):
        self._x = x
        self._y = y

    def get_x(self):
        return self._x

    def set_x(self, value):
        self._x = value

    def get_y(self):
        return self._y

    def set_y(self, value):
        self._y = value
Copied!

In this example, you create a Point class with two non-public attributes ._x and ._y to hold the Cartesian coordinates of the point at hand.

Note: Python doesn’t have the notion of access modifiers, such as private, protected, and public, to restrict access to attributes and methods. In Python, the distinction is between public and non-public class members.

If you want to signal that a given attribute or method is non-public, then you have to use the well-known Python convention of prefixing the name with an underscore (_). That’s the reason behind the naming of the attributes ._x and ._y.

Note that this is just a convention. It doesn’t stop you and other programmers from accessing the attributes using dot notation, as in obj._attr. However, it’s bad practice to violate this convention.

To access and mutate the value of either ._x or ._y, you can use the corresponding getter and setter methods. Go ahead and save the above definition of Point in a Python module and import the class into an interactive session. Then run the following code:

Python 
>>> from point_v1 import Point

>>> point = Point(12, 5)
>>> point.get_x()
12
>>> point.get_y()
5

>>> point.set_x(42)
>>> point.get_x()
42

>>> # Non-public attributes are still accessible
>>> point._x
42
>>> point._y
5
Copied!

With .get_x() and .get_y(), you can access the current values of ._x and ._y. You can use the setter method to store a new value in the corresponding managed attribute. From the two final examples, you can confirm that Python doesn’t restrict access to non-public attributes. Whether or not you access them directly is up to you.

Read the full article at https://realpython.com/python-property/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 21, 2024 02:00 PM UTC

Python Bytes

#406 What's on Django TV tonight?

<strong>Topics covered in this episode:</strong><br> <ul> <li><a href="https://opensourcepledge.com?featured_on=pythonbytes"><strong>Open Source Pledge</strong></a></li> <li><strong>Jeff Triplet's <a href="https://djangotv.com?featured_on=pythonbytes">DjangoTV</a></strong></li> <li><strong><a href="https://peps.python.org/pep-0735/?featured_on=pythonbytes">PEP 735 – Dependency Groups in pyproject.toml</a></strong></li> <li><strong><a href="https://livereload.readthedocs.io/en/latest/index.html?featured_on=pythonbytes">livereload</a></strong></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=XzXG6cRP2Mk' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="406">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by ScoutAPM: <a href="https://pythonbytes.fm/scout"><strong>pythonbytes.fm/scout</strong></a></p> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Monday</strong> at 10am PT. Older video versions available there too.</p> <p>Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it. </p> <p><strong>Brian #1:</strong> <a href="https://opensourcepledge.com?featured_on=pythonbytes"><strong>Open Source Pledge</strong></a></p> <ul> <li>Learned about this because of this post <ul> <li><a href="https://www.djangoproject.com/weblog/2024/oct/08/why-django-supports-the-open-source-pledge/?featured_on=pythonbytes">Why Django supports the Open Source Pledge</a></li> </ul></li> <li>Steps <ul> <li>Pay Open Source maintainers. <ul> <li>Min to participate is 2k/year/dev at your company</li> </ul></li> <li>Self-report annually <ul> <li>Publish a blog post outlining your payments</li> </ul></li> </ul></li> <li><a href="https://lucumr.pocoo.org/2024/10/14/mixing-oss-and-money/?featured_on=pythonbytes">Armin’s post</a> about launching Open Source Pledge and mixing money with open source</li> </ul> <p><strong>Michael #2:</strong> Jeff Triplet's <a href="https://djangotv.com?featured_on=pythonbytes">DjangoTV</a></p> <ul> <li>A nice aggregation of lots of Django conference talks</li> <li>Filter by conference</li> <li>Good search as well</li> </ul> <p><strong>Brian #3:</strong> <a href="https://peps.python.org/pep-0735/?featured_on=pythonbytes">PEP 735 – Dependency Groups in pyproject.toml</a></p> <ul> <li>Author: Stephen Rosen, Sponsor: Brett Cannon, PEP-Delegate: Paul Moore</li> <li>Accepted. Resolotion Oct 10, 2024</li> <li>“This PEP specifies a mechanism for storing package requirements in pyproject.toml files such that they are not included in any built distribution of the project.”</li> <li>Allow us to define named groups of dependencies that can be independent of the main project.</li> <li><p>ex:</p> <pre><code>[dependency-groups] test = ["pytest", "coverage"] docs = ["sphinx", "sphinx-rtd-theme"] typing = ["mypy", "types-requests"] typing-test = [{include-group = "typing"}, {include-group = "test"}, "useful-types"] </code></pre></li> <li><p>“might” work like this: pip install --dependency-groups=test,typing</p> <ul> <li>but tool venders are able to define how they use groups. Of course.</li> </ul></li> <li>Similar solutions <ul> <li>multiple requirements.txt files: requirements_test.txt, requirements_docs.txt, etc. <ul> <li>no standard naming convention, not standardized </li> </ul></li> <li>package extras: <ul> <li>not gauranteed to be statically defined (TIL)</li> <li>additional to main dependencies, so not independent</li> </ul></li> </ul></li> </ul> <p><strong>Michael #4:</strong> <a href="https://livereload.readthedocs.io/en/latest/index.html?featured_on=pythonbytes">livereload</a></p> <ul> <li>Example from talkpython.fm: <a href="https://gist.github.com/mikeckennedy/4e1378477a6d174aa8d59921f8db89c3?featured_on=pythonbytes"><strong>asset_bundler_watcher.py</strong></a></li> <li><a href="https://livereload.readthedocs.io/en/latest/index.html?featured_on=pythonbytes">The docs are sparse</a>, so see the gist above</li> </ul> <p><strong>Extras</strong> </p> <p>Brian:</p> <ul> <li><a href="https://kjaymiller.com/blog/personal-blogs-are-no-longer-personal-when-ai-gets-too-involved.html?featured_on=pythonbytes">Personal Blogs are no longer personal when AI gets too involved</a> - KJayMiller</li> <li><a href="https://stefaniemolin.com/articles/devx/pre-commit/exif-stripper/?featured_on=pythonbytes">Mind Your Image Metadata</a> - Stefanie Molin</li> </ul> <p>Michael:</p> <ul> <li>14% of our listeners are in Germany, thanks Germany! <ul> <li>Prost!</li> </ul></li> <li><a href="https://www.hetzner.com/cloud/?featured_on=pythonbytes">Hetzner comes to the US</a></li> </ul> <p><strong>Joke:</strong> </p> <ul> <li>A programmer’s partner asks them: “Would you go get a loaf of bread from the store? And if they have eggs, get a dozen.” </li> <li>A while later, the programmer returns with 12 loaves of bread and says “They had eggs.” <ul> <li>From https://savvyprogrammer.io/software-jokes/</li> </ul></li> </ul>

October 21, 2024 08:00 AM UTC

Zato Blog

HL7 FHIR Security with Basic Auth, OAuth and SSL/TLS

HL7 FHIR Security with Basic Auth, OAuth and SSL/TLS

2024-10-21, by Dariusz Suchojad

HL7 FHIR Security

Preliminary reading: HL7 FHIR Integrations in Python

FHIR servers offer their APIs using REST, which in turn means that they are HTTP servers under the hood. As a result, a few common security mechanisms can be employed to secure access to the servers when you invoke them.

When you integrate with a FHIR server, consult with its maintainers what of these, if any, should be used.

Note that Basic Auth and OAuth, when used over HTTP, are mutually exclusive. The HTTP protocol reserves only one header, called Authorization, for the authorization credentials and there is no standard way in the protocol to use more than one authorization mechanism.

On the other hand, the SSL/TLS encryption is independent of the credentials used and can be employed with Basic Auth, OAuth, or in scenarios when no authorization header is sent at all.

This is why, when you create or update a FHIR connection, you can set Basic Auth or OAuth and SSL/TLS options independently, as in the screenshot below:

Basic Auth

Basic Auth is a combination of username and password credentials. They are allocated in advance by the maintainers of a FHIR server.

To use them in your integrations, create a new Basic Auth definition in the Dashboard, set its password, and assign it to an existing or new outgoing FHIR connection. No restarts nor reloads are required.

OAuth

Authentication with OAuth is built around a notion of short-lived tokens, which are simple strings. Your Zato server has credentials, much like a username and password although they are called a client ID and secret, based on which an authentication server issues tokens that let Zato prove that in fact it does have the correct credentials and that it is allowed to invoke a particular API as it is defined through scopes.

Scopes can be likened to permissions to access a subset of resources from a FHIR server. When Zato receives a token from an authentication server the token is inherently linked to specific scopes within which it can interact with the FHIR server.

That tokens are short-lived means that they need to be refreshed periodically, e.g. at least once per hour Zato needs to ask the authentication server for a new token based on its credentials. This process takes place under the hood and requires no configuration on your part.

To use OAuth in your integrations, create a new OAuth definition in the Dashboard, set its secret, and assign it to an existing or new outgoing FHIR connection. No restarts nor reloads are required.

SSL/TLS encryption

If the FHIR server that a connection points to uses SSL/TLS then a question arises of how to validate the certificate of the Certificate Authority (CA) that signed the certificate that the FHIR server uses. There are several options:

Next steps:

➀ Read about how to use Python to build and integrate enterprise APIs that your tests will cover
➀ Python API integration tutorial
➀ Python Integration platform as a Service (iPaaS)
➀ What is an Enterprise Service Bus (ESB)? What is SOA?

More blog posts➀

October 21, 2024 07:43 AM UTC

Julien Tayon

Hello world part II : actually recoding print

In part I we explored the pre-requisite in order to code print : having a grasp on the framebuffer.

Here, we are gonna deep inside one of the most overlooked object oriented abastraction : a file and actually print what we can of hello world in 100 lines of code.



The file handler and the file descriptor



These two abstractions are the low level and high level abstractions of the same thing : a view on something more complex which access has been encapsulated in generic methods. Actually when you code a framebuffer driver you provide function pointers that are specialized to your device and you may omit those common to the class. This is done with a double lookup on the major node, minor node number. Of those « generic » methods you have : seek, write, tell, truncate, open, read, close ...
The file handler in python also handles extra bytes (pun) of facilities : like character encoding, stats, and buffering.

Here, we work with the low level abstraction : the file which we access with fileno through its file descriptor. And thanks to this abstraction, you don't care if the underlying implementation fragments the file itself (ex: on a hard drive), you can magically always ask without caring for the gory details to read any arbitrary block or chararacters at any given position.

Two of the most used methods on files here are seek and write.

The file descriptor method write is sensitive to the positionning set by seek. Hence, we can write a line, and position ourselves one line below to write the next line in a character.

matrices as a view of a row based array



When I speak of rows and columns I evocate abstractions that are not related to the framebuffer.

The coordinates are an abstraction we build for convenience to say I want to write from this line at this column.
And since human beings bug after 2 dimensions we split the last dimnension in a vector of dimension 4 called a pixel.
get_at function illustrates our use of this trick to position the (invisible) cursor at any given position on the screen expressed for clarity in size of glyphes.
We could actually code all this exercice through a 3D view of the framebuffer. I just wouldn't be able to pack the code in less than 100 lines of code and would introduce useless abstractions.

But if you have doubt on the numerous seek I do and why I mutiply lines and columns value the way I do check the preceding link for an understanding of raw based array nth matricial dimensionnal views.

fonts, chars glyphs...



Here we are gonna take matrices defining the glyphes (what you actually see on screen) by 8x8 = 64 1D array and map them on the screen with put_char. Put char does a little bit of magic by relying on python to do the chararcter to glyph conversion through the dict lookup that expecting strings does a de factor codepoint to glyph conversion without having to pass the codepoint conversion.

The set of characters to glyphs conversion with their common property is a font.

The hidden console



The console is an abstraction that keeps track of the global states such as : what is the current line we print at. Thus, here, being lazy I use the global variables instead of a singleton named « console » or « term » to keep track of them. But first and foremost, these « abstractions » are just expectations we share in a common mental mode. Like we expect « print » to add a newline at the end of the string and begin the printing at the next line.

The to be finished example



I limited the code to 100 lines so that it's fairly readable. I let as an exercise the following points : I want to point out the true python print function is WAY MORE COMPLEX than this terse example and also handle magic conversion from memory objects to their string counterpart (like integers that are converted to their decimal representation), it handles buffering, encoding, and so much more. This is merely a toy to dive into the complexity of the mission at hand.
This example is a part of a project to write « hello world » on the framebuffer in numerous languages, bash included.

Annexe : the code




#!/usr/bin/env python3
from struct import pack
from os import SEEK_CUR, lseek as  seek, write
w,h =map(int, open("/sys/class/graphics/fb0/virtual_size").read().split(","))
so_pixel = 4
stride = w * so_pixel

encode = lambda b,g,r,a : pack("4B",b,g,r,a)

font = {
    "height":8,
    "width":8,
    'void' : [ 
        0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 1, 1, 1, 1, 0, 0,
        0, 1, 0, 0, 0, 0, 1, 0, 
        0, 0, 1, 0, 0, 1, 1, 0,
        0, 0, 0, 0, 1, 0, 0, 0,
        0, 0, 0, 0, 1, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 1, 0, 0, 0, 
       ],
    "l":[ 
        0, 0, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0,
        0, 0, 1, 1, 1, 1, 0, 0,
        ],
    "o": [
        0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 1, 1, 1, 1, 0, 0,
        0, 1, 0, 0, 0, 0, 1, 0,
        0, 1, 0, 0, 0, 0, 1, 0,
        0, 1, 0, 0, 0, 0, 1, 0,
        0, 1, 0, 0, 0, 0, 1, 0,
        0, 0, 1, 0, 0, 1, 0, 0,
        0, 0, 0, 1, 1, 0, 0, 0],
    "h": [
        0, 0, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0,
        0, 1, 1, 1, 1, 0, 0, 0,
        0, 1, 0, 0, 1, 0, 0, 0,
        0, 1, 1, 0, 1, 0, 0, 0,
        0, 1, 0, 0, 1, 0, 0, 0],
}

def go_at(fh, x, y): 
   global stride
   seek(fh.fileno(),x*so_pixel + y *stride, 0)

def next_line(fh, reminder):
    seek(fh.fileno(), stride - reminder, SEEK_CUR)

def put_char(fh, x,y, letter):
    go_at(fh, x, y)
    black = encode(0,0,0,255)
    white = encode(255,255,255,255)
    char = font.get(letter, None) or font["void"]
    line = ""
    for col,pixel in enumerate(char):
        write(fh.fileno(), white if pixel else black)
        if (col%font["width"]==font["width"]-1):
            next_line(fh, so_pixel * font["width"])
COL=0
LIN=0

OUT = open("/dev/fb0", "bw")
FD = OUT.fileno()

def newline():
    global OUT,LIN,COL
    LIN+=1
    go_at(OUT, 0, LIN * font["height"])

def print_(line):
    global OUT, COL, LIN
    COL=0
    for c in line:
        if c == "\n":
            newline()
        put_char(OUT,COL * font["width"] , LIN * font['height'], c)
        COL+=1
    newline() 

for i in range(30):
    print_("hello lol")

October 21, 2024 05:10 AM UTC

Seth Michael Larson

Python and Sigstore

Python and Sigstore

About ‱ Blog ‱ Cool URLs

Python and Sigstore

Published 2024-10-21 by Seth Larson
Reading time: minutes

This critical role would not be possible without funding from the Alpha-Omega project. Massive thank-you to Alpha-Omega for investing in the security of the Python ecosystem!

I was a guest on the Open Source Security podcast this week talking about Sigstore and Python among other things I'm working on at the Python Software Foundation.

Sigstore is a digital signing method that has been used by CPython since 3.11.0 which focuses on ergonomics and uses short-lived keys with strongly-bound human-readable identities via OpenID Connect.

CPython also provides digital signatures using PGP and has been doing so for much longer. Below is a diagram showing the current state of affairs:

Distro (Offline?)   
Distro (Offline?)   
python.org
python.org
GitHub
GitHub
Source Code (tar.gz)
Source Code (tar.gz)
PGP Signature
PGP Signature
Sigstore Bundle
...@python.org
Sigstore Bundle...
CPython
Release
Infra
CPython...
Distro
Package
(deb, rpm)
Distro...
Distro
Build
Infra
Distro...
PGP Keys
PGP Keys
Accounts (GitHub, Gmail)
Accounts (Git...



Release
Manager
ReleaseMan...
Rebuild from source
Rebuild from...
Not used for verification
Not used for...Text is not SVG - cannot display
CPython offers two verification methods: PGP and Sigstore. Verifiers choose which method to use.

From this diagram you can see the two "sources" of identity provided by PGP and Sigstore both link back to release managers. PGP relies on private keys which are maintained and protected by individual release managers to create signatures of artifacts. Sigstore uses third-parties which support OpenID Connect such as GitHub and Google to bind a human-readable identity like "thomas@python.org" to a signing certificate and signature that can be later verified.

Why Sigstore?

The problem is that securely maintaining PGP private keys is not an easy task and the burden rests on volunteers for a minimum of 7 years (1 year of pre-releases then 5 years of bug and security fixes across at least two consecutive releases). Contrast that experience to Sigstore where release managers only need to click a button to OAuth sign-in during the release process.

Sigstore externalizes the operational burden of maintaining long-lived signing keys from individual volunteers to teams of people who run an identity platform professionally and the public-good code-signing certificate authority Fulcio. This seems like a pretty good trade-off, considering we already trust these platforms' identity management teams for things like GitHub accounts.

For this reason, I've authored PEP 761 providing a deprecation and discontinuance plan of PGP signatures for CPython artifacts. You can join the discussion of this PEP on discuss.python.org. Big thank-you to Hugo van Kemenade, the release manager for Python 3.14 for sponsoring my first PEP and helping me with the process and thanks to William Woodruff for reviewing the PEP draft and explaining the nitty-gritty details of Sigstore.

This PEP deprecates the expectation that future CPython releases will provide PGP signatures and sets a timeline for discontinuance (Python 3.14) and a mechanism for that timeline to be extended by a vote of the Steering Council, if necessary.

What do verifiers need?

Signatures are only useful if they are verified, so we must weigh the needs of verifiers!

CPython's expected downstream verifiers are primarily "distributions" of CPython, such as through a Linux distro (Debian, Fedora, Gentoo, etc) or in a container image. There are other distributions of Python such as pyenv and python-build-standalone.

These users of CPython's source code are great places to verify signatures, as they're likely to be high value targets themselves and can provide a consistent stream of verifications. If any one signature verification were to fail, it would signal that something is wrong with upstream CPython artifacts or python.org and would likely be investigated and remediated quickly.

This further constrains attackers looking to affect CPython downstream users, as compromising python.org would no longer be enough. Instead, attackers would need to compromise the build infrastructure or CPython source code.

From discussions, the requirements that I've gathered from verifiers are:

  • Need a tool for verification that can be packaged by distros. The recommended tool for verifying with a CLI is either Cosign or sigstore-python, both of which have challenges for Linux distro packagers.
  • OS packages would require Cosign to be packaged in those OS package managers. This isn't trivial as it requires the Go toolchain to build.
  • Docker and other container images want verification tools to be available at the OS level. Needing to pull these externally (from Cosign's GitHub releases) would require multi-stage builds which they want to avoid if possible. Today Cosign is available in Alpine, but not yet in Debian (but there is the beginnings of support), Gentoo, or Fedora.
  • Offline verification is important. Many package ecosystems ship the signatures inside the packages to enable "build from source" (such as Gentoo), and there's no guarantee that the package is being built online. It's okay that revocation or root of trust updates can't happen when offline. Cosign and other Sigstore verification tools support offline verification if the root of trust is "bundled" as a file locally.
  • Online periodic updates to revocations and root of trust. Cosign and other Sigstore verifiers supports this use-case as the default behavior.

Overall, the availability of Cosign in OS package managers appears to be the biggest blocker to see adoption for verifying CPython's Sigstore signatures.

Why adopt a new signature method?

So why have CPython's Sigstore signatures not seen adoption despite being available for multiple years? Here's a short list of self-reinforcing reasons:

  • Verifiers don't need to adopt the new signature method (Sigstore), because the existing one (PGP) works and there's no expectation for discontinuance.
  • Signers can't migrate away from the old signature method because there's apparent demand from verifiers for the old signature method.
  • Verifiers don't try or test the new signature method, so the maintainers of signature tooling can't learn about or improve verifier use-cases.
  • Concurrently supporting multiple signature methods is more work for both signers and verifiers.
  • There are fewer available signatures using the new signing method, so the value of adopting the method as a verifier is less (but maybe this will change soon?).

Keep in mind that almost everyone involved in the above scenarios are volunteers. Doing work to adopt a new process when existing processes are working can feel like "busy-work", but I don't think that's the case for Sigstore.

Sigstore's benefits for ergonomics paired with its ability to use workload identity are two stand-out features compared to PGP. Workload identity being extra important now that many projects are moving to hosted build infrastructure for releases.

Workload Identity

Sigstore supporting workload identity means that release manager accounts can no longer be hijacked to produce bad signatures. Artifacts get signed by the build infrastructure provider directly:

Distro (Offline?)
Distro (Offline?)
python.org
python.org
GitHub
GitHub
Source Code (tar.gz)
Source Code (tar.gz)
Sigstore Bundle
python/release
Sigstore Bundle...
CPython
Release
Infra
CPython...
Distro
Package
(deb, rpm)
Distro...
Distro
Build
Infra
Distro...
Accounts (GitHub, Gmail)
Accounts (Git...



Release
Manager
ReleaseMan...
Rebuild from source
Rebuild from...
Workload identity: verify artifacts came from CPython release infra

Switching to workload identity also means downstream verifiers no longer need to make changes when new release managers join the project, the expected identity would always be gh/python/release-tools/....

We still have a ways to go to adopt workload identity for CPython because our macOS and Windows release processes don't use hosted build platforms that support OpenID Connect and Sigstore. That means that for now we'll keep using release manager identities.

But this future may not be far off for Python packages hosted on PyPI...

Many more signatures are coming!

William Woodruff and the team at Trail of Bits have authored PEP 740 which is provisionally accepted. The PEP specifies how attestations that can be verified by PyPI (like Sigstore) using workload identities specified with the secure publishing feature "Trusted Publishers" and then served alongside artifacts on PyPI.

There's a lot more to this story (but it's not for me to tell). Given Trusted Publishers' success, there clearly are exciting times are ahead. Subscribe to the PyPI blog to learn more once the project is complete.

That's all for this post! 👋 If you're interested in more you can read the last report.

Have thoughts or questions? Let's chat over email or social:

Want more articles like this one? Get notified of new posts by subscribing to the RSS feed or the email newsletter. I won't share your email or send spam, only whatever this is!

Want more content now? This blog's archive has ready-to-read articles. I also curate a list of cool URLs I find on the internet.

Find a typo? This blog is open source, pull requests are appreciated.

Thanks for reading! ♡ This work is licensed under CC BY-SA 4.0

︎

October 21, 2024 12:00 AM UTC

October 20, 2024

Real Python

Quiz: Pydantic: Simplifying Data Validation in Python

In this quiz, you’ll test your understanding of Pydantic. Pydantic is a powerful data validation library for Python. You can also use a related library, pydantic-settings, for settings management.

By working through this quiz, you’ll revisit how to work with data schemas with Pydantic’s BaseModel, write custom validators for complex use cases, validate function arguments with Pydantic’s @validate_call, and manage settings and configure applications with pydantic-settings.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 20, 2024 12:00 PM UTC

Julien Tayon

Revisiting hello world : coding print from scratch part I

The « hello world » example is about standing on the shoulders of the giant and learn how to use function as tools.

Most coders will use print during their whole life without actually coding it. However, it is a fun exercise.

The framebuffer



Given you are on linux you probably have a device named /dev/fb0 if you don't, you can't do this. The framebuffer is a view of the linear graphical memory used by your video card where what you see on the screen is stored ... at the condition you are in console mode and you have the rights.

On my debian centric distribution, to give the_user permissions to write in the framebuffer I must add the_user to group_video. This can be done with sudo adduser the_user video or sudo vigr.

Then, you have to be in console mode. To switch from xorg/wayland to the console back and forth I use the Ctrl + Alt + Fn combination to switch off from X and Alt + Fn to switch back to X (it's called switching the virtual console).

Once this is done you check you have rights by doing 
cat /dev/urandom > /dev/fb0
which should fill your screen with random colors and insult you stating there is no more room left on the device. SNAFU : everyhting works as intended.


The pixel



Framebuffer don't know about pixels made of Red, Green, Blue and alpha (given you have a video card that is less than 20 years old), they are just made of memory. We will have to slowly build up our understanding of what this is all about.

The in memory layout may differ according to the hardware, some are having a RGBA layout, mine, the i915 is having a BGRA layout. The following example may need to be rewritten with different hardware if the output is not consistent with your assumption.

Determining the memory layout and coordinates



We will do a test and validate code session : first we make assumption on where the colours are by writting 3 squares of Red, Blue and Green on the screen, then, we will snapshot the screen.

$ cat fb.py
#!/usr/bin/env python3
from struct import pack
w,h =map(int, open("/sys/class/graphics/fb0/virtual_size").read().split(","))
midx = w//2
midy = h//2

encode = lambda b,g,r,a : pack("4B",b,g,r,a)

with open("/dev/fb0", "wb") as f:
    for y in range(0,h):
        for x in range(0,w):
            f.write(encode(
                not x%100 and 0xA0 or x<midx and 0xFF or 0, #blue
                y<midy and 0xFF or 0,                       #green
                x>midx and y>midy and 0xFF or 0,            #red
                0,
            ))
The only « trick » is the use of pack to encode the four colour bytes in a byte array that is written to the framebuffer filehandler. If the code works correctly we should validate the following assumptions: And if the world is consistent we can read from the framebuffer and snapshot it in the most trivial picture encoding witch is Portable PixMap format. A portable Pixmap is made of
The code for this is straigh forward : 
$ cat snap.py
#!/usr/bin/env python
from struct import unpack
w,h = map( int,open("/sys/class/graphics/fb0/virtual_size").read().split(","))

# returns b g r a
decode = lambda pixel : unpack("4B", pixel)

def pr(b,g,r,a):
    print("%d %d %d" % (r,g,b))

print(f"""P3
{w} {h}
255
""")

with open("/dev/fb0", "rb") as fin:
    while pixel := fin.read(4):
        pr(*decode(pixel))
Here the only trick is we use the symetrical function of pack, unpack to decode the pixel in the four colour bytes.

wrapping up part one



Asumming you can install fim the framebuffer image wiewer and you installed imagemagick : you can now do 
./fb.py && snap.py > this.ppm  && convert this.ppm this.jpg && fim this.jpg
Doing so, you should have the same picture showing twice without an error like this :
As an exercise, you can vary the fb.py to make funny output, or code a PPM viewer that print back your ppm to the screen.

October 20, 2024 06:01 AM UTC

October 19, 2024

Python Does What?!

Enums make good singletons

It's simple and common to allocate a marker object to represent missing or null data.

MISSING = object()
There's a slightly more verbose construct with some advantages:
import enum

class MissingType(enum.Enum):
    MISSING = "MISSING"

MISSING = MissingEnum.MISSING
Type checkers understand that MISSING is the only possible value of MissingType; so you can use is checks:
def or_1(val: float | MissingType = MISSING) -> float:
    if scale is not MISSING:
        return 1.0
    return scale
mypy understands this is type correct.
More broadly, the semantics of a single-value enum are the same as a singleton. For example, neither singletons nor enums should have additional instances allocated. Instead of fixing bugs one by one with custom __init__ and __deepcopy__, the correct behaviors come for free.

October 19, 2024 04:20 PM UTC

Armin Ronacher

Serendipity

Life begins.
An empty page.
A hollow echo on an barren stage.
We wander through the quiet air,
Unsure of what awaits us there.
With work and study, I filled the days,
But purpose wanes in fading haze.
An emptiness remained inside,
A restless longing I tried to hide.
But then a hand, warm in the night,
Turns the dark to softest light.
Two souls meet and pledge their vow,
To fill what life is missing now.
By chance, our paths became entwined,
Through joy and laughter, lives combined.
Soon tiny hands, starry-eyed,
Fill our home with love and pride.
With every laugh, with every tear,
You fill the space, bring kindness near.
Two hearts become one place,
Of love and hope,
a shared embrace.
My gratitude grows deep and wide,
For every moment, side by side.
Through joy, sorrow, pain, persistence,
You bring meaning to my existence.

October 19, 2024 12:00 AM UTC

October 18, 2024

Real Python

The Real Python Podcast – Episode #224: Narwhals: Expanding DataFrame Compatibility Between Libraries

How does a Python tool support all types of DataFrames and their various features? Could a lightweight library be used to add compatibility for newer formats like Polars or PyArrow? This week on the show, we speak with Marco Gorelli about his project, Narwhals.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 18, 2024 12:00 PM UTC

Programiz

Python match
case Statement

The match
case statement allows us to execute different actions based on the value of an expression. In this tutorial, you will learn how to use the Python match
case with the help of examples.

October 18, 2024 04:04 AM UTC

Matt Layman

Epic Debugging, Hilarious Outcome - Building SaaS #205

In this episode, I dug into an issue with sign up. What I thought was going to be a minor issue to fix turned into a massively intense debugging session. We ended up going very deep with the django-allauth package to understand what was going on.

October 18, 2024 12:00 AM UTC

October 17, 2024

Real Python

Quiz: Single and Double Underscores in Python Names

In this quiz, you’ll test your understanding of Single and Double Underscores in Python Names.

By working through this quiz, you’ll revisit Python naming conventions that rely on using underscores (_), how to differentiate public and non-public names by using a single leading underscore, how to use double leading underscores to leverage name mangling in Python classes, and other common uses of underscores in Python names.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 17, 2024 12:00 PM UTC

Talk Python to Me

#481: Python Opinions and Zeitgeist with Hynek

Hynek has been writing and speaking on some of the most significant topics in the Python space and I've enjoyed his takes. So I invited him on the show to share them with all of us. This episode really epitomizes one of the reasons I launched Talk Python 9 years ago. It's as if we run into each other at a bar during a conference and I ask Hynek, "So what are your thoughts on ..." and we dive down the rabbit hole for an hour. I hope you enjoy it.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/workos'>WorkOS</a><br> <a href='https://talkpython.fm/bluehost'>Bluehost</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Hynek Schlawack on Mastodon</b>: <a href="https://mastodon.social/@hynek?featured_on=talkpython" target="_blank" >@hynek</a><br/> <br/> <b>Why I Still Use Python Virtual Environments in Docker</b>: <a href="https://hynek.me/articles/docker-virtualenv/?featured_on=talkpython" target="_blank" >hynek.me</a><br/> <b>Production-ready Python Docker Containers with uv</b>: <a href="https://hynek.me/articles/docker-uv/?featured_on=talkpython" target="_blank" >hynek.me</a><br/> <b>Attrs</b>: <a href="https://github.com/python-attrs/attrs?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>uv</b>: <a href="https://docs.astral.sh/uv/?featured_on=talkpython" target="_blank" >astral.sh</a><br/> <b>What’s New In Python 4</b>: <a href="https://docs.python.org/3.13/whatsnew/3.13.html?featured_on=talkpython" target="_blank" >python.org</a><br/> <b>BusyBox</b>: <a href="https://www.busybox.net?featured_on=talkpython" target="_blank" >busybox.net</a><br/> <b>Hynek's YouTube Channel</b>: <a href="https://www.youtube.com/@The_Hynek" target="_blank" >youtube.com</a><br/> <b>MOPUp for macOS</b>: <a href="https://github.com/glyph/MOPUp?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>Homebrew Python Is Not For You</b>: <a href="https://justinmayer.com/posts/homebrew-python-is-not-for-you/?featured_on=talkpython" target="_blank" >justinmayer.com</a><br/> <b>argon2-cffi: Argon2 for Python</b>: <a href="https://github.com/hynek/argon2-cffi?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>pytest-freethreaded</b>: <a href="https://github.com/tonybaloney/pytest-freethreaded?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>LM Studio</b>: <a href="https://lmstudio.ai?featured_on=talkpython" target="_blank" >lmstudio.ai</a><br/> <b>StackOverflow Trends Graph</b>: <a href="https://trends.stackoverflow.co/?tags=java,c,python,c%23,vb.net,javascript,assembly,php,perl,ruby,swift,r,objective-c&featured_on=talkpython" target="_blank" >trends.stackoverflow.co</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=Tdt4Xa5sCik" target="_blank" >youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/481/python-opinions-and-zeitgeist-with-hynek" target="_blank" >talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

October 17, 2024 08:00 AM UTC

Julien Tayon

3D ploter in python-tk with matplotlib.

Wishing to prove my assertion wrong on python-tk that piping python directly into tk/tcl interpreter is simple I tried to contradict myself by making a full GUI in matplotlib. Because, if you are not aware : matplotlib supports multi-target (Wx, Qt, gtk, tk, html) multi-platform widgets (Button, checkbox, text entry and so much more).

The challenge seemed pretty easy, to assemble an easy demo with simple example :

Thus, the added value resided in letting people fill in the min/max/step information for the relevant dimensions.

Without the colorbar I would have just been slightly annoyed by the slowness of the reaction of matplotlib as a GUI, but the colorbar posed a new challenge because it would either stack for each drawing or make plt.clf/ax.cla erase too much (see this great resource on when to use cla/clf in matplotlib).

So ... I tried python-tk with matplotlib knowing all too well that you can embed natively matplotlib in tkinter interface.

And since it was working I kept it.

Here is a screenshot of the interface :
WARNING this code should not be let in inventive hands (such as bored teenagers) because there is an evil eval; it requires to be in care of consenting adults.

Some highlights of the code : This said, we have a pretty compact code of 128 lines of code that is pretty more reactive than using matplolib's widget for a 3D ploter.

October 17, 2024 07:48 AM UTC

Spyder IDE

Officially announcing the release of Spyder 6!

After two years in development and more than 2600 commits from over two dozen authors around the world, Spyder 6.0.0 had its stable release on September 3, 2024! Now that 6.0.1 is out and the dust has settled, we'd like to formally announce the release here, thank those who've contributed to it, and introduce a series of posts highlighting its major new features and improvements that haven't already been showcased here.

October 17, 2024 12:00 AM UTC

October 16, 2024

Django Weblog

Announcing weekly DSF office hours

For the last year, Thibaud Colas and I have had a weekly DSF co-working session — we get on a video call and spend an hour quietly working together on DSF things. It's worked well to help us carve out time to work on DSF initiatives, so we'd like to expand into an open-to-everyone weekly "office hours" format.

These will be Wednesdays at 6PM UTC (convert to other time zones). (Yes, that means the first one will be in just about 4 hours, short notice I know, so maybe mark it down for next week.)

All you need to do is bring something DSF-related to work on. This is intentionally broad, as long as it's vaguly DSF-related you're welcome to come. It's not a general-purpose Django coding session (you're welcome to be writing code but it should be related the DSF, e.g. working on djangoproject.com or something.)

This week and next, we'll probably be focusing on nominations for the DSF Board -- nominations close October 25th.

For now, we're deliberately not publishing the video call information publicly — we're a bit worried about spammers and scammers. So if you want to join, you'll need to contact the board, or someone on the board, to get the info. You can use the DSF contact form, and anyone's welcome to contact me directly: — email jacob@djangoproject.com, Signal jacobian.01, or @jacob@jacobian.org on Mastodon.

(Yes, this introduces some friction which is at odds with the "everyone's welcome" ethos. If/when we figure out a better way to moderate these calls, we'll change this.)

I look forward to seeing you there!

October 16, 2024 02:27 PM UTC

Python Insider

Python 3.14.0 alpha 1 is now available

It's now time for a new alpha of a new version of Python!

https://www.python.org/downloads/release/python-3140a1/

This is an early developer preview of Python 3.14

Major new features of the 3.14 series, compared to 3.13

Python 3.14 is still in development. This release, 3.14.0a1 is the first of seven planned alpha releases.

Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process.

During the alpha phase, features may be added up until the start of the beta phase (2025-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2025-07-22). Please keep in mind that this is a preview release and its use is not recommended for production environments.

Many new features for Python 3.14 are still being planned and written. Among the new major new features and changes so far:

The next pre-release of Python 3.14 will be 3.14.0a2, currently scheduled for 2024-11-19.

More resources

Enjoy the new release

Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation.

Regards from a grey yet colourful Helsinki,

Your release team,
Hugo van Kemenade
Ned Deily
Steve Dower
Ɓukasz Langa

October 16, 2024 02:17 PM UTC

Mike Driscoll

SSH Scripting with Fabric and Python

Reading and writing files is a basic task that most software applications need to do. You will also find that you sometimes need to read and write files to a remote machine or perhaps run commands on a remote machine as well. Python is well-suited for this type of activity using tools such as Paramiko. However, in this tutorial, you will spend some time learning how to use a different package called Fabric.

Fabric is a high-level Python package designed especially to execute shell commands remotely over SSH and then yielding useful Python objects in return. This article will focus on the latest version of Fabric, which is 3.2.2 at the time of writing.

Getting Fabric

Fabric is a third party package. That means you need to install Fabric to be able to use it. Fortunately, you can use Python’s pip tool to do so.

Open up a terminal application and run the following command:

python -m pip install fabric

If you don’t want to clutter up your main Python environment, then you should use a Python virtual environment. You can learn more about those in An Intro to Python Virtual Environments.

Once you are finished installing Fabric, you can move on to learning how to use it!

Connecting to the Server with Fabric

The Fabric documentation uses the following as a super simple example of running a command on a remote machine using SSH:

from fabric import Connection
result = Connection('web1.example.com').run('uname -s', hide=True)

You only need two lines of code to start running commands on a remote machine. But what if the remote machine requires credntials?

In that case, you need to create a Config object and update your instantiation of Connection like this:

from fabric import Connection, Config

config = Config(overrides={"sudo": {"password": "MyAmazingPassword123"}})
conn = Connection("mike@10.10.166.128:22", connect_kwargs={"password": "MyAmazingPassword123!"}, config=config)

If you have a machine that uses an SSH key pair, you can use this alternate connect_kwargs dictionary:

connect_kwargs={
        "key_filename": "/home/myuser/.ssh/private.key",
    }

Then simply update the call to Connection and you’re good to go.

Running Commands with Fabric

Now that you have the knowledge needed to connect to the remote machine, you probably want to start running more complex commands. Here is an example of running a ping command:

from fabric import Connection, Config

config = Config(overrides={"sudo": {"password": "MyAmazingPassword123"}}) 
conn = Connection("mike@10.10.166.128:22", connect_kwargs={"password": "MyAmazingPassword123!"}, config=config)

conn.run("ping -c 2 www.google.com")

What if you want to be able to use the super user (i.e. root) when running a command? Fabric makes that easy by using the sudo() method:

from fabric import Connection, Config

config = Config(overrides={"sudo": {"password": "MyAmazingPassword123"}}) 
conn = Connection("mike@10.10.166.128:22", connect_kwargs={"password": "MyAmazingPassword123!"}, config=config)

conn.sudo("systemctl restart nfs-kernel-server")

Transferring Files with Fabric

If you want to download a file from a remote machine, Fabric makes this rudimentary task even easier. Here’s how to do it:

from fabric import Connection, Config 

config = Config(overrides={"sudo": {"password": "MyAmazingPassword123"}})  
conn = Connection("mike@10.10.166.128:22", connect_kwargs={"password": "MyAmazingPassword123!"}, config=config)

conn.get("remote_file_path", "local_file_path")

Note that all you need to do is call get() while specifying the location of the remote file that you want for the first argument and the local path for where you want to download the file as the second argument.

Sending a file to the remote server is done using the put() method:

from fabric import Connection, Config 

config = Config(overrides={"sudo": {"password": "MyAmazingPassword123"}})  
conn = Connection("mike@10.10.166.128:22", connect_kwargs={"password": "MyAmazingPassword123!"}, config=config)

conn.put("local_file_path", "remote_file_path")

You reverse the arguments for put() versus get(). The local path is passed in first, followed by the remote location that you want to upload to.

But what if you want to upload to a restricted area of the file system on the remote machine? You know, like to the etc folder or any of the other root-owned folders?

Fabric doesn’t have a built-in way to do that. Instead you use a two-step process:

Here’s an example:

from fabric import Connection, Config 

config = Config(overrides={"sudo": {"password": "MyAmazingPassword123"}})  
conn = Connection("mike@10.10.166.128:22", connect_kwargs={"password": "MyAmazingPassword123!"}, config=config) 

# Send the file to a user directory
conn.put("local_file_path", "remote_file_path")

# Use sudo to move that file to a root location
conn.sudo("mv remote_file_path root_location_path")

Wrapping Up

Fabric is a great tool that greatly simplifies running SSH commands on remote computers. If you know how to use common Linux commands or know Python well, you can do lots of different things. For example, you could even upload a Python script to the remote server, run it, and then remove the file. At that point, you could do just about anything that you needed to.

Give Fabric a try and see what you can do!

The post SSH Scripting with Fabric and Python appeared first on Mouse Vs Python.

October 16, 2024 02:08 PM UTC

Real Python

Structural Pattern Matching in Python

Structural pattern matching is a powerful control flow construct invented decades ago that’s traditionally used by compiled languages, especially within the functional programming paradigm.

Most mainstream programming languages have since adopted some form of pattern matching, which offers concise and readable syntax while promoting a declarative code style. Although Python was late to join the party, it introduced structural pattern matching in the 3.10 release.

In this tutorial, you’ll:

  • Master the syntax of the match statement and case clauses
  • Explore various types of patterns supported by Python
  • Learn about guards, unions, aliases, and name binding
  • Extract values from deeply nested hierarchical data structures
  • Customize pattern matching for user-defined classes
  • Identify and avoid common pitfalls in Python’s pattern matching

To get the most out of this tutorial, you should have a basic understanding of conditional statements, loops, functions, and classes in Python. Additionally, familiarity with Python’s built-in data structures, such as tuples, lists, and dictionaries, will be beneficial.

Get Your Free Code: Click here to download the free sample code that shows you how to use structural pattern matching in Python.

Take the Quiz: Test your knowledge with our interactive “Structural Pattern Matching” quiz. You’ll receive a score upon completion to help you track your learning progress:

Structural Pattern Matching in Python

Interactive Quiz

Structural Pattern Matching

In this quiz, you'll test your understanding of structural pattern matching in Python. This powerful control flow construct, introduced in Python 3.10, offers concise and readable syntax while promoting a declarative code style.

Getting to Know Structural Pattern Matching

Before taking advantage of structural pattern matching in your code, make sure that you’re running Python 3.10 or later, as you won’t be able to use it in earlier Python versions. Note that although the name structural pattern matching is often shortened to just pattern matching, the qualifier structural is crucial to understanding the use cases for this feature. In this section, you’ll get a high-level overview of structural pattern matching.

What Is Pattern Matching?

You can think of pattern matching as a form of syntactic sugar built on top of existing language constructs, including conditional statements and tuple unpacking. While you can absolutely live without pattern matching, it gives you new superpowers, making this feature more convenient than the conventional syntax in some situations.

Pattern matching often leads to more elegant, concise, and readable code written in a declarative style. To get a taste of it, take a quick look at the following example without trying to fully understand how it works just yet:

Python 
import json

def log(event):
    match json.loads(event):
        case {"keyboard": {"key": {"code": code}}}:
            print(f"Key pressed: {code}")
        case {"mouse": {"cursor": {"screen": [x, y]}}}:
            print(f"Mouse cursor: {x=}, {y=}")
        case _:
            print("Unknown event type")
Copied!

The match statement takes a subject, which can be any valid Python expression, such as a string literal or a function call, and compares the resulting value to one or more patterns listed in the case clauses. The first pattern that matches the given subject will trigger the corresponding case block to run. You’ll learn more about the match statement and case clauses later in this tutorial.

At first glance, the syntax of structural pattern matching in Python looks a bit like the switch statement found in the C-family programming languages if you squint your eyes:

C 
void log_event(enum Event event) {
    switch (event) {
        case KEYBOARD:
            printf("Keyboard event\n");
            break;
        case MOUSE:
            printf("Mouse event\n");
            break;
        default:
            printf("Unknown event\n");
    }
}
Copied!

This resemblance is deceptive, though. The classic switch statement controls the execution flow based on the exact value stored in a variable. It effectively works as a chained sequence of mutually exclusive if..elif... equality comparisons, but with a more succinct and readable syntax.

Although you can use pattern matching this way, you’d be missing out on its true power and flexibility. Structural pattern matching was designed to go beyond value comparisons. In particular, it combines conditional statements or branching based on a logical predicate with destructuring or object deconstruction, which is the inverse of object construction. You’ll see examples of destructuring in the next section.

Note: Because pattern matching does two things at once, the Python interpreter can take advantage of this to optimize the underlying bytecode with specialized opcodes, making the code run slightly faster.

The brief code snippet above merely scratches the surface of what you can achieve with pattern matching, but it already shows you its expressiveness, especially when you compare it with the traditional if...elif... statements and isinstance() checks. Here’s one of the many ways you can implement the equivalent logic using standard Python:

Python 
import json

def log(event):
    parsed_event = json.loads(event)
    if (
        "keyboard" in parsed_event and
        "key" in parsed_event["keyboard"] and
        "code" in parsed_event["keyboard"]["key"]
    ):
        code = parsed_event["keyboard"]["key"]["code"]
        print(f"Key pressed: {code}")
    elif (
        "mouse" in parsed_event and
        "cursor" in parsed_event["mouse"] and
        "screen" in parsed_event["mouse"]["cursor"]
    ):
        screen = parsed_event["mouse"]["cursor"]["screen"]
        if isinstance(screen, list) and len(screen) == 2:
            x, y = screen
            print(f"Mouse cursor: x={x}, y={y}")
        else:
            print("Unknown event type")
    else:
        print("Unknown event type")
Copied!

This code is functionally identical to the previous version but is longer and has more indentation levels than before. Additionally, it looks more verbose and imperative in style, describing not only what to do but also how to perform the individual steps. Granted, you could try making it slightly shorter by using the Walrus operator and following the EAFP principle without explicit checks, but it’d remain somewhat convoluted.

It’s worth noting that structural pattern matching first emerged in compiled functional languages with static typing. The attempt to implement it in Python, which is a dynamic language, presented completely new and unique challenges. You can read more about them in the paper entitled Dynamic Pattern Matching with Python, which was co-authored by Guido van Rossum and published in the proceedings of the Dynamic Languages Symposium in 2020.

Now that you’ve seen the most basic form of pattern matching in Python, it’s time to unravel the meaning of a structural pattern.

What Is a Structural Pattern?

Read the full article at https://realpython.com/structural-pattern-matching/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 16, 2024 02:00 PM UTC

October 15, 2024

PyCoder’s Weekly

Issue #651 (Oct. 15, 2024)

#651 – OCTOBER 15, 2024
View in Browser »

The PyCoder’s Weekly Logo

Exploring the New Features of Python 3.13

Python 3.13 is here! Our regular guests, Geir Arne Hjelle and Christopher Trudeau, return to discuss the new version. This year, Geir Arne coordinated a series of preview articles with members of the Real Python team and a showcase tutorial, “Python 3.13: Cool New Features for You to Try.” Christopher’s video course “What’s New in Python 3.13” covers the topics from the article and shows the new features in action.
REAL PYTHON podcast

HPy: A Better C API for Python

The HPy project is a wrapper to the Python C-API meant to make it easier to integrate with Python code. It allows for universal binaries and has a debug mode. Associated HN Discussion
HPYPROJECT.ORG

Posit Connect: Share the Work You Make With Streamlit, FastAPI, & Other FOSS Frameworks

alt

People use Posit Connect to publish, host, & manage interactive apps, dashboards, Python models, APIs, & much more. It provides a centralized, self-service place to share the products of data science teams. Increase the impact of your work by making it easier for others to integrate with your work →
POSIT sponsor

PyCon Africa 2024: A Summary

This was Vuyisile’s first visit to PyCon Africa and so he posted a summary of the event.
VUYISILE NDLOVU

PEP 760: No More Bare Excepts (Withdrawn)

PYTHON.ORG

PEP 735: Dependency Groups in pyproject.toml (Accepted)

PYTHON.ORG

Django Bugfix Release Issued: 5.1.2

DJANGO SOFTWARE FOUNDATION

Python Developers Survey 2024

PSF

Quiz: When to Use a List Comprehension in Python

REAL PYTHON

Quiz: Modern Python String Formatting Tools

REAL PYTHON

Articles & Tutorials

__init__.py Files Are Optional. You Should Still Use Them

If you’ve ever googled the question “Why do Python packages have empty __init__.py files?”, you could get the idea that Python packages wouldn’t work without them. This is a common misconception—they’ve been optional since Python 3.3! Why then, do most Python projects still have them?
BOVENBERG.NET ‱ Shared by Arie Bovenberg

Build a Contact Book App With Python, Textual, and SQLite

In this tutorial, you’ll be guided step by step through the process of building a basic contact book application. You’ll use Python and Textual to build the application’s text-based user interface (TUI), and then use SQLite to manage the database.
REAL PYTHON

Save 100+ Engineering Hours With the Highest Performing GPU Clusters on the Market

alt

FluidStack provides GPU clusters for LLM training & inference for the top AI labs including Poolside and CharacterAI. Clusters are built on the latest Nvidia GPUs (A100s, H100s, H200s, GB200s) and are deployed on fully managed Kubernetes/Slurm with 24/7 support, 15 min response time and 99% uptime →
FLUIDSTACK sponsor

PEP 777: How to Re-Invent the Wheel

“The current wheel 1.0 specification was written over a decade ago, and has been extremely robust to changes in the Python packaging ecosystem… this PEP prescribes compatibility requirements on future wheel revisions.”
PYTHON.ORG

PEP 758: Allow except and except* Expressions Without Parentheses

“This PEP proposes to allow unparenthesized except and except* blocks in Python’s exception handling syntax. Currently, when catching multiple exceptions, parentheses are required around the exception types.”
PYTHON.ORG

PEP 761: Deprecating PGP Signatures for CPython Artifacts

Since Python 3.11.0, CPython has provided two verifiable digital signatures for all CPython artifacts: PGP and Sigstore. This PEP proposes moving to Sigstore as the only way of signing artifacts.
PYTHON.ORG

In the Making of Python Fitter and Faster

This post details how Python’s recent performance improvements work under the hood. It covers changes to the interpreter, better memory management, and the newly experimental JIT compiler.
SUMER CIP

The Ultimate Guide to Error Handling in Python

This detailed post covers the variety of ways to deal with errors in your code, why you might choose between the approaches, and what all this theory actually means in the real world.
MIGUEL GRINBERG

TypedDicts Are Better Than You Think

TypedDict was introduced in PEP-589 which landed in Python 3.8. The primary use case was to create type annotations for dictionaries. This post explains why you should use them.
CHANGS.CO.UK

Narrow State of a Django Model Using Python TypeGuard

Bruno came across a problem with type checking for a Django project which led him to use TypeGuard for the first time. This post explains why.
BRUNO ALLA

If We Had $1,000,000


Jacob ponders what the Django Software Foundation would look like if they had 4x their current budget.
JACOB KAPLAN-MOSS

Projects & Code

srgn: Grep-Like Tool That Understands Code

GITHUB.COM/ALEXPOVEL

streamable: Stream-Like Manipulation of Iterables

GITHUB.COM/EBONNAL

httpdbg: Debug HTTP(S) Requests in a Python Program

GITHUB.COM/CLE-B

Secure.py: HTTP Security Headers Made Easy

GITHUB.COM/TYPEERROR ‱ Shared by Caleb Kinney

django-admin-tui: Django Admin in the Terminal!

GITHUB.COM/VALBERG

Events

Weekly Real Python Office Hours Q&A (Virtual)

October 16, 2024
REALPYTHON.COM

Python Brasil 2024

October 16 to October 21, 2024
PYTHONBRASIL.ORG.BR

PyCon PanamĂĄ 2024

October 16 to October 19, 2024
PYCON.PA

Swiss Python Summit 2024

October 17 to October 19, 2024
PYTHON-SUMMIT.CH

PyData Bristol Meetup

October 17, 2024
MEETUP.COM

PyLadies Dublin

October 17, 2024
PYLADIES.COM

PyCon APAC 2024

October 25 to October 27, 2024
PYCON.ID

PyCon Korea 2024

October 25 to October 28, 2024
PYCON.KR

PythonHo Conference 2024

October 26 to October 28, 2024
PYTHONHO.COM

Happy Pythoning!
This was PyCoder’s Weekly Issue #651.
View in Browser »

alt

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

October 15, 2024 07:30 PM UTC

Real Python

Using Type Hints for Multiple Return Types in Python

In Python, type hinting is an optional yet useful feature for making your code easier to read, reason about, and debug. With type hints, you let other developers know the expected data types for variables, function arguments, and return values. As you write code for applications that require greater flexibility, you may need to specify multiple return types to make your code more robust and adaptable to different situations.

You’ll encounter different use cases where you may want to annotate multiple return types within a single function in Python. In other words, the data returned can vary in type. In this video course, you’ll walk through examples of how to specify multiple return types for a function that parses a string from an email address to grab the domain name.

In addition, you’ll see examples of how to specify type hints for callback functions or functions that take another function as input. With these examples, you’ll be ready to express type hints in functional programming.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 15, 2024 02:00 PM UTC