Planet Python

Last update: November 06, 2024 09:43 PM UTC

November 06, 2024

Real Python

How to Reset a pandas DataFrame Index

In this tutorial, you’ll learn how to reset a pandas DataFrame index, the reasons why you might want to do this, and the problems that could occur if you don’t.

Before you start your learning journey, you should familiarize yourself with how to create a pandas DataFrame. Knowing the difference between a DataFrame and a pandas Series will also prove useful to you.

In addition, you may want to use the data analysis tool Jupyter Notebook as you work through the examples in this tutorial. Alternatively, JupyterLab will give you an enhanced notebook experience, but feel free to use any Python environment you wish.

As a starting point, you’ll need some data. To begin with, you’ll use the band_members.csv file included in the downloadable materials that you can access by clicking the link below:

Get Your Code: Click here to download the free sample code you’ll use to learn how to reset a pandas DataFrame index.

The table below describes the data from band_members.csv that you’ll begin with:

Column Name	PyArrow Data Type	Description
`first_name`	`string`	First name of member
`last_name`	`string`	Last name of member
`instrument`	`string`	Main instrument played
`date_of_birth`	`string`	Member’s date of birth

As you’ll see, the data has details of the members of the rock band The Beach Boys. Each row contains information about its various members both past and present.

Note: In case you’ve never heard of The Beach Boys, they’re an American rock band formed in the early 1960s.

Throughout this tutorial, you’ll be using the pandas library to allow you to work with DataFrames, as well as the newer PyArrow library. The PyArrow library provides pandas with its own optimized data types, which are faster and less memory-intensive than the traditional NumPy types that pandas uses by default.

If you’re working at the command line, you can install both pandas and pyarrow using the single command python -m pip install pandas pyarrow. If you’re working in a Jupyter Notebook, you should use !python -m pip install pandas pyarrow. Regardless, you should do this within a virtual environment to avoid clashes with the libraries you use in your global environment.

Once you have the libraries in place, it’s time to read your data into a DataFrame:

Python
      
>>> import pandas as pd

>>> beach_boys = pd.read_csv(
...     "band_members.csv"
... ).convert_dtypes(dtype_backend="pyarrow")
Copied!

First, you used import pandas to make the library available within your code. To construct the DataFrame and read it into the beach_boys variable, you used pandas’ read_csv() function, passing band_members.csv as the file to read. Finally, by passing dtype_backend="pyarrow" to .convert_dtypes() you convert all columns to pyarrow types.

If you want to verify that pyarrow data types are indeed being used, then beach_boys.dtypes will satisfy your curiosity:

Python
      
        
      
    
>>> beach_boys.dtypes
first_name            string[pyarrow]
last_name             string[pyarrow]
instrument            string[pyarrow]
date_of_birth         string[pyarrow]
dtype: object
Copied!

As you can see, each data type contains [pyarrow] in its name.

If you wanted to analyze the date information thoroughly, then you would parse the date_of_birth column to make sure dates are read as a suitable pyarrow date type. This would allow you to analyze by specific days, months or years, and so on, as commonly found in pivot tables.

The date_of_birth column is not analyzed in this tutorial, so the string data type it’s being read as will do. Later on, you’ll get the chance to hone your skills with some exercises. The solutions include the date parsing code if you want to see how it’s done.

Now that the file has been loaded into a DataFrame, you’ll probably want to take a look at it:

Python
      
        
      
    
>>> beach_boys
  first_name last_name instrument date_of_birth
0      Brian    Wilson       Bass   20-Jun-1942
1       Mike      Love  Saxophone   15-Mar-1941
2         Al   Jardine     Guitar   03-Sep-1942
3      Bruce  Johnston       Bass   27-Jun-1942
4       Carl    Wilson     Guitar   21-Dec-1946
5     Dennis    Wilson      Drums   04-Dec-1944
6      David     Marks     Guitar   22-Aug-1948
7      Ricky    Fataar      Drums   05-Sep-1952
8    Blondie   Chaplin     Guitar   07-Jul-1951
Copied!

DataFrames are two-dimensional data structures similar to spreadsheets or database tables. A pandas DataFrame can be considered a set of columns, with each column being a pandas Series. Each column also has a heading, which is the name property of the Series, and each row has a label, which is referred to as an element of its associated index object.

The DataFrame’s index is shown to the left of the DataFrame. It’s not part of the original band_members.csv source file, but is added as part of the DataFrame creation process. It’s this index object you’re learning to reset.

The index of a DataFrame is an additional column of labels that helps you identify rows. When used in combination with column headings, it allows you to access specific data within your DataFrame. The default index labels are a sequence of integers, but you can use strings to make them more meaningful. You can actually use any hashable type for your index, but integers, strings, and timestamps are the most common.

Note: Although indexes are certainly useful in pandas, an alternative to pandas is the new high-performance Polars library, which eliminates them in favor of row numbers. This may come as a surprise, but aside from being used for selecting rows or columns, indexes aren’t often used when analyzing DataFrames. Also, row numbers always remain sequential when rows are added or removed in a Polars DataFrame. This isn’t the case with indexes in pandas.

Read the full article at https://realpython.com/pandas-reset-index/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 06, 2024 02:00 PM UTC

Julien Tayon

The crudest CRUD of them all : the smallest CRUD possible in 150 lines of python

Right now, I am on a never ending quest that requires me to think of building a full fledge MVC controller : an anti-jira tracker that would favours HARD CHECKED facts over wishful thinking.

For this to begin, I am not really motivated in beginning with a full fledged MVC (Model View Controller) à la django because there is a lot of boilerplates and actions to do before a result. But, it has a lot of feature I want, including authentication, authorization and handling security.

For prototypes we normally flavours lightweight framework (à la flask), and CRUD.

CRUD approach is a factorisation of all framework in a single dynamic form that adapts itself to the model to generate HTML forms to input data, tabulate, REST endpoints and search them from the python class declaration and generate the database model. One language to rule them all : PYTHON. You can easily generate even the javascript to handle autocompletion on the generated view from python with enough talent.

But before using a CRUD framework, we need a cruder one, ugly, disgusting but useful for a human before building the REST APIs, writing the class in python, the HTML form, and the controlers.

I call this the crudest CRUD of them all.

Think hard at what you want when prototyping ...

to write no CONTROLLERS ; flask documentation has a very verbose approach to exposing routes and writing them, writing controller for embasing and searching databases is boring
to write the fewer HTML views possible, one and only onle would be great ;
to avoid having to fiddle the many files reflecting separation of concerns : the lesser python files and class you touch the better;
to avoid having to write SQL nor use an ORM (at least a verbose declarative one) ;
show me your code and you can mesmerize and even fool me, however show me your data structure and I'll know everthing I have to know about your application : data structure should be under your nose in a readable fashion in the code;/
to have AT LEAST one end point for inserting and searching so that curl can be used to begin automation and testing, preferably in a factorisable fashion;
only one point of failure is accepted

Once we set these few condition we see whatever we do WE NEED a dynamic http server at the core. Python being the topic here, we are gonna do it in python.

What is the simplest dynamic web server in python ?

The reference implementation of wsgi that is the crudest wsgi server of them all : wsgiref. And you don't need to download it since it's provided in python stdlib.

First thing first, we are gonna had a default view so that we can serve an HTML static page with the list of the minimal HTML we need to interact with data : sets of input and forms.

Here, we stop. And we see that these forms are describing the data model.

Wouldn't it be nice if we could parse the HTML form easily with a tool from the standard library : html.parser and maybe deduce the database model and even more than fields coud add relationship, and well since we are dreaming : what about creating the tables on the fly from the form if they don't exists ?

The encoding of the relationship do require an hijack of convention where when the parser cross a name of the field in the form whatever_id it deduces it is a foreign key to table « whatever », column « id ».
Once this is done, we can parse the html, do some magick to match HTML input types to database types (adapter) and it's almost over. We can even dream of creating the database if it does not exists in a oneliner for sqlite.

We just need to throw away all the frugality of dependencies by the window and spoil our karma of « digital soberty » by adding the almighty sqlalchemy the crudest (but still heavy) ORM when it comes of the field of the introspective features of an ORM to map a database object to a python object in a clear consistent way. With this, just one function is needed in the controller to switch from embasing (POST method) and searching (GET).

Well, if the DOM is passed in the request. So of course I see the critics here :

we can't pass the DOM in the request because the HTML form ignores the DOM
You are not scared of error 415 (request too large) in the get method if you pass the DOM ?

That's where we obviously need two important tools : 1) javascript, 2) limitations.

Since we are human we would also like the form to be readable when served, because, well, human don't read the source and can't see the name attributes of the input. A tad of improving the raw html would be nice. It would also give consistency. It will also diminishes the required size of the formular to send. Here, javascript again is the right anwser. Fine, we serve the static page in the top of the controller. Let's use jquery to make it terse enough. Oh, if we have Javascript, wouldn't il be able to clone the part of the invented model tag inside every form so now we can pass the relevant part of the DOM to the controller ?

I think we have everything to write the crudest CRUD server of them all :D

Happy code reading :


import multipart
from wsgiref.simple_server import make_server
from json import dumps
from sqlalchemy import create_engine, MetaData, Table, Column
from sqlalchemy import Integer, String, Float, Date, DateTime,UnicodeText, ForeignKey
from html.parser import HTMLParser
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session
from sqlalchemy import select
from sqlalchemy import create_engine
from sqlalchemy_utils import database_exists, create_database
from urllib.parse import parse_qsl, urlparse

engine = create_engine("postgresql://jul@192.168.1.32/pdca")
if not database_exists(engine.url):
    create_database(engine.url)

tables = dict()

class HTMLtoData(HTMLParser):
    def __init__(self):
        global engine, tables
        self.cols = []
        self.table = ""
        self.tables= []
        self.engine= engine
        self.meta = MetaData()
        super().__init__()

    def handle_starttag(self, tag, attrs):
        attrs = dict(attrs)
        if tag == "input":
            if attrs.get("name") == "id":
                self.cols += [ Column('id', Integer, primary_key = True), ]
                return
            try:
                if attrs.get("name").endswith("_id"):
                    table,_=attrs.get("name").split("_")
                    self.cols += [ Column(attrs["name"], Integer, ForeignKey(table + ".id")) ]
                    return
            except Exception as e: print(e)
            if attrs["type"] in ("email", "url", "phone", "text"):
                self.cols += [ Column(attrs["name"], UnicodeText ), ]
            if attrs["type"] == "number":
                if attrs["step"] == "any":
                    self.cols+= [ Columns(attrs["name"], Float), ]
                else:
                    self.cols+= [ Column(attrs["name"], Integer), ]
            if attrs["type"] == "date":
                self.cols += [ Column(attrs["name"], Date) ]
            if attrs["type"] == "datetime":
                self.cols += [ Column(attrs["name"], DateTime) ]
            if attrs["type"] == "time":
                self.cols += [ Column(attrs["name"], Time) ]
        if tag== "form":
            self.table = urlparse(attrs["action"]).path[1:]

    def handle_endtag(self, tag):
        if tag=="form":
            self.tables += [ Table(self.table, self.meta, *self.cols), ]
            tables[self.table] = self.tables[-1]
            self.table = ""
            self.cols = []
            with engine.connect() as cnx:
                self.meta.create_all(engine)
                cnx.commit()
html = """
<!doctype html>
<html>
<head>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.7.1/jquery.min.js"></script>
<script>
$(document).ready(function() {
    $("form").each((i,el) => {
        $(el).wrap("<fieldset>"+ el.action + "</fieldset>"  );
        $(el).append("<input type=submit value=insert formmethod=post ><input type=submit value=search formmethod=get />");
    });
    $("input:not([type=hidden],[type=submit])").each((i,el) => {
        $(el).before("<label>" + el.name+ "</label><br/>");
        $(el).after("<br>");
    });
});
</script>
</head>
<body>
    <form action=/user  >
        <input type=number name=id />
        <input type=text name=name />
        <input type=email name=email >
    </form>
    <form action=/event >
        <input type=number name=id />
        <input type=date name=date />
        <input type=text name=text />
        <input type=number name=user_id />
    </form>
</body>
</html>
"""

router = dict({"" : lambda fo: html,})

def simple_app(environ, start_response):
    fo,fi=multipart.parse_form_data(environ)
    fo.update(**{ k: dict(
            name=fi.filename,
            content=fi.file.read().decode('utf-8', 'backslashreplace'),
            content_type=fi.content_type,
        ) for k,v in fi.items()})
    table = route = environ["PATH_INFO"][1:]
    fo.update(**dict(parse_qsl(environ["QUERY_STRING"])))
    start_response('200 OK', [('Content-type', 'text/html; charset=utf-8')])
    try:
        HTMLtoData().feed(html)
    except KeyError: pass
    metadata = MetaData()
    metadata.reflect(bind=engine)
    Base = automap_base(metadata=metadata)
    Base.prepare()
    if route in tables.keys():
        with Session(engine) as session:
            Item = getattr(Base.classes, table)
            if environ.get("REQUEST_METHOD", "GET") == "POST":
                new_item = Item(**{ k:v for k,v in fo.items() if v and not k.startswith("_")})
                session.add(new_item)
                ret=session.commit()
                fo["insert_result"] = new_item.id
            if environ.get("REQUEST_METHOD") == "GET":
                result = []
                for elt in session.execute(
                    select(Item).filter_by(**{ k : v for k,v in fo.items() if v and not k.startswith("_")})).all():
                    result += [{ k.name:getattr(elt[0],k.name) for k  in tables[table].columns}]
                fo["search_result"] = result
    return [ router.get(route,lambda fo:dumps(fo.dict, indent=4, default=str))(fo).encode() ]

print("Crudest CRDU of them all on port 5000...")
make_server('', 5000, simple_app).serve_forever()

November 06, 2024 12:04 PM UTC

Programiz

Python match…case Statement

The match…case statement allows us to execute different actions based on the value of an expression. In this tutorial, you will learn how to use the Python match…case with the help of examples.

November 06, 2024 10:10 AM UTC

November 05, 2024

TestDriven.io

Avoid Counting in Django Pagination

This article looks at how to avoid the count query in Django's paginator.

November 05, 2024 10:28 PM UTC

PyCoder’s Weekly

Issue #654 (Nov. 5, 2024)

#654 – NOVEMBER 5, 2024
View in Browser »

PySheets: Spreadsheets in the Browser Using PyScript

What goes into building a spreadsheet application in Python that runs in the browser? How do you make it launch quickly, and where do you store the cells of data? This week on the show, we speak with Chris Laffra about his project, PySheets, and his book “Communication for Engineers.”
REAL PYTHON podcast

Adding Keyboard Shortcuts to the Python REPL

Python 3.13 included a new version of the REPL which has the ability to define keyboard shortcuts. This article shows you how to create one and warns you about potential hangups.
TREY HUNNER

Tired of Being Paged? Worry Less With Temporal

Say goodbye to managing failures, network outages, flaky endpoints, and long-running processes. Temporal ensures your code never fails. Period. PLUS, you can get started today on Temporal Cloud with $1,000 free credits on us →
TEMPORAL TECHNOLOGIES sponsor

Running a Million Empty Tests

To better understand just where the performance cost of running tests comes from, Anders ran a million empty tests. This post talks about what he did and the final results.
ANDERS HOVMOLLER

Pillow Release 11.0.0

GITHUB.COM/PYTHON-PILLOW

PEP 750: Template Strings (Major Updates)

PSF

PEP 756: Add `PyUnicode_Export()` and `PyUnicode_Import()` C Functions (Withdrawn)

PSF

Discussions

Thinking of Rewriting Our Go / Java API in Python

Best GUI for Local Client App?

Articles & Tutorials

Move to Sigstore Complicates Linux Distros

Currently, CPython signs its artifacts with both PGP and Sigstore. Removing the PGP signature has been proposed, but that has implications: Sigstore is still new enough that many Linux distributions don’t support it yet.
JOE BROCKMEIER

Python’s Magic Methods in Classes

In this video course, you’ll learn what magic methods are in Python, how they work, and how to use them in your custom classes to support powerful features in your object-oriented code.
REAL PYTHON course

[Webinar] How to Build Secure, Ethical, and Scalable AI Operations

As GenAI and LLMs rapidly evolve, the impact of data leaks and unsafe AI outputs makes it critical to secure your AI infrastructure. Learn how MLOps and ML Platform teams can use the newly launched Guardrails Pro to secure AI operations — enabling faster, safer adoption of LLMs at scale →
GUARDRAILS sponsor

Make It Ephemeral: Software Should Decay and Lose Data

In the real world, things decay over time. In the digital world things get kept forever, and sometimes that shouldn’t be so. Designing for deletion is hard.
ARMIN RONACHER

Python 3.13, t-Strings, Dep Groups…

Bite code! does their monthly Python news wrap-up. Check out stories on 3.13, proposed template strings, dependency groups in pyproject.toml, and more.
BITE CODE!

Identifying Products From Images

This project uses computer vision solution to automate doing inventory of products in retail, using YOLOv8 and image embeddings for precise detection.
ALBERT FERRÉ • Shared by Albert Ferré

Write More Pythonic Code With Context Managers

Context managers enable you to create “template” code with initialization and clean up to make the code that uses them easier to read and understand.
JUHA-MATTI SANTALA

Django Girls 10th Birthday!

This post celebrating ten years of Django Girls talks about how it got started, what they’re hoping to do, and how you can get involved.
DJANGO GIRLS

`pytest` Selection Arguments for Failing Tests

This quick TIL post talks about five useful pytest options that let you control what tests to run with respect to failing tests.
RODRIGO GIRÃO SERRÃO

Asyncio `gather()` Return Values

This post shows you how to return values from coroutines that have been concurrently executed using asyncio.gather().
JASON BROWNLEE

PyBay 2024

This list contains the recorded talks from the PyBay 2024 conference.
YOUTUBE video

Projects & Code

wimsey: Data Contract Library

GITHUB.COM/BENRUTTER

libcom: Image Composition Toolbox

GITHUB.COM/BCMI

simplemind: Experimental Client for AI Providers

GITHUB.COM/KENNETHREITZ

PyChrono: Multi-Physics Simulation in Python

CRISTIANOPIZZAMIGLIO.COM • Shared by Cristiano Pizzamiglio

jamesql: In-Memory NoSQL Database in Python

GITHUB.COM/CAPJAMESG

Events

Weekly Real Python Office Hours Q&A (Virtual)

November 6, 2024
REALPYTHON.COM

Canberra Python Meetup

November 7, 2024
MEETUP.COM

Sydney Python User Group (SyPy)

November 7, 2024
SYPY.ORG

DFW Pythoneers 2nd Saturday Teaching Meeting

November 9, 2024
MEETUP.COM

PiterPy Meetup

November 12, 2024
PITERPY.COM

PyCon Sweden 2024

November 14 to November 16, 2024
PYCON.SE

PyCon Hong Kong 2024

November 16 to November 17, 2024
PYCON.HK

PyCon Mini Tokai 2024

November 16 to November 17, 2024
PYCON.JP

PyCon Ireland 2024

November 16 to November 18, 2024
PYTHON.IE

Happy Pythoning!
This was PyCoder’s Weekly Issue #654.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

November 05, 2024 07:30 PM UTC

Real Python

Introduction to Web Scraping With Python

Web scraping is the process of collecting and parsing raw data from the Web, and the Python community has come up with some pretty powerful web scraping tools.

The Internet hosts perhaps the greatest source of information on the planet. Many disciplines, such as data science, business intelligence, and investigative reporting, can benefit enormously from collecting and analyzing data from websites.

In this video course, you’ll learn how to:

Parse website data using string methods and regular expressions
Parse website data using an HTML parser
Interact with forms and other website components

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 05, 2024 02:00 PM UTC

Quiz: Variables in Python: Usage and Best Practices

In this quiz, you’ll test your understanding of Variables in Python: Usage and Best Practices.

By working through this quiz, you’ll revisit how to create and assign values to variables, change a variable’s data type dynamically, use variables to create expressions, counters, accumulators, and Boolean flags, follow best practices for naming variables, and create, access, and use variables in their scopes.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 05, 2024 12:00 PM UTC

Talk Python to Me

#484: From React to a Django+HTMX based stack

Have you heard about HTMX? We've discussed it a time or two on this show. We're back with another episode on HTMX, this time with a real-world success story and lessons learned. We have Sheena O'Connell on to tell us how she moved from a React-Django app to pure Django with HTMX. Episode sponsors <a href='https://talkpython.fm/posit'>Posit</a> <a href='https://talkpython.fm/bluehost'>Bluehost</a> <a href='https://talkpython.fm/training'>Talk Python Courses</a> Links from the show <div>Sheena O'Connell: <a href="https://www.sheenaoc.com?featured_on=talkpython" target="_blank" >sheenaoc.com</a> An HTMX success story essay: <a href="https://www.sheenaoc.com/articles/2024-06-30-htmx?featured_on=talkpython" target="_blank" >sheenaoc.com</a> Sheena's HTMX Workshop: <a href="https://prelude.tech/upcoming_workshops?featured_on=talkpython" target="_blank" >prelude.tech - discount code: talk_python</a> Talk Python's HTMX Courses HTMX + Flask course: <a href="https://training.talkpython.fm/courses/htmx-flask-modern-python-web-apps-hold-the-javascript" target="_blank" >training.talkpython.fm</a> HTMX + Django course: <a href="https://training.talkpython.fm/courses/htmx-django-modern-python-web-apps-hold-the-javascript" target="_blank" >training.talkpython.fm</a> Build An Audio AI App course: <a href="https://training.talkpython.fm/courses/build-an-audio-ai-app-with-python-and-assemblyai" target="_blank" >training.talkpython.fm</a> HTMX: <a href="https://htmx.org?featured_on=talkpython" target="_blank" >htmx.org</a> Playwright: <a href="https://playwright.dev?featured_on=talkpython" target="_blank" >playwright.dev</a> django-template-partials: <a href="https://github.com/carltongibson/django-template-partials?featured_on=talkpython" target="_blank" >github.com</a> Michael's jinja_partials: <a href="https://github.com/mikeckennedy/jinja_partials?featured_on=talkpython" target="_blank" >github.com</a> django-guardian: <a href="https://github.com/django-guardian/django-guardian?featured_on=talkpython" target="_blank" >github.com</a> Talk Python Courses HTMX Example: <a href="https://training.talkpython.fm/courses/all" target="_blank" >training.talkpython.fm/courses/all</a> Alpine.js: <a href="https://alpinejs.dev?featured_on=talkpython" target="_blank" >alpinejs.dev</a> David Guillot SaaS video: <a href="https://www.youtube.com/watch?v=3GObi93tjZI" target="_blank" >youtube.com</a> awesome-htmx: <a href="https://github.com/rajasegar/awesome-htmx?featured_on=talkpython" target="_blank" >github.com</a> Guild of Educators: <a href="https://guildofeducators.org?featured_on=talkpython" target="_blank" >guildofeducators.org</a> The big rewrite song: <a href="https://www.youtube.com/watch?v=xCGu5Z_vaps" target="_blank" >youtube.com</a> Watch this episode on YouTube: <a href="https://www.youtube.com/watch?v=ZTAEkPRNbi4" target="_blank" >youtube.com</a> Episode transcripts: <a href="https://talkpython.fm/episodes/transcript/484/from-react-to-a-django-htmx-based-stack" target="_blank" >talkpython.fm</a> --- Stay in touch with us --- Subscribe to us on YouTube: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a> Follow Talk Python on Mastodon: <a href="https://fosstodon.org/web/@talkpython" target="_blank" >talkpython</a> Follow Michael on Mastodon: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" >mkennedy</a> </div>

November 05, 2024 08:00 AM UTC

Tryton News

Tryton Release 7.4

We are proud to announce the 7.4 release of Tryton .
This release provides many bug fixes, performance improvements and some fine tuning.
You can give it a try on the demo server, use the docker image or download it here.
As usual upgrading from previous series is fully supported.

Here is a list of the most noticeable changes:

Changes for the User

Clients

The Many2Many widget now has a restore button to revert the removal of records before saving.

The CSV export window stays open after the export is done so you can refine your export without having the redo all of the configuration.
It also supports exporting and importing translatable fields with a language per column.
The error messages displayed when there is a problem with the CSV import have been improved to include the row and column number of the value that caused the error.

The management window for the favourites has been removed and replaced by a simple “last favorite first” order.

The focus goes back to the search entry after performing a search/refresh.

You can now close a tab by middle clicking on it (as is common in other software).

Web Client

The left menu and the attachment preview can now be resized so the user can make them the optimal size for their screen.

Accounting

The minimal chart of accounts has been replaced by the a universal chart of accounts which is a good base for IFRS and US GAAP.

It is now possible to copy an accounting move from a closed period. The closed period will be replaced by the current period after accepting the warning.

The payments are now numbered to make it easier to identify them inside the application.
An option has been added to the parties to allow direct debits to be created based on the balance instead of the accounting lines.
We’ve added a button on the Stripe payments and Stripe and Braintree customers to allow an updated to be forced. This helps when fixing missed webhooks.

When a stock move is cancelled, the corresponding stock account move is now cancelled automatically.
But it now no longer possible to cancel a done stock move which has been included in a calculation used for anglo-saxon accounting.

Commission

It is now possible to deactivate an agent so that they are no longer used for future orders.

Company

It is now possible to add a company logo. This is then displayed in the header of generated documents.

Incoterm

A warning is now raised when the incoterm of a shipment is different from the original document (such as the sale or purchase).

Party

We’ve added more identifiers for parties like the United Kingdom Unique Taxpayer Reference, Taiwanese Tax Number, Turkish tax identification number, El Salvador Tax Number, Singapore’s Unique Entity Number, Montenegro Tax Number and Kenya Tax Number.

Product

We’ve added a wizard to manage the replacement of products. Once there is no more stock of the replaced product in any of the warehouses, all the stock on all pending orders are replaced automatically.

A description can now be set for each product image.

There is now a button on the price list form to open the list of lines. This is helpful when the price list has a lot of lines.

Production

It is now possible to cancel a done production. All its stock moves are then cancelled.

The Bill of Materials now have an auto-generated internal code.

Purchase

The wizard to handle exceptions has been improved to clearly display the list of lines to recreate and the list of lines to ignore.

The menu entry Parties associated to Purchases has been removed in favour of the per party reporting.

The purchase amendment now supports amending the quantity of a purchase line using the secondary unit.

Quality

It is now no longer possible to delete non-pending inspections.

Sale

The wizards to handle exceptions have been improved to clearly display the list of lines to recreate and the list of lines to ignore.

The menu entry Parties associated to Sales has been removed in favor of the per party reporting.

A warning is now raised when the user tries to submit a complaint for the same origin as an existing complaint.

The reporting can be grouped per promotion.

From a promotion, it is now possible to list of the sales related to it.
The coupon number of promotion can now be reused once the previous promotion has expired.

The sale amendment now supports amending the quantity of a sale line using the secondary unit.

Stock

It is now possible to cancel a done shipment. When this happens the stock moves of the shipment are cancelled.

The task to reschedule late shipments now includes any shipment that is not yet done.

The supplier shipments no longer have a default planned date.

The customer shipments now have an extra state, Shipped, before the Done state.

The lot trace now shows the inventory as a document.

The package weight and the warehouse are now criteria that can be used when selecting a shipping method.

Changes for the System Administrator

The clients automatically retry 5 times on a 503 Service Unavailable response. They respect the Retry-After value if it is set in the response header. This is useful when performing short maintenance on the server without causing an interruption for the users.

The scheduled tasks now show when they are running and prevent the user from editing them (as they are locked anyway).
We also store their last duration for a month by default. So the administrator can analyze and find slow tasks.

It is now possible to configure a license key for the TinyMCE editor.
Also TinyMCE has been updated to version 7.

It is now possible to configure the command to use to convert a report to a different format. This allows the use of an external service like document-converter.

Accounting

The Accounting Party group has been merged into the *Accounting" group.

We now raise a warning when the user is changing one of the configured credentials used on external services. This is to prevent accidental modification.

Document Incoming

It is now possible to set a maximum size for the content of the document incoming requests.

Inbound Email

It is now possible to set a maximum size for the inbound email requests.

Web Shop

There is now a scheduled task that updates the cache that contains the product data feeds.

Changes for the Developer

Server

The ORM supports SQL Range functions and operators to build exclusion constraints. This allows, for example, the use of non-overlapping constraints using an index.
On PostgreSQL the btree_gist extension may be needed otherwise the ORM will fallback to locking querying the table.
The SQLite backend adds simple SQL constraints to the table schema.

The relational fields with a filter are no longer copied by default. This was a frequent source of bugs as the same relational field without the filter was already copied so it generated duplicates.

We’ve added a sparkline tool to generate textual sparklines. This allows the removal of the pygal dependency.

The activate_modules from testing now accepts a list of setup methods that are run before taking the backup. This speeds up any other tests which restore the backup as they then do not need to run those setup methods.

The backend now has a method to estimate the number of rows in a table. This is faster than counting when we only need an estimate, for example when choosing between a join and a sub-query.

We’ve added a ModelSQL.__setup_indexes__ method that prepares the indexes once the Pool has been loaded.

It is now possible to generate many sequential numbers in a single call. This allows, for example, to number a group of invoices with a single call.

The backend now uses JSONB by default for MultiSelection fields. It was already supported, but the database needed to be altered to activate the feature.

You can now define the cardinality (low, normal or high) for the index usage. This allows the backend to choose an optimal type of index to create.

We now have tools that apply the typing to columns of an SQLite query. This is needed because SQLite doesn’t do a good job of supporting CAST.

The RPC responses are now compressed if their size if large enough and the client accepts it.

The ModelView._changed_values and ModelStorage._save_values are now methods instead of properties. This makes it is easier to debug errors because AttributeError exceptions are no longer hidden.

The scheduled task runner now uses a pool of processes for better parallelism and management. Only the running task is now locked.

We’ve added an environment variable TEST_NETWORK so we can avoid running tests that require network access.

There is now a command line option for exporting translations and storing them as a po file in the corresponding module.
Tryton sets the python-format flag in the po file for the translations containing python formats. This allows Weblate (our translation service) to check if the translations keep the right placeholders.

Accounting

The payment amounts are now cached on the account move line to improve the performance when searching for lines to pay.
The payment amounts now have to be greater or equal to zero.

Purchase

Only purchase lines of type line can be used as an origin for a stock move.

Sale

Only sales lines of type line can be used as an origin for a stock move.

The fields from the Sale Shipment Cost Module are now all prefixed with sale_.

Stock

Cancelled moves are no longer included in the shipment and package measurements.

2 posts - 1 participant

Read full topic

November 05, 2024 07:00 AM UTC

Django Weblog

Django bugfix release issued: 5.1.3

Today we've issued the 5.1.3 bugfix release.

The release package and checksums are available from our downloads page, as well as from the Python Package Index. The PGP key ID used for this release is Mariusz Felisiak: 2EF56372BA48CD1B.

November 05, 2024 06:04 AM UTC

November 04, 2024

James Bennett

Three Django wishes

’Tis the season when people are posting their “Django wishlists”, for specific technical or organizational or community initiatives they’d like to see undertaken. Here are a few examples from around the Django community:

So, in the spirit of the season, here is my own list, which I’ve narrowed down to three wishes (in the tradition of many stories about wishes), consisting of one organizational item and two technical ones.

Pass the torch

This one requires a bit of background, so please bear with me.

The Django Software Foundation — usually just abbreviated “DSF” — is the nonprofit organization which officially “owns” Django. It’s the legal holder of all the intellectual property, including both the copyright to the original Django codebase (generously donated by the Lawrence Journal-World, where it was first developed) and the trademarks, such as the registered trademark on the name “Django” itself. The DSF does a lot (and could do more, with a bigger budget) to support the Django community, and offers financial support to the development of Django itself, but does not directly develop Django, or oversee Django’s development or technical direction.

Originally, that job went to Django co-creators Adrian Holovaty and Jacob Kaplan-Moss. They granted commit permissions to a growing team of collaborators, but remained the technical leaders of the Django project until 2014, when they stepped aside and a new body, called the Technical Board, was introduced to replace them. The Technical Board was elected by the Django committers — by this point usually referred to as “Django Core” — and although the committers continued to have broad authority to make additions or changes to Django’s codebase, the Technical Board became the ultimate decision-maker for things that needed a tie-breaking vote, or that were too large for a single committer to do solo (usually via Django Enhancement Proposals, or DEPs, modeled on the processes of many other open-source projects, including Python’s “PEPs”).

One thing the DSF has done is use some of its funds on the Django Fellowship program, which pays contractors (the “Django Fellows”) to carry out tasks like ticket triage, pull-request review, etc. which would otherwise rely on volunteer labor (with all the problems that involves).

But the system of “Django Core” committers and Technical Board did not work out especially well. Many of the committers were either intermittently active or just completely inactive, new committers were added rarely if ever, and it was unclear what sort of path there was (or even if there was a path) for a motivated contributor to work their way toward committer status. About the only thing that did work well was the Fellowship program, which largely was what kept Django running as a software project toward the end of that era.

This caused a lot of debates focused on the theme of what to do about “Django Core” and how to reform the project and get it back on a healthy footing. The end result of that was a Django Enhancement Proposal numbered as DEP 10, which I spent most of 2018 and 2019 working on. I wrote an explanation at the time, and I’ll just link it here and mention that DEP 10 (which passed in early 2020) kept the Technical Board as a tie-breaking and oversight body, and introduced two other main roles — “Mergers” and “Releasers” — which have mostly but not exclusively been filled by the Django Fellows. The first DEP 10 Technical Board drafted and passed another DEP, DEP 12, renaming themselves to “Steering Council” (similar to Python’s technical governing body, but a name I’ve never liked because the Django version doesn’t meaningfully “steer” Django) and making a few tweaks.

So, that brings us to the present day. Where, sadly, the DEP 10/12 era is looking like as much of a failure as the preceding “Django Core” + committer-elected Technical Board era. The DEP 10 Technical Boards/Steering Councils have been dysfunctional at best, and there’s been no influx of new people from outside the former “Django Core”. A stark example: I ran for the the Steering Council last year to try to work on fixing some of this, but the Steering Council election attracted only four total candidates for five seats, all of them former “Django Core” members.

Recently there was a lot of discussion on the DSF members’ forum about what to do with the Steering Council, and a few attempts to take action which failed in frustrating ways. The end result was the resignation of two Steering Council members, which brought the group below quorum and has automatically triggered an election (though one that will run under the existing DEP 10/12 rules, since triggering an election locks the eligibility and election rules against changes).

I believe the ongoing inability to develop stable technical governance and meaningful turnover of technical leadership is the single greatest threat to Django’s continued viability as a project. This is an unfortunate vindication of what I said six years ago in that blog post about developing DEP 10:

Django’s at risk of being badly off in the future; for some time now, the project has not managed to bring in new committers at a sufficient rate to replace those who’ve become less active or even entirely inactive, and that’s not sustainable for much longer.

The good news is there’s a new generation of contributors who I believe are more than ready to take up the technical leadership of Django, and even a structured program — not run by former “Django Core”! — for recruiting and mentoring new contributors on an ongoing basis and helping them build familiarity with working on and contributing to Django. The bad news is there’s a huge obstacle in their way: all of us old-time “Django Core” folks who keep occupying all the official leadership positions. Just recruiting people to run against such long-time well-known names in the project is difficult, and actually winning against us probably close to impossible.

So the biggest thing I’d like for Django, right now, is for the entire former “Django Core” group — myself included! — to simply get out of the way. I thought I could come back last year and help fix things after stepping down post-DEP-10, but doing so was a mistake and only prolonged the problem. I will not be running in the upcoming Steering Council election and I beg my “Django Core” colleagues to all do likewise. There are qualified, motivated folks out there who should be given their chance to step up and run things, and we should collectively place Django into their capable hands. Then they can sort out the rest of the technical governance however they see fit.

And honestly, I’ve been in and out of just about every formal role the Django project has for (checks calendar) seventeen years now. It’s time. It’s time for me, and the rest of the old guard, to give way to new folks before we do serious harm to Django by continuing to hold on to leadership roles.

Give Django a hint

Python 3.0 introduced the ability to add “annotations” to function and method declarations, and though it didn’t specify what they were to be used for, people almost immediately started developing ways to specify static type information via annotations, which came to be known as “type hints”. Python 3.5 formalized this and introduced the typing module in the standard library with tools to make the type-hint use case easier, and Python 3.6 introduced the ability to annotate other names, including standalone variables and class attributes.

Django has a complicated history with this feature of modern Python. There’ve been multiple efforts to add type annotations directly in Django’s own code, there’s a third-party package which provides annotations as an add-on, a proposed DEP never went anywhere because the Technical Board at the time was against it, and it’s just been stuck as a frequently-requested feature ever since.

Let me be absolutely clear: I don’t have any issue with statically-typed programming languages as a concept. I’ve used both statically- and dynamically-typed languages and liked and disliked examples of each. If I weren’t writing Python, personally I probably would be writing C# (statically-typed). But I also have absolutely no interest in static type checking for Python as a feature or a use case.

What I do have an interest in is all the other use cases type hints enable. There’s a whole booming ecosystem of modern Python tools out there now which use type hints to enable all sorts of interesting runtime behavior. Pydantic and msgspec do runtime derivation of validation and serialization/deserialization behavior from type hints. FastAPI and Litestar are web frameworks which use type hints to drive input/output schemas, dependency injection and more. SQLAlchemy as of version 2.0 can use type hints to drive ORM class definitions.

I am very interested in those sorts of things, and right now they’re not available from vanilla Django because Django doesn’t do type hints (you can use a third-party package to turn Django into something resembling one of the newer type-hint-driven frameworks, but it’s an extra package and a whole new way of doing things that doesn’t “feel like Django”).

Compare, for example, this Django ORM model:

from django.db import models

class Person(models.Model):
    name = models.CharField()
    date_of_birth = models.DateField()

With its modern SQLAlchemy equivalent:

from datetime import date   
from sqlalchemy.orm import DeclarativeBase, Mapped

Base = DeclarativeBase()

class Person(Base):
    __tablename__ = "person"
    name: Mapped[str]
    date_of_birth: Mapped[date]

You can use SQLAlchemy’s mapped_column() function to be more verbose and specify a bunch more information, but for a basic column you don’t have to. Just write a type hint and it does the right thing.

I think type hint support in Django has the potential to unlock a huge variety of useful new features and conveniences, and the lack of it is causing Django to fall well behind the current state of the art in Python web development. So if I could somehow wave a magic wand and get any single technical change instantly made to Django, type hints would be it.

More generic Django

Django includes a feature known as “generic views” (keep in mind that Django doesn’t strictly follow regular MVC terminology, and so a Django “view” is what most pure MVC implementations would call the “controller”), which are reusable implementations of common operations like “CRUD” (create, retrieve, update, delete operations — including both individual-result and list-of-results), date-based archives of data, etc.

And basically everybody agrees Django’s generic views are too complicated. There’s a giant complex inheritance tree of classes involved, with a huge mess of different attributes and methods you can set or override to affect behavior depending on exactly which set of classes you’re inheriting from, creating a steep learning curve and requiring even experienced developers to spend a lot of time with both official and unofficial documentation (ccbv.co.uk is the usual reference people are directed to).

There’s a reason for this complexity: originally, Django’s generic views were functions, not classes, and you customized their behavior by passing arguments to them. The class-based generic views were introduced in Django 1.3 (released in 2011), and for compatibility and ease of migration at the time, were implemented in a way which precisely mirrored the functionality of the function-based views. Which means that for every thing you could do via an argument to the function-based views, there is a mixin class, method, or attribute on the class-based ones corresponding to it.

This made some sense at the time, because it was a big migration to ask people to go through. It makes much less sense now, over 13 years later, when the complexity of Django’s hierarchy of class-based views mostly just scares people and makes them not want to use what is otherwise a pretty useful feature: class-based views are a huge reduction in repetitive/boilerplate code when you know how to use them (for example, see the views used by this site for date-based browsing of entries and detail/list views of entries by category — that really is all the code needed to provide all the backend logic).

At this point the overcomplexity of Django’s generic views is basically a meme in the community, and is one of the things I see most often cited by new Django users as making their experience difficult. So if I were going to be given the magic wand a second time and allowed to make another instant technical change, it’d be to finally deprecate the complicated generic-view class hierarchy and replace it with a ground-up rewrite aimed at providing a clear, powerful API rather than maintaining compatibility with a set of older functions that were deprecated nearly a decade and a half ago.

What do you wish for?

Of course, there’s a lot more that could be done to or for Django besides the three items I’ve outlined here. I’d encourage anyone who uses Django to think about what they’d like to see, to post about it, and, ideally, to get involved with Django’s development. That’s not just a way to get bits of your own wishlist implemented; it’s also the way to make sure Django continues to be around for people to have wishes about, and I hope that continues for many years to come.

November 04, 2024 11:21 PM UTC

Python Engineering at Microsoft

Announcing GitHub Copilot in Data Wrangler

AI did not write this blog post, but it will make your exploratory data analysis with Data Wrangler better!

Today, we’re excited to introduce our first step of integrating the power of Copilot into Data Wrangler.

With this first integration of Copilot with Data Wrangler, you’ll be able to:

Use natural language to clean and transform your data
Get help with fixing errors in your data transformation code

An example of using Copilot in Data Wrangler to filter for listings that allow dogs/cats

A common limitation of using AI tools for exploratory data analysis tasks today is the lack of data context provided to the AI. Responses are typically more generalized and not tailored to the specific task or data at hand. In addition, there’s always the manual and tedious task of verifying the correctness of the generated code.

What makes Copilot with Data Wrangler different is twofold. First, this integration allows you to choose to provide Copilot with your data context, enabling it to generate more relevant and specific code for the exact dataset you have open. Second, you get to preview the exact behavior of the code on your dataset with the Data Wrangler interface to visually validate Copilot’s response, along with all the benefits that the Data Wrangler tool provides.

Data transformations

With Copilot in Data Wrangler, you can ask it to perform ambiguous, open-ended transformations or a specific task you have in mind. Below we’ve included three examples of the many possibilities you can achieve with Copilot in Data Wrangler:

Formatting a datetime column

Removing any column(s) with over 40% missing values

Fixing an error in a data transformation

Getting started today

To use Copilot with Data Wrangler, you will need the following 3 prerequisites.

You must have the Data Wrangler extension for VS Code installed.
You must have the GitHub Copilot extension for VS Code installed.
You must have an active subscription for GitHub Copilot in your personal account, or you need to be assigned a seat by your organization. Sign up for a GitHub Copilot free trial in your personal account.

Follow these steps to Set up GitHub Copilot in VS Code.

Once the prerequisites are met, you will see the Copilot interface within Data Wrangler by default (customizable in the Data Wrangler settings) when you are in Editing Mode. You can then either select the input box or use the default Copilot keyboard shortcut of CMD/CTRL + I.

Responsible AI

AI is not perfect (neither are we!) and it will improve over time. Microsoft and GitHub Copilot follow Responsible AI principles and employ controls to ensure that your experience with the service is appropriate, pleasant, and useful. We understand there is hesitation and concern surrounding the rapid expansion of AI’s capabilities, and fully respect those who don’t want or can’t use Copilot.

If you have any feedback around the Copilot experience in Data Wrangler, please file an issue in our Data Wrangler public GitHub repository here.

Next Steps

We are just getting started. This is the first experience in Data Wrangler that we are enhancing with Copilot. Stay tuned for more AI-powered experiences in Data Wrangler to help with your data analysis needs soon!

The post Announcing GitHub Copilot in Data Wrangler appeared first on Python.

November 04, 2024 07:02 PM UTC

Real Python

Variables in Python: Usage and Best Practices

In Python, variables are symbolic names that refer to objects or values stored in your computer’s memory. They allow you to assign descriptive names to data, making it easier to manipulate and reuse values throughout your code.

Understanding variables is key for Python developers because variables are essential building blocks for any Python program. Proper use of variables allows you to write clear, readable, and maintainable code.

In this tutorial, you’ll learn how to:

Create and assign values to variables
Change a variable’s data type dynamically
Use variables to create expressions, counters, accumulators, and Boolean flags
Follow best practices for naming variables
Create, access, and use variables in their scopes

To get the most out of this tutorial, you should be familiar with Python’s basic data types and have a general understanding of programming concepts like loops and functions.

Don’t worry if you don’t have all this knowledge yet and you’re just getting started. You won’t need this knowledge to benefit from working through the early sections of this tutorial.

Get Your Code: Click here to download the free sample code that shows you how to use variables in Python.

Take the Quiz: Test your knowledge with our interactive “Variables in Python: Usage and Best Practices” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Variables in Python: Usage and Best Practices

In this quiz, you'll test your understanding of variables in Python. Variables are symbolic names that refer to objects or values stored in your computer's memory, and they're essential building blocks for any Python program.

Getting to Know Variables in Python

In Python, variables are names associated with concrete objects or values stored in your computer’s memory. By associating a variable with a value, you can refer to the value using a descriptive name and reuse it as many times as needed in your code.

Variables behave as if they were the value they refer to. To use variables in your code, you first need to learn how to create them, which is pretty straightforward in Python.

Creating Variables With Assignments

The primary way to create a variable in Python is to assign it a value using the assignment operator and the following syntax:

Python Syntax
      
    
variable_name = value
Copied!

In this syntax, you have the variable’s name on the left, then the assignment (=) operator, followed by the value you want to assign to the variable at hand. The value in this construct can be any Python object, including strings, numbers, lists, dictionaries, or even custom objects.

Note: To learn more about assignments, check out Python’s Assignment Operator: Write Robust Assignments.

Here are a few examples of variables:

Python
      
>>> word = "Python"

>>> number = 42

>>> coefficient = 2.87

>>> fruits = ["apple", "mango", "grape"]

>>> ordinals = {1: "first", 2: "second", 3: "third"}

>>> class SomeCustomClass: pass
>>> instance = SomeCustomClass()
Copied!

In this code, you’ve defined several variables by assigning values to names. The first five examples include variables that refer to different built-in types. The last example shows that variables can also refer to custom objects like an instance of your SomeCustomClass class.

Setting and Changing a Variable’s Data Type

Apart from a variable’s value, it’s also important to consider the data type of the value. When you think about a variable’s type, you’re considering whether the variable refers to a string, integer, floating-point number, list, tuple, dictionary, custom object, or another data type.

Python is a dynamically typed language, which means that variable types are determined and checked at runtime rather than during compilation. Because of this, you don’t need to specify a variable’s type when you’re creating the variable. Python will infer a variable’s type from the assigned object.

Note: In Python, variables themselves don’t have data types. Instead, the objects that variables reference have types.

For example, consider the following variables:

Python
      
        
      
    
>>> name = "Jane Doe"
>>> age = 19
>>> subjects = ["Math", "English", "Physics", "Chemistry"]

>>> type(name)
<class 'str'>
>>> type(age)
<class 'int'>
>>> type(subjects)
<class 'list'>
Copied!

Read the full article at https://realpython.com/python-variables/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 04, 2024 02:00 PM UTC

Robin Wilson

Join the GeoTAM hackathon to work out business turnovers!

Summary: I’m involved in organising a hackathon, and I’d love you to take part. The open-source GeoTAM hackathon focuses on estimating turnover for individual business locations in the UK, from a variety of open datasets. Please checkout the hackathon page and sign up. There are prizes of up to £2,000!

(Click image for a larger version)

I’m currently working with Rebalance Earth, a boutique asset manager who are focused on making nature an investable asset. Our aim is to mobilise investment in UK natural infrastructure – for example, by arranging investment to undertake river restoration and reduce the risk of flooding. We will do this by finding businesses at risk of flooding, designing restoration schemes that will reduce this risk, and setting up ‘Nature-as-a-Service’ contracts with businesses to pay for the restoration.

I’m the Lead Geospatial Developer at Rebalance Earth, and am leading the development of our Geospatial Predictive Analytics Platform (GPAP), which helps us assess businesses at risk of flooding and design schemes to reduce this flooding.

An important part of deciding which areas to focus on is estimating the total business value at risk from flooding. A good way of establishing this is to use an estimate of the business turnover. However, there are no openly-available datasets showing business turnover in the UK – which is where the hackathon comes in.

We’re looking for participants to bring their expertise in programming, data science, machine learning and more to take some datasets we provide, combine them with other open data and try and estimate turnover. Specifically, we’re interested in turnover of individual business locations – for example, the turnover of a specific supermarket, not the whole supermarket chain.

The hackathon runs from 20th – 26th November 2024. We’ll provide some datasets, some ideas, and a Discord server to communicate through. We’d like you to bring your expertise and see what you can produce. This is a tricky task, and we’re not expecting fully polished solutions; proof-of-concept solutions are absolutely fine. You can enter as a team or an individual.

Most importantly, there are prizes:

£2,000 for the First Prize
£1,000 for the Second Prize
£500 for the Third Prize

and there’s a possibility that we might even hire you to continue work on your idea!

So, please sign up and tell your friends!

November 04, 2024 11:04 AM UTC

ListenData

How to Automate WordPress using Python

This tutorial explains how to use Python to automate tasks in WordPress. It includes various functions to perform tasks such as creating, extracting, updating and deleting WordPress posts, pages, comments and media items (images) directly from Python.

Table of Contents

To read this article in full, please click here

November 04, 2024 08:29 AM UTC

Python Bytes

#408 python-preference only-managed 3.13t

Topics covered in this episode: <ul> <li><a href="https://nedbatchelder.com/blog/202410/github_action_security_zizmor.html?featured_on=pythonbytes">GitHub action security: zizmor</a></li> <li><a href="https://github.blog/news-insights/octoverse/octoverse-2024/?featured_on=pythonbytes">Python is now the top language on GitHub</a></li> <li><a href="https://www.bitecode.dev/p/python-313-what-didnt-make-the-headlines?featured_on=pythonbytes">Python 3.13, what didn't make the headlines</a></li> <li><a href="https://us.pycon.org/2025/?featured_on=pythonbytes">PyCon US 2025</a></li> <li>Extras</li> <li>Joke</li> </ul><a href='https://www.youtube.com/watch?v=9pyPp5lLSfI' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="408">Watch on YouTube</a> About the show Sponsored by: <ul> <li><a href="https://pythonbytes.fm/scout">ScoutAPM</a> - Django Application Performance Monitoring</li> <li><a href="https://pythonbytes.fm/codeium">Codeium</a> - Free AI Code Completion & Chat </li> </ul> Connect with the hosts <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy">@mkennedy@fosstodon.org</a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken">@brianokken@fosstodon.org</a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes">@pythonbytes@fosstodon.org</a></li> </ul> Join us on YouTube at <a href="https://pythonbytes.fm/stream/live">pythonbytes.fm/live</a> to be part of the audience. Usually Monday at 10am PT. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it. Brian #1: <a href="https://nedbatchelder.com/blog/202410/github_action_security_zizmor.html?featured_on=pythonbytes">GitHub action security: zizmor</a> <ul> <li>Article: Ned Batchelder </li> <li>zizmor: William Woodruff & others</li> <li>“a new tool to check your GitHub action workflows for security concerns.”</li> <li>Install with cargo or brew, then point it at workflow yml files.</li> <li>It reports security concerns.</li> </ul> Michael #2: <a href="https://github.blog/news-insights/octoverse/octoverse-2024/?featured_on=pythonbytes">Python is now the top language on GitHub</a> <ul> <li>Thanks to Pat Decker for the heads up.</li> <li>A rapidly growing number of developers worldwide <ul> <li>This suggests AI isn’t just helping more people learn to write code or build software faster—it’s also attracting and helping more people become developers. First-time open source contributors continue to show wide-scale interest in AI projects. But we aren’t seeing signs that AI has hurt open source with low-quality contributions.</li> </ul></li> <li>Python is now the most used language on GitHub as global open source activity continues to extend beyond traditional software development. <ul> <li>The rise in Python usage correlates with large communities of people joining the open source community from across the STEM world rather than the traditional community of software developers.</li> </ul></li> <li>There’s a continued increase in first-time contributors to open source projects. 1.4 million new developers globally joined open source with a majority contributing to commercially backed and generative AI projects. <ul> <li>Notably, we did not see a rise in rejected pull requests. This could indicate that quality remains high despite the influx of new contributors.</li> </ul></li> </ul> Brian #3: <a href="https://www.bitecode.dev/p/python-313-what-didnt-make-the-headlines?featured_on=pythonbytes">Python 3.13, what didn't make the headlines</a> <ul> <li>Some pretty cool updates to pdb : the command line Python debugger <ul> <li>multiline editing</li> <li>code completion </li> </ul></li> <li>pathlib has a bunch of performance updates</li> <li>python -m venv adds a .gitignore file that auto ignores the venv.</li> </ul> Michael #4: <a href="https://us.pycon.org/2025/?featured_on=pythonbytes">PyCon US 2025</a> <ul> <li>Site is live with CFP and dates</li> <li><a href="https://us.pycon.org/2025/about/health-safety-guidelines/?featured_on=pythonbytes">Health code</a> is finally reasonable: “Masks are Encouraged but not Required”</li> <li>PyCon US 2025 Dates <ul> <li>Tutorials - May 14-15, 2025</li> <li>Sponsor Presentations - May 15, 2025</li> <li>Opening Reception - May 15, 2025</li> <li>Main Conference and Online - May 16-18, 2025</li> <li>Job Fair - May 18, 2025</li> <li>Sprints - May 19-May 22, 2025</li> </ul></li> </ul> Extras Brian: <ul> <li><a href="https://micro.webology.dev/2024/11/02/please-publish-and.html?featured_on=pythonbytes">Please publish and share more</a> - Jeff Triplett</li> </ul> Michael: <ul> <li><a href="https://github.com/tox-dev/pre-commit-uv?featured_on=pythonbytes">pre-commit-uv</a> <ul> <li>Just spoke with Sefanie Molin <a href="https://talkpython.fm/episodes/show/482/pre-commit-hooks-for-python-devs?featured_on=pythonbytes">about pre-commit hooks on Talk Python</a></li> </ul></li> <li><a href="https://blog.omnivore.app/p/omnivore-is-joining-elevenlabs?featured_on=pythonbytes">Curse you Omnivore</a>!</li> <li>We have moved to <a href="https://www.hetzner.com/cloud/?featured_on=pythonbytes">hetzner</a> </li> <li><a href="https://typora.io?featured_on=pythonbytes">Typora markdown app</a></li> <li>free-threaded Python is now available via <a href="https://docs.astral.sh/uv?featured_on=pythonbytes">uv</a> <pre><code>uv self update uv python install --python-preference only-managed 3.13t </code></pre></li> </ul> Joke: <a href="https://devhumor.com/media/coding-chair-vs-debugging-chair?featured_on=pythonbytes">Debugging char</a>

November 04, 2024 08:00 AM UTC

Zato Blog

Meaningful automation in Python

2024-11-04, by Dariusz Suchojad

This article is an introduction to meaningful automation, integrations and interoperability with Zato, service-oriented thinking and Python.

Zato is a convenient and secure, Python-based, open-source, service-oriented platform for automation, integrations and interoperability. It is used to connect distributed systems or data sources and to build API-focused, middleware and backend applications.

The platform is designed and built specifically with Python users in mind - often working in, and for, industries such as telecommunications, defense, health care and others that require automation, integrations and interoperability of multiple systems and processes.

Sample real-world, mission-critical Zato environments include:

Systems for telecommunication operators integrating CRM, ERP, Charging Systems, Billing and other OSS/BSS applications internal or external to the operators, including network automation of packet brokers and other network visibility and cybersecurity tools from Keysight
Enterprise services buses for government, helping in the digital transformation of legacy systems and processes towards modern capabilities
AI, ML and data science systems that analyze and improve acquisition and supply chain activities in government processes
Applied observability automation that enables meaningful decision making through the orchestration and coordination of the collection, distribution and presentation of data spread across a pool of independent systems
Platforms for health care and public administration systems, helping to achieve data interoperability through the integration of independent data sources, databases and health information exchanges (HIE)
Global IoT platforms for hybrid integrations of medical devices and software both in the cloud and on premises
Cybersecurity automation, including IT/OT hardware and software assets
Robotic process automation (RPA) of message flows and events produced by early warning systems

Zato offers connectors to all the popular technologies and vendors, such as REST, Cloud, task scheduling, Microsoft 365, Salesforce, Atlassian, SAP, Odoo, SQL, HL7, FHIR, AMQP, IBM MQ, LDAP, Redis, MongoDB, WebSockets, SOAP, Caching and many more.

Running in the cloud, on premises, or under Docker, Kubernetes and other container technologies, Zato services are optimized for high performance and security - it is easily possible to run hundreds and thousands of services on typical server instances as offered by Amazon, Google Cloud, Azure or other cloud providers.

Zato servers offer high availability and no-downtime deployment. Servers form clusters that are used to scale systems both horizontally and vertically.

The product is commercial open-source software with training, professional services and enterprise 24x7x365 support available.

A platform and language for interesting, reusable and atomic services

Zato promotes the design of, and helps you build, solutions composed of services that are interesting, reusable and atomic (IRA).

What does it really mean in practice that something is interesting, reusable and atomic? In particular, how do we define what is interesting?

Each interesting service should make its users want to keep using it more and more. People should immediately see the value of using the service in their processes. An interesting service strikes everyone as immediately useful in wider contexts, preferably with few or no conditions, prerequisites and obligations.

An interesting service is aesthetically pleasing, both in terms of its technical usage as well as its relevance to, and potential applicability in, fields broader than originally envisioned. If people check the service and say "I know, we will definitely use it" or "Why don't we use it" you know that the service is interesting. If they say "Oh no, not this one again" or "No, thanks, but no" then it is the opposite.

Note that focus here is on the value that the service brings for the user. You constantly need to keep in mind that people generally want to use services only if they allow them to fulfill their plans or execute some bigger ideas. Perhaps they already have them in mind and they are only looking for technical means of achieving that or perhaps it is your services that will make a person realize that something is possible at all, but the point is the same, your service should serve a grander purpose.

This mindset, of wanting to build things that are useful and interesting is not specific to Python or, indeed, to software and technology. Even if you are designing and implementing services for your own purposes, you need to act as if you were a consultant that can always see a bigger vision, a bigger architecture, and who can envision results that are still ahead in the future while at the same time not forgetting that it is always a series of small interesting actions, that everyone can relate to, that lead to success.

A curious observation can be made, particularly when you consider all the various aspects of the digital transformation that companies and organizations go through, is that many people to whom the services are addressed, or who sponsor their development, are surprised when they see what automation and integrations are capable.

Put differently, many people can only begin to visualize bigger designs once they see in practice smaller, practical results that further their missions, careers and otherwise help them at work. This is why, again, the focus on being interesting is essential.

At the same, it can be at times advantageous to you that people will not see automation or integrations coming. That lets you take the lead and build a center of such a fundamental shift around yourself. This is a great position to be in, a blue ocean of possibilities, because it means little to no competition inside an organization that you are a part of.

If you are your own audience, that is, if you build services for your own purposes, the same principles apply and it is easy to observe that thinking in services lets you build a toolbox of reusable, complementary capabilities, a portfolio, that you can take with you as you progress in your career. For instance, your services, and your work, can concentrate on a particular vendor and with a set of services that automate their products, you will be always able to put that into use, shorting your own development time, no matter who employs you and in what way.

Regardless of who the clients that you build the solutions for are, observe that automation and integrations with services are evolutionary and incremental in their nature, at least initially. Yes, the resulting value can often be revolutionary but you do not intend to incur any massive changes until there are clear, interesting results available. Trying to integrate and change existing systems at the same time is doable, but not trivial, and it is best left to later stages, once your automation gets the necessary buy-in from the organization.

Services should be ready to be used in different, independent processes. The processes can be strictly business ones, such as processing of orders or payments, or they can be of a deep, technical nature, e.g. automating cybersecurity hardware. What matters in either case is that reusability breeds both flexibility and stability.

There is inherent flexibility in being able to compose bigger processes out of smaller blocks with clearly defined boundaries, which can easily translate to increased competitive advantage when services are placed into more and more areas. A direct result of this is a reduction in R&D time as, over time, you are able to choose from a collection of loosely-coupled components, the services, that hide implementation details of a particular system or technology that they automate or integrate with.

Through their continued use in different processes, services can reduce overall implementation risks that are always part of any kind of software development - you will know that you can keep reusing stable functionality that has been already well tested and that is used elsewhere.

Because services are reusable, there is no need for gigantic, pure waterfall-style implementations of automation and integrations in an organization. Each individual project can contribute a smaller set of services that, as a whole, constitute the whole integrated environment. Conversely, each new project can start to reuse services delivered by the previous ones, hence allowing you to quickly, incrementally, prove the value of the investment in service-oriented thinking.

To make them reusable, services are designed in a way that hides their implementation details. Users only need to know how to invoke the service; the specific systems or processes it automates or integrates are not necessarily important for them to know as long as a specific business goal is achieved. Thanks to that, both services and what they integrate can be replaced without disrupting other parts - in reality, this is exactly what happens - systems with various kinds of data will be changed or modernized but the service will stay the same and the user will not notice anything.

Each service fulfills a single, atomic business need. Each service is deployed independently and, as a whole, they constitute an implementation of business processes taking place in your company or organization. Note that the definition of what the business need is, again, specific to your own needs. In purely market-oriented integrations, this may mean, for instance, the opening of a bank account. In IT or OT automation, on the other hand, it may mean the reconfiguration of a specific device.

That services are atomic also means that they are discrete and that their functionality is finely grained. You will recognize whether a design goes in this direction if consider the names of the services for a moment. An atomic service will invariably use a short name consisting of a single verb and noun. For instance, "Create Customer Account", "Stop Firewall", "Conduct Feasibility Study", it is easy to see that we cannot break them down into smaller part, they are atomic.

At the same time, you will keep creating composite services that invoke other services; this is natural and as expected but you will not consider services such as "Create Customer Account and Set Up a SIM Card" as atomic ones because, in that form, they will not be very reusable, and a major part of why being atomic is important is that it promotes reusability. For instance, having separate services to create customer accounts, independently of setting up their SIM cards, is that one can without difficulty foresee situations when an account is created but a SIM card is purchased at a later time and, conversely, surely one customer account should be able to potentially have multiple SIM cards. Think of it as being similar to LEGO bricks where just a few basic shapes can form millions of interesting combinations.

The point about service naming conventions is well worth remembering because this lets you maintain a vocabulary that is common to both technical and business people. A technical person will understand that such naming is akin to the CRUD convention from the web programming world while a business person will find it easy to map the meaning to a specific business function within a broader business process.

With Zato, you use Python to focus on the business logic exclusively and the platform takes care of scalability, availability communications protocols, messaging, security or routing. This lets you concentrate only on what is the very core of systems integrations - making sure their services are interesting, reusable and atomic.

Python is the perfect choice for this job because it hits the sweet spot under several key headings:

It is a very high level language, with a syntax close to how grammar of various spoken languages works, which makes it easy to translate business requirements into implementation
It is a solid, mainstream and full-featured, real programming language rather than a domain-specific one which means that it offers a great degree of flexibility and choice in expressing their needs
It is difficult to find universities without Python courses. Most people entering the workforce already know Python, it is a new career language. In fact, it is becoming more and more difficult to find new talent who would not prefer to use Python.
Yet, one does not need to be a developer or a full-time programmer to use Python. In fact, most people who use Python are not programmers at all. They are specialists in other fields who also need to use a programming language to automate or integrate their work in a meaningful way.
Many Python users come from backgrounds in network and cybersecurity engineering - fields that naturally require a lot of automation using a real language that is convenient and easy to get started with
Many Python users are scientists with a background in AI, ML and data science, applying their domain-specific knowledge in processes that, by their very nature, require them to collect and integrate data from independent sources, which again leads to automation and integrations
Many Python users have a strong web programming background which means that it takes little effort to take a step further, towards automation and integrations. In turn, this means that it is easy to find good people for API projects.
Many Python users know multiple programming languages - this is very useful in the context of integration projects where one is typically faced with dozens of technologies, vendors or integration methods and techniques.
Lower maintenance costs - thanks to the language's unique design, Python programmers tend to produce code that is easy to read and understand. From the perspective of multi-year maintenance, reading and analyzing code, rather than writing it, is what most programmers do most of the time, making sense to use a language that makes it easy to carry out the most common tasks.

In short, Python can be construed as executable pseudo-code with many of its users already having roots in modern server-side programming so Zato, both from a technical and strategic perspective, is a natural choice for both simple and complex, sophisticated automation, integration and interoperability solutions as a platform built in the language and designed for Python people from day one.

Next steps:

➤ Read about how to use Python to build and integrate enterprise APIs that your tests will cover
➤ Python API integration tutorial
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?

November 04, 2024 08:00 AM UTC

Michael Foord

Python Knowledge Sharing Videos Online

.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }

I’ve been teaching Python in one hour knowledge sharing sessions, some of which I’ve put online on youtube.

This is the link to the playlist of the sessions:

Python Knowledge Share Videos on YouTube

The slides for each of the sessions, along with some example code, can be found in this github repository:

https://github.com/voidspace/talks

So far there are seven one-hour sessions (with more planned) on:

Python Core Object Model
- Python objects
- Slots
- Attribute lookup and the MRO
- Inheritance, multiple inheritance and super
- Inside Python objects and classes
Closures and decorators (functional programming)
- Functional programming: higher order functions and functions as objects
- Lambdas
- Closures: functions that build functions
- Variable scoping: global, local and nonlocal
- Decorators: functions wrapping functions
- Decorator factories (decorators that take arguments)
- Class decorators
- Decorator order and using functools.wraps
Generators and Iterators
- The iteration protocol
- Stateful iteration with generators
- Adding iteration support to objects
- References, assignment and mutability
- Identity versus equality
- Call by object
- Object copying
Unicode, Floats and regex
- Floating point numbers
- Unicode, encodings and strings
- Regular expressions
Concurrency (async, threads, processes, the GIL)
- The history of concurrency from AmigaOS to a multi-core world
- Python and the Global Interpreter Lock
- I/O bound and CPU bound tasks
- Threads and processes
- Async programming (green threading, coroutines)
- Concurrency with threads
- Concurrency with multiprocessing
- Looking to the future (Python 3.13): optional GIL (PEP 703) and subinterpreters (PEP 554)
Testing with pytest
- virtual environments and pipenv (installing pytest)
- pytest command line for collecting and running tests
- Simple test functions and asserts
- Test fixtures and conftest.py
- Testing exceptions
- Test parameterisation for test combinations
- Test marking for running test subsets
- Principles of testing (unit tests versus end to end testing, building test helpers etc)
- Mocking and patching
Modules and Namespaces
- Import syntax variations
- namespaces and variable lookups
- sys.modules and the import cache
- Module objects
- Module level functionality: __dir__ and __getattr__
- Packages and the filesystem
- Relative import syntax
- Module reloading (how to do it and why not to do it)
- Circular imports, avoiding and fixing
- Executable modules and packages

Other Talks

A selection of some of the talks and interviews I’ve given on Python and software engineering across my career.

November 04, 2024 12:00 AM UTC

Agile Alliance Scrummaster Certification

Scrummaster

I’ve been a fan of Agile ever since my first programming job with Resolver Systems back in 2006. I had taught myself programming and there I really learned engineering, how to build software products whilst caring about quality. I became passionate about testing as a way to ensure a minimal level of quality and about agile processes which are able to change quickly.

My only experience of Scrum was for a year at a heavily waterfall shop which layered Scrum for project management on top of heavily waterfall software development processes. It wasn’t a fun experience but I still learned a great deal.

I’ve been working with Gigaclear, as team lead on backend API servers, including One Touch Switch, for about a year now. I enjoy our software development practises and processes; we use a combination of agile for software development, devops (devsecops of course) for software deployment and maintenance, and Scrum for project management. It’s a very effective combination.

I recently attended a course with the Agile Alliance, led by John McFadyen, and became certified as a Scrummaster.

The course was fantastic and very inspiring. At Gigaclear we’re systematically evaluating all our systems, systems architecture and processes. This process and the course have been hugely inspirational and I have an article on software development processes coming shortly…

November 04, 2024 12:00 AM UTC

November 03, 2024

Glyph Lefkowitz

The Federation Deathmatch

It’s the weekend, and I have some Thoughts about federated social media. So, buckle up, I guess, it’s time to start some fights.

Recently there has been some discourse about Bluesky’s latest fundraising round. I’ve been participating in conversations about this on Mastodon, and I think I might sometimes come across as a Mastodon partisan, but my feelings are complex and I really don’t want to be boosting the ActivityPub Fediverse without qualification.

So here are some qualifications.

Bluesky Is Evil

To the extent that I am an ActivityPub partisan in the discourse between ActivityPub and ATProtocol, it is because I do not believe that Bluesky is a meaningfully decentralized social network. It is a social network, run by a company, which has a public API with some elements that might, one day, make it possible for it to be decentralized. But today, it is not, either practically or theoretically.

The Bluesky developers are putting in a ton of effort to maybe make it decentralized, hypothetically, someday. A lot of people think they will succeed. But ActivityPub (and, of course, Mastodon specifically) are already, today, meaningfully decentralized, as you can see on FediDB, there are instances with hundreds of thousands of people on them, before we even get to esoterica like the integrations Threads, Wordpress, Flipboard, and Ghost are doing.

The inciting incident for this post — that a lot of people are also angry about Bluesky raising millions of dollars from Evil Guys Doing Evil Stuff Capital — is indeed a serious concern. It lights the fuse that burns towards their eventual, inevitable incredible journey. ATProtocol is just an API, and that API will get shut off one day, whenever their funders get bored of the pretense of their network being “decentralized”.

At time of writing, it is also interesting that 3 of the 4 times that the CEO of Bluesky has even skeeted the word “blockchain” is to say “no blockchain”, to reassure users that the scam magnet of “Blockchain” is not actually near their product or protocol, which is a much harder position to maintain when your lead investor is “Blockchain Capital”.

I think these are all valid criticisms of Bluesky. But I also think that the actual engineers working on the product are aware of these issues, and are making a significant effort to address them or mitigate them in any way they can. All that work can still be easily incinerated by a slow quarter in terms of user growth numbers or a missed revenue forecast when the VCs are getting impatient, but it’s not nothing, it is a life’s work.

Really, who among us could not have our life’s ambitions trivially destroyed in an afternoon, simply because a billionaire decided that they should be? If you feel like you are safe from this, I have some bad news about how money works. So we are all doing our best in an imperfect system and maybe Bluesky is on to something here. That’s eminently possible. They’re certainly putting forth an earnest effort.

Mastodon Is Stupid

Meanwhile, not nearly as much has been made recently of Mastodon refusing funding from a variety of sources, when all indications are that funding is low, and plummeting, far below the level required to actually sustain the site, and they haven’t done a financial transparency report for over a year, and that report was already nearly a year late.

Mastodon and the fediverse are not nearly in a position to claim moral superiority over Bluesky. Sure, taking blockchain VC money might seem like a rookie mistake, but going out of business because you are spurning every possible source of funding is not that wise either.

Some might think that, sure, Mastodon the company might die but at least the Fediverse as a whole will keep going strong, right? Lots of people run their own instances! I even find elements of this argument convincing, and I think there is probably some truth to it. But to really believe this argument as claimed, that it’s a fait accompli that the fediverse will survive in some form, that all those self-run servers will be a robust network that will self-repair, requires believing some obviously false stuff. It is frankly unprofitable to run a Fediverse instance. Realistically, if you want to operate a mastodon server for yourself, it is going to cost at least $100/year once you include stuff like having a domain name, and managing the infrastructure costs is a complex problem that keeps getting harder to manage as the software itself gets slower.

Cory Doctorow has recently argued that this is all worth it, because at least on Mastodon, you’re in control, not at the whims of centralized website operators like Bluesky. In his words,

On Mastodon (and other services based on Activitypub), you can easily leave one server and go to another, and everyone you follow and everyone who follows you will move over to the new server. If the person who runs your server turns out to be imperfect in a way that you can’t endure, you can find another server, spend five minutes moving your account over, and you’re back up and running on the new server

He concludes:

Any system where users can leave without pain is a system whose owners have high switching costs and whose users have none

(Emphasis mine).

This is a beautiful vision. It is, however, an incorrect assessment of the state of the Fediverse as it stands today. It’s not true in two important ways:

First, if you look at any account of a user’s fediverse account migration, like this one from Steve Bate or this one from the Ente project or this one from Erin Kissane, you will see that it is “painful for the foreseeable future” or “wasn’t as seamless as advertised”, and that “the best time to […] migrate instances […] is never”. This language does not presage a pleasant experience, as Doctorow puts it, “without pain”.

Second, migration is an active process that requires engagement from the instance that hosts you. If you have been blocked or banned, or had your account terminated, you are just out of luck. You do not have control over your data or agency over your online identity unless you’ve shelled out the relatively exorbitant amount of money to actually operate your own instance.

In short, ActivityPub is no panacea. A federated system is not really a “decentralized” system, as much as it is a bunch of smaller centralized systems that all talk to each other. You still need to know, and care, about your social and financial relationship to the operators of your instance. There is probably no getting away from this, like, just generally on the Internet, no matter how much peer-to-peer software we deploy, but there certainly isn’t in the incomplete mess that is ActivityPub.

JOIN, or DIE.

Neither Mastodon (or ActivityPub) nor Bluesky (or ATProtocol) has a comprehensive solution to the problem of decentralized social media. These companies, and these protocols, are both deeply flawed and if everything keeps bumping along as it is, I believe both are likely to fail. At different times, on different timelines, and for different reasons, but fail nonetheless.

However, these networks are both small and growing, and we are not yet in the phase of enshittification where margins are shrinking and audiences are captured and the screws must be tightened to juice revenue. There are stil possibilities. Mastodon is crowdfunded and what they lack in resources they make up for in flexibility and scrappiness. Bluesky has money and while there will eventually be a need to monetize somehow, they have plenty of runway to come up with that answer, and a lot of sophisticated protocol work has been done. Not enough to make a complete circut and allow users true, practical decentralization, but it’s not nothing, either.

Mastodon and Bluesky are both organizations with humans in them, and piles of data that is roughly schema-compatible even if the nuances and details are different. I know that there is a compatible model becuse thanks to both platforms being relatively open, there is a functioning ActivityPub/ATProtocol bridge in the form of Brid.gy Fed. You can use it today, and I highly recommend that you do so, so that “choice of protocol” does not fully define your audience. If you’re on bluesky, follow this account, and if you’re on Mastodon or elsewhere on the Fediverse, search for and follow @bsky.brid.gy@bsky.brid.gy.

The reality that fans of decentralized, independent social media must confront is that we are a tiny audicence right now. Whichever site we are looking at, we are talking about a few million monthly active users at best, in a world where even the pathetic husk of Twitter still has hundreds of millions and Facebook has billions. Interneceine fights are not going to get us anywhere. We need to build bridges and links and connect our networks as densely as possible. If I’m being honest, Bridgy Fed looks like a pretty janky solution, but it’s something, and we need to start doing something soon, so we do not collectively become a permanent minority that mass markets can safely ignore.

As users, we need to set an example, so that the developers of the respective platforms get their shit together and work together directly so that workarounds like Bridgy are not required. Frankly, this is mostly on the ActivityPub and Mastodon devs, as far as I can tell. Unfortunately, not a lot of this seems to be public, or at least I haven’t witnessed a lot of it directly, but I have heard repeatedly that the ActivityPub developers are prickly, and this is one high-profile public example where an ActivityPub partisan is incredibly, pointlessly hostile and borderline harrassing towards someone — Mike Masnick, a long-time staunch advocate for open protocols and open patents, someone with a Mastodon account, and thus as good a prospective ally as the ActivityPub fediverse might reasonably find — explaining some of the relative benefits of Bluesky.

Most of us are technology nerds in one way or another. In that way we can look at signifiers like “ActivityPub” and “ATProtocol”, and feel like these are hard boundaries around different all-encompassing structures for the future, and thus tribes we must join and support.

A better way to look at this, however, is to see social entities like Mastodon gGmbH and Bluesky PBC — or, more to the point, Fosstodon, SFBA Social, Hachyderm (and maybe, one day, even an instance which isn’t fully just for software development nerds), as groups that deploy these protocols to access some data that they publish, just as they might publish their website over HTTP or their newsletters over SMTP. There are technical challenges involved in bridging between mutually unintelligible domain models, but that is, like, network software's whole deal. Most software is just some kind of translation from one format or context to another. The best possible future for the fediverse is the one where users care as much about the distinction between ATProtocol and ActivityPub as they do about the distinction between POP3 and IMAP.

To both developers and users of these systems, I say: get it together. Be nice to each other. Because the rest of the social media ecosystem is sure as shit not going to be nice to us if we ever see even a hint of success and start to actually cut into their user base.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor!

November 03, 2024 09:49 PM UTC

Real Python

The Python Square Root Function

The Python square root function, sqrt(), is part of the math module and is used to calculate the square root of a given number. To use it, you import the math module and call math.sqrt() with a non-negative number as an argument. For example, math.sqrt(9) returns 3.0.

This function works with both integers and floats and is essential for mathematical operations like solving equations and calculating geometric properties. In this tutorial, you’ll learn how to effectively use the square root function in Python.

By the end of this tutorial, you’ll understand how:

Python’s sqrt() function calculates square roots using Python’s math.sqrt() for quick and accurate results in your programs.
math.sqrt() calculates the square root of positive numbers and zero but raises an error for negative inputs.
Python’s square root function can be used to solve real-world problems like calculating distances using the Pythagorean theorem.

Time to dive in!

Python Pit Stop: This tutorial is a quick and practical way to find the info you need, so you’ll be back to your project in no time!

Free Bonus: Click here to get our free Python Cheat Sheet that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Square Roots in Mathematics

In algebra, a square, x, is the result of a number, n, multiplied by itself: x = n²

You can calculate squares using Python:

Python
      
        
      
    
>>> n = 5
>>> x = n**2
>>> x
25
Copied!

The Python ** operator is used for calculating the power of a number. In this case, 5 squared, or 5 to the power of 2, is 25.

The square root, then, is the number n, which when multiplied by itself yields the square, x.

In this example, n, the square root of 25, is 5.

25 is an example of a perfect square. Perfect squares are the squares of integer values:

Python
      
>>> 1**2
1

>>> 2**2
4

>>> 3**2
9
Copied!

You might have memorized some of these perfect squares when you learned your multiplication tables in an elementary algebra class.

If you’re given a small perfect square, it may be straightforward enough to calculate or memorize its square root. But for most other squares, this calculation can get a bit more tedious. Often, an estimation is good enough when you don’t have a calculator.

Read the full article at https://realpython.com/python-square-root-function/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 03, 2024 02:00 PM UTC

Michael Foord

Gigaclear One Touch Switch Service

OneTouchSwitch

For the last year I’ve been working as a team lead for backend API development with Gigaclear a UK rural ISP who own and run fibre internet to rural communities across the UK. This is alongside my training work.

This image shows the main project I’ve been working on since joining Gigaclear, One Touch Switch. A regulatory requirement for all ISPs to allow automated switching between ISPs. When you sign up with a new internet provider your account is automatically ceased with the old provider and VOIP numbers can be automatically ported.

Our OTS project is just part of the Gigaclear One Touch Switch system which interfaces with Salesforce and Netadmin and the website order flow (the Online Buying Journey) and represents an impressive engineering effort. We were one of the first ISPs with a system ready to take part in industry trials a few months ago, both OTS and our underlying systems passed pen testing with flying colours, and the switch on has been smooth.

Something I’m proud to have been part of. My current project is preparing security awareness training materials based on OWASP for our various engineering departments whilst we also undertake a systematic review of all of our systems and processes.

In the diagram I’m team lead for the Sphinx engineering team.

November 03, 2024 12:00 AM UTC

Adventures with MicroPython

ZumoBot

My first blog post in a few years! I have some articles I’d like to publish, and some adventures to share, so I thought it was time to fire up a blog engine again.

My nine year old son, Benjamin, is really into programming with Scratch and he’s keen to play with electronics and learn MicroPython. Which is awesome because there’s almost nothing I would love to do more with him.

MicroPython is an extremely impressive implementation of Python that will run on embedded devices and microcontrollers, as well as bigger tiny computers like the Raspberry Pi.

I’ve dug out an old MicroBit I had, purchased a Raspberry Pi Pico board/kit and also a ZumoBot 2040 robot which uses the same microcontroller as the Pico, to play with.

I’m now starting to get to grips with the basics, using the Thonny IDE.

I have a bunch of Neopixel LEDs, including a long light strip, I’d like to wire up in my living room controlled by a Pico board and an Android App using Kivy. That’s my goal number 1.

I’d like to program the ZumoBot to explore and map my flat. Goal 2.

Meanwhile Benjamin is enjoying playing with electronics (switches, LEDs, potentiometer and now a motor) with the Pico and on his own he’s programming the MicroBit with Scratch (or at least “blocks” which is the Microsoft equivalent). I’ve also done my first soldering in over a decade.

I have a github repository to track my tinkering, but I’d like to write up some recipes and post them on this blog as I go. (The biggest hurdle is I can’t easily create circuit diagrams. Time to explore.)

The github repository and ZumoBot links:

November 03, 2024 12:00 AM UTC

November 02, 2024

Brett Cannon

Don't return named tuples in new APIs

In my opinion, you should only introduce a named tuple to your code when you&aposre updating a preexisting API that was already returning a tuple or you are wrapping a tuple return value from another API.

Let&aposs start with when you should use named tuples. Usually an API that returns a tuple does so when you only have a couple of items in your tuple and the name of the function returning the tuple id enough to explain what each item in the tuple does. But sometimes your API expands and you find that your tuple is no longer self-documenting purely based on the name of the API (e.g., get_mouse_position() very likely has a two-item tuple of X and Y coordinates of the screen while app_state() could be a tuple of anything). When you find yourself in the situation of needing your return type to describe itself and a tuple isn&apost cutting it anymore, then that&aposs when you reach for a named tuple.

So why not start out that way? In a word: simplicity. Now, some of you might be saying to yourself, "but I use named tuples because they are so simple to define!" And that might be true for when you define your data structure (and I&aposll touch on this "simplicity of definition" angle later), but it actually makes your API more complex for both you and your users to use. For you, it doubles the data access API surface for your return type as you have to now support index-based and attribute-based data access forever (or until you choose to break your users and change your return type so it doesn&apost support both approaches). This leads to writing tests for both ways of accessing your data, not just one of them. And you shouldn&apost skimp on this because you don&apost know if your users will use indexes or attribute names to access the data structure, nor can you guarantee someone won&apost break your code in the future by dropping the named tuple and switching to some custom type (thanks to Python&aposs support of structural typing (aka duck typing), you can&apost assume people are using a type checker and thus the structure of your return type becomes your API contract). And so you need to test both ways of using your return type to exercise that contract you have with your users, which is more work than had you not used a named tuple and instead chose just a tuple or just a class.

Named tuples are also a bit more complex for users. If you&aposre reaching for a named tuple you&aposre essentially signalling upfront that the data structure is too big/complex for a tuple alone to work. And yet by using a named tuple means you are supporting the tuple approach even if you don&apost think it&aposs a good idea from the start. On top of that, the tuple API allows for things that you probably don&apost want people doing with your return type, like slicing, iterating over all the items as if they are homogeneous, etc. Basically my argument is the "flexibility" of having the index-based access to the data on top of the attribute-based access isn&apost flexible in a good way.

So why do people still reach for named tuples when defining return types for new APIs? I think it&aposs because people find them faster to define a new type than writing out a new class. Compare this:

Point = namedtuple(&aposPoint&apos, [&aposx&apos, &aposy&apos, &aposz&apos])

To this:

class Point:
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

So there is a clear difference in the amount of typing. But there are three more ways to do the same data structure that might not be so burdensome. One is dataclasses:

@dataclasses.dataclass
class Point:
    x: int
    y: int
    z: int

Another is simply a dictionary, although I know some prefer attribute-based access to data so much that they won&apost use this option). Toss in a TypedDict and you also get editor support as well:

class Point(typing.TypedDict):
    x: int
    y: int
    z: int

# Alternatively ...
Point = typing.TypedDict("Point", {"x": int, "y": int, "z": int})

A third option is types.SimpleNamespace if you really want attributes without defining a class:

Point = lambda x, y, z: types.SimpleNamespace(x=x, y=y, z=z)

If none of these options work for you then you can always hope that somehow I convince enough people that my record/struct idea is a good one and get into the language. 😁

My key point in all of this is to prefer readability and ergonomics over brevity in your code. That means avoiding named tuples except where you are expanding to tweaking an existing API where the named tuple improves over the plain tuple that&aposs already being used.

November 02, 2024 10:00 PM UTC

Ned Batchelder

Coverage.py originally

Something many people don’t realize is that I didn’t write the original coverage.py. It was written by Gareth Rees in 2001. I’ve been extending and maintaining it since 2004. This ancient history came up this week, so I grabbed the 2001 version from archive.org to keep it here for posterity.

I already had a copy of Gareth’s original page about coverage.py, which now links to my local copy of coverage.py from 2001. BTW: that page is itself a historical artifact now, with the header from this site as it looked when I first copied the page.

The original coverage.py was a single file, so the “coverage.py” name was literal: it was the name of the file. It only had about 350 lines of code, including a few to deal with pre-2.0 Python! Some of those lines remain nearly unchanged to this day, but most of it has been heavily refactored and extended.

Coverage.py now has about 20k lines of Python in about 100 files. The project now has twice the amount of C code as the original file had Python. I guess in almost 20 years a lot can happen!

It’s interesting to see this code again, and to reflect on how far it’s come.

November 02, 2024 08:27 PM UTC