Planet Python
Last update: February 12, 2025 07:43 AM UTC
February 12, 2025
Kushal Das
pass using stateless OpenPGP command line interface
Yesterday I wrote about how
I am using a different tool for git
signing and verification. Next, I
replaced my pass
usage. I have a small
patch to use
stateless OpenPGP command line interface (SOP). It is an implementation
agonostic standard for handling OpenPGP messages. You can read the whole SPEC
here.
Installation
cargo install rsop rsop-oct
And copied the bash script from my repository to the path somewhere.
The rsoct
binary from rsop-oct
follows the same SOP standard but uses the
card to signing/decryption. I stored my public key in
~/.password-store/.gpg-key
file, which is in turn used for encryption.
Usage
Here nothing changed related my daily pass usage, except the number of time I am typing my PIN :)
February 12, 2025 05:26 AM UTC
February 11, 2025
PyCoder’s Weekly
Issue #668: NumPy, Compiling Python 1.0, BytesIO, and More (Feb. 11, 2025)
#668 – FEBRUARY 11, 2025
View in Browser »
NumPy Techniques and Practical Examples
In this video course, you’ll learn how to use NumPy by exploring several interesting examples. You’ll read data from a file into an array and analyze structured arrays to perform a reconciliation. You’ll also learn how to quickly chart an analysis & turn a custom function into a vectorized function.
REAL PYTHON course
Let’s Compile Python 1.0
As part of the celebration of 31 years of Python, Bite Code compiles the original Python 1.0 and plays around with it.
BITE CODE!
Postman AI Agent Builder Is Here: The Quickest Way to Build AI Agents. Start Building
Postman AI Agent Builder is a suite of solutions that accelerates agent development. With centralized access to the latest LLMs and APIs from over 18,000 companies, plus no-code workflows, you can quickly connect critical tools and build multi-step agents — all without writing a single line of code →
POSTMAN sponsor
Save Memory With BytesIO
If you want to save memory when reading from a BytesIO
object, getvalue()
is surprisingly a good choice.
ITAMAR TURNER-TRAURING
Discussions
Python Jobs
Backend Software Engineer (Anywhere)
Articles & Tutorials
How to Split a String in Python
This tutorial will help you master Python string splitting. You’ll learn to use .split()
, .splitlines()
, and re.split()
to effectively handle whitespace, custom delimiters, and multiline text, which will level up your data parsing skills.
REAL PYTHON
The Mutable Trap: Avoiding Unintended Side Effects in Python
“Ever had a Python function behave strangely, remembering values between calls when it shouldn’t? You’re not alone! This is one of Python’s sneakiest pitfalls—mutable default parameters.”
CRAIG RICHARDS • Shared by Bob
Posit Package Manager: Secure Python Library Management
Python developers use Posit Package Manager to mirror public & internally developed repos within their firewalls. Get reporting on known vulnerabilities to proactively address potential threats. High-security environments can even run air-gapped.
POSIT sponsor
Decorator JITs: Python as a DSL
There are several Just In Time compilation tools out there that allow you to decorate a function to indicate you want it compiled. This article shows you how that works.
ELI BENDERSKY
Better Unit-Tests for Email With Django 5.2
Django 5.2 contains a new helper on the email class to make it easier to write unit-tests validating that your email contains the content you expect it to contain.
MEDIUM.COM/AMBIENT-INNOVATION • Shared by Ronny Vedrilla
Rendering Form Fields as Group in Django
Django 5.0 added the concept of field groups which make it easier to customize the layout of Django forms. This article covers what groups are and how to use them.
VALENTINO GAGLIARDI
Developer Philosophy
The author was recently invited with other senior devs to give a lightning talk on their personal development philosophy. This post captures those thoughts.
QNTM
Interrupting Scripts Without Tracebacks
This Things-I’ve-Learned post talks about how you can suppress the KeyboardInterrupt
expression so your program doesn’t exit with a traceback.
RODRIGO GIRÃO SERRÃO
PEP 772: Packaging Governance Process
This PEP proposes a Python Packaging Council with broad authority over packaging standards, tools, and implementations.
PYTHON.ORG
Python Terminology: An Unofficial Glossary
“Definitions for colloquial Python terminology (effectively an unofficial version of the Python glossary).”
TREY HUNNER
Projects & Code
Events
Weekly Real Python Office Hours Q&A (Virtual)
February 12, 2025
REALPYTHON.COM
Python Atlanta
February 14, 2025
MEETUP.COM
Python Barcamp Karlsruhe 2025
February 15 to February 17, 2025
BARCAMPS.EU
Inland Empire Python Users Group Monthly Meeting
February 19, 2025
MEETUP.COM
PyData Bristol Meetup
February 20, 2025
MEETUP.COM
DjangoCongress JP 2025
February 22 to February 23, 2025
DJANGOCONGRESS.JP
PyConf Hyderabad 2025
February 22 to February 24, 2025
PYCONFHYD.ORG
Happy Pythoning!
This was PyCoder’s Weekly Issue #668.
View in Browser »
[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]
February 11, 2025 07:30 PM UTC
Python Insider
Python 3.14.0 alpha 5 is out
Here comes the antepenultimate alpha.
https://www.python.org/downloads/release/python-3140a5/
This is an early developer preview of Python 3.14
Major new features of the 3.14 series, compared to 3.13
Python 3.14 is still in development. This release, 3.14.0a5, is the fifth of seven planned alpha releases.
Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process.
During the alpha phase, features may be added up until the start of the beta phase (2025-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2025-07-22). Please keep in mind that this is a preview release and its use is not recommended for production environments.
Many new features for Python 3.14 are still being planned and written. Among the new major new features and changes so far:
- PEP 649: deferred evaluation of annotations
- PEP 741: Python configuration C API
- PEP 761: Python 3.14 and onwards no longer provides PGP signatures for release artifacts. Instead, Sigstore is recommended for verifiers.
- Improved error messages
- A new type of interpreter. For certain newer compilers, this interpreter provides significantly better performance. Opt-in for now, requires building from source.
- Python removals and deprecations
- C API removals and deprecations
- (Hey, fellow core developer, if a feature you find important is missing from this list, let Hugo know.)
The next pre-release of Python 3.14 will be the penultimate alpha, 3.14.0a6, currently scheduled for 2025-03-14.
More resources
- Online documentation
- PEP 745, 3.14 Release Schedule
- Report bugs at github.com/python/cpython/issues
- Help fund Python and its community
And now for something completely different
2025-01-29 marked the start of a new lunar year, the Year of the Snake 🐍 (and the Year of Python?).
For centuries, π was often approximated as 3 in China. Some time between the years 1 and 5 CE, astronomer, librarian, mathematician and politician Liu Xin (劉歆) calculated π as 3.154.
Around 130 CE, mathematician, astronomer, and geographer Zhang Heng (張衡, 78–139) compared the celestial circle with the diameter of the earth as 736:232 to get 3.1724. He also came up with a formula for the ratio between a cube and inscribed sphere as 8:5, implying the ratio of a square’s area to an inscribed circle is √8:√5. From this, he calculated π as √10 (~3.162).
Third century mathematician Liu Hui (刘徽) came up with an algorithm for calculating π iteratively: calculate the area of a polygon inscribed in a circle, then as the number of sides of the polygon is increased, the area becomes closer to that of the circle, from which you can approximate π.
This algorithm is similar to the method used by Archimedes in the 3rd century BCE and Ludolph van Ceulen in the 16th century CE (see 3.14.0a2 release notes), but Archimedes only went up to a 96-sided polygon (96-gon). Liu Hui went up to a 192-gon to approximate π as 157/50 (3.14) and later a 3072-gon for 3.14159.
Liu Hu wrote a commentary on the book The Nine Chapters on the Mathematical Art which included his π approximations.
In the fifth century, astronomer, inventor, mathematician, politician, and writer Zu Chongzhi (祖沖之, 429–500) used Liu Hui’s algorithm to inscribe a 12,288-gon to compute π between 3.1415926 and 3.1415927, correct to seven decimal places. This was more accurate than Hellenistic calculations and wouldn’t be improved upon for 900 years.
Happy Year of the Snake!
Enjoy the new release
Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organisation contributions to the Python Software Foundation.
Regards from a remarkably snowless Helsinki,
Your release team,
Hugo van Kemenade
Ned Deily
Steve Dower
Łukasz Langa
February 11, 2025 04:25 PM UTC
Real Python
Building a Python Command-Line To-Do App With Typer
Building an application to manage your to-do list can be an interesting project when you’re learning a new programming language or trying to take your skills to the next level. In this video course, you’ll build a functional to-do application for the command line using Python and Typer, which is a relatively young library for creating powerful command-line interface (CLI) applications in almost no time.
With a project like this, you’ll apply a wide set of core programming skills while building a real-world application with real features and requirements.
In this video course, you’ll learn how to:
- Build a functional to-do application with a Typer CLI in Python
- Use Typer to add commands, arguments, and options to your to-do app
- Test your Python to-do application with Typer’s
CliRunner
and pytest
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
February 11, 2025 02:00 PM UTC
Kushal Das
Using openpgp-card-tool-git with git
One of the power of Unix systems comes from the various small tools and how
they work together. One such new tool I am using for some time is for git
signing
& verification
using OpenPGP and my Yubikey for the actual signing
operation via
openpgp-card-tool-git. I
replaced the standard gpg
for this usecase with the oct-git
command from this
project.
Installation & configuration
cargo install openpgp-card-tool-git
Then you will have to configuration your (in my case the global configuration) git configuration.
git config --global gpg.program <path to oct-git>
I am assuming that you already had it configured before for signing, otherwise you have to run the following two commands too.
git config --global commit.gpgsign true
git config --global tag.gpgsign true
Usage
Before you start using it, you want to save the pin in your system keyring.
Use the following command.
oct-git --store-card-pin
That is it, now your git commit
will sign the commits using oct-git
tool.
In the next blog post I will show how to use the other tools from the author for various different OpenPGP oeprations.
February 11, 2025 11:12 AM UTC
Django Weblog
DSF member of the month - Lily Foote
For February 2025, we welcome Lily Foote (@lilyf) as our DSF member of the month! ⭐
Lily Foote is a contributor to Django core for many years, especially on the ORM. She is currently a member of the Django 6.x Steering Council and she has been a DSF member since March 2021.
You can learn more about Lily by visiting her GitHub profile.
Let’s spend some time getting to know Lily better!
Can you tell us a little about yourself (hobbies, education, etc)
My name is Lily Foote and I’ve been contributing to Django for most of my career. I’ve also recently got into Rust and I’m excited about using Rust in Python projects. When I’m not programming, I love hiking, climbing and dancing (Ceilidh)! I also really enjoying playing board games and role playing games (e.g. Dungeons and Dragons).
How did you start using Django?
I’d taught myself Python in my final year at university by doing Project Euler problems and then decided I wanted to learn how to make a website. Django was the first Python web framework I looked at and it worked really well for me.
What other framework do you know and if there is anything you would like to have in Django if you had magical powers?
I’ve done a small amount with Flask and FastAPI. More than any new features, I think the thing that I’d most like to see is more long-term contributors to spread the work of keeping Django awesome.
What projects are you working on now?
The side project I’m most excited about is Django Rusty Templates, which is a re-implementation of Django’s templating language in Rust.
Which Django libraries are your favorite (core or 3rd party)?
The ORM of course!
What are the top three things in Django that you like?
Django Conferences, the mentorship programme Djangonaut Space and the whole community!
You have been a mentor multiple times with GSoC and Djangonaut Space program, what is required according to you to be a good mentor?
I think being willing to invest time is really important. Checking in with your mentees frequently and being an early reviewer of their work. I think this helps keep their motivation up and allows for small corrections early on.
Any advice for future contributors?
Start small and as you get more familiar with Django and the process of contributing you can take on bigger issues. Also be patient with reviewers – Django has high standards, but is mostly maintained by volunteers with limited time.
You are now part of the Steering Council, congratulations again! Do you have any words to share related to that?
Yes! It’s a huge honour! Since January, we’ve been meeting weekly and it feels like we’ve hardly scratched the surface of what we want to achieve. The biggest thing we’re trying to tackle is how to improve the contribution experience – especially evaluating new feature ideas – without draining everyone’s time and energy.
You have a lot of knowledge in the Django ORM, how did you start to contribute to this part?
I added the Greatest and Least expressions in Django 1.9, with the support of one of the core team at the time. After that, I kept showing up (especially at conference sprints) and finding a new thing to tackle.
Is there anything else you’d like to say?
Thanks for having me on!
Thank you for doing the interview, Lily!
February 11, 2025 04:51 AM UTC
Seth Michael Larson
Building software for connection (#2: Consensus)
This is the second article in a series about “software for connection”.
In the previous article we concluded that a persistent always-on internet connection isn't required for software to elicit feelings of connection between humans.
Building on this conclusion: let's explore how Animal Crossing software was able to intercommunicate without requiring a centralized server and infrastructure and the trade-offs for these design decisions.
![](https://dodo.ac/np/images/e/e2/Tom_Nook%27s_Special_Delivery.png)
Image of Tom Nook from an Animal Crossing online contest (Nookipedia)
Distributing digital goods without the internet
Animal Crossing has over 1,000 unique items that need to be collected for a complete catalog, including furniture, wallpapers, clothing, parasols, and carpets. Many of these items are quite rare or were only programmed to be accessible through an official Nintendo-affiliated distribution such as a magazine or online contest.
Beyond official distributions, it's clear Animal Crossings' designer, Katsuya Eguchi, wanted players to cooperate to complete their catalogs. The game incentivized trading items between towns by assigning one “native fruit” (Apple, Orange, Cherry, Peach, or Pear) and randomly making a subset of items harder to find than others depending on a hidden “item group” variable (either A, B, or C).
Items could be exchanged between players when one player visits another town, but this required physically bringing your memory card to another players' GameCube. The GameCube might have come with a handle, but the 'cube wasn't exactly a portable console. Sharing a physical space isn't something you can do with everyone or on a regular basis.
So what did Katsuya Eguchi design for Animal Crossing? To allow for item distributions from magazines and contests and to make player-to-player item sharing easier Animal Crossing included a feature called “secret codes”.
This feature worked by allowing players to exchange 28-character codes with Tom Nook for items. Players could also generate codes for their friends to “send” an item from their own game to a different town. Codes could be shared by writing them on a paper note, instant message, or text message.
Huntr R. explaining how “secret codes” are implemented. A
surprising amount of cryptography!
The forgotten durability of offline software
This Reddit comment thread from the GameCube subreddit was the initial inspiration for this entire series. The post is about someone's niece who just started playing Animal Crossing for the first time. The Redditor asked folks to send items to their nieces' town using the secret code system.
This ended up surprising many folks that this system still worked in a game that was over 23 years old! For reference, Nintendo Wi-Fi Connection and Nintendo Network were only available for 8 and 13 years respectively. Below are a handful of the comments from the thread:
- “That's still online???”
- “It was online???!”
- “For real does this still work lol?”
- “...Was it ever online?”
secret code for my favorite Animal Crossing NES game Wario's Woods:
Xvl5HeG&C9prXu
IWhuzBinlVlqOg
It's hard not to take these comments as indicators that something is very wrong with internet-connected software today. What had to go wrong for a system continuing to work to be met with surprise? Many consumers' experience with software products today is that they become useless e-waste after some far-away service is discontinued a few years after purchase.
My intuition from this is that software that requires centralized servers and infrastructure to function will have shorter lifetimes than software which is offline or only opportunistically uses online functionality.
I don't think this is particularly insightful, more dependencies always means less resilience. But if we're building software for human connection then the software should optimally only be limited by the availability of humans to connect.
What is centralization good for?
Data layout of secret codes before being encrypted (Animal Crossing decompilation project)
Animal Crossings' secret code system is far from perfect. The system is easily abusable, as the same secret codes can be reused over-and-over by the same user to duplicate items without ever expiring. The only limit was that 3 codes could be used per day.
Secret codes are tied to a specific town and recipient name, but even this stopgap can be defeated by setting your name and town name to specific values to share codes across many different players.
Not long after Animal Crossing's release the secret code algorithm was reverse-engineered so secret codes for any item could be created for any town and recipient name as if they came from an official Nintendo distribution. This was possible because the secret code system relied on "security through obscurity".
Could centralization be the answer to preventing these abuses?
The most interesting property that a centralized authority approach provides is global consensus: forcing everyone to play by the same rules. By storing the “single source-of-truth” a central authority is able to prevent abuses like the ones mentioned above.
For example, a centralized “secret code issuing server” could generate new unique codes per-use and check each code's validity against a database to prevent users from generating their own illegitimate codes or codes being re-used multiple times.
The problem with centralized consensus is it tends to be viral to cover the entire software state. A centralized server can generate codes perfectly, but how can that same server know that the items you're exchanging for codes were obtained legitimately? To know this the server would also need to track item legitimacy, leading to software which requires an internet connection to operate.
This is optimal from a correctness perspective, but as was noted earlier, I suspect that if such a server was a mandatory part of the secret code system in Animal Crossing that the system would likely not be usable today.
This seems like a trade-off, which future would you rather have?
Redesigning Animal Crossing secret codes
If I were designing Animal Crossings' secret code system with modern hardware, what would it look like? How can we keep the offline fall-back while providing consensus and being less abusable, especially for official distributions.
I would likely use a public-key cryptographic system for official distributions, embedding a certificate that could be used to “verify” that specific secret codes originated from the expected centralized entity. Codes that are accepted would be recorded to prevent reusing the same code multiple times in the same town. Using public-key cryptography prevents the system from being reverse-engineered to distribute arbitrary items until the certificate private key was cracked.
For sharing items between players I would implement a system where each town generated a public and private key and the public key was shared to other towns whenever the software was able to, such as when a player visited the other town. Players would only be able to send items to players that they have visited (which for Animal Crossing required physical presence, more on this later!)
Each sender could store a nonce value for each potential recipient. Embedding that nonce into the secret code would allow the recipients' software to verify that the specific code hadn't been used yet. The nonce wouldn't have to be long to avoid simple reusing of codes.
Both above systems would require much more data to be embedded into each “secret code” compared to the 28-character codes from the GameCube. For this I would use QR codes to embed over 2KB of data into a single QR code. Funnily enough, Animal Crossing New Leaf and onwards use QR code technology for players to share design patterns.
This design is still abusable if users can modify their software or hardware but doesn't suffer from the trivial-to-exploit flaws of Animal Crossing's secret code system.
Decentralized global consensus?
What if we could have the best of both worlds: we want consensus that is both global and decentralized. At least today, we are out of luck.
Decentralized global consensus is technologically feasible, but the existing solutions (mostly blockchains) are expensive (both in energy and capital) and can't handle throughput on any sort of meaningful scale.
Pick two: Decentralized, Global, and Efficient
There are many other decentralized consensus systems that are able to form “pockets” of useful peer-to-peer consensus using a fraction of the resources, such as email, BitTorrent, ActivityPub, and Nostr. These systems are only possible by adding some centralization or by only guaranteeing local consensus.
When is global consensus needed?
Obviously global consensus is important for certain classes of software like financial, civics, and infrastructure, but I wonder how the necessity of consensus in software changes for software with different risk profiles.
For software which has fewer risks associated with misuse is there as much need for global consensus? How can software for connection be designed to reduce risk and require less consensus to be effective? If global consensus and centralized servers become unnecessary, can we expect software for connection to be usable on much longer timescales, essentially for as long as there are users?
February 11, 2025 12:00 AM UTC
Quansight Labs Blog
PEP 517 build system popularity
Analysis of PEP 517 build backends used in 8000 top PyPI packages
February 11, 2025 12:00 AM UTC
February 10, 2025
Python Morsels
Newlines and escape sequences in Python
Python allows us to represent newlines in strings using the \n
"escape sequence" and Python uses line ending normalization when reading and writing with files.
![](https://i.vimeocdn.com/filter/overlay?src0=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F1977087207-0c74471d1dbc86bcf57a09f6e904a1675d7e8d9f7d286f62f34276e569301211-d_1920x1080&src1=http%3A%2F%2Ff.vimeocdn.com%2Fp%2Fimages%2Fcrawler_play.png)
Table of contents
Newline characters
This string contains a newline character:
>>> text = "Hello\nworld"
>>> text
'Hello\nworld'
That's what \n
represents: a newline character.
If we print this string, we'll see that \n
becomes an actual newline:
>>> print(text)
Hello
world
Why does Python represent a newline as \n
?
Escape sequences in Python
Every character in a Python …
Read the full article: https://www.pythonmorsels.com/newlines-and-escape-sequences/
February 10, 2025 03:17 PM UTC
Real Python
How to Join Strings in Python
Python’s built-in string method .join()
lets you combine string elements from an iterable into a single string, using a separator that you specify. You call .join()
on the separator, passing the iterable of strings to join.
By the end of this tutorial, you’ll understand that:
- You use
.join()
in Python to combine string elements with a specified separator. - A separator is the piece of text you want inserted between each substring.
- To join list elements, you call
.join()
on a separator string, passing the list as the argument. .join()
inserts the separator between each list element to form a single string.- The
.join()
method returns a new string that is the concatenation of the elements in the iterable, separated by the specified string. - For smaller string concatenation tasks, you can use the concatenation operator (
+
) or f-strings instead of.join()
.
Python’s built-in str.join()
method gives you a quick and reliable way to combine multiple strings into a single string. Whether you need to format output or assemble data for storage, .join()
provides a clean and efficient approach for joining strings from an iterable.
In the upcoming sections, you’ll learn the basic usage of .join()
to concatenate strings effectively. You’ll then apply that knowledge to real-world scenarios, from building CSV files to constructing custom log outputs. You’ll also discover some surprising pitfalls and learn how to avoid them.
Get Your Code: Click here to download the free sample code that shows you how to join strings in Python.
Take the Quiz: Test your knowledge with our interactive “How to Join Strings in Python” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
How to Join Strings in PythonTest your understanding of Python's .join() string method for combining strings, handling edge cases, and optimizing performance.
How to Join Strings in Python Using .join()
To use the string method .join()
, you call .join()
on a separator string and pass an iterable of other strings as the argument. The method returns a single string, where it has inserted the separator string between each element of the iterable:
>>> words = ["Time", "flies", "like", "an", "arrow!"]
>>> " ".join(words)
'Time flies like an arrow!'
In this example, you joined a list of words into one sentence, separated by spaces.
At first glance, this usage might look a little backward. In many other string operations, you call the method on the main string that you want to manipulate. However, with .join()
, you call the method on the separator string, then pass the iterable of strings that you want to combine:
>>> separator = " "
>>> separator.join(words)
'Time flies like an arrow!'
This example achieves the same result as the earlier one but splits the process into two steps. Defining separator
separately makes the code more readable and avoids the potentially odd-looking syntax of calling .join()
directly on a short string literal.
Note: Remember that .join()
is a string method, which means that you’ll need to call it on a single string object. Keeping that in mind may help you remember why you need to call it on the separator string.
You rarely see code that’s written in multiple steps where you assign the separator string to a variable, like you did in the example above.
In typical usage, you call .join()
directly on the separator string, all in one line. This approach is more concise and highlights that any valid string can be your separator, whether it’s whitespace, a dash, or a multicharacter substring.
Join With an Empty String
What if you don’t want any separator at all, but just want to concatenate the items? One valid approach is to use an empty string (""
) as the separator:
>>> letters = ["A", "B", "C", "D"]
>>> "".join(letters)
'ABCD'
This code snippet concatenates the letters in the list, forming a single string "ABCD"
. Using an empty string as the separator is a great way to assemble strings without a delimiter between them.
Combine Strings of Characters
Since .join()
can take any iterable of strings—not just lists—you can even pass a string as an argument. Because strings are iterable, Python iterates over each character in that string, considering each character as a separate element:
>>> characters = "ABCD"
>>> ",".join(characters)
'A,B,C,D'
By calling .join()
on ","
and passing the string characters
, you effectively place a comma between every single character in "ABCD"
. This might not always be what you intend, but it’s a neat trick to keep in mind if you ever need to treat each character as a separate element.
Read the full article at https://realpython.com/python-join-string/ »
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
February 10, 2025 02:00 PM UTC
Quansight Labs Blog
Two years of contributions to conda-forge: work done during our CZI EOSS 5 grant
In 2022 we were awarded a CZI EOSS grant for conda-forge. The proposal, co-submitted by Quansight Labs and QuantStack, targeted three areas: maintaining and improving conda-forge's infrastructure, creating a new maintainer's dashboard, and implementing OCI-based mirroring for the packages. This work has now concluded and we would like to publish a summary of what we achieved!
February 10, 2025 12:00 AM UTC
meejah.ca
Terminal Visualizer
Showing numbers is fun and hard
February 10, 2025 12:00 AM UTC
February 09, 2025
Anarcat
A slow blogging year
Well, 2024 will be remembered, won't it? I guess 2025 already wants to make its mark too, but let's not worry about that right now, and instead let's talk about me.
A little over a year ago, I was gloating over how I had such a great blogging year in 2022, and was considering 2023 to be average, then went on to gather more stats and traffic analysis... Then I said, and I quote:
I hope to write more next year. I've been thinking about a few posts I could write for work, about how things work behind the scenes at Tor, that could be informative for many people. We run a rather old setup, but things hold up pretty well for what we throw at it, and it's worth sharing that with the world...
What a load of bollocks.
A bad year for this blog
2024 was the second worst year ever in my blogging history, tied with 2009 at a measly 6 posts for the year:
anarcat@angela:anarc.at$ curl -sSL https://anarc.at/blog/ | grep 'href="\./' | grep -o 20[0-9][0-9] | sort | uniq -c | sort -nr | grep -v 2025 | tail -3
6 2024
6 2009
3 2014
I did write about my work though, detailing the migration from Gitolite to GitLab we completed that year. But after August, total radio silence until now.
Loads of drafts
It's not that I have nothing to say: I have no less than five drafts in my working tree here, not counting three actual drafts recorded in the Git repository here:
anarcat@angela:anarc.at$ git s blog
## main...origin/main
?? blog/bell-bot.md
?? blog/fish.md
?? blog/kensington.md
?? blog/nixos.md
?? blog/tmux.md
anarcat@angela:anarc.at$ git grep -l '\!tag draft'
blog/mobile-massive-gallery.md
blog/on-dying.mdwn
blog/secrets-recovery.md
I just don't have time to wrap those things up. I think part of me is disgusted by seeing my work stolen by large corporations to build proprietary large language models while my idols have been pushed to suicide for trying to share science with the world.
Another part of me wants to make those things just right. The "tagged drafts" above are nothing more than a huge pile of chaotic links, far from being useful for anyone else than me, and even then.
The on-dying
article, in particular, is becoming my nemesis. I've
been wanting to write that article for over 6 years now, I think. It's
just too hard.
Writing elsewhere
There's also the fact that I write for work already. A lot. Here are the top-10 contributors to our team's wiki:
anarcat@angela:help.torproject.org$ git shortlog --numbered --summary --group="format:%al" | head -10
4272 anarcat
423 jerome
117 zen
116 lelutin
104 peter
58 kez
45 irl
43 hiro
18 gaba
17 groente
... but that's a bit unfair, since I've been there half a decade. Here's the last year:
anarcat@angela:help.torproject.org$ git shortlog --since=2024-01-01 --numbered --summary --group="format:%al" | head -10
827 anarcat
117 zen
116 lelutin
91 jerome
17 groente
10 gaba
8 micah
7 kez
5 jnewsome
4 stephen.swift
So I still write the most commits! But to truly get a sense of the amount I wrote in there, we should count actual changes. Here it is by number of lines (from commandlinefu.com):
anarcat@angela:help.torproject.org$ git ls-files | xargs -n1 git blame --line-porcelain | sed -n 's/^author //p' | sort -f | uniq -ic | sort -nr | head -10
99046 Antoine Beaupré
6900 Zen Fu
4784 Jérôme Charaoui
1446 Gabriel Filion
1146 Jerome Charaoui
837 groente
705 kez
569 Gaba
381 Matt Traudt
237 Stephen Swift
That, of course, is the entire history of the git repo, again. We
should take only the last year into account, and probably ignore the
tails
directory, as sneaky Zen Fu imported the entire docs from
another wiki there...
anarcat@angela:help.torproject.org$ find [d-s]* -type f -mtime -365 | xargs -n1 git blame --line-porcelain 2>/dev/null | sed -n 's/^author //p' | sort -f | uniq -ic | sort -nr | head -10
75037 Antoine Beaupré
2932 Jérôme Charaoui
1442 Gabriel Filion
1400 Zen Fu
929 Jerome Charaoui
837 groente
702 kez
569 Gaba
381 Matt Traudt
237 Stephen Swift
Pretty good! 75k lines. But those are the files that were modified in the last year. If we go a little more nuts, we find that:
anarcat@angela:help.torproject.org$ $ git-count-words-range.py | sort -k6 -nr | head -10
parsing commits for words changes from command: git log '--since=1 year ago' '--format=%H %al'
anarcat 126116 - 36932 = 89184
zen 31774 - 5749 = 26025
groente 9732 - 607 = 9125
lelutin 10768 - 2578 = 8190
jerome 6236 - 2586 = 3650
gaba 3164 - 491 = 2673
stephen.swift 2443 - 673 = 1770
kez 1034 - 74 = 960
micah 772 - 250 = 522
weasel 410 - 0 = 410
I wrote 126,116 words in that wiki, only in the last year. I also deleted 37k words, so the final total is more like 89k words, but still: that's about forty (40!) articles of the average size (~2k) I wrote in 2022.
(And yes, I did go nuts and write a new log parser, essentially from scratch, to figure out those word diffs. I did get the courage only after asking GPT-4o for an example first, I must admit.)
Let's celebrate that again: I wrote 90 thousand words in that wiki in 2024. According to Wikipedia, a "novella" is 17,500 to 40,000 words, which would mean I wrote about a novella and a novel, in the past year.
But interestingly, if I look at the repository analytics. I certainly didn't write that much more in the past year. So that alone cannot explain the lull in my production here.
Arguments
Another part of me is just tired of the bickering and arguing on the internet. I have at least two articles in there that I suspect is going to get me a lot of push-back (NixOS and Fish). I know how to deal with this: you need to write well, consider the controversy, spell it out, and defuse things before they happen. But that's hard work and, frankly, I don't really care that much about what people think anymore.
I'm not writing here to convince people. I have stop evangelizing a long time ago. Now, I'm more into documenting, and teaching. And, while teaching, there's a two-way interaction: when you give out a speech or workshop, people can ask questions, or respond, and you all learn something. When you document, you quickly get told "where is this? I couldn't find it" or "I don't understand this" or "I tried that and it didn't work" or "wait, really? shouldn't we do X instead", and you learn.
Here, it's static. It's my little soapbox where I scream in the void. The only thing people can do is scream back.
Collaboration
So.
Let's see if we can work together here.
If you don't like something I say, disagree, or find something wrong or to be improved, instead of screaming on social media or ignoring me, try contributing back. This site here is backed by a git repository and I promise to read everything you send there, whether it is an issue or a merge request.
I will, of course, still read comments sent by email or IRC or social media, but please, be kind.
You can also, of course, follow the latest changes on the TPA wiki. If you want to catch up with the last year, some of the "novellas" I wrote include:
- TPA-RFC-33: Monitoring: Nagios to Prometheus conversion, see also the extensive Prometheus documentation we wrote
- TPA-RFC-45: email architecture: draft of long-term email
services at
torproject.org
- TPA-RFC-62: TPA password manager: switch to password-store
- TPA-RFC-63: storage server budget: buy a new backup storage server (5k$ + 100$/mth)
- TPA-RFC-65: PostgreSQL backups: switching from legacy to pgBackRest for database backups
- TPA-RFC-68: Idle canary servers: provision test servers that sit idle to monitor infrastructure and stage deployments
- TPA-RFC-71: Emergency email deployments, phase B: deploy a new sender-rewriting mail forwarder, migrate mailing lists off the legacy server to a new machine, migrate the remaining Schleuder list to the Tails server, upgrade eugeni.
- TPA-RFC-76: Puppet merge request workflow: how to mirror our private Puppet repo to GitLab safely
- TPA-RFC-79: General merge request workflows: how to use merge requests, assignees, reviewers, draft and threads on GitLab projects
(Well, no, you can't actually follow changes on a GitLab wiki. But we have a wiki-replica git repository where you can see the latest commits, and subscribe to the RSS feed.)
See you there!
February 09, 2025 04:19 PM UTC
Talk Python to Me
#493: Quarto: Open-source technical publishing
In this episode, I'm joined by JJ Allaire, founder and executive chairman at Posit, and Carlos Scheidegger, a software engineer at Posit, to explore Quarto, an open-source tool revolutionizing technical publishing. We discuss how Quarto empowers users to seamlessly transform Jupyter notebooks into polished reports, dashboards, e-books, websites, and more. JJ shares his journey from creating RStudio to developing Quarto as a versatile, multi-language tool, while Carlos delves into its roots in reproducibility and the challenges of academic publishing. Don't miss this deep dive into a tool that's shaping the future of data-driven storytelling!<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br> <a href='https://talkpython.fm/digitalocean-partner'>DigitalOcean</a><br/> <br/> <h2>Links from the show</h2> <div><strong>JJ Allaire</strong><br/> <strong>JJ on LinkedIn</strong>: <a href="https://www.linkedin.com/in/jjallaire/?featured_on=talkpython" target="_blank" >linkedin.com</a><br/> <strong>JJ on GitHub</strong>: <a href="https://github.com/jjallaire?featured_on=talkpython" target="_blank" >github.com</a><br/> <br/> <strong>Carlos Scheidegger</strong><br/> <strong>Personal site</strong>: <a href="https://cscheid.net?featured_on=talkpython" target="_blank" >cscheid.net</a><br/> <strong>Mastodon</strong>: <a href="https://mastodon.social/@scheidegger?featured_on=talkpython" target="_blank" >@scheidegger</a><br/> <br/> <strong>Fast AI</strong>: <a href="https://fast.ai?featured_on=talkpython" target="_blank" >fast.ai</a><br/> <strong>nbdev</strong>: <a href="https://nbdev.fast.ai/?featured_on=talkpython" target="_blank" >nbdev.fast.ai</a><br/> <strong>nbsanity - Share Notebooks as Polished Web Pages in Seconds</strong>: <a href="https://www.answer.ai/posts/2024-12-13-nbsanity.html?featured_on=talkpython" target="_blank" >answer.ai</a><br/> <strong>Pandoc</strong>: <a href="https://pandoc.org/?featured_on=talkpython" target="_blank" >pandoc.org</a><br/> <strong>Observable</strong>: <a href="https://github.com/observablehq/framework?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>Quarto Pub</strong>: <a href="https://quartopub.com/?featured_on=talkpython" target="_blank" >quartopub.com</a><br/> <strong>Deno</strong>: <a href="https://deno.com/?featured_on=talkpython" target="_blank" >deno.com</a><br/> <strong>Real World Data Science site</strong>: <a href="https://realworlddatascience.net/?featured_on=talkpython" target="_blank" >realworlddatascience.net</a><br/> <strong>Typst</strong>: <a href="https://typst.app/?featured_on=talkpython" target="_blank" >typst.app</a><br/> <strong>Github Actions for Quarto</strong>: <a href="https://github.com/quarto-dev/quarto-actions?featured_on=talkpython" target="_blank" >github.com</a><br/> <strong>Watch this episode on YouTube</strong>: <a href="https://www.youtube.com/watch?v=AXkwy8dzCrA" target="_blank" >youtube.com</a><br/> <strong>Episode transcripts</strong>: <a href="https://talkpython.fm/episodes/transcript/493/quarto-open-source-technical-publishing" target="_blank" >talkpython.fm</a><br/> <br/> <strong>--- Stay in touch with us ---</strong><br/> <strong>Subscribe to Talk Python on YouTube</strong>: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a><br/> <strong>Talk Python on Bluesky</strong>: <a href="https://bsky.app/profile/talkpython.fm" target="_blank" >@talkpython.fm at bsky.app</a><br/> <strong>Talk Python on Mastodon</strong>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <strong>Michael on Bluesky</strong>: <a href="https://bsky.app/profile/mkennedy.codes?featured_on=talkpython" target="_blank" >@mkennedy.codes at bsky.app</a><br/> <strong>Michael on Mastodon</strong>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>
February 09, 2025 08:00 AM UTC
Anarcat
Qalculate hacks
This is going to be a controversial statement because some people are absolute nerds about this, but, I need to say it.
Qalculate is the best calculator that has ever been made.
I am not going to try to convince you of this, I just wanted to put out my bias out there before writing down those notes. I am a total fan.
This page will collect my notes of cool hacks I do with
Qalculate. Most examples are copy-pasted from the command-line
interface (qalc(1)
), but I typically use the graphical interface as
it's slightly better at displaying complex formulas. Discoverability
is obviously also better for the cornucopia of features this fantastic
application ships.
Qalc commandline primer
On Debian, Qalculate's CLI interface can be installed with:
apt install qalc
Then you start it with the qalc
command, and end up on a prompt:
anarcat@angela:~$ qalc
>
Then it's a normal calculator:
anarcat@angela:~$ qalc
> 1+1
1 + 1 = 2
> 1/7
1 / 7 ≈ 0.1429
> pi
pi ≈ 3.142
>
There's a bunch of variables to control display, approximation, and so on:
> set precision 6
> 1/7
1 / 7 ≈ 0.142857
> set precision 20
> pi
pi ≈ 3.1415926535897932385
When I need more, I typically browse around the menus. One big issue I
have with Qalculate is there are a lot of menus and features. I had
to fiddle quite a bit to figure out that set precision
command
above. I might add more examples here as I find them.
Bandwidth estimates
I often use the data units to estimate bandwidths. For example, here's what 1 megabit per second is over a month ("about 300 GiB"):
> 1 megabit/s * 30 day to gibibyte
(1 megabit/second) × (30 days) ≈ 301.7 GiB
Or, "how long will it take to download X", in this case, 1GiB over a 100 mbps link:
> 1GiB/(100 megabit/s)
(1 gibibyte) / (100 megabits/second) ≈ 1 min + 25.90 s
Password entropy
To calculate how much entropy (in bits) a given password structure,
you count the number of possibilities in each entry (say, [a-z]
is
26 possibilities, "one word in a 8k dictionary" is 8000), extract the
base-2 logarithm, multiplied by the number of entries.
For example, an alphabetic 14-character password is:
> log2(26*2)*14
log₂(26 × 2) × 14 ≈ 79.81
... 80 bits of entropy. To get the equivalent in a Diceware password with a 8000 word dictionary, you would need:
> log2(8k)*x = 80
(log₂(8 × 000) × x) = 80 ≈
x ≈ 6.170
... about 6 words, which gives you:
> log2(8k)*6
log₂(8 × 1000) × 6 ≈ 77.79
78 bits of entropy.
Exchange rates
You can convert between currencies!
> 1 EUR to USD
1 EUR ≈ 1.038 USD
Even fake ones!
> 1 BTC to USD
1 BTC ≈ 96712 USD
This relies on a database pulled form the internet (typically the central european bank rates, see the source). It will prompt you if it's too old:
It has been 256 days since the exchange rates last were updated.
Do you wish to update the exchange rates now? y
As a reader pointed out, you can set the refresh rate for currencies, as some countries will require way more frequent exchange rates.
The graphical version has a little graphical indicator that, when you mouse over, tells you where the rate comes from.
Other conversions
Here are other neat conversions extracted from my history
> teaspoon to ml
teaspoon = 5 mL
> tablespoon to ml
tablespoon = 15 mL
> 1 cup to ml
1 cup ≈ 236.6 mL
> 6 L/100km to mpg
(6 liters) / (100 kilometers) ≈ 39.20 mpg
> 100 kph to mph
100 kph ≈ 62.14 mph
> (108km - 72km) / 110km/h
((108 kilometers) − (72 kilometers)) / (110 kilometers/hour) ≈
19 min + 38.18 s
Completion time estimates
This is a more involved example I often do.
Background
Say you have started a long running copy job and you don't have the
luxury of having a pipe you can insert pv(1) into to get a nice
progress bar. For example, rsync
or cp -R
can have that problem
(but not tar
!).
(Yes, you can use --info=progress2
in rsync
, but that estimate is
incremental and therefore inaccurate unless you disable the
incremental mode with --no-inc-recursive
, but then you pay a huge
up-front wait cost while the entire directory gets crawled.)
Extracting a process start time
First step is to gather data. Find the process start time. If you were
unfortunate enough to forget to run date --iso-8601=seconds
before
starting, you can get a similar timestamp with stat(1)
on the
process tree in /proc
with:
$ stat /proc/11232
File: /proc/11232
Size: 0 Blocks: 0 IO Block: 1024 directory
Device: 0,21 Inode: 57021 Links: 9
Access: (0555/dr-xr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2025-02-07 15:50:25.287220819 -0500
Modify: 2025-02-07 15:50:25.287220819 -0500
Change: 2025-02-07 15:50:25.287220819 -0500
Birth: -
So our start time is 2025-02-07 15:50:25
, we shave off the
nanoseconds there, they're below our precision noise floor.
If you're not dealing with an actual UNIX process, you need to figure out a start time: this can be a SQL query, a network request, whatever, exercise for the reader.
Saving a variable
This is optional, but for the sake of demonstration, let's save this as a variable:
> start="2025-02-07 15:50:25"
save("2025-02-07T15:50:25"; start; Temporary; ; 1) =
"2025-02-07T15:50:25"
Estimating data size
Next, estimate your data size. That will vary wildly with the job
you're running: this can be anything: number of files, documents being
processed, rows to be destroyed in a database, whatever. In this case,
rsync
tells me how many bytes it has transferred so far:
# rsync -ASHaXx --info=progress2 /srv/ /srv-zfs/
2.968.252.503.968 94% 7,63MB/s 6:04:58 xfr#464440, ir-chk=1000/982266)
Strip off the weird dots in there, because that will confuse qalculate, which will count this as:
2.968252503968 bytes ≈ 2.968 B
Or, essentially, three bytes. We actually transferred almost 3TB here:
2968252503968 bytes ≈ 2.968 TB
So let's use that. If you had the misfortune of making rsync silent,
but were lucky enough to transfer entire partitions, you can use df
(without -h
! we want to be more precise here), in my case:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_hdd-srv 7512681384 7258298036 179205040 98% /srv
tank/srv 7667173248 2870444032 4796729216 38% /srv-zfs
(Otherwise, of course, you use du -sh $DIRECTORY
.)
Digression over bytes
Those are 1 K
bytes which is actually (and rather unfortunately)
Ki
, or "kibibytes" (1024 bytes), not "kilobytes" (1000 bytes). Ugh.
> 2870444032 KiB
2870444032 kibibytes ≈ 2.939 TB
> 2870444032 kB
2870444032 kilobytes ≈ 2.870 TB
At this scale, those details matter quite a bit, we're talking about a 69GB (64GiB) difference here:
> 2870444032 KiB - 2870444032 kB
(2870444032 kibibytes) − (2870444032 kilobytes) ≈ 68.89 GB
Anyways. Let's take 2968252503968 bytes
as our current progress.
Our entire dataset is 7258298064 KiB
, as seen above.
Solving a cross-multiplication
We have 3 out of four variables for our equation here, so we can already solve:
> (now-start)/x = (2996538438607 bytes)/(7258298064 KiB) to h
((actual − start) / x) = ((2996538438607 bytes) / (7258298064
kibibytes))
x ≈ 59.24 h
The entire transfer will take about 60 hours to complete! Note that's not the time left, that is the total time.
To break this down step by step, we could calculate how long it has taken so far:
> now-start
now − start ≈ 23 h + 53 min + 6.762 s
> now-start to s
now − start ≈ 85987 s
... and do the cross-multiplication manually, it's basically:
x/(now-start) = (total/current)
so:
x = (total/current) * (now-start)
or, in Qalc:
> ((7258298064 kibibytes) / ( 2996538438607 bytes) ) * 85987 s
((7258298064 kibibytes) / (2996538438607 bytes)) × (85987 secondes) ≈
2 d + 11 h + 14 min + 38.81 s
It's interesting it gives us different units here! Not sure why.
Now and built-in variables
The now
here is actually a built-in variable:
> now
now ≈ "2025-02-08T22:25:25"
There is a bewildering list of such variables, for example:
> uptime
uptime = 5 d + 6 h + 34 min + 12.11 s
> golden
golden ≈ 1.618
> exact
golden = (√(5) + 1) / 2
Computing dates
In any case, yay! We know the transfer is going to take roughly 60 hours total, and we've already spent around 24h of that, so, we have 36h left.
But I did that all in my head, we can ask more of Qalc yet!
Let's make another variable, for that total estimated time:
> total=(now-start)/x = (2996538438607 bytes)/(7258298064 KiB)
save(((now − start) / x) = ((2996538438607 bytes) / (7258298064
kibibytes)); total; Temporary; ; 1) ≈
2 d + 11 h + 14 min + 38.22 s
And we can plug that into another formula with our start time to figure out when we'll be done!
> start+total
start + total ≈ "2025-02-10T03:28:52"
> start+total-now
start + total − now ≈ 1 d + 11 h + 34 min + 48.52 s
> start+total-now to h
start + total − now ≈ 35 h + 34 min + 32.01 s
That transfer has ~1d left, or 35h24m32s, and should complete around 4 in the morning on February 10th.
But that's icing on top. I typically only do the cross-multiplication and calculate the remaining time in my head.
I mostly did the last bit to show Qalculate could compute dates and time differences, as long as you use ISO timestamps. Although it can also convert to and from UNIX timestamps, it cannot parse arbitrary date strings (yet?).
Other functionality
Qalculate can:
- Plot graphs;
- Use RPN input;
- Do all sorts of algebraic, calculus, matrix, statistics, trigonometry functions (and more!);
- ... and so much more!
I have a hard time finding things it cannot do. When I get there, I typically need to resort to programming code in Python, use a spreadsheet, and others will turn to more complete engines like Maple, Mathematica or R.
But for daily use, Qalculate is just fantastic.
And it's pink! Use it!
Further reading and installation
This is just scratching the surface, the fine manual has more information, including more examples. There is also of course a qalc(1) manual page which also ships an excellent EXAMPLES section.
Qalculate is packaged for over 30 Linux distributions, but also ships packages for Windows and MacOS. There are third-party derivatives as well including a web version and an Android app.
February 09, 2025 04:09 AM UTC
February 08, 2025
Armin Ronacher
Seeking Purity
The concept of purity — historically a guiding principle in social and moral contexts — is also found in passionate, technical discussions. By that I mean that purity in technology translates into adherence to a set of strict principles, whether it be functional programming, test-driven development, serverless architectures, or, in the case of Rust, memory safety.
Memory Safety
Rust positions itself as a champion of memory safety, treating it as a non-negotiable foundation of good software engineering. I love Rust: it's probably my favorite language. It probably won't surprise you that I have no problem with it upholding memory safety as a defining feature.
Rust aims to achieve the goal of memory safety via safe abstractions, a compile time borrow checker and a type system that is in service of those safe abstractions. It comes as no surprise that the Rust community is also pretty active in codifying a new way to reason about pointers. In many ways, Rust pioneered completely new technical approaches and it it widely heralded as an amazing innovation.
However, as with many movements rooted in purity, what starts as a technical pursuit can evolve into something more ideological. Similar to how moral purity in political and cultural discourse can become charged, so does the discourse around Rust, which has been dominated by the pursuit of memory safety. Particularly within the core Rust community itself, discussion has moved beyond technical merits into something akin to ideological warfare. The fundamental question of “Is this code memory safe?”, has shifted to “Was it made memory safe in the correct way?”. This distinction matters because it introduces a purity test that values methodology over outcomes. Safe C code, for example, is often dismissed as impossible, not necessarily because it is impossible, but because it lacks the strict guarantees that Rust's borrow checker enforces. Similarly, using Rust’s unsafe blocks is increasingly frowned upon, despite their intended purpose of enabling low-level optimizations when necessary.
This ideological rigidity creates significant friction when Rust interfaces with other ecosystems (or gets introduced there), particularly those that do not share its uncompromising stance. For instance, the role of Rust in the Linux kernel has been a hot topic. The Linux kernel operates under an entirely different set of priorities. While memory safety is important there is insufficient support for adopting Rust in general. The kernel is an old project and it aims to remain maintainable for a long time into the future. For it to even consider a rather young programming language should be seen as tremendous success for Rust and also for how open Linus is to the idea.
Yet that introduction is balanced against performance, maintainability, and decades of accumulated engineering expertise. Many of the kernel developers, who have found their own strategies to write safe C for decades, are not accepting the strongly implied premise that their work is inherently flawed simply because it does not adhere to Rust's strict purity rules.
Tensions rose when a kernel developer advocating for Rust's inclusion took to social media to push for changes in the Linux kernel development process. The public shaming tactic failed, leading the developer to conclude:
“If shaming on social media does not work, then tell me what does, because I'm out of ideas.”
It's not just the kernel where Rust's memory safety runs up against the complexities of the real world. Very similar feelings creep up in the gaming industry where people love to do wild stuff with pointers. You do not need large disagreements to see the purist approach create some friction. A recent post of mine for instance triggered some discussions about the trade-offs between more dependencies, and moving unsafe to centralized crates.
I really appreciate that Rust code does not crash as much. That part of Rust, among many others, makes it very enjoyable to work with. Yet I am entirely unconvinced that memory safety should trump everything, at least at this point in time.
What people want in the Rust in Linux situation is for the project leader to come in to declare support for Rust's call for memory safety above all. To make the detractors go away.
Python's Migration Lesson
Hearing this call and discussion brings back memories. I have lived through a purity driven shift in a community before. The move from Python 2 to Python 3 started out very much the same way. There was an almost religious movement in the community to move to Python 3 in a ratcheting motion. The idea that you could maintain code bases that support both 2 and 3 were initially very loudly rejected. I took a lot of flak at the time (and for years after) for advocating for a more pragmatic migration which burned me out a lot. That feedback came both in person and online and it largely pushed me away from Python for a while. Not getting behind the Python 3 train was seen as sabotaging the entire project. However, a decade later, I feel somewhat vindicated that it was worth being pragmatic about that migration.
At the root of that discourse was a idealistic view of how Unicode could work in the language and that you can move an entire ecosystem at once. Both those things greatly clashed with the lived realities in many projects and companies.
I am a happy user of Python 3 today. This migration has also taught me the important lesson not be too stuck on a particular idea. It would have been very easy to pick one of the two sides of that debate. Be stuck on Python 2 (at the risk of forking), or go all in on Python 3 no questions asked. It was the path in between that was quite painful to advocate for, but it was ultimately the right path. I wrote about my lessons of that migration a in 2016 and I think most of this still rings true. That was motivated by even years later people still reaching out to me who did not move to Python 3, hoping for me to embrace their path. Yet Python 3 has changed! Python 3 is a much better language than it was when it first released. It is a great language because it's used by people solving real, messy problems and because it over time found answers for what to do, if you need to have both Python 2 and 3 code in the wild. While the world of Python 2 is largely gone, we are still in a world where Unicode and bytes mix in certain contexts.
The Messy Process
Fully committing to a single worldview can be easier because you stop questioning everything — you can just go with the flow. Yet truths often reside on both sides. Allowing yourself to walk the careful middle path enables you to learn from multiple perspectives. You will face doubts and open yourself up to vulnerability and uncertainty. The payoff, however, is the ability to question deeply held beliefs and push into the unknown territory where new things can be found. You can arrive at a solution that isn't a complete rejection of any side. There is genuine value in what Rust offers—just as there was real value in what Python 3 set out to accomplish. But the Python 3 of today isn't the Python 3 of those early, ideological debates; it was shaped by a messy, slow, often contentious, yet ultimately productive transition process.
I am absolutely sure that in 30 years from now we are going to primarily program in memory safe languages (or the machines will do it for us) in environments where C and C++ prevail. That glimpse of a future I can visualize clearly. The path to there however? That's a different story altogether. It will be hard, it will be impure. Maybe the solution will not even involve Rust at all — who knows.
We also have to accept that not everyone is ready for change at the same pace. Forcing adoption when people aren't prepared only causes the pendulum to swing back hard. It's tempting to look for a single authority to declare “the one true way,” but that won't smooth out the inevitable complications. Indeed, those messy, incremental challenges are part of how real progress happens. In the long run, these hard-won refinements tend to produce solutions that benefit all sides—if we’re patient enough to let them take root. The painful and messy transition is here to stay, and that's exactly why, in the end, it works.
February 08, 2025 12:00 AM UTC
February 07, 2025
ListenData
4 Ways to Use ChatGPT API in Python
In this tutorial, we will explain how to use ChatGPT API in Python, along with examples.
Please follow the steps below to access the ChatGPT API.
- Visit the OpenAI Platform and sign up using your Google, Microsoft or Apple account.
- After creating your account, the next step is to generate a secret API key to access the API. The API key looks like this -
sk-xxxxxxxxxxxxxxxxxxxx
- If your phone number has not been associated with any other OpenAI account previously, you may get free credits to test the API. Otherwise you have to add atleast 5 dollars into your account and charges will be based on the usage and the type of model you use. Check out the pricing details in the OpenAI website.
- Now you can call the API using the code below.
February 07, 2025 04:11 PM UTC
Real Python
The Real Python Podcast – Episode #238: Charlie Marsh: Accelerating Python Tooling With Ruff and uv
Are you looking for fast tools to lint your code and manage your projects? How is the Rust programming language being used to speed up Python tools? This week on the show, we speak with Charlie Marsh about his company, Astral, and their tools, uv and Ruff.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
February 07, 2025 12:00 PM UTC
Test and Code
Mocking in Python with unittest.mock - Michael Foord
This episode is a replay of a 2021 interview I did with Michael Foord.
We lost Michael in January, and I'd like to revisit this interview as a tribute.
Michael Foord was a pivotal figure in the Python community and the creator of the mock library that's now unittest.mock.
But he did so much more as well.
His contributions continue to resonate within the developer community.
This interview is just a small peek at his influence.
In this episode
- Introduction to Michael Foord
- The Mock Library Origins
- Mocking and Testing Philosophy
- Career Path and Consulting
- Understanding Mocking
- The Power of Patch
- Testing Strategies and Design
- Mocking External Dependencies
- Teaching Testing and Mocking
python.org has put up an "In memoriam" page for Michael Foord, and many people have shared stories and memories.
Links:
February 07, 2025 07:00 AM UTC
Daniel Roy Greenfeld
TIL: Typer commands defaulting to help
If you save this code to cli.py
:
import typer
app = typer.Typer()
@app.command()
def create():
"""Creates a user"""
typer.echoint("Creating user")
@app.command()
def delete():
"""Deletes a user"""
typer.echo("Deleting user")
if __name__ == "__main__":
app()
and run it you get:
$ python cli.py
Usage: cli.py [OPTIONS] COMMAND [ARGS]...
Try 'cli.py --help' for help.
╭─ Error ────────────╮
│ Missing command. │
╰────────────────────╯
That's not bad, but it forces you to request help before doing anything. Here's how it can be made so much better through the @app.callback
command. See below:
import typer
app = typer.Typer()
# Setup the helper default
@app.callback(invoke_without_command=True)
def helper(ctx: typer.Context):
"""
Awesome CLI app
"""
if ctx.invoked_subcommand is None:
typer.echo(ctx.get_help())
@app.command()
def create():
"""Creates a user"""
typer.echoint("Creating user")
@app.command()
def delete():
"""Deletes a user"""
typer.echo("Deleting user")
if __name__ == "__main__":
app()
Now we get something as a default that really shows off the charm of typer
:
$ python cli.py
Usage: cli.py [OPTIONS] COMMAND [ARGS]...
Awesome CLI app
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion Install completion for the current shell. │
│ --show-completion Show completion for the current shell, to copy it or customize the installation.│
│ --help Show this message and exit. │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────╮
│ create Creates a user │
│ delete Deletes a user │
╰─────────────────────────╯
February 07, 2025 12:50 AM UTC
February 06, 2025
Python Morsels
Python Terminology: an unofficial glossary
Definitions for colloquial Python terminology (effectively an unofficial version of the Python glossary).
Table of contents
Looping
These terms are all about looping and objects you can loop over.
- Iteration
-
Looping over an iterable object.
A
for
loop is the epitome of iteration, but there are many other ways to iterate over an iterable in Python.List comprehensions iterate. Tuple unpacking iterates. Using
*
operator in a function call iterates (see Unpacking iterables into function arguments). Using*
operator in a list literal iterates (see Unpacking iterables into iterables). - Iterable
-
(Our perspective as Python users) Anything you can loop over with a
for
loop or by various other forms of iteration. More on iterables in What is an iterable. -
(Python's perspective) Anything that can be passed to the built-in
iter
function to get an iterator from it. If you're inventing your own iterable class, also see How to make an iterable.
Data Types
These terms are related to …
Read the full article: https://www.pythonmorsels.com/terms/
February 06, 2025 09:31 PM UTC
Python Engineering at Microsoft
Python in Visual Studio Code – February 2025 Release
We’re excited to announce the February 2025 release of the Python, Pylance and Jupyter extensions for Visual Studio Code!
This release includes the following announcements:
- No-config debugging
- Test discovery cancellation
- Launch the Native REPL from the terminal
- Go to Implementation with Pylance
- AI Code Action: Generate Symbol (Experimental)
If you’re interested, you can check the full list of improvements in our changelogs for the Python, Jupyter and Pylance extensions.
No-config debugging
After listening to community feedback that configuring the debugger can be difficult, we are introducing a no-config debugging experience! We are excited to say that this is now available in the Python Debugger extension!
As the name implies, this workflow allows you to start the debugger without creating or managing a Python debug configuration in launch.json
. Simply replace python
with debugpy
prefixed in your run command in the terminal, for example debugpy <script.py or module>
, to start a debug session.
NOTE: We also recommend adding the following to your User
settings.json
file:"python.experiments.optOutFrom": ["pythonTerminalEnvVarActivation"]
as this experiment is known to conflict with this new feature. You can still activate your terminals using normal activation commands.
You’ll see feedback in the terminal that a debug session is starting and the debug toolbar pop up in the editor. Set breakpoints in the UI and open up the debug toolbar to use all of VS Code’s debugging functionality.
For more information and troubleshooting tips, check out the no-config debugging wiki. We would love to hear your feedback on this new feature – you can open issues and feature requests in our repository.
Test discovery cancellation
When triggering test discovery from the Test Explorer UI, you can now cancel an ongoing test discovery call. Use the Cancel Test Refresh button, which appears in replacement of the Refresh button during discovery.
Launch the Native REPL from the terminal
You are now able to launch a VS Code Native REPL from your terminal REPL. By setting python.terminal.shellIntegration.enabled
to true
, a clickable link will display in the Python REPL in the terminal, allowing you to directly open the VS Code Native REPL from the terminal. This will eventually be set to true
by default, but for now you can enable it manually.
The Native REPL allows you to iteratively run Python code similar to the Python REPL located in the terminal enhanced with features such as IntelliSense and syntax highlighting.
Go to Implementation with Pylance
Pylance now has support for Go to Implementation, which allows you to more quickly navigate to the implementation of a function or method directly from its usage. This feature is particularly helpful when working with inherited classes.
AI Code Action: Generate Symbol (Experimental)
There’s a new experimental AI Code Action for generating symbols with Pylance and Copilot. To try it out, you can enable the following setting: "python.analysis.aiCodeActions": {"generateSymbol": true}
.
With this enabled, once you define a new symbol, such as a class, function, or variable, you can select the Generate Symbol with Copilot Code Action and let Copilot handle the implementation! Afterward, you can leverage Pylance’s Move Symbol Code Actions to relocate it to a different file.
Other Changes and Enhancements
We have also added small enhancements and fixed issues requested by users that should improve your experience working with Python and Jupyter Notebooks in Visual Studio Code. Some notable changes include:
- This release is the last release with support for Python 3.8 in all our extensions (Python extension, Python Debugger, Python Environments, and Python tooling extensions).
- The Python Environments extension APIs now support grouping environments by environment managers and allows package managers to control the common package lists, providing more flexibility to providers.
We would also like to extend special thanks to this month’s contributors:
- Use fnmatch instead of pathlibs match by @Mutantpenguin in @vscode-flake8#256
- Change the flake8 version string in
readme.md
to be pip compatible by @safiyat in @vscode-flake8#257 - Fix link in README.md by @maxg203 in @vscode-flake8#269
- Use global settings for
ignorePatterns
default by @taesungh in @vscode-flake8#327 - Workaround for memory leak caused by pylint bug by @DetachHead in @vscode-pylint#585
- Run mypy in the directory of the nearest pyproject.toml or mypy.ini by @jwhitaker-gridcog in @vscode-mypy#316
- Use global settings for
ignorePatterns
default by @taesungh in @vscode-mypy#325 - Fix: address dev-dependency issues reported by
npm audit
by @hamirmahal in @vscode-mypy#327 - Fix: usage of
node12 which is deprecated
in CI by @hamirmahal in @vscode-mypy#336 - Fix for conda listing in README.md by @jezdez in @vscode-python-environments#80
- Added note of ms-python.python pre-release requirement by @robwoods-cam in @vscode-python-environments#111
- Untildify conda path from settings by @almarouk in @vscode-python-environments#122
- Quote arguments systematically when calling conda by @almarouk in @vscode-python-environments#121
- Handle all shells’ de/activation by @flying-sheep in @vscode-python-environments#137
Try out these new improvements by downloading the Python extension and the Jupyter extension from the Marketplace, or install them directly from the extensions view in Visual Studio Code (Ctrl + Shift + X or ⌘ + ⇧ + X). You can learn more about Python support in Visual Studio Code in the documentation. If you run into any problems or have suggestions, please file an issue on the Python VS Code GitHub page.
The post Python in Visual Studio Code – February 2025 Release appeared first on Python.
February 06, 2025 09:29 PM UTC
PyPy
PyPy v7.3.18 release
PyPy v7.3.18: release of python 2.7, 3.10 and 3.11 beta
The PyPy team is proud to release version 7.3.18 of PyPy.
This release includes a python 3.11 interpreter. We are labelling it "beta" because it is the first one. In the next release we will drop 3.10 and remove the "beta" label. There are a particularly large set of bugfixes in this release thanks to @devdanzin using fusil on the 3.10 builds, originally written by Victor Stinner. Other significant changes:
We have updated libffi shipped in our portable builds. We also now statically link to libffi where possible which reduces the number of shared object dependencies.
We have added code to be able to show the native function names when profiling with VMProf. So far only Linux supports this feature.
We have added a PEP 768-inspired remote debugging facility.
The HPy backend has been updated to latest HPy HEAD
The release includes three different interpreters:
PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the
+
is for backported security updates)PyPy3.10, which is an interpreter supporting the syntax and the features of Python 3.10, including the stdlib for CPython 3.10.19.
PyPy3.11, which is an interpreter supporting the syntax and the features of Python 3.11, including the stdlib for CPython 3.11.11.
The interpreters are based on much the same codebase, thus the triple release. This is a micro release, all APIs are compatible with the other 7.3 releases. It follows after 7.3.17 release on August 28, 2024.
We recommend updating. You can find links to download the releases here:
We would like to thank our donors for the continued support of the PyPy project. If PyPy is not quite good enough for your needs, we are available for direct consulting work. If PyPy is helping you out, we would love to hear about it and encourage submissions to our blog via a pull request to https://github.com/pypy/pypy.org
We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: bug fixes, PyPy and RPython documentation improvements, or general help with making RPython's JIT even better.
If you are a python library maintainer and use C-extensions, please consider making a HPy / CFFI / cppyy version of your library that would be performant on PyPy. In any case, both cibuildwheel and the multibuild system support building wheels for PyPy.
VMProf Native Symbol Names
When running VMProf profiling with native profiling enabled, PyPy did so far not produce function names for C functions. The output looked like this:
pypy -m vmprof ~/projects/gitpypy/lib-python/2.7/test/pystone.py Pystone(1.1) time for 50000 passes = 0.0109887 This machine benchmarks at 4.55011e+06 pystones/second vmprof output: %: name: location: 100.0% entry_point <builtin>/app_main.py:874 100.0% run_command_line <builtin>/app_main.py:601 100.0% run_toplevel <builtin>/app_main.py:93 100.0% _run_module_as_main /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:150 100.0% _run_code /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:62 100.0% <module> /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/site-packages/vmprof/__main__.py:1 100.0% main /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/site-packages/vmprof/__main__.py:30 100.0% run_path /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:238 100.0% _run_module_code /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:75 100.0% <module> /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:3 100.0% main /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:60 100.0% pystones /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:67 100.0% Proc0 /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:79 76.9% <unknown code> 69.2% <unknown code> 53.8% <unknown code> 53.8% <unknown code> 46.2% <unknown code> 46.2% <unknown code> 38.5% <unknown code> 38.5% Proc8 /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:212 30.8% <unknown code> ...
We can now symbolify these C functions and give function names and which shared library they come from, at least on Linux:
Pystone(1.1) time for 50000 passes = 0.218967 This machine benchmarks at 228345 pystones/second vmprof output: %: name: location: 100.0% entry_point <builtin>/app_main.py:889 100.0% run_command_line <builtin>/app_main.py:616 100.0% run_toplevel <builtin>/app_main.py:95 100.0% _run_module_as_main /home/user/projects/gitpypy/lib-python/2.7/runpy.py:150 100.0% _run_code /home/user/projects/gitpypy/lib-python/2.7/runpy.py:62 100.0% <module> /home/user/projects/gitpypy/site-packages/vmprof/__main__.py:1 100.0% main /home/user/projects/gitpypy/site-packages/vmprof/__main__.py:30 100.0% run_module /home/user/projects/gitpypy/lib-python/2.7/runpy.py:179 100.0% _run_module_code /home/user/projects/gitpypy/lib-python/2.7/runpy.py:75 100.0% <module> /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:3 100.0% main /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:60 100.0% pystones /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:67 100.0% Proc0 /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:79 95.5% n:pypy_g_execute_frame:0:pypy-c 91.4% n:pypy_g_PyFrame_dispatch:0:pypy-c 63.8% n:pypy_g_PyFrame_dispatch_bytecode:0:pypy-c 49.8% Proc1 /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:137 17.6% copy /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:53 13.6% n:pypy_g_PyFrame_CALL_FUNCTION:0:pypy-c 10.4% Proc8 /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:212 8.6% n:pypy_g_STORE_ATTR_slowpath:0:pypy-c
This becomes even more useful when using the VMProf Firefox converter, which uses the Firefox Profiler Web UI to visualize profiling output:
![/images/2025-vmprof-firefox.png](https://www.pypy.org/images/2025-vmprof-firefox.png)
What is PyPy?
PyPy is a Python interpreter, a drop-in replacement for CPython It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
We provide binary builds for:
x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
64-bit ARM machines running Linux (
aarch64
) and macos (macos_arm64
).
PyPy supports Windows 32-bit, Linux PPC64 big- and little-endian, Linux ARM 32 bit, RISC-V RV64IMAFD Linux, and s390x Linux but does not release binaries. Please reach out to us if you wish to sponsor binary releases for those platforms. Downstream packagers provide binary builds for debian, Fedora, conda, OpenBSD, FreeBSD, Gentoo, and more.
What else is new?
For more information about the 7.3.18 release, see the full changelog.
Please update, and continue to help us make pypy better.
Cheers, The PyPy Team
February 06, 2025 12:00 PM UTC
Real Python
Quiz: How to Join Strings in Python
In this quiz, you’ll test your understanding of How to Join Strings in Python.
By working through this quiz, you’ll review how to use .join()
to combine strings with a specified separator and handle non-string data in iterables. You’ll also revisit how .join()
compares to other concatenation methods and how it’s optimized in CPython.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
February 06, 2025 12:00 PM UTC
Quiz: Python "for" Loops: The Pythonic Way
In this quiz, you’ll test your understanding of Python’s for
loop.
By working through this quiz, you’ll revisit how to iterate over items in a data collection, how to use range()
for a predefined number of iterations, and how to use enumerate()
for index-based iteration.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]