Planet Python
http://planetpython.org/
Planet Python - http://planetpython.org/Real Python: Reading and Writing WAV Files in Python
https://realpython.com/python-wav-files/
<div><p>There’s an abundance of third-party tools and libraries for manipulating and analyzing audio WAV files in Python. At the same time, the language ships with the little-known <code>wave</code> module in its standard library, offering a quick and straightforward way to read and write such files. Knowing Python’s <code>wave</code> module can help you dip your toes into digital audio processing.</p>
<p>If topics like audio analysis, sound editing, or music synthesis get you excited, then you’re in for a treat, as you’re about to get a taste of them!</p>
<p><strong>In this tutorial, you’ll learn how to:</strong></p>
<ul>
<li><strong>Read</strong> and <strong>write</strong> WAV files using pure Python</li>
<li>Handle the 24-bit <strong>PCM encoding</strong> of audio samples</li>
<li><strong>Interpret</strong> and <strong>plot</strong> the underlying <strong>amplitude levels</strong></li>
<li>Record <strong>online audio streams</strong> like Internet radio stations</li>
<li>Animate <strong>visualizations</strong> in the <strong>time</strong> and <strong>frequency</strong> domains</li>
<li><strong>Synthesize</strong> sounds and apply <strong>special effects</strong></li>
</ul>
<p>Although not required, you’ll get the most out of this tutorial if you’re familiar with <a href="https://realpython.com/numpy-tutorial/">NumPy</a> and <a href="https://realpython.com/python-matplotlib-guide/">Matplotlib</a>, which greatly simplify working with audio data. Additionally, knowing about <a href="https://realpython.com/python-array/">numeric arrays</a> in Python will help you better understand the underlying data representation in computer memory.</p>
<p>Click the link below to access the bonus materials, where you’ll find <strong>sample audio files</strong> for practice, as well as the <strong>complete source code</strong> of all the examples demonstrated in this tutorial:</p>
<div class="alert alert-warning">
<p><strong>Get Your Code:</strong> <a href="https://realpython.com/bonus/python-wav-files-code/" class="alert-link">Click here to download the free sample code</a> that shows you how to read and write WAV files in Python.</p>
</div>
<p>You can also take the quiz to test your knowledge and see how much you’ve learned:</p>
<div class="alert alert-primary">
<p><strong><span class="icon baseline"></span> Take the Quiz:</strong> Test your knowledge with our interactive “Reading and Writing WAV Files in Python” quiz. Upon completion you will receive a score so you can track your learning progress over time:</p>
<p class="text-center my-2"><a class="btn btn-primary" href="https://realpython.com/quizzes/python-wav-files/" target="_blank">Take the Quiz »</a></p>
</div>
<h2 id="understand-the-wav-file-format">Understand the WAV File Format<a class="headerlink" href="https://realpython.com/atom.xml#understand-the-wav-file-format" title="Permanent link"></a></h2>
<p>In the early nineties, Microsoft and IBM jointly developed the <a href="https://en.wikipedia.org/wiki/WAV">Waveform Audio File Format</a>, often abbreviated as WAVE or WAV, which stems from the file’s extension (<code>.wav</code>). Despite its older age in computer terms, the format remains relevant today. There are several good reasons for its wide adoption, including:</p>
<ul>
<li><strong>Simplicity:</strong> The WAV file format has a straightforward structure, making it relatively uncomplicated to decode in software and understand by humans.</li>
<li><strong>Portability:</strong> Many software systems and hardware platforms support the WAV file format as standard, making it suitable for data exchange.</li>
<li><strong>High Fidelity:</strong> Because most WAV files contain raw, uncompressed audio data, they’re perfect for applications that require the highest possible sound quality, such as with music production or audio editing. On the flipside, WAV files take up significant storage space compared to <a href="https://en.wikipedia.org/wiki/Lossy_compression">lossy compression</a> formats like MP3.</li>
</ul>
<p>It’s worth noting that WAV files are specialized kinds of the <a href="https://en.wikipedia.org/wiki/Resource_Interchange_File_Format">Resource Interchange File Format (RIFF)</a>, which is a container format for audio and video streams. Other popular file formats based on RIFF include <a href="https://en.wikipedia.org/wiki/Audio_Video_Interleave">AVI</a> and <a href="https://en.wikipedia.org/wiki/MIDI">MIDI</a>. RIFF itself is an extension of an even older <a href="https://en.wikipedia.org/wiki/Interchange_File_Format">IFF</a> format originally developed by Electronic Arts to store video game resources.</p>
<p>Before diving in, you’ll deconstruct the WAV file format itself to better understand its structure and how it represents sounds. Feel free to jump ahead if you just want to see <a href="https://realpython.com/atom.xml#get-to-know-pythons-wave-module">how to use the <code>wave</code> module</a> in Python.</p>
<h3 id="the-waveform-part-of-wav">The Waveform Part of WAV<a class="headerlink" href="https://realpython.com/atom.xml#the-waveform-part-of-wav" title="Permanent link"></a></h3>
<p>What you perceive as sound is a disturbance of pressure traveling through a physical medium, such as air or water. At the most fundamental level, every sound is a <a href="https://en.wikipedia.org/wiki/Wave">wave</a> that you can describe using three attributes:</p>
<ol>
<li><a href="https://en.wikipedia.org/wiki/Amplitude">Amplitude</a> is the measure of the sound wave’s strength, which you perceive as <a href="https://en.wikipedia.org/wiki/Loudness">loudness</a>.</li>
<li><a href="https://en.wikipedia.org/wiki/Frequency">Frequency</a> is the reciprocal of the <a href="https://en.wikipedia.org/wiki/Wavelength">wavelength</a> or the number of oscillations per second, which corresponds to the <a href="https://en.wikipedia.org/wiki/Pitch_(music)">pitch</a>.</li>
<li><a href="https://en.wikipedia.org/wiki/Phase_(waves)">Phase</a> is the point in the wave cycle at which the wave starts, not registered by the human ear directly.</li>
</ol>
<p>The word <a href="https://en.wikipedia.org/wiki/Waveform">waveform</a>, which appears in the WAV file format’s name, refers to the graphical depiction of the audio signal’s shape. If you’ve ever opened a sound file using audio editing software, such as <a href="https://www.audacityteam.org/">Audacity</a>, then you’ve likely seen a visualization of the file’s content that looked something like this:</p>
<a href="https://files.realpython.com/media/audacity_waveform.d53954eea4a1.png" target="_blank"><img class="img-fluid mx-auto d-block border " src="https://files.realpython.com/media/audacity_waveform.d53954eea4a1.png" width="1920" height="1080" alt="Waveform in Audacity" /></a>Waveform in Audacity
<p>That’s your audio waveform, illustrating how the amplitude changes over time.</p>
<p>The vertical axis represents the <strong>amplitude</strong> at any given point in time. The midpoint of the graph, which is a horizontal line passing through the center, represents the baseline amplitude or the point of silence. Any deviation from this equilibrium corresponds to a higher positive or negative amplitude, which you experience as a louder sound.</p>
<p>As you move from left to right along the graph’s horizontal scale, which is the <strong>timeline</strong>, you’re essentially moving forward in time through your audio track.</p>
<p>Having such a view can help you visually inspect the characteristics of your audio file. The series of the amplitude’s peaks and valleys reflect the volume changes. Therefore, you can leverage the waveform to identify parts where certain sounds occur or find quiet sections that may need editing.</p>
<p>Coming up next, you’ll learn how WAV files store these amplitude levels in digital form.</p>
<h3 id="the-structure-of-a-wav-file">The Structure of a WAV File<a class="headerlink" href="https://realpython.com/atom.xml#the-structure-of-a-wav-file" title="Permanent link"></a></h3>
</div><h2><a href="https://realpython.com/python-wav-files/?utm_source=realpython&utm_medium=rss">Read the full article at https://realpython.com/python-wav-files/ »</a></h2>
<hr />
<p><em>[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>2024-03-27T14:00:00+00:00Python GUIs: Q&A: How Do I Display Images in PySide6? — Using QLabel to easily add images to your applications
https://www.pythonguis.com/faq/adding-images-to-pyside6-applications/
<p>Adding images to your application is a common requirement, whether you're building an image/photo viewer, or just want to add some decoration to your GUI. Unfortunately, because of how this is done in Qt, it can be a little bit tricky to work out at first.</p>
<p>In this short tutorial, we will look at how you can insert an external image into your PySide6 application layout, using both code and Qt Designer.</p>
<div class="toc"><span class="toctitle">Table of Contents</span><ul>
<li><a href="https://www.pythonguis.com/feeds/all.atom.xml#which-widget-to-use">Which widget to use?</a></li>
<li><a href="https://www.pythonguis.com/feeds/all.atom.xml#using-qt-designer">Using Qt Designer</a></li>
<li><a href="https://www.pythonguis.com/feeds/all.atom.xml#using-code">Using Code</a></li>
<li><a href="https://www.pythonguis.com/feeds/all.atom.xml#conclusion">Conclusion</a></li>
</ul>
</div>
<h3 id="which-widget-to-use">Which widget to use?</h3>
<p>Since you're wanting to insert an image you might be expecting to use a widget named <code>QImage</code> or similar, but that would make a bit too much sense! <code>QImage</code> is actually Qt's image <em>object</em> type, which is used to store the actual image data for use within your application. The <em>widget</em> you use to display an image is <code>QLabel</code>.</p>
<p>The primary use of <code>QLabel</code> is of course to add labels to a UI, but it also has the ability to display an image — or <em>pixmap</em> — instead, covering the entire area of the widget. Below we'll look at how to use <code>QLabel</code> to display a widget in your applications.</p>
<h3 id="using-qt-designer">Using Qt Designer</h3>
<p>First, create a <em>MainWindow</em> object in Qt Designer and add a "Label" to it. You can find Label at in <em>Display Widgets</em> in the bottom of the left hand panel. Drag this onto the <code>QMainWindow</code> to add it.</p>
<p><img alt="MainWindow with a single QLabel added" src="https://www.pythonguis.com/static/faq/adding-images-to-applications/1.png" width="1917" height="1027" />
<em>MainWindow with a single QLabel added</em></p>
<p>Next, with the Label selected, look in the right hand <code>QLabel</code> properties panel for the <code>pixmap</code> property (scroll down to the blue region). From the property editor dropdown select "Choose File…" and select an image file to insert.</p>
<p>
<div></div>
</p>
<p>As you can see, the image is inserted, but the image is kept at its original size, cropped to the boundaries of the<code>QLabel</code> box. You need to resize the <code>QLabel</code> to be able to see the entire image.</p>
<p>In the same controls panel, click to enable <code>scaledContents</code>.</p>
<p>
<div></div>
</p>
<p>When <code>scaledContents</code> is enabled the image is resized to the fit the bounding box of the <code>QLabel</code> widget. This shows the entire image at all times, although it does not respect the aspect ratio of the image if you resize the widget.</p>
<p>You can now save your UI to file (e.g. as <code>mainwindow.ui</code>).</p>
<p>To view the resulting UI, we can use the standard application template below. This loads the <code>.ui</code> file we've created (<code>mainwindow.ui</code>) creates the window and starts up the application.</p>
<div class="code-block">
<span class="code-block-language code-block-PySide6">PySide6</span>
<pre><code class="PySide6">import sys
from PySide6 import QtWidgets
from PySide6.QtUiTools import QUiLoader
loader = QUiLoader()
app = QtWidgets.QApplication(sys.argv)
window = loader.load("mainwindow.ui", None)
window.show()
app.exec()
</code></pre>
</div>
<p>Running the above code will create a window, with the image displayed in the middle.</p>
<p><img alt="QtDesigner application showing a Cat" src="https://www.pythonguis.com/static/faq/adding-images-to-applications/5.png" width="802" height="639" />
<em>QtDesigner application showing a Cat</em></p>
<h3 id="using-code">Using Code</h3>
<p>Instead of using Qt Designer, you might also want to show an image in your application through code. As before we use a <code>QLabel</code> widget and add a <em>pixmap</em> image to it. This is done using the <code>QLabel</code> method <code>.setPixmap()</code>. The full code is shown below.</p>
<div class="code-block">
<span class="code-block-language code-block-PySide6">PySide6</span>
<pre><code class="PySide6">import sys
from PySide6.QtGui import QPixmap
from PySide6.QtWidgets import QMainWindow, QApplication, QLabel
class MainWindow(QMainWindow):
def __init__(self):
super(MainWindow, self).__init__()
self.title = "Image Viewer"
self.setWindowTitle(self.title)
label = QLabel(self)
pixmap = QPixmap('cat.jpg')
label.setPixmap(pixmap)
self.setCentralWidget(label)
self.resize(pixmap.width(), pixmap.height())
app = QApplication(sys.argv)
w = MainWindow()
w.show()
sys.exit(app.exec())
</code></pre>
</div>
<p>The block of code below shows the process of creating the <code>QLabel</code>, creating a <code>QPixmap</code> object from our file <code>cat.jpg</code> (passed as a file path), setting this <code>QPixmap</code> onto the <code>QLabel</code> with <code>.setPixmap()</code> and then finally resizing the window to fit the image.</p>
<div class="code-block">
<span class="code-block-language code-block-python">python</span>
<pre><code class="python">label = QLabel(self)
pixmap = QPixmap('cat.jpg')
label.setPixmap(pixmap)
self.setCentralWidget(label)
self.resize(pixmap.width(), pixmap.height())
</code></pre>
</div>
<p>Launching this code will show a window with the cat photo displayed and the window sized to the size of the image.</p>
<p><img alt="QMainWindow with Cat image displayed" src="https://www.pythonguis.com/static/faq/adding-images-to-applications/4.png" width="602" height="439" />
<em>QMainWindow with Cat image displayed</em></p>
<p>Just as in Qt designer, you can call <code>.setScaledContents(True)</code> on your <code>QLabel</code> image to enable scaled mode, which resizes the image to fit the available space.</p>
<div class="code-block">
<span class="code-block-language code-block-python">python</span>
<pre><code class="python">label = QLabel(self)
pixmap = QPixmap('cat.jpg')
label.setPixmap(pixmap)
label.setScaledContents(True)
self.setCentralWidget(label)
self.resize(pixmap.width(), pixmap.height())
</code></pre>
</div>
<p class="admonition admonition-note"><span class="admonition-kind"><i class="fas fa-sticky-note"></i></span> Notice that you set the scaled state on the <code>QLabel</code> widget and not the image pixmap itself.</p>
<h3 id="conclusion">Conclusion</h3>
<p>In this quick tutorial we've covered how to insert images into your Qt UIs using <code>QLabel</code> both from Qt Designer and directly from PySide6 code.</p>2024-03-27T06:00:00+00:00PyCoder’s Weekly: Issue #622 (March 26, 2024)
https://pycoders.com/issues/622
<p> <span>#622 – MARCH 26, 2024</span><br /> <span><a href="https://pycoders.com/issues/622/feed">View in Browser »</a></span> </p> <p><a href="https://pycoders.com"><img alt="The PyCoder’s Weekly Logo" src="https://cdn.pycoders.com/37bdf31dc645f968ffb90196e5d38ff5" /></a></p> <hr /> <div> <h3><a href="https://pycoders.com/link/12449/feed" target="_blank">Build a Python Turtle Game: Space Invaders Clone</a></h3> <p> In this step-by-step tutorial, you’ll use Python’s turtle module to write a Space Invaders clone. You’ll learn about techniques used in animations and games, and consolidate your knowledge of key Python topics.<br /> <span><a href="https://pycoders.com/link/12449/feed" target="_blank">REAL PYTHON</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12442/feed" target="_blank">Getting Help (In Python)</a></h3> <p> When trying to remember just where <code>sleep()</code> was in the Python standard library, Ishaan stumbled through the built-in help and learned how to use it to answer just these kinds of questions.<br /> <span><a href="https://pycoders.com/link/12442/feed" target="_blank">ISHAAN ARORA</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12415/feed" target="_blank">Reporting Appsec Risk up to Your CISO Body</a></h3> <a href="https://pycoders.com/link/12415/feed" target="_blank"><img src="https://cdn.pycoders.com/1772f532693d9dba1509bc4f8f05a9a7" alt="alt" /></a> <p> Master concise risk reporting for a stronger partnership with your CISO. Translate technical jargon into actionable insights for your CISO with Snyk’s guide on strategies on how to <a href="https://pycoders.com/link/12415/feed" target="_blank">bridge visibility gaps and provide meaningful risk reports →</a><br /> <span><a href="https://pycoders.com/link/12415/feed" target="_blank">SNYK.IO</a></span> <span>sponsor</span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12431/feed" target="_blank">Every Dunder Method in Python</a></h3> <p> Ever wonder just how many special methods there are in Python? This post explains all of Python’s 100+ dunder methods and 50+ dunder attributes.<br /> <span><a href="https://pycoders.com/link/12431/feed" target="_blank">TREY HUNNER</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12433/feed" target="_blank">PyOhio 2024 Announced</a></h3> <p> <span><a href="https://pycoders.com/link/12433/feed" target="_blank">PYOHIO</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12420/feed" target="_blank">Django REST Framework 3.15</a></h3> <p> <span><a href="https://pycoders.com/link/12420/feed" target="_blank">DRF</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12446/feed" target="_blank">EuroPython 2024: Community Voting Is Now Live!</a></h3> <p> <span><a href="https://pycoders.com/link/12446/feed" target="_blank">EUROPYTHON</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12429/feed" target="_blank">Python 3.10.14, 3.9.19, and 3.8.19 Security Releases</a></h3> <p> <span><a href="https://pycoders.com/link/12429/feed" target="_blank">CPYTHON DEV BLOG</a></span> </p> </div> <h2>Discussions</h2> <div> <h3><a href="https://pycoders.com/link/12422/feed" target="_blank">Ideas: Make <code>super()</code> Work in a Class Definition</a></h3> <p> <span><a href="https://pycoders.com/link/12422/feed" target="_blank">PYTHON DISCUSS</a></span> </p> </div> <h2>Articles & Tutorials</h2> <div> <h3><a href="https://pycoders.com/link/12430/feed" target="_blank">SQLite and SQLAlchemy in Python: Beyond Flat Files</a></h3> <p> In this video course, you’ll learn how to store and retrieve data using Python, SQLite, and SQLAlchemy as well as with flat files. Using SQLite with Python brings with it the additional benefit of accessing data with SQL. By adding SQLAlchemy, you can work with data in terms of objects and methods.<br /> <span><a href="https://pycoders.com/link/12430/feed" target="_blank">REAL PYTHON</a></span> <span>course</span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12418/feed" target="_blank">Why Programming Languages Need a Style Czar</a></h3> <p> The more flexible the language, the more likely you’re going to have a variety of styles in the code. The larger the project the harder it is to manage. This opinion piece explains why having someone dictate how code should look at the language level can be valuable.<br /> <span><a href="https://pycoders.com/link/12418/feed" target="_blank">ADAM GORDON BELL</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12416/feed" target="_blank">Elevate Your Python Coding Game with 250 Pythonic Tips!</a></h3> <a href="https://pycoders.com/link/12416/feed" target="_blank"><img src="https://cdn.pycoders.com/1ff4edc46112de01eec1545e45887abf" alt="alt" /></a> <p> Discover how to write elegant, efficient Python code with our FREE eBook “Pybites Python Tips”. From basics to advanced techniques, these 250 actionable insights will transform your coding approach. <a href="https://pycoders.com/link/12416/feed" target="_blank">Perfect for Pythonistas aiming for mastery →</a><br /> <span><a href="https://pycoders.com/link/12416/feed" target="_blank">PYBITES</a></span> <span>sponsor</span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12440/feed" target="_blank">Python Basics Exercises: Dictionaries</a></h3> <p> One of the most useful data structures in Python is the dictionary. In this video course, you’ll practice working with Python dictionaries, see how dictionaries differ from lists and tuples, and define and use dictionaries in your own code.<br /> <span><a href="https://pycoders.com/link/12440/feed" target="_blank">REAL PYTHON</a></span> <span>course</span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12436/feed" target="_blank">The (Hidden) Danger of Notebooks in Production</a></h3> <p> An opinion piece on the perils of utilizing notebooks in a production system. It highlights some of their inherent challenges and presents an alternative approach where notebooks can co-exist with a production system.<br /> <span><a href="https://pycoders.com/link/12436/feed" target="_blank">CHASE GRECO</a> • Shared by Chase Greco</span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12443/feed" target="_blank">MVC in Python Web Apps: Explained With Lego</a></h3> <p> This tutorial conceptually explains the Model-View-Controller (MVC) pattern in Python web apps using Lego bricks. Finally understand this important architecture to streamline your web development process.<br /> <span><a href="https://pycoders.com/link/12443/feed" target="_blank">REAL PYTHON</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12419/feed" target="_blank">Parsing URLs in Python</a></h3> <p> Correctly parsing a URL can be tough, in fact the built-in Python functions aren’t fully compliant with the RFC. This post talks about how that is, and a library that gets it right.<br /> <span><a href="https://pycoders.com/link/12419/feed" target="_blank">TYLER KENNEDY</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12417/feed" target="_blank">Rapid Prototyping in Python</a></h3> <p> This post talks about using Python as a prototyping language for more complex projects in other languages. Rather than write pseudo-code, write actual code to test your ideas.<br /> <span><a href="https://pycoders.com/link/12417/feed" target="_blank">AMJITH</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12424/feed" target="_blank">Go, Python, Rust, and Production AI Applications</a></h3> <p> Sameer talks about his use of Go, Python, and Rust, and how their approaches effect your application’s safety, along with how that impacts coding for AI systems.<br /> <span><a href="https://pycoders.com/link/12424/feed" target="_blank">SAMEER AJMANI</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12448/feed" target="_blank">20 Django Packages That I Use in Every Project</a></h3> <p> An opinionated list of Django third-party packages that Will (author of Django for Beginners) uses to add features to his Django web projects.<br /> <span><a href="https://pycoders.com/link/12448/feed" target="_blank">WILL VINCENT</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12447/feed" target="_blank">The Wrong Way to Speed Up Your Code With Numba</a></h3> <p> Numba can make your numeric code faster, but only if you use it right. Learn what “right” means and what to avoid.<br /> <span><a href="https://pycoders.com/link/12447/feed" target="_blank">ITAMAR TURNER-TRAURING</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12427/feed" target="_blank">4 Ways to Correct Grammar With Python</a></h3> <p> This tutorial explains various methods for checking and correcting English language grammatical errors using Python.<br /> <span><a href="https://pycoders.com/link/12427/feed" target="_blank">DEEPANSHU BHALLA</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12439/feed" target="_blank">State of WASI Support for CPython: March 2024</a></h3> <p> Progress on WASI and CPython continues. Brett gives a summary of changes since last year’s post.<br /> <span><a href="https://pycoders.com/link/12439/feed" target="_blank">BRETT CANNON</a></span> </p> </div> <h2>Projects & Code</h2> <div> <h3><a href="https://pycoders.com/link/12432/feed" target="_blank">likeprogramming: A Python Superset With Slang</a></h3> <p> <span><a href="https://pycoders.com/link/12432/feed" target="_blank">GITHUB.COM/STARINGISPOLITE</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12444/feed" target="_blank">wifi-heat-mapper: Benchmark Wi-Fi Networks</a></h3> <p> <span><a href="https://pycoders.com/link/12444/feed" target="_blank">GITHUB.COM/NISCHAY-PRO</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12445/feed" target="_blank">Slightly Simplified Subprocesses</a></h3> <p> <span><a href="https://pycoders.com/link/12445/feed" target="_blank">GITHUB.COM/POMPONCHIK</a> • Shared by Evgeniy Blinov</span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12435/feed" target="_blank">hancho: A Simple, Pleasant Build System in Python</a></h3> <p> <span><a href="https://pycoders.com/link/12435/feed" target="_blank">GITHUB.COM/AAPPLEBY</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12421/feed" target="_blank">flect: Python Framework for Full-Stack Web Applications</a></h3> <p> <span><a href="https://pycoders.com/link/12421/feed" target="_blank">GITHUB.COM/CHAOYINGZ</a></span> </p> </div> <h2>Events</h2> <div> <h3><a href="https://pycoders.com/link/12438/feed" target="_blank">Weekly Real Python Office Hours Q&A (Virtual)</a></h3> <p> March 27, 2024<br /> <span><a href="https://pycoders.com/link/12438/feed" target="_blank">REALPYTHON.COM</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12426/feed" target="_blank">SPb Python Drinkup</a></h3> <p> March 28, 2024<br /> <span><a href="https://pycoders.com/link/12426/feed" target="_blank">MEETUP.COM</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12423/feed" target="_blank">PyCamp Spain 2024</a></h3> <p> March 29 to April 2, 2024<br /> <span><a href="https://pycoders.com/link/12423/feed" target="_blank">PYCAMP.ES</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12441/feed" target="_blank">PyLadies Amsterdam</a></h3> <p> March 29, 2024<br /> <span><a href="https://pycoders.com/link/12441/feed" target="_blank">MEETUP.COM</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12428/feed" target="_blank">PythOnRio Meetup</a></h3> <p> March 30, 2024<br /> <span><a href="https://pycoders.com/link/12428/feed" target="_blank">PYTHON.ORG.BR</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12434/feed" target="_blank">PyCon Lithuania 2024</a></h3> <p> April 2 to April 7, 2024<br /> <span><a href="https://pycoders.com/link/12434/feed" target="_blank">PYCON.LT</a></span> </p> </div> <div> <h3><a href="https://pycoders.com/link/12437/feed" target="_blank">PyCascades 2024</a></h3> <p> April 5 to April 9, 2024<br /> <span><a href="https://pycoders.com/link/12437/feed" target="_blank">PYCASCADES.COM</a></span> </p> </div> <hr /> <p>Happy Pythoning!<br />This was PyCoder’s Weekly Issue #622.<br /><a href="https://pycoders.com/issues/622/feed">View in Browser »</a></p> <img src="https://pycoders.com/issues/622/open/feed" width="1" height="1" alt="alt" />
<hr />
<p><em>[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week <a href="https://pycoders.com/?utm_source=pycoders&utm_medium=feed&utm_campaign=footer">>> Click here to learn more</a> ]</em></p>2024-03-26T19:30:00+00:00PyPy: Fixing a Bug in PyPy's Incremental GC
https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html
<h2>Introduction</h2>
<p>Since last summer, I've been looking on and off into a weird and hard to
reproduce <a href="https://github.com/pypy/pypy/issues/3959">crash bug in PyPy</a>. It was
manifesting only on CI, and it seemed to always happen in the AST rewriting
phase of <a href="https://pytest.org">pytest</a>, the symptoms being that PyPy would crash
with a segfault. All my attempts to reproduce it locally failed, and my
attempts to try to understand the problem by dumping the involved ASTs lead
nowhere.</p>
<p>A few weeks ago, we got <a href="https://github.com/PyO3/pyo3/issues/3766">two more</a>
<a href="https://github.com/orgs/pypy/discussions/4923">bug reports</a>, the last one by
the authors of the <a href="https://nanobind.readthedocs.io/">nanobind</a> binding
generator, with the same symptoms: crash in AST rewriting, only on CI. I
decided to make a more serious push to try to find the bug this time.
Ultimately the problem turned out to be several bugs in PyPy's garbage
collector (GC) that had been there since its inception in
<a href="https://www.pypy.org/posts/2013/10/incremental-garbage-collector-in-pypy-8956893523842234676.html">2013</a>.
Understanding the
situation turned out to be quite involved, additionally complicated by this
being the first time that I was working on this particular aspect of PyPy's GC.
Since the bug was so much work to find, I thought I'd write a blog post about
it.</p>
<p>The blog post consists of three parts: first a chronological description of
what I did to find the bug, a technical explanation of what goes wrong, some
reflections on the bug (and then a bonus bug I also found in the process).</p>
<h2>Finding the Bug</h2>
<p>I started from the failing <a href="https://github.com/wjakob/nanobind/actions/runs/8234561874/job/22516568891">nanobind CI
runs</a>
that ended with a segfault of the PyPy interpreter. This was only an
intermittent problem, not every run was failing. When I tried to just run the
test suite locally, I couldn't get it to fail. Therefore at first I tried to
learn more about what was happening by looking on the CI runners.</p>
<h3>Running on CI</h3>
<p>I forked the nanobind repo and hacked the CI script in order to get it to use a
PyPy build with <a href="https://doc.pypy.org/en/latest/build.html#making-a-debug-build-of-pypy">full debug information and more assertions turned on</a>. In order
to increase the probability of seeing the crash I added an otherwise unused
<a href="https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs">matrix</a>
variable to the CI script that just contained 32 parameters. This means every
build is done 32 times (sorry Github for wasting your CPUs 😕). With that
amount of repetition, I got at least one job of every build that was crashing.</p>
<p>Then I added the <code>-Xfaulthandler</code> option to the PyPy command which will use the
<a href="https://docs.python.org/3.11/library/faulthandler.html">faulthandler</a> module
try to print a Python stacktrace if the VM segfaults to confirm that PyPy was
indeed crashing in the <a href="https://docs.python.org/3/library/ast.html">AST</a>
<a href="https://github.com/pytest-dev/pytest/blob/main/src/_pytest/assertion/rewrite.py">rewriting
phase</a>
of pytest, which pytest uses for <a href="https://docs.pytest.org/en/7.1.x/how-to/assert.html#asserting-with-the-assert-statement">nicer
assertions</a>.
I experimented with hacking our faulthandler implementation to also give me a
C-level callstack, but that didn't work as well as I hoped.</p>
<p>Then I tried to run <a href="https://sourceware.org/gdb/">gdb</a> on CI to try to get it
to print a C callstack at the crash point. You can get gdb to execute commands
as if typed at the prompt with the <code>-ex</code> commandline option, I used something
like this:</p>
<div class="code"><pre class="code literal-block">gdb -ex "set confirm off" -ex "set pagination off" -ex \
"set debuginfod enabled off" -ex run -ex where -ex quit \
--args <command> <arguments>
</pre></div>
<p>But unfortunately the crash never occurred when running in gdb.</p>
<p>Afterwards I tried the next best thing, which was configuring the CI runner to
<a href="https://github.com/itamarst/gha-upload-cores">dump a core file and upload it as a build
artifact</a>, which worked. Looking
at the cores locally only sort of worked, because I am running a different
version of Ubuntu than the CI runners. So I used
<a href="https://mxschmitt.github.io/action-tmate/">tmate</a> to be able to log into the
CI runner after a crash and interactively used gdb there. Unfortunately what I
learned from that was that the bug was some kind of <strong>memory corruption</strong>,
which is always incredibly unpleasant to debug. Basically the header word of a
Python object had been corrupted somehow at the point of the crash, which means
that it's <a href="https://en.wikipedia.org/wiki/Virtual_method_table">vtable</a> wasn't
usable any more.</p>
<p>(Sidenote: <a href="https://www.pypy.org/posts/2009/10/gc-improvements-6174120095428192954.html#unifying-the-vtable-ptr-with-the-gc-header">PyPy doesn't really use a vtable
pointer</a>,
instead it uses half a word in the header for the vtable, and the other half
for flags that the GC needs to keep track of the state of the object.
Corrupting all this is still bad.)</p>
<h3>Reproducing Locally</h3>
<p>At that point it was clear that I had to push to reproduce the problem on my
laptop, to allow me to work on the problem more directly and not to always have
to go via the CI runner. Memory corruption bugs often have a lot of randomness
(depending on which part of memory gets modified, things might crash or more
likely just happily keep running). Therefore I decided to try to brute-force
reproducing the crash by simply running the tests many many times. Since the
crash happened in the AST rewriting phase of pytest, and that happens only if
no <a href="https://stackoverflow.com/questions/2998215/if-python-is-interpreted-what-are-pyc-files">pyc
files</a>
of the bytecode-compiled rewritten ASTs exist, I made sure to delete them
before every test run.</p>
<p>To repeat the test runs I used
<a href="https://tratt.net/laurie/src/multitime/">multitime</a>, which is a simple program
that runs a command repeatedly. It's meant for lightweight benchmarking
purposes, but it also halts the execution of the command if that command exits
with an error (and it sleeps a small random time between runs, which might help
with randomizing the situation, maybe). Here's a demo:</p>
<p>(<a href="https://bernsteinbear.com/">Max</a> pointed out
<a href="https://github.com/silentbicycle/autoclave">autoclave</a> to me when reviewing
this post, which is a more dedicated tool for this job.)</p>
<p>Thankfully, running the tests repeatedly eventually lead to a crash, solving my
"only happens on CI" problem. I then tried various variants to exclude possible
sources of errors. The first source of errors to exclude in PyPy bugs is the
just-in-time compiler, so I reran the tests with <code>--jit off</code> to see whether I
could still get it to crash, and thankfully I eventually could (JIT bugs are
often very annoying).</p>
<p>Next source of bugs to exclude where C-extensions. Since those were the tests
of nanobind, a framework for creating C-extension modules I was a bit worried
that the bug might be in our emulation of CPython's C-API. But running PyPy
with the <code>-v</code> option (which will print all the imports as they happen)
confirmed that at the point of crash no C-extension had been imported yet.</p>
<h3>Using <code>rr</code></h3>
<p>I still couldn't get the bug to happen in GDB, so the tool I tried next was
<a href="https://rr-project.org/">rr, the "reverse debugger"</a>. rr can record the execution of a program and
later replay it arbitrarily often. This gives you a time-traveling debugger
that allows you to execute the program backwards in addition to forwards.
Eventually I managed to get the crash to happen when running the tests with
<code>rr record --chaos</code> (<code>--chaos</code> randomizes some decisions that rr takes, to try to
increase the chance of reproducing bugs).</p>
<p>Using rr well is quite hard, and I'm not very good at it. The main approach I
use with rr to debug memory corruption is to replay the crash, then set a
<a href="https://sourceware.org/gdb/current/onlinedocs/gdb.html/Set-Watchpoints.html">watchpoint</a>
for the corrupted memory location, then use the command <code>reverse-continue</code> to
find the place in the code that mutated the memory location. <code>reverse-continue</code>
is like <code>continue</code>, except that it will execute the program backwards from the
current point. Here's a little demo of this:</p>
<p>Doing this for my bug revealed that the object that was being corrupted was
erroneously collected by the garbage collector. For some reason the GC had
wrongly decided that the object was no longer reachable and therefore put the
object into a freelist by writing a pointer to the next entry in the freelist
into the first word of the object, overwriting the object's header. The next
time the object was used things crashed.</p>
<h3>Side-quest: wrong GC assertions</h3>
<p>At this point in the process, I got massively side-tracked. PyPy's GC has a
number of debug modes that you can optionally turn on. Those slow down the
program execution a lot, but they should in theory help to understand why the
GC goes wrong. When I turned them on, I was getting a failing assertion really
early in the test execution, complaining about an invariant violation in the GC
logic. At first this made me very happy. I thought that this would help me fix
the bug more quickly.</p>
<p>Extremely frustratingly, after two days of work I concluded that the assertion
logic itself was wrong. I have fixed that in the meantime too, the details
of that are in the bonus section at the end of the post.</p>
<h3>Using GDB scripting to find the real bug</h3>
<p>After that disaster I went back to the earlier rr recording without GC assertions
and tried to understand in more detail why the GC decided to free an object
that was still being referenced. To be able to do that I used the <a href="https://sourceware.org/gdb/current/onlinedocs/gdb.html/Python-API.html">GDB Python
scripting
API</a> to
write some helper commands to understand the state of the GC heap (rr is an
extension of GDB, so the GDB scripting API works in rr too).</p>
<p>The first (small) helper command I wrote with the GDB scripting API was a way
to pretty-print the currently active GC flags of a random PyPy object, starting
just from the pointer. The more complex command I wrote was an object tracer,
which follows pointers to GC objects starting from a root object to explore the
object graph. The object tracer isn't complete, it doesn't deal with all the
complexities of PyPy's GC. But it was good enough to help me with my problem, I
found out that the corrupted object was stored in an array.</p>
<p>As an example, here's a function that uses the GDB API to walk one of the
helper data structures of the GC, a stack of pointers:</p>
<div class="code"><pre class="code literal-block"><span class="k">def</span> <span class="nf">walk_addr_stack</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span>
<span class="w"> </span><span class="sd">""" walk an instance of the AddressStack class (which is a linked list of</span>
<span class="sd"> arrays of 1019 pointers).</span>
<span class="sd"> the first of the arrays is only partially filled with used_in_last_chunk</span>
<span class="sd"> items, all the other chunks are full."""</span>
<span class="k">if</span> <span class="n">obj</span><span class="o">.</span><span class="n">type</span><span class="o">.</span><span class="n">code</span> <span class="o">==</span> <span class="n">gdb</span><span class="o">.</span><span class="n">TYPE_CODE_PTR</span><span class="p">:</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">obj</span><span class="o">.</span><span class="n">dereference</span><span class="p">()</span>
<span class="n">used_in_last_chunk</span> <span class="o">=</span> <span class="n">lookup</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="s2">"used_in_last_chunk"</span><span class="p">)</span>
<span class="n">chunk</span> <span class="o">=</span> <span class="n">lookup</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="s2">"inst_chunk"</span><span class="p">)</span><span class="o">.</span><span class="n">dereference</span><span class="p">()</span>
<span class="k">while</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">items</span> <span class="o">=</span> <span class="n">lookup</span><span class="p">(</span><span class="n">chunk</span><span class="p">,</span> <span class="s2">"items"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">used_in_last_chunk</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">items</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="n">chunk</span> <span class="o">=</span> <span class="n">lookup</span><span class="p">(</span><span class="n">chunk</span><span class="p">,</span> <span class="s2">"next"</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">chunk</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">chunk</span> <span class="o">=</span> <span class="n">chunk</span><span class="o">.</span><span class="n">dereference</span><span class="p">()</span>
<span class="n">used_in_last_chunk</span> <span class="o">=</span> <span class="mi">1019</span>
</pre></div>
<p>The full file of supporting code I wrote can be found in <a href="https://gist.github.com/cfbolz/13cadcbbef321d93fc9790dff6f60a6a">this
gist</a>. This is
pretty rough throw-away code, however.</p>
<p>In the following recording I show a staged debugging session with some of the
extra commands I wrote with the Python API. The details aren't important, I
just wanted to give a bit of a flavor of what inspecting objects looks like:</p>
<p>The next step was to understand why the array content wasn't being correctly
traced by the GC, which I eventually managed with some <a href="https://www.fayewilliams.com/2011/07/13/gdb-conditional-breakpoints/">conditional
breakpoints</a>,
more watchpoints, and using <code>reverse-continue</code>. It turned out to be a bug that
occurs when the content of one array was memcopied into another array. The
technical details of why the array wasn't traced correctly are described in
detail in the next section.</p>
<h3>Writing a unit test</h3>
<p>To try to make sure I really understood the bug correctly I then wrote a GC
unit test that shows the problem. Like most of PyPy, our GC is written in
RPython, a (somewhat strange) subset/dialect of Python2, which can be compiled
to C code. However, since it is also valid Python2 code, it can be <a href="https://www.pypy.org/posts/2022/04/how-is-pypy-tested.html">unit-tested
on top of a Python2
implementation</a>
(which is one of the reasons why we keep maintaining PyPy2).</p>
<p>In the GC unit tests you have a lot of control about what order things happen
in, e.g. how objects are allocated, when garbage collection phases happen, etc.
After some trying I managed to write a test that crashes with the same kind of
memory corruption that my original crash exhibited: an object that is still
reachable via an array is collected by the GC. To give you a flavor of what
this kind of test looks like, here's an (edited for clarity) version of the
test I eventually managed to write</p>
<div class="code"><pre class="code literal-block"><span class="k">def</span> <span class="nf">test_incrementality_bug_arraycopy</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">source</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">malloc</span><span class="p">(</span><span class="n">VAR</span><span class="p">,</span> <span class="mi">8</span><span class="p">)</span> <span class="c1"># first array</span>
<span class="c1"># the stackroots list emulates the C stack</span>
<span class="bp">self</span><span class="o">.</span><span class="n">stackroots</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">source</span><span class="p">)</span>
<span class="n">target</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">malloc</span><span class="p">(</span><span class="n">VAR</span><span class="p">,</span> <span class="mi">8</span><span class="p">)</span> <span class="c1"># second array</span>
<span class="bp">self</span><span class="o">.</span><span class="n">stackroots</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">target</span><span class="p">)</span>
<span class="n">node</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">malloc</span><span class="p">(</span><span class="n">S</span><span class="p">)</span> <span class="c1"># unrelated object, will be collected</span>
<span class="n">node</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">5</span>
<span class="c1"># store reference into source array, calling the write barrier</span>
<span class="bp">self</span><span class="o">.</span><span class="n">writearray</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">node</span><span class="p">)</span>
<span class="n">val</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">gc</span><span class="o">.</span><span class="n">collect_step</span><span class="p">()</span>
<span class="n">source</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">stackroots</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c1"># reload arrays, they might have moved</span>
<span class="n">target</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">stackroots</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="c1"># this GC step traces target</span>
<span class="n">val</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">gc</span><span class="o">.</span><span class="n">collect_step</span><span class="p">()</span>
<span class="c1"># emulate what a memcopy of arrays does</span>
<span class="n">res</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">gc</span><span class="o">.</span><span class="n">writebarrier_before_copy</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">res</span>
<span class="n">target</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">source</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c1"># copy two elements of the arrays</span>
<span class="n">target</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">source</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="c1"># now overwrite the reference to node in source</span>
<span class="bp">self</span><span class="o">.</span><span class="n">writearray</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">lltype</span><span class="o">.</span><span class="n">nullptr</span><span class="p">(</span><span class="n">S</span><span class="p">))</span>
<span class="c1"># this GC step traces source</span>
<span class="bp">self</span><span class="o">.</span><span class="n">gc</span><span class="o">.</span><span class="n">collect_step</span><span class="p">()</span>
<span class="c1"># some more collection steps, crucially target isn't traced again</span>
<span class="c1"># but node is deleted</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">gc</span><span class="o">.</span><span class="n">collect_step</span><span class="p">()</span>
<span class="c1"># used to crash, node got collected</span>
<span class="k">assert</span> <span class="n">target</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">x</span> <span class="o">==</span> <span class="mi">5</span>
</pre></div>
<p>One of the good properties of testing our GC that way is that all the memory is
emulated. The crash in the last line of the test isn't a segfault at all,
instead you get a nice exception saying that you tried to access a freed chunk
of memory and you can then debug this with a python2 debugger.</p>
<h3>Fixing the Bug</h3>
<p>With the unit test in hand, fixing the test was relatively straightforward (the
diff in its simplest form is anyway only a <a href="https://github.com/pypy/pypy/commit/78bbeb93471b5f38438004e971f4b4f84ab17a84">single line
change</a>).
After this first version of my fix, I
<a href="https://github.com/pypy/pypy/issues/4925#issuecomment-2014459454">talked to Armin
Rigo</a> who
helped me find different case that was still wrong, in the same area of the
code.</p>
<p>I also got help by the developers at <a href="https://portaone.com/">PortaOne</a>
who are using PyPy on their servers and had seen some <a href="https://github.com/pypy/pypy/issues/4900">mysterious PyPy
crashes</a>
recently, that looked related to the GC. They did test deployments of my fixes
in their various stages to their servers to try to see whether stability
improved for them. Unfortunately in the end it turned out that their crashes
are an unrelated GC bug related to object pinning, which we haven't resolved
yet.</p>
<h3>Writing a GC fuzzer/property based test</h3>
<p>Finding bugs in the GC is always extremely disconcerting, particularly since
this one manged to hide for so long (more than ten years!). Therefore I wanted
to use these bugs as motivation to try to find more problems in PyPy's GC. Given
the ridiculous effectiveness of fuzzing, I used
<a href="https://hypothesis.readthedocs.io/en/latest/">hypothesis</a> to write a
property-based test. Every test performs a sequence of randomly chosen steps
from the following list:</p>
<ul>
<li>allocate an object</li>
<li>read a random field from a random object</li>
<li>write a random reference into a random object</li>
<li>drop a random stack reference</li>
<li>perform one GC step</li>
<li>allocate an array</li>
<li>read a random index from a random array</li>
<li>write to an array</li>
<li>memcopy between two arrays</li>
</ul>
<p>This approach of doing a sequence of steps is pretty close to the <a href="https://hypothesis.readthedocs.io/en/latest/stateful.html">stateful
testing</a> approach of
hypothesis, but I just implemented it manually with the <a href="https://hypothesis.readthedocs.io/en/latest/data.html#drawing-interactively-in-tests">data
strategy</a>.</p>
<p>Every one of those steps is always performed on both the tested GC, and on some
regular Python objects. The Python objects provide the "ground truth" of what
the heap should look like, so we can compare the state of the GC objects
with the state of the Python objects to find out whether the GC made a mistake.</p>
<p>In order to check whether the test is actually useful, I reverted my bug fixes
and made sure that the test re-finds both the spurious GC assertion error and the
problems with memcopying an array.</p>
<p>In addition, the test also found corner cases in my fix. There was a situation
that I hadn't accounted for, which the test found after eventually.
I also plan on adding a bunch of other GC features as steps in the
test to stress them too (for example weakrefs, identity hashes, pinning, maybe
finalization).</p>
<p>At the point of publishing this post, the fixes got merged to the 2.7/3.9/3.10
branches of PyPy, and will be part of the next release (v7.3.16).</p>
<h2>The technical details of the bug</h2>
<p>In order to understand the technical details of the bug, I need to give some
background explanations about PyPy's GC.</p>
<h3>PyPy's incremental GC</h3>
<p>PyPy uses an incremental generational mark-sweep GC. It's
<a href="https://en.wikipedia.org/wiki/Tracing_garbage_collection#Generational_GC_(ephemeral_GC)">generational</a>
and therefore has minor collections (where only young objects get collected)
and major collections (collecting long-lived objects eventually, using a
<a href="https://en.wikipedia.org/wiki/Tracing_garbage_collection#Na%C3%AFve_mark-and-sweep">mark-and-sweep</a>
algorithm). Young objects are allocated in a nursery using a
bump-pointer allocator, which makes allocation quite efficient. They are moved
out of the nursery by minor collections. In order to find references from old
to young objects the GC uses a write barrier to detect writes into old objects.</p>
<p>The GC is also
<a href="https://en.wikipedia.org/wiki/Tracing_garbage_collection#Stop-the-world_vs._incremental_vs._concurrent">incremental</a>,
which means that its major collections aren't done all at once (which would
lead to long pauses). Instead, major collections are sliced up into small
steps, which are done directly after a minor collection (the GC isn't
<em>concurrent</em> though, which would mean that the GC does work in a separate
thread).</p>
<p>The incremental GC uses <a href="https://en.wikipedia.org/wiki/Tracing_garbage_collection#Tri-color_marking">tri-color
marking</a>
to reason about the reachable part of the heap during the marking phase, where
every old object can be:</p>
<ul>
<li>black: already marked, reachable, definitely survives the collection</li>
<li>grey: will survive, but still needs to be marked</li>
<li>white: potentially dead</li>
</ul>
<p>The color of every object is encoded by setting flags
in the object header.</p>
<p>The GC maintains the <strong>invariant</strong> that black objects must never point to white
objects. At the start of a major collection cycle the stack roots are turned
gray. During the mark phase of a major collection cycle, the GC will trace gray
objects, until
none are left. To trace a gray object, all the objects it references have to be
marked grey if they are white so far. After a grey object is traced, it can be
marked black (because all the referenced objects are now either black or gray).
Eventually, there are no gray objects left. At that point (because no white
object can be reached from a black one) all the white objects are known to be
unreachable and can therefore be freed.</p>
<p>The GC is incremental because every collection step will only trace a limited
number of gray objects, before giving control back to the program. This leads to
a problem: if an already traced (black) object is changed between two marking
steps of the GC, the program can mutate that object and write a new reference
into one of its fields. This could lead to an invariant violation, if the
referenced object is white. Therefore, the GC uses the write barrier (which it
needs anyway to find references from old to young objects) to mark all black
objects that are modified gray, and then trace them again at one of the
later collection steps.</p>
<h3>The special write barrier of memcopy</h3>
<p>Arrays use a different kind of write barrier than normal objects. Since they
can be arbitrarily large, tracing them can take a long time. Therefore it's
potentially wasteful to trace them fully at a minor collection. To fix this,
the array write barrier keeps more granular information about which parts of
the array have been modified since the last collection step. Then only the
modified parts of the array need to be traced, not the whole array.</p>
<p>In addition, there is another optimization for arrays, which is that memcopy is
treated specially by the GC. If memcopy is implemented by simply writing a loop
that copies the content of one array to the other, that will invoke the write
barrier every single loop iteration for the write of every array element,
costing a lot of overhead. Here's some pseudo-code:</p>
<div class="code"><pre class="code literal-block"><span class="k">def</span> <span class="nf">arraycopy</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">dest</span><span class="p">,</span> <span class="n">source_start</span><span class="p">,</span> <span class="n">dest_start</span><span class="p">,</span> <span class="n">length</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">length</span><span class="p">):</span>
<span class="n">value</span> <span class="o">=</span> <span class="n">source</span><span class="p">[</span><span class="n">source_start</span> <span class="o">+</span> <span class="n">i</span><span class="p">]</span>
<span class="n">dest</span><span class="p">[</span><span class="n">dest_start</span> <span class="o">+</span> <span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span> <span class="c1"># <- write barrier inserted here</span>
</pre></div>
<p>Therefore the GC has a special memcopy-specific
write barrier that will perform the GC logic once before the memcopy loop, and
then use a regular (typically SIMD-optimized) memcopy implementation from
<code>libc</code>. Roughly like this:</p>
<div class="code"><pre class="code literal-block"><span class="k">def</span> <span class="nf">arraycopy</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">dest</span><span class="p">,</span> <span class="n">source_start</span><span class="p">,</span> <span class="n">dest_start</span><span class="p">,</span> <span class="n">length</span><span class="p">):</span>
<span class="n">gc_writebarrier_before_array_copy</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">dest</span><span class="p">,</span> <span class="n">source_start</span><span class="p">,</span> <span class="n">dest_start</span><span class="p">,</span> <span class="n">length</span><span class="p">)</span>
<span class="n">raw_memcopy</span><span class="p">(</span><span class="n">cast_to_voidp</span><span class="p">(</span><span class="n">source</span><span class="p">)</span> <span class="o">+</span> <span class="n">source_start</span><span class="p">,</span>
<span class="n">cast_to_voidp</span><span class="p">(</span><span class="n">dest</span><span class="p">)</span> <span class="o">+</span> <span class="n">dest_start</span><span class="p">,</span>
<span class="n">sizeof</span><span class="p">(</span><span class="n">itemtype</span><span class="p">(</span><span class="n">source</span><span class="p">))</span> <span class="o">*</span> <span class="n">length</span><span class="p">)</span>
</pre></div>
<p>(this is really a rough sketch. The <a href="https://github.com/pypy/pypy/blob/789f964fff59c722b0872abcdc56d2b1373a9f3b/rpython/rlib/rgc.py#L365">real
code</a>
is much more complicated.)</p>
<h3>The bug</h3>
<p>The bugs turned out to be precisely in this memcopy write barrier. When we
implemented the current GC, we adapted our previous GC, which was a
generational mark-sweep GC but <em>not</em> incremental. We started with most of the
previous GC's code, including the write barriers. The regular write barriers
were adapted to the new incremental assumptions, in particular the need for the
write barrier to also turn black objects back to gray when they are modified
during a marking phase. This was simply not done at all for the memcopy write
barrier, at least in two of the code paths. Fixing this problem fixes the unit
tests and stops the crashes.</p>
<h2>Reflections</h2>
<p>The way the bug was introduced is really typical. A piece of code (the memcopy
write barrier) was written under a set of assumptions. Then those assumptions
changed later. Not all the code pieces that relied on these assumptions to be
correct were updated. It's pretty hard to prevent this in all situations.</p>
<p>I still think we could have done more to prevent the bug occurring. Writing a
property-based test for the GC would have been a good idea given the complexity
of the GC, and definitely something we did in other parts of our code at the
time (just using the <code>random</code> module mostly, we started using hypothesis
later).</p>
<p>It's a bit of a mystery to me why this bug managed to be undetected for so
long. Memcopy happens in a lot of pretty core operations of e.g. lists in
Python (<code>list.extend</code>, to name just one example). To speculate, I would suspect
that all the other preconditions for the bug occurring made it pretty rare:</p>
<ul>
<li>the content of an old list that is not yet marked needs to be copied into
another old list that is marked already</li>
<li>the source of the copy needs to also store an object that has no other
references</li>
<li>the source of the copy then needs to be overwritten with other data</li>
<li>then the next collection steps need to be happening at the right points</li>
<li>...</li>
</ul>
<p>Given the complexity of the GC logic I also wonder whether some lightweight
formal methods would have been a good idea. Formalizing some of the core
invariants in <a href="https://en.wikipedia.org/wiki/B-Method">B</a> or
<a href="https://en.wikipedia.org/wiki/TLA%2B">TLA+</a> and then <a href="https://en.wikipedia.org/wiki/Model_checking">model
checking</a> them up to some number
of
objects would have found this problem pretty quickly. There are also correctness
proofs for GC algorithms in some research papers, but I don't have a good
overview of the literature to point to any that are particularly good or bad.
Going such a more formal route might have fixed this and probably a whole bunch
of other bugs, but of course it's a pretty expensive (and tedious) approach.</p>
<p>While it was super annoying to track this down, it was definitely good to learn
a bit more about how to use rr and the GDB scripting interface.</p>
<h2>Bonus Section: The Wrong Assertion</h2>
<p>Some more technical information about the wrong assertion is in this section.</p>
<h3>Background: pre-built objects</h3>
<p>PyPy's VM-building bootstrapping process can "freeze" a bunch of heap objects
into the final binary. This allows the VM to start up quickly, because those
frozen objects are loaded by the OS as part of the binary.</p>
<p>Those frozen pre-built objects are parts of the 'roots' of the garbage
collector and need to be traced. However, tracing all the pre-built objects at
every collection would be very expensive, because there are a lot of them
(about 150,000 in a PyPy 3.10 binary). Tracing them all is also not necessary,
because most of them are never modified. Unmodified pre-built objects can only reference
other pre-built objects, which can never be deallocated anyway. Therefore we
have an optimization that uses the write barrier (which we need anyway to find
old-to-young pointers) to notice when a pre-built object gets modified for the
very first time. If that happens, it gets added to the set of pre-built objects
that gets counted as a root, and is traced as a root at collections
from then on.</p>
<h3>The wrong assertion</h3>
<p>The assertion that triggered when I turned on the GC debug mode was saying that
the GC found a reference from a black to a white object, violating its
invariant. Unmodified pre-built objects count as black, and they aren't roots,
because they can only ever reference other pre-built objects. However, when a
pre-built object gets modified for the first time, it becomes part of the root
set and will be marked gray. This logic works fine.</p>
<p>The wrong assertion triggers if a pre-built object is mutated for the very
first time in the middle of an incremental marking phase. While the pre-built
object gets added to the root set just fine, and will get traced before the
marking phase ends, this is encoded slightly differently for pre-built objects,
compared to "regular" old objects. Therefore, the invariant checking code
wrongly reported a black->white pointer in this situation.</p>
<p>To fix it I also wrote a unit test checking the problem, made sure that the GC
hypothesis test also found the bug, and then fixed the wrong assertion to take
the color encoding of pre-built objects into account.</p>
<p>The bug managed to be invisible because we don't tend to turn on the GC
assertions very often. We only do that when we find a GC bug, which is of
course also when we need it the most to be correct.</p>
<h2>Acknowledgements</h2>
<p>Thanks to Matti Picus, Max Bernstein, Wouter van Heyst for giving me feedback on drafts of the
post. Thanks to Armin Rigo for reviewing the code and pointing out holes in my
thinking. Thanks to the original reporters of the various forms of the bug,
including Lily Foote, David Hewitt, Wenzel Jakob.</p>2024-03-26T19:14:09+00:00Real Python: Finding Python Easter Eggs
https://realpython.com/courses/finding-python-easter-eggs/
<p>In this <strong>Code Conversation</strong>, you’ll follow a chat between Philipp and Bartosz as they go on an Easter egg hunt. Along the way, you’ll:</p>
<ul>
<li>Learn about Easter egg hunt traditions</li>
<li>Uncover the first Easter egg in software</li>
<li>Explore Easter eggs in Python</li>
</ul>
<p>There won’t be many code examples in this Code Conversation, so you can lean back and join Philipp and Bartosz on their Easter egg hunt.</p>
<hr />
<p><em>[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>2024-03-26T14:00:00+00:00Robin Wilson: My geospatial PDF talk at FOSS4G 2021
https://blog.rtwilson.com/my-geospatial-pdf-talk-at-foss4g-2021/
<p>This is only about 3 years late – but I gave a talk at <a href="https://2021.foss4g.org/">FOSS4G 2021</a> on geospatial PDFs. The full title was:</p>
<p><strong>From static PDFs to interactive, geospatial PDFs, or, ‘I never knew PDFs could do that!’</strong></p>
<p>The video is below:</p>
<p></p>
<p>In the talk I cover what a geospatial PDF is, how to export as a geospatial PDF from QGIS, how to import that PDF again to extract the geospatial data from it, how to create geospatial PDFs using GDAL (including styling vector data) – and then take things to the nth degree by showing a fully interactive geospatial PDF, providing a UI within the PDF file. Some people attending the talk described it as "the best talk of the conference"!</p>
<p>A few relevant resources are below:</p>
<ul>
<li><a href="https://gdal.org/programs/gdal_create.html">gdal_create documentation</a></li>
<li><a href="https://gdal.org/drivers/vector/pdf.html#vector-pdf">GDAL PDF driver documentation – vector</a></li>
<li><a href="https://gdal.org/drivers/raster/pdf.html">GDAL PDF driver documentation – raster</a></li>
<li><a href="https://gdal.org/drivers/raster/pdf.html#creation-of-pdf-file-from-a-xml-composition-file-gdal-3-0">GDAL PDF Composition File documentation</a></li>
<li><a href="https://gdal.org/user/ogr_feature_style.html#ogr-feature-style">GDAL feature style specification</a></li>
</ul>2024-03-26T10:32:51+00:00Python Bytes: #376 Every dunder method in a Python Lockbox
https://pythonbytes.fm/episodes/show/376/every-dunder-method-in-a-python-lockbox
<strong>Topics covered in this episode:</strong><br>
<ul>
<li><a href="https://micro.webology.dev/2024/03/20/on-robotstxt.html?utm_source=pocket_saves"><strong>🤖</strong></a> <a href="https://micro.webology.dev/2024/03/20/on-robotstxt.html"><strong>On Robots.txt</strong></a></li>
<li><a href="https://github.com/jawah/niquests"><strong>niquests</strong></a></li>
<li><a href="https://www.pythonmorsels.com/every-dunder-method/"><strong>Every dunder method in Python</strong></a></li>
<li><a href="https://github.com/mkjt2/lockbox"><strong>Lockbox</strong></a></li>
<li><strong>Extras</strong></li>
<li><strong>Joke</strong></li>
</ul><a href='https://www.youtube.com/watch?v=wohUfOSl18Q' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="376">Watch on YouTube</a><br>
<p><strong>About the show</strong></p>
<p>Sponsored by ScoutAPM: <a href="https://pythonbytes.fm/scout"><strong>pythonbytes.fm/scout</strong></a></p>
<p><strong>Connect with the hosts</strong></p>
<ul>
<li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a></li>
<li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a></li>
<li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a></li>
</ul>
<p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too.</p>
<p><strong>Brian #1:</strong> <a href="https://micro.webology.dev/2024/03/20/on-robotstxt.html?utm_source=pocket_saves"><strong>🤖</strong></a> <a href="https://micro.webology.dev/2024/03/20/on-robotstxt.html"><strong>On Robots.txt</strong></a></p>
<ul>
<li>Jeff Triplett</li>
<li>“In theory, this file helps control what search engines and AI scrapers are allowed to visit, but I need more confidence in its effectiveness in the post-AI apocalyptic world.”</li>
<li>Resources to get started
<ul>
<li><a href="https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/">Block the Bots that Feed “AI” Models by Scraping Your Website</a></li>
<li><a href="https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers/">Go ahead and block AI web crawlers</a></li>
<li><a href="https://darkvisitors.com/">Dark Visitors</a></li>
<li>Django
<ul>
<li><a href="https://learndjango.com/tutorials/add-robotstxt-django-website">Add robots.txt to a Django website</a></li>
<li><a href="https://adamj.eu/tech/2020/02/10/robots-txt/">How to add a robots.txt to your Django site</a></li>
</ul></li>
<li>Hugo
<ul>
<li><a href="https://gohugo.io/templates/robots/">Hugo robots.txt</a></li>
</ul></li>
</ul></li>
<li>Podcast questions:
<ul>
<li>Should content creators block AI from our work?</li>
<li>Should’t we set up a standard way to do this?</li>
<li>I still haven’t found a way to block GitHub repositories.
<ul>
<li>Is there a way?</li>
<li>Licensing is one thing (not easy), but I don’t think any bots respect any protocol for repos.</li>
</ul></li>
</ul></li>
</ul>
<p><strong>Michael #2:</strong> <a href="https://github.com/jawah/niquests"><strong>niquests</strong></a></p>
<ul>
<li>Requests but with HTTP/3, HTTP/2, Multiplexed Connections, System CAs, Certificate Revocation, DNS over HTTPS / TLS / QUIC or UDP, Async, DNSSEC, and (much) pain removed!</li>
<li><strong>Niquests</strong> is a simple, yet elegant, HTTP library. It is a drop-in replacement for <strong>Requests</strong>, which is under feature freeze.</li>
<li><strong>See why you should switch:</strong> <a href="https://medium.com/dev-genius/10-reasons-you-should-quit-your-http-client-98fd4c94bef3">Read about 10 reasons why</a></li>
</ul>
<p><strong>Brian #3:</strong> <a href="https://www.pythonmorsels.com/every-dunder-method/"><strong>Every dunder method in Python</strong></a></p>
<ul>
<li>Trey Hunner</li>
<li>Sure, there’s <code>__repr__()</code>, <code>__str__()</code>, and <code>__init__()</code>, but how about dunder methods for:
<ul>
<li>Equality and hashability</li>
<li>Orderability</li>
<li>Type conversions and formatting</li>
<li>Context managers</li>
<li>Containers and collections</li>
<li>Callability</li>
<li>Arithmetic operators</li>
<li>… and so much more … even a cheat sheet.</li>
</ul></li>
</ul>
<p><strong>Michael #4:</strong> <a href="https://github.com/mkjt2/lockbox"><strong>Lockbox</strong></a></p>
<ul>
<li>Lockbox is a forward proxy for making third party API calls.</li>
<li>Why? Automation or workflow platforms like Zapier and IFTTT allow "webhook" actions for interacting with third party APIs.</li>
<li>They require you to provide your third party API keys so they can act on your behalf. You are trusting them to keep your API keys safe, and that they do not misuse them.</li>
<li>How Lockbox helps: When a workflow platform needs to make a third party API call on your behalf, it makes a Lockbox API call instead. Lockbox makes the call to the third party API, and returns the result to the workflow platform.</li>
</ul>
<p><strong>Extras</strong> </p>
<p>Brian:</p>
<ul>
<li><a href="https://adamj.eu/tech/2024/02/10/django-join-community-mastodon/"><strong>Django: Join the community on Mastodon</strong></a> - Adam Johnson</li>
<li><a href="https://unmaintained.tech/"><strong>No maintenance intended</strong></a> - Sent in from Kim van Wyk</li>
</ul>
<p>Michael:</p>
<ul>
<li>US sues Apple
<ul>
<li><a href="https://www.youtube.com/watch?v=_O5XMMvGJ1M">Good video on pluses and minuses</a></li>
<li><a href="https://www.youtube.com/watch?v=A69-8XxLbJ4">The hot water just the day before</a> [<a href="https://www.youtube.com/watch?v=4ut-de57A2c">and this one</a>]</li>
<li><a href="https://9to5mac.com/2024/03/25/app-store-proposals-rejected/">https://9to5mac.com/2024/03/25/app-store-proposals-rejected/</a> </li>
</ul></li>
<li><a href="https://twitter.com/thepsf/status/1770528868111130683?s=12&t=RL7Nk7OAFSptvENxe1zIqA">PyPI Support Specialist job</a></li>
<li><a href="https://www.youtube.com/watch?v=Jh24NVM2FDY">VS Code AMA</a>, please <a href="https://forms.gle/thh3pYteN3dGYYvN9">submit your question here</a> </li>
<li><a href="https://fosstodon.org/@gthomas/112158142020246243">PyData Eindhoven 2024</a> has a date and open CFP</li>
</ul>
<p><strong>Joke:</strong> <a href="https://ioc.exchange/@rye/112079906909625874"><strong>Windows Certified</strong></a></p>2024-03-26T08:00:00+00:00Armin Ronacher: On Tech Debt: My Rust Library is now a CDO
http://lucumr.pocoo.org/2024/3/26/rust-cdo
<p>You're probably familiar with tech debt. There is a joke that if there is
tech debt, surely there must be derivatives to work with that debt? I'm
happy to say that the Rust ecosystem has created an environment where it
looks like one solution for tech debt is collateralization.</p>
<p>Here is how this miracle works. Say you have a library <a class="reference external" href="https://github.com/mitsuhiko/insta">stuff</a> which depends on some other
library <a class="reference external" href="https://github.com/chyh1990/yaml-rust">learned-rust-this-way</a>.
The author of <cite>learned-rust-this-way</cite> at one point lost interest in this
thing and issues keep piling up. Some of those issues are feature
requests, others are legitimate bugs. However you as the person that
wrote <cite>stuff</cite> never ran into any of those problems. Yet it's hard to
argue that <cite>learned-rust-this-way</cite> isn't tech debt. It's one that does
not bother you all that much, but it's debt nonetheless.</p>
<p>At one point someone else figures out that <cite>learned-rust-this-way</cite> is debt.
One of the ways in which this happens is because the name is great.
Clearly that's not the only person that learned Rust this way and someone
else also wants that name. Except the original author is unreachable. So
now there is one more reason for that package <a class="reference external" href="https://github.com/rustsec/advisory-db/issues/1921">to get added to the RUSTSEC
database</a> and all
the sudden all hell breaks lose. Within minutes CI will start failing for
a lot of people that directly or indirectly use <cite>learned-rust-this-way</cite>
notifying them that something happened. That's because <cite>RUSTSEC</cite> is
basically a rating agency and they decided that your debt is now junk.</p>
<p>What happens next? As the maintainer of <cite>stuff</cite> your users all the sudden
start calling you out for using <cite>learned-rust-this-way</cite> and you suffer.
Stress levels increase. You gotta unload that shit. Why? Not because it
does not work for you, but someone called you out of that debt. If we
really want to stress the financial terms this is your margin call. Your
users demand action to deal with your debt.</p>
<p>So what can you do? One option is to move to alternatives (unload the
debt). In this particular case for whatever reason all the alternatives
to <cite>learned-rust-this-way</cite> are not looking very appealing either. One is
a fork of that thing which also only has a single maintainer, but all the
sudden pulls in 3 more dependencies, one of which already have a "B-"
rating. Another option in the ecosystem just decided <a class="reference external" href="https://github.com/dtolnay/serde-yaml/commit/3ba8462f7d3b603d832e0daeb6cfc7168a673d7a">to default</a>
before they are called out.</p>
<p>Remember you never touched <cite>learned-rust-this-way</cite> actively. It worked
for you in the unmaintained way of the last four years. If you now fork
that library (and name it <cite>learned-rust-this-way-and-its-okay</cite>) you are
now subject to the same demands. Forking that library is putting cash on
the pile of debt. Except if you don't act on the bug reports there,
you will eventually be called out like <cite>learned-rust-this-way</cite> was. So
while that might buy you time, it does not really solve the issue.</p>
<p>However here is what actually does work: you just merge that code into
your own library. Now that junk tech debt is suddenly rated “AAA”. For
as long as you never touch that code any more, you never reveal to anyone
that you did that, and you just keep maintaining your library like you did
before, the world keeps spinning on.</p>
<p>So as of today: I collateralized <cite>yaml-rust</cite> by vendoring it in <cite>insta</cite>.
It's now an amalgamation of insta code and yaml-rust. And by doing so, I
successfully upgraded this junk tech debt to a perfect AAA.</p>
<p>Who won? I think nobody really.</p>
<small><p>As for the title: a <a class="reference external" href="https://en.wikipedia.org/wiki/Collateralized_debt_obligation">CDO</a>
is a financial instrument that became pretty infamous during the financial
crisis of 2007. An entertaining explanation of that can be found in
“<a class="reference external" href="https://en.wikipedia.org/wiki/The_Big_Short_(film)">The Big Short</a>”.</p>
</small>2024-03-26T00:00:00+00:00PyCharm: PyCharm 2024.1 Release Candidate Is Out!
https://blog.jetbrains.com/pycharm/2024/03/pycharm-2024-1-release-candidate-is-out/
<p>The PyCharm 2024.1 RC is now available!</p>
<p>You can get the latest build from our <a href="https://www.jetbrains.com/pycharm/nextversion/" target="_blank" rel="noreferrer noopener">website</a>, through the free <a href="https://www.jetbrains.com/toolbox-app/" target="_blank" rel="noreferrer noopener">Toolbox App</a>, or via snaps for Ubuntu.</p>
<p align="center"><a class="jb-download-button" href="https://www.jetbrains.com/pycharm/nextversion/" target="_blank" rel="noopener">Download PyCharm 2024.1 RC</a></p>
<p><em><strong>T</strong></em><strong><em>o use this build, you need to have an active subscription to PyCharm Professional.</em></strong></p>
<p>With the major release on the horizon, there’s no better time to explore the newly introduced features before the official launch.</p>
<p>Our latest build integrates all of the significant updates introduced during the <a href="https://blog.jetbrains.com/pycharm/tag/2024-1-eap/">PyCharm 2024.1 Early Access Program</a>. Here’s a short recap of the new features aimed at enhancing various aspects of your development workflows: </p>
<ul>
<li>Full line code completion, now for Python, JavaScript, and TypeScript</li>
<li>A revamped <em>Terminal</em> tool window</li>
<li>Sticky lines in the editor </li>
<li>In-editor code reviews</li>
<li>Enriched support for GitHub Actions</li>
<li>WireMock server support </li>
<li>And many more </li>
</ul>
<p>To learn more about these and other improvements, check out the posts tagged under the <a href="https://blog.jetbrains.com/pycharm/tag/2024-1-eap/">PyCharm 2024.1 EAP</a> section on our blog.</p>
<p>Although the addition of new features has finished and the team is now refining those included in v2024.1, we still have updates to share. Take a closer look!</p>
<h2 class="wp-block-heading" id="ai-assistant">AI Assistant </h2>
<p>Beginning with the Beta version of PyCharm 2024.1, AI Assistant has been unbundled and is now available as a separate <a href="https://plugins.jetbrains.com/plugin/22282-ai-assistant" target="_blank" rel="noreferrer noopener">plugin</a>. This change is driven by the need to offer greater flexibility and control over your various preferences and requirements, enabling you to choose if and when you’d like to use AI-powered technologies in your working environments.</p>
<p>That’s a wrap! For the full list of updates in the latest build, please refer to the <a href="https://youtrack.jetbrains.com/articles/PY-A-233538193/PyCharm-2024.1-RC-241.14494.200-build-Release-Notes" target="_blank" rel="noreferrer noopener">release notes</a>.</p>
<p>As we put the final touches to ensure a flawless release, we’d like to thank all participants who actively contributed to the Early Access Program for version 2024.1.</p>
<p>You can drop us a line in the comments below or reach out to us on <a href="https://twitter.com/pycharm" target="_blank" rel="noreferrer noopener">X (formerly Twitter)</a> – we’re always looking to benefit from your input. Finally, if you happen to spot any bugs, please report them using our <a href="https://youtrack.jetbrains.com/issues/PY" target="_blank" rel="noreferrer noopener">issue tracker</a>.</p>2024-03-25T22:02:24+00:00Real Python: Prompt Engineering: A Practical Example
https://realpython.com/practical-prompt-engineering/
<div><p>You’ve used <a href="https://realpython.com/chatgpt-coding-mentor-python/">ChatGPT</a>, and you understand the potential of using a <strong>large language model (LLM)</strong> to assist you in your tasks. Maybe you’re already working on an LLM-supported application and have read about <strong>prompt engineering</strong>, but you’re unsure how to translate the theoretical concepts into a practical example.</p>
<p>Your text prompt instructs the LLM’s responses, so tweaking it can get you vastly different output. In this tutorial, you’ll apply multiple <strong>prompt engineering techniques</strong> to a real-world example. You’ll experience prompt engineering as an iterative process, see the effects of applying various techniques, and learn about related concepts from machine learning and data engineering.</p>
<p><strong>In this tutorial, you’ll learn how to:</strong></p>
<ul>
<li>Work with OpenAI’s <strong>GPT-3.5</strong> and <strong>GPT-4</strong> models through their <strong>API</strong></li>
<li>Apply prompt engineering techniques to a <strong>practical, real-world example</strong></li>
<li>Use <strong>numbered steps</strong>, <strong>delimiters</strong>, and <strong>few-shot prompting</strong> to improve your results</li>
<li>Understand and use <strong>chain-of-thought prompting</strong> to add more context</li>
<li>Tap into the power of <strong>roles</strong> in messages to go beyond using singular <strong>role prompts</strong></li>
</ul>
<p>You’ll work with a <strong>Python script</strong> that you can repurpose to fit your own LLM-assisted task. So if you’d like to use practical examples to discover how you can use prompt engineering to get better results from an LLM, then you’ve found the right tutorial!</p>
<div class="alert alert-warning">
<p><strong>Get Your Code:</strong> <a href="https://realpython.com/bonus/prompt-engineering-code/" class="alert-link">Click here to download the sample code </a> that you’ll use to get the most out of large language models through prompt engineering.</p>
</div>
<div class="alert alert-primary">
<p><strong><span class="icon baseline"></span> Take the Quiz:</strong> Test your knowledge with our interactive “Practical Prompt Engineering” quiz. Upon completion you will receive a score so you can track your learning progress over time:</p>
<p class="text-center my-2"><a class="btn btn-primary" href="https://realpython.com/quizzes/practical-prompt-engineering/" target="_blank">Take the Quiz »</a></p>
</div>
<h2 id="understand-the-purpose-of-prompt-engineering">Understand the Purpose of Prompt Engineering<a class="headerlink" href="https://realpython.com/atom.xml#understand-the-purpose-of-prompt-engineering" title="Permanent link"></a></h2>
<p>Prompt engineering is more than a buzzword. You can get vastly different output from an LLM when using different prompts. That may seem obvious when you consider that you get different output when you ask different questions—but it also applies to phrasing the same conceptual question differently. Prompt engineering means constructing your text input to the LLM using specific approaches.</p>
<p>You can think of prompts as arguments and the LLM as the function to which you pass these arguments. Different input means different output:</p>
<div class="codeblock mb-3 w-100">
<div class="codeblock__header d-flex justify-content-between codeblock--blue">
<span class="mr-2 noselect">Python</span>
<div class="noselect">
<span class="codeblock__output-toggle" title="Toggle prompts and output"><span class="icon baseline js-codeblock-output-on codeblock__header--icon-lower"></span></span>
</div>
</div>
<div>
<div class="highlight highlight--with-header"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">hello</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Hello, </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">!"</span><span class="p">)</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">hello</span><span class="p">(</span><span class="s2">"World"</span><span class="p">)</span>
<span class="go">Hello, World!</span>
<span class="gp">>>> </span><span class="n">hello</span><span class="p">(</span><span class="s2">"Engineer"</span><span class="p">)</span>
<span class="go">Hello, Engineer!</span>
</code></pre></div>
<button class="codeblock__copy btn btn-outline-secondary border m-1 px-1 d-hover-only" title="Copy to clipboard"><span class="icon baseline"></span></button>
<span class="small"><span class="icon baseline mr-1 text-success"></span>Copied!</span>
</div>
</div>
<p>While an LLM is much more complex than the toy function above, the fundamental idea holds true. For a successful function call, you’ll need to know exactly which argument will produce the desired output. In the case of an LLM, that argument is text that consists of many different <strong>tokens</strong>, or <a href="https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them">pieces of words</a>.</p>
<div class="alert alert-primary">
<p><strong>Note:</strong> The analogy of a function and its arguments has a caveat when dealing with OpenAI’s LLMs. While the <code>hello()</code> function above will always return the same result given the same input, the results of your LLM interactions won’t be 100 percent deterministic. This is currently inherent to how these models operate.</p>
</div>
<p>The field of prompt engineering is still changing rapidly, and there’s a lot of active research happening in this area. As LLMs continue to evolve, so will the prompting approaches that will help you achieve the best results.</p>
<p>In this tutorial, you’ll cover some prompt engineering techniques, along with approaches to iteratively developing prompts, that you can use to get better text completions for your own LLM-assisted projects:</p>
<ul>
<li><a href="https://realpython.com/atom.xml#describe-your-task">Zero-Shot Prompting</a></li>
<li><a href="https://realpython.com/atom.xml#use-few-shot-prompting-to-improve-output">Few-Shot Prompting</a></li>
<li><a href="https://realpython.com/atom.xml#use-delimiters-to-clearly-mark-sections-of-your-prompt">Delimiters</a></li>
<li><a href="https://realpython.com/atom.xml#describe-your-request-in-numbered-steps">Numbered Steps</a></li>
<li><a href="https://realpython.com/atom.xml#add-a-role-prompt-to-set-the-tone">Role Prompts</a></li>
<li><a href="https://realpython.com/atom.xml#walk-the-model-through-chain-of-thought-prompting">Chain-of-Thought (CoT) Prompting</a></li>
<li><a href="https://realpython.com/atom.xml#structure-your-output-format-as-json">Structured Output</a></li>
<li><a href="https://realpython.com/atom.xml#improve-your-output-with-the-power-of-conversation">Labeled Conversations</a></li>
</ul>
<p>There are more techniques to uncover, and you’ll also find links to additional resources in the tutorial. Applying the mentioned techniques in a practical example will give you a great starting point for improving your LLM-supported programs. If you’ve never worked with an LLM before, then you may want to peruse <a href="https://platform.openai.com/docs/guides/gpt">OpenAI’s GPT documentation</a> before diving in, but you should be able to follow along either way.</p>
<h2 id="get-to-know-the-practical-prompt-engineering-project">Get to Know the Practical Prompt Engineering Project<a class="headerlink" href="https://realpython.com/atom.xml#get-to-know-the-practical-prompt-engineering-project" title="Permanent link"></a></h2>
<p>You’ll explore various prompt engineering techniques in service of a practical example: <a href="https://en.wikipedia.org/wiki/Data_sanitization">sanitizing</a> customer chat conversations. By practicing different prompt engineering techniques on a single real-world project, you’ll get a good idea of why you might want to use one technique over another and how you can apply them in practice.</p>
<p>Imagine that you’re the resident Python developer at a company that handles thousands of customer support chats on a daily basis. Your job is to format and sanitize these conversations. You also help with deciding which of them require additional attention.</p>
<h3 id="collect-your-tasks">Collect Your Tasks<a class="headerlink" href="https://realpython.com/atom.xml#collect-your-tasks" title="Permanent link"></a></h3>
<p>Your big-picture assignment is to help your company stay on top of handling customer chat conversations. The conversations that you work with may look like the one shown below:</p>
<div class="codeblock mb-3 w-100">
<div class="codeblock__header d-flex justify-content-between codeblock--grey">
<span class="mr-2 noselect">Text</span>
<div class="noselect">
</div>
</div>
<div>
<div class="highlight highlight--with-header"><pre><span></span><code>[support_tom] 2023-07-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2023-07-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2023-07-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2023-07-24T10:04:03+00:00 : Blast! You're right!
</code></pre></div>
<button class="codeblock__copy btn btn-outline-secondary border m-1 px-1 d-hover-only" title="Copy to clipboard"><span class="icon baseline"></span></button>
<span class="small"><span class="icon baseline mr-1 text-success"></span>Copied!</span>
</div>
</div>
<p>You’re supposed to make these text conversations more accessible for further processing by the customer support department in a few different ways:</p>
<blockquote>
<ul>
<li>Remove personally identifiable information.</li>
<li>Remove swear words.</li>
<li>Clean the date-time information to only show the date.</li>
</ul>
</blockquote>
</div><h2><a href="https://realpython.com/practical-prompt-engineering/?utm_source=realpython&utm_medium=rss">Read the full article at https://realpython.com/practical-prompt-engineering/ »</a></h2>
<hr />
<p><em>[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>2024-03-25T14:00:00+00:00Zato Blog: Systems Automation in Python
https://zato.io/en/blog/automation-in-python.html
<h1 class="insights-header">
<a class="rounded nav-title" href="https://zato.io/en/blog/automation-in-python.html">Systems Automation in Python</a>
</h1>
<div class="article-meta">
2024-03-25, by Dariusz Suchojad
<img src="https://upcdn.io/kW15bqq/raw/root/static/blog/authors/dsuch.webp" alt="" width="35" class="insights-author" />
</div>
<p>How to automate systems in Python and how the Zato <a href="https://zato.io/index.html">Python integration platform</a> differs from a network
automation tool, how to start using it, along with a couple of examples of integrations with
Office 365 and Jira, is what the latest article is about.</p>
<p><strong>➤ Read it here: <a href="https://zato.io/articles/automation-in-python.html">Systems Automation in Python</a>.</strong></p>
<p><a href="https://zato.io/articles/automation-in-python.html"><img src="https://upcdn.io/kW15bqq/raw/root/en/docs/3.2/gfx/api/networks/network-architecture.png" class="img-responsive docs-article" alt="Systems Automation in Python" />
</a></p>
<p><a href="https://zato.io/articles/automation-in-python.html"><img src="https://upcdn.io/kW15bqq/raw/root/en/docs/3.2/gfx/api/networks/scope-comparison.png" class="img-responsive docs-article" alt="Systems Automation in Python" />
</a></p>
<div class="nav-title footer">
<a class="rounded nav-title" href="https://zato.io/en/blog/">More blog posts</a><span class="arrow-footer">➤</span>
</div>2024-03-25T08:00:00+00:00Amjith Ramanujam: Rapid Prototyping in Python
https://blog.amjith.com/rapid-prototyping-in-python
<div class="posthaven-post-body"><p>I was recently assigned to a new project at work. Like any good software engineer I started writing the pseudocode for the modules. We use C++ at work to write our programs.</p>
<p>I quickly realized it's not easy to translate programming ideas to English statements without a syntactic structure. When I was whining about it to Vijay, he told me to try prototyping it in Python instead of writing pseudocode. Intrigued by this, I decided to write a prototype in Python to test how various modules will come together.</p>
<p>Surprisingly it took me a mere 2 hours to code up the prototype. I can't emphasize enough, how effortless it was in Python.</p>
<h2>What makes Python an ideal choice for prototyping:</h2>
<p><strong>Dynamically typed language:</strong></p>
<p>Python doesn't require you to declare the datatype of a variable. This lets you write a function that is generic enough to handle any kind of data. For eg:</p>
<div class="CodeRay">
<div class="code"><pre><span class="keyword">def</span> <span class="function">max_val</span>(a,b):
<span class="keyword">return</span> a <span class="keyword">if</span> a >b <span class="keyword">else</span> b</pre></div>
</div>
<p>This function can take integers, floats, strings, a combination of any of those, or lists, dictionaries, tuples, whatever.</p>
<p>A list in Python need not be homogenous. This is a perfectly good list:</p>
<div class="CodeRay">
<div class="code"><pre>[<span class="integer">1</span>, <span class="string"><span class="delimiter">'</span><span class="content">abc</span><span class="delimiter">'</span></span>, [<span class="integer">1</span>,<span class="integer">2</span>,<span class="integer">3</span>]]</pre></div>
</div>
<p>This lets you pack data in unique ways on the fly which can later be translated to a class or a struct in a statically typed language like C++.</p>
<div class="CodeRay">
<div class="code"><pre><span class="keyword">class</span> <span class="class">newDataType</span>
{
<span class="predefined-type">int</span> i;
String str;
Vector vInts;
};</pre></div>
</div>
<p><strong>Rich Set to Data-Structures:</strong></p>
<p>Built-in support for lists, dictionaries, sets, etc reduces the time involved in hunting for a library that provides you those basic data-structures.</p>
<p><strong>Expressive and Succinct:</strong></p>
<p>The algorithms that operate on the data-structures are intuitive and simple to use. The final code is more readable than a pseudocode.</p>
<p>For example: Lets check if a list has an element</p>
<div class="CodeRay">
<div class="code"><pre>>>> lst = [<span class="integer">1</span>,<span class="integer">2</span>,<span class="integer">3</span>] <span class="comment"># Create a list</span>
>>> res = <span class="integer">2</span> <span class="keyword">in</span> lst <span class="comment"># Check if 2 is in 'lst'</span>
<span class="predefined-constant">True</span></pre></div>
</div>
<p>If we have to do it in C++.</p>
<div class="CodeRay">
<div class="code"><pre>list lst;
lst.push_back(<span class="integer">3</span>);
lst.push_back(<span class="integer">1</span>);
lst.push_back(<span class="integer">7</span>);
list::iterator result = find(lst.begin(), lst.end(), <span class="integer">7</span>);
<span class="predefined-type">bool</span> res = (result != lst.end())</pre></div>
</div>
<p><strong>Python Interpreter and Help System:</strong></p>
<p>This is a huge plus. The presence of interpreter not only aids you in testing snippets of code, but it acts as an help system. Lets say we want to look up the functions that operate on a List.</p>
<div class="CodeRay">
<div class="code"><pre>>>> <span class="predefined">dir</span>([])
[<span class="string"><span class="delimiter">'</span><span class="content">__add__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__class__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__contains__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__delattr__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__delitem__</span><span class="delimiter">'</span></span>,
<span class="string"><span class="delimiter">'</span><span class="content">__delslice__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__doc__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__eq__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__format__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__ge__</span><span class="delimiter">'</span></span>,
<span class="string"><span class="delimiter">'</span><span class="content">__getattribute__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__getitem__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__getslice__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__gt__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__hash__</span><span class="delimiter">'</span></span>,
<span class="string"><span class="delimiter">'</span><span class="content">__iadd__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__imul__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__init__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__iter__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__le__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__len__</span><span class="delimiter">'</span></span>,
<span class="string"><span class="delimiter">'</span><span class="content">__lt__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__mul__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__ne__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__new__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__reduce__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__reduce_ex__</span><span class="delimiter">'</span></span>,
<span class="string"><span class="delimiter">'</span><span class="content">__repr__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__reversed__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__rmul__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__setattr__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__setitem__</span><span class="delimiter">'</span></span>,
<span class="string"><span class="delimiter">'</span><span class="content">__setslice__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__sizeof__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__str__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">__subclasshook__</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">append</span><span class="delimiter">'</span></span>,
<span class="string"><span class="delimiter">'</span><span class="content">count</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">extend</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">index</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">insert</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">pop</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">remove</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">reverse</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">sort</span><span class="delimiter">'</span></span>]
>>> help([].sort)
Help on built-<span class="keyword">in</span> function sort:
sort(...)
L.sort(cmp=<span class="predefined-constant">None</span>, key=<span class="predefined-constant">None</span>, reverse=<span class="predefined-constant">False</span>) -- stable sort *IN PLACE*;
<span class="predefined">cmp</span>(x, y) -> -<span class="integer">1</span>, <span class="integer">0</span>, <span class="integer">1</span></pre></div>
</div>
<p><strong> Advantages of prototyping instead of pseudocode: </strong></p>
<ul>
<li>The type definition of the datastructures emerge as we code. </li>
<li>The edge cases start to emerge when you prototype. </li>
<li>A set of required supporting routines. </li>
<li>A better estimation of the time required to complete a task. </li>
</ul></div>2024-03-24T16:35:37+00:00Kay Hayen: Nuitka Release 2.1
https://nuitka.net/posts/nuitka-release-21.html
<p>This is to inform you about the new stable release of <a class="reference external" href="https://nuitka.net">Nuitka</a>. It is the extremely compatible Python compiler,
<a class="reference external" href="https://nuitka.net/doc/download.html">“download now”</a>.</p>
<p>This release had focus on new features and new optimization. There is a
also a large amount of compatibility with things newly added to support
anti-bloat better, and workaround problems with newer package versions
that would otherwise need source code at run-time.</p>
<h2>Bug Fixes</h2>
<ul>
<li><p>Windows: Using older MSVC before 14.3 was not working anymore. Fixed
in 2.0.1 already.</p></li>
<li><p>Compatibility: The <code class="docutils literal notranslate"><span class="pre">dill-compat</span></code> plugin didn’t work for functions
with closure variables taken. Fixed in 2.0.1 already.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_local_closure</span><span class="p">(</span><span class="n">b</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">_local_multiply</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="k">return</span> <span class="n">x</span> <span class="o">*</span> <span class="n">y</span> <span class="o">+</span> <span class="n">b</span>
<span class="k">return</span> <span class="n">_local_multiply</span>
<span class="n">fn</span> <span class="o">=</span> <span class="n">get_local_closure</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">fn2</span> <span class="o">=</span> <span class="n">dill</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">dill</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">fn</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="n">fn2</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
</pre></div>
</div>
</li>
<li><p>Windows: Fix, sometimes <code class="docutils literal notranslate"><span class="pre">kernel32.dll</span></code> is actually reported as a
dependency, remove assertion against that. Fixed in 2.0.1 already.</p></li>
<li><p>UI: The help output for <code class="docutils literal notranslate"><span class="pre">--output-filename</span></code> was not formatted
properly. Fixed in 2.0.1 already.</p></li>
<li><p>Standalone: Added support for the <code class="docutils literal notranslate"><span class="pre">scapy</span></code> package. Fixed in 2.0.2
already.</p></li>
<li><p>Standalone: Added <code class="docutils literal notranslate"><span class="pre">PonyORM</span></code> implicit dependencies. Fixed in 2.0.2
already.</p></li>
<li><p>Standalone: Added support for <code class="docutils literal notranslate"><span class="pre">cryptoauthlib</span></code>, <code class="docutils literal notranslate"><span class="pre">betterproto</span></code>,
<code class="docutils literal notranslate"><span class="pre">tracerite</span></code>, <code class="docutils literal notranslate"><span class="pre">sklearn.util</span></code>, and <code class="docutils literal notranslate"><span class="pre">qt_material</span></code> packages. Fixed
in 2.0.2 already.</p></li>
<li><p>Standalone: Added missing data file for <code class="docutils literal notranslate"><span class="pre">scipy</span></code> package. Fixed in
2.0.2 already.</p></li>
<li><p>Standalone: Added missing DLLs for <code class="docutils literal notranslate"><span class="pre">speech_recognition</span></code> package.
Fixed in 2.0.2 already.</p></li>
<li><p>Standalone: Added missing DLL for <code class="docutils literal notranslate"><span class="pre">gmsh</span></code> package. Fixed in 2.0.2
already.</p></li>
<li><p>UI: Using reporting path in macOS dependency scan error message,
otherwise these contain home directory paths for no good reason.
Fixed in 2.0.2 already.</p></li>
<li><p>UI: Fix, could crash when compiling directories with trailing slashes
used. At least on Windows, this happened for the “/” slash value.
Fixed in 2.0.2 already.</p></li>
<li><p>Module: Fix, convenience option <code class="docutils literal notranslate"><span class="pre">--run</span></code> was not considering
<code class="docutils literal notranslate"><span class="pre">--output-dir</span></code> directory to load the result module. Without this,
the check for un-replaced module was always triggering for module
source in current directory, despite doing the right thing and
putting it elsewhere. Fixed in 2.0.2 already.</p></li>
<li><p>Python2: Avoid values for <code class="docutils literal notranslate"><span class="pre">__file__</span></code> of modules that are unicode
and solve a TODO that restores consistency over modules mode
<code class="docutils literal notranslate"><span class="pre">__file__</span></code> values. Fixed in 2.0.2 already.</p></li>
<li><p>Windows: Fix, short paths with and without dir name cached wrongly,
which could lead to shorted paths even where not asked for them.
Fixed in 2.0.2 already.</p></li>
<li><p>Fix, comparing list values that changed could segfault. This is a bug
fix Python did, that we didn’t follow yet and that became apparent
after using our dedicated list helpers more often. Fixed in 2.0.2
already.</p></li>
<li><p>Standalone: Added support for <code class="docutils literal notranslate"><span class="pre">tiktoken</span></code> package. Fixed in 2.0.2
already.</p></li>
<li><p>Standalone: Fix, namespace packages had wrong runtime <code class="docutils literal notranslate"><span class="pre">__path__</span></code>
value. Fixed in 2.0.2 already.</p></li>
<li><p>Python3.11: Fix, was using tuples from freelist of the wrong size</p>
<ul class="simple">
<li><p>CPython changed the index for the size, to not use zero, which was
wasteful when introduced with 3.10, but to <code class="docutils literal notranslate"><span class="pre">size-1</span></code> but we did
not follow that and then used a tuple one bit larger than
necessary.</p></li>
<li><p>As a result, code producing a lot short living tuples could end up
creating new ones over and over, causing bad memory allocations
and slow performance.</p></li>
</ul>
<p>Fixed in 2.0.2 already.</p>
</li>
<li><p>macOS: Fix, need to allow non-existent and versioned dependencies of
DLLs to themselves. Fixed in 2.0.2 already.</p></li>
<li><p>Windows: Fix PGO (Profile Guided Optimization) build errors with
MinGW64, this feature is not yet ready for general use, but these
errors shouldn’t happen. Fixed in 2.0.2 already.</p></li>
<li><p>Plugins: Fix, do not load <code class="docutils literal notranslate"><span class="pre">importlib_metadata</span></code> unless really
necessary.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">pkg_resources</span></code> plugin used to load it, and that then had
harmful effects for our handling of distribution information in some
configurations. Fixed in 2.0.3 already.</p>
</li>
<li><p>Plugins: Avoid warnings from plugin evaluated code, it could happen
that a <code class="docutils literal notranslate"><span class="pre">UserWarning</span></code> would be displayed during compilation. Fixed
in 2.0.3 already.</p></li>
<li><p>Fix, loading pickles with compiled functions in module mode was not
working. Fixed in 2.0.3 already.</p></li>
<li><p>Standalone: Added data files for <code class="docutils literal notranslate"><span class="pre">h2o</span></code> package. Fixed in 2.0.3
already.</p></li>
<li><p>Fix, variable assignment from variables that started to raise were
not recognized.</p>
<p>When a variable assignment from a variable became a raise expression,
that wasn’t caught and propagated as it should have been. Fixed in
2.0.3 already.</p>
</li>
<li><p>Make the <code class="docutils literal notranslate"><span class="pre">NUITKA_PYTHONPATH</span></code> usage more robust. Fixed in 2.0.3
already.</p></li>
<li><p>Fix, PySide2/6 argument name for slot connection and disconnect
should be <code class="docutils literal notranslate"><span class="pre">slot</span></code>, wasn’t working with keyword argument calls. Fixed
in 2.0.3 already.</p></li>
<li><p>Standalone: Added support for <code class="docutils literal notranslate"><span class="pre">paddle</span></code> and <code class="docutils literal notranslate"><span class="pre">paddleocr</span></code> packages.
Fixed in 2.0.4 already.</p></li>
<li><p>Standalone: Added support for <code class="docutils literal notranslate"><span class="pre">diatheke</span></code>. Fixed in 2.0.4 already.</p></li>
<li><p>Standalone: Added support for <code class="docutils literal notranslate"><span class="pre">zaber-motion</span></code> package. Fixed in
2.0.4 already.</p></li>
<li><p>Standalone: Added support for <code class="docutils literal notranslate"><span class="pre">plyer</span></code> package. Fixed in 2.0.4
already.</p></li>
<li><p>Fix, added handling of <code class="docutils literal notranslate"><span class="pre">OSError</span></code> for metadata read, otherwise
corrupt packages can have Nuitka crashing. Fixed in 2.0.4 already.</p></li>
<li><p>Fix, need to annotate potential exception exit when making a fixed
import from hard module attribute. Fixed in 2.0.4 already.</p></li>
<li><p>Fix, didn’t consider Nuitka project options with <code class="docutils literal notranslate"><span class="pre">--main</span></code> and
<code class="docutils literal notranslate"><span class="pre">--script-path</span></code>. This is of course the only way Nuitka-Action does
call it, so they didn’t work there at all. Fixed in 2.0.4 already.</p></li>
<li><p>Scons: Fix, need to close progress bar when about to error exit.
Otherwise error outputs will be garbled by incomplete progress bar.
Fixed in 2.0.4 already.</p></li>
<li><p>Fix, need to convert relative from imports to hard imports too, or
else packages needed to be followed are not included. Fixed in 2.0.5
already.</p></li>
<li><p>Standalone: Added <code class="docutils literal notranslate"><span class="pre">pygame_menu</span></code> data files. Fixed in 2.0.6 already.</p></li>
<li><p>Windows: Fix, wasn’t working when compiling on network mounted drive
letters. Fixed in 2.0.6 already.</p></li>
<li><p>Fix, the <code class="docutils literal notranslate"><span class="pre">.pyi</span></code> parser was crashing on some comments with a leading
<code class="docutils literal notranslate"><span class="pre">from</span></code> in the line, recognize these better. Fixed in 2.0.6 already.</p></li>
<li><p>Actions: Fix, some yaml configs could fail to load plugins. Fixed in
2.0.6 already.</p></li>
<li><p>Standalone: Added support for newer <code class="docutils literal notranslate"><span class="pre">torch</span></code> packages that otherwise
require source code.</p></li>
<li><p>Fix, inline copies of <code class="docutils literal notranslate"><span class="pre">tqdm</span></code> etc. left sub-modules behind, removing
only the top level <code class="docutils literal notranslate"><span class="pre">sys.modules</span></code> entry may not be enough.</p></li>
</ul>
<h2>New Features</h2>
<ul>
<li><p>Plugins: Added support for <code class="docutils literal notranslate"><span class="pre">constants</span></code> in Nuitka package
configurations. We can now using <code class="docutils literal notranslate"><span class="pre">when</span></code> clauses, define variable
values to be defined, e.g. to specify the DLL suffix, or the DLL
path, based on platform dependent properties.</p></li>
<li><p>Plugins: Make <code class="docutils literal notranslate"><span class="pre">relative_path</span></code>, <code class="docutils literal notranslate"><span class="pre">suffix</span></code>, <code class="docutils literal notranslate"><span class="pre">prefix</span></code> in DLL Nuitka
package configurations allowed to be an expression rather than just a
constant value.</p></li>
<li><p>Plugins: Make not only booleans related to the python version
available, but also strings <code class="docutils literal notranslate"><span class="pre">python_version_str</span></code> and
<code class="docutils literal notranslate"><span class="pre">python_version_full_str</span></code>, to use them when constructing e.g. DLL
paths in Nuitka package configuration.</p></li>
<li><p>Plugins: Added helper function <code class="docutils literal notranslate"><span class="pre">iterate_modules</span></code> for producing the
submodules of a given package, for using in expressions of Nuitka
package configuration.</p></li>
<li><p>macOS: Added support for Tcl/Tk detection on Homebrew Python.</p></li>
<li><p>Added <code class="docutils literal notranslate"><span class="pre">module</span></code> attribute to <code class="docutils literal notranslate"><span class="pre">__compiled__</span></code> values</p>
<p>So far it was impossible to distinguish non-standalone, i.e.
accelerated mode and module compilation by looking at the
<code class="docutils literal notranslate"><span class="pre">__compiled__</span></code> attribute, so we add an indicator for module mode
that closes this gap.</p>
</li>
<li><p>Plugins: Added <code class="docutils literal notranslate"><span class="pre">appdirs</span></code> and <code class="docutils literal notranslate"><span class="pre">importlib</span></code> for use in Nuitka
package config expressions.</p></li>
<li><p>Plugins: Added ability to specify modules to not follow when a module
is used. This <code class="docutils literal notranslate"><span class="pre">nofollow</span></code> configuration is for rare use cases only.</p></li>
<li><p>Plugins: Added values <code class="docutils literal notranslate"><span class="pre">extension_std_suffix</span></code> and
<code class="docutils literal notranslate"><span class="pre">extension_suffix</span></code> for use in expressions, to e.g. construct DLL
suffix patterns from it.</p></li>
<li><p>UI: Added more control over caching with per cache category
environment variables, as <a class="reference external" href="https://nuitka.net/doc/user-manual.html#control-where-caches-live">documented in the User Manual.</a>.</p></li>
<li><p>Plugins: Added support for reporting module detections</p>
<p>The <code class="docutils literal notranslate"><span class="pre">delvewheel</span></code> plugin now puts the version of that packaging tool
used by a particular module in the report rather than tracing it to
the user, that in the normal case won’t care. This is more for
debugging purposes of Nuitka.</p>
</li>
</ul>
<h2>Optimization</h2>
<ul>
<li><p>Scalability: Do not make loop analysis at all for very trusted value
traces, their point is to not change, and waiting for that to be
confirmed has no point.</p></li>
<li><p>Use very trusted value traces in functions not just as mere assign
traces or else expected optimization will not be done on them in many
cases. With this a lot more cases of hard values are optimized
leading also to generally more compact and correct results in terms
of imports, metadata, code avoided on the wrong OS, etc.</p></li>
<li><p>Scalability: When specializing assignments, make sure to have the
proper value trace immediately.</p>
<p>When changing to a hard value, the value trace was still an assign
trace and not very trusted for one for micro pass of the module.</p>
<p>This had the effect to need one more micro pass to get to benefiting
of the unescapable nature of those values, which meant more micro
passes than necessary and those being more complex due to escaped
traces, and therefore taking longer for affected modules.</p>
</li>
<li><p>Scalability: The code trying avoid merge traces of merge traces, and
to instead flatten merge traces was only handling part of these
correctly, and correcting it reduced optimization time for some
functions from infinite to instant. Less memory usage should also
come out of this, even where this was not affecting compile time as
much. Added in 2.0.1 already.</p></li>
<li><p>Scalability: Some codes that checked for variables were testing for
temporary variable and normal variable both one after another, making
some optimization steps and code generation slower than necessary due
to the extra calls.</p></li>
<li><p>Scalability: A variable assignment from variable that were later
recognized to become a raise was not recognized as such, and this
then wasn’t caught and propagated as it should, preventing more
optimization of the affected code. Make sure to convert more directly
when observing things to change, rather than doing it one pass later.</p></li>
<li><p>The fix proper reuse of tuples released to the freelist with matching
sizes causes less memory usage and faster performance for the 3.11
version. Added in 2.0.2 already.</p></li>
<li><p>Statically optimize <code class="docutils literal notranslate"><span class="pre">sys.exit</span></code> into exception raise of
<code class="docutils literal notranslate"><span class="pre">SystemExit</span></code>.</p>
<p>This should make a bunch of dead code obvious to Nuitka, it can now
tell this aborts execution of a branch, potentially eliminating
imports, etc.</p>
</li>
<li><p>macOS: Enable python static link library for Homebrew too. Added in
2.0.1 already. Added in 2.0.3 already.</p></li>
<li><p>Avoid compiling bloated module namespace of <code class="docutils literal notranslate"><span class="pre">altair</span></code> package. Added
in 2.0.3 already.</p></li>
<li><p>Anti-Bloat: Avoid including <code class="docutils literal notranslate"><span class="pre">kubernetes</span></code> for <code class="docutils literal notranslate"><span class="pre">tensorflow</span></code> unless
used otherwise. Added in 2.0.3 already.</p></li>
<li><p>Anti-Bloat: Avoid including setuptools for <code class="docutils literal notranslate"><span class="pre">tqdm</span></code>. Added in 2.0.3
already.</p></li>
<li><p>Anti-Bloat: Avoid <code class="docutils literal notranslate"><span class="pre">IPython</span></code> in <code class="docutils literal notranslate"><span class="pre">fire</span></code> package. Added in 2.0.3
already.</p></li>
<li><p>Anti-Bloat: Avoid including <code class="docutils literal notranslate"><span class="pre">Cython</span></code> for <code class="docutils literal notranslate"><span class="pre">pydantic</span></code> package.
Added in 2.0.3 already.</p></li>
<li><p>Anti-Bloat: Changes to avoid <code class="docutils literal notranslate"><span class="pre">triton</span></code> in newer <code class="docutils literal notranslate"><span class="pre">torch</span></code> as well.
Added in 2.0.5 already.</p></li>
<li><p>Anti-Bloat: Avoid <code class="docutils literal notranslate"><span class="pre">setuptools</span></code> via <code class="docutils literal notranslate"><span class="pre">setuptools_scm</span></code> in
<code class="docutils literal notranslate"><span class="pre">pyarrow</span></code>.</p></li>
<li><p>Anti-Bloat: Made more packages equivalent to using <code class="docutils literal notranslate"><span class="pre">setuptools</span></code>
which we want to avoid, all of <code class="docutils literal notranslate"><span class="pre">Cython</span></code>, <code class="docutils literal notranslate"><span class="pre">cython</span></code>, <code class="docutils literal notranslate"><span class="pre">pyximport</span></code>,
<code class="docutils literal notranslate"><span class="pre">paddle.utils.cpp_extension</span></code>, <code class="docutils literal notranslate"><span class="pre">torch.utils.cpp_extension</span></code> were
added for better reports of the actual causes.</p></li>
</ul>
<h2>Organisational</h2>
<ul>
<li><p>Moved the changelog of Nuitka to the website, just point to there
from Nuitka repo.</p></li>
<li><p>UI: Proper error message from Nuitka when scons build fails with a
detail mnemonic page. Read more on <a class="reference internal" href="https://nuitka.net/blog/info/scons-backend-failure.html"><span class="doc">the info page</span></a> for detailed information.</p></li>
<li><p>Windows: Reject all MinGW64 that are not are not the <code class="docutils literal notranslate"><span class="pre">winlibs</span></code> that
Nuitka itself downloaded. As these packages break very easily, we
need to control if it’s a working set of <code class="docutils literal notranslate"><span class="pre">ccache</span></code>, <code class="docutils literal notranslate"><span class="pre">make</span></code>,
<code class="docutils literal notranslate"><span class="pre">binutils</span></code> and gcc with all the necessary workarounds and features
like <code class="docutils literal notranslate"><span class="pre">LTO</span></code> working on Windows properly.</p></li>
<li><p>Quality: Added auto-format of PNG and JPEG images. This aims at
making it simpler to add images to our repositories, esp. Nuitka
Website. This now makes <code class="docutils literal notranslate"><span class="pre">optipng</span></code> and <code class="docutils literal notranslate"><span class="pre">jpegoptim</span></code> calls as
necessary. Previously this was manual steps for the website to be
applied.</p></li>
<li><p>User Manual: Be more clear about compiler version needs on Windows
for Python 3.11.</p></li>
<li><p>User Manual: Added examples for error message with low C compiler
memory, such that maybe they can be found via search by users.</p></li>
<li><p>User Manual: Removed sections that are unnecessary or better
maintained as separate pages on the website.</p></li>
<li><p>Quality: Avoid empty <code class="docutils literal notranslate"><span class="pre">no-auto-follow</span></code> values, for silently ignoring
it there is a dedicated string <code class="docutils literal notranslate"><span class="pre">ignore</span></code> that must be used.</p></li>
<li><p>Quality: Enforce normalized paths for <code class="docutils literal notranslate"><span class="pre">dest_path</span></code> and
<code class="docutils literal notranslate"><span class="pre">relative_path</span></code>. Users were uncertain if a leading dot made sense,
but we now disallow it for clarity.</p></li>
<li><p>Quality: Check more keys with expressions for syntax errors, to catch
these mistakes in configuration sooner.</p></li>
<li><p>Quality: Scanning through all files with the auto-format tool should
now be faster, and CPython test suite directories (test submodules)
if present are ignored.</p></li>
<li><p>Release: Remove month from manpage generation, that’s only noise in
diffs.</p></li>
<li><p>Removed digital art folders, these were only making checkouts larger
for no good reason. We will have better ones on the website in the
future.</p></li>
<li><p>Scons: Allow C warnings when compiling for running in debugger
automatically.</p></li>
<li><p>UI: The macOS app bundle option is not experimental at all. This has
been untrue for years now, remove that cautioning.</p></li>
<li><p>macOS: Discontinue support for PyQt6.</p>
<p>With newer PyQt6 we would have to package frameworks properly, and we
don’t have that yet and it will be a lot of developer time to get it.</p>
<p>Instead point people to PySide6 which is the better choice and is
perfectly supported by Qt company and Nuitka.</p>
</li>
<li><p>Removed version numbering, month of creation, etc. from the man pages
generated.</p></li>
<li><p>Moved <code class="docutils literal notranslate"><span class="pre">Credits.rst</span></code> file to be on the website and maintain it there
rather than syncing of from the Nuitka repository.</p></li>
<li><p>Bumped copyright year and split the license text such that it is now
at the bottom of the files rather than eating up the first page, this
is aimed at making the code more readable.</p></li>
</ul>
<h2>Cleanups</h2>
<ul>
<li><p>With <code class="docutils literal notranslate"><span class="pre">sys.exit</span></code> being optimized, we were able to make our trick to
avoid following <code class="docutils literal notranslate"><span class="pre">nuitka</span></code> because of accidentally finding the
<code class="docutils literal notranslate"><span class="pre">setup</span></code> as an import more simple.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># Don't allow importing this, and make recognizable that</span>
<span class="c1"># the above imports are not to follow. Sometimes code imports</span>
<span class="c1"># setup and then Nuitka ends up including itself.</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">!=</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="s2">"Cannot import 'setup' module of Nuitka"</span><span class="p">)</span>
</pre></div>
</div>
</li>
<li><p>Scons: Don’t scan for <code class="docutils literal notranslate"><span class="pre">ccache</span></code> on Windows, the <code class="docutils literal notranslate"><span class="pre">winlibs</span></code> package
contains it nowadays, and since it’s now required to be used, there
is no point for this code anymore.</p></li>
<li><p>Minor cleanups coming from trying out <code class="docutils literal notranslate"><span class="pre">ruff</span></code> as a linter on Nuitka,
it found a few uses of not using <code class="docutils literal notranslate"><span class="pre">not</span> <span class="pre">in</span></code>, but that was it.</p></li>
</ul>
<h2>Tests</h2>
<ul class="simple">
<li><p>Removed test with chinese filenames, we need to avoid chinese names
in the repo. These have been seen as preventing installation on some
systems that are not capable of handling them in the git, zip, pip
tooling, so lets avoid them entirely now that Nuitka handles these
just fine.</p></li>
<li><p>Tests: More macOS standalone tests that need to be bundles were
getting the project configuration to do it.</p></li>
</ul>
<h2>Summary</h2>
<p>This release added much needed tools for our Nuitka Package
configuration, but also cleans up scalability and optimization that was
supposed to work, but did not yet, or not anymore.</p>
<p>The usability improved again, as it does always, but the big
improvements for scalability that will implement existing algorithms
more efficient, are yet to come, this release was mainly driven by the
need to get <code class="docutils literal notranslate"><span class="pre">torch</span></code> to work in its latest version out of the box with
stable Nuitka, but this couldn’t be done as a hotfix</p>2024-03-23T14:11:00+00:00Python Morsels: Unnecessary else statements
https://www.pythonmorsels.com/unnecessary-else-statements/
<p>When your function ends in an <code>else</code> block with a <code>return</code> statement in it, should you remove that <code>else</code>?</p>
<div>
<img width="480" height="270" src="https://i.vimeocdn.com/filter/overlay?src0=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F1823075550-2a9f40e068ac5d480a98e820bc1987933fd75b20daa27db32f2915e417517d8c-d_1920x1080&src1=http%3A%2F%2Ff.vimeocdn.com%2Fp%2Fimages%2Fcrawler_play.png" />
<p>
<strong>Table of contents</strong>
<ol>
<li><a href="https://www.pythonmorsels.com/unnecessary-else-statements/#a-function-where-both-if-and-else-return" target="_blank">A function where both <code>if</code> and <code>else</code> return</a></li>
<li><a href="https://www.pythonmorsels.com/unnecessary-else-statements/#is-that-else-statement-unnecessary" target="_blank">Is that <code>else</code> statement unnecessary?</a></li>
<li><a href="https://www.pythonmorsels.com/unnecessary-else-statements/#sometimes-else-improves-readability" target="_blank">Sometimes <code>else</code> improves readability</a></li>
<li><a href="https://www.pythonmorsels.com/unnecessary-else-statements/#when-should-you-remove-an-else-statement" target="_blank">When should you remove an <code>else</code> statement?</a></li>
<li><a href="https://www.pythonmorsels.com/unnecessary-else-statements/#considering-readability-with-if-else-statements" target="_blank">Considering readability with <code>if</code>-<code>else</code> statements</a></li>
</ol>
</p>
</div>
<div>
<h2>A function where both <code>if</code> and <code>else</code> return</h2>
<p>This <code>earliest_date</code> function uses the <a href="https://pypi.org/project/python-dateutil/" target="_blank">python-dateutil</a> third-party library to parse two strings as dates:</p>
<div class="codehilite"><pre><span></span><code><span class="kn">from</span> <span class="nn">dateutil.parser</span> <span class="kn">import</span> <span class="n">parse</span>
<span class="k">def</span> <span class="nf">earliest_date</span><span class="p">(</span><span class="n">date1</span><span class="p">,</span> <span class="n">date2</span><span class="p">):</span>
<span class="sd">"""Return the string representing the earliest date."""</span>
<span class="k">if</span> <span class="n">parse</span><span class="p">(</span><span class="n">date1</span><span class="p">,</span> <span class="n">fuzzy</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="o"><</span> <span class="n">parse</span><span class="p">(</span><span class="n">date2</span><span class="p">,</span> <span class="n">fuzzy</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="k">return</span> <span class="n">date1</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">date2</span>
</code></pre></div>
<p>This function returns the string which represents the earliest given date:</p>
<div class="codehilite"><pre><span></span><code><span class="gp">>>> </span><span class="n">earliest_date</span><span class="p">(</span><span class="s2">"May 3 2024"</span><span class="p">,</span> <span class="s2">"June 5 2025"</span><span class="p">)</span>
<span class="go">'May 3 2024'</span>
<span class="gp">>>> </span><span class="n">earliest_date</span><span class="p">(</span><span class="s2">"Feb 3 2026"</span><span class="p">,</span> <span class="s2">"June 5 2025"</span><span class="p">)</span>
<span class="go">'June 5 2025'</span>
</code></pre></div>
<p>Note that this function uses an <a href="https://www.pythonmorsels.com/if-statements/" target="_blank">if statement</a> that returns, and an <code>else</code> that also returns.</p>
<h2>Is that <code>else</code> statement unnecessary?</h2>
<p>We don't necessarily <em>need</em> that …</p>
</div>
<h3><a href="https://www.pythonmorsels.com/unnecessary-else-statements/" target="_blank">Read the full article: https://www.pythonmorsels.com/unnecessary-else-statements/</a></h3>2024-03-22T22:00:00+00:00Django Weblog: Welcome our new Fellow - Sarah Boyce
https://www.djangoproject.com/weblog/2024/mar/22/welcome-our-new-fellow-sarah-boyce/
<p>The DSF Board and Fellows Committee are pleased to introduce Sarah Boyce as our new Django Fellow. Sarah will be joining Natalia Bidart who is continuing her excellent tenure as a Fellow.</p>
<p>Sarah is a senior developer and developer advocate with 5 years of experience developing with Django under her belt. She graduated with a first class honours degree in Mathematics from the University of Bath, and transitioned in software development in her first job out of school.</p>
<p>Sarah first worked as a client project focused developer, where she gained experience directly dealing with requests from clients as well as managing our own internal ticketing system for feature/bug reports. A stint as a backend developer using Django and DRF provided a grounding in working on long term challenges on a single project. Most recently Sarah has been a developer advocate focused on creating content on and about Django and Django development.</p>
<p>For the past several years, Sarah has been a very active member of the Django community. She has a history of producing well researched and written patches for Django, as well as on a number of highly used third party packages. Sarah is a member of the Django Review and Triage team, helping others to get their patches over the line and into Django. She also finds time to participate in and create content for Django meetups, conferences, and the Django News newsletter.</p>
<p>Sarah is also a Co-Founder and Co-Organiser of Djangonaut Space, the mentorship program developing future contributors to Django and other Django related packages. Djangonaut Space was awarded the <a href="https://www.djangoproject.com/weblog/2023/dec/08/2023-malcolm-tredinnick-memorial-prize-awarded-to/">2023 Malcolm Tredinnick Memorial Prize</a>.</p>
<p>Please join me in welcoming and wishing Sarah well as the new Fellow.</p>
<hr />
<p>Thank you to all of the applicants to the Fellowship. We hope that we will be able to expand the Fellowship program in the future, and knowing that there are more excellent candidates gives us confidence in working towards that goal.</p>
<hr />
<p>Finally our deepest thanks and gratitude goes to Mariusz Felisiak. Mariusz is stepping down from the Fellowship after 5 years of dedicated service in order to focus on other areas of the Django and wider world. We wish you well Mariusz.</p>2024-03-22T16:54:02+00:00Daniel Roy Greenfeld: Keynote at PyCon Lithuania 2024
https://daniel.feldroy.com/posts/2024-pycon-lithuania
<p>From April 2nd to April 6th I'll be at <a href="https://pycon.lt/2024">PyCon Lithuania 2024</a> in Vilnius to present a keynote about 25 years of glorious coding mistakes (mostly in Python). Audrey and Uma will be accompanying me, making us the first members of the Lithuanian side of my family to return there in over 100 years!</p>
<p>At the conference I'll be joined by my old friend Tom Christie, author of <a href="https://daniel.roygreenfeld.com/feeds/python.atom.xml">HTTPX</a>, <a href="https://www.starlette.io">Starlette</a>, and <a href="https://www.django-rest-framework.org/">Django REST Framework</a>. I hope to meet many new friends, specifically everyone there. At the sprints I'll be joined by my awesome wife, Audrey, author of <a href="https://github.com/cookiecutter/cookiecutter">Cookiecutter</a>.</p>
<p>Come and join us!</p>2024-03-22T13:00:00+00:00Real Python: The Real Python Podcast – Episode #197: Using Python in Bioinformatics and the Laboratory
https://realpython.com/podcasts/rpp/197/
<p>How is Python being used to automate processes in the laboratory? How can it speed up scientific work with DNA sequencing? This week on the show, Chemical Engineering PhD Student Parsa Ghadermazi is here to discuss Python in bioinformatics.</p>
<hr />
<p><em>[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>2024-03-22T12:00:00+00:00PyCharm: PyCharm 2023.3.5 Is Out!
https://blog.jetbrains.com/pycharm/2024/03/2023-3-5/
<p>PyCharm 2023.3.5 is an important bug-fix update.</p>
<p>You can update to this version from inside the IDE, using the <a href="https://www.jetbrains.com/toolbox-app/" target="_blank" rel="noreferrer noopener">Toolbox App</a>, or via snaps, if you’re using Ubuntu. You can also download it directly from our <a href="https://www.jetbrains.com/pycharm/download/" target="_blank" rel="noreferrer noopener">website</a>.</p>
<p>Here are some of the notable fixes in v2023.3.5: </p>
<ul>
<li>The “Problems” tool window no longer displays outdated project errors that have already been resolved. [<a href="https://youtrack.jetbrains.com/issue/PY-71058" target="_blank" rel="noopener">PY-71058</a>]</li>
<li>PyCharm now supports Docker 2.25, eliminating errors that occurred when attempting to create a Docker-compose interpreter with Docker 2.25. [<a href="https://youtrack.jetbrains.com/issue/PY-71131/Error-on-adding-a-docker-compose-interpreter-after-update-to-Docker-25-on-Linux-Cannot-deserialize-value-of-type" target="_blank" rel="noopener">PY-71131</a>]</li>
<li>We’ve introduced a workaround to reduce the likelihood of IDE crashes following an update to macOS Sonoma 14.4. [<a href="https://youtrack.jetbrains.com/issue/JBR-6802" target="_blank" rel="noopener">JBR-6802</a>]</li>
<li>We’ve fixed the issue causing erratic screen scaling on Linux. [<a href="https://youtrack.jetbrains.com/issue/IDEA-341318/" target="_blank" rel="noreferrer noopener">IDEA-341318</a>]</li>
</ul>
<p>For the full list of issues addressed in PyCharm 2023.3.5, please see the <a href="https://youtrack.jetbrains.com/articles/PY-A-233538189/PyCharm-2023.3.5-233.15026.15-build-Release-Notes" target="_blank" rel="noreferrer noopener">release notes</a>.</p>2024-03-22T06:52:40+00:00Talk Python to Me: #454: Data Pipelines with Dagster
https://talkpython.fm/episodes/show/454/data-pipelines-with-dagster
Do you have data that you pull from external sources or is generated and appears at your digital doorstep? I bet that data needs processed, filtered, transformed, distributed, and much more. One of the biggest tools to create these data pipelines with Python is Dagster. And we are fortunate to have Pedram Navid on the show this episode. Pedram is the Head of Data Engineering and DevRel at Dagster Labs. And we're talking data pipelines this week at Talk Python.<br/>
<br/>
<strong>Episode sponsors</strong><br/>
<br/>
<a href='https://talkpython.fm/training'>Talk Python Courses</a><br>
<a href='https://talkpython.fm/posit'>Posit</a><br/>
<br/>
<strong>Links from the show</strong><br/>
<br/>
<div><b>Rock Solid Python with Types Course</b>: <a href="https://training.talkpython.fm/courses/python-type-hint-course-with-hands-on-examples?ref=podcast" target="_blank" rel="noopener">training.talkpython.fm</a><br/>
<br/>
<b>Pedram on Twitter</b>: <a href="https://twitter.com/pdrmnvd" target="_blank" rel="noopener">twitter.com</a><br/>
<b>Pedram on LinkedIn</b>: <a href="https://linkedin.com/in/pedramnavid" target="_blank" rel="noopener">linkedin.com</a><br/>
<b>Ship data pipelines with extraordinary velocity</b>: <a href="https://dagster.io" target="_blank" rel="noopener">dagster.io</a><br/>
<b>dagster-open-platform</b>: <a href="https://github.com/dagster-io/dagster-open-platform" target="_blank" rel="noopener">github.com</a><br/>
<b>The Dagster Master Plan</b>: <a href="https://dagster.io/blog/dagster-master-plan" target="_blank" rel="noopener">dagster.io</a><br/>
<b>data load tool (dlt)</b>: <a href="https://dlthub.com" target="_blank" rel="noopener">dlthub.com</a><br/>
<b>DataFrames for the new era</b>: <a href="https://pola.rs" target="_blank" rel="noopener">pola.rs</a><br/>
<b>Apache Arrow</b>: <a href="https://arrow.apache.org" target="_blank" rel="noopener">arrow.apache.org</a><br/>
<b>DuckDB is a fast in-process analytical database</b>: <a href="https://duckdb.org" target="_blank" rel="noopener">duckdb.org</a><br/>
<b>Ship trusted data products faster</b>: <a href="https://www.getdbt.com" target="_blank" rel="noopener">www.getdbt.com</a><br/>
<b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=vRVhDfQPHBM" target="_blank" rel="noopener">youtube.com</a><br/>
<b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/454/data-pipelines-with-dagster" target="_blank" rel="noopener">talkpython.fm</a><br/>
<br/>
<b>--- Stay in touch with us ---</b><br/>
<b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/>
<b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/>
<b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>2024-03-21T08:00:00+00:00Programiz: Python Program to Capitalize the First Character of a String
https://www.programiz.com/python-programming/examples/capitalise-first-character
In this example, you will learn to capitalize the first character of a string.2024-03-21T05:19:28+00:00Python⇒Speed: The wrong way to speed up your code with Numba
https://pythonspeed.com/articles/slow-numba/
<p>If your NumPy-based code is too slow, you can sometimes use Numba to
speed it up. Numba is a compiled language that uses the same syntax as
Python, and it compiles at runtime, so it’s very easy to write. And
because it re-implements a large part of the NumPy APIs, it can also
easily be used with existing NumPy-based code.</p>
<p>However, Numba’s NumPy support can be a trap: it can lead you to missing
huge optimization opportunities by sticking to NumPy-style code. So in
this article we’ll show an example of:</p>
<ul>
<li>The wrong way to use Numba, writing NumPy-style full array transforms.</li>
<li>The right way to use Numba, namely <code class="language-plaintext highlighter-rouge">for</code> loops.</li>
</ul>
<a href="https://pythonspeed.com/articles/slow-numba/">Read more...</a>2024-03-21T00:00:00+00:00Matt Layman: Post-launch Punchlist - Building SaaS with Python and Django #186
https://www.mattlayman.com/blog/2024/post-launch-punchlist-building-saas-with-python-and-django-186/
In this episode, we had a bunch of issues to resolve post-launch. I set the code that causes trials to expire, made updates to who receives prompt emails, and added some polish to the sign up process and interface to make it clear what will happen in the flow. After those modifications, we worked through a set of smaller changes like setting up Dependabot and adding a missing database index.2024-03-21T00:00:00+00:00EuroPython: EuroPython 2024: Community Voting is now live! Go Vote!
https://blog.europython.eu/europython-2024-community-voting-is-now-live-go-vote/
<p>Hey hey,</p><p>With 110 days remaining until the big day, the EuroPython programme team is working full steam ahead to put together a power-packed schedule. And what *YOU* want to see at the conference is our guiding light in the process. </p><p>With that, we are excited to announce the EuroPython 2024 Community Voting: <a href="https://ep2024.europython.eu/voting">https://ep2024.europython.eu/voting</a> 🎉<br /><br />All past EuroPython attendees between 2015-2024 & prospective speakers from this year are eligible to vote. <br /><br />You can help us spread the word by forwarding this email to your fellow EuroPython friends.<br /><br />The more votes we have, the better informed decisions the programme team can make!</p><p>Head over to <a href="https://ep2024.europython.eu/voting">https://ep2024.europython.eu/voting</a> to make your voice heard!<br /><br />Thank you for your continued support,<br /><em>EuroPython 2024 Organisers</em><br /><br /></p>2024-03-20T22:00:43+00:00Python Software Foundation: Announcing a PyPI Support Specialist
https://pyfound.blogspot.com/2024/03/announcing-pypi-support-specialist.html
<p>We launched the Python Package Index (PyPI) in 2003 and for most of its history a robust and dedicated volunteer community kept it running. Eventually, we put a bit of PSF staff time into the maintenance of the Index, and last year with support from AWS we <a href="https://pyfound.blogspot.com/2023/08/announcing-our-new-pypi-safety-security.html">hired Mike Fiedler</a> to work full-time on PyPI’s urgent security needs. <br /></p><p>PyPI has grown enormously in the last 20+ years, and in recent years it has reached a truly massive scale with growth only continuing upward. In 2022 alone, PyPI saw a 57% growth and as of this writing, there are over a half a million packages <a href="https://pypi.org/">on PyPI</a>. The impact PyPI has these days is pretty breathtaking. Running a free public service of that size has come with challenges, too. As PyPI has grown, the work of communicating with users and solving account issues here has grown in tandem and out-stripped our current volunteer plus one tenth of a staff person capacity. We also know that some community members have noticed and expressed frustration with the time-frame that goes with tasks that don't have sufficient staffing. <br /><br />Much of this work is sensitive and complex such that it needs to be performed by a PSF staff person. It involves personal information and verification processes to make sure we’re giving access and names to the correct entities. Work like this needs to be done by a person who is here day after day to carry out multi-step verification procedures and is accountable to the PSF. </p><p>We are very happy to share the news that <a href="https://pythonsoftwarefoundation.applytojob.com/apply/nyYHuOha9h/PyPI-Support-Specialist">we are hiring a person</a> to help us manage the increased capacity and allow us to keep pace with PyPI’s seemingly unstoppable growth. This is an associate role that is 100% remote. Please take a look at <a href="https://pythonsoftwarefoundation.applytojob.com/apply/nyYHuOha9h/PyPI-Support-Specialist">this posting for a PyPI Support Specialist</a> and share it with your networks. </p>2024-03-20T15:08:22+00:00Real Python: Build a Python Turtle Game: Space Invaders Clone
https://realpython.com/build-python-turtle-game-space-invaders-clone/
<div><p>In this tutorial, you’ll use Python’s <code>turtle</code> module to build a Space Invaders clone. The game <a href="https://en.wikipedia.org/wiki/Space_Invaders">Space Invaders</a> doesn’t need any introduction. The original game was released in 1978 and is one of the most recognized video games of all time. It undeniably defined its own video game genre. In this tutorial, you’ll create a basic clone of this game.</p>
<p>The <code>turtle</code> module you’ll use to build the game is part of Python’s standard library, and it enables you to draw and move sprites on the screen. The <code>turtle</code> module is not a game-development package, but it gives instructions about creating a <em><code>turtle</code> game</em>, which will help you understand how video games are built.</p>
<p><strong>In this tutorial, you’ll learn how to:</strong></p>
<ul>
<li><strong>Design and build</strong> a classic video game</li>
<li>Use the <code>turtle</code> module to <strong>create animated sprites</strong></li>
<li>Add <strong>user interaction</strong> in a graphics-based program</li>
<li><strong>Create a game loop</strong> to control each frame of the game</li>
<li><strong>Use functions</strong> to represent key actions in the game</li>
</ul>
<p>This tutorial is ideal for anyone who is familiar with the core Python topics and wants to use them to build a classic video game from scratch. You don’t need to be familiar with the <code>turtle</code> module to work through this tutorial. You can download the code for each step by clicking on the link below:</p>
<div class="alert alert-warning">
<p><strong>Get Your Code:</strong> <a href="https://realpython.com/bonus/build-python-turtle-game-space-invaders-clone-code/" class="alert-link">Click here to download the free sample code</a> that shows you how to build a Python turtle game.</p>
</div>
<p>In the next section, you can have a look at the version of the game you’ll build as you follow the steps outlined in this tutorial.</p>
<h2 id="demo-a-python-turtle-space-invaders-game">Demo: A Python Turtle Space Invaders Game<a class="headerlink" href="https://realpython.com/atom.xml#demo-a-python-turtle-space-invaders-game" title="Permanent link"></a></h2>
<p>You’ll build a simplified version of the classic Space Invaders game and control the laser cannon with the keys on your keyboard. You’ll shoot lasers from the cannon by pressing the spacebar, and aliens will appear at regular intervals at the top of the screen and move downwards. Your task is to shoot the aliens before they reach the bottom of the screen. The game ends when one alien reaches the bottom.</p>
<p>This is what your <code>turtle</code> game will look like when you complete this tutorial:</p>
<div class="embed-responsive embed-responsive-16by9 rounded mb-3 ">
</div>
<p>Here you can see the main game play for this game, as the laser cannon moves back and forth and shoots the falling aliens. The game also displays the elapsed time and the number of aliens shot down on the screen.</p>
<h2 id="project-overview">Project Overview<a class="headerlink" href="https://realpython.com/atom.xml#project-overview" title="Permanent link"></a></h2>
<p>In this project, you’ll start by creating the screen that will contain the game. In each step, you’ll create game components such as the laser cannon, lasers, and aliens, and you’ll add the features required to make a functioning game.</p>
<p>To create this <code>turtle</code> game, you’ll work through the following steps:</p>
<ol>
<li>Create the game <strong>screen</strong> and the <strong>laser cannon</strong></li>
<li><strong>Move the cannon</strong> left and right using keys</li>
<li><strong>Shoot lasers</strong> with the spacebar</li>
<li><strong>Create aliens</strong> and move them towards the bottom of the screen</li>
<li>Determine when <strong>a laser hits an alien</strong></li>
<li><strong>End the game</strong> when an alien reaches the bottom</li>
<li>Add a <strong>timer</strong> and a <strong>score</strong></li>
<li><strong>Improve the cannon’s movement</strong> to make the game smoother</li>
<li>Set the game’s <strong>frame rate</strong></li>
</ol>
<p>You’ll start with a blank screen, and then see the game come to life one feature at a time as you work through each step in this tutorial.</p>
<h2 id="prerequisites">Prerequisites<a class="headerlink" href="https://realpython.com/atom.xml#prerequisites" title="Permanent link"></a></h2>
<p>To complete this tutorial, you should be comfortable with the following concepts:</p>
<ul>
<li>Repeating code using <a href="https://realpython.com/python-for-loop/"><code>for</code> loops</a> and <a href="https://realpython.com/python-for-loop/"><code>while</code> loops</a></li>
<li>Using <a href="https://realpython.com/python-conditional-statements/"><code>if</code> statements</a> to control what happens in different conditions</li>
<li><a href="https://realpython.com/defining-your-own-python-function/">Defining functions</a> to encapsulate code</li>
<li>Using <a href="https://realpython.com/python-lists-tuples/">lists</a> to store multiple items</li>
</ul>
<p>You don’t need to be familiar with Python’s <code>turtle</code> to start this tutorial. However, you can <a href="https://realpython.com/beginners-guide-python-turtle/">read an overview of the <code>turtle</code> module</a> to find out more about the basics.</p>
<p>If you don’t have all of the prerequisite knowledge before you start, that’s okay! In fact, you might learn more by going ahead and getting started! You can always stop and review the resources linked here if you get stuck.</p>
<h2 id="step-1-set-up-the-turtle-game-with-a-screen-and-a-laser-cannon">Step 1: Set Up the Turtle Game With a Screen and a Laser Cannon<a class="headerlink" href="https://realpython.com/atom.xml#step-1-set-up-the-turtle-game-with-a-screen-and-a-laser-cannon" title="Permanent link"></a></h2>
<p>You can’t have a game without a screen where all the action happens. So, the first step is to create a blank screen. Then, you can add sprites to represent the items in the game. In this project, you can run your code at any point to see the game in its current state.</p>
<p>You can download the code as it’ll look at the end of this step from the folder named <code>source_code_step_1/</code> in the link below:</p>
</div><h2><a href="https://realpython.com/build-python-turtle-game-space-invaders-clone/?utm_source=realpython&utm_medium=rss">Read the full article at https://realpython.com/build-python-turtle-game-space-invaders-clone/ »</a></h2>
<hr />
<p><em>[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. <a href="https://realpython.com/python-tricks/?utm_source=realpython&utm_medium=rss&utm_campaign=footer">>> Click here to learn more and see examples</a> ]</em></p>2024-03-20T14:00:00+00:00