Planet Python

Last update: October 05, 2022 10:41 AM UTC

October 04, 2022

Guido van Rossum

Reasoning about asyncio.Semaphore

In Silicon Valley is a very exclusive fast-food restaurant, which is always open. There is one table, where one guest at a time is served an absolutely fabulous hamburger. When you arrive, you wait in line until the table is available. Then the host takes you to the table and, this being America, you are asked a seemingly endless series of questions about how you would like your hamburger to be cooked and served.

But today we're not talking about culinary delights. We're talking about the queuing system used by the restaurant. If you are lucky to arrive at the restaurant when the table is available and there are no other guests waiting, you are seated right away. Otherwise, the host gives you a buzzer (from an infinite stack of buzzers!) and you are free to roam the neighborhood until your buzzer goes off. It is the host's job to ensure that guests are seated in order of arrival. When it is your turn, the host will cause your buzzer go off and you make your way back to the restaurant, where you will be seated.

If you change your mind, you can return the buzzer to the host, who will take it back without lifting an eyebrow. If your buzzer has already gone off, the host will buzz the next guest, if any. Guests are always polite and don't abscond with their buzzers. The host is always fair and doesn't seat another guest ahead of you even if you take your time making it back.

The above description fits that of a Lock. A guest arriving corresponds to the acquire() call; leaving is a release() call. Changing your mind is like getting cancelled while waiting in acquire(). You can change your mind before or after your buzzer goes off, i.e., you can be cancelled before or after the lock has awakened your call (but before you return from acquire()).

One day the restaurant expands, hiring extra sous-chefs and opening several new tables. There is still only one host, whose job is not really changed. However, since multiple guests can be seated concurrently, a Semaphore must now be used instead of a simple Lock.

It turns out that implementing synchronization primitives is hard. This is somewhat surprising in the case of asyncio, since only one task can be executing at a time, and task switches only happen at await. But in the past year its fairness, correctness, semantics and performance have all been challenged. In fact, the last three complaints happened in the last month, and, being the last asyncio expert standing, I had to learn in a hurry what's the best way to think about semaphores.

The restaurant metaphor was very useful. For example, there is a difference between the number of open tables and the number of guests who may be seated immediately, and it equals the number of guests whose buzzer has gone off but who haven't come back to the host yet.

There was one particular challenge to fairness, where a task that released a semaphore and then immediately tried to acquire it again could starve other tasks. This is like a guest walking out, turning around, and getting seated again ahead of other waiting guests.

And there was a bug where a cancelled acquire() call could leave the lock in a bad state. This is like the host getting confused when a guest with a buzzing buzzer returns it but declines to be seated.

The restaurant metaphor didn't help with everything: cancellation behavior in asyncio is just complex. In Python 3.11 we have started putting extra strain on cancellation, because of two new asynchronous context managers we added:

Class TaskGroup for managing a group of related tasks. When one task fails, the others are cancelled, and the context manager waits for all tasks to exit.
timeout() function for managing timeouts. When the timeout goes off, the current task is cancelled.

Here is the main complication of cancellation handling:

When waiting for a Future, that Future may be cancelled, and then the await operation fails, raising CancelledError.
But when awaiting a Future raises CancelledError you cannot assume that the Future was cancelled! It is also possible that the Future was already marked as having a result (so it can no longer be cancelled), and your task has been marked as runnable, but another (also runnable) task runs first and cancels your task. I am grateful to Cyker Way for pointing out this corner case.

It helps to think of Futures as being in one of four states:

Waiting
Done, holding a result
Done, holding an exception
Done, but cancelled

From the waiting state a Future can transition to one of the other states, and then it cannot change state again. (Insert cute picture of state diagram here. :-)

The semaphore manages a FIFO queue of waiters. It does not use the exception state, but it does use the other three states:

Waiting: a guest with a buzzer that hasn't gone off yet
Holding a result: a guest who has been buzzed
Cancelled: a guest who returns their buzzer before it goes off

Fairness is supposed to be ensured by always appending a new Future to the queue to the end when acquire() finds the semaphore locked, and by always marking the leftmost (i.e., oldest) Future in the queue as holding a result when release() is called while queue isn't empty. The fairness bug was due acquire() taking a shortcut when the Semaphore's level (the number of open tables) is nonzero. It should not do this when there are still Futures in the queue. In other words we were sometimes seating a newly arrived guest when there was an open table even though there was already a guest waiting.

Guess what caused the cancellation bug? The scenario where a Future is holding a result (guest with buzzer buzzing) but the task awaiting that Future gets cancelled (guest declining to be seated).

I struggled to visualize the state of the Semaphore for myself, with its level and FIFO queue of waiting Futures. I also struggled with the definition of locked(). If the level variable had been public I would have struggled with its semantics too. In the end I came up with the following definitions:

W, the list of waiting futures, or [f for f in queue if not f.done()]
R, the list of futures holding a result, or [f for f in queue if f.done() and not f.cancelled()]
C, the list of cancelled futures, or [f for f in queue if f.cancelled()]

and some invariants:

set(W + R + C) == set(queue) — all futures are either waiting, have a result, or are cancelled.
level >= len(R) — we must have at least as many open tables as there are guests holding buzzing buzzers.
define locked() as (len(W) > 0 or len(R) > 0 or level == 0) — we cannot immediately seat anyone unless (a) there are no guests waiting for their buzzer to go off, (b) there are no guests holding a buzzing buzzer, and (c) there is at least one open table.

I leave you with a final link, to the current code.

October 04, 2022 11:39 PM UTC

Python Engineering at Microsoft

Contribute to Open Source with the Pythonistas at Microsoft – Hacktoberfest 2022

The Python Cloud Developer Advocate Team and friends will be getting together October 10th at 2PM Pacific on Microsoft Developer Channel Twitch to talk Python and Hactoberfest! Hang out with Sarah Kaiser, Pamela Fox, Dawn Wages, Jay Miller and Anthony Shaw while we share some of our favorite projects to contribute to and where we have set our sights this year.

Tips for Open Source contribution

1. Find a project by searching open repositories, chatting with Open Source contributors or checking out third party sites.

If you’re interested in checking out Python projects on GitHub that are participating in Hacktoberfest you can search them by seeing which projects are tagged hacktoberfest. Using GitHub’s advanced search box, you’re able to search for projects that are interesting to you. To learn more about finding a project to work on, check out Hacktoberfest and Microsoft Learn guide.

2. When looking for a project, look for repositories with:

Issues labels titled Good First Issue and Hacktoberfest.
A README with a “Getting Started” and “How To Contribute” sections. Sometimes contribution documents are located at https://github.com/[project_org]/project/contribute. NOTE: Maintainers of these repositories may call them different things and may be inside the repository or described in external documentation that is linked. It serves as a place with instructions on how to get started and setting up the project. It can also go into detail about how bugs or features are submitted, how to structure your pull request or Python coding guidelines.
A Code of Conduct: This document helps protect participants by establishing expectations for behavior in the contributing community. The enforcement of this document makes sure that the community supporting the project feels safe and respected.
Dependencies that sound familiar to you: If you are looking for a good place to start contributing, it may be helpful to start searching for repositories that rely on packages with which you have some familiarity.

3. Find a helpful list to guide you to the right project.

If you’re unfamiliar with projects that could be a good fit, try out some of these sites:

Good First Issues dot com
Good First Issue dot dev
Awesome For Beginners – MunGell GitHub Repo
Good First Issue – GitHub Topic
First Contributions
Pycon 2022, 2021 or 2020 Conference Sprints

4. Figuring out how to get un-stuck can be tough. Don’t be afraid to ask for help respectfully.

Maintainers of popular projects may receive calls for assistance frequently, however, if a project is signed up for Hacktoberfest, that’s a sign that the maintainers are interested in helping out new contributors, and they often have a lot of experience with guiding contributors in the right direction. Sometimes they will direct you to a Slack or Discord channel where people are having conversations regularly. Commenting on issues or pull requests are the most common ways to start to engage with maintainers. Sometimes they will point you to a piece of documentation or an old issue that already addresses your question. Sometimes it may take time for them to reply to your question. These are all normal parts of contributing to Open Source. At the end of the day maintainers of Open Source Software are human and are excited for your contribution. Being helpful and respectful when reaching out is the best way to make friends and get help.

Our favorite projects accepting contributions

The Python Cloud Developer Advocates Sarah, Pamela, Jay and Anthony and Dawn, the Python Community PM, will be streaming together to talk about how we go into Open Source Contribution. Don’t miss our chat October 10th at 2PM Pacific on the Microsoft Developer Channel.

Here are some of our favorite projects and which are participating in ⭐Hacktoberfest 2022:

Pyright is a static type checker for large Python source bases, and powers Pylance the default language server for Python development in Visual Studio Code and Visual Studio. It can run a “watch” mode and performs fast incremental updates when files are modified.
If you’re interested in contributing and maintaining efforts, the VS Code team could use the continued support in the Python community for vscode-pylint, vscode-flake8, vscode-isort, and vscode-black-formatter which are VS Code integrations for Pylint, Flake8, isort and Black respectively.
Black is “the uncompromising Python code formatter”, maintained by the Python Software Foundation, that removes the burden off developers to hand-format their code. There is also a VS Code extension for Black, which reports format errors in your code and fixes them. Black is participating in Hacktoberfest 2022.
CPython is the name of the core Python repository. The Python Developer’s Guide is the comprehensive resource for contributing to Python for both new and experienced developers. As of the release of this post, the latest is Python 3.12 of which its documentation is updated daily! A good How-To Guide written by Real Python or this talk Contributing to Core Python [YouTube] by Carol Willing’s, a Python Software Foundation Fellow, former Director and core developer for CPython, could also help point you in the right direction. The Python Software Foundation also has Python Core Mentorship that is “predicated on the idea that Python core, and Python as a whole would be served by further lowering the barrier to contribution to Python core. From fixing code bugs to translating documentation to improving Python’s official site with some front-end skills, there are lots of ways to contribute.
NumFOCUS affiliated projects are some of the most popular scientific Python packages in the world. The list of sponsored projects includes Numpy, pandas,IPython and scikit-learn. Each of these projects have their own community that contribute back their efforts into the scientific community at large.
IPython is a command shell that allows code to be interpreted for interactive computing in multiple programming languages and is an alternative to using the default Python interpreter with enhanced features like syntax highlighting and object introspection during runtime. IPython is participating in Hacktoberfest 2022.
pytest is a popular test framework to write and run tests in Python. Learn more about pytest in Testing with pytest Microsoft Learn Module or contribute to pytest this Hacktoberfest. Pytest is participating in Hacktoberfest 2022.
picologging beta is a high-performance logging library for Python that is 4-10x faster than the logging module in the standard library.
Wagtail CMS is a Django Content Management System focused on flexibility and user experience. It has an active community on Stack Overflow and Slack of developers, designers and editors that are eager to help. They participated in Google Summer of Code 2021 and 2022 where core team members help first time contributors work on summer long projects, which fosters a community that welcomes new participants. Their contribution docs go into development set up, testing and committing code. Wagtail CMS is participating in Hacktoberfest 2022.
Check out some of the Python and Azure demo projects maintained by our teammates! fire-map uses Azure Functions to map MODIS Fire Detection. diversity-orgs-django is a Django implementation deployed on Azure which aggregates tech orgs that support diversity initiatives. AZ-Queue-Tweeter is a demo application that sends images to Twitter using text from Azure Queue Storage.

We love to hear from you!

The Pythonistas at Microsoft spend their working days thinking about how to help you.

Drop us a line, keep in touch:

Talk Python projects on Azure and VS Code:

Upcoming events:

The post Contribute to Open Source with the Pythonistas at Microsoft – Hacktoberfest 2022 appeared first on Python.

October 04, 2022 08:17 PM UTC

PyCoder’s Weekly

Issue #545 (Oct. 4, 2022)

#545 – OCTOBER 4, 2022
View in Browser »

Building Chat With Django Channels and WebSockets

Building stateful web applications can be tricky, unless you use a framework, of course. Django to the rescue! In this article, learn how to build a real-time chat app using Django Channels and WebSockets.
MUHAMMED ALI

Generate a QR Code With Python

QR codes are two dimensional bar codes that allow you to embed URLs in images. Learn how to make a QR code using the qrcode library.
JERRY ZHU

Built for Seamless, Continuous Deployment - Platform.sh Is Your Web Fleet Management Solution

Platform.sh is a secure cloud hosting platform that allows you to effortlessly manage, monitor and optimize all your websites. With our solution, your development team can avoid wasting precious time updating your websites individually or fixing deployment bugs and focus on what truly matters →
PLATFORM.SH sponsor

Sneaky REST APIs With Django Ninja

In this video course, you’ll learn how to use Django Ninja, a FastAPI inspired tool for turning Django views in REST API endpoints. With Ninja, you can quickly build API endpoints.
REAL PYTHON course

Discussions

How to Fix Python

Al appears to be bored and decided to start a flame war. As his suggestions include 1-based indexing and using “x” instead of “*” for multiplication, he’s likely being sarcastic. The joy of text is you’re never quite sure.
AL SWEIGART

Python Jobs

Articles & Tutorials

PEP 698: Override Decorator for Static Typing

This Python Enhancement Proposal describes the use of a new decorator, @override, which would be used as a type-hint for methods in a sub-class that override a parent’s method. This type hint would introduce a level of safety if the parent method is refactored without corresponding changes to the child method.
PYTHON.ORG

5 Ways to Schedule Jupyter Notebooks

A Jupyter Notebook is an excellent tool for creating computational documents. There is often a requirement to update the notebook’s results at a selected time interval and publish it to the rest of the team. Piotr summarizes five different approaches for Jupyter Notebook scheduling.
PIOTR PŁOŃSKI • Shared by Piotr Płoński

Get Started Debugging Python Apps in PyCharm With This Tutorial

PyCharm is one of the most popular IDEs for Python development. This walkthrough for debugging will have you moving quickly →
ROOKOUT sponsor

Adventures With `mypyc`

The type annotation tool, mypy comes with a compiler called mypyc which uses Python type hints to generate C-extensions automatically. This blog posting describes how Steve used it to compile some of his code, the problems he ran into, and how he fixed them.
STEVE BRAZIER

Resiliency in Distributed Systems

This blog posting contains a brief summary and then two chapters from the book “Understanding Distributed Systems” by Roberto Vitillo. It describes how to make distributed systems more resilient from both the client’s and server’s perspectives.
GERGELY OROSZ

Custom Python Strings: Inheriting From `str` vs `UserString`

In this tutorial, you’ll learn how to create custom string-like classes in Python by inheriting from the built-in str class or by subclassing UserString from the collections module.
REAL PYTHON

Production Tips for Django Apps

This blog post describes seven concepts Django programmers should keep in mind when writing production ready code. Hints include information on custom user models, configuring Gunicorn, problems with serializers, and more.
RAUNAQ SINGH

Image Generation Using Stable Diffusion in KerasCV

Stable Diffusion is a powerful, open-source text-to-image generation model. This guide shows you how to generate novel images using the KerasCV Python interface.
CHOLLET, ET AL

Deploying Python Web Apps as AWS Lambda Functions

A step-by-step guide on how to deploy Python in the AWS Lambda environment. Details include managing dependencies and dealing with an ASGI application.
SIMON WILLISON’

Pytest for Beginners

This article introduces you to using pytest for testing Python code. Learn how to improve your development process and write better automated tests.
ŠPELA GIACOMELLI • Shared by Michael Herman

Hybiscus - Use a JSON API to Create Eye-Catching PDFs Without Any Design Skills

Use a simple API to build stunning PDFs from a set of pre-styled components. Choose from customisable components for layout, content or data visualisation. Get started with automating building PDFs in the cloud now, for any use case.
HYBISCUS sponsor

Want Cleaner Code? Use the Rule of Six

This article outlines six things every developer should keep in mind when coding, hopefully resulting in easy to read, more maintainable Python.
DAVIDAMOS.DEV

Developing RESTful APIs With Python and Flask

A step-by-step article that guides you through everything you need to know to write your first REST API using Flask. From installing the requirements, through mapping models, to implementing security.
KREBS & MARTINEZ • Shared by Juan Cruz Martinez

Projects & Code

difftastic: A Structural Diff That Understands Syntax

GITHUB.COM/WILFRED

minestrone: Search, Modify, and Parse Messy HTML

GITHUB.COM/ADAMGHILL

refurb: Refurbish and Modernize Python Codebases

GITHUB.COM/DOSISOD

isort: Sort Your Imports

GITHUB.COM/PYCQA

Events

STL Python

October 5, 2022
MEETUP.COM

Crafting Software

October 5, 2022
MEETUP.COM

Weekly Real Python Office Hours Q&A (Virtual)

October 5, 2022
REALPYTHON.COM

Smart Iterator Challenge (Week 2)

October 8 to October 17, 2022
MEETUP.COM

PyCon MEA @ Global DevSlam 2022

October 10 to October 14, 2022
GLOBALDEVSLAM.COM

PyCon Ghana 2022

October 13 to October 16, 2022
PYCON.ORG

PyCon ZA 2022

October 13 to October 15, 2022
PYCON.ORG

Happy Pythoning!
This was PyCoder’s Weekly Issue #545.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

October 04, 2022 07:30 PM UTC

Real Python

Python Basics: Strings and String Methods

Many programmers, regardless of their specialty, deal with text on a daily basis. For example, web developers work with text input from web forms. Data scientists process text to extract data and perform tasks like sentiment analysis, which can help identify and classify opinions in a body of text.

Collections of text in Python are called strings. Special functions called string methods are used to manipulate strings. There are string methods for changing a string from lowercase to uppercase, removing whitespace from the beginning or end of a string, replacing parts of a string with different text, and much more.

In this video course, you’ll learn how to:

Manipulate strings with string methods
Work with user input
Deal with strings of numbers
Format strings for printing

This video course is part of the Python Basics series, which accompanies Python Basics: A Practical Introduction to Python 3. You can also check out the other Python Basics courses.

Note that you’ll be using IDLE to interact with Python throughout this course.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 04, 2022 02:00 PM UTC

Talk Python to Me

#384: Python Data Visualization - Where To Start?

Do you struggle to know where to start in the wide range of Python's visualization frameworks? Not sure when to use Plotly vs. Matplotlib vs. Altair? Then this episode is for you. We have Chris Moffitt, a Talk Python course author and founder of Practical Business Python, back on the show to discuss getting started with Python's data visualization frameworks. Links from the show <div>Chris on Twitter: <a href="https://twitter.com/chris1610" target="_blank" rel="noopener">@chris1610</a> Python Data Visualization course: <a href="https://talkpython.fm/dataviz" target="_blank" rel="noopener">talkpython.fm</a> 10 tips to move from Excel to Python episode: <a href="https://talkpython.fm/episodes/show/288/10-tips-to-move-from-excel-to-python" target="_blank" rel="noopener">talkpython.fm</a> Escaping Excel Hell with Python and Pandas episode: <a href="https://talkpython.fm/episodes/show/200/escaping-excel-hell-with-python-and-pandas" target="_blank" rel="noopener">talkpython.fm</a> PB Python: <a href="https://pbpython.com" target="_blank" rel="noopener">pbpython.com</a> matplotlib: <a href="https://matplotlib.org/stable/index.html" target="_blank" rel="noopener">matplotlib.org</a> Seaborn example: <a href="https://seaborn.pydata.org/examples/multiple_bivariate_kde.html" target="_blank" rel="noopener">seaborn.pydata.org</a> Altair: <a href="https://altair-viz.github.io/" target="_blank" rel="noopener">altair-viz.github.io</a> Plotly sunburst: <a href="https://plotly.com/python/sunburst-charts/" target="_blank" rel="noopener">plotly.com</a> Plotly treemap: <a href="https://plotly.com/python/treemaps/" target="_blank" rel="noopener">plotly.com</a> streamlit: <a href="https://streamlit.io/gallery" target="_blank" rel="noopener">streamlit.io</a> Dash: <a href="https://dash.gallery/Portal/" target="_blank" rel="noopener">dash.gallery</a> Streamlit Talk Python episode: <a href="https://talkpython.fm/episodes/show/260/from-basic-script-to-interactive-data-sci-app-with-streamlit" target="_blank" rel="noopener">talkpython.fm</a> splink package: <a href="https://github.com/moj-analytical-services/splink" target="_blank" rel="noopener">github.com</a> redframes package: <a href="https://github.com/maxhumber/redframes" target="_blank" rel="noopener">github.com</a> Edward Tufte book: <a href="https://www.edwardtufte.com/tufte/books_vdqi" target="_blank" rel="noopener">edwardtufte.com</a> Watch this episode on YouTube: <a href="https://www.youtube.com/watch?v=5aER107ELt0" target="_blank" rel="noopener">youtube.com</a> Episode transcripts: <a href="https://talkpython.fm/episodes/transcript/384/python-data-visualization-where-to-start" target="_blank" rel="noopener">talkpython.fm</a> --- Stay in touch with us --- Subscribe to us on YouTube: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a> Follow Talk Python on Twitter: <a href="https://twitter.com/talkpython" target="_blank" rel="noopener">@talkpython</a> Follow Michael on Twitter: <a href="https://twitter.com/mkennedy" target="_blank" rel="noopener">@mkennedy</a> </div> Sponsors <a href='https://talkpython.fm/foundershub'>Microsoft</a> <a href='https://talkpython.fm/training'>Talk Python Training</a> <a href='https://talkpython.fm/assemblyai'>AssemblyAI</a>

October 04, 2022 08:00 AM UTC

Django Weblog

Django security releases issued: 4.1.2, 4.0.8, and 3.2.16

In accordance with our security release policy, the Django team is issuing Django 4.1.2, Django 4.0.8, and Django 3.2.16. These releases addresses the security issue detailed below. We encourage all users of Django to upgrade as soon as possible.

CVE-2022-41323: Potential denial-of-service vulnerability in internationalized URLs

Internationalized URLs were subject to potential denial of service attack via the locale parameter. This is now escaped to avoid this possibility.

This issue has medium severity, according to the Django security policy.

Thanks to Benjamin Balder Bach for the report.

Affected supported versions

Django main branch
Django 4.1
Django 4.0
Django 3.2

Resolution

Patches to resolve the issue have been applied to Django's main branch and the 4.1, 4.0, and 3.2 release branches. The patches may be obtained from the following changesets:

The following releases have been issued:

Django 4.1.2 (download Django 4.1.2 | 4.1.2 checksums)
Django 4.0.8 (download Django 4.0.8 | 4.0.8 checksums)
Django 3.2.16 (download Django 3.2.16 | 3.2.16 checksums)

The PGP key ID used for this release is Carlton Gibson: E17DF5C82B4F9D00.

General notes regarding security reporting

As always, we ask that potential security issues be reported via private email to security@djangoproject.com, and not via Django's Trac instance or the django-developers list. Please see our security policies for further information.

October 04, 2022 07:59 AM UTC

IslandT

Python RegEx Module

In this chapter let us create a few examples with Python built-in RegEx module’s methods and see what this particular module can do!

As always, you need to import the module into your python file before using it.

import re

Find whether a phrase is contained within a string or not and uses case insensitive search.

text = "Well, again, it is Hello WORLD!..."
result = (re.match(".*Hello World.*", text, re.I))

Below is the return Match object.

<re.Match object; span=(0, 34), match='Well, again, it is Hello WORLD!...'>

Which shows it matches our search.

Besides using to search a string, re module also can be used to split the text into a list. The below line will split the text into a list at each white-space character.

text = "Well, again, it is Hello WORLD!..."
result = re.split("\s", text) # ['Well,', 'again,', 'it', 'is', 'Hello', 'WORLD!...']

At last, let us replaced hello world with a good morning phrase.

text = "Well, again, it is Hello WORLD!..."
result = re.sub("Hello WORLD", "good morning", text) # Well, again, it is good morning!...

There is still more to learn from this module which I will leave to you all to explore by yourself.

October 04, 2022 04:43 AM UTC

Matt Layman

Episode 17 - Accepting Files

On this episode, we’re going to dig into file management. Unlike the static files that you create for the app yourself, you may want your app to accept files from your users. Profile pictures are a good example of user files. You’ll see how Django handles those kinds of files and how to deal with them safely. Listen at djangoriffs.com or with the player below. Last Episode On the last episode, we looked at how to manage settings on your Django site.

October 04, 2022 12:00 AM UTC

October 03, 2022

Codementor

Getting started with Rocksdb and Python

In this post, I am going to discuss RocksDB. RocksDB is an embeddable persistent key-value store system developed by Facebook. It was originally forked from LevelDB which was created by Google.…

October 03, 2022 05:03 PM UTC

Zato Blog

HL7 FHIR Integrations in Python

HL7 FHIR, pronounced “fire”, is a data model and message transfer protocol designed to facilitate the exchange of information among systems used in health care settings.

In such environments, a FHIR server will assume the role of a central repository of health records with other systems integrating with it, potentially in a hub-and-spoke fashion, thus letting the FHIR server become a unified and consistent source of data that would otherwise stay confined to a silo of each individual health information system.

While FHIR is the way forward, the current reality of health care systems is that much of the useful and actionable information is distributed and scattered among many individual data sources - paper-based directories, applications or data bases belonging to the same or different enterprises - and that directly hampers the progress towards delivering good health care. Anyone witnessing health providers copy-and-pasting the same information from one application to another, not having access to the already existing data, not to mention people not having an easy way to access their own data about themselves either, can understand what the lack of interoperability looks like externally.

The challenges that integrators face are two-fold. On the one hand, the already existing systems, including software as well as medical appliances, were often not, or are still not being, designed for the contemporary inter-connected world. On the other hand, FHIR in itself is a relatively new technology which means that it is not straightforward to re-use the existing skills and competencies.

Zato is an open-source platform that makes it possible to integrate systems with FHIR using Python. Specifically, its support for FHIR enables quick on-boarding of integrators who may be new to health care interoperability, who are coming to FHIR with previous experience or interest in web development technologies, and who need an easy way to get started with and to navigate the complex landscape of health care integrations.

Connecting to FHIR servers

Outgoing FHIR connections are what allows Python-based services to communicate with FHIR servers. Throughout the rest of the chapter, the following definition will be used. It connects to a live, publicly available FHIR server.

Filling out the form below will suffice, there is no need for any server restarts. This principle, that restarts are not needed, applies all throughout the platform, whenever you change any piece of configuration, it will be automatically propagated as necessary.

Name: FHIR.Sample
Address: https://simplifier.net/zato
Security: No security definition (we will talk about security later)
TLS CA Certs: Default bundle

Retrieving data from FHIR servers

In Python code, you obtain client connections to FHIR servers through self.out.hl7.fhir objects, as in the example below which first refers to the server by its name and then looks up all the patients in the server.

The structure of the Patient resource that we expect to receive can be found here.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class FHIService1(Service):
    name = 'demo.fhir.1'

    def handle(self) -> 'None':

        # Connection to use
        conn_name = 'FHIR.Sample'

        with self.out.hl7.fhir[conn_name].conn.client() as client:

            # This is how we can refer to patients
            patients = client.resources('Patient')

            # Get all active patients, sorted by their birth date
            result = patients.sort('active', '-birthdate')

            # Log the result that we received
            for elem in result:
                self.logger.info('Received -> %s', elem['name'])

Invoking the service will store in logs the data expected:

INFO - Received -> [{'use': 'official', 'family': 'Chalmers', 'given': ['Peter', 'James']}]

For comparison, this is what the FHIR server displays in its frontend.

the information is the same.

Storing data in FHIR servers

To save information in a FHIR server, create the required resources and call .save to permanently store the data in the server. Resources can be saved either individually (as in the example below) or as a bundle.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class CommandsService(Service):
    name = 'demo.fhir.2'

    def handle(self) -> 'None':

        # Connection to use
        conn_name = 'FHIR.Sample'

        with self.out.hl7.fhir[conn_name].conn.client() as client:

            # First, create a new patient
            patient = client.resource('Patient')

            # Save the patient in the FHIR server
            patient.save()

            # Create a new appointment object
            appointment = client.resource('Appointment')

            # Who will attend it
            participant = {
                'actor': patient,
                'status':'accepted'
            }

            # Fill out the information about the appointment
            appointment.status = 'booked'
            appointment.participant = [participant]
            appointment.start  = '2022-11-11T11:11:11.111+00:00'
            appointment.end    = '2022-12-22T22:22:22.222+00:00'

            # Save the appointment in the FHIR server
            appointment.save()

Learning what FHIR resources to use

The “R” in FHIR stands for “Resources” and the sample code above uses resources such a Patient or Appointment but how does one learn what other resources exist and what they look like? In other words, how does one learn the underlying data model?

First, you need to get familiar with the spec itself which, in addition to textual information, offers visualizations of the data model. For instance, here is the description of the Observation object, including details such as all the attributes an Observation is composed of as well as their multiplicities.

Secondly, do spend time with FHIR servers such as Simplifier. Use Zato services to create test resources, look them up and compare the results with what the spec says. There is no substitute for experimentation when learning a new data model.

FHIR security

Outgoing FHIR connections can be secured in several ways, depending on what a given FHIR requires:

With Basic Auth definitions
With OAuth definitions
With SSL/TLS. If the server is not a public one (e.g. it is in a private network with a private IP address), you may need to upload the server’s certificate to Zato first if you plan to use SSL/TLS because, without it, the server’s certificate may be rejected.

MLLP, HL7 v2 and v3

While FHIR is what new deployments use, it is worth to add that there are still other HL7 versions frequently seen in integrations:

Version 2, using its own MLLP protocol
Version 3, using XML

Both of them can be used in Zato services, in both directions. For instance, it is possible to both receive HL7 v2 messages as well as to send them to external applications. It is also possible to send v2 messages using REST in addition to MLLP.

Next steps

Start the tutorial to learn more technical details about Zato, including its architecture, installation and usage. After completing it, you will have a multi-protocol service representing a sample scenario often seen in banking systems with several applications cooperating to provide a single and consistent API to its callers.
Check more resources for developers and screenshots.
Para aprender más sobre las integraciones de Zato y API en español, haga clic aquí

October 03, 2022 04:41 PM UTC

Python Morsels

TypeError: can only concatenate str (not "int") to str

Python crashed with the error TypeError: can only concatenate str (not "int") to str. Essentially Python's saying you've used + between a string and a number and it's unhappy about it. Let's talk about how to fix this issue and how to avoid it more generally.

Table of contents

What does that error message mean?

The last line is in a Python traceback message is the most important one.

TypeError: can only concatenate str (not "int") to str

TypeError: can only concatenate str (not "int") to str

That last line says the exception type (TypeError) and the error message (can only concatenate str (not "int") to str).

This error message says something similar:

TypeError: unsupported operand type(s) for +: 'int' and 'str'

TypeError: unsupported operand type(s) for +: 'int' and 'str'

Both of these error messages are trying to tell us that we're trying to use the + operator (used for string concatenation in Python) between a string and a number.

You can use + between a string and a string (to concatenate them) or a number and a number (to add them). But in Python, you cannot use + between a string and a number because Python doesn't do automatic type coercion. Type conversions are usually done manually in Python.

How do we fix this?

How we fix this will …

Read the full article: https://www.pythonmorsels.com/can-only-concatenate-str/

October 03, 2022 03:00 PM UTC

Real Python

Build Enumerations of Constants With Python's Enum

Some programming languages, like Java and C++, include syntax that supports a data type known as enumerations, or just enums. This data type allows you to create sets of semantically related constants that you can access through the enumeration itself. Python doesn’t have a dedicated syntax for enums. However, the Python standard library has an enum module that supports enumerations through the Enum class.

If you come from a language with enumerations, and you’re used to working with them, or if you just want to learn how to use enumerations in Python, then this tutorial is for you.

In this tutorial, you’ll learn how to:

Create enumerations of constants using Python’s Enum class
Work with enumerations and their members in Python
Customize enumeration classes with new functionalities
Code practical examples to understand why you would use enumerations

Additionally, you’ll explore other specific enumeration types that live in enum, including IntEnum, IntFlag, and Flag. They’ll help you create specialized enums.

To follow along with this tutorial, you should be familiar with object-oriented programming and inheritance in Python.

Source Code: Click here to download the free source code that you’ll use to build enumerations in Python.

Getting to Know Enumerations in Python

Several programming languages, including Java and C++, have a native enumeration or enum data type as part of their syntax. This data type allows you to create sets of named constants, which are considered members of the containing enum. You can access the members through the enumeration itself.

Enumerations come in handy when you need to define an immutable and discrete set of similar or related constant values that may or may not have semantic meaning in your code.

Days of the week, months and seasons of the year, Earth’s cardinal directions, a program’s status codes, HTTP status codes, colors in a traffic light, and pricing plans of a web service are all great examples of enumerations in programming. In general, you can use an enum whenever you have a variable that can take one of a limited set of possible values.

Python doesn’t have an enum data type as part of its syntax. Fortunately, Python 3.4 added the enum module to the standard library. This module provides the Enum class for supporting general-purpose enumerations in Python.

Enumerations were introduced by PEP 435, which defines them as follows:

An enumeration is a set of symbolic names bound to unique, constant values. Within an enumeration, the values can be compared by identity, and the enumeration itself can be iterated over. (Source)

Before this addition to the standard library, you could create something similar to an enumeration by defining a sequence of similar or related constants. To this end, Python developers often used the following idiom:

>>>

>>> RED, GREEN, YELLOW = range(3)

>>> RED
0

>>> GREEN
1

Even though this idiom works, it doesn’t scale well when you’re trying to group a large number of related constants. Another inconvenience is that the first constant will have a value of 0, which is falsy in Python. This can be an issue in certain situations, especially those involving Boolean tests.

Note: If you’re using a Python version before 3.4, then you can create enumerations by installing the enum34 library, which is a backport of the standard-library enum. The aenum third-party library could be an option for you as well.

In most cases, enumerations can help you avoid the drawbacks of the above idiom. They’ll also help you produce more organized, readable, and robust code. Enumerations have several benefits, some of which relate to ease of coding:

Allowing for conveniently grouping related constants in a sort of namespace
Allowing for additional behavior with custom methods that operate on either enum members or the enum itself
Providing quick and flexible access to enum members
Enabling direct iteration over members, including their names and values
Facilitating code completion within IDEs and editors
Enabling type and error checking with static checkers
Providing a hub of searchable names
Mitigating spelling mistakes when using the members of an enumeration

They also make your code robust by providing the following benefits:

Ensuring constant values that can’t be changed during the code’s execution
Guaranteeing type safety by differentiating the same value shared across several enums
Improving readability and maintainability by using descriptive names instead of mysterious values or magic numbers
Facilitating debugging by taking advantage of readable names instead of values with no explicit meaning
Providing a single source of truth and consistency throughout the code

Now that you know the basics of enumerations in programming and in Python, you can start creating your own enum types by using Python’s Enum class.

Creating Enumerations With Python’s `Enum`

Python’s enum module provides the Enum class, which allows you to create enumeration types. To create your own enumerations, you can either subclass Enum or use its functional API. Both options will let you define a set of related constants as enum members.

Read the full article at https://realpython.com/python-enum/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

October 03, 2022 02:00 PM UTC

Python for Beginners

How to Use Comments in Python?

Writing code in Python can be challenging. That’s why it’s important to be as clear as possible when naming variables and functions. A good name will explain what a variable is used for, or what a function does. However, good naming conventions will only get you so far. What if we’re dealing with complex logic or an otherwise confusing algorithm? Or what if we need to share and explain our intentions with another programmer? It would be helpful to leave notes for ourselves, or others, in the code. That’s what Python comments are for.

With comments, we can make our programs easier to understand, and easier to make changes to later. Python comments are ignored by the interpreter, so they won’t interfere with how your code runs.

Topics we’ll cover:

Why writing comments is important
How to use comments to improve your Python code
Best practices for commenting in Python

When to Write Python Comments?

Let’s face it, programming isn’t easy; we could all use some help. Comments allow a programmer to add some extra information that explains what’s happening in the code.

If you’re working on a large project, it can be difficult to keep track of everything that’s going on. This is especially true when returning to a project after a long absence.

Such a scenario is more common than you might think. Programmers are often required to juggle many projects at once.

Coming back to a project is often painful. Updating a project is much easier if the code is clear, follows styling guides, and makes use of well placed comments.

When it comes to working on a team, comments are even more crucial. It can be hard enough to decipher your own code, let alone something written by a coworker, or stranger. Following some basic guidelines will help you write clear, effective comments.

How to Write a Comment in Python?

Comments are usually written in blocks and explain the code that follows them. Typically, comments are used to explain complex formulas, algorithms, or design decisions.

Definition: A comment is code that is meant to be programmer-readable, and ignored by the computer. It’s a message to ourselves, or other coders working on a project.

Usage: Use comments to explain functions, algorithms, maths, design choices, or anything else you may need to refer to in the future.

Syntax: Most comments in Python are written using the # characters. Python ignores any code that follows a hash sign (#) when the program runs.

There are three different types of comments available in Python: block, inline, and docstring. Knowing when to use each is important. For instance, docstrings can be used to automatically generate the documentation for a project.

Writing Single Line Comments in Python

A single line comment, known as a block comment in Python, starts with the # character. The block comment is used to explain code that follows the comment.

Remember to keep comments as short as possible. It’s easy to overdo it, adding comments about everything under the impression that the program becomes clearer the more words there are. That isn’t true.

Brevity is the soul of wit, as they say. The same is true of commenting your code. It’s a good idea to keep it simple.

Example 1: Python block comments

# this is a block comment in python, it uses a single line

# a list of keywords provided by the client
keywords = ["tv shows", "best streaming tv","hit new shows"]

Example 2: The Pythagorean Theorem

import math

# the sides of a triangle opposite the hypotenuse
a = 2
b = 3

# the Pythagorean Theorem
c = math.sqrt(a**2 + b**2)

print(c)

Output

3.605551275463989

Inline Comments

In order to save space, it’s sometimes a good idea to put a comment on the same line as the code you’re describing. This is known as an inline comment.

Inline comments are also written with the # character. They are placed at the end of a code statement. Inline comments are useful when you want to include more information about variable names, or explain a function in greater detail.

Example 3: Writing Inline Comments in Python

income = 42765 # annual taxable income in USD
tax_rate = 0.22 # 22% annual tax rate
print("You owe ${}".format(income*tax_rate))

Writing Multiline Comments in Python

In Python, the most direct path to writing a comment that’s more than one line long is to use multiple block comments. Python will ignore the hash signs (#) and you’ll be left with multiple lines of comments.

Example 4: Multiline Comments

chessboard = {}
# we need to use a nested for loop to build a grid of tiles
# the grid of tiles will serve as a basis for the game board
# 0 will stand for an empty space
for i in range(1,9):
    for j in range(1,9):
        chessboard[i,y] = 0

Unfortunately, with this method it’s necessary to add a hashtag before every line.

While you can’t technically write a comment that’s longer than a single line, Python docstrings offer a way of getting around this rule.

Docstrings are meant to be associated with functions and classes. They’re used to create documentation for your project as the docstring will become associated with the functions they appear under.

Docstrings use two sets of triple quotes to designate that the text between them is special. These triple quotes are meant to be placed after functions and classes. They are used to document Python code.

Example 5: Using a docstring with a Function

def string_reverse(a_string):
    """A helper function to reverse strings

    Args:
        a_string (sring):

    Returns:
        string: A reversed copy of the input string
    """
    reverse = ""
    i = len(a_string)
    while i > 0:
        reverse += a_string[i-1]
        i = i-1

    return reverse

# driver code
print(string_reverse("Is this how you turn a phrase?"))

Output

?esarhp a nrut uoy woh siht sI

We can also use these triple quotes to create comments in Python. Because they are read as strings, they won’t affect our program when it runs.

""" this is a multiline comment in Python

    it's basically just a string

    but you have to be careful when using this method.
"""

Suggested Reading: To read about more computer science topics, you can read this article on dynamic role-based authorization using ASP.net. You can also read this article on user activity logging using Asp.net.

Making Comments For Yourself

Some comments are left for the developers themselves. For instance, you might leave a comment to remind yourself where you left off on something you were unable to finish on a previous work day.

Large programs are difficult to manage. When you open a project, especially if it’s one you haven’t looked at in a while, you’ll sometimes find that you need to “reload” the code. Essentially, you’ll need to remind yourself of the details of the project.

Comments can help save time in this regard. Some well placed comments can help you avoid those head-scratching moments where you ask, “What exactly was I doing here?”

Commenting for Others

Anytime you’re working on a team, Python or otherwise, communication is vital. Confusion here only leads to wasted time and frustration.

That’s why programmers should use comments to explain their code. Following style guides and appropriately commenting your work will make sure everyone is on the same page.

If you are into machine learning, you can read this article on regression in machine learning. You might also like this article on polynomial regression using sklearn in python.

Best Practices for Commenting in Python

While Python comments usually make a program easier to understand, that isn’t always the case. There are some unwritten rules that one should follow if they hope to write productive comments.

It’s true that adding more comments doesn’t necessarily improve your code’s readability. In fact, having too many comments is just as bad as having too few. So how do you know how much to comment, when to comment, or when NOT to comment?

Commenting Guidelines:

Avoid redundancy
Be respectful
Less is more

Knowing when comments are unnecessary is an important skill to master. Often, you’ll see students repeat function names in variables, as if writing them again will make their meaning clearer.

Don’t make unnecessary comments like this one:


# the name of the author
author_name = "Le Guin, Ursula K"

Let’s take a look at a longer example with multiple comments and a docstring. Use the comments to navigate the bubble sort algorithm.

Example 7: Commenting the bubble sort algorithm

def BubbleSort(nums):
    """A method for quickly sorting small arrays."""
    total_nums = len(nums)

    for i in range(total_nums):
        for j in range(0, total_nums-i-1):
            # traverse the array backwards
            # swap elements that are out of order
            if nums[j] > nums[j+1]:
                temp = nums[j]
                nums[j] = nums[j+1]
                nums[j+1] = temp

nums = [100,7,19,83,63,97,1]

BubbleSort(nums)
print(nums)

Output

[1, 7, 19, 63, 83, 97, 100]

Helpful Links

Hopefully I’ve impressed upon you the importance of properly commenting your Python code. It will save you time and make your work easier.
Hungry for more? Sharpen your Python coding skills by following these links to more Python For Beginners tutorials.

The post How to Use Comments in Python? appeared first on PythonForBeginners.com.

October 03, 2022 01:00 PM UTC

IslandT

Python math Module

math is a Python built-in module which means it comes together with the python package and you do not need to install it separately. In the following examples let us create various little programs to use the math module to solve some problems.

In order to use the math module, you need to import it into the Python file.

import math

Find the square root of 32.

math.sqrt(32) #5.656854249492381

Find the degree of pi.

math.degrees(math.pi) # 180 degrees

Find the sin of 45.

math.sin(45) # 0.8509035245341184

Find the cos of 45.

math.cos(45) # 0.5253219888177297

Find the tan of 45.

math.tan(45) # 1.6197751905438615

Find log 10 of 13.

math.log(13,10) # 1.1139433523068367

Find the exponential of 13.

math.exp(13) # 442413.3920089205

Truncate the decimal part of 3.78.

math.trunc(3.78) # 3

Get the ceiling value of 3.78.

math.ceil(3.78) # 4

Get the floor value of 3.78.

math.floor(3.78) # 3

Get the pi value.

math.pi # 3.141592653589793

Get the exponential value.

math.e # 2.718281828459045

Find the hypotenuse of the triangle with x = 3, y = 4.

math.hypot(3,4) # 5

That is it, there are lots more other methods of the math module that awaiting you to explore thus it is your turn to try them out!

October 03, 2022 12:52 PM UTC

Mike Driscoll

PyDev of the Week: Mark Mikofski

This week we welcome Mark Mikofski (@bwanamarko) as our PyDev of the Week! Mark is a core developer on the pvlib tool, which is used "for simulating the performance of photovoltaic energy systems." You can check out his Github.io page to see Mark's blog, talks, and more!

pvlib logo

Let's spend some time getting to know Mark better.

Can you tell us a little about yourself (hobbies, education, etc):

I was raised by a single mother, who herself was raised by a single mother, my grandmother, who was a Cuban immigrant. My mother was a teacher and shared her love of learning with me, especially mathematics. I majored in mechanical engineering, and after 3-years in the US Peace Corps in Tanzania, East Africa, where I taught high school math, I eventually returned to school to complete a master's and doctorate in mechanical engineering focusing on the formation of harmful emissions from combustion and air quality. My grad school studies were a natural segue into renewables, which is where I've been working for the past 15+ years. In my free time, I love to run, climb, read, play guitar, and spend time with my family.

Why did you start using Python?

As an undergraduate, we were taught Pascal, which hasn't been a particularly useful language, but I had been programming since middle school when I used BASIC on my Atari 600xl, so I loved Pascal anyway. Later I learned MATLAB which became my analysis tool of choice for many years. I had heard rumors about "Python" and even had to install it during grad school to use Cantera a library for studying the thermodynamics of chemical reactions like combustion. However, I thought that Python was a "serious" programming language, in the same league as C/C++ and FORTRAN. I had been exposed to FORTRAN as part of my grad school research, and it was terrifying, so I didn't want to go anywhere near it. I finally tried Python in 2012, converting an Excel analysis to what eventually became PVMismatch, and I immediately had that a-ha! moment described so well in XKCD's famous comic. After that I quickly became a Python enthusiast, and started using it for all my analysis.

What other programming languages do you know and which is your favorite?

Other than Python I'm not proficient in any other languages except maybe MATLAB, which I've used since undergrad. I've dabbled in C# for work, but not because I really wanted to. I also experimented once with Java to make an Android app, but it wasn't for me. I've learned just enough HTML, CSS, and JS so I can make web apps using Django/Flask and Bootstrap, but mainly just for proofs of concepts, and then I let the real developers take over. I can also use SQL but I often have to search the web for help on writing simple queries. I had to write some FORTRAN in grad school, but it was terrifying. I've written a few lines of C but I've never tried to understand C++. I also sometimes have to wade through some R, but only long enough to rewrite it in Python. I think Julia sounds cool, but I haven't learned to use it yet. So I guess my favorite language is still just Python.

What projects are you working on now?

pvlib is the main project I contribute to outside of work. It's a community project for modelling energy from solar-power systems. It's been really amazing to collaborate with the other maintainers. pvlib started as a MATLAB toolbox developed by Sandia National Laboratories researcher Cliff Hansen and maybe some other folks. Later Rob Andrews, then a PhD student, ported it to Python, and soon after University of Arizona research professor Will Holmgren contributed enough for the v0.1.0 release in 2015. I didn't make my first contribution until 2016, and it was just a tiny change in the package overview example. Since then, I've made more contributions, and I've used pvlib more and more in my everyday work. Sometime in 2020 I took a formal role with pvlib, joining the maintainers and focusing mostly on building the pvlib community which has grown to over 600 members from academia, national labs, and industry. Around this time NREL researcher and frequent collaborator, Kevin Anderson also joined the maintainers, we published our JOSS paper, and pvlib became a NumFOCUS-affiliated project. In the last few years, it's been remarkable to see the enthusiasm of renewable energy analysts when they begin to feel empowered using pvlib and Python in their everyday work to answer questions they didn't realize were at their fingertips, and to automate processes that previously took hours. We've given three pvlib tutorials, including one at PyData Global in 2021 which uses JupyterHub to make it easy and accessible to anyone to get started with pvlib without needing to install anything. I also volunteer with NumFOCUS on the affiliated projects selection committee and Diversity and Inclusion in Scientific Computing (DISC) committee, to make sure these opportunities are available to as many folks as possible and to make sure we are using all of our talents and getting as much cognitive diversity as we can.

Which Python libraries are your favorite (core or 3rd party)?

My favorite 3rd party library is Seaborn for making amazing statistical plots. I'm always so proud of myself when I manage to make what I think is a really amazing Seaborn plot. You might be surprised, but my favorite core library is pathlib. I just love the intuitive reuse of the slash operator to join paths, which is so much more elegant than the older os.path.join(). I also like the calendar core library. No one should ever have to remember how many days are in a particular month or if it's a leap year. Seriously

What is your elevator pitch for pvlib?

I've already said a lot about pvlib, so I'll use this question to give my elevator pitch for devs, quants, and data scientists in renewables. For folks looking for a new challenge, looking for a way to make a meaningful impact, now is the time to pivot hard to renewables. We don't have a lot of time before the climate crisis becomes irreversibly much worse than it already is. And the good news is we have solutions, renewable energy, battery storage, green hydrogen, and other clean, carbon-free alternatives are ready today, but we need to deploy them at a massive scale that will require collaboration from industry, government, investors, grid operators, construction, labor, manufacturing, and basically all corners of the US workforce and economy. We will need to deploy over a terawatt of renewable energy into the United States grid by 2035 or basically all of the installed capacity from the last 10 years every year for the next 10 years. That's over 100 gigawatts per year. We need everyone to get involved, and devs, quants, and data scientists can play a key role. So come join the energy transition.

What are the top 3 things you find most exciting about solar energy or solar technology?

1. The technology is constantly improving so there's always something new to learn
2. Working in solar energy makes me feel better about the climate crisis, because rather than focusing on our impending doom, I'm working on accelerating the energy transition, and I believe that if enough of us get involved in all sectors of industry and government, that we can make a positive impact.
3. The people in solar are really great. I am constantly overcome with awe by how much people care about each other and about the planet.

Is there anything else you’d like to say?

I just want people to know that they don't have to know or do anything special to use Python. Anyone can use Python to unlock the potential of digital and computing to supercharge whatever work they're doing. You don't have to be a software engineer or developer or computer scientist to use Python. You can replace existing tools in your analysis toolchain with Python, and automate manual processes to analyze many orders of magnitude more data to make statistically significant conclusions. So don't be afraid, and don't wait like I did. Just give Python a try today. I really like this zero to hero presentation by Annalise Miller at the PV Performance Modelling Collaborative workshop last month: https://youtu.be/-6TzTVWBeOo.

Thanks for doing the interview, Mark!

The post PyDev of the Week: Mark Mikofski appeared first on Mouse Vs Python.

October 03, 2022 12:30 PM UTC

Anwesha Das

Running Snowflake proxy

Snowflake is a technology to allow people from all over the world to access censored applications and websites.

Similar to how VPNs assist users in getting around Internet censorship, Snowflake helps you avoid being noticed by Internet censors by making your Internet activity appear as though you&aposre using the Internet for a regular video or voice call.

I am running a Snowflake proxy for sometime now. Installed on a server using the Ansible role. I also sent in couple of patches to the role.

$  systemctl status snowflake-proxy.service 
● snowflake-proxy.service - snowflake-proxy
     Loaded: loaded (/etc/systemd/system/snowflake-proxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-09-29 07:39:32 UTC; 4 days ago
       Docs: man:snowflake-proxy
             https://snowflake.torproject.org/
   Main PID: 1188368 (proxy)
      Tasks: 9 (limit: 9495)
     Memory: 138.8M
        CPU: 2h 50min 12.358s
     CGroup: /system.slice/snowflake-proxy.service
             └─1188368 /usr/bin/proxy -capacity 0

Oct 03 02:39:33 XXXX proxy[1188368]: 2022/10/03 02:39:33 In the last 1h0m0s, there were 5 connections. Traffic Relayed ↑ 61 MB, ↓ 7 MB.
Oct 03 03:39:33 XXXX proxy[1188368]: 2022/10/03 03:39:33 In the last 1h0m0s, there were 5 connections. Traffic Relayed ↑ 4 MB, ↓ 714 KB.
Oct 03 04:39:33 XXXX proxy[1188368]: 2022/10/03 04:39:33 In the last 1h0m0s, there were 5 connections. Traffic Relayed ↑ 25 MB, ↓ 2 MB.
Oct 03 05:30:51 XXXX proxy[1188368]: sctp ERROR: 2022/10/03 05:30:51 [0xc0014eea80] stream 1 not found)
Oct 03 05:30:51 XXXX proxy[1188368]: sctp ERROR: 2022/10/03 05:30:51 [0xc0014eea80] stream 1 not found)
Oct 03 05:39:33 XXXX proxy[1188368]: 2022/10/03 05:39:33 In the last 1h0m0s, there were 12 connections. Traffic Relayed ↑ 39 MB, ↓ 7 MB.
Oct 03 06:39:33 XXXX proxy[1188368]: 2022/10/03 06:39:33 In the last 1h0m0s, there were 17 connections. Traffic Relayed ↑ 83 MB, ↓ 19 MB.
Oct 03 07:39:33 XXXX proxy[1188368]: 2022/10/03 07:39:33 In the last 1h0m0s, there were 13 connections. Traffic Relayed ↑ 180 MB, ↓ 26 MB.
Oct 03 08:39:33 XXXX proxy[1188368]: 2022/10/03 08:39:33 In the last 1h0m0s, there were 17 connections. Traffic Relayed ↑ 101 MB, ↓ 99 MB.
Oct 03 09:39:33 XXXX proxy[1188368]: 2022/10/03 09:39:33 In the last 1h0m0s, there were 32 connections. Traffic Relayed ↑ 238 MB, ↓ 21 MB.

You can even run a proxy in your browser.

October 03, 2022 09:56 AM UTC

The Digital Cat

Delegation: composition and inheritance in object-oriented programming

Introduction¶

Object-oriented programming (OOP) is a methodology that was introduced in the 60s, though as for many other concepts related to programming languages it is difficult to give a proper date. While recent years have witnessed a second youth of functional languages, object-oriented is still a widespread paradigm among successful programming languages, and for good reasons. OOP is not the panacea for all the architectural problems in software development, but if used correctly can give a solid foundation to any system.

It might sound obvious, but if you use an object-oriented language or a language with strong OOP traits, you have to learn this paradigm well. Being very active in the Python community, I see how many times young programmers are introduced to the language, the main features, and the most important libraries and frameworks, without a proper and detailed description of OOP and how OOP is implemented in the language.

The implementation part is particularly important, as OOP is a set of concepts and features that are expressed theoretically and then implemented in the language, with specific traits or choices. It is very important, then, to keep in mind that the concepts behind OOP are generally shared among OOP languages, but are not tenets, and are subject to interpretation.

What is the core of OOP? Many books and tutorials mention the three pillars encapsulation, delegation, and polymorphism, but I believe these are traits of a more central concept, which is the collaboration of entities. In a well-designed OO system, we can observe a set of actors that send messages to each other to keep the system alive, responsive, and consistent.

These actors have a state, the data, and give access to it through an interface: this is encapsulation. Each actor can use functionalities implemented by another actor sending a message (calling a method) and when the relationship between the two is stable we have delegation. As communication happens through messages, actors are not concerned with the nature of the recipients, only with their interface, and this is polymorphism.

Alan Kay, in his "The Early History of Smalltalk", says

In computer terms, Smalltalk is a recursion on the notion of computer itself. Instead of dividing "computer stuff" into things each less strong than the whole — like data structures, procedures, and functions which are the usual paraphernalia of programming languages — each Smalltalk object is a recursion on the entire possibilities of the computer. Thus its semantics are a bit like having thousands and thousands of computers all hooked together by a very fast network.

I find this extremely enlightening, as it reveals the idea behind the three pillars, and the reason why we do or don't do certain things in OOP, why we consider good to provide some automatic behaviours or to forbid specific solutions.

By the way, if you replace the word "object" with "microservice" in the quote above, you might be surprised by the description of a very modern architecture for cloud-based systems. Once again, concepts in computer science are like fractals, they are self-similar and pop up in unexpected places.

In this post, I want to focus on the second of the pillars of object-oriented programming: delegation. I will discuss its nature and the main two strategies we can follow to implement it: composition and inheritance. I will provide examples in Python and show how the powerful OOP implementation of this language opens the door to interesting atypical solutions.

For the rest of this post, I will consider objects as mini computers and the system in which they live a "very fast network", using the words of Alan Kay. Data contained in an object is the state of the computer, its methods are the input/output devices, and calling methods is the same thing as sending a message to another computer through the network.

Delegation in OOP¶

Delegation is the mechanism through which an actor assigns a task or part of a task to another actor. This is not new in computer science, as any program can be split into blocks and each block generally depends on the previous ones. Furthermore, code can be isolated in libraries and reused in different parts of a program, implementing this "task assignment". In an OO system the assignee is not just the code of a function, but a full-fledged object, another actor.

The main concept to retain here is that the reason behind delegation is code reuse. We want to avoid code repetition, as it is often the source of regressions; fixing a bug in one of the repetitions doesn't automatically fix it in all of them, so keeping one single version of each algorithm is paramount to ensure the consistency of a system. Delegation helps us to keep our actors small and specialised, which makes the whole architecture more flexible and easier to maintain (if properly implemented). Changing a very big subsystem to satisfy a new requirement might affect other parts system in bad ways, so the smaller the subsystems the better (up to a certain point, where we incur in the opposite problem, but this shall be discussed in another post).

There is a dichotomy in delegation, as it can be implemented following two different strategies, which are orthogonal from many points of view, and I believe that one of the main problems that object-oriented systems have lies in the use of the wrong strategy, in particular the overuse of inheritance. When we create a system using an object-oriented language we need to keep in mind this dichotomy at every step of the design.

There are four areas or points of views that I want to introduce to help you to visualise delegation between actors: visibility, control, relationship, and entities. As I said previously, while these concepts apply to systems at every scale, and in particular to every object-oriented language, I will provide examples in Python.

The first way to look at delegation is through the lenses of state sharing. As I said before the data contained in an object can be seen as its state, and if hearing this you think about components in a frontend framework or state machines you are on the right path. The state of a computer, its memory or the data on the mass storage, can usually be freely accessed by internal systems, while the access is mediated for external ones. Indeed, the level of access to the state is probably one of the best ways to define internal and external systems in a software or hardware architecture.

When using inheritance, the child class shares its whole state with the parent class. Let's have a look at a simple example

class Parent:
    def __init__(self, value):
        self._value = value 3

    def describe(self): 1
        print(f"Parent: value is {self._value}")

class Child(Parent):
    pass

>>> cld = Child(5)
>>> print(cld._value)
5
>>> cld.describe() 2
Parent: value is 5

As you can see, describe is defined in Parent 1, so when the instance cld calls it 2, its class Child delegates the call to the class Parent. This, in turn, uses _value as if it was defined locally 3, while it is defined in cld. This works because, from the point of view of the state, Parent has complete access to the state of Child. Please note that the state is not even enclosed in a name space, as the state of the child class becomes the state of the parent class.

Composition, on the other side, keeps the state completely private and makes the delegated object see only what is explicitly shared through message passing. A simple example of this is

class Logger:
    def log(self, value):
        print(f"Logger: value is {value}")


class Process:
    def __init__(self, value):
        self._value = value 1
        self.logger = Logger()

    def info(self):
        self.logger.log(self._value) 2

>>> prc = Process(5)
>>> print(prc._value)
5
>>> prc.info()
Logger: value is 5

Here, instances of Process have an attribute _value 1 that is shared with the classLogger only when it comes to calling Logger.log 2 inside their info method. Logger objects have no visibility of the state of Process objects unless it is explicitly shared.

Note for advanced readers: I'm clearly mixing the concepts of instance and class here, and blatantly ignoring the resulting inconsistencies. The state of an instance is not the same thing as the state of a class, and it should also be mentioned that classes are themselves instances of metaclasses, at least in Python. What I want to point out here is that access to attributes is granted automatically to inherited classes because of the way __getattribute__ and bound methods work, while in composition such mechanisms are not present and the effect is that the state is not shared.

Control: implicit and explicit delegation

Another way to look at the dichotomy between inheritance and composition is that of the control we have over the process. Inheritance is usually provided by the language itself and is implemented according to some rules that are part of the definition of the language itself. This makes inheritance an implicit mechanism: when you make a class inherit from another one, there is an automatic and implicit process that rules the delegation between the two, which makes it run outside our control.

Let's see an example of this in action using inheritance

class Window:
    def __init__(self, title, size_x, size_y):
        self._title = title
        self._size_x = size_x
        self._size_y = size_y

    def resize(self, new_size_x, new_size_y):
        self._size_x = new_size_x
        self._size_y = new_size_y
        self.info()

    def info(self): 2
        print(f"Window '{self._title}' is {self._size_x}x{self._size_y}")


class TransparentWindow(Window):
    def __init__(self, title, size_x, size_y, transparency=50):
        self._title = title
        self._size_x = size_x
        self._size_y = size_y
        self._transparency = transparency

    def change_transparency(self, new_transparency):
        self._transparency = new_transparency

    def info(self): 1
        super().info() 3
        print(f"Transparency is set to {self._transparency}")

At this point we can instantiate and use TransparentWindow

>>> twin = TransparentWindow("Terminal", 640, 480, 80)
>>> twin.info()
Window 'Terminal' is 640x480
Transparency is set to 80
>>> twin.change_transparency(70)
>>> twin.resize(800, 600)
Window 'Terminal' is 800x600
Transparency is set to 70

When we call twin.info, Python is running TransparentWindow's implementation of that method 1 and is not automatically delegating anything to Window even though the latter has a method with that name 2. Indeed, we have to explicitly call it through super when we want to reuse it 3. When we use resize, though, the implicit delegation kicks in and we end up with the execution of Window.resize. Please note that this delegation doesn't propagate to the next calls. When Window.resize calls self.info this runs TransparentWindow.info, as the original call was made from that class.

Composition is on the other end of the spectrum, as any delegation performed through composed objects has to be explicit. Let's see an example

class Body:
    def __init__(self, text):
        self._text = text

    def info(self):
        return {
            "length": len(self._text)
        }


class Page:
    def __init__(self, title, text):
        self._title = title
        self._body = Body(text)

    def info(self):
        return {
            "title": self._title,
            "body": self._body.info() 1
        }

When we instantiate a Page and call info everything works

>>> page = Page("New post", "Some text for an exciting new post")
>>> page.info()
{'title': 'New post', 'body': {'length': 34}}

but as you can see, Page.info has to explicitly mention Body.info through self._body 1, as we had to do when using inheritance with super. Composition is not different from inheritance when methods are overridden, at least in Python.

Relationship: to be vs to have

The third point of view from which you can look at delegation is that of the nature of the relationship between actors. Inheritance gives the child class the same nature as the parent class, with specialised behaviour. We can say that a child class implements new features or changes the behaviour of existing ones, but generally speaking, we agree that it is like the parent class. Think about a gaming laptop: it is a laptop, only with specialised features that enable it to perform well in certain situations. On the other end, composition deals with actors that are usually made of other actors of a different nature. A simple example is that of the computer itself, which has a CPU, has a mass storage, has memory. We can't say that the computer is the CPU, because that is reductive.

This difference in the nature of the relationship between actors in a delegation is directly mapped into inheritance and composition. When using inheritance, we implement the verb to be

class Car:
    def __init__(self, colour, max_speed):
        self._colour = colour
        self._speed = 0
        self._max_speed = max_speed

    def accelerate(self, speed):
        self._speed = min(speed, self._max_speed)


class SportsCar(Car):
    def accelerate(self, speed):
        self._speed = speed

Here, SportsCar is a Car, it can be initialised in the same way and has the same methods, though it can accelerate much more (wow, that might be a fun ride). Since the relationship between the two actors is best described by to be it is natural to use inheritance.

Composition, on the other hand, implements the verb to have and describes an object that is "physically" made of other objects

class Employee:
    def __init__(self, name):
        self._name = name


class Company:
    def __init__(self, ceo_name, cto_name):
        self._ceo = Employee(ceo_name)
        self._cto = Employee(cto_name)

We can say that a company is the sum of its employees (plus other things), and we easily recognise that the two classes Employee and Company have a very different nature. They don't have the same interface, and if they have methods with the same name is just by chance and not because they are serving the same purpose.

Entities: classes or instances

The last point of view that I want to explore is that of the entities involved in the delegation. When we discuss a theoretical delegation, for example saying "This Boeing 747 is a plane, thus it flies" we are describing a delegation between abstract, immaterial objects, namely generic "planes" and generic "flying objects".

class FlyingObject:
    pass


class Plane(FlyingObject):
    pass


>>> boeing747 = Plane()

Since Plane and FlyingObject share the same underlying nature, their relationship is valid for all objects of that type and it is thus established between classes, which are ideas that become concrete when instantiated.

When we use composition, instead, we are putting into play a delegation that is not valid for all objects of that type, but only for those that we connected. For example, we can separate gears from the rest of a bicycle, and it is only when we put together that specific set of gears and that bicycle that the delegation happens. So, while we can think theoretically at bicycles and gears, the actual delegation happens only when dealing with concrete objects.

class Gears:
    def __init__(self):
        self.current = 1

    def up(self):
        self.current = min(self.current + 1, 8)

    def down(self):
        self.current = max(self.current - 1, 0)


class Bicycle:
    def __init__(self):
        self.gears = Gears() 1

    def gear_up(self):
        self.gears.up() 2

    def gear_down(self):
        self.gears.down() 3

>>> bicycle = Bicycle()

As you can see here, an instance of Bicycle contains an instance of Gears 1 and this allows us to create a delegation in the methods gear_up 2 and gear_down 3. The delegation, however, happens between bicycle and bicycle.gears which are instances.

It is also possible, at least in Python, to have composition using pure classes, which is useful when the class is a pure helper or a simple container of methods (I'm not going to discuss here the benefits or the disadvantages of such a solution)

class Gears:
    @classmethod
    def up(cls, current):
        return min(current + 1, 8)

    @classmethod
    def down(cls, current):
        return max(current - 1, 0)


class Bicycle:
    def __init__(self):
        self.gears = Gears
        self.current_gear = 1

    def gear_up(self):
        self.current_gear = self.gears.up(self.current_gear)

    def gear_down(self):
        self.current_gear = self.gears.down(self.current_gear)

>>> bicycle = Bicycle()

Now, when we run bicycle.gear_up the delegation happens between bicycle, and instance, and Gears, a class. We might extend this forward to have a class which class methods call class methods of another class, but I won't give an example of this because it sounds a bit convoluted and probably not very reasonable to do. But it can be done.

So, we might devise a pattern here and say that in composition there is no rule that states the nature of the entities involved in the delegation, but that most of the time this happens between instances.

Note for advanced readers: in Python, classes are instances of a metaclass, usually type, and type is an instance of itself, so it is correct to say that composition happens always between instances.

Bad signs¶

Now that we looked at the two delegations strategies from different points of view, it's time to discuss what happens when you use the wrong one. You might have heard of the "composition over inheritance" mantra, which comes from the fact that inheritance is often overused. This wasn't and is not helped by the fact that OOP is presented as encapsulation, inheritance, and polymorphism; open a random OOP post or book and you will see this with your own eyes.

Please, bloggers, authors, mentors, teachers, and overall programmers: stop considering inheritance the only delegation system in OOP.

That said, I think we should avoid going from one extreme to the opposite, and in general learn to use the tools languages give us. So, let's learn how to recognise the "smell" of bad code!

You are incorrectly using inheritance when:

There is a clash between attributes with the same name and different meanings. In this case, you are incorrectly sharing the state of a parent class with the child one (visibility). With composition the state of another object is namespaced and it's always clear which attribute you are dealing with.
You feel the need to remove methods from the child class. This is typically a sign that you are polluting the class interface (relationship) with the content of the parent class. using composition makes it easy to expose only the methods that you want to delegate.

You are incorrectly using composition when:

You have to map too many methods from the container class to the contained one, to expose them. The two objects might benefit from the automatic delegation mechanism (control) provided by inheritance, with the child class overriding the methods that should behave differently.
You are composing instances, but creating many class methods so that the container can access them. This means that the nature of the delegation is more related to the code and the object might benefit from inheritance, where the classes delegate the method calls, instead of relying on the relationship between instances.

Overall, code smells for inheritance are the need to override or delete attributes and methods, changes in one class affecting too many other classes in the inheritance tree, big classes that contain heavily unrelated methods. For composition: too many methods that just wrap methods of the contained instances, the need to pass too many arguments to methods, classes that are too empty and that just contain one instance of another class.

Domain modelling¶

We all know that there are few cases (in computer science as well as in life) where we can draw a clear line between two options and that most of the time the separation is blurry. There are many grey shades between black and white.

The same applies to composition and inheritance. While the nature of the relationship often can guide us to the best solution, we are not always dealing with the representation of real objects, and even when we do we always have to keep in mind that we are modelling them, not implementing them perfectly.

As a colleague of mine told me once, we have to represent reality with our code, but we have to avoid representing it too faithfully, to avoid bringing reality's limitations into our programs.

I believe this is very true, so I think that when it comes to choosing between composition an inheritance we need to be guided by the nature of the relationship in our system. In this, object-oriented programming and database design are very similar. When you design a database you have to think about the domain and the way you extract information, not (only) about the real-world objects that you are modelling.

Let's consider a quick example, bearing in mind that I'm only scratching the surface of something about which people write entire books. Let's pretend we are designing a web application that manages companies and their owners, and we started with the consideration that and Owner, well, owns the Company. This is a clear composition relationship.

class Company:
    def __init__(self, name):
        self.name = name

class Owner:
    def __init__(self, first_name, last_name, company_name):
        self.first_name = first_name
        self.last_name = last_name
        self.company = Company(company_name)

>>> owner1 = Owner("John", "Doe", "Pear")

Unfortunately, this automatically limits the number of companies owned by an Owner to one. If we want to relax that requirement, the best way to do it is to reverse the composition, and make the Company contain the Owner.

class Owner:
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name

class Company:
    def __init__(self, name, owner_first_name, owner_last_name):
        self.name = name
        self.owner = Owner(owner_first_name, owner_last_name)

>>> company1 = Company("Pear", "John", "Doe")
>>> company2 = Company("Pulses", "John", "Doe")

As you can see this is in direct contrast with the initial modelling that comes from our perception of the relationship between the two in the real world, which in turn comes from the specific word "owner" that I used. If I used a different word like "president" or "CEO", you would immediately accept the second solution as more natural, as the "president" is one of many employees.

The code above is not satisfactory, though, as it initialises Owner every time we create a company, while we might want to use the same instance. Again, this is not mandatory, it depends on the data contained in the Owner objects and the level of consistency that we need. For example, if we add to the owner an attribute online to mark that they are currently using the website and can be reached on the internal chat, we don't want have to cycle between all companies and set the owner's online status for each of them if the owner is the same. So, we might want to change the way we compose them, passing an instance of Owner instead of the data used to initialise it.

class Owner:
    def __init__(self, first_name, last_name, online=False):
        self.first_name = first_name
        self.last_name = last_name
        self.online = online

class Company:
    def __init__(self, name, owner):
        self.name = name
        self.owner = owner

>>> owner1 = Owner("John", "Doe")
>>> company1 = Company("Pear", owner1)
>>> company2 = Company("Pulses", owner1)

Clearly, if the class Company has no other purpose than having a name, using a class is overkill, so this design might be further reduced to an Owner with a list of company names.

class Owner:
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name
        self.companies = []

>>> owner1 = Owner("John", "Doe")
>>> owner1.companies.extend(["Pear", "Pulses"])

Can we use inheritance? Now I am stretching the example to its limit, but I can accept there might be a use case for something like this.

class Owner:
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name

class Company(Owner):
    def __init__(self, name, owner_first_name, owner_last_name):
        self.name = name
        super().__init__(owner_first_name, owner_last_name)

>>> company1 = Company("Pear", "John", "Doe")
>>> company2 = Company("Pulses", "John", "Doe")

As I showed in the previous sections, though, this code smells as soon as we start adding something like the email address.

class Owner:
    def __init__(self, first_name, last_name, email):
        self.first_name = first_name
        self.last_name = last_name
        self.email = email

class Company(Owner):
    def __init__(self, name, owner_first_name, owner_last_name, email):
        self.name = name
        super().__init__(owner_first_name, owner_last_name, email)

>>> company1 = Company("Pear", "John", "Doe")
>>> company2 = Company("Pulses", "John", "Doe")

Is email that of the company or the personal one of its owner? There is a clash, and this is a good example of "state pollution": both attributes have the same name, but they represent different things and might need to coexist.

In conclusion, as you can see we have to be very careful to discuss relationships between objects in the context of our domain and avoid losing connection with the business logic.

Mixing the two: composed inheritance¶

Speaking of blurry separations, Python offers an interesting hook to its internal attribute resolution mechanism which allows us to create a hybrid between composition and inheritance that I call "composed inheritance".

Let's have a look at what happens internally when we deal with classes that are linked through inheritance.

class Parent:
    def __init__(self, value):
        self.value = value

    def info(self):
        print(f"Value: {self.value}")

class Child(Parent):
    def is_even(self):
        return self.value % 2 == 0

>>> c = Child(5)
>>> c.info()
Value: 5
>>> c.is_even()
False

This is a trivial example of an inheritance relationship between Child and Parent, where Parent provides the methods __init__ and info and Child augments the interface with the method is_even.

Let's have a look at the internals of the two classes. Parent.__dict__ is

mappingproxy({'__module__': '__main__',
              '__init__': <function __main__.Parent.__init__(self, value)>,
              'info': <function __main__.Parent.info(self)>,
              '__dict__': <attribute '__dict__' of 'Parent' objects>,
              '__weakref__': <attribute '__weakref__' of 'Parent' objects>,
              '__doc__': None}

and Child.__dict__ is

mappingproxy({'__module__': '__main__',
              'is_even': <function __main__.Child.is_even(self)>,
              '__doc__': None})

Finally, the bond between the two is established through Child.__bases__, which has the value (__main__.Parent,).

So, when we call c.is_even the instance has a bound method that comes from the class Child, as its __dict__ contains the function is_even. Conversely, when we call c.info Python has to fetch it from Parent, as Child can't provide it. This mechanism is implemented by the method __getattribute__ that is the core of the Python inheritance system.

As I mentioned before, however, there is a hook into this system that the language provides us, namely the method __getattr__, which is not present by default. What happens is that when a class can't provide an attribute, Python first tries to get the attribute with the standard inheritance mechanism but if it can't be found, as a last resort it tries to run __getattr__ passing the attribute name.

An example can definitely clarify the matter.

class Parent:
    def __init__(self, value):
        self.value = value

    def info(self):
        print(f"Value: {self.value}")

class Child(Parent):
    def is_even(self):
        return self.value % 2 == 0

    def __getattr__(self, attr):
        if attr == "secret":
            return "a_secret_string"

        raise AttributeError

>>> c = Child(5)

Now, if we try to access c.secret, Python would raise an AttributeError, as neither Child nor Parent can provide that attribute. As a last resort, though, Python runs c.__getattr__("secret"), and the code of that method that we implemented in the class Child returns the string "a_secret_string". Please note that the value of the argument attr is the name of the attribute as a string.

Because of the catch-all nature of __getattr__, we eventually have to raise an AttributeError to keep the inheritance mechanism working, unless we actually need or want to implement something very special.

This opens the door to an interesting hybrid solution where we can compose objects retaining an automatic delegation mechanism.

class Parent:
    def __init__(self, value):
        self.value = value

    def info(self):
        print(f"Value: {self.value}")

class Child:
    def __init__(self, value):
        self.parent = Parent(value)

    def is_even(self):
        return self.value % 2 == 0

    def __getattr__(self, attr):
        return getattr(self.parent, attr)

>>> c = Child(5)
>>> c.value
5
>>> c.info()
Value: 5
>>> c.is_even()
False

As you can see, here Child is composing Parent and there is no inheritance between the two. We can nevertheless access c.value and call c.info, thanks to the face that Child.__getattr__ is delegating everything can't be found in Child to the instance of Parent stored in self.parent.

Note: don't confuse getattr with __getattr__. The former is a builtin function that gets an attribute provided its name, a replacement for the dotted notation when the name of the attribute is known as a string. The latter is the hook into the inheritance mechanism that I described in this section.

Now, this is very powerful, but is it also useful?

I think this is not one of the techniques that will drastically change the way you write code in Python, but it can definitely help you to use composition instead of inheritance even when the amount of methods that you have to wrap is high. One of the limits of composition is that you are at the extreme spectrum of automatism; while inheritance is completely automatic, composition doesn't do anything for you. This means that when you compose objects you need to decide which methods or attributes of the contained objects you want to wrap, in order to expose then in the container object. In the previous example, the class Child might want to expose the attribute value and the method info, which would result in something like

class Parent:
    def __init__(self, value):
        self.value = value

    def info(self):
        print(f"Value: {self.value}")

class Child:
    def __init__(self, value):
        self.parent = Parent(value)

    def is_even(self):
        return self.value % 2 == 0

    def info(self):
        return self.parent.info()

    @property
    def value(self):
        return self.parent.value

As you can easily see, the more Child wants to expose of the Parent interface, the more wrapper methods and properties you need. To be perfectly clear, in this example the code above smells, as there are too many one-liner wrappers, which tells me it would be better to use inheritance. But if the class Child had a dozen of its own methods, suddenly it would make sense to do something like this, and in that case, __getattr__ might come in handy.

Final words¶

Both composition and inheritance are tools, and both exist to serve the bigger purpose of code reuse, so learn their strength and their weaknesses, so that you might be able to use the correct one and avoid future issues in your code.

I hope this rather long discussion helped you to get a better picture of the options you have when you design an object-oriented system, and also maybe introduced some new ideas or points of view if you are already comfortable with the concepts I wrote about.

Updates¶

2021-03-06 Following the suggestion of Tim Morris I added the console output to the source code to make the code easier to understand. Thanks Tim for the feedback!

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

October 03, 2022 08:00 AM UTC

Podcast.init

Complete Your Hardware "Weekend Projects" In An Actual Weekend With Belay

Working on hardware projects often has significant friction involved when compared to pure software. Brian Pugh enjoys tinkering with microcontrollers, but his "weekend projects" often took longer than a weekend to complete, so he created Belay. In this episode he explains how Belay simplifies the interactions involved in developing for MicroPython boards and how you can use it to speed up your own experimentation.

Summary

Announcements

Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great!
When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. Select Star’s data discovery platform solves that out of the box, with a fully automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Just connect it to your dbt, Snowflake, Tableau, Looker, or whatever you’re using and Select Star will set everything up in just a few hours. Go to pythonpodcast.com/selectstar today to double the length of your free trial and get a swag package when you convert to a paid plan.
Your host as usual is Tobias Macey and today I’m interviewing Brian Pugh about Belay, a python library that enables the rapid development of projects that interact with hardware via a micropython-compatible board.

Interview

Introductions
How did you get introduced to Python?
Can you describe what Belay is and the story behind it?
Who are the target users for Belay?
What are some of the points of friction involved in developing for hardware projects?
- What are some of the features of Belay that make that a smoother process?
What are some of the ways that simplifying the develop/debug cycles can improve the overall experience of developing for hardware platforms?
- What are some of the inherent limitations of constrained hardware that Belay is unable to paper over?
Can you describe how Belay is implemented?
What does the workflow look like when using Belay as compared to using MicroPython directly?
What are some of the ways that you are using Belay in your own projects?
What are the most interesting, innovative, or unexpected ways that you have seen Belay used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Belay?
When is Belay the wrong choice?
What do you have planned for the future of Belay?

Keep In Touch

BrianPugh on GitHub
LinkedIn

Picks

Tobias
- Gunnar Computer Glasses

Closing Announcements

Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

October 03, 2022 02:15 AM UTC

October 02, 2022

IslandT

Python Tutorial — Chapter 15

In this chapter let us create a function based on another function with the python lambda function!

Let’s say you want to create a seed value that will be used to find the modulo with another value then you will create the function below with the help of the lambda anonymous function.

def modfunc(n):
  return lambda a : n % a

Now you can call the above function and use the lambda anonymous function to perform the modulo operation.

myfunction = modfunc(12)
print(myfunction(5))

Basically the above is equal to 12 % 5 which is equal to 2.

This is the last tutorial of the Python beginner series, I might continue to write more python tutorials in the future that will deal with more complex python modules such as the Math and the RegEx module but for the python basic part, I think I have covered almost everything that you will need to start writing your own python program! Don’t forget to share the entire tutorial series with your friend to show your appreciation of my hard work.

October 02, 2022 01:37 PM UTC

Python Tutorial — Chapter 14

In this tutorial let us create a python class and then create the child class of that main class.

Create a Python class that will multiply two numbers.

class Multiply:
    def __init__(self, parameter1, parameter2):
        self.parameter1 = parameter1
        self.parameter2 = parameter2

    def multiplying(self):
        return self.parameter1 * self.parameter2

As you can see from above the Multiply class has been created with the keyword class and the method of multiplying two numbers which is called multiplying has also been created.

Now you can create an object from the above class and call its method as follows:-

mul = Multiply(10,2)
print(mul.multiplying()) #20

When the object has been created for the first time, it will call the init method and put the two parameters into the init method as parameter1 and parameter2 accordingly. The self keyword inside the init or other methods will be used to refer to that object itself so you can access the elements of that object in the future, for example in the multiplying method above.

Next let us create the child class of that main class.

First of all, let us modify the above class a little bit.

class Multiply:
    def __init__(self, parameter1, parameter2):
        self.parameter1 = parameter1
        self.parameter2 = parameter2

    def multiplying(self):
        self.parameter = self.parameter1 * self.parameter2

As you can see the multiplying function no longer returns the result but instead assigns the outcome to the parameter variable which will be used later to multiply with the third parameter of the child object!

Next, let us create the child object.

class Multiply2(Multiply):
    def __init__(self, parameter1, parameter2, parameter3):
        super().__init__(parameter1, parameter2)
        self.parameter3 = parameter3

    def result(self):
        self.multiplying() # multiplying the first two parameters
        return self.parameter * self.parameter3

Now the child class will inherit all the methods and properties of the parent class. Notice the super() method is used to call the init method of the parent class and then passed the first two parameters into that parent class.

At last, by using the self keyword under the result method you can access the parameter variable from the parent class which is actually the property of the child class itself.

Now you can create the object from the child class and get the result of three parameters that will multiply together.

mul = Multiply2(10,2, 10)
print(mul.result()) # 200

That is it, and now let us all get to the last chapter of the Python tutorial.

October 02, 2022 01:06 PM UTC

Planet Python

October 04, 2022

Tips for Open Source contribution

1. Find a project by searching open repositories, chatting with Open Source contributors or checking out third party sites.

2. When looking for a project, look for repositories with:

3. Find a helpful list to guide you to the right project.

4. Figuring out how to get un-stuck can be tough. Don’t be afraid to ask for help respectfully.

Our favorite projects accepting contributions

We love to hear from you!

Discussions

Python Jobs

Articles & Tutorials

Projects & Code

Events

CVE-2022-41323: Potential denial-of-service vulnerability in internationalized URLs

Affected supported versions

Resolution

General notes regarding security reporting

October 03, 2022

Connecting to FHIR servers

Retrieving data from FHIR servers

Storing data in FHIR servers

Learning what FHIR resources to use

FHIR security

MLLP, HL7 v2 and v3

Next steps

What does that error message mean?

How do we fix this?

Getting to Know Enumerations in Python

Creating Enumerations With Python’s Enum

Topics we’ll cover:

When to Write Python Comments?

How to Write a Comment in Python?

Writing Single Line Comments in Python

Example 1: Python block comments

Example 2: The Pythagorean Theorem

Inline Comments

Example 3: Writing Inline Comments in Python

Writing Multiline Comments in Python

Example 4: Multiline Comments

Example 5: Using a docstring with a Function

Making Comments For Yourself

Commenting for Others

Best Practices for Commenting in Python

Example 7: Commenting the bubble sort algorithm

Helpful Links

Introduction¶

Delegation in OOP¶

Visibility: state sharing

Control: implicit and explicit delegation

Relationship: to be vs to have

Entities: classes or instances

Bad signs¶

Domain modelling¶

Mixing the two: composed inheritance¶

Final words¶

Updates¶

Feedback¶

Summary

Announcements

Interview

Keep In Touch

Picks

Closing Announcements

Links

October 02, 2022

October 01, 2022

Creating Enumerations With Python’s `Enum`