Planet Python

Last update: January 27, 2022 04:41 AM UTC

January 27, 2022

David Amos

Why Can't You Reverse A String With a Flag Emoji?

What do you think is the output of the following Python code?

>>> flag = "🇺🇸"
>>> reversed_flag = flag[::-1]
>>> print(reversed_flag)

Questions like this make me want to immediately open a Python REPL and try the code out because I think I know what the answer is, but I&aposm not very confident in that answer.

Here&aposs my line of thinking when I first saw this question:

• The flag string contains a single character.
• The [::-1] slice reverses the flag string.
• The reversal of a string with a single character is the same as the original string.
• Therefore, reversed_flag must be "🇺🇸".

That&aposs a perfectly valid argument. But is the conclusion true? Take a look:

>>> flag = "🇺🇸"
>>> reversed_flag = flag[::-1]
>>> print(reversed_flag)
🇸🇺

What in the world is going on here?

Does "🇺🇸" Really Contain a Single Character?

When the conclusion of a valid argument is false, one of its premises must be false, too. Let&aposs start from the top:

The flag string contains a single character.

Is that so? How can you tell how many characters a string has?

In Python, you can use the built-in len() function to get the total number of characters in a string:

>>> len("🇺🇸")
2

Oh.

That&aposs weird. You can only see a single thing in the string "🇺🇸" — namely the US flag — but a length of 2 jives with the result of flag[::-1]. Since the reverse of "🇺🇸" is "🇸🇺", this seems to imply that somehow "🇺🇸" == "🇺 🇸".

How Can You Tell What Characters Are In a String?

There are a few different ways that you can see all of the true characters in a string using Python:

>>> # Convert a string to a list
>>> list("🇺🇸")
[&apos🇺&apos, &apos🇸&apos]

>>> # Loop over each character and print
>>> for character in "🇺🇸":
...     print(character)
...
🇺
🇸

The US flag emoji isn’t the only flag emoji with two characters:

>>> list("🇿🇼")  # Zimbabwe
[&apos🇿&apos, &apos🇼&apos]

>>> list("🇳🇴")  # Norway
[&apos🇳&apos, &apos🇴&apos]

>>> list("🇨🇺")  # Cuba
[&apos🇨&apos, &apos🇺&apos]

>>> # What do you notice?

And then there’s the Scottish flag:

>>> list("🏴󠁧󠁢󠁳󠁣󠁴󠁿")
[&apos🏴&apos, &apos\U000e0067&apos, &apos\U000e0062&apos, &apos\U000e0073&apos, &apos\U000e0063&apos,
 &apos\U000e0074&apos, &apos\U000e007f&apos]

OK, what is that all about?

The unnerving thing about these examples is that they imply that you can&apost tell what characters are in a string just by looking at your screen.

Or, perhaps more deeply, it makes you question your understanding of the term character.

What Is a Character, Anyway?

The term character in computer science can be confusing. It tends to get conflated with the word symbol, which, to be fair, is a synonym for the word character as it&aposs used in English vernacular.

In fact, when I googled character computer science, the very first result I got was a link to a Technopedia article that defines a character as:

“[A] display unit of information equivalent to one alphabetic letter or symbol."

— Technopedia, “Character (Char)”

That definition seems off, especially in light of the US flag example that indicates that a single symbol may be comprised of at least two characters.

The second Google result I get is Wikipedia. In that article, the definition of a character is a bit more liberal:

”[A] character is a unit of information that roughly corresponds to a grapheme, grapheme-like unit, or symbol, such as in an alphabet or syllabary in the written form of a natural language.

— Wikipedia, "Character (computing)”

Hmm... using the word "roughly" in a definition makes the definition feel, shall I say, non-definitive.

But the Wikipedia article goes on to explain that the term character has been used historically to "denote a specific number of contiguous bits.”

Then, a significant clue to the question about how a string with one symbol can contain two or more characters:

“A character is most commonly assumed to refer to 8 bits (one byte) today... All [symbols] can be represented with one or more 8-bit code units with UTF-8.”

— Wikipedia, "Character (computing)”

OK! Maybe things are starting to make a little bit more sense. A character is one byte of information representing a unit of text. The symbols that we see in a string can be made up of multiple 8-bit (1 byte) UTF-8 code units.

Characters are not the same as symbols. It seems reasonable now that one symbol could be made up of multiple characters, just like flag emojis.

But what is a UTF-8 code unit?

A little further down the Wikipedia article on characters, there’s a section called Encoding that explains:

“Computers and communication equipment represent characters using a character encoding that assigns each character to something – an integer quantity represented by a sequence of digits, typically – that can be stored or transmitted through a network. Two examples of usual encodings are ASCII and the UTF-8 encoding for Unicode.”

— Wikipedia, "Character (computing)”

There’s another mention of UTF-8! But now I need to know what a character encoding is.

What Exactly Is a Character Encoding?

According to Wikipedia, a character encoding assigns each character to a number. What does that mean?

Doesn’t it mean that you can pair each character with a number? So, you could do something like pair each uppercase letter in the English alphabet with an integer 0 through 25.

You can represent this pairing using tuples in Python:

>>> pairs = [(0, "A"), (1, "B"), (2, "C"), ..., (25, "Z")]
>>> # I&aposm omitting several pairs here -----^^^

Stop for a moment and ask yourself: “Can I create a list of tuples like the one above without explicitly writing out each pair?"

One way is to use Python’s enumerate() function. enumerate() takes an argument called iterable and returns a tuple containing a count that defaults to 0 and the values obtained from iterating over iterable.

Here’s a look at enumerate() in action:

>>> letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
>>> enumerated_letters = list(enumerate(letters))
>>> enumerated_letters
[(0, &aposA&apos), (1, &aposB&apos), (2, &aposC&apos), (3, &aposD&apos), (4, &aposE&apos), (5, &aposF&apos), (6, &aposG&apos),
(7, &aposH&apos), (8, &aposI&apos), (9, &aposJ&apos), (10, &aposK&apos), (11, &aposL&apos), (12, &aposM&apos), (13, &aposN&apos),
(14, &aposO&apos), (15, &aposP&apos), (16, &aposQ&apos), (17, &aposR&apos), (18, &aposS&apos), (19, &aposT&apos), (20, &aposU&apos),
(21, &aposV&apos), (22, &aposW&apos), (23, &aposX&apos), (24, &aposY&apos), (25, &aposZ&apos)]

There’s an easier way to make all of the letters, too.

Python’s string module has a variable called ascii_uppercase that points to a string containing all of the uppercase letters in the English alphabet:

>>> import string
>>> string.ascii_uppercase
&aposABCDEFGHIJKLMNOPQRSTUVWXYZ&apos

>>> enumerated_letters = list(enumerate(string.ascii_uppercase))
>>> enumerated_letters
[(0, &aposA&apos), (1, &aposB&apos), (2, &aposC&apos), (3, &aposD&apos), (4, &aposE&apos), (5, &aposF&apos), (6, &aposG&apos),
 (7, &aposH&apos), (8, &aposI&apos), (9, &aposJ&apos), (10, &aposK&apos), (11, &aposL&apos), (12, &aposM&apos), (13, &aposN&apos),
 (14, &aposO&apos), (15, &aposP&apos), (16, &aposQ&apos), (17, &aposR&apos), (18, &aposS&apos), (19, &aposT&apos),
 (20, &aposU&apos), (21, &aposV&apos), (22, &aposW&apos), (23, &aposX&apos), (24, &aposY&apos), (25, &aposZ&apos)]

OK, so we’ve associated characters to integers. That means we’ve got a character encoding!

But, how do you use it?

To encode the string ”PYTHON” as a sequence of integers, you need a way to look up the integer associated with each character. But, looking things up in a list of tuples is hard. It’s also really inefficient. (Why?)

Dictionaries are good for looking things up. If we convert enumerated_letters to a dictionary, we can quickly look up the letter associated with an integer:

>>> int_to_char = dict(enumerated_letters)

>>> # Get the character paired with 1
>>> int_to_char[1]
&aposB&apos

>>> # Get the character paired with 15
>>> int_to_char[15]
&aposP&apos

However, to encode the string ”PYTHON” you need to be able to look up the integer associated with a character. You need the reverse of int_to_char.

How do you swap keys and values in a Python dictionary?

One way is use the reversed() function to reverse key-value pairs from the int_to_char dictionary:

>>> # int_to_char.items() is a "list" of key-value pairs
>>> int_to_char.items()
dict_items([(0, &aposA&apos), (1, &aposB&apos), (2, &aposC&apos), (3, &aposD&apos), (4, &aposE&apos), (5, &aposF&apos),
(6, &aposG&apos), (7, &aposH&apos), (8, &aposI&apos), (9, &aposJ&apos), (10, &aposK&apos), (11, &aposL&apos), (12, &aposM&apos),
(13, &aposN&apos), (14, &aposO&apos), (15, &aposP&apos), (16, &aposQ&apos), (17, &aposR&apos), (18, &aposS&apos),
(19, &aposT&apos), (20, &aposU&apos), (21, &aposV&apos), (22, &aposW&apos), (23, &aposX&apos), (24, &aposY&apos),
(25, &aposZ&apos)])

>>> # The reversed() function can reverse a tuple
>>> pair = (0, "A")
>>> tuple(reversed(pair))
(&aposA&apos, 0)

You can write a generator expression that reverses all of the pairs in int_to_char.items() and use that generator expression to populate a dictionary:

>>> char_to_int = dict(reversed(pair) for pair in int_to_char.items())
>>> # Reverse the pair-^^^^^^^^^^^^^^
>>> # For every key-value pair--------^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

>>> # Get the integer associated with B
>>> char_to_int["B"]
1

>>> # Get the integer associated with P
>>> char_to_int["P"]
15

It’s good that you paired each letter with a unique integer. Otherwise, this dictionary reversal wouldn’t have worked. (Why?)

Now you can encode strings as list of integers using the char_to_int dictionary and a list comprehension:

>>> [char_to_int[char] for char in "PYTHON"]
[15, 24, 19, 7, 14, 13]

And you can convert a list of integers into a string of uppercase characters using int_to_char in a generator expression with Python&aposs string .join() method:

>>> "".join(int_to_char[num] for num in [7, 4, 11, 11, 14])
&aposHELLO&apos

But, there’s a problem.

Your encoding can’t handle strings with things like punctuation, lowercase letters, and whitespace:

>>> [char_to_int[char] for char in "monty python!"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
KeyError: &aposm&apos
>>> # ^^^^^^^-----char_to_int has no "m" key

One way to fix this is to create an encoding using a string containing all of the lowercase letters, punctuation marks, and whitespace characters that you need.

But, in Python, there’s almost always a better way. Python’s string module contains a variable called printable that gives you a string containing a whole bunch of printable characters:

>>> string.printable
&apos0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\&apos()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c&apos

Would you have included all of those characters if you were making your own string from scratch?

Now you can make new dictionaries for encoding and decoding the characters in string.printable:

>>> int_to_printable = dict(enumerate(string.printable))
>>> printable_to_int = dict(reversed(item) for item in int_to_printable.items())

You can use these dictionaries to encode and decode more complicated strings:

>>> # Encode the string "monty python!"
>>> encoded_string = [printable_to_int[char] for char in "monty python!"]
>>> encoded_string
[22, 24, 23, 29, 34, 94, 25, 34, 29, 17, 24, 23, 62]

>>> # Decode the encoded string
>>> decoded_string = "".join(int_to_printable[num] for num in encoded_string)
>>> decoded_string
&aposmonty python!&apos

You’ve now made two different character encodings! And they really are different. Just look at what happens when you decode the same list of integers using both encodings:

>>> encoded_string = [15, 24, 19, 7, 14, 13]

>>> # Decode using int_to_char (string.ascii_uppercase)
>>> "".join(int_to_char[num] for num in encoded_string)
&aposPYTHON&apos

>>> # Decode using int_to_printable(string.printable)
>>> "".join(int_to_printable[num] for num in encoded_string)
&aposfoj7ed&apos

Not even close!

So, now we know a few things about character encodings:

• A character encoding pairs characters with unique integers.
• Some character encodings exclude characters that are included in other character encodings.
• Two different character encodings may decode the same integers into two different strings.

What does any of this have to do with UTF-8?

What Is UTF-8?

Wikipedia&aposs article on characters mentions two different character encodings:

“Two examples of usual encodings are ASCII and the UTF-8 encoding for Unicode.”

— Wikipedia, "Character (computing)”

OK, so ASCII and UTF-8 are specific kinds of character encoding.

According to the Wikipedia article on ASCII:

ASCII was the most common character encoding on the World Wide Web until December 2007, when UTF-8 encoding surpassed it; UTF-8 is backward compatible with ASCII.

— Wikipedia, "ASCII”

UTF-8 isn’t just the dominant character encoding for the web. It’s also the primary character encoding for Linux and macOS operating systems and is even the default for Python code.

In fact, you can see how UTF-8 encodes characters as integers using the .encode() method on Python string objects. But .encode() doesn&apost return a list of integers. Instead, encode() returns a bytes object:

>>> encoded_string = "PYTHON".encode()

>>> # The encoded string *looks* like a string still,
>>> # but notice the b in front of the first quote
>>> encoded_string
b&aposPYTHON&apos

>>> # b stands for bytes, which is the type of
>>> # object returned by .encode()
>>> type(encoded_string)
<class &aposbytes&apos>

The Python docs describe a bytes object as “an immutable sequence of integers in the range 0 <= x < 256.” That seems a little weird considering that the encoded_string object displays the characters in the string “PYTHON” and not a bunch of integers.

But let’s accept this and see if we can tease out the integers somehow.

The Python docs say that bytes is a "sequence," and Python&aposs glossary defines a sequence as “[a]n iterable which supports efficient element access using integer indices.”

So, it sounds like you can index a bytes object the same way that you can index a Python list object. Let&aposs try it out:

>>> encoded_string[0]
80

Aha!

What happens when you convert encoded_string to a list?

>>> list(encoded_string)
[80, 89, 84, 72, 79, 78]

Bingo. It looks like UTF-8 assigns the letter ”P”  to the integer 80, ”Y” to the integer 89, ”T” to the integer 84, and so on.

Let’s see what happens when we encode the string ”🇺🇸” using UTF-8:

>>> list("🇺🇸".encode())
[240, 159, 135, 186, 240, 159, 135, 184]

Huh. Did you expect ”🇺🇸” to get encoded as eight integers?

”🇺🇸” is made up of two characters, namely “🇺”  and ”🇸". Let&aposs see how those get encoded:

>>> list("🇺".encode())
[240, 159, 135, 186]

>>> list("🇸".encode())
[240, 159, 135, 184]

OK, things are making more sense now. Both “🇺” and ”🇸" get encoded as four integers, and the four integers corresponding to “🇺” appear first in the list of integers corresponding to ”🇺🇸”, while the four integers corresponding to ”🇸" appear second.

This raises a question, though.

Why Does UTF-8 Encode Some Characters As Four Integers and Others as One Integer?

The character “🇺” is encoded as a sequence of four integers in UTF-8, while the character ”P” gets encoded as a single integer. Why is that?

There’s a hint at the top of Wikipedia’s UTF-8 article:

UTF-8 is capable of encoding all 1,112,064 valid character code points in Unicode using one to four one-byte (8-bit) code units. Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes.

— Wikipedia, “UTF-8”

OK, so that makes it sound like UTF-8 isn’t encoding characters to integers, but instead to something called a Unicode code point. And each code unit can apparently be one to four bytes.

There are a couple of questions we need to answer now:

1. What is a byte?
2. What is a Unicode code point?

The word byte has been floating around a lot, so let’s go ahead and give it a proper definition.

A bit is the smallest unit of information. A bit has two states, on or off, that are usually represented by the integers 0 and 1, respectively. A byte is a sequence of eight bits.

You can interpret bytes as integers by viewing their component bits as expressing a number in binary notation.

Binary notation can look pretty exotic the first time you see it. It&aposs a lot like the usual decimal representation you use to write numbers, though. The difference is that each digit can only be a 0 or a 1, and the value of each place in the number is a power of 2, not a power of 10:

Since a byte contains eight bits, the largest number you can represent with a single byte is 11111111 in binary or 255 in decimal notation.

A character encoding that uses one byte for each character can encode a maximum of 255 characters since the maximum 8-bit integer is 255.

255 characters might be enough to encode everything in the English language. Still, there’s no way that it can handle all of the characters and symbols used in written and electronic communication worldwide.

So what do you do? Allowing characters to be encoded as multiple bytes seems like a reasonable solution, and that’s exactly what UTF-8 does.

UTF-8 is an acronym for Unicode Transformation Format — 8-bit. There&aposs that word Unicode again.

According to the Unicode website:

“Unicode provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language.”

— Unicode website, “What is Unicode?”

Unicode is massive. The goal of Unicode is to provide a universal representation for all written language. Every character gets assigned to a code point — a fancy word for “integer" with some additional organization — and there are a total of 1,112,064 possible code points.

How Unicode code points actually get encoded depends, though. UTF-8 is just one character encoding implementing the Unicode standard. It divides code points into groups of one to four 8-bit integers.

There are other encodings for Unicode. UTF-16 divides Unicode code points into one or two 16-bit numbers and is the default encoding used by Microsoft Windows. UTF-32 can encode every Unicode code point as a single 21-bit integer.

But wait, UTF-8 encodes symbols as code points using one to four bytes. OK, so… why does the 🇺🇸 symbol get encoded with eight bytes?

>>> list("🇺🇸".encode())
[240, 159, 135, 186, 240, 159, 135, 184]

>>> # There are eight integers in the list, a total of eight bytes!

Remember, two characters make up the US flag emoji: 🇺 and 🇸. These characters are called regional indicator symbols. There are twenty-six regional indicators in the Unicode standard representing A–Z English letters. They’re used to encode ISO 3166-1 two-letter country codes.

Here’s what Wikipedia has to say about regional indicator symbols:

These were defined in October 2010 as part of the Unicode 6.0 support for emoji, as an alternative to encoding separate characters for each country flag. Although they can be displayed as Roman letters, it is intended that implementations may choose to display them in other ways, such as by using national flags. The Unicode FAQ indicates that this mechanism should be used and that symbols for national flags will not be directly encoded.

— Wikipedia, “Regional indicator symbol”

In other words, the 🇺🇸 symbol — indeed, the symbol for any country&aposs flag — is not directly supported by Unicode. Operating systems, web browsers, and other places where digital text is used, may choose to render pairs of regional indicators as flags.

Let’s take stock of what we know so far:

• Strings of symbols get converted to sequences of integers by a character encoding, usually UTF-8.
• Some characters are encoded as a single 8-bit integer by UTF-8, and others require two, three, or four 8-bit integers.
• Some symbols, such as flag emojis, are not directly encoded by Unicode. Instead, they are renders of sequences of Unicode characters and may or may not be supported by every platform.

So, when you reverse a string, what gets reversed? Do you reverse the entire sequence of integers in the encoding, or do you reverse the order of the code points, or something different?

How Do You Actually Reverse A String?

Can you think of a way to answer this question with a code experiment rather than trying to look up the answer?

You saw earlier that UTF-8 encodes the string ”PYTHON” as a sequence of six integers:

>>> list("PYTHON".encode())
[80, 89, 84, 72, 79, 78]

What happens if you encode the reversal of the string ”PYTHON”?

>>> list("PYTHON"[::-1].encode())
[78, 79, 72, 84, 89, 80]

In this case, the order of the integers in the list was reversed. But what about other symbols?

Earlier, you saw that the “🇺" symbol is encoded as a sequence of four integers. What happens when you encode its reversal?

>>> list("🇺".encode())
[240, 159, 135, 186]

>>> list("🇺"[::-1].encode())
[240, 159, 135, 186]

Huh. The order of the integers in both lists is the same!

Let’s try reversing the string with the US flag:

>>> list("🇺🇸".encode())
[240, 159, 135, 186, 240, 159, 135, 184]
>>> # ^^^^^^^^^^^^^^---Code point for 🇺
>>> #                ^^^^^^^^^^^^^^^^^^---Code point for 🇸

>>> # The code points get swapped!
>>> list("🇺🇸"[::-1].encode())
[240, 159, 135, 184, 240, 159, 135, 186]
>>> # ^^^^^^^^^^^^^^---Code point for 🇸
>>> #                ^^^^^^^^^^^^^^^^^^---Code point for 🇺

The order of the integers isn’t reversed! Instead, the groups of four integers representing the Unicode code points for 🇺and 🇸get swapped. The orders of the integers in each code point stay the same.

What Does All Of This Mean?

The title of this article is a lie! You can reverse a string with a flag emoji. But the reversing symbols composed of multiple code points can have surprising results. Especially if you&aposve never heard of things like character encodings and code points before.

Why Is Any Of This Important?

There are a couple of important lessons to take away from this investigation.

First, if you don&apost know which character encoding was used to encode some text, you can&apost guarantee that the decoded text accurately represents the original text.

Second, although UTF-8 is widely adopted, there are still many systems that use different character encodings. Keep this in mind when reading text from a file, especially when shared from a different operating system or across international borders. Be explicit and always indicate which encoding is being used to encode or decode text.

For example, Python’s open() function has an encoding parameter that specifies the character encoding to use when reading or writing text to a file. Make use of it.

Where Do You Go From Here?

We’ve covered a lot of ground, but there are still a lot of questions left unanswered. So write down some of the questions you still have and use the investigative techniques you saw in this article to try and answer them.

Here are some questions you might want to explore:

• When you convert ”🏴󠁧󠁢󠁳󠁣󠁴󠁿” to a list, you end up with a bunch of strings that start with ”\U”. What are those strings, and what do they represent?
• The UTF-8 encoding for ”🏴󠁧󠁢󠁳󠁣󠁴󠁿” contains a whopping 28 bytes of information. What makes 🏴󠁧󠁢󠁳󠁣󠁴󠁿 different from 🇺🇸? What other flags get encoded as 28 bytes?
• Are there any flag emojis that get encoded as a single code point?
• Many platforms support colored emojis, such as a thumbs-up emoji that can be rendered with different skin tones. How does the same symbol with different colors get encoded?
• How can you check if a string containing emojis is a palindrome?

Thanks for reading! Stay curious out there!

Join my free weekly newsletter "Curious About Code" to get curiosity-inducing content in your inbox every Friday.

January 27, 2022 02:23 AM UTC

January 26, 2022

Real Python

Raining Outside? Build a Weather CLI App With Python

Read the full article at https://realpython.com/build-a-python-weather-app-cli/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

January 26, 2022 02:00 PM UTC

Python for Beginners

Delete All Elements Of A List In Python

In python, we use lists in almost every program. We have already discussed ways to compare two lists and to reverse a list in python. In this article, we will discuss different ways to delete all the elements of a list in python.

Delete All Elements Of A List In Python Using The clear() Method

The pop() method is used to delete the last element of a list. When invoked on a list, the pop() method returns the last element of the list and deletes it from the list. We can use the while loop and the pop() method to remove all the elements of the list. 

For this, we can keep invoking the pop() method on the list inside the while loop until the list becomes empty.  Once the list becomes empty, the while loop will stop its execution and we will get a list that has no elements. You can observe this in the following program.

myList = [1, 2, 3, 4, 5, 6, 7, 8, 9]
print("The original list is:")
print(myList)
# deleting elements using the pop() method
while myList:
    myList.pop()
print("List after deleting all the elements:",myList)

Output:

The original list is:
[1, 2, 3, 4, 5, 6, 7, 8, 9]
List after deleting all the elements: []

Instead of using the pop() method several times, we can use the clear() method to delete all the elements from a list. The clear() method, when invoked on a list, deletes all the elements of the list. 

myList = [1, 2, 3, 4, 5, 6, 7, 8, 9]
print("The original list is:")
print(myList)
# deleting elements using the clear() method
myList.clear()
print("List after deleting all the elements:",myList)

Output:

The original list is:
[1, 2, 3, 4, 5, 6, 7, 8, 9]
List after deleting all the elements: []

Delete All Elements Of A List In Python Using The del Statement

The del statement is used to delete an object in python. We can also use the del statement to remove the elements of a list. For this, we will create a slice of the list that contains all the elements. After that, we will delete the slice. As the slice contains references to the elements in the original list, all the elements in the original list will be deleted and we will get an empty list. You can do it as follows.

myList = [1, 2, 3, 4, 5, 6, 7, 8, 9]
print("The original list is:")
print(myList)
# deleting elements using the del method
del myList[:]
print("List after deleting all the elements:", myList)

Output:

The original list is:
[1, 2, 3, 4, 5, 6, 7, 8, 9]
List after deleting all the elements: []

Delete All Elements Of A List In Python Using The * Operator

This is one of the least used ways to remove elements from a list in Python. You might be knowing that multiplying a list to any number N repeats the elements of the list N times. In the same way, when we multiply a list to 0, all the elements of the list are deleted. So, we can multiply the given list with 0 to remove all of its elements as follows.

myList = [1, 2, 3, 4, 5, 6, 7, 8, 9]
print("The original list is:")
print(myList)
# deleting elements using *
myList = myList * 0
print("List after deleting all the elements:", myList)

Output:

The original list is:
[1, 2, 3, 4, 5, 6, 7, 8, 9]
List after deleting all the elements: []

Conclusion

In this article, we have discussed four different ways to delete all the elements of a list in python. To know more about lists in python, you can read this article on list comprehension. You might also like this article on how to get the last element of a list in python.

The post Delete All Elements Of A List In Python appeared first on PythonForBeginners.com.

January 26, 2022 01:18 PM UTC

death and gravity

Dealing with YAML with arbitrary tags in Python

... in which we use PyYAML to safely read and write YAML with any tags, in a way that's as straightforward as interacting with built-in types.

If you're in a hurry, you can find the code at the end.

• Why is this useful?
• A note on PyYAML extensibility
• Preserving tags
  • Constructing unknown objects
  • A better wrapper
  • Representing tagged objects
• Unhashable keys
  • Constructing pairs
  • Representing pairs
• Conclusion
• Bonus: hashable wrapper
• Bonus: broken YAML

Why is this useful? #

People mostly use YAML as a friendlier alternative to JSON¹, but it can do way more.

Among others, it can represent user-defined and native data structures.

Say you need to read (or write) an AWS Cloud Formation template:

EC2Instance:
  Type: AWS::EC2::Instance
  Properties:
    ImageId: !FindInMap [
      AWSRegionArch2AMI,
      !Ref 'AWS::Region',
      !FindInMap [AWSInstanceType2Arch, !Ref InstanceType, Arch],
    ]
    InstanceType: !Ref InstanceType

>>> yaml.safe_load(text)
Traceback (most recent call last):
  ...
yaml.constructor.ConstructorError: could not determine a constructor for the tag '!FindInMap'
  in "<unicode string>", line 4, column 14:
        ImageId: !FindInMap [
                 ^

... or, you need to safely read untrusted YAML that represents Python objects:

!!python/object/new:module.Class { attribute: value }

>>> yaml.safe_load(text)
Traceback (most recent call last):
 ...
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/new:module.Class'
  in "<unicode string>", line 1, column 1:
    !!python/object/new:module.Class ...
    ^

Warning

Historically, yaml.load(thing) was unsafe for untrusted data, because it allowed running arbitrary code. Consider using use safe_load() instead.

For example, you could do this:

>>> yaml.load("!!python/object/new:os.system [echo WOOSH. YOU HAVE been compromised]")
WOOSH. YOU HAVE been compromised
0

There were a bunch of CVEs about it.

To address the issue, load() requires an explicit Loader since PyYAML 6. Also, version 5 added two new functions and corresponding loaders:

• full_load() resolves all tags except those known to be unsafe (note that this was broken before 5.4, and thus vulnerable)
• unsafe_load() resolves all tags, even those known to be unsafe (the old load() behavior)

safe_load() resolves only basic tags, remaining the safest.

Can't I just get the data, without it being turned into objects?

You can! The YAML spec says:

In a given processing environment, there need not be an available native type corresponding to a given tag. If a node’s tag is unavailable, a YAML processor will not be able to construct a native data structure for it. In this case, a complete representation may still be composed and an application may wish to use this representation directly.

And PyYAML obliges:

>>> text = """\
... one: !myscalar string
... two: !mysequence [1, 2]
... """
>>> yaml.compose(text)
MappingNode(
    tag='tag:yaml.org,2002:map',
    value=[
        (
            ScalarNode(tag='tag:yaml.org,2002:str', value='one'),
            ScalarNode(tag='!myscalar', value='string'),
        ),
        (
            ScalarNode(tag='tag:yaml.org,2002:str', value='two'),
            SequenceNode(
                tag='!mysequence',
                value=[
                    ScalarNode(tag='tag:yaml.org,2002:int', value='1'),
                    ScalarNode(tag='tag:yaml.org,2002:int', value='2'),
                ],
            ),
        ),
    ],
)
>>> print(yaml.serialize(_))
one: !myscalar 'string'
two: !mysequence [1, 2]

... the spec didn't say the representation has to be concise. ¯\_(ツ)_/¯

Here's how YAML processing works, to give you an idea what we're looking at:

YAML Processing Overview

The output of compose() above is the representation (node graph).

From that, safe_load() does its best to construct objects, but it can't do anything for tags it doesn't know about.

There must be a better way!

Thankfully, the spec also says:

That said, tag resolution is specific to the application. YAML processors should therefore provide a mechanism allowing the application to override and expand these default tag resolution rules.

We'll use this mechanism to convert tagged nodes to almost-native types, while preserving the tags.

A note on PyYAML extensibility #

PyYAML is a bit unusual.

For each processing direction, you have a corresponding Loader/Dumper class.

For each processing step, you can add callbacks, stored in class-level registries.

The callbacks are method-like – they receive the Loader/Dumper as the first argument:

Dice = namedtuple('Dice', 'a b')

def dice_representer(dumper, data):
    return dumper.represent_scalar(u'!dice', u'%sd%s' % data)

yaml.Dumper.add_representer(Dice, dice_representer)

You may notice the add_...() methods modify the class in-place, for everyone, which isn't necessarily great; imagine getting a Dice from safe_load(), when you were expecting only built-in types.

We can avoid this by subclassing, since the registry is copied from the parent. Note that because of how copying is implemented, registries from two direct parents are not merged – you only get the registry of the first parent in the MRO.

So, we'll start by subclassing SafeLoader/Dumper:

4
5
6
7
8

class Loader(yaml.SafeLoader):
    pass

class Dumper(yaml.SafeDumper):
    pass

Preserving tags #

Constructing unknown objects #

For now, we can use named tuples for objects with unknown tags, since they are naturally tag/value pairs:

12
13
14

class Tagged(typing.NamedTuple):
    tag: str
    value: object

Tag or no tag, all YAML nodes are either a scalar, a sequence, or a mapping. For unknown tags, we delegate construction to the loader's default constructors, and wrap the resulting value:

17
18
19
20
21
22
23
24
25
26
27
28

def construct_undefined(self, node):
    if isinstance(node, yaml.nodes.ScalarNode):
        value = self.construct_scalar(node)
    elif isinstance(node, yaml.nodes.SequenceNode):
        value = self.construct_sequence(node)
    elif isinstance(node, yaml.nodes.MappingNode):
        value = self.construct_mapping(node)
    else:
        assert False, f"unexpected node: {node!r}"
    return Tagged(node.tag, value)

Loader.add_constructor(None, construct_undefined)

Constructors are registered by tag, with None meaning "unknown".

Things look much better already:

>>> yaml.load(text, Loader=Loader)
{
    'one': Tagged(tag='!myscalar', value='string'),
    'two': Tagged(tag='!mysequence', value=[1, 2]),
}

A better wrapper #

That's nice, but every time we use any value, we have to check if it's tagged, and then go through value if is:

>>> one = _['one']
>>> one.tag
'!myscalar'
>>> one.value.upper()
'STRING'

We could subclass the Python types corresponding to core YAML tags (str, list, and so on), and add a tag attribute to each. We could subclass most of them, anyway – neither bool nor NoneType can be subclassed.

Or, we could wrap tagged objects in a class with the same interface, that delegates method calls and attribute access to the wrapee, with a tag attribute on top.

Tip

This is known as the decorator pattern design pattern (not to be confused with Python decorators).

Doing this naively entails writing one wrapper per type, with one wrapper method per method and one property per attribute. That's even worse than subclassing!

There must be a better way!

Of course, this is Python, so there is.

We can use an object proxy instead (also known as "dynamic wrapper"). While they're not perfect in general, the one wrapt provides is damn near perfect enough²:

12
13
14
15
16
17
18
19
20
21
22

class Tagged(wrapt.ObjectProxy):

    # tell wrapt to set the attribute on the proxy, not the wrapped object
    tag = None

    def __init__(self, tag, wrapped):
        super().__init__(wrapped)
        self.tag = tag

    def __repr__(self):
        return f"{type(self).__name__}({self.tag!r}, {self.__wrapped__!r})"

>>> yaml.load(text, Loader=Loader)
{
    'one': Tagged('!myscalar', 'string'),
    'two': Tagged('!mysequence', [1, 2]),
}

The proxy behaves identically to the proxied object:

>>> one = _['one']
>>> one.tag
'!myscalar'
>>> one.upper()
'STRING'
>>> one[:3]
'str'

...up to and including fancy things like isinstance():

>>> isinstance(one, str)
True
>>> isinstance(one, Tagged)
True

And now you don't have to care about tags if you don't want to.

Representing tagged objects #

The trip back is exactly the same, but much shorter:

39
40
41
42
43
44
45

def represent_tagged(self, data):
    assert isinstance(data, Tagged), data
    node = self.represent_data(data.__wrapped__)
    node.tag = data.tag
    return node

Dumper.add_representer(Tagged, represent_tagged)

Representers are registered by type.

>>> print(yaml.dump(Tagged('!hello', 'world'), Dumper=Dumper))
!hello 'world'

Since we still have stuff to do, we parametrize the tests from the start.

 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

BASIC_TEXT = """\
one: !myscalar string
two: !mymapping
  three: !mysequence [1, 2]
"""

BASIC_DATA = {
    'one': Tagged('!myscalar', 'string'),
    'two': Tagged('!mymapping', {'three': Tagged('!mysequence', [1, 2])}),
}

DATA = [
    (BASIC_TEXT, BASIC_DATA),
]

Loading works:

23
24
25

@pytest.mark.parametrize('text, data', DATA)
def test_load(text, data):
    assert yaml.load(text, Loader=Loader) == data

And dumping works:

28
29
30
31

@pytest.mark.parametrize('text', [t[0] for t in DATA])
def test_roundtrip(text):
    data = yaml.load(text, Loader=Loader)
    assert data == yaml.load(yaml.dump(data, Dumper=Dumper), Loader=Loader)

... but only for known types:

34
35
36

def test_dump_error():
    with pytest.raises(yaml.representer.RepresenterError):
        yaml.dump(object(), Dumper=Dumper)

Unhashable keys #

Let's try an example from the PyYAML documentation:

>>> text = """\
... ? !!python/tuple [0,0]
... : The Hero
... ? !!python/tuple [1,0]
... : Treasure
... ? !!python/tuple [1,1]
... : The Dragon
... """

This is supposed to result in something like:

>>> yaml.unsafe_load(text)
{(0, 0): 'The Hero', (1, 0): 'Treasure', (1, 1): 'The Dragon'}

Instead, we get:

>>> yaml.load(text, Loader=Loader)
Traceback (most recent call last):
  ...
TypeError: unhashable type: 'list'

That's because the keys are tagged lists, and neither type is hashable:

>>> yaml.load("!!python/tuple [0,0]", Loader=Loader)
Tagged('tag:yaml.org,2002:python/tuple', [0, 0])

This limitation comes from how Python dicts are implemented,³ not from YAML; quoting from the spec again:

The content of a mapping node is an unordered set of key/value node pairs, with the restriction that each of the keys is unique. YAML places no further restrictions on the nodes. In particular, keys may be arbitrary nodes, the same node may be used as the value of several key/value pairs and a mapping could even contain itself as a key or a value.

Constructing pairs #

What now?

Same strategy as before: wrap the things we can't handle.

Specifically, whenever we have a mapping with unhashable keys, we return a list of pairs instead. To tell it apart from plain lists, we use a subclass:

48
49
50

class Pairs(list):
    def __repr__(self):
        return f"{type(self).__name__}({super().__repr__()})"

Again, we let the loader do most of the work:

53
54
55
56
57
58
59
60
61

def construct_mapping(self, node):
    value = self.construct_pairs(node)
    try:
        return dict(value)
    except TypeError:
        return Pairs(value)

Loader.construct_mapping = construct_mapping
Loader.add_constructor('tag:yaml.org,2002:map', Loader.construct_mapping)

We set construct_mapping so that any other Loader constructor wanting to make a mapping gets to use it (like our own construct_undefined() above). Don't be fooled by the assignment, it's a method like any other.⁴ But we're changing the class from outside anyway, it's best to stay consistent.

Note that overriding construct_mapping() is not enough: we have to register the constructor explictly, otherwise SafeDumper's construct_mapping() will be used (since that's what was in the registry before).

Note

In case you're wondering, this feature is orthogonal from handling unknown tags; we could have used different classes for them. However, as mentioned before, the constructor registry breaks multiple inheritance, so we couldn't use the two features together.

Anyway, it works:

>>> yaml.load(text, Loader=Loader)
Pairs(
    [
        (Tagged('tag:yaml.org,2002:python/tuple', [0, 0]), 'The Hero'),
        (Tagged('tag:yaml.org,2002:python/tuple', [1, 0]), 'Treasure'),
        (Tagged('tag:yaml.org,2002:python/tuple', [1, 1]), 'The Dragon'),
    ]
)

Representing pairs #

Like before, the trip back is short and uneventful:

64
65
66
67
68
69

def represent_pairs(self, data):
    assert isinstance(data, Pairs), data
    node = self.represent_dict(data)
    return node

Dumper.add_representer(Pairs, represent_pairs)

>>> print(yaml.dump(Pairs([([], 'one')]), Dumper=Dumper))
[]: one

Because the tests are parametrized, we just need to add more data:

18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

UNHASHABLE_TEXT = """\
[0,0]: one
!key {0: 1}: {[]: !value three}
"""

UNHASHABLE_DATA = Pairs(
    [
        ([0, 0], 'one'),
        (Tagged('!key', {0: 1}), Pairs([([], Tagged('!value', 'three'))])),
    ]
)

DATA = [
    (BASIC_TEXT, BASIC_DATA),
    (UNHASHABLE_TEXT, UNHASHABLE_DATA),
]

Conclusion #

YAML is extensible by design. I hope that besides what it says on the tin, this article shed some light on how to customize PyYAML for your own purposes, and that you've learned at least one new Python thing.

You can get the code here, and the tests here.

Learned something new today? Share this with others, it really helps!

Want more? Get updates via email or Atom feed.

Bonus: hashable wrapper #

You may be asking, why not make the wrapper hashable?

Most unhashable (data) objects are that for a reason: because they're mutable.

We have two options:

• Make the wrapper hash change with the content. This this will break dictionaries in strange and unexpected ways (and other things too) – the language requires mutable objects to be unhashable.

• Make the wrapper hash not change with the content, and wrappers equal only to themselves – that's what user-defined classes do by default anyway.

  This works, but it's not very useful, because equal values don't compare equal anymore (data != load(dump(data))). Also, it means you can only get things from a dict if you already have the object used as key:

  >>> data = {Hashable([1]): 'one'}
  >>> data[Hashable([1])]
  Traceback (most recent call last):
    ...
  KeyError: Hashable([1])
  >>> key = list(data)[0]
  >>> data[key]
  'one'
  

  I'd file this under "strange and unexpected" too.

  (You can find the code for the example above here.)

Bonus: broken YAML #

We can venture even farther, into arguably broken YAML. Let's look at some examples.

First, there are undefined tag prefixes:

>>> yaml.load("!m!xyz x", Loader=Loader)
Traceback (most recent call last):
  ...
yaml.parser.ParserError: while parsing a node
found undefined tag handle '!m!'
  in "<unicode string>", line 1, column 1:
    !m!xyz x
    ^

A valid version:

>>> yaml.load("""\
... %TAG !m! !my-
... ---
... !m!xyz x
... """, Loader=Loader)
Tagged('!my-xyz', 'x')

Second, there are undefined aliases:

>>> yaml.load("two: *anchor", Loader=Loader)
Traceback (most recent call last):
  ...
yaml.composer.ComposerError: found undefined alias 'anchor'
  in "<unicode string>", line 1, column 6:
    two: *anchor
         ^

A valid version:

>>> yaml.load("""\
... one: &anchor [1]
... two: *anchor
... """, Loader=Loader)
{'one': [1], 'two': [1]}

It's likely possible to handle these in a way similar to how we handled undefined tags, but we'd have to go deeper – the exceptions hint to which processing step to look at.

Since I haven't actually encountered them in real life, we'll "save them for later" :)

1. Of which YAML is actually a superset. ^[return]

2. Timothy 20:9. ^[return]

3. Using a hash table. For nice explanation of how it all works, complete with a pure-Python implementation, check out Raymond Hettinger's talk Modern Python Dictionaries: A confluence of a dozen great ideas (code). ^[return]

4. Almost. The zero argument form of super() won't work for methods defined outside of a class definition, but we're not using it here. ^[return]

January 26, 2022 12:14 PM UTC

Martin Fitzpatrick

DiffCast: Automatic Python Screencast Creator — Create reproducible programming screencasts without typos or edits

Programming screencasts are a popular way to teach programming and demo tools. Typically people will open up their favorite editor and record themselves tapping away. But this has a few problems. A good setup for coding isn't necessarily a good setup for video -- with text too small, a window too big, or too many distractions. Typing code in live means mistakes, which means more time editing or confusion for the people watching.

DiffCast eliminates that, automatically generating screencasts from Python source files you prepare ahead of time.

Given the following Python source files:

print('Hello, world!')

name = input('What is your name?\n')
print(f'Hello, {name}!')

friends = ['Emelia', 'Jack', 'Bernardina', 'Jaap']

name = input('What is your name?\n')

if name in friends:
    print(f'Hello, {name}!')
else:
    print("I don't know you.")

friends = ['Emelia', 'Jack', 'Bernardina', 'Jaap']

while True:
    name = input('What is your name?\n')

    if name in friends:
        print(f'Hello, {name}!')
    else:
        print("I don't know you.")
        friends.append(name)

DiffCast will generate the following screencast (editor can be captured separately).

The editor view is configured to be easily readable in video, without messing with your IDE settings. Edits happen at a regular speed, without mistakes, making them easy to follow. Each diffcast is completely reproducible, with the same files producing the same output every time. You can set defined window sizes, or remove the titlebar to record.

Designed for creating reproducible tutoring examples, library demos or demonstrating code to a class. You can also step forwards and backwards through each file manually, using the control panel.

Finally, you can write out edits to a another file, and show the file location in a file listing for context. Run the intermediate files to demonstrate the effect of the changes.

DiffCast is open source (GPL licensed) & free to use. For bug reports, feature requests see the Github project.

January 26, 2022 11:00 AM UTC

ItsMyCode

Calculate Euclidean Distance in Python

In this article, we will be using the NumPy and SciPy modules to Calculate Euclidean Distance in Python.

In mathematics, the Euclidean Distance refers to the distance between two points in the plane or 3-dimensional space. In short, we can say that it is the shortest distance between 2 points irrespective of dimensions.

How to Calculate Euclidean Distance in Python?

The formula to calculate the distance between two points (x1 1 , y1 1 ) and (x2 2 , y2 2 ) is d = √[(x2 – x1)2 + (y2 – y1)2].

There are 4 different approaches for finding the Euclidean distance in Python using the NumPy and SciPy libraries.

1. Using linalg.norm()
2. Using dot() and sqrt()
3. Using square() and sum() 
4. Using distance.euclidean() from SciPy Module

Method 1: Using linalg.norm() Method in NumPy

The NumPy module has a norm() method, which can be used to find the required distance when the data is provided in the form of an array.

The norm() method returns the vector norm of an array. You can learn more about the linalg.norm() method here.

Example

# Python code to find Euclidean distance
# using linalg.norm()

# Import NumPy Library
import numpy as np

# initializing points in
# numpy arrays
point1 = np.array((4, 4, 2))
point2 = np.array((1, 2, 1))

# calculate Euclidean distance
# using linalg.norm() method
dist = np.linalg.norm(point1 - point2)

# printing Euclidean distance
print(dist)

Output

3.7416573867739413

Method 2: Using dot() and sqrt() methods

We can leverage the NumPy dot() method for finding the dot product of the difference of points, and by doing the square root of the output returned by the dot() method, we will be getting the Euclidean distance. 

# Python code to find Euclidean distance
# using dot() and sqrt() methods

# Import NumPy Library
import numpy as np

# initializing points in
# numpy arrays
point1 = np.array((4, 4, 2))
point2 = np.array((1, 2, 1))

# subtracting both the vectors
temp = point1 - point2
 
# Perform dot product
# and do the square root
dist = np.sqrt(np.dot(temp.T, temp))
 
# printing Euclidean distance
print(dist)

Output

3.7416573867739413

Method 3: Using square() and sum() methods

Another alternate way is to apply the mathematical formula (d = √[(x2 – x1)2 + (y2 – y1)2]) using the NumPy Module to Calculate Euclidean Distance in Python

The sum() function will return the sum of elements, and we will apply the square root to the returned element to get the Euclidean distance.

# Python code to find Euclidean distance
# using square() and sum() methods

# Import NumPy Library
import numpy as np

# initializing points in
# numpy arrays
point1 = np.array((4, 4, 2))
point2 = np.array((1, 2, 1))

# finding sum of squares
sum_vectors = np.sum(np.square(point1 - point2))
 
# perform the squareroot and
# print Euclidean distance
print(np.sqrt(sum_vectors))

Output

3.7416573867739413

Method 4: Using distance.euclidean() from SciPy Module

We discussed several methods to Calculate Euclidean distance in Python using the NumPy module. These methods can be slower when it comes to performance, and hence we can use the SciPy library, which is much more performance efficient.

The SciPy module is mainly used for mathematical and scientific calculations. It has a built-in distance.euclidean() method that returns the Euclidean Distance between two points.

# Python code to find Euclidean distance
# using distance.euclidean() method

# Import SciPi Library
from scipy.spatial import distance

# initializing points in
# numpy arrays
point1 = (4, 4, 2)
point2 = (1, 2, 1)
 
# print Euclidean distance 
print(distance.euclidean(point1,point2))

Output

3.7416573867739413

January 26, 2022 10:51 AM UTC

Python Print Variable

Python is one of the most versatile programming languages, and we can use the print statement in several ways to print a variable in Python.

In this article, let us look at different approaches available in Python to print a variable with examples.

How to use the print() function in Python?

In Python, we can use the print statement to print any variable, strings, and other data types provided in Python, or we can use it for debugging the code.

The syntax of the print statement differs from Python 2 to Python 3. 

In Python 2, the print statement is pretty straightforward. You need to use the print keyword and input the message, as shown in the example.

Example – Print statement in Python 2 

# print statement in Python 2
print "Welcome to Python Tutorials"

Output

Welcome to Python Tutorials

In Python 3, you need to use the print statement as a method, which means the message you need to print should be wrapped in the parenthesis, as shown in the example.

Example – Print statement in Python 3

# print statement in Python 3
print("Welcome to Python Tutorials")

Output

Welcome to Python Tutorials

Note: In Python 2, we can run the print statement with and without parenthesis. However, in Python 3, we can run the print statement as a method; otherwise, you will get an error.

How to Print variable in Python?

There are 5 different approaches to print variables in Python. Let us explore each of the methods in depth with examples.

• Using comma , character 
• Using string Formatting with the help of %
• Using string Formatting with the help of {}
• Using the + operator
• Using the f-string 

Method 1: Using comma , character to separate the variables in a print statement

The simplest and naive way to print the variable in Python is to separate the variable using the comma character in the print statement.

The following example will print the variable and string in the same line in Python code.

Example

# printing string with print statement using comma ","
first_name = "Bing"
last_name = "Chandler"
print("The full name of the Employee is ", last_name, first_name)

Output

The full name of the Employee is  Chandler Bing

Method 2: Using the String Formatting with the help of % character

The string formatting is popular in other languages, and Python also gives multiple ways to string formatting and print them.

String Formatting gives the user more options to customize, substitute, and place the variable in the right place to make the output of the print message more meaningful.

For example, we can use %s to substitute the string variable and %d to substitute the numerical value.

# print statement using string formatting 
first_name = "Bing"
last_name = "Chandler"
age =25
print("The full name of the Employee is %s %s and the age is %d " %(last_name, first_name,age))

Output

The full name of the Employee is Chandler Bing and  age is 25 

Method 3: Using the string formatting with positional arguments {}

We can use Python format specifiers as replacement fields within the string format. Starting from Python 3.1, the positional argument specifiers can be omitted.

For example, until Python 3.1, we had to pass the positional specifiers as arguments to the print function as shown below.

Example

# string formatting without positional specifiers
first_name = "Bing"
last_name = "Chandler"
age =25
print("The full name of the Employee is {0} {1} and the age is {2} ".format(last_name, first_name,age))
print("The full name of the Employee is {1} {0} and the age is {2} ".format(last_name, first_name,age))

Output

The full name of the Employee is Chandler Bing and the age is 25 
The full name of the Employee is Bing Chandler and the age is 25 

From Python 3.1 onwards, the positional specifiers can be omitted, and the same can be written as shown in the below example.

Example

# string formatting without positional specifiers
first_name = "Bing"
last_name = "Chandler"
age =25
print("The full name of the Employee is {} {} and the age is {} ".format(last_name, first_name,age))

Output

The full name of the Employee is Chandler Bing and the age is 25 

Note: We can use the positional specifiers in Python 3.1 and above as it has its own advantage. In fact, it works in all the Python versions.

Method 4: Using the f-string in the print statement

Another simplest way is to use f-string denoting with the letter f, which indicates Python, to format the string.

The f-string is another method of achieving string formatting, and it is comparatively faster when compared to other string formatting methods.

Example

# print statement using f-string
first_name = "Bing"
last_name = "Chandler"
age =25
print(f"The full name of the Employee is {last_name} {first_name} and the age is {age} ")

Output

The full name of the Employee is Chandler Bing and the age is 25 

Method 5: Using the + operator to join variables

The + operator is a mathematical operator, but you can use it in the print statement to join the variables. This basically is a string concatenation that concatenates the variables and string in Python.

Example

# print statement using the + character
first_name = "Bing "
last_name = "Chandler "
age =25
print("The full name of the Employee is " +last_name + first_name +" and the age is "+str(age))

Output

The full name of the Employee is Chandler Bing  and the age is 25

January 26, 2022 08:01 AM UTC

Codementor

A Senior Engineer at Google Reveals “The Best Programming Language To Learn in 2022”

I am a senior software engineer at Google Singapore and very often I am asked about which programming language to learn.

January 26, 2022 01:46 AM UTC

January 25, 2022

PyCoder’s Weekly

Issue #509 (Jan. 25, 2022)

#509 – JANUARY 25, 2022
View in Browser »

How to Quickly Label Data for Machine Learning

With Toloka, you can control the accuracy and speed of data labeling to develop high performing ML models. Our platform supports annotation for image classification, semantic segmentation, object detection, named entity recognition, sentiment analysis, speech recognition, text classification →
TOLOKA AI sponsor

Python Jobs

Senior Full-Stack Web Developer (Anywhere)

MonetizeMore

Python Engineer (Web Scraping) (Asia, TX, America)

NewsCatcher

PyCoder's Weekly Curator and Podcast Co-Host (Anywhere)

Real Python

Senior Software Engineer (Anywhere)

Brew

More Python Jobs >>>

Articles & Tutorials

Scrape Your Web Data From Any Target

With Oxylabs Scraper APIs, extract public data from the most complex targets. Our Scraper APIs handle JavaScript-heavy websites and support large data requests using over 102 million global proxy infrastructure. Receive data in JSON or CSV format and pay per successful request only. Start free trial →
OXYLABS sponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #509.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

January 25, 2022 07:30 PM UTC

Python Morsels

How to remove spaces in Python

Need to remove spaces from your a string in Python? Let's talk about how to remove all spaces, remove duplicate spaces, remove spaces from the ends of our strings, remove trailing newlines, and remove spaces from the beginning and end of each line.

Remove spaces from a string

Need to remove all spaces from a string in Python?

If it's just space characters you could use the string replace method to replace all spaces by an empty string.

If we call the replace method on this greeting string:

>>> greeting = " Hello world! "
>>> no_spaces = greeting.replace(" ", "")

The resulting no_spaces string will have all space characters removed:

>>> no_spaces
'Helloworld!'

Remove all whitespace from a string

If you're trying to remove all sorts of whitespace characters (space, tab, newline, etc.) you could use the string split and join methods:

If we call split on this version string, Python will split on all consecutive whitespace characters:

>>> version = "\tpy 310\n"
>>> version.split()
['py', '310']

The string join method can join an iterable of strings by a delimiter (see turning a list into a string). If we join with a delimiter of empty string (""), we'll effectively remove all spaces:

>>> no_spaces = "".join(version.split())
>>> no_spaces
'py310'

If you're comfortable with regular expressions, you can also use a regular expression to replace all consecutive whitespace by an empty string:

>>> import re
>>> no_spaces = re.sub(r"\s+", r"", version)
>>> no_spaces
'py310'

Remove duplicate whitespace

What if you just need to get rid of extra spaces (collapsing consecutive spaces)?

We could use the string split and join methods, as before, but join on a space character instead of an empty string:

>>> version = "\tpy 310\n"
>>> normalized_spaces = " ".join(version.split())
>>> normalized_spaces
'py 310'

Note that this normalizes all whitespace characters (so newlines and tabs will be converted as well) and it removes spaces from the ends of our string.

Strip whitespace from the beginning and end of a string

What if you only need to remove whitespace from the beginning and end of your string?

You can use the string strip method:

>>> version = "\tpy 310\n"
>>> stripped_version = version.strip()
>>> stripped_version
'py 310'

By default the strip method removes all whitespace characters (not just spaces).

The strip method also accepts an optional argument if you prefer to strip just a specific character. It also has two cousin methods: lstrip (for splitting from the left-hand side) and rstrip (for splitting from the right-hand side).

If you just need to remove an optional trailing newline character from the end of your string, you can use strip (passing in a \n character):

>>> version = "\tpy 310\n"
>>> no_trailing_newline = version.rstrip("\n")
>>> no_trailing_newline
'\tpy 310'

Strip whitespace from the beginning and end of each line

What if you need to strip whitespace from the beginning and end of each line in your string?

You could split your lines with splitlines, use a comprehension to call the strip method on each line, and then use the join method to join your lines back together with newline characters:

>>> string = " Line 1\nLine 2  \n Line 3 \n"
>>> stripped = "\n".join([
...     line.strip()
...     for line in string.splitlines()
... ])
...
>>> stripped
'Line 1\nLine 2\nLine 3'

Although this is complex enough that I'd usually start to reach for regular expressions at this point:

>>> import re
>>> stripped = re.sub(r"^\s+|\s+$", r"", string, flags=re.MULTILINE)
>>> stripped
'Line 1\nLine 2\nLine 3'

January 25, 2022 04:00 PM UTC

The Open Sourcerer

The Software Upgrade Threadmill and Life’s crazy chain of dependencies — an epic tale about Firefox, GTG, Python, and Linux distros

This blog post talks about the general types of hurdles I’ve been encountering on the desktop side of the “upgrade threadmill” when running Linux. It is part two of a three-part blog post série. If you’re into servers, you may be interested in part one, the Linux server landscape’s post-2020 metamorphosis.

Modern software development happens at a breakneck pace, and while staying on ancient versions (hello, Debian Stable / Ubuntu LTS / Android users) is not really a safe and realistic option anymore (try reporting bugs without getting laughed out of the room by upstream maintainers), it is becoming a challenge for users to keep up. When it works, it works… but when something breaks down in the upgrade threadmill, the chain of dependencies to get back on track can become absolutely ludicrous and throw your digital life in turmoil. Just like needing to replace that one light bulb…

Case in point: I’m finally publishing this article in 2022, while I initially meant to blog about this way back in 2017… but more stuff kept breaking all the time, resetting my productivity and accidentally adding more potential content for this blog post. More value for you, dear reader! So grab your popcorn and read on.

As someone who has been running Linux for 19 years (as of 2022), I think I know my way around most hurdles you can possibly encounter. Undoubtedly, running Linux-based operating systems on desktop/laptop computers has overall gotten incredibly easier compared to 2003, but also, as one gradually becomes highly dependent on specific tools and committed to well-oiled workflows, the upgrade threadmill can become a real high-stakes pursuit.

I’ll skim over minor anecdotes like the fact that I once had to “dnf versionlock alsa” for a while because ALSA was misbehaving with my Sound Blaster™ Audigy™ (don’t laugh! That’s how you get analog surround sound when your motherboard doesn’t do 5.1) while testing on a live USB version of a new Fedora release but, as it turns out, it worked without problems after installation… or the fact that I just realized, in 2022, that nautilus-sendto doesn’t exist anymore and since most apps haven’t yet migrated to the Flatpak-compatible replacement “portal”, that means I can’t right-click a file in Nautilus to send it via email now… and I’ll instead dive into the rather more interesting reason why I ran an old insecure web browser—and operating system—for nearly two years.

A tale of foxes

In 2017, Mozilla unleashed Firefox Quantum, quite possibly the most awesome and groundbreaking release they did in that decade. Except there was just one little problem: it also completely changed the extensions system (for good technical reasons).

I happened to be critically dependent, for my day-to-day productivity, on the now incompatible TabGroups extension (which was, itself, a reimplementation of Mozilla’s previously deprecated “Panorama” feature from Firefox 4, previously known as “Tab Candy“), whose author had burned out and given up completely on porting it to the new WebExtensions API.

As a power user, I was utterly foxed.

There was an awfully long period of time where Firefox had no replacement and no migration path for users like me.

Because the modern web (a.k.a. the web obesity crisis) was unbearable, I needed to stay on an Electrolysis-capable-but-pre-Quantum version of Firefox, and that meant specifically Firefox 57, and no other. Not even Firefox ESR. I had to deep-freeze my Firefox version to prevent any upgrade. Thankfully, operating systems like Fedora allow you to do that easily, with DNF’s “versionlock” feature.

This is what “dnf versionlock firefox” feels like (soft warning: this photo may shock animal lovers):

Locking your Firefox version is something that you should never do from a security standpoint (yes I can hear your lamentations, infosec crowd), but I had no other choice and I was too busy in my day-to-day mercenary CMO job to be messing with my personal productivity tooling. So I kept waiting, checking and hoping that someday, someone would write a replacement WebExtension for it.

I therefore spent a painfully long period of time stuck on the same old (and eventually EoL‘ed) version of Fedora Workstation, because upgrading Fedora eventually became impossible due to dependency issues arising from the versionlocked Firefox. While Fedora 27 was released in 2017, I had to remain on that frozen-in-time, unmaintained, insecure version until the end of 2019.

This is what waiting in suspense, on an EoL’ed operating system, looks like:

Yes, I can hear all of you from the InfoSec crowd gasping for air, in shock, after having stared at the screen for a solid 30 seconds wondering how I could have subjected myself to such risk in this day and age. What can I say, I have NERV_es of S_tEELE.

Thankfully, one day in 2019, Simple Tab Groups was recommended to me and, as I tested it extensively to see how solid it actually was, I determined it to be the sole viable, Nekohayo-Proof™, Enterprise-Grade™ replacement for Panorama/TabGroups. It provided a migration path and it was pretty much bullet-proof (especially compared to things like Conex, which I found to be unreliable for my needs). I’ve been happily using it since 2019 and it works better than anything else ever did. With a couple of tweaks in the settings, it has much better performance than its predecessors—especially when you disable thumbnails, use the tabs “discarding” (unloading) feature, and after my suggested search activation technique was implemented—so that’s neat.

A tale of pure hard drive

Eventually, Firefox being a solved problem at last, and Fedora 27 reaching end-of-life status, I really had to migrate, not just for security reasons, but also because my computer’s hard drive partitions were badly space-bound, as I had really outgrown the layout I had made all the way back in… 2010.

Repartitioning my workstation, and upgrading some SSDs in the process, was also blocking my ability to reassign the older/smaller SSDs to my server, which in turn was blocking my ability to migrate from Centos 7 to Centos 8, which was hurting my ability to upgrade PHP sustainably, which was blocking my ability to upgrade Matomo.

If your head hurts while you’re reading this, at this point, it’s not just because of the hard drives. But wait, I’m not done yet!

A tale of serpents

Fedora, as a fast-moving operating system, generally favors leanness and innovation over backwards compatibility (“Friends, Features, First“). This means that if you want to get rid of Python 2 for example, you hunt down and exile any application that still hasn’t ported to Python 3 (they were not alone by the way; OpenSUSE and Sabayon Linux, among others, did the same thing roughly a year later). This works well in an ideal world where application developers have the time and resources to quickly port to new technologies, but the Python 2 to 3 migration has been a notoriously long process in terms of ecosystem. And sometimes, developers have to shave two dozen massive yaks in lockstep in order to achieve that. I know, because I have lived through, as a co-maintainer, the multi-years effort that was porting Pitivi to Python 3, GTK 3, GObject Introspection and GStreamer 1.0 simultaneously.

As a result, when Fedora decided to cut the cord for Python 2 apps in version 31, I had a number off desktop applications simply cease to exist on Fedora. Among them were DisplayCAL (which I had just discovered as the only way to actually make my Colormunki spectrophotometer work for display calibration without requiring a PhD in quantum physics, as colord unfortunately never worked reliably for me), and… Getting Things GNOME.

I could live without DisplayCAL (“we had barely just met!”), but I could not live without GTG, so I had to go begging for a good samaritan to emergency-package it as a Flatpak. Thankfully, Bilal Elmoussaoui answered the call to package the old version. Whew, I had some breathing room and could finally upgrade from the EoL’ed Fedora 27 to latest stable version at the time, Fedora 30, while I figured out a way to put the GTG project back on track.

• I eventually did put the GTG project back onto the rails (with Diego’s admirable bugfixing frenzy, the much-awaited 0.4 version was released in 2020 from development hell). Two years down the road, GTG is now slated to re-enter Fedora (hopefully) 🤞
• DisplayCAL, on the other hand, is still a Python 2 application so… as Larry “Solo Wing Pixy” Foulke said in the introduction scene of The Belkan War, “It’s going to take a while.”

As you can see, even for an experienced Linux user used to troubleshooting cutting-edge technology, the upgrade threadmill can be a problem. I mean, just look at how absolutely ridiculous the “chain of dependencies” has been for me here just to solve the “critical path”:

• Objective: “run the latest Fedora version!”
  • Blocked by Firefox version
    • Blocked by needing to wait for a viable replacement for Panorama / Tab Groups
  • Blocked by GTG
    • Needs a temporary Flatpak package to live through the winter
    • Needs a new Python 3 + GObject Introspection + GTK3-based release
      • Needs a new community of coders to reform around the project
        • Needs product management, including proper open-source processes and an opiniated direction
          • Needs updating/rewriting the majority of contributors’ documentation
          • Needs talking to previous maintainers and the public to ascertain what led to previous failure
        • Needs remarketing & open-source tech evangelism, including outreach & community management
          • Needs to evaluate the feasibility and available resources
            • Need to ascertain the technical status of the previously stalled development efforts (by persistently tracking down and pressing previous maintainers for answers via personal email exchanges) and see if any of the onlookers of this ticket would be interested in embarking on that project
            • Needs to survey the market to gauge interest, with a marketing piece that also provides historical context and a pitch on GTG’s unique value proposition, preceded by:
              • Pre-market with a theory writeup explaining my worldview with a typology of workers
              • Pre-market with a theory writeup explaining the unique value of the GTD methodology for personal productivity
          • Needs a clear, actionable, lean, rapidly achievable “MVP” roadmap
            • Needs heavy bug triaging
              • Needs building & testing code that had been abandoned for 7 years
              • Needs claiming administrative ownership of the multiple bug trackers
          • Needs to build a social media presence from scratch
          • Needs to rewrite the project’s front page to market it to users (who may become potential contributors)
        • The things in bold above are done as a public service, but they also are what companies would hire me for, if they have enough Japanese Yen. I didn’t spend years training as a mercenary in Restricted Airspace B7R for nothin’!

That’s not even counting the repartitioning of all my drives (which was another yak shave that had its own chain of dependencies), nor the years-long epic investigation to solve weird hardware issues some years prior to all this. Goodness gracious, it’s a miracle we get anything done around here.

You may think the story ends here. After all, with those issues resolved, I finally was able to thaw everything and upgrade Fedora… and it was a huge sigh of relief indeed; late 2019 to late 2020 was pretty great in terms of being able to run a normal up-to-date Fedora Workstation experience. But nothing is eternal (except Doom)…

Keep your popcorn nearby. Next, I will write about the Curious Case of Fedora and Unison, which warrants its own blog post. Keep your eyes open for a new article on Monday, January 31st!

January 25, 2022 02:30 PM UTC

Real Python

Looping With Python enumerate()

In Python, a for loop is usually written as a loop over an iterable object. This means that you don’t need a counting variable to access items in the iterable. Sometimes, though, you do want to have a variable that changes on each loop iteration. Rather than creating and incrementing a variable yourself, you can use Python’s enumerate() to get a counter and the value from the iterable at the same time!

In this course, you’ll see how to:

• Use enumerate() to get a counter in a loop
• Apply enumerate() to display item counts
• Implement your own equivalent function to enumerate()
• Unpack values returned by enumerate()

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

January 25, 2022 02:00 PM UTC

Mike Driscoll

PySimpleGUI - An Intro to Laying Out Elements

PySimpleGUI is a Python GUI that wraps other Python GUI toolkits (Tkinter, PySide, wxPython). By abstracting away the complexities of the other GUIs into a common API, you can quickly write code that can be rendered using any of those other toolkits just by changing the import at the top of your program.

If you were to use wxPython and you wanted to layout your widgets, you would use a wx.Sizer. In PySide or PyQt, you would use Layout object. Tkinter uses geometry managers. The general concept is the same. When you use these tools, the GUI toolkit uses relative positioning of the widgets to arrange them onscreen.

In PySimpleGUI, the same thing is done using nested lists. You will look at a couple of different examples in this tutorial that will give you a general idea of how this all works.

Creating a Horizontal Layout

Creating a series of Elements that are horizontally oriented (left-to-right) is done with a small list of lists in PySimpleGUI.

Open up a Python editor and add the following code:

import PySimpleGUI as sg

# Horizontal layout
layout = [[sg.Button(f"OK {num}") for num in range(1, 6)]]

# Create the window
window = sg.Window("Demo", layout)

# Create an event loop
while True:
    event, values = window.read()
    # End program if user closes window or
    # presses the OK button
    if event == "OK" or event == sg.WIN_CLOSED:
        break

window.close()

Here you use a Python list comprehension to create a nested list that contains five Button objects in it. When you run this code, you will see the buttons aligned horizontally, with one next to the other going from left-to-right.

Now you are ready to see how you can rewrite the code to create a vertical layout!

Creating a Vertical Layout

To create a vertically oriented layout with PySimpleGUI, you need to make the layout contain a series of nested lists that each contain a one or more Elements in them.

Create a new Python file and add this code to it:

import PySimpleGUI as sg

layout = [[sg.Button("OK")],
          [sg.Button("OK 2")],
          [sg.Button("OK 3")],
          [sg.Button("OK 4")],
          [sg.Button("OK 5")]]

# Create the window
window = sg.Window("Demo", layout)

# Create an event loop
while True:
    event, values = window.read()
    # End program if user closes window or
    # presses the OK button
    if event == "OK" or event == sg.WIN_CLOSED:
        break

window.close()

When you run this code, the Button Elements will be stacked from top to bottom vertically, instead of horizontally.

Using Columns for Complex Layouts

If you need to add two or more columns of Elements next to each other in PySimpleGUI, you can use a sg.Column Element. It is a type of container Element specifically made for creating stacked sets of Elements!

Create another Python file and add the following code:

import PySimpleGUI as sg
import os.path


# First the window layout in 2 columns
file_list_column = [
    [sg.Text("Image Folder"),
     sg.In(size=(25, 1), enable_events=True, key="-FOLDER-"),
     sg.FolderBrowse(),],
    [sg.Listbox(values=[], enable_events=True, size=(40, 20), key="-FILE LIST-")],
]


# For now will only show the name of the file that was chosen
image_viewer_column = [
    [sg.Text("Choose an image from list on left:")],
    [sg.Text(size=(40, 1), key="-TEXT-")],
    [sg.Image(key="-IMAGE-")],
]

# ----- Full layout -----
layout = [
    [sg.Column(file_list_column),
     sg.VSeperator(),
     sg.Column(image_viewer_column),]
]


window = sg.Window("Column Demo", layout)


# Run the Event Loop
while True:
    event, values = window.read()

    if event == "Exit" or event == sg.WIN_CLOSED:
        break

    # Folder name was filled in, make a list of files in the folder
    if event == "-FOLDER-":
        folder = values["-FOLDER-"]
        try:
            # Get list of files in folder
            file_list = os.listdir(folder)
        except:
            file_list = []

        fnames = [
            f
            for f in file_list
            if os.path.isfile(os.path.join(folder, f))
            and f.lower().endswith((".png", ".gif"))
        ]

        window["-FILE LIST-"].update(fnames)

    elif event == "-FILE LIST-":  # A file was chosen from the listbox

        try:
            filename = os.path.join(values["-FOLDER-"], values["-FILE LIST-"][0])
            window["-TEXT-"].update(filename)
            window["-IMAGE-"].update(filename=filename)
        except:
            pass

window.close()

In this example, you create two layouts! The first layout is called file_list_column and contains four Elements in it. The second layout is called image_viewer_column and contains three Elements. Then you put these layouts inside of Columns that are themselves inside a layout. As you can see, PySimpleGUI layouts are lists inside of lists all the way down!

Wrapping Up

PySimpleGUI removes some of the complexity of creating layouts by using Python lists. The PyQt / PySide and wxPython GUI toolkits use an object-oriented approach which includes a significant amount of boilerplate and flags to do much the same thing. They each have their tradeoffs, but PySimpleGUI's learning curve is that much flatter because of their design choice.

Related Reading

• PySimpleGUI: Using an Image as a Button

• PySimpleGUI - How to Draw Shapes on an Image with a GUI

• A Brief Intro to PySimpleGUI

• The Demos for PySimpleGUI

The post PySimpleGUI - An Intro to Laying Out Elements appeared first on Mouse Vs Python.

January 25, 2022 01:30 PM UTC

Python for Beginners

String Indexing in Python

Strings are used for processing text data in Python. While processing strings, we often need to access a certain part of the string. In this article, we will see how we can extract parts of a string using indexing in Python. 

What is String Indexing ?

If we have an ordered sequence or container object such as a string, a list, or a tuple, we can access the elements of the objects using their relative position in the sequence. The relative position of the elements in the ordered sequence is termed as index. With Indexing, we can access any element from an ordered sequence using the indices. 

In python, string indexing is zero based. It means that we start the counting from 0 and the first character of the string is assigned the index 0, the second character is assigned the index 1, the third character is assigned the index 2 and so on. 

We can understand this using the following example.

Suppose that we have a string “PythonForBeginners”

Here, the index of the letter “P” is 0. The index of the letter “y” is 1. The index of letter ”t” is 2, The index of letter “h” is 3 and so on. The index of the last letter “s” is 17.

In python, we can use positive as well as negative numbers for string indexing. Let us discuss them one by one.

String Indexing using Positive Numbers

As we have seen above, Strings are indexed using positive numbers from 0 to string length -1. We can access a character at any position between 0 to (length of the string) -1 using positive indices as follows.

myString = "PythonForbeginners"
index = 0
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))
index = 1
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))
index = 2
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))
index = 3
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))
index = 17
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))

Output:

Character at index 0 in the string 'PythonForbeginners' is P.
Character at index 1 in the string 'PythonForbeginners' is y.
Character at index 2 in the string 'PythonForbeginners' is t.
Character at index 3 in the string 'PythonForbeginners' is h.
Character at index 17 in the string 'PythonForbeginners' is s.

Always remember that an index greater than or equal to the string length will cause an IndexError exception as follows.

myString = "PythonForbeginners"
index = 20
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))

Output:

Traceback (most recent call last):
  File "/home/aditya1117/PycharmProjects/pythonProject/string12.py", line 3, in <module>
    character = myString[index]
IndexError: string index out of range

You can either avoid the IndexError exception by checking the value of index before accessing any character in the string. Alternatively, you can use python try except block to handle the exceptions if they occur.

Here, I will suggest you to use the try except blocks. Checking the index every time we access a character may be redundant and costly if we are accessing using indices less than the length of the string. While using try except blocks, the program will not check the value of index each time we access a character from the string. If IndexError occurs, it will be handled by the code in the except block. 

Indexing using Negative Numbers

We can also use negative indices for accessing characters from a string. In python, the last character of the string is assigned an index -1. The second last character is assigned an index -2. Similarly, the first character of the string is assigned an index -(length of the string).

We can understand this using the following example.

Suppose that we have a string “PythonForBeginners”

Here, the index of the letter “s” is -1. The index of the letter “r” is -2. The index of letter ”n” is -3 , The index of letter “n” is -4 and so on. The index of the first letter “P” is -18.

You can verify this using the following program.

myString = "PythonForbeginners"
index = -1
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))
index = -2
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))
index = -3
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))
index = -4
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))
index = -18
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))

Output:

Character at index -1 in the string 'PythonForbeginners' is s.
Character at index -2 in the string 'PythonForbeginners' is r.
Character at index -3 in the string 'PythonForbeginners' is e.
Character at index -4 in the string 'PythonForbeginners' is n.
Character at index -18 in the string 'PythonForbeginners' is P.

While using negative numbers as indices, Make sure that you do not pass an index less than  -(length of the string). Otherwise, your program will run into an IndexError as follows.

myString = "PythonForbeginners"
index = -20
character = myString[index]
print("Character at index {} in the string '{}' is {}.".format(index, myString, character))

Output:

Traceback (most recent call last):
  File "/home/aditya1117/PycharmProjects/pythonProject/string12.py", line 3, in <module>
    character = myString[index]
IndexError: string index out of range

Conclusion

In this article, we have studied string indexing in python. We have seen how we can use negative as well as positive numbers to access characters from a string. To study more about strings in python, you can read this article on string concatenation.

The post String Indexing in Python appeared first on PythonForBeginners.com.

January 25, 2022 01:30 PM UTC

Python⇒Speed

The fastest way to read a CSV in Pandas

You have a large CSV, you’re going to be reading it in to Pandas—but every time you load it, you have to wait for the CSV to load. And that slows down your development feedback loop, and might meaningfully slows down your production processing.

But it’s faster to read the data in faster. Let’s see how.

In this article we’ll cover:

1. Pandas’ default CSV reading.
2. The faster, more parallel CSV reader introduced in v1.4.
3. A different approach that can make things even faster.

January 25, 2022 12:00 AM UTC

January 24, 2022

PyCharm

Together, We Supported Python!

Last November, PyCharm joined forces with the Python Software Foundation (PSF) for their end-of-the-year fundraiser. From November 9 to December 1, all proceeds from every new PyCharm Professional Edition license purchased with the discount code ‘SUPPORTPYTHON21’ went to the PSF’s general fund.

The Python Software Foundation is the main organization behind the Python programming language. As a non-profit organization, the PSF depends on sponsorships and donations to support its work. You can always donate yourself through their website.

JetBrains and the PSF would like to thank all of you who took part in this campaign. Together, we raised $25,000 in less than a month! Contributions like those from PyCharm users help the PSF maintain a healthy balance and continue to support the Python community and its various outreach and diversity programs.

We hope that you enjoy using Python in 2022 and that PyCharm will be a powerful and reliable partner for your journey!

The PyCharm Team

January 24, 2022 04:03 PM UTC

ItsMyCode

Adding new column to existing DataFrame in Pandas

In this article, we will look at different ways to adding new column to existing DataFrame in Pandas. 

Let us create a simple DataFrame that we will use as a reference throughout this article to demonstrate adding new columns into Pandas DataFrame.

# import pandas library
import pandas as pd

# create pandas DataFrame
df = pd.DataFrame({'team': ['India', 'South Africa', 'New Zealand', 'England'],
                   'points': [10, 8, 3, 5],
                   'runrate': [0.5, 1.4, 2, -0.6],
                   'wins': [5, 4, 2, 2]})

# print the DataFrame
print(df)

Output

           team  points  runrate  wins
0         India      10      0.5     5
1  South Africa       8      1.4     4
2   New Zealand       3      2.0     2
3       England       5     -0.6     2

Now that we have created a DataFrame let’s assume that we need to add a new column called “lost”, which holds the count of total matches each team has lost.

Method 1: Declare and assign a new list as a column

The simplest way is to create a new list and assign the list to the new DataFrame column. Let us see how we can achieve this with an example.

# import pandas library
import pandas as pd

# create pandas DataFrame
df = pd.DataFrame({'team': ['India', 'South Africa', 'New Zealand', 'England'],
                   'points': [10, 8, 3, 5],
                   'runrate': [0.5, 1.4, 2, -0.6],
                   'wins': [5, 4, 2, 2]})

# print the DataFrame
print(df)

# declare a new list and add the values into the list
match_lost = [2, 1, 3, 4]

# assign the list to the new DataFrame Column
df["lost"] = match_lost

# Print the new DataFrame
print(df)

Output

           team  points  runrate  wins  lost
0         India      10      0.5     5     2
1  South Africa       8      1.4     4     1
2   New Zealand       3      2.0     2     3
3       England       5     -0.6     2     4

Method 2: Using the DataFrame.insert() method

The disadvantage of the above approach is that we cannot add the column at the specified position, and by default, the column is inserted towards the end, making it the last column.

We can overcome the issue using the pandas.DataFrame.insert() method. This method is useful when you need to insert a new column in a specific position or index.

In the below example, let us insert the new column “lost” before the “wins” column. We can achieve this by inserting a new column at index 2.

# import pandas library
import pandas as pd

# create pandas DataFrame
df = pd.DataFrame({'team': ['India', 'South Africa', 'New Zealand', 'England'],
                   'points': [10, 8, 3, 5],
                   'runrate': [0.5, 1.4, 2, -0.6],
                   'wins': [5, 4, 2, 2]})

# print the DataFrame
print(df)


# insert the new column at the specific position
df.insert(3, "lost", [2, 1, 3, 4], True)

# Print the new DataFrame
print(df)

Output

           team  points  runrate  lost  wins
0         India      10      0.5     2     5
1  South Africa       8      1.4     1     4
2   New Zealand       3      2.0     3     2
3       England       5     -0.6     4     2

Method 3: Using the DataFrame.assign() method

The pandas.DataFrame.assign() method is used if we need to create multiple new columns in a DataFrame.

This method returns a new object with all original columns in addition to new ones. All the existing columns that are re-assigned will be overwritten.

In the below example, we are adding multiple columns to Pandas DataFrame.

# import pandas library
import pandas as pd

# create pandas DataFrame
df = pd.DataFrame({'team': ['India', 'South Africa', 'New Zealand', 'England'],
                   'points': [10, 8, 3, 5],
                   'runrate': [0.5, 1.4, 2, -0.6],
                   'wins': [5, 4, 2, 2]})

# print the DataFrame
print(df)

# append multiple columns to Pandas DataFrame
df2 = df.assign(lost=[2, 1, 3, 4], matches_remaining=[2, 3, 1, 1])

# Print the new DataFrame
print(df2)

Output

           team  points  runrate  wins  lost  matches_remaining
0         India      10      0.5     5     2                  2
1  South Africa       8      1.4     4     1                  3
2   New Zealand       3      2.0     2     3                  1
3       England       5     -0.6     2     4                  1

Method 4: Using the pandas.concat() method

We can also leverage the pandas.concat() method to concatenate a new column to a DataFrame by passing axis=1 as an argument. This method returns a new DataFrame after concatenating the columns.

# import pandas library
import pandas as pd

# create pandas DataFrame
df = pd.DataFrame({'team': ['India', 'South Africa', 'New Zealand', 'England'],
                   'points': [10, 8, 3, 5],
                   'runrate': [0.5, 1.4, 2, -0.6],
                   'wins': [5, 4, 2, 2]})

# print the DataFrame
print(df)

# create a new DataFrame
df2 = pd.DataFrame([[1, 2], [2, 1], [3, 4], [0, 3]],
                   columns=['matches_left', 'lost'])

# concat and Print the new DataFrame
print(pd.concat([df, df2], axis=1))

Output

           team  points  runrate  wins  matches_left  lost
0         India      10      0.5     5             1     2
1  South Africa       8      1.4     4             2     1
2   New Zealand       3      2.0     2             3     4
3       England       5     -0.6     2             0     3

Method 5: Using the Dictionary

Another trick is to create a dictionary to add a new column in Pandas DataFrame. We can use the existing columns as Key to the dictionary and assign values respectively to the new column.

# import pandas library
import pandas as pd

# create pandas DataFrame
df = pd.DataFrame({'team': ['India', 'South Africa', 'New Zealand', 'England'],
                   'points': [10, 8, 3, 5],
                   'runrate': [0.5, 1.4, 2, -0.6],
                   'wins': [5, 4, 2, 2]})

# print the DataFrame
print(df)

# Create a new dictionary with keys as existing column
# and the values of new column
match_lost = {2: 'India', 1: 'South Africa', 3: 'New Zealand', 0: 'England'}

# assign the dictionary to the DataFrame Column
df['lost'] = match_lost

# print Dataframe
print(df)

Output

           team  points  runrate  wins  lost
0         India      10      0.5     5     2
1  South Africa       8      1.4     4     1
2   New Zealand       3      2.0     2     3
3       England       5     -0.6     2     0

Conclusion

In this article, we saw the 5 approaches creating and assigning a list, insert(), assign(), concat() and dictionary to insert new columns into Pandas DataFrame or overwrite the existing ones. Depending on the need and the requirement, you can choose one of the methods specified which are more suitable.

January 24, 2022 03:49 PM UTC

Real Python

Modulo String Formatting in Python

If you’re writing modern Python code with Python 3, you’ll probably want to format your strings with Python f-strings. However, if you’re working with older Python codebases, you’re likely to encounter the string modulo operator for string formatting.

If you’re reading or writing Python 2 code, it’ll help if you’re familiar with this technique. Because the syntax still works in Python 3, you might even see developers use it in modern Python codebases.

In this tutorial, you’ll learn how to:

• Use the modulo operator (%) for string formatting
• Convert values into specific types before inserting them into your string
• Specify the horizontal space a formatted value occupies
• Fine-tune the display using conversion flags
• Specify values using dictionary mapping instead of tuples

If you’re acquainted with the printf() family of functions of C, Perl, or Java, then you’ll see that these don’t exist in Python. However, there’s quite a bit of similarity between printf() and the string modulo operator, so if you’re familiar with printf(), then a lot of the following will feel familiar.

On the other hand, if you aren’t familiar with printf(), don’t worry! You don’t need any prior knowledge of printf() to master modulo string formatting in Python.

Free Bonus: Click here to get our free Python Cheat Sheet that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Use the Modulo Operator for String Formatting in Python

You’ve probably used the modulo operator (%) before with numbers, in which case it computes the remainder from a division:

>>>

>>> 11 % 3
2

With string operands, the modulo operator has an entirely different function: string formatting.

Note: These two operations aren’t very much alike. They only share the same name because they are represented by the same symbol (%).

Here’s what the syntax of the string modulo operator looks like:

<format_string> % <values>

On the left side of the % operator, <format_string> is a string containing one or more conversion specifiers. The <values> on the right side get inserted into <format_string> in place of the conversion specifiers. The resulting formatted string is the value of the expression.

Get started with an example where you call print() to display a formatted string using the string modulo operator:

>>>

>>> print("%d %s cost $%.2f" % (6, "bananas", 1.74))
6 bananas cost $1.74

In addition to representing the string modulo operation itself, the % character also denotes the beginning of a conversion specifier in the format string—in this case, there are three: %d, %s, and %.2f.

In the output, Python converted each item from the tuple of values to a string value and inserted it into the format string in place of the corresponding conversion specifier:

• The first item in the tuple is 6, a numeric value that replaces %d in the format string.
• The next item is the string value "bananas", which replaces %s.
• The last item is the float value 1.74, which replaces %.2f.

The resulting string is 6 bananas cost $1.74, as demonstrated in the following diagram:

The String Modulo Operator

If there are multiple values to insert, then they must be enclosed in a tuple, as illustrated above. If there’s only one value, then you can write it by itself without the surrounding parentheses:

>>>

>>> print("Hello, my name is %s." % "Graham")
Hello, my name is Graham.

Notice also that string modulo operation isn’t only for printing. You can also format values and assign them to another string variable:

Read the full article at https://realpython.com/python-modulo-string-formatting/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

January 24, 2022 02:00 PM UTC

Podcast.__init__

Improve Your Productivity By Investing In Developer Experience Design For Your Projects

When we are creating applications we spend a significant amount of effort on optimizing the experience of our end users to ensure that they are able to complete the tasks that the system is intended for. A similar effort that we should all consider is optimizing the developer experience for ourselves and other engineers who contribute to the projects that we work on. Adam Johnson recently wrote a book on how to improve the developer experience for Django projects and in this episode he shares some of the insights that he has gained through that project and his work with clients to help you improve the experience that you and your team have when collaborating on software development.

January 24, 2022 11:30 AM UTC

Zato Blog

Remote API debugging with VS Code

• Each Zato environment ships with default configuration that lets its servers be started from a Visual Studio Code’s debugger

• Servers started in this way can run remotely, e.g. your local IDE may be on Mac or Windows while your Zato servers will be in a remote Linux instance. This will still let you debug your services deployed remotely.

• It does not matter whether the server is under Docker or if it runs in a Linux VM directly

• This article contains step-by-step instructions on how to start a Zato server in such a manner and how to debug your code remotely from VS Code

Prerequisites

• Ensure that your remote Zato server runs in a Linux system that has at least 2 CPUs and 2 GB of RAM. If you use AWS, a medium instance will be the correct one to choose.

• Make sure that there is SSH connectivity between your localhost and the remote Linux server, that is, you can ssh into the system where the Zato server is. As a reminder, if you use Docker Quickstart, the default port is 22022.

• If the server runs in a Docker Quickstart instance, there are no further prerequisites and you can skip to the next section

• If you created a Zato cluster yourself, check if these two files exist:

  /path/to/your/environment/.vscode/launch.json
  /path/to/your/environment/.vscode/settings.json
  

• If the files do not exist, download them here: launch.json and settings.json

• Once downloaded, save them to the locations above; if the .vscode directory does not exist, create it. The end result should be that if, for instance, your environment is in /opt/zato/env/dev, the files will go to /opt/zato/env/dev/.vscode.

SSH connections in VS Code

• In VS Code, install an extension called Remote - SSH

• After you install it, there will be a new button available in the bottom-left hand side corner of your IDE. The button will let you open SSH connections. Click it and then click “Connect to Host”

• Choose the host where your Zato server is and the IDE will open a new window to connect to that host using SSH. Enter SSH credentials if necessary. Note that you will be in a new IDE window now.

Opening a remote environment

• Once you are connected in a new window, choose “Open Folder”, select the directory where your environment is and click OK

• Under Docker Quickstart, the path will be /opt/zato/env/qs-1/. In other environments, navigate to your environment’s path accordingly.

• After opening the remote directory with a Zato environment, your IDE window will look like below:

Starting the server

• Click the Run and Debug icon:

• Click the play icon next to the Remote Zato Main option:

• The IDE will work now on installing all of its own components in the remote Linux server - that may take a couple of minutes the first time around. The process is CPU-intensive which is why 2 CPUs are a prerequisite.

• If VS Code tells you that its Python IDE extension is not installed in the remote SSH system, choose to install it over SSH. This may also take a couple of minutes.

• Allow for some time until the IDE completes the installation of its remote components - there will be feedback provided in the IDE window’s footer. Once they are installed, proceed to the next section below.

Deploying a test service

• Save the code below as demo.py and hot-deploy it in the now-already-started remote server. Note the highlighted line, we are going to add a breakpoint to it soon.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):
    def handle(self):
        msg = 'Hello, I am a demo service'
        self.logger.info(msg)

Debugging a service

• In the IDE, find the deployed service on a remote server under the path of /path/to/server/work/hot-deploy/current/demo.py and open it:

• Add a breakpoint in line 9, as indicated below:

• Invoke the service in any way you prefer, e.g. through REST, Dashboard or from command line

• The debugger will stop at line 9, showing the local variables, the call stack and other details, exactly as if it was a local server

Congratulations! This concludes the process, everything is set up, you can debug your Zato API services remotely now.

Next steps

• Start the tutorial to learn how to integrate APIs and build systems. After completing it, you will have a multi-protocol service representing a sample scenario often seen in banking systems with several applications cooperating to provide a single and consistent API to its callers.

• Visit the support page if you need assistance.

• Para aprender más sobre las integraciones de Zato y API en español, haga clic aquí

• Pour en savoir plus sur les intégrations API avec Zato en français, cliquez ici

January 24, 2022 09:31 AM UTC

IslandT

Starting a python language chess game project

Hello everyone, this is a new chess game project which I am going to start to create and update the code on this website weekly. In this first chapter of the project report, I am going to 1) render out the chessboard 2) write a code to recognize the square which I am touching on 3) Put a pawn on the board.

Without further due, let me start by explaining to you first what this project is all about and what is the final outcome of the project.

The reason I have started this project is that I want to create a chess game application that I can play with as a training exercise for myself to sharpen my chess skill. Stockfish is the chess engine that runs this application and the python code will use one of the Stockfish wrappers written in python to communicate with the Stockfish chess engine. Pygame will be used to create the chess game interface as well as to move pieces along the chessboard. The finished project will contain a common interface together with the chessboard and possibly also includes the AI and analysis part later on but not in this project.

The below python program will perform the above three goals, first, render the chessboard, then prints out the square on the command line and highlight the square area (I am using pycharm to write this program but you can certainly use another editor if you have one) each time the user pressed on one of the squares, and finally, put a pawn on one of the squares on the chessboard.

import sys, pygame
import math

pygame.init()

size = width, height = 512, 512
white = 255, 178, 102
black = 126, 126, 126
hightlight = 192, 192, 192
title = "IslandT Chess"

width = 64 # width of the square
original_color = ''

#empty chess dictionary

chess_dict = {}

#chess square list
chess_square_list = [
                    "a8", "b8", "c8", "d8", "e8", "f8", "g8", "h8",
                    "a7", "b7", "c7", "d7", "e7", "f7", "g7", "h7",
                    "a6", "b6", "c6", "d6", "e6", "f6", "g6", "h6",
                    "a5", "b5", "c5", "d5", "e5", "f5", "g5", "h5",
                    "a4", "b4", "c4", "d4", "e4", "f4", "g4", "h4",
                    "a3", "b3", "c3", "d3", "e3", "f3", "g3", "h3",
                    "a2", "b2", "c2", "d2", "e2", "f2", "g2", "h2",
                    "a1", "b1", "c1", "d1", "e1", "f1", "g1", "h1"
                    ]

# chess square position
chess_square_position = []

#pawn image
pawn0 = pygame.image.load("pawn.png")

# create a list to map name of column and row
for i in range(0, 8) : # control row
    for j in range(0, 8): # control column
        chess_square_position.append((j * width, i * width))

# create a dictionary to map the name of column and row

for n in range(0, len(chess_square_position)):
    chess_dict[chess_square_list[n]] = chess_square_position[n]

screen = pygame.display.set_mode(size)
pygame.display.set_caption(title)

rect_list = list() # this is the list of brown rectangle

# used this loop to create a list of brown rectangles
for i in range(0, 8): # control the row
    for j in range(0, 8): # control the column
        if i % 2 == 0: # which means it is an even row
            if j % 2 != 0: # which means it is an odd column
                rect_list.append(pygame.Rect(j * width, i * width, width, width))
        else:
            if j % 2 == 0: # which means it is an even column
                rect_list.append(pygame.Rect(j * width, i * width, width, width))


# create main surface and fill the base color with light brown color
chess_board_surface = pygame.Surface(size)
chess_board_surface.fill(white)

# next draws the dark brown rectangles on the chessboard surface
for chess_rect in rect_list:
    pygame.draw.rect(chess_board_surface, black, chess_rect)

while True:
    for event in pygame.event.get():

        if event.type == pygame.QUIT: sys.exit()
        elif event.type == pygame.MOUSEBUTTONDOWN:

            pos = event.pos
            x = math.floor(pos[0] / width)
            y = math.floor(pos[1] / width)

            # print the square name which you have clicked on
            for key, value in chess_dict.items():
                if (x * width, y * width) == (value[0],value[1]):
                    print(key)

            original_color = chess_board_surface.get_at((x * width, y * width ))
            pygame.draw.rect(chess_board_surface, hightlight, pygame.Rect((x) * width, (y) * width, 64, 64))
        elif event.type == pygame.MOUSEBUTTONUP:
            pos = event.pos
            x = math.floor(pos[0] / width)
            y = math.floor(pos[1] / width)
            pygame.draw.rect(chess_board_surface, original_color, pygame.Rect((x) * width, (y) * width, 64, 64))

    # displayed the chess surface
    screen.blit(chess_board_surface, (0, 0))
    screen.blit(pawn0, (0, 64)) # just testing...
    pygame.display.update()

The result is as follows…

At the moment only one piece of pawn is on the chessboard and our next goal is to move that piece along the board which is going to happen in the next coming chapter!

If you are interested in another topic besides programming then do visit my blog and become my friend on Blogspot.

January 24, 2022 07:41 AM UTC

Mike Driscoll

PyDev of the Week: Julian Sequeira

This week we welcome Julian Sequeira (@juliansequeira) as our PyDev of the Week! Julian is one of the co-founders of PyBites. They post articles, courses, run a podcast and have a fun Code Challenge site too.

You can connect with Julian on LinkedIn if you'd like to. Now let's spend some time getting to know him better!

Can you tell us a little about yourself (hobbies, education, etc):

I'm Julian Sequeira, an entrepreneur obsessed with Python and everything Mindset related. I've spent my life in technology and am currently the Co-Founder of PyBites as well as a Program Manager at Amazon Web Services. Off the books, I love to tinker with tech, play the guitar, spend time with my kids and dive into a casual video game or three.

Why did you start using Python?

I started using Python when I was working as a Field Engineer at Oracle in 2016. By then I'd already befriended my now best mate and business partner, Bob Belderbos, who I was looking to partner with on a project. At the same time, I had the need to create a simple app to track the over time I was doing as part of the day job.

While using Python to create the app, I found myself wanting and needing to take notes on the concepts I was learning (virtual environments blew my mind!). This led to Bob and I deciding to create a blog so we could both learn Python and record what we learned at the same time. PyBites was born, the Python learning continued, all manner of material was created and here we are today, with PyBites as our very own company.

Oh and yes, I finished creating the overtime tracker and while I found no discrepancies in *my* pay, it found some gaps in someone else's! Win!

What other programming languages do you know and which is your favorite?

This is a tough one! I've spent the vast majority of my time on Python but do have some rudimentary Javascript and C++ experience. C++ holds a special (painful) place in my heart as it was the very first programming language I learned back in high school. I remember making a  text-based Black Jack game which I was super proud of. Given the relative complexities, I still feel like a champion just writing "#include <iostream>".

What projects are you working on now?

Not as many projects as I'd like due to moving houses but I've just started working on a Python app with my son so we can categorise and record his Pokemon card collection. We're using the PyBites Developer Mindset approach and starting with an MVP. It's slow going as I'm teaching him Python at the same time but it's not the speed that counts, we're just enjoying the journey and every little win along the way.

Outside of that, there are always projects in the works for PyBites. I'm not coding anything right now but am working on some content for schools that are using our Python exercises platform to teach students how to code. This is what gets me up in the morning!

Which Python libraries are your favorite (core or 3rd party)?

The kind with books! ... Anyone?

Fine! While I'm not an expert with it in any way, I've come to appreciate OpenCV. To me it just unlocks so much from a creative perspective and allows me to take dreams I've had my entire life and make them a reality. I remember following a tutorial from Adrian Rosebrock to create my own OpenCV based Pokedex. It blew my mind and really reminded me that Python can be used not just to create the usual "serious" app but to set us free and really use our imaginations to make the world a brighter place.

What were we talking about? Oh yeah, OpenCV would be it!

How is the PyBites Podcast going? How is that different from the other things you do at PyBites?

The PyBites Podcast is going strong! It has to be the most fun we have second only to working with our clients. As we record each episode, I honestly forget that it's being recorded and just enjoy the conversation. It allows me to share openly and honestly without feeling tied down by a script. I feel like this sets it apart from the rest of the things we do at PyBites. It's candid, light and raw and allows listeners to hear us at our best.

Also, while PyBites can be quite technical on the blog, the podcast tends to lean toward the mindset side of things. Mindset topics can be dry in text form but when discussed over audio people can hear the passion and enthusiasm we have regarding the theme of the episode.

Is there anything else you’d like to say?

As you may or may not know, Bob and I coach people through their Python journey. One thing that comes up over and over is how important it is to have a strong mindset as you push for your Python goals. It's not enough to just code. You have to learn to effectively work with the successes *and* the failures, how to work with others, how to take criticism, deal with tutorial paralysis and imposter syndrome... the list goes on.

Ultimately, you need to find a balance between the mindset and the tech skills. This is when real progress is made.

And to wrap it up, it may sound lame but the reality is that it starts and ends with you. Trust in yourself and your abilities and go get it.

Thanks for doing the interview, Julian!

Related Articles

• PyDev of the Week: Bob Belderbos

• PyDev of the Week: Rodrigo Girão Serrão

• PyDev of the Week: Lais Carvalho

• PyDev of the Week: Martin Manns

The post PyDev of the Week: Julian Sequeira appeared first on Mouse Vs Python.

January 24, 2022 06:05 AM UTC

James Bennett

For hire

As I write this it’s the evening of January 23, 2022. A little over two weeks ago I gave notice at my now-former employer, and as of two days ago I am officially on the job market.

If you already know me and are interested in talking about an opportunity, please get in touch. Or if you want to know a bit more first, read on…

Who I am

It’s a bit tricky to pin down when …

Read full entry

January 24, 2022 01:51 AM UTC

January 23, 2022

Anarcat

Switching from OpenNTPd to Chrony

A friend recently reminded me of the existence of chrony, a "versatile implementation of the Network Time Protocol (NTP)". The excellent introduction is worth quoting in full:

It can synchronise the system clock with NTP servers, reference clocks (e.g. GPS receiver), and manual input using wristwatch and keyboard. It can also operate as an NTPv4 (RFC 5905) server and peer to provide a time service to other computers in the network.

It is designed to perform well in a wide range of conditions, including intermittent network connections, heavily congested networks, changing temperatures (ordinary computer clocks are sensitive to temperature), and systems that do not run continuosly, or run on a virtual machine.

Typical accuracy between two machines synchronised over the Internet is within a few milliseconds; on a LAN, accuracy is typically in tens of microseconds. With hardware timestamping, or a hardware reference clock, sub-microsecond accuracy may be possible.

Now that's already great documentation right there. What it is, why it's good, and what to expect from it. I want more. They have a very handy comparison table between chrony, ntp and openntpd.

My problem with OpenNTPd

Following concerns surrounding the security (and complexity) of the venerable ntp program, I have, a long time ago, switched to using openntpd on all my computers. I hadn't thought about it until I recently noticed a lot of noise on one of my servers:

jan 18 10:09:49 curie ntpd[1069]: adjusting local clock by -1.604366s
jan 18 10:08:18 curie ntpd[1069]: adjusting local clock by -1.577608s
jan 18 10:05:02 curie ntpd[1069]: adjusting local clock by -1.574683s
jan 18 10:04:00 curie ntpd[1069]: adjusting local clock by -1.573240s
jan 18 10:02:26 curie ntpd[1069]: adjusting local clock by -1.569592s

You read that right, openntpd was constantly rewinding the clock, sometimes in less than two minutes. The above log was taken while doing diagnostics, looking at the last 30 minutes of logs. So, on average, one 1.5 seconds rewind per 6 minutes!

That might be due to a dying real time clock (RTC) or some other hardware problem. I know for a fact that the CMOS battery on that computer (curie) died and I wasn't able to replace it (!). So that's partly garbage-in, garbage-out here. But still, I was curious to see how chrony would behave... (Spoiler: much better.)

But I also had trouble on another workstation, that one a much more recent machine (angela). First, it seems OpenNTPd would just fail at boot time:

anarcat@angela:~(main)$ sudo systemctl status openntpd
● openntpd.service - OpenNTPd Network Time Protocol
     Loaded: loaded (/lib/systemd/system/openntpd.service; enabled; vendor pres>
     Active: inactive (dead) since Sun 2022-01-23 09:54:03 EST; 6h ago
       Docs: man:openntpd(8)
    Process: 3291 ExecStartPre=/usr/sbin/ntpd -n $DAEMON_OPTS (code=exited, sta>
    Process: 3294 ExecStart=/usr/sbin/ntpd $DAEMON_OPTS (code=exited, status=0/>
   Main PID: 3298 (code=exited, status=0/SUCCESS)
        CPU: 34ms

jan 23 09:54:03 angela systemd[1]: Starting OpenNTPd Network Time Protocol...
jan 23 09:54:03 angela ntpd[3291]: configuration OK
jan 23 09:54:03 angela ntpd[3297]: ntp engine ready
jan 23 09:54:03 angela ntpd[3297]: ntp: recvfrom: Permission denied
jan 23 09:54:03 angela ntpd[3294]: Terminating
jan 23 09:54:03 angela systemd[1]: Started OpenNTPd Network Time Protocol.
jan 23 09:54:03 angela systemd[1]: openntpd.service: Succeeded.

After a restart, somehow it worked, but it took a long time to sync the clock. At first, it would just not consider any peer at all:

anarcat@angela:~(main)$ sudo ntpctl -s all
0/20 peers valid, clock unsynced

peer
   wt tl st  next  poll          offset       delay      jitter
159.203.8.72 from pool 0.debian.pool.ntp.org
    1  5  2    6s    6s             ---- peer not valid ----
138.197.135.239 from pool 0.debian.pool.ntp.org
    1  5  2    6s    7s             ---- peer not valid ----
216.197.156.83 from pool 0.debian.pool.ntp.org
    1  4  1    2s    9s             ---- peer not valid ----
142.114.187.107 from pool 0.debian.pool.ntp.org
    1  5  2    5s    6s             ---- peer not valid ----
216.6.2.70 from pool 1.debian.pool.ntp.org
    1  4  2    2s    8s             ---- peer not valid ----
207.34.49.172 from pool 1.debian.pool.ntp.org
    1  4  2    0s    5s             ---- peer not valid ----
198.27.76.102 from pool 1.debian.pool.ntp.org
    1  5  2    5s    5s             ---- peer not valid ----
158.69.254.196 from pool 1.debian.pool.ntp.org
    1  4  3    1s    6s             ---- peer not valid ----
149.56.121.16 from pool 2.debian.pool.ntp.org
    1  4  2    5s    9s             ---- peer not valid ----
162.159.200.123 from pool 2.debian.pool.ntp.org
    1  4  3    1s    6s             ---- peer not valid ----
206.108.0.131 from pool 2.debian.pool.ntp.org
    1  4  1    6s    9s             ---- peer not valid ----
205.206.70.40 from pool 2.debian.pool.ntp.org
    1  5  2    8s    9s             ---- peer not valid ----
2001:678:8::123 from pool 2.debian.pool.ntp.org
    1  4  2    5s    9s             ---- peer not valid ----
2606:4700:f1::1 from pool 2.debian.pool.ntp.org
    1  4  3    2s    6s             ---- peer not valid ----
2607:5300:205:200::1991 from pool 2.debian.pool.ntp.org
    1  4  2    5s    9s             ---- peer not valid ----
2607:5300:201:3100::345c from pool 2.debian.pool.ntp.org
    1  4  4    1s    6s             ---- peer not valid ----
209.115.181.110 from pool 3.debian.pool.ntp.org
    1  5  2    5s    6s             ---- peer not valid ----
205.206.70.42 from pool 3.debian.pool.ntp.org
    1  4  2    0s    6s             ---- peer not valid ----
68.69.221.61 from pool 3.debian.pool.ntp.org
    1  4  1    2s    9s             ---- peer not valid ----
162.159.200.1 from pool 3.debian.pool.ntp.org
    1  4  3    4s    7s             ---- peer not valid ----

Then it would accept them, but still wouldn't sync the clock:

anarcat@angela:~(main)$ sudo ntpctl -s all
20/20 peers valid, clock unsynced

peer
   wt tl st  next  poll          offset       delay      jitter
159.203.8.72 from pool 0.debian.pool.ntp.org
    1  8  2    5s    6s         0.672ms    13.507ms     0.442ms
138.197.135.239 from pool 0.debian.pool.ntp.org
    1  7  2    4s    8s         1.260ms    13.388ms     0.494ms
216.197.156.83 from pool 0.debian.pool.ntp.org
    1  7  1    3s    5s        -0.390ms    47.641ms     1.537ms
142.114.187.107 from pool 0.debian.pool.ntp.org
    1  7  2    1s    6s        -0.573ms    15.012ms     1.845ms
216.6.2.70 from pool 1.debian.pool.ntp.org
    1  7  2    3s    8s        -0.178ms    21.691ms     1.807ms
207.34.49.172 from pool 1.debian.pool.ntp.org
    1  7  2    4s    8s        -5.742ms    70.040ms     1.656ms
198.27.76.102 from pool 1.debian.pool.ntp.org
    1  7  2    0s    7s         0.170ms    21.035ms     1.914ms
158.69.254.196 from pool 1.debian.pool.ntp.org
    1  7  3    5s    8s        -2.626ms    20.862ms     2.032ms
149.56.121.16 from pool 2.debian.pool.ntp.org
    1  7  2    6s    8s         0.123ms    20.758ms     2.248ms
162.159.200.123 from pool 2.debian.pool.ntp.org
    1  8  3    4s    5s         2.043ms    14.138ms     1.675ms
206.108.0.131 from pool 2.debian.pool.ntp.org
    1  6  1    0s    7s        -0.027ms    14.189ms     2.206ms
205.206.70.40 from pool 2.debian.pool.ntp.org
    1  7  2    1s    5s        -1.777ms    53.459ms     1.865ms
2001:678:8::123 from pool 2.debian.pool.ntp.org
    1  6  2    1s    8s         0.195ms    14.572ms     2.624ms
2606:4700:f1::1 from pool 2.debian.pool.ntp.org
    1  7  3    6s    9s         2.068ms    14.102ms     1.767ms
2607:5300:205:200::1991 from pool 2.debian.pool.ntp.org
    1  6  2    4s    9s         0.254ms    21.471ms     2.120ms
2607:5300:201:3100::345c from pool 2.debian.pool.ntp.org
    1  7  4    5s    9s        -1.706ms    21.030ms     1.849ms
209.115.181.110 from pool 3.debian.pool.ntp.org
    1  7  2    0s    7s         8.907ms    75.070ms     2.095ms
205.206.70.42 from pool 3.debian.pool.ntp.org
    1  7  2    6s    9s        -1.729ms    53.823ms     2.193ms
68.69.221.61 from pool 3.debian.pool.ntp.org
    1  7  1    1s    7s        -1.265ms    46.355ms     4.171ms
162.159.200.1 from pool 3.debian.pool.ntp.org
    1  7  3    4s    8s         1.732ms    35.792ms     2.228ms

It took a solid five minutes to sync the clock, even though the peers were considered valid within a few seconds:

jan 23 15:58:41 angela systemd[1]: Started OpenNTPd Network Time Protocol.
jan 23 15:58:58 angela ntpd[84086]: peer 142.114.187.107 now valid
jan 23 15:58:58 angela ntpd[84086]: peer 198.27.76.102 now valid
jan 23 15:58:58 angela ntpd[84086]: peer 207.34.49.172 now valid
jan 23 15:58:58 angela ntpd[84086]: peer 209.115.181.110 now valid
jan 23 15:58:59 angela ntpd[84086]: peer 159.203.8.72 now valid
jan 23 15:58:59 angela ntpd[84086]: peer 138.197.135.239 now valid
jan 23 15:58:59 angela ntpd[84086]: peer 162.159.200.123 now valid
jan 23 15:58:59 angela ntpd[84086]: peer 2607:5300:201:3100::345c now valid
jan 23 15:59:00 angela ntpd[84086]: peer 2606:4700:f1::1 now valid
jan 23 15:59:00 angela ntpd[84086]: peer 158.69.254.196 now valid
jan 23 15:59:01 angela ntpd[84086]: peer 216.6.2.70 now valid
jan 23 15:59:01 angela ntpd[84086]: peer 68.69.221.61 now valid
jan 23 15:59:01 angela ntpd[84086]: peer 205.206.70.40 now valid
jan 23 15:59:01 angela ntpd[84086]: peer 205.206.70.42 now valid
jan 23 15:59:02 angela ntpd[84086]: peer 162.159.200.1 now valid
jan 23 15:59:04 angela ntpd[84086]: peer 216.197.156.83 now valid
jan 23 15:59:05 angela ntpd[84086]: peer 206.108.0.131 now valid
jan 23 15:59:05 angela ntpd[84086]: peer 2001:678:8::123 now valid
jan 23 15:59:05 angela ntpd[84086]: peer 149.56.121.16 now valid
jan 23 15:59:07 angela ntpd[84086]: peer 2607:5300:205:200::1991 now valid
jan 23 16:03:47 angela ntpd[84086]: clock is now synced

That seems kind of odd. It was also frustrating to have very little information from ntpctl about the state of the daemon. I understand it's designed to be minimal, but it could inform me on his known offset, for example. It does tell me about the offset with the different peers, but not as clearly as one would expect. It's also unclear how it disciplines the RTC at all.

Compared to chrony

Now compare with chrony:

jan 23 16:07:16 angela systemd[1]: Starting chrony, an NTP client/server...
jan 23 16:07:16 angela chronyd[87765]: chronyd version 4.0 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBUG)
jan 23 16:07:16 angela chronyd[87765]: Initial frequency 3.814 ppm
jan 23 16:07:16 angela chronyd[87765]: Using right/UTC timezone to obtain leap second data
jan 23 16:07:16 angela chronyd[87765]: Loaded seccomp filter
jan 23 16:07:16 angela systemd[1]: Started chrony, an NTP client/server.
jan 23 16:07:21 angela chronyd[87765]: Selected source 206.108.0.131 (2.debian.pool.ntp.org)
jan 23 16:07:21 angela chronyd[87765]: System clock TAI offset set to 37 seconds

First, you'll notice there's none of that "clock synced" nonsense, it picks a source, and then... it's just done. Because the clock on this computer is not drifting that much, and openntpd had (presumably) just sync'd it anyways. And indeed, if we look at detailed stats from the powerful chronyc client:

anarcat@angela:~(main)$ sudo chronyc tracking
Reference ID    : CE6C0083 (ntp1.torix.ca)
Stratum         : 2
Ref time (UTC)  : Sun Jan 23 21:07:21 2022
System time     : 0.000000311 seconds slow of NTP time
Last offset     : +0.000807989 seconds
RMS offset      : 0.000807989 seconds
Frequency       : 3.814 ppm fast
Residual freq   : -24.434 ppm
Skew            : 1000000.000 ppm
Root delay      : 0.013200894 seconds
Root dispersion : 65.357254028 seconds
Update interval : 1.4 seconds
Leap status     : Normal

We see that we are nanoseconds away from NTP time. That was ran very quickly after starting the server (literally in the same second as chrony picked a source), so stats are a bit weird (e.g. the Skew is huge). After a minute or two, it looks more reasonable:

Reference ID    : CE6C0083 (ntp1.torix.ca)
Stratum         : 2
Ref time (UTC)  : Sun Jan 23 21:09:32 2022
System time     : 0.000487002 seconds slow of NTP time
Last offset     : -0.000332960 seconds
RMS offset      : 0.000751204 seconds
Frequency       : 3.536 ppm fast
Residual freq   : +0.016 ppm
Skew            : 3.707 ppm
Root delay      : 0.013363549 seconds
Root dispersion : 0.000324015 seconds
Update interval : 65.0 seconds
Leap status     : Normal

Now it's learning how good or bad the RTC clock is ("Frequency"), and is smoothly adjusting the System time to follow the average offset (RMS offset, more or less). You'll also notice the Update interval has risen, and will keep expanding as chrony learns more about the internal clock, so it doesn't need to constantly poll the NTP servers to sync the clock. In the above, we're 487 micro seconds (less than a milisecond!) away from NTP time.

(People interested in the explanation of every single one of those fields can read the excellent chronyc manpage. That thing made me want to nerd out on NTP again!)

On the machine with the bad clock, chrony also did a 1.5 second adjustment, but just once, at startup:

jan 18 11:54:33 curie chronyd[2148399]: Selected source 206.108.0.133 (2.debian.pool.ntp.org) 
jan 18 11:54:33 curie chronyd[2148399]: System clock wrong by -1.606546 seconds 
jan 18 11:54:31 curie chronyd[2148399]: System clock was stepped by -1.606546 seconds 
jan 18 11:54:31 curie chronyd[2148399]: System clock TAI offset set to 37 seconds 

Then it would still struggle to keep the clock in sync, but not as badly as openntpd. Here's the offset a few minutes after that above startup:

System time     : 0.000375352 seconds slow of NTP time

And again a few seconds later:

System time     : 0.001793046 seconds slow of NTP time

I don't currently have access to that machine, and will update this post with the latest status, but so far I've had a very good experience with chrony on that machine, which is a testament to its resilience, and it also just works on my other machines as well.

Extras

On top of "just working" (as demonstrated above), I feel that chrony's feature set is so much superior... Here's an excerpt of the extras in chrony, taken from comparison table:

• source frequency tracking
• source state restore from file
• temperature compensation
• ready for next NTP era (year 2036)
• replace unreachable / falseticker servers
• aware of jitter
• RTC drift tracking
• RTC trimming
• Restore time from file w/o RTC
• leap seconds correction, in slew mode
• drops root privileges

I even understand some of that stuff. I think.

So kudos to the chrony folks, I'm switching.

Caveats

One thing to keep in mind in the above, however is that it's quite possible chrony does as bad of a job as openntpd on that old machine, and just doesn't tell me about it. For example, here's another log sample from another server (marcos):

jan 23 11:13:25 marcos ntpd[1976694]: adjusting clock frequency by 0.451035 to -16.420273ppm

I get those basically every day, which seems to show that it's at least trying to keep track of the hardware clock.

In other words, it's quite possible I have no idea what I'm talking about and you definitely need to take this article with a grain of salt. I'm not an NTP expert.

Switching to chrony

Because the default configuration in chrony (at least as shipped in Debian) is sane (good default peers, no open network by default), installing it is as simple as:

apt install chrony

And because it somehow conflicts with openntpd, that also takes care of removing that cruft as well.

January 23, 2022 09:55 PM UTC

ItsMyCode

How to Import CSV Files into R?

A comma-separated values (CSV) file is a delimited text file that uses a comma to separate the values. CSV files are popular formats for storing tabular data, i.e. data is composed of rows and columns.

In this article, we will learn how to import CSV files into R with the help of examples.

Importing CSV Files in R

There are 3 popular methods available to import CSV files into R. 

• Using read.csv() method
• Using read_csv() method
• Using fread() method

In this tutorial, we will explore all the 3 methods and see how we can import the CSV file.

Using read.csv() method

The read.csv() method is used to import a CSV file, and it is best suitable for the small CSV files.

The contents of the CSV files are stored into a variable for further manipulation. We can even import multiple CSV files and store them into different variables.

The output returned will be in the format of DataFrame, where row numbers are assigned with integers.

Syntax: 

read.csv(path, header = TRUE, sep = “,”)

Arguments: 

• path: CSV file path that needs to be imported.
• header: Indicates whether to import headers in CSV. By default, it is set to TRUE.
• sep: the field separator character

R often uses a concept of factors to re-encode strings. Hence it is recommended to set stringsAsFactors=FALSE so that R doesn’t convert character or categorical variables into factors.

# read the data from the CSV file
data <- read.csv("C:\\Personal\\IMS\\cricket_points.csv", header=TRUE)

# print the data variable (outputs as DataFrame)
data

Output

      ï..Teams Wins Lose Points
1        India    5     2     10
2 South Africa    3     4      6
3  West Indies    1     6      2
4      England    2     4      4
5    Australia    4     2      8
6  New Zealand    2     5      4

Method 2: Using read_csv() method

The read_csv() method is the most recommended way of reading the CSV file in R. It reads a CSV file one line at a time. 

The data is read in the form of Tibble, and only 10 rows are displayed at once, and the rest are available after expanding.

It also displays the percentage of the file read into the system making it more robust when compared to the read.csv() method.

If you are working with large CSV files, it’s recommended to use the read_csv() method. 

Syntax:

read_csv (path , col_names , n_max , col_types , progress )

Arguments : 

• path: CSV file path that needs to be imported.
• col_names: Indicates whether to import headers in CSV. By default, it is set to TRUE.
• n_max: The maximum number of rows to read.
• col_types: If any column succumbs to NULL, then the col_types can be specified in a compact string format.
• progress: A progress meter to analyse the percentage of files read into the system

# import data.table library 
library(data.table)

#import data
data2 <- read_csv("C:\\Personal\\IMS\\cricket_points.csv")

Output

      ï..Teams Wins Lose Points
1        India    5     2     10
2 South Africa    3     4      6
3  West Indies    1     6      2
4      England    2     4      4
5    Australia    4     2      8
6  New Zealand    2     5      4

Method 3: Using fread() method

If the CSV files are extremely large, the best way to import into R is using the fread() method from the data.table package.

The output of the data will be in the form of Data table in this case.

# import data.table library 
library(data.table)

# read the CSV file
data3 <- fread("C:\\Personal\\IMS\\cricket_points.csv")

          Teams Wins Lose Points
1:        India    5     2     10
2: South Africa    3     4      6
3:  West Indies    1     6      2
4:      England    2     4      4
5:    Australia    4     2      8
6:  New Zealand    2     5      4

Note: It is recommended to use double backlashes (\\) while providing the file path. Else you may get below error.

Error: '\U' used without hex digits in character string starting ""C:\U"

January 23, 2022 07:46 PM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Brasil
  • Planet Python Poland
• Python Libraries
  • OLPC
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Mathspp Pydon'ts"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Ahmed Bouchefra
  • Al-Ahmadgaid Asaad
  • Alec Munro
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • Armin Ronacher
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Automating OSINT
  • Awesome Python Applications
  • Baiju Muthukadan
  • Bajusz Tamás
  • BeDjango
  • Ben Bass
  • Ben Cook
  • Ben Rousch
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • BioPython News
  • Bit of Cheese
  • Bojan Mihelac
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Okken
  • Bruno Oliveira
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Düvel
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Casey Duncan
  • Catherine Devlin
  • Ching-Hwa Yu
  • Chris Hager
  • Chris Miles
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Heimes
  • Christian Scholz
  • Christoph Zwerschke
  • CodeGrades
  • CodeSnipers
  • Codementor
  • CodersLegacy
  • Coding Diet
  • Corey Gallon
  • Corey Goldberg
  • Cormoran Project
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dallas Fort Worth Pythoneers
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Data Community DC
  • Data School
  • Dave Beazley
  • David Amos
  • David Caron
  • David Goodger
  • David Lindelof
  • David MacIver
  • David Malcolm
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • Denis Kurov
  • Django Weblog
  • Django Weekly
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Dougal Matthews
  • Duncan McGreggor
  • Ed Crewe
  • Edward K. Ream
  • Eli Bendersky
  • Eniram Ltd.
  • Eray Özkural (examachine)
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Evennia
  • Everyday Superpowers
  • Fabio Zadrozny
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • Gocept Weblog
  • Godson Gera
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Gustavo Narea
  • Gustavo Niemeyer
  • Gökhan Sever
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Humberto Rocha
  • Hynek Schlawack
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Inspired Python
  • Ionel Cristian Maries
  • IronPython-URLs
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • ItsMyCode
  • Ivan Velichko
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • Janusworx
  • Jarrod Millman
  • Jean-Paul Calderone
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Burns
  • John Cook
  • John Jacobsen
  • John Ludhi/nbshare.io
  • Jon Parise
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan Street
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Julien Danjou
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kodnito
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • LAAC Technology
  • Laurent Szyster
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lintel Technologies
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Lucas Cimon
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mats Kindahl
  • Matt Layman
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Matthew Wright
  • Mattias Brändström
  • Mauveweb
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Nelson
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mirek Długosz
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Naomi Ceder
  • Natan Zabkar
  • Ned Batchelder
  • Neil Schemenauer
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • Nsukami Patrick
  • Obey the Testing Goat
  • Ofosos
  • Omaha Python Users Group
  • Ondřej Čertík
  • Paolo Amoroso
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patrick Kennedy
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • Pedro Lima
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyATL Bytecode
  • PyBites
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyPy
  • PyTennessee
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python GUIs
  • Python Insider
  • Python Morsels
  • Python Piedmont Triad User Group
  • Python Pool
  • Python Software Foundation
  • Python Sweetness
  • Python User Groups
  • Python for Beginners
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • Pythonicity
  • Pythonology
  • Python⇒Speed
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Rene Dudfield
  • Reuven Lerner
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rob Galanakis
  • Robert Collins
  • Robert Zaremba
  • Robin Parmar
  • Robin Wilson
  • Rodrigo Araúj
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Sandipan Dey
  • Sandro Tosi
  • Sean McGrath
  • Sebastian Pölsterl
  • Sebastian Witowski
  • Selena Deckelmann
  • Senthil Kumaran
  • Shannon -jj Behrens
  • ShiningPanda
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Holden
  • Steven Klass
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Suresh Dasari/Tutlane.com
  • Swisscom ICT
  • Talk Python to Me
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Terri Oda
  • Terry Jones
  • Test and Code
  • TestDriven.io
  • The Data Scientist
  • The Digital Cat
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Open Sourcerer
  • The Parcon Blog
  • The Python Coding Blog
  • The Python Papers
  • The Three of Wands
  • Thibauld Nion
  • ThisHosting.Rocks
  • Thomas Guest
  • Thomas Vander Stichele
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Knapp
  • Tim Lesher
  • Tobias Ivarsson
  • Tom Christie
  • Tomasz Ducin
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinayak Mehta
  • Virgil Dupras
  • Vladimir Perić
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • Wing Tips
  • Wingware
  • Wyatt Baldwin
  • Yaniv Aknin
  • Yann Larrivée
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero to Mastery
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • death and gravity
  • eGenix.com
  • hypothesis.works articles
  • kdev-python
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • testmon
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal:
    Open an issue on github