Planet Python

Last update: May 21, 2022 10:41 AM UTC

May 21, 2022

death and gravity

The unreasonable effectiveness of f‍-‍strings and re.VERBOSE

... in which we look at one or two ways to make life easier when working with Python regular expressions.

tl;dr: You can compose verbose regular expressions using f‍-‍strings.

Here's a real-world example – instead of this:

1

pattern = r"((?:\(\s*)?[A-Z]*H\d+[a-z]*(?:\s*\+\s*[A-Z]*H\d+[a-z]*)*(?:\s*[\):+])?)(.*?)(?=(?:\(\s*)?[A-Z]*H\d+[a-z]*(?:\s*\+\s*[A-Z]*H\d+[a-z]*)*(?:\s*[\):+])?(?![^\w\s])|$)"

... do this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

code = r"""
[A-Z]*H  # prefix
\d+      # digits
[a-z]*   # suffix
"""

multicode = fr"""
(?: \( \s* )?               # maybe open paren and maybe space
{code}                      # one code
(?: \s* \+ \s* {code} )*    # maybe followed by other codes, plus-separated
(?: \s* [\):+] )?           # maybe space and maybe close paren or colon or plus
"""

pattern = fr"""
( {multicode} )             # code (capture)
( .*? )                     # message (capture): everything ...
(?=                         # ... up to (but excluding) ...
    {multicode}             # ... the next code
        (?! [^\w\s] )       # (but not when followed by punctuation)
    | $                     # ... or the end
)
"""

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

pattern = r"""
(                       # code (capture)
    # BEGIN multicode

    (?: \( \s* )?       # maybe open paren and maybe space

    # code
    [A-Z]*H  # prefix
    \d+      # digits
    [a-z]*   # suffix

    (?:                 # maybe followed by other codes,
        \s* \+ \s*      # ... plus-separated

        # code
        [A-Z]*H  # prefix
        \d+      # digits
        [a-z]*   # suffix
    )*

    (?: \s* [\):+] )?   # maybe space and maybe close paren or colon or plus

    # END multicode
)

( .*? )                 # message (capture): everything ...

(?=                     # ... up to (but excluding) ...
    # ... the next code

    # BEGIN multicode

    (?: \( \s* )?       # maybe open paren and maybe space

    # code
    [A-Z]*H  # prefix
    \d+      # digits
    [a-z]*   # suffix

    (?:                 # maybe followed by other codes,
        \s* \+ \s*      # ... plus-separated

        # code
        [A-Z]*H  # prefix
        \d+      # digits
        [a-z]*   # suffix
    )*

    (?: \s* [\):+] )?   # maybe space and maybe close paren or colon or plus

    # END multicode

        # (but not when followed by punctuation)
        (?! [^\w\s] )

    # ... or the end
    | $
)

"""

It's better than the non-verbose one, but even with careful formatting and comments, the repetition makes it pretty hard to follow – and wait until you have to change something!

Read on for details and some caveats.

Prerequisites #

Formatted string literals (f‍-‍strings) were added in Python 3.6¹, and provide a way to embed expressions inside string literals, using a syntax similar to that of str.format():

>>> name = "world"
>>>
>>> "Hello, {name}!".format(name=name)
'Hello, world!'
>>>
>>> f"Hello, {name}!"
'Hello, world!'

Verbose regular expressions (re.VERBOSE) have been around since forever², and allow writing regular expressions with non-significant whitespace and comments:

>>> text = "H1 code (AH2b+EUH3) fancy code"
>>>
>>> code = r"[A-Z]*H\d+[a-z]*"
>>> re.findall(code, text)
['H1', 'AH2b', 'EUH3']
>>>
>>> code = r"""
... [A-Z]*H  # prefix
... \d+      # digits
... [a-z]*   # suffix
... """
>>> re.findall(code, text, re.VERBOSE)
['H1', 'AH2b', 'EUH3']

The "one weird trick" #

Once you see it, it's obvious – you can use f‍-‍strings to compose regular expressions:

>>> multicode = fr"""
... (?: \( )?         # maybe open paren
... {code}            # one code
... (?: \+ {code} )*  # maybe other codes, plus-separated
... (?: \) )?         # maybe close paren
... """
>>> re.findall(multicode, text, re.VERBOSE)
['H1', '(AH2b+EUH3)']

It's so obvious, it only took me three years to do it after I started using Python 3.6+, despite using both features during all that time.

Of course, there's any number of libraries for building regular expressions; the benefit of this is that it has zero dependencies, and zero extra things you need to learn.

Caveats #

Hashes and spaces need to be escaped #

Because a hash is used to mark the start of a comment, and spaces are mostly ignored, you have to represent them in some other way.

The documentation of re.VERBOSE is quite helpful:

When a line contains a # that is not in a character class and is not preceded by an unescaped backslash, all characters from the leftmost such # through the end of the line are ignored.

That is, this won't work as the non-verbose version:

>>> re.findall("\d+#\d+", "1#23a")
['1#23']
>>> re.findall("\d+ # \d+", "1#23a", re.VERBOSE)
['1', '23']

... but these will:

>>> re.findall("\d+ [#] \d+", "1#23a", re.VERBOSE)
['1#23']
>>> re.findall("\d+ \# \d+", "1#23a", re.VERBOSE)
['1#23']

The same is true for spaces:

>>> re.findall("\d+ [ ] \d+", "1 23a", re.VERBOSE)
['1 23']
>>> re.findall("\d+ \  \d+", "1 23a", re.VERBOSE)
['1 23']

Hashes need extra care #

When composing regexes, ending a pattern on the same line as a comment might accidentally comment the following line in the enclosing pattern:

>>> one = "1 # comment"
>>> onetwo = f"{one} 2"
>>> re.findall(onetwo, '0123', re.VERBOSE)
['1']
>>> print(onetwo)
1 # comment 2

This can be avoided by always ending the pattern on a new line:

>>> one = """\
... 1 # comment
... """
>>> onetwo = f"""\
... {one} 2
... """
>>> re.findall(onetwo, '0123', re.VERBOSE)
['12']

While a bit cumbersome, in real life most patterns would span multiple lines anyway, so it's not really an issue.

(Note that this is only needed if you use comments.)

Brace quantifiers need to be escaped #

Because f‍-‍strings already use braces for replacements, to represent brace quantifiers you must double the braces:

>>> re.findall("m{2}", "entire mm but only two of mmm")
['mm', 'mm']
>>> letter = "m"
>>> pattern = f"{letter}{{2}}"
>>> re.findall(pattern, "entire mm but only two of mmm")
['mm', 'mm']

I don't control the flags #

Maybe you'd like to use verbose regexes, but don't control the flags passed to the re functions (for example, because you're passing the regex to an API).

Worry not! The regular expression syntax supports inline flags:

(?aiLmsux)

(One or more letters [...]) The group matches the empty string; the letters set the corresponding flags: [...] re.X (verbose), for the entire regular expression. [...] This is useful if you wish to include the flags as part of the regular expression, instead of passing a flag argument to the re.compile() function. Flags should be used first in the expression string.

(?aiLmsux-imsx:...)

[...] The letters set or remove the corresponding flags [...] for the part of the expression. [...]

So, you can do this:

>>> onetwo = """\
... (?x)
... 1 # look, ma
... 2 # no flags
... """
>>> re.findall(onetwo, '0123')
['12']

... or this:

>>> onetwo = """\
... (?x:
...     1 # verbose until the close paren
... )2"""
>>> re.findall(onetwo, '0123')
['12']

That's it for now.

Learned something new today? Share this with others, it really helps!

Want to know when new articles come out? Subscribe here to get new stuff straight to your inbox!

Bonus: I don't use Python #

Lots of other languages support the inline verbose flag, too! You can build a pattern in whichever language is more convenient, and use it in any other one.³ Languges like...

C (with PCRE – and by extension, C++, PHP, and many others):

echo '0123' | pcregrep -o '(?x)
1 2  # such inline
'

char *pattern =
    "(?x)\n"
    "1 2  # much verbose\n"
;
char *subject = "0123";
int subject_length = strlen(subject);

int errornumber;
PCRE2_SIZE erroroffset;

pcre2_code *re = pcre2_compile(
    (PCRE2_SPTR)pattern,
    PCRE2_ZERO_TERMINATED,
    0,
    &errornumber,
    &erroroffset,
    NULL
);

pcre2_match_data *match_data = pcre2_match_data_create_from_pattern(re, NULL);

pcre2_match(
    re,
    (PCRE2_SPTR)subject,
    subject_length,
    0,
    0,
    match_data,
    NULL
);

PCRE2_SIZE *ovector = pcre2_get_ovector_pointer(match_data);
PCRE2_SPTR substring_start = (PCRE2_SPTR)subject + ovector[0];
size_t substring_length = ovector[1] - ovector[0];
printf("%.*s\n", (int)substring_length, (char *)substring_start);

C#:

Console.WriteLine(new Regex(@"(?x)
1 2  # wow
").Match("0123"));

grep (only the GNU one):

echo '0123' | grep -Po '(?x) 1 2  # no line'

Java (and by extension, lots of JVM languages, like Scala):

var p = Pattern.compile(
    "(?x)\n" +
    "1 2  # much class\n"
);
var m = p.matcher("0123");
m.find();
System.out.println(m.group(0));

Perl:

"0123" =~ /(?x)(
1 2  # no scare
)/;
print $1 . "\n";

PostgreSQL:

select substring(
  '0123' from
    $$(?x)
    1 2  # such declarative
    $$
);

Ruby:

puts /(?x)
1 2  # nice
/.match('0123')

Rust:

let re = Regex::new(
    r"(?x)
    1 2  # much safe
    "
).unwrap();
println!("{}", re.find("0123").unwrap().as_str());

Swift:

let string = "0123"
let range = string.range(
    of : """
    (?x)
    1 2  # omg hi
    """,
    options : .regularExpression
)
print(string[range!])

Notable languages that don't support inline verbose flags out of the box:

• C (regex.h – POSIX regular expressions)
• C++ (regex)
• Go (regexp)
• Javascript
• Lua

1. In PEP 498 – Literal String Interpolation. ^[return]

2. That is, since at least Python 1.5.2, released in 1998 – for all except a tiny minority of Python users, that's before forever. ^[return]

3. If they both support inline flags, they likely share most other features. ^[return]

May 21, 2022 08:10 AM UTC

May 20, 2022

Python for Beginners

Check For Sorted List in Python

Lists are one of the most used data structures in python. In this article, we will discuss different ways to check for sorted list in python.

How to Check For a Sorted List in Python?

A sorted list has all its elements in non-decreasing order. In other words, the element at index i is always less than or equal to the element at index i+1 if the list is sorted list. For instance, consider the following lists.

myList1=[1,2,3,4,5,6,7]

myList2=[1,2,3,3,4,5,5]

myList3=[1,2,3,4,2,5,6]

Here, myList1 has all the elements in increasing order if we move from left to right. Hence, it is a sorted list. myList2 doesn’t have its elements in an increasing order but all the elements are in non-decreasing order if we move from left to right. Hence, myList2 is also a sorted list. However, the elements in myList3 are neither in increasing order nor in non-decreasing order. Hence, it is not a sorted list. 

Check For Sorted List in Python

To check for a sorted list, we just have to traverse through the list and check if all the elements are in a non-decreasing order or not. For this, we will use a variable isSorted and a for loop. First, we will initialize the variable isSorted to True assuming that the list is already sorted. After that, we will traverse through the list and check if all the elements at index “i” are less than or equal to the element at the index “i+1” from index 0 to the end. If we find any element at index “i” that is greater than the element at index “i+1”, we will assign the value False to the isSorted variable denoting that the list is not sorted. Then we will come out of the loop using the break statement as we have already found that the list is not sorted. 

If we do not find any element that is greater than the element at its right, the variable isSorted will remain True and will denote that the list is sorted. You can observe this in the following example.

def checkSortedList(newList):
    isSorted = True
    l = len(newList)
    for i in range(l - 1):
        if newList[i] > newList[i + 1]:
            isSorted = False
    return isSorted


myList1 = [1, 2, 3, 4, 5, 6, 7]
myList2 = [1, 2, 3, 3, 4, 5, 5]
myList3 = [1, 2, 3, 4, 2, 5, 6]
print("The list {} is sorted: {} ".format(myList1, checkSortedList(myList1)))
print("The list {} is sorted: {} ".format(myList2, checkSortedList(myList2)))
print("The list {} is sorted: {} ".format(myList3, checkSortedList(myList3)))

Output:

The list [1, 2, 3, 4, 5, 6, 7] is sorted: True 
The list [1, 2, 3, 3, 4, 5, 5] is sorted: True 
The list [1, 2, 3, 4, 2, 5, 6] is sorted: False

Conclusion

In this article, we have discussed how we can check for sorted list. To know more about lists, you can read this article on list comprehension in python. You might also like this article on dictionary comprehension in python. 

The post Check For Sorted List in Python appeared first on PythonForBeginners.com.

May 20, 2022 01:00 PM UTC

Real Python

The Real Python Podcast – Episode #110: Advantages of Protobuf for Serialization in Python

Would you like a way to send structured serialized data between different platforms and languages? What if the data was self-documenting, could automatically generate Python code, and would validate itself? This week on the show, Liran Haimovitch talks about protocol buffers and communicating with microservices through Remote Procedure Calls (RPC).

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 20, 2022 12:00 PM UTC

Mike Driscoll

How to do a Simple Cache with Python and functools (Video)

Learn how to use Python's amazing standard library to add caching to your functions. In this video tutorial, you will learn how to use functools.cache to add caching to a function

If you'd like to learn more about the functools module, you can check out this other video of mine:

The post How to do a Simple Cache with Python and functools (Video) appeared first on Mouse Vs Python.

May 20, 2022 11:37 AM UTC

Talk Python to Me

#366: Optimizing PostgreSQL DB Queries with pgMustard

Does your app have a database? Does that database play an important role in how the app operations and users perceive its quality? Most of you probably said yes to the first, and definitely to the second. But what if your DB isn't doing as well as it should? How would you know? And once you know, what do you do about it? <br/> <br/> On this episode, we're joined by Michael Christofides, co-creator of pgMustard, to discuss and explore the EXPLAIN command for Postgres and other databases as well as all the recommendations you might dig into as a result of understanding exactly what's happening with you queries.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Michael Christofides</b>: <a href="https://twitter.com/michristofides" target="_blank" rel="noopener">@michristofides</a><br/> <b>Datagrip</b>: <a href="https://www.jetbrains.com/datagrip/" target="_blank" rel="noopener">jetbrains.com</a><br/> <b>pgMustard</b>: <a href="https://pgmustard.com" target="_blank" rel="noopener">pgmustard.com</a><br/> <b>pgMustard example 1</b>: <a href="https://app.pgmustard.com/#/explore/example-1" target="_blank" rel="noopener">app.pgmustard.com</a><br/> <b>pgMustard example 2</b>: <a href="https://app.pgmustard.com/#/explore/example-2" target="_blank" rel="noopener">app.pgmustard.com</a><br/> <b>pgMustard example 3</b>: <a href="https://app.pgmustard.com/#/explore/example-3" target="_blank" rel="noopener">app.pgmustard.com</a><br/> <b>Arctype</b>: <a href="https://www.arctype.com" target="_blank" rel="noopener">arctype.com</a><br/> <b>Postico</b>: <a href="https://eggerapps.at/postico/" target="_blank" rel="noopener">eggerapps.at/postico</a><br/> <b>Laetitia Avrot Secrets of 'psql'— Video</b>: <a href="https://www.youtube.com/watch?v=znPL6iDDou8" target="_blank" rel="noopener">youtube.com</a><br/> <b>Beekeeper Studio</b>: <a href="https://www.beekeeperstudio.io" target="_blank" rel="noopener">beekeeperstudio.io</a><br/> <b>DBeaver</b>: <a href="https://dbeaver.io" target="_blank" rel="noopener">dbeaver.io</a><br/> <b>SQLite Browser</b>: <a href="https://sqlitebrowser.org" target="_blank" rel="noopener">sqlitebrowser.org</a><br/> <br/> <b>Michael's new Up and Running with Git course</b>: <a href="https://talkpython.fm/git" target="_blank" rel="noopener">talkpython.fm/git</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=0-hZrPTqN9A" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/366/optimizing-postgresql-db-queries-with-pgmustard" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Twitter</b>: <a href="https://twitter.com/talkpython" target="_blank" rel="noopener">@talkpython</a><br/> <b>Follow Michael on Twitter</b>: <a href="https://twitter.com/mkennedy" target="_blank" rel="noopener">@mkennedy</a><br/></div><br/> <strong>Sponsors</strong><br/> <a href='https://talkpython.fm/sentry'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a><br> <a href='https://talkpython.fm/assemblyai'>AssemblyAI</a>

May 20, 2022 08:00 AM UTC

May 19, 2022

Mike Driscoll

How to Create a Command-line Application with argparse

When you are creating an application, you will usually want to be able to tell your application how to do something. There are two popular methods for accomplishing this task. You can make your application accept command-line arguments or you can create a graphical user interface. Some applications support both.

Command-line interfaces are helpful when you need to run your code on a server. Most servers do not have a monitor hooked up, especially if they are Linux servers. In those cases, you might not be able to run a graphical user interface even if you wanted to.

Python comes with a built-in library called argparse that you can use to create a command-line interface. In this article, you will learn about the following:

• Parsing Arguments
• Creating Helpful Messages
• Adding Aliases
• Using Mutually Exclusive Arguments
• Creating a Simple Search Utility

There is a lot more to the argparse module than what will be covered in this article. If you would like to know more about it, you can check out the documentation.

Now it's time to get started with parsing arguments from the command-line!

Parsing Arguments

Before you learn how to use argparse, it's good to know that there is another way to pass arguments to a Python script. You can pass any arguments to a Python script and access those arguments by using the sys module.

To see how that works, create a file named sys_args.py and enter the following code into it:

# sys_args.py

import sys

def main():
    print('You passed the following arguments:')
    print(sys.argv)

if __name__ == '__main__':
    main()

This code imports sys and prints out whatever is in sys.argv. The argv attribute contains a list of everything that was passed to the script with the first item being the script itself.

Here's an example of what happens when you run this code along with a couple of sample arguments:

$ python3 sys_args.py --s 45
You passed the following arguments:
['sys_args.py', '--s', '45']

The problem with using sys.argv is that you have no control over the arguments that can be passed to your application:

• You can't ignore arguments
• You can't create default arguments
• You can't really tell what is a valid argument at all

This is why using argparse is the way to go when working with Python's standard library. The argparse module is very powerful and useful. Let's think about a common process that a command line application follows:

• pass in a file
• do something to that file in your program
• output the result

Here is a generic example of how that might work. Go ahead and create file_parser.py and add the following code:

# file_parser.py

import argparse

def file_parser(input_file, output_file=''):
    print(f'Processing {input_file}')
    print('Finished processing')
    if output_file:
        print(f'Creating {output_file}')

def main():
    parser = argparse.ArgumentParser('File parser')
    parser.add_argument('--infile', help='Input file')
    parser.add_argument('--out', help='Output file')
    args = parser.parse_args()
    if args.infile:
        file_parser(args.infile, args.out)

if __name__ == '__main__':
    main()

The file_parser() function is where the logic for the parsing would go. For this example, it only takes in a file name and prints it back out. The output_file argument defaults to an empty string.

The meat of the program is in main() though. Here you create an instance of argparse.ArgumentParser() and give your parser a name. Then you add two arguments, --infile and --out. To use the parser, you need to call parse_args(), which will return whatever valid arguments were passed to your program. Finally, you check to see if the user used the --infile flag. If they did, then you run file_parser().

Here is how you might run the code in your terminal:

$ python file_parser.py --infile something.txt
Processing something.txt
Finished processing

Here you run your script with the --infile flag along with a file name. This will run main() which in turns calls file_parser().

The next step is to try your application using both command-line arguments you declared in your code:

$ python file_parser.py --infile something.txt --out output.txt
Processing something.txt
Finished processing
Creating output.txt

This time around, you get an extra line of output that mentions the output file name. This represents a branch in your code logic. When you specify an output file, you can have your code go through the process of generating that file using a new block of code or a function. If you do not specify an output file, then that block of code would not run.

When you create your command-line tool using argparse, you can easily add messages that help your users when they are unsure of how to correctly interact with your program.

Now it's time to find out how to get help from your application!

Creating Helpful Messages

The argparse library will automatically create a helpful message for your application using the information that you provided when you create each argument. Here is your code again:

# file_parser.py

import argparse

def file_parser(input_file, output_file=''):
    print(f'Processing {input_file}')
    print('Finished processing')
    if output_file:
        print(f'Creating {output_file}')

def main():
    parser = argparse.ArgumentParser('File parser')
    parser.add_argument('--infile', help='Input file')
    parser.add_argument('--out', help='Output file')
    args = parser.parse_args()
    if args.infile:
        file_parser(args.infile, args.out)

if __name__ == '__main__':
    main()

Now try running this code with the -h flag and you should see the following:

$ file_parser.py -h
usage: File parser [-h] [--infile INFILE] [--out OUT]

optional arguments:
  -h, --help       show this help message and exit
  --infile INFILE  Input file
  --out OUT        Output file

The help parameter to add_argument() is used to create the help message above. The -h and --help options are added automatically by argparse. You can make your help more informative by giving it a description and an epilog.

Let's use them to improve your help messages. Start by copying the code from above into a new file named file_parser_with_description.py, then modify it to look like this:

# file_parser_with_description.py

import argparse

def file_parser(input_file, output_file=''):
    print(f'Processing {input_file}')
    print('Finished processing')
    if output_file:
        print(f'Creating {output_file}')

def main():
    parser = argparse.ArgumentParser(
            'File parser',
            description='PyParse - The File Processor',
            epilog='Thank you for choosing PyParse!',
            )
    parser.add_argument('--infile', help='Input file for conversion')
    parser.add_argument('--out', help='Converted output file')
    args = parser.parse_args()
    if args.infile:
        file_parser(args.infile, args.out)

if __name__ == '__main__':
    main()

Here you pass in the description and epilog arguments to ArgumentParser. You also update the help arguments to add_argument() to be more descriptive.

When you run this script with -h or --help after making these changes, you will see the following output:

$ python file_parser_with_description.py -h
usage: File parser [-h] [--infile INFILE] [--out OUT]

PyParse - The File Processor

optional arguments:
  -h, --help       show this help message and exit
  --infile INFILE  Input file for conversion
  --out OUT        Converted output file

Thank you for choosing PyParse!

Now you can see the new description and epilog in your help output. This gives your command-line application some extra polish.

You can also disable help entirely in your application via the add_help argument to ArgumentParser. If you think that your help text is too wordy, you can disable it like this:

# file_parser_no_help.py

import argparse

def file_parser(input_file, output_file=''):
    print(f'Processing {input_file}')
    print('Finished processing')
    if output_file:
        print(f'Creating {output_file}')

def main():
    parser = argparse.ArgumentParser(
            'File parser',
            description='PyParse - The File Processor',
            epilog='Thank you for choosing PyParse!',
            add_help=False,
            )
    parser.add_argument('--infile', help='Input file for conversion')
    parser.add_argument('--out', help='Converted output file')
    args = parser.parse_args()
    if args.infile:
        file_parser(args.infile, args.out)

if __name__ == '__main__':
    main()

By setting add_help to False, you are disabling the -h and --help flags.

You can see this demonstrated below:

$ python file_parser_no_help.py --help
usage: File parser [--infile INFILE] [--out OUT]
File parser: error: unrecognized arguments: --help

In the next section, you'll learn about adding aliases to your arguments!

Adding Aliases

An alias is a fancy word for using an alternate flag that does the same thing. For example, you learned that you can use both -h and --help to access your program's help message. -h is an alias for --help, and vice-versa

Look for the changes in the parser.add_argument() methods inside of main():

# file_parser_aliases.py

import argparse

def file_parser(input_file, output_file=''):
    print(f'Processing {input_file}')
    print('Finished processing')
    if output_file:
        print(f'Creating {output_file}')

def main():
    parser = argparse.ArgumentParser(
            'File parser',
            description='PyParse - The File Processor',
            epilog='Thank you for choosing PyParse!',
            add_help=False,
            )
    parser.add_argument('-i', '--infile', help='Input file for conversion')
    parser.add_argument('-o', '--out', help='Converted output file')
    args = parser.parse_args()
    if args.infile:
        file_parser(args.infile, args.out)

if __name__ == '__main__':
    main()

Here you change the first add_argument() to accept -i in addition to --infile and you also added -o to the second add_argument(). This allows you to run your code using two new shortcut flags.

Here's an example:

$ python3 file_parser_aliases.py -i something.txt -o output.txt
Processing something.txt
Finished processing
Creating output.txt

If you go looking through the argparse documentation, you will find that you can add aliases to subparsers too. A subparser is a way to create sub-commands in your application so that it can do other things. A good example is Docker, a virtualization or container application. It has a series of commands that you can run under docker as well as docker compose and more. Each of these commands has separate sub-commands that you can use.

Here is a typical docker command to run a container:

docker exec -it container_name bash

This will launch a container with docker. Whereas if you were to use docker compose, you would use a different set of commands. The exec and compose are examples of subparsers.

The topic of subparsers are outside the scope of this tutorial. If you are interested in more details dive right into the documentation.

Using Mutually Exclusive Arguments

Sometimes you need to have your application accept some arguments but not others. For example, you might want to limit your application so that it can only create or delete files, but not both at once.

The argparse module provides the add_mutually_exclusive_group() method that does just that!

Change your two arguments to be mutually exclusive by adding them to a group object like in the example below:

# file_parser_exclusive.py

import argparse

def file_parser(input_file, output_file=''):
    print(f'Processing {input_file}')
    print('Finished processing')
    if output_file:
        print(f'Creating {output_file}')

def main():
    parser = argparse.ArgumentParser(
            'File parser',
            description='PyParse - The File Processor',
            epilog='Thank you for choosing PyParse!',
            add_help=False,
            )
    group = parser.add_mutually_exclusive_group()
    group.add_argument('-i', '--infile', help='Input file for conversion')
    group.add_argument('-o', '--out', help='Converted output file')
    args = parser.parse_args()
    if args.infile:
        file_parser(args.infile, args.out)

if __name__ == '__main__':
    main()

First, you created a mutually exclusive group. Then, you added the -i and -o arguments to the group instead of to the parser object. Now these two arguments are mutually exclusive.

Here is what happens when you try to run your code with both arguments:

$ python3 file_parser_exclusive.py -i something.txt -o output.txt
usage: File parser [-i INFILE | -o OUT]
File parser: error: argument -o/--out: not allowed with argument -i/--infile

Running your code with both arguments causes your parser to show the user an error message that explains what they did wrong.

After covering all this information related to using argparse, you are ready to apply your new skills to create a simple search tool!

Creating a Simple Search Utility

Before starting to create an application, it is always good to figure out what you are trying to accomplish. The application you want to build in this section should be able to search for files of a specific file type. To make it more interesting, you can add an additional argument that allows you to optionally search for specific file sizes as well.

You can use Python's glob module for searching for file types. You can read all about this module here:

• https://docs.python.org/3/library/glob.html

There is also the fnmatch module, which glob itself uses. You should use glob for now as it is easier to use, but if you're interested in writing something more specialized, then fnmatch may be what you are looking for.

However, since you want to be able to optionally filter the files returned by the file size, you can use pathlib which includes a glob-like interface. The glob module itself does not provide file size information.

You can start by creating a file named pysearch.py and entering the following code:

# pysearch.py

import argparse
import pathlib


def search_folder(path, extension, file_size=None):
    """
    Search folder for files
    """
    folder = pathlib.Path(path)
    files = list(folder.rglob(f'*.{extension}'))

    if not files:
        print(f'No files found with {extension=}')
        return

    if file_size is not None:
        files = [
                f
                for f in files
                if f.stat().st_size >= file_size
                ]

    print(f'{len(files)} *.{extension} files found:')
    for file_path in files:
        print(file_path)

You start the code snippet above by importing argparse and pathlib. Next you create the search_folder() function which takes in three arguments:

• path - The folder to search within
• extension - The file extension to look for
• file_size - What file size to filter on in bytes

You turn the path into a pathlib.Path object and then use its rglob() method to search in the folder for the extension that the user passed in. If no files are found, you print out a meaningful message to the user and exit.

If any files are found, you check to see whether file_size has been set. If it was set, you use a list comprehension to filter out the files that are smaller than the specified file_size.

Next, you print out the number of files that were found and finally loop over these files to print out their names.

To make this all work correctly, you need to create a command-line interface. You can do that by adding a main() function that contains your argparse code like this:

def main():
    parser = argparse.ArgumentParser(
            'PySearch',
            description='PySearch - The Python Powered File Searcher',
            )
    parser.add_argument('-p', '--path',
                        help='The path to search for files',
                        required=True,
                        dest='path')
    parser.add_argument('-e', '--ext',
                        help='The extension to search for',
                        required=True,
                        dest='extension')
    parser.add_argument('-s', '--size',
                        help='The file size to filter on in bytes',
                        type=int,
                        dest='size',
                        default=None)

    args = parser.parse_args()
    search_folder(args.path, args.extension, args.size)

if __name__ == '__main__':
    main()

This ArgumentParser() has three arguments added to it that correspond to the arguments that you pass to search_folder(). You make the --path and --ext arguments required while leaving the --size argument optional. Note that the --size argument is set to type=int, which means that you cannot pass it a string.

There is a new argument to the add_argument() function. It is the dest argument which you use to tell your argument parser where to save the arguments that are passed to them.

Here is an example run of the script:

$ python3 pysearch.py -p /Users/michael/Dropbox/python101code/chapter32_argparse -e py -s 650
6 *.py files found:
/Users/michael/Dropbox/python101code/chapter32_argparse/file_parser_aliases2.py
/Users/michael/Dropbox/python101code/chapter32_argparse/pysearch.py
/Users/michael/Dropbox/python101code/chapter32_argparse/file_parser_aliases.py
/Users/michael/Dropbox/python101code/chapter32_argparse/file_parser_with_description.py
/Users/michael/Dropbox/python101code/chapter32_argparse/file_parser_exclusive.py
/Users/michael/Dropbox/python101code/chapter32_argparse/file_parser_no_help.py

$ python3 pysearch.py -p /Users/michael/Dropbox/python101code/chapter32_argparse -e py -s python
usage: PySearch [-h] -p PATH -e EXTENSION [-s SIZE]
PySearch: error: argument -s/--size: invalid int value: 'python'

The post How to Create a Command-line Application with argparse appeared first on Mouse Vs Python.

May 19, 2022 12:30 PM UTC

PyBites

Learn by building, how we built our own collaboration tool in Django

Listen now:

In this episode we tell the story about Pybites CMS, our new powertool we use to better streamline our business.

We tell why we built it, how it was an opportunity for Julian to brush up his coding skills, how we applied the PDM philosophy of learning by building, scratching your own itch.

Join our Facebook group here (live training coming soon!): https://www.facebook.com/groups/pybites

And if you want to take your Python and dev skills to the next level, check out our PDM coaching program here:
https://pybit.es/catalogue/the-pdm-program/

We could not find the mentioned article about shipping fast, but we certainly have spoken about this before, here are some additional resources:
https://pybit.es/pareto/
https://www.pybitespodcast.com/1501156/9135351-040-ship-your-code
https://www.linkedin.com/feed/update/urn:li:activity:6925359287103844352/

Thanks for listening, for any feedback email us at info@pybit.es

May 19, 2022 10:51 AM UTC

Python GUIs

PyQt6, PySide6, PyQt5 and PySide2 Books: Create GUI Applications with Python & Qt

Hello! Today I have released new digital editions of my PyQt5, PyQt6, PySide2 and PySide6 book Create GUI Applications with Python & Qt.

• PyQt6 Book 2nd Edition, Create GUI Applications with Python & Qt6
• PySide6 Book, 2nd Edition, Create GUI Applications with Python & Qt6
• PyQt5 Book 5th Edition, Create GUI Applications with Python & Qt5
• PySide2 Book 5th Edition, Create GUI Applications with Python & Qt5

This update adds over 200 pages of Qt examples and exercises - the book is now 800 pages long! - and continues to be updated and extended. The latest additions include:

• Built-in dialogs, including QMessageBox and QFileDialog
• Working with multiple windows, cross-window communication
• Using QThreadPool.start() to execute Python functions
• Long-running threads with QThread
• Using custom widgets in Qt Designer
• Recurring & single shot timers
• Managing data files, working with paths
• Packaging with PyInstaller on Windows, macOS & Linux
• Creating distributable installers on Windows, macOS & Linux

This update marks the 2nd Edition of the PyQt6 & PySide2 books, and the 5th Edition of PyQt5 & PySide2.

As always, if you've previously bought a copy of the book you get these updates for free! Just go to the downloads page and enter the email you used for the purchase. If you have problems getting this update just get in touch.

Enjoy!

For an in-depth guide to building GUIs with Python see my PySide2 book.

May 19, 2022 09:00 AM UTC

ListenData

Wish Christmas with Python and R

This post is dedicated to all the Python and R Programming Lovers...Flaunt your knowledge in your peer group with the following programs. As a data science professional, you want your wish to be special on eve of christmas. If you observe the code, you may also learn 1-2 tricks which you can use later in your daily tasks.

paste(intToUtf8(acos(log(1))*180/pi-13),
toupper(substr(month.name[2],2,2)),
paste(rep(intToUtf8(acos(exp(0)/2)*180/pi+2^4+3*2),2), collapse = intToUtf8(0)),
LETTERS[5^(3-1)], intToUtf8(atan(1/sqrt(3))*180/pi+2),
toupper(substr(month.abb[10],2,2)),
intToUtf8(acos(log(1))*180/pi-(2*3^2)),
toupper(substr(month.name[4],3,4)),
intToUtf8(acos(exp(0)/2)*180/pi+2^4+3*2+1),
intToUtf8(acos(exp(0)/2)*180/pi+2^4+2*4),
intToUtf8(acos(log(1))*180/pi-13),
LETTERS[median(0:2)],
intToUtf8(atan(1/sqrt(3))*180/pi*3-7),
sep = intToUtf8(0)
)

import math
import datetime

(chr(int(math.acos(math.log(1))*180/math.pi-13)) \
 + datetime.date(1900, 2, 1).strftime('%B')[1]  \
 + 2 * datetime.date(1900, 2, 1).strftime('%B')[3]  \
 + datetime.date(1900, 2, 1).strftime('%B')[7]  \
 + chr(int(math.atan(1/math.sqrt(3))*180/math.pi+2)) \
 + datetime.date(1900, 10, 1).strftime('%B')[1] \
 + chr(int(math.acos(math.log(1))*180/math.pi-18)) \
 + datetime.date(1900, 4, 1).strftime('%B')[2:4]  \
 + chr(int(math.acos(math.exp(0)/2)*180/math.pi+2**4+3*2+1)) \
 + chr(int(math.acos(math.exp(0)/2)*180/math.pi+2**4+2*4)) \
 + chr(int(math.acos(math.log(1))*180/math.pi-13)) \
 + "{:c}".format(97) \
 + chr(int(math.atan(1/math.sqrt(3))*180/math.pi*3-7))).upper()
 

install.packages("audio")
library(audio)
christmas_file - tempfile()
download.file("https://github.com/deepanshu88/Datasets/raw/master/UploadedFiles/merrychristmas1.wav", christmas_file, mode = "wb")
xmas <- load.wave(christmas_file)
play(xmas)

#Install packages
!pip install requests
!pip install playsound

import requests 
from playsound import playsound

audio_url = "https://github.com/deepanshu88/Datasets/raw/master/UploadedFiles/merrychristmas1.wav"
r = requests.get(audio_url)  

with open("christmas.wav",'wb') as f: 
      f.write(r.content)

playsound('christmas.wav')

May 19, 2022 08:15 AM UTC

May 18, 2022

Python Software Foundation

The 2022 Python Language Summit: Achieving immortality

What does it mean to achieve immortality? At the 2022 Python Language Summit, Eddie Elizondo, an engineer at Instagram, and Eric Snow, CPython core developer, set out to explain just that.

Only for Python objects, though. Not for humans. That will have to be another PEP.

---

Objects in Python, as they currently stand

In Python, as is well known, everything is an object. This means that if you want to calculate even a simple sum, such as 194 + 3.14, the Python interpreter must create two objects: one object of type int representing the number 194, and another object of type float representing the number 3.14.

All objects in Python maintain a reference count: a running total of the number of active references to that object that currently exist in the program. If the reference count of an object drops to 0, the object is eventually destroyed (through a process known as garbage collection). This process ensures that programmers writing Python don’t normally need to concern themselves with manually deleting an object when they’re done with it. Instead, memory is automatically freed up.

The need to keep reference counts for all objects (along with a few other mutable fields on all objects) means that there is currently no way of having a “truly immutable” object in Python.

This is a distinction that only really applies at the C level. For example, the None singleton cannot be mutated at runtime at the Python level:

>>> None.__bool__ = lambda self: True
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object attribute '__bool__' is read-only

However, at the C level, the object representing None is mutating constantly, as the reference count to the singleton changes constantly.

---

Immortal objects

An “immortal object”, according to PEP 683 (written by Elizondo/Snow), is an object marked by the runtime as being effectively immutable, even at the C level. The reference count for an immortal object will never reach 0; thus, an immortal object will never be garbage-collected, and will never die.

“The fundamental improvement here is that now an object can be truly immutable.”

– Eddie Elizondo and Eric Snow, PEP 683

 

The lack of truly immutable objects in Python, PEP 683 explains, “can have a large negative impact on CPU and memory performance, especially for approaches to increasing Python’s scalability”.

---

The benefits of immortality

At their talk at the Python Language Summit, Elizondo and Snow laid out a number of benefits that their proposed changes could bring.

Guaranteeing “true memory immutability”, Elizondo explained, “we can simplify and enable larger initiatives,” including Eric Snow’s proposal for a per-interpreter GIL, but also Sam Gross’s proposal for a version of Python that operates without the GIL entirely. The proposal could also unlock new optimisation techniques in the future by helping create new ways of thinking about problems in the CPython code base.

---

The costs

A naive implementation of immortal objects is costly, resulting in performance regeressions of around 6%. This is mainly due to adding a new branch of code to the logic keeping track of an object’s reference count.

With mitigations, however, Elizondo and Snow explained that the performance regression could be reduced to around 2%. The question they posed to the assembled developers in the audience was whether this was an “acceptable” performance regression – and, if not, what would be?

---

Reception

The proposal was greeted with a mix of curious interest and healthy scepticism. There was agreement that certain aspects of the proposal would reach wide support among the community, and consensus that a performance regression of 1-2% would be acceptable if clear benefits could be shown. However, there was also concern that parts of the proposal would change semantics in a backwards-incompatible way.

Pablo Galindo Salgado, Release Manager for Python 3.10/3.11 and CPython Core Developer, worried that all the benefits laid out by the speakers were only potential benefits, and asked for more specifics. He pointed out that changing the semantics of reference-counting would be likely to break an awful lot of projects, given that popular third-party libraries such as numpy, for example, use C extensions which continuously check reference counts.

Thomas Wouters, CPython Core Developer and Steering Council Member, concurred, saying that it probably “wasn’t possible” to make these changes without changing the stable ABI. Kevin Modzelewski, a maintainer of Pyston, a performance-oriented fork of Python 3.8, noted that Pyston had had immortal objects for a while – but Pyston had never made any promise to support the stable ABI, freeing the project of that constraint.

May 18, 2022 07:29 PM UTC

Real Python

Build a URL Shortener With FastAPI and Python

Read the full article at https://realpython.com/build-a-python-url-shortener-with-fastapi/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 18, 2022 02:00 PM UTC

Python for Beginners

Convert Tuple to String in Python

In python, a tuple can contain different elements. On the other hand, a string is just a sequence of characters. In this article, we will discuss how to convert a tuple to a string when we are given a tuple of numbers of characters in python. 

How to Convert Tuple of Characters to a String?

Suppose that we have the following tuple of characters.

myTuple = ("P", "y", "t", "h", "o", "n")

Now, we have to convert this tuple to the string “Python”.

To convert the tuple to string, we will first create an empty string named output_string. After that, We will traverse the tuple using a for loop. While traversal, we will add each character in the tuple to the output_string using the string concatenation operation. After execution of the for loop, we will get the desired string in the variable output_string. You can observe this in the following example.

myTuple = ("P", "y", "t", "h", "o", "n")
output_string = ""
for character in myTuple:
    output_string += character
print("The input tuple is:", myTuple)
print("The output string is:", output_string)

Output:

The input tuple is: ('P', 'y', 't', 'h', 'o', 'n')
The output string is: Python

Instead of using the for loop, we can use the join() method to convert the tuple of characters into a string. The join() method, when invoked on a string, takes an iterable object containing strings as an input argument. After execution, it returns the modified string containing the new character along with the characters in the original string. 

To convert a tuple of characters to a string, we will first create an empty string named  output_string. After that, we will invoke the join() method on the output_string with the tuple as input argument. After execution, the join() method will return the desired string as shown below.

myTuple = ("P", "y", "t", "h", "o", "n")
output_string = ""
output_string = output_string.join(myTuple)
print("The input tuple is:", myTuple)
print("The output string is:", output_string)

Output:

The input tuple is: ('P', 'y', 't', 'h', 'o', 'n')
The output string is: Python

Convert Tuple of Numbers to a String

If you try to convert a tuple of numbers to a string using any of the methods discussed above, the program will raise TypeError exception. You can observe this in the following example.

myTuple = (1, 2, 3, 4, 5, 6)
output_string = ""
for character in myTuple:
    output_string += character
print("The input tuple is:", myTuple)
print("The output string is:", output_string)

Output:

Traceback (most recent call last):
  File "/home/aditya1117/PycharmProjects/pythonProject/string1.py", line 4, in <module>
    output_string += character
TypeError: can only concatenate str (not "int") to str

Similarly, when we use the second approach, the program runs into TypeError exception as shown below.

myTuple = (1, 2, 3, 4, 5, 6)
output_string = ""
output_string = output_string.join(myTuple)
print("The input tuple is:", myTuple)
print("The output string is:", output_string)

Output:

Traceback (most recent call last):
  File "/home/aditya1117/PycharmProjects/pythonProject/string1.py", line 3, in <module>
    output_string = output_string.join(myTuple)
TypeError: sequence item 0: expected str instance, int found

To avoid the error, we just have to convert each element of the tuple into a string before adding it to the output_string. In the first approach, we will first convert each element of the tuple into a string using the str() function. After that, we will perform the concatenation operation. In this way, we can convert a tuple of numbers to a string.

myTuple = (1, 2, 3, 4, 5, 6)
output_string = ""
for element in myTuple:
    character = str(element)
    output_string += character
print("The input tuple is:", myTuple)
print("The output string is:", output_string)

Output:

The input tuple is: (1, 2, 3, 4, 5, 6)
The output string is: 123456

For the approach using the join() method, we will first convert the tuple of numbers into a tuple of strings. For this, we will use the map() function and the str() function. 

The map() function takes a function as its first argument and an iterable object as the second input argument. After execution, it returns a map object in which the function is applied to each element of the iterable object. We will pass the str() function and the tuple of numbers to the map() function as input arguments. After that, we will convert the map object using the tuple() constructor. Hence, we will get a tuple of strings as shown below.

myTuple = (1, 2, 3, 4, 5, 6)
newTuple = tuple(map(str, myTuple))
print("The input tuple is:", myTuple)
print("The output tuple is:", newTuple)

Output:

The input tuple is: (1, 2, 3, 4, 5, 6)
The output tuple is: ('1', '2', '3', '4', '5', '6')

After getting the tuple of strings from the tuple of numbers, we can directly obtain the string as follows.

myTuple = (1, 2, 3, 4, 5, 6)
newTuple = tuple(map(str, myTuple))
output_string = ''.join(newTuple)
print("The input tuple is:", myTuple)
print("The output string is:", output_string)

Output:

The input tuple is: (1, 2, 3, 4, 5, 6)
The output string is: 123456

Conclusion

In this article, we have discussed how to convert a tuple to a string in python. To know more about strings in python, you can read this article on string formatting in python. You might also like this article on list comprehension in python.

The post Convert Tuple to String in Python appeared first on PythonForBeginners.com.

May 18, 2022 01:00 PM UTC

PyBites

Configure a Linux Development Environment on Windows with WSL and VS Code

About WSL

It seems like everyone is using Linux or Mac for software development these days, but if you’re a windows user, you may have looked into what you needed to do to be able to use Linux on your PC and found that dual-booting or virtual machines sounded like too much trouble. But these days, most employers want their devs to know their way around the Linux command line. There’s never been a better time for Windows users to start learning those skills since there’s finally a way to have our cake and eat it too. It’s called Windows Subsystem for Linux (WSL).

The Windows Subsystem for Linux lets developers run a GNU/Linux environment — including most command-line tools, utilities, and applications — directly on Windows, unmodified, without the overhead of a traditional virtual machine or dualboot setup.”

What is Windows Subsystem for Linux | Microsoft Docs

If this sounds interesting, read on as I walk you through setting up a Python development environment in WSL.

Enable and install WSL

In order to use WSL, you must be using Windows version 10 (build 19041 or higher) or any version of Windows 11. For this tutorial I’m using Windows 11 and am also going to integrate Visual Studio Code with WSL so you’ll want to install this too. (WSL is not just for VS Code users, but at the time of this writing, it is the only lightweight code editor that integrates WSL natively.)

To get started on Windows 11, open a Powershell prompt as administrator and run
wsl --install

Note: if you’re using Windows 10 you’ll need to specify the distribution you want to install, e.g.: wsl --install -d Ubuntu

When prompted, reboot your computer

You should get a PowerShell window prompting you for your credentials to use within the Ubuntu distribution. After entering your new username and password, you should arrive at a Bash prompt.

You can use this shell the same as you would in the Linux terminal and can open it in the future by going to the Start Menu and searching for the Ubuntu app.

Launch your Windows Explorer by pressing WIN + E and you’ll see that you have a new Linux icon at the bottom of your navigation pane. This is a mounted drive that contains all your Linux files.

Congratulations, you have installed Linux on Windows!

Install Python, Pip, and Venv on Ubuntu

The included Ubuntu distribution already has python installed. Run python3 –version to ensure the version. As of the time of this writing, Python 3.8.2 is included.

Let’s ensure your python is up to date in Linux and you have the basics by returning to the Bash shell (Start Menu -> Ubuntu) and following the next 3 steps

• If you need to update your version of Python, first update your Ubuntu version by entering: 
  sudo apt update && sudo apt upgrade
  then update Python using sudo apt upgrade python3.
• Install pip by entering: sudo apt install python3-pip.
• If you use a virtual environment, install it by entering: sudo apt install python3-venv.

Run VS Code on WSL

Now that you have Linux installed on Windows and have your basic Python installation ready to go, we can start setting up a real Linux development environment in VS Code

Open VS Code and install the extension called Remote WSL

You will notice a new icon in the lower-left corner of the application. Click that icon to open a remote window.

From here you have multiple options:

New WSL Window will give you a clean session of VS Code in Linux. Alternatively, you can launch a new session in a different distro (see Windows Store for more distros to download if you wish).

Open Folder in WSL will open any folder on your Windows or Linux drives in VS Code on WSL. Although you can open files in WSL from the Windows file system, I recommend you work with projects and files actually stored within your Linux file system for compatibility. However, if you have a rather large project, porting it over could potentially be a lot of work, and guidance on that process is beyond the scope of this article, always make sure you have your code backed up in case something goes wrong.

The final option is Reopen Folder in WSL which will reload a folder opened in VS Code under Windows and open it under WSL instead.

For the purposes of this tutorial, click on New WSL Window. A new instance of VS Code will open and it will load into Ubuntu. You know you are running your VS Code instance in Linux by the icon in the lower left.

NOTE: There’s a possibility you may have to adjust some of your settings or reinstall/reconfigure some of your extensions through WSL.

Now that you have the hard part done, there’s just one more thing to walk through.

Creating and running a sample project

You’re all set up and ready to rock with your new Linux environment, but before you go, let’s run a simple hello world.

Create a new python file, write a hello world statement, then File -> Save. Notice VS Code is prompting you to save a file in the Linux file system.

After saving the file, press CTRL + ` to open the integrated terminal. VS Code now opens the default Ubuntu shell (aka “Bash”). If you have never used Linux before, you’ll want to learn the terminal as it’s one of the main benefits you’re getting by going through this process!

Type in the following command and press Enter to see your first program running in Linux on Windows

Enjoy your new development environment! To learn more about Linux, see Linux.org

May 18, 2022 12:12 PM UTC

Zato Blog

Web scraping as an API service

Overview

In systems-to-systems integrations, there comes an inevitable time when we have to employ some kind of a web scraping tool to integrate with a particular application. Despite its not being our first choice, it is good to know what to use at such a time - in this article, I provide a gentle introduction to my favourite tool of this kind, called Playwright, followed by sample Python code that integrates it with an API service.

Naturally, in the context of backend integrations, web scraping should be avoided and, generally, it should be considered the last resort. The basic issue here is that while the UI term contains the “interface” part, it is not really the “Application Programming” Interface that we would like to have.

It is not that the UI cannot be programmed against. After all, a web browser does just that, it takes a web page and renders it as expected. Same goes for desktop or mobile applications. Also, anyone integrating with mainframe computers will recognize that this is basically what 3270 can be used for too.

Rather, the fundamental issue is that web scraping goes against the principles of separation of layers and roles across frontend, middleware and backend, which in turn means that authors of resources (e.g. HTML pages) do not really expect for many people to access them in automated ways.

Perhaps they actually should expect it, and web pages should finally start to resemble genuine knowledge graphs, easy to access by humans, be it manually or through automation tools, but the reality today is that it is not the case and, in comparison with backend systems, the whole of the web scraping space is relatively brittle, which is why we shun this approach in integrations.

Yet, another part of reality, particularly in enterprise integrations, is that people may be sometimes given access to a frontend application on an internal network and that is it. No API, no REST, no JSON, no POST data, no real data formats, and one is simply supposed to fill out forms as part of a business process.

Typically, such a situation will result in an integration gap. There will be fully automated parts in the business process preceding this gap, with multiple systems coordinated towards a specific goal and there will be subsequent steps in the process, also fully automated.

Or you may be given access only to a specific frontend and only through VPN via a single remote Windows desktop. Getting access to a REST API may take months or may be never realized because of some high level licensing issues. This is not uncommon in the real life.

Such a gap can be a jarring and sore point, truly ruining the whole, otherwise fluid, integration process. This creates a tension and to resolve the tension, we can, should all the attempts to find a real API fail, finally resort to web scraping.

It is mostly in this context that I am looking at Playwright below - the tool is good and it has many other uses that go beyond the scope of this text, and it is well worth knowing it, for instance for frontend testing of your backend systems, but, when we deal with API integrations, we should not overdo with web scraping.

Needless to say, if web scraping is what you do primarily, your perspective will be somewhat different - you will not need any explanation of why it is needed or when, and you may be only looking for a way to enclose up your web scraping code in API services. This article will explain that too.

Introducing Playwright

The nice part of Playwright is that we can use it to visually prepare a draft of Python code that will scrape a given resource. That is, instead of programming it in Python, we go to an address, fill out a form, click buttons and otherwise use everything as usually and Playwright generates for us code that will be later used in integrations.

That code will require a bit of clean-up work, which I will talk about below, but overall it works very nicely and is certainly useful. The result is not one of these do-not-touch auto-generated pieces of code that are better left to their own.

While there are better ways to integrate with Jira, I chose that application as an example of Playwright’s usage simply because I cannot show you any internal application in a public blog post.

Below, there are two windows. One is Playwright’s emulating a Blackberry device to open a resource. I was clicking around, I provided an email address and then I clicked the same email field once more. To the right, based on my actions, we can find the generated Python code, which I consider quite good and readable.

The Playwright Inspector, the tool that gave us the code, will keep recording all of our actions until we click the “Record” button which then allows us to click the button next to “Record” which is “Copy code to clipboard”. We can then save the code to a separate file and run it on demand, automatically.

But first, we will need to install Playwright.

Installing and starting Playwright

The tools is written in TypeScript and can be installed using npx, which in turn is part of NodeJS.

Afterwards, the “playwright install” call is needed as well because that will potentially install runtime dependencies, such as Chrome libraries.

Finally, we install Playwright using pip as well because we want to access with Python. Note that if you are installing Playwright under Zato, the “/path/to/pip” will be typically “/opt/zato/code/bin/pip”.

npx -g --yes playwright install
playwright install
/path/to/pip install playwright

We can now start it as below. I am using BlackBerry as an example of what Playwright is capable of. Also, it is usually more convenient to use a mobile version of a site when the main window and Inspector are opened side by side, but you may prefer to use Chrome, Firefox or anything else.

playwright codegen https://example.atlassian.net/jira --device "BlackBerry Z30"

That is practically everything as using Playwright to generate code in our context goes. Open the tool, fill out forms, copy code to a Python module, done.

What is still needed, though, is cleaning up the resulting code and embedding it in an API integration process.

Code clean-up

After you keep using Playwright for a while with longer forms and pages, you will note that the generated code tends to accumulate parts that repeat.

For instance, in the module below, which I already cleaned up, the same “[placeholder="Enter email"]” reference to the email field is used twice, even if a programmer developing this could would prefer to introduce a variable for that.

There is not a good answer to the question of what to do about it. On the one hand, obviously, being programmers we would prefer not to repeat that kind of details. On the other hand, if we clean up the code too much, this may result in too much of a maintenance burden because we need to keep it mind that we do not really want to invest to much in web scraping and, should there be a need to repeat the whole process, we do not want to end up with Playwright’s code auto-generated from scratch once more, without any of our clean-up.

A good compromise position is to at least extract any kind of credentials from the code to environment variables or a similar place and to remove some of the code comments that Playwright generates. The result as below is what it should like at the end. Not too much effort without leaving the whole code as it was originally either.

Save the code below as “play1.py” as this is what the API service below will use.

# -*- coding: utf-8 -*-

# stdlib
import os

# Playwright
from playwright.sync_api import Playwright, sync_playwright

class Config:
    Email = os.environ.get('APP_EMAIL', 'zato@example.com')
    Password = os.environ.get('APP_PASSWORD', '')
    Headless = bool(os.environ.get('APP_HEADLESS', False))

def run(playwright: Playwright) -> None:

    browser = playwright.chromium.launch(headless=Config.Headless) # type: ignore
    context = browser.new_context()

    # Open new page
    page = context.new_page()

    # Open project boards
    page.goto("https://example.atlassian.net/jira/software/projects/ABC/boards/1")
    page.goto("https://id.atlassian.com/login?continue=https%3A%2F%2Fexample.atlassian.net%2Flogin%3FredirectCount%3D1%26dest-url%3D%252Fjira%252Fsoftware%252Fprojects%252FABC%252Fboards%252F1%26application%3Djira&application=jira")

    # Fill out the email
    page.locator("[placeholder=\"Enter email\"]").click()
    page.locator("[placeholder=\"Enter email\"]").fill(Config.Email)

    # Click #login-submit
    page.locator("#login-submit").click()

with sync_playwright() as playwright:
    run(playwright)

Web scraping as a standalone activity

We have the generated code so the first thing to do with it is to run it from command line. This will result in a new Chrome window’s accessing Jira - it is Chrome, not Blackberry, because that is the default for Playwright.

The window will close soon enough but this is fine, that code only demonstrates a principle, it is not a full integration task.

python /path/to/play1.py

It is also useful that we can run the same Python module from our IDE, giving us the ability to step through the code line by line, observing what changes when and why.

Web scraping as an API service

Finally, we are ready to invoke the standalone module from an API service, as in the following code that we are also going to make available as a REST channel.

A couple of notes about the Python service below:

• We invoke Playwright in a subprocess, as a shell command
• We accept input through data models although we do not provide any output definition because it is not needed here
• When we invoke Playwright, we set the APP_HEADLESS to True which will ensure that it does not attempt to actually display a Chrome window. After all, we intend for this service to run on Linux servers, in backend, and such a thing will be unlikely to work in this kind of an environment.

Other than that, this is a straightforward Zato service - it receives input, carries out its work and a reply is returned to the caller (here, empty).

# -*- coding: utf-8 -*-

# stdlib
from dataclasses import dataclass

# Zato
from zato.server.service import Model, Service

# ###########################################################################

@dataclass(init=False)
class WebScrapingDemoRequest(Model):
    email: str
    password: str

# ###########################################################################

class WebScrapingDemo(Service):
    name = 'demo.web-scraping'

    class SimpleIO:
        input  = WebScrapingDemoRequest

    def handle(self):

        # Path to a Python installation that Playwright was installed under
        py_path = '/path/to/python'

        # Path to a Playwright module with code to invoke
        playwright_path = '/path/to/the-playwright-module.py'

        # This is a template script that we will invoke in a subprocess
        command_template = """
        APP_EMAIL={app_email} APP_PASSWORD={app_password} APP_HEADLESS=True {py_path} {playwright_path}
        """

        # This is our input data
        input = self.request.input # type: WebScrapingDemoRequest

        # Extract credentials from the input ..
        email = input.email
        password = input.password

        # .. build the full command, taking all the config into account ..
        command = command_template.format(
            app_email = email,
            app_password = password,
            py_path = py_path,
            playwright_path = playwright_path,
        )

        # .. invoke the command in a subprocess  ..
        result = self.commands.invoke(command)

        # .. if it was not a success, log the details received ..
        if not result.is_ok:
            self.logger.info('Exit code -> %s', result.exit_code)
            self.logger.info('Stderr -> %s',    result.stderr)
            self.logger.info('Stdout -> %s',    result.stdout)

# ###########################################################################

Now, the REST channel:

The last thing to do is to invoke the service - I am using curl from the command line below but it could very well be Postman or a similar option.

curl localhost:17010/demo/web-scraping -d '{"email":"hello@example.com", "password":"abc"}' ; echo

There will be no Chrome window this time around because we run Playwright in the headless mode. There will be no output from curl either because we do not return anything from the service but in server logs we will find details such as below.

We can learn from the log that the command took close to 4 seconds to complete, that the exit code was 0 (indicating success) and that is no stdout or stderr at all.

INFO - Command `
APP_EMAIL=hello@example.com APP_PASSWORD=abc APP_HEADLESS=True
  /path/to/python
  /path/to/the-playwright-module.py
` completed in 0:00:03.844157,
  exit_code -> 0; len-out=0 (0 Bytes); len-err=0 (0 Bytes);
  cid -> zcmdc5422816b2c6ff9f10742134

We are now ready to continue to work on it - for instance, you will notice that the password is visible in logs and this should not be allowed.

But, all such works are extra in comparison with the main theme - we have Playwright, which is a a tool that allows us to quickly integrate with frontend applications and we can automate it through API services. Just as expected.

Next steps

• Start the tutorial to learn how to integrate APIs and build systems. After completing it, you will have a multi-protocol service representing a sample scenario often seen in banking systems with several applications cooperating to provide a single and consistent API to its callers.

• Visit the support page if you need assistance.

• Para aprender más sobre las integraciones de Zato y API en español, haga clic aquí.

• Pour en savoir plus sur les intégrations API avec Zato en français, cliquez ici.

May 18, 2022 09:31 AM UTC

Python Bytes

#284 Spicy git for Engineers

<p><strong>Watch the live stream:</strong></p> <a href='https://www.youtube.com/watch?v=Go2-6sboFS4' style='font-weight: bold;'>Watch on YouTube</a><br> <br> <p><strong>About the show</strong></p> <p>Sponsored by us! Support our work through:</p> <ul> <li>Our <a href="https://training.talkpython.fm/"><strong>courses at Talk Python Training</strong></a></li> <li><a href="https://testandcode.com/"><strong>Test &amp; Code</strong></a> Podcast</li> <li><a href="https://www.patreon.com/pythonbytes"><strong>Patreon Supporters</strong></a></li> </ul> <p><strong>Brian #1:</strong><a href="https://github.com/alan-turing-institute/distinctipy"><strong>distinctipy</strong></a></p> <ul> <li>“<em>distinctipy</em> is a lightweight python package providing functions to generate colours that are visually distinct from one another.”</li> <li>Small, focused tool, but really cool.</li> <li>Say you need to plot a dynamic number of lines.</li> <li>Why not let distinctipy pick colors for you that will be distinct?</li> <li>Also can display the color swatches.</li> <li>Some example palettes here: https://github.com/alan-turing-institute/distinctipy/tree/main/examples <pre><code>from distinctipy import distinctipy # number of colours to generate N = 36 # generate N visually distinct colours colors = distinctipy.get_colors(N) # display the colours distinctipy.color_swatch(colors) </code></pre></li> </ul> <p><strong>Michael #2:</strong> <a href="https://docs.soda.io/soda-sql/concepts.html"><strong>Soda SQL</strong></a></p> <ul> <li>Soda SQL is a free, open-source command-line tool.</li> <li>It utilizes user-defined input to prepare SQL queries that run tests on dataset in a data source to find invalid, missing, or unexpected data.</li> <li>Looks good for data pipelines and other CI/CD work!</li> </ul> <p><strong>Daniel #3:</strong> <a href="https://www.nature.com/articles/s41586-020-2649-2"><strong>Python in Nature</strong></a></p> <ul> <li>There’s a review article from Sept 2020 on array programming with NumPy in the research journal Nature.</li> <li>For reference, in grad school we had a fancy paper on quantum entanglement that got rejected from Nature Communications, a sub-journal to Nature. Nature is hard to get into.</li> <li>List of authors includes Travis Oliphant who started NumPy. Covers NumPy as the foundation, building up to specialized libraries like QuTiP for quantum computing.</li> <li>If you search “Python” on their site, many papers come up. Interesting to see their take on publishing software work.</li> </ul> <p><strong>Brian #4:</strong> <a href="https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/"><strong>Supercharging GitHub Actions with Job Summaries</strong></a></p> <ul> <li><p>From a tweet by <a href="https://twitter.com/simonw/status/1526337395334885377?s=20&amp;t=pFgZ2Ruklh8MLNlSiUmIcA">Simon Willison</a></p> <ul> <li>and an article: <a href="https://til.simonwillison.net/github-actions/job-summaries">GH Actions job summaries</a></li> </ul></li> <li><p>Also, <a href="https://twitter.com/nedbat/status/1526338136699281408?s=20&amp;t=pFgZ2Ruklh8MLNlSiUmIcA">Ned Batchelder</a> is using it for Coverage reports</p></li> <li><p>“You can now output and group custom Markdown content on the Actions run summary page.”</p></li> <li><p>“Custom Markdown content can be used for a variety of creative purposes, such as:</p> <ul> <li>Aggregating and displaying test results</li> <li>Generating reports</li> <li>Custom output independent of logs”</li> </ul></li> <li><p><a href="https://github.com/nedbat/coveragepy/blob/ad824b4585c88d0a153dd248f4585084dea33189/.github/workflows/coverage.yml#L218-L221">Coverage.py example:</a></p> <pre><code>- name: "Create summary" run: | echo '### Total coverage: ${{ env.total }}%' &gt;&gt; $GITHUB_STEP_SUMMARY echo '[${{ env.url }}](${{ env.url }})' &gt;&gt; $GITHUB_STEP_SUMMARY </code></pre></li> </ul> <p><strong>Michael #5:</strong><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit_01678898482.html"><strong>Language Summit is write up out</strong></a></p> <ul> <li>via Itamar, by Alex Waygood <ul> <li><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-python_11.html"><strong>Python without the GIL</strong></a>: A talk by Sam Gross</li> <li><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-per.html"><strong>Reaching a per-interpreter GIL</strong></a>: A talk by Eric Snow</li> <li><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit_2.html"><strong>The "Faster CPython" project: 3.12 and beyond</strong></a>: A talk by Mark Shannon</li> <li><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-python.html"><strong>WebAssembly: Python in the browser and beyond</strong></a>: A talk by Christian Heimes</li> <li><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-f.html"><strong>F-strings in the grammar</strong></a><strong>:</strong> A talk by Pablo Galindo Salgado</li> <li><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit_60.html"><strong>Cinder Async Optimisations</strong></a>: A talk by Itamar Ostricher</li> <li><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-dealing.html"><strong>The issue and PR backlog</strong></a>: A talk by Irit Katriel</li> <li><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit_11.html"><strong>The path forward for immortal objects</strong></a>: A talk by Eddie Elizondo and Eric Snow</li> <li><a href="https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit.html"><strong>Lightning talks</strong></a>, featuring short presentations by Carl Meyer, Thomas Wouters, Kevin Modzelewski, Samuel Colvin and Larry Hastings</li> </ul></li> </ul> <p><strong>Daniel #6:</strong><a href="https://www.allspice.io"><strong>AllSpice is Git for EEs</strong></a></p> <ul> <li>Software engineers have Git/SVN/Mercurial/etc</li> <li>None of the other engineering disciplines (mechanical, electrical, optical, etc), have it nearly as good. Altium has their Vault and “365,” but there’s nothing with a Git-like UX.</li> <li>Supports version history, diffs, all the things you expect. Even self-hosting and a Gov Cloud version.</li> <li>“Bring your workflow to the 21st century, finally.”</li> </ul> <p><strong>Extras</strong> </p> <p>Brian:</p> <ul> <li><a href="https://testandcode.com/188">Will McGugan talks about Rich, Textual, and Textualize on Test &amp; Code 188</a></li> <li>Also 3 other episodes since last week. (I have a backlog I’m working through.)</li> </ul> <p>Michael:</p> <ul> <li><a href="https://www.youtube.com/watch?v=je21yaRV_xc"><strong>Power On-Xbox Documentary | Full Movie</strong></a></li> <li><a href="https://www.youtube.com/watch?v=7sxUu-tYcIA"><strong>The 4 Reasons To Branch with Git - Illustrated Examples with Python</strong></a></li> <li><a href="https://www.easypost.com"><strong>A Python spotting</strong></a> - via Jason Pecor</li> <li><a href="https://twitter.com/btskinn/status/1524507904929370114"><strong>2022 StackOverflow Developer Survey</strong></a> is live, via Brian</li> <li><a href="https://www.textsniper.app"><strong>TextSniper macOS App</strong></a></li> <li>PandasTutor on webassembly </li> </ul> <p>Daniel: </p> <ul> <li>I know Adafruit’s a household name, shout-out to <a href="https://www.sparkfun.com">Sparkfun</a>, <a href="https://www.seeedstudio.com">Seeed Studio</a>, <a href="https://openmv.io">OpenMV</a>, and other companies in the field.</li> </ul> <p><strong>Joke:</strong> </p> <p><a href="https://www.reddit.com/r/ProgrammerHumor/comments/un1pmg/i_can_explain_this/"><strong>A little awkward</strong></a></p>

May 18, 2022 08:00 AM UTC

Django Weblog

Django 4.1 alpha 1 released

Django 4.1 alpha 1 is now available. It represents the first stage in the 4.1 release cycle and is an opportunity for you to try out the changes coming in Django 4.1.

Django 4.1 has an profusion of new features which you can read about in the in-development 4.1 release notes.

This alpha milestone marks the feature freeze. The current release schedule calls for a beta release in about a month and a release candidate about a month from then. We'll only be able to keep this schedule if we get early and often testing from the community. Updates on the release schedule are available on the django-developers mailing list.

As with all alpha and beta packages, this is not for production use. But if you'd like to take some of the new features for a spin, or to help find and fix bugs (which should be reported to the issue tracker), you can grab a copy of the alpha package from our downloads page or on PyPI.

The PGP key ID used for this release is Carlton Gibson: E17DF5C82B4F9D00.

May 18, 2022 05:58 AM UTC

Python⇒Speed

CPUs, cloud VMs, and noisy neighbors: the limits of parallelism

Sometimes your program is slow not because of your code, but because of where it’s running. If you have other processes competing for the same limited hardware resources, your code will run more slowly.

Once you add virtualization into the mix, those competing processes might be invisible… but they’re still there.

In this article we’ll cover:

• The hardware limits of CPUs’ cores and “hyperthreads”.
• How operating systems deal with multiple processes that want to use limited number of CPU cores.
• The impact of virtualization on these various resources, including what cloud vCPUs mean.
• CPU sharing limits available via containers.

May 18, 2022 12:00 AM UTC

scikit-learn

The Value of Open Source Sprints, the scikit-learn Experience

Author: Reshama Shaikh

With contributions from: Gaël Varoquaux, Andreas Mueller, Olivier Grisel, Julien Jerphanion, Guillaume LeMaitre

Top Line Summary

Sprints are working sessions to contribute to an open source library. The goals and achievements differ between Developer and Community sprints. The long-term impact of open source sprints, particularly community events, is not easily quantifiable or measurable. Positive outcomes of sprints have slowly been emerging, and for that reason, to realize the value of open source sprints requires playing the “long game”.

Introduction

The scikit-learn project has a long and extraordinary legacy of open source sprints. Since 2010, when its first public version was released, there have been as many as 45 sprints organized. The number 45 is a lower bound, since there are likely more sprints that have not been listed.

To date, more than 2300 people have contributed to scikit-learn. The number of contributors to scikit-learn exceeds those of other related libraries such as numpy, scipy and matplotlib, with the exception of the pandas, which has a greater number of contributors (See Appendix A).

The public discourse on open source has expanded to explore topics of sustainability, funding models, and diversity and inclusion, to name a few. A reasonable, yet ”difficult to answer” question that has been posed is:

What is the effectiveness of sprint models and what is the long-term engagement as a result of these sprints?

Due to technological limitations of GitHub and privacy concerns, we do not hold precise data on how many scikit-learn contributors connected to the project via a sprint. We have no formal data collection process which records statistics on how many sprint participants are recurring or information on their contributions to other open source projects or other long term positive ripple effects. A scientific look at the correlation between the number of sprints and contributors is beyond the scope of this article. What we will examine in this article are the objectives, results and aspirations of running the scikit-learn sprints.

The queries from other open-source projects requesting guidance on sprints and diversity and inclusions have been increasing. We share these experiences and lessons learned with the community, potential funders and open source project maintainers, particularly those projects which are nascent in their quest to build community, sustainability and diversity and inclusion.

Outline

In this article we examine the following:

• What is a “sprint”?
• What are the differences between “Developer” and “Community” sprints?
• What are the goals of the open source sprints?
• What value do open source sprints bring to the project and community?
• What are the aspirations of the scikit-learn project, in terms of connecting with the community?

Definition of a scikit-learn Sprint

A scikit-learn sprint has traditionally been an event where contributors come together to work on issues in the scikit-learn repository. A sprint can be as short as a few hours, or last over several days, even a week or longer. They may be in-person, online, hybrid or partially asynchronous. Sprints may be organized by the developers of the library, community groups (such as Meetups), scheduled alongside scientific or Python conferences, or even at home with a few friends. They can more simply and less dauntingly be described as working sessions to contribute to the open source library.

Developer vs Community Sprint

We distinguish between a Developer (Dev) and Community sprint because the goals and results differ significantly between the two.

Developer (Dev) Sprint

A Developer, or “Dev”, sprint is one that is typically organized by the maintainers of the library. A Dev sprint is one where the developers or maintainers of the library gather to work on issues and to discuss the resolution of ongoing complex issues. This also provides the team an opportunity to focus on tasks related to the long-term roadmap of the project.

The first early Dev sprints were organized at Inria. The first major Dev sprint was held in Granada after the NIPS 2011 conference (now renamed NeurIPS). It was the first time that most of the team had met in real life after months or years of online collaboration, and over a dozen developers participated. Later, Dev sprints were often hosted in the offices of partnering tech companies, typically from 3 to 7 days, once a year, in pre-COVID times.

Community Sprint

A Community sprint can be a collaboration by individuals, by affinity communities such as Meetup Groups (Data Umbrella, PyLadies, etc.), by conferences (SciPy, PyCon, PyData Global, JupyterCon, etc.). A Community sprint is one that is with the general public and it may be beginners, experts, or a combination of both.

For scikit-learn, the early Community sprints were alongside the SciPy conferences and the practice has continued for over a decade.

At a Developer sprint, a contributor may work on a PR that has been ongoing for three months. Conversely, Community sprints require curated issues which newcomers can complete in a shorter period of time (such as 1 day, or 1 day with 1-2 months follow-up).

The landscape of Dev and Community sprints with other scientific python libraries is unknown.

Goals of the Sprints

Goals of Dev Sprints

• To get maintainers in one room to efficiently discuss open issues and pull requests
• To move along contributions in a synchronous fashion
• To foster existing collaborations with external developers synchronously
• To build rapport: Maintainers reside in various continents and the in-person sprints build rapport within the team. Social interactions are critical in having a productive team.
• To foster collaborations with the project’s corporate sponsors (members of the scikit-learn Consortium)

Goals of Community & Beginner Sprints

• To broaden the project’s contributor base
• To build community and connect the project maintainers with its users
• To obtain interactive feedback from new scikit-learn users and contributors
• To onboard new contributors to scikit-learn and PyData generally
• To onboard new contributors who would become recurring contributors
• To collaborate with community groups to increase diversity of contributor base with intentional outreach
• To strengthen and support existing contributors in order to maintain recurring community contributors

scikit-learn Team Members Who Connected to the Project Via a Sprint

It is notable that a number of the current maintainers of the library found their way to the project via a sprint. Additionally, some members of the Contributor Experience Team connected to the scikit-learn project via the sprints.

Olivier Grisel

Olivier Grisel has been a contributor and maintainer for more than 12 years. Olivier met Gaël Varoquaux at a local conference organized in Paris by the French speaking Python users group AFPy.org. After chatting 5 minutes about toy ML experiments in Python, Gaël invited Olivier to join the first sprint organized at Inria in March 2010:

Olivier shares:

At the time, scikit-learn coding sprints gathered only 6 people sitting around a table with some wifi and a coffee machine :)

Andreas Mueller

Andreas Mueller has been a maintainer of the project since 2011. He joined a sprint at a conference because he was a user and wanted to contribute. He shares in a 2017 interview:

While working on my Ph.D. in computer vision and learning, the scikit-learn library became an essential part of my toolkit. My initial participation in open source began in 2011 at the NeurIPS conference in Granada, Spain, where I had attended a scikit-learn sprint. The scikit-learn release manager at the time had to leave, and the project leads asked me to become release manager; that’s how it all got started.

Julien Jerphanion

Julien Jerphanion participated in a sprint in February 2019 at AXA as a first time contributor while interning at Dataiku. The sprint provided Julien an opportunity to experience scikit-learn and meet the maintainers. Prior to the sprint, he had only used the library in a few projects. He contributed code, reviews, and documentation since March 2021, joined Inria in April 2021 and in October 2021, Julien became a core developer.

Other Maintainers

There are other maintainers and emeritus contributors who had participated in a Developer or Community sprint along their journey with the scikit-learn team, such as Vlad Nicolae (current maintainer), Gilles Loupe (Emeritus), Thouis (Ray) Jones (Emeritus).

Reshama Shaikh

Reshama Shaikh has organized nine scikit-learn community sprints from 2017 to 2021. She first contributed code and documentation fixes to scikit-learn in September 2018. In September 2020, she was invited to join the scikit-learn team.

In her PyConDE PyData Berlin keynote from April 2022, 5 Years, 10 Sprints, a scikit-learn Open Source Journey, she shares a history and progression of the Community sprints.

Juan Martín Loyola

Juan Martín Loyola started contributing to scikit-learn as preparation for the Data Umbrella Latin America, June 2021 sprint. He continued to contribute prolifically after the sprint, and he was invited to join the team in December 2021. Given his location in Argentina, he will be providing support at the 2022 SciPy Latin America sprint.

Second Degree Impact

Lauren Burke joined the scikit-learn Communications Team in November 2021 at the recommendation of Reshama Shaikh, and this can be considered a network effect. This demonstrates that sprints can result in valuable contributions other than code.

Sprints: Observed Impact and Lessons Learned

There are a number of observed favorable outcomes from the sprints for both the project and contributors.

Onboarding

The sprints help the community discover the open source process and get started with contributing.

Building community

Sprint participants, whether one-time or recurring, become ambassadors for the project.

Open source workflow knowledge

Users learn a range of tools such as: virtual environment setup, version control systems (i.e. Git), testing (flake8, pytest, continuous integration) and unit tests. They also learn software development best practices. For many users of scikit-learn, the sprint is the first time they navigate through the codebase and structure of scikit-learn, dig into functions and learn about errors. They develop experience in collaborative open source workflow. For employers, letting their team contribute to open-source might be a plus as they learn how to collaborate properly and learn about the internals of the library. The sprint experience assists contributors in developing a wider set of technical skills that can be shared across projects, networking, on to jobs and more.

Overcoming barriers to entry

The sprints, as a “hands-on working session”, provide an avenue for potential contributors to overcome common barriers to entry, particularly “getting started”, and moving from the possibility to an actuality stage.

Providing an avenue for advanced contributions

As sprints provide an on-ramp for new contributors, it similarly provides an opportunity for returning contributors to advance their contributing skills to the next level in a structured environment and with mentorship.

Building confidence

The sprints help to build confidence for both new and returning contributors.

Gaël shares:

I believe those sprints helped resourceful people (like Juan Martín) to gain confidence and provide valuable contributions (especially reviews).

Increase open-source literacy

The sprints are a forum for users to gain a greater understanding of how an open source project functions and for the user/contributor to learn of an actual contribution, from start to finish.

Value of synchronous interaction

Typically, open source contributions to scikit-learn occur on the GitHub repository in asynchronous fashion. The sprints provide real-time synchronous interaction. This experience provides more direct access to technical assistance and feedback to the contributor, and in a direct, efficient, and time-saving manner.

Julien shares:

I think having a setup like this [beginner/community sprint] is valuable for first time contributors because they can synchronously get specific information they would hardly have got otherwise. To me, this allow giving feedback which is immediate, specific and exact, making contributing to open-source enjoyable and preventing frustration: giving such feedback is what we should aim for and in this regard this setup is convenient.

Online Sprints

Since the start of the pandemic, Data Umbrella organized 4 online sprints. Additionally, there were 2 online sprints with SciPy and EuroPython.

These have been the observed benefits of the online sprints, which began in 2020 due to the global pandemic:

Networking

Online sprints make it easier to meet new people with different backgrounds.

International collaboration

Collaborating with affinity communities can attract more candidates from various backgrounds. In particular, online sprints help break geographical barriers.

Pair programming

The pairing of contributors seems to work well. Pair programming was consistently ranked as a positive experience by online sprint participants.

Increases accessibility

The use of online tools makes it possible to interact with people who would not have joined community events traditionally organized in North America or western Europe e.g. because of the travel costs and complexity of obtaining a visa in time. Attending the online events is probably also less disruptive for people with young children.

For the scikit-learn project itself, it made it possible to “recruit” a couple of new recurring contributors who attend regular office hours after the original sprints.

Office Hours

Actually the fact that we now have community office hours on Discord is probably a consequence of us attending the Data Umbrella online sprints.

Olivier shares:

I think they [the sprints] were the most interesting online events I attended during the COVID-19 crisis when all traditional on-site tech events were canceled. In particular the active planning by the Data Umbrella team for participants to work in pairs with audio rooms on Discord + a central help desk audio room worked really well.

The pre-sprint and post-sprint office hours also made it possible to limit the time spent on helping fix setup issues compared to what we experience in traditional sprints. They also forced us as maintainers to review and fix our documentation before the event.

Creation of supplementary resources in different media types

Data Umbrella coordinated the creation of a series of videos and transcripts that provided learning materials for the community to prepare for the sprint. These resources are available to the public and have a wide reach:

This is the Contributing to scikit-learn list of videos that were created for the sprints:

• Andreas Mueller: Crash Course in Contributing to scikit-learn
• Reshama Shaikh: Example of scikit-learn Pull Request
• Andreas Mueller: Sprint FAQs
• Thomas Fan: 3 Components for Reviewing a Pull Request
• Melissa Weber Mendonca: Sphinx for Python Documentation

Aspirations for Future scikit-learn Sprints

One of the primary goals of the Community sprints was to onboard new contributors who would become recurring contributors. This goal has generally not been realized. scikit-learn is a complex and advanced project, and a one-time sprint does not provide sufficient opportunity and support to sprint participants to become recurring contributors. A few sprint participants have progressed to become returning contributors, and it is a very small number relative to the number of sprint participants.

Onboarding a first-time contributor takes time. People who are contributing for the first time need to go through a lot of information simultaneously regarding both technical and organizational aspects of contributions. People may run into unexpected issues at the start depending on their setup and experience, might get frustrated and or discouraged and might not report the problem they are having (thinking it is their fault). Pre-event office hours have been successful at alleviating some of these roadblocks, for those sprint participants who have completed their pre-work.

Here are some adjustments that can be made in the future to reach the goal of recruiting recurring contributors:

• Provide mentoring
• Improve onboarding process
• Improve issues definitions
• Have sprints alongside tutorials
• Expand types of contributions that new contributors can make
• Have smaller sprint events

Mentoring
Sprints may not be sufficient for onboarding people. Mentoring is needed to take to the next level, and mentoring relationships can be established during sprint events.

Improve the onboarding process

While the scikit-learn project has improved significantly in the past few years as a result of feedback and learnings from the sprints, there is still room for improvement.

The scikit-learn project is complex, the contributor learning curve is steeper, and it has been getting more difficult to contribute to scikit-learn.

Improve issues definitions

There are 1600+ issues in the GitHub repository. Issues can be better defined and it would be valuable to break the issues into smaller steps which would be more approachable.

Sprints alongside tutorials

Scheduling sprints alongside tutorial sessions would be conducive in allowing users to connect the open source tool use cases with the motivation and product vision of scikit-learn.

Expand types of contributions

While the sprints have typically focused on documentation and code contributions, the project needs support in other areas. There is a backlog of open issues (1600+ !) and open pull requests (650+). The project needs support in triaging issues and reviewing pull requests. It would be beneficial to have sprint contributors work on increasingly complex issues.

Julien shares from personal experience:

In particular and in my opinion,reviewing pull requests is as valuable as authoring them. I also find it a preferable way to learn about scikit-learn internals rather than solving issues.

Have smaller sprints

Julien suggests:

Would sprints with a really small number of people (e.g. 2 mentees per mentor) be more valuable in the long term? Personally, I would prefer mentoring one or two people closely instead (ideally in-person) as I think it is more achievable, enjoyable and fruitful experience (this is something I am trying to do at the moment when I can get some time but I currently have limited of it).

Finally, I would also really treasure having in-person sprints [in Paris] with external (recurring) contributors (with a specific expertise) on advanced subjects when it is possible in the future.

Conclusion

Connecting and Supporting scikit-learn

To connect with the scikit-learn project, these are the most active social media platforms:

• Twitter: @scikit_learn
• LinkedIn: @scikit-learn

It is most welcome for users to “star” the code repository on GitHub: scikit-learn/scikit-learn

Our office hours, in addition to public developers and triage meetings are all posted on our Community Calendar.

The next Community sprint may be held at EuroScipy 2022 in Basel Switzerland in early September. Information on past and upcoming sprints are shared on our community site.

Contributing to scikit-learn

To contribute to scikit-learn, we have resources available here:

• English
• Spanish

There are additional resources for contributing:

• Contributing Videos
• English, Spanish and some Portuguese language transcripts

Appendix A: GitHub Contributors Comparison of Libraries

A comparison of the contributor base to other related libraries in the same space (May 2022):

• pandas: ~2560
• scikit-learn: ~ 2300 contributors
• numpy: ~ 1300 contributors
• matplotlib: ~1150
• scipy: ~1120

References

• Behind the Scenes: What It Takes to Run Data Umbrella’s scikit-learn Open Source Sprints
• Data Umbrella sprint reports
• Data Umbrella community sprint blogs
• Interview with Maren Westermann: Extending the Impact of the scikit-learn Sprints to the Community
• Interview with scikit-learn Triage Team Member: Juan Martín Loyola

May 18, 2022 12:00 AM UTC

May 17, 2022

PyCoder’s Weekly

Issue #525 (May 17, 2022)

#525 – MAY 17, 2022
View in Browser »

Analyze Code-Level Performance Across Your App’s Environment With Minimal Performance Overhead

Datadog’s profiler allows you to capture code profiles for all of your production instances. Compare those profiles in the profile comparison view to see how the performance of your code changes over time. You can quantify the changes you’ve made to fix a bottleneck. Try Datadog APM free →
DATADOG sponsor

Discussions

Python Jobs

Academic Innovation Developer (Ann Arbor, MI, USA)

University of Michigan

Software Development Lead (Ann Arbor, MI, USA)

University of Michigan

Senior Backend Engineer (Anywhere)

Doist

Senior Storytelling Framework Engineer - Python (France)

GoPro

Senior Software Engineer - Python Full Stack (USA)

Blenderbox

Gameful Learning Developer (Ann Arbor, MI, USA)

University of Michigan

Data & Operations Engineer (Ann Arbor, MI, USA)

University of Michigan

Lead Software Engineer (Anywhere)

Right Side Up

More Python Jobs >>>

Articles & Tutorials

Pulumi: Developer-First Infrastructure with Python

Developing on the cloud is complex. What if you could use your existing programming knowledge to build, deploy, and manage infrastructure on any cloud using your favorite languages and tools? With Pulumi you can. Get started for free at pulumi.com →
PULUMI sponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #525.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

May 17, 2022 07:30 PM UTC

Real Python

Using Python Class Constructors

Class constructors are a fundamental part of object-oriented programming in Python. They allow you to create and properly initialize objects of a given class, making those objects ready to use. Class constructors internally trigger Python’s instantiation process, which runs through two main steps: instance creation and instance initialization.

If you want to dive deeper into how Python internally constructs objects and learn how to customize the process, then this video course is for you.

In this video course, you’ll:

• Understand Python’s internal instantiation process
• Customize object initialization using .__init__()
• Fine-tune object creation by overriding .__new__()

With this knowledge, you’ll be able to tweak the creation and initialization of objects in your custom Python classes, which will give you control over the instantiation process at a more advanced level.

To better understand the examples and concepts in this tutorial, you should be familiar with object-oriented programming and special methods in Python.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

May 17, 2022 02:00 PM UTC

Python Insider

Python 3.9.13 is now available

May 17, 2022 01:33 PM UTC

Test and Code

188: Python's Rich, Textual, and Textualize - Innovating the CLI

Will McGugan has brought a lot of color to CLIs within Python due to Rich.
Then Textual started rethinking full command line applications, including layout with CSS.
And now Textualize, a new startup, is bringing CLI apps to the web.

Special Guest: Will McGugan.

Sponsored By:

• Rollbar: With Rollbar, developers deploy better software faster.

Links:

• rich
• rich-cli
• textual
• Textualize.io
• Rich Gallery
• Textualize Gallery
• Python Bytes Podcast

May 17, 2022 01:15 PM UTC

Python Software Foundation

The 2022 Python Language Summit: Python without the GIL

May 17, 2022 07:54 AM UTC

Nicola Iarocci

Eve-Swagger v0.2 released

I just released Eve-Swagger v0.2 on PyPI. Eve-Swagger is a Swagger/OpenAPI extension for Eve powered RESTful APIs. This maintenance release addresses a few issues and adds support for eve-auth-jwt. Many thanks to Roberto Romero for his contributions to this release. Subscribe to the newsletter, the RSS feed, or follow @nicolaiarocci on Twitter

May 17, 2022 06:05 AM UTC

Hynek Schlawack

Better Python Object Serialization

The Python standard library is full of underappreciated gems. One of them allows for simple and elegant function dispatching based on argument types. This makes it perfect for serialization of arbitrary objects – for example to JSON in web APIs and structured logs.

May 17, 2022 12:00 AM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Brasil
  • Planet Python Poland
• Python Libraries
  • OLPC
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Mathspp Pydon'ts"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Ahmed Bouchefra
  • Al-Ahmadgaid Asaad
  • Alec Munro
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • Armin Ronacher
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Automating OSINT
  • Awesome Python Applications
  • Baiju Muthukadan
  • Bajusz Tamás
  • BeDjango
  • Ben Bass
  • Ben Cook
  • Ben Rousch
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • BioPython News
  • Bit of Cheese
  • Bojan Mihelac
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Okken
  • Bruno Oliveira
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Düvel
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Casey Duncan
  • Catherine Devlin
  • Ching-Hwa Yu
  • Chris Hager
  • Chris Miles
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Heimes
  • Christian Scholz
  • Christoph Zwerschke
  • CodeGrades
  • CodeSnipers
  • Codementor
  • CodersLegacy
  • Coding Diet
  • Corey Gallon
  • Corey Goldberg
  • Cormoran Project
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dallas Fort Worth Pythoneers
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Data Community DC
  • Data School
  • Dave Beazley
  • David Amos
  • David Caron
  • David Goodger
  • David Lindelof
  • David MacIver
  • David Malcolm
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • Denis Kurov
  • Django Weblog
  • Django Weekly
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Dougal Matthews
  • Duncan McGreggor
  • Ed Crewe
  • Edward K. Ream
  • Eli Bendersky
  • Eniram Ltd.
  • Eray Özkural (examachine)
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Evennia
  • Everyday Superpowers
  • Fabio Zadrozny
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • Gocept Weblog
  • Godson Gera
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Gustavo Narea
  • Gustavo Niemeyer
  • Gökhan Sever
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Humberto Rocha
  • Hynek Schlawack
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Inspired Python
  • Ionel Cristian Maries
  • IronPython-URLs
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • ItsMyCode
  • Ivan Velichko
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • Janusworx
  • Jarrod Millman
  • Jean-Paul Calderone
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Burns
  • John Cook
  • John Jacobsen
  • John Ludhi/nbshare.io
  • Jon Parise
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan Street
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Julien Danjou
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kodnito
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • LAAC Technology
  • Laurent Szyster
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lintel Technologies
  • Linux Stans
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Lucas Cimon
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mats Kindahl
  • Matt Layman
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Matthew Wright
  • Mattias Brändström
  • Mauveweb
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Nelson
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mirek Długosz
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Naomi Ceder
  • Natan Zabkar
  • Ned Batchelder
  • Neil Schemenauer
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • Nsukami Patrick
  • Obey the Testing Goat
  • Ofosos
  • Omaha Python Users Group
  • Ondřej Čertík
  • Paolo Amoroso
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patrick Kennedy
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • Pedro Lima
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyATL Bytecode
  • PyBites
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyPy
  • PyTennessee
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python GUIs
  • Python Insider
  • Python Morsels
  • Python Piedmont Triad User Group
  • Python Pool
  • Python Software Foundation
  • Python Sweetness
  • Python User Groups
  • Python for Beginners
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • Pythonicity
  • Pythonology
  • Python⇒Speed
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Rene Dudfield
  • Reuven Lerner
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rob Galanakis
  • Robert Collins
  • Robert Zaremba
  • Robin Parmar
  • Robin Wilson
  • Rodrigo Araúj
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Sandipan Dey
  • Sandro Tosi
  • Sean McGrath
  • Sebastian Pölsterl
  • Sebastian Witowski
  • Selena Deckelmann
  • Senthil Kumaran
  • Shannon -jj Behrens
  • ShiningPanda
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Holden
  • Steven Klass
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Suresh Dasari/Tutlane.com
  • Swisscom ICT
  • Talk Python to Me
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Terri Oda
  • Terry Jones
  • Test and Code
  • TestDriven.io
  • The Data Scientist
  • The Digital Cat
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Open Sourcerer
  • The Parcon Blog
  • The Python Coding Blog
  • The Python Papers
  • The Three of Wands
  • Thibauld Nion
  • ThisHosting.Rocks
  • Thomas Guest
  • Thomas Vander Stichele
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Knapp
  • Tim Lesher
  • Tobias Ivarsson
  • Tom Christie
  • Tomasz Ducin
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinayak Mehta
  • Virgil Dupras
  • Vladimir Perić
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • Wing Tips
  • Wingware
  • Wyatt Baldwin
  • Yaniv Aknin
  • Yann Larrivée
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero to Mastery
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • death and gravity
  • eGenix.com
  • hypothesis.works articles
  • kdev-python
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • scikit-learn
  • testmon
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal:
    Open an issue on github