skip to navigation
skip to content

Planet Python

Last update: June 18, 2019 04:47 AM UTC

June 17, 2019

Zato Blog

Zato 3.1 Released - Open-Source Python-based API Integrations and Backend Application Server

The newest version of Zato, the open-source Python-based enterprise API integrations platform and backend application server, is out with a lot of interesting features, changes and additions.

The full changelog is here and below is a summary of what is new in 3.1:

What is Zato?

Zato is an open-source API integrations platform and backend application server composed of several major blocks of functionality:

Its HA architecture is highly-scalable and everything comes with a web-based GUI along with a command-line interface and admin APIs.

If you are looking for a highly productive Python-based open-source platform designed specifically to integrate systems or expose APIs in many protocols or data formats, to be used by other servers, frontends or mobile, Zato is the choice.

Quick links:

June 17, 2019 10:01 PM UTC

NumFOCUS

Google Renews Corporate Sponsorship of NumFOCUS

The post Google Renews Corporate Sponsorship of NumFOCUS appeared first on NumFOCUS.

June 17, 2019 03:34 PM UTC

Real Python

Python Community Interview With Marlene Mhangami

We are joined today by Marlene Mhangami. Marlene is a passionate Pythonista who is not only using tech to facilitate social change and empower Zimbabwean women but is also the chair of the very first PyCon Africa. Join me as we talk about her non-traditional start in tech, as well as her passion for using technology to create social change for good.

Ricky: Welcome, Marlene! I’m so glad you could join me for this interview. Let’s start with the usual first question. How’d you get into programming, and when did you start using Python?

Marlene Mhangami

Marlene: Well, my background is in molecular biology, so in college, although I had some interaction with code, it wasn’t really something I used regularly. I’ve always been interested in technology and the impact it has on the world, but growing up I didn’t really have anyone to introduce me to the concept of coding. We had one computer that we shared in my house, and my brothers were super into gaming. They were on it all the time.

In high school, I took a computer class, but my teachers, who I’m sure were really nice people, didn’t go much further than the basics of using Microsoft Office and having us memorize names of different parts of hardware. So, in general, I focused more on life and physical sciences, and I’m still really interested in those fields today.

I studied for a bit in the United States for college, and I remember coming home one summer and being really aware of how different Zimbabwe was from the U.S. From really small cultural differences like how people address conflict (which is something I’m still trying to figure out with my US friends) to much more impactful things like access to knowledge and education.

I decided I wanted to start being more involved with my local community, and for some reason, I decided that I wanted to leverage technology to help me do that. I also had a really vivid dream that made me re-evaluate what I was doing with my life.

After googling around and actually organizing a meetup, I got introduced to one of my co-founders, Ronald Maravanykia, who was at the time running a Django girls workshop in Harare. He introduced me to Python as a great educational tool for teaching programming to people who don’t have a computer science background. That was almost three years ago, and I’m super grateful Python ended up being the language I chose to focus on. It’s really great!

Ricky: As a Zimbabwean, you are passionate about empowering Zimbabwean women in tech and helping today’s young girls pursue a career in tech. So much so that you are the programs director at the non-profit ZimboPy?

Marlene: Yes, as I mentioned earlier, using technology to create social change for good is something that’s consistently at the forefront of my mind. My co-founders and I decided to create ZimboPy as a way to give girls in Zim access to knowledge and experiences they wouldn’t necessarily get anywhere else in the country.

I think for me, the end goal is to create a model where we introduce girls to programming, and if they enjoy it, offering them a pathway to get jobs in the field. The tech world is not perfect, but unlike most fields, there are ways that you can by-pass traditional gatekeepers and be successful regardless of whether or not you have the privilege of receiving formal higher education.

We are still a small non-profit, and I’m learning new things every day, but we’ve had some great success stories that give me so much hope! Also, our mentorship weeks, where we introduce girls to women who are doing incredible things in the field and help them build something new, are my favorite thing to do all year! We’ve had incredible women like Ashley McNamara and Emily Freeman speak. It’s always so much fun!

Ricky: You are one of the 13 members of the Python Software Foundation’s (PSF) Board of Directors. How did you come to be involved with the PSF, and what do you think is the biggest challenge as we look towards 2020 and beyond?

Marlene: I’m actually up for re-election this year, so hopefully, I’ll still be on the board for 2020. I really had such a positive experience with it so far!

I was actually introduced to the PSF by Lorena Mesa, who was volunteering to speak at one of our ZimboPy mentorship weeks in early 2017. She was on the board at the time, and we spoke about it around the time when elections were coming up. She really felt that I would be a great addition to the board. At that time, no one from Africa had ever held a seat, so she was super encouraging.

I remember looking through the list of the nominees and thinking that I didn’t belong there. I had done some work with the local Python community in Zimbabwe, but I felt really intimidated by everyone’s profiles. I ended up agreeing to let Lorena nominate me, mainly because I had nothing to lose, haha! Surprisingly, I was voted in, which I think is purely a testament to the Python community’s commitment to diversity and global representation.

So many changes have taken place in the last few years: the formation of the steering council, the exponential growth of PyCon, and the spread of the language globally. All of these are things we’ve been able to discuss as a board.

I think defining the biggest challenge is tricky because it genuinely depends on each individual’s perspective. There are Directors on the board who are infinitely more technical than I am, and they can probably speak to all sorts of challenges on the future of Python from a more technical perspective.

For me, my focus is to help the growth of Python and its community to be as inclusive and representative as possible. I recently became the board’s communications co-chair, along with Lorena, and have been working with the blogging team. I’d like to see an increase in the representation of voices from communities around the world, maybe one day having blog posts and content on the official site in various languages.

All of these are things I feel are important and a challenge that we have to tackle well.

Ricky: I had the good fortune to meet you in person at PyCon 2019. It was your first PyCon (mine, too). How was your experience there? What was the standout moment for you?

Marlene: I genuinely loved Pycon 2019, and of course meeting you, Ricky, and the rest of the Real Python team! I’ve been to several PyCons around Africa and other parts of the world, but PyCon US is definitely a unique experience.

One of my favorite things about the conference was how friendly and laid back everyone was. I’d literally pass by people that I follow on Twitter, who have done amazing things, and they’d just be sitting there, eating a sandwich! I’m being a little funny here, but honestly, everyone was so welcoming and fun. I still can’t fully wrap my head around it.

I had so many standout moments, but I think I’d say Sha Wallace and Jessica Meckeller’s joint keynote was out of this world. I was tearing up because it’s such a great example of how we as a community can create positive change! I took away so many notes from that talk that I’d like to implement here in Zimbabwe.

This is an additional thing, I know, but I’d also like to say that I had so many standout moments outside the convention center, like having dinner with people from around the world and going to the local art gallery and trying bubble tea, also generally just wandering about the city. I had the best time, and I’m really looking forward to Pittsburg in 2020!

Ricky: If my memory serves right, at the end of PyCon, Microsoft and Adafruit donated a bunch of Circuit Playground Lunchbox Kits for you to take back to the girls you mentor in Zimbabwe. I’m curious if you managed to put them to use yet? Or what your plans are, outside of using them to teach them how to write some CircuitPython?

Marlene: Yes! That was another standout moment (to add to the twenty others above, haha). I got to chat with Nina Zakharenko and Katni from Adafruit, who are some of the loveliest human beings I’ve ever met! We had dinner together at some point and afterwards, they decided to donate the kits for me to bring back home and use for our ZimboPy programs.

I only recently got back to Zimbabwe, and I’ve started to play around with the kits myself. The plan at the moment is to host a workshop specifically centered around CircuitPython to show the girls that they can be playful and have fun while they code. I’m super pumped about it!

I was also talking to Kat about the potential of me hosting a tutorial at PyCon Africa with the Adafruit boards, similar to what she was demoing at the Microsoft booth. If I feel confident enough to do it by July, I think that’s something attendees at the conference should look out for!

Ricky: As if all the community work you do already isn’t enough, you’re also the organizer of the very first PyCon Africa! How’s the planning coming on? Can you share any details?

Marlene: I’m really excited that we’re in the final stages of planning for the conference. PyCon Africa is the first regional / pan-African gathering of the Python community! We have a great organizing committee, with people who have helped organize PyCons in Nigeria, Ghana, Namibia, Uganda, South Africa, and even the UK, so in that respect, I’ve been really lucky to land on this team.

The response from the global Python community has also been really encouraging. Real Python is actually one of our sponsors, which is amazing. Shout out to you! I think we’re moving in the right direction.

I’m this year’s chair, and although everything is running smoothly so far, I’ve been surprised by how all-consuming the process of planning can feel at times. Each time I look up it’s as if 3 months have passed! At PyCon US, I was speaking to a few other regional conference organizers and was glad to find I wasn’t the only one that felt like this. It can feel pretty overwhelming sometimes, but for the most part, I’m happy with the progress we’ve made and optimistic that we will put on a great event!

The conference is going to be held in Accra, Ghana, this year from August 6 to 10. We’re looking for sponsors, and there’s an option for individuals to buy a sharing ticket to sponsor a Pythonista who doesn’t have the means to buy their own ticket! You can find out more info on all this on our website: africa.pycon.org

Ricky: Now for my last questions. What else do you get up to in your spare time? Any other hobbies and interests, aside from Python and coding?

Marlene: I really enjoy long walks on the beach, haha. I’m joking, but I also genuinely do! I’ve been getting into running recently, and I’m hoping to try to get myself into good enough form to do a marathon at some point.

I also really like painting, but not in a cultured, cool way. I sometimes just like to paint and draw as a way to relax. Meditation and gratefulness journaling is also something I like to do each morning, as a good way to start the day.

It was an absolute pleasure speaking to Marlene. If you want to attend PyCon Africa, tickets are still available. If you want to find out more about Marlene, or reach out to her and discuss any of the amazing things she is doing, then you can find her on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

June 17, 2019 02:00 PM UTC

Stack Abuse

Introduction to OpenCV with Python

Introduction

In this tutorial, we are going to learn how to use OpenCV library in Python. OpenCV is an open source library which is supported by multiple platforms including Windows, Linux, and MacOS, and is available for use in multiple other languages as well; however, it is most commonly used in Python for Machine Learning applications, specifically in the Computer Vision domain.

Apart from its cross-platform support and availability in multiple other computer languages, which allows applications developed in it to be used on different systems, OpenCV is also, in comparison to other such libraries, fairly efficient in terms of computations, as it uses vector operations for most of its functions.

In this tutorial, we'll cover OpenCV installation on Mac, Windows, and Linux, image operations, image arithmetics, image smoothing, and geometric transformations using OpenCV. So without further ado, let's start.

Installation

Note: Since we are going to use OpenCV in the Python language, it is an implicit requirement that you already have Python (version 3) installed on your workstation. Depending upon your OS, execute one of the following commands to install OpenCV library on your system:

Windows

$ pip install opencv-python

MacOS

$ brew install opencv3 --with-contrib --with-python3

Linux

$ sudo apt-get install libopencv-dev python-opencv

To check if your installation was successful or not, run the following command in either a Python shell, or your command prompt/ terminal:

import cv2

If you do not get an error on importing cv2 then it was installed correctly.

Basic Image Operations

Now that we have installed OpenCV on our workstations, let's get our hands dirty with some of the functionalities that OpenCV offers.

Display an Image

Displaying an image using OpenCV is a two-step process; first, we have to load it, and then we can display it. Both operations are done in sequence using different functions.

To display an image, we need to know two things:

  1. Image Path (both absolute and relative paths work)
  2. Read Mode (read, write, etc.)

The function we'll use for reading/loading an image is cv2.imread(), which has two variations. First one is IMREAD_GRAYSCALE, which as the name suggests, converts the image to grayscale before reading it. The second one is IMREAD_UNCHANGED, which loads the image without cutting out the alpha channel. The default is IMREAD_COLOR, which simply reads the colored image using the RGB channels only.

Let's code an example:

import cv2

my_bike = cv2.imread('bike.png')

This will load the image of a bike from the file system and store it in the my_bike variable for further operations

Note: If you get an error from the above code, there are only three possible reasons for it. The first one is that the path you specified is incorrect, second is that the image file you specified doesn't exist, and the last one is that the image type (jpg/jpeg/png) in the image path is incorrect.

Let's now display the image we just read. It can be done by the cv2.imshow() function. If you have used Matlab, you may be familiar with these image operations.

cv2.imshow('my_bike', my_bike)

The first parameter to the imshow() function is the string name that you want to display on the image window. The second parameter is the image handler we created using the cv2.imread() function.

Saving an Image

Saving an image is a very commonly used feature, as we may need to update our image and save the changes to the file system for later use. OpenCV has an cv2.imwrite() function to save images.

Here is an example:

cv2.imwrite('bike.png', my_bike)

Here we specify the name and current location of the image. The resulting image is automatically saved in the current working directory.

Arithmetic Operations on Images

Arithmetic operations on images refer to adding, subtracting, multiplying, or dividing multiple images to generate a new image which is an arithmetic combination of the input images. Image arithmetics has a lot of applications, like adding a watermark to an image, creating a blended combination of two images, applying different types of image filters, etc.

While there are many operations you can perform, we will only be showing two examples here, as this will then allow you to apply the concept to other arithmetic operations available in OpenCV. The first example will be the addition of two images, and the second example will be blending two images.

Let's code these two examples:

Adding Images

import cv2

# Read in the two images
image_1 = cv2.imread('bike.jpg')  
image_2 = cv2.imread('car.jpg')

# Sum the two image arrays for all channels
result = cv2.add(image_1, image_2)

cv2.imshow('result', result)  
cv2.waitKey(0)  
cv2.destroyAllWindows()

The waitKey command will wait for you to press a key before it moves on to the next command. This is useful so that the program will continue to display your image until a key is pressed, otherwise it will be displayed for a split second and then quickly disappear once the program has stopped executing.

Blending Images

Blending images is similar to image addition, except each image's contribution to the new resulting image can be controlled. Basically, if we want one image to be more focused, and the other one to be more faint when they get merged, we will go with blending, instead of simple addition.

Lets code it to clarify further:

import cv2

# Read in the two images
image_1 = cv2.imread('bike.jpg')  
image_2 = cv2.imread('car.jpg')

result = cv2.addWeighted(image_1, 0.9, image_2, 0.1)

cv2.imshow('result', result)  
cv2.waitKey(0) # Wait for the user to press a key before continuing  
cv2.destroyAllWindows()

The sum of the weights given to the addWeighted function should be equal to 1.0. You can also give a scalar value at the end, which would be added to all the pixel values of the resultant image.

Note: The images can be of any type; however, the type should be the same for all images. For instance, if you are using PNG format, all images being used for computation should be in PNG format as well.

Image Smoothing

Image smoothing is a very helpful feature, which is mostly performed before the images are passed on to a machine learning model. It is mostly done to remove noise/high-frequency elements from images by passing the image through a low-pass filter. There are many filters, including box filter (averaging filter), median filter, mode filter, Gaussian filter, and many more; however, to understand image smoothing and how to do it using OpenCV, we will only cover the box filter.

Let's say you have an image of 10x10, and you want to pass it through a 3x3 box/averaging filter, how would you do it?

You'll start with the top left of the image, place your 3x3 filter there, and replace the central element with the average of all 9 elements. This was the first step, now you will move your filter one step to the right, and repeat the same process until you have covered the whole image. An example of 10x10 image, and 3x3 averaging filter are shown below for your reference:

Filter/Mask:

Filter being applied on 10x10 Image:

Now that we have discussed how it works, lets try and see how we can apply different filters on our image using OpenCV; please read the comments thoroughly to know which line of code is used for which filter:

import cv2

# Load the original image
original_image = cv2.imread('my_bike.png')

# Filter by passing image through 3x3 averaging filter
average_image = cv2.blur(original_image,(3,3))

# Apply 3x3 gaussian filter on the original image
gaussian_image = cv2.GaussianBlur((original_image,(3,3),0))

# Apply 3x3 median filter on the original image
median_image = cv2.medianBlur(original_image,3)

Note: You can view the resulting images by using the following additional code:

import matplotlib.pyplot as plt

plt.imshow(average_image)  
plt.show()

Image Transformations

Image transformation is the last, but one of the most important topics that we are going to cover with OpenCV. It has a lot of applications, but one of the most common ones nowadays is in Machine Learning for Data Augmentation, i.e. when you have a shortage of dataset, you augment/transform the currently available images to make them different from your existing dataset. This effectively increases your dataset size and might help in improving your model accuracy.

The list of possible transformations is a long one, including scaling, affine, rotation, translation, etc. We will only cover two of them using OpenCV to get a general idea; however, OpenCV provides supporting functions for a wide range of them. Let's start with scaling.

Scaling

To put it in simple words, scaling is basically just resizing your image, i.e. either making it bigger or smaller. resize is the function used for scaling the images in OpenCV. Resizing has three types: INTER_CUBIC, INTER_LINEAR, and INTER_AREA. Let's code an example using these functions for scaling; please read through the code, comments, and descriptions carefully as they will explain what exactly is going on in the code:

import cv2  
import numpy as np  
import matplotlib.pyplot as plt

image = cv2.imread('my_bike.jpg')

# Scale up/expand both width and height by factor of 2
result_1 = cv2.resize(image, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC)

# Scale down/shrink both width and height by factor of 2
result_2 = cv2.resize(image, None, fx=2, fy=2, interpolation=cv2.INTER_AREA)

# Display the resulting images
plt.imshow(result_1)  
plt.imshow(result_2)  
plt.show()

Here in the resize function, the fx parameter in represents the scale factor for width, fy represents the scale factor height, and interpolation specifies the function to be used for scaling (shrinking or expansion).

Rotation

Rotation allows us to move an image about the axis for a certain specified angle.

Before we learn how to rotate our images using code, we should know that there is a rotation matrix that is used for performing this transformation; we will not go in details of that, as OpenCV makes it very simple for us to calculate that matrix using a single function call. You will see that in the code below:

import cv2  
import matplotlib.pyplot as plt

# Load the image of a bike
image = cv2.imread('my_bike.jpg',0)

# Rows and columns
r, c = image.shape

matrix = cv2.getRotationMatrix2D((cols/2,rows/2), 180, 1)  
result = cv2.warpAffine(image,matrix,(c,r))

# Display resulting rotation
plt.imshow(result)  
plt.show()

In the getRotationMatrix2D function, 180 specifies the degree by which the image should be rotated, 1 is the scaling factor, the function call would return the rotation matrix in the matrix variable.

The warpAffine function call uses the matrix we calculated from the previous method to rotate the image according to our specifications.

Conclusion

To conclude it all, let's reiterate over some important points that we discussed in this article. OpenCV is a library available in multiple languages and is mostly used in conjunction with NumPy, SciPy and Matplotlib, as we saw in some of the examples above, as well. Some of its functions are the same as in Matlab, and it also supports vectorized operations, hence increasing computational efficiency.

Furthermore, OpenCV is one of the best libraries out there for the Computer Vision domain, and after reading this article, you should be able to go ahead and search up some computer vision / machine learning applications that have been developed using OpenCV.

Let it be known that this article was just the tip of the iceberg, and OpenCV has a lot more to offer. Reading this should enable you to dive deeper and learn about other advanced features that OpenCV has to offer.

June 17, 2019 12:26 PM UTC

Reuven Lerner

Reminder: Early-bird pricing for Weekly Python Exercise ends tomorrow

This is just a quick reminder that if you want to join the advanced cohort of Weekly Python Exercise starting July 2nd, you should do it by tomorrow (Tuesday, June 18th).

Don’t miss this opportunity to improve your Python coding skills! We’ll be talking about iterators, generators, decorators, threads, and functional programming, and helping you to improve your skills.

Questions? Just e-mail me at reuven@lerner.co.il. But hurry, before the price goes up!

The post Reminder: Early-bird pricing for Weekly Python Exercise ends tomorrow appeared first on Reuven Lerner.

June 17, 2019 12:01 PM UTC

Catalin George Festila

Python Qt5 - the most simple QTreeWidget - part 002.

This tutorial uses PyQt5 and Python version 3.7.3. Let's install the PyQt5 python module with the pip tool: C:\Python373\Scripts>pip install PyQt5 Collecting PyQt5 ... Successfully installed PyQt5-5.12.2 PyQt5-sip-4.19.17Let's see one simple example with comments about how to use QTreeWidget. import sys from PyQt5.QtWidgets import QTreeWidget, QTreeWidgetItem, QApplication, QWidget if __name__ =

June 17, 2019 11:41 AM UTC

Stanislas Morbieu

AI Paris 2019 in one picture

Last week, I was at the AI Paris 2019 event to represent Kernix. We had great talks with so many people, and I barely had time to go around to look what other companies were working on. That is why I look at this afterwards. Can we have a big picture of the event without being there?

It happens that the list of the companies is on the website of the AI Paris 2019. With a little bit of Python code we can retrieve it. The following Python code retrieves the html content of the page we are interested in:

import urllib


url = "https://aiparis.fr/2019/sponsors-et-exposants/"

with urllib.request.urlopen(url) as response:
    html = response.read().decode('utf-8')

Note that response.read() returns bytes, so we have to decode it to get a string. The encoding used is utf-8, which can be seen in the charset attribute of the meta tag (and fortunately it is the correct encoding, which is not always the case). Looking at the html, we can notice that the data is in a javascript variable. Let's get the line where it is defined:

import re


line = re.findall(".*Kernix.*", html)[0]

By default, the regex does not span multiple lines. The line variable is almost a json. We can get the data as a Python dictionary with the following:

import json


i = line.index('[')
d = '{"all":' + line[i:-1] + '}'
data = json.loads(d)

Let's now get the descriptions of the companies concatenated into a single string:

import re


def clean_html(html):
    regex = re.compile('<.*?>')
    text = re.sub(regex, '', html)

    return text

corpus = [clean_html(e['description']) for e in data['all']]
text = " ".join(corpus)

The clean_html function is created since the descriptions contain html code and removing the tags make it more readable. A way to go when dealing with text is to remove "stopwords", i.e. words that do not contain meaningfull information:

from nltk.corpus import stopwords


stop_words = (set(stopwords.words('french'))
              | {'le', 'leur', 'afin', 'les', 'leurs'}
              | set(stopwords.words('english')))

NLTK library provides lists of stopwords. We use both the english and french lists since the content is mosly written in french, but some descriptions are in english. I added some french stopwords.

I am not a big fan of word clouds since we can only spot quickly a few words, but for this use case, we can get a pretty good big picture in very few lines:

AI Paris 2019 word cloud

Here is the code to generate it:

import random

import matplotlib.pyplot as plt
from wordcloud import WordCloud


def grey_color_func(word, font_size,
                    position, orientation,
                    random_state=None,
                    **kwargs):
    return "hsl(0, 0%%, %d%%)" % random.randint(60, 100)


wordcloud = WordCloud(width=1600, height=800,
                      background_color="black",
                      stopwords=stop_words,
                      random_state=1,
                      color_func=grey_color_func).generate(text)
fig = plt.figure(figsize=(20, 10))
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis("off")
plt.tight_layout(pad=0)
fig.savefig('wordcloud.png', dpi=fig.dpi)
plt.show()

The grey_color_func function enable a greyscale picture. A fixed random state enables reproductibility. Setting both the width, height and the figsize enable a high quality output. The margins are removed with the two lines:

plt.axis("off")
plt.tight_layout(pad=0)

Let's get back to the word cloud. We can spot immediately the words "solution" (it is the same in french and english), "data" (both in french "donnée" and english "data") and "company" ("entreprise" in french). The expressions "AI" ("IA" in french) and "artificial intelligence" ("intelligence artificielle") come only later which is quite surprising for an event focusing on artificial intelligence as a whole. In fact, AI is often seen as a solution to use data in an useful way.

Many actors are working on how to store data, manage and process it (which is very important), but only a few, from what I saw and heard, focus on creating value for people. I am always delighted to create something useful for people in they day-to-day life, whether it is a recommender system to discover things we never thought exist, or automate boring tasks, improve research in the pharmaceutical sector, and so many others. AI is about people!

June 17, 2019 08:00 AM UTC

Mike Driscoll

PyDev of the Week: Meredydd Luff

This week we welcome Meredydd Luff (@meredydd) as our PyDev of the Week! Meredydd is the co-founder of Anvil and a core developer for the Skulpt package. You can learn more about Meredydd on his website. Let’s take a few moments to get to know him better!

Can you tell us a little about yourself (hobbies, education, etc):

I’ve loved programming since I was first introduced to BASIC at the age of 7. I come from Cambridge (the old one in the UK, not the relatively-new one near Boston), and I studied here too. I actually started out as a biologist, but then switched to computer science for my PhD.

I think programming is the closest thing to magic we have, and I love watching and helping people get their hands on this power. My PhD research was about building usable parallel programming systems, and now I work on Anvil, a tool to make web programming faster and easier for everyone (with Python!).

When I’m not programming, I fly light aeroplanes, which I guess is what happens when your inner six-year-old makes your life decisions. I used to dance competitively (including a few years on England’s top Latin formation team), but it turns out international competitions and startups don’t play well together, so the startup won.

Why did you start using Python?

I’d dabbled in Python a bit, but I only really started using it in earnest when we started creating Anvil. We wanted to make web development easier, by replacing the mess of five(!) different programming languages with one language and a sensible visual designer. Python was the obvious choice – it’s accessible, it’s predictable, and it has a huge and powerful ecosystem.

What other programming languages do you know and which is your favorite?

I’m a big fan of Clojure. It’s sort of the diametrical opposite of Python. Python is simple, concrete and predictable – it’s really a programming language designed for people. By contrast, Lisps like Clojure turn the abstraction up to 11, and make the person program like the compiler thinks.

I also have to tip my hat to C – if I’m using C, I must be having an adventure close to the hardware 🙂

What projects are you working on now?

These days I spend all my time on Anvil, a platform for building full-stack web apps with nothing but Python. There’s a drag-and-drop designer for your UIs, we run your client-side Python in the browser, and your server-side Python runs in our own serverless environment. We even have a Python-native database you can use.

So, whereas previously you’d need to learn HTML+CSS+JS+Python+SQL (plus all the frameworks, AWS, etc), now anyone who can write Python can build and deploy a web application with Anvil.

Which Python libraries are your favorite (core or 3rd party)?

It’s tough, but I’d have to choose Skulpt, the Python-to-Javascript compiler. We’d used before in an educational context, but we use it really heavily in Anvil. Obviously Skulpt is how we run client-side Python in the browser, but we use it in other ways too – for example, we use Skulpt’s parser to drive our Python code completion! (I talked briefly about how our Python autocompleter works at PyCon UK.)

I’m one of the core maintainers these days – I’m currently working on a tear-down-and-rebuild of the front end, which is great fun for compiler nerds. If you want to join in, please drop us a line on GitHub!

Where did the idea behind Skulpt come from?

I can’t claim credit for Skulpt’s existence – the project was started by Scott Graham, and these days there’s a whole team of us. The original impetus was around education: When you’re first learning to code, setting up a Python environment is a big hassle, and so having a playground “just there” in the browser is a massive win. I suppose Anvil is one step further – we put a full-strength application development and deployment platform “just there” in your browser.

Can you tell us the story behind Anvil?

My cofounder Ian and I both learned to program using Visual Basic and similar tools. The 90s were a sort of golden age for that: Anyone who could use one (fairly simple) programming language could build apps that looked and worked like everything else on their desktop.

These days, everything is on the web, but the barrier to entry is huge: you need to learn all these languages and frameworks, plus Linux system administration, just to build your first app. It’s exhausting – and it cuts off so much opportunity for people like data scientists or electronic engineers, who have a job to do and don’t have time to learn all that stuff. Eventually, Ian and I got fed up of moaning about the situation, and decided to build something to fix it!

Anvil’s goal is to make web programming usable by everyone, but still powerful enough for seasoned professionals. We cut out the incidental complexity, but we keep the powerful programming language and the huge ecosystem.

Is there anything else you’d like to say?

Oh, yes – Use autocomplete!

Thanks for doing the interview, Meredydd!

The post PyDev of the Week: Meredydd Luff appeared first on The Mouse Vs. The Python.

June 17, 2019 05:05 AM UTC

Kushal Das

DMARC, mailing list, yahoo and gmail

Last Friday late night, I suddenly started getting a lot of bounced emails from the dgplug mailing list. Within a few minutes, I received more than a thousand emails. A quick look inside of the bounce emails showed the following error:

Unauthenticated email from yahoo.in is not accepted due to     domain's 550-5.7.1 DMARC policy.

Gmail was blocking one person’s email via our list (he sent that using Yahoo and from his iPhone client), and caused more than 1700 gmail users in our list in the nomail block unless they check for the mailman’s email and click to reenable their membership.

I panicked for a couple of minutes and then started manually clicking on the mailman2 UI for each user to unblock them. However, that was too many clicks. Suddenly I remembered the suggestion from Saptak about using JavaScript to do this kind of work. Even though I tried to learn JavaScript 4 times and failed happily, I thought a bit searching on Duckduckgo and search/replace within example code can help me out.

$checkboxes = document.querySelectorAll("[name$=nomail]");
for (var i=0; i<$checkboxes.length; i++)  {
      $checkboxes[i].checked = false;
}

The above small script helped me to uncheck 50 email addresses at a time, and I managed to unblock the email addresses without spending too many hours clicking.

I have also modified the mailing list DMARC settings as suggested. Now, have to wait and see if this happens again.

June 17, 2019 05:02 AM UTC

Podcast.__init__

Algorithmic Trading In Python Using Open Tools And Open Data

Algorithmic trading is a field that has grown in recent years due to the availability of cheap computing and platforms that grant access to historical financial data. QuantConnect is a business that has focused on community engagement and open data access to grant opportunities for learning and growth to their users. In this episode CEO Jared Broad and senior engineer Alex Catarino explain how they have built an open source engine for testing and running algorithmic trading strategies in multiple languages, the challenges of collecting and serving currrent and historical financial data, and how they provide training and opportunity to their community members. If you are curious about the financial industry and want to try it out for yourself then be sure to listen to this episode and experiment with the QuantConnect platform for free.

Summary

Algorithmic trading is a field that has grown in recent years due to the availability of cheap computing and platforms that grant access to historical financial data. QuantConnect is a business that has focused on community engagement and open data access to grant opportunities for learning and growth to their users. In this episode CEO Jared Broad and senior engineer Alex Catarino explain how they have built an open source engine for testing and running algorithmic trading strategies in multiple languages, the challenges of collecting and serving currrent and historical financial data, and how they provide training and opportunity to their community members. If you are curious about the financial industry and want to try it out for yourself then be sure to listen to this episode and experiment with the QuantConnect platform for free.

Announcements

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, scalable shared block storage, node balancers, and a 40 Gbit/s public network, all controlled by a brand new API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they just launched dedicated CPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • And to keep track of how your team is progressing on building new features and squashing bugs, you need a project management system designed by software engineers, for software engineers. Clubhouse lets you craft a workflow that fits your style, including per-team tasks, cross-project epics, a large suite of pre-built integrations, and a simple API for crafting your own. With such an intuitive tool it’s easy to make sure that everyone in the business is on the same page. Podcast.init listeners get 2 months free on any plan by going to pythonpodcast.com/clubhouse today and signing up for a trial.
  • You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to pythonpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • The Python Software Foundation is the lifeblood of the community, supporting all of us who want to run workshops and conferences, run development sprints or meetups, and ensuring that PyCon is a success every year. They have extended the deadline for their 2019 fundraiser until June 30th and they need help to make sure they reach their goal. Go to pythonpodcast.com/psf today to make a donation. If you’re listening to this after June 30th of 2019 then consider making a donation anyway!
  • Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at pythonpodcast.com/chat
  • Your host as usual is Tobias Macey and today I’m interviewing Jared Broad and Alex Catarino about QuantConnect, a platform for building and testing algorithmic trading strategies on open data and cloud resources

Interview

  • Introductions
  • How did you get introduced to Python?
  • Can you start by explaining what QuantConnect is and how the business got started?
  • What is your mission for the company?
  • I know that there are a few other entrants in this market. Can you briefly outline how you compare to the other platforms and maybe characterize the state of the industry?
  • What are the main ways that you and your customers use Python?
  • For someone who is new to the space can you talk through what is involved in writing and testing a trading algorithm?
  • Can you talk through how QuantConnect itself is architected and some of the products and components that comprise your overall platform?
  • I noticed that your trading engine is open source. What was your motivation for making that freely available and how has it influenced your design and development of the project?
  • I know that the core product is built in C# and offers a bridge to Python. Can you talk through how that is implemented?
    • How do you address latency and performance when bridging those two runtimes given the time sensitivity of the problem domain?
  • What are the benefits of using Python for algorithmic trading and what are its shortcomings?
    • How useful and practical are machine learning techniques in this domain?
  • Can you also talk through what Alpha Streams is, including what makes it unique and how it benefits the users of your platform?
  • I appreciate the work that you are doing to foster a community around your platform. What are your strategies for building and supporting that interaction and how does it play into your product design?
  • What are the categories of users who tend to join and engage with your community?
  • What are some of the most interesting, innovative, or unexpected tactics that you have seen your users employ?
  • For someone who is interested in getting started on QuantConnect what is the onboarding process like?
    • What are some resources that you would recommend for someone who is interested in digging deeper into this domain?
  • What are the trends in quantitative finance and algorithmic trading that you find most exciting and most concerning?
  • What do you have planned for the future of QuantConnect?

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

June 17, 2019 02:04 AM UTC

Wingware Blog

Extending Wing with Python (Part Two)

In this issue of Wing Tips we continue to look at how to extend Wing's functionality, by setting up a project that can be used to develop and debug extension scripts written for (and with) Wing, just as you would work with any other Python code.

Creating a Scripting Project

To debug extension scripts written for Wing, you will need to set up a new project that is configured so that Wing can debug itself. The manual steps for doing this are documented in Debugging Extension Scripts. However, let's use an extension script to do this automatically.

Copy the following Python code and paste it into a new file in Wing, using the File > New menu item:

import sys
import os
import collections
import wingapi

def new_scripting_project():
  """Create a new Wing project that is set up for developing and
  debugging scripts using the current instance of Wing"""

  app = wingapi.gApplication

  def setup_project():

    # Set the Python Executable to the Python that runs Wing
    proj = app.GetProject()
    proj.SetPythonExecutable(None, sys.executable)
    if not __debug__:
      from proj.attribs import kPyRunArgs
      app.fSingletons.fFileAttribMgr.SetValue(kPyRunArgs, None,
                                              ('custom', '-u -O'))

    # Add Wing's installation directory to the project
    proj.AddDirectory(app.GetWingHome())

    # Set main debug file to Wing's entry point
    wingpy = os.path.join(app.GetWingHome(), 'bootstrap', 'wing.py')
    proj.SetMainDebugFile(wingpy)

    # Setup the environment for auto-completion on the API and some additional
    # values needed only on macOS; use OrderedDict because later envs depend
    # on earlier ones
    env = collections.OrderedDict()
    env['PYTHONPATH'] = os.path.join(app.GetWingHome(), 'src')
    if sys.platform == 'darwin':
      runtime_dir = os.path.join(app.GetWingHome(), 'bin', '__os__', 'osx')
      files = os.listdir(runtime_dir)
      for fn in files:
        if fn.startswith('runtime-qt'):
          qt_ver = fn[len('runtime-'):]
          break
      env.update(collections.OrderedDict([
        ('RUNTIMES', runtime_dir),
        ('QTVERSION', qt_ver),
        ('QTRUNTIME', '${RUNTIMES}/runtime-${QTVERSION}'),
        ('SCIRUNTIME', '${RUNTIMES}/runtime-scintillaedit-${QTVERSION}'),
        ('DYLD_LIBRARY_PATH', '${QTRUNTIME}/lib:${SCIRUNTIME}/lib'),
        ('DYLD_FRAMEWORK_PATH', '${DYLD_LIBRARY_PATH}'),
      ]))
    app.GetProject().SetEnvironment(None, 'startup', env)

  # Create a new project; this prompts the user to save any unsaved files first
  # and only calls the completion callback if they don't cancel
  app.NewProject(setup_project)
new_scripting_project.contexts = [wingapi.kContextNewMenu('Scripts')]

Save this to a file named utils.py or any other name ending in .py in the scripts directory that is located inside Wing's Settings Directory, as listed in Wing's About box. Then select Reload All Scripts from the Edit menu to get Wing to discover the new script file. Once that's done, Wing monitors the file and will reload it automatically if you edit it and save changes to disk.

You can now create your project with New scripting project from the Scripting menu that should appear in the menu bar:

/images/blog/scripting-2/new-scripting-menu-item.png

This prompts to save any edited files before closing the current project, and then creates and configures the new project.

Debugging Extension Scripts

Now you're ready to develop and debug extension scripts with Wing. Try this now by appending the following to the script file that you created in the previous section:

import wingapi
def hello_world():
  wingapi.gApplication.ShowMessageDialog("Hello", "Hello world!")
hello_world.contexts = [wingapi.kContextNewMenu("Scripts")]

Save this to disk and then set a breakpoint on the third line, on the call to ShowMessageDialog.

Next, uncheck the Projects > Auto-reopen Last Project preference so that the debugged copy of Wing opens the default project and not your already-open scripting project:

/images/blog/scripting-2/no-auto-reopen.png

Then select Start/Continue from the Debug menu to start up a copy of Wing running under its own debugger. This copy of Wing should also contain a Scripts menu in its menu bar, with an item Hello world:

/images/blog/scripting-2/scripts-menu.png

Select that and you will reach the breakpoint you set in the outer instance of Wing (the one you started debug from):

/images/blog/scripting-2/breakpoint.png

You will see and can navigate the entire stack, but Wing will not be able to show source files for most of its internals. If you need to see the source code of Wing itself, you will have to obtain the source code as described in Advanced Scripting.



That's it for now! In the next Wing Tip we'll look more closely at the scripting API.

June 17, 2019 01:00 AM UTC

June 16, 2019

Catalin George Festila

Python 3.7.3 : Using the pycryptodome python module.

This python module can be used with python 3. More information can be found here. PyCryptodome is a self-contained Python package of low-level cryptographic primitives. It supports Python 2.6 and 2.7, Python 3.4 and newer, and PyPy. The install of this python module is easy with pip tool: C:\Python373\Scripts>pip install pycryptodome Collecting pycryptodome ... Installing collected packages:

June 16, 2019 05:54 AM UTC

June 15, 2019

Codementor

HackerRank: Circular Array Rotation in Python

How I solved my daily recommended HackerRank challenge problem and a small rant on problem specificity

June 15, 2019 08:19 PM UTC

Weekly Python StackOverflow Report

(clxxxii) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2019-06-15 20:04:06 GMT

  1. How to use random to choose colors - [11/3]
  2. Conditional Cumulative Sums in Pandas - [9/3]
  3. Efficient way to loop over 2D array - [9/2]
  4. Will passing ignore_index=True to pd.concat preserve index succession within dataframes that I'm concatenating? - [9/1]
  5. How do I properly use a function under a class? - [8/4]
  6. Why are f-strings faster than str() to parse values? - [7/1]
  7. How to efficiently calculate triad census in undirected graph in python - [6/4]
  8. Convert elements of list in pandas series using a dict - [6/3]
  9. Python Regex escape operator \ in substitutions & raw strings - [6/2]
  10. Rewrite to dictionary comprehensions - [6/2]

June 15, 2019 08:04 PM UTC

Doug Hellmann

sphinxcontrib.sqltable 2.0.0

sphinxcontrib-sqltable is a Sphinx extension for embedding database contents in documents What’s new in 2.0.0? drop python 2 support fix documentation build fix flake8 errors add tox and travis config

June 15, 2019 07:15 PM UTC

sphinxcontrib.sqltable 1.1.0

sphinxcontrib-sqltable is a Sphinx extension for embedding database contents in documents Update packaging metadata to use pbr for building packages.

June 15, 2019 06:21 PM UTC

Learn PyQt

Gradient Designer, a custom PyQt5/PySide2 widget

This custom PyQt5/PySide2-compatible widget provides a gradient designer providing a handy interface to design linear gradients in your applications. A new gradient can be created simply by creating an instance of the object.

gradient = Gradient()

The default gradient is black to white. The stop points are marked by a red box with a white line drawn vertically through it so they are visible on any gradient.

Initial state of the gradient designer Initial state of the gradient designer

User Interface

The widget allows editing of the gradient using the mouse. The controls are —

  • Double-click to add a new stop to the gradient at the clicked location. This is set to the same colour as the point to the right.
  • Right-click on a stop marker to edit the colour of that stop, by opening up a platform-native colour selector tool.
  • Click & drag a stop to move it, you can drag a stop past another stop to reverse the order. The two outermost points cannot be dragged.

Setting a Gradient

The gradient is defined as a list of 2-tuple containing a stop point as float between 0 and 1, and a colour as either a hex str or QColor or colour name. These can be set/retrieved through the API. 

gradient.setGradient([(0, 'black'), (1, 'red')])

>>> gradient.gradient()
[(0, 'black'), (1, 'red')]

If you set a gradient out of order it will be sorted.

gradient.setGradient([(0, 'black'), (1, 'green'), (0.5, 'red')])

>>> gradient.gradient()
[(0.0, 'black'), (0.5, 'red'), (1.0, 'green')]

If any stop is outside the range 0..1 it will throw an assertion.

Gradient auto-sorted when set. Gradient auto-sorted when set.

Modifying the Gradient

Alongside the GUI interface you can edit the gradient by adding/removing stop points through the API. The methods available are —

.addStop(stop, color=None) to add a new stop to the gradient. The stop is a float between 0 and 1, and the optional second parameter is a colour (hex, QColor, color name) for that point. If no colour is provided it defaults to the same colour as the following stop.

.removeStopAtPosition(n) removes a stop by index (i.e. order, the first stop would be zero). You cannot remove the end stop points.

.setColorAtPosition(n, color) set the color of a given stop by index (i.e. order).

.chooseColorAtPosition(n, current_color=None) pops up a window to choose the colour for the specified stop by index. If optional parameter current_color is provided the colour chooser will default to this initially.

The colour picker popped-up by the Gradient widget The colour picker popped-up by the Gradient widget

Continue reading: “Palette”

June 15, 2019 05:24 PM UTC

Ian Ozsvald

“A starter data science process for software engineers” – talk at PyLondinium 2019

I’ve just spoken on “A starter data science process for software engineers” (slides linked) at PyLondinium 2019, this talk is aimed at software engineers who are starting to ask data related questions and who are starting a data science journey. I’ve noted that many software engineers – without a formal data science background – are joining our PyData/data science world but lack useful transitionary resources. [note – video to come]

In this talk (based in part upon my current training courses and my recent PyDataCambridge talk) I cover:

The Notebook lives in github and this link should start a live Binder version (in which Altair is interactive and the slider Widget at the bottom of the Notebook live-drives scikit-learn predictions).

After the talk it seems that both Altair and the message “make a project spec” were the main winners, with Voila as a close third.

PyLondinium were also kind enough to organise a book signing for my High Performance Python book where I got to talk a bit about our in-preparation 2nd edition (for January).

This conference builds on last year’s inaugural event, it has grown and has a lovely feel. You may want to think on putting in a talk for next year’s PyLondinium!

 

Ian is a Chief Interim Data Scientist via his Mor Consulting. Sign-up for Data Science tutorials in London and to hear about his data science thoughts and jobs. He lives in London, is walked by his high energy Springer Spaniel and is a consumer of fine coffees.

The post “A starter data science process for software engineers” – talk at PyLondinium 2019 appeared first on Entrepreneurial Geekiness.

June 15, 2019 05:22 PM UTC

Learn PyQt

Graphic Equalizer, a custom PyQt5/PySide2 visualization widget

This custom PyQt5/PySide2-compatible widget provides a frequency visualizer output for audio applications. It is completely configurable from the number of bars, the number of segments and colours to the animated decay. It's ready to drop into your Python-Qt5 applications.

The download package includes a demo animation using random data and some custom colour configuration so you can explore how the widget works.

EqualizerBar Demo

Basic setup

To create an Equalizer object pass in the number of bars and a number of segments per bar. 

equalizer.Equalizer(5, 10)
The default appearance of the widget. The default appearance of the widget.

The default range for each of the equalizer bars is 0...100. This can be adjusted using .setRange. 

equalizer.setRange(0, 1000)

Decay animation

The equalizer bar includes a decay animation where peaks of values slowly fade over time. This is created by gradually subtracting a set value from the current value of each bar, using a recurring timer.

The frequency of the decay timer (in milliseconds) can be configured using .setDecayFrequencyMs. The default value is 100ms.

equalizer.setDecayFrequencyMs()

Passing 0 will disable the decay mechanism.

On each tick a set value is removed from the current value of each bar. This is configured by using .setDecay. Adjust this based on your equalizer's range of possible values. 

equalizer.setDecay(0, 1000)

Bar style configuration

The number of bars to display and the number of segments in the bars/the colours of those bars are defined at startup.

equalizer = EqualizerBar(10,  ["#2d004b", "#542788", "#8073ac", "#b2abd2", "#d8daeb", "#f7f7f7", "#fee0b6", "#fdb863", "#e08214", "#b35806", "#7f3b08"])
Purple Orange theme with 10 bars Purple Orange theme with 10 bars

To set the colours after startup, either provide a list of colors (QColor or hex values) at startup, or use the .setColors method. Passing a list of colours to this method will change the number of segments to match the colours provided. 

equalizer.setColors(["#810f7c", "#8856a7", "#8c96c6", "#b3cde3", "#edf8fb"])

You can also use .setColor to set a single colour for all segments, without changing the number of bars.

equalizer.setColor('red')

The spacing around the equalizer graphic is configured with .setBarPadding providing an integer value to add in pixels.

equalizer.setBarPadding(20)

The proportion of each segment in the bar which is solid/empty is configured using .setBarSolidPercent with a float value between 0...1. 

equalizer.setBarSolidPercent(0.4)

Finally, the background colour can be configured using .setBackgroundColor

equalizer.setBackgroundColor('#0D1F2D')
Customising bar padding and sizing. Customising bar padding and sizing.

Continue reading: “Gradient”

June 15, 2019 02:57 PM UTC

Codementor

How I learned that a lambda can't be stopped

AWS tips and tricks!

June 15, 2019 01:27 PM UTC

Ian Ozsvald

“On the Delivery of Data Science Projects” – talk at PyDataCambridge meetup

A few weeks I got to speak at PyDataCambridge (thanks for having me!), slides are here for “On The Delivery of Data Science Projects“.

This talk is based on my experiences coaching teams (whilst building IP for clients) to help them derisk, design and deliver working data science products. This talk is really in two halves – it takes the important lessons from my two training classes and boils them down into a 30 minute talk. We cover:

Let me know if you found this talk useful? I really think the ideas around successful project delivery need to be collected and shared, we’re still in the “wild west” and I’m keen to collate more examples of successful process.

 

Ian is a Chief Interim Data Scientist via his Mor Consulting. Sign-up for Data Science tutorials in London and to hear about his data science thoughts and jobs. He lives in London, is walked by his high energy Springer Spaniel and is a consumer of fine coffees.

The post “On the Delivery of Data Science Projects” – talk at PyDataCambridge meetup appeared first on Entrepreneurial Geekiness.

June 15, 2019 11:34 AM UTC

J. Pablo Fernández

Converting a Python data into a ReStructured Text table

This probably exist but I couldn’t find it. I wanted to export a bunch of data from a Python/Django application into something a non-coder could understand. The data was not going to be a plain CSV, but a document, with various tables and explanations of what each table is. Because ReStructured Text seems to be … Continue reading Converting a Python data into a ReStructured Text table →

June 15, 2019 10:26 AM UTC

June 14, 2019

NumFOCUS

NumFOCUS Hires First Ever Development Director

The post NumFOCUS Hires First Ever Development Director appeared first on NumFOCUS.

June 14, 2019 09:37 PM UTC

Doug Hellmann

sphinxcontrib-spelling 4.3.0

sphinxcontrib-spelling is a spelling checker for Sphinx-based documentation. It uses PyEnchant to produce a report showing misspelled words. What’s new in 4.3.0? Logging: use warning() instead of its deprecated alias (contributed by Sergey Kolosov) Support additional contractions (contributed by David Baumgold) require sphinx >= 2.0.0 declare support for python 3.6

June 14, 2019 09:05 PM UTC

Python Anywhere

Using MongoDB on PythonAnywhere with MongoDB Atlas

.jab-post img { border: 2px solid #eeeeee; padding: 5px; }

This requires a paid PythonAnywhere account

Lots of people want to use MongoDB with PythonAnywhere; we don't have support for it built in to the system, but it's actually pretty easy to use with a database provided by MongoDB Atlas -- and as Atlas is a cloud service provided by Mongo's creators, it's probably a good option anyway :-)

If you're experienced with MongoDB and Atlas, then our help page has all of the details you need for connecting to them from our systems.

But if you'd just like to dip your toe in the water and find out what all of this MongoDB stuff is about, this blog post explains step-by-step how to get started so that you can try it out.

Prerequisites

The first important thing to mention is that you'll need a paid PythonAnywhere account to access Atlas. Free accounts can only access the external Internet using HTTP or HTTPS, and unfortunately MongoDB uses it's own protocol which is quite different to those.

Apart from that, in order to follow along you'll need at least a basic understanding of PythonAnywhere, writing website code and of databases in general -- if you've been through our Beginners' Flask and MySQL tutorial you should be absolutely fine.

Signing up to MongoDB atlas

Unsurprisingly, the first step in using Atlas is to sign up for an account (if you don't already have one). Go to their site and click the "Try Free" button at the top right. That will take you to a page with the signup form; fill in the appropriate details, and sign up.

Atlas' user interface may change a little after we publish this post (their sign-up welcome page has already changed between out first draft yesterday and the publication today!), so we won't give super-detailed screenshots showing what to click, but as of this writing, they present you with a "Welcome" window that has some "help getting started" buttons. You can choose those if you prefer, but we'll assume that you've clicked the "I don't need help getting started" button. That should land you on a page that looks like this, which is where you create a new MongoDB cluster.

Creating your first cluster and adding a user

A MongoDB cluster is essentially the same as a database server for something more traditional like MySQL or PostgreSQL. It's called a cluster because it can potentially spread over multiple computers, so it can in theory scale up much more easily than an SQL database.

To create the cluster:

This will take you to a page describing your cluster. Initially it will have text saying something like "Your cluster is being created" -- wait until that has disappeared, and you'll have a page that will look something like this:

Now we need to add a user account that we'll use to connect to the cluster from Python:

Click the button to create the user, and you'll come back to the user admin page, with your user listed in the list.

Setting up the whitelist

Access to your MongoDB cluster is limited to computers on a whitelist; this provides an extra level of security beyond the username/password combination we just specified.

Just to get started, we'll create a whitelist that comprises every IP address on the Internet -- that is, we won't have any restrictions at all. While this is not ideal in the long run, it makes taking the first steps much easier. You can tighten things up later -- more information about that at the end of this post.

Here's how to configure the whitelist to allow global access:

Getting the connection string

Now we have a MongoDB cluster running, and it's time to connect to it from PythonAnywhere. The first step is to get the connection string:

Connecting to the cluster from a PythonAnywhere console

Next, go to PythonAnywhere, and log in if necessary. The first thing we need to do here is make sure that we have the correct packages installed to connect to MongoDB -- we'll be using Python 3.7 in this example, so we need to use the pip3.7 command. Start a Bash console, and run:

pip3.7 install --user --upgrade pymongo dnspython

Once that's completed, let's connect to the cluster from a command line. Run ipython3.7 in your console, and when you have a prompt, import pymongo and connect to the cluster:

import pymongo
client = pymongo.MongoClient("<the atlas connection string>")

...replacing <the atlas connection string> with the actual connection string we got from the Atlas site earlier, with <password> replaced with the password you used when setting up the user.

That's created a connection object, but hasn't actually connected to the cluster yet -- pymongo only does that on an as-needed basis.

A good way to connect and at least make sure that what we've done so far has worked is to ask the cluster for a list of the databases that it currently has:

client.list_database_names()

If all is well, you'll get a result like this:

['admin', 'local']

...just a couple of default databases created by MongoDB itself for its own internal use.

Now let's add some data. MongoDB is much more free-form than a relational database like MySQL or PostgreSQL. There's no such thing as a table, or a row -- instead, a database is just a bunch of things called "collections", each of which is comprised of a set of "documents" -- and the documents are just objects, linking keys to values.

That's all a bit abstract; I find a useful way to imagine it is that a MongoDB database is like a directory on a disk; it contains a number of subdirectories (collections), and each of those contains a number of files (each one being a document). The files just store JSON data -- basically, Python dictionary objects.

Alternatively, you can see it by comparison with an SQL database:

For this tutorial, we're going to create one super-simple database, called "blog". Unsurprisingly for a blog, it will contain a number of "post" documents, each of which will contain a title, body, a slug (for the per-article URL) and potentially more information.

The first question is, how do we create a database? Neatly, we don't need to explicitly do anything -- if we refer to a database that doesn't exist, MongoDB will create it for us -- and likewise, if we refer to a collection inside the database that doesn't exist, it will create that for us too. So in order to add the first row to the "posts" collection in the "blog" database we just do this:

db = client.blog
db.posts.insert_one({"title": "My first post", "body": "This is the body of my first blog post", "slug": "first-post"})

No need for CREATE DATABASE or CREATE TABLE statements -- it's all implicit! IPython will print out the string representation of the MongoDB result object that was returned by the insert_one method; something like this:

<pymongo.results.InsertOneResult at 0x7f9ae871ea88>

Let's add a few more posts:

db.posts.insert_one({"title": "Another post", "body": "Let's try another post", "slug": "another-post", "extra-data": "something"})
db.posts.insert_one({"title": "Blog Post III", "body": "The blog post is back in another summer blockbuster", "slug": "yet-another-post", "author": "John Smith"})

Now we can inspect the posts that we have:

for post in client.blog.posts.find():
    print(post)

You'll get something like this:

{'_id': ObjectId('5d0395dcbf76b2ab4ed67948'), 'title': 'My first post', 'body': 'This is the body of my first blog post', 'slug': 'first-post'}
{'_id': ObjectId('5d039611bf76b2ab4ed67949'), 'title': 'Another post', 'body': "Let's try another post", 'slug': 'another-post', 'extra-data': 'something'}
{'_id': ObjectId('5d039619bf76b2ab4ed6794a'), 'title': 'Blog Post III', 'body': 'The blog post is back in another summer blockbuster', 'slug': 'yet-another-post', 'author': 'John Smith'}

The find function is a bit like a SELECT statement in SQL; with no parameters, it's like a SELECT with no WHERE clause, and it just returns a cursor that allows us to iterate over every document in the collection. Let's try it with a more restrictive query, and just print out one of the defined values in the object we inserted:

for post in client.blog.posts.find({"title": {"$eq": "Another post"}}):
    print(post["body"])

You'll get something like this:

Let's try another post

MongoDB's query language is very rich -- you can see a list of the query operators here. We won't go into any more detail here -- there are many excellent MongoDB tutorials on the Internet, so if you google for "mongodb python tutorial" you're bound to find something useful!

Now we've connected to a MongoDB database, and created some data, so let's do something with that.

Connecting to the cluster from a Flask website

The next step is to connect to our cluster from a website's code. We'll use Flask for this -- because it's database-agnostic, it's a better fit for MongoDB than Django, which is quite tied to the SQL model of representing data.

We're also going to use a Flask extension called Flask-PyMongo to make our connections -- the raw PyMongo package has a few problems with the way PythonAnywhere runs websites, and while there are ways around those (see the help page), the Flask extension handles everything smoothly for us. So, in your Bash console, exit IPython and run

pip3.7 install --user Flask-PyMongo

Once that's done, let's create a website: head over to the "Web" page inside PythonAnywhere, and create yourself a Python 3.7 Flask app.

When it's been created, edit the flask_app.py file that was auto-generated for you, and replace the contents with this:

from flask import Flask, render_template
from flask_pymongo import PyMongo

app = Flask(__name__)
app.config["MONGO_URI"] = "<the atlas connection string>"

mongo = PyMongo(app)

@app.route('/')
def index():
    return render_template("blog.html", posts=mongo.db.posts.find())

@app.route('/<post_slug>')
def item(post_slug):
    return render_template("blog.html", posts=mongo.db.posts.find({"slug": post_slug}))

...replacing <the atlas connection string> with the connection string as before, with one change -- the original connection string will have /test in it, like this:

mongodb+srv://admin:iW8qWskQGJcEpZdu9ZUt@cluster0-zakwe.mongodb.net/test?retryWrites=true&w=majority

That /test means "connect to the database called test on the cluster". We've put our data into a database called blog, so just replace the test with blog, so that it looks like this:

mongodb+srv://admin:iW8qWskQGJcEpZdu9ZUt@cluster0-zakwe.mongodb.net/blog?retryWrites=true&w=majority

All of the rest of the code in that file should be pretty obvious if you're familiar with Flask -- the MongoDB-specific stuff is very similar to the code we ran in a console earlier, and also to the way we would connect to MySQL via SQLAlchemy. The only really new thing is the abbreviated syntax for searching for an exact match:

mongo.db.posts.find({"slug": post_slug})

...is just a shorter way of saying this:

mongo.db.posts.find({"slug": {"$eq": post_slug}})

To go with this Flask app, we need a template file called blog.html in a new templates subdirectory of the directory containing flask_app.py -- here's something basic that will work:

<html>
    <head>
        <meta charset="utf-8">
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" integrity="sha512-dTfge/zgoMYpP7QbHy4gWMEGsbsdZeCXz7irItjcC3sPUFtf0kuFbDz/ixG7ArTxmDjLXDmezHubeNikyKGVyQ==" crossorigin="anonymous">
        <title>My blog</title>
    </head>

    <body>
        <div class="container">
            <div class="row">
                <h1><a href="/">My Blog</a></h1>
            </div>

            {% for post in posts %}
                <div class="row">
                    <h2><a href="/{{ post.slug }}">{{ post.title }}</a></h2>

                    <p>
                    {{ post.body }}
                    </p>
                </div>
            {% endfor %}

        </div>

    </body>
</html>

Note that there's one simple difference to the way we reference the post document to how we'd do it in Python code -- in Python we have to say (for example) post["title"], while dictionary lookups in a Flask template require us to use post.title.

Once you've created that file, reload the website using the button on the "Web" page, and you should see a website with your blog posts on it:

Let's add a new post: keep the tab showing your website open, but in another tab go to your Bash console, start ipython3.7 again, connect to the database, and add a new post:

import pymongo
client = pymongo.MongoClient("<the atlas connection string>")
db = client.blog
db.posts.insert_one({"title": "Blog Post Goes Forth", "body": "...but I thought we were coding Python?", "slug": "bad-blackadder-nerd-joke"})

Head back to the tab showing the site, and hit the browser's refresh button -- your new post will appear!

All done!

So now we have a working super-simple blog running on PythonAnywhere, backed by an Atlas MongoDB cluster.

The one remaining issue is that the whitelist we specified is a little broad. If someone gets hold of your MongoDB username and password, they can access the database. It's possible to set things up so that you have an initially-empty whitelist, and then every time you run your code, it automatically whitelists the IP address it's running on using the Atlas API -- that's a slightly more advanced topic, though, so if you want to learn about that, head over to our MongoDB help page.

We hope this post has been useful -- if you have any questions or comments, please leave them below. Also, if there are other things you'd like to connect to from PythonAnywhere that you think could benefit from having a blog post explaining how to do it, please do let us know!

June 14, 2019 12:59 PM UTC