skip to navigation
skip to content

Planet Python

Last update: December 17, 2018 04:47 AM UTC

December 17, 2018

James Bennett

How to break Python

Edited December 2018: added sections for Python 3.7 and upcoming 3.8, updated Python 3.6 section since Python 3.6 has been released, and updated Python 3.3 section since 3.3 has reached end-of-life.

Don’t worry, this isn’t another piece about Python 3. I’m fully in favor of Python 3, and on record as to why. And if you’re still not convinced, I suggest this thoroughly comprehensive article on the topic, which goes ...

Read full entry

December 17, 2018 04:01 AM UTC

Podcast.__init__

Polyglot: Multi-Lingual Natural Language Processing with Rami Al-Rfou

Using computers to analyze text can produce useful and inspirational insights. However, when working with multiple languages the capabilities of existing models are severely limited. In order to help overcome this limitation Rami Al-Rfou built Polyglot. In this episode he explains his motivation for creating a natural language processing library with support for a vast array of languages, how it works, and how you can start using it for your own projects. He also discusses current research on multi-lingual text analytics, how he plans to improve Polyglot in the future, and how it fits in the Python ecosystem.

Summary

Using computers to analyze text can produce useful and inspirational insights. However, when working with multiple languages the capabilities of existing models are severely limited. In order to help overcome this limitation Rami Al-Rfou built Polyglot. In this episode he explains his motivation for creating a natural language processing library with support for a vast array of languages, how it works, and how you can start using it for your own projects. He also discusses current research on multi-lingual text analytics, how he plans to improve Polyglot in the future, and how it fits in the Python ecosystem.

Preface

Interview

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

December 17, 2018 03:13 AM UTC

Techiediaries - Django

Django 2 Ajax CRUD with Python 3.7 and jQuery

In this tutorial, you'll learn how to send Ajax requests in Django 2 and Python 3.7 to add CRUD operations in your application and manipulate your Django models and database without having to refresh your web pages each time.

Ajax stands for Asynchronous JavaScript and XML and it's a way for getting data from the server and updating the page on the fly without refreshing the page.

Creating a Virtual Environment

Make sure you have Python 3 installed (Python 3.7 is the latest as of this writing) and start by creating a virtual environment for your project's packages:

$ python -m venv myenv

Next, activate your virtual environment using:

$ source myenv/bin/activate

Installing Django 2 and Creating a Project

Now, you need to install Django using pip:

$ python -m pip install django

Next, create a Django project using:

$ django-admin startproject djangoajaxdemo

Next you need to create a Django application using the manage.py script:

$ cd djangoajaxdemo
$ python manage.py startapp rooms

Next you need to add it to your project's installed apps array in the settings.py file:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rooms'
]

Adding jQuery

In this tutorial, we'll be using jQuery to send Ajax requests to Django. You can also use any other HTTP client like Axios, the JavaScript Fetch API available on modern browsers or the XMLHttpRequest interface.

First of all, you need to get jquery from the official website and include it in your project or use a CDN. Go to the official website and get the CDN of the version of jQuery you want to use.

In my case, I'll be using jQuery 3.3.1 from https://code.jquery.com/jquery-3.3.1.min.js.

Inside the rooms application, create a templates/rooms folder and create a base.html file:

{% load static %}
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title> Django Ajax CRUD with jQuery</title>
<link  rel="stylesheet"  href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css"   crossorigin="anonymous">
</head>
  <body>
    <div class="container d-flex h-100">
      <div class="row justify-content-center">
        <div class="col-10">
          {% block main %}
          {% endblock %}
        </div>
      </div>
    </div>
{% block js %}
<script src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
{% endblock %}
{% block extrajs %}
{% endblock %}
  </body>
</html>

Adding a Model

We'll be adding CRUD operations against a Room model. Open the rooms/models.py file and add the following code:

from django.db import models



class  Room(models.Model):
    ROOM_TYPES = (
        (1, 'Single'),
        (2, 'Double'),
        (3, 'Triple'),
    )

    name = models.CharField(max_length=50)
    status = models.CharField(max_length=30, blank=True)
    room_number = models.IntegerField(blank=True, null=True)
    nobeds = models.IntegerField(blank=True, null=True)
    room_type = models.PositiveSmallIntegerField(choices=ROOM_TYPES)

Adding CRUD Views

In the rooms/views.py file add the following class based and generic views for performing CRUD operations:

from django.views.generic import View
from django.http import JsonResponse
from django import forms
from django.views.decorators.csrf import csrf_exempt
from django.utils.decorators import method_decorator
from django.forms.models import model_to_dict
from .models import Room

class  RoomForm(forms.ModelForm):
    class  Meta:
        model = Room
        fields =  '__all__'

class  RoomList(View):
    def  get(self, request):
        rooms =  list(Room.objects.all().values())
        data =  dict()
        data['rooms'] = rooms
        return JsonResponse(data)

class  RoomDetail(View):
    def  get(self, request, pk):
        room = get_object_or_404(Room, pk=pk)
        data =  dict()
        data['room'] = model_to_dict(room)
        return JsonResponse(data)

@method_decorator(csrf_exempt, name='dispatch')
class  RoomCreate(CreateView):
    def  post(self, request):
        data =  dict()
        form = RoomForm(request.POST)
        if form.is_valid():
            room = form.save()
            data['room'] = model_to_dict(room)
        else:
            data['error'] =  "form not valid!"
        return JsonResponse(data)

class  RoomUpdate(View):
    def  post(self, request, pk):
        data =  dict()
        room = Room.objects.get(pk=pk)
        form = RoomForm(instance=room, data=request.POST)
        if form.is_valid():
            room = form.save()
            data['room'] = model_to_dict(room)
        else:
            data['error'] =  "form not valid!"
        return JsonResponse(data)

class  RoomDelete(View):
    def  post(self, request, pk):
        data =  dict()
        room = Room.objects.get(pk=pk)
        if room:
            room.delete()
            data['message'] =  "Room deleted!"
        else:
            data['message'] =  "Error!"
        return JsonResponse(data)

Next, let's add the urls. Open the urls.py file and add:

from django.urls import path, include
from django.views.generic.base import TemplateView
from rooms import views

urlpatterns = [
    path('rooms/', TemplateView.as_view(template_name="rooms/main.html"), name='room_main'),
    path('rooms/list', views.RoomList.as_view(), name='room_list'),
    path('rooms/create', views.RoomCreate.as_view(), name='room_create'),
    path('rooms/update/<int:pk>', views.RoomUpdate.as_view(), name='room_update'),
    path('rooms/delete/<int:pk>', views.RoomDelete.as_view(), name='room_delete'),
    path('rooms/<int:pk>', views.RoomDetail.as_view(), name='room_detail'), 
]

Adding a Template

Since we'll be using Ajax for making CRUD operations in our Django application we will not need multiple pages or templates but instead we'll conceive our application as a Single Page Application.

Let's create a main.html file inside the rooms/templates/rooms folder with the following content:

{% extends 'rooms/base.html' %}
{% block main %}  
{% endblock %}

{% block extrajs %}
<script  src="{% static 'js/app.js' %}"></script>
{% endblock %}

Next under your project's root folder create the static/js/ folder and add an app.js file with the following content:

$(function () {
    console.log("Hello!");
});

In your urls.py file, add the following url pattern so you can access your static files in development mode:

from django.conf import settings
# [...]

if settings.DEBUG:
    from django.contrib.staticfiles.urls import staticfiles_urlpatterns
    urlpatterns += staticfiles_urlpatterns()

In the settings.py file, add the following setting to configure your static folder:

STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)

If your run your application and visit the http://127.0.0.1:8000/rooms/ you should see a Hello! in the console of your browser which means you static files are configured correctly.

In this tutorial, we'll implement the list and delete operations. For create and update operations we'll see them in the next tutorial.

Getting Rooms with jQuery.ajax()

In your app.js file, add the following code to get data from the rooms/list endpoint by sending a GET Ajax request:

$.ajax({
    url:  '/rooms/list',
    type:  'get',
    dataType:  'json',
    success: function  (data) {
        let rows =  '';
        data.rooms.forEach(room => {
        rows += `
        <tr>
            <td>${room.room_number}</td>
            <td>${room.name}</td>
            <td>${room.nobeds}</td>
            <td>${room.room_type}</td>
            <td>
                <button class="btn deleteBtn" data-id="${room.id}">Delete</button>
                <button class="btn updateBtn" data-id="${room.id}">Update</button>
            </td>
        </tr>`;
    });
    $('[#myTable](https://paper.dropbox.com/?q=%23myTable) > tbody').append(rows);
    $('.deleteBtn').each((i, elm) => {
        $(elm).on("click",  (e) => {
            deleteRoom($(elm))
        })
    })
    }
});

Deleting Rooms with jQuery.ajax()

Next, you need to add an implementation for the deleteRoom(e) method:

function  deleteRoom(el){
    roomId  =  $(el).data('id')
    $.ajax({
        url:  `/rooms/delete/${roomId}`,
        type:  'post',
        dataType:  'json',
        success:  function (data) {
            $(el).parents()[1].remove()
        }
    });
}

Now go back to your rooms/templates/rooms/main.html template and add the table:

{% block main %}
<table  class="table table-bordered"  id="myTable">
<thead>
<th>
Room Number
</th>
<th>
Name
</th>
<th>
Number of Beds
</th>
<th>
Type
</th>
<th>
Actions
</th>
</thead>
<tbody>
</tbody>
</table>

<div  id="roomform">
    <button  id="createRoom"  class="btn"> Create Room </button>
</div>

{% endblock %}

This is a screenshot of our page at this point:

Django Ajax and jQuery

In the next tutorial, we'll see how to create a form and send it with jQuery and Ajax to our Django endpoints to create and update rooms.

December 17, 2018 12:00 AM UTC

December 16, 2018

Vasudev Ram

Dynamic function creation at run time with Python's exec built-in

By Vasudev Ram

Hi, readers,
I was browsing Paul Graham's essay about his book "On Lisp". It's about advanced uses of Lisp and some of the features that make it unique.

In it, he talks about many Lisp features, like the ability to treat code as data, the fact that functions are first-class objects, so, just like other objects such as scalar variables, lists, maps, etc., they too can be passed as arguments to functions, returned from functions, stored in variables, etc., and the fact that Lisp programs can write Lisp programs. (There's a lot more to the book, of course.) This made me think of experimenting a bit with Python's dynamic features, whether similar to Lisp or not. One man's Lisp is another man's Python ...

I did know from a while ago that Python has many dynamic features, and have blogged about some of them in the past. (Search my blog for topics like Python tricks, the trace module and chained decorators, Python introspection, the inspect module, finding the caller of a Python function, Python generators are pluggable, function-based callbacks, class-based callbacks, and other fun stuff. Try using 'site:jugad2.blogspot.com' and suchlike search engine techniques.) This is one more such exploration.

I got the idea of writing a program that can dynamically create a function at run time, based on some user input, and then run that function. The program uses the exec built-in of Python.

Here is the program, dyn_func_with_exec.py:
# dyn_func_with_exec.py 
# Purpose: To dynamically create (and run) a function at run time. 
# Author: Vasudev Ram
# Copyright 2018 Vasudev Ram
# Web site: https://vasudevram.github.io
# Product store: https://gumroad.com/vasudevram

from __future__ import print_function
import sys

major = sys.version_info.major
if major == 2:
    input_func = raw_input
elif major == 3:
    input_func = input
else:
    print("Unsupported version, go back to the future.")
    sys.exit(0)

func_name = input_func("Enter name of function to create dynamically: ")
func_def = """
def {}(args):
    print("In function", func_name, ": args:", args)
""".format(func_name)

exec(func_def)
print(func_name, ":", eval(func_name))

args1 = (1, "a"); args2 = (2, "b")

# Two ways of calling the dynamically created function:
print("\nCalling {} via eval():".format(func_name))
eval(func_name + "(args1)")
print("\nCalling {} via globals() dict:".format(func_name))
globals()[func_name](args2)
Here is the output of 3 runs:
(py is the Python launcher for Windows.)
$ py -2 dyn_func_with_exec.py
Enter name of function to create dynamically: mu
mu : <function mu at 0x0000000002340518>

Calling mu via eval():
In function mu : args: (1, 'a')

Calling mu via globals() dict:
In function mu : args: (2, 'b')

$ py -3 dyn_func_with_exec.py
Enter name of function to create dynamically: _
_ : <function _ at 0x000000000039C1E0>

Calling _ via eval():
In function _ : args: (1, 'a')

Calling _ via globals() dict:
In function _ : args: (2, 'b')

$ py dyn_func_with_exec.py
Enter name of function to create dynamically: |
Traceback (most recent call last):
  File "dyn_func_with_exec.py", line 28, in <module>
    exec(func_def)
  File "<string> line 2
    def |(args):
        ^
SyntaxError: invalid syntax
So it works in both Python 2 and 3. The last run (with an invalid function name) shows how the user input gets interpolated into the function definition string func_def.

The if statement used shows another dynamic aspect of Python: the ability to call different functions (via the same name, input_func), based on some condition which is only known at run time (the Python version, in this case).

Why mu for the input? Well, if foo, why not mu? :)
See my HN comment here

Another way to dynamically decide which function to run:

Driving Python function execution from a text file

Enjoy.

- Vasudev Ram - Online Python training and consulting
I conduct online courses on Python programming, Unix / Linux commands and shell scripting and SQL programming and database design, with course material and personal coaching sessions.

The course details and testimonials are here.

Contact me for details of course content, terms and schedule.

Try FreshBooks: Create and send professional looking invoices in less than 30 seconds.

Getting a new web site or blog, and want to help preserve the environment at the same time? Check out GreenGeeks.com web hosting.

Sell your digital products via DPD: Digital Publishing for Ebooks and Downloads.

Learning Linux? Hit the ground running with my vi quickstart tutorial. I wrote it at the request of two Windows system administrator friends who were given additional charge of some Unix systems. They later told me that it helped them to quickly start using vi to edit text files on Unix. Of course, vi/vim is one of the most ubiquitous text editors around, and works on most other common operating systems and on some uncommon ones too, so the knowledge of how to use it will carry over to those systems too.

Teachable: feature-packed course creation platform, with unlimited video, courses and students.

Check out WP Engine, powerful WordPress hosting.

Creating online products for sale? Check out ConvertKit, email marketing for online creators.

Get a fast web site with A2 Hosting.

Posts about: Python * DLang * xtopdf

My ActiveState Code recipes

Follow me on:

Vasudev Ram

December 16, 2018 04:47 PM UTC

Davy Wybiral

Nordic Thingy:52 Dev Kit (First impression)

Today I'm playing around with a Nordic Thingy:52 Bluetooth 5 development kit from Nordic Semiconductor.


December 16, 2018 09:40 AM UTC

codingdirectional

Tidy up the user interface

Welcome back to the multitas project which we have already created two features for this application in the previous articles 1) Remove duplicate files 2) Move file from one folder to another. In this chapter, we are just going to tidy up the buttons on the user interface before we continue to build the next task in the next chapter. What we are going to do here is to create a button container to keep all the buttons that we will use in our program. Below is the full source code which will result in this below outcome.

The user interface of this application 
from tkinter import *
from tkinter import filedialog
from Remove import Remove
import os
from tkinter import messagebox
import webbrowser
from Copy import Copy

win = Tk() # 1 Create instance
win.title("Multitas") # 2 Add a title
#win.iconbitmap(os.path.abspath('icon.ico'))
win.resizable(0, 0) # 3 Disable resizing the GUI
win.configure(background='black') # 4 change background color
#win.iconbitmap(('icon.ico'))

# 5 Create a label
aLabel = Label(win, text="Select a task", anchor="center", padx=13, pady=10, relief=RAISED,)
aLabel.grid(column=0, row=0, sticky=W+E)
aLabel.configure(foreground="white")
aLabel.configure(background="black")
aLabel.configure(wraplength=90)
aLabel.message = ''

# 6 Create a selectFile function to be used by action button
def selectFile():

    fullfilenames = filedialog.askopenfilenames(initialdir="/", title="Select a file") # select multiple files from the hard drive

    if(fullfilenames != ''):

        fullfilenamelist = win.tk.splitlist(fullfilenames)
        directory_path = os.path.dirname(os.path.abspath(fullfilenamelist[0]))

        os.chdir(directory_path)  # change the directory to the selected file directory

        folder = filedialog.askdirectory()  # 7 open a folder then create and start a new remove thread to delete the duplicate file
        folder = folder.replace('/', '\\')  # 8 this is for the windows separator only

        if(folder != '' and folder != os.getcwd()):

            for fullfilename in fullfilenamelist:

                if(fullfilename != ''):
                    filename = fullfilename.split('/')[-1]
                    remove = Remove(folder, filename, fullfilename, directory_path, aLabel)
                    remove.start()
                    remove.join()
                    messagebox.showinfo('Remove the duplicate files', aLabel.message)

        else:
            messagebox.showinfo("Error", "Kindly select one folder and it must be a different one")
            return

    else:
        messagebox.showinfo("Select file", "You need to select a file!")
        return

# 6 Create a copyFile function to move a file from one folder to another
def copyFile():

    fullfilenames = filedialog.askopenfilenames(initialdir="/", title="Select a file") # select multiple files from the hard drive

    if(fullfilenames != ''):

        fullfilenamelist = win.tk.splitlist(fullfilenames)
        directory_path = os.path.dirname(os.path.abspath(fullfilenamelist[0]))

        os.chdir(directory_path)  # change the directory to the selected file directory

        folder = filedialog.askdirectory()  # 7 open a folder then create and start a new Copy thread to move the file from one directory to another one
        folder = folder.replace('/', '\\')  # 8 this is for the windows separator only

        if(folder != '' and folder != os.getcwd()):

            for fullfilename in fullfilenamelist:

                if(fullfilename != ''):
                    filename = fullfilename.split('/')[-1]
                    copy = Copy(folder, filename, fullfilename)
                    copy.start()
                    copy.join()
                    messagebox.showinfo('Move the file ', 'File has been moved to new destination')

        else:
            messagebox.showinfo("Error", "Kindly select one folder and it must be a different one")
            return

    else:
        messagebox.showinfo("Select file", "You need to select a file!")
        return

def openLink(): # start a new link

    webbrowser.open_new("http://codingdirectional.info/2018/12/12/remove-duplicate-files-project-is-ready/")

# 9 Adding a Button

button_frame = Frame(win, bg='black')
button_frame.grid(column=0, row=1, sticky=E+W)

action = Button(button_frame, text="Remove", command=selectFile, padx=2)
action.grid(column=0, row=0, sticky=E+W)
action.configure(background='black')
action.configure(foreground='white')

action_move = Button(button_frame, text="Move", command=copyFile, padx=2)
action_move.grid(column=1, row=0, sticky=E+W)
action_move.configure(background='black')
action_move.configure(foreground='white')

action_link = Button(button_frame, text="Manual", command=openLink, padx=2)
action_link.grid(column=2, row=0, sticky=E+W)
action_link.configure(background='black')
action_link.configure(foreground='white')

win.mainloop()  # 10 start GUI

That is basically it, now the buttons will be more organize than before and it is time for us to move on to the next chapter.

December 16, 2018 07:54 AM UTC

gamingdirectional

Create a power bar for pygame project

In this chapter, we are going to create the last piece of game feature which is the player’s power bar, after this, I will do all the touches up to this game project which certainly includes to tidy up the game code before uploading the game to the pygame portal. Alright, let’s get to work. The first file which we will need to edit is the score manager file where we will create a power...

Source

Related posts:

What is your top achievements? Create an exit button on the main game menu scene Open a text file then write the game level onto it with python Create the pause scene for the game Create the third level for this pygame project Lets move on to the next game level! Create a pool object for player missile Create a game over scene for pygame project Setting up PyCharm for Pygame development in Windows 10 OS Rotate an object in Pygame

December 16, 2018 05:43 AM UTC

Toshio Kuratomi

Python, signal handlers, and exceptions

Never ever, ever raise a regular exception in a Python signal handler.

This is probably the best advice that I had never heard.  But after looking into an initial analysis of a timeout decorator bug it’s advice that I wish was prominently advertised.  So I’m publishing this little tale to let others know about this hidden gotcha so that, just maybe, when you have an opportunity to do this, a shiver will run down your spine, your memory will be tickled, and you’ll be able to avoid the pitfalls that I’ve recorded here.

When will I need to be reminded of this?

Signals are a means provided by an operating system to inform programs of  events that aren’t part of their normal program flow.  If you’re on a UNIX-like operating system and hit Control-C to cancel a program running in the shell, you’ve used a signal.  Control-C in the shell sends a SIGINT (Interrupt) signal to the program.  The program receives the SIGINT and a signal handler (a function which is written to take appropriate action when a signal is received) is invoked to handle it.  In most cases, for SIGINT, the signal handler tries to complete any pressing work (for instance, to finish writing data to a file so that the file isn’t left in a half-written state) and then causes the program to exit.

Python provides a signal library which is very similar to the C API underneath Python.  Python does a little bit of behind the scenes work to make sure that signals appear to be interacting with Python code rather than interfering with the interpreter’s implementation (meaning that signals will appear to happen between Python’s byte code instructions rather than leaving a Python instruction half-completed).

Your mission: exit when a signal occurs

If you search the internet for how to implement a timeout in Python, you’ll find tons of examples using signals, including one from the standard library documentation and one which is probably where the design for our original decorator came from.  So let’s create some quick test code to show how the signal works to implement a timeout

import signal
import time

def handler(signum, frame):
    print('Signal handler called with signal',
          signum)
    raise OSError("timeout exceeded!")

def long_function():
    time.sleep(10)

# Set the signal handler and a 1-second alarm
old_handler = signal.signal(signal.SIGALRM,
                            handler)
signal.alarm(1)
# This sleeps for longer than the alarm
start = time.time()
try:
    long_function()
except OSError as e:
    duration = time.time() - start
    print('Duration: %.2f' % duration)
    raise
finally:
    signal.signal(signal.SIGALRM, old_handler)
    signal.alarm(0) # Disable the alarm

This code is adapted from the example of implementing a timeout in the standard library documentation.  We first define a signal handler named handler() which will raise an OSError when it’s invoked. We then define a function, long_function() which is designed to take longer to run than our timeout. The we hook everything together:

When we run the code we see that the signal handler raises the OSError as expected:

$ /bin/time -p python3 test.py
Signal handler called with signal 14
Duration: 1.00
Traceback (most recent call last):
  File "test.py", line 18, in
    long_function()
  File "test.py", line 11, in long_function
    time.sleep(2)
  File "test.py", line 8, in handler
    raise OSError("timeout exceeded!")
OSError: timeout exceeded!
real 1.04

Although long_function() takes 10 seconds to complete, the SIGALRM fires after 1 second.  That causes handler() to run which raises the OSError with the message timeout exceeded!.  The exception propagates to our toplevel where it is caught and prints Duration: 1.00 before re-raising so we can see the traceback.  We see that the output of /bin/time roughly agrees with the duration we calculated within the program… just a tad over 1 second to run the whole thing.

Exceptions seem to be working…?

It’s time to make our long_function code a little less trivial.

import signal
import time

def handler(signum, frame):
    print('Signal handler called with signal',
signum)
    raise OSError("timeout exceeded!")

def long_function():
    try:
        with open('/etc/passwd', 'r') as f:
            data = f.read()
            # Simulate reading a lot of data
            time.sleep(10)
    except OSError:
        # retry once:
        with open('/etc/passwd', 'r') as f:
            data = f.read()
            time.sleep(10)

# Set the signal handler and a 5-second alarm
old_handler = signal.signal(signal.SIGALRM,
handler)
signal.alarm(1)
start = time.time()
# This sleeps for longer than the alarm
try:
    long_function()
except OSError as e:
    duration = time.time() - start
    print('Duration: %.2f' % duration)
    raise
finally:
    signal.signal(signal.SIGALRM, old_handler)
    signal.alarm(0)        # Disable the alarm

We’ve changed long_function(), lines 9-20, to do two new things:

  1. It now reads from a file.  We still have time.sleep(), but that’s just to simulate a long read operation which will trigger our timeout.
  2. Open() and read() can raise a variety of exceptions.  If open() or read() raise an OSError exception, we want to retry reading the file once before we give up.

So what happens when we run this version?

$ /bin/time -p python3 test.py
Signal handler called with signal 14
real 11.07

That was unexpected…

As you can see from the output, our program still fired the SIGALRM.  And the signal handler still ran.  But after it ran, everything else seems to be different.  Apparently, the OSError didn’t propagate up the stack to our toplevel so we didn’t print the duration.  Furthermore, we ended up waiting for approximately 10 seconds in addition to the 1 second we waited for the timeout. What happened?

The key to understanding this is to understand that when the signal handler is invoked, Python adds the call to the signal handler onto its current call stack (The call stack roughly represents the nesting of functions that lead up to where the code is currently executing.  So, in our example, inside of the signal handler, the call stack would start with the module toplevel, then long_function(), and finally handler().  If you look at the traceback from the first example, you can see exactly that call stack leading you through all the function calls to the point where the exception was raised.)  When the signal handler raises its exception, Python unwinds the call stack one function at a time to find an exception handler (not to be confused with our signal handler) which can handle the exception.

Where was the program waiting when SIGALRM was emitted?  It was on line 14, inside of long_function().  So Python acts as though handler() was invoked directly after that line.  And when handler() raises its OSError exception, Python then unwinds the call stack from handler() to long_function() and sees that on line 15, there’s an except OSError: and so Python lets it catch the signal handler’s OSError instead of propagating it up the stack further.  And in our code, that exception handler decides to retry reading the file which is where there is a second 10 second delay as we read the file.  Since the SIGALRM was already used up, the timeout doesn’t fire this time.  So the rest of the bug progresses from there: long_function() now waits the full 10 seconds before returning because there’s no timeout to stop it.  It then returns normally to its caller.  The caller doesn’t receive an OSError exception.  So it doesn’t fire its own OSError exception handler which would have printed the Duration.

Okay, this isn’t good. Can it get worse?

There are even less intuitive ways that this bug can be provoked. For instance:

def long_function():
    time.sleep(0.8)
    try:
        time.sleep(0.1)
        time.sleep(0.1)
        time.sleep(0.1)
    except Exception:
        pass

In this version, we don’t have the context clue that we’re raising OSError in the signal handler and mistakenly catching it in long_function(). An experienced Python coder will realize that except Exception is sufficiently broad to be catching OSError, but in the clutter of other function calls and without the same named exception to provoke their suspicions initially, they might not realize that the occasional problems with the timeout not working could be triggered by this late exception handler.

This is also problematic if the programmer is having to orient themselves off of a traceback. That would lead them to look inside of long_function() for the source of a OSError. They won’t find it there because it’s inside of the signal handler which is outside of the function.

import zipfile
def long_function():
    time.sleep(0.9)
    zipfile.is_zipfile('my_file.zip')

In this case, there’s no exception handling in our code to clue us in. If you look at the implementation of zipfile.is_zipfile(), though, you’ll see that there’s an except OSError: inside of there. If the timeout’s alarm ends up firing while you are inside of that code, is_zipfile() will just as happily swallow the OSError as an exception handler inside of long_function() would have.

So I think I can solve this if I….

There are ways to make the timeout functionality more robust. For instance, let’s define our handler like this:

import functools

class OurTimeout(BaseException):
    pass

def handler(timeout, signum, frame):
    print('Signal handler called with signal',
          signum)
    signal.alarm(timeout)
    raise OurTimeout("timeout exceeded!")

# toplevel code:
one_second_timeout = functools.partial(handler, 1)
old_handler = signal.signal(signal.SIGALRM, one_second_timeout)
signal.alarm(1)
try:
    long_function()
except OurTimeout:
    print('Timeout from our timeout function)

The first thing you can see is that this now raises a custom timeout which inherits from BaseException. This is more robust than using standard timeouts which are rooted at Exception because well written Python code won’t catch exceptions inheriting directly from BaseException. It’s not foolproof, however, because it’s only convention that prevents someone from catching BaseException. And no matter how good a Python programmer you are, I’m sure you’ve been lazy enough to write code like this at least once:

def long_function():
    try:
        with open('/etc/passwd', 'r') as f:
            f.read()
    except:
        # I'll fix the possible exception classes later
        rollback()

Bare except: catches any exception, including BaseException so using a bare except will still break this timeout implementation.

The second thing this implementation changes is to reset the alarm inside of the signal handler. This is helpful as it means that the code will be able to timeout multiple times to get back to the toplevel exception handler. In our retry example, Both the initial attempt and the retry attempt would timeout so long_function() would end up taking only two seconds and fail due to our timeout in the end. However, there are still problems. For instance, this code ends up taking timeout * 3 seconds instead of just timeout seconds because the exception handler prevents us from hitting the break statement so we’ll have to keep hitting the timeout:

def long_function():
    for i in range(0, 3):
        try:
            time.sleep(10)
        except OSError:
            pass
        else:
            break

The following code, (arguably buggy because you *should* disable the alarm as the first thing in the recovery code instead of the last) can end up aborting the toplevel code’s recovery effort if timeout is too short:

try:
    long_function()
except OSError:
   rollback_partial_long_function()
   signal.alarm(0)

So even though this code is better than before, it is still fragile, error prone, and can do the wrong thing even with code that isn’t obviously wrong.

Doctor, is there still hope?

When I was looking at the problem with the timeout decorator in Ansible what struck me was that when using the decorator, the decorator was added outside of a function telling it to timeout but the timeout was occurring and potentially being processed inside of the function. That meant that it would always be unintuitive and error prone to someone trying to use the decorator:

@timeout.timeout(1)
def long_function():
    try:
        time.sleep(10)
    except:
        # Try an alternative
        pass

try:
    long_function()
except timeout.TimeoutError:
    print('long_function did not complete')

When looking at the code, the assumption is that on the inside there’s long_function, then outside of it, the timeout code, and outside of that the caller. So the expectation is that an exception raised by the timeout code should only be processed by the caller since exceptions in Python only propagate up the call stack. Since the decorator’s functionality was implemented via a signal handler, though, that expectation was disappointed.

To solve this, I realized that the way signals and exceptions interact would never allow exceptions to propagate correctly. So I switched from using a signal to using one thread for the timeout and one thread for running the function. Simplified, that flow looks like this (You can look at the code for the new decorator in Ansible if you’re okay with the GPLv3+ license. The following code is all mine in case you want to re-implement the idea without the GPLv3+ restrictions:

import multiprocessing
import multiprocessing.pool

def timeout(seconds=10):
    def decorator(func):
        def wrapper(*args, **kwargs):
            pool = multiprocessing.pool.ThreadPool(processes=1)
            results = pool.apply_async(func, args, kwargs)
            pool.close()
            try:
                return results.get(seconds)
            except multiprocessing.TimeoutError:
                raise OSError('Timeout expired after: %s' % seconds)
        return wrapper
    return decorator

@timeout(1)
def long_func():
    try:
        time.sleep(10)
    except OSError:
        print('Failure!')
    print('end of long_func')

try:
    long_func()
except OSError as e:
    print('Success!')
    raise

As you can see, I create a multiprocessing.pool.ThreadPool with a single thread in it. Then I run the decorated function in that thread. I use res.get(timeout) with the timeout we set to get the results or raise an exception if the timeout is exceeded. If the timeout was exceeded, then I throw the OSError to be like our first example.

If all goes well, the exception handling inside of long_func won’t have any effect on what happens. Let’s see:

$ python2 test.py
Success!
Traceback (most recent call last):
  File "test.py", line 49, in
    long_func()
  File "test.py", line 35, in wrapper
    raise OSError('Timeout expired after: %s' % seconds)
OSError: Timeout expired after: 1

Yep, as you can see from the stack trace, the OSError is now being thrown from the decorator, not from within long_func(). So only the toplevel exception handler has a chance to handle the exception. This leads to more intuitive code and hopefully less bugs down the road.

So are there ever times when a signal handler is the right choice?

Signal handlers can be used for more than just timeouts. And they can do other things besides trying to cancel an ongoing function. For instance, the Apache web server has a signal handler for the HUP (HangUP) signal. When it receives that, it reloads its configuration from disk. If you take care to catch any potential exceptions during that process, you shouldn’t run across these caveats because these problems only apply to raising an exception from the signal handler.

When you do want to exit the function via a signal handler, I would be a little hesitant because you can never entirely escape from the drawbacks above and threading provides a more intuitive interface for the called and calling code.  I think that a few practices make it more robust, however.  As mentioned above:

And one that wasn’t mentioned before:

It’s better to make an ad hoc exception-raising signal handler that handles a specific problem inside of a function rather than attempting to place it outside of the function.  For instance:

def timeout_handler():
    raise BaseException('Timeout')

def long_function():
    try:
        old_handler = signal.signal(signal.SIGALRM,
                                    timeout_handler)
        signal.alarm(1)
        time.sleep(10)   # Placeholder for the real,
                         # potentially blocking code
    except BaseException:
        print('Timed out!')
    finally:
        signal.signal(signal.SIGALRM, old_handler)
        signal.alarm(0)

long_function()

Why is it better to do it this way? If you do this, you localize the exception catching due to the signal handler with the code that could potentially raise the exception. It isn’t foolproof as you are likely still going to call helper functions (even in the stdlib) which could handle a conflicting exception but at least you are more clearly aware of the fact that the exception could potentially originate from any of your code inside this function rather than obfuscating all of that behind a function call.

December 16, 2018 12:23 AM UTC

December 15, 2018

"Morphex's Blogologue"

Python quick-fix of broken router

I have a router which seems to "take the day off" every once in a while, and this started after I filled up all 4 Ethernet ports.

Rebooting, the only fix I've found so far, fixes the problem, so that all 4 Ethernet ports start working again.

Rebooting the router gets boring and annoying after a while, so I decided to write a script to automatically reboot the router every hour.

That script is here:

https://github.com/morphex/netgear_reboot/

Didn't need to write any Python 2 or 3 specific hacks to make it work on either Python 2 or 3 either.

I was reading a Python 2-3 cheat sheet here:

http://python-future.org/compatible_idioms.html

And after getting used to using for example print(1), I like the idea of making things simpler and more consistent, an easier to learn syntax.

An added benefit is that I, and other who use two or more keyboard layouts, don't have to remember where the backticks are located on no_nb and en_us keyboard layouts.

December 15, 2018 04:06 PM UTC

Codementor

Simple way to get data from web page using python

Can you guess a simple way you can get data from a web page? It’s through a technique called web scraping.

December 15, 2018 03:03 PM UTC

Weekly Python StackOverflow Report

(clvi) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2018-12-15 14:32:38 GMT

  1. Is is safe to use next within a for loop in Python? - [18/3]
  2. Performant cartesian product (CROSS JOIN) with pandas - [14/3]
  3. Why does pandas merge on NaN? - [14/1]
  4. Use columns 1 and 2 to populate column 3 - [10/4]
  5. Setting flag column depending on whether column contains a given string - [10/2]
  6. __init__ function definition without self - [10/1]
  7. Why does isinstance require a tuple instead of any iterable? - [9/0]
  8. Is there an efficient method of checking whether a column has mixed dtypes? - [8/4]
  9. Python random.Random(bytearray) TypeError: unhashable type - [8/2]
  10. Pythonic way to fill rows with date range - [7/3]

December 15, 2018 02:32 PM UTC

gamingdirectional

What is your top achievements?

After the previous article, we have finally been able to create a score scene as I had promised you before in the last article. What this scene does is to list out the latest 5 levels that the player has achieved, if the level count has reached it’s maximum value then the earliest level from the list will be removed and get replaced by the latest level at the end of the list.

Source

Related posts:

Create an exit button on the main game menu scene Open a text file then write the game level onto it with python Create the pause scene for the game Create the third level for this pygame project Lets move on to the next game level! Create a pool object for player missile Lets put the game instruction online instead The modify version of the Pygame Missile Manager Class Group all the sprites together with Pygame Drawing an Ellipse with Pygame

December 15, 2018 12:33 PM UTC

codingdirectional

Move files from one folder to another folder with python

After I have finished writing the python program which will remove the duplicate files in another folder, the first part of this tkinter’s project has come to an end and the second part will follow. Just in case you miss out the first part of the project completely then make sure to read the previous articles about it, you can also download the remove duplicate application bundle for windows os through this link then either start the application file or run it’s main file on the windows command prompt by following the instructions on this previous article.

With that we are now ready to move on to the second part of the project which is to move a file or files from one folder to another. If you have followed my previous articles then you know that we have created a remove thread to remove all the duplicate files, we will use back the same skeleton of the remove thread to create a thread class which we will call it the copy class which has only one simple mission — move a file or files from one folder to another. First of all lets create the Copy class.

import threading
import os
from shutil import copyfile

class Copy(threading.Thread):

    def __init__(self, massage,  filename, fullfilename,):

        threading.Thread.__init__(self)
        self.massage = massage
        self.src = fullfilename
        self.dst = os.path.join(self.massage, filename)

    def run(self):

        copyfile(self.src, self.dst)
        os.remove(self.src)

As you can see from above the copy class is using the same skeleton as the remove class but this time we have simplified the process, the run method of the thread will perform this two tasks 1) Move the old file to the new directory 2) Delete the old file in the old directory.

Next, we will edit the main file by including a copy file method which will be called when the user clicks on the Move button, the user will need to select a file or files from a folder and then select another folder which he wishes to move those files to. The content of the copy file method will mostly be the same as the select file method which we used to select a file or files that we wish to remove their duplicate copies from. Here is the source code for the main file.

from tkinter import *
from tkinter import filedialog
from Remove import Remove
import os
from tkinter import messagebox
import webbrowser
from Copy import Copy

win = Tk() # 1 Create instance
win.title("Multitas") # 2 Add a title
#win.iconbitmap(os.path.abspath('icon.ico'))
win.resizable(0, 0) # 3 Disable resizing the GUI
win.configure(background='black') # 4 change background color
#win.iconbitmap(('icon.ico'))

# 5 Create a label
aLabel = Label(win, text="Select a task", anchor="center", padx=13, pady=10, relief=RAISED)
aLabel.grid(column=0, row=0, sticky=W+E)
aLabel.configure(foreground="white")
aLabel.configure(background="black")
aLabel.configure(wraplength=96)
aLabel.message = ''

# 6 Create a selectFile function to be used by action button
def selectFile():

    fullfilenames = filedialog.askopenfilenames(initialdir="/", title="Select a file") # select multiple files from the hard drive

    if(fullfilenames != ''):

        fullfilenamelist = win.tk.splitlist(fullfilenames)
        directory_path = os.path.dirname(os.path.abspath(fullfilenamelist[0]))

        os.chdir(directory_path)  # change the directory to the selected file directory

        folder = filedialog.askdirectory()  # 7 open a folder then create and start a new remove thread to delete the duplicate file
        folder = folder.replace('/', '\\')  # 8 this is for the windows separator only

        if(folder != '' and folder != os.getcwd()):

            for fullfilename in fullfilenamelist:

                if(fullfilename != ''):
                    filename = fullfilename.split('/')[-1]
                    remove = Remove(folder, filename, fullfilename, directory_path, aLabel)
                    remove.start()
                    remove.join()
                    messagebox.showinfo('Remove the duplicate files', aLabel.message)

        else:
            messagebox.showinfo("Error", "Kindly select one folder and it must be a different one")
            return

    else:
        messagebox.showinfo("Select file", "You need to select a file!")
        return

# 6 Create a copyFile function to move a file from one folder to another
def copyFile():

    fullfilenames = filedialog.askopenfilenames(initialdir="/", title="Select a file") # select multiple files from the hard drive

    if(fullfilenames != ''):

        fullfilenamelist = win.tk.splitlist(fullfilenames)
        directory_path = os.path.dirname(os.path.abspath(fullfilenamelist[0]))

        os.chdir(directory_path)  # change the directory to the selected file directory

        folder = filedialog.askdirectory()  # 7 open a folder then create and start a new Copy thread to move the file from one directory to another one
        folder = folder.replace('/', '\\')  # 8 this is for the windows separator only

        if(folder != '' and folder != os.getcwd()):

            for fullfilename in fullfilenamelist:

                if(fullfilename != ''):
                    filename = fullfilename.split('/')[-1]
                    copy = Copy(folder, filename, fullfilename)
                    copy.start()
                    copy.join()
                    messagebox.showinfo('Move the file ', 'File has been moved to new destination')

        else:
            messagebox.showinfo("Error", "Kindly select one folder and it must be a different one")
            return

    else:
        messagebox.showinfo("Select file", "You need to select a file!")
        return

def openLink(): # start a new link

    webbrowser.open_new("http://codingdirectional.info/2018/12/12/remove-duplicate-files-project-is-ready/")

# 9 Adding a Button

action = Button(text="Remove", command=selectFile)
action.grid(column=0, row=1, sticky=S+W)
action.configure(background='black')
action.configure(foreground='white')

action_move = Button(text="Move", command=copyFile)
action_move.grid(column=0, row=1, sticky=S+E)
action_move.configure(background='black')
action_move.configure(foreground='white')

action_link = Button(text="Manual", command=openLink)
action_link.grid(column=1, row=1, sticky=S+E)
action_link.configure(background='black')
action_link.configure(foreground='white')

win.mainloop()  # 10 start GUI

With that we have finished the second part of the project and is now ready to move on to the third part of the project. But before we move on, we will do some modifications on the arrangement of the buttons in the next chapter which we will create a container to contain more buttons for the future usage.

Again if you are new to this project then don’t forget to download the first part of the project through this link. This application is only for the windows os user but if you are using linux and are familiar with python programming then you should be able to easily modify this program for your own use.

December 15, 2018 05:22 AM UTC

Techiediaries - Django

Django 2 CRUD Tutorial: Generic Class-Based Views

Throughout this tutorial, you'll create a Django 2 CRUD example application using generic class based views. You will be using the latest Python 3.7 version.

You'll learn to quickly create CRUD views by using the Django built-in generic views such as ListView, DetailView, CreateView, UpdateView and DeleteView.

CRUD stands for create, read, update and delete and it simply refers to a set of common operations that are used in most web applications to interact with data from a database. It basically provides a CRUD interface that allows your users to create, read, update and delete data in your application database like MySQL, PostgreSQL or Oracle etc.

Prerequisites

You'll need to have the following prerequisites:

In this tutorial, you'll learn about:

Creating and Activating a Virtual Environment

When working with new Python projects, it's recommended that you create a virtual and isolated environment for your project's packages. In Python 3.7, you can use the venv module to create virtual environments.

Go to your terminal and run the following command:

$ python -m venv env

A virtual environment called env is created.

Next, you'll need to activate the environment using the source command:

$ source env/bin/activate

Now, you can install your project's dependencies without worrying about interfering with the other projects.

Installing Django and Creating a New Project

Let's install Django 2 into our virtual environment using pip:

$ pip install django

Next, create a new project using django-admin.py:

$ django-admin.py startproject django_crud_cbv

Creating a New Application

Now, let's create a new application in our Django project. Navigate inside your project's folder and run the following command:

$ cd django_crud_cbv
$ python manage.py startapp contacts

This will create an application called contacts.

You need to add this application to your project. Open the settings.py file and add contacts to the INSTALLED_APPS array:

INSTALLED_APPS = ( 
 # [...]
 'contacts',
 )

Creating a Model and Migrating the Database

After creating the contacts application, you'll need to a model that represents contacts in the database. Open the contacts/models.py file and add the following code:

from django.contrib.gis.db import models

class  Contact(models.Model):
    name = models.CharField(max_length=100)
    email = models.EmailField()
    address = models.CharField(max_length=100)
    phone = models.CharField(max_length=50)

Now, that you have defined your model, you can migrate your database and run your development server.

Go to your terminal and run these commands:

$ python manage.py makemigrations
$ python manage.py migrate

Enabling the Admin Interface

At this point of your project, you can already have a CRUD interface by registering your model with the admin application with isa built-in Django application that comes with every project.

Open the contacts/admin.py file and register the Contact model:

from django.contrib import admin 
from .models import Contact
admin.site.register(Contact)

You can access the admin interface from http://localhost:8000/admin.

Creating the CRUD Views

Instead of using the admin interface to perform CRUD operations against our database, let's create our own CRUD views.

We'll be using the Django class based generic views to define our views. Open the contacts/views.py file and start by adding the following imports:

from django.views.generic import ListView, DetailView 
from django.views.generic.edit import CreateView, UpdateView, DeleteView
from .models import Contact

Now, you can add a first view for listing the contacts using the ListView generic view:

class ContactList(ListView): 
    model = Contact

Next, create a detail view using the DetailView generic view class:

class ContactDetail(DetailView): 
    model = Contact

Next, you need to add the create view for creating contacts using the CreateView generic view:

class ContactCreate(CreateView): 
    model = Contact

Next, you need to add the update view for updating contacts using the UpdateView view:

class ContactUpdate(UpdateView): 
    model = Contact

Finally, you need to add the delete view for deleting contacts using the DeleteView class-based generic view:

class ContactDelete(DeleteView): 
    model = Contact

Adding the Templates

After defining the CRUD views, you next need to add the template for each of your views. Each view expects a template with a specific name in the templates folder of your application.

Inside the contacts folder, create a templates/contacts/ folder and start by adding the contact_list.html file with the following content:

<h1>Contacts</h1>

<table>
<thead>
<tr>
<th>Name</th>
<th>Email</th>
<th>Address</th>
<th>Phone</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
{% for contact in object_list %}
<tr>
<td>{{ contact.name }}</td>
<td>{{ contact.email }}</td>
<td>{{ contact.address }}</td>
<td>
<a  href="{% url "contact_detail"  contact.id  %}">details</a>
<a  href="{% url "contact_edit"  contact.id  %}">edit</a>
<a  href="{% url "contact_delete"  contact.id  %}">delete</a>
</td>
</tr>

{% endfor %}
</tbody>
</table>

Next, create the contact_details.html file with the following content:

<h1>Contact Details</h1>

<p>Name: {{object.name}}</p>
<p>Email: {{object.email}}</p>
<p>Address: {{object.address}}</p>
<p>Phone: {{object.phone}}</p>

Next, let's create the contact_form.html file that will be used by the update view:

<h1>Contact Update</h1>

<form  method="post">
{% csrf_token %}
{{ form.as_p }}
<input  type="submit"  value="Submit"  />
</form>

Finally you need to create the contact_confirm_delete.html file that will be used by the delete view:

<h1>Contact Delete?</h1>
<form  method="post">
{% csrf_token %}
Are you sure you want to delete this contact?
<input  type="submit"  value="Submit"  />
</form>

Adding the URLs

Finally, you need to add various URLs to the views you have defined. Open the urls.py file of your project and add the following URLs:

from django.urls import path
from . import views

urlpatterns = [
    # [...]
    path('contacts', views.ContactList.as_view(), name='contact_list'),
    path('contact/<int:pk>', views.ContactDetail.as_view(), name='contact_detail'),
    path('create', views.ContactCreate.as_view(), name='contact_create'),
    path('update/<int:pk>', views.ContactUpdate.as_view(), name='contact_update'),
    path('delete/<int:pk>', views.ContactDelete.as_view(), name='contact_delete'),
]

Finally, you can you run development server using:

$ python manage.py runserver

You can then access your CRUD interface from http://localhost:8000/contacts/

Conclusion

In this tutorial, you have created a CRUD project with Django and Python 3.7.

You have used the various class-based generic views provided by Django such as ListView, DetailView, CreateView, UpdateView and DeleteView to create CRUD views that allow your users to create, read, update and deete contacts from your database. In the example, we have used SQLite but you can very easily switch to PostgreSQL, MySQL or any database you want without changing anything in your code thanks to Django ORM.

December 15, 2018 12:00 AM UTC

Angular 6|7 Tutorial — CRUD & Python REST API

Angular 7 is released! Read about its new features and how to update Angular 6 to v7.

This tutorial is designed for developers that want to use Angular 6/7 to build front-end apps for their back-end REST APIs.

You will see by example how to build a CRUD REST API with Angular and Python.

The new features of Angular 7 include better performance, new powerful CLI additions and a new way to inject services. Throughout this tutorial, designed for beginners, you'll learn Angular by example by building a full-stack CRUD—Create, Read, Update and Delete—web application using the latest version of the most popular framework and platform for building mobile and desktop client side applications (also called SPAs or Single Page Applications), created and used internally by Google.

In the back-end we'll use Python with Django, the most popular pythonic web framework designed for perfectionists with deadlines.

In nutshel, you'll learn to generate Angular 6 apps, generate components and services and add routing. You'll also learn to use various features such as HttpClient for sending AJAX requests and HTTP calls and subscribing to RxJS 6 observables etc.

By the end of this Angular 6 tutorial, you'll learn by building a real world example application:

The Django Example CRUD Back-End

We'll make use of a simple CRUD API built with Django and Django REST framework. Since this is an Angular tutorial we'll not focus on building the API as this will be the subject of a separate tutorial but you can grab the source code of the back-end API from this repository

You can use the following commands to start the development server:

# Clone the project and navigate into it
$ git clone https://github.com/techiediaries/django-crm
$ cd django-crm

# Create a virtual environment and install packages
$ pipenv install

# Activate the virtual environment
$ pipenv shell 

# Create and migrate the database then run the local development server
$ python manage.py migrate
$ python manage.py runserver

You server will be running from http://localhost:8000.

We are using pipenv, the officially recommended package management tool for Python so you'll need to have it installed. The process is quite simple depending on your operating system.

Angular 6 CRUD Example

The example Angular 6 application we'll be building is the front-end for a CRM RESTful API that will allow you to create accounts, leads, opportunities and contacts. It's a perfect example for a CRUD (Create, Read, Update and Delete) application built as an SPA (Single Page Application). The example application is work on progress so we'll be building it through a series of tutorials and will be updated to contain advanced features such as RxJS 6 and JWT authentication. We'll also use Bootstrap 4 and Angular 6 Material for building and styling the UI components. You either need Bootstrap 4 or Angular Material for styling so depending on your choice you can follow separate tutorials:

Installing the Angular CLI (v6.0.0)

Make sure you have Node.js installed, next run the following command in your terminal to install Angular CLI v 6.0.0.

npm -g install @angular/cli

You can check the installed version by running the following command:

ng version

This is the output I'm getting:

    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/


Angular CLI: 6.0.0
Node: 8.11.1
OS: linux x64
Angular: 
... 

Package                      Version
------------------------------------------------------
@angular-devkit/architect    0.6.0
@angular-devkit/core         0.6.0
@angular-devkit/schematics   0.6.0
@schematics/angular          0.6.0
@schematics/update           0.6.0
rxjs                         6.1.0
typescript                   2.7.2

Now, you're ready to create a project using Angular CLI v6. Simply run the following command in your terminal:

ng new crmapp

The CLI will automatically generate a bunch of files common to most Angular 6 projects and install the required dependencies for your project.

We will mostly be working inside the src/app folder. This is the directory structure of the project:

Angular 6 tutorial

You can serve your application locally by running the following commands:

# Navigate inside your project's folder
$ cd crmapp

# Serve your application
$ ng serve

You application will be running from http://localhost:4200.

This is a screen-shot of home page of the application:

Angular 6 tutorial

Components in Angular 6|5|4

Now what's a component?

A component is a TypeScript class with an HTML template and an optional set of CSS styles that control a part of the screen.

Components are the most important concept in Angular 6. An Angular 6 application is basically a tree of components with a root component (the famous AppComponent). The root component is the one contained in the bootstrap array in the main NgModule module app.module.ts.

One important aspect of components is re-usability. A component can be re-used throughout the application and even in other applications. Common and repeatable code that performs a certain task can be encapsulated into a re-usable component that can be called whenever we need the functionality it provides.

Each bootstrapped component is the base of its own tree of components. Inserting a bootstrapped component usually triggers a cascade of component creations that fill out that tree. source

Component-Based Architecture

An Angular application is made of several components forming a tree structure with parent and child components.

A component is an independent block of a big system (web application) that communicates with the other building blocks (components) of the system using inputs and outputs. A component has associated view, data and behavior and may have parent and child components.

Components allow maximum re-usability, easy testing, maintenance and separation of concerns.

Let's now see this practically. Head over to your Angular application project folder and open the src/app folder. You will find the following files:

Except for the last file which contains the declaration of the application main (root) Module, all these files are used to create a component. It's the AppComponent: The root component of our application. All other components we are going to create next will be direct or un-direct children of the root component.

Demystifying the AppComponent (The Root Component of Angular Applications)

Go ahead and open the src/app/app.component.ts file and let's understand the code behind the main/root component of the application.

First, this is the code:

import { Component } from  '@angular/core';
@Component({
    selector:  'app-root',
    templateUrl:  './app.component.html',
    styleUrls: ['./app.component.css']
})
export  class  AppComponent {
    title  =  'app';
}

We first import the Component decorator from @angular/core then we use it to decorate the TypeScript class AppComponent. The Component decorator takes an object with many parameters such as:

The export keyword is used to export the component so that it can be imported from other components and modules in the application.

The title variable is a member variable that holds the string 'app'. There is nothing special about this variable and it's not a part of the canonical definition of an Angular component.

Now let's see the corresponding template for this component. If you open src/app/app.component.html this is what you'll find:

<div  style="text-align:center">
<h1>
Welcome to !
</h1>
    <img  width="300"  alt="Angular Logo"  src="data:image/svg+xml;....">
</div>

    <h2>Here are some links to help you start: </h2>
<ul>
    <li>
    <h2><a  target="_blank"  rel="noopener"  href="https://angular.io/tutorial">Tour of Heroes</a></h2>
    </li>
    <li>
    <h2><a  target="_blank"  rel="noopener"  href="https://github.com/angular/angular-cli/wiki">CLI Documentation</a></h2>
    </li>
    <li>
    <h2><a  target="_blank"  rel="noopener"  href="https://blog.angular.io/">Angular blog</a></h2>
    </li>
</ul>

The template is a normal HTML file (almost all HTML tags are valid to be used inside Angular templates except for some tags such as <script>, <html> and <body> etc.) with the exception that it can contain template variables (in this case the title variable) or expressions ({{...}}) that can be used to insert values in the DOM dynamically. This is called interpolation or data binding. You can find more information about templates from the docs.

You can also use other components directly inside Angular templates (via the selector property) just like normal HTML.

If you are familiar with the MVC (Model View Controller) pattern, the component class plays the role of the Controller and the HTML template plays the role of the View.

Angular 6 Components by Example

After getting the theory behind Angular components, let's now create the components for our simple CRM application.

Our REST API, built with Django, exposes these endpoints:

Before adding routing to our application we first need to create the application's components so based on the exposed REST API architecture we can initially divide our application into these components:

Let's use the Angular CLI to create the components

ng generate component AccountList
ng generate component AccountCreate

ng generate component ContactList
ng generate component ContactCreate

ng generate component LeadList
ng generate component LeadCreate

ng generate component OpportunityList
ng generate component OpportunityCreate

This is the output of the first command:

CREATE src/app/account-list/account-list.component.css (0 bytes)
CREATE src/app/account-list/account-list.component.html (31 bytes)
CREATE src/app/account-list/account-list.component.spec.ts (664 bytes)
CREATE src/app/account-list/account-list.component.ts (292 bytes)
UPDATE src/app/app.module.ts (418 bytes)

You can see that the command generates all the files to define a component and also updates src/app/app.module.ts.

If you open src/app/app.module.ts after running all commands, you can see that all components are automatically added to the AppModule declarations array.:

import { BrowserModule } from  '@angular/platform-browser';

import { NgModule } from  '@angular/core';



import { AppComponent } from  './app.component';

import { AccountListComponent } from  './account-list/account-list.component';

import { AccountCreateComponent } from  './account-create/account-create.component';

import { ContactListComponent } from  './contact-list/contact-list.component';

import { ContactCreateComponent } from  './contact-create/contact-create.component';

import { LeadListComponent } from  './lead-list/lead-list.component';

import { LeadCreateComponent } from  './lead-create/lead-create.component';

import { OpportunityListComponent } from  './opportunity-list/opportunity-list.component';

import { OpportunityCreateComponent } from  './opportunity-create/opportunity-create.component';

@NgModule({

declarations: [
    AppComponent,
    AccountListComponent,
    AccountCreateComponent,
    ContactListComponent,
    ContactCreateComponent,
    LeadListComponent,
    LeadCreateComponent,
    OpportunityListComponent,
    OpportunityCreateComponent
],
imports: [
    BrowserModule
],
providers: [],
bootstrap: [AppComponent]
})
export  class  AppModule { }

If you are creating components manually, you need to make sure to include manually so they can be recognized as part of the module.

Adding Angular 6 Routing

Angular CLI provides the --routing switch (ng new crmapp --routing) that enables you to add routing automatically but we're going to add routing manually for the sake of understanding the various pieces involved in adding component routing to your Angular application.

In fact, adding routing is quite simple:

This is the initial content of app-routing.module.ts:

import { NgModule } from '@angular/core';
import { Routes, RouterModule } from '@angular/router';

const routes: Routes = [
];

@NgModule({
  imports: [RouterModule.forRoot(routes)],
  exports: [RouterModule]
})
export class AppRoutingModule { }

The routes will contain all the routes of the application. After creating the components we'll see how to add routes to this array.

For now, we want to redirect the visitor to the /accounts path when the home URL is visited so the first path we'll add is:

{ path:  '', redirectTo:  'accounts', pathMatch:  'full' },

The pathMatch specifies the matching strategy. full means that we want to fully match the path.

Next let's add the other paths:

{ path:  '', redirectTo:  'accounts', pathMatch:  'full' },
{
    path:  'accounts',
    component:  AccountListComponent
},
{
    path:  'create-account',
    component:  AccountCreateComponent
},
{
    path:  'contacts',
    component:  ContactListComponent
},
{
    path:  'create-contact',
    component:  ContactCreateComponent
},
{
    path:  'leads',
    component:  LeadListComponent
},
{
    path:  'create-lead',
    component:  LeadCreateComponent
},
{
    path:  'opportunities',
    component:  OpportunityListComponent
},
{
    path:  'create-opportunity',
    component:  OpportunityCreateComponent
}

];

Now open src/app/app.module.ts and import the routing module then add it to the imports array:

import {AppRoutingModule} from  './app-routing.module';
[...]

@NgModule({

declarations: [

AppComponent,
[...]
],

imports: [
    BrowserModule,
    AppRoutingModule
],
[...]
})
export  class  AppModule { }

Finally, open src/app/app.component.html then add the navigation links and the router outlet:

<a [routerLink]="'/accounts'"> Accounts </a>
<a [routerLink]="'/create-account'"> Create Account </a>

<a [routerLink]="'/contacts'"> Contacts </a>
<a [routerLink]="'/create-contact'"> Create Contact </a>

<a [routerLink]="'/leads'"> Leads </a>
<a [routerLink]="'/create-lead'"> Create Lead </a>

<a [routerLink]="'/opportunities'"> Opportunities </a>
<a [routerLink]="'/create-opportunity'"> Create Opportunity </a>

<div>
    <router-outlet></router-outlet>
</div>

An Example for Consuming the REST API Using Angular 6 HttpClient

Now that we've created the different components and added routing and navigation, let's see an example of how to use the HttpClient of Angular 6 to consume the RESTful API back-end.

First, you need to add the HttpClientModule module to the imports array of the main application module

[..]
import { HttpClientModule } from  '@angular/common/http';

@NgModule({

declarations: [
..
],

imports: [

[..]

HttpClientModule
],
providers: [],
bootstrap: [AppComponent]

})

export  class  AppModule { }

Create an Angular 6 Service/Provider

A service is a global class that can be injected in any component. It's used to encapsulate code that can be common between multiple components in one place instead of repeating it throughout various components.

Now, lets create a service that encapsulates all the code needed for interacting with the REST API. Using Angulat CLI run the following command:

ng g service api

Two files: src/app/api.service.ts and src/app/api.service.spec.ts will be generated. The first contains code for the service and the second contains tests.

Open src/app/api.service.ts then import and inject the HttpClient class.

import { Injectable } from  '@angular/core';
import { HttpClient} from  '@angular/common/http';

@Injectable({
providedIn:  'root'
})

export  class  APIService {

    constructor(private  httpClient:  HttpClient) {}

}

Angular 6 provides a way to register services/providers directly in the @Injectable() decorator by using the new providedIn attribute. This attribute accepts any module of your application or 'root' for the main app module. Now you don't have to include your service in the providers array of your module.

Getting Contacts/Sending HTTP GET Request Example

Let's start with the contacts API endpoint.

Open src/app/api.service.ts and add the following method:

export  class  APIService {
API_URL  =  'http://localhost:8000';
constructor(private  httpClient:  HttpClient) {}
getContacts(){
    return  this.httpClient.get(`${this.API_URL}/contacts`);
}

Next, open src/app/contact-list/contact-list.component.ts and inject the APIService then call the getContacts() method:

import { Component, OnInit } from  '@angular/core';
import { APIService } from  '../api.service';

@Component({
    selector:  'app-contact-list',
    templateUrl:  './contact-list.component.html',
    styleUrls: ['./contact-list.component.css']
})

export  class  ContactListComponent  implements  OnInit {

private  contacts:  Array<object> = [];
constructor(private  apiService:  APIService) { }
ngOnInit() {
    this.getContacts();
}
public  getContacts(){
    this.apiService.getContacts().subscribe((data:  Array<object>) => {
        this.contacts  =  data;
        console.log(data);
    });
}
}

Now let's display the contacts in the template. Open src/app/contact-list/contact-list.component.html and add the following code:

<h1>
My Contacts
</h1>
<div>
<table  style="width:100%">
<tr>
    <th>First Name</th>
    <th>Last Name</th>
    <th>Phone</th>
    <th>Email</th>
    <th>Address</th>
</tr>
<tr *ngFor="let contact of contacts">
    <td> { { contact.first_name } } </td>
    <td> { { contact.last_name } } </td>
    <td> { { contact.phone } } </td>
    <td> { { contact.email } } </td>
    <td> { { contact.address } } </td>
</tr>
</table>

</div>

This is a screen-shot of the component:

Angular 6 tutorial

Creating Contacts/Sending HTTP POST Request Example

Now let's create a method to send HTTP Post request to create a random contact. Open the API service file and add the following method:

createContact(contact){
    return  this.httpClient.post(`${this.API_URL}/contacts/`,contact);
}

Next let's call this method from the ContactCreateComponent to create a contact. First open src/app/contact-create/contact-create.component.ts and add the following code:

import { Component, OnInit } from  '@angular/core';
import { APIService } from  '../api.service';


@Component({

selector:  'app-contact-create',

templateUrl:  './contact-create.component.html',

styleUrls: ['./contact-create.component.css']

})

export  class  ContactCreateComponent  implements  OnInit {
constructor(private  apiService:  APIService) { }

ngOnInit() {}

createContact(){

var  contact  = {
    account:  1,
    address:  "Home N 333 Apartment 300",
    createdBy:  1,
    description:  "This is the third contact",
    email:  "abbess@email.com",
    first_name:  "kaya",
    isActive: true,
    last_name: "Abbes",
    phone: "00121212101"
};
this.apiService.createContact(contact).subscribe((response) => {
    console.log(response);
});
};
}
}

For now, we're simply hard-coding the contact info for the sake of simplicity.

Next open src/app/contact-create/contact-create.component.html and add a button to call the method to create a contact:

<h1>
Create Contact
</h1>
<button (click)="createContact()">
    Create Contact
</button>

Conclusion

Throught this Angular 6 tutorial for beginners, we've seen, by building a simple real world CRUD example, how to use different Angular concepts to create simple full-stack CRUD application with Angular and Django. You can find the source code in this repository.

December 15, 2018 12:00 AM UTC

December 14, 2018

Continuum Analytics Blog

Python Data Visualization 2018: Where Do We Go From Here?

This post is the third in a three-part series on the current state of Python data visualization and the trends that emerged from SciPy 2018. By James A. Bednar As we saw in Part I and Part II of this series, having so many separate Python visualization libraries to choose from often is confusing to …
Read more →

The post Python Data Visualization 2018: Where Do We Go From Here? appeared first on Anaconda.

December 14, 2018 05:57 PM UTC

Stack Abuse

Introduction to Web Scraping with Python

Introduction

Web-scraping is an important technique, frequently employed in a lot of different contexts, especially data science and data mining. Python is largely considered the go-to language for web-scraping, the reason being the batteries-included nature of Python. With Python, you can create a simple scraping script in about 15 minutes and in under 100 lines of code. So regardless of usage, web-scraping is a skill that every Python programmer must have under his belt.

Before we start getting hands-on, we need to step back and consider what is web-scraping, when should we use it, and when to avoid using it.

As you already know, web-scraping is a technique employed to automatically extract data from websites. What's important to understand is, web-scraping is a somewhat crude technique to extract data from various sources - typically web pages. If the developers of a website are generous enough to provide an API to extract data, that would be a lot more stable and robust way to do access the data. So, as a rule of thumb, if a website provides an API to programmatically retrieve their data, use that. If an API is not available, only then use web-scraping.

Be sure to also comply with any rules or restrictions regarding web scraping for each website you use, as some do not allow it. With that being clear, let's jump right into the tutorial.

For this tutorial, we're going to scrape http://quotes.toscrape.com/, a site that lists famous quotes by renowned authors.

The Web Scraping Pipeline

We can understand web-scraping as a pipeline containing 3 components:

  1. Downloading: Downloading the HTML web-page
  2. Parsing: Parsing the HTML and retrieving data we're interested in
  3. Storing: Storing the retrieved data in our local machine in a specific format

Downloading HTML

It only seems logical that to extract any data from a web-page, we first have to download it. There are two ways we can go about doing this:

1. Using Browser Automation Libraries

You can use browser automation libraries such as Selenium to download the HTML from a web page. Selenium lets you open a browser, let's say Chrome, and control it however you want. You can open the web-page in a browser and then get the HTML code of that page, all automated using Selenium.

However, this method has a huge drawback — it is significantly slower. The reason being the overhead of running the browser and rendering the HTML in the browser. This method should only be used in exceptional cases — cases where the content we want to scrape uses JavaScript code in the browser, or requires us to click buttons/links to get to the data, which Selenium can do for us.

2. Using HTTP Libraries

HTTP libraries, such as the Requests module or Urllib, let you send the HTTP request, bypassing the need to open any browser at all, unlike the first method. This method should always be preferred, as it is much faster than Selenium.

Now let me show you how can we achieve this component of the pipeline using Selenium and Requests libraries:

Using Requests

Install the requests module with the following:

$ pip install requests

And now you can use it in your code, like this:

import requests

result = requests.get('http://quotes.toscrape.com/')  
page = result.text

Here, an HTTP GET request is made to the URL, which is almost synonymous to downloading the webpage. Then, we can get the HTML source of the page by accessing the result object returned by the requests.get() method.

Using Selenium

You can install the selenium module via pip:

$ pip install selenium

from selenium import webdriver

driver = webdriver.Chrome()  
driver.get('http://quotes.toscrape.com/')  
page = driver.page_source

Here, we first start by creating a webdriver object, which represents the browser. Doing this will open the Chrome browser on the computer running the code. Then, by calling the get method of the webdriver object, we can open our URL. And finally, we get the source code by accessing the page_source property of the webdriver object.

In both the cases, the HTML source of the URL is stored in the page variable as a string.

Parsing HTML and Extracting Data

Without getting into theoretical computer science, we can define parsing as the process of analyzing a string so that we can understand its contents and thus access data within it easily.

In Python, there are two libraries that can help us with Parsing HTML: BeautifulSoup and Lxml. Lxml is a more lower-level framework than BeautifulSoup, and we can use Lxml as a backend in BeautifulSoup, so for simple HTML parsing purposes, BeautifulSoup would be the preferred library.

But before we dive into parsing, we have to analyze the web-page's HTML and see how the data we want to scrape is structured and located. Only when we're armed with that information, we can get the information we want from the parsed HTML. But thankfully, we won't have to open the source code in an editor and manually understand and correlate each HTML element with the corresponding data in the rendered page. Most browsers offer an inspector, like Chrome's Developer Tools, which enables us to quickly look at the HTML code of any element just by clicking on them.

To do this in Chrome, open the web page in Chrome, then right-click on the data you want to scrape and select Inspect. In Firefox, this option is called Inspect Element - which is the same thing, but just a different name.

inspect.png

You'll notice a pane opened at the bottom of the Chrome window, containing the source code of the element you clicked on. Browse through the source code a bit to get an idea of how the data which we want to scrape is structured in the HTML code.

source.png

As you can understand after a little bit of inspection, each quote on http://quotes.toscrape.com/ is contained in a div with the class="quote" attribute. Within that div, the text of the quote is in a span with class="text" and the author's name is in a small tag with class="author". This info will be required when we get to actually parsing the HTML and extracting our data.

structure.png

Now, let's start parsing the HTML page using BeautifulSoup. But first, we must install it:

$ pip install beautifulsoup4

Once installed, you can call it in your code like this:

from bs4 import BeautifulSoup

soup = BeautifulSoup(page, 'html.parser')

First of all, we create a parsed version of the page by passing it to the BeautifulSoup class constructor. As you can see, we also pass a second argument to the constructor, html.parser. That is the name of the parser that Beautiful Soup is going to use to parse the string you passed to it. You could've also used the parser lxml, which we previously talked about, given that you have the Lxml library installed.

quotes = soup.find_all('div', class_='quote')

Then, we extract all the div tags in the page containing class="quote", as we know those are the divs containing quotes. To do this, Beautiful Soup 4 offers a find_all function. We passed the tag name and the class name to the find_all function, and it returned all the tags satisfying the conditions, i.e. the tags containing our quotes.

An important thing to note here is, we're working with tree structures here. The variable soup, and also each element of quotes, are trees. In a way, the elements of quotes are parts of the larger soup tree. Anyway, without drifting off into a different discussion, let's carry on.

scraped = []  
for quote in quotes:  
    text = quote.find('span', class_='text').text
    author = quote.find('small', class_='author').text
    scraped.append([text, author])

We know that the text of the quote is in a span tag with class="text" and the author is in a small tag with class="author". To extract them from the quote elements, we again employ a similar function, find. The find function takes the same arguments as the find_all function. The only difference is that it returns the first tag satisfying the conditions, whereas find_all returned a list of tags. Also, we want to access the text property of the returned object, which contains the text enclosed within that tag.

So, as you can see in the code, we loop through all the elements of the list quotes, and extract the quote text and author name, storing them as a list of lists with the name scraped. The scraped list, when printed on the console, looks like this:

[['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
  'Albert Einstein'],
 ['“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
  'J.K. Rowling'],
 ['“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',
  'Albert Einstein'],
 ['“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”',
  'Jane Austen'],
 ["“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”",
  'Marilyn Monroe'],
 ['“Try not to become a man of success. Rather become a man of value.”',
  'Albert Einstein'],
 ['“It is better to be hated for what you are than to be loved for what you are not.”',
  'André Gide'],
 ["“I have not failed. I've just found 10,000 ways that won't work.”",
  'Thomas A. Edison'],
 ["“A woman is like a tea bag; you never know how strong it is until it's in hot water.”",
  'Eleanor Roosevelt'],
 ['“A day without sunshine is like, you know, night.”',
  'Steve Martin']]

Storing the Retrieved Data

Once we have acquired the data, we can store it in whatever format we want e.g, a CSV file, SQL database, or NoSQL database. To be strict, this step shouldn't count as a part of the scraping process, but still, I'll cover it briefly for the sake of completeness.

I'd say the most popular way of storing scraped data is storing them as CSV spreadsheets, so I'll show you how to do just that, very briefly. I won't go into the details, for that you should refer to the official Python documentation. So without further ado, let's jump into the code.

import csv

with open('quotes.csv', 'w') as csv_file:  
    writer = csv.writer(csv_file, delimiter=',')
    for quote in scraped:
        writer.writerow(quote)

As we can see, the code is pretty self-explanatory. We are creating a CSV writer object from the opened quotes.csv file, and then writing the quotes one by one using the writerow function. As it's evident, the writerow function accepts a list as input and then writes that to the CSV as a row.

Conclusion and Next Steps

This tutorial should help you understand what scraping is basically about while learning to implement a simple scraper yourself. This kind of scraper should suffice for simple automation or small-scale data retriever. But if you want to extract large amounts of data efficiently, you should look into scraping frameworks, especially Scrapy. It'll help you write very fast, efficient scrapers using a few lines of code. Whatever framework you use, underneath that shiny surface that framework is also using these very basic scraping principles, so understanding this tutorial should help you build the foundational knowledge for your scrapping adventures.

December 14, 2018 04:11 PM UTC

Will Kahn-Greene

Socorro: migrating to Python 3

Summary

Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the Breakpad crash reporter asks the user if the user would like to send a crash report. If the user answers "yes!", then the Breakpad crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro. Socorro saves the crash report, processes it, and provides an interface for aggregating, searching, and looking at crash reports.

This blog post talks about the project migrating Socorro to Python 3. It covers the incremental steps we did and why we chose that path plus some of the technical problems we hit.

Read more… (16 mins to read)

December 14, 2018 03:00 PM UTC

qutebrowser development blog

Happy birthday, qutebrowser!

5 years ago today, this happened:

commit 11a94957dc038fc27c5ff976197ad2b2d0352d20
Author: Florian Bruhin <git@the-compiler.org>
Date:   Sat Dec 14 22:15:16 2013 +0100

    Initial commit

That's how qutebrowser looked a day after that (and that commit still seems to run!): https://imgur.com/a/xoG1r4G

Exactly a year later, things …

December 14, 2018 08:53 AM UTC

Test and Code

58: Testing REST APIs with Docker containers and pytest

Let's say you've got a web application you need to test.
It has a REST API that you want to use for testing.

Can you use Python for this testing even if the application is written in some other language? Of course.
Can you use pytest? duh. yes. what else?
What if you want to spin up docker instances, get your app running in that, and run your tests against that environment?
How would you use pytest to do that?
Well, there, I'm not exactly sure. But I know someone who does.

Dima Spivak is the Director of Engineering at StreamSets, and he and his team are doing just that.
He's also got some great advice on utilizing code reviews across teams for test code, and a whole lot more.

Special Guest: Dima Spivak.

Sponsored By:

Support Test & Code

Links:

<p>Let&#39;s say you&#39;ve got a web application you need to test.<br> It has a REST API that you want to use for testing. </p> <p>Can you use Python for this testing even if the application is written in some other language? Of course.<br> Can you use pytest? duh. yes. what else?<br> What if you want to spin up docker instances, get your app running in that, and run your tests against that environment? <br> How would you use pytest to do that?<br> Well, there, I&#39;m not exactly sure. But I know someone who does.</p> <p>Dima Spivak is the Director of Engineering at StreamSets, and he and his team are doing just that.<br> He&#39;s also got some great advice on utilizing code reviews across teams for test code, and a whole lot more.</p><p>Special Guest: Dima Spivak.</p><p>Sponsored By:</p><ul><li><a rel="nofollow" href="https://testandcode.com/digitalocean">DigitalOcean</a>: <a rel="nofollow" href="https://testandcode.com/digitalocean">Get started with a free $100 credit </a></li></ul><p><a rel="payment" href="https://www.patreon.com/testpodcast">Support Test &amp; Code</a></p><p>Links:</p><ul><li><a title="Introducing the StreamSets Test Framework" rel="nofollow" href="https://streamsets.com/blog/introducing-the-streamsets-test-framework/">Introducing the StreamSets Test Framework</a></li><li><a title="pytest-benchmark · PyPI" rel="nofollow" href="https://pypi.org/project/pytest-benchmark/">pytest-benchmark · PyPI</a></li><li><a title="StreamSets Test Framework-based tests for StreamSets Data Collector" rel="nofollow" href="https://github.com/streamsets/datacollector-tests">StreamSets Test Framework-based tests for StreamSets Data Collector</a></li><li><a title="StreamSets: Where DevOps Meets Data Integration" rel="nofollow" href="https://streamsets.com/">StreamSets: Where DevOps Meets Data Integration</a></li><li><a title="slack channel for Test &amp; Code" rel="nofollow" href="https://testandcode.com/slack">slack channel for Test &amp; Code</a></li></ul>

December 14, 2018 07:00 AM UTC

December 13, 2018

Python Engineering at Microsoft

Python in Visual Studio Code – December 2018 Release

We are pleased to announce that the December 2018 release of the Python Extension for Visual Studio Code is now available. You can download the Python extension from the Marketplace, or install it directly from the extension gallery in Visual Studio Code. You can learn more about Python support in Visual Studio Code in the documentation.

This release was a short release, where we primarily focused on two top-requested features for the data science experience shipped in November: remote Jupyter support and export Python files as Jupyter Notebooks. We have also fixed many issues reported on GitHub, and you can see the full list in our changelog.

Keep on reading to learn more!

Remote Jupyter Support

This release enables you to connect to remote Jupyter servers for execution, so you can offload intensive computation to other machines that have more compute power or have a specific hardware spec that you need.

To connect to a remote Jupyter server, run this new command Python: Specify Jupyter server URI in the Visual Studio Code Command Palette.

Then you will be asked to enter the hostname and a token for authentication.

You can find the token in the terminal when you start a Jupyter Notebook server with token authentication enabled. The token is generated and logged to the terminal. For example:

[I 11:59:16.597 NotebookApp] The Jupyter Notebook is running at:
http://localhost:8888/?token=c8de56fa4deed24899803e93c227592aef6538f93025fe01

Copy the URL in the terminal and then paste into VS Code.

Cells will now be executed using the remote Jupyter server, and you can see that information in the Interactive window. Here is an example of a Windows machine connecting to a Linux VM for execution (the real URI is blurred out for security reasons 😊):

Token-based authentication is on by default for notebook 4.3 and later. This document has more details on Security in the Jupyter notbeook server.

Export Python files as Jupyter Notebooks

This update adds two commands for exporting Python files as Jupyter Notebooks. Along with the export run results command that was shipped in the previous release, the Python extension now offers three Export as Jupyter Notebook options and you can choose the one that is right for your use-case.

After exporting you will see a message box with an “Open in browser” button, which will open the exported notebook file locally in a browser.

Other Changes and Enhancements

We have also added small enhancements and fixed issues requested by users that should improve your experience working with Python in Visual Studio Code. The full list of improvements is listed in our changelog; some notable changes include:

Be sure to download the Python extension for Visual Studio Code now to try out the above improvements. If you run into any problems, please file an issue on the Python VS Code GitHub page.

 

December 13, 2018 10:53 PM UTC

NumFOCUS

Double Your Donation: NumFOCUS Receives $5,000 Matching Gift Challenge

The post Double Your Donation: NumFOCUS Receives $5,000 Matching Gift Challenge appeared first on NumFOCUS.

December 13, 2018 05:13 PM UTC

gamingdirectional

Create an exit button on the main game menu scene

In this article we will continue with our previous pygame project by creating an exit button on the main game menu scene which when the user clicks on it the program will close the game. We will create the score scene in the next chapter as I have promised in the previous article but in this article we will create the exit button which is also very important for this project. This button will only...

Source

Related posts:

Open a text file then write the game level onto it with python Create the pause scene for the game Create the third level for this pygame project Lets move on to the next game level! Create a pool object for player missile Create an explosion manager and explosion class for pygame project Pygame loads image and background graphic on game scene The game state engine is ready! Playing background music with Pygame Draw lines with Pygame

December 13, 2018 10:58 AM UTC

Jeff Knupp

How To Do Just About Anything With Python Lists

Python's list is one of the built-in sequence types (i.e. "it holds a sequence of things") is a wonderfully useful tool. I figured I'd try to determine what people are most often trying to do with lists (by analyzing Google's query data on the topic) and just bang out examples of "How do I do X with a list in Python?"

Reverse/Sort A List In Python

There are two ways to reverse a list in Python, and which one you use depends on what you want to do with the resulting reversed data. If you're only going iterate over the items in the reversed list (say, to print them out), use the Python built-in function reversed(seq). Here's an example of reversed in action:

original_list = [1, 2, 3, 4, 5]
for element in reversed(original_list):
    print(element)

If you need the reversed list itself, use the Python built-in function sorted(iterable, *, key=None, reverse=False). Let's see some examples:

In [10]: original_list = [3, 1, 2, 5, 4]

In [11]: sorted(original_list)
Out[11]: [1, 2, 3, 4, 5]

In [15]: sorted(original_list, reverse=True)
Out[15]: [5, 4, 3, 2, 1]

But what if your list contains more than just simple integers? How does one sort, say, a list of temperature readings over a given time if those daily readings are each stored as a tuples of the form (<date>, <daily high>, <daily low>)? Look at the following example:

readings = [('1202', 45.0, 28.1), ('1201', 44.0, 33.0), ('1130', 45.0, 32.6)]

Calling sorted(readings) will give us a new list with the elements ordered by the <date> portion of the tuple (the 0-th element, since Python compares tuples lexicographically; each item is compared in order, starting with the first elements). But what if we wanted to sort by <daily high> or <daily low>? Simple! Just pass the key parameter a function that takes a single argument and returns the key for sorted() to use for comparisons. For example, I could sort by daily low temperatures like so:

In [25]: sorted(readings, key=lambda reading: reading[2])
Out[25]: [('1202', 45.0, 28.1), ('1130', 45.0, 32.6), ('1201', 44.0, 33.0)]

In that example, we passed the key parameter a lambda function which accepted one argument and returned a value, in our case the third part of our temperature recording tuple (the reading[2] part). If we had wanted to sort by the daily high in reverse order, we would just change the call like so:

In [26]: sorted(readings, key=lambda reading: reading[1], reverse=True)
Out[26]: [('1202', 45.0, 28.1), ('1130', 45.0, 32.6), ('1201', 44.0, 33.0)]

Accessing elements of a tuple or class is such a common task that Python provides a set of convenience functions in the operator built-in module. To access a specific field in the tuple (as we did above for daily high as reading[1] and daily low as reading[2]), use the field's index in the tuple as an argument to operator.itemgetter:

In [29]: sorted(readings, key=itemgetter(1), reverse=True)
Out[29]: [('1202', 45.0, 28.1), ('1130', 45.0, 32.6), ('1201', 44.0, 33.0)]

But notice that the first two entries have the same high temp recordings (45.0). What if we wanted to first sort by high temp and then by low temp? itemgetter allows for multiple levels of sorting by simply passing in multiple index values. So let's sort by high temp first, then low temp:

In [31]: sorted(readings, key=itemgetter(1,2), reverse=True)
Out[31]: [('1130', 45.0, 32.6), ('1202', 45.0, 28.1), ('1201', 44.0, 33.0)]

Notice that the ('1130', 45.0, 32.6) tuple is now first, as it had an equal high temp and a greater low temp than ('1202', 45.0, 28.1).

Split A Python List Into Chunks

Splitting a list into equally sized sub-lists (for processing data in parallel, perhaps) is a common task. It's so common, in fact, that the itertools module (a module practically begging to be used in these kinds of tasks, by the way) gives actual code for how to accomplish this in Python in the itertools recpipes section of the docs. Here is the relevant code:

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

The code may look confusing at first, but it's simply creating n (or 3 in the example in the comments) separate iterators over the iterable argument and then cleverly zipping them back together using zip_longest from the itertools module, to collect the elements of iterable in a series of n-sized chunks.

Flatten A Python List Of Lists Into One List

itertools recipes FTW again! Straight out of the section of the Python docs that gave us grouper above, the recipe for "flatten" is:

def flatten(list_of_lists):
    "Flatten one level of nesting."
    return chain.from_iterable(list_of_lists)

It simply calls itertools.chain.from_iterable() on the list_of_lists. A call to flatten([[1, 2, 3], [4, 5], [6, 7, 8]]) will give us an iterator that yields each element individually. We can say flattened_list = list(flatten([[1, 2, 3], [4, 5], [6, 7, 8]])) if what we need is an actual list and not just an iterator.

Insert Into A List In Python

The word "insert" here is vague (insert where?), but let's roll with it. Here are some of the flavors of list assignment mentioned in the Python docs about operations on "Mutable Sequence Types":

Operation   Result

s[i] = x    item i of s is replaced by x     
s[i:j] = t  slice of s from i to j is replaced by the contents of the iterable t     
s[i:j:k] = t    the elements of s[i:j:k] are replaced by those of t (as long as t is the same length as the slice it's replacing)
s.append(x) appends x to the end of the sequence (same as s[len(s):len(s)] = [x])
s.extend(t) or s += t   extends s with the contents of t (for the most part the same as s[len(s):len(s)] = t)    
s.insert(i, x)  inserts x into s at the index given by i (same as s[i:i] = [x])

So six different ways to insert into a list in Python. That doesn't seem very Pythonic!? Let's see which we might use on a case by case basis:

Insert Into The Beginning Of A List In Python

s.insert(0, value)

Insert Into The End Of A List In Python

s.append(value)

Insert Into An Existing Index Of A Python List

s[index] = value

Concatenating Two Python Lists

s += t  # Where s and t are both lists

Python "List Index Out Of Range" Error Message

This entry is less a "how-to" and more of a "what to do when things go wrong" type of entry, but it's nonetheless searched for very often in conjunction with Python lists (for obvious reasons). The message "list index out of range" is brought to you by a specific type of built-in Exception, namely the IndexError. It is simply saying that you tried to access a list, perhaps using code like the following:

def my_function(some_list):
    """Return something really interesting..."""
    third_value = some_list[2]
    # ... possibly more code following

2 is not a valid index to use because some_list does not have three values (we don't know how many it does have, just less than 3 in this case).

Want to know how many values some_list has? The Python built-in len(s) function will give you exactly that.

A simple debugging exercise of the "list index out of range" message might look like this:

def my_function(some_list):
    try:
        third_value = some_list[2]
    except IndexError:
        print(f'some_list only has {len(some_list)} entries')
        raise

Here we put our code in a try...except block and catch any IndexError exceptions that are raised. When we see one, we just print out the length of some_list and re-raise the exception (since we can't exactly handle that in any useful way). That gives me the following output:

In [50]: def my_function(some_list):
    ...:         try:
    ...:             third_value = some_list[2]
    ...:         except IndexError:
    ...:             print(f'some_list only has {len(some_list)} entries')
    ...:             raise
    ...:

In [51]: my_function([1,2])
some_list only has 2 entries
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-51-3f735c70ddb9> in <module>
----> 1 my_function([1,2])

<ipython-input-50-137ffeaa7c70> in my_function(some_list)
    1 def my_function(some_list):
    2         try:
----> 3             third_value = some_list[2]
    4         except IndexError:
    5             print(f'some_list only has {len(some_list)} entries')

IndexError: list index out of range

How To Make A List In Python

The section topic is taken directly from search query volume, so rest assured this isn't me just throwing random facts about Python lists at you. There are a few ways to create a list in Python, but the two you'll use the most is using the "square-brace" syntax:

Generally speaking, prefer the use of the first two methods unless you can think of a very good reason not to.

Apply A Function To A List In Python

Remember when I said list comprehensions were out of scope for this article. I lied. Even though they're a little trickier to grasp, they're very powerful and applying a function to each element in a list in Python is basically their bread and butter. Imagine we have a list of the numbers one through five and want to create a second list whose elements exactly each element in the original list multiplied by two. To construct such a list using a for loop, we'd need the following code:

original_list = [1,2,3,4,5]
double_list = []
for element in original_list:
    double_list.append(element * 2)

The need to do this occurs so frequently that (especially for performance-sensitive applications) a highly optimized general form of the above as part of the Python language in list comprehensions. Rather than go over the topic in great detail here, I'll simply link to the Python documentation on list comprehensions and note that we could achieve the same as code above using a list comprehension, like so:

original_list = [1,2,3,4,5]
double_list = [element * 2 for element in original_list]

I've grown to appreciate the precision of the grammar when reading it aloud, but it definitely takes some getting used to (and this is only the simplest use of a list comprehension).

Of course, there's always the Python built-in function map(), which will do much the same thing in a slightly easier to grok way, but there is an important difference between the two: map() yields the values back one-at-a-time (it's a generator) whereas a list comprehension will always create a new list from the existing list given as an argument.

It may seem difficult, but try to get to a place where you're using list comprehensions rather than map() as the former are both more powerful and more Pythonic.

Summary

This article was a quick hit, but it's good to go back to basics occasionally and see if there's anything you might have missed your first time through (I'll wager that very few Python developers know and use the itertools module as well and often as they should). I tentatively plan on adding to this article over time (a change log would be provided). Let me know if you think there's a common "Python list" Google query I missed or with suggestions on the other parts of Python lists that often trip up newcomers.

December 13, 2018 08:02 AM UTC

Yasoob Khalid

Research Writeup: Deanonymization and Proximity Detection Using Wi-Fi

Hi everyone! If you have been following my blog for a while you will know that I did research at Colgate University over the summers. My research was on Wi-Fi and how I can do some interesting stuff using it. The university just published its annual catalogue of all the research projects which happened over the summer.  My research was done under the mentorship of Aaron Gember-Jacobson. I could not have asked for a better advisor. Here is the writeup of my project:

According to RAINN (Rape, Abuse & Incest National Network), 23.1% of female and 5.4% of male undergraduate students experience rape or sexual assault, with only a minute percentage reporting their assault to law enforcement1. In certain cases, survivors can forget who the perpetrator was due to trauma and/or intoxication. I want to use technology to counter this problem. My hope is to reduce the number of potential culprits when such an incident occurs to make it easier for the survivor to identify the perpetrator.

This can be made possible by using a device that most people carry at all times – a smartphone. The idea is to save the device identifier and the distance between your phone and that of each person who comes near you in a searchable database. This allows you, the user, to search for which device was near you at a particular time. The research is further divided into two parts. The first involved finding a way to effectively calculate the relative distance between two smartphones and the second involved information storage and querying. I focused mainly on the first part, which turned out to be more difficult and involved than I anticipated.

The cornerstone of this idea is Wi-Fi and the information your smartphone emits when the Wi-Fi is turned on, though not necessarily connected to an access point. The formal requirements of this system are as follows: it should be passive so you don’t have to actively monitor it; it shouldn’t require other people’s smartphones to run any specific application; the error in distance estimation should be less than 1 meter so the algorithm can accurately identify a human interaction; the system needs to work in NLOS (Non-line-of-sight) scenarios since people often have their smartphones in their pockets; finally, it should not require more than three devices, including your smartphone, a nearby smartphone, and a Wi-Fi Access Point to which both phones are connected, because the system should be portable.

Previous research in relative distance estimation offers varying levels of precision. One method involves using RSSI (Received Signal Strength Indication) readings from multiple access points (4+ for accuracy) and triangulating smartphone position based on that. We cannot use this method because 4+ devices are required. Another method involves using Time-of-Flight (ToF) measurements.  There are multiple variations of this method, but the basic idea is to send data from your device to the device being localized, and recording the time taken for the data to travel from one device to another and for an acknowledgment to be received. Based on this timing measurement and the required time delay (known as SIFS, or Short Interframe Space) between a device receiving data and sending an acknowledgement, we can estimate the distance between two devices. This gives the best accuracy but is not directly applicable to this situation, because it requires a direct connection between the two smartphones.

I sought to develop a modified version of the ToF method, because it offers the best precision and requires the least number of devices to work effectively. The method I developed was to send unsolicited control packets (a special type of data frame) to the target mobile device and force it to send an acknowledgement (see figure). The major research question is: how do we force the target device to send an acknowledgement even if we are not directly connected to it?

I set up a testbed with three desktops equipped with Wi-Fi cards and running Ubuntu Linux. I used Scapy (a Python program for generating network packets) to generate and send control packets from one desktop to another and tcpdump on the third desktop to monitor and analyze the wireless communication taking place. I was able to send the control packets and solicit an acknowledgment from the target mobile (Ubuntu desktop) without being directly connected to it.

However, there was a bug in the networking drivers of Ubuntu that generated acknowledgments even in cases where no acknowledgment was supposed to be sent by the target device. Currently, I am investigating the bug and trying to figure out the most suitable way forward. Through this research, I found that the process of distance estimation is more complicated than it seems. There are several variables and timing issues that need to be taken into account.  In the future, I plan on finding a workaround for this bug, with the eventual goal of making this system usable in everyday life.

If you have any questions about my research or anything in general please write them in the comments below. Looking forward to hearing your views! Have a great day/night! 🙂 

December 13, 2018 05:54 AM UTC