skip to navigation
skip to content

Planet Python

Last update: December 10, 2016 09:48 PM

December 10, 2016


Codementor

Building a Chatbot using Telegram and Python (Part 1)

Chatbots are all the rage at the moment, with some predicting that they will be bigger than mobile apps. The main idea of chatbots is that instead of having to dig through awkward mobile menus and learn UIs, you’ll simply have a conversation with a bot through a familiar
instant messaging interface. If you want to order a Pizza, you start a conversation with the Domino’s Pizza bot and have the same conversation with it that you might have with a human.

There are a few different platforms that allow you to build your own chatbot. One of these, which is arguably the simplest to use and is also growing steadily in popularity, is Telegram.

In this tutorial, we’ll walk through building a simple Telegram Bot using Python. At first, our bot will simply echo back any message we send it, but then we’ll extend it to add a database and persist information across chat sessions.

We’ll use Python to power our Bot and SQLite to store information persistently across sessions. In summary, this is a tutorial series that will:

Although creating an Echo Bot is simple enough, and you can find various scripts and frameworks online that will give you this as a starting point—we will do everything from scratch and explain every piece of code we write. We’ll also look at some subtleties in the Telegram API, and talk about what these mean for us as developers. If you just want to create a Telegram bot as quickly as possible, this tutorial is probably not what you’re looking for, but if you want to gain a deeper understanding of how chatbots work and how to build one from scratch, then you’re in the right place.

What you need

You’ll need to have:

All of the code is aimed at Python 3.5, but it should be easily adaptable to other versions of Python.

Why Python?

You can write a Telegram chat bot in any language you want. Some of the main options apart from Python would be Java, PHP, or Ruby. If you are more familiar with a different high-level programming language, then you might prefer to use that instead, but Python is a good choice for several reasons:

Setting up

Nearly everything we do will be achievable using only the standard Python libraries, but we’ll also be using the third-party requests module which provides a number of improvements to Python’s urllib, and allows us to make HTTP requests very simply and concisely. Install this through pip using a command similar to the following (you may need to use pip instead of pip3 and/or add the --user flag, based on how you usually install Python libraries).

pip3 install requests

If you usually use a virtual environment for new Python projects, then set one of those up first, and install requests inside that.

Creating a Telegram Bot

The first step is to tell Telegram that you want to create a new bot. All the messages that our Bot sends and receives will go through Telegram’s infrastructure. Our code will periodically make a request to retrieve all new messages to our Bot from Telegram’s servers, and will then send responses to each message as necessary. In order to register a bot with Telegram, you first need to create a personal Telegram account. Visit web.telegram.org and enter your phone number. Telegram will send you a text message (SMS), and you can then create an account by following the instructions on the screen. If you already have a Telegram account, then you can simply use that one, and you can also use any of the Telegram Desktop and Mobile apps available from telegram.org, instead of the Web app that we’ll be using for all examples in this tutorial.

Once you have a Telegram account, you can register a new Telegram Bot by using Bot Father. Visit telegram.me/botfather to start a conversation with Telegram’s bot that creates other bots. Telegram bots can receive messages or commands. The former are simply text that you send as if you were sending a message to another person, while the latter are prefixed with a / character. To create a new bot, send the following command to Bot Father as a chat (exactly as if you were talking to another person on Telegram).

/newbot

You should get a reply instantly that asks you to choose a name for your Bot. We’ll call our Bot To Do Bot because, by the end of this tutorial, it’ll function as a simple “to do” list. Send the following message to Bot Father when it prompts you for a name:

To Do Bot

Bot Father will now ask you to pick a username for your Bot. This username has to end in bot, and be globally unique. As Telegram has grown more popular, it has become more difficult to find a short and relevant username for your Bot. In this tutorial, we’ll be using exampletodo_bot, but for the rest of this tutorial, we’ll indicate the Bot’s username with <your-bot-username>, so you’ll have to substitute your chosen username wherever relevant from now on. Send your chosen username to Bot Father:

<your-bot-username>

Now Bot Father will send you a “Congratulations” message, which will include a token. The token should look something like this:

2483457814:AAHrlCx234_VskzWEJdWjTsdfuwejHyu5mI

For the rest of this tutorial, we’ll indicate where you need to put your token by using <your-bot-token>.

Take note of the token, as we’ll need it in the code that we’re about to write.

Interacting with our Bot through our web browser

We can control our Bot by sending HTTPS requests to Telegram. This means that the simplest way to interact with our Bot is through a web browser. By visiting different URLs, we send different commands to our Bot. The simplest command is one where we get information about our Bot. Visit the following URL in your browser (substituting the bot token that you got before)

https://api.telegram.org/bot<your-bot-token>/getme

The first part of the URL indicates that we want to communicate with the Telegram API (api.telegram.org). We follow this with /bot to say that we want to send a command to our Bot, and immediately after we add our token to identify which bot we want to send the command to and to prove that we own it. Finally, we specify the command that we want to send (/getme) which in this case just returns basic information about our Bot using JSON. The response should look similar to the following:

{"ok":true,"result":{"id":248718785,"first_name":"To Do Bot","username":"exampletodo_bot"}}

Retrieving messages sent to our Bot

The simplest way for us to retrieve messages sent to our Bot is through the getUpdates call. If you visit https://api.telegram.org/bot<your-bot-token>/getUpdates, you’ll get a JSON response of all the new messages sent to your Bot. Our Bot is brand new and probably hasn’t received any messages yet, so if you visit this now, you should see an empty response.

Telegram Bots can’t talk to users until the user first initiates a conversation (this is to reduce spam). In order to try out the getUpdates call, we’ll first send a message to our Bot from our own Telegram account. Visit telegram.me/<your-bot-username> to open a conversation with your Bot in the web client (or search for @<your-bot-username> in any of the Telegram clients). You should see your Bot displayed with a /start button at the bottom of the screen. Click this button to start chatting with your Bot. Send your Bot a short message, such as “hello”.

Now visit the https://api.telegram.org/bot<your-bot-token>/getUpdates URL again, and you should see a JSON response showing the messages that your bot has received (including one from when you pressed the start button). Let’s take a look at an example of this and highlight the import data that we’ll be writing code to extract in the next section.

{"ok":true,"result":[{"update_id":625407400,
"message":{"message_id":1,"from":{"id":24860000,"first_name":"Gareth","last_name":"Dwyer (sixhobbits)","username":"sixhobbits"},"chat":{"id":24860000,"first_name":"Gareth","last_name":"Dwyer (sixhobbits)","username":"sixhobbits","type":"private"},"date":1478087433,"text":"\/start","entities":[{"type":"bot_command","offset":0,"length":6}]}},{"update_id":625407401,
"message":{"message_id":2,"from":{"id":24860000,"first_name":"Gareth","last_name":"Dwyer (sixhobbits)","username":"sixhobbits"},"chat":{"id":24860000,"first_name":"Gareth","last_name":"Dwyer (sixhobbits)","username":"sixhobbits","type":"private"},"date":1478087624,"text":"test"}}]}

The result section of the JSON is a list of updates that we haven’t acknowledged yet (we’ll talk about how to acknowledge updates later). In this example, our Bot has two new messages. Each message contains a bunch of data about who sent it, what chat it is part of, and the contents of the message. The two pieces of information that we’ll focus on for now are the chat ID, which will allow us to send a reply message and the message text which contains the text of the message. In the next section, we’ll see how to extract these two pieces of data using Python.

Sending a message from our Bot

The final API call that we’ll try out in our browser is that used to send a message. To do this, we need the chat ID for the chat where we want to send the message. There are a bunch of different IDs in the JSON response from the getUpdates call, so make sure you get the right one. It’s the id field which is inside the chat field (24860000 in the example above, but yours will be different). Once you have this ID, visit the following URL in your browser, substituting <chat-id> for your chat ID.

https://api.telegram.org/bot<your-bot-id>/sendMessage?chat_id=<chat-id>&text=TestReply

Once you’ve visited this URL, you should see a message from your Bot sent to your which says “TestReply”.

Now that we know how to send and receive messages using the Telegram API, we can get going with automating this process by writing some logic in Python.

Writing the Python code for our Bot

Now we can get to writing Python. Create the file echobot.py and add the following code:

import json 
import requests

TOKEN = "<your-bot-token>"
URL = "https://api.telegram.org/bot{}/".format(TOKEN)


def get_url(url):
    response = requests.get(url)
    content = response.content.decode("utf8")
    return content


def get_json_from_url(url):
    content = get_url(url)
    js = json.loads(content)
    return js


def get_updates():
    url = URL + "getUpdates"
    js = get_json_from_url(url)
    return js


def get_last_chat_id_and_text(updates):
    num_updates = len(updates["result"])
    last_update = num_updates - 1
    text = updates["result"][last_update]["message"]["text"]
    chat_id = updates["result"][last_update]["message"]["chat"]["id"]
    return (text, chat_id)


def send_message(text, chat_id):
    url = URL + "sendMessage?text={}&chat_id={}".format(text, chat_id)
    get_url(url)
    

text, chat = get_last_chat_id_and_text(get_updates())
send_message(text, chat)

Let’s pull apart what this code does:

The final two lines bring everything we have written together to actually receive and send a message. First, we get the text and the chat ID from the most recent message sent to our Bot. Then, we call send_message using the same text that we just received, effectively “echoing” the last message back to the user.

At the moment, our script doesn’t listen for new messages and immediately reply. Instead, when we run it, our bot will fetch only the most recent message sent to that and echo it. We can test it out by sending our bot a message, and then running the script. Give this a go!

Flaws with our bot

The most obvious problem with our Bot is that we have to run a Python script manually every time we want to interact with it. Also, as mentioned before, we always download the entire message history that Telegram provides. This is both inefficient and unreliable, as we don’t want to unnecessarily download the entire message history if we only want a single message, and because Telegram only keeps this list of updates for 24 hours. Another issue is that we pass our message as a string, but because this is converted to a URL before being sent to Telegram, you’ll notice that some unexpected things happen if you send messages to the bot with special characters (for example, the + symbol will disappear from all echoed messages). Finally, the Bot throws an index error if we try to run it when there are no new messages to receive.

We’ll now update our bot to:
* Constantly listen for new messages and reply to each.
* Acknowledge each message as it receives it and tells Telegram to not send us that message again.
* Use Long Polling so that we don’t have to make too many requests.
* Correctly encode our messages to account for URL formatting.

Listening for new messages

We don’t want to manually start our Bot every time that we want it to reply to the latest message, so the first thing to do is to wrap our code that receives new messages and echoes them back in a loop. We’ll also put this in a main function and use the Pythonic if __name__ == '__main__' statement so that we could import our functions into another script without running anything. We don’t want to ask for new updates as fast as possible, so we’ll also put a small delay between requests (this is kinder to Telegram’s servers and better for our own network resources, too).

At the top of the file, add a new import for the Python time module

import time

And change the last two lines of the file to read as follows:

def main():
    last_textchat = (None, None)
    while True:
        text, chat = get_last_chat_id_and_text(get_updates())
        if (text, chat) != last_textchat:
            send_message(text, chat)
            last_textchat = (text, chat)
        time.sleep(0.5)


if __name__ == '__main__':
    main()

This code now gets the most recent messages from Telegram every half second. We now also need to remember the most recent message that we replied to (we save this in the last_textchat variable) so that we don’t keep on sending the echoes every second to messages that we’ve already processed. This is again a very crude way of achieving what we want (for example, if we send the same message to our bot twice in a row, it won’t reply to the second one), but we’ll see a more elegant way to achieve this below. For now, you can run this code and now instead of the script terminating, you’ll see that it keeps running. You can now send your Bot a series of messages, and (as long as you don’t send more than one per half second), you’ll see each of them getting echoed back again.

Acknowledging the messages we’ve already seen

Instead of asking Telegram for all our recent messages with every call, and then trying to figure out which ones we are interested in, we can tell Telegram that we’ve already processed certain messages and that we want to stop receiving them as part of the getUpdates calls. Each update has an update_id field, and these are incremental (later messages have higher numbers). When we make the getUpdates API call, we can optionally pass an offset argument and give an update_id as the value. This tells Telegram that we’ve already seen and processed that message and that we don’t want it again. This also means that Telegram will never send us any of the previous messages (messages with a lower update_id) again either, so we need to make sure that we really are finished with all of the messages before doing this.

Modify our bot as follows:

def get_updates(offset=None):
    url = URL + "getUpdates"
    if offset:
        url += "?offset={}".format(offset)
    js = get_json_from_url(url)
    return js
def get_last_update_id(updates):
    update_ids = []
    for update in updates["result"]:
        update_ids.append(int(update["update_id"]))
    return max(update_ids)

This simply loops through each of the updates that we get from Telegram and then returns the biggest ID. We need this so that we can call getUpdates again, passing this ID, and indicate which messages we’ve already seen.

def echo_all(updates):
    for update in updates["result"]:
        text = update["message"]["text"]
        chat = update["message"]["chat"]["id"]
        send_message(text, chat)
def main():
    last_update_id = None
    while True:
        updates = get_updates(last_update_id)
        if len(updates["result"]) > 0:
            last_update_id = get_last_update_id(updates) + 1
            echo_all(updates)
        time.sleep(0.5)

Our main code no longer needs to worry about duplicate messages, as each time we get new messages, we send the biggest update_id along with the next request, ensuring that we only ever receive messages that we haven’t seen before.

Note that we have to check if there are new updates (which we do in the third line of main()), and that we have to always send an update ID which is one bigger than the previous one we’ve seen (i.e. we’re actually telling Telegram which ID we’re expecting, not which one we’ve seen).

Try out the changes by restarting the Python script and sending some messages to your Bot—you should see that it works as before, but now it doesn’t matter if you send duplicate messages or send messages too quickly, both of which are big improvements.

Using Long Polling

The last major problem with our Echo Bot is that it has to make a web request every 0.5 seconds. This is not great for Telegram’s servers (they explicitly ask people not to do this outside of testing scenarios) and not great for our resources either. Long Polling takes advantage of the fact that most of the time, we are receiving “empty” responses. Because our Bot is probably not going to be receiving messages every half second, most of the time when we ask for updates, there aren’t any. With Long Polling, instead of Telegram telling us that there aren’t updates, it simply keeps the connection open until there are updates, and then sends these down the open pipe. Of course, it’s impractical to keep a connection open forever, so we can specify the number of seconds that we want to wait for. This is done by passing another optional argument to the getUpdates call, namely timeout.

To make our code use Long Polling, simply update our get_updates method as follows:

def get_updates(offset=None):
    url = URL + "getUpdates?timeout=100"
    if offset:
        url += "&offset={}".format(offset)
    js = get_json_from_url(url)
    return js

Now we always pass along the timeout argument. Because we now have two arguments, we also need to change where we previously had ?offset={} to &offset={} (in URLs, we specify that the argument list is starting with a ? but further arguments are separated with &).

Run the bot again, and it should run exactly as before, but now it’ll be making far fewer requests and using less of your machine’s resources. If you want to check that this is working, simply add a line like print("getting updates") directly below the while True in the main function and run the bot with and without the timeout argument that we just added. Without the timeout, you’ll see that the bot checks for updates every 0.5 seconds. While with the timeout, it will only initiate a new check every 100 seconds, or whenever a new message is received.

Correctly encoding our message text

The final problem of our echo bot is that it acts strangely if we send it messages containing special characters. For example, all + signs disappear from our messages, and all text after an & sign disappears, too. This is caused by these symbols having special meanings in the context of URLs. To fix this, we need to encode any special characters in our message. Luckily, the standard Python urllib has a function that handles this for us, so we only need to import that and add a single line of code.

Add the following line at the top of your .py file

import urllib

And now modify the send_message function to read as follows:

def send_message(text, chat_id):
    text = urllib.parse.quote_plus(text)
    url = URL + "sendMessage?text={}&chat_id={}".format(text, chat_id)
    get_url(url)

Restart the Bot once more, and send it some messages that were problematic before, such as:

+
Hello+
Hello&test

Now it should be able to reply to all of these messages (and pretty much anything else you throw at it, including emoji) flawlessly.

End of Part 1

That brings us to the end of the first part of this tutorial. We built a simple Echo Bot using the Telegram Bot API from scratch and implemented some more advanced features such as keeping track of which messages we’d already processed, using Long Polling, and correctly encoding our messages for URLs. In the next part, we’ll add a database and turn our Bot into something more useful—a To Do List.

The final code listing for the Echo Bot presented here can be found at https://github.com/sixhobbits/python-telegram-tutorial.

Part 2 of this tutorial can be found here: https://www.codementor.io/garethdwyer/tutorials/building-a-chatbot-using-telegram-and-python-part-2-sqlite-databse-backend-m7o96jger

More to come: Watch out for Part 3 of this 3-part tutorial.

December 10, 2016 12:09 PM


hypothesis.works articles

How Hypothesis Works

Hypothesis has a very different underlying implementation to any other property-based testing system. As far as I know, it’s an entirely novel design that I invented.

Central to this design is the following feature set which every Hypothesis strategy supports automatically (the only way to break this is by having the data generated depend somehow on external global state):

  1. All generated examples can be safely mutated
  2. All generated examples can be saved to disk (this is important because Hypothesis remembers and replays previous failures).
  3. All generated examples can be shrunk
  4. All invariants that hold in generation must hold during shrinking ( though the probability distribution can of course change, so things which are only supported with high probability may not be).

(Essentially no other property based systems manage one of these claims, let alone all)

The initial mechanisms for supporting this were fairly complicated, but after passing through a number of iterations I hit on a very powerful underlying design that unifies all of these features.

It’s still fairly complicated in implementation, but most of that is optimisations and things needed to make the core idea work. More importantly, the complexity is quite contained: A fairly small kernel handles all of the complexity, and there is little to no additional complexity (at least, compared to how it normally looks) in defining new strategies, etc.

This article will give a high level overview of that model and how it works.

Read more...

December 10, 2016 11:00 AM


Malthe Borch

A Letter for Laura

Sat, 10 Dec 201608:30:00 GMT

Laura was my recruiter. She'd found me from a pile of discarded resumes and thought perhaps it was a mistake.

I think she is no longer entirely of that persuasion – I didn't get the job – but I do think she knows that tech hiring is broken and not just for old farts.

Because it's still true that the tech interview is in most respects a reverse turing-test: prove that you're not human.

It's well-known that you've got to concentrate to do good work. You're unlikely to get to that state in an interview. The assumption is that it doesn't matter because it's the same for everyone. That we can in fact use your bad performance vis-a-vis your peers as a heuristic for future performance.

I think I'm a false negative because I can do it. My past performance shows it and certainly, anyone can visit my open source profile on Github and see for themselves.

I realize that most developers just aren't very good and do need to be weeded out and that not everyone will have a portfolio of past work. And perhaps we're only ever as good as our last two year's work. I'll argue that if you can give good answers to the following questions then you're ahead of most other candidates.

Of course, I'm biased. I think the answer to the last question is "not at all". But I'll accept your answer if it's good. I want to work with people who care about software – philosophy included. It's ultimately about getting things done, but it's equally important to sometimes get out of the weeds, take a step back and talk about what we think of this field.

After all, the first step in gettings things done is not having so much to do.

December 10, 2016 10:48 AM


Catalin George Festila

The python modules pygobject, pycairo and pygtk under Windows OS.

I used the python version 2.7 32 bits under .
C:\Python27>python
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (
Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

The issue is to instal PyGTK python module but you can see I got errors.
C:\>cd Python27

C:\Python27>cd script
The system cannot find the path specified.

C:\Python27>cd Scripts

C:\Python27\Scripts>pip install PyGTK
Collecting PyGTK
Downloading pygtk-2.24.0.tar.bz2 (2.4MB)
100% |################################| 2.4MB 224kB/s
Complete output from command python setup.py egg_info:
ERROR: Could not import dsextras module: Make sure you have installed pygobj
ect.

----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in c:\users\my_account
\appdata\local\temp\pip-build-os60hf\PyGTK\

C:\Python27\Scripts>pip install pygobject
Collecting pygobject
Downloading pygobject-2.28.3.tar.bz2 (889kB)
100% |################################| 890kB 550kB/s
Complete output from command python setup.py egg_info:
ERROR: Could not find pkg-config: Please check your PATH environment variabl
e.

----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in c:\users\my_account
\appdata\local\temp\pip-build-lt0gbh\pygobject\

I got the pygobject-2.28.3.win32-py2.7.msi, pycairo-1.8.10.win32-py2.7.msi and pygtk-2.24.0.win32-py2.7.msi from gnome website.
I install this python modules using registry and this option: Python from another location and I set my python path: C:\Python27\
>>> import gobject
>>> dir(gobject)
['GBoxed', 'GEnum', 'GError', 'GFlags', 'GInterface', 'GObject', 'GObjectMeta',
...
>>> import cairo
>>> dir(cairo)
['ANTIALIAS_DEFAULT', 'ANTIALIAS_GRAY', 'ANTIALIAS_NONE', 'ANTIALIAS_SUBPIXEL',
>>> import pygtk
>>> dir(pygtk)
['__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_
get_available_versions', '_our_dir', '_pygtk_2_0_dir', '_pygtk_dir_pat', '_pygtk
_required_version', 'fnmatch', 'glob', 'os', 'require', 'require20', 'sys']


As you can see this python modules works well.


December 10, 2016 07:16 AM


Mike C. Fletcher

PyCon.ca talk on Mistakes in Life/AI is up

December 10, 2016 04:56 AM


Michał Bultrowicz

My talk about TDD of microservices

This year at EuroPython (and not only there), I gave a talk about Test Driven Development of Python microservices. I guess you can check it out, if you’re into that kind of stuff. It’s also available in Polish.

December 10, 2016 12:00 AM

December 09, 2016


Calvin Spealman

Let's Make A Game! Watch me make "Off And On Again"

Let's Make A Game! I've been making games to one degree or another for a few years, and I've really gotten into a rhythm in the last few months. This month I'm trying a new experiment and posting the entire process of building the month's game development project. I'll be publishing a stream of every single step I take in the process, broken down into 25 one hour episodes.

I'm doing this for a few reasons.

Firstly, I'm doing this for myself. The one hour sessions and the focus recording gives me on the task at hand are helpful. I think this is an interesting way to go about a project. I'd also like the strict schedule to help me study and learn more about the aspects of budgeting and scoping game projects that I have too little experience to be very good at.

Secondly, I'm hoping someone out there gets some value out of this. My skill level as a game developer is still pretty low, but maybe its helpful for someone to see how someone learns to solve problems, rather than only watching experts that already know the answers.

Thirdly, I want to see if I like the idea and to figure out if I'll keep doing it. Maybe this will be the only time I do this. Maybe I'll scale back the number of hours and continue it on a monthly basis or just occasionally, in the future. Or, also likely, this may be a one time thing.

If you're interested, check out the Youtube channel for my one-man game studio, Stonebird Games.

You can watch the first three episodes right here!






December 09, 2016 11:20 PM


Semaphore Community

Dockerizing a Python Django Web Application

This article is brought with ❤ to you by Semaphore.

Introduction

This article will cover building a simple 'Hello World'-style web application written in Django and running it in the much talked about and discussed Docker. Docker takes all the great aspects of a traditional virtual machine, e.g. a self contained system isolated from your development machine, and removes many of the drawbacks such as system resource drain, setup time, and maintenance.

When building web applications, you have probably reached a point where you want to run your application in a fashion that is closer to your production environment. Docker allows you to set up your application runtime in such a way that it runs in exactly the same manner as it will in production, on the same operating system, with the same environment variables, and any other configuration and setup you require.

By the end of the article you'll be able to:

What is Docker, Anyway?

Docker's homepage describes Docker as follows:

"Docker is an open platform for building, shipping and running distributed applications. It gives programmers, development teams, and operations engineers the common toolbox they need to take advantage of the distributed and networked nature of modern applications."

Put simply, Docker gives you the ability to run your applications within a controlled environment, known as a container, built according to the instructions you define. A container leverages your machines resources much like a traditional virtual machine (VM). However, containers differ greatly from traditional virtual machines in terms of system resources. Traditional virtual machines operate using Hypervisors, which manage the virtualization of the underlying hardware to the VM. This means they are large in terms of system requirements.

Containers operate on a shared Linux operating system base and add simple instructions on top to execute and run your application or process. The difference being that Docker doesn't require the often time-consuming process of installing an entire OS to a virtual machine such as VirtualBox or VMWare. Once Docker is installed, you create a container with a few commands and then execute your applications on it via the Dockerfile. Docker manages the majority of the operating system virtualization for you, so you can get on with writing applications and shipping them as you require in the container you have built. Furthermore, Dockerfiles can be shared for others to build containers and extend the instructions within them by basing their container image on top of an existing one. The containers are also highly portable and will run in the same manner regardless of the host OS they are executed on. Portability is a massive plus side of Docker.

Prerequisites

Before you begin this tutorial, ensure the following is installed to your system:

Setting Up a Django web application

Starting a Django application is easy, as the Django dependency provides you with a command line tool for starting a project and generating some of the files and directory structure for you. To start, create a new folder that will house the Django application and move into that directory.

$ mkdir project
$ cd project

Once in this folder, you need to add the standard Python project dependencies file which is usually named requirements.txt, and add the Django and Gunicorn dependency to it. Gunicorn is a production standard web server, which will be used later in the article. Once you have created and added the dependencies, the file should look like this:

$ cat requirements.txt                                                              
Django==1.9.4
gunicorn==19.6.0

With the Django dependency added, you can then install Django using the following command:

$ pip install -r requirements.txt

Once installed, you will find that you now have access to the django-admin command line tool, which you can use to generate the project files and directory structure needed for the simple "Hello, World!" application.

$ django-admin startproject helloworld

Let's take a look at the project structure the tool has just created for you:

.
├── helloworld
│   ├── helloworld
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   └── manage.py
└── requirements.txt

You can read more about the structure of Django on the official website. django-admin tool has created a skeleton application. You control the application for development purposes using the manage.py file, which allows you to start the development test web server for example:

$ cd helloworld
$ python manage.py runserver

The other key file of note is the urls.py, which specifies what URL's route to which view. Right now, you will only have the default admin URL which we won't be using in this tutorial. Lets add a URL that will route to a view returning the classic phrase "Hello, World!".

First, create a new file called views.py in the same directory as urls.py with the following content:

from django.http import HttpResponse

def index(request):
    return HttpResponse("Hello, world!")

Now, add the following URL url(r'', 'helloworld.views.index') to the urls.py, which will route the base URL of / to our new view. The contents of the urls.py file should now look as follows:

from django.conf.urls import url
from django.contrib import admin

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'', 'helloworld.views.index'),
]

Now, when you execute the python manage.py runserver command and visit http://localhost:8000 in your browser, you should see the newly added "Hello, World!" view.

The final part of our project setup is making use of the Gunicorn web server. This web server is robust and built to handle production levels of traffic, whereas the included development server of Django is more for testing purposes on your local machine only. Once you have dockerized the application, you will want to start up the server using Gunicorn. This is much simpler if you write a small startup script for Docker to execute. With that in mind, let's add a start.sh bash script to the root of the project, that will start our application using Gunicorn.

#!/bin/bash

# Start Gunicorn processes
echo Starting Gunicorn.
exec gunicorn helloworld.wsgi:application \
    --bind 0.0.0.0:8000 \
    --workers 3

The first part of the script writes "Starting Gunicorn" to the command line to show us that it is starting execution. The next part of the script actually launches Gunicorn. You use exec here so that the execution of the command takes over the shell script, meaning that when the Gunicorn process ends so will the script, which is what we want here.

You then pass the gunicorn command with the first argument of helloworld.wsgi:application. This is a reference to the wsgi file Django generated for us and is a Web Server Gateway Interface file which is the Python standard for web applications and servers. Without delving too much into WSGI, the file simply defines the application variable, and Gunicorn knows how to interact with the object to start the web server.

You then pass two flags to the command, bind to attach the running server to port 8000, which you will use to communicate with the running web server via HTTP. Finally, you specify workers which are the number of threads that will handle the requests coming into your application. Gunicorn recommends this value to be set at (2 x $num_cores) + 1. You can read more on configuration of Gunicorn in their documentation.

Finally, make the script executable, and then test if it works by changing directory into the project folder helloworld and executing the script as shown here. If everything is working fine, you should see similar output to the one below, be able to visit http://localhost:8000 in your browser, and get the "Hello, World!" response.

$ chmod +x start.sh
$ cd helloworld
$ ../start.sh
Starting Gunicorn.
[2016-06-26 19:43:28 +0100] [82248] [INFO]
Starting gunicorn 19.6.0
[2016-06-26 19:43:28 +0100] [82248] [INFO]
Listening at: http://0.0.0.0:8000 (82248)
[2016-06-26 19:43:28 +0100] [82248] [INFO]
Using worker: sync
[2016-06-26 19:43:28 +0100] [82251] [INFO]
Booting worker with pid: 82251
[2016-06-26 19:43:28 +0100] [82252] [INFO]
Booting worker with pid: 82252
[2016-06-26 19:43:29 +0100] [82253] [INFO]
Booting worker with pid: 82253

Dockerizing the Application

You now have a simple web application that is ready to be deployed. So far, you have been using the built-in development web server that Django ships with the web framework it provides. It's time to set up the project to run the application in Docker using a more robust web server that is built to handle production levels of traffic.

Installing Docker

One of the key goals of Docker is portability, and as such is able to be installed on a wide variety of operating systems.

For this tutorial, you will look at installing Docker Machine on MacOS. The simplest way to achieve this is via the Homebrew package manager. Instal Homebrew and run the following:

$ brew update && brew upgrade --all && brew cleanup && brew prune
$ brew install docker-machine

With Docker Machine installed, you can use it to create some virtual machines and run Docker clients. You can run docker-machine from your command line to see what options you have available. You'll notice that the general idea of docker-machine is to give you tools to create and manage Docker clients. This means you can easily spin up a virtual machine and use that to run whatever Docker containers you want or need on it.

You will now create a virtual machine based on VirtualBox that will be used to execute your Dockerfile, which you will create shortly. The machine you create here should try to mimic the machine you intend to run your application on in production. This way, you should not see any differences or quirks in your running application neither locally nor in a deployed environment.

Create your Docker Machine using the following command:

$ docker-machine create development --driver virtualbox
--virtualbox-disk-size "5000" --virtualbox-cpu-count 2
--virtualbox-memory "4096"

This will create your machine and output useful information on completion. The machine will be created with 5GB hard disk, 2 CPU's and 4GB of RAM.

To complete the setup, you need to add some environment variables to your terminal session to allow the Docker command to connect the machine you have just created. Handily, docker-machine provides a simple way to generate the environment variables and add them to your session:

$ docker-machine env development
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://123.456.78.910:1112"
export DOCKER_CERT_PATH="/Users/me/.docker/machine/machines/development"
export DOCKER_MACHINE_NAME="development"
# Run this command to configure your shell:
# eval "$(docker-machine env development)"

Complete the setup by executing the command at the end of the output:

$(docker-machine env development)

Execute the following command to ensure everything is working as expected.

$ docker images
REPOSITORY   TAG   IMAGE  ID   CREATED   SIZE

You can now dockerize your Python application and get it running using the docker-machine.

Writing the Dockerfile

The next stage is to add a Dockerfile to your project. This will allow Docker to build the image it will execute on the Docker Machine you just created. Writing a Dockerfile is rather straightforward and has many elements that can be reused and/or found on the web. Docker provides a lot of the functions that you will require to build your image. If you need to do something more custom on your project, Dockerfiles are flexible enough for you to do so.

The structure of a Dockerfile can be considered a series of instructions on how to build your container/image. For example, the vast majority of Dockerfiles will begin by referencing a base image provided by Docker. Typically, this will be a plain vanilla image of the latest Ubuntu release or other Linux OS of choice. From there, you can set up directory structures, environment variables, download dependencies, and many other standard system tasks before finally executing the process which will run your web application.

Start the Dockerfile by creating an empty file named Dockerfile in the root of your project. Then, add the first line to the Dockerfile that instructs which base image to build upon. You can create your own base image and use that for your containers, which can be beneficial in a department with many teams wanting to deploy their applications in the same way.

# Dockerfile

# FROM directive instructing base image to build upon
FROM python:2-onbuild

It's worth noting that we are using a base image that has been created specifically to handle Python 2.X applications and a set of instructions that will run automatically before the rest of your Dockerfile. This base image will copy your project to /usr/src/app, copy your requirements.txt and execute pip install against it. With these tasks taken care of for you, your Dockerfile can then prepare to actually run your application.

Next, you can copy the start.sh script written earlier to a path that will be available to you in the container to be executed later in the Dockerfile to start your server.

# COPY startup script into known file location in container
COPY start.sh /start.sh

Your server will run on port 8000. Therefore, your container must be set up to allow access to this port so that you can communicate to your running server over HTTP. To do this, use the EXPOSE directive to make the port available:

# EXPOSE port 8000 to allow communication to/from server
EXPOSE 8000

The final part of your Dockerfile is to execute the start script added earlier, which will leave your web server running on port 8000 waiting to take requests over HTTP. You can execute this script using the CMD directive.

# CMD specifcies the command to execute to start the server running.
CMD ["/start.sh"]
# done!

With all this in place, your final Dockerfile should look something like this:

# Dockerfile

# FROM directive instructing base image to build upon
FROM python:2-onbuild

# COPY startup script into known file location in container
COPY start.sh /start.sh

# EXPOSE port 8000 to allow communication to/from server
EXPOSE 8000

# CMD specifcies the command to execute to start the server running.
CMD ["/start.sh"]
# done!

You are now ready to build the container image, and then run it to see it all working together.

Building and Running the Container

Building the container is very straight forward once you have Docker and Docker Machine on your system. The following command will look for your Dockerfile and download all the necessary layers required to get your container image running. Afterwards, it will run the instructions in the Dockerfile and leave you with a container that is ready to start.

To build your container, you will use the docker build command and provide a tag or a name for the container, so you can reference it later when you want to run it. The final part of the command tells Docker which directory to build from.

$ cd <project root directory>
$ docker build -t davidsale/dockerizing-python-django-app .

Sending build context to Docker daemon 237.6 kB
Step 1 : FROM python:2-onbuild
# Executing 3 build triggers...
Step 1 : COPY requirements.txt /usr/src/app/
 ---> Using cache
Step 1 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Using cache
Step 1 : COPY . /usr/src/app
 ---> 68be8680cbc4
Removing intermediate container 75ed646abcb6
Step 2 : COPY start.sh /start.sh
 ---> 9ef8e82c8897
Removing intermediate container fa73f966fcad
Step 3 : EXPOSE 8000
 ---> Running in 14c752364595
 ---> 967396108654
Removing intermediate container 14c752364595
Step 4 : WORKDIR helloworld
 ---> Running in 09aabb677b40
 ---> 5d714ceea5af
Removing intermediate container 09aabb677b40
Step 5 : CMD /start.sh
 ---> Running in 7f73e5127cbe
 ---> 420a16e0260f
Removing intermediate container 7f73e5127cbe
Successfully built 420a16e0260f

In the output, you can see Docker processing each one of your commands before outputting that the build of the container is complete. It will give you a unique ID for the container, which can also be used in commands alongside the tag.

The final step is to run the container you have just built using Docker:

$ docker run -it -p 8000:8000 davidsale/djangoapp1
Starting Gunicorn.
[2016-06-26 19:24:11 +0000] [1] [INFO]
Starting gunicorn 19.6.0
[2016-06-26 19:24:11 +0000] [1] [INFO]
Listening at: http://0.0.0.0:9077 (1)
[2016-06-26 19:24:11 +0000] [1] [INFO]
Using worker: sync
[2016-06-26 19:24:11 +0000] [11] [INFO]
Booting worker with pid: 11
[2016-06-26 19:24:11 +0000] [12] [INFO]
Booting worker with pid: 12
[2016-06-26 19:24:11 +0000] [17] [INFO]
Booting worker with pid: 17

The command tells Docker to run the container and forward the exposed port 8000 to port 8000 on your local machine. After you run this command, you should be able to visit http://localhost:8000 in your browser to see the "Hello, World!" response. If you were running on a Linux machine, that would be the case. However, if running on MacOS, then you will need to forward the ports from VirtualBox, which is the driver we use in this tutorial so that they are accessible on your host machine.

$ VBoxManage controlvm "development" natpf1
  "tcp-port8000,tcp,,8000,,8000";

This command modifies the configuration of the virtual machine created using docker-machine earlier to forward port 8000 to your host machine. You can run this command multiple times changing the values for any other ports you require.

Once you have done this, visit http://localhost:8000 in your browser. You should be able to visit your dockerized Python Django application running on a Gunicorn web server, ready to take thousands of requests a second and ready to be deployed on virtually any OS on planet using Docker.

Next Steps

After manually verifying that the appication is behaving as expected in Docker, the next step is the deployment. You can use Semaphore's Docker platform for automating this process.

Conclusion

In this tutorial, you have learned how to build a simple Python Django web application, wrap it in a production grade web server, and created a Docker container to execute your web server process.

If you enjoyed working through this article, feel free to share it and if you have any questions or comments leave them in the section below. We will do our best to answer them, or point you in the right direction.

This article is brought with ❤ to you by Semaphore.

December 09, 2016 06:36 PM


Weekly Python StackOverflow Report

(xlix) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2016-12-09 17:47:28 GMT


  1. Is it possible to break a long function name across multiple lines? - [49/7]
  2. What are the names of the magic methods for the operators "is" and "in"? - [15/3]
  3. Why does Python 2 allow comparisons between lists and numbers? - [9/3]
  4. Why modifying dict during iteration doesn't always raise exception? - [9/3]
  5. Why is print("text" + str(var1) + "more text" + str(var2)) described as "disapproved"? - [9/2]
  6. Why can Linux accept sockets in multiprocessing? - [9/1]
  7. why is len so much more efficient on DataFrame than on underlying numpy array? - [8/2]
  8. Why is 'new_file += line + string' so much faster than 'new_file = new_file + line + string'? - [7/2]
  9. Python - Create Counter() from mapping, non-integer values - [7/1]
  10. Fonts Corrupted - [7/1]

December 09, 2016 05:48 PM


Graham Dumpleton

Backdoors for becoming root in a Docker container.

In my last post, the main issue I looked at was whether you can trust what a Docker-formatted image says about the user it will run as. What we found was that if the ‘USER’ statement is used in a Dockefile, but is set to a name, you have no idea what UNIX user ID the application in the container will run as. This is because the name could be mapped to any user ID by the UNIX passwd file. Setting

December 09, 2016 04:34 PM


Python Software Foundation

"Kurt Doesn't Drop The Ball": Thanks For Kurt Kaiser's 10 Years As PSF Treasurer

"We were all amateurs," recalls Guido van Rossum. In the early years of the Python Software Foundation, its founders were more interested in writing code than running a nonprofit. Guido says they only loosely tracked who had access to the bank account and who was responsible for filing paperwork on time. There were some close calls, and the foundation nearly missed some deadlines. That changed when Kurt Kaiser became treasurer, ten years ago. "Kurt straightened all that out."

When Kaiser took over the job, the PSF was a small foundation giving out a few thousand dollars a year in grants. The accounting was handed over to him without much explanation or support; he learned the system and grew it to what it is today. The PSF now awards dozens of grants a year totalling over a quarter million dollars. The foundation accepts money from PSF sponsors, PyCon sponsors, members, and donors; it provides financial aid to hundreds of conference attendees, covers Meetup.com fees, and acts as a fiscal sponsor to groups like PyLadies, lending its nonprofit status to these smaller organizations. Kaiser manages all these transactions. When processes become routine, he automates them with code. Board member Diana Clarke says, "He's more than just your typical treasurer; he's a developer plus treasurer."

Kurt Kaiser was a physicist throughout his career, and picked up Python as a hobby. His first great contribution was the code he wrote for IDLE, the development GUI that comes bundled with Python. At first he was a user and contributor to the project, but that changed in the wake of conflicts among its maintainers. "After the combatants of the IDLE war withdrew," says Guido, "Kurt became a lead developer. He had a much smoother personality." When the PSF needed a new treasurer, Kaiser was entrusted with the role. "He just got so into the cause," says Director of Operations Ewa Jodlowska. "He believed he was helping the greater good by helping the PSF."

This year alone, through the end of October, the PSF has awarded $260,000 to more than 50 entities. Kaiser administers this swelling stream of grants. The work is not all routine: a recent grant to PyCon CZ, for example, required special handling, because Czech bank account IDs have a different format than American accounts. Some grants pose extraordinary hurdles, like the PSF's support for a PyCon in Zimbabwe. Kaiser researched the United States' financial embargo of Zimbabwe, and the PSF concluded it could not sponsor the conference, but it could still lend support by sending a keynote speaker.

Each year, a new Board of Directors brings new ideas for funding Python advocacy, but these ideas can raise knotty questions about financial rules. According to Diana Clarke, it is tempting to balk at these complexities and simply say no, but Kaiser is always willing to research new options that meet the community's changing needs.

The PSF's mission is to advance the Python language, and increasingly it uses Python as a vehicle for promoting access to computer science for women, minorities, and members of underserved groups. For example, the PSF collects money on behalf of PyLadies, which allows them to operate under the PSF's nonprofit umbrella without going through all the paperwork. Clarke says, "The PyLadies auction raises money and then turns that around in the form of scholarships, and Kurt is the one who enables that. It's a nontrivial amount of work and it's very important."

The PyCon Financial Aid program is another way the PSF spends money to promote access to computer science. Guido van Rossum says, "What better use of that money, than to pay for people to come to the conference who couldn't otherwise come, people who are a good investment for the community?" The PSF awards $100,000 in scholarships to help people come to PyCon from all over the world. Kaiser administers these scholarships one-on-one: he disburses money, collects receipts, and receives conference write-ups from recipients.

When Diana Clarke worked with Kaiser at PyCon, she was struck at his singular role in the conference. He has to carry a briefcase of cash, and he manually writes hundreds of checks to ensure attendees receive their awards. Instead of being free to attend talks, Kaiser often sits in the hallway to be available when scholarship recipients need to pick up their money.

Kurt Kaiser is among the PSF's longest-serving officers. As Diana Clarke says, "He does so much for us, and most people don't have a lot of insight into what he does." On his tenth anniversary as treasurer, the PSF congratulates him and thanks him. From its early years as a small foundation through today, Kaiser has been a consistent and reliable steward of the organization's finances.  "For a long time Kurt was the only stable point," says Guido. "We know that Kurt doesn't drop the ball."


From left: Kurt Kaiser, Ewa Jodlowska, Meagan Sneeringer, Doug Napoleone, 

Hannes Hapke at The Multnomah Whiskey Library in Portland OR, PyCon 2016


December 09, 2016 09:00 AM


Jeff Knupp

How Python Linters Will Save Your Large Python Project

A Python project I'm working on at Enigma is starting to grow rather large. I spent a good deal of effort yesterday getting a five line change added to our Makefile (which is run as part of our CI and CD pipeline on every pull request and merge). After the PR was merged, I gloated to others how awesome my team's project was. Here are the five lines:

lint:
    pylint --rcfile=.pylintrc api -f parseable -r n && \
    mypy --silent-imports api && \
    pycodestyle api --max-line-length=120 && \
    pydocstyle api

For completeness sake, here is the test target that actually gets run as part of CI tests:

test: install lint
    python3 setup.py test --addopts="--cov=api"
    coverage xml -i -o coverage/cobertura-coverage.xml --omit=$(VIRTUAL_ENV)/*,.eggs/*

(So, for those not versed in "Makefile-ese", the test target won't run successfully if the lint target doesn't do so first).

Why was I happy enough to be a jerk to other engineers and brag about those five lines? It all comes down to complexity.

As Python projects grow, maintenance becomes a nightmare (I'm more referring to enterprise-sized projects rather than "large" personal projects, but the same idea holds). Code becomes disorganized, messy, reflects the style of the author (even for teams doing a decent job enforcing PEP-8 and PEP-257, and docstrings fall by the wayside. It takes new developers longer and longer to ramp up on the project. Simple fixes and feature changes become not so simple when it comes time to actually make them. All of these are due to the necessary increase in complexity as a project grows. So how to we reduce that complexity as much as possible?

No Panacea

Obviously, there's no one answer here, but there's a set of actions you can take right now which requires low overhead and pays huge dividends:

Make code "linters" part of your build process. If any of them report any issues, fail your build.

So what is a code "linter"?

A code "linter" is a tool that performs static analysis on your code and reports coding style violations, possible design flaws, outright bugs, and code patterns known to be dangerous. Static analysis means that your code is analyzed but not executed, so there are no external dependencies and running a linter will never harm your environment. You can think of them as a poor-man's AI code reviewer.

Linters have various output formats and modes they can be run in (and vary in complexity and usefulness). Here's example output from pylint, one of the most popular Python linters:

$ pylint sandman2
************* Module sandman2.app
C:117, 0: Line too long (81/80) (line-too-long)
C:163, 0: Line too long (88/80) (line-too-long)
C: 24, 0: Invalid constant name "auth" (invalid-name)
R: 27, 0: Too many arguments (6/5) (too-many-arguments)
************* Module sandman2.decorators
W: 42,24: Duplicate key 'status' in dictionary (duplicate-key)
************* Module sandman2.model
C: 49, 0: Line too long (89/80) (line-too-long)
C: 50, 0: Line too long (113/80) (line-too-long)
C:141, 0: Invalid constant name "DeclarativeModel" (invalid-name)
C: 10, 0: Imports from package sqlalchemy are not grouped (ungrouped-imports)
************* Module sandman2.service
C: 98, 0: Trailing whitespace (trailing-whitespace)
C:208, 0: Line too long (87/80) (line-too-long)
C:215, 0: Line too long (93/80) (line-too-long)
C:224, 0: Line too long (82/80) (line-too-long)
R:231, 4: Method could be a function (no-self-use)
************* Module sandman2.scripts.sandman2ctl
C: 40, 0: Line too long (98/80) (line-too-long)


Report
======
364 statements analysed.

Statistics by type
------------------

+---------+-------+-----------+-----------+------------+---------+
|type     |number |old number |difference |%documented |%badname |
+=========+=======+===========+===========+============+=========+
|module   |9      |9          |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+
|class    |12     |12         |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+
|method   |21     |21         |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+
|function |18     |18         |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+



External dependencies
---------------------
::

    flask (sandman2.app,sandman2.decorators,sandman2.service)
      \-views (sandman2.service)
    flask_sqlalchemy (sandman2.model)
    sandman2 (sandman2.scripts.sandman2ctl)
      \-admin (sandman2.app)
      \-app (sandman2)
      \-decorators (sandman2.service)
      \-exception (sandman2.app,sandman2.decorators,sandman2.service)
      \-model (sandman2.app,sandman2.service)
      \-service (sandman2.app)
    sqlalchemy
      \-ext
      | \-automap (sandman2.model)
      | \-declarative (sandman2.model)
      \-inspection (sandman2.model)
      \-sql
        \-sqltypes (sandman2.app)



Raw metrics
-----------

+----------+-------+------+---------+-----------+
|type      |number |%     |previous |difference |
+==========+=======+======+=========+===========+
|code      |459    |54.38 |459      |=          |
+----------+-------+------+---------+-----------+
|docstring |243    |28.79 |243      |=          |
+----------+-------+------+---------+-----------+
|comment   |20     |2.37  |22       |-2.00      |
+----------+-------+------+---------+-----------+
|empty     |122    |14.45 |118      |+4.00      |
+----------+-------+------+---------+-----------+



Duplication
-----------

+-------------------------+------+---------+-----------+
|                         |now   |previous |difference |
+=========================+======+=========+===========+
|nb duplicated lines      |0     |0        |=          |
+-------------------------+------+---------+-----------+
|percent duplicated lines |0.000 |0.000    |=          |
+-------------------------+------+---------+-----------+



Messages by category
--------------------

+-----------+-------+---------+-----------+
|type       |number |previous |difference |
+===========+=======+=========+===========+
|convention |12     |13       |-1.00      |
+-----------+-------+---------+-----------+
|refactor   |2      |2        |=          |
+-----------+-------+---------+-----------+
|warning    |1      |3        |-2.00      |
+-----------+-------+---------+-----------+
|error      |0      |0        |=          |
+-----------+-------+---------+-----------+



% errors / warnings by module
-----------------------------

+-----------------------------+------+--------+---------+-----------+
|module                       |error |warning |refactor |convention |
+=============================+======+========+=========+===========+
|sandman2.decorators          |0.00  |100.00  |0.00     |0.00       |
+-----------------------------+------+--------+---------+-----------+
|sandman2.service             |0.00  |0.00    |50.00    |33.33      |
+-----------------------------+------+--------+---------+-----------+
|sandman2.app                 |0.00  |0.00    |50.00    |25.00      |
+-----------------------------+------+--------+---------+-----------+
|sandman2.model               |0.00  |0.00    |0.00     |33.33      |
+-----------------------------+------+--------+---------+-----------+
|sandman2.scripts.sandman2ctl |0.00  |0.00    |0.00     |8.33       |
+-----------------------------+------+--------+---------+-----------+



Messages
--------

+--------------------+------------+
|message id          |occurrences |
+====================+============+
|line-too-long       |8           |
+--------------------+------------+
|invalid-name        |2           |
+--------------------+------------+
|ungrouped-imports   |1           |
+--------------------+------------+
|trailing-whitespace |1           |
+--------------------+------------+
|too-many-arguments  |1           |
+--------------------+------------+
|no-self-use         |1           |
+--------------------+------------+
|duplicate-key       |1           |
+--------------------+------------+



Global evaluation
-----------------
Your code has been rated at 9.59/10 (previous run: 9.50/10, +0.08)

Of course, this is super verbose, and is definitely the most comprehensive output of any of the linters I've used, but it gives a good sense of what they look for and what they're capable of. pylint provides a helpful (if somewhat arbitrary) "score" at the end of the output which is saved between runs, so you know if your code is getting better or worse by pylint standards.

So How Does This Relate To Continuous Integration?

In their most basic mode, most linters will print out all violations to STDOUT and return a non-zero exit code if any violations were encountered. That means, if everything went perfectly, the command just runs to success with no output. If you take a look at my Makefile change at the top of the article, that's why we can chain together calls to successive linters and the first one to report a violation will abort the entire set of commands.

By hooking this up to your CI platform, you can enforce good style, catch bugs, and actually require proper documentation for everything (and I do mean everything) at build time. Someone doesn't follow PEP-8? Build fails. Sorry, your PR can't be merged. Didn't write a docstring for that package/module/class/function? Sorry, build failed. Wrote a function that takes 14 arguments and named it foo? You guessed it... Build failed.

This is huge for a number of reasons:

I'm Convinced! Let's Lint This Puppy!

Python (like most languages) has a number of linters available, all of which do similar (but not necessarily exactly the same) things. To confuse matters, everyone thinks it's a great idea every once in a while to write a tool that just packages all the linters that are currently en vogue and fires all of them at your code like a fire hose. But this happens every few years, so you get meta-linters made of meta-linters, until who knows what tool is actually looking at your code (and if it's still maintained).

So I'm going to be "that guy" and just tell you what linters you should run. The following list was chosen very carefully, but I'll get into linter holy wars in the comments if anyone is game.

The linters you should use are:

And that's it. I'll briefly cover each one below.

pylint

pylint has been around forever and is almost certainly the most comprehensive Python linter available. Some complain that it's too picky or too verbose, but that's what config files are for! One of the lesser-known (awesome) features of pylint is the .pylintrc file, which pylint will even create a skeleton with sane defaults for you if you run

$ pylint --generate-rcfile

I still marvel that more CLI tools don't do this. It's simply awesome.

You can greatly reduce pylint's verbosity by running with the -r n flag to suppress that giant report.

pycodestyle and pydocstyle

Both of these linters used to have different names: the former was called pep8 and the latter pep257 (after the PEPs they check conformance against). In a somewhat hilarious GitHub issue, GvR himself asked them to kindly change the names of the tools to avoid people hating on PEP-8 just because the tool of the same name was bugging them. Regardless, these tools are comprehensive and easy to get started with. Just pip install them and then run $ pycodestyle <some-module-or-package-or-file> and find out everything you've been doing wrong all this time!

As one might expect, pycodestyle handles PEP-8 (the official Style Guide for Python Code) conformance while pydocstyle handles PEP-257 (the official Docstring Conventions for Python). The latter is awesomely strict. When and if you get your project to pass it without error (took me a day on the project in question) you'll never have to worry about someone forgetting to write a docstring (or forgetting how to write a docstring) again. It even checks for proper capitalization, punctuation, and prescriptive vs. descriptive comments. I am in love with its pedantry.

Automate All The Things

In all seriousness, Enigma is a startup. We have limited resources. If we want to keep a large project sane, the only way to do that is through automation (because we're too busy building awesome stuff with public data). I would go so far as to argue that, between our testing suite (unit tests and E2E browser tests), continuous integration pipeline (Jenkins + GitHub + Codecov + a lot of homegrown glue), our continuous deployment pipeline (10% Jenkins/GitHub, 90% Ansible), and tools like the linters mentioned above, we are saving the cost of two full-time engineers to manage our development and deployment processes while making our code base more readable, more testable, and more maintainable.

All of that from a couple of automated tools you can install in the next five minutes... What are you waiting for!?

December 09, 2016 08:53 AM


Kushal Das

10 years of dgplug summer training

In 2017 dgplug summer training will be happening for the 10th time. Let me tell you honestly, I had no clue that we will reach here when I started this back in 2008. The community gathered together, and we somehow managed to continue. In case, you do not know about this training, dgplug summer training is a 3 months long online IRC based course where we help people to become contributors to upstream projects. The sessions start around 6:30PM IST, and continues till 9PM (or sometimes till very late at night) for 3 months. You can read the logs of the sessions here.

The beginning

I remember looking at how people were trying to apply for Google Summer Code back in 2008, but many of them never had minimum experience with software development in Open Source world. dgplug started in 2004, and I think our motto “Learn yourself, teach others” is still as relevant now, as it was back in 2004. I was thinking about one online training where we can teach the basics of using Linux, and also basics of programming in Linux. How to create, and submit patches etc. I chose to use IRC as the place of the training because of bandwidth issue. Looking at how we conducted regular Fedora meetings on IRC, I was sure it would be doable. Of course there were many self doubts, too many open questions for which I never had any answer. When I shared this idea in different mailing lists, and asked for volunteers, Shakthi Kannan (irc: mbuf) also jumped in to mentor in the training. This gave me some hope. We had around 10+ participants in the first year of this training. Mailing lists guidelines, and how to communicate online were important from the beginning. Python was my primary language back then, and we choose to use that in the training too. With help from Prasad J. Pandit (irc: pjp) we had sessions on programming using C. But we found out that debugging segfaults over IRC was just not possible. We can try to do that if there were 2 or 3 people, but for a large group it was just not possible. As I mentioned earlier, contribution is the keyword for this training, we had other sessions than programming from the very first year. Máirín Duffy took a session on Fedora Arts, Paul Frields (then Fedora Project leader) took a session on Documentation. Debarshi took the sessions on shell scripting, and few of our sessions went up to 2AM (starting at 6pm).

Continuing & experimenting

Many participants from the 2008 session were regulars in the IRC channel. A few of them were already contributing in upstream projects. We decided to continue with the training. The number of participants grew from 10 to 170+ (in 2016). We did various kinds of experiments in between these years. Many of our ideas failed. Many ideas have gone through iterations until we found what works better. Below are few ideas we found difficult to implement in a training like this:

Lessons learned (the good points)

Start from scratch

Every year we start from scratch. There are always many participants who does not know much of command line tools, or how to reply inline to a mail in the mailing list. This is why we start from communication tools, basic shell commands, and using editors (we teach Vim). From there the sessions slowly start becoming more intense.

IRC works well on low bandwidth

IRC is still the best medium for a training where people are joining in from all over the world. It works in 2G connection too. Breaking the first entry barrier is difficult, but once we breach that, people generally stays back on the channels.

Stay in touch

Stay in touch with the participants even after the course is finished. Most of the regular participants in the #dgplug channel are actual ex-participants of the summer training. Some were trainees, some were trainers. Talking about other things in life is important to increase the participation in IRC.

Have documentation

We were slow in this point. Even though from the first year we had logs of each session, we never had one single place for the participants to learn from. Now we maintain a Sphinx docs. It is always nice to be able to point people to one link from where they can start.

Meet in real world

Every year we meet during PyCon India. This meeting is known as staircase meeting as we used to block most of the staircase at the venue in Bangalore. Meeting people face to face, talking to them in real life, sharing food & thoughts help to break many hidden barriers. It is true that not all of us can meet in one place, but we try to meet at least once every year. In 2017 we will be meeting during PyCon Pune in February in Pune, then once again in PyCon India in Delhi.

Give them hope, give them examples

Other than our regular sessions, we also get sessions from other known upstream contributors in various projects. They talk about how they have started their journey in Open Source world. Sometimes they sessions on their field. We had sessions from documentation, art work, experiences of past participants, Creative Commons licenses, leaders from various communities. This year Python creator Guido Van Rossum also joined in, and took a very interactive session.

Automate as much as possible

We now have a bot which helps the trainer during the sessions. This helps to reduce chaos during the sessions. As we all know the nature of IRC, it is very common that 10 people can try to talk at the same time. Stopping that was a big win for us. We still do not upload the logs automatically, as we go through the logs once manually before uploading.

Have a group of volunteers whom the community can trust

Most of the volunteers of dgplug are ex-participants of this summer training. They stayed back in the channel after their training is over. They help people almost 247 (depending their availability). There are the people whom we all trust. They become leaders by their own effort, no one comes and announces new leaders. Delegation of various work is important, getting fresh blood in the volunteer pool is very important. We constantly look for new volunteers. But at the same time you will need some people who can come in at the time of any emergency (with authority).

Enforce some rules

Rules are important for any community, call it Code of Conduct, or something else. But without these maintained rules, any community can get into chaos. Enforce these rules. We generally force people to write in full English words all the time (instead of sms language). The participants do pull legs of anyone who types u or r in the IRC channel. We also try to push people for inline reply than top posting in the mailing list.

Solve real life problem

Do not try to stay in only examples from books while teaching something. Always try to solve problems which are closer to the hearts of the participants. These solutions will give them much more encouragement than anything else. It can as small as finding photos in their computer, or building a full scale application to manage one’s video pool. We try to use Python to solve many of these issues, as we have experienced Python programmers in the channel . Python3 is our default language to do anything on the sessions.

Outcome of the training

I think we are the highest concentration of upstream contributors in one community in India. We had participants from Singapore, South Korea, to Afghanistan, Pakistan, to Nigeria (a big shoutout to Tosin), to many parts in US and Canada. We had school teachers, college students, lawyers, sys-admins, housewives, school students, music students, participants from various backgrounds in the last 10 years. We have companies personally inquiring about ex-participants before hiring them. Many of the participants are now my colleague in Red Hat. We have people in the channel who are willing to help anyone, everyone learns from everyone else. People are part many other communities, but #dgplug binds us all.

December 09, 2016 06:03 AM


tryexceptpass

You’re correct in that I generalized too much here, I shouldn’t have used the word parallelism…

December 09, 2016 03:54 AM

Woops… Fixed! Thanks for pointing it out.

December 09, 2016 03:44 AM

December 08, 2016


Python Data

Vagrant on Windows

There are many different ways to install python and work with python on Windows. You can install Canopy or Anaconda to have an entire python ecosystem self-contained or you can install python directly onto your machine and configure all the bits and bytes yourself. My current recommendation is to use Vagrant on Windows combined with Virtualbox to virtualize your development environment.

While I use a mac or the majority of my development, I do find myself using Windows 10 more and more, and may be moving to a Windows machine in the future for my day-to-day laptop.  I have and do use Canopy and/or Anaconda but I’ve recently moved the majority of my python development on Windows into Linux (Ubuntu) virtual machines using Vagrant and Virtualbox. You can use other products like VMWare’s virtual machine platform, but Virtualbox is free and does a good enough job for day-to-day development.1

One Caveat: if you’re doing processor / memory intensive development with python, this may not be the best option for you. That said, it can work for those types of development efforts if you configure your virtual machine with enough RAM and processors.

To get started, you’ll need to download and install Vagrant and Virtualbox for your machine.   I am using Vagrant 1.90 and Virtualbox 5.1.10 at the time of this post.

Feel free to ‘run’ either of the programs, but there’s no need to enter either program just yet.    To really use the Vagrant and the linux virtual machine, you’ll need to download a *nix emulator to allow you to do the things you need to with vagrant.

I use Git’s “bash” to interface with my virtual machines and Vagrant.  You could use putty or CygWin or any other emulator, but I’ve found Git’s bash to be the easiest and simplest to install and use.  Jump over and download git for your machine and install it. At the time of writing, I’m using Git 2.11.0.

While installing Git, I recommend leaving everything checked on the ‘select’ components window if you don’t currently have any git applications installed. If you want to use other git applications, you can uncheck the “associate .git* configuration files…” option.  There is one ‘gotcha’ when installing git that you should be aware of.

On the “adjusting your path” section (see figure 1), you’ll need to think about how you want to use git on the command line.

git command line pathFigure 1: Adjusting your path

I selected the third option when installing git. I do not use git from the windows command line though…I use a git GUI along with git from the command line within my virtual environment.

Another screen to consider is the “Configuring the terminal emulator…” screen (figure 2).  I selected and use the MinTTY option because it gives me a much more *nix feel. This is personal preference. If you are going to be doing a lot of interactive python work in the console, you might want to select the 2nd option to use the windows default console window.

Configuring your terminal emulatorFigure 2: Configuring your terminal emulator

During the remainder of the installation, I left the rest of the options at the defaults.

Now that git (and bash) is installed, you can launch Git Bash to start working with Vagrant. You should see a window similar to Figure 3.

Git BashFigure 3: Git Bash

From this point, you can do your ‘vagrant init’, ‘vagrant up’ and ‘vagrant ssh’ to initialize, create and ssh into your vagrant machine.

Setting up Vagrant on Windows

For those of you that haven’t used Vagrant in the past, here’s how I set it up and use it. I generally use vagrant in this way to run jupyter, so I’ll walk you through setting things up for jupyter, pandas, etc.

First, set up a directory for your project. At the Bash command line, change into the directory you want to work from and type “mkdir vagrant_project” (or whatever name you want to use). Now, initialize your vagrant project by typing:

vagrant init

This will create a Vagrantfile in the directory you’re in. This will allow you to set the configuration of your virtual machine and Vagrant. The Vagrantfile should look something like this:

VagrantFile ExampleFigure 4: VagrantFile Example

Before we go any further, open up your Vagrantfile and change the following line:

config.vm.box = "base"

change “base” to “ubuntu/xenial64” to run Ubuntu 16.04. The line should now read:

config.vm.box = "ubuntu/xenial64"

If you want to run other flavors of linux or other OS’s, you can find others at https://atlas.hashicorp.com/search.

Since I’m setting this VM up to work with jupyter, I also want to configure port forwarding in the Vagrantfile. Look for the line that reads:

# config.vm.network "forwarded_port", guest: 80, host: 8080

and add a line directly below that line to read:

config.vm.network "forwarded_port", guest: 8888, host: 8888

This addition creates a forwarded port on your system from port 8888 on your host (your windows machine) to port 8888 on your guest (the virtual machine). This will allow you to access your jupyter notebooks from your Windows browser.

At this point, you could also configure lots of other vagrant options, but these are the bare minimums that you need to get started.

At the Bash command line, you can now type “vagrant up” to build your virtual machine. Since this is the first time you’ve run the command on this directory, it will go out and download the ubuntu/xenial64 ‘box’ and then build the virtual machine with the defaults.  You might see a Windows alert asking to ‘approve’ vagrant to make some changes…go ahead and allow that.

Once the ‘vagrant up’ command is complete, you should see something similar to Figure 5 below.

Output of Vagrant UpFigure 5: Output of ‘vagrant up’

Now, you can ‘vagrant ssh’ to get into the virtual machine.  You should then see something similar to Figure 6. Now your running vagrant on windows!

Vagrant SSH - Vagrant on WindowsFigure 6: Output of ‘vagrant ssh’

One of the really cool things that vagrant does by default is set up shared folders. This allows you to do your development work in your favorite IDE or editor and have the changes show up automatically in your vagrant virtual machine.

At the Bash command line, type:

cd /vagrant

You should see a directory listing that has your Vagrantfile and a log file. If you visit your project directory using windows explorer, you should see the same two files. Shared folders for the win! I know its just a small thing, but it makes things easier for initial setup.

You now have vagrant on windows!

Configure the Python Environment

Time to set up your python environment.

First, install pip.

sudo apt install python-pip

Even though you’ve set up a virtual machine for development, it is still a good idea to use virtualenv to separate multiple projects requirements.  Install install virtualenv  with the following command:

sudo apt install virtualenv

In your project directory, set up your virtual environment by typing:

virtualenv env

Note: You may run unto an error while running this command. It will be something like like the message below:

OSError: [Errno 71] Protocol error

If this happens, delete the ‘env’ folder and then add ‘–always-copy’ to the command and re-run it. See here for more details.

#### run this only if you had an error in the step above
virtualenv env --always-copy

Activate your virtualenv by typing:

source env/bin/activate

We’re ready to install pandas and jupyter using the command below. This will install both modules as well as their dependencies.

pip install pandas jupyter

Now you’re ready to run jupyter.

jupyter notebook --ip=0.0.0.0

In the above command, we start jupyter notebook with an extra config line of ‘–ip=0.0.0.0’. This tells jupyter to listen on any IP address. It may not always be necessary, but I find it cuts out a lot of issues when I’m running it in vagrant like this.

In your windows browser, visit ‘http://localhost:8888/tree’ and  – assuming everything went the way it should – you should see your jupyter notebook tree.

Jupyter via Vagrant VMFigure 7: Jupyter via Vagrant VM

From here, you can create your notebooks and run them just like you would with any other platform.

 

The post Vagrant on Windows appeared first on Python Data.

December 08, 2016 07:54 PM


Enthought

Webinar: Solving Enterprise Python Deployment Headaches with the New Enthought Deployment Server

JOIN US FOR A WEBINAR:

Solving Enterprise Python Deployment Headaches with the NEW Enthought Deployment Server

enthought-deployment-server-webinar-1024x512

Built on 15 years of experience of Python packaging and deployment for Fortune 500 companies, the NEW Enthought Deployment Server provides enterprise-grade tools groups and organizations using Python need, including:

  1. Secure, onsite access to a private copy of the proven 450+ package Enthought Python Distribution
  2. Centralized management and control of packages and Python installations
  3. Private repositories for sharing and deployment of proprietary Python packages
  4. Support for the software development workflow with Continuous Integration and development, testing, and production repositories

In this webinar, Enthought’s product team will demonstrate the key features of the Enthought Deployment Server and how it can take the pain out of Python deployment and management at your organization.


WHEN
Wed, December 14, 2016, 1-1:45 PM CT. REGISTER NOW
Tues, December 20, 2016, 9-9:45 AM CT (3:00-3:45 PM BT). REGISTER NOW

Can’t attend a live session? Register here and we’ll send you a recording.


Who Should Attend this Webinar:

If you answer “yes” to any of the questions below, then you (or someone at your organization) should attend this webinar:

  1. Are you using Python in a high-security environment (firewalled or air gapped)?
  2. Are you concerned about how to manage open source software licenses or compliance management?
  3. Do you need multiple Python environment configurations or do you need to have consistent standardized environments across a group of users?
  4. Are you producing or sharing internal Python packages and spending a lot of effort on distribution?
  5. Do you have a “guru” (or are you the guru?) who spends a lot of time managing Python package builds and / or distribution?

In this webinar, we’ll demonstrate how the Enthought Deployment Server can help your organization address these situations and more. Register here today to learn more (if you can’t attend, we’ll send you a recording).

December 08, 2016 03:39 PM


Catalin George Festila

Noise 2D, 3D, 4Dwith opensimplex python module.

OpenSimplex noise is an n-dimensional gradient noise function that was developed in order to overcome the patent-related issues surrounding Simplex noise, while continuing to also avoid the visually-significant directional artifacts characteristic of Perlin noise.
Let's start with instalation:
C:\Python27\Scripts>pip install OpenSimplex
Collecting OpenSimplex
Downloading opensimplex-0.2.tar.gz
Installing collected packages: OpenSimplex
Running setup.py install for OpenSimplex ... done
Successfully installed OpenSimplex-0.2

Test some examples from official page:
C:\Python27>python
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (
Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from opensimplex import OpenSimplex
>>> tmp = OpenSimplex()
>>> print (tmp.noise2d(x=10, y=10))
-0.297251513589
>>> tmp = OpenSimplex(seed=1)
>>> print (tmp.noise2d(x=10, y=10))
-0.734782324747
>>> dir(OpenSimplex)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribut
e__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_e
x__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_
_weakref__', '_extrapolate2d', '_extrapolate3d', '_extrapolate4d', 'noise2d', 'n
oise3d', 'noise4d']

Let's make a image example with noise 2D:
from opensimplex import OpenSimplex
from PIL import Image

height = int(input("Enter in the map height: "))
width = int(input("Enter in the map width: "))

def main():
    simplex = OpenSimplex()
    im = Image.new('L', (width, height))
    for y in range(0, height):
        for x in range(0, width):
            value = simplex.noise2d(x , y )
            color = int((value + 1) * 128)
            im.putpixel((x, y), color)
    im.save('noise2d.png')
    im.show()
if __name__ == '__main__':
    main()

December 08, 2016 03:15 PM


Import Python

ImportPython Issue 102 - asyncio and coroutines, chatbots, tensorflow and more

Worthy Read

Singapore’s MRT Circle Line was hit by a spate of mysterious disruptions in recent months, causing much confusion and distress to thousands of commuters. Like most of my colleagues, I take a train on the Circle Line to my office at one-north every morning. So on November 5, when my team was given the chance to investigate the cause, I volunteered without hesitation.
pandas

9 DBs to choose from, 5 min setup, auto-scaling, Cloud hosted. FREE for 30 Days.
Sponsor

A dive into Python’s asyncio tasks and event loops. The asyncio package allows us to define coroutines. These are code blocks that have the ability of yielding execution to other blocks. They run inside an event loop which iterates through the scheduled tasks and executes them one by one. A task switch occurs when it reaches an await statement or when the current task completes.
python3
,
asyncio

Fun article by Florin Badita using BigQuery over entire github hosted Python project's code base.
github
,
bigquery

We are going to explore two parts of using a ML model in production. How to export a model and have a simple self-sufficient file for it ? How to build a simple python server (using flask) to serve it with TF ?.
tensorflow
,
machinelearning

What does the future hold for Python, aside from new versions of the language ?

Move fast, with confidence. Learn more about Infrastructure at an upcoming webinar.
Sponsor

ChatBot is the new buzz word for a while. Microsoft Cognitive Services API allows you to built ones that allow your app to process natural language and learn how to recognize what users want.
chatbots

I created a semi-practical application that reads from the Twitter stream, parses tweets, and does some machine learning magic to score the tweet’s sentiment and project it into a two-dimensional grid, where tweets with similar content will appear closer to each other. It does all of this more or less in real time using asynchronous messaging.
data-mining
,
gensim
,
tweets

As I’m preparing for a project that will involve Python programming, I need to get up to speed with at least a basic level of Python mastery. However, I’m not a big fan of using the interactive Python REPL. I decided to use Xcode instead, and I’m finding it a much better solution for my needs:
xcode
,
apple

Introductory article on usage of pickle module.
pickle

In a recent project, we decided to use Lambda to execute some fairly math-y Python code in response to user click events on a webpage. Originally this back-end Python code had utilized the Numba library to speed up its execution. However, we quickly found that it was not trivial to make Lambda and Numba play nicely together.
aws
,
lambda
,
numba

From time to time I like to read documentation of modules I think I know well. The python documentation is not a pleasant read but sometimes you strike a gem.
subTest

FuzzyWuzzy is a fantastic Python package which uses a distance matching algorithm to calculate proximity measures between string entries.
string_maching

In this short screencast I’ll walk you through a simple code example that demonstrates how you can use custom exception classes in your Python code to make it easier to understand, easier to debug, and more maintainable.
video
,
screencast
,
exception_handling


Projects

universe - 2867 Stars, 213 Fork
Universe is a software platform for measuring and training an AI's general intelligence across the world's supply of games, websites and other applications. This is the universe open-source library, which provides a simple Gym interface to each Universe environment. Universe allows anyone to train and evaluate AI agents on an extremely wide range of real-time, complex environments.

tensorflow_chatbot - 120 Stars, 36 Fork
This is the full code for 'How to Make an Amazing Tensorflow Chatbot Easily' by @Sirajology on Youtube. In this demo code, we implement Tensorflows Sequence to Sequence model to train a chatbot on the Cornell Movie Dialogue dataset. After training for a few hours, the bot is able to hold a fun conversation.

showmemore - 9 Stars, 1 Fork
ShowMeMore is an automated researcher. Given a list of tags to start with, it goes hunting for images, and over time grows its model in response to reactions, slowly reaching out to find things you weren't aware you already liked.

speedtest-tweet-bot - 8 Stars, 1 Fork
monitors internet speed and tweets when its slow

alfredworkflow-capkeystone - 7 Stars, 1 Fork
Alfred Workflow to convert hex string to assembly and vice versa

December 08, 2016 03:01 PM


Django Weekly

Django Weekly 16 - Djangocon Europe, cookiecutter, caching, djangogirls and more ...

Worthy Read

2017’s DjangoCon Europe takes place against the gorgeous backdrop of Florence in the springtime. The event will be held in the historic Odeon Cinema in the centre of the city. Ticket sales and call for proposal is open.
djangocon
,
europe

9 DBs to choose from, 5 min setup, auto-scaling, Cloud hosted. FREE for 30 Days.
sponsor

Cookiecutter is ” A command-line utility that creates projects from cookiecutters (project templates). E.g. Python package projects, jQuery plugin projects.” In particular I explored some of the Django-specific cookiecutter templates. List includes Cookiecutter-Django, Django-crud, Django-paas, Simple Django.
cookiecutter
,
two_scoops_of_django

Basic tutorial showing us where django admin's recent actions are stored i.e. django_admin_log / LogEntry. Followed up with how to browse all entries from admin panel itself. Also shows how to disable edits and delete.
admin-panel

If you are looking for minutes of meeting of Django Software Foundation this is page to check.
DSF

Curators Note - Django Girls is phenomenal in it's execution and impact. I was at PyLadies SFO in March 2016 and met 3 journalist that credited Django Girls for creating an environment that was welcoming of women and newbies. Subsequently at Pycon India I met four college students who credited Django Girls for helping get internships. If you look at the no of cities and grassroot level events they do it's amazing.
djangogirls

Move fast, with confidence. Learn more about Infrastructure at an upcoming webinar.
sponsor

Background and Explanation of a Django Caching Decorator
Cache

In a simple way we can say that Generic Relations is a foreign key that can refer to any instance of models. Good beginners tutorial explaining Generic Relations.
orm


documentation

Code snippet
code_snippet

Continuous delivery with Bitbucket Pipelines and custom CentOS7 PostgreSQL Docker image.
cookiecutter
,
continous delivery


Projects

Simple django app that add autocomplete to search inside admin panel.

December 08, 2016 10:28 AM


hypothesis.works articles

Compositional shrinking

In my last article about shrinking, I discussed the problems with basing shrinking on the type of the values to be shrunk.

In writing it though I forgot that there was a halfway house which is also somewhat bad (but significantly less so) that you see in a couple of implementations.

This is when the shrinking is not type based, but still follows the classic shrinking API that takes a value and returns a lazy list of shrinks of that value. Examples of libraries that do this are theft and QuickTheories.

This works reasonably well and solves the major problems with type directed shrinking, but it’s still somewhat fragile and importantly does not compose nearly as well as the approaches that Hypothesis or test.check take.

Ideally, as well as not being based on the types of the values being generated, shrinking should not be based on the actual values generated at all.

This may seem counter-intuitive, but it actually works pretty well.

Read more...

December 08, 2016 09:00 AM


Swisscom ICT

Smart Contracts

Paying is easy and seamless in Switzerland. And thanks to the public land register, even real estate transactions carry virtually no risk of fraud.  Contrast this with other countries, where you have to obtain insurance against an unpaid mortgage by a previous owner being discovered on the house you just bought. Or simply transferring money in rural Africa, where the risk of theft was high before the advent of mobile payment services.

So there’s an obvious need for efficient transaction solutions (like Monetas) in markets that lack an efficient infrastructure. But what would be the benefit in a country like Switzerland, where the population is also reluctant to try new things that don’t have an obvious benefit?

I believe that a very important element of the transaction landscape is the Smart Contract. The Smart Contract is basically an agreement that is defined by executable code instead of natural language text. So a mortgage contract would actually effect the interest payment instead of describing when it is due; a rental car contract would unlock the car instead of describing when you can use it.

 

The crypto transaction landscape

Quite a few solutions and frameworks have been proposed that make use of cryptographic technology to ease transactions. Most notably are blockchains, which form the basis of public ecosystems like Bitcoin or Ethereum and restricted access frameworks like Hyperledger.

Smart Contracts rely on a blockchain to add code in a non-disputable transaction. They do not necessarily involve a currency like Bitcoin, but may use one or more such currencies or other payment systems during the execution of a contract.

Blockchain ecosystem

Interactive contracts: Queries and Simulation

Smart contracts are defined by executable code. They will need a human-readable high-level programming language and suitable execution environment. I fear this will be a custom language or JavaScript and I hope it will rather be Python [see inset below]. A very high level of abstraction is necessary: There will be contracts built upon other contracts, contracts reacting to input events. Contracts may provide a query API, answering questions about possible behavior (Was this agreed? What did we agree?).

Contract conditions can also be simulated to check how contracts react. This may be used to discover gaps and expose potential risks. Will contracts ever negotiate with a mediator bot appointed by their parties to close discovered gaps?

Insurance

While reading the description of a structured financial product may seem complicated, smart contracts may cause an explosion of legal complexity. But their interactive nature and simulation possibilities help taming the complexity. Instead of paying lawyers fighting for their interpretation of the contract’s text, you might simply buy insurance to protect you against contractual risks. And the regulation, instead of issuing more complex regulations, might mandate that contract holders are insured, much in the same way that car owners have to be insured.

The insurance will of course issue a smart contract, accessing the contracts it protects, along with other information about your life.

Breaking the chain of transparency

If an insurance company needs to see all my contracts and certain other life conditions, will this lead to a horror world of full transparency, as depicted by Dave Eggers in The Circle?

Not necessarily: Simulation can be done without my identity, by someone I trust. The insurance doesn’t need to know me, as long as it gets the aggregated, anonymized data. But this needs a network of trust, similar to the banking network.

Essential to this is an open way to express smart contracts. Open in the sense of open source, so everyone can inspect and vet the contractual building blocks.

Technical Opinion: Why Python for smart contracts

A full language with a broad community has advantages over a more restricted domain-specific language such as Ethereum’s  Solidity, which is JavaScript-like.

Python is an expressive language with a low conceptual overhead.  It is guided by the Zen of Python, which also stipulates good practice for development in Python. I find these principles very appropriate when applied to contracts.

Notably,

  • Simple is better than complex: The most important contracts (like constitutions) should be as simple as possible.
  • Readability counts: You shouldn’t need a law degree or a computer science degree to understand a contract.
  • Explicit is better than implicit: For contracts, implicit is the worst case. You want things stated, not implied.
  • There should be one– and preferably only one –obvious way to do it: Following accepted practices helps understanding.

Python is often used as a “glue language” – which is also appropriate for contracts, as they usually bridge different worlds, notably physical world objects and cash flows.

 

Will crypto-based transactions find widespread acceptance? Written contracts are in wide use, and there is a huge open ecosystem built around them. As the lackadaisical use of end-to-end encryption of email shows, it’s not easy to add features in an open ecosystem unless something is severely broken.  Only if there is a clearly apparent value added, a new paradigm might succeed. I believe that insurance of contracts might be this feature, as protection against uncertainty and fear has always driven humanity.

Recommended Resources

December 08, 2016 06:00 AM


Calvin Spealman

Finding "One Game A Month"

I was really excited about the One Game A Month challenge as soon as I heard about it.

For about two years I've struggled in fits and starts to make my way into game development. This hasn't been productive in any of the ways I hoped when I started. Its really difficult to be fairly experienced as a developer, which I believe I am in my day job as a web developer, while struggling really hard at an area in which your experience just doesn't exist.

Its like being a pilot who doesn't know how to drive.


But this challenge provided a new breath to this little hobby of mine. It gave me a scaffolding to experiment, to learn, to reflect on finished projects. I had spent far too much time on game projects that stretched on far past their exciting phases, bogged down by bad decisions and regret.

And it has worked.

I have a lot to learn. I have a lot of experience to gain through trial and error and mistake and discovery. I have a lot of fun to be had making more small games that scratch the little itches that have been building up. I have a really exciting 2017 planned for game development!

I've been doing One Game A Month for three months now. I've released three small games. These aren't great games by any stretch of the definition, of course. But they're games that I'm learning from. They're something I'm really excited about and I have needed something personally exciting as a developer for a long time. These are good times for me, and I hope to have more to share about them soon.

December 08, 2016 01:25 AM


Daniel Bader

Cool new features in Python 3.6

Cool new features in Python 3.6

Python 3.6 adds a couple of new features and improvements that’ll affect the day to day work of Python coders. In this article I’ll give you an overview of the new features I found the most interesting.

Improved numeric literals

This is a syntactic tweak that makes numeric literals easier to read. You can now add underscores to numbers in order to group them to your liking. This is handy for expressing large quantities or constants in binary or hexadecimal:

>>> six_figures = 100_000
>>> six_figures
100000
>>> programmer_error = 0xbad_c0ffee
>>> flags = 0b_0111_0101_0001_0101

Remember, this change doesn’t introduce any new semantics. It’s just a way to represent numeric literals differently in your source code. A small but neat addition.

You can learn more about this change in PEP 515.

String interpolation

Python 3.6 adds yet another way to format strings called Formatted String Literals. This new way of formatting strings lets you use embedded Python expressions inside string constants. Here’s are two simple examples to give you a feel for the feature:

>>> name = 'Bob'
>>> f'Hello, {name}!'
'Hello, Bob!'
>>> a = 5
>>> b = 10
>>> f'Five plus ten is {a + b} and not {2 * (a + b)}.'
'Five plus ten is 15 and not 30.'

String literals also support the existing format string syntax of the str.format() method. That allows you to do things like:

>>> error = 50159747054
>>> f'Programmer Error: {error:#x}'
'Programmer Error: 0xbadc0ffee'

Python’s new Formatted String Literals are similar to the JavaScript Template Literals added in ES2015/ES6. I think they’re quite a nice addition to the language and I look forward to using them in my day to day work.

You can learn more about this change in PEP 498.

Type annotations for variables

Starting with Python 3.5 you could add type annotations to functions and methods:

>>> def my_add(a: int, b: int) -> int:
>>>    return a + b

In Python 3.6 you can use a syntax similar to type annotations for function arguments to type-hint standalone variables:

>>> python_version : float = 3.6

Nothing has changed in terms of the semantics–CPython simply records the type as a type annotation but doesn’t validate or check types in any way.

Type-checking is purely optional and you’ll need a tool like Mypy for that, which basically works like a code linter.

You can learn more about this change in PEP 526.

Other notable changes

I think Python 3.6 will be an interesting release. There are many more interesting additions and improvements that are worth checking out. You can learn more about them in the links below or by reading the official “What’s New In Python 3.6” announcement.

December 08, 2016 12:00 AM

December 07, 2016


Machinalis

PyConAr 2016

A few days ago (November 25, 26 and 27th) the 8th edition of PyCon Argentina took place in Bahía Blanca, a city 600km to the south of Buenos Aires, and I was lucky enough to be able to go with a group of other people from Machinalis.

PyConAr is the biggest Python related conference we have in Argentina, and one of the biggest programming conferences in the country too. It’s organized by Python Argentina (PyAr), a large community of programmers from all kind of backgrounds, which also holds smaller PyDays and one PyCamp each year. And quite important too: each year it’s organized in a different city by a different group of people, because as a community we want to give everyone the chance to participate, especially those who live far away from the usual places where conferences are held.

The talks and workshops

The topics were diverse, ranging from machine learning to asyncio, web APIs, mobile development, remote working, docker, and much more. And while most of the talks and workshops were very technical, I was glad to find some non-technical ones too. Specially the keynotes, which weren’t python-specific but dealt with topics of central interest in our community.

Some quite subjective highlights:

I loved the deep learning workshop organized by LinuxChix Argentina and guided by Celia Cintas. The example used in the workshop was Star Wars spaceships classification using neural networks and real time video. How awesome is that!

/static/media/uploads/pyconar_star_wars.jpg

[photo by Cynthia Monastirsky]

I was happy about how our own 3 talks combined each other to give a general overview of the Python data science ecosystem. Rafael Carrascosa presented a general overview of the data science toolbelt, Javier Mansilla dived a little deeper into the deep learning scenario (pun intended), and I covered technical details of neural networks and Keras (the talks were filmed by the conference and will be uploaded, we will add the links later on).

/static/media/uploads/pyconar_rafa.jpg

[photo by Juan Pedro Fisanotti]

The sprints were quite productive! People helped solve issues and implemented new features into three not so known but very useful projects Pysenteishon, SimpleAI, and Twistorpy.

/static/media/uploads/pyconar_sprints.jpg

[photo by PyConAr]

I learned some interesting things regarding high speed access to large volumes of shared data from Claudio Freire’s talk.

And finally, there was a workshop for blind people which I wasn’t able to be in, but others spoke of how useful it was for the participants!

The community

PyAr is known to be a very friendly and welcoming community. As we like to say, “PyAr es amor” (PyAr is love). This PyConAr wasn’t the exception to the rule.

Experienced people guiding new ones in the workshops and sprints. Large groups of people who didn’t know each other before, enjoying a beer or a pizza after the conference with the mandatory geeky conversations (yes, we tried to define monads). Knowledge and smiles shared in the halls. I really enjoy being part of this big family that is PyAr :)

Maybe one of the most emotive moments was the lightning talk given by one of the youngest participants of the workshop for blind people, in which she expressed how important it was for her. As one of the organizers said, "that alone is worth all the effort".

/static/media/uploads/pyconar_all.jpg
[photo by Yamila Cuestas]

Thanks

For that and all the other things not included in this post, we would like to thank specially to this year organizers! You did an awesome job, and the whole PyAr community owes you their gratitude.

And also to the sponsors, speakers, and everyone who helped in some way.

See you in the next PyConAr! (or PyDay, or PyCamp :)

/static/media/uploads/pyconar_machinalis.jpg

The Machi-group :) [photo by Nicolás Demarchi]

December 07, 2016 08:02 PM