skip to navigation
skip to content

Planet Python

Last update: September 25, 2016 04:51 AM

September 25, 2016

Python 4 Kids

Python for Kids: Python 3 – Project 10

Using Python 3 in Project 10 of Python For Kids For Dummies

In this post I talk about the changes that need to be made to the code of Project 10 of my book Python for Kids for Dummies in order for it to work with Python 3. The main difference between the Python 2.7 and Python 3 code for this project is that Python 3 uses raw_input and that has been renamed to input in Python 3. Most of the code in project 10 will work with this one change. However, in a lot of cases what Python outputs in Python 3 is different from the output in Python 2.7. This project has a lot of code. In order to shorten the length of this post I am only showing the Python 3 versions of the longer pieces (rather than both Python 2.7 (from the book) and Python 3). Look at the book to see the Python 2.7 code (it’s very similar).

Disclaimer

Some people want to use my book Python for Kids for Dummies to learn Python 3. I am working through the code in the existing book, highlighting changes from Python 2 to Python 3 and providing code that will work in Python 3. If you are using Python 2.7 you can ignore this post. This post is only for people who want to take the code in my book Python for Kids for Dummies and run it in Python 3.

######## Page 283

The code on this page uses raw_input, which has been renamed to input in Python 3. You can either replace all occurrences of raw_input with input or add a line:

raw_input = input

at the start of the relevant code. In order to reduce the amount of code being repeated, I am adding raw_input = input to the Constants section of the code. You will need to remember that all of the later code assumes that this line has been added. 

  
"""
Python 2.7
math_trainer.py
Train your times tables.
Initial Features:
* Print out times table for a given number.
* Limit tables to a lower number (default is 1)
and an upper number (default is 12).
* Pose test questions to the user
* Check whether the user is right or wrong
* Track the user's score.
Brendan Scott
February 2015
"""
#### Constants Section
TEST_QUESTION = (4, 6)
QUESTION_TEMPLATE = "What is %sx%s? "
#### Function Section
#### Testing Section
question = TEST_QUESTION
prompt = QUESTION_TEMPLATE%question
correct_answer = question[0]*question[1] # indexes start from 0
answer = raw_input(prompt)
if int(answer)== correct_answer:
    print("Correct!")
else:
    print("Incorrect")

>>> ================================ RESTART ================================
>>>
What is 4x6? 24
Correct!
>>> ================================ RESTART ================================
>>>
What is 4x6? 25
Incorrect


     

"""
Python 3
math_trainer.py
Train your times tables.
Initial Features:
* Print out times table for a given number.
* Limit tables to a lower number (default is 1)
and an upper number (default is 12).
* Pose test questions to the user
* Check whether the user is right or wrong
* Track the user's score.
Brendan Scott
February 2015
"""
#### Constants Section
raw_input = input # this line added
TEST_QUESTION = (4, 6)
QUESTION_TEMPLATE = "What is %sx%s? "

#### Function Section

#### Testing Section
question = TEST_QUESTION
prompt = QUESTION_TEMPLATE%question
correct_answer = question[0]*question[1] # indexes start from 0
answer = raw_input(prompt)
if int(answer)== correct_answer:
    print("Correct!")
else:
    print("Incorrect")

>>> ================================ RESTART ================================
>>>
What is 4x6? 24
Correct!
>>> ================================ RESTART ================================
>>>
What is 4x6? 25
Incorrect

######## Page 286-296

All code on this page is the same, and all outputs from the code is the same in Python 3 as in Python 2.7.
Remember that for the code to work in Python 3 code an additional line

raw_input = input

as added in the Constants section of the code.

######## Page 297

All code on this page is the same, and all outputs from the code is the same in Python 3 as in Python 2.7.

######## Page 298
The code in this section is different in Python 2.7 v Python 3.
The Python 2.7 code assumed that there was a list and that a while loop repeatedly removed things from that list. When everything was removed then the loop stopped. This was achieved by a test

batch != []

that is, stop when the variable batch is an empty list.
Ultimately, what is in batch comes from a call to the range builtin:

tables_to_print = range(1, upper+1)

In Python 2.7 this is a list which is generated in full and stored in tables_to_print. In Python 3 it’s not. Rather the range builtin generates the values that are needed at the time they are needed – not before. In Python 3 batch is a “range object”, not a list. And, while batch gets shorter and shorter, it’s never going to be an empty list (it would need to stop being a range and start being a list), no matter how long the program runs. To get this code working in Python 2.7 you can either:
(A) explicitly make batch a list by changing the line:

tables_to_print = range(1, upper+1)

to

tables_to_print = list(range(1, upper+1))

this changes all the relevant variables (and, in particular batch) into lists so the condition in the while loop will evaluate as you expect; or

(B) change the condition in the while loop to check the length of batch rather than whether or not it is an empty list. That is change:

 
    while batch != []: # stop when there's no more to print

to

 
    while len(batch) > 0: # stop when there's no more to print

That is, once the length is 0 (ie no more elements to display), stop the loop. I think this is the better of the two options because it makes the test independent of the type of variable used to keep track of batches.

Remember that the Python 3 code has an additional line

 
raw_input = input

in the Constants section of the code. 

 
#Python 2.7

TIMES_TABLE_ENTRY = "%2i x %2i = %3i "

def display_times_tables(upper=UPPER):
    """
    Display the times tables up to UPPER
    """
    tables_per_line = 5
    tables_to_print = range(1, upper+1)
    # get a batch of 5 to print
    batch = tables_to_print[:tables_per_line]
    # remove them from the list
    tables_to_print = tables_to_print[tables_per_line:]
    while batch != []: # stop when there's no more to print
        for x in range(1, upper+1):
            # this goes from 1 to 12 and is the rows
            accumulator = []
            for y in batch:
                # this covers only the tables in the batch
                # it builds the columns
                accumulator.append(TIMES_TABLE_ENTRY%(y, x, x*y))
            print("".join(accumulator)) # print one row
        print("\n") # vertical separation between blocks of tables.
        # now get another batch and repeat.
        batch = tables_to_print[:tables_per_line]
        tables_to_print = tables_to_print[tables_per_line:]


     

#Python 3                            
TIMES_TABLE_ENTRY = "%2i x %2i = %3i "

def display_times_tables(upper=UPPER):
    """
    Display the times tables up to UPPER
    """
    tables_per_line = 5
    tables_to_print = list(range(1, upper+1))
    # get a batch of 5 to print
    batch = tables_to_print[:tables_per_line]
    # remove them from the list
    tables_to_print = tables_to_print[tables_per_line:]
    while len(batch)>0: # stop when there's no more to print
        for x in range(1, upper+1):
            # this goes from 1 to 12 and is the rows
            accumulator = []
            for y in batch:
                # this covers only the tables in the batch
                # it builds the columns
                accumulator.append(TIMES_TABLE_ENTRY%(y, x, x*y))
            print("".join(accumulator)) # print one row
        print("\n") # vertical separation between blocks of tables.
        # now get another batch and repeat.
        batch = tables_to_print[:tables_per_line]
        tables_to_print = tables_to_print[tables_per_line:]

######## Page 302, 304
All code on this page is the same, and all outputs from the code is the same in Python 3 as in Python 2.7.
Remember that the Python 3 code has an additional line

   
raw_input = input

in the Constants section of the code.

######## Page 305-306

All code on this page is the same, and all outputs from the code is the same in Python 3 as in Python 2.7.

######## Page 307
All code on this page is the same, and all outputs from the code is the same in Python 3 as in Python 2.7.
Remember that the Python 3 code has an additional line

   
raw_input = input

in the Constants section of the code.

#########################################
### Full Code:
#########################################

The code in this section is different in Python 2.7 v Python 3.
The Python 3 code has an additional line
raw_input = input
in the Constants section of the code and the line 

   
    while batch != []: # stop when there's no more to print

has been changed to

   
    while len(batch) > 0: # stop when there's no more to print

     
"""
math_trainer.py
Train your times tables.
Initial Features:
* Print out times table for a given number.
* Limit tables to a lower number (default is 1) and
an upper number (default is 12).
* Pose test questions to the user
* Check whether the user is right or wrong
* Track the user's score.
Brendan Scott
February 2015
"""

#### Imports Section
import random
import sys
import time

#### Constants Section
TEST_QUESTION = (4, 6)
QUESTION_TEMPLATE = "What is %sx%s? "
LOWER = 1
UPPER = 12
MAX_QUESTIONS = 10 # for testing, you can increase it later
TIMES_TABLE_ENTRY = "%2i x %2i = %3i "

INSTRUCTIONS = """Welcome to Math Trainer
This application will train you on your times tables.
It can either print one or more of the tables for you
so that you can revise (training) or you it can test
you on your times tables.
"""
CONFIRM_QUIT_MESSAGE = 'Are you sure you want to quit (Y/n)? '
SCORE_TEMPLATE = "You scored %s (%i%%) in %.1f seconds"

#### Function Section
def make_question_list(lower=LOWER, upper=UPPER, random_order=True):
    """ prepare a list of questions in the form (x,y)
    where x and y are in the range from LOWER to UPPER inclusive
    If random_order is true, rearrange the questions in a random
    order
    """
    spam = [(x+1, y+1) for x in range(lower-1, upper)
                       for y in range(lower-1, upper)]
    if random_order:
        random.shuffle(spam)
    return spam

def display_times_tables(upper=UPPER):
    """
    Display the times tables up to UPPER
    """
    tables_per_line = 5
    tables_to_print = range(1, upper+1)
    # get a batch of 5 to print
    batch = tables_to_print[:tables_per_line]
    # remove them from the list 
    tables_to_print = tables_to_print[tables_per_line:]
    while batch != []: # stop when there's no more to print
        for x in range(1, upper+1):
            # this goes from 1 to 12 and is the rows 
            accumulator = []
            for y in batch:
                # this covers only the tables in the batch
                # it builds the columns
                accumulator.append(TIMES_TABLE_ENTRY%(y, x, x*y))
            print("".join(accumulator)) # print one row
        print("\n") # vertical separation between blocks of tables.
        # now get another batch and repeat. 
        batch = tables_to_print[:tables_per_line]
        tables_to_print = tables_to_print[tables_per_line:]

    
def do_testing():
    """ conduct a round of testing """
    question_list = make_question_list()
    score = 0
    start_time = time.time()
    for i, question in enumerate(question_list):
        if i >= MAX_QUESTIONS:
            break
        prompt = QUESTION_TEMPLATE%question
        correct_answer = question[0]*question[1]
        # indexes start from 0
        answer = raw_input(prompt)

        if int(answer) == correct_answer:
            print("Correct!")
            score = score+1
        else:
            print("Incorrect, should have "+\
                  "been %s"%(correct_answer))

    end_time = time.time()
    time_taken = end_time-start_time
    percent_correct = int(score/float(MAX_QUESTIONS)*100)
    print(SCORE_TEMPLATE%(score, percent_correct, time_taken))

def do_quit():
    """ quit the application"""
    if confirm_quit():
        sys.exit()
    print("In quit (not quitting, returning)")

def confirm_quit():
    """Ask user to confirm that they want to quit
    default to yes 
    Return True (yes, quit) or False (no, don't quit) """
    spam = raw_input(CONFIRM_QUIT_MESSAGE)
    if spam == 'n':
        return False
    else:
        return True    


#### Testing Section

#do_testing()
##display_times_tables()

#### Main Section

if __name__ == "__main__":
    while True:
        print(INSTRUCTIONS)
        raw_input_prompt = "Press: 1 for training,"+\
                           " 2 for testing, 3 to quit.\n"
        selection = raw_input(raw_input_prompt)
        selection = selection.strip()
        while selection not in ["1", "2", "3"]:
            selection = raw_input("Please type either 1, 2, or 3: ")
            selection = selection.strip()

        if selection == "1":
            display_times_tables()
        elif selection == "2":
            do_testing()
        else:  # has to be 1, 2 or 3 so must be 3 (quit)
            do_quit()


     

"""
math_trainer.py (Python 3)
Train your times tables.
Initial Features:
* Print out times table for a given number.
* Limit tables to a lower number (default is 1) and
an upper number (default is 12).
* Pose test questions to the user
* Check whether the user is right or wrong
* Track the user's score.
Brendan Scott
February 2015
"""

#### Imports Section
import random
import sys
import time

#### Constants Section
raw_input = input
TEST_QUESTION = (4, 6)
QUESTION_TEMPLATE = "What is %sx%s? "
LOWER = 1
UPPER = 12
MAX_QUESTIONS = 10 # for testing, you can increase it later
TIMES_TABLE_ENTRY = "%2i x %2i = %3i "

INSTRUCTIONS = """Welcome to Math Trainer
This application will train you on your times tables.
It can either print one or more of the tables for you
so that you can revise (training) or you it can test
you on your times tables.
"""
CONFIRM_QUIT_MESSAGE = 'Are you sure you want to quit (Y/n)? '
SCORE_TEMPLATE = "You scored %s (%i%%) in %.1f seconds"

#### Function Section
def make_question_list(lower=LOWER, upper=UPPER, random_order=True):
    """ prepare a list of questions in the form (x,y)
    where x and y are in the range from LOWER to UPPER inclusive
    If random_order is true, rearrange the questions in a random
    order
    """
    spam = [(x+1, y+1) for x in range(lower-1, upper)
                       for y in range(lower-1, upper)]
    if random_order:
        random.shuffle(spam)
    return spam

def display_times_tables(upper=UPPER):
    """
    Display the times tables up to UPPER
    """
    tables_per_line = 5
    tables_to_print = range(1, upper+1)
    # get a batch of 5 to print
    batch = tables_to_print[:tables_per_line]
    # remove them from the list 
    tables_to_print = tables_to_print[tables_per_line:]
    while len(batch) > 0: # stop when there's no more to print
        for x in range(1, upper+1):
            # this goes from 1 to 12 and is the rows 
            accumulator = []
            for y in batch:
                # this covers only the tables in the batch
                # it builds the columns
                accumulator.append(TIMES_TABLE_ENTRY%(y, x, x*y))
            print("".join(accumulator)) # print one row
        print("\n") # vertical separation between blocks of tables.
        # now get another batch and repeat. 
        batch = tables_to_print[:tables_per_line]
        tables_to_print = tables_to_print[tables_per_line:]

    
def do_testing():
    """ conduct a round of testing """
    question_list = make_question_list()
    score = 0
    start_time = time.time()
    for i, question in enumerate(question_list):
        if i >= MAX_QUESTIONS:
            break
        prompt = QUESTION_TEMPLATE%question
        correct_answer = question[0]*question[1]
        # indexes start from 0
        answer = raw_input(prompt)

        if int(answer) == correct_answer:
            print("Correct!")
            score = score+1
        else:
            print("Incorrect, should have "+\
                  "been %s"%(correct_answer))

    end_time = time.time()
    time_taken = end_time-start_time
    percent_correct = int(score/float(MAX_QUESTIONS)*100)
    print(SCORE_TEMPLATE%(score, percent_correct, time_taken))

def do_quit():
    """ quit the application"""
    if confirm_quit():
        sys.exit()
    print("In quit (not quitting, returning)")

def confirm_quit():
    """Ask user to confirm that they want to quit
    default to yes 
    Return True (yes, quit) or False (no, don't quit) """
    spam = raw_input(CONFIRM_QUIT_MESSAGE)
    if spam == 'n':
        return False
    else:
        return True    


#### Testing Section

#do_testing()
##display_times_tables()

#### Main Section

if __name__ == "__main__":
    while True:
        print(INSTRUCTIONS)
        raw_input_prompt = "Press: 1 for training,"+\
                           " 2 for testing, 3 to quit.\n"
        selection = raw_input(raw_input_prompt)
        selection = selection.strip()
        while selection not in ["1", "2", "3"]:
            selection = raw_input("Please type either 1, 2, or 3: ")
            selection = selection.strip()

        if selection == "1":
            display_times_tables()
        elif selection == "2":
            do_testing()
        else:  # has to be 1, 2 or 3 so must be 3 (quit)
            do_quit()

September 25, 2016 04:06 AM

Podcast.__init__

Episode 76 - PsychoPy with Jonathan Peirce

Summary

We’re delving into the complex workings of your mind this week on Podcast.__init__ with Jonathan Peirce. He tells us about how he started the PsychoPy project and how it has grown in utility and popularity over the years. We discussed the ways that it has been put to use in myriad psychological experiments, the inner workings of how to design and execute those experiments, and what is in store for its future.

Brief Introduction

Linode Sponsor Banner

Use the promo code podcastinit20 to get a $20 credit when you sign up!

Rollbar Logo

I’m excited to tell you about a new sponsor of the show, Rollbar.

One of the frustrating things about being a developer, is dealing with errors… (sigh)

  • Relying on users to report errors
  • Digging thru log files trying to debug issues
  • A million alerts flooding your inbox ruining your day…

With Rollbar’s full-stack error monitoring, you get the context, insights and control you need to find and fix bugs faster. It’s easy to get started tracking the errors and exceptions in your stack.You can start tracking production errors and deployments in 8 minutes - or less, and Rollbar works with all major languages and frameworks, including Ruby, Python, Javascript, PHP, Node, iOS, Android and more.You can integrate Rollbar into your existing workflow such as sending error alerts to Slack or Hipchat, or automatically create new issues in Github, JIRA, Pivotal Tracker etc.

We have a special offer for Podcast.__init__ listeners. Go to rollbar.com/podcastinit, signup, and get the Bootstrap Plan free for 90 days. That’s 300,000 errors tracked for free.Loved by developers at awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch and more. Help support Podcast.__init__ and give Rollbar a try a today. Go to rollbar.com/podcastinit

Hired Logo

On Hired software engineers & designers can get 5+ interview requests in a week and each offer has salary and equity upfront. With full time and contract opportunities available, users can view the offers and accept or reject them before talking to any company. Work with over 2,500 companies from startups to large public companies hailing from 12 major tech hubs in North America and Europe. Hired is totally free for users and If you get a job you’ll get a $2,000 “thank you” bonus. If you use our special link to signup, then that bonus will double to $4,000 when you accept a job. If you’re not looking for a job but know someone who is, you can refer them to Hired and get a $1,337 bonus when they accept a job.

Interview with Jonathan Peirce

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Summary We're delving into the complex workings of your mind this week on Podcast.__init__ with Jonathan Peirce. He tells us about how he started the PsychoPy project and how it has grown in utility and popularity over the years. We discussed the ways that it has been put to use in myriad psychological experiments, the inner workings of how to design and execute those experiments, and what is in store for its future.Brief IntroductionHello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable.Hired is sponsoring us this week. If you're looking for a job as a developer or designer then Hired will bring the opportunities to you. Sign up at hired.com/podcastinit to double your signing bonus.Once you land a job you can check out our other sponsor Linode for running your awesome new Python apps. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next projectYou want to make sure your apps are error-free so give our last sponsor, Rollbar, a look. Rollbar is a service for tracking and aggregating your application errors so that you can find and fix the bugs in your application before your users notice they exist. Use the link rollbar.com/podcastinit to get 90 days and 300,000 errors for free on their bootstrap plan.Visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch.By leaving a review on iTunes, or Google Play Music it becomes easier for other people to find us.Join our community! Visit discourse.pythonpodcast.com to help us grow and connect our wonderful audience.Your hosts as usual are Tobias Macey and Chris PattiToday we're interviewing Jonathan Peirce about PsychoPy, an open source application for the presentation and collection of stimuli for psychological experimentation Use the promo code podcastinit20 to get a $20 credit when you sign up! I’m excited to tell you about a new sponsor of the show, Rollbar. One of the frustrating things about being a developer, is dealing with errors… (sigh)Relying on users to report errorsDigging thru log files trying to debug issuesA million alerts flooding your inbox ruining your day...With Rollbar’s full-stack error monitoring, you get the context, insights and control you need to find and fix bugs faster. It's easy to get started tracking the errors and exceptions in your stack.You can start tracking production errors and deployments in 8 minutes - or less, and Rollbar works with all major languages and frameworks, including Ruby, Python, Javascript, PHP, Node, iOS, Android and more.You can integrate Rollbar into your existing workflow such as sending error alerts to Slack or Hipchat, or automatically create new issues in Github, JIRA, Pivotal Tracker etc. We have a special offer for Podcast.__init__ listeners. Go to rollbar.com/podcastinit, signup, and get the Bootstrap Plan free for 90 days. That's 300,000 errors tracked for free.Loved by developers at awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch and more. Help support Podcast.__init__ and give Rollbar a try a today. Go to rollbar.com/podcastinit On Hired software engineers designers can get 5+ interview requests in a week and each offer has salary and equity upfront. With full time and contract opportunities available, users can view the offers and accept or reject them before talking to any company. Work with over 2,500 companies from startups to large public companies hailing from 12 major tech hubs in North America and Europe. Hired is totally free for users and If you get a job you’ll get a $2,000 “thank you” bonus. If you use our special link to signup, then that bonus will double to $4,000 when you accept a job. If you’re not looking for a job but know someone who is, you can refer them to Hired and get a $1,337 bonus

September 25, 2016 12:15 AM

September 24, 2016

Hynek Schlawack

Sharing Your Labor of Love: PyPI Quick and Dirty

A completely incomplete guide to packaging a Python module and sharing it with the world on PyPI.

September 24, 2016 12:00 PM

Brian Okken

22: Converting Manual Tests to Automated Tests

How do you convert manual tests to automated tests? This episode looks at the differences between manual and automated tests and presents two strategies for converting manual to automated. Support Special thanks to my wonderful Patreon supporters and those who have supported the show by purchasing Python Testing with unittest, nose, pytest

The post 22: Converting Manual Tests to Automated Tests appeared first on Python Testing.

September 24, 2016 08:00 AM

Weekly Python StackOverflow Report

(xxxviii) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2016-09-24 07:49:04 GMT

  1. Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't? - [85/4]
  2. Remove the first N items that match a condition in a Python list - [46/6]
  3. Short-circuit evaluation like Python's "and" while storing results of checks - [21/14]
  4. Python Recursion Challenge - [13/5]
  5. How to maintain different country versions of same language in Django? - [8/1]
  6. Why does Python 3 exec() fail when specifying locals? - [8/1]
  7. Is there a difference between str function and percent operator in Python - [8/1]
  8. How to handle SQLAlchemy Connections in ProcessPool? - [8/0]
  9. Python vectorizing nested for loops - [7/2]
  10. Double loop takes time - [6/3]

September 24, 2016 07:49 AM

September 23, 2016

Simon Wittber

Python3 Asyncio PubSub Plaything


#!/usr/bin/env python
import io
import asyncio
import websockets
import logging
import collections

logger = logging.getLogger('websockets.server')
logger.setLevel(logging.ERROR)
logger.addHandler(logging.StreamHandler())
events = collections.defaultdict(lambda: set())
#-----------------------------------------------------------------------
async def handle_outgoing_queue(websocket):
    while websocket.open:
        msg = await websocket.outbox.get()
        await websocket.send(msg)
#-----------------------------------------------------------------------
async def pubsub(websocket, path):
    websocket.prefix = path.encode()
    websocket.outbox = asyncio.Queue()
    websocket.subscriptions = set()
    sender_task = asyncio.ensure_future(handle_outgoing_queue(websocket))
    while True:
        msg = await websocket.recv()
        if msg is None: break
        if isinstance(msg, str): msg = msg.encode()
        stream = io.BytesIO(msg)
        await handle_message(websocket, stream)
    sender_task.cancel()
    for name in websocket.subscriptions:
        try:
            events[name].remove(websocket)
        except KeyError:
            pass
#-----------------------------------------------------------------------
async def handle_message(websocket, stream):
    cmd = stream.readline().strip()
    name = websocket.prefix + stream.readline().strip()
    print(cmd, name);
    if cmd == b"SUB":
        events[name].add(websocket)
        websocket.subscriptions.add(name)
    elif cmd == b"UNS":
        subscribers = events[name]
        try:
            websocket.subscriptions.remove(name)
        except KeyError:
            pass
        try:
            subscribers.remove(websocket)
        except KeyError:
            pass
    elif cmd == b"PUB":
        stream.seek(0)
        msg = stream.read()
        for subscriber in events[name]:
            await subscriber.outbox.put(msg)
#-----------------------------------------------------------------------
start_server = websockets.serve(pubsub, '0.0.0.0', 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()


September 23, 2016 09:08 PM

Anwesha Das

See you PyLadies at PyCon India

PyCon India 2016 has started today. The Day 0 was the devsprint. Reached the venue late ( actually never thought would be able to attend due to my food pipe infection issue). Met friends which is one of the most attractive point of attending the conference apart from learning new things and other. Checked the Red Hat booth where we, PyLadies will be standing. Made preparations and plans regarding the set up. Discussed plans for the conference with other PyLadies. I have already asked them to give some lighting talks about what they have learnt and what do we do in PyLadies Pune since I have rebooted the PyLadies Pune Chapter.

Tomorrow there will be my talk on PyPI Licensing Scenario. This year's PyCon, we will be having a good number of PyLadies attending the conference. Meet us, Trishna Guha, Nisha Menon Poyarekar,Rupali Talwatkar,Pooja Yadav at PyCon beside Red Hat booth. This is for the first time I left Py(my 21 months old daughter) at home. Hopefully this PyCon will turn out to be a worthwhile experience. I know Nisha Menon Poyarekar has the same feeling. So, see you at PyCon India, 2016.

September 23, 2016 04:49 PM

Continuum Analytics News

Why Your Company Needs a Chief Data Science Officer

Company Blog
Friday, September 23, 2016
Michele Chambers
EVP Anaconda Business Unit & CMO
Continuum Analytics

This article was originally posted in CMSWire and has been edited for length and clarity. 

Ten years ago, the Chief Data Science Officer (CDSO) role was non-existent. It came into being when D.J. Patil was named the first US Chief Data Scientist by President Obama in 2015. A product of the Chief Technology Officer (CTO), who is responsible for focusing on scientific and technological matters within an organization including the company’s hardware and software, the CDSO takes technology within the enterprise to a whole new level. Companies have been motivated to get on the data train since 1990, when they began implementing big data collection.

However, making sense of the data was a challenge. With no dedicated person to own and manage the huge piles of datasets being collected, organizations began to flail and sink under the weight of all their data. It didn’t happen overnight, but data scientists and, subsequently, the role of the CDSO, came to life once companies realized that proper data analysis was key to finding correlations needed to spot business trends and, ultimately, exploit the power of big data to deliver value.

The CDSO role confirms the criticality of collecting data properly to capitalize on it and make certain it is stored securely in the event of a disaster or emergency (some businesses have yet to recover data following Hurricane Sandy).

Fast forward to 2016. Big data has exploded, but companies are still struggling with how best to organize around it — as an activity, a business function and a capability.

But what exactly can we achieve with it?

CDSOs (and their team of data scientists) are key to the skill set needed to apply analytics to their business, explain how to use data to create a competitive advantage and surpass competitors and understand how to find true value from data by acting on it.

Empowering Data Science Teams

Today, businesses are equipped with data science teams made up of a variety of roles––business analysts, machine learning experts, data engineers and more.

With the CDSO at the helm, the data science team can collaborate and centralize these skills, becoming a hub of intelligence and adding value to each business they serve. With a multifaceted perspective on data science as a whole, the CDSO allows for more innovative ideas and solutions for companies.

Staying Cost Efficient

It’s no secret that how businesses handle data has a direct impact on the bottom line. An interesting example occurred at DuPont, a company that defines itself as “a science company dedicated to solving challenging global problems” and is well known for its distribution of Corian solid surface countertops across the world. When asked if it believed it was covering its entire total addressable market (TAM), company executives were definitive in their response: a resounding yes.

Executives knew they had covered every region in the market and had great insight into analytics via distributors. What they hadn’t taken into consideration, however, was the vast amounts of data embedded within end-customer insights Without knowing exactly where the product was being installed — literally, DuPont had no insight into locations where it had not saturated.

DuPont took this information and created countertops that embedded sensors driven by Internet of Things (IoT) technology. By not simply relying on the data provided by its suppliers, DuPont seized the opportunity to increase its pool of knowledge significantly, by adding data science into its product.

This is just one example of how data science and the CDSO can implement previously non-existent processes and drive increased business intelligence in the most beneficial way –– with increased value to its re-sellers and a direct impact on revenue.

Changing the World

There is no room for doubt: it’s proven that innovation in the field of Open Data Science has led to the need for a CDSO to derive as much value from data as possible and help companies make an impact on the world.

John Deere, a 180-year-old company, is now revolutionizing farming by incorporating “smart farms.” Big data and IoT solutions allow farmers to make educated decisions based on real-time analysis of captured data. Giving this company the ability to put its data to good use, resulted in industry-wide and, in many areas, worldwide, positive changes — another reason why technology driven by the CDSO is an integral part of any organization.

The need for an executive-level decision maker proves to be an essential piece of the puzzle. The CDSO deserves a seat at the executive table to empower data science teams, drive cost efficiency and previously unimagined results and, most importantly, help companies change the world.

 

September 23, 2016 03:35 PM

Python Insider

Python Core Development Sprint 2016: 3.6 and beyond!

From September 5th to the 9th a group of Python core developers gathered for a sprint hosted at Instagram and sponsored by Instagram, Microsoft, and the Python Software Foundation. The goal was to spend a week working towards the Python 3.6b1 release, just in time for the Python 3.6 feature freeze on Monday, September 12, 2016. The inspiration for this sprint was the Need for Speed sprint held in Iceland a decade ago, where many performance improvements were made to Python 2.5. How time flies!


By any measurement, the sprint was extremely successful. All participants left feeling accomplished both in the work they did and in the discussions they held. Being in the same room encouraged many discussions related to various PEPs, and many design decisions were made. There was also the camaraderie of working in the same room together — typically most of us only see each other at the annual PyCon US, where there are other distractions that prevent getting to simply spend time together. (This includes the Python development sprints at PyCon, where the focus is more on helping newcomers contribute to Python — that’s why this sprint was not public.)


From a quantitative perspective, the sprint was the most productive week for Python ever! According to the graphs from the GitHub mirror of CPython, the week of September 4th saw more commits than the preceding 7 weeks combined! And in terms of issues, the number of open issues dropped by 62 with a total of 166 issues closed.


A large portion of the work performed during the sprint week revolved around various PEPs that had either not yet been accepted or were not yet fully implemented. In the end, 12 PEPs were either implemented from scratch or had their work completed during the week, out of Python 3.6’s total of 16 PEPs:


  1. Preserving the order of **kwargs in a function (PEP 468)
  2. Add a private version to dict (PEP 509)
  3. Underscores in Numeric Literals (PEP 515)
  4. Adding a file system path protocol (PEP 519)
  5. Preserving Class Attribute Definition Order (PEP 520)
  6. Adding a frame evaluation API to CPython (PEP 523)
  7. Make os.urandom() blocking on Linux, add os.getrandom() (PEP 524)
  8. Asynchronous Generators (PEP 525)
  9. Syntax for Variable Annotations (PEP 526)
  10. Change Windows console encoding to UTF-8 (PEP 528)
  11. Change Windows filesystem encoding to UTF-8 (PEP 529)
  12. Asynchronous Comprehensions (PEP 530)

Some large projects were also worked on that are not represented as PEPs. For instance, Python 3.6 now contains support for DTrace and SystemTap. This will give people more tools to introspect and monitor Python. See the HOWTO for usage instructions and examples showing some of the new possibilities.


CPython also gained a more memory efficient dictionary implementation at the sprint. The new implementation shrinks memory usage of dictionaries by about 25% and also preserves insertion order, without speed penalties. Based on a proposal by Raymond Hettinger, the patch was written by INADA Naoki prior to the sprint but it was reviewed and heavily discussed at the sprint, as changing the underlying implementation of dictionaries can have profound implications on Python itself. In the end, the patch was accepted, directly allowing for PEP 468 to be accepted and simplifying PEP 520.


Work was also done on the Gilectomy (see the presentation on the topic from PyCon US for more background info on the project). Progress was made such that Python would run without any reference counting turned on (i.e. Python turned into a huge memory leak). Work then was started on trying the latest design on how to turn reference counting back on in a way that would allow Python to scale with the number of threads and CPU cores used. There’s still a long road ahead before the Gilectomy will be ready to merge though, and we even jokingly considered branding the result as Python 4.


Much of the work done during the sprint led not only to improvements in the language and library, but to better performance as well. A quick performance comparison between Python 3.5.2+ and 3.6b1+ under OS X shows that 3.6 is generally faster, with double-digit speed improvements not uncommon. Similar benchmarking under Windows 10 has been reported to show similar performance gains.


A huge thanks goes out to the participants of the sprints! They are listed below in alphabetical order, along with thanks to the organizations that helped finance their attendance. Many of them traveled to attend and gave up the US Labor Day holiday with their families to participate. In the end, we had participants from 3 countries on 2 continents (We actually invited more people from other countries and continents, but not everybody invited could attend.)



September 23, 2016 02:11 PM

Omaha Python Users Group

September Wrap Up

Wes Turner did a great presentation demonstrating “HTML parsing, JSON-LD, RDFa, and schema.org in order to add Schema.org RDFa markup to @TheGlobalGoals for Sustainable Development at http://www.globalgoals.org/“.

https://github.com/westurner/pyglobalgoals/blob/develop/notebooks/globalgoals-pyglobalgoals.py.ipynb

And you can check here to see Wes’s other Python endeavors and tools:
https://westurner.org/

September 23, 2016 02:05 PM

Ian Ozsvald

Practical ML for Engineers talk at #pyconuk last weekend

Last weekend I had the pleasure of introducing Machine Learning for Engineers (a practical walk-through, no maths) [YouTube video] at PyConUK 2016. Each year the conference grows and maintains a lovely vibe, this year it was up to 600 people! My talk covered a practical guide to a 2 class classification challenge (Kaggle’s Titanic) with scikit-learn, backed by a longer Jupyter Notebook (github) and further backed by Ezzeri’s 2 hour tutorial from PyConUK 2014.

Debugging slide from my talk (thanks Olivia)

Debugging slide from my talk (thanks Olivia)

Topics covered include:

I had to cover the above in 20 minutes, obviously that was a bit of a push! I plan to cover this talk again at regional meetups, probably with 30-40 minutes. As it stands the talk (github) should lead you into the Notebook and that’ll lead you to Ezzeri’s 2 hour tutorial. This should be enough to help you start on your own 2 class classification challenge, if your data looks ‘somewhat like’ the Titanic data.

I’m generally interested in the idea of helping more engineers get into data science and machine learning. If you’re curious – I have a longer set of notes called Data Science Delivered and some vague plans to maybe write a book (maybe) – for the book join the mailing list here if you’d like to hear more (no hard sell, almost no emails at the moment, I’m still figuring out if I should do this).

You might also want to follow-up on Katharine Jarmul’s data wrangling talk and tutorial, Nick Radcliffe’s Test Driven Data Analysis (with new automated TDD-for-data tool to come in a few months), Tim Vivian-Griffiths’ SVM Diagnostics, Dr. Gusztav Belteki’s Ventilator medical talk, Geoff French’s Deep Learning tutorial and Marco Bonzanini and Miguel ‘s Intro to ML tutorial. The videos are probably in this list.

If you like the above then do think on coming to our monthly PyDataLondon data science meetups near London Bridge.

PyConUK itself has grown amazingly – the core team put in a huge amount of effort. It was very cool to see the growth of the kids sessions, the trans track, all the tutorials and the general growth in the diversity of our community’s membership. I was quite sad to leave at lunch on the Sunday – next year I plan to stay longer, this community deserves more investment. If you’ve yet to attend a PyConUK then I strongly urge you to think on submitting a talk for next year and definitely suggest that you attend.

The organisers were kind enough to let Kat and myself do a book signing, I suggest other authors think on joining us next year. Attendees love meeting authors and it is yet another activity that helps bind the community together.

Book signing at PyConUK

Book signing at PyConUK

Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

September 23, 2016 11:28 AM

Talk Python to Me

#77 20 Python Libraries You Aren't Using (But Should)

Many of you write to me and tell me how you appreciate the way my guests and I highlight a particular Python package at the end of each episode. Well if you enjoy that little segment, you're going to love this episode. <br/> <br/> This week you'll meet Caleb Hattingh who wrote a great book called 20 Python Libraries You Aren't Using (But Should). He and I spend an hour digging into all the very powerful and interesting packages that you probably haven't heard of but will be super excited to use after you learn about them. <br/> <br/> Links from the show: <br/> <div style="font-size: .85em;"> <br/> <b>Caleb on twitter</b>: <a href='https://twitter.com/caleb_hattingh' target='_blank'>@caleb_hattingh</a> <br/> <b>Book: 20 Python Libraries You Aren't Using (But Should)</b>: <br/> <a href='http://www.oreilly.com/programming/free/20-python-libraries-you-arent-using-but-should.csp ' target='_blank'>oreilly.com/programming/free/20-python-libraries-you-arent-using-but-should.csp</a> <br/> <b>Learning Cython course</b>: <a href='http://shop.oreilly.com/product/0636920046813.do' target='_blank'>shop.oreilly.com/product/0636920046813.do</a> <br/> <b>Python-specific Slack group online (~ 2.5k members)</b>: <a href='http://pythondevelopers.herokuapp.com/' target='_blank'>pythondevelopers.herokuapp.com</a> <br/> </div>

September 23, 2016 08:00 AM

Daniel Bader

How do I make my own command line commands using Python?

How do I make my own command line commands using Python?

The Python script you just wrote would make a great little command line tool – but having to type “python myscript.py” all the time to launch your program gets daunting fast. Here’s how you can make your Python script feel like a real shell command.

At the end of this short tutorial you’ll know how to write a Python program that simulates the Unix echo command and can be started from the command line just like it:

$ myecho Hello, World!

Sweet, let’s jump right in!

Imagine you have the following short Python script called myecho.py that just prints the command line arguments you pass to it back to the console:

import sys

for arg in sys.argv:
    print(arg)

You can run this command just fine by passing it to the Python interpreter like so:

$ python myecho.py Hello, World!
myecho.py
Hello,
World!

But how can you give your users a more polished experience that allows them simply type myecho Hello, World! and get the same result?

Easy – there are three things you need to do:

#1. Mark your Python file as executable

The first thing you’ll need to do is mark your script as executable in the file system, like so:

$ chmod +x myecho.py

This sets the executable flag on myecho.py, which tells the shell that it’s a program that can be run directly from the command line. Let’s try it:

$ ./myecho.py Hello, World!

We need to prefix our command with ./ because usually the current directory is not included in the PATH environment variable on Unix. This is a security feature1.

Anyway – the result will be that you get a crazy error message when you try to run myecho.py. It’ll probably look something like this:

./myecho.py: line 4: syntax error near unexpected token `print'
./myecho.py: line 4: `    print(arg)'

The reason for that is that now the system doesn’t know it’s supposed to execute a Python script. So instead it takes a wild guess and tries to run your Python script like a shell script with the /bin/sh interpreter.

That’s why you’re getting these odd syntax errors. But there’s an easy fix for this in the next step. You just need to …

#2. Add an interpreter “shebang”

Okay, admittedly this sounds completely crazy if you’ve never heard of Unix shebangs before…😃 But it’s actually a really simple concept and super useful:

Whenever you run a script file on an Unix-like operating system (like Linux or macOS) the program loader responsible for loading and executing your script checks the first line for an interpreter directive. Here’s an example:

#!/bin/sh

You’ve probably seen those before. These interpreter directives are also called shebangs in Unix jargon2. They tell the program loader which interpreter should execute the script.

You can use this mechanism to your advantage by adding a shebang line that points to the system Python interpreter:

#!/usr/bin/env python

You may be wondering why you should be using env to load the Python interpreter instead of simply using an absolute path like /usr/local/bin/python.

The reason for that is that the Python interpreter will be installed in different locations on different systems. On a Mac using Homebrew it might be in /usr/local/bin/python. On a Ubuntu Linux box it might be in /usr/bin/python.

Using another level of indirection through env you can select the Python interpreter that’s on the PATH environment variable. That’s usually the right way to go about it3.

Okay, so now that you’ve added that #!/usr/bin/env python line your script should look like this:

#!/usr/bin/env python
import sys

for arg in sys.argv:
    print(arg)

Let’s try to run it again!

$ ./myecho.py Hello, World!
./myecho.py
Hello,
World!

Yes! Success!

Now that you’re using the interpreter directive shebang in the script you can also drop the .py extension. This will make your script look even more like a system tool:

$ mv myecho.py myecho

This is starting to look pretty good now:

$ ./myecho Hello, World!
./myecho
Hello,
World!

#3. Make sure your program is on the PATH

The last thing you need to change to make your Python script really seem like a shell command or system tool is to make sure it’s on your PATH.

That way you’ll be able to launch it from any directory by simply running myecho Hello, World!, just like the “real” echo command.

Here’s how to achieve that.

I don’t recommend that you try to copy your script to a system directory like /usr/bin/ or /usr/local/bin because that can lead to all kinds of odd naming conflicts (and, in the worst case, break your operating system install).

So instead, what you’ll want to do is to create a bin directory in your user’s home directory and then add that to the PATH.

First, you need to create the ~/bin directory:

$ mkdir -p ~/bin

Next, copy your script to ~/bin:

$ cp myecho ~/bin

Finally, add ~/bin to your PATH:

export PATH=$PATH":$HOME/bin"

Adding ~/bin to the PATH like this is only temporary, however. It won’t stick across terminal sessions or system restarts. If you want to make your command permanently available on a system, do the following:

Okay great, now we finally get the result we wanted all along:

$ myecho Hello, World!
/Users/youruser/bin/myecho
Hello,
World!

Whew. When I write these tutorials I’m always surprised how much work seemingly “simple” things take when you’re trying to get to the bottom of them. So don’t be too hard on yourself if some of these steps felt a little arcane at first 😃. All of this stuff becomes second nature once you’ve dealt with it a few times.

  1. Here’s an awesome in-depth explanation for this security feature and why it is necessary: www.linfo.org/dot_slash.html ↩

  2. You can read all about Unix shebangs here: wikipedia.org/wiki/Shebang_(Unix) ↩

  3. Learn more about env and its merits here: wikipedia.org/wiki/Env ↩

September 23, 2016 12:00 AM

hypothesis.works articles

3.5.0 and 3.5.1 Releases of Hypothesis for Python

This is a combined release announcement for two releases. 3.5.0 was released yesterday, and 3.5.1 has been released today after some early bug reports in 3.5.0

Changes

3.5.0 - 2016-09-22

This is a feature release.

Additionally there have been some minor bug fixes:

3.5.1 - 2016-09-23

This is a bug fix release.

Notes

Aside from the above changes, there are a couple big things behind the scenes of this release that make it a big deal.

The first is that the flagship chunk of work, statistics, is a long-standing want to have that has never quite been prioritised. By funding it, Jean-Louis and Adfinis-SyGroup successfully bumped it up to the top of the priority list, making it the first funded feature in Hypothesis for Python!

Another less significant but still important is that this release marks the first real break with an unofficial Hypothesis for Python policy of not having any dependencies other than the standard library and backports. This release adds a dependency on the uncompyle6 package. This may seem like an odd choice, but it was invaluable for fixing the repr behaviour, which in turn was really needed for providing good statistics for filter and recursive strategies.

Read more...

September 23, 2016 12:00 AM

September 22, 2016

Dataquest

Learn Python the right way in 5 steps

Python is an amazingly versatile programming language. You can use it to build websites, machine learning algorithms, and even autonomous drones. A huge percentage of programmers in the world use Python, and for good reason. It gives you the power to create almost anything. But – and this is a big but – you have to learn it first. Learning any programming language can be intimidating. I personally think that Python is better to learn than most, but learning it was still a rocky journey for me.

One of the things that I found most frustrating when I was learning Python was how generic all the learning resources were. I wanted to learn how to make websites using Python, but it seemed like every learning resource wanted me to spend 2 long, boring, months on Python syntax before I could even think about doing what interested me.

This mismatch made learning Python quite intimidating for me. I put it off for months. I got a couple of lessons into the Codecademy tutorials, then stopped. I looked at Python code, but it was foreign and confusing:

from django.http import HttpResponse def...

September 22, 2016 04:51 PM

Semaphore Community

Dockerizing a Python Django Web Application

This article is brought with ❤ to you by Semaphore.

Introduction

This article will cover building a simple 'Hello World'-style web application written in Django and running it in the much talked about and discussed Docker. Docker takes all the great aspects of a traditional virtual machine, e.g. a self contained system isolated from your development machine, and removes many of the drawbacks such as system resource drain, setup time, and maintenance.

When building web applications, you have probably reached a point where you want to run your application in a fashion that is closer to your production environment. Docker allows you to set up your application runtime in such a way that it runs in exactly the same manner as it will in production, on the same operating system, with the same environment variables, and any other configuration and setup you require.

By the end of the article you'll be able to:

What is Docker, Anyway?

Docker's homepage describes Docker as follows:

"Docker is an open platform for building, shipping and running distributed applications. It gives programmers, development teams, and operations engineers the common toolbox they need to take advantage of the distributed and networked nature of modern applications."

Put simply, Docker gives you the ability to run your applications within a controlled environment, known as a container, built according to the instructions you define. A container leverages your machines resources much like a traditional virtual machine (VM). However, containers differ greatly from traditional virtual machines in terms of system resources. Traditional virtual machines operate using Hypervisors, which manage the virtualization of the underlying hardware to the VM. This means they are large in terms of system requirements.

Containers operate on a shared Linux operating system base and add simple instructions on top to execute and run your application or process. The difference being that Docker doesn't require the often time-consuming process of installing an entire OS to a virtual machine such as VirtualBox or VMWare. Once Docker is installed, you create a container with a few commands and then execute your applications on it via the Dockerfile. Docker manages the majority of the operating system virtualization for you, so you can get on with writing applications and shipping them as you require in the container you have built. Furthermore, Dockerfiles can be shared for others to build containers and extend the instructions within them by basing their container image on top of an existing one. The containers are also highly portable and will run in the same manner regardless of the host OS they are executed on. Portability is a massive plus side of Docker.

Prerequisites

Before you begin this tutorial, ensure the following is installed to your system:

Setting Up a Django web application

Starting a Django application is easy, as the Django dependency provides you with a command line tool for starting a project and generating some of the files and directory structure for you. To start, create a new folder that will house the Django application and move into that directory.

$ mkdir project
$ cd project

Once in this folder, you need to add the standard Python project dependencies file which is usually named requirements.txt, and add the Django and Gunicorn dependency to it. Gunicorn is a production standard web server, which will be used later in the article. Once you have created and added the dependencies, the file should look like this:

$ cat requirements.txt                                                              
Django==1.9.4
gunicorn==19.6.0

With the Django dependency added, you can then install Django using the following command:

$ pip install -r requirements.txt

Once installed, you will find that you now have access to the django-admin command line tool, which you can use to generate the project files and directory structure needed for the simple "Hello, World!" application.

$ django-admin startproject helloworld

Let's take a look at the project structure the tool has just created for you:

.
├── helloworld
│   ├── helloworld
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   └── manage.py
└── requirements.txt

You can read more about the structure of Django on the official website. django-admin tool has created a skeleton application. You control the application for development purposes using the manage.py file, which allows you to start the development test web server for example:

$ cd helloworld
$ python manage.py runserver

The other key file of note is the urls.py, which specifies what URL's route to which view. Right now, you will only have the default admin URL which we won't be using in this tutorial. Lets add a URL that will route to a view returning the classic phrase "Hello, World!".

First, create a new file called views.py in the same directory as urls.py with the following content:

from django.http import HttpResponse

def index(request):
    return HttpResponse("Hello, world!")

Now, add the following URL url(r'', 'helloworld.views.index') to the urls.py, which will route the base URL of / to our new view. The contents of the urls.py file should now look as follows:

from django.conf.urls import url
from django.contrib import admin

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'', 'helloworld.views.index'),
]

Now, when you execute the python manage.py runserver command and visit http://localhost:8000 in your browser, you should see the newly added "Hello, World!" view.

The final part of our project setup is making use of the Gunicorn web server. This web server is robust and built to handle production levels of traffic, whereas the included development server of Django is more for testing purposes on your local machine only. Once you have dockerized the application, you will want to start up the server using Gunicorn. This is much simpler if you write a small startup script for Docker to execute. With that in mind, let's add a start.sh bash script to the root of the project, that will start our application using Gunicorn.

#!/bin/bash

# Start Gunicorn processes
echo Starting Gunicorn.
exec gunicorn helloworld.wsgi:application \
    --bind 0.0.0.0:8000 \
    --workers 3

The first part of the script writes "Starting Gunicorn" to the command line to show us that it is starting execution. The next part of the script actually launches Gunicorn. You use exec here so that the execution of the command takes over the shell script, meaning that when the Gunicorn process ends so will the script, which is what we want here.

You then pass the gunicorn command with the first argument of helloworld.wsgi:application. This is a reference to the wsgi file Django generated for us and is a Web Server Gateway Interface file which is the Python standard for web applications and servers. Without delving too much into WSGI, the file simply defines the application variable, and Gunicorn knows how to interact with the object to start the web server.

You then pass two flags to the command, bind to attach the running server to port 8000, which you will use to communicate with the running web server via HTTP. Finally, you specify workers which are the number of threads that will handle the requests coming into your application. Gunicorn recommends this value to be set at (2 x $num_cores) + 1. You can read more on configuration of Gunicorn in their documentation.

Finally, make the script executable, and then test if it works by changing directory into the project folder helloworld and executing the script as shown here. If everything is working fine, you should see similar output to the one below, be able to visit http://localhost:8000 in your browser, and get the "Hello, World!" response.

$ chmod +x start.sh
$ cd helloworld
$ ../start.sh
Starting Gunicorn.
[2016-06-26 19:43:28 +0100] [82248] [INFO]
Starting gunicorn 19.6.0
[2016-06-26 19:43:28 +0100] [82248] [INFO]
Listening at: http://0.0.0.0:8000 (82248)
[2016-06-26 19:43:28 +0100] [82248] [INFO]
Using worker: sync
[2016-06-26 19:43:28 +0100] [82251] [INFO]
Booting worker with pid: 82251
[2016-06-26 19:43:28 +0100] [82252] [INFO]
Booting worker with pid: 82252
[2016-06-26 19:43:29 +0100] [82253] [INFO]
Booting worker with pid: 82253

Dockerizing the Application

You now have a simple web application that is ready to be deployed. So far, you have been using the built-in development web server that Django ships with the web framework it provides. It's time to set up the project to run the application in Docker using a more robust web server that is built to handle production levels of traffic.

Installing Docker

One of the key goals of Docker is portability, and as such is able to be installed on a wide variety of operating systems.

For this tutorial, you will look at installing Docker Machine on MacOS. The simplest way to achieve this is via the Homebrew package manager. Instal Homebrew and run the following:

$ brew update && brew upgrade --all && brew cleanup && brew prune
$ brew install docker-machine

With Docker Machine installed, you can use it to create some virtual machines and run Docker clients. You can run docker-machine from your command line to see what options you have available. You'll notice that the general idea of docker-machine is to give you tools to create and manage Docker clients. This means you can easily spin up a virtual machine and use that to run whatever Docker containers you want or need on it.

You will now create a virtual machine based on VirtualBox that will be used to execute your Dockerfile, which you will create shortly. The machine you create here should try to mimic the machine you intend to run your application on in production. This way, you should not see any differences or quirks in your running application neither locally nor in a deployed environment.

Create your Docker Machine using the following command:

$ docker-machine create development --driver virtualbox
--virtualbox-disk-size "5000" --virtualbox-cpu-count 2
--virtualbox-memory "4096"

This will create your machine and output useful information on completion. The machine will be created with 5GB hard disk, 2 CPU's and 4GB of RAM.

To complete the setup, you need to add some environment variables to your terminal session to allow the Docker command to connect the machine you have just created. Handily, docker-machine provides a simple way to generate the environment variables and add them to your session:

$ docker-machine env development
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://123.456.78.910:1112"
export DOCKER_CERT_PATH="/Users/me/.docker/machine/machines/development"
export DOCKER_MACHINE_NAME="development"
# Run this command to configure your shell:
# eval "$(docker-machine env development)"

Complete the setup by executing the command at the end of the output:

$(docker-machine env development)

Execute the following command to ensure everything is working as expected.

$ docker images
REPOSITORY   TAG   IMAGE  ID   CREATED   SIZE

You can now dockerize your Python application and get it running using the docker-machine.

Writing the Dockerfile

The next stage is to add a Dockerfile to your project. This will allow Docker to build the image it will execute on the Docker Machine you just created. Writing a Dockerfile is rather straightforward and has many elements that can be reused and/or found on the web. Docker provides a lot of the functions that you will require to build your image. If you need to do something more custom on your project, Dockerfiles are flexible enough for you to do so.

The structure of a Dockerfile can be considered a series of instructions on how to build your container/image. For example, the vast majority of Dockerfiles will begin by referencing a base image provided by Docker. Typically, this will be a plain vanilla image of the latest Ubuntu release or other Linux OS of choice. From there, you can set up directory structures, environment variables, download dependencies, and many other standard system tasks before finally executing the process which will run your web application.

Start the Dockerfile by creating an empty file named Dockerfile in the root of your project. Then, add the first line to the Dockerfile that instructs which base image to build upon. You can create your own base image and use that for your containers, which can be beneficial in a department with many teams wanting to deploy their applications in the same way.

# Dockerfile

# FROM directive instructing base image to build upon
FROM python:2-onbuild

It's worth noting that we are using a base image that has been created specifically to handle Python 2.X applications and a set of instructions that will run automatically before the rest of your Dockerfile. This base image will copy your project to /usr/src/app, copy your requirements.txt and execute pip install against it. With these tasks taken care of for you, your Dockerfile can then prepare to actually run your application.

Next, you can copy the start.sh script written earlier to a path that will be available to you in the container to be executed later in the Dockerfile to start your server.

# COPY startup script into known file location in container
COPY start.sh /start.sh

Your server will run on port 8000. Therefore, your container must be set up to allow access to this port so that you can communicate to your running server over HTTP. To do this, use the EXPOSE directive to make the port available:

# EXPOSE port 8000 to allow communication to/from server
EXPOSE 8000

The final part of your Dockerfile is to execute the start script added earlier, which will leave your web server running on port 8000 waiting to take requests over HTTP. You can execute this script using the CMD directive.

# CMD specifcies the command to execute to start the server running.
CMD ["/start.sh"]
# done!

With all this in place, your final Dockerfile should look something like this:

# Dockerfile

# FROM directive instructing base image to build upon
FROM python:2-onbuild

# COPY startup script into known file location in container
COPY start.sh /start.sh

# EXPOSE port 8000 to allow communication to/from server
EXPOSE 8000

# CMD specifcies the command to execute to start the server running.
CMD ["/start.sh"]
# done!

You are now ready to build the container image, and then run it to see it all working together.

Building and Running the Container

Building the container is very straight forward once you have Docker and Docker Machine on your system. The following command will look for your Dockerfile and download all the necessary layers required to get your container image running. Afterwards, it will run the instructions in the Dockerfile and leave you with a container that is ready to start.

To build your container, you will use the docker build command and provide a tag or a name for the container, so you can reference it later when you want to run it. The final part of the command tells Docker which directory to build from.

$ cd <project root directory>
$ docker build -t davidsale/dockerizing-python-django-app .

Sending build context to Docker daemon 237.6 kB
Step 1 : FROM python:2-onbuild
# Executing 3 build triggers...
Step 1 : COPY requirements.txt /usr/src/app/
 ---> Using cache
Step 1 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Using cache
Step 1 : COPY . /usr/src/app
 ---> 68be8680cbc4
Removing intermediate container 75ed646abcb6
Step 2 : COPY start.sh /start.sh
 ---> 9ef8e82c8897
Removing intermediate container fa73f966fcad
Step 3 : EXPOSE 8000
 ---> Running in 14c752364595
 ---> 967396108654
Removing intermediate container 14c752364595
Step 4 : WORKDIR helloworld
 ---> Running in 09aabb677b40
 ---> 5d714ceea5af
Removing intermediate container 09aabb677b40
Step 5 : CMD /start.sh
 ---> Running in 7f73e5127cbe
 ---> 420a16e0260f
Removing intermediate container 7f73e5127cbe
Successfully built 420a16e0260f

In the output, you can see Docker processing each one of your commands before outputting that the build of the container is complete. It will give you a unique ID for the container, which can also be used in commands alongside the tag.

The final step is to run the container you have just built using Docker:

$ docker run -it -p 8000:8000 davidsale/djangoapp1
Starting Gunicorn.
[2016-06-26 19:24:11 +0000] [1] [INFO]
Starting gunicorn 19.6.0
[2016-06-26 19:24:11 +0000] [1] [INFO]
Listening at: http://0.0.0.0:9077 (1)
[2016-06-26 19:24:11 +0000] [1] [INFO]
Using worker: sync
[2016-06-26 19:24:11 +0000] [11] [INFO]
Booting worker with pid: 11
[2016-06-26 19:24:11 +0000] [12] [INFO]
Booting worker with pid: 12
[2016-06-26 19:24:11 +0000] [17] [INFO]
Booting worker with pid: 17

The command tells Docker to run the container and forward the exposed port 8000 to port 8000 on your local machine. After you run this command, you should be able to visit http://localhost:8000 in your browser to see the "Hello, World!" response. If you were running on a Linux machine, that would be the case. However, if running on MacOS, then you will need to forward the ports from VirtualBox, which is the driver we use in this tutorial so that they are accessible on your host machine.

$ VBoxManage controlvm "development" natpf1
  "tcp-port8000,tcp,,8000,,8000";

This command modifies the configuration of the virtual machine created using docker-machine earlier to forward port 8000 to your host machine. You can run this command multiple times changing the values for any other ports you require.

Once you have done this, visit http://localhost:8000 in your browser. You should be able to visit your dockerized Python Django application running on a Gunicorn web server, ready to take thousands of requests a second and ready to be deployed on virtually any OS on planet using Docker.

Next Steps

After manually verifying that the appication is behaving as expected in Docker, the next step is the deployment. You can use Semaphore's Docker platform for automating this process.

Conclusion

In this tutorial, you have learned how to build a simple Python Django web application, wrap it in a production grade web server, and created a Docker container to execute your web server process.

If you enjoyed working through this article, feel free to share it and if you have any questions or comments leave them in the section below. We will do our best to answer them, or point you in the right direction.

This article is brought with ❤ to you by Semaphore.

September 22, 2016 04:16 PM

Import Python

ImportPython Issue 91 - asynq from quora, python packaging ecosystem and more


Worthy Read

I will be attending Pycon India 2016 in Delhi
importpython
Hey guys, this is Ankur. Curator behind ImportPython. Will be attending PyconIndia. Happy to meet you all and discuss all things Python. Get your opinion on the newsletter, How to make it better ?. Ping me on ankur at outlook dot com or just reply to this email. I will respond back. See you there.

Asynchronous Programming in Python at Quora and asynq
async-io
asynq is a library for asynchronous programming in Python with a focus on batching requests to external services. It also provides seamless interoperability with synchronous code, support for asynchronous context managers, and tools to make writing and testing asynchronous code easier. asynq was developed at Quora and is a core component of Quora's architecture. See the original blog post here.

Python has come a long way. So has job hunting.

Try Hired and get in front of 4,000+ companies with one application. No more pushy recruiters, no more dead end applications and mismatched companies, Hired puts the power in your hands.
Sponsor

The Python Packaging Ecosystem - Nick Coghlan
packaging
There have been a few recent articles reflecting on the current status of the Python packaging ecosystem from an end user perspective, so it seems worthwhile for me to write-up my perspective as one of the lead architects for that ecosystem on how I characterise the overall problem space of software publication and distribution, where I think we are at the moment, and where I'd like to see us go in the future.

Deploying modern Python apps to ancient infrastructure with pkgsrc
infrastructure
This team is responsible for supplying a variety of web apps built on a modern stack (mostly Celery, Django, nginx and Redis), but have almost no control over the infrastructure on which it runs, and boy, is some of that infrastructure old and stinky. We have no root access to these servers, most software configuration requires a ticket with a lead time of 48 hours plus, and the watchful eyes of a crusty old administrator and obtuse change management process. The machines are so old that many are still running on real hardware, and those that are VMs still run some ancient variety of Red Hat Linux, with, if we’re lucky, Python 2.4 installed.

Making publication ready Python notebooks
ipython
The notebook functionality of Python provides a really amazing way of analyzing data and writing reports in one place. However in the standard configuration, the pdf export of the Python notebook is somewhat ugly and unpractical. In the following I will present my choices to create almost publication ready reports from within IPython/Jupyter notebook.

SF Pybay conference videos
Paul Bailey, "A Guide to Bad Programming", at PyBay2016 was my fav talk amongst all. Check out the youtube channel.

Compressing and enhancing hand-written notes
image processing
I wrote a program to clean up scans of handwritten notes while simultaneously reducing file size. Some of my classes don’t have an assigned textbook. For these, I like to appoint weekly “student scribes” to share their lecture notes with the rest of the class, so that there’s some kind written resource for students to double-check their understanding of the material. The notes get posted to a course website as PDFs.

Image Manipulation in Python - Tutorial
image processing
This tutorial will show you how to transform an image with different filters and techniques to deliver different outputs. These methods are still in use and part of a process known as Computer-To-Plate (CTP), used to create a direct output from an image file to a photographic film or plate (depending on the process). Note - It's a pretty good article that makes uses of Python 3, Pillow and is well written.

Weekly Python Chat: Tips for learning Django
video
This is a Weekly Python Chat live video chat events. These events are hosted by Trey Hunner. This week Melanie Crutchfield and he are going to chat about things you'll wish you knew earlier when making your first website with Django. Much watch for newbies building websites in Django.

2 Factor Authentication in Django
security
If you are looking to implement 2 Factor Authentication as part of your product and don't know where to start read this.



Upcoming Conference / User Group Meet







Projects

streamlink - 59 Stars, 10 Fork
CLI for extracting streams from various websites to video player of your choosing

encore.ai - 40 Stars, 6 Fork
Generate new lyrics in the style of any artist using LSTMs and TensorFlow

September 22, 2016 10:41 AM

PayPal Engineering Blog

Python by the C side

C shells by the C shoreMahmoud’s note: This will be my last post on the PayPal Engineering blog. If you’ve enjoyed this sort of content subscribe to my blog/pythondoeswhat.com or follow me on Twitter. It’s been fun!

All the world is legacy code, and there is always another, lower layer to peel away. These realities cause developers around the world to go on regular pilgrimage, from the terra firma of Python to the coasts of C. From zlib to SQLite to OpenSSL, whether pursuing speed, efficiency, or features, the waters are powerful, and often choppy. The good news is, when you’re writing Python, C interactions can be a day at the beach.

 

A brief history

As the name suggests, CPython, the primary implementation of Python used by millions, is written in C. Python core developers embraced and exposed Python’s strong C roots, taking a traditional tack on portability, contrasting with the “write once, debug everywhere” approach popularized elsewhere. The community followed suit with the core developers, developing several methods for linking to C. Years of these interactions have made Python a wonderful environment for interfacing with operating systems, data processing libraries, and everything the C world has to offer.

This has given us a lot of choices, and we’ve tried all of the standouts:

Approach Vintage Representative User Notable Pros Notable Cons
C extension modules 1991 Standard library Extensive documentation and tutorials. Total control. Compilation, portability, reference management. High C knowledge.
SWIG 1996 crfsuite Generate bindings for many languages at once Excessive overhead if Python is the only target.
ctypes 2003 oscrypto No compilation, wide availability Accessing and mutating C structures cumbersome and error prone.
Cython 2007 gevent, kivy Python-like. Highly mature. High performance. Compilation, new syntax and toolchain.
cffi 2013 cryptography, pypy Ease of integration, PyPy compatibility New/High-velocity.

There’s a lot of history and detail that doesn’t fit into a table, but every option falls into one of three categories:

  1. Writing C
  2. Writing code that translates to C
  3. Writing code that calls into libraries that present a C interface

Each has its merits, so we’ll explore each category, then finish with a real, live, worked example.

Writing C

Python’s core developers did it and so can you. Writing C extensions to Python gives an interface that fits like a glove, but also requires knowing, writing, building, and debugging C. The bugs are much more severe, too, as a segmentation fault that kills the whole process is much worse than a Python exception, especially in an asynchronous environment with hundreds of requests being handled within the same process. Not to mention that the glove is also tailored to CPython, and won’t fit quite right, or at all, in other execution environments.

At PayPal, we’ve used C extensions to speed up our service serialization. And while we’ve solved the build and portability issue, we’ve lost track of our share of references and have moved on from writing straight C extensions for new code.

Translating to C

After years of writing C, certain developers decide that they can do better. Some of them are certainly onto something.

Going Cythonic

Cython is a superset of the Python programming language that has been turning type-annotated Python into C extensions for nearly a decade, longer if you count its predecessor, Pyrex. Apart from its maturity, the points that matters to us are:

Not to mention that the generated C often makes use of performance tricks that are too tedious or arcane to write by hand, partially motivated by scientific computing’s constant push. And through all that, Cython code maintains a high level of integration with Python itself, right down to the stack trace and line numbers.

PayPal has certainly benefitted from their efforts through high-performance Cython users like gevent, lxml, and NumPy. While our first go with Cython didn’t stick in 2011, since 2015, all native extensions have been written and rewritten to use Cython. It wasn’t always this way however.

A sip, not a SWIG

An early contributor to Python at PayPal got us started using SWIG, the Simplified Wrapper and Interface Generator, to wrap PayPal C++ infrastructure. It served its purpose for a while, but every modification was a slog compared to more Pythonic techniques. It wasn’t long before we decided it wasn’t our cup of tea.

Long ago SWIG may have rivaled extension modules as Python programmers’ method of choice. These days it seems to suit the needs of C library developers looking for a fast and easy way to wrap their C bindings for multiple languages. It also says something that searching for SWIG usage in Python nets as much SWIG replacement libraries as SWIG usage itself.

Calling into C

So far all our examples have involved extra build steps, portability concerns, and quite a bit of writing languages other than Python. Now we’ll dig into some approaches that more closely match Python’s own dynamic nature: ctypes and cffi.

Both ctypes and cffi leverage C’s Foreign Function Interface (FFI), a sort of low-level API that declares callable entrypoints to compiled artifacts like shared objects (.so files) on Linux/FreeBSD/etc. and dynamic-link libraries (.dll files) on Windows. Shared objects take a bit more work to call, so ctypes and cffi both use libffi, a C library that enables dynamic calls into other C libraries.

Shared libraries in C have some gaps that libffi helps fill. A Linux .so, Windows .dll, or OS X .dylib is only going to provide symbols: a mapping from names to memory locations, usually function pointers. Dynamic linkers do not provide any information about how to use these memory locations. When dynamically linking shared libraries to C code, header files provide the function signatures; as long as the shared library and application are ABI compatible, everything works fine. The ABI is defined by the C compiler, and is usually carefully managed so as not to change too often.

However, Python is not a C compiler, so it has no way to properly call into C even with a known memory location and function signature. This is where libffi comes in. If symbols define where to call the API, and header files define what API to call, libffi translates these two pieces of information into how to call the API. Even so, we still need a layer above libffi that translates native Python types to C and vice versa, among other tasks.

ctypes

ctypes is an early and Pythonic approach to FFI interactions, most notable for its inclusion in the Python standard library.

ctypes works, it works well, and it works across CPython, PyPy, Jython, IronPython, and most any Python runtime worth its salt. Using ctypes, you can access C APIs from pure Python with no external dependencies. This makes it great for scratching that quick C itch, like a Windows API that hasn’t been exposed in the os module. If you have an otherwise small module that just needs to access one or two C functions, ctypes allows you to do so without adding a heavyweight dependency.

For a while, PayPal Python code used ctypes after moving off of SWIG. We found it easier to call into vanilla shared objects built from C++ with an extern C rather than deal with the SWIG toolchain. ctypes is still used incidentally throughout the code for exactly this: unobtrusively calling into certain shared objects that are widely deployed. A great open-source example of this use case is oscrypto, which does exactly this for secure networking. That said, ctypes is not ideal for huge libraries or libraries that change often. Porting signatures from headers to Python code is tedious and error-prone.

cffi

cffi, our most modern approach to C integration, comes out of the PyPy project. They were seeking an approach that would lend itself to the optimization potential of PyPy, and they ended up creating a library that fixes many of the pains of ctypes. Rather than handcrafting Python representations of the function signatures, you simply load or paste them in from C header files.

For all its convenience, cffi’s approach has its limits. C is really almost two languages, taking into account preprocessor macros. A macro performs string replacement, which opens a Fun World of Possibilities, as straightforward or as complicated as you can imagine. cffi’s approach is limited around these macros, so applicability will depend on the library with which you are integrating.

On the plus side, cffi does achieve its stated goal of outperforming ctypes under PyPy, while remaining comparable to ctypes under CPython. The project is still quite young, and we are excited to see where it goes next.

A Tale of 3 Integrations: PKCS11

We promised an example, and we almost made it three.

PKCS11 is a cryptography standard for interacting with many hardware and software security systems. The 200-plus-page core specification includes many things, including the official client interface: A large set of C header-style information. There are a variety of pre-existing bindings, but each device has its own vendor-specific quirks, so what are we waiting for?

Metaprogramming

As stated earlier, ctypes is not great for sprawling interfaces. The drudgery of converting function signatures invites transcription bugs. We somewhat automated it, but the approach was far from perfect.

Our second approach, using cffi, worked well for our first version’s supported feature subset, but unfortunately PKCS11 uses its own CK_DECLARE_FUNCTION macro instead of regular C syntax for defining functions. Therefore, cffi’s approach of skipping #define macros will result in syntactically invalid C code that cannot be parsed. On the other hand, there are other macro symbols which are compiler or operating system intrinsics (e.g. __cplusplus, _WIN32, __linux__). So even if cffi attempted to evaluate every macro, we would immediately runs into problems.

So in short, we’re faced with a hard problem. The PKCS11 standard is a gnarly piece of C. In particular:

  1. Many hundreds of important constant values are created with #define
  2. Macros are defined, then re-defined to something different later on in the same file
  3. pkcs11f.h is included multiple times, even once as the body of a struct

In the end, the solution that worked best was to write up a rigorous parser for the particular conventions used by the slow-moving standard, generate Cython, which generates C, which finally gives us access to the complete client, with the added performance bonus in certain cases. Biting this bullet took all of a day and a half, we’ve been very satisfied with the result, and it’s all thanks to a special trick up our sleeves.

Parsing Expression Grammars

Parsing expression grammars (PEGs) combine the power of a true parser generating an abstract syntax tree, not unlike the one used for Python itself, with the convenience of regular expressions. One might think of PEGs as recursive regular expressions. There are several good libraries for Python, including parsimonious and parsley. We went with the former for its simplicity.

For this application, we defined a two grammars, one for pkcs11f.h and one for pkcs11t.h:

PKCS11F GRAMMAR

    file = ( comment / func / " " )*
    func = func_hdr func_args
    func_hdr = "CK_PKCS11_FUNCTION_INFO(" name ")"
    func_args = arg_hdr " (" arg* " ); #endif"
    arg_hdr = " #ifdef CK_NEED_ARG_LIST" (" " comment)?
    arg = " " type " " name ","? " " comment
    name = identifier
    type = identifier
    identifier = ~"[A-Z_][A-Z0-9_]*"i
    comment = ~"(/\*.*?\*/)"ms

PKCS11T GRAMMAR

    file = ( comment / define / typedef / struct_typedef / func_typedef / struct_alias_typedef / ignore )*
    typedef = " typedef" type identifier ";"
    struct_typedef = " typedef struct" identifier " "? "{" (comment / member)* " }" identifier ";"
    struct_alias_typedef = " typedef struct" identifier " CK_PTR"? identifier ";"
    func_typedef = " typedef CK_CALLBACK_FUNCTION(CK_RV," identifier ")(" (identifier identifier ","? comment?)* " );"    member = identifier identifier array_size? ";" comment?
    array_size = "[" ~"[0-9]"+ "]"
    define = "#define" identifier (hexval / decval / " (~0UL)" / identifier / ~" \([A-Z_]*\|0x[0-9]{8}\)" )
    hexval = ~" 0x[A-F0-9]{8}"i
    decval = ~" [0-9]+"
    type = " unsigned char" / " unsigned long int" / " long int" / (identifier " CK_PTR") / identifier
    identifier = " "? ~"[A-Z_][A-Z0-9_]*"i
    comment = " "? ~"(/\*.*?\*/)"ms
    ignore = ( " #ifndef" identifier ) / " #endif" / " "

Short, but dense, in true grammatical style. Looking at the whole program, it’s a straightforward process:

  1. Apply the grammars to the header files to get our abstract syntax tree.
  2. Walk the AST and sift out the semantically important pieces, function signatures in our case.
  3. Generate code from the function signature data structures.

Using only 200 lines of code to bring such a massive standard to bear, along with the portability and performance of Cython, through the power of PEGs ranks as one of the high points of Python in practice at PayPal.

Wrapping up

It’s been a long journey, but we stayed afloat and we’re happy to have made it. To recap:

All of this encapsulates perfectly why we love Python so much. Python is a great starter language, but it also has serious chops as a systems language and ecosystem. That bottom-to-top, rags-to-riches, books-to-bits story is what makes it the ineffable, incomparable language that it is.

C you around!

Kurt and Mahmoud

September 22, 2016 04:42 AM

Catalin George Festila

Another learning python post with pygame.

This is a simple python script with pygame python module.
I make it for for educational purposes for the children.
I used words into romanian language for variables, functions and two python class.
See this tutorial here

September 22, 2016 04:03 AM

Matthew Rocklin

Dask Cluster Deployments

This work is supported by Continuum Analytics and the XDATA Program as part of the Blaze Project

All code in this post is experimental. It should not be relied upon. For people looking to deploy dask.distributed on a cluster please refer instead to the documentation instead.

Dask is deployed today on the following systems in the wild:

These systems provide users access to cluster resources and ensure that many distributed services / users play nicely together. They’re essential for any modern cluster deployment.

The people deploying Dask on these cluster resource managers are power-users; they know how their resource managers work and they read the documentation on how to setup Dask clusters. Generally these users are pretty happy; however we should reduce this barrier so that non-power-users with access to a cluster resource manager can use Dask on their cluster just as easily.

Unfortunately, there are a few challenges:

  1. Several cluster resource managers exist, each with significant adoption. Finite developer time stops us from supporting all of them.
  2. Policies for scaling out vary widely. For example we might want a fixed number of workers, or we might want workers that scale out based on current use. Different groups will want different solutions.
  3. Individual cluster deployments are highly configurable. Dask needs to get out of the way quickly and let existing technologies configure themselves.

This post talks about some of these issues. It does not contain a definitive solution.

Example: Kubernetes

For example, both Olivier Griesl (INRIA, scikit-learn) and Tim O’Donnell (Mount Sinai, Hammer lab) publish instructions on how to deploy Dask.distributed on Kubernetes.

These instructions are well organized. They include Dockerfiles, published images, Kubernetes config files, and instructions on how to interact with cloud providers’ infrastructure. Olivier and Tim both obviously know what they’re doing and care about helping others to do the same.

Tim (who came second) wasn’t aware of Olivier’s solution and wrote up his own. Tim was capable of doing this but many beginners wouldn’t be.

One solution would be to include a prominent registry of solutions like these within Dask documentation so that people can find quality references to use as starting points. I’ve started a list of resources here: dask/distributed #547 comments pointing to other resources would be most welcome..

However, even if Tim did find Olivier’s solution I suspect he would still need to change it. Tim has different software and scalability needs than Olivier. This raises the question of “What should Dask provide and what should it leave to administrators?” It may be that the best we can do is to support copy-paste-edit workflows.

What is Dask-specific, resource-manager specific, and what needs to be configured by hand each time?

Adaptive Deployments

In order to explore this topic of separable solutions I built a small adaptive deployment system for Dask.distributed on Marathon, an orchestration platform on top of Mesos.

This solution does two things:

  1. It scales a Dask cluster dynamically based on the current use. If there are more tasks in the scheduler then it asks for more workers.
  2. It deploys those workers using Marathon.

To encourage replication, these two different aspects are solved in two different pieces of code with a clean API boundary.

  1. A backend-agnostic piece for adaptivity that says when to scale workers up and how to scale them down safely
  2. A Marathon-specific piece that deploys or destroys dask-workers using the Marathon HTTP API

This combines a policy, adaptive scaling, with a backend, Marathon such that either can be replaced easily. For example we could replace the adaptive policy with a fixed one to always keep N workers online, or we could replace Marathon with Kubernetes or Yarn.

My hope is that this demonstration encourages others to develop third party packages. The rest of this post will be about diving into this particular solution.

Adaptivity

The distributed.deploy.Adaptive class wraps around a Scheduler and determines when we should scale up and by how many nodes, and when we should scale down specifying which idle workers to release.

The current policy is fairly straightforward:

  1. If there are unassigned tasks or any stealable tasks and no idle workers, or if the average memory use is over 50%, then increase the number of workers by a fixed factor (defaults to two).
  2. If there are idle workers and the average memory use is below 50% then reclaim the idle workers with the least data on them (after moving data to nearby workers) until we’re near 50%

Think this policy could be improved or have other thoughts? Great. It was easy to implement and entirely separable from the main code so you should be able to edit it easily or create your own. The current implementation is about 80 lines (source).

However, this Adaptive class doesn’t actually know how to perform the scaling. Instead it depends on being handed a separate object, with two methods, scale_up and scale_down:

class MyCluster(object):
    def scale_up(n):
        """
        Bring the total count of workers up to ``n``

        This function/coroutine should bring the total number of workers up to
        the number ``n``.
        """
        raise NotImplementedError()

    def scale_down(self, workers):
        """
        Remove ``workers`` from the cluster

        Given a list of worker addresses this function should remove those
        workers from the cluster.
        """
        raise NotImplementedError()

This cluster object contains the backend-specific bits of how to scale up and down, but none of the adaptive logic of when to scale up and down. The single-machine LocalCluster object serves as reference implementation.

So we combine this adaptive scheme with a deployment scheme. We’ll use a tiny Dask-Marathon deployment library available here

from dask_marathon import MarathonCluster
from distributed import Scheduler
from distributed.deploy import Adaptive

s = Scheduler()
mc = MarathonCluster(s, cpus=1, mem=4000,
                     docker_image='mrocklin/dask-distributed')
ac = Adaptive(s, mc)

This combines a policy, Adaptive, with a deployment scheme, Marathon in a composable way. The Adaptive cluster watches the scheduler and calls the scale_up/down methods on the MarathonCluster as necessary.

Marathon code

Because we’ve isolated all of the “when” logic to the Adaptive code, the Marathon specific code is blissfully short and specific. We include a slightly simplified version below. There is a fair amount of Marathon-specific setup in the constructor and then simple scale_up/down methods below:

from marathon import MarathonClient, MarathonApp
from marathon.models.container import MarathonContainer


class MarathonCluster(object):
    def __init__(self, scheduler,
                 executable='dask-worker',
                 docker_image='mrocklin/dask-distributed',
                 marathon_address='http://localhost:8080',
                 name=None, cpus=1, mem=4000, **kwargs):
        self.scheduler = scheduler

        # Create Marathon App to run dask-worker
        args = [
            executable,
            scheduler.address,
            '--nthreads', str(cpus),
            '--name', '$MESOS_TASK_ID',  # use Mesos task ID as worker name
            '--worker-port', '$PORT_WORKER',
            '--nanny-port', '$PORT_NANNY',
            '--http-port', '$PORT_HTTP'
        ]

        ports = [{'port': 0,
                  'protocol': 'tcp',
                  'name': name}
                 for name in ['worker', 'nanny', 'http']]

        args.extend(['--memory-limit',
                     str(int(mem * 0.6 * 1e6))])

        kwargs['cmd'] = ' '.join(args)
        container = MarathonContainer({'image': docker_image})

        app = MarathonApp(instances=0,
                          container=container,
                          port_definitions=ports,
                          cpus=cpus, mem=mem, **kwargs)

        # Connect and register app
        self.client = MarathonClient(marathon_address)
        self.app = self.client.create_app(name or 'dask-%s' % uuid.uuid4(), app)

    def scale_up(self, instances):
        self.client.scale_app(self.app.id, instances=instances)

    def scale_down(self, workers):
        for w in workers:
            self.client.kill_task(self.app.id,
                                  self.scheduler.worker_info[w]['name'],
                                  scale=True)

This isn’t trivial, you need to know about Marathon for this to make sense, but fortunately you don’t need to know much else. My hope is that people familiar with other cluster resource managers will be able to write similar objects and will publish them as third party libraries as I have with this Marathon solution here: https://github.com/mrocklin/dask-marathon (thanks goes to Ben Zaitlen for setting up a great testing harness for this and getting everything started.)

Adaptive Policies

Similarly, we can design new policies for deployment. You can read more about the policies for the Adaptive class in the documentation or the source (about eighty lines long). I encourage people to implement and use other policies and contribute back those policies that are useful in practice.

Final thoughts

We laid out a problem

We proposed two solutions:

  1. Maintain a registry of links to solutions, supporting copy-paste-edit practices
  2. Develop an API boundary that encourages separable development of third party libraries.

It’s not clear that either solution is sufficient, or that the current implementation of either solution is any good. This is is an important problem though as Dask.distributed is, today, still mostly used by super-users. I would like to engage community creativity here as we search for a good solution.

September 22, 2016 12:00 AM

September 21, 2016

Kushal Das

Dgplug contributor grant recipient Trishna Guha

I am happy to announce that Trishna Guha is the recipient of a dgplug contributor grant for 2016. She is an upstream contributor in Fedora Cloud SIG, and hacks on Bodhi in her free time. Trishna started her open source journey just a year back during the dgplug summer training 2015, you can read more about her work in a previous blog post. She has also become an active member of the local Pune PyLadies chapter.

The active members of dgplug.org every year contribute funding, which we then use to help out community members as required. For example, we previously used this fund to pay accommodation costs for our women contributors during PyCon. This year we are happy to be able to assist Trishna Guha to attend PyCon India 2016. Her presence and expertise with upstream development will help diversity efforts at various levels. As she is still a college student, we found many students are interested to talk to and learn from her. So, if you are coming down to PyCon India this weekend, remember to visit the Red Hat booth, and have a chat with Trishna.

September 21, 2016 02:20 PM

Martijn Faassen

Is Morepath Fast Yet?

Morepath is a Python web framework. But is it fast enough for your purposes?

Does performance matter?

Performance is one of the least important criteria you should use when you pick a Python web framework. You're using Python for web development after all; you are already compromising performance for ease of use.

But performance makes things seem easy. It boils down the whole choice between web frameworks to a single seemingly easy to understand variable: how fast is this thing? All web frameworks are the same anyway, right? (Wrong). We don't want the speed of our application be dragged down by the web framework. So we should just pick the one that is fastest. Makes total sense.

It makes total sense until you take a few minutes to think about it. Performance, sure, but performance doing what? Performance is notoriously difficult to measure. Sending a single "hello world" response? Parsing complex URLs with multiple variables in them? HTML template rendering? JSON serialization? Link generation? What aspect of performance matters to you depends on the application you're building. Why do we worry so much about performance and not about features, anyway?

Choosing a web framework based on performance makes no sense for most people. For most applications, application code dominates the CPU time spent. Pulling stuff out of a database can take vastly more time than rendering a web response.

What matters

So it makes sense to look at other factors when picking a web framework. Is there documentation? Can it do what I need it to do? Will it grow with me over time? Is it flexible? Is it being maintained? What's the community like? Does it have a cool logo?

Okay, I'm starting to sound like someone who doesn't want to admit the web framework I work on, Morepath, is atrociously slow. I'm giving you all kinds of reasons why you should use it despite its performance, which you would guess is pretty atrocious. It's true that the primary selling point of Morepath isn't performance -- it's flexibility. It's a micro-framework that is easy to learn but that doesn't let you down when your requirements become more complex.

Morepath performance

I maintain a very simple benchmark tool that measures just one aspect of performance: how fast a web framework at the Python WSGI level can generate simple "hello world" responses.

https://github.com/faassen/welcome-bench

I run it against Morepath once every while to see how we're doing with performance. I actually care more about what the framework is doing when Morepath generates the response than I care about the actual requests per second it can generate. I want Morepath's underlying complexity to be relatively simple. But since performance is so easy to think about I take advantage of that as a shortcut. I treat performance as an approximation of implementation complexity. Plus it's cool when your framework is fast, I admit it.

The current Morepath development version takes about 5200 ms per 100,000 requests, which translates to about 19200 requests per second. Let's see how that compares to some of the friendly competition:

                ms     rps  tcalls  funcs
django       10992    9098     190     85
flask        15854    6308     270    125
morepath      5204   19218     109     80

So at this silly benchmark, Morepath is more than twice as fast as Django and more than three times faster than Flask!

Let me highlight that for marketing purposes and trick those who aren't reading carefully:

Morepath is more than 2 times faster than Django and more than 3 times faster than Flask

Yay! End of story. Well, I gave a bit of a selective view just now. Here are some other web frameworks:

                ms     rps  tcalls  funcs
bottle        2172   46030      53     31
falcon        1539   64961      26     24
morepath      5204   19218     109     80
pyramid       3920   25509      72     57
wheezy.web    1201   83247      25     23

I'm not going to highlight that Bottle is more than two times faster at this silly benchmark nor that Falcon is more than three times faster. Let's not even think about wheezy.web.

I think this performance comparison actually highlights my point that in practice web framework performance is usually irrelevant. People aren't flocking to wheezy.web just because it's so fast. People aren't ignoring Flask because it's comparatively slow. I suspect many are surprised are surprised Flask is one of the slowest frameworks in this benchmark, as it's a lightweight framework.

Flask's relatively slow performance hasn't hurt its popularity. This demonstrates my point that web framework performance isn't that important overall. I don't fully understand why Flask is relatively slow, but I know part of the reason is werkzeug, its request/response implementation. Morepath is actually doing a lot more sophisticated stuff underneath than Flask and it's still faster. That Pyramid is faster than Morepath is impressive, as what it needs to do during runtime is similar to Morepath.

Let's look at the tcalls column: how many function calls get executed during a request. There is a strong correlation between how many Python function calls there are during a request and requests per second. This is why performance is a decent approximation of implementation complexity. It's also a clear sign we're using an interpreted language.

How Morepath performance has changed

So how has Morepath's performance evolved over time? Here's a nice graph:

Morepath performance over time

So what does this chart tell us? Before its 0.1 release when it still used werkzeug, Morepath was actually about as slow as Flask. After we switched to webob, Morepath became faster than Flask, but was still slower than Django.

By release 0.4.1 a bunch of minor improvements had pushed performance slightly beyond Django's -- but I don't have a clear idea of the details. I also don't understand exactly why there's a performance bump for 0.7, though I suspect it has to do with a refactor of application mounting behavior I did around that time -- while that code isn't exercised in this benchmark, it's possible it simplified a critical path.

I do know what caused the huge bump in performance in 0.8. This marked the switch to Reg 0.9, which is a dispatch library that is used heavily by Morepath. Reg 0.9 got faster, as this is when Reg switched to a more flexible and efficient predicate dispatch approach.

Performance was stable again until version 0.11, when it went down again. In 0.11 we introduced a measure to make the request object sanitize potentially dangerous input, and this cost us some performance. I'm not sure what caused the slight performance drop in 0.14.

And then there's a vast performance increase in current master. What explains this? Two things:

  • We've made some huge improvements to Reg again. Morepath benefits because it uses Reg heavily.
  • I cheated. That is, I found work that could be skipped in the case no URL parameters are in play, as in this benchmark.

Skipping unnecessary work was a legitimate improvement of Morepath. The code now avoids accessing the relatively expensive GET attribute on the webob request, and also avoids a for loop through an empty list and a few if statements. In Python, performance is sensitive to even a few extra lines of code on the critical path.

But when you do have URL parameters, Morepath's feature that lets you convert and validate them automatically is pretty nice -- in almost all circumstances you should be glad to pay the slight performance penalty for the convenience. Features are usually worth their performance cost.

So is Morepath fast yet?

Is Morepath fast yet? Probably. Does it matter? It depends. What benchmark? But for those just skimming this article, I'll do another sneaky highlight:

Morepath is fast. Morepath outperforms the most popular Python web frameworks while offering a lot more flexibility.

September 21, 2016 11:36 AM

Codementor

Image Manipulation in Python

Color Space Background

CMYK Color Representation:

When we see a printed advertisement or poster, the colors are printed with color spaces based on the CMYK color model, using the subtractive primary colors of pigment Cyan, Magenta, Yellow, and blacK. This is also known as the four-colors print process.

python image processing

The “primary” and “secondary” colors in a four-color print process are exhibited below.

python image processing

RGB Color Representation:

While printed colors are represented with the use of the four-color process, monitors represent color using the RGB color model which is an additive color model in which Red, Green and Blue light are added together in various ways to reproduce a broad array of colors.

python image processing

The Difference Between Print and Monitor Color

Offset lithography is one of the most common ways of creating printed materials. A few of its applications include newspapers, magazines, brochures, stationery, and books. This model requires the image to be converted or made in the CMYK color model. All printed material relies on creating pigments of colors that when combined, forms the color as shown below.

python image processing

The ink’s semi-opacity property is in conjunction with the halftone technology and this is responsible for allowing pigments to mix and create new colors with just four primary ones.

On the other hand, media that transmit light (such as the monitor on your PC, tablet, or phone) use additive color mixing, which means that every pixel is made from three colors (RGB color model) by displaying different intensity the colors get produced.

Why Should I Care?

The first thing you might be wondering is why am I telling you how traditional press works, the existence of CMYK and RGB color models, or the pigment process (ink opacity plus halftoning) to print a brochure.

The rest of the tutorial will show you how to transform an image with different filters and techniques to deliver different outputs. These methods are still in use and part of a process known as Computer-To-Plate (CTP), used to create a direct output from an image file to a photographic film or plate (depending on the process), which are employed in industrial machines like the ones from Heidelberg.

This tutorial will give you insight into the filters and techniques used to transform images, some international image standards, and hopefully, some interest in the history of printing.

Quick Notes

  1. Pillow is a fork of PIL (Python Imaging Library)
  2. Pillow and PIL cannot co-exist in the same environment. Before installing Pillow, uninstall PIL.
  3. libjpeg-dev is required to be able to process jpeg’s with Pillow.
  4. Pillow >= 2.0.0 supports Python versions 2.6, 2.7, 3.2, 3.3, and 3.4
  5. Pillow >= 1.0 no longer supports import Image. Use from PIL import Image instead.

Installing Required Library

Before we start, we will need Python 3 and Pillow. If you are using Linux, Pillow will probably be there already, since major distributions including Fedora, Debian/Ubuntu, and ArchLinux include Pillow in packages that previously contained PIL.

The easiest way to install it is to use pip:

pip install Pillow

The installation process should look similar to the following.

python image processing

If any trouble happens, I recommend reading the documentation from the Pillow Manual.

Basic Methods

Before manipulating an image, we need to be able to open the file, save the changes, create an empty picture, and to obtain individual pixels color. For the convenience of this tutorial, I have already made the methods to do so, which will be used in all subsequent sections.

These methods rely on the imported image library from the Pillow package.

# Imported PIL Library
from PIL import Image



# Open an Image
def open_image(path):
    newImage = Image.open(path)
    return newImage



# Save Image
def save_image(image, path):
    image.save(path, 'png')



# Create a new image with the given size
def create_image(i, j):
    image = Image.new("RGB", (i, j), "white")
    return image



# Get the pixel from the given image
def get_pixel(image, i, j):
    # Inside image bounds?
    width, height = image.size
    if i > width or j > height:
        return None

    # Get Pixel
    pixel = image.getpixel((i, j))
    return pixel

Grayscale Filter

The traditional grayscale algorithm transforms an image to grayscale by obtaining the average channels color and making each channel equals to the average.

python image processing

A better choice for grayscale is the ITU-R Recommendation BT.601-7, which specifies methods for digitally coding video signals by normalizing the values. For the grayscale transmissions, it defines the following formula.

BT.601-7

# Create a Grayscale version of the image
def convert_grayscale(image):
    # Get size
    width, height = image.size

    # Create new Image and a Pixel Map
    new = create_image(width, height)
    pixels = new.load()

    # Transform to grayscale
    for i in range(width):
        for j in range(height):
            # Get Pixel
            pixel = get_pixel(image, i, j)

            # Get R, G, B values (This are int from 0 to 255)
            red =   pixel[0]
            green = pixel[1]
            blue =  pixel[2]

            # Transform to grayscale
            gray = (red * 0.299) + (green * 0.587) + (blue * 0.114)

            # Set Pixel in new image
            pixels[i, j] = (int(gray), int(gray), int(gray))

    # Return new image
    return new

By applying the filter with the above code, and using the BT.601-7 recommendation, we get the following result.

python image processing

Half-tone Filter

The halftoning filter is the traditional method of printing images. It is a reprographic technique that simulates continuous tone imagery through the use of dots.

To generate the effect of shades of gray using only dots of black, we will define a size, in this case, a two by two matrix, and depending on the saturation we will draw black dots in this matrix.

# Create a Half-tone version of the image
def convert_halftoning(image):
    # Get size
    width, height = image.size

    # Create new Image and a Pixel Map
    new = create_image(width, height)
    pixels = new.load()

    # Transform to half tones
    for i in range(0, width, 2):
        for j in range(0, height, 2):
            # Get Pixels
            p1 = get_pixel(image, i, j)
            p2 = get_pixel(image, i, j + 1)
            p3 = get_pixel(image, i + 1, j)
            p4 = get_pixel(image, i + 1, j + 1)

            # Transform to grayscale
            gray1 = (p1[0] * 0.299) + (p1[1] * 0.587) + (p1[2] * 0.114)
            gray2 = (p2[0] * 0.299) + (p2[1] * 0.587) + (p2[2] * 0.114)
            gray3 = (p3[0] * 0.299) + (p3[1] * 0.587) + (p3[2] * 0.114)
            gray4 = (p4[0] * 0.299) + (p4[1] * 0.587) + (p4[2] * 0.114)

            # Saturation Percentage
            sat = (gray1 + gray2 + gray3 + gray4) / 4

            # Draw white/black depending on saturation
            if sat > 223:
                pixels[i, j]         = (255, 255, 255) # White
                pixels[i, j + 1]     = (255, 255, 255) # White
                pixels[i + 1, j]     = (255, 255, 255) # White
                pixels[i + 1, j + 1] = (255, 255, 255) # White
            elif sat > 159:
                pixels[i, j]         = (255, 255, 255) # White
                pixels[i, j + 1]     = (0, 0, 0)       # Black
                pixels[i + 1, j]     = (255, 255, 255) # White
                pixels[i + 1, j + 1] = (255, 255, 255) # White
            elif sat > 95:
                pixels[i, j]         = (255, 255, 255) # White
                pixels[i, j + 1]     = (0, 0, 0)       # Black
                pixels[i + 1, j]     = (0, 0, 0)       # Black
                pixels[i + 1, j + 1] = (255, 255, 255) # White
            elif sat > 32:
                pixels[i, j]         = (0, 0, 0)       # Black
                pixels[i, j + 1]     = (255, 255, 255) # White
                pixels[i + 1, j]     = (0, 0, 0)       # Black
                pixels[i + 1, j + 1] = (0, 0, 0)       # Black
            else:
                pixels[i, j]         = (0, 0, 0)       # Black
                pixels[i, j + 1]     = (0, 0, 0)       # Black
                pixels[i + 1, j]     = (0, 0, 0)       # Black
                pixels[i + 1, j + 1] = (0, 0, 0)       # Black

    # Return new image
    return new

By applying the filter using the code above, we are separating in five ranges and coloring the pixels on the array white or black depending on the saturation, which will produce the following result.

python image processing

Dithering Filter

Dithering is an intentionally applied form of noise; it is used for processing an image to generate the illusion of colors by using the halftone filter on each color channel.

This method is used in traditional print as explained earlier.
In our approach, we will set the saturation for each channel,
in a direct fashion by replicating the halftone algorithm in each channel.
For a more advanced dither filter, you can read about the Floyd–Steinberg dithering.

# Return color value depending on quadrant and saturation
def get_saturation(value, quadrant):
    if value > 223:
        return 255
    elif value > 159:
        if quadrant != 1:
            return 255

        return 0
    elif value > 95:
        if quadrant == 0 or quadrant == 3:
            return 255

        return 0
    elif value > 32:
        if quadrant == 1:
            return 255

        return 0
    else:
        return 0



# Create a dithered version of the image
def convert_dithering(image):
    # Get size
    width, height = image.size

    # Create new Image and a Pixel Map
    new = create_image(width, height)
    pixels = new.load()

    # Transform to half tones
    for i in range(0, width, 2):
        for j in range(0, height, 2):
            # Get Pixels
            p1 = get_pixel(image, i, j)
            p2 = get_pixel(image, i, j + 1)
            p3 = get_pixel(image, i + 1, j)
            p4 = get_pixel(image, i + 1, j + 1)

            # Color Saturation by RGB channel
            red   = (p1[0] + p2[0] + p3[0] + p4[0]) / 4
            green = (p1[1] + p2[1] + p3[1] + p4[1]) / 4
            blue  = (p1[2] + p2[2] + p3[2] + p4[2]) / 4

            # Results by channel
            r = [0, 0, 0, 0]
            g = [0, 0, 0, 0]
            b = [0, 0, 0, 0]

            # Get Quadrant Color
            for x in range(0, 4):
                r[x] = get_saturation(red, x)
                g[x] = get_saturation(green, x)
                b[x] = get_saturation(blue, x)

            # Set Dithered Colors
            pixels[i, j]         = (r[0], g[0], b[0])
            pixels[i, j + 1]     = (r[1], g[1], b[1])
            pixels[i + 1, j]     = (r[2], g[2], b[2])
            pixels[i + 1, j + 1] = (r[3], g[3], b[3])

    # Return new image
    return new

By processing each channel, we set the colors to the primary and secondary ones. This is a total of eight colors, but as you notice in the processed image, they give the illusion of a wider array of colors.

python image processing

Complete Source

The complete code to process images takes a PNG file in RGB color mode (with no transparency), saving the output as different images.

Due to limitations with JPEG support on various operating systems, I choose the PNG format.

'''
    This Example opens an Image and transform the image into grayscale, halftone, dithering, and primary colors.
    You need PILLOW (Python Imaging Library fork) and Python 3.5
    -Isai B. Cicourel
'''


# Imported PIL Library
from PIL import Image



# Open an Image
def open_image(path):
    newImage = Image.open(path)
    return newImage



# Save Image
def save_image(image, path):
    image.save(path, 'png')



# Create a new image with the given size
def create_image(i, j):
    image = Image.new("RGB", (i, j), "white")
    return image



# Get the pixel from the given image
def get_pixel(image, i, j):
    # Inside image bounds?
    width, height = image.size
    if i > width or j > height:
        return None

    # Get Pixel
    pixel = image.getpixel((i, j))
    return pixel



# Create a Grayscale version of the image
def convert_grayscale(image):
    # Get size
    width, height = image.size

    # Create new Image and a Pixel Map
    new = create_image(width, height)
    pixels = new.load()

    # Transform to grayscale
    for i in range(width):
        for j in range(height):
            # Get Pixel
            pixel = get_pixel(image, i, j)

            # Get R, G, B values (This are int from 0 to 255)
            red =   pixel[0]
            green = pixel[1]
            blue =  pixel[2]

            # Transform to grayscale
            gray = (red * 0.299) + (green * 0.587) + (blue * 0.114)

            # Set Pixel in new image
            pixels[i, j] = (int(gray), int(gray), int(gray))

    # Return new image
    return new



# Create a Half-tone version of the image
def convert_halftoning(image):
    # Get size
    width, height = image.size

    # Create new Image and a Pixel Map
    new = create_image(width, height)
    pixels = new.load()

    # Transform to half tones
    for i in range(0, width, 2):
        for j in range(0, height, 2):
            # Get Pixels
            p1 = get_pixel(image, i, j)
            p2 = get_pixel(image, i, j + 1)
            p3 = get_pixel(image, i + 1, j)
            p4 = get_pixel(image, i + 1, j + 1)

            # Transform to grayscale
            gray1 = (p1[0] * 0.299) + (p1[1] * 0.587) + (p1[2] * 0.114)
            gray2 = (p2[0] * 0.299) + (p2[1] * 0.587) + (p2[2] * 0.114)
            gray3 = (p3[0] * 0.299) + (p3[1] * 0.587) + (p3[2] * 0.114)
            gray4 = (p4[0] * 0.299) + (p4[1] * 0.587) + (p4[2] * 0.114)

            # Saturation Percentage
            sat = (gray1 + gray2 + gray3 + gray4) / 4

            # Draw white/black depending on saturation
            if sat > 223:
                pixels[i, j]         = (255, 255, 255) # White
                pixels[i, j + 1]     = (255, 255, 255) # White
                pixels[i + 1, j]     = (255, 255, 255) # White
                pixels[i + 1, j + 1] = (255, 255, 255) # White
            elif sat > 159:
                pixels[i, j]         = (255, 255, 255) # White
                pixels[i, j + 1]     = (0, 0, 0)       # Black
                pixels[i + 1, j]     = (255, 255, 255) # White
                pixels[i + 1, j + 1] = (255, 255, 255) # White
            elif sat > 95:
                pixels[i, j]         = (255, 255, 255) # White
                pixels[i, j + 1]     = (0, 0, 0)       # Black
                pixels[i + 1, j]     = (0, 0, 0)       # Black
                pixels[i + 1, j + 1] = (255, 255, 255) # White
            elif sat > 32:
                pixels[i, j]         = (0, 0, 0)       # Black
                pixels[i, j + 1]     = (255, 255, 255) # White
                pixels[i + 1, j]     = (0, 0, 0)       # Black
                pixels[i + 1, j + 1] = (0, 0, 0)       # Black
            else:
                pixels[i, j]         = (0, 0, 0)       # Black
                pixels[i, j + 1]     = (0, 0, 0)       # Black
                pixels[i + 1, j]     = (0, 0, 0)       # Black
                pixels[i + 1, j + 1] = (0, 0, 0)       # Black

    # Return new image
    return new



# Return color value depending on quadrant and saturation
def get_saturation(value, quadrant):
    if value > 223:
        return 255
    elif value > 159:
        if quadrant != 1:
            return 255

        return 0
    elif value > 95:
        if quadrant == 0 or quadrant == 3:
            return 255

        return 0
    elif value > 32:
        if quadrant == 1:
            return 255

        return 0
    else:
        return 0



# Create a dithered version of the image
def convert_dithering(image):
    # Get size
    width, height = image.size

    # Create new Image and a Pixel Map
    new = create_image(width, height)
    pixels = new.load()

    # Transform to half tones
    for i in range(0, width, 2):
        for j in range(0, height, 2):
            # Get Pixels
            p1 = get_pixel(image, i, j)
            p2 = get_pixel(image, i, j + 1)
            p3 = get_pixel(image, i + 1, j)
            p4 = get_pixel(image, i + 1, j + 1)

            # Color Saturation by RGB channel
            red   = (p1[0] + p2[0] + p3[0] + p4[0]) / 4
            green = (p1[1] + p2[1] + p3[1] + p4[1]) / 4
            blue  = (p1[2] + p2[2] + p3[2] + p4[2]) / 4

            # Results by channel
            r = [0, 0, 0, 0]
            g = [0, 0, 0, 0]
            b = [0, 0, 0, 0]

            # Get Quadrant Color
            for x in range(0, 4):
                r[x] = get_saturation(red, x)
                g[x] = get_saturation(green, x)
                b[x] = get_saturation(blue, x)

            # Set Dithered Colors
            pixels[i, j]         = (r[0], g[0], b[0])
            pixels[i, j + 1]     = (r[1], g[1], b[1])
            pixels[i + 1, j]     = (r[2], g[2], b[2])
            pixels[i + 1, j + 1] = (r[3], g[3], b[3])

    # Return new image
    return new



# Create a Primary Colors version of the image
def convert_primary(image):
    # Get size
    width, height = image.size

    # Create new Image and a Pixel Map
    new = create_image(width, height)
    pixels = new.load()

    # Transform to primary
    for i in range(width):
        for j in range(height):
            # Get Pixel
            pixel = get_pixel(image, i, j)

            # Get R, G, B values (This are int from 0 to 255)
            red =   pixel[0]
            green = pixel[1]
            blue =  pixel[2]

            # Transform to primary
            if red > 127:
                red = 255
            else:
                red = 0
            if green > 127:
                green = 255
            else:
                green = 0
            if blue > 127:
                blue = 255
            else:
                blue = 0

            # Set Pixel in new image
            pixels[i, j] = (int(red), int(green), int(blue))

    # Return new image
    return new



# Main
if __name__ == "__main__":
    # Load Image (JPEG/JPG needs libjpeg to load)
    original = open_image('Hero_Prinny.png')

    # Example Pixel Color
    print('Color: ' + str(get_pixel(original, 0, 0)))

    # Convert to Grayscale and save
    new = convert_grayscale(original)
    save_image(new, 'Prinny_gray.png')

    # Convert to Halftoning and save
    new = convert_halftoning(original)
    save_image(new, 'Prinny_half.png')

    # Convert to Dithering and save
    new = convert_dithering(original)
    save_image(new, 'Prinny_dither.png')

    # Convert to Primary and save
    new = convert_primary(original)
    save_image(new, 'Prinny_primary.png')

The source code takes an image, then applies each filter and saves the output as a new image, producing the following results.

Don’t forget to specify the path to the image in original = open_image('Hero_Prinny.png') and on the outputs. Unless you have that image, which would mean you are a Disgaea fan.

python image processing

Wrapping Up

Image filters are not only something we use to make our pictures on social networking sites look cool, they are useful and powerful techniques for processing videos and images not only for printing in an offset; but also to compress and improve playback and speed of on-demand services.

Have You Tried The Following?

Now that you know some image filters, how about applying several of them on the same picture?

On a large image, what happens with a filter when you “fit to screen”?

Did you notice the extra filter in the complete source code?

Have you checked the size of the different output images?

Play around and check what happens. Create your filter or implement a new one, the idea is to learn new things. If you like this tutorial, share it with your friends. ;-)

##Other tutorials you might be interested in:

September 21, 2016 08:39 AM

tryexceptpass

On Democratizing the Next Generation of User Interfaces

Musings of a dev trying to plan for the future

Continue reading on »

September 21, 2016 04:30 AM

Ruslan Spivak

Let’s Build A Simple Interpreter. Part 11.

I was sitting in my room the other day and thinking about how much we had covered, and I thought I would recap what we’ve learned so far and what lies ahead of us.

Up until now we’ve learned:

With all that knowledge and experience under our belt, we’ve built an interpreter that can scan, parse, and build an AST and interpret, by walking the AST, our very first complete Pascal program. Ladies and gentlemen, I honestly think if you’ve reached this far, you deserve a pat on the back. But don’t let it go to your head. Keep going. Even though we’ve covered a lot of ground, there are even more exciting parts coming our way.

With everything we’ve covered so far, we are almost ready to tackle topics like:

But before we cover those topics, we need to build a solid foundation and infrastructure.

This is where we start diving deeper into the super important topic of symbols, symbol tables, and scopes. The topic itself will span several articles. It’s that important and you’ll see why. Okay, let’s start building that foundation and infrastructure, then, shall we?

First, let’s talk about symbols and why we need to track them. What is a symbol? For our purposes, we’ll informally define symbol as an identifier of some program entity like a variable, subroutine, or built-in type. For symbols to be useful they need to have at least the following information about the program entities they identify:

Today we’ll tackle variable symbols and built-in type symbols because we’ve already used variables and types before. By the way, the “built-in” type just means a type that hasn’t been defined by you and is available for you right out of the box, like INTEGER and REAL types that you’ve seen and used before.

Let’s take a look at the following Pascal program, specifically at the variable declaration part. You can see in the picture below that there are four symbols in that section: two variable symbols (x and y) and two built-in type symbols (INTEGER and REAL).

How can we represent symbols in code? Let’s create a base Symbol class in Python:

class Symbol(object):
    def __init__(self, name, type=None):
        self.name = name
        self.type = type

As you can see, the class takes the name parameter and an optional type parameter (not all symbols may have a type associated with them). What about the category of a symbol? We’ll encode the category of a symbol in the class name itself, which means we’ll create separate classes to represent different symbol categories.

Let’s start with basic built-in types. We’ve seen two built-in types so far, when we declared variables: INTEGER and REAL. How do we represent a built-in type symbol in code? Here is one option:

class BuiltinTypeSymbol(Symbol):
    def __init__(self, name):
        super(BuiltinTypeSymbol, self).__init__(name)

    def __str__(self):
        return self.name

    __repr__ = __str__

The class inherits from the Symbol class and the constructor requires only a name of the type. The category is encoded in the class name, and the type parameter from the base class for a built-in type symbol is None. The double underscore or dunder (as in “Double UNDERscore”) methods __str__ and __repr__ are special Python methods and we’ve defined them to have a nice formatted message when you print a symbol object.

Download the interpreter file and save it as spi.py; launch a python shell from the same directory where you saved the spi.py file, and play with the class we’ve just defined interactively:

$ python
>>> from spi import BuiltinTypeSymbol
>>> int_type = BuiltinTypeSymbol('INTEGER')
>>> int_type
INTEGER
>>> real_type = BuiltinTypeSymbol('REAL')
>>> real_type
REAL

How can we represent a variable symbol? Let’s create a VarSymbol class:

class VarSymbol(Symbol):
    def __init__(self, name, type):
        super(VarSymbol, self).__init__(name, type)

    def __str__(self):
        return '<{name}:{type}>'.format(name=self.name, type=self.type)

    __repr__ = __str__

In the class we made both the name and the type parameters required parameters and the class name VarSymbol clearly indicates that an instance of the class will identify a variable symbol (the category is variable.)

Back to the interactive python shell to see how we can manually construct instances for our variable symbols now that we know how to construct BuiltinTypeSymbol class instances:

$ python
>>> from spi import BuiltinTypeSymbol, VarSymbol
>>> int_type = BuiltinTypeSymbol('INTEGER')
>>> real_type = BuiltinTypeSymbol('REAL')
>>>
>>> var_x_symbol = VarSymbol('x', int_type)
>>> var_x_symbol
<x:INTEGER>
>>> var_y_symbol = VarSymbol('y', real_type)
>>> var_y_symbol
<y:REAL>

As you can see, we first create an instance of a built-in type symbol and then pass it as a parameter to VarSymbol‘s constructor.

Here is the hierarchy of symbols we’ve defined in visual form:

So far so good, but we haven’t answered the question yet as to why we even need to track those symbols in the first place.

Here are some of the reasons:

Take a look at the following incorrect Pascal program, for example:

There are two problems with the program above (you can compile it with fpc to see it for yourself):

  1. In the expression “x := 2 + y;” we assigned a decimal value to the variable “x” that was declared as integer. That wouldn’t compile because the types are incompatible.
  2. In the assignment statement “x := a;” we referenced the variable “a” that wasn’t declared - wrong!

To be able to identify cases like that even before interpreting/evaluating the source code of the program at run-time, we need to track program symbols. And where do we store the symbols that we track? I think you’ve guessed it right - in the symbol table!

What is a symbol table? A symbol table is an abstract data type (ADT) for tracking various symbols in source code. Today we’re going to implement our symbol table as a separate class with some helper methods:

class SymbolTable(object):
    def __init__(self):
        self._symbols = OrderedDict()

    def __str__(self):
        s = 'Symbols: {symbols}'.format(
            symbols=[value for value in self._symbols.values()]
        )
        return s

    __repr__ = __str__

    def define(self, symbol):
        print('Define: %s' % symbol)
        self._symbols[symbol.name] = symbol

    def lookup(self, name):
        print('Lookup: %s' % name)
        symbol = self._symbols.get(name)
        # 'symbol' is either an instance of the Symbol class or 'None'
        return symbol

There are two main operations that we will be performing with the symbol table: storing symbols and looking them up by name: hence, we need two helper methods - define and lookup.

The method define takes a symbol as a parameter and stores it internally in its _symbols ordered dictionary using the symbol’s name as a key and the symbol instance as a value. The method lookup takes a symbol name as a parameter and returns a symbol if it finds it or “None” if it doesn’t.

Let’s manually populate our symbol table for the same Pascal program we’ve used just recently where we were manually creating variable and built-in type symbols:

PROGRAM Part11;
VAR
   x : INTEGER;
   y : REAL;

BEGIN

END.

Launch a Python shell again and follow along:

$ python
>>> from spi import SymbolTable, BuiltinTypeSymbol, VarSymbol
>>> symtab = SymbolTable()
>>> int_type = BuiltinTypeSymbol('INTEGER')
>>> symtab.define(int_type)
Define: INTEGER
>>> symtab
Symbols: [INTEGER]
>>>
>>> var_x_symbol = VarSymbol('x', int_type)
>>> symtab.define(var_x_symbol)
Define: <x:INTEGER>
>>> symtab
Symbols: [INTEGER, <x:INTEGER>]
>>>
>>> real_type = BuiltinTypeSymbol('REAL')
>>> symtab.define(real_type)
Define: REAL
>>> symtab
Symbols: [INTEGER, <x:INTEGER>, REAL]
>>>
>>> var_y_symbol = VarSymbol('y', real_type)
>>> symtab.define(var_y_symbol)
Define: <y:REAL>
>>> symtab
Symbols: [INTEGER, <x:INTEGER>, REAL, <y:REAL>]

If you looked at the contents of the _symbols dictionary it would look something like this:

How do we automate the process of building the symbol table? We’ll just write another node visitor that walks the AST built by our parser! This is another example of how useful it is to have an intermediary form like AST. Instead of extending our parser to deal with the symbol table, we separate concerns and write a new node visitor class. Nice and clean. :)

Before doing that, though, let’s extend our SymbolTable class to initialize the built-in types when the symbol table instance is created. Here is the full source code for today’s SymbolTable class:

class SymbolTable(object):
    def __init__(self):
        self._symbols = OrderedDict()
        self._init_builtins()

    def _init_builtins(self):
        self.define(BuiltinTypeSymbol('INTEGER'))
        self.define(BuiltinTypeSymbol('REAL'))

    def __str__(self):
        s = 'Symbols: {symbols}'.format(
            symbols=[value for value in self._symbols.values()]
        )
        return s

    __repr__ = __str__

    def define(self, symbol):
        print('Define: %s' % symbol)
        self._symbols[symbol.name] = symbol

    def lookup(self, name):
        print('Lookup: %s' % name)
        symbol = self._symbols.get(name)
        # 'symbol' is either an instance of the Symbol class or 'None'
        return symbol

Now onto the SymbolTableBuilder AST node visitor:

class SymbolTableBuilder(NodeVisitor):
    def __init__(self):
        self.symtab = SymbolTable()

    def visit_Block(self, node):
        for declaration in node.declarations:
            self.visit(declaration)
        self.visit(node.compound_statement)

    def visit_Program(self, node):
        self.visit(node.block)

    def visit_BinOp(self, node):
        self.visit(node.left)
        self.visit(node.right)

    def visit_Num(self, node):
        pass

    def visit_UnaryOp(self, node):
        self.visit(node.expr)

    def visit_Compound(self, node):
        for child in node.children:
            self.visit(child)

    def visit_NoOp(self, node):
        pass

    def visit_VarDecl(self, node):
        type_name = node.type_node.value
        type_symbol = self.symtab.lookup(type_name)
        var_name = node.var_node.value
        var_symbol = VarSymbol(var_name, type_symbol)
        self.symtab.define(var_symbol)

You’ve seen most of those methods before in the Interpreter class, but the visit_VarDecl method deserves some special attention. Here it is again:

def visit_VarDecl(self, node):
    type_name = node.type_node.value
    type_symbol = self.symtab.lookup(type_name)
    var_name = node.var_node.value
    var_symbol = VarSymbol(var_name, type_symbol)
    self.symtab.define(var_symbol)

This method is responsible for visiting (walking) a VarDecl AST node and storing the corresponding symbol in the symbol table. First, the method looks up the built-in type symbol by name in the symbol table, then it creates an instance of the VarSymbol class and stores (defines) it in the symbol table.

Let’s take our SymbolTableBuilder AST walker for a test drive and see it in action:

$ python
>>> from spi import Lexer, Parser, SymbolTableBuilder
>>> text = """
... PROGRAM Part11;
... VAR
...    x : INTEGER;
...    y : REAL;
...
... BEGIN
...
... END.
... """
>>> lexer = Lexer(text)
>>> parser = Parser(lexer)
>>> tree = parser.parse()
>>> symtab_builder = SymbolTableBuilder()
Define: INTEGER
Define: REAL
>>> symtab_builder.visit(tree)
Lookup: INTEGER
Define: <x:INTEGER>
Lookup: REAL
Define: <y:REAL>
>>> # Let’s examine the contents of our symbol table
…
>>> symtab_builder.symtab
Symbols: [INTEGER, REAL, <x:INTEGER>, <y:REAL>]

In the interactive session above, you can see the sequence of “Define: …” and “Lookup: …” messages that indicate the order in which symbols are defined and looked up in the symbol table. The last command in the session prints the contents of the symbol table and you can see that it’s exactly the same as the contents of the symbol table that we’ve built manually before. The magic of AST node visitors is that they pretty much do all the work for you. :)

We can already put our symbol table and symbol table builder to good use: we can use them to verify that variables are declared before they are used in assignments and expressions. All we need to do is just extend the visitor with two more methods: visit_Assign and visit_Var:

def visit_Assign(self, node):
    var_name = node.left.value
    var_symbol = self.symtab.lookup(var_name)
    if var_symbol is None:
        raise NameError(repr(var_name))

    self.visit(node.right)

def visit_Var(self, node):
    var_name = node.value
    var_symbol = self.symtab.lookup(var_name)

    if var_symbol is None:
        raise NameError(repr(var_name))

These methods will raise a NameError exception if they cannot find the symbol in the symbol table.

Take a look at the following program, where we reference the variable “b” that hasn’t been declared yet:

PROGRAM NameError1;
VAR
   a : INTEGER;

BEGIN
   a := 2 + b;
END.

Let’s see what happens if we construct an AST for the program and pass it to our symbol table builder to visit:

$ python
>>> from spi import Lexer, Parser, SymbolTableBuilder
>>> text = """
... PROGRAM NameError1;
... VAR
...    a : INTEGER;
...
... BEGIN
...    a := 2 + b;
... END.
... """
>>> lexer = Lexer(text)
>>> parser = Parser(lexer)
>>> tree = parser.parse()
>>> symtab_builder = SymbolTableBuilder()
Define: INTEGER
Define: REAL
>>> symtab_builder.visit(tree)
Lookup: INTEGER
Define: <a:INTEGER>
Lookup: a
Lookup: b
Traceback (most recent call last):
  ...
  File "spi.py", line 674, in visit_Var
    raise NameError(repr(var_name))
NameError: 'b'

Exactly what we were expecting!

Here is another error case where we try to assign a value to a variable that hasn’t been defined yet, in this case the variable ‘a’:

PROGRAM NameError2;
VAR
   b : INTEGER;

BEGIN
   b := 1;
   a := b + 2;
END.

Meanwhile, in the Python shell:

>>> from spi import Lexer, Parser, SymbolTableBuilder
>>> text = """
... PROGRAM NameError2;
... VAR
...    b : INTEGER;
...
... BEGIN
...    b := 1;
...    a := b + 2;
... END.
... """
>>> lexer = Lexer(text)
>>> parser = Parser(lexer)
>>> tree = parser.parse()
>>> symtab_builder = SymbolTableBuilder()
Define: INTEGER
Define: REAL
>>> symtab_builder.visit(tree)
Lookup: INTEGER
Define: <b:INTEGER>
Lookup: b
Lookup: a
Traceback (most recent call last):
  ...
  File "spi.py", line 665, in visit_Assign
    raise NameError(repr(var_name))
NameError: 'a'

Great, our new visitor caught this problem too!

I would like to emphasize the point that all those checks that our SymbolTableBuilder AST visitor makes are made before the run-time, so before our interpreter actually evaluates the source program. To drive the point home if we were to interpret the following program:

PROGRAM Part11;
VAR
   x : INTEGER;
BEGIN
   x := 2;
END.

The contents of the symbol table and the run-time GLOBAL_MEMORY right before the program exited would look something like this:

Do you see the difference? Can you see that the symbol table doesn’t hold the value 2 for variable “x”? That’s solely the interpreter’s job now.

Remember the picture from Part 9 where the Symbol Table was used as global memory?

No more! We effectively got rid of the hack where symbol table did double duty as global memory.

Let’s put it all together and test our new interpreter with the following program:

PROGRAM Part11;
VAR
   number : INTEGER;
   a, b   : INTEGER;
   y      : REAL;

BEGIN {Part11}
   number := 2;
   a := number ;
   b := 10 * a + 10 * number DIV 4;
   y := 20 / 7 + 3.14
END.  {Part11}

Save the program as part11.pas and fire up the interpreter:

$ python spi.py part11.pas
Define: INTEGER
Define: REAL
Lookup: INTEGER
Define: <number:INTEGER>
Lookup: INTEGER
Define: <a:INTEGER>
Lookup: INTEGER
Define: <b:INTEGER>
Lookup: REAL
Define: <y:REAL>
Lookup: number
Lookup: a
Lookup: number
Lookup: b
Lookup: a
Lookup: number
Lookup: y

Symbol Table contents:
Symbols: [INTEGER, REAL, <number:INTEGER>, <a:INTEGER>, <b:INTEGER>, <y:REAL>]

Run-time GLOBAL_MEMORY contents:
a = 2
b = 25
number = 2
y = 5.99714285714

I’d like to draw your attention again to the fact that the Interpreter class has nothing to do with building the symbol table and it relies on the SymbolTableBuilder to make sure that the variables in the source code are properly declared before they are used by the Interpreter.

Check your understanding

That’s all for today. In the next article, I’ll talk about scopes and we’ll get our hands dirty with parsing nested procedures. Stay tuned and see you soon! And remember that no matter what, “Keep going!”

P.S. My explanation of the topic of symbols and symbol table management is heavily influenced by the book Language Implementation Patterns by Terence Parr. It’s a terrific book. I think it has the clearest explanation of the topic I’ve ever seen and it also covers class scopes, a subject that I’m not going to cover in the series because we will not be discussing object-oriented Pascal.

P.P.S.: If you can’t wait and want to start digging into compilers, I highly recommend the freely available classic by Jack Crenshaw “Let’s Build a Compiler.”


By the way, I’m writing a book “Let’s Build A Web Server: First Steps” that explains how to write a basic web server from scratch. You can get a feel for the book here, here, and here. Subscribe to the mailing list to get the latest updates about the book and the release date.

OPTIN_FORM_PLACEHOLDER


All articles in this series:

September 21, 2016 01:15 AM