Planet Python
Last update: January 21, 2022 04:40 PM UTC
January 21, 2022
Python Circle
Python Snippets - A mini project built using Django - Github Repository made public
mini project, python snippet sharing website, Django project free, Final year project free, Python Django project free,
Real Python
The Real Python Podcast – Episode #94: Designing for Users and Building a Social Network With Django
Are you looking for a project to practice your Django skills? Designing the fundamental interactions of a social network is an instructive way to explore models and relationships while learning advanced Django skills. This week on the show, we talk with previous guest Martin Breuss about his new four-part series, "Build a Social Network With Django".
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Kushal Das
Using Python to access a Solid Pod
From the project website:
Solid is a specification that lets people store their data securely in decentralized data stores called Pods. Pods are like secure personal web servers for your data.
We can host these Pods in personal servers or at any provider. Everything is tied up based on the user identity, called WebID. It is an HTTP URI described as RDF document.
You can decide who/what can access your data. Applications/humans can use Solid authentication and identify to the Pod server to access the data using open protocols.
How to get a Pod?
The website lists current vendors who provide Pod services. If you want to play around locally, you can run community server on your local system. Or just create an account at solidcommunity.net, and use that to learn more.
Using Python with Solid
We already have a solid-file Python module. You can install it via pip in a virtual environment.
$ python3 -m venv .venv
$ source .venv/bin/activate
$ python3 -m pip install solid-file
For the rest of the example code, I am going to use my Pod at the solidcommunity.net, feel free to replace the username/password and URL values in the code as required.
USERNAME = "kushaldas"
PASSWORD = "******************************************"
IDP = 'https://solidcommunity.net'
POD_ENDPOINT = "https://kushaldas.solidcommunity.net/public/"
from solid.auth import Auth
from solid.solid_api import SolidAPI
auth = Auth()
api = SolidAPI(auth)
auth.login(IDP, USERNAME, PASSWORD)
Here we are importing the module, creating an Auth object and identify using username and password.
Then we will check if a folder exist or not (it does not exist yet), and create the folder in this case.
folder_url = f"{POD_ENDPOINT}/languages/"
if not api.item_exists(folder_url):
print(api.create_folder(folder_url))
The output is <Response [201 Created]>.
Next, we create two text files.
data = io.BytesIO("I ❤️ 🦀".encode("utf-8"))
file_url = f"{folder_url}hello.txt"
print(api.put_file(file_url, data, 'text/plain'))
data = io.BytesIO(b"Already 10 years of SOPA blackout")
msg_url = f"{folder_url}message.txt"
print(api.put_file(msg_url, data, 'text/plain'))
We can then list all of the items under our subfolder.
folder_data = api.read_folder(folder_url)
files = "\n".join(list(map(lambda x: x.name, folder_data.files)))
print(f'Files in the folder: \n{files}')
We can then try to read one of the files we just now created.
resp = api.get(file_url)
print(f"**{resp.text}**")
Output:
Files in the folder:
hello.txt
message.txt
Why am I looking at Solid?
Solid as the specification is evolving along with the community. One usecase for any government organization would be if the citizens can control the access to their own data, and people can authorize who gets to access their data. Solid can become answer to that question. The specification is loose enough to allow building things easily on top of it.
I would love to see Python as a major part of this ecosystem. The solid-file project maintainers are doing a great job. But, we need more related projects, including proper examples of various usecases. Maybe a server too.
Django Weblog
Announcing DjangoCon Europe 2022
We are happy to announce DjangoCon Europe 2022 will take place in Porto, Portugal 🇵🇹 hopefully! Let us explain, and believe us that there is a lot to explain.
DjangoCon Europe is hosted annually by an independent volunteer team. This volunteer team submits a proposal to DSF, which will then select the most suitable one from all the proposals. We knew from the start it would be daunting for a new team to submit for 2022, in these uncertain times. So, we pledged our availability to host another DjangoCon, but only if there was no other team submitting a proposal.
Do not get us wrong, we love the idea of welcoming you all in our city and country, we have been trying for the last two years, but we felt we should give other teams the chance, so we have not submitted an official proposal, just a backup plan, most of all we didn’t want to spend a year without a DjangoCon Europe.
Being here means no other team was available, understandingly so. Our main advantage is: having organised the previous two editions we are able to quickly set up keeping up the momentum.
We will try for the third time to host an in-person event, but a safe one. This means it will be hybrid from the start, both for speakers and participants. We want to give freedom of choice as well as being prepared for unforeseeable issues, which seem to be the new normal. Please keep in mind this might change in a heartbeat and we might end up with yet another online-only event, but let us hope for a bit more.
On another note, you might have noticed the unusual dates for DjangoCon Europe, which usually takes place during the first half of the year. DSF tried to find a new team to host, but due to the lack of viable proposals eventually contacted us in mid-November. Finding a suitable date on such short notice, avoiding other events, and booking a venue left us with little to no choices. To this end, we would like to thank both DEFNA and DSF for the exceptional permission as it will be unusually close to DjangoCon US.
So, DjangoCon Europe 2022 is back again and it’s going to be 5 full days of talks, tutorials and sprints - from September 21 to 25:
- Conference talks: September 21-23 (Wednesday-Friday)
- Sprints: September 24-25 (Saturday and Sunday)
In the near future, we will have more info about the conference, which we will publish on the website. This will include more details about the tickets, talks, workshops, grants, code of conduct, etc. For now, here is a summary of that info.
Grants
As with past years, there will be a travel grants program to assist people with financial difficulties, people who are under-represented or from marginalised groups - allowing access to an event that otherwise would be very difficult position for them to attend;
Sponsors
If you're interested in sponsoring the event, please get in touch at sponsors@djangocon.eu.
Talk proposals
You can already start to prepare your talk, and for that, we recommend that you watch the talk “How To Get On This Stage (And What To Do When You Get There)” by Mark Smith. If you think you have something great to talk about – start to prepare your talk! If you are unsure, talk it over with somebody, or go to Slack to find previous speakers and participants to discuss your idea with. When in doubt, submit your talk!
Volunteers
As you can imagine there is a lot to do, but it's very much worth it – DjangoCon Europe is an extremely friendly, open, inclusive, and informative (for beginners and advanced users alike) conference. Join us regardless of your prior experience: this is also an opportunity to learn! In other words, you don't have to be an expert to join. Below are the teams and their activities/responsibilities that we seek help with:
- Communications: Press, community relations, announcements, social media, attendee tools, volunteer coordination
- Support and hospitality: Helpdesk, attendee support contact, visa help, travel management, chat support for attendees, on-site volunteer organization, speaker support;
- Financial aid and diversity advocate: Setup, grant selection, aid organization, accessibility considerations, outreach on-site;
- Sponsors: Outreach to companies, organizing their logistics at the event and other types of visibility;
- Program: Committee work, talk selection, scheduling, session chairs, sprint/open space/keynote/lightning talks session organization (we will open the CFP soon!);
- Code of Conduct: Drafting documents, handling of requests and issues.
You can apply through this form here.
Your location before and during the event is not significant, since it will be hosted in a hybrid format. We can do all things that need to be done in Porto ourselves. The only important thing is that you have the energy and free time to help organize a wonderful DjangoCon Europe. The official language of all these prior activities will be English, as well as the conference itself.
--We expect new challenges but pledge our hearts and minds to do the best DjangoCon Europe we can, never giving up under these strenuous conditions. Please consider volunteering and join us, we need you!
We hope we'll see you all at DjangoCon Europe 2022, and don't forget to follow us @djangoconeurope on Twitter, and also join our dedicated Slack channel.
Hoping for the best,
The DjangoCon Europe 2022 Organisers
January 20, 2022
Python for Beginners
Check For Perfect Number In Python
We have named numbers based on their specialties. One such number is a perfect number. In this article, we will discuss the properties of perfect numbers. We will also implement a program to check for a perfect number in python.
What Is A Perfect Number?
A number is called a perfect number if it is equal to the sum of all of its factors excluding the number itself. If we consider the number itself in the sum, we can say that the sum of all the factors of a perfect number is double the given number.
For example, consider the number 6. It has four factors i.e. 1,2,3, and 6. As we are excluding the number itself, the sum of other factors i.e. 1, 2, and 3 is 6. Hence, 6 is a perfect number.
Alternatively, the sum of all the factors of 6 is 1+2+3+6 i.e. 12, which is double the number itself. Hence, 6 is a perfect number.
Let us take another number 10. The factors of 10 are 1,2,5, and 10. The sum of all the factors of 10 equals 18, which is not double the given number. Hence, 10 is not a perfect number.
Check For Perfect Number In Python
To check for a perfect number, we will first find its factors. After that, we will check if the sum of all the factors is double the given number or not.
To find the factors of the given number N, we will divide the number by all the numbers starting from 1 to N. The numbers that completely divide the given number will be declared as factors of N. We will store these factors in a list as follows.
dedef calculate_factors(N):
factors = []
for i in range(1, N + 1):
if N % i == 0:
factors.append(i)
return factors
input_number = 10
output = calculate_factors(input_number)
print("factors of {} are {}".format(input_number, output))
Output:
factors of 10 are [1, 2, 5, 10]After finding the factors of the given number, we will find the sum of the factors using the sum() function. After finding the sum, we will check if the sum is double the given number or not. If yes, we will say that the given number is a perfect number. Otherwise not.
We can implement this logic to check for a Perfect number in python as follows.
def calculate_factors(N):
factors = []
for i in range(1, N + 1):
if N % i == 0:
factors.append(i)
return factors
def check_perfect_number(N):
factors = calculate_factors(N)
sumOfFactors = sum(factors)
if sumOfFactors // 2 == N:
return True
else:
return False
input_number = 10
output = check_perfect_number(input_number)
print("{} is a perfect number: {}".format(input_number, output))
input_number = 6
output = check_perfect_number(input_number)
print("{} is a perfect number: {}".format(input_number, output))Output:
10 is a perfect number: False
6 is a perfect number: TrueConclusion
In this article, we have discussed what a perfect number is. We have also implemented a program to check whether a given number is a perfect number or not. To know more about numbers in python, you can read this article on decimal numbers in python. You might also like this article on complex numbers in python.
The post Check For Perfect Number In Python appeared first on PythonForBeginners.com.
Artem Rys
Detect unused secrets in your Github repositories
With the increasing number of security threats, it is important to use (and follow the best practices) only those “secrets” that you…
Stack Abuse
Image Classification with Transfer Learning in Keras - Create Cutting Edge CNN Models
Introduction
Deep Learning models are very versatile and powerful - they're routinely outperforming humans in narrow tasks, and their generalization power is increasing at a rapid rate. New models are being released and benchmarked against community-accepted datasets frequently, and keeping up with all of them is getting harder.
Most of these models are open source, and you can implement them yourself as well.
This means that the average enthusiast can load in and play around with the cutting edge models in their home, on very average machines, not only to gain a deeper understanding and appreciation of the craft, but also to contribute to the scientific discourse and publish their own improvements whenever they're made.
In this guide, you'll learn how to use pre-trained, cutting edge Deep Learning models for Image Classification and repurpose them for your own specific application. This way, you're leveraging their high performance, ingenious architectures and someone else's training time - while applying these models to your own domain.
All of the code written in the guide is also available on GitHub.
Transfer Learning for Computer Vision and Convolutional Neural Networks (CNNs)
Knowledge and knowledge representations are very universal. A computer vision model trained on one dataset learns to recognize patterns that might be very prevalent in many other datasets.
Notably, in "Deep Learning for the Life Sciences", by Bharath Ramsundar, Peter Eastman, Patrick Walters and Vijay Pande, it's noted that:
"There have been multiple studies looking into the use of recommendation system algorithms for use in molecular binding prediction. Machine learning architectures used in one field tend to carry over to other fields, so it’s important to retain the flexibility needed for innovative work."
For instance, straight and curved lines, which are typically learned at a lower level of a CNN hierarchy are bound to be present in practically all datasets. Some high-level features, such as the ones that distinguish a bee from an ant are going to be represented and learned much higher in the hierarchy:
The "fine line" between these is what you can reuse! Depending on the level of similarity between your dataset and the one a model's been pre-trained on, you may be able to reuse a small or large portion of it.
A model that classifies human-made structures (trained on a dataset such as the Places365) and a model that classifies animals (trained on a dataset such as ImageNet) are bound to have some shared patterns, although, not a lot.
You might want to train a model to distinguish, say, buses and cars for a self-driving car's vision system. You may also reasonably choose to use a very performant architecture that has proven to work well on datasets similar to yours. Then, the long process of training begins, and you end up having a performant model of your own!
However, if another model is likely to have similar representations on lower and higher levels of abstraction, there's no need to re-train a model from scratch. You may decide to use some of the already pre-trained weights, which are just as applicable to your own application of the model as they were applicable to the creator of the original architecture. You'd be transferring some of the knowledge from an already existing model to a new one, and this is known as Transfer Learning.
The closer the dataset of a pre-trained model is to your own, the more you can transfer. The more you can transfer, the more of your own time and computation you can save. It's worth remembering that training neural networks does have a carbon footprint, so you're not only saving time!
Typically, Transfer Learning is done by loading a pre-trained model, and freezing its layers. In many cases, you can just cut off the classification layer (the final layers, or, head) and just re-train the classification layer, while keeping all of the other abstraction layers intact. In other cases, you may decide to re-train several layers in the hierarchy instead, and this is typically done when the datasets contain sufficiently different data points that re-training multiple layers is warranted. You may also decide to re-train the entirety of the model to fine-tune all of the layers.
These two approaches can be summarized as:
- Using the Convolutional Network as a Feature Extractor
- Fine-Tuning the Convolutional Network
In the former, you use the underlying entropic capacity of the model as a fixed feature extractor, and just train a dense network on top to discern between these features. In the latter, you fine-tune the entire (or portion of the) convolutional network, if it doesn't already have representative feature maps for some other more specific dataset, while also relying on the already trained feature maps.
Here's a visual representation of how Transfer Learning works:
Established and Cutting Edge Image Classification Models
Many models exist out there, and for well-known datasets, you're likely to find hundreds of well-performing models published in online repositories and papers. A good holistic view of models trained on the ImageNet dataset can be seen at PapersWithCode.
Some of the well-known published architectures that have subsequently been ported into many Deep Learning frameworks include:
- EfficientNet
- SENet
- Xception
- ResNet
- VGGNet
- AlexNet
- LeNet-5
The list of models on PapersWithCode is constantly being updated, and you shouldn't hang up on the position of these models there. Many of them are outperformed by various other models as of writing, and many of the new models are actually based on the ones outlined in the list above. It's worth noting that Transfer Learning actually played an important role in the newer, higher accuracy models!
The downside is - a lot of the newest models aren't ported as pre-trained models within frameworks such as Tensorflow and PyTorch. It's not like you'll be losing out on a lot of the performance, so going with any of the well-established ones isn't really bad at all.
Transfer Learning with Keras - Adapting Existing Models
With Keras, the pre-trained models are available under the tensorflow.keras.applications module. Each model has its own sub-module and class. When loading a model in, you can set a couple of optional arguments to control how the models are being loaded in.
For instance, the weights argument, if present, defines the pre-trained weights. If omitted, only the architecture (untrained network) will be loaded in. If you supply the name of a dataset - a pre-trained network will be returned for that dataset.
Additionally, since you'll most likely be removing the top layer(s) for Transfer Learning, the include_top argument is used to define whether the top layer(s) should be present or not!
import tensorflow.keras.applications as models
# 98 MB
resnet = models.resnet50.ResNet50(weights='imagenet', include_top=False)
# 528MB
vgg16 = models.vgg16.VGG16(weights='imagenet', include_top=False)
# 23MB
nnm = models.NASNetMobile(weights='imagenet', include_top=False)
# etc...
Note: If you've never loaded pre-trained models before, they'll be downloaded over an internet connection. This may take anywhere between a few seconds and a couple of minutes, depending on your internet speed and the size of the models. The size of pre-trained models spans from as little as 14MB (typically lower for Mobile models) to as high as 549MB.
EfficientNet is a family of networks that are quite performant, scalable and, well, efficient. They were made with reducing learnable parameters in mind, so they only have 4M parameters to train. While 4M is still a large number, consider that VGG16, for instance, has 20M. On a home setup, this also helps with training times significantly!
Let's load in one of the members of the EfficientNet family - EfficientNet-B0:
effnet = keras.applications.EfficientNetB0(weights='imagenet', include_top=False)
effnet.summary()
This results in:
Model: "efficientnetb0"
________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
========================================================================================
input_1 (InputLayer) [(None, 224, 224, 3) 0
________________________________________________________________________________________
rescaling (Rescaling) (None, 224, 224, 3) 0 input_1[0][0]
________________________________________________________________________________________
...
...
block7a_project_conv (Conv2D) (None, 7, 7, 320) 368640 block7a_se_excite[0][0]
________________________________________________________________________________________
block7a_project_bn (BatchNormal (None, 7, 7, 320) 1280 block7a_project_conv[0][0]
========================================================================================
Total params: 3,634,851
Trainable params: 3,592,828
Non-trainable params: 42,023
________________________________________________________________________________________
On the other hand, if we were to load in EfficientNet-B0 with the top included, we'd also have a few new layers at the end, that were trained to classify the data for ImageNet. This is the top of the model that we'll be training ourselves for our own application:
effnet = keras.applications.EfficientNetB0(weights='imagenet', include_top=True)
effnet.summary()
This would include the Flatten and Dense layers, which then prop up the parameter size significantly:
Model: "efficientnetb0"
_________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
=========================================================================================
input_11 (InputLayer) [(None, 224, 224, 3) 0
_________________________________________________________________________________________
...
...
_________________________________________________________________________________________
top_conv (Conv2D) (None, 7, 7, 1280) 409600 block7a_project_bn[0][0]
_________________________________________________________________________________________
top_bn (BatchNormalization) (None, 7, 7, 1280) 5120 top_conv[0][0]
_________________________________________________________________________________________
top_activation (Activation) (None, 7, 7, 1280) 0 top_bn[0][0]
_________________________________________________________________________________________
avg_pool (GlobalAveragePooling2 (None, 1280) 0 top_activation[0][0]
_________________________________________________________________________________________
top_dropout (Dropout) (None, 1280) 0 avg_pool[0][0]
_________________________________________________________________________________________
predictions (Dense) (None, 1000) 1281000 top_dropout[0][0]
=========================================================================================
Total params: 5,330,571
Trainable params: 5,288,548
Non-trainable params: 42,023
_________________________________________________________________________________________
Again, we won't be using the top layers, as we'll be adding our own top to the EfficientNet model and re-training only the ones we add on top. It is worth noting what the architecture is already built with! They seem to be using a Conv2D layer, followed by a BatchNormalization, GlobalAveragePooling2D and Dropout before the final Dense classification layer. While we don't have to strictly follow this approach (and other approaches may prove to be better for another dataset), it's reasonable to remember how the original top looked like.
Note: Data preprocessing plays a crucial role in model training, and most models will have different preprocessing pipelines. You don't have to perform guesswork here! Where applicable, a model comes with its own preprocess_input() function.
The preprocess_input() function applies the same preprocessing steps to the input as they were applied during training. You can import the function from the respective module of the model, if a model resides in its own module. For instance, VGG16 has its own preprocess_input function:
from keras.applications.vgg16 import preprocess_input
That being said, loading in a model, preprocessing input for it and predicting a result in Keras is as easy as:
import tensorflow.keras.applications as models
from keras.applications.vgg16 import preprocess_input
vgg16 = models.vgg16.VGG16(weights='imagenet', include_top=True)
img = # get data
img = preprocess_input(img)
pred = vgg16.predict(img)
Note: Not all models have a dedicated preprocess_input() function, because the preprocessing is done within the model itself. For instance, EfficientNet that we'll be using doesn't have its own dedicated preprocessing function, as the Rescaling layer takes care of that.
That's it! Now, since the pred array doesn't really contain human-readable data, you can also import the decode_predictions() function alongside the preprocess_input() function from a module. Alternatively, you can import the generic decode_predictions() function that also applies to models that don't have their dedicated modules:
from keras.applications.model_name import preprocess_input, decode_predictions
# OR
from keras.applications.imagenet_utils import decode_predictions
# ...
print(decode_predictions(pred))
Tying this together, let's get an image of a black bear via urllib, save that file into a target size suitable for EfficientNet (the input layer expects a shape of (None, 224, 224, 3)) and classify it with the pre-trained model:
from tensorflow import keras
from keras.applications.vgg16 import preprocess_input, decode_predictions
from tensorflow.keras.preprocessing import image
import urllib.request
import matplotlib.pyplot as plt
import numpy as np
# Public domain image
url = 'https://upload.wikimedia.org/wikipedia/commons/0/02/Black_bear_large.jpg'
urllib.request.urlretrieve(url, 'bear.jpg')
# Load image and resize (doesn't keep aspect ratio)
img = image.load_img('bear.jpg', target_size=(224, 224))
# Turn to array of shape (224, 224, 3)
img = image.img_to_array(img)
# Expand array into (1, 224, 224, 3)
img = np.expand_dims(img, 0)
# Preprocess for models that have specific preprocess_input() function
# img_preprocessed = preprocess_input(img)
# Load model and run prediction
effnet = keras.applications.EfficientNetB0(weights='imagenet', include_top=True)
pred = effnet.predict(img)
print(decode_predictions(pred))
This results in:
[[
('n02133161', 'American_black_bear', 0.6024658),
('n02132136', 'brown_bear', 0.1457715),
('n02134418', 'sloth_bear', 0.09819221),
('n02510455', 'giant_panda', 0.0069221947),
('n02509815', 'lesser_panda', 0.005077324)
]]
It's fairly certain that the image is an image of an American Black Bear, which is right! When preprocessed with a preprocessing function, the image may change significantly. For instance, VGG16's preprocessing function would change the color of the bear's fur:
It looks a lot more brown now! If we were to feed this image into EfficientNet, it'd think it's a brown bear:
[[
('n02132136', 'brown_bear', 0.7152758),
('n02133161', 'American_black_bear', 0.15667434),
('n02134418', 'sloth_bear', 0.012813852),
('n02134084', 'ice_bear', 0.0067828503), ('n02117135', 'hyena', 0.0050422684)
]]
Awesome! The model works. Now, let's add a new top to it and re-train the top to perform classification for something outside of the ImageNet set.
Adding a New Top to a pre-trained Model
When performing transfer learning, you'll be loading models without tops, or remove them manually:
# Load without top
# When adding new layers, we also need to define the input_shape
# so that the new Dense layers have a fixed input_shape as well
effnet_base = keras.applications.EfficientNetB0(weights='imagenet',
include_top=False,
input_shape=((224, 224, 3)))
# Or load the full model
full_effnet = keras.applications.EfficientNetB0(weights='imagenet',
include_top=True,
input_shape=((224, 224, 3)))
# And then remove X layers from the top
trimmed_effnet = keras.Model(inputs=full_effnet.input, outputs=full_effnet.layers[-3].output)
We'll be going with the first option since it's more convenient. Depending on whether you'd like to fine-tune the convolutional blocks or not - you'll either freeze or won't freeze them. Say we want to use the underlying pre-trained feature maps and freeze the layers so that we only re-train the new classification layers at the top:
effnet_base.trainable = False
You don't need to iterate through the model and set each layer to be trainable or not, though you also can. If you'd like to turn off the first n layers, and allow some higher-level feature maps to be fine-tuned, but leave the lower-level ones untouched, you can:
for layer in effnet_base.layers[:-2]:
layer.trainable = False
Here, we've set all layers in the base model to be untrainable, except for the last two. If we check the model, there are only ~2.5K trainable parameters now:
effnet_base.summary()
# ...
=========================================================================================
Total params: 4,049,571
Trainable params: 2,560
Non-trainable params: 4,047,011
_________________________________________________________________________________________
Now, let's define a Sequential model that'll be put on top of this effnet_base. Fortunately, chaining models in Keras is as easy as making a new model and putting it on top of another one! You can leverage the Functional API and just chain a few new layers on top of a model.
Let's add a Conv2D layer, a BatchNormalization layer, a GlobalAveragePooling2D layer, some Dropout and a couple of fully connected layers after a Flatten:
conv2d = keras.layers.Conv2D(7, 7)(effnet_base.output, training=False)
bn = keras.layers.BatchNormalization()(conv2d)
gap = keras.layers.GlobalAveragePooling2D()(bn)
do = keras.layers.Dropout(0.2)(gap)
flatten = keras.layers.Flatten()(gap)
fc1 = keras.layers.Dense(512, activation='relu')(flatten)
output = keras.layers.Dense(100, activation='softmax')(fc1)
new_model = keras.Model(inputs=effnet_base.input, outputs=output)
Note: When adding the layers of the EfficientNet, we set the training to False. This puts the network in inference mode instead of training mode and it's a different parameter than the trainable we've set to False earlier. This is a crucial step if you wish to unfreeze layers later on! BatchNormalization, is computes moving statistics. When unfrozen, it'll start applying updates to parameters again, and will "undo" the training done before fine-tuning. Since TF 2.0, setting the model's trainable as False also turns training to False but only for BatchNormalization layers, so this step is unnecessary for versions after TF 2.0.
Alternatively, you can use the Sequential API and call the add() method multiple times:
new_model = keras.Sequential()
new_model.add(effnet_base) # Add entire model
new_model.add(keras.layers.Conv2D(7,7))
new_model.add(keras.layers.BatchNormalization())
new_model.add(keras.layers.GlobalAveragePooling2D())
new_model.add(keras.layers.Dropout(0.3))
new_model.add(keras.layers.Flatten())
new_model.add(keras.layers.Dense(512, activation='relu'))
new_model.add(keras.layers.Dense(100, activation='softmax'))
This adds the entire model as a layer itself, so it's treated as one entity:
Layer: 0, Trainable: False # Entire EfficientNet model
Layer: 1, Trainable: True
Layer: 2, Trainable: True
...
On the other hand, you can extract all of the layers and add them instead as separate entities, by adding the output of effnet_base:
new_model = keras.Sequential()
new_model.add(effnet_base.output) # Add unwrapped layers
new_model.add(keras.layers.Conv2D(7,7))
new_model.add(keras.layers.BatchNormalization())
new_model.add(keras.layers.GlobalAveragePooling2D())
new_model.add(keras.layers.Dropout(0.3))
new_model.add(keras.layers.Dense(100, activation='softmax'))
In any of these cases - we've added 10 output classes, since we'll be using the CIFAR10 dataset later on, which has 10 classes! Let's take a look at the trainable layers in the network:
for index, layer in enumerate(new_model.layers):
print("Layer: {}, Trainable: {}".format(index, layer.trainable))
This results in:
Layer: 0, Trainable: False
Layer: 1, Trainable: False
Layer: 2, Trainable: False
...
Layer: 235, Trainable: False
Layer: 236, Trainable: False
Layer: 237, Trainable: True
Layer: 238, Trainable: True
Layer: 239, Trainable: True
Layer: 240, Trainable: True
Layer: 241, Trainable: True
Awesome! Let's load in the dataset, preprocess it and re-train the classification layers on it.
Loading and Preprocessing Data
We'll be working with the CIFAR10 dataset. This is a dataset that's not too hard to classify since it only has 10 classes, and we'll be leveraging a well-received architecture to help us in that endeavor.
It's "older brother", CIFAR100 is a genuinely hard one to work with. It has 50.000 images, with 100 labels, meaning each class has only 100 samples. This is extremely hard to get right on so few labels, and almost all well-performing models on the dataset use heavy data augmentation.
Data augmentation is an art and science in and of itself, and is out of scope for this guide - so we'll only be diversifying the dataset with a couple of random transformations.
For brevity's sake, we'll stick to CIFAR10, to emulate the dataset you'll be working with yourself!
Note: Keras' datasets module contains a few datasets, but these are mainly meant for benchmarking and learning. We can use tensorflow_datasets to get access to a much larger corpora of datasets! Alternatively, you can use any other source, such as Kaggle or academic repositories.
We'll be using tensorflow_datasets to download the CIFAR10 dataset, get the labels and the number of classes:
import tensorflow_datasets as tfds
import tensorflow as tf
dataset, info = tfds.load("cifar10", as_supervised=True, with_info=True)
# Save class_names and n_classes for later
class_names = info.features["label"].names
n_classes = info.features["label"].num_classes
print(class_names) # ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
print(n_classes) # 10
You can get to know the dataset like this, but we won't be diving into that right now. Let's split it into a train_set, valid_set and test_set instead:
test_set, valid_set, train_set = tfds.load("cifar10",
split=["train[:10%]", "train[10%:25%]", "train[25%:]"],
as_supervised=True)
print("Train set size: ", len(train_set)) # Train set size: 37500
print("Test set size: ", len(test_set)) # Test set size: 5000
print("Valid set size: ", len(valid_set)) # Valid set size: 7500
Note: The split argument expects train and test keywords, and there's no valid keyword that can be used to extract a validation set. Because of this, we need to perform the slightly awkward and clunky split as we have - with a 10/15/75 split.
Now, the CIFAR10 images are significantly different from the ImageNet images! Namely, CIFAR10 images are just 32x32, while our EfficientNet model expects 224x224 images. We'll want to resize the images in any case. We might also want to apply some transformation functions on duplicate images to artificially expand the sample size per class if the dataset doesn't have enough of them. In the case of CIFAR10, this isn't an issue, as there are enough images per class, but with CIFAR100 - it's a different story. It's worth noting that, when upscaling images that are this small, even humans have a significant difficulty discerning what's on some of the images.
For instance, here are a few non-resized images:
Can you tell what's on these with confidence? Consider the lifelong amount of context you have for these images, as well, which the model doesn't have. It's worth keeping this in mind when you train it and observe the accuracy.
Let's define a preprocessing function for each image and its associated label:
def preprocess_image(image, label):
# Resize to EfficientNet size
resized_image = tf.image.resize(image, [224, 224])
# Random flips and rotations (fully optional, and doesn't impact performance much no augmentation takes place)
# If we run this function multiple times, it'll net different results
img = tf.image.random_flip_left_right(resized_image)
img = tf.image.random_flip_up_down(img)
img = tf.image.rot90(img)
# Preprocess image with model-specific function if it has one
# img = preprocess_input(img)
return img, label
And finally, we'll want to apply this function to each image in the sets! We haven't performed augmentation by expanding the sets here, though you could. For brevity's sake, we'll avoid performing data augmentation.
This is easily done via the map() function. Since the input into the network also expects batches ((None, 224, 224, 3) instead of (224, 224, 3)) - we'll also batch() the datasets after mapping:
train_set = train_set.map(preprocess_image).batch(32).prefetch(1)
test_set = test_set.map(preprocess_image).batch(32).prefetch(1)
valid_set = valid_set.map(preprocess_image).batch(32).prefetch(1)
Note: The prefetch() function is optional but helps with efficiency. As the model is training on a single batch, the prefetch() function pre-fetches the next batch so it's not waited upon when the training step is finished.
Finally, we can train the model!
Training a Model
With the data loaded, preprocessed and split into adequate sets - we can train the model on it. The optimizer as well as its hyperparameters, loss function and metrics generally depend on your specific task.
Since we're doing sparse classification, a sparse_categorical_crossentropy loss should work well, and the Adam optimizer is a reasonable default loss function. Let's compile the model, and train it on a few epochs. It's worth remembering that most of the layers in the network are frozen! We're only training the new classifier on top of the extracted feature maps.
Only once we train the top layers, we may decide to unfreeze the feature extraction layers, and let them fine-tune a bit more. This step is optional, and in many cases, you won't unfreeze them (mainly when working with really large networks). A good rule of thumb is to try and compare the datasets and guesstimate which levels of the hierarchy you can re-use without re-training.
If they're really different, you probably chose a network pre-trained on the wrong dataset. It wouldn't be efficient to use feature extraction of Places365 (man-made objects) for classifying animals. However, it would make sense to use a network trained on ImageNet (which has various objects, animals, plants and humans) and then use it for a different dataset with relatively similar categories, such as CIFAR10.
Note: Depending on the architecture you're using, unfreezing the layers might be a bad idea, due to their size. There's a good chance that your local machine will run out of memory when trying to tackle a 20M parameter model and loading a training step into the RAM/VRAM. When possible, try to find an architecture pre-trained on a dataset that's sufficiently similar to yours that you don't have to change the feature extractors. If you have to, it's not impossible but does make the process much slower. We'll cover that later.
Let's train the new network (really, only the top of it) for 10 epochs:
optimizer = keras.optimizers.Adam(learning_rate=2e-5)
new_model.compile(loss="sparse_categorical_crossentropy",
optimizer=optimizer,
metrics=["accuracy"])
history = new_model.fit(train_set,
epochs=10,
validation_data=valid_set)
Note: This may take some time and is ideally done on a GPU. Depending on how large the model is, and the dataset being fed into it. If you don't have access to a GPU, it's advised to run this code on any of the cloud providers that give you access to a free GPU, such as Google Collab, Kaggle Notebooks, etc. Each epoch can take anywhere from 60 seconds on stronger GPUs to 10 minutes, on weaker ones.
This is the point in which you sit back and go grab a coffee (or tea)! After 10 epochs, the train and validation accuracy are looking good:
Epoch 1/10
1172/1172 [==============================] - 92s 76ms/step - loss: 1.6582 - accuracy: 0.6373 - val_loss: 1.1582 - val_accuracy: 0.7935
...
Epoch 10/10
1172/1172 [==============================] - 89s 76ms/step - loss: 0.0911 - accuracy: 0.9781 - val_loss: 0.3847 - val_accuracy: 0.8792
~98% on the training set and ~88% on the validation set - it clearly overfit, but not too badly. Let's test it out and plot the learning curves.
Testing a Model
Let's first test this model out, before trying to unfreeze all of the layers and seeing if we can fine-tune it then:
new_model.evaluate(test_set)
# 157/157 [==============================] - 10s 61ms/step - loss: 0.3778 - accuracy: 0.8798
# [0.3778476417064667, 0.879800021648407]
88% on the testing set, and extremely close to the accuracy on the validation set! Looks like our model is generalizing well, but there's still room for improvement. Let's take a look at the learning curves.
The training curves are to be expected - they're pretty short since we only trained for 10 epochs, but they've quickly plateaued, so we probably wouldn't have gotten much better performance with more epochs. While oscillations do occur and the accuracy could very well rise in Epoch 11 - it's not too likely, so we'll miss out on the chance:
Can we fine-tune this network further? We've replaced and re-trained the top layers concerned with classification of feature maps. Let's try unfreezing the convolutional layers and fine-tuning them as well!
Unfreezing Layers - Fine-Tuning a Network Trained with Transfer Learning
Once you've finished re-training the top layers, you can close the deal and be happy with your model. For instance, suppose you got a 95% accuracy - you seriously don't need to go further. However, why not?
If you can squeeze out an additional 1% in accuracy, it might not sound like a lot, but consider the other end of the trade. If your model has a 95% accuracy on 100 samples, it misclassified 5 samples. If you up that to 96% accuracy, it misclassified 4 samples.
The 1% of accuracy translates to a 25% decrease in false classifications.
Whatever you can further squeeze out of your model can actually make a significant difference on the number of incorrect classifications. We have a pretty satisfactory 88% accuracy with our model, but we can most probably squeeze more out of it if we just slightly re-train the feature extractors. Again, the images in CIFAR10 are much smaller than ImageNet images, and it's almost as if someone with great eyesight suddenly gained a huge prescription and only saw the world through blurry eyes. The feature maps have to be at least somewhat different!
Let's save the model into a file so we don't lose the progress, and unfreeze/fine-tune a loaded copy, so we don't accidentally mess up the weights on the original one:
new_model.save('effnet_transfer_learning.h5')
loaded_model = keras.models.load_model('effnet_transfer_learning.h5')
Now, we can fiddle around and change the loaded_model without impacting new_model! To start out, we'll want to change the loaded_model from inference mode back to training mode - i.e. unfreeze the layers so that they're trainable again.
Note: Again, if a network uses BatchNormalization (and most do), you'll want to keep it frozen before fine-tuning a network. Since we're not freezing the entire base network anymore, we'll just freeze the BatchNormalization layers instead and allow other layers to be altered.
Let's turn off the BatchNormalization layers so our training doesn't go down the drain:
for layer in loaded_model.layers:
if isinstance(layer, keras.layers.BatchNormalization):
layer.trainable = False
else:
layer.trainable = True
for index, layer in enumerate(loaded_model.layers):
print("Layer: {}, Trainable: {}".format(index, layer.trainable))
Let's check if that worked:
Layer: 0, Trainable: True
Layer: 1, Trainable: True
Layer: 2, Trainable: True
Layer: 3, Trainable: True
Layer: 4, Trainable: True
Layer: 5, Trainable: False
Layer: 6, Trainable: True
Layer: 7, Trainable: True
Layer: 8, Trainable: False
...
Awesome! Before we can do anything with the model, to "solidify" the trainability, we have to recompile it. This time around, we'll be using a smaller learning_rate, since we don't want to alter the network much at all, and just want to fine-tune some of the feature extracting capabilities and the new classification layer on top:
optimizer = keras.optimizers.Adam(learning_rate=1e-6)
# Recompile after turning to trainable
loaded_model.compile(loss="sparse_categorical_crossentropy",
optimizer=optimizer,
metrics=["accuracy"])
history = loaded_model.fit(train_set,
epochs=10,
validation_data=valid_set)
Again, this may take some time - so sip on another hot beverage of your choice while this runs in the background. Once it finishes, it should reach up to 90% in validation accuracy and 91% on the test set:
Epoch 1/10
1172/1172 [==============================] - 389s 328ms/step - loss: 0.0552 - accuracy: 0.9863 - val_loss: 0.3493 - val_accuracy: 0.8941
...
Epoch 10/10
1172/1172 [==============================] - 376s 321ms/step - loss: 0.0196 - accuracy: 0.9955 - val_loss: 0.3373 - val_accuracy: 0.9043
Again, this isn't a huge jump from the perspective of accuracy itself, but it is a much more significant reduction in the proportion of misclassifications. Let's evaluate it and visualize some of the predictions:
loaded_model.evaluate(test_set)
# 157/157 [==============================] - 10s 61ms/step - loss: 0.3177 - accuracy: 0.9108
# [0.3176877200603485, 0.9107999801635742]
fig = plt.figure(figsize=(15, 10))
i = 1
for entry in test_set.take(25):
# Predict, get the raw Numpy prediction probabilities
# Reshape entry to the model's expected input shape
pred = np.argmax(loaded_model.predict(entry[0].numpy()[0].reshape(1, 224, 224, 3)))
# Get sample image as numpy array
sample_image = entry[0].numpy()[0]
# Get associated label
sample_label = class_names[entry[1].numpy()[0]]
# Get human label based on the prediction
prediction_label = class_names[pred]
ax = fig.add_subplot(5, 5, i)
# Plot image and sample_label alongside prediction_label
ax.imshow(np.array(sample_image, np.int32))
ax.set_title("Actual: %s\nPred: %s" % (sample_label, prediction_label))
i = i+1
plt.tight_layout()
plt.show()
Here, the only misclassification we can see in the first 25 images is a truck being misclassified as a horse. This is likely due to the context - it's in a forest and is brown and elongated. This also fits the description of a horse to a degree, so it's not too surprising that in a blurry, small (224x224) image, the truck was misclassified.
Another thing that definitely didn't help is that it looks like the gate of the truck is open, which could look like the neck of a horse as it feeds on grass.
Conclusion
Transfer Learning is the process of transferring already learned knowledge representations from one model to another, when applicable.
This concludes this guide to Transfer Learning for Image Classification with Keras and Tensorflow. We've started out with taking a look at what Transfer Learning is and how knowledge representations can be shared between models and architectures.
Then, we've taken a look at some of the most popular and cutting edge models for Image Classification released publically, and piggy-backed on one of them - EfficientNet - to help us in classifying some of our own data. We've taken a look at how to load and examine pre-trained models, how to work with their layers, predict with them and decode the results, as well as how to define your own layers and intertwine them with the existing architecture.
Finally, we've loaded and preprocessed a dataset, and trained our new classification top layers on it, before unfreezing the layers and fine-tuning it further through several additional epochs.
John Ludhi/nbshare.io
Python Type Isinstance
Instance and Type both are used to check the type of object. Instance can check the type of subclass too where as type can't.
!python --version
Check if type is Integer
x = 1
print(type(x))
Check if type is Float
x = 1.5
print(type(x))
Check if type is String
x = 'john'
print(type(x))
x = 'j'
print(type(x))
Check if type is Class
class demo():
pass
print(type(demo()))
type(demo())==demo
isinstance can be used to check the type of object.
x = 1
isinstance(x,int)
isinstance(x,float)
x = 1.2
isinstance(x,float)
isinstance can check the type of subclass too.
class computer():
pass
class laptop(computer):
pass
isinstance(computer(),computer)
isinstance(laptop(),computer)
Note, type can not check the type of subclass.
type(laptop())==computer
January 19, 2022
David Amos
How To Stay Curious as a Coder
It&aposs easy to be complacent about curiosity. Our lives are filled with stress. We craft routines around that stress. Sometimes those routines turn life into an inescapable turnstile. It isn&apost easy to be curious when your life depends on keeping the wheel turning.
I spend a lot of my time solving problems with Python. Problem-solving demands creativity. Curiosity breeds creativity. The demand for creative software solutions is constant, but curiosity comes and goes.
When your curiosity wanes, don&apost resign yourself to the idea that some people are naturally more curious about things and that perhaps you&aposre not one of them. Curiosity is not an immutable trait. You can learn how to be more curious.
How To Cultivate Curiosity
Curiosity is contagious. Follow curious people, and you&aposll likely catch the bug. Once your mind is infected with a framework for curiosity, you instinctively apply that framework to everything.
Mystery is the key to curiosity. Embrace feeling puzzled because puzzles are everywhere. Question absolutes and axioms, even the ones in this article. Devise experiments to test premises.
Learn to recognize when your curiosity ebbs. I know that my curiosity is critically low whenever I stop reading consistently. I&aposm not talking about reading code — I mean reading anything. It&aposs a case of intellectual malnutrition, and the cure is a content-rich diet.
Consume Content Voraciously
Get into a habit of passive consumption. Listen to podcasts during your commute or household chores. Watch documentaries during your lunch break. If your mind wanders, let it. Don&apost focus on one topic. A diverse content diet is more likely to reveal a mystery that stirs you.
Actively consume content about mysteries that particularly draw your attention. Read long-form articles and books. Listen intently and take notes. But refrain from forcing yourself into active consumption. Forced activity stifles joy.
Dive Deeply Into Things That Interest You Right Now
Unlike diets that nourish our bodies, a curiosity-inducing content diet has no plan and no schedule. Explore topics haphazardly, but also explore them deeply. Don&apost delay your curiosity. Pursue topics that interest you right now.
Get lost in rabbit holes, but avoid ones generated algorithmically. Algorithms have a knack for surfacing similar content. Actively search for answers to questions that arise.
Embrace Learning as a Conversation
Curiosity is a tug-of-war between selfishness and humility. You&aposre compelled by a primal urge to know more. Yet learning requires you to expose a gap in your understanding, to recognize that there is still more to be known.
Learning is a conversation. Participate in that conversation. Ask questions. A good question synthesizes existing knowledge and addresses an acknowledged gap in understanding.
- Read: Consume Content
- Evaluate: Think critically, ask questions, and discover solutions
- Print: Share what you&aposve learned
- Loop: Interpret feedback and start over
How To Practice Curiosity as a Coder
Getting better at being curious requires practicing curiosity. Fortunately, coding is full of puzzles. There are packages and tools to discover, languages to learn, and implementation details to explore.
Here are two techniques I&aposve used to practice curiosity while coding.
Use Things In Ways They Weren&apost Intended to Be Used
Push boundaries and perform experiments.
I once worked as a programmer for a commercial audio/visual installation company. On one job, we installed a distributed video switching system controlled by an app. The app sent TCP packets from an iPad to a network switch and used VLAN untagging to connect video receivers on the network to broadcasts streamed by transmitters.
The app was incompatible with the latest switch firmware. The IT department refused to downgrade the firmware, and the app developer couldn&apost deliver a patch until well after the project deadline. But I could configure the app to send commands to any IP address.
I could write a TCP server to accept commands from the app, convert them to the network switch&aposs protocol, and forward them to the switch. I needed a machine to host the script. The audio processing unit (APU) was a Linux box with plenty of storage and memory. Installing a custom script would void the warranty.
The APU was programmed by a drag-and-drop visual coding interface. One of the available "blocks" could run Lua scripts inside the application&aposs sandbox. But, could a script running in the sandbox receive commands from the app and communicate with the network switch?
There were no documented restrictions, so I ran an experiment. It worked! Even better, script blocks automatically started whenever the APU booted. Video switching worked effortlessly even after reboots from power outages and other incidents.
My curiosity paid off. We completed the job on time. My discoveries created new solutions for clients and spawned internal tools that saved time and money on installations.
Try To Break Things
Destruction can be productive.
I enjoy finding ways to "break" a language. It isn&apost about finding bugs. It&aposs about finding code examples that exhibit surprising behavior. It always starts with a question.
"How does a Python set determine if two objects are distinct?"
Knowing where to start looking for answers is a crucial skill. The Python docs on sets are pretty good.
"Oh, ok. Sets use an algorithm that checks if two objects have the same hash value and compare equal to each other. If both checks are true, then those objects are indistinguishable from each other, and the set can only contain one of them."
A little bit of knowledge opens the door to new questions.
"Can you &apostrick&apos a set into thinking two different objects are nondistinct?"
Now there&aposs a fun little puzzle. Think about it. An integer hashes to its value. You can control a custom object&aposs hash value with the .__hash__() method. That means you can create an object that hashes to the same value as the integer 1:
class NotOne:
def __hash__(self):
return 1
To confuse a set into thinking NotOne instances are equal to 1, you need them to compare equal to 1. You can make NotOne objects compare equal to any object by implementing an .__eq__() method that always returns True:
class NotOne:
def __hash__(self):
return 1
def __eq__(self, other):
return True
Now see what happens when you create a set containing 1 and an instance of NotOne:
>>> n = NotOne()
>>> # Create a set S containing 1 and n
>>> S = {1, n}
>>> # S only contains 1
>>> S
{1}
>>> # But somehow n is in S!
>>> n in S
True
n is very much a distinct object from 1. n isn&apost even a number. It doesn&apost support arithmetic operations. But you&aposll never be able to put n and 1 in a set together because they fail to meet a set&aposs criteria for distinctness. It feels weird that any set containing 1 also contains n.
"That&aposs pretty weird. I wonder if there&aposs a way to trick a set into thinking two nondistinct objects are distinct from each other?"
For such a thing to be possible requires an object that doesn&apost compare equal to itself. If you&aposve ever worked with IEEE 754 NaN objects before, you know they fit the bill.
"What happens when you try to put several NaN values into a Python set?"
Let&aposs find out.
>>> S = {float("nan"), float("nan"), float("nan")}
>>> S
{nan, nan, nan}
"Ok, that is weird. But surely you can verify that the set contains a NaN object. Right?"
Behold:
>>> float("nan") in S
False
I love examples like this. In all honesty, nothing is broken — except maybe my brain for a little bit. Seeking out and understanding examples like these strengthens your intuition about code and surfaces new ideas for solving problems.
Where To Go From Here
Start practicing curiosity today. Ask more questions and do more experiments. Be thankful that it&aposs impossible to know everything. There&aposs always a new puzzle to be solved. There are always new things to wonder about.
Are you interested in learning something new? I offer private one-on-one coaching for Python programming and technical writing. Click here for more information.
ItsMyCode
Pandas : How to Find Unique Values in a Column
ItsMyCode |
We can find unique values of a column in Pandas DataFrame using the unique() function.
The unique() method filters out only unique values from a dataframe column. In this tutorial, we will learn how to use the unique() method to find unique values in Pandas DataFrame columns with examples.
In Pandas rename column of DataFrame can be done using pandas.DataFrame.rename() method.
We have a simple DataFrame with the dictionary of lists, indicates the fruits, price and quantity as the column names.
# import pandas library
import pandas as pd
# create DataFrame
df = pd.DataFrame({'fruits': ['orange', 'mango', 'apple', 'grapes', 'orange', 'mango'],
'price': ['40', '80', '30', '40', '30', '80'],
'quantity': ['200', '300', '300', '400', '200', '800']
})
print(df)Output
fruits price quantity
0 orange 40 200
1 mango 80 300
2 apple 30 300
3 grapes 40 400
4 orange 30 200
5 mango 80 800Find unique values of a single column in Pandas DataFrame
Let’s say if we need to find the unique values of a fruits column, we can use the unique() method as shown in the below code.
# import pandas library
import pandas as pd
# create DataFrame
df = pd.DataFrame({'fruits': ['orange', 'mango', 'apple', 'grapes', 'orange', 'mango'],
'price': ['40', '80', '30', '40', '30', '80'],
'quantity': ['200', '300', '300', '400', '200', '800']
})
# get the unique value of column fruits
print(df.fruits.unique())
Output
['orange' 'mango' 'apple' 'grapes']Find unique values in all columns in Pandas DataFrame
If we need to find the unique values of all the columns in Pandas DataFrame, we need to iterate the columns using the for loop and then use the unique() method on each column name.
# import pandas library
import pandas as pd
# create pd DataFrame
df = pd.DataFrame({'fruits': ['orange', 'mango', 'apple', 'grapes', 'orange', 'mango'],
'price': ['40', '80', '30', '40', '30', '80'],
'quantity': ['200', '300', '300', '400', '200', '800']
})
# get the unique value of all columns
for col in df:
print(df
.unique())
Output
['orange' 'mango' 'apple' 'grapes']
['40' '80' '30']
['200' '300' '400' '800']Find and count unique values of a single column in Pandas DataFrame
We can even count the occurrence of unique values in a single column using the method value_counts() method.
# import pandas library
import pandas as pd
# create DataFrame
df = pd.DataFrame({'fruits': ['orange', 'mango', 'apple', 'grapes', 'orange', 'mango'],
'price': ['40', '80', '30', '40', '30', '80'],
'quantity': ['200', '300', '300', '400', '200', '800']
})
# get the count unique values of column fruits
print(df.fruits.value_counts())
Output
orange 2
mango 2
apple 1
grapes 1
Name: fruits, dtype: int64The post Pandas : How to Find Unique Values in a Column appeared first on ItsMyCode.
Paolo Amoroso
Publishing Python Documentation to Read The Docs With Jupyter Books
I published the documentation of Suite8080, a suite of Intel 8080 Assembly cross-development tools I’m writing in Python with Replit.
 |
| The Suite8080 documentation website built with Jupyter Book and Sphinx and hosted at Read The Docs. |
Like many Python projects, the documentation comprises a website hosted at Read The Docs. But, unlike most projects, I didn’t use Sphinx directly to generate the website. Instead, I created the documentation with Jupyter Book by The Executable Book Project, well known for Jupyter Notebook. Jupyter Book is a new system for producing publication-quality books and documents for software and research projects.
Jupyter Book is significant and promising because, by building on top of Sphinx as a backend and offering a Markdown-based formatting language, it hides the complexity of Sphinx and reduces friction when writing documentation.
Before this project I checked out Sphinx directly, but its arcane formatting language and complex setup didn’t make me go far. Sphinx is an amazing tool though, so I promised myself I’d try it again. Discovering Jupyter Book changed my plans.
The integration between Jupyter Book and Sphinx is at an early stage. Indeed, I had to overcome a few minor issues before making the systems work with my setup. Therefore, I’m posting these notes to document and share my experience. How meta.
Motivation and goals
As a hobby project, Suite8080 is a small and simple system. Still, I wanted to document it for the usual reasons this is a best practice, such as allowing others to use and understand the system.
But I had additional motivations.
First off, I’m one of those who needs to understand and use Suite8080. I’m working on and extending the system. But sometimes I put it aside and come back to it later, thus needing to re-immerse in its internals. The necessity to understand my own code is no joke.
The Python community has a tradition of producing good documentation, which is a key resource for helping me learn the language and leverage its ecosystem. I’m a text guy and love reading books and software documentation, to the point I often read manuals cover to cover (don’t judge).
Publishing the Suite8080 documentation is a way of giving back to the community and practicing technical writing, hoping to improve my ability to write manuals like the ones I read and love. Additionally, it’s an opportunity to practice the Python publishing toolchain.
Text formatting and publishing toolchain
Producing documentation with my setup relies on two text formatting tools, Jupyter Book and Sphinx, and the Read The Docs publishing platform. My development environment Replit runs and controls the other tools, letting them access the manuscript to process.
Although Jupyter Book, Sphinx, and Read The Docs can work together, there’s no single information source on how to do that. I had to experiment a bit to figure out the steps.
Let’s go over the tools and how to use them with my setup.
Jupyter Book
Jupyter Book provides a formatting language and tools for producing books and software documentation containing text, media, and executable content such as Jupyter notebooks. For text, Jupyter Book supports its own Markdown dialect, MyST, but accepts also reStructuredText files for Sphinx. This is no coincidence as Jupyter Book runs Shpinx as a backend.
The documentation Jupyter Book formats is ready for browsing on a local machine, or for publication as a static website to Read The Docs and other platforms. I love the clean and modern design of the sites Jupyter Book produces.
Jupyter Book is a Python system installable with pip. The Suite8080 REPL on Replit, the cloud workspace I develop Suite8080 with, allows a few ways of installing Jupyter Book.
The first is to run pip from the REPL’s Linux shell, which makes Jupyter Book available only for the current session. Running pip on Replit to install Jupyter Book issues a missing dependency error but a simple fix is to run pip a second time, when the error goes away.
A disadvantage of this option is the need to reinstall Jupyter Book at each new session. This is less of an annoyance than it seems as building or editing the documentation, and hence keeping Jupyter Book ready, is less frequent than working on the Python code.
There are two more ways: installing Jupyter Book with the Replit package manager and making the REPL always-on to leave the tool loaded in the workspace. Both have drawbacks, i.e. the lengthy download of Jupyter Book and its many dependencies (e.g. Sphinx) with the Replit package manager, and using up a precious always-on REPL slot with enabling always-on for a REPL.
I work on the Suite8080 documentation less frequently than on the Python code, so I go with pip-installing Jupyter Book on-demand.
Sphinx
Jupyter Book produces documentation from source files written in a variety of text formatting languages, including its own Markdown dialect MyST and reStructuredText for Sphinx, the well-known text formatting language and static site generator. Despite the ample choice, I use only MyST mostly because of the simplicity and minimalism of Markdown.
Under the hood, Jupyter Book translates the sources to the Sphinx format and runs the latter to build the documentation. So it’s possible to work almost entirely from Jupyter Book.
The build process still requires writing a few Sphinx directives or editing the Sphinx configuration file, for example, to produce API reference documentation with sphinx.ext.autodoc. I converted the Suite8080 Python docstrings to the Google style and configured sphinx.ext.autodoc to include the sphinx.ext.napoleon extension.
Read The Docs
Although I can build the Suite8080 documentation with Jupiter Book and browse it locally on my development system, the final, public result is a website anyone interested in Suite8080 may visit.
I upload that website to Read The Docs, a popular platform for publishing Python documentation formatted with Sphinx. The process is made possible by the integration between Jupyter Book and Read The Docs. To get started, I only had to configure Read The Docs via an online form to tell the platform where to find the Suite8080 GitHub repository.
Once configured, Read The Docs automatically discovers the Jupyter Book documentation sources, builds them, and publishes the resulting website. Then, each time I commit to the GitHub repository, Read The Docs detects any changes to the documentation sources and rebuilds them if necessary. All that is left to do is to check the build logs for any issues.
Replit
I write and document Suite8080 in the cloud with the Replit development and computing environment. Replit is the best fit for my Chrome OS fully cloud lifestyle.
Three Replit features help build and publish documentation with Jupyter Book.
The first is the Linux shell of the Python REPL where I develop Suite8080. Configuring and building the documentation involves running a couple of command line scripts.
The second feature is the version control client built into the REPL, which synchronizes with the Suite8080 GitHub project repository. Assuming Read The Docs is properly configured, as soon as I commit source changes in the REPL, Read The Docs automatically rebuilds the documentation and publishes it. When committing changes, the .gitignore file takes care of skipping the Sphinx build directories.
 |
| The Suite8080 documentation previewed in the project’s Replit workspace (REPL), with the Python http.server module serving the locally generated website. |
Finally, Replit lets me preview the formatted documentation in the REPL, both in the Markdown pane as I type and as HTML built by Sphinx.
Although Replit runs fully in the browser, it can’t render HTML files in a REPL unless an HTTP server is running in the REPL. In Python REPLs, there’s no need to install external servers as the native Python http.server module does a good job of displaying the static sites Sphinx generates. All I have to do is navigate to the directory with the HTML files and run http.server from the shell:
$ cd docs/_build/html
$ python3 -m http.server
This opens in the REPL an HTML pane showing the website. For convenience, I pop out the HTML pane into a full browser tab.
Structuring the documentation
In the early stages of the Suite8080 development, I began documenting the project in a series of Markdown files in the source tree: a README.md file as well as usage and design notes files usage.md and design.md. Anticipating I’d produce more files, I gathered the Markdown sources into a docs directory.
This turned out to be a good arrangement as the structure of the documentation was ready for Jupyter Book. A Jupyter Book book usually comprises a directory holding the manuscript files in Markdown and other formats, as well as some configuration files.
To adapt the Markdown files in the docs directory to Jupyter Book, I repurposed part of README.md as an introductory chapter, lightly edited the usage and design notes as separate chapters, created a chapter with the API reference processed by sphinx.ext.autodoc, and created the configuration files, _toc.yml describing the table of contents and _config.yml for the configuration.
There’s another important configuration file, docs/requirements.txt, needed to fix a Read The Docs build issue a missing dependency causes.
Read The Docs can automatically discover the directory holding the documentation sources to build from. Here is what the Suite8080 directory tree looks like in the Replit REPL from the shell’s home directory:
.
├── asm
├── docs
│ └── _build
│ └── html
│ ├── _images
│ ├── _panels_static
│ ├── _sources
│ └── _static
│ ├── css
│ ├── images
│ ├── js
│ ├── __pycache__
│ └── vendor
│ └── fontawesome
│ └── 5.13.0
│ ├── css
│ └── webfonts
├── suite8080
│ └── __pycache__
└── tests
└── __pycache__
docs is the documentation directory, the other top-level ones hold the Python and Assembly files. The root of the Sphinx build tree is in directory docs/_build. The source tree in the GitHub repository is the same, except for the missing documentation build and compilation cache directories.
Experimenting with a sandbox
Aside from some experimentation with Sphinx, I never used Jupyter Book, Sphinx, and Read The Docs before. Suite8080 is the right opportunity for checking out and learning the tools. Given the different tools and the ways they interact, I wanted to be sure not to mess with the Replit REPL and the GitHub repository that make up my Suite8080 development environment.
Therefore, I practiced publishing the documentation within a sandbox, a dummy project with a new REPL and a separate GitHub repository.
Forking the Suite8080 REPL wasn’t convenient as the forked REPL would have accessed the same GitHub repository, so I ended up copying the Suite8080 files I needed, the Markdown and Python sources as well as the directory structure.
Working in the sandbox gave me the confidence to learn without breaking my main development environment. Once confident I mastered the tools and process, I copied the changed files back to the main Suite8080 project and continued from there.
I could have created a new version control branch from the main REPL, but learning branching models will be a project for another time.
Workflow
The integration between Jupyter Book and Read The Docs is new and a work in progress.
After some experimentation, I figured out what steps to take and which ones aren’t documented. I put together a workflow comprising the initial Read The Docs configuration, and a series of steps to go through in the Suite8080 REPL each time I update the documentation sources.
Initial configuration
I kicked off the work with a couple of onetime actions to configure Read The Docs using only the browser.
First, with the documentation files ready in the source tree and committed to GitHub, I created a Read The Docs account. The platform supports signing up with a GitHub account, as I did. This helps a lot as Read The Docs needs access to the Suite8080 GitHub repository to build the documentation from source.
Next, I created a Read The Docs project for Suite8080. The defaults are fine, but there are two key fields to fill in the creation form, the URL of the GitHub repository and the Read The Docs project slug. I chose suite8080 as the slug, which is associated with the suite8080.readthedocs.io documentation website.
As soon as I entered these settings, Read The Docs cloned the GitHub repository and ran the first build, which failed with a missing dependency error. To satisfy the dependencies and fix the build error, I added to the source tree the file docs/requirements.txt with a reference to the Jupyter Book master branch:
https://github.com/executablebooks/jupyter-book/archive/refs/heads/master.zipTo tell Sphinx about the dependency I added another file, .readthedocs.yaml:
python:
install:
- requirements: docs/requirements.txt
formats:
- pdf
- epub
With the initial configuration complete, Read The Docs was ready to automatically rebuild the documentation and publish the website each time I would commit to the Suite8080 GitHub repository.
Rebuilding the documentation
Whenever I change the Markdown sources, I go through a few steps for rebuilding and publishing the documentation to Read The Docs. First, in the REPL shell I change to the project root directory ~/suite8080 and run Jupyter Book to generate the conf.py file the Sphinx build needs:
~/suite8080$ jupyter-book config sphinx docsNext comes the generation of the API reference. A current limitation of the toolchain is there’s no way of having Jupiter Book specify to Sphinx where sphinx.ext.autodoc can find the Python modules. A workaround is to edit docs/conf.py to add this code:
import os
import sys
sys.path.insert(0, os.path.abspath(".."))
The code adds to the Python path the parent directory of the docs directory, which is where the Python modules can be imported from from.
Having to manually edit conf.py for every documentation build is an error-prone kludge that interferes with automation. But at this time there’s no better sphinx.ext.autodoc integration between Jupyter Book and Sphinx. Ideally, Jupyter Book should allow configuring the module path via, say, _config.yml or directly take care of this.
Finally, everything is ready to run Sphinx to build the HTML site:
~/suite8080$ sphinx-build docs docs/_build/html -b htmlThis leaves the HTML files in the build directory docs/_build/html. Most of the times, I change to that directory and run the Python HTTP server in the REPL to check the resulting website for any issues:
~/suite8080$ cd docs docs/_build/html
~/.../_build/html$ python3 -m http.server
I pop out the small HTML preview pane from the REPL into a full-screen tab and browse the website on the desktop from Chrome OS, as well as from my Android devices.
When I’m satisfied everything looks good on the desktop and on mobile, I commit the changes to version control from the REPL. The commit triggers the rebuild of the website on Read The Docs, after which I do a final check there to make sure there are no issues in the build logs and the site still looks good.
The documentation is now ready for user consumption.
Publishing the Suite8080 documentation is another step in my journey to use Python with Replit in the cloud. I’m continuing to experiment with Replit, write new Python code, try libraries and tools, and share my experience in the Python with Replit blog series. Follow along to not miss a post.
Real Python
Build a Dice-Rolling Application With Python
Building small projects, like a text-based user interface (TUI) dice-rolling application, will help you level up your Python programming skills. You’ll learn how to gather and validate the user’s input, import code from modules and packages, write functions, use for loops and conditionals, and neatly display output by using strings and the print() function.
In this project, you’ll code an application that simulates dice-rolling events. To do so, you’ll use Python’s random module.
In this tutorial, you’ll learn how to:
- Use
random.randint()to simulate dice-rolling events - Ask for the user’s input using the built-in
input()function - Parse and validate the user’s input
- Manipulate strings using methods, such as
.center()and.join()
You’ll also learn the basics of how to structure, organize, document, and run your Python programs and scripts.
Click the link below to download the entire code for this dice-rolling application and follow along while you build the project yourself:
Get Source Code: Click here to get the source code you’ll use to build your Python dice-rolling app.
Demo
In this step-by-step project, you’ll build an application that runs dice-rolling simulations. The app will be able to roll up to six dice, with each die having six faces. After every roll, the application will generate an ASCII diagram of dice faces and display it on the screen. The following video demonstrates how the app works:
When you run your dice-rolling simulator app, you get a prompt asking for the number of dice you want to roll. Once you provide a valid integer from 1 to 6, inclusive, then the application simulates the rolling event and displays a diagram of dice faces on the screen.
Project Overview
Your dice-rolling simulator app will have a minimal yet user-friendly text-based user interface (TUI), which will allow you to specify the number of six-sided dice that you’d like to roll. You’ll use this TUI to roll the dice at home without having to fly to Las Vegas.
Here’s a description of how the app will work internally:
| Tasks to Run | Tools to Use | Code to Write |
|---|---|---|
| Prompt the user to choose how many six-sided dice to roll, then read the user’s input | Python’s built-in input() function |
A call to input() with appropriate arguments |
| Parse and validate the user’s input | String methods, comparison operators, and conditional statements | A user-defined function called parse_input() |
| Run the dice-rolling simulation | Python’s random module, specifically the randint() function |
A user-defined function called roll_dice() |
| Generate an ASCII diagram with the resulting dice faces | Loops, list.append(), and str.join() |
A user-defined function called generate_dice_faces_diagram() |
| Display the diagram of dice faces on the screen | Python’s built-in print() function |
A call to print() with appropriate arguments |
Keeping these internal workings in mind, you’ll code three custom functions to provide the app’s main features and functionalities. These functions will define your code’s public API, which you’ll call to bring the app to life.
To organize the code of your dice-rolling simulator project, you’ll create a single file called dice.py in a directory of your choice in your file system. Go ahead and create the file to get started!
Prerequisites
You should be comfortable with the following concepts and skills before you start building this dice-rolling simulation project:
- Ways to run scripts in Python
- Python’s
importmechanism - The basics of Python data types, mainly strings and integer numbers
- Basic data structures, especially lists
- Python variables and constants
- Python comparison operators
- Boolean values and logical expressions
- Conditional statements
- Python
forloops - The basics of input, output, and string formatting in Python
If you don’t have all of the prerequisite knowledge before starting this coding adventure, then that’s okay! You might learn more by going ahead and getting started! You can always stop and review the resources linked here if you get stuck.
Step 1: Code the TUI of Your Python Dice-Rolling App
In this step, you’ll write the required code to ask for the user’s input of how many dice they want to roll in the simulation. You’ll also code a Python function that takes the user’s input, validates it, and returns it as an integer number if the validation was successful. Otherwise, the function will ask for the user’s input again.
To download the code for this step, click the following link and navigate to the source_code_step_1/ folder:
Get Source Code: Click here to get the source code you’ll use to build your Python dice-rolling app.
Read the full article at https://realpython.com/python-dice-roll/ »
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Python for Beginners
Find The Sum Of Digits Of An Integer In Python
Integers are used almost everywhere when you write a program. In this article, we will discuss ways to find the sum of digits of a given integer in python.
How to find the sum of digits of an integer?
To find the sum of digits of a number N, we can extract the digits of a number and add them one by one. For this, we will extract the digits from right to left. For extracting the digits, we will divide the number by 10 and store the remainder during each division operation. We will keep dividing the number by 10 until it becomes 0. In this way, we can extract digits one by one.
For example, If we are given the number 12345, we will calculate the sum of digits as follows.
- First, we will initialize a variable sumOfDigits to zero.
- After that, we will divide 12345 by 10.
- After division, we will add reminder 5 to sumOfDigits and update the number to 1234.
- Then, we will divide 1234 by 10.
- After division, we will add reminder 4 to sumOfDigits and update the number to 123.
- Again, we will divide 123 by 10.
- After division, we will add reminder 3 to sumOfDigits and update the number to 12.
- Again, we will divide 12 by 10.
- After division, we will add reminder 2 to sumOfDigits and update the number to 1.
- Now, we will divide 1 by 10.
- After division, we will add reminder 1 to sumOfDigits and update the number to 0.
- As the number has become 0, we will be having the sum of digits of the number 12345 in the variable sumOfDigits i.e. 15.
Implementation In Python
As we have seen above, to find the sum of digits of an integer in python, we just have to divide the number by 10 until it becomes 0. At the same time, we have to add the remainder in each division to obtain the sum of digits. We can implement the program to perform this operation as follows.
def calculate_sum_of_digits(N):
sumOfDigits = 0
while N > 0:
digit = N % 10
sumOfDigits = sumOfDigits + digit
N = N // 10
return sumOfDigits
input_number = 12345
output = calculate_sum_of_digits(input_number)
print("Sum of digits of {} is {}.".format(input_number, output))
input_number = 126
output = calculate_sum_of_digits(input_number)
print("Sum of digits of {} is {}.".format(input_number, output))
Output:
Sum of digits of 12345 is 15.
Sum of digits of 126 is 9.Conclusion
In this article, we have discussed and implemented a program to find the sum of digits of an integer in Python. To learn more about numbers in python, you can read this article on decimal numbers in python. You might also like this article on complex numbers in python.
The post Find The Sum Of Digits Of An Integer In Python appeared first on PythonForBeginners.com.
The Three of Wands
cattrs I: un/structuring speed
Over the years, I&aposve put a lot of effort into making cattrs fast.
Let&aposs take a look at how the cattrs structuring code evolved during this time.
cattrs is a library for transforming data, and one of its main uses is transforming attrs classes to and from dictionaries. (The dictionaries are then fed to a serialization library, like json, ujson or msgpack.)
For the purposes of this article, imagine a class like this:
from attrs import define
@define
class Test:
a: int
b: int
c: float
d: float
e: str
f: strNow imagine, having read a JSON payload from an HTTP request as a dictionary, you wish to transform that JSON payload into an instance of this class.
v1: The Early Days
Back in the early days of cattrs, say circa 2019 (which is simultaneously a lifetime ago, and yesterday), if you asked cattrs to structure your payload into an instance of this class, cattrs would perform something roughly similar to this (very simplified) code:
from attrs import fields
def structure_attrs_fromdict(payload, cls):
res = {}
for field in fields(cls):
attribute_converter = get_converter(field.type)
res[field.name] = attribute_converter(payload[field.name])
return cls(**res)Even though this code path is considered suboptimal nowadays, it still exists in the (now, legacy and soon to be renamed) Converter class of cattrs.
This code does an OK job, but the problem with it is that it&aposs somewhat slow. The converter of each field (in this case, two ints, two floats and two strings) needs to be looked up each and every time you call this function, even though they will almost certainly not change during the lifetime of the converter. And there&aposs also the issue of the for loop, creating an iterator and iterating over it every time.
Another problem with it is that any additional features are going to slow it down further. Maybe you want to rename a field - more processing needs to occur. Maybe you want to omit a field altogether - an additional if statement for every attribute. I figured there had to be a better way.
As a side note, on my MacBook Pro and on CPython 3.9, applying this structure function to an appropriate dictionary takes around 4.9 microseconds. That&aposs not too bad, but keep in mind this is a very simple case. In real life classes have more fields, the classes are often nested, and the field types and converters are also more complex.
v2: The GenConverter
Back in those days, I had this (in retrospective, very silly) notion that people wouldn&apost use cattrs if it was slower than what they could write themselves. I asked myself, given an instance of Test, how would a human write the structuring function. Then I remembered this is Python, and in Python code can write code. If you can write an optimized structuring function for your class, so could cattrs. And the GenConverter (generating converter) was born.
When the GenConverter sees Test for the first time, it&aposll generate a custom structuring function for it. By using Python&aposs inspect module, we can look at the actual generated source ourselves.
>>> from inspect import getsourcelines
>>> from cattrs import GenConverter
>>> c = GenConverter()
>>> f = c._structure_func.dispatch(Test)
>>> for l in getsourcelines(f)[0]: print(l)
def structure_Test(o, *_):
res = {
&aposa&apos: structure_a(o[&aposa&apos], type_a),
&aposb&apos: structure_b(o[&aposb&apos], type_b),
&aposc&apos: structure_c(o[&aposc&apos], type_c),
&aposd&apos: structure_d(o[&aposd&apos], type_d),
&apose&apos: structure_e(o[&apose&apos], type_e),
&aposf&apos: structure_f(o[&aposf&apos], type_f),
}
return __cl(**res)As you see, this is a lot like what you&aposd write yourself. When cattrs generates this function, it provides a dictionary of global variables to it - that&aposs where the structure_a et al come from.
The main benefit of this approach is it does the work in two phases - the generation phase and the run phase. The generation phase takes significantly longer, but it happens only once. The generation phase outputs the structure_Test function, which is then cached and run on every structure call.
This also means we can do more work in the generation phase, which allows an entire class of features for essentially no cost. Renaming fields and handling generic classes falls into this category, alongside resolving the individual field converters.
(Note that if the app you&aposre working on cannot afford to pay the upfront cost of compiling the function, you can still use the old Converter code path instead. For example, if you&aposre writing a CLI app where fast startup is crucial.)
This code path has been the default since cattrs 1.7, released in May 2021. On my Mac, it takes ~2.3 microseconds to run. That&aposs more than twice the speed of the old approach.
For a while, I imagined this is as optimized as Python code can get. A few days ago while reading a thread from the Python-Dev mailing list, I realized I was wrong.
v3: Better Living Through Better Bytecode
cattrs generates Python code as simple lines of texual source code. The Python interpreter then ingests this text using the compile and eval builtins and produces a function object we can call.
There&aposs another layer though, sitting in the middle and not as obvious: the function bytecode. This bytecode is a list of low level instructions the Python interpreter executes when a function is called. Let&aposs take a look at the bytecode of our generated structure_Test using Python&aposs dis module.
>>> from dis import dis
>>> dis(f)
3 0 LOAD_GLOBAL 0 (structure_a)
2 LOAD_FAST 0 (o)
4 LOAD_CONST 1 (&aposa&apos)
6 BINARY_SUBSCR
8 LOAD_GLOBAL 1 (type_a)
10 CALL_FUNCTION 2
4 12 LOAD_GLOBAL 2 (structure_b)
14 LOAD_FAST 0 (o)
16 LOAD_CONST 2 (&aposb&apos)
18 BINARY_SUBSCR
20 LOAD_GLOBAL 3 (type_b)
22 CALL_FUNCTION 2
... similar lines omitted...
2 72 LOAD_CONST 7 ((&aposa&apos, &aposb&apos, &aposc&apos, &aposd&apos, &apose&apos, &aposf&apos))
74 BUILD_CONST_KEY_MAP 6
76 STORE_FAST 2 (res)
10 78 LOAD_GLOBAL 12 (__cl)
80 BUILD_TUPLE 0
82 BUILD_MAP 0
84 LOAD_FAST 2 (res)
86 DICT_MERGE 1
88 CALL_FUNCTION_EX 1
90 RETURN_VALUEThis function uses LOAD_GLOBAL a lot - we provide a lot of data through the function globals, so no wonder. It also turns out Python functions essentially have access to two scopes - the global scope and the local scope - and loading objects from the global scope is a lot slower than loading them from the local scope!
When an object is loaded from the local scope, you&aposll see the LOAD_FAST instruction instead of the LOAD_GLOBAL instruction. The local scope is mostly used for variables defined in the function, hence the name. Wouldn&apost it be great if we could generate our function to read all the objects we&aposve prepared for it from the local namespace, though?
We can use a trick. We can get all these objects into the local scope by setting them as default values to dummy parameters. The expected interface of the structure function is simple, so no one will know or mind if we stick a bunch of keyword-only parameters with defaults into the function signature.
So the next version of cattrs, 22.1.0 (oh yeah, we&aposre switching to CalVer, just like attrs), will do just that.
First, let&aposs take a look at the new Python source:
>>> for l in getsourcelines(f)[0]: print(l)
def structure_Test(o, _, *, __cl=__cl, __c_structure_a=__c_structure_a, __c_structure_b=__c_structure_b, __c_structure_c=__c_structure_c, __c_structure_d=__c_structure_d, __c_structure_e=__c_structure_e, __c_structure_f=__c_structure_f):
return __cl(
__c_structure_a(o[&aposa&apos]),
__c_structure_b(o[&aposb&apos]),
__c_structure_c(o[&aposc&apos]),
__c_structure_d(o[&aposd&apos]),
__c_structure_e(o[&apose&apos]),
__c_structure_f(o[&aposf&apos]),
)As I mentioned, we&aposve stuck a bunch of variables into the function signature so it&aposs super messy right now. We can apply some deduplication to it later, but it doesn&apost really matter.
I&aposve also applied some more optimizations to it: we don&apost generate a temporary dictionary to hold the individual fields if we don&apost need to, and we don&apost need to in this case.
Now if we look at the bytecode:
>>> dis(f)
2 0 LOAD_FAST 2 (__cl)
3 2 LOAD_FAST 3 (__c_structure_a)
4 LOAD_FAST 0 (o)
6 LOAD_CONST 1 (&aposa&apos)
8 BINARY_SUBSCR
10 CALL_FUNCTION 1
4 12 LOAD_FAST 4 (__c_structure_b)
14 LOAD_FAST 0 (o)
16 LOAD_CONST 2 (&aposb&apos)
18 BINARY_SUBSCR
20 CALL_FUNCTION 1
... similar lines omitted ...
8 52 LOAD_FAST 8 (__c_structure_f)
54 LOAD_FAST 0 (o)
56 LOAD_CONST 6 (&aposf&apos)
58 BINARY_SUBSCR
60 CALL_FUNCTION 1
2 62 CALL_FUNCTION 6
64 RETURN_VALUESweet, sweet LOAD_FAST, and no LOAD_GLOBAL in sight. But what are the fruits of our labor?
On my Mac, this version takes ~1.39 microseconds to run. That&aposs 60% of the v2 solution, and 28% of the v1 approach. Not too bad, if I do say so myself.
CPython 3.11a4 and PyPy
I&aposve been using CPython 3.9 to run these benchmarks, as that is the version I&aposm using for work at this time. Just out of curiosity, I&aposve run these benchmarks on the latest alpha of CPython 3.11 (a4) and the latest stable PyPy release (7.3.7), to see how they compare to my old workhorse the 3.9.
CPython 3.11a4
- v1: 3.57 us +- 0.14 us
- v2: 1.70 us +- 0.04 us
- v3: 1.09 us +- 0.02 us
Looks like 3.11 is ~27% faster than 3.9 for the v3 approach. Great news!
PyPy
- v1: 1.68 us +- 0.05 us
- v2: 479 ns +- 13 ns
- v3: 228 ns +- 6 ns
PyPy is still the king of crunching CPU-bound code. 228 nanoseconds to structure a 6-field class, that&aposs gotta be close to native, right?
PyBites
10 Tips to Write Better Functions in Python
Functions are your primary building blocks in Python. They can be deceptively easy, yet they are profound.
Right off the bat there are some major benefits of using functions:
- Functions make your code more modular, re-usable and DRY (don’t repeat yourself).
- This will make your code easier to test!
- Functions better isolate functionality through scoping (as per the Zen of Python: namespaces are one honking great idea).
- They have documentation built in through docstrings.
- They combat the everything needs to be a class bias you sometimes see, which certainly does not hold true, specially when getting anything off the ground.
- And lastly, their flow is easy to reason about: 1) receive something (input) -> 2) do something (transform) -> 3) return something (output)
So it’s not hard to pitch using more functions in your code, and I am sure most of you already do (hence why I am not covering the syntax in this article, although I might do a follow up article).
So let’s look at what makes for better functions? Here are 10 tips:
- Every function name is an opportunity to self document your code. You can use various words separated by underscores (as per PEP8 convention). Having multiple words and not putting types in them (e.g. id_list) can greatly increase the readability of your code.
- As per the Single-responsibility principle, a function should do one thing only. This makes them easier to test and reuse. It’s hard to define a length restriction but going over 15 lines should make you pause and reflect.
Functions usually grow longer because they are doing too many things. In that case you can abstract behavior in one or more new functions and call those inside the original function. Again, this will make your code easier to test and gives you more opportunities to name each chunk. - Keep the function’s interface (that is, the arguments that go into it) small. If you’re defining functions with similar arguments, can you group those arguments into one object (e.g. a namedtuple or dataclass)?
Also, at that point, a class might be more appropriate. Usually I ask myself if I need to keep some sort of state?
Also use arbitrary args/ keyword args sparingly, they allow any number of arguments to be passed into your function which might hamper the function’s readability and maintenance because it decreases the strictness of the interface (less boundaries). - Check input arguments early and raise an exception if they are off, this will limit your levels of nesting (Zen: Flat is better than nested).
This also offers a clear contract for the caller of the function (“this is how we are going to play the game”). For more on exception handling, check out this article. - An easy win is to define variables as close to where you are using them. Less scrolling, more readability.
- Python 3 allows for positional-only arguments and keyword-only arguments. Specially the latter can make your code a lot more readable and I have seen it used in FastAPI and SQLModel for example.
- Adding type hints can greatly enhance the readability of you code and allows tooling to catch bugs earlier. If you’re new to them, start here.
- Make return types consistent. For example, when returning a sequence, in the “0 items” case, return an empty list instead of None (which is “falsy” too). Type hints can enforce this.
To wrap it up, here are two more tips related to common pitfalls to avoid:
- First, don’t use “global” in your functions because it lets you access (change) variables in the outer scope. Functions should be “pure”, meaning they don’t have “side effects”. Changing something in the outer scope violates this and can lead to insidious bugs.
- Secondly, arguments are evaluated once, so using a mutable default argument (e.g. a list or a dict), you might be surprised that the same data structure will be operated on with subsequent calls.
This is often not what you want, so it’s recommended to use None and make a new list inside the function which creates a new copy upon each invocation. Here is a code example to make this clearer.
I hope this helps you write better functions. If you have any feedback or like to discuss this further join our Slack Community and start a conversation in the #codequality channel.
If you want some dedicated guidance building your dream app end-to-end (MVP ready), we help people reach that goal through our coaching.
And other than that, keep calm and code in Python 
– Bob
Quansight Labs Blog
Conda and Grayskull, the Masters of Software Packaging
Python might be the most popular snake out there, but most of us have also heard of that other serpent: Conda. And some of us have wondered what it really is. In this post we’ll learn about Conda, software packages and package recipes. Most importantly we’ll learn about Grayskull — a conda recipe generator.
Read more… (6 min remaining to read)
Codementor
10 Tools Python Programmers Can Learn
These are the awesome tools a Python programmer can learn to improve his productivity and become a better,
Wing Tips
Turbo Completion Mode in Wing Pro
In this issue of Wing Tips we look at how to use Wing Pro's turbo completion mode for Python. Auto-completion normally requires pressing a completion key, as configured by the Editor > Auto-completion > Completion Keys preference, before a completion is entered into the editor.
Wing Pro also provides a Python Turbo Mode for auto-completion, where completion occurs on any key that cannot be part of a symbol. This allows you to type just enough of a symbol until it is selected in the completer, then move right on to typing the code that follows that symbol. It takes some getting used to, but it can greatly reduce typing once you become comfortable with it.
The following example shows the code that can be produced simply by typing the characters p=x.p.e.P( without pressing any completion keys:
This particular case requires only a single character to match each symbol in the auto-completer. In a more cluttered name space, it may take more characters to match the desired symbol, but in many cases two or three characters is all that is needed before you can move on to the next part of the expression you are trying to type.
Notice that turbo mode distinguishes between contexts where a new symbol may be defined and those where an existing symbol must be used. For example, if you type c followed by = on a new line, Wing knows that the = indicates you may be defining a new symbol and does not place the current selection from the auto-completer. When you do want completion to occur in a defining context, you can press Tab or another completion key.
In a context where you are trying to type something other than what is in the completer, you can press Ctrl, Alt or Command briefly by itself to hide the auto-completer and thus disable turbo-completion until you type more symbol characters and the completer is displayed once again.
Turbo completion is enabled with the Editor > Auto-completion > Python Turbo Mode preference in Wing Pro.
That's it for now! We'll be back next week with more Wing Tips for Wing Python IDE.
Matt Layman
Go Fast With Django
In the last Understand Django article, we learned about commands. Commands are the way to execute scripts that interact with your Django app. With this article, we’re going to dig into performance. How do you make your Django site faster? Keep reading to find out. From Browser To DjangoURLs Lead The WayViews On ViewsTemplates For User InterfacesUser Interaction With FormsStore Data With ModelsAdminister All The ThingsAnatomy Of An ApplicationUser AuthenticationMiddleware Do You Go?
January 18, 2022
PyCoder’s Weekly
Issue #508 (Jan. 18, 2022)
#508 – JANUARY 18, 2022
View in Browser »
Unravelling “[]” Subscriptions in Python
“It’s quite possible you’re not familiar with this formal term, but you are probably familiar with the syntax: the square brackets used for indexing lists and tuples (sequence[4]), accessing the value of a specified dictionary (dictionary["key"]), etc. To cover this topic we will break up into three parts: general subscriptions, slicing, and multiple arguments.”
BRETT CANNON
Working With Pipenv
Pipenv is a packaging tool for Python that solves some common problems associated with the typical workflow using pip, virtualenv, and requirements.txt. In this course, you’ll go over what problems Pipenv solves and how to manage your Python dependencies with it.
REAL PYTHON course
Start Monitoring Service Dependencies to Eliminate Latency and Errors
Datadog APM empowers developer teams to identify anomalies, resolve issues, and improve application performance. Begin collecting stack traces, visualizing them as flame graphs, and organizing them into profile types, such as CPU, I/O, and more. Try Datadog APM free with a 14 day trial →
DATADOG sponsor
A Probabilistic Programming Language in 70 Lines of Python
Learn how Probabilistic Programming Languages (PPLs) work by building a simple one in Python.
ANDREA COGNOLATO
Python News: What’s New From December 2021?
The Python community elected its fourth steering council in December 2021. In this article, you’ll catch up on the results of the election as well as news about new maintenance releases of Python and about end-of-life for Python 3.6.
REAL PYTHON
Discussions
IPython 8 CLI Runs “Black” Autoformatter on REPL Code
This Twitter thread explores concerns about using the IPython REPL with autoformatting enabled for education purposes and interactive math.
RAYMOND HETTINGER
Python Jobs
Senior Full-Stack Web Developer (Anywhere)
Python Engineer Web Scraping (Asia, TX, America)
PyCoder's Weekly Curator and Podcast Co-Host (Anywhere)
Senior Software Engineer (Anywhere)
Sr DevOps Engineer (San Jose, CA, USA)
Senior Software Engineer (Anywhere)
Articles & Tutorials
Problems With Testing Python Pre-Releases and Pip Caching
“Pip was caching a wheel built using CFFI and Python 3.11.0-alpha2 and was installing that same wheel for the now Python 3.11.0-alpha3 both on our local machines and in our test suite. By clearing the pip cache on our machines and our CI we were able to get a passing test suite.”
SETH MICHAEL LARSON
Launching Python, Virtual Environments, and Locking Dependencies With Brett Cannon
Would you like a simple command to launch your Python programs using v the newest version of the language installed on your machine? This week on the show, we continue our conversation with Brett Cannon. Brett discusses his project, the Python Launcher for Unix.
REAL PYTHON podcast
Application Performance Monitoring, Built for Developers by Developers
Scout is an application performance monitoring tool designed to help Python developers find and fix performance issues quickly. With unlimited seats and applications, users of Scout’s APM tool find that anyone can be a performance expert. Click to sign up for your free 14-day trial today →
SCOUT APM sponsor
Build and Submit HTML Forms With Django (Part 4 of 4)
In the final part of this tutorial series, you’ll continue to build out a social network with Django. You’ll wrap up by working with Django forms, learning how to display helpful error messages, setting up dynamic URLs in your app, and more.
REAL PYTHON
Introduction to Data Verification & Storage With Pydbantic
Walk through the basics of pydbantic and discover how to create, query, update, and remove data from models. Review how pydbantic handles migrations after detecting any changes and how to integrate pydbantic with Redis for caching.
JOSHUA JAMISON • Shared by Joshua Jamison
5 Ways to Use Python on an iPad
“It turns out that it’s actually pretty easy to code in Python on the iPad, especially if you’re willing to work in Jupyter Notebooks. […] Here are five ways you can code in Python on any iPad right now.”
DAVID AMOS • Shared by David Amos
Generate Your Text Files With Python Using Cog
An interview with Ned Batchelder about the interesting history of the Cog project and how you can use it to automate the work of generating text with arbitrary Python code.
PODCAST.__INIT__ podcast
Don’t Wrap Instance Methods With the functools.lru_cache Decorator
Wrapping instance method in Python suffers from one giant gotcha—class instances don’t get garbage collected properly. This post explores that.
REDOWAN DELOWAR • Shared by Redowan Delowar
Analyzing Stock Data Near Events With Pandas
Stock market returns are heavily impacted by some events, like FOMC announcements, and we can use Pandas to understand this impact.
MATT WRIGHT
What’s Your Favorite Programming Language?
Take the Developer Nation survey, share your views and make an impact on the developer ecosystem. Plus, you get a chance to win cool prizes, licenses, gaming accessories, vouchers and many more.
SLASHDATA LTD sponsor
Make Simple Mocks With SimpleNamespace
“When testing Python code you may need a mock object. That’s okay! But what’s the best way to build a simple mock?”
ADAM JOHNSON
Projects & Code
Events
Weekly Real Python Office Hours Q&A (Virtual)
January 18, 2021
REALPYTHON.COM
PyData Bristol Meetup
January 20, 2022
MEETUP.COM
Python Northwest
January 20, 2022
PYNW.ORG.UK
PyLadies Dublin
January 20, 2022
PYLADIES.COM
MadPUG
January 20 to January 21, 2022
MEETUP.COM
Karlsruhe Python User Group (KaPy)
January 21, 2022
BL0RG.NET
Happy Pythoning!
This was PyCoder’s Weekly Issue #508.
View in Browser »
[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]
Real Python
Starting With Python IDLE
If you’ve recently downloaded Python onto your computer, then you may have noticed a new program on your machine called IDLE. You might be wondering, “What is this program doing on my computer? I didn’t download that!” While you may not have downloaded this program on your own, IDLE comes bundled with every Python installation. It’s there to help you get started with the language right out of the box. In this course, you’ll learn how to work in Python IDLE and a few cool tricks you can use on your Python journey!
In this course, you’ll learn:
- What Python IDLE is
- How to interact with Python directly using IDLE
- How to edit, execute, and debug Python files with IDLE
- How to customize Python IDLE to your liking
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Python for Beginners
Check For Moran Number In Python
Numbers have magic in them. People have named different numbers based on their specialties. In this article, we will discuss what a Moran Number is. We will also implement a program in python to check if a number is a Moran number or not.
What Is A Moran Number?
A Moran number is a number, which when divided by the sum of its digits, gives a prime number as the result. In other words, if we take a Moran number, calculate the sum of its digits, and divide the number with the sum of the digits, the result will be a prime number. Moran numbers are a subset of Harshad numbers.
For example, 42 is a Moran number. If you calculate its sum of digits, it is 6. 42 divided by 6 gives the result 7 which is a prime number. Hence, 42 is a Moran number.
On the other hand, if we take 20, its sum of digits is 2. 20 divided by 2 gives the result 10 which is not a prime number. Hence, 20 is not a Moran number.
Check For Moran Number in Python
To check for a Moran number in Python, we will have to perform the following operation.
- Calculate the sum of the digits of the given number.
- Divide the number by the sum of digits.
- Check whether the result of division is a prime number.
Let us discuss how to find the sum of digits of the number first.
To find the sum of digits of the given number, we will divide the number by 10 until the number becomes 0. During each division, we will get the rightmost digit as a remainder. We will use the remainder to calculate the sum of digits by adding the remainders during each division as follows.
def calculate_sum_of_digits(N):
sumOfDigits = 0
while N > 0:
digit = N % 10
sumOfDigits = sumOfDigits + digit
N = N // 10
return sumOfDigitsAfter finding the sum of digits, you can divide the given number by the sum of digits to find the result. Now, we have to check if the number is a prime number or not. For this, we will divide the result by all the numbers from 2 to the square root of the result. If the result is divisible by any of the numbers in this range, the number will not be a prime number. The isPrime() function given below performs this operation. It takes a number as an input argument and returns True if the given number is a prime number. Otherwise, it returns False.
def isPrime(N):
count = 2
while count ** 2 <= N:
if N % count == 0:
return False
count = count + 1
return TrueProgram To Check For Moran Number in Python
After defining functions for calculating the sum of digits and checking for prime numbers, we can write a program to check for a Moran number in Python as follows.
def calculate_sum_of_digits(N):
sumOfDigits = 0
while N > 0:
digit = N % 10
sumOfDigits = sumOfDigits + digit
N = N // 10
return sumOfDigits
def isPrime(N):
count = 2
while count ** 2 <= N:
if N % count == 0:
return False
count = count + 1
return True
def check_for_moran_number(N):
sumOfDigits = calculate_sum_of_digits(N)
if N % sumOfDigits == 0:
result = N // sumOfDigits
return isPrime(result)
else:
return False
input_number = 42
output = check_for_moran_number(input_number)
print("{} is a Moran Number:{}".format(input_number, output))
input_number = 20
output = check_for_moran_number(input_number)
print("{} is a Moran Number:{}".format(input_number, output))
Output:
42 is a Moran Number:True
20 is a Moran Number:FalseConclusion
In this article, we have discussed what a Moran number is. We also discussed the steps to check for a Moran number and implemented it in Python. To learn more about numbers in python, you can read this article on decimal numbers in python. You might also like this article on complex numbers in python.
The post Check For Moran Number In Python appeared first on PythonForBeginners.com.
eGenix.com
Python Meeting Düsseldorf - 2022-01-19
The following text is in German, since we're announcing a regional user group meeting in Düsseldorf, Germany.
Ankündigung
Das nächste Python Meeting Düsseldorf findet an folgendem Termin statt:
19.01.2022, 18:00 Uhr
Raum 1, 2.OG im Bürgerhaus Stadtteilzentrum Bilk
Düsseldorfer Arcaden, Bachstr. 145, 40217 Düsseldorf
Programm
Bereits angemeldete Vorträge
Dr. Klaus Bremer"Vergleich namedtuple, dataclasses, slots" (LT)
Marc-Andre Lemburg
"Parsen der NRW Landtagsprotokolle -- Technik und Fehleranalyse"
Oliver Stapel
"Erste Schritte in der Auswertung der Parlamentsreden im Landtag NRW"
Charlie Clark
"Templates und große Excel-Dateien mit Openpyxl"
Marc-Andre Lemburg
"Eigene Template Filter für Ansible schreiben" (LT)
Hayley Spire
"Introducing ReDI School NRW" (LT)
Weitere Vorträge können gerne noch angemeldet werden. Bei Interesse, bitte unter info@pyddf.de melden.
Startzeit und Ort
Wir treffen uns um 18:00 Uhr im Bürgerhaus in den Düsseldorfer Arcaden.
Das Bürgerhaus teilt sich den Eingang mit dem Schwimmbad und befindet
sich an der Seite der Tiefgarageneinfahrt der Düsseldorfer Arcaden.
Über dem Eingang steht ein großes "Schwimm’ in Bilk" Logo. Hinter der Tür
direkt links zu den zwei Aufzügen, dann in den 2. Stock hochfahren. Der
Eingang zum Raum 1 liegt direkt links, wenn man aus dem Aufzug kommt.
>>> Eingang in Google Street View
Corona
Es gelten: 2G-Regelung, sowie die üblichen Regelungen zum Abstand und dem Tragen einer Maske. Der Raum kann gut belüftet werden und bietet mit 83 m² genügend Platz für 15 Personen.
Wichtig: Bitte nur dann anmelden, wenn ihr absolut sicher seid, dass ihr auch kommt. Angesichts der geringen Anzahl Plätze haben wir kein Verständnis für kurzfristige Absagen oder No-Shows.
Einleitung
Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python Begeisterte aus der Region wendet.
Einen guten Überblick über die Vorträge bietet unser PyDDF YouTube-Kanal, auf dem wir Videos der Vorträge nach den Meetings veröffentlichen.Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf:
Programm
Das Python Meeting Düsseldorf nutzt eine Mischung aus (Lightning) Talks und offener Diskussion.
Vorträge können vorher angemeldet werden, oder auch spontan während des Treffens eingebracht werden. Ein Beamer mit XGA Auflösung steht zur Verfügung.(Lightning) Talk Anmeldung bitte formlos per EMail an info@pyddf.de
Kostenbeteiligung
Das Python Meeting Düsseldorf wird von Python Nutzern für Python Nutzer veranstaltet.
Da Tagungsraum, Beamer, Internet und Getränke Kosten produzieren, bitten wir die Teilnehmer um einen Beitrag in Höhe von EUR 10,00 inkl. 19% Mwst. Schüler und Studenten zahlen EUR 5,00 inkl. 19% Mwst.
Wir möchten alle Teilnehmer bitten, den Betrag in bar mitzubringen.
Anmeldung
Da wir nur 15 Personen in dem angemieteten Raum empfangen dürfen, möchten wir bitten, sich vorher anzumelden.
Meeting Anmeldung bitte per Meetup
Weitere Informationen
Weitere Informationen finden Sie auf der Webseite des Meetings:
http://pyddf.de/
Viel Spaß !
Marc-Andre Lemburg, eGenix.com
Test and Code
176: SaaS Side Projects
The idea of having a software as a service product sound great, doesn't it?
Solve a problem with software. Have a nice looking landing page and website. Get paying customers.
Eventually have it make enough revenue so you can turn it into your primary source of income.
There's a lot of software talent out there. We could solve lots of problems.
But going from idea to product to first customer is non-trivial.
Especially as a side hustle.
This episode discusses some of the hurdles from idea to first customer.
Brandon Braner is building Released.sh. It's a cool idea, but it's not done yet.
Brandon and I talk about building side projects:
- finding a target audience
- limiting scope to something doable by one person
- building a great looking landing page
- finding time to work on things
- prioritizing and planning
- learning while building
- even utilizing third party services to allow you to launch faster
- and last, but not least, having fun
Special Guest: Brandon Braner.
Sponsored By:
- PyCharm Professional: Try PyCharm Pro for 4 months and learn how PyCharm will save you time. Promo Code: TESTANDCODE22
- Sauce Labs: Visit saucelabs.com/testbetter for more information and a free trial. Sauce Labs. Test Continuously. Test Smarter. Develop with confidence.
Links:
<p>The idea of having a software as a service product sound great, doesn't it?<br><br> Solve a problem with software. Have a nice looking landing page and website. Get paying customers.<br><br> Eventually have it make enough revenue so you can turn it into your primary source of income. </p> <p>There's a lot of software talent out there. We could solve lots of problems. <br> But going from idea to product to first customer is non-trivial. <br> Especially as a side hustle. <br> This episode discusses some of the hurdles from idea to first customer. </p> <p>Brandon Braner is building Released.sh. It's a cool idea, but it's not done yet. </p> <p>Brandon and I talk about building side projects:</p> <ul> <li>finding a target audience</li> <li>limiting scope to something doable by one person</li> <li>building a great looking landing page</li> <li>finding time to work on things</li> <li>prioritizing and planning</li> <li>learning while building</li> <li>even utilizing third party services to allow you to launch faster</li> <li>and last, but not least, having fun</li> </ul><p>Special Guest: Brandon Braner.</p><p>Sponsored By:</p><ul><li><a href="https://testandcode.com/pycharm" rel="nofollow">PyCharm Professional</a>: <a href="https://testandcode.com/pycharm" rel="nofollow">Try PyCharm Pro for 4 months and learn how PyCharm will save you time.</a> Promo Code: TESTANDCODE22</li><li><a href="https://saucelabs.com/testbetter" rel="nofollow">Sauce Labs</a>: <a href="https://saucelabs.com/testbetter" rel="nofollow">Visit saucelabs.com/testbetter for more information and a free trial. Sauce Labs. Test Continuously. Test Smarter. Develop with confidence.</a></li></ul><p><a href="https://www.patreon.com/testpodcast" rel="payment">Support Test & Code in Python</a></p><p>Links:</p><ul><li><a href="https://www.released.sh/" title="Released" rel="nofollow">Released</a></li><li><a href="https://tailwindcss.com/" title="Tailwind CSS " rel="nofollow">Tailwind CSS </a></li><li><a href="https://tailwindui.com/" title="Tailwind UI" rel="nofollow">Tailwind UI</a></li><li><a href="https://www.figma.com/" title="Figma" rel="nofollow">Figma</a></li><li><a href="https://www.heroku.com/" title="Heroku" rel="nofollow">Heroku</a></li><li><a href="https://cloud.google.com/appengine" title="Google App Engine" rel="nofollow">Google App Engine</a></li></ul>Python⇒Speed
How vectorization speeds up your Python code
Python is not the fastest programming language. So when you need to process a large amount of homogeneous data quickly, you’re told to rely on “vectorization.”
This leads to more questions:
- What does “vectorization” actually mean?
- When does it apply?
- How does vectorization actually make code faster?
To answer that question, we’ll consider interesting performance metrics, learn some useful facts about how CPUs work, and discover that NumPy developers are working hard to make your code faster.
Read more...