A Short Primer On “extra_requires“ in setup.py

To include optional installation capabilities in your Python module’s setup.py file, you can use the extras_require parameter. The extras_require parameter allows you to define groups of optional dependencies that users can install by specifying an extra name when running pip install.

Here’s an example setup.py file that includes an optional dependency group for running tests:

from setuptools import setup, find_packages

setup(
    name='mymodule',
    version='0.1.0',
    description='My awesome module',
    packages=find_packages(),
    install_requires=[
        # Required dependencies go here
        'numpy',
        'pandas',
    ],
    extras_require={
        'test': [
            # Optional dependencies for testing go here
            'pytest',
            'coverage',
        ]
    },
)

In this example, the install_requires parameter lists the required dependencies for your module, which are required for installation regardless of which optional dependency groups are installed.

The extras_require parameter defines an optional dependency group called test, which includes the pytest and coverage packages. Users can install these packages by running pip install mymodule[test].

One can define multiple optional dependency groups by adding additional keys to the extras_require dictionary.

Using optional dependencies with the extras_require parameter in your Python module’s setup.py file has several advantages:

• It allows users to install only the dependencies they need: By defining optional dependency groups, users can choose which additional dependencies to install based on their needs. This can help to reduce the amount of disk space used and minimize potential conflicts between packages.
• It makes your module more flexible: By offering optional dependency groups, your module becomes more flexible and can be used in a wider range of contexts. Users can customize their installation to fit their needs, which can improve the overall user experience.
• It simplifies dependency management: By clearly defining which dependencies are required and which are optional, you can simplify dependency management for your module. This can make it easier for users to understand what they need to install and help to prevent dependency-related issues.
• It can improve module performance: By offering optional dependencies, you can optimize your module’s performance for different use cases. For example, you can include additional packages for visualization or data processing that are only needed in certain scenarios. This can help to improve performance and reduce memory usage for users who don’t need these features.

Related

Advertisement

How to Hide the Console When Using Subprocess

My raspberry pi is always sitting in the corner and creating timelapse videos. The entire video capturing, collating, and publishing is automated. The cleanup and other scripts also keep running on time and the system work near perfect.

The only issue I have faced with it is if someone switched it off, I get to know about it for a long time.

To compensate for this I wrote a python script on my home laptop, that monitors if the raspberry is offline and tells me when this happens.

I am using the trust winsay library.

Here’s the snippet of the script where the script tells me

subprocess.run(["say", "Check RaspberryPi, its not working"] )

The only issue I have with this subprocess call is that whenever this thing is triggered, a window appears before the user this is distracting and annoying.

So how to stop subprocess to stop opening the window is simple, just use this

si = subprocess.STARTUPINFO()
si.dwFlags |= subprocess.STARTF_USESHOWWINDOW

And call subprocess as

subprocess.run(["say", "Check RaspberryPi, its not working"] ,startupinfo=si)

That’s it. Hope this helps.

Some Related posts

Combine a Bunch Of Videos With Python

Suppose you have a bunch of videos you want to concatenate or join together. How will you do this in python?

I had the same problem. Lot of dashcam videos that I wanted to join together as a single file

Here’s how I did it.

import moviepy.editor as mpy
import os

allfiles = os.listdir("path/to/video/files")
mp4files = [f for f in allfiles if f.endswith(".mp4")]
vc = []
output_fname = "DashCam_day_22June2022.mp4"
for vid in mp4files:
    a = mpy.VideoFileClip(vid)
    vc.append(a)
clip = mpy.concatenate_videoclips(vc)
clip.write_videofile(output_fname)
# if you want the audio too use the below line
#clip.write_videofile(output_fname, temp_audiofile="out.m4a", audio_codec="aac")
_ = [a.close() for a in vc]

Posting it here for anyone’s reference. Have you used moviepy? How is your experience with it, do let me know?

Related Posts

Put an Image Behind your matplotlib plots

Here’s a quick one.

Problem.

You want to add pretty graphics in the back of your data. How to do this with matplotlib?

Solution

import numpy as np
import matplotlib.pyplot as plt

# Path to the image
fpath =r"C:\Users\sukhbinder.singh\Desktop\day.jpg"

# Read the image
img = plt.imread(fpath)

# Plot the image
fig, ax = plt.subplots()
ax.imshow(img)
a, b, c,d = plt.axis('off')

# Now plot your appropriatly scalled data. We will plot some 
# random numbers
xx = np.random.randint(a,b, size=100)
yy = np.random.randint(d,c, size=100)
plt.plot(xx,yy, "r.")
plt.savefig("wall.png")
plt.show()

Simple. Here’s the result

Image as Background in a Matplotlib Plot

Rendering Matplotlib Graphs in Django

If you have seen my post on the expense Django app, the dashboard is rendered using matplotlib’s static files which are passed on runtime to the webpage HTML.

See this here in action. All the png images are generated on the fly and sent to the browser to display.

Here’s the simple code stript down code to demonstrate this with a simple dummy graph

# DJANGO View code

from django.shortcuts import render
import matplotlib.pyplot as plt
import io
import urllib, base64

def home(request):
    plt.plot(range(10))
    fig = plt.gcf()
    #convert graph into dtring buffer and then we convert 64 bit code into image
    buf = io.BytesIO()
    fig.savefig(buf,format='png')
    buf.seek(0)
    string = base64.b64encode(buf.read())
    uri =  urllib.parse.quote(string)
    return render(request,'home.html',{'data':uri})

#HTML Template

<!DOCTYPE html>
<html lang="en" dir="ltr">
  <head>
    <meta charset="utf-8">
    <title>matplot</title>
  </head>
  <body>
    <img src="data:image/png;base64,{{ data }}" alt="" height="250" ,width="250">
  </body>
</html>

Using matplotlib, io, base64 library to accomplish the whole thing. Matplotlib to plot, io to hold the plot byte buffer and then base64 to convert the bytes to string representations that the browser can use to display.

I could have used chartsjs or other javascript libraries which will give interactive plots, but this matplotlib implementation is a good place to start and it works without any more complexity.

Checkout these Similar Posts

Error While migrating a Django Project

Here’s an error I encountered while migrating a Django project from my PC to Mac. Nothing related to the difference in architecture but as I later learned, the solution was quite simple and comes out of the box in Django.

PROBLEM

OperationalError at /admin/exp/expense/add/
no such table: exp_expense
Request Method:	POST
Request URL:	http://127.0.0.1:8000/admin/exp/expense/add/
Django Version:	2.2.16
Exception Type:	OperationalError
Exception Value:	
no such table: exp_expense
Exception Location:	D:\apps\anaconda3\lib\site-packages\django\db\backends\sqlite3\base.py in execute, line 383
Python Executable:	D:\apps\anaconda3\python.exe
Python Version:	3.8.3
Python Path:	
['C:\\Users\\Sukhbinder\\Desktop\\PROJECTS\\exptest\\test_exp',

Solution

(sdh_env) (base) C:\Users\Sukhbinder\Desktop\PROJECTS\exptest\test_exp>python manage.py migrate --run-syncdb
Operations to perform:
Synchronize unmigrated apps: exp, messages, rest_framework, staticfiles
Apply all migrations: admin, auth, contenttypes, sessions
Synchronizing apps without migrations:
Creating tables…
Creating table exp_expense
Running deferred SQL…
Running migrations:
No migrations to apply.

That’s it. Addition of –run_syncdb in the migrate command.

python manage.py migrate --run_syncdb

Have you faced anything similar?

Python is Beautiful

The code is already shared here and here but by just adding two words you can create these animations.

Yes, just two words.

marker=r'$\heartsuit$'

The one line that you have to change in the code is the scatter plot line where we add the additional marker keyword.

Here’s a very good resource on markers in matplotlib. I love matplotlib.

Let It Snow- the python code.

Last month posted this gif image of the snow animation done using matplotlib. As promised, here’s the code. Done using excellent animation capabilities of matplotlib.

The code can be used as a module as follows

from snowfall import Snowfall

snow = Snowfall("Happy Holidays").show()

Here’s the output.

Love the ease with which it can be done. The code is very generic and can be used to make any generic message with snowfall animation.

Here is the code for this. Also available at this gist

Treemap in Python

Tree Maps are primarily used to display data that is grouped and nested in a hierarchical. A treemap is one method to simultaneously display the magnitude of the major categories – as well as the magnitude of the larger subcategories in one visualization.

Ben Shneiderman introduced treemaps in the early 90s as a visualization metaphor for displaying hierarchical trees. A treemap arranges hierarchical items in rectangular bounding boxes representing the tree structure. The rectangles of a treemap are called cells.

Cells on the treemap have three visual attributes:

• The colour of a cell is a visual representation of a measure
• Its size can also represent another measure.
• The location of a cell indicates which parent cell it belongs to (Spatial Relationship)

This is what Ben Shneiderman had to say on his motivation to come up with this visualization.

During 1990, in response to the common problem of a filled hard disk, I became obsessed with the idea of producing a compact visualization of directory tree structures. Since the 80 Megabyte hard disk in the HCIL was shared by 14 users it was difficult to determine how and where space was used. Finding large files that could be deleted, or even determining which users consumed the largest shares of disk space were difficult tasks.

source

Python has an excellent package called squarify, which can be installed from pypi

Here’s a simple example to use squarify to create a treemap in python.

import squarify
import matplotlib.pyplot as plt

data={'Reserve': 363.0,
 'OtherA': 240.0,
 'Assets': 154.0,
 'Other': 117.0,
 'Debt': 109.0,
 'SC': 88.0,
 'Investments': 72.0,
 'CWIP': 10}

squarify.plot(data.values(), label=data.keys())
plt.axis("off"); plt.show()

I am using this a lot in the last few days, really good.

Read more

Comet Plot in Python

In the summer of 2016, while working on a project with Matlab, I discovered the cool plotting function in Matlab, comet.

comet(x,y) displays a comet plot of y versus x.

A comet is an animation of a marker (head) and a line (tail) tracing a growing line over the data points. The tail is a solid line that traces the entire function.

You have to see the plot to experience this.

Ever since wanted to recreate that plot in python, so here we go.

Here’s a code to display the above plot

t=np.arange(0, 4*np.pi, np.pi/50)
x = -np.sin(t) - np.sin(t/2)
y = -np.cos(t) + np.cos(t/2)

comet(x,y)

Here’s the code for comet

import matplotlib.pyplot as plt
import numpy as np


def comet(x,y=None, time=0.05):
    """
    Displays a comet plot

    by Sukhbinder
    date: 15 Feb 2021
    """
    x = np.asarray(x)
    plt.ion()
    plt.xlim(x.min(), x.max())
    if y is not None:
        y = np.asarray(y)
        plt.ylim(y.min(), y.max())
    else:
        plt.ylim(0, len(x))
    if y is not None:
        plot = plt.plot(x[0], y[0])[0]
    else:
        plot = plt.plot(x[0])[0]

    for i in range(len(x)+1):
        if y is not None:
            plot.set_data(x[0:i], y[0:i])
        else:
            plot.set_xdata(x[0:i])
        plt.draw()
        plt.pause(time)
    plt.ioff()

Have you used comet or a similar plot? Let me know.

Two Important Reason for Testing are …

Back in 2020 during the start of the first wave in the UK, before the lockdown, I gave a quick talk on testing with Pytest to a few developer’s colleagues in the office.

Few slides were titled why test and elaborated on various (more like 8) reasons for why we should write tests?

Two reasons that I believe are most important are….

Testing enables better design of the software.

Testing is not about writing tests. It is about writing a testable code.

In order to test a single unit, we need to be able to instantiate that unit standalone – without the rest of the system.
This leads to a loosely coupled code with explicit dependencies.
This leads to a better separation of concerns, and also follows the single responsibility principle

Testing enables quick feedback

Testing provides quick feedback and makes development more productive leading to more confidence when making any changes and eventually makes refactoring a codebase easier.

Whats your reason for writing or not writing tests?

Few Things to Know in Unittest

Spellings: How to get good at it?

As my school friends will attest, spellings were never a strong suit for me. Most of this could be because I was lazy and never paid attention to it when reading. I liked reading but never looked at the spellings.

I forgot about this until few years back when I began finding my kids getting spelling tests at school. To give them extra practice, I turned to python.

This is the app that I developed which is based on the concept of spaced revision. According to wikipedia

Spaced repetition is an evidence-based learning technique that is usually performed with flashcards. Newly introduced and more difficult flashcards are shown more frequently while older and less difficult flashcards are shown less frequently in order to exploit the psychological spacing effect.

Here’s a long demo of the app being used by my older kid.

Another fun thing added to the program is voice, so my kids really do love doing this.

My kids have been using this consistently for a over a year now and they have improved their spellings.

If you want to give it a try, please download this wheel or visit this github project to download the code.

Do give it a try if you have kids and let me know how it goes?

Determining screen locked of a system using python’s standard library

I think, apart from the ease of use, python’s batteries include philosopy is one of the reason its has become so popular.

Here’s another cool functionality that we needed in one of our app that was trying to maximise the usage of computing resources when the user has locked his computer.

The problem,

Get to know if the screen is locked

def screen_locked():
    """
    Find if the user has locked their screen.
    """
    user32 = ctypes.windll.User32
    OpenDesktop = user32.OpenDesktopA
    SwitchDesktop = user32.SwitchDesktop
    DESKTOP_SWITCHDESKTOP = 0x0100

    hDesktop = OpenDesktop("default", 0, False, DESKTOP_SWITCHDESKTOP)
    result = SwitchDesktop(hDesktop)
    if result:
        return False
    else:
        return True

File and Folder Comparison with Python

Python standard library modules are incedible. There’s a small gem to compare files and directories.

Its useful

Say you have two ascii files and you want to do a file comparision, don’t worry, use python.

import filecmp

# Check two files
assert filecmp.cmp(base_reduced_bdd, bdd_file, shallow=False) is True

To compare two directories, use

x = filecmp.dircmp(dir1, dir2)

# prints a report on the differences between dir1 and dir2
x.report() 

filecmp module has utilities for comparing files and directories.

It consists of the following.

Classes:
    dircmp

Functions:
    cmp(f1, f2, shallow=True) -> int
    cmpfiles(a, b, common) -> ([], [], [])
    clear_cache()



Signature: filecmp.cmp(f1, f2, shallow=True)
Docstring:
Compare two files.

Arguments:

f1 -- First file name

f2 -- Second file name

shallow -- Just check stat signature (do not read the files).
           defaults to True.

Return value:

True if the files are the same, False otherwise.

This function uses a cache for past comparisons and the results,
with cache entries invalidated if their stat information

Refer docs for more usage.

One-liner in Python to get the Directories

Python’s standard library is powerful and can be used for a lot of good things.

Here’s a simple one-liner to collect all the directories from a path.

inner_dirs = next(os.walk("."))[1]

What are your favourite one-liners that you tend to regularly use in python?

Memory Profile Your code with Ipython

Lets say you have a function that you want check the memory usage in python. This can be evaluated with another IPython extension, the memory_profiler.

The memory profiler extension contains two useful magic functions: the %memit magic and the %mprun function.

%memit magic gives the peak and total memory used by a function, while %mprun provides a line by line usage of memory.

file: temp_interp.py

import numpy as np
from scipy.interpolate import interp1d
def test(n):
	a = np.random.rand(n,4000,30)
	x = np.arange(n)
	xx = np.linspace(0,n, 2*n)
	f= interp1d(x,a, axis=0, copy=False, fill_value="extrapolate", assume_sorted=True)
	b = f(xx)

To test this function with %mprun

from test_interp import test
%mprun -f test test(1000)

This shows a line-by-line description of memory use.

Before using, we need to load the extension:

%load_ext memory_profiler

To install the extension use the following

pip install memory_profiler

Get Activation from scikit-learn’s neural network model

I have a simple multilayer perceptron that I use on my work computer. It works well with the and has 95% accuracy on top 5 basis.

It’s a delight to see it work, but I want to get more insights on what is happening inside it, one way to make it work is to see how and what neurons are firing.

Unlike keras, sklearn doesn’t give back activations for each layer on by itself, but there is a way to get the activations,

Following is code that helps get the activation from a sklearn neural network model

def get_activations(clf, X):
        hidden_layer_sizes = clf.hidden_layer_sizes
        if not hasattr(hidden_layer_sizes, "__iter__"):
            hidden_layer_sizes = [hidden_layer_sizes]
        hidden_layer_sizes = list(hidden_layer_sizes)
        layer_units = [X.shape[1]] + hidden_layer_sizes + \
            [clf.n_outputs_]
        activations = [X]
        for i in range(clf.n_layers_ - 1):
            activations.append(np.empty((X.shape[0],
                                         layer_units[i + 1])))
        clf._forward_pass(activations) 
return activations

via stackoverflow

Shutil make_archive to Rescue

Shutil to rescue

I am amazed at the versatility of the shutil library, one usage that i discovered recently was its ability to create archives.

Previously I was always using zipfile module from python but make_archive function of shutil is such an intuitive function to use.

With a single line you can take backup of a folder.

Example

shutil.make_archive(output_filename, 'zip', dir_name)

shutil.make_archive(base_name, format, root_dir=None, base_dir=None,
verbose=0, dry_run=0, owner=None, group=None, logger=None)

Create an archive file (eg. zip or tar).

For more info check docs.