Some Useful pytest Command-line Options

I love pytest.

Pytest is a testing framework which allows us to write test codes using functional python and functional python is awesome.

Why use PyTest?

There are many reasons to use pytest here are some that I feel are important.

  • Very easy to start with because of its simple and easy syntax.
  • Less Boilerplate
  • Can run a specific test or a subset of tests
  • and many more useful features

Here’s a list of command-line options that can be used while using pytest.

Simple use
pytest

Too unorganised, lets’ fix this

pytest -v

Much better.

oh there’s a failure but there is too much information on the failure, let’s fix that with

pytest -v –tb=line

This is good, but just a line of info is too little. Ok lets try this.

pytest -v –tb=short

Thats good.

What if I want to run a specific test. No problem just use “-k” option

pytest -v -k “SOMENAME”

Thats cool. What if I want to just run the last failed test or tests. Simple use “–lf

pytest -v –lf

And if you want to debug the failed tests, well use “–pdb

pytest -v –pdb

On failure, it will bring the debugger.

Well, that’s it for this post. Hope this helps.

More posts like this that you might want to explore.

Calendar in Cmd

Calendar in CMD using python

In my quest to make my windows system mostly behave like Mac, i keep adding things that i miss in windows like the ability to see the calendar in terminal.

In macbook pro, typing cal anywhere in the terminal gives you the calendar.

Python comes to my rescue in all these quests to turn windows more like the mac os.

The first step to do that is use alias and create a cal keyword. This post talks about setting alias in windows using doskey.

And the meat of the application is this python one-liner that gives you the calendar output in text format.

python -m calendar 2021 10

If you want the calendar for entire year, just skip the year and month

python -m calendar

Hope this helps someone else.

Simple Gems of Python

Like last week’s post, this functionality is one I have applied in many of my pet projects and scripts. The use case is to get the string values from the user clipboard.


How to get clipboard contents only using standard python?


There are few methods, but the method shown below is the one that I have used successfully across windows, Mac, and Linux systems.

def get_clipboard():
    import tkinter as tk
    try:
        root = tk.Tk()
        root.withdraw()
        return root.clipboard_get()
    except Exception:
        return None

Here’s a quick demo. 🙂

Get clipboard contents only using standard python

Do you know of any other gems like this?

Two Important Reason for Testing are …

Back in 2020 during the start of the first wave in the UK, before the lockdown, I gave a quick talk on testing with Pytest to a few developer’s colleagues in the office.

Few slides were titled why test and elaborated on various (more like 8) reasons for why we should write tests?

Two reasons that I believe are most important are….

Testing enables better design of the software.

Testing is not about writing tests. It is about writing a testable code.

In order to test a single unit, we need to be able to instantiate that unit standalone – without the rest of the system.
This leads to a loosely coupled code with explicit dependencies.
This leads to a better separation of concerns, and also follows the single responsibility principle


Testing enables quick feedback

Testing provides quick feedback and makes development more productive leading to more confidence when making any changes and eventually makes refactoring a codebase easier.

Whats your reason for writing or not writing tests?

Reversi Game in Python

I started using python in 2010 but it never entered my mainstream work till 2012/13. Till that time, most of my work was in C and Fortran.

In April 2013, while taking a break from C and FORTRAN, I began my days by writing python games.
Not with pygame, but games for consoles, one of the first games I wrote was revesi. 

It’s a simple one-player strategy game where the aim is to arrange the numbers in the sequence. I saw this on one of the javascript websites and liked it so much that I decided to implement this in python.

I have stumbled upon this game last week on an old hard disk. Was delighted to see it was working. So did what  I should have done a long time ago, put it out on GitHub for anyone interested to play.

Here’s a demo of me playing the first round.

Simple game but it turns out to be very engaging.


In April 2013, wrote many other games, which I will push online as I get some time. so watch this space.


Do let me know if you have played anything similar?

Determining screen locked of a system using python’s standard library

I think, apart from the ease of use, python’s batteries include philosopy is one of the reason its has become so popular.

Here’s another cool functionality that we needed in one of our app that was trying to maximise the usage of computing resources when the user has locked his computer.

The problem,

Get to know if the screen is locked

def screen_locked():
    """
    Find if the user has locked their screen.
    """
    user32 = ctypes.windll.User32
    OpenDesktop = user32.OpenDesktopA
    SwitchDesktop = user32.SwitchDesktop
    DESKTOP_SWITCHDESKTOP = 0x0100

    hDesktop = OpenDesktop("default", 0, False, DESKTOP_SWITCHDESKTOP)
    result = SwitchDesktop(hDesktop)
    if result:
        return False
    else:
        return True




Starter Argparse and Setup.py templates

I tend to develop my python scripts as small apps, this way I can call and use them from every where in command line without relying on the calling the python scripts

Here’s a template of the setup.py and argparse template that I always start with.

Setup.py example is for making the application as a package and installable.

Install_requires is a list of packages required by the app.
Entry_points is optional if given, app can be called using the ‘app_name’ from command line.

import os
from setuptools import find_packages, setup
 
setup(
    name= 'name',
    version="3.0",
    packages=find_packages(),
    include_package_data=True,
    zip_safe=False,
    license="Private",
    description= "This is the description",
    author= "author_name",
    author_email= "author_contact_email",
 
    install_requires=["psutil", "pywin32"],
 
    entry_points={
        'console_scripts': ['app_name = app:main']
    }
)
 

Argparse example template

Following a starter template for using argparse.

def main():
    parser = argparse.ArgumentParser(
        description="Pro Thermals: Uses Iges from NX Update to perform thermal analysis on scenery Model")
    parser.add_argument("indir", type=str, help="Input dir ")
    parser.add_argument("outdir", type=str, help="Output dir ")
    parser.add_argument("-s", "--scpath", type=str, help="simulation executable path ",
                        default="W:\\simulation_app\\MSWindows\\bin\\x64")
    parser.add_argument("-o", "--omp-threads", type=int, help="Specifies how many OMP_NUM_THREADS app uses while running ",
                        default=1)
 
    args = parser.parse_args()
 
    input_dir = args.indir or os.getcwd()
    output_dir = args.outdir
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)
 
    run_ok = db_thermals(input_dir,
                         output_dir,
                         args.scpath, args.omp_threads)
 
if __name__ == "__main__":
    main()

In the above example, indir and outdir are required positional arguments, while scpath and omp-threads are optional arguments with some defaults.

.

Few Useful Conda Commands To Get Started.

Conda is a tool for managing and deploying applications, environments and packages.

I use conda a lot. It has lot of functionalities and there is lot to learn in it but for regular usage, there are just few things that they need to know to use it productively.

so here’s a list of commands that I had shared with my colleagues to get them started in conda.

This commands will just be enough to get the ground running and do most of the stuff.

Few useful conda commands

  • conda env export –f env.yml -> Exports the environment in env.yml file
  • conda env remove -n env_name -> Remove a environment
  • conda create –yes -n env_name python=3.5 -> Create an environment with python 3.5
  • conda info –envs -> List all environments
  • activate env_name -> activates environment
  • deactivate -> deactivates the current environment
  • conda install package_name -> install a package in the current environment
  • conda uninstall package_name -> uninstall a package in the current environment

Joint plot with Matplotlib

Today I am releasing a simple module to create joint plot with Matplotlib on github. Joint plot is available in the excellent seaborn library but unfortunately it’s not always available on many systems. Recently I needed this functionality, so wrote this simple module with matplotlib.

The functionality is almost similar to seaborn but with limited feature. This has helped me in my work, releasing it in the hope that others might find it useful.

Sample usage:

Import Jointplot
tips=pd.read_csv(r'tests/tips_p.csv')
data=np.c_[tips['total_bill'].values,tips['tip'].values]
jointPlot(data,kde=True)

Find the code at this github repository.

Quarterly Results Analysis with Python

Every quarter when results come out, I spend sufficient hours looking at the financial results of the stocks I am tracking to elicit nagging comments from my better half. So developed this simple python script which does the analysis and generates me the PDF which I email to myself to look at during office commute.

The code as always is available at my github page here.

Some sample plots

Intend to update the scripts for other analysis as and when I get some time.

Simple trick to read mixed format data with numpy’s genfromtxt

Suppose your data is like the following


DD,MM,YY,AMT,WHERE
30,5,2015,2,TRAVEL
30,5,2015,50,TRAVEL
30,5,2015,5,PHONE
31,5,2015,6.62,TESCO
31,5,2015,5,POUNDSHOP
31,5,2015,4.51,SAINSBURY
31,5,2015,1,PHONE

Let’s load it with numpy using genfromtxt


expdata=np.genfromtxt(fname, delimiter=',',skip_header=1,usecols=[0,1,2,3,4])

print expdata[0]
print expdata[-1]

This prints the following


[  3.00000000e+01   5.00000000e+00   2.01500000e+03   2.00000000e+00             NaN]
[   30.     5.  2015.    50.    NaN]

Numeric data is read but not quite as what we wanted. Notice the NaN.

You can supply genfromtxt the datatypes using the keyword format like

dtype=([(‘f0’, ‘<i4’), (‘f1’, ‘<i4’), (‘f2’, ‘<i4’), (‘f3’, ‘<f8’), (‘f4’, ‘|S14’)])


expdata=np.genfromtxt(fname, delimiter=',',skip_header=1,usecols=[0,1,2,3,4],dtype=([('f0', '&amp;amp;lt;i4'), ('f1', '&amp;amp;lt;i4'), ('f2', '&amp;amp;lt;i4'), ('f3', '&amp;amp;lt;f8'), ('f4', '|S14')]))

(30, 5, 2015, 2.0, 'TRAVEL')
(30, 5, 2015, 50.0, 'TRAVEL')

But that looks too much work but there’s a simple smart way to do this. Use dtype=None

 


expdata=np.genfromtxt(fname, delimiter=',',skip_header=1,usecols=[0,1,2,3,4],dtype=None)

(30, 5, 2015, 2.0, 'TRAVEL')
(30, 5, 2015, 50.0, 'TRAVEL')

Using dtype=None is a good trick if you don’t know what your columns should be and you need some help in getting the format. This might be slower but it does work and once you have the data you can replace the dtype none with the appropriate arguments.

Quick Gantt Chart with Matplotlib

ProjectPlan_GANTT_CHart_MatplotlibThe problem: You need a quick Gantt chart for a quick proposal report and you dint have any project planner software installed.

Solution:

While there are many different ways, you only have access to python, well Here’s a simple Gantt chart plotter with just matplotlib.

Not for rigorous use but a good substitute to make quick Gantt plot for quick report.

Continue reading

Biplot  in Python revisited. 

Mark sent a recent email complaining  the previous biplot code not working. Though I was not able to replicate his errors,but from the error message figured the error was due to the PCA numbers supplied.

That reminded me of the simplification task that I intended to do on the previous version to make it work like it works in matlab.

No dependency on PCA data structure, send to variables, scores and coefficient and plot the biplot.

Continue reading

Matlab to Python – some code examples

Two years back, I was converting a matlab script to python, here are some of the errors that I encountered during the conversion. Found them documented in that converted script, posting them here for wider audience.



Matlab to Python

1. () to []

nd=topo(j,i)
X(j,i)=coords(nd,1)
Y(j,i)=coords(nd,2)

SyntaxError: can't assign to function call

nd=topo[j,i]
X[j,i]=coords[nd,1]
Y[j,i]=coords[nd,2]

2. 1 to 0

X(j,i)=coords(nd,1)
Y(j,i)=coords(nd,2)

IndexError: index (2) out of range (0<=index<2) in dimension 1

X[j,i]=coords[nd,0]
Y[j,i]=coords[nd,1]

3. zeros to zeros

B=zeros(2,edof)

In Python

B=np.zeros((2,edof))

4. array to array

D=G*[1 0; 0 1];

in python

D_ps=G* np.array([[1.0,0.0],[0.0,1.0]])

5. % to #

% is comments in matlab

Python

# is comments in python

6. For to for

for i=1:nnel
node(i)=nodes(iel,i);
end

python

for i in range(nnel):
node[i]=nodes[iel,i]

7. Matlab find to python find

L1 = find(coordinates(:,2)==min(coordinates(:,2)))

in python

l1 = np.where(coords[:,1]==np.min(coords[:,1])

Pdf With Matplotlib

I thought everyone knew about but i was surprised this this little feature of matplotlib is not that known as widely as I assumed.

We all know we can save a plot from matplotlib to pdf but there other little feature hiding in the backends where we can write out a multiple page pdf

Here’s a simple code to write out multiple page pdf using matplotlib.


import numpy as np
import matplotlib.pyplot as plt
from matplotlib.backends.backend_pdf import PdfPages

pdf=PdfPages('matplotlibplot.pdf')

fig=plt.figure()
plt.plot(np.random.rand(10));
pdf.savefig(fig)

fig=plt.figure()
plt.hist(np.random.randn(1000));
pdf.savefig(fig)

pdf.close()

So easy. Instantiate pdfpages, Open the pdf file, save figures. Works from version 0.99 of matplotlib.
With little use of plt.text, plt.annotate, can be used to produce quick pdf reports. In fact,I have pandas and matplotlib workflow to pump out full blown management report on continuous improvement initiative.
Here’s a simple full example of creating a simple pdf report using the above feature

Continue reading

Biplot with Python

biplot in python

I have plotted Biplot in Matlab and have created it using fortran in the past. Last month, while playing with PCA, needed to plot biplots in python. Unlike MATLAB, there is no straight forward implementation of biplot in python, so wrote a simple python function to plot it given score and coefficients from a principal component analysis.

Here’s the function.


def biplot(score,coeff,pcax,pcay,labels=None):
    pca1=pcax-1
    pca2=pcay-1
    xs = score[:,pca1]
    ys = score[:,pca2]
    n=score.shape[1]
    scalex = 1.0/(xs.max()- xs.min())
    scaley = 1.0/(ys.max()- ys.min())
    plt.scatter(xs*scalex,ys*scaley)
    for i in range(n):
        plt.arrow(0, 0, coeff[i,pca1], coeff[i,pca2],color='r',alpha=0.5)
        if labels is None:
            plt.text(coeff[i,pca1]* 1.15, coeff[i,pca2] * 1.15, "Var"+str(i+1), color='g', ha='center', va='center')
        else:
            plt.text(coeff[i,pca1]* 1.15, coeff[i,pca2] * 1.15, labels[i], color='g', ha='center', va='center')
    plt.xlim(-1,1)
    plt.ylim(-1,1)
    plt.xlabel("PC{}".format(pcax))
    plt.ylabel("PC{}".format(pcay))
    plt.grid()

Biplot_PCA_in_Python

Plotted using

biplot(score,pca.components_,1,2,labels=categories)

What is Biplot?

Biplot is one of the most useful and versatile methods of multivariate data visualisation. The bipolar extends the idea of a simple scatter plot of two variables to the case of many variables, with the objective of visualising the maximum possible information in the data.

From wikipedia

A biplot allows information on both samples and variables of a data matrix to be displayed graphically. Samples are displayed as points while variables are displayed either as vectors, linear axes or nonlinear trajectories.

If you would like to dig deeper, here’s a link on a comprehensive introduction to Biplots [PDF].

What can Software Teams Learn From Aviation?

This pycon talk combines my two passions, aviation and software development. Grab a cold coffee and watch this very interesting talk. Just a little over half an hour, if you have any interest in any of this two stream, you will like the talk.

What can Python-based software teams learn from aviation? Why should software always fail hard? What’s wrong with too many error logs? And why are ops people already like pilots? Learn all this, and about planes, too.

If in hurry, you can download the PDF of the presentation.