Standard setup.py

Yes yes I know we should use poetry and other packaging mechanisms, but for simple small projects at home or for a simple application the setup.py is a good place to begin

Here’s a sample setup.py that I have used in many of my personal projects

Standard Setup.py

import pathlib
from setuptools import find_packages, setup


# The directory containing this file
HERE = pathlib.Path(__file__).parent

# The text of the README file
README = (HERE / "README.md").read_text()

setup(
    name="winsay",
    version="1.1",
    packages=find_packages(),
    license="Private",
    description="say in windows",
    long_description=README,
    long_description_content_type="text/markdown",
    author="sukhbinder",
    author_email="sukh2010@yahoo.com",
    url = 'https://github.com/sukhbinder/winsay',
    keywords = ["say", "windows", "mac", "computer", "speak",],
    entry_points={
        'console_scripts': ['say = winsay.winsay:main', ],
    },
    install_requires=["pywin32"],
    classifiers=[
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.7",
    ],

)

This is useful and is always a good starting point for my project files.

The Most Important Lesson

Covid 19 has changed the world, accelerated tech adoption, made things possible we didn’t think was possible. Fastest vaccine development ever! But what were our biggest lessons from this? Thankfully as covid subsides we should all ponder on its important question.

But as explained in the below passage from Morgan Housely, we would be better off taking a 30k foot view than selecting the first specific thing that comes to mind.

A good lesson from the dot-com bust was the perils of overconfidence. But the lesson most people took away was “the stock market becomes overvalued when it trades at a P/E ratio over 30.” It was hyperspecific, so many of the same investors who lost their shirts in 2002 got up and walked straight into the housing bubble, where they lost again.

The most important lessons from a big event are usually the broad, 30,000-foot takeaways. They’re more likely to apply to the next iteration of crisis.

Morgan Housely

JSON to Named Tuple

You have a JSON file and you are tired of getting the JSON just as a plain vanilla dictionary, then the following code using the namedtuple available in the collections module in standard python can come to your rescue

Here’s an example

from collections import namedtuple
import json
fname =r"D:\pool\JobFolder\INLT2916\1\run_1\sample_1.json"
with open(fname, "r") as fin:
    data = json.load(fin)
def convert(dictionary):
    for key, value in dictionary.items():
            if isinstance(value, dict):
                dictionary[key] = convert(value) 
    return namedtuple('GenericDict', dictionary.keys())(**dictionary)
objdata = convert(data)

Observe the convert definition. Now one can access its elements like

objdata.tasks.NXUpdate.start_date
objdata.metadata.running_tasks

Hope this helps someone.

Similar posts

Terrified to do anything…..

Sometimes Twitter is a gem. Only social media that I follow. Here’s one thought that has remained with me since I read it.

If you went back in time before your birth you’d be terrified to do anything, because you’d know that even the smallest nudges to the present can have major impacts on the future.

Applied to today, a reminder that how you live each day really does matter. It changes the future.

via this

Rendering Matplotlib Graphs in Django

If you have seen my post on the expense Django app, the dashboard is rendered using matplotlib’s static files which are passed on runtime to the webpage HTML.

See this here in action. All the png images are generated on the fly and sent to the browser to display.

Here’s the simple code stript down code to demonstrate this with a simple dummy graph

# DJANGO View code

from django.shortcuts import render
import matplotlib.pyplot as plt
import io
import urllib, base64

def home(request):
    plt.plot(range(10))
    fig = plt.gcf()
    #convert graph into dtring buffer and then we convert 64 bit code into image
    buf = io.BytesIO()
    fig.savefig(buf,format='png')
    buf.seek(0)
    string = base64.b64encode(buf.read())
    uri =  urllib.parse.quote(string)
    return render(request,'home.html',{'data':uri})
#HTML Template

<!DOCTYPE html>
<html lang="en" dir="ltr">
  <head>
    <meta charset="utf-8">
    <title>matplot</title>
  </head>
  <body>
    <img src="data:image/png;base64,{{ data }}" alt="" height="250" ,width="250">
  </body>
</html>

Using matplotlib, io, base64 library to accomplish the whole thing. Matplotlib to plot, io to hold the plot byte buffer and then base64 to convert the bytes to string representations that the browser can use to display.

I could have used chartsjs or other javascript libraries which will give interactive plots, but this matplotlib implementation is a good place to start and it works without any more complexity.

Checkout these Similar Posts

Room For Error

Photo by Pixabay on Pexels.com

The recent market declines and my stock WhatsApp groups falling silent reminded me of this piece that I had saved on my desktop, written by Morgan Housely

Many bets fail not because they were wrong, but because they were mostly right in a situation that required things to be exactly right. Room for error – often called the margin of safety – is one of the most underappreciated forces in business.

It comes in many forms:
A frugal budget, flexible thinking, and a loose timeline – anything that lets you live happily with a range of outcomes.

It’s different from being conservative.

Conservative is avoiding a certain level of risk. Margin of safety is raising the odds of success at a given level of risk by increasing your chances of survival.

Its magic is that the higher your margin of safety, the smaller your edge needs to be to have a favourable outcome. And small edges are where big payoffs tend to live, since most people don’t have the patience to wait around for them.

Here are a few posts related to the same theme

Error While migrating a Django Project

Here’s an error I encountered while migrating a Django project from my PC to Mac. Nothing related to the difference in architecture but as I later learned, the solution was quite simple and comes out of the box in Django.

PROBLEM

OperationalError at /admin/exp/expense/add/
no such table: exp_expense
Request Method:	POST
Request URL:	http://127.0.0.1:8000/admin/exp/expense/add/
Django Version:	2.2.16
Exception Type:	OperationalError
Exception Value:	
no such table: exp_expense
Exception Location:	D:\apps\anaconda3\lib\site-packages\django\db\backends\sqlite3\base.py in execute, line 383
Python Executable:	D:\apps\anaconda3\python.exe
Python Version:	3.8.3
Python Path:	
['C:\\Users\\Sukhbinder\\Desktop\\PROJECTS\\exptest\\test_exp',

Solution

(sdh_env) (base) C:\Users\Sukhbinder\Desktop\PROJECTS\exptest\test_exp>python manage.py migrate --run-syncdb
Operations to perform:
Synchronize unmigrated apps: exp, messages, rest_framework, staticfiles
Apply all migrations: admin, auth, contenttypes, sessions
Synchronizing apps without migrations:
Creating tables…
Creating table exp_expense
Running deferred SQL…
Running migrations:
No migrations to apply.

That’s it. Addition of –run_syncdb in the migrate command.

python manage.py migrate --run_syncdb

Have you faced anything similar?

Will Tech Giants of Today Last?

In a recent Daily Journal Annual Meeting, Charlie Munger was asked

Question: Do the great tech giant franchises of our day, specifically Microsoft, Apple, and Alphabet, have the same long-term durability that Coca-Cola had 30 to 40 years ago?

Charlie Munger: It’s a lot easier to predict who flourished in the past because we know what happened in the past. But now I want to compare what’s gonna happen in the future. Of course, that’s harder.

It’s very hard for me to imagine—that doesn’t mean it couldn’t happen—but I would expect Microsoft, Apple, and Alphabet to be strong 50 years from now—really strong, still strong. But if you’d asked me when I was young what was gonna happen to the department stores that went broke or the newspapers which were broke, and so on, I wouldn’t have predicted that either. I think it’s hard to predict how your world is going to change if you’re going to talk about 70, 80, 90 years.

Just imagine, they wiped out the shareholders of General Motors, they wiped out the shareholders at Kodak. Who in the hell would have predicted that? This technological change can destroy a lot of people. It’s hard to predict for sure in advance.

Use H5REPACK to Reclaim the Space

h5repack is a command-line tool that applies HDF5 filters to an input file file1, saving the output in a new file, file2.

Removing entire nodes (groups or datasets) from an hdf5 file should be no problem.
However, if you want to reclaim the space you have to run the h5repack tool.

h5repack -h

From the hdf5 docs:

Deleting a Dataset from a File and Reclaiming Space

HDF5 does not at this time provide an easy mechanism to remove a dataset from a file or to reclaim the storage space occupied by a deleted object.

Removing a dataset and reclaiming the space it used can be done with the H5Ldelete function and the h5repack utility program. With the H5Ldelete function, links to a dataset can be removed from the file structure. After all the links have been removed, the dataset becomes inaccessible to any application and is effectively removed from the file. The way to recover the space occupied by an unlinked dataset is to write all of the objects of the file into a new file. Any unlinked object is inaccessible to the application and will not be included in the new file. Writing objects to a new file can be done with a custom program or with the h5repack utility program.

Few related posts

Thermodynamics, Life and Universe

Thermodynamics was my favourite subject when I was in college. The subject felt close to something I can relate to. I did not know why I like it better than others, I liked fluid dynamics too but thermodynamics was always my top one. Reading the book Einsteins Fridge did rekindled that love of that subject.

Here’s a quote from the book.

At its heart are three concepts energy, entropy, and temperature. Without an understanding of these and the laws they obey, all science physics, chemistry, and biology would be incoherent. The laws of thermodynamics govern everything from the behavior of atoms to that of living cells, from the engines that power our world to the black hole at the center of our galaxy.

Thermodynamics explains why we must eat and breathe, how the lights come on, and how the universe will end.

From the book

Some Related Posts:

To Get Conda in cmd System-Wide

You have installed Miniconda3 on my Windows (10/7) laptop. You can run python through the Anaconda Prompt but python is not recognised in the Windows Command Prompt.

From Windows Command Prompt if you type in ‘conda info’ you get this because it doesn’t even recognise conda:

‘conda’ is not recognized as an internal ….

How to solve this?

Sometimes having conda in cmd line is a useful thing to have. Opening Anaconda prompts just for accessing conda utilities is a hassle, so I always have conda available in cmd systemwide.

Here’s are two steps to follow.

To get conda in cmd system-wide

Step 1

If Anaconda is installed for the current user only, add %USERPROFILE%\Anaconda3\condabin (I mean condabin, not Scripts) into the environment variable PATH (the user one). If Anaconda is installed for all users on your machine, add C:\ProgramData\Anaconda3\condabin into PATH.

How do I set system environment variables on Windows?

set path=%path%%USERPROFILE%\Anaconda3\condabin

Step 2
Open a new Powershell or CMD, run the following command once to initialize conda.

conda init


These steps make sure the conda command is exposed to your cmd.exe and Powershell.

Hope this helps someone.

Pair this post with these useful posts to dig deeper.

It’s Not What You Want…

Photo by Pixabay on Pexels.com

Johannes Gensfleisch was a German inventor, printer, publisher, and goldsmith who introduced printing to Europe with his mechanical movable-type printing press. His work started the Printing Revolution in Europe and is regarded as a milestone of the second millennium.

Here’s a story from the book From Gutenberg to Google: The History of Our Future Hardcover by Tom Wheeler on what he might have felt after inventing the movable type and how it would be Guternber day.

It is worthwhile pausing at this point to savor Gutenberg’s success.

Imagine the exultation and celebration that must have gripped Johannes Gutenberg as his first printed book was bound!

More than a decade in development, Gutenberg’s understanding that a page of information was the sum of its parts had required a “secret art” to both discover a revolutionary new process and find the means of adjusting a seemingly endless number of variables into harmonious production.

Now it was done. Success had been achieved in twenty-eight pages of Latin grammar instruction.

The Western world had never before seen the rapid production of hundreds of perfect-quality pages, each one identical to the others. It was a moment to be savored, a decade-long quest with a transformative result.

Unfortunately, the exultation would be short-lived.

Other mass-market documents flowed from Gutenberg’s printing shop. The earliest dated work was a papal indulgence of 1454. Having spent more than a decade perfecting his technique, however, Gutenberg, it would appear, was not satisfied with such run-of-the-mill products. He wanted a monument. Today we call that monument the Gutenberg Bible.

It would be his downfall.

The Gutenberg Bible, Tim Hartford says in his book called Adapt: Why Success Always Starts With Failure., was a failure. It’s a pretty strange example of failure since it was the first book printed with movable type which started 500 years of mass communication. What we do now is pretty much the result of Gutenberg bible.

Believe it or not, the Gutenberg Bible was a total flop for Johannes Gutenberg, the father of the printing press. He went bankrupt trying to make money printing this book. Just like with any new technology, it was very expensive to print books. It was actually so expensive that you might as well hand write them rather then print them. The business model which eventually worked out a bit after Gutenberg’s time was printing leaflets for the church. These early leaflets for the church kept the early printing industry afloat.

Pair this with these posts.

Use FORFILES to Automate and Simplify Windows workflow

If you are using windows command line cmd, you need to know about the for command. This command is useful and can help you automate many common tasks with the windows batch command. I have talked about the for command a lot.

But recently I was surprised to find another gem of a command that makes using files a lot more convenient. The command is forfiles.

Don’t know when this was introduced but it’s a command worth learning about if you manipulate files from the command line of windows.

Forfiles helps your select a file or set of files and execute a command on that file. Really really useful if you work on batch jobs involving a number of files.

FORFILES [/P pathname] [/M searchmask] [/S]
         [/C command] [/D [+ | -] {MM/dd/yyyy | dd}]

Description:
    Selects a file (or set of files) and executes a
    command on that file. This is helpful for batch jobs.

Parameter List:
    /P    pathname      Indicates the path to start searching.
                        The default folder is the current working
                        directory (.).

    /M    searchmask    Searches files according to a searchmask.
                        The default searchmask is '*' .

    /S                  Instructs forfiles to recurse into
                        subdirectories. Like "DIR /S".

    /C    command       Indicates the command to execute for each file.
                        Command strings should be wrapped in double
                        quotes.

                        The default command is "cmd /c echo @file".

                        The following variables can be used in the
                        command string:
                        @file    - returns the name of the file.
                        @fname   - returns the file name without
                                   extension.
                        @ext     - returns only the extension of the
                                   file.
                        @path    - returns the full path of the file.
                        @relpath - returns the relative path of the
                                   file.
                        @isdir   - returns "TRUE" if a file type is
                                   a directory, and "FALSE" for files.
                        @fsize   - returns the size of the file in
                                   bytes.
                        @fdate   - returns the last modified date of the
                                   file.
                        @ftime   - returns the last modified time of the
                                   file.

                        To include special characters in the command
                        line, use the hexadecimal code for the character
                        in 0xHH format (ex. 0x09 for tab). Internal
                        CMD.exe commands should be preceded with
                        "cmd /c".

    /D    date          Selects files with a last modified date greater
                        than or equal to (+), or less than or equal to
                        (-), the specified date using the
                        "MM/dd/yyyy" format; or selects files with a
                        last modified date greater than or equal to (+)
                        the current date plus "dd" days, or less than or
                        equal to (-) the current date minus "dd" days. A
                        valid "dd" number of days can be any number in
                        the range of 0 - 32768.
                        "+" is taken as default sign if not specified.

    /?                  Displays this help message.

Examples:
    FORFILES /?
    FORFILES
    FORFILES /P C:\WINDOWS /S /M DNS*.*
    FORFILES /S /M *.txt /C "cmd /c type @file | more"
    FORFILES /P C:\ /S /M *.bat
    FORFILES /D -30 /M *.exe
             /C "cmd /c echo @path 0x09 was changed 30 days ago"
    FORFILES /D 01/01/2001
             /C "cmd /c echo @fname is new since Jan 1st 2001"
    FORFILES /D +6/9/2021 /C "cmd /c echo @fname is new today"
    FORFILES /M *.exe /D +1
    FORFILES /S /M *.doc /C "cmd /c echo @fsize"
    FORFILES /M *.txt /C "cmd /c if @isdir==FALSE notepad.exe @file"

Pair this post with these posts.

Precision versus Approximation

In the book How Long is a Piece of String by Eastaway Rob

How can you tell the difference between a mathematician and an engineer? Ask them what pi is. Mathematician: “It is a ratio describing the circumference of a circle to its diameter, a transcendental number which begins 3.14 and continues for an infinite number of digits.”

Engineer: “It’s about 3, but let’s call it 10 just to be on the safe side.”

Happy PI day!!

Read More

Error: invalid command ‘bdist_wheel’

This is a common problem that I have seen while building a wheel in python. Mostly this is the case because wheel is not a standard python library distributed with python.

Problem

(django) (base) C:\Users\sukhbinder\Desktop\PROJECTS\exp>python setup.py build bdist_wheel
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'bdist_wheel'

Solution

(django) (base) C:\Users\sukhbinder\Desktop\PROJECTS\exp>pip install wheel
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.36.2

That’s it. Install wheel from pypi.

Pair this with these posts.

Thought Experiments

On November 9 in 1921, German-born theoretical physicist Albert Einstein was named the winner of the Nobel Prize for Physics for his explanation of the photoelectric effect.

Albert Einstein famous instrument was his thought experiments. Here are a few examples of this from the book Einstein’s Fridge: How the Difference Between Hot and Cold Explains the Universe by Paul Sen

An example Einstein gives is the so-called photoelectric effect when a light beam (or any electromagnetic radiation) strikes certain substances, it creates an electric current. The relationship between the frequency of the light and the amount of electric current generated was puzzling. In many cases, bright red light produces no current, whereas dim blue light, which oscillates at a higher frequency, does produce some. Faint ultraviolet light of an even higher frequency produces even more.

Einstein explained this as follows: Light consists of lumps of energy, but the amount of energy in each lump depends on the frequency of the light. So, a lump of red light is smaller (i.e., contains less energy) than a lump of blue light. A lump of blue light is smaller than a lump of ultraviolent light. Shining red light on a substance, therefore, is akin to bombarding it with feathers. If a hundred feathers blew into your face, you could brush them off with little difficulty. Shining ultraviolet on something is equivalent to firing bullets at it. A single bullet will do far more damage than a hundred feathers. In the same way, a few particles of ultraviolet light will generate far more electric current than a large number of red-light particles.

Einstein’s 1905 paper is regarded as one of the founding texts of quantum physics. But it would take another two decades before its ideas were fully accepted and understood by the scientific community. Only in the late 1920s was the word photon adopted to describe light particles. And light’s wavelike behaviour did not go away.

Photons exhibit both particle-like and wave-like behaviour hence the phrase wave-particle duality and the many mysteries of quantum physics.

Another example is seeing sugar water

In Einstein’s imagination, sugar water isn’t an unvarying, indivisible liquid. Rather, it is a mass of tiny water molecules, which he visualized as tiny spheres bumping into and jostling with one another. Interspersed within the water molecules are the larger spheres of the sugar molecules, which prevent the smaller spheres from moving freely.

In effect, the water molecules are slowed down as they collide with the large sugar molecules. The consequence is that sugar water is considerably stickier than pure water. Einstein then went on to show how this description made good quantitative predictions.

His thesis is packed with mathematical equations, which guide the reader to an extraordinary conclusion that two simple measurements of the behaviour of sugar water allow one to estimate the diameter of an individual sugar molecule.

The two measurements are 1. Compare the viscosity of a sugar solution containing a known amount of sugar with that of pure water. 2. Measure the osmotic pressure of the sugar solution.

A Common Problem When Using a Virtual Machine

Here’s a common problem when using a virtual machine. You go out for lunch or a small errand to get the food order from the security gate of your apartment and the virtual machine locks out.

If you are especially unlucky the VDI machine completely disconnects due to aggressive inactivity monitoring. And if the VDI machines are in short supply or you are in peak working hours, good luck in getting the system back.

Hours wasted. Usually, this ruins the work mood and spoils the experience of working.

To mitigate this frequent issue, wanted to have my VDI machine always running, so needed a system to make this work.

Python was my first choice, but due to security, the system was stripped to a bare minimum and even python was not installed in the virtual system and was unreliable.

So turned to the trusty windows command line cmd.

My first attempt was this

@echo off

:loop
RunDll32.exe user32.dll,SetCursorPos
timeout /t 45
goto loop

This simple windows batch moves the mouse by a tiny amount every 45 seconds. Yes, it’s annoying but I was hoping this will work, but alas I was wrong. Bloody windows.

Searched online and someone said that moving the mouse is not enough. Windows is looking for user input, not just mouse movement. Damm it.

So the next solution was two parts.

Created a VBScript neversleep.vbs

set shell = CreateObject("WScript.Shell")
WScript.Sleep 1500
shell.SendKeys"{SCROLLLOCK}"
WScript.Sleep 1500
shell.SendKeys"{SCROLLLOCK}"

This script mimics the pressing of a key on the keyboard.

And the second file is the command line batch script to fire the VBScript after every 45 seconds.

@echo off

:loop
cscript "C:\Users\sukhbinder\Desktop\neverSleep2.vbs" //nologo
timeout /t 45
goto loop

I used the most useless key on the keyboard Scroll Lock and it is nice to see it briefly blink now and then.

If the VDI servers are working properly this solution works like a charm. Happy!! But not for long as…….

Well, let’s park that for another post.

If you found this post useful, you might like these too.