Grep in windows

Mac and Linux have great grep command to search within files, but in windows, this is difficult.

I use the following, not well advertised, command to do things that grep does.

findstr /s /m /i /c:”reference condition” *

Searches for strings in files.

FINDSTR [/B] [/E] [/L] [/R] [/S] [/I] [/X] [/V] [/N] [/M] [/O] [/P] [/F:file]
        [/C:string] [/G:file] [/D:dir list] [/A:color attributes] [/OFF[LINE]]
        strings [[drive:][path]filename[ ...]]

  /B         Matches pattern if at the beginning of a line.
  /E         Matches pattern if at the end of a line.
  /L         Uses search strings literally.
  /R         Uses search strings as regular expressions.
  /S         Searches for matching files in the current directory and all
             subdirectories.
  /I         Specifies that the search is not to be case-sensitive.
  /X         Prints lines that match exactly.
  /V         Prints only lines that do not contain a match.
  /N         Prints the line number before each line that matches.
  /M         Prints only the filename if a file contains a match.
  /O         Prints character offset before each matching line.
  /P         Skip files with non-printable characters.
  /OFF[LINE] Do not skip files with offline attribute set.
  /A:attr    Specifies color attribute with two hex digits. See "color /?"
  /F:file    Reads file list from the specified file(/ stands for console).
  /C:string  Uses specified string as a literal search string.
  /G:file    Gets search strings from the specified file(/ stands for console).
  /D:dir     Search a semicolon delimited list of directories
  strings    Text to be searched for.
  [drive:][path]filename
             Specifies a file or files to search.

Thankfully windows 10 has bash command’line, I haven’t used it but hope all this will be unnecessary in  it.

Do you have a commandline trick that you always come back to?

Few Things to Know in Unittest

I have completely shifted to pytest now, but it was standard unittest module that I started, like many others.

Pytest is simple, intuitive and easy to work with but it always have to be installed as an additional package, sometime installing new things is not possible. (like on rigid office installations :()

So for that unittest, the standard package is a good option..

Apart from the usual unittests things, I have found the followings useful for real world unit testing of apps.

Putting them here as a reference

    @unittest.skip("NotDef.")
    def test_gauss_lobatto_quadrature(self):
        pass

    @unittest.expectedFailure
    def test_mc_orthogonality(self):
     super(TestLaguerreAlpha0Function,self).test_mc_orthogonality()

The following decorators and exception implement test skipping and expected failures:

@unittest.skip(reason)
Unconditionally skip the decorated test. reason should describe why the test is being skipped.

@unittest.skipIf(condition, reason)
Skip the decorated test if condition is true.

@unittest.skipUnless(condition, reason)¶
Skip the decorated test unless condition is true.

@unittest.expectedFailure
Mark the test as an expected failure. If the test fails it will be considered a success. If the test passes, it will be considered a failure.

What other things will you add to this list?

Solving UnicodeDecodeError while reading csv in python

Recently tried reading a csv file with pandas and got this error

fname = r"C:\Users\singh\Desktop\prediction.csv"
data  = pd.read_csv(fname)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 7: invalid start byte

After much head scrating, reading various stanckoverflow post and the reading the pandas docs, found the solution .

data = pd.read_csv(fname, encoding = 'unicode_escape')

yes just a single paramenter encoding = ‘unicode_escape‘

Posting it here in the hope it helps others.

Copying Entire Folders Using Standard Libaries

Python standard libraries has lot of little functions that are really useful

One such function is copy_tree

from distutils.dir_util import copy_tree
copy_tree("/a/b/c", "/x/y/z")

Availablity of these simple nuggets makes working with python a pleasant experience.

Crontab Time String Syntax

schtasks is my goto workhorse for scheduleing tasks in windows environment and on mac and raspberrypi, crontab is an excellent choice.

I use crontab a lot in lot of the things I do in raspberry, but I keep forgetting the cron time sting syntax, so here’s one that I have kept as a reference

The Cron time string is five values separated by spaces, based on the following information:

Character Descriptor Acceptable values

1 Minute 0 to 59, or * (no specific value)
2 Hour 0 to 23, or * for any value. All times UTC.
3 Day of the month 1 to 31, or * (no specific value)
4 Month 1 to 12, or * (no specific value)
5 Day of the week 0 to 7 (0 and 7 both represent Sunday), or * (no specific value)

Cron time string Description

30 * * * * Execute a command at 30 minutes past the hour, every hour.
0 13 * * 1 Execute a command at 1:00 p.m. UTC every Monday.
*/5 * * * * Execute a command every five minutes.
0 */2 * * * Execute a command every second hour, on the hour.

Posting it here for my reference. Hope others will find it useful

Few Useful Conda Commands To Get Started.

Conda is a tool for managing and deploying applications, environments and packages.

I use conda a lot. It has lot of functionalities and there is lot to learn in it but for regular usage, there are just few things that they need to know to use it productively.

so here’s a list of commands that I had shared with my colleagues to get them started in conda.

This commands will just be enough to get the ground running and do most of the stuff.

Few useful conda commands

• conda env export –f env.yml -> Exports the environment in env.yml file
• conda env remove -n env_name -> Remove a environment
• conda create –yes -n env_name python=3.5 -> Create an environment with python 3.5
• conda info –envs -> List all environments
• activate env_name -> activates environment
• deactivate -> deactivates the current environment
• conda install package_name -> install a package in the current environment
• conda uninstall package_name -> uninstall a package in the current environment

Python 2 to Python 3

Python 2 has officially retired, you should be using python 3 now.

2to3

If you have few code and scripts in python2, there’s a tool available with standard python installation, which can help you convert your scripts to python 3 without much effort.

2to3 is the tool available your in command line.

Suppose my python2 code is in the folder python2-version/mycode, the following line will create a python3 version of the script.

2to3 --output-dir=python3-version/mycode -W -n python2-version/mycode

And as the docs mention, you can use some flags for more customization:

• the -w flag to enable writeback, which applies the changes to the file
• the -n to disable backups

There are a few more flags; see the docs for more information

If you don’t have 2to3 on your path, you can directly invoke lib2to3:

python -m lib2to3 -W -n directory\file.py --output-dir=python3-version/mycode

Bacteria and Virus

Everyone I talk to sometime in the conversation talks about this.

Why don’t we know how to cure/tame this virus when we do have cure for so many diseases?

I can’t explain this better than the below note.

There are two reasons for this failure to have anything on the shelf that can be used to treat viruses: one biological, the other economic. The biological problem, as Amesh Adalja of Johns Hopkins University argued in a prescient call to arms just before the pandemic struck, is that viruses do not have their own biochemistry, because they borrow ours.

So unlike, say, tuberculosis, there is not much to attack. As any doctor will tell you, antibiotics are no use in fighting a virus. They interfere with machinery found only in bacteria, but there is no equivalent machinery in viruses — which are just a bunch of genes (15 of them in the case of Sars-CoV-2) that borrow our body’s machinery to replicate themselves.

The problem is that viruses differ from each other, so treatments that work for one seldom work for another. The drugs that work against HIV-1, the main cause of Aids, sometimes do not even work against HIV-2,a milder version of the virus. Those that work against herpes don’t kill the very similar cytomegalovirus. One influenza drug works only against influenza A and not B. One antiviral kills just one genotype of hepatitis C. It is no coincidence that the antiviral treatments capable of attacking more kinds of virus, such as ribavirin, are also the most toxic to the patient, because they tend to attack the machinery of the host as well.

Do invest some time to read to this? Also if you decide to read more you will discover how Fujifilms (not Kodak) can help fight this pandemic.

http://www.rationaloptimist.com/blog/race-to-cure-covid

Inactivity matters

For me investing is like gardening, it doesn’t need to always active. Work the soil, sow the seeds and then later let nature/time take its course.

Of course, you need to regularly water the plants, prune out the weeds and sometime, move the plants in shed, but most of the work is in the beginning and for the rest of the time inactivity matters. Staying away is better.

Was looking at this old post and this prompted me to analyze my activity in investing

Green is buying activity and orange is the selling.

Memory Profile Your code with Ipython

Lets say you have a function that you want check the memory usage in python. This can be evaluated with another IPython extension, the memory_profiler.

The memory profiler extension contains two useful magic functions: the %memit magic and the %mprun function.

%memit magic gives the peak and total memory used by a function, while %mprun provides a line by line usage of memory.

file: temp_interp.py

import numpy as np
from scipy.interpolate import interp1d
def test(n):
	a = np.random.rand(n,4000,30)
	x = np.arange(n)
	xx = np.linspace(0,n, 2*n)
	f= interp1d(x,a, axis=0, copy=False, fill_value="extrapolate", assume_sorted=True)
	b = f(xx)

To test this function with %mprun

from test_interp import test
%mprun -f test test(1000)

This shows a line-by-line description of memory use.

Before using, we need to load the extension:

%load_ext memory_profiler

To install the extension use the following

pip install memory_profiler

Creating Shortcuts in Windows in Batch Mode

Problem

A few months back, as part of an application deployment, needed to create shortcut in windows in batch mode

Solution

With google search and trials, this is the batch script that does that for me.

! create_shortcut.bat

@echo off
set WD=%1
set TARGET="%2"
set SHORTCUT="$home\Desktop\%3.lnk"
set PWS=powershell.exe -ExecutionPolicy Bypass -NoLogo -NonInteractive -NoProfile
%PWS% -Command "$ws = New-Object -ComObject WScript.Shell; $s = $ws.CreateShortcut(\"%SHORTCUT%\"); $S.TargetPath = \"%TARGET%\"; $S.WorkingDirectory = \"%WD%\"; $S.Save()"

How to call this:

create_shortcut.bat d:\apps http://www.google.com google

This will create shortcut to google on users desktop

• First argument is start-in folder name
• Second argument is the app or website the shortcut is pointing to.
• Third argument is the name of the shortcut,

Stay Safe and Stay Away

This is a repost of the blog post from the stevepavlina blog.

This is not remotely a “business as usual” situation. Please open your eyes to what’s happening. But more importantly, please open your heart to what’s about to happen.

This isn’t a time to be working on your launches. This isn’t a time to work on your campaigns. The focus on business needs to be put on hold right now.

We all have something much more important to work on right now. We have the leverage to greatly reduce a tremendous amount of suffering and loss of life that’s about to unfold. That this will happen is unfortunately inevitable, but we can absolutely have a meaningful impact in reducing it.

And if there are any marketing and influencing challenges that are worth the committed application of the very best of our skills, this is such a challenge.

The seriousness of our collective situation is going to intensify in the coming weeks – way more dramatically than it already has.

By the end of the month, many more hospitals across Europe and the USA will be where Northern Italy is – overwhelmed and with doctors and nurses dying from the disease too. Hospitals in the USA are already preparing for this by adding tents in parking lots. It’s a race against time, and we have much less time than most people realize. Seattle is already showing us a glimpse of what will soon hit many other cities.

This won’t simply blow over in a couple of weeks. We’re talking months, including the inevitability of 2+ months of full lockdown, which will almost certainly begin by the end of this month.

The best thing you can do now is share the critical importance of social distancing to reduce the spread of the disease, which has a long, silent incubation period. If you do that, you can save a lot of lives.

What if your launches and products are beneficial to people? Of course they are. But consider the opportunity cost of what you could be doing right now instead if you shift your focus into life-saving mode.

I can certainly claim that my work benefits people too, and it surely does, but I think I can be of greater service right now if I respond to the tremendous opportunity of taking direct action to help lower the disease progression in whatever way I can.

One way where I may be able to exert some leverage is to invite more people to join me in helping to get the word out about just how critical it is for all of us to shift gears right now. I’ve already been doing this behind the scenes for days, starting around February 26th. And that’s having some impact. But we still need to do more.

Every day that you work on your usual business instead of promoting the critical importance of social distancing is another day you could have saved lives and reduced suffering. Every day you delay is a missed opportunity for you to truly step up, not merely with personal ambition but with caring, compassion, and courage as well.

This is an extraordinary time for you to take your best expression of ambition and get it aligned with your best expression of caring. This is an invitation for us to collectively change the world in a way that really matters to a lot of people, especially those who will be hit hardest by this disease in the weeks and months ahead.

Where else are you going to encounter such an extraordinary opportunity to do the most meaningful work of your life?

Don’t worry about perfection. Speed of messaging is way more important than elegance right now. Just get the word out as fast as you can. Even if all you do is tell people to stay the fuck homeor share the hashtag #staythefuckhome, that will help. Just start with one Tweet. Please.

Don’t worry about being judged or criticized for being an alarmist. By the end of the month, it will be obvious that you did the right thing. You’ll probably wish you’d acted sooner and more powerfully. I already feel that way now.

Please put normal business on hold. Please use your marketing and influencing skills to save lives right now. Every day counts.

It doesn’t matter how big your reach is. Even if you’re just starting a new YouTube channel and only have a few viewers, please get the word out. That’s just as honorable a deed as for those who may reach millions.

Once we’re on full lockdown and people are no longer still going out risking other people’s lives, then maybe it will be an okay time to think about business again. But there’s a good chance your priorities will shift too as the body count continues to rise. Will you still care about business in the same way when people you know start having breathing difficulties, and they’re scared to go to an overcrowded nearby hospital with an exhausted staff and no more beds or ventilators available?

It hurts my heart to see marketer friends brainstorming and pondering the money-making opportunities that may come from everyone being stuck at home, salivating over that captive audience. I can understand that mindset, but I can’t pretend it’s okay. I realize that I must personally do something about that… and encourage others to help us change course as well.

At this time it’s just not okay to be blogging and emailing and tweeting and advertising as if we’re still in business as usual. It’s downright wicked to do such things right now because you’re encouraging others to engage in denial about what’s happening. We need to do the opposite of that. We absolutely must be sounding the alarm to wake more people up to what’s happening. We must do our best to turn the ship of ignorance and denial towards saving lives and reducing suffering.

Years from now when you look back on these days, do you really think you’ll reflect upon how brilliant you were for capturing those extra sales while people were getting sick and dying? Will you feel proud of yourself for your pandemic launches and marketing campaigns?

If you promote stuff that comforts people, that’s great – but save it for later once we’re all under lockdown. Our priorities right now should be to get people to stop silently spreading the disease and to push towards full lockdown to the extent that we can influence that too.

It’s your business and your decision of course. Just please take a peak into your heart and see what it wants to do about this situation. If your current business activities still feel very aligned and if what I’m sharing here doesn’t change anything for you, that’s fine. You have nothing to defend.

While I normally feel very aligned with my business and it’s very flexible anyway, I’m stretching it to go where it needs to go right now in order to push it to be of more service than usual. I’m not willing to be haunted for the rest of my life by the knowledge that I could have tried harder to save lives and reduce suffering – and didn’t. Are you willing to go that route?

This is one of the most extreme times you’ll ever live through. Who do you want to be in this situation? A decade from now, how will you wish you’d responded?

For Everyone

If you see anyone doing “business as usual” posts, videos, promotions, launches, and ads that don’t seem to be helping with the current situation, please actively encourage – no, implore – them to stop immediately and shift gears to save lives and reduce suffering.

Immediately reply to their emails as soon as you receive them. Comment on their posts, videos, and ads right away. Call them out on their Facebook Lives and webinars. Tell them to please focus on saving lives instead of trying to do anything less important right now. Be gentle but please be firm too. And be brave.

Please don’t shame anyone. Please invite them to step up to serve a greater need.

Realize that if you can influence even one more marketer or influencer to help spread the word about practicing social distancing just one day earlier than they otherwise would, you’re saving lives too. You do that, and you’re my hero.

Here are some further resources to help, both for your own education and for further re-sharing:

• Coronavirus stats – This page shows frequently updated virus stats for all countries, so you can see at a glance what’s happening. Also follow some of the links for deeper stats and data.
• Stay the Fuck Home – This is a good page for referring people. It shows simple actions to be taken immediately and handle multiple translations for different languages.
• Coronavirus: Why You Must Act Now – Tomas Pueyo’s powerfully persuasive piece with lots of math-based reasoning, charts, and graphs (more than 28M views in the past week). I’ve been sharing this one a lot and feel grateful for Tomas’ efforts.
• Jason Warner’s Facebook post – A solid math-based post to help persuade people of the importance of changing behaviors immediately. Please reshare this on Facebook. I did, as did 119K other people so far.
• Messages from Italy – This is a short YouTube video of Italians under lockdown sharing what they wished they’d known 10 days earlier.

Please care

Command Line Parameters for Your Batch Script

When using python as a driver program to lot of other apps, usage of batch files in windows system is something everyone encounters

And when it comes to batch files, sending command line arguments to batch file is one thing i tend to google a lot. A command line argument (or parameter) is any value passed into a batch script. So for convenience, I am listing the most common usages that I needed last year

%* for all command line parameters (excluding the script name itself).

%0 - the command used to call the batch file (could be foo, ..\foo, c:\bats\foo.bat, etc.)
%1 is the first command line parameter,
%2 is the second command line parameter,
and so on till %9 (and SHIFT can be used for those after the 9th).

%~nx0 - the actual name of the batch file, regardless of calling method (some-batch.bat)
%~dp0 - drive and path to the script (d:\scripts)
%~dpnx0 - is the fully qualified path name of the script (d:\scripts\some-batch.bat)

More info examples at https://www.ss64.com/nt/syntax-args.html

Computer Name in Python 3 ways

Problem:

Get the computer name in python without using any external module or library

Solution:

import platform
import socket
import os

# first 
platform.node()
# second
socket.gethostname()
# third
os.environ['COMPUTERNAME']

Find Disk Usage with shutil

As I move to python as my primary programming language, I am discovering new things about the modules that i have used for many years, one such module is the shutil.

To check free disk space, one can use the following command

  _, _, free = shutil.disk_usage("")

shutil.disk_usage returns the disk usage statistics about a given path

Returned values is a named tuple with attributes ‘total’, ‘used’, ‘free’, which are the amount of total, used and free space in bytes.

How to set persistent environment variables with command line in windows

Environment variables are not often seen directly when using Windows. However there are cases, especially when using the command line, that setting and updating environment variables is a necessity.

I always new the set command line command to set environment variables in windows, but to make a system wide effect of the environment variable, this command needs users to logout and login again.

To set persistent environment variables at the command line, we can use setx.exe. It became part of Windows as of Vista/Windows Server 2008.

setx.exe does not set the environment variable in the current command prompt, but it will be available in subsequent command prompts.

for more info type the following in the windows command

set /?

or

setx /?

Shelve it with python

One of the little gems hidden in python standard library is shelve.

The shelve module can be used as a simple persistent storage option for Python objects when a relational database is overkill. The shelf is accessed by keys, just as with a dictionary. The values are pickled and written to a database created and managed by dbm.

import shelve

with shelve.open('test_shelf.db') as s:
    s['key1'] = {
        'int': 310,
        'float': 3.14.5,
        'string': 'Sample string data',
	'array': [[1,2,3],[4,5,6]],
    }

I mostly work with large simulation data and run simulation from python and these simulations take time to run sometimes days, so a simple persistent storage option provided by shelve is an intuitive way to restore my work.

An advantage of is we do not have to remember the order in which the objects are pickled, since shelve gives a dictionary-like object.

Here’s a sample code I use to store my long running results and latter to restore those values at a later time.

filename=r"flake_results.out"

my_shelf = shelve.open(filename, "n")

for key in ["stress", "strain", "plas", "creep","temp"]:
	try:	
		my_shelf[key]=globals()[key]
	except Exception:
		print ("Error shelving: {}".format(key))

my_shelf.close()

To restore

my_shelf = shelve.open(filename)
for key in my_shelf:
    globals()[key]=my_shelf[key]

my_shelf.close()

for more info: Visit This

Kill Tasks in Windows

Linux, Mac users have it easy. Top and kill are two commands that can help one take control of the system.

For windows user, until recently I was stuck with the task manager. Like all manager this one demands too much attention and is not batch able.

Summoned Google Gennie and discovered.

Tasklist and Taskkill commands in CMD.

Neat. Where were these commands hiding?

Here is two gifs for how they work