Remember you are a black swan

I am sometimes taken aback by how people can have a miserable day or get angry because they feel cheated by a bad meal, cold coffee, a social rebuff or a rude reception. We are quick to forget that just being alive is an extraordinary piece of good luck, a remote event, a chance of occurrence of monstrous proportions. Imagine a speck of dust next to a planet a billion times the size of earth. The speck of dust represents the odds in favor of your being born; the huge planet would be the odds against it. So stop sweating the small stuff. Don’t be like the ingrate who got a castle as a present and worried about the mildew in the bathroom. Stop looking at the gift horse in the mouth – remember you are a Black Swan. [Source: Taleb; The Black Swan]

Zigs and Zags

The markets are all time high, despite the situation on the ground. Take any stock portfolio and you will see high returns.

I receive at least one call from my acquaintance on investing in stocks. When the markets are booming like, everyone wants to join in. No one wants to be left behind.

I am not immune to these behaviors. I am looking at my portfolio more often than I should. Have read and clicked on news articles of stock more than I did last year.

I like revisiting the books on investing to stay grounded.

Legendary investor Warren Buffett has this aphorism: My favorite holding period is forever. If we think this way, never planning to sell, we force ourselves to value stocks based on the cash they generate, instead of being distracted by guesses about future prices.

A Buffett variation on this theme is, I never attempt to make money on the stock market. I buy on the assumption that they could close the market the next day and not reopen it for five years. If we think this way, we stop speculating about zigs and zags in stock prices and focus on the cash generated by the money machine.

The idea is simple and powerful, but often elusive. It is very hard to buy a stock without looking at what its price has been in the past and thinking about what its price might be in the future. It is very hard to think about waiting patiently for cash to accumulate when it is so tempting to think about making a quick killing by flipping stocks.

From the book Money Machine by Gary Smith

Strip Doxygen Comments

A few years back a teammate wrote a vba based productivity tool to automate writing of doxygen comments in large legacy fortran code.

The tool worked perfectly except that only worked on a clean no doxygen commented fortran code.

Everytime new code was added, it meant we couldn’t use the tool, so wrote the following batch script that I used to strip all the code of all doxygen comments.

Strip doxygen commands and replace the file

@echo off
REM Deletes Doxygen comments from fortran file
REM Written by Sukhbinder
REM 5 AUG 2015
for /f %%i IN ('dir /b *.F') DO find /V /I "c>" %%i > new_%%i & del %%i & rename new_%%i %%i

Starter code for Using Tensorboard

I have been using keras for multiple years now at home on my personal projects, one things that I have used very little is the tensorboard, mostly because I run the models on my home system and the console output more than satisfies my needs.

But now I am planning to use it regularly. So here’s the starter code that I always start with.

Its dead easy to add tensorboard with keras.callbacks.

  • Initialize TensorBoard object
  • attach it to the model fit

Thats it.

Here’s a quick code.

from keras.callbacks import TensorBoard

tensorboard = TensorBoard(log_dir='./logs', histogram_freq=0,
                          write_graph=True, write_images=False)
# define model, Y_train,
          validation_data=(X_test, Y_test),

Be sure to check other callbacks in keras. They are rich in functionality!

Pandoc – a Swiss Army Knife

Most of the documentation in office is in Microsoft docs but they are not conducive to version control. So there is a drive to have them in RST format which is easier to version control and gives the same or similar flexibility.

Tool that does the job is called pandoc

If you need to convert files from one markup format into another, pandoc is your swiss-army knife. Pandoc can convert between mandy different formats.

A simple batch script to convert all .docx files within a folder is below.

Note that first you need to delete cropped areas of pictures from the document (select any one picture and go to Format/Compress Pictures).

Docx to markdown

pandoc.exe -f docx doc_file.docx -t markdown -o --extract-media=. --wrap=none

Docx to RST

pandoc.exe -f docx doc_file.docx -t rst -o proposal.rst --extract-media=. --wrap=none

Kudo’s to the team for such a wonderful tool. Well designed and versatile.

Hope to explore it more in the near future.

Few ipython %run magics I can’t live without

If you are not using ipython I strongly encourage you to stop reading and get ipython.

From the time I have discovered it, 8 years back, i haven’t looked back. This is my default python console. Even jupyter hasn’t replaced it.

Why? you ask.

IPython offers a combination of convenient shell features, special commands and a history mechanism for both input (command history) and output (results caching, similar to Mathematica). It is intended to be a fully compatible replacement for the standard Python interpreter, while offering vastly improved functionality and flexibility.

And here’s a list of ipython’s %run magics options that I regularly use.

%run -t   some_cool_function()  # To time a code
%run -d some_cool_function() # for debuging and stepping if there is an error
%run -i some_cool_function()  # for interactive execution

Here’s the explanation from the docs

__name__ is NOT set to ‘__main__’, but to the running file’s name without extension (as python does under import). This allows running scripts and reloading the definitions in them without calling code protected by an if __name__ == "__main__" clause.
run the file in IPython’s namespace instead of an empty one. This is useful if you are experimenting with code written in a text editor which depends on variables defined interactively.
print timing information at the end of the run. IPython will give you an estimated CPU time consumption for your script, which under Unix uses the resource module to avoid the wraparound problems of time.clock(). Under Unix, an estimate of time spent on system tasks is also given (for Windows platforms this is reported as 0.0).
run your program under the control of pdb, the Python debugger. This allows you to execute your program step by step, watch variables, etc. Internally, what IPython does is similar to calling:
run program under the control of the Python profiler module (which prints a detailed report of execution times, function calls, etc).

Do you have any favorite magic in ipython?

Converting xls/xml Files to Proper Excel file

My office project management system is an a java app and the for some reasons, the excel export from it always comes out as excel-xml-xls output.

Whenever this file is opened in excel, it throws a warning about conversion. This is annoyng but manageble, but if you want to automate reporting from this files, its annnoying, pandas doesn’t read it directly, so user is stuck.

We have raised a bug on this but it has still not resolve, but then I found this vb script that does the conversion and Ii have never looked back.

Posting it here in case others need it.

if WScript.Arguments.Count < 3 Then
    WScript.Echo "Please specify the source and the destination files. Usage: ExcelToCsv <xls/xlsx source file> <csv destination file> <worksheet number (starts at 1)>"
End If

csv_format = 51

Set objFSO = CreateObject("Scripting.FileSystemObject")

src_file = objFSO.GetAbsolutePathName(Wscript.Arguments.Item(0))
dest_file = objFSO.GetAbsolutePathName(WScript.Arguments.Item(1))
worksheet_number = CInt(WScript.Arguments.Item(2))

Dim oExcel
Set oExcel = CreateObject("Excel.Application")

Dim oBook
Set oBook = oExcel.Workbooks.Open(src_file)

oBook.SaveAs dest_file, csv_format

oBook.Close False

Via this post.

Simple script but a god send for me. now my reporting of numbers is just a click always, as this enables me to write a python workflow which works like a charm.

Bias of Thinking in Straight Lines

Photo by Miguel u00c1. Padriu00f1u00e1n on

My favorite text book in college was the introduction to flight by John D Anderson. Peppered with historical notes and each chapter starting like a novel, I loved that book then and do so now.

Last week, I stumbled upon this note saved on my phone from the book E=mc2: A Biography of the World’s Most Famous Equation by David Bodanis.

Re-reading this reminded me of my younger self at college reading the flights book.

When a subject reaches an advanced level, a lack of education usually makes it impossible for outsiders to get started. The doors are closed, the papers unreadable. But in these early days of understanding energy, it was a different story. Most science students had been trained to show that any complicated motion could be broken down into a mix of pushes and pulls that worked in straight lines. It was natural for them, accordingly, to try to see if there were any straight-line pulls between magnets and electricity. But this approach didn’t show how the power of electricity might tunnel through space to affect magnetism.

Because Faraday did not have that bias of thinking in straight lines, he could turn to the Bible for inspiration. The Sandemanian religious group he belonged to believe in a different geometric pattern: the circle. Humans are holy, they said, and we all owe an obligation to one another based on our holy nature. I will help you, and you will help the next person, and that person will help another, and so on until the circle is complete. This circle wasn’t merely an abstract concept. Faraday had spent much of his free time for years either at the church talking about this circular relation, or engaged in charity and mutual helping to carry it out.

He got to work studying the relationship between electricity and magnetism in the late summer of 1821. From his religious background, he imagined a whirling tornado of invisible circular lines swirling around it. If he were right, then a loosely dangling wire could be tugged along, caught in those mystical circles like a small boat getting caught up in a whirlpool. He connected the battery. And immediately he had the discovery of the century.

What Faraday had invented, in his basement laboratory, was the basis of the electric engine. A single dangled wire, whirling around and around, doesn’t seem like much. But Faraday had only a small magnet, and was feeding in very little power. Rev it up, and that whirling wire will still doggedly follow the circular patterns he had mapped out in seemingly empty air. Ultimately one could attach heavy objects to a similar wire, and they would be tugged along as well—that’s how an electric engine works. It doesn’t matter whether it is the featherweight spinning plate of a computer drive that’s being dragged along, or the pumps that pour tons of fuel into a jet engine.

Working with PDF in python

I have a love hate relationship with the pdf format. I like the convenience of the portability but its hard to work with you if you need the data.

And because of the portability, most of data coming my ways mostly is in pdf format, so i have come to rely on the an excellent library that I use to work with pdf’s in python pyPDF2

PyPDF2 is a pure-python PDF library capable of splitting, merging together, cropping, and transforming the pages of PDF files. It can also add custom data, viewing options, and passwords to PDF files. It can retrieve text and metadata from PDFs as well as merge entire files together.

Sample code to demonstrate simple usage of the library

import PyPDF2

pdf_file = open('today.pdf', 'rb')
read_pdf = PyPDF2.PdfFileReader(pdf_file)

# Get number of pages
number_of_pages = read_pdf.getNumPages()

# Read a page content
page = read_pdf.getPage(0)
page_content = page.extractText()
print (page_content.encode('utf-8'))

Single Definition of Success

Photo by Tim Gouw on

For I have a single definition of success: you look in the mirror every evening, and wonder if you disappoint the person you were at 18, right before the age when people start getting corrupted by life. Let him or her be the only judge; not your reputation, not your wealth, not your standing in the community, not the decorations on your lapel. If you do not feel ashamed, you are successful. All other definitions of success are modern constructions; fragile modern constructions.

source : Nassim Taleb commencement address

Installing espeak/python in chromebook

A friend of my had a minor surgery last month. Their 8 year old daughter was staying with us for the time being.

She has brought her google chromebook and after seeing my kids work on spelling app she asked me if she can have the same for her on her chromebook.

“Sure, why not”.

So here’s the rough process followed I followed to install python, espeak and the spelling app on Linux in chromeos

  1. Install Linux virtual machine. Which was easy. Just clicked few button and the virtual machine was ready.
  2. sudo apt-get update
  3. sudo apt-get python3-pip
  4. sudo apt espeak asaf-utils ffmpeg
  5. sudo pip3 install pyttsx3

It was a good experience to work on this. Previous to this i have not seen chomeos. I liked the way it worked out. definitely recommend chromos.

Slow and Gradual

Photo by Nita on

Put a frog in a pot of boiling water, it will immediately jump out. But if we place the frog in a tepid water and don’t scare it, it will continue to stay. If we heat the pot and gradually increase the temperature, the frog will do nothing. As the heat gradually increases, it will continue to stay put. Eventually it will boil to its death?

Why?  Because its internal apparatus for sensing threats to survival is geared to sudden changes in his environment, not to slow, gradual ones. 

This fable illustrates how, like a frog, we humans are geared to not recognise, slow and gradual changes.

This is why we don’t see our belly expand, see how gradually kids are growing. We need contrast and slow and gradual changes stumps us.

Slow changes, both in improvements and in deteriorations, get magnified over time. But over short periods of time, they are barely noticeable.

As 2020 is entering its last leg, let us look around and notice the slow and gradual changes unfolding.

Build Wisely

Photo by Pixabay on

Last month my daughter had her mid-term exams. Following story was part of her exam. I liked it hence posting it here as a reminder.

An elderly carpenter was ready to retire. He told his employer of his plans to leave the house building business and live a more leisurely life with his family. He would miss the pay cheque, but he needed to retire. The employer was sorry to see his good worker go and asked if he could build just one more house as a personal favour.

The carpenter said ‘ yes ‘,but in time it was easy to see that his heart was not in his work. He resorted to shoddy workmanship and used inferior material. It was an unfortunate way to end his career.

When the carpenter finished his work, his employer handed over the key of the house to the carpenter—“This is your house, my gift to you.”

What a shock! What a shame! If he had only known he was building his own house, he would have done it so differently! Now he had to live in a poorly built house.

So it is with us. We build our lives, a day at a time, often putting less than our best into the building. Then, with a shock, we realize we have to live in the house we have built. If we could do it over, we would do it much differently. But, you cannot go back.

You are the carpenter, and every day you hammer a nail, place a board, or erect a wall. Someone once said, your attitude, and the choices you make today, help build the “house” you will live in tomorrow. Therefore, build wisely!

Record Screen With Python

There a tons of screen recording software available in the market, but you don’t want to install one more software for just one usage. Or sometimes installing a freeware or a paid ware is not allowed.

What to do? you improvise. If you know and have python on your system, you are in luck.

This is what I did, when I had few occasions where I wanted to show my work. A quick gif or a small movie goes a long way, so created this app using python to records shot clips .

Here’s a demo of the same using the tool.

The entire code and tool is available at my gihub here