Panda Time index vs DateTime and Inconveniences

Sometimes convenience becomes a handicap. Saw this first hand later last month.

Was so used to using Pandas DataFrame and the Timeindex object that when I had to move back to a system which didn’t have pandas I was struggling to get a simplified day of year, day of week and week of year from python’s standard datetime module.

Here is how all this available in Pandas from a Timeindex column or index.

If your looks like this, with index as time

You can get all the convenience functions like this

However, this gave an opportunity to explore datetime and here is the code to get all this and more from date time.

import datetime
today=datetime.datetime.today()
day_of_year = today.timetuple().tm_yday
weekofyear=today.isocalendar()[1]
dayofweek=today.isocalendar()[2]

Inconveniences are good; they always end up teaching you something.

Advertisements

The Best Debugging Tool of All

Rob Pike (one of the authors of the Go language) has this to say on debugging…..:

A year or two after I’d joined the Labs, I was pair programming with Ken Thompson on an on-the-fly compiler for a little interactive graphics language designed by Gerard Holzmann. I was the faster typist, so I was at the keyboard and Ken was standing behind me as we programmed. We were working fast, and things broke, often visibly—it was a graphics language, after all. When something went wrong, I’d reflexively start to dig in to the problem, examining stack traces, sticking in print statements, invoking a debugger, and so on. But Ken would just stand and think, ignoring me and the code we’d just written. After a while I noticed a pattern: Ken would often understand the problem before I would, and would suddenly announce, “I know what’s wrong.” He was usually correct. I realized that Ken was building a mental model of the code and when something broke it was an error in the model. By thinking about *how* that problem could happen, he’d intuit where the model was wrong or where our code must not be satisfying the model.

Ken taught me that thinking before debugging is extremely important. If you dive into the bug, you tend to fix the local issue in the code, but if you think about the bug first, how the bug came to be, you often find and correct a higher-level problem in the code that will improve the design and prevent further bugs.

I recognize this is largely a matter of style. Some people insist on line-by-line tool-driven debugging for everything. But I now believe that thinkingwithout looking at the code—is the best debugging tool of all, because it leads to better software.

Updated Binary STL File Reader

bent_plate

A recent comment on the post titled binary stl file reader in numpy prompted me to revisit the code and while I was looking at it , I updated the same for code to view the loaded stl file.

The above picture shows the result.

The code as usual available on github from here

from binarySTLreader import BinarySTL,ShowSTLFile
h,p,n,v1,v2,v3=BinarySTL('bent_plate.stl')
ShowSTLFile(v1,v2,v3)

Switch from Discrete Mathematics to Probability, Statistics

From the book Foundations of Data Science by John Hopcroft and Ravindran Kannan

Computer science as an academic discipline began in the 60’s. Emphasis was on programming languages, compilers, operating systems, and the mathematical theory that supported these areas. Courses in theoretical computer science covered finite automata,regular expressions, context free languages, and computability.

In the 70’s, algorithms was added as an important component of theory. The emphasis was on making computers useful. Today, a fundamental change is taking place and the focus is more on applications.

There are many reasons for this change. The merging of computing and communications has played an important role. The enhanced ability to observe, collect and store data in the natural sciences, in commerce, and in other fields calls for a change in our understanding of data and how to handle it in the modern setting. The emergence of the web and social networks, which are by far the largest such structures, presents both opportunities and challenges for theory

All this entails, there is the switch from discrete mathematics to more emphasis on probability and statistics.

Joint plot with Matplotlib

Today I am releasing a simple module to create joint plot with Matplotlib on github. Joint plot is available in the excellent seaborn library but unfortunately it’s not always available on many systems. Recently I needed this functionality, so wrote this simple module with matplotlib.

The functionality is almost similar to seaborn but with limited feature. This has helped me in my work, releasing it in the hope that others might find it useful.

Sample usage:

Import Jointplot
tips=pd.read_csv(r'tests/tips_p.csv')
data=np.c_[tips['total_bill'].values,tips['tip'].values]
jointPlot(data,kde=True)

Find the code at this github repository.

Installing UQTkV3.0 on MAC OS x 10

Version 3 of UQ Toolkit is out.

The UQ Toolkit (UQTk) is a collection of libraries and tools for the quantification of uncertainty in numerical model predictions. Just like version 2, version 3.0 offers intrusive and non-intrusive methods for propagating input uncertainties through computational models, tools for sensitivity analysis, methods for sparse surrogate construction, and Bayesian inference tools for inferring parameters from experimental data.

Installing UQTk version 2 on a Mac was easy. Use gcc or clang and fire up the make file, but for version 3, the team has now shifted to cmake and if you want python interface you also need SWIG.

Just like in life, additional dependencies bring additional complexity and chances of things going wrong. Well this was my experience while installing UQtk version 3 on my mac.

Here are my running notes while installing the library on OS X 10.

Cmake doesn’t come preinstalled on a Mac. So get it from here. Installation is easy.


I didn’t have swig so had to download swig from here

Installing is 4 steps

  • Go to the untarred directory,
  • ./configure
  • make
  • sudo make install

I forgot the 4th step and got the following error while installing uqtk

CMake Error at /Applications/CMake.app/Contents/share/cmake-3.6/Modules/FindPackageHandleStandardArgs.cmake:148(message):

Could NOT find SWIG (missing: SWIG_EXECUTABLE SWIG_DIR)

Call Stack (most recent call first): /Applications/CMake.app/Contents/share/cmake-3.6/Modules/

— Configuring incomplete, errors occurred!

See also “/Users/sukhbindersingh/PROJECTS/UQTk_v3.0/builddir/CMakeFiles/CMakeOutput.log”.

 

So make sure, you do all the four steps…

 

SWIG needs PCRE-Perl compatible Regular expression library, which was not available on my mac, so download that too from here

Download PCRE not PCRE2. I wasted time installing PCRE2 to find that swig doesn’t recognise it.

 

Make CMAKE work in command line

 

Now cmake opens up as GUI in Mac OS by defauly, but we need command line version, so we have following 3 options

One may add CMake to the PATH:

PATH=”/Applications/CMake.app/Contents/bin”:”$PATH”

Or, to install symlinks to ‘/usr/local/bin’, run:

sudo “/Applications/CMake.app/Contents/bin/cmake-gui” –install

Or, to install symlinks to another directory, run:

sudo “/Applications/CMake.app/Contents/bin/cmake-gui” –install=/path/to/bin

 

I opted for the second option, mostly because it was simplest and I want cmake to available everywhere and every time I need it

sudo “/Applications/CMake.app/Contents/bin/cmake-gui” –install

 

Where are your gcc, gfortran, and clang?

 

Once we have all these and before you start, we have to check few things. In this installation, I wanted to use gcc, gfortran, and clang for my setup, so get to know where they are located in your system by using the ‘which’ command

The Installations

 

In the terminal, go to the UQTk_v3.0 directory

  • mkdir builddir
  • cd builddir/
  • cmake -DCMAKE_Fortran_COMPILER=/usr/local/bin/gfortran -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/clang ../../UQTk_v3.0

 

Replace green text with your own path.

Once the cmake configuration is complete successfully, type make

All will go ok, until you hit this error.

In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:23:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/

MacOSX10.11.sdk/usr/include/unistd.h:510:14:error:declaration of ‘optarg’ has a different language linkage

extern char *optarg; /*

getopt(3) external variables */

^

/Users/sukhbindersingh/PROJECTS/UQTk_v3.0/cpp/lib/include/gen_defs.h:49:14: note:

previous declaration is here

extern char *optarg;

^

1 error generated.

make[2]: *** [cpp/lib/CMakeFiles/uqtk.dir/tools/multiindex.cpp.o] Error 1

 

Well this is easy to solve and I think it’s an error in the file. Open gen_defs.h file from cpp/lib/include and comment the following line as shown.

extern char *optarg

 

 

Another error that I faced was the missing gfortran library because of no LIBRARY_PATH.

ld: library not found for -lgfortran

clang: error: linker command failed with exit code 1 (use -v to see invocation)

make[2]: *** [cpp/app/gen_mi/gen_mi] Error 1

make[1]: *** [cpp/app/gen_mi/CMakeFiles/

gen_mi.dir/all] Error 2

make: *** [all] Error 2

 

This meant my libgfortran.a was not in the LIBRARY_PATH, so let’s find where the library is and then set the LIBRARY_PATH environment variable pointing it to that folder.

Locate libgfortran*.a

sudo find /usr -iname ‘libgfortran*.a’

 

Found them here.

/usr/local/gfortran/lib/i386/libgfortran.a

/usr/local/gfortran/lib/libgfortran.a

Set the environment variable to that folder.

export LIBRARY_PATH=/usr/local/gfortran/lib/

 

Compilation should continue smoothly and at last type make install and you should have everything installed.


Type ctest to check your installation


Recap

 

So here’s recap of the steps.

  1. mkdir builddir
  2. cd builddir
  3. cmake -DCMAKE_Fortran_COMPILER=/usr/local/bin/gfortran -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/clang ../../UQTk_v3.0 [Without python]

    or cmake -DPyUQTk=ON -P-DCMAKE_Fortran_COMPILER=/usr/local/bin/gfortran -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/clang ../../UQTk_v3.0 [With python]

  4. make
  5. make install

I faced two problems,

  1. error:declaration of ‘optarg’ has a different language linkage Solution: comment the offending line.
  2. ld: library not found for –lgfortran clang: error: linker command failed with exit code 1 Solution: export LIBRARY_PATH=/usr/local/gfortran/lib/

Using PyUQTk option gave me lot of problem, have delayed installing python binding for now due to lack of time.

Well now this is done, will spend couple of weekends looking at the library, you can fork the entire library from my github page. It has all the libs, so if the compliers are similar, they can be used as is without going through all this steps.

Get Mount Drive from mount path in windows

Things that are simple in Linux and UNIX based system always end up becoming a head ache in windows, but unfortunately most of our work-life runs on windows.

And if you don’t have admin rights to install specialized modules in your python installation, then here’s a hacked version of routine that should do the job.


def get_mount_drive(path):
    os.system(r'net use > t.txt')
    lines=open('t.txt').readlines()
    for line in lines:
    if path in line:
       return lines.split()[1]
    return None