How to resolve this pandas ValueError: arrays must all be same length

Consider the following code.

import numpy as np
import pandas as pd

in_dict = dict(a=np.random.rand(3), b=np.random.rand(6), c=np.random.rand(2))

df = pd.DataFrame.from_dict(in_dict)

This fails with the following error

df = pd.DataFrame.from_dict(in_dict)
ValueError                                Traceback (most recent call last)
<ipython-input-4-2c9e8bf1abe9> in <module>
----> 1 df = pd.DataFrame.from_dict(in_dict)

~\Anaconda3\lib\site-packages\pandas\core\ in from_dict(cls, data, orient, dtype, columns)
   1371             raise ValueError("only recognize index or columns for orient")
-> 1373         return cls(data, index=index, columns=columns, dtype=dtype)
   1375     def to_numpy(

~\Anaconda3\lib\site-packages\pandas\core\ in __init__(self, data, index, columns, dtype, copy)
    528         elif isinstance(data, dict):
--> 529             mgr = init_dict(data, index, columns, dtype=dtype)
    530         elif isinstance(data, ma.MaskedArray):
    531             import as mrecords

~\Anaconda3\lib\site-packages\pandas\core\internals\ in init_dict(data, index, columns, dtype)
    285             arr if not is_datetime64tz_dtype(arr) else arr.copy() for arr in arrays
    286         ]
--> 287     return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)

~\Anaconda3\lib\site-packages\pandas\core\internals\ in arrays_to_mgr(arrays, arr_names, index, columns, dtype, verify_integrity)
     78         # figure out the index, if necessary
     79         if index is None:
---> 80             index = extract_index(arrays)
     81         else:
     82             index = ensure_index(index)

~\Anaconda3\lib\site-packages\pandas\core\internals\ in extract_index(data)
    399             lengths = list(set(raw_lengths))
    400             if len(lengths) > 1:
--> 401                 raise ValueError("arrays must all be same length")
    403             if have_dicts:

ValueError: arrays must all be same length

The solution is simple. I have faced this situation a lot, so posting this here on the blog for easy reference

use orient=’index’

df = pd.DataFrame.from_dict(in_dict, orient='index')


          0         1         2         3         4         5
a  0.409699  0.098402  0.399315       NaN       NaN       NaN
b  0.879116  0.460574  0.971645  0.147398  0.939485  0.222164
c  0.747605  0.123114       NaN       NaN       NaN       NaN


          a         b         c
0  0.409699  0.879116  0.747605
1  0.098402  0.460574  0.123114
2  0.399315  0.971645       NaN
3       NaN  0.147398       NaN
4       NaN  0.939485       NaN
5       NaN  0.222164       NaN

Some related posts you might like


Error While migrating a Django Project

Here’s an error I encountered while migrating a Django project from my PC to Mac. Nothing related to the difference in architecture but as I later learned, the solution was quite simple and comes out of the box in Django.


OperationalError at /admin/exp/expense/add/
no such table: exp_expense
Request Method:	POST
Request URL:
Django Version:	2.2.16
Exception Type:	OperationalError
Exception Value:	
no such table: exp_expense
Exception Location:	D:\apps\anaconda3\lib\site-packages\django\db\backends\sqlite3\ in execute, line 383
Python Executable:	D:\apps\anaconda3\python.exe
Python Version:	3.8.3
Python Path:	


(sdh_env) (base) C:\Users\Sukhbinder\Desktop\PROJECTS\exptest\test_exp>python migrate --run-syncdb
Operations to perform:
Synchronize unmigrated apps: exp, messages, rest_framework, staticfiles
Apply all migrations: admin, auth, contenttypes, sessions
Synchronizing apps without migrations:
Creating tables…
Creating table exp_expense
Running deferred SQL…
Running migrations:
No migrations to apply.

That’s it. Addition of –run_syncdb in the migrate command.

python migrate --run_syncdb

Have you faced anything similar?

HDF5 lib from Visual Studio

During October last year, my work involved using HDF library with FORTRAN and C and use them to store mounds of data other systems were generating.

The first step was to use the library with visual studio 2008 in windows. Companywide HDF5 had many users in Linux cluster but no formal implementation was available to use in visual studio. While that made the library available but building, a program using them in visual studio was a different matter. I faced couple of issues and with much googling and stack overflowing, solved them, here is a running note I took for one of the issue, just in case someone else is in the same situation.

Hope this helps.

1>libhdf5.lib(H5I.c.obj) : error LNK2001: unresolved external symbol _forceCRTManifestCUR

1>libhdf5.lib(H5.c.obj) : error LNK2001: unresolved external symbol _forceCRTManifestCUR

1>libhdf5.lib(H5D.c.obj) : error LNK2001: unresolved external symbol _forceCRTManifestCUR

1>libhdf5.lib(H5F.c.obj) : error LNK2001: unresolved external symbol _forceCRTManifestCUR

1>libhdf5.lib(H5S.c.obj) : error LNK2001: unresolved external symbol _forceCRTManifestCUR


  • The various header files associated with the visual C++ runtime embed a number of directives into the compiled code. This power used to be used for good: appropriate #pragma’s could ensure that the resulting .lib file automatically had a dependency on the correct runtime libraries.

    However, a kind of dependency ____ comes about when one tries to mix projects built with different build settings in the same version of dev studio, and becomes even worse when pre-built libs made by another version of Dev Studio are used.

    The best thing to do, frankly, would be to rebuild your .libs in the same version of Dev Studio. There are some project settings that can be used when building a .lib that can ‘sanitize it’ a bit and make it’s use more compatible with different versions of dev studio – libraries should generally be built with the multi threaded runtime selected (NOT the DLL runtime) and the option “Omit Default Library Names” selected.

    In this case, __forceCRTManifestCUR, is a result of a #pragma comment(linker, “/INCLUDE=__forceCRTManifestCUR”) directive in one of the c-runtime header files.

    You can work around this by simply including a like like this in your main.cpp file:

    int __forceCRTManifestCUR=0;

    Doing this will “defeat” an attempt by the headers to get a manifest dependency to the “current” version of the CRT dlls embedded – but don’t worry – the correct CRT manifest is already specified correctly using a different mechanism, so you can generally quite safely define this symbol (by declaring it as an int or anything really) without causing any problems for the project.

The Best Debugging Tool of All

Rob Pike (one of the authors of the Go language) has this to say on debugging…..:

A year or two after I’d joined the Labs, I was pair programming with Ken Thompson on an on-the-fly compiler for a little interactive graphics language designed by Gerard Holzmann. I was the faster typist, so I was at the keyboard and Ken was standing behind me as we programmed. We were working fast, and things broke, often visibly—it was a graphics language, after all. When something went wrong, I’d reflexively start to dig in to the problem, examining stack traces, sticking in print statements, invoking a debugger, and so on. But Ken would just stand and think, ignoring me and the code we’d just written. After a while I noticed a pattern: Ken would often understand the problem before I would, and would suddenly announce, “I know what’s wrong.” He was usually correct. I realized that Ken was building a mental model of the code and when something broke it was an error in the model. By thinking about *how* that problem could happen, he’d intuit where the model was wrong or where our code must not be satisfying the model.

Ken taught me that thinking before debugging is extremely important. If you dive into the bug, you tend to fix the local issue in the code, but if you think about the bug first, how the bug came to be, you often find and correct a higher-level problem in the code that will improve the design and prevent further bugs.

I recognize this is largely a matter of style. Some people insist on line-by-line tool-driven debugging for everything. But I now believe that thinkingwithout looking at the code—is the best debugging tool of all, because it leads to better software.

Error spawning ‘cmd.exe’ and how to solve it


Yes I know the world have moved beyond visual studio 2013, but at office we are stuck with Visual studio 8 and after working for some time, it spits the following error.


Error 4 Error spawning ‘cmd.exe’.

Then, I have to close the visual studio and restart again. This works. Although not ideal, I was happy doing this as writing the method was more important than tinkering with visual studio.

Last week I had enough and decided to google and get this sorted.. So here’s the solution from ever useful stack overflow that worked like a charm for me.


Just open Tools -> Options -> Projects and Solutions -> VC++ Directories

and add these lines :


Now back to work!!!

Solving ‘ld: library not found’ issues after upgrading to OS X El Capitain


Upgrading to a new OS is always exciting, but if one uses a system for development, then its not always smooth.

Here’s one issue i recently faced after upgrading to OS X El Capitain.

I was recently converting a fortran program to python on macbook using the f2py and got the following error.

ld: library not found for -lbundle1.o

The code complied but the bundle library was missing during linking , this code worked last time I did it, so the suspect was the new OS X El Capitain.

More specifically the suspect was the Xcode library, so I reinstalled the developer tools using the command:

xcode-select –install

Took 15 minutes but after those minutes, faced a new problem, now gcc_s.10.5 library was missing..  

ld: library not found for -lgcc_s.10.5

This was more tricky than the first problem but quickly found this fix from here.

cd /usr/local/lib

sudo ln -s ../../lib/libSystem.B.dylib libgcc_s.10.5.dylib

These two problems solved. Now back to work. 

GDB commands by function

In office, doing a lot of linux code debugging. Missing the luxury of visual studio. Fed up with having multiple print statements all over the code, have switched to gdb and this excellent text file is the best ready reference I found on gdb commands.

Great resource if you are stuck with debugging in linux. [yes, yes, can’t install anything on the company system, if a big process all together!!!]

from: by James R. Binkley

GDB commands by function - simple guide
More important commands have a (*) by them.

% gdb -help         	print startup help, show switches
*% gdb object      	normal debug 
*% gdb object core 	core debug (must specify core file)
%% gdb object pid  	attach to running process
% gdb        		use file command to load object 

*(gdb) help        	list command classes
(gdb) help running      list commands in one command class
(gdb) help run        	bottom-level help for a command "run" 
(gdb) help info         list info commands (running program state)
(gdb) help info line    help for a particular info command
(gdb) help show         list show commands (gdb state)
(gdb) help show commands        specific help for a show command

*(gdb) break main       set a breakpoint on a function
*(gdb) break 101        set a breakpoint on a line number
*(gdb) break basic.c:101        set breakpoint at file and line (or function)
*(gdb) info breakpoints        show breakpoints
*(gdb) delete 1         delete a breakpoint by number
(gdb) delete        	delete all breakpoints (prompted)
(gdb) clear             delete breakpoints at current line
(gdb) clear function    delete breakpoints at function
(gdb) clear line        delete breakpoints at line
(gdb) disable 2         turn a breakpoint off, but don't remove it
(gdb) enable 2          turn disabled breakpoint back on
(gdb) tbreak function|line        set a temporary breakpoint
(gdb) commands break-no ... end   set gdb commands with breakpoint
(gdb) ignore break-no count       ignore bpt N-1 times before activation
(gdb) condition break-no expression         break only if condition is true
(gdb) condition 2 i == 20         example: break on breakpoint 2 if i equals 20
(gdb) watch expression        set software watchpoint on variable
(gdb) info watchpoints        show current watchpoints

Running the program
*(gdb) run        	run the program with current arguments
*(gdb) run args redirection        run with args and redirection
(gdb) set args args...        set arguments for run 
(gdb) show args        show current arguments to run
*(gdb) cont            continue the program
*(gdb) step            single step the program; step into functions
(gdb) step count       singlestep \fIcount\fR times
*(gdb) next            step but step over functions 
(gdb) next count       next \fIcount\fR times
*(gdb) CTRL-C          actually SIGINT, stop execution of current program 
*(gdb) attach process-id        attach to running program
*(gdb) detach        detach from running program
*(gdb) finish        finish current function's execution
(gdb) kill           kill current executing program 

Stack backtrace
*(gdb) bt        	print stack backtrace
(gdb) frame        	show current execution position
(gdb) up        	move up stack trace  (towards main)
(gdb) down        	move down stack trace (away from main)
*(gdb) info locals      print automatic variables in frame
(gdb) info args         print function parameters 

Browsing source
*(gdb) list 101        	list 10 lines around line 101
*(gdb) list 1,10        list lines 1 to 10
*(gdb) list main  	list lines around function 
*(gdb) list basic.c:main        list from another file basic.c
*(gdb) list -        	list previous 10 lines
(gdb) list *0x22e4      list source at address
(gdb) cd dir        	change current directory to \fIdir\fR
(gdb) pwd          	print working directory
(gdb) search regexpr    forward current for regular expression
(gdb) reverse-search regexpr        backward search for regular expression
(gdb) dir dirname       add directory to source path
(gdb) dir        	reset source path to nothing
(gdb) show directories        show source path

Browsing Data
*(gdb) print expression        print expression, added to value history
*(gdb) print/x expressionR        print in hex
(gdb) print array[i]@count        artificial array - print array range
(gdb) print $        	print last value
(gdb) print *$->next    print thru list
(gdb) print $1        	print value 1 from value history
(gdb) print ::gx        force scope to be global
(gdb) print 'basic.c'::gx        global scope in named file (>=4.6)
(gdb) print/x &main     print address of function
(gdb) x/countFormatSize address        low-level examine command
(gdb) x/x &gx        	print gx in hex
(gdb) x/4wx &main       print 4 longs at start of \fImain\fR in hex
(gdb) x/gf &gd1         print double
(gdb) help x        	show formats for x
*(gdb) info locals      print local automatics only
(gdb) info functions regexp         print function names
(gdb) info variables  regexp        print global variable names
*(gdb) ptype name        print type definition
(gdb) whatis expression       print type of expression
*(gdb) set variable = expression        assign value
(gdb) display expression        display expression result at stop
(gdb) undisplay        delete displays
(gdb) info display     show displays
(gdb) show values      print value history (>= gdb 4.0)
(gdb) info history     print value history (gdb 3.5)

Object File manipulation
(gdb) file object      		load new file for debug (sym+exec)
(gdb) file             		discard sym+exec file info
(gdb) symbol-file object        load only symbol table
(gdb) exec-file object 		specify object to run (not sym-file)
(gdb) core-file core   		post-mortem debugging

Signal Control
(gdb) info signals        	print signal setup
(gdb) handle signo actions      set debugger actions for signal
(gdb) handle INT print          print message when signal occurs
(gdb) handle INT noprint        don't print message
(gdb) handle INT stop        	stop program when signal occurs
(gdb) handle INT nostop         don't stop program
(gdb) handle INT pass        	allow program to receive signal
(gdb) handle INT nopass         debugger catches signal; program doesn't
(gdb) signal signo        	continue and send signal to program
(gdb) signal 0        		continue and send no signal to program

Machine-level Debug
(gdb) info registers        	print registers sans floats
(gdb) info all-registers        print all registers
(gdb) print/x $pc        	print one register
(gdb) stepi        		single step at machine level
(gdb) si        		single step at machine level
(gdb) nexti        		single step (over functions) at machine level
(gdb) ni        		single step (over functions) at machine level
(gdb) display/i $pc        	print current instruction in display
(gdb) x/x &gx        		print variable gx in hex
(gdb) info line 22        	print addresses for object code for line 22
(gdb) info line *0x2c4e         print line number of object code at address
(gdb) x/10i main        	disassemble first 10 instructions in \fImain\fR
(gdb) disassemble addr          dissassemble code for function around addr

History Display
(gdb) show commands        	print command history (>= gdb 4.0)
(gdb) info editing       	print command history (gdb 3.5)
(gdb) ESC-CTRL-J        	switch to vi edit mode from emacs edit mode
(gdb) set history expansion on       turn on c-shell like history
(gdb) break class::member       set breakpoint on class member. may get menu
(gdb) list class::member        list member in class
(gdb) ptype class               print class members
(gdb) print *this        	print contents of this pointer
(gdb) rbreak regexpr     	useful for breakpoint on overloaded member name

(gdb) define command ... end        define user command
*(gdb) RETURN        		repeat last command
*(gdb) shell command args       execute shell command 
*(gdb) source file        	load gdb commands from file
*(gdb) quit        		quit gdb

Calling Fortran from C ScreenCast

Was just browsing and came across this website that allows creating screen cast from web without any installation.

I decided to check this website. And here’s the result.

The video demonstrates calling fortran from c. The fortran is compiled with gfortran and C with Gcc.

Why did I choose this example? Well these two programs were lying around on my netbook’s desktop.

The service is great. It loads a java program and the intuitive GUI is easy to use. Easy upload to YouTube from within the app is topper. 😉

Happy 25th Birthday GNU GCC!

GNU compiler collection or GCC turned 25 last Friday. Gfortran is part of GCC.

To that end, Richard Guenther, who works in tool chain development at Suse Labs, and spends his days working on GCC, sent an email to the GCC development newsgroup to mark this historic occasion. Here a part of it, reprinted, for your reading pleasure.

Today the GCC development team celebrates the 25th anniversary of the GNU Compiler Collection.

When Richard Stallman announced the first public release of GCC in 1987, few could have imagined the broad impact that it has had. It has prototyped many language features that later were adopted as part of their respective standards — everything from "long long" type to transactional memory. It deployed an architecture-neutral automatic vectorization facility, OpenMP, and Polyhedral loop nest optimization. It has provided the toolchain infrastructure for the GNU/Linux ecosystem used everywhere from Google and Facebook to financial markets and stock exchanges. We salute and thank the hundreds of developers who have contributed over the years to make GCC one of the most long-lasting and successful free software projects in the history of this industry.

Whether you know it or not, GCC has probably affected how you go about your daily life with computers.

You might want to read this

A Stochastic Encounter

I use gfortran fortran compiler at home and intel fortran compiler in office.

Its by random chance I found a vital difference in gfortran and intel compiler’s interpretation of using the intrinsic subroutine Random_number.

In gg95 fortran forum, someone posted this query.

I have the following code to generate a random number in between 0 and 1.

program random
implicit none
real :: R
call random_seed()

call random_number(R)
end program

But it always returns a same number. Why this happens, can any body suggest me?

I checked this in gfortran and I got the same result. Everytime it returned the same random number.

With Intel compiler the result was as expected. In sun solaris it was as expected. The program gave different random number on each call.

But gfortran was different. I thought it was a bug in gfortran.

But then Tobius Burnus explained this

No, it’s actually a bug in the Fortran standard: The standard does not
specify whether calling “random_number()” should produce by default the same sequence or every time a different sequence.

There are proponents for both.

One group argues that having a random_number() function should produce random results. Others argue that when running, e.g., an Monte Carlo algorithm, by default the results should be reproducible. As the standard does not mandate either choice, one has to live with having compiles in both groups. (I heard that the person responsible for adding the intrinsic believed that he had
written down the choice – but obviously he hasn’t.)

I think its worth keeping in mind. This can help solve lot of pain if you are using gfortran and intel compilers for your stochastic programs.

Ten minutes to setup Modern Fortran 2003 / 2008 with CB IDE on Windows


The attached file demonstrates a step by step setup of modern Fortran 2003/2008 using CodeBlocks IDE on windows.

This was prepared and uploaded by Mr. Mohammad Rahmani.

The introduction of using the Codeblock IDE was new to me. Thank you Mohammad for the nice introduction on how to setup gfortran and C::B IDE.

I am making the pdf widely available through this link.

Modern Fortran With CodeBlocks IDE

10 minutes to setup Modern Fortran 2003 / 2008 with CB on Windows

Which one to use? -KPIC or -Kpic

So far my code was compiling perfectly in all the platform but after a recent new addition, the compiler gave this error while compiling.

too many symbols require ‘small’ PIC references :

Have 1092, maximum 1024 –recompile some modules –K PIC.

As I researched this.  I found my code has hit a rare case event.

KPIC and Kpic compiler options produces position-independent code for use in shared libraries.
Generally the Kpic works fine and does the job, but in rare case Kpic hits the limit and then KPIC has to be used.

What is Position-Independent Code?

The code within a dynamic executable is typically position-dependent, and is tied to a fixed address in memory. Shared objects, on the other hand, can be loaded at different addresses in different processes. Position-independent code is not tied to a specific address.

This independence allows the code to execute efficiently at a different address in each process that uses the code and thus position-independent code is always recommended for the creation of shared objects.

Here’s are the bits that helped me figure this out.

What is -KPIC and -Kpic?

Produces position-independent code for use in shared libraries. Each reference to a global datum is generated as a dereference of a pointer in the global offset table. Each function call is generated in pc-relative addressing mode through a procedure linkage table.

(SPARC) (PowerPC) With this option, the global offset table spans the range of 32-bit addresses in those rare cases where there are too many global data objects for -Kpic.

(Intel) -KPIC is identical to -Kpic.

Produces position-independent code for use in shared libraries.

(SPARC) (PowerPC) It is similar to -KPIC, but the size of the global offset table is limited to 8Kbytes.

Performance costs with -Kpic and -KPIC
There are two nominal performance costs with -Kpic and -KPIC:

A routine compiled with either -Kpic or -KPIC executes a few extra instructions upon entry to set a register to point at a table (_GLOBAL_OFFSET_TABLE_) used for accessing a shared library’s global or static variables.

Each access to a global or static variable involves an extra indirect memory reference through _GLOBAL_OFFSET_TABLE_. If the compile is done with -KPIC, there are two additional instructions per global and static memory reference.

When considering the above costs, remember that the use of -Kpic and -KPIC can significantly reduce system memory requirements, due to the effect of library code sharing. Every page of code in a shared library compiled -Kpic or -KPIC can be shared by every process that uses the library.

If a page of code in a shared library contains even a single non-pic (that is, absolute) memory reference, the page becomes non sharable, and a copy of the page must be created each time a program using the library is executed.

How to tell whether or not an object is compiled with -Kpic or –KPIC?
The easiest way to tell whether or not a .o file has been compiled with -Kpic or -KPIC is with the nm command:

 % nm file.o | grep _GLOBAL_OFFSET_TABLE_


A .o file containing position-independent code contains an unresolved external reference to _GLOBAL_OFFSET_TABLE_, as indicated by the letter U.

How to decide what to use -Kpic or –KPIC?
To determine whether to use -Kpic or -KPIC, use nm to identify the number of distinct global and static variables used or defined in the library.If the size of _GLOBAL_OFFSET_TABLE_ is under 8,192 bytes, you can use -Kpic. Otherwise, you must use -KPIC.


Primary Source: cc compiler options

To know more about Global Offset Table click here.

Fortran STL file reader and the compiler options.

In the post STL files and fortran, I posted a fortran binary stl file reader program.

Thomas was having problem using the code. He suspected that maybe it’s the problem of 32 and 64 incompatibility issue. I promised that I will check.

Last week I dug out the executable I created then and sure enough it worked. I fed it different 64 and 32 bit written stl files and it read them all.

After this, I downloaded the code posted on the blog, suspecting something is wrong in the uploaded code.

I compiled it. Tried reading the previously tested stl files and it failed. Confused I began debugging.

As I went through the code line by line, suddenly it clicked.

I recompiled the code and it worked.

The magic was in using the compiler option /assume:byterecl with the intel fortran compiler.

So the compilation was done as
Ifort /assume:byterecl stlreader.f90

Since the code is opening the binary file with RECL specifier, the assume:byterecl specifies that the units for the OPEN statement RECL specifier ie record length value are in bytes for unformatted data files, not longwords.

Thanks Thomas for the comment which led to this. 😉

The binary stl file reader fortran code is available here, make sure you use the assume byterecl option while compiling.

A Very Beginner’s Tutorial of DBX in Unix for Windows Users

Most of us work in windows. Most of our coding is done in visual studio and its a great tool to code and debug. The ability to add visual bookmarks, breakpoint is great.

But there are times when we are needed to work in Unix and Linux and sometime client requirements are such that we need to work from within a telnet account where there is practically no access to any GUI environment, then working with DBX is the only option.

I learnt DBX just because of that restriction and here is a simple presentation for anyone coming from windows development environment to begin using DBX, the powerful and capable debugging system available in Unix and Linux.

Its a very basic tutorial to start DBX debugging, meant primarily for first time users of dbx. Hope it will roll your ball and make your life a little easier.

Click here to download the original ppt file

or see all the slides as pictures posted here.