Why a "Lost Paradise" in code, that ran many months in py2.7, now np.ndarray() not found in namespaces, while [import numpy as np] was executed?

Question

My question is: was there any namespace handling change between 2.7.9 and 2.7.11?

(I strived to narrow/isolate the issue, but for below-mentioned reasons, it is practically impossible to post an MCVE).

Below posted in-vivo diagnostics has proved, there are no namespace-issues with np.ndarray calls until the very first line trying to access / use numpy ( np ) symbol a line 14644 (14630 in the original numbering)

  aDSegFLOAT = np.ndarray( ( 80, 7 ), ...

where a thereto working numpy call suddenly raises an exception:

UnboundLocalError: local variable 'np' referenced before assignment

---

Intro:
For the purpose of a quantitative modelling, a rather extensive python module code is being re-used in a distributed processing framework operated for technical analyses and predictions. Due to a fact, that quant-models ( being used for predictions ) are generated at a remarkable cost of small tens-of-CPU-core-hours, there were deployed some additional measures allowing a stateful module reload( QuantFX )-s, so that these protect the already elaborated instances of the quant-models, but allow for module functionalities to be updated, tweaked and re-tested on-the-fly.

Facts:
import QuantFX ceased to work and throws exceptions on numpy calls, that were not modified:

• module code was used about a last 12 months, updating just [QUANT-TOOLS],[MODEL] sections
• the code is a Python 2.7 part of a multi-party distributed processing, so an attempt for a MCVE will never reflect / replicate the ecosystem behaviour

When the problem started:
After loading a recent Anaconda ( miniconda 4.0.0 / 4.0.5 ) package manager for a VM02/wXP host, also python got re-dressed into 2.7.11 and a module called QuantFX.py, being until this update run without any problems in other Anaconda 2.2.0 (32-bit) installations in python 2.7.9 stopped working with a strange looking exception

What demonstrates an ill-functioning state-of-operations:
The very import QuantFX goes well as it does for dozen months, week by week. A call to QuantFX.aMiniRESPONDER() now results in an exception / yields a Traceback as if the numpy were not import-ed at all ( ref. Line 221 ) but assumed to be some local variable.

    ... 
    [aMiniRESPONDER]: messaging & signalling sockets RTO:<infrastructure-setup>
    START:
           Sun Apr 17 20:19:33 2016 ...
    [aMiniRESPONDER]: final attempt to de-block a remote-processing failed at aXmitSOCKET.send()
    
    SIG->: Sun Apr 17 20:19:33 2016 PUB-lished a <_|SIG_EXIT|_> signal to peers, will sleep( 3 ) to allow for reception
    ZMQ:
           Sun Apr 17 20:19:36 2016 Graceful Exit Done. RET(0)
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "QuantFX.py", line 14630, in aMiniRESPONDER
        aDSegFLOAT= np.ndarray( ( 80, 7 ),
    UnboundLocalError: local variable 'np' referenced before assignment

( while a manual call to the very same numpy function on interpreter command line prompt,
respecting the namespace into which the numpy was import-ed by the module ( Line 221 below )
went well

|>>> QuantFX.np.ndarray( ( 2, 3 ), dtype = QuantFX.np.float32, order = 'F' )
array([[  1.22325565e+10,  -2.81789980e-05,   1.22325565e+10],
       [  1.91215026e+00,  -1.81719875e+00,   1.91215026e+00]], dtype=float32)
+0:00:13.282000
22:20:54
|
|>>> QuantFX.np.__version__
'1.10.4'
+0:00:24.438000
22:21:18
|

).

Line 14630 context of the Traceback and details:

Line 221 import numpy as np + the concept for State-Full reloads:

While QuantFX module coding might attract objections with regard to PEP* et al, the syntax-constructors were decided so as to allow both a context-full modus operandi via import/reload() and also fast and safe context-less light-weight operations in minimalistic RAM/CPU consuming remote terminals via select/copy/paste just a single component-code for a diagnostic or a control-CLI purposes ( like a aRemoteKEYBOARD() or aMiniRESPONDER() ) but without a need to import the whole QuantFX module ( provisioning late-imports, including the said numpy ( Line 14426 ) for such QuantFX-Context-less modus operandi ).

---

What is sought:

Any explanations about changes between 2.7.9 and 2.7.11, that could be related to the observed np.ndarray() collisions, introduced just last Friday.

Any recommendations for an alternative module syntax-constructors' layout is also of a high value if such proposal preserves both the statefulness during reload( QuantFX ) and permits both a context-full and a context-less code-base component usage.

---

Update:

in contrast to its possible appearance, this post was preceded with due review and debugging efforts for adding in-vivo self-diagnostics so as to allow for a problem isolation

   """ DEBUG: VM02, after about-a-year working here stable... THROWS EXC. HERE: "UnboundLocalError: local variable 'np' referenced before assignment"                                 ????? .EXC on np.ndarray()
            >>> import QuantFX
            Is QuantFX_FLAG seen in dir() during <module> import:  True
            SECTION: ____IDENTIFY____ [TRY].OK
            SECTION: ____IDENTIFY____ [FIN]
            SECTION: import           [TRY].OK
            SECTION: import           [FIN]
            SECTION: FX-MARKET CONTEXTs [TRY].OK
            SECTION: FX-MARKET CONTEXTs [FIN]
            SECTION:    GENERAL TOOLS [TRY].OK
            SECTION:    GENERAL TOOLS [FIN]
            SECTION:     DATA INPUTs [TRY].OK
            SECTION:     DATA INPUTs [FIN]
            SECTION: v41 PROCESS [TRY].OK
            SECTION: v41 PROCESS [FIN]
            SECTION: v41 QUANT TOOLS [TRY].OK
            SECTION: v41 QUANT TOOLS [FIN]
            QuantFX.py:10966: RuntimeWarning: invalid value encountered in divide
            ST2  = np.where( ST2 != 0, ( ST[-LLV.shape[0]:] - LLV ) /  ST2, ST2 ) # / ST2 ( == HHV - LLV )          #        __main__:4162: RuntimeWarning: invalid value encountered in divide
            SECTION: v41 MODEL [TRY].OK
            SECTION: v41 MODEL [FIN]
            SECTION: v41 PREDICTOR [TRY].OK
            SECTION: v41 PREDICTOR [FIN]
            SECTION: MetaTrader RESPONDER [TRY].OK
            SECTION: MetaTrader RESPONDER [FIN]
            SECTION: ____TEMPLATE____ [TRY].OK
            SECTION: ____TEMPLATE____ [FIN]
            SECTION: ____TEMPLATE____ [TRY].OK
            SECTION: ____TEMPLATE____ [FIN]
            SECTION: ____TEMPLATE____ [TRY].OK
            SECTION: ____TEMPLATE____ [FIN]
            +0:00:01.219000
            18:20:14
            |
            |>>> QuantFX.aMiniRESPONDER( aTarget2Bind2_URL = "tcp://10.0.0.62:9999", anInstrumentDictOfPARAMs = QuantFX.anFxCTX[ QuantFX.aCtxID ] )
            [aMiniRESPONDER]: runs in it's own full QuantFX context
            [aMiniRESPONDER]: imports DONE
            [aMiniRESPONDER]: variables DONE
            [aMiniRESPONDER]: messaging & signalling sockets RTO:[aZmqCONTEXT] <class 'zmq.sugar.context.Context'> setup with 15.2.0 version
            [aMiniRESPONDER]: messaging & signalling sockets     [aXmitSOCKET]-pre
            [aMiniRESPONDER]: messaging & signalling sockets     (aXmitSOCKET)-to be instantiated-via a call to aZmqCONTEXT.socket( zmq.PAIR )
            [aMiniRESPONDER]: messaging & signalling sockets     [aXmitSOCKET]-post
            [aMiniRESPONDER]: messaging & signalling sockets     [aXmitSOCKET]-bind()
            [aMiniRESPONDER]: messaging & signalling sockets RTO:[aXmitSOCKET]
            [aMiniRESPONDER]: messaging & signalling sockets RTO:[aCtrlSOCKET]
            [aMiniRESPONDER]: messaging & signalling sockets RTO:[aSIGsSOCKET]
            [aMiniRESPONDER]: messaging & signalling sockets RTO:<infrastructure-setup>
            START:
                Sun Apr 17 20:19:33 2016 ...
            [aMiniRESPONDER]: final attempt to de-block a remote-processing failed at aXmitSOCKET.send()
            
            SIG->: Sun Apr 17 20:19:33 2016 PUB-lished a <_|SIG_EXIT|_> signal to peers, will sleep( 3 ) to allow for reception
            ZMQ:
                Sun Apr 17 20:19:36 2016 Graceful Exit Done. RET(0)
            Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
            File "QuantFX.py", line 14630, in aMiniRESPONDER
                aDSegFLOAT= np.ndarray( ( 80, 7 ),                    
            UnboundLocalError: local variable 'np' referenced before assignment

In-vivo numpy-nd.array-symbol-checker

With an in-vivo checker idea proposal from @viraptor

    def debug_check_np_ndarray( aCallerSideInspectFrameINFO ):      # DEBUG-CHECK
        #ebug_check_np_ndarray( aCallerSideInspectFrameINFO = getframeinfo( currentframe() ) )
        #rom inspect import currentframe, getframeinfo              # EXTERNAL RESPONSIBILITY TO import + setup aCallerSideInspectFrameINFO .OnCall
        
        #ef         np_check():                                     # DEF:

        try:                                                        # TRY: np.*
            np.ndarray
            print 70*" ", "* NP: np.ndarray call [OK], FILE: ", aCallerSideInspectFrameINFO.filename, " LINE: ", aCallerSideInspectFrameINFO.lineno
            
        except:                                                     # EXC: found the issue
            print 70*" ", "* NP: np.ndarray call [**], FILE: ", aCallerSideInspectFrameINFO.filename, " LINE: ", aCallerSideInspectFrameINFO.lineno
            exc_type, exc_value, exc_traceback = sys.exc_info()             # prepare a traceback detail
            traceback.print_tb(  exc_traceback, limit = 5, file = sys.stdout )
            raise
            
        else:                                                       # ELSE: ok, passed
            #rint("* NP check ok")  # optionally add current function name via a traceback module
            return

On launch provides a detailed line-by-line confirmation there was no np.ndarray symbol-masking until the Line 14644 crashes on aDSegFLOAT = np.ndarray( ... ):

|>>> QuantFX.aMiniRESPONDER()
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14387
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14391
[aMiniRESPONDER]: runs in it's own full QuantFX context
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14395
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14404
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14410
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14412
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14419
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14423
[aMiniRESPONDER]: runs in a full QuantFX mode, few imports DONE
[aMiniRESPONDER]: imports DONE
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14441
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14443
[aMiniRESPONDER]: variables DONE
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14454
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14527
[aMiniRESPONDER]: messaging & signalling sockets RTO:[aZmqCONTEXT] <class 'zmq.sugar.context.Context'> setup with 15.2.0 version
[aMiniRESPONDER]: messaging & signalling sockets RTO:<infrastructure-setup>
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14626
START:
    Tue Apr 19 16:02:12 2016 ...
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14632
                                                                    * NP: np.ndarray call [OK], FILE:  QuantFX.py  LINE:  14643

[aMiniRESPONDER].FINALLY: will start attempts to de-block a remote-processing ( after a prior intentional SIG_EXIT or an unhandled EXC )
[aMiniRESPONDER]: final attempt to de-block a remote-processing failed at aXmitSOCKET.send()
[aMiniRESPONDER]: final attempt to propagate a signal to peers failed at aSIGsSOCKET.send()

EXC. ZmqError(ZMQError('No such file or directory')) on aSIGsSOCKET .setsockopt( zmq.LINGER, 0 ) / .close( aSIGsSOCKET )
EXC. ZmqError(ZMQError('No such file or directory')) on aCtrlSOCKET .setsockopt( zmq.LINGER, 0 ) / .close( aCtrlSOCKET )
EXC. ZmqError(ZMQError('No such file or directory')) on aXmitSOCKET .setsockopt( zmq.LINGER, 0 ) / .close( aXmitSOCKET )

ZMQ:
    Tue Apr 19 16:02:15 2016 Graceful Exit Done. RET(0)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "QuantFX.py", line 14644, in aMiniRESPONDER
    aDSegFLOAT          = np.ndarray( ( 80, 7 ),                    #           #  DSegFLOAT[][]                                  ### ToDo.MS: size-independent ( 80+ )                   np.*()  :::::::::::::::
UnboundLocalError: local variable 'np' referenced before assignment
+0:00:10.734000
16:02:15
|            

Answer 1

From the code you're showing, it's unlikely/impossible that the python update changed anything in the screenshoted code.

What could change is either the environment, or code in other places that was supposed to not set aMiniRESPONDER_in_full_QuantFX_CONTEXT. In case that variable is set, the import numpy as np is commented and you'd get the exception you're showing here.

Some debugging ideas: either run in a debugger, or use print-debug and break/print in places where you think numpy should be imported - likely you'll find that you're just not going into those branches.

Another one: Change one thing at a time. If I understand the question correctly, you changed: python, package manager and deployment environment all in one go and discovered that stuff doesn't work anymore in a way you don't understand. Go back to the original environment and retest. Then upgrade python and nothing else. Retest. Then upgrade package manager (but not packages!). Retest. ... Until you find exactly which change breaks your app.

Another one: When something fails and you can't recover from it (for example failed import), don't continue. On the second screenshot you've got a lot of import statements in a try block, which will simply ignore the errors and continue execution. (L 328-332). If you continue, something will just explode later on. If the rest of the code looks like that - yes, you'll end up with silent failures and no idea where they come from.

Another one to check @hpaulj's suggestion. Very crude approach: create a function like:

def np_check():
    try:
        np.ndarray
    except:
        # found the issue
        raise
    else:
        print("* NP check ok")  # optionally add current function name via traceback module

Sprinkle that around the code to find out where's the first place when the np name "disappears". Put the first call right after the import you expect to work, so that you see at least one "* NP check ok" before the error. Then you just need to add more checks between the place of last success and the first failure, until you isolate the line which causes issues.

Answer 2

I can reproduce your error with a simple script - I import a module and use it once fine. But the 2nd time I use it, the global assignment gets masked by a local one. This masking occurs even though the local assignment occurs after the attempted use.

import collections    
def foo1():
   dd=collections.defaultdict(list)
   return dd

def foo2():
   dd=collections.defaultdict(int)
   collections = []
   return dd

print foo1()
print foo2()

produces

2112:~/mypy$ python stack1.py
defaultdict(<type 'list'>, {})
Traceback (most recent call last):
  File "stack1.py", line 13, in <module>
    print foo2()
  File "stack1.py", line 8, in foo2
    dd=collections.defaultdict(int)
UnboundLocalError: local variable 'collections' referenced before assignment

Search your code, or possibly something that you call, for a np=... like statement. For a start look at QuantFX.aMiniRESPONDER, since that is the immediate context of the error.

Changing my test function to

def foo2():
   dd = foo1()
   collections = []
   return dd

does not produce the error. So the masking has a rather limited scope.