I'm trying to process data received via API using a loop, but the loop is very slow. Is this a Dyalog bug?
res←getBinanceSymbols
res←⍬
baseToken←'USDT'
tmpSymbols←(⎕JSON(HttpCommand.Get'api.binance.com/api/v3/exchangeInfo').Data).symbols
:For sym1 :In tmpSymbols
⎕←sym1.symbol
:EndFor
see here → https://www.youtube.com/watch?v=oUl4oixd3Ds
Anyway, my complete task is to form pairs of the form XXXYYY XXXZZZ ZZZYYY, and I've tried to do this using nested loops (see below), but it takes forever...
res←getBinanceSymbols
res←⍬
baseToken←'USDT'
tmpSymbols←(⎕JSON(HttpCommand.Get'api.binance.com/api/v3/exchangeInfo').Data).symbols
:For sym1 :In tmpSymbols
Sym1_Token1←sym1.baseAsset
Sym1_Token2←sym1.quoteAsset
:If Sym1_Token1≡baseToken
:For sym2 :In tmpSymbols
Sym2_Token1←sym2.baseAsset
Sym2_Token2←sym2.quoteAsset
:If Sym1_Token1≡Sym2_Token2
:For sym3 :In tmpSymbols
Sym3_Token1←sym3.baseAsset
Sym3_Token2←sym3.quoteAsset
:If Sym2_Token1≡Sym3_Token1
:AndIf Sym3_Token2≡Sym1_Token2
:AndIf Sym1_Token1≢Sym1_Token2
:AndIf Sym2_Token1≢Sym1_Token1
:AndIf Sym2_Token1≢Sym1_Token2
res,←⊂((Sym1_Token1⍪Sym1_Token2)(Sym2_Token1⍪Sym1_Token2)(Sym2_Token1⍪Sym1_Token2))
:EndIf
:EndFor
:EndIf
:EndFor
:EndIf
:EndFor
Question 1: Why is the loop so slow? I work on a computer with Intel Core i5 11th and 40GB DDR4. It's just unthinkable!
Question 2: How can I solve my problem without resorting to nested loops? In a more the APL way
The reason for unreasonable slowness is what I call a dangling reference. Dyalog objects contain a reference to their parent object.
tmpSymbols
therefore contains a reference toData
butData
doesn't otherwise live anywhere per se. This continuously causes the garbage collector to kick in and investigate whetherData
can be expunged, only to find, every time, that it cannot, becausetmpSymbols
needs it to remain in existence. It is this contant garbage collection activity that is the cause of the slowdown.So, how do you work around this? Keep a direct reference to
Data
, as then there's no question that it doesn't need to be expunged: