Alternatives to mutable lists to store tracking information

Question

I have some calculations. In this example, a very simple one it is a for loop. In reality, after testing, I will have a loop that will process live streaming data.

open System.Collections.Generic

let updates = new List<int>()

let appendToAList number = 
    updates.Add number
    printfn "Added: %i" number

let test = 
    for num in [1..100] do
        appendToAList num
        num * 2 |> ignore

My goal is to implement appendToAList, a function that takes some additional information about the ongoing computation (which is needed to track its execution) and stacks the results in a suitable datastructure. Is there a functional alternative to mutable lists?

Thank you.

EDIT: Some more info on the achitecture

Not sure this helps, but - long story short - I am building a trading system.

Some snippets from my code:

// defines the action steps when new data is received
let processFeed (date, feed:Matrix) = stateMonad<Matrix, Matrix> {
    do! strategy.ApplyPnL feed 
    do! strategy.RecalibrateFilter feed 
    let! signal' = strategy.Signal feed 
    do! strategy.Trade feed signal' }

This part takes a timestamp and fresh market quotes. Strategy wraps some mathematical functions. According to the client needs, the strategy can recalibrate, extract a signal and eventually shoot orders. In order to make things leaner, a state monad carries along the state of the algorithm and the portfolio (money, nr shares, etc ..)

At the moment, I have to run simulations (back-tests) in order to assess that things work fine. Basically, I loop through the historical data.

// process the historical data
let rows = seq {  for idx = 0 to 3 do yield (dataframe.GetRowKeyAt(idx), data.[[idx], [0..3]])  }  // for debugging purposes here I have replaced data.Size.[0]-1 with 3
let simulation = stateMonad<Matrix, Matrix> {
    for row in rows do
        do! processFeed row }

The exact same code that run in the for loop should run, in the future, in a loop that could look like

...
let runLive = 
    while (isConnected) 
        let newFeed = GetSomehowDataFromTheMarket
        do! processFeed newFeed

I have not yet worked out this part and I am learning F# by doing so, please forgive this non-sense pseudocode.

What I need is a neat way to get information out of the loop and process it (display it in a gui, chart some data, etc). In other words, I need to sketch how the algorithm will report to the user.

I have started trying to stack the state of the algorithm in a list so I could eventually start charting some stuff.

It would be great if I could run the function that takes care of the reporting on a separate thread, so I was interested in learning a functional way to do all that, something to facilitate the design of something that could easily parallelized.

Answer 1

Even if you have some global state (e.g. values produced by some live stream), it might make sense to use immutable functional list, but have a mutable reference to it.

The idea is, that you can very cheaply append elements (to the front) and, at any point in time, you can safely get a reference to the current list (and process it concurrently, while new state is constructed - without affecting the process that processes an earlier snapshot of the list).

So you could use something like:

let mutable updates = []

let appendToAList number = 
    updates <- number::updates
    printfn "Added: %i" number

let test = 
    for num in [1..100] do
        appendToAList num
        num * 2 |> ignore

The best choice really depends on the actual scenario that you have in mind - so if you want to get a better answer, try describing what kind of system/architecture are you designing.

Answer 2

It would be best to remove your loop and do something like this (unless your other requirements prevent it):

let test = [1..100] |> List.fold (fun state number-> printfn "Added: %i" number
                                                     number::state) []

The F# standard list (not the same as the List type in System.Collections.Generic) is immutable. Also, notice how there are no longer any mutable variables. In case you're not familiar with the F# list type, [] creates an empty list and putting :: between an item and a list creates a new list with the item prepended.

Answer 3

If I were dealing with the processing of live data streams, I'd probably use some kind of event streaming solution whereby the result of each calculation triggers an event.

Depending on the result of the calculation (maybe there are errors, or special cases for values within certain ranges etc.) you could use the Observable module to build an event pipeline with your specific rules etc. Your pipelines could end by storing results and time information in a database, or graphing to a UI or something.

This way you could do away with your mutable state altogether.

let event = new Event<int>()
let observable = event.Publish

// Example rules: Even numbers are successes and odd numbers are errors
let evens, odds = observable |> Observable.partition (fun i -> i%2=0)
evens |> Observable.subscribe (printfn "Success: %i")
odds |> Observable.subscribe (printfn "Error: %i")


let appendToList number = 
    event.Trigger(number)

let test = 
    for num in [1..100] do
        appendToList num
        num * 2 |> ignore

This might be completely off the mark, and perhaps you need to fit a certain design in order to plug into an existing system or whatever. But then again, it might also help! - hopefully it is the latter.