Reducing memory requirements of a pyomo model

Question

I am building a big pyomo model with over 1 million constraints and 2 million variables.

And I am looking for suggestions to reduce the memory requirements of the model that I am building.

At the moment it requires over 20gb's of RAM.

How would I reduce this?

I've never tested defining variables with/without within=pyomo.NonNegativeReals. But I am assuming it would reduce the amount of memory required for a given variable. Is there other things I could do without reducing the amount of the variables or constraints.

Eg:

Following var will need X bytes of memory

m.var = pyomo.Var(
    m.index)

And maybe following will need X-1 bytes of memory

m.var = pyomo.Var(
    m.index,
    within=pyomo.NonNegativeReals)

Of course this is a speculation. Without testing one cannot be sure about this. However, I am willing to try anything if some1 has an idea or more experience regarding this issue.

Any ideas?

Some Tests:

Keep in mind that is not the real model but the example builded with an other data. But still the same script.

index=1000 // Full Consts         // 347580 KB (commit) // 370652 KB (working set)
              0 Const Full Rules  // 282416 KB (commit) // 305252 KB (working set)
              0 Const 0 Rule      // 282404 KB (commit) // 305200 KB (working set)
              1 Const 1 Rule      // 290408 KB (commit) // 313136 KB (working set)

index=8760 // Full Consts         // 1675860 KB (commit) // 1695676 KB (working set)

Answer 1

I've used pympler to analyze the test case you pointed me to. Here is what I've found:

After pyomo_model_prep (loads data and places it onto empty ConcreteModel):

1. memory usage is 13.2 MB

After adding all Set and Param objects:

2. memory usage is 13.3 MB

After adding all Var objects:

3. memory usage is 14.3 MB

After adding all Constraint objects:

4. memory usage is 15.0 MB

When I set the timesteps to 60, the results are

1. memory usage is 13.2 MB (data)
2. memory usage is 13.3 MB (after sets, params)
3. memory usage is 19.6 MB (after vars)
4. memory usage is 23.6 MB (after constraints)

So the variables do have a pretty big impact on model memory when there are a larger number of timesteps. The only obvious place I can see for reducing memory usage is to not store all of the data on the model (or delete it from the model after it is no longer needed), then perhaps what is unused will be cleaned up by the garbage collector.

Unfortunately, there isn't really any easy way to reduce the memory of the variable declarations.

Update 1: Just an FYI, pretty much all of the memory usage for the variable declarations is a result of the e_pro_in and e_pro_out indexed variables.

Update 2: If a large number of indices of the e_pro_in and e_pro_out variables are not used in the model, you can reduce memory requirements by building a reduced index set for each of them. Here is how that might look:

e_pro_in_index = []
for t in m.tm:
    for i,j in m.pro_tuples:
        for c in m.com:
            if ...:
                e_pro_in_index.append((t,i,j,c))
m.e_pro_in_index = Set(dimen=4, initialize=e_pro_in_index)
m.e_pro_in = pyomo.Var(
    m.e_pro_in_index,
    within=pyomo.NonNegativeReals,
    doc='Power flow of commodity into process (MW) per timestep')

You would need to extract the logic from constraint rules to figure out what indices are not needed.