Get DataTable Value from Adjacent Cell without ForEach loop?

258 views Asked by At

I am working my way around this problem: Create multiple results from single row joining either 2 or 3 tables based off conditional of table1 results? and while I wish I could take Strawberry's advice, I can't, so I am trying now to do more in C# rather than through the DB, but also trying to be smart about how much CPU I utilize.

For scale, table1 may have 50,000 records which have those 20 codetype fields which have to be evaluated before they can be matched to table2, which has about 2,000 rows, or table3, which could have 200,000 rows. To avoid hammering the DB, I am going to store what is possible in memory, limit results by date as much as possible, but I want to avoid 2,000 foreach loops per 20 codetype matches.

To start off, I am getting the results I need from table2 and loading them into a C# DataTable stored as the variable named descriptionLookup:

id,    description
13    Item 13 Description
15    Item 15 Description
17    Item 17 Description
18    Item 18 Description
21    Item 21 Description
28    Item 28 Description
45    Item 45 Description

And table3 as lookupTable:

id,  table2id
1    15
33   17
21   28

doing a simple (not showing all surrounding code, just relevant):

var rawData = new DataTable();
using (OdbcCommand com = new OdbcCommand("SELECT id, description from table2", conn))
{
    using (OdbcDataReader reader = com.ExecuteReader())
    {
        rawData.Load(reader);
        conn.Close();
        return rawData;
    }

}

I then have that assigned to a variable that called the function. Now I have to deal with table1:

codeid1,codeid2,codeid3,...codeid20 ... codetype1,codetype2,codetype3,.....codetype20
18      13      1          33           0         0         1              1
13      21      45         0            0         1         0              0

Using a foreach row, I need to evaluate each codetype column for a 1 or a 0. When codetype=1 I need to grab the associated codeid, and then do a lookup from the data I am holding in memory as descriptionLookup to see what the table2id is that matches the id in lookuptable and then use that to lookup the description of the associated field in table2.

If codetype is 0, I just need to match codeid with the associated description field in table2.

I am looking at how to lay this out, and all I can think of is:

DataTable descriptionLookup= DB.ExecuteQuery("SELECT id, description from table2");
DataTable lookupTable= DB.ExecuteQuery("SELECT id, table2id from table3");
DataTable mainData= DB.ExecuteQuery("SELECT * from from table1");

foreach (var row in mainData)
{
    var manDataId = row.GetColumn("id");
    var subroutine = new Dictionary<string, string>();
    for (var index = 1; index < 20; index++)
    {
        string description;
        if (row.GetColumn("codetype" + index) == "1")
        {
            int idLookup = row.GetColumn(["codeid" +index]);
            foreach (var row2 in lookupTable)
            {
                if (row3.GetColumn("id") == idLookup)
                {
                    descriptionId = row3.GetColumn("table2id");
                    foreach (var row2 in descriptionLookup)
                    {
                        if (row.GetColumn("id") == descriptionId)
                        {
                        description = row2.GetColumn("description").ToString();
                        }
                    }
                }
            }
        }elseif (row.GetColumn("codetype" + index) == "0")
        {
            descriptionId = row.GetColumn(["codeid" +index]);
            foreach (var row2 in descriptionLookup)
            {
                if (row.GetColumn("id") == descriptionId)
                {
                description = row2.GetColumn("description").ToString();
                }
            }
        }

        subroutine.Add(manDataId.ToString(), description.ToString());
    }

ArrayData.Add(subroutine);
}

I haven't tried running the above code, so there is probably a problem or two in there, but it gets the point across of looping through thousands of records using foreach (var row3 in idLookup). The alternative seems to be making a DB query, but that seems more intensive than just looping through what is in memory, but it seems like there is a better way that I am missing on how to get the id or table2id without using a foreach.

To make this the longest looking question in history :) Here is my SQL that I have so far:

SELECT table1.id, table1.field1, table1.field2,
table2.description, fee.amt as fee FROM table2
INNER JOIN table1
ON table2.id = table1.codeid1
OR table2.id = table1.codeid2
OR table2.id = table1.codeid3
OR table2.id = table1.codeid4
OR table2.id = table1.codeid5
OR table2.id = table1.codeid6
OR table2.id = table1.codeid7
OR table2.id = table1.codeid8
OR table2.id = table1.codeid9
OR table2.id = table1.codeid10
OR table2.id = table1.codeid11
OR table2.id = table1.codeid12
OR table2.id = table1.codeid13
OR table2.id = table1.codeid14
OR table2.id = table1.codeid15
OR table2.id = table1.codeid16
OR table2.id = table1.codeid17
OR table2.id = table1.codeid18
OR table2.id = table1.codeid19
OR table2.id = table1.codeid20
INNER JOIN fee ON table2.id = fee.id
WHERE table1.codetype1 = 0
AND table1.codetype2 = 0
AND table1.codetype3 = 0
AND table1.codetype4 = 0
AND table1.codetype5 = 0
AND table1.codetype6 = 0
AND table1.codetype7 = 0
AND table1.codetype8 = 0
AND table1.codetype9 = 0
AND table1.codetype10 = 0
AND table1.codetype11 = 0
AND table1.codetype12 = 0
AND table1.codetype13 = 0
AND table1.codetype14 = 0
AND table1.codetype15 = 0
AND table1.codetype16 = 0
AND table1.codetype17 = 0
AND table1.codetype18 = 0
AND table1.codetype19 = 0
AND table1.codetype20 = 0

This works great as long as there isn't any of the codetype that have a 1, otherwise that record will be skipped. There will likely not be a single row where all codeid / codetype are filled out, nor will there ever be a case where codetype will be 1 across the board to match the inverse of this query.

0

There are 0 answers