stop words removal using arrays c#

Question

I have a string array of stopWords and string array of input texts i.e.

string[] stopWords = File.ReadAllLines(@"C:\stopWords.txt");

and

con.Open();
SqlCommand query = con.CreateCommand();
query.CommandText = "select p_abstract from aminer_paper where pid between 1 and 500 and DATALENGTH(p_abstract) != 0";

SqlDataReader reader = query.ExecuteReader();

var summary = new List<string>();
while(reader.Read())
{
    summary.Add(reader["p_abstract"].ToString());
}

reader.Close();

string[] input_Texts = summary.ToArray();

Now, I have to use these stopWords array to remove from input_Texts array. I have used following technique but not working, weird while accessing both arrays index. For example, take first text at index 0 of input_Texts array i.e.

input_Texts[0]

and then match all the word strings in stopWords array i.e.

// have to match all the indexes of stopWords[] with input_Texts[0]
stopWords[]   

then after removing all the stopWords from index 0 text of input_Texts array, have to repeat it for all the texts in input_Texts array.

Any suggestions and code samples with modifications will be highly appreciated with acknowledgment.

Thanks.

Answer 1

You can use Linq to do this

        //string[] input_Text = new string[] { "Ravi Kumar", "Ravi Kumar", "Ravi Kumar" }; 
        //string[] stopWords = new string[] { "Ravi" }; 
        for(int i=0;i<input_Text.Count();i++)
        {
            for (int j = 0; j < stopWords.Count(); j++)
            {
                   input_Text[i] = input_Text[i].Replace(stopWords[j]," ");
            }
        }

Answer 2

using System;
using System.IO;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using System.Data;
using System.Data.SqlClient;

namespace StopWords_Removal
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {

                string[] stopWords = File.ReadAllLines(@"C:\stopWords.txt");

                SqlConnection con = new SqlConnection("Data Source=ABC;Initial Catalog=xyz;Integrated Security=True");

                con.Open();
                SqlCommand query = con.CreateCommand();
                query.CommandText = "select text from table where id between 1 and 500 and DATALENGTH(text) != 0";

                SqlDataReader reader = query.ExecuteReader();

                var summary = new List<string>();
                while(reader.Read())
                {
                    summary.Add(reader["p_abstract"].ToString());
                }

                reader.Close();
                string[] input_Texts = summary.ToArray();

                for (int i = 0; i < input_Texts.Length; i++)
                {
                    for (int j = 0; j < input_Texts.Length; j++)
                    {
                        input_Texts[j] = string.Join(" ", input_Texts[j].Split(' ').Except(input_Texts[j].Split(' ').Intersect(stopWords)));
                    }
                }

                for (int d = 0; d < input_Texts.Length; d++)
                {
                    Console.WriteLine(input_Texts[d]); 
                    Console.ReadLine();
                }

            }
            catch (Exception e)
            {
                Console.WriteLine("Exception: " + e.Message);
            }
            finally
            {
                Console.WriteLine("Executing finally block.");
            } 
        }
    }
}

Answer 3

try this:

string[] result = input_Texts.Except(stopWords).ToArray();

Answer 4

Can also be done like this:

for(int i=0;i<input_Texts.Length;i++)
  {
    input_Texts[i]=string.Join(" ", input_Texts[i].Split(' ').Except(input_Texts[i].Split(' ').Intersect(stopWords)));
  }

This will process each text in input_Texts and remove all the stop words from it.