Creating an elasticsearch index from logstash

740 views Asked by At

I am trying to load data from an Sql Server into ElasticSearch. I am using Logstash with the jdbc plugin and the elastic-search plugin. I am loading my data in ElasticSearch but can not figure out how to set my index. I am using a template index to try this. Below is what I am using but whenever I search I do not get any results.

logstash.config

# contents of logstash\bin\logstash.config

input {
  jdbc {
    jdbc_driver_library => ".\Microsoft JDBC Driver 6.2 for SQL Server\sqljdbc_6.2\enu\mssql-jdbc-6.2.1.jre8.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://mydbserver;databaseName=mydb;"
    jdbc_user => "******"
    jdbc_password => "******"
    schedule => "* * * * *"
    parameters => { "classification" => "EMPLOYEE" }
    statement => "SELECT Cost_Center, CC_Acct_1, CC_Acct_2, CC_Acct_3 from dbo.Cost_Center where CC_Classification = :classification"
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "allocation-testweb"
    template => "index_template.json"
  }
  #stdout { codec => rubydebug }
}

index_template.json

{
"template": "allocation-*",
"order":1,
"settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1,
    "analysis": {
        "analyzer": {
            "substring_analyzer": {
                "tokenizer": "ngram_tokenizer",
                "filter": ["lowercase"]
            }
        },
        "tokenizer": {
            "ngram_tokenizer": {
                "type": "edge_ngram",
                "min_gram": 2,
                "max_gram": 10,
                "token_chars": ["letter","digit"]
            }
        }
    }
},
"mappings":{
    "costcenter": {
        "properties": {
            "cc_acct_1": {
                "type": "string",
                "analyzer": "substring_analyzer"
            },
            "cc_acct_2": {
                "type": "string",
                "analyzer": "substring_analyzer"
            }
        }
    }
}

I have created a similar index in code while doing some initial research. Is my index_template incorrect or is there another way I should be doing this?

Update: I had the mismatched index names between my 2 files. I'm now able to search using Postman and curl. However when I try to get data using a NEST client I can never get data back. Below is the code snippet for the query.

  var searchResult = client.Search<CostCenter>(s => s  
    .Size(1000)
    .Index("allocation_testweb")
    .MatchAll());

This previously worked with the same data loaded from a file. CostCenter is simply an object with members called Cost_Center, CC_Acct_1, CC_Acct_2, and CC_Acct_3. I'm sure again I am over complicating the issue and missing something obvious.

UPDATE II:

I have made the changes suggested by @RussCam below and still do not get any results back. Below is my updated code.

var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node); 
     //.InferMappingFor<CostCenter>(m => m.IndexName("allocation_testweb"));
var client = new ElasticClient(settings);

var searchResult = client.Search<CostCenter>(s => s
    .Type("costCenter")
    .Size(1000)
    .Index("allocation_testweb")
    .MatchAll());

I commented out the InferMappingFor<> since it was not providing a result.

Mapping image requested by @RussCam. I've also included my costcenter class (I have tried naming all variations of costcenter).

public class costcenter
{
  public string cost_center { get; set; }
  public string cc_acct_1 { get; set; }
  public string cc_acct_2 { get; set; }
  public string cc_acct_3 { get; set; }
}

enter image description here

0

There are 0 answers