Facing logstash compatibility issues

27 views Asked by At

I am trying to move data from AWS ElasticSearch 1.5 to AWS OpenSearch 2.11 with the help of logstash. I have downloaded the logstash from here :

https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.16.3-x86_64.rpm

Further, I am using the below conf file :

input {
    elasticsearch {
        hosts => "https://search-es15domain-blog.us-east-1.es.amazonaws.com:443"
        index => "index-28-feb"
    }
}
 
 
output {
    opensearch {
        hosts       => "https://search-os211domain-blog.us-east-1.es.amazonaws.com:443"
        index       => "logstash-28-feb"
        ecs_compatibility => disabled
        ssl_certificate_verification => false
    }
}

When running logstash it is returning and failing with the following logs :

[2024-03-28T06:56:54,606][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-03-28T06:56:54,618][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2024-03-28T06:56:56,875][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-03-28T06:56:57,762][INFO ][org.reflections.Reflections] Reflections took 109 ms to scan 1 urls, producing 119 keys and 417 values 
[2024-03-28T06:56:59,087][INFO ][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["https://search-os211domain-blog.us-east-1.es.amazonaws.com:443"]}
[2024-03-28T06:56:59,140][WARN ][logstash.outputs.opensearch][main] ** WARNING ** Detected UNSAFE options in opensearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
[2024-03-28T06:56:59,473][INFO ][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://search-os211domain-blog.us-east-1.es.amazonaws.com:443/]}}
[2024-03-28T06:56:59,823][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://search-os211domain-blog.us-east-1.es.amazonaws.com:443/"}
[2024-03-28T06:56:59,873][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.11.0) {:version=>2}
[2024-03-28T06:57:00,020][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:disabled}
[2024-03-28T06:57:00,037][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x13ead8c7 run>"}
[2024-03-28T06:57:00,921][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.88}
[2024-03-28T06:57:01,908][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of OpenSearch>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-opensearch-1.0.0/lib/logstash/inputs/opensearch.rb:391:in `test_connection!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-opensearch-1.0.0/lib/logstash/inputs/opensearch.rb:224:in `register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:390:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:315:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x13ead8c7 run>"}
[2024-03-28T06:57:01,910][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2024-03-28T06:57:01,926][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2024-03-28T06:57:02,013][INFO ][logstash.runner          ] Logstash shut down.
[2024-03-28T06:57:02,022][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

Looking forward to your suggestions.

Thanks

0

There are 0 answers