What is a file pending_segments_runc under /usr/share/elasticsearch/data/nodes/0/_state/ in an Elasticsearch node

Question

Some security scan against one of our k8s clusters found a file called "pending_segments_runc" under folder /usr/share/elasticsearch/data/nodes/0/_state/ in one Elasticsearch master pod. The scanning software is reporting this as a possible sign that the container has tampered with the "runc" binary file, and can potentially lead to container escape.

I need to find proves supporting this to be either a true positive or a false positive. I've looked in to other ES pods in other ES clusters, in the corresponding folder. They do not have any file starting with "pending_segments", nor anything ending with "runc". All Elasticearch clusters are of version 7.17 and are deployed by an ECK operator.

So the question is, can anyone with knowledge of Elasticsearch internals point out a possibility that a file with the name "pending_segements_runc" can be created under that folder? Or reversely, confirm that it is impossible a file with that name can be produced by Elasticsearch itself?

Answer 1

Following source code links in @Val's answer. I found concrete proof that file name can be created by Elasticsearch / Lucene. First, ES passed a numeric generation parameter to IndexFileNames.fileNameFromGeneration when composing a "pending_segements" file name: https://github.com/elastic/elasticsearch/blob/96184ddb133829b7d87eadaf5a7b087aef2001f2/x-pack/plugin/core/src/main/java/org/elasticsearch/snapshots/sourceonly/SourceOnlySnapshot.java#L129

This method in turn called Long.toString(gen, Character.MAX_RADIX)) to convert the generation number to a string. https://github.com/apache/lucene/blob/ea272d0eda140a76c70f2d3befbe85e9e74a74e1/lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java#L69 . Because of the second parameter Character.MAX_RADIX, the result is not a decimal but alphanumeric.

Answer 2

Looking at the Elasticsearch source code and Lucene source code, it's definitely possible that such a file can be written.

However, just note from the code that the suffix is supposed to be numeric and not alphanumeric like runc, so that's still not clear. Hard to tell without checking what's inside. It could be a naughty segment file waiting to be merged with other legit segments. Since runC is a container executor, I'd be very careful since that could be some bad code disguised as a legitimately named segment file waiting to be merged.