Using SFTP Outbound Channel Adapter to stream data to a remote file server over SFTP

Question

In my spring boot app, I have to read a lot of data from the DB convert it to a CSV file and upload it to a SFTP server. Since the file can be huge I cannot read the whole file in memory and then upload. It will to be streamed and appended on the SFTP server.

I am using the atmoz/sftp docker container as my SFTP server and it works fine.

For Spring boot client side - Spring Integration - SFTP Outbound Channel Adapter seems promising to me. Using the following code I was able to upload the file to the SFTP server:

@SpringBootApplication
public class SftpIngApplication {

    public static void main(String[] args) throws FileNotFoundException {
        ConfigurableApplicationContext context =
                new SpringApplicationBuilder(SftpIngApplication.class)
                        .web(WebApplicationType.SERVLET)
                        .run(args);
        MyGateway gateway = context.getBean(MyGateway.class);
        gateway.sendToSftp(new File("/home/john/sftp-ing/notes.txt"));
    }

    @Bean
    public SessionFactory<SftpClient.DirEntry> sftpSessionFactory() {
        DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
        factory.setHost("localhost");
        factory.setPort(2222);
        factory.setUser("foo");
        factory.setPassword("pass");
        factory.setAllowUnknownKeys(true);
        return new CachingSessionFactory<>(factory);
    }

    @MessagingGateway
    public interface MyGateway {

        @Gateway(requestChannel = "toSftpChannel")
        void sendToSftp(File file);

    }

    @Bean
    public SftpRemoteFileTemplate sftpRemoteFileTemplate(SessionFactory<SftpClient.DirEntry> sftpSessionFactory) {
        return new SftpRemoteFileTemplate(sftpSessionFactory);
    }

    @Bean
    public IntegrationFlow sftpOutboundFlow(SessionFactory<SftpClient.DirEntry> sftpSessionFactory) {
        return IntegrationFlow.from("toSftpChannel")
                .handle(Sftp.outboundAdapter(sftpSessionFactory, FileExistsMode.REPLACE)

                        .useTemporaryFileName(false)
                        .remoteDirectory("upload")
                ).get();
    }

}

However this example uses new File("/home/john/sftp-ing/notes.txt") to upload the file. Simply replacing it with FileInputStream also uploads the file but not with the same name and extension i.e. bar.txt. The file is uploaded as 67157c1b-439d-9f3b-f2be-88eff908e39c.msg.

It seems to me that the stream is open and some how needs to closed probably however I am not sure what would be the best way to do this. Can someone please help me out with this?

Answer 1

It is indeed correct to use an InputStream for your use-case as a payload for transferring to SFTP. What you are missing is a file name to be provided as well.

The default FileNameGenerator in the RemoteFileTemplate is a DefaultFileNameGenerator with the logic like this:

public String generateFileName(Message<?> message) {
    Object filename = evaluateExpression(this.expression, message);
    if (filename instanceof String name && StringUtils.hasText(name)) {
        return name;
    }
    Object payload = message.getPayload();
    if (payload instanceof File file) {
        return file.getName();
    }
    return message.getHeaders().getId() + ".msg";
}

The mentioned expression is this:

private volatile Expression expression =
        new FunctionExpression<Message<?>>((message) -> message.getHeaders().get(FileHeaders.FILENAME));

Since you don't provide that FileHeaders.FILENAME, you end up with that generated name based on message id.

Just consider to populated a FileHeaders.FILENAME header with expected file name before you send an InputStream to your sftpOutboundFlow!