how to read all file names from S3 Bucket using Mule 4?

Question

I have a requirement where I need to list down only file names from S3 Bucket.

I have tried a simple POC where I used "list Objects" connector of mule 4 and configured with my bucket and folder name. but its giving me only 101 objects (objects contains key names which is actually file names)

the code XML:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core" xmlns:s3="http://www.mulesoft.org/schema/mule/s3"
    xmlns:http="http://www.mulesoft.org/schema/mule/http"
    xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/s3 http://www.mulesoft.org/schema/mule/s3/current/mule-s3.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
    <http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="9b051304-71f4-4199-8895-0ada11e3abde" >
        <http:listener-connection host="0.0.0.0" port="8081" />
    </http:listener-config>
    <s3:config name="Amazon_S3_Configuration" doc:name="Amazon S3 Configuration" doc:id="bb8a9880-5b6d-4f17-a768-54e02a26b163" >
        <s3:basic-connection accessKey="${Accesskey}" secretKey="${Secretkey}" />
    </s3:config>
    <configuration-properties doc:name="Configuration properties" doc:id="d6105df6-f925-49c8-9b07-688637897e05" file="dev.properties" />
    <flow name="tes-s3Flow" doc:id="3d752d7d-e210-4e73-9d34-240933ed6086" >
        <http:listener doc:name="Listener" doc:id="79565bb4-3a69-495c-ae95-756cb9ca27f4" config-ref="HTTP_Listener_config" path="/test"/>
        <logger level="INFO" doc:name="Logger" doc:id="3f6b2e97-00d7-41bd-902e-2c526aae8ddb" />
        <s3:list-objects doc:name="List objects" doc:id="1ae98c37-c376-4a78-a1d3-2bad6ae01ac3" config-ref="Amazon_S3_Configuration" bucketName="manager-files" prefix="system-api/prd/440/"/>
        <ee:transform doc:name="Transform Message" doc:id="c9e9ad29-dcaa-46c1-a761-bd166fb71958" >
            <ee:message >
                <ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload]]></ee:set-payload>
            </ee:message>
        </ee:transform>
    </flow>
</mule>

The screen shot of connector configuration is as below.

I tried with property Max Keys = 500 0r 1 but it always gives me 101 objects.

note: I have total 300 000 files and I want names for all

Is it possible to get that or any workaround possible?

Answer 1

I've conducted a test on my end where I added over 200 files to an Amazon S3 bucket. Using the List Object operation with the S3 connector, I was able to list all the files. Here's the code I employed:

<?xml version="1.0" encoding="UTF-8"?>
<mule
    xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
    xmlns:s3="http://www.mulesoft.org/schema/mule/s3"
    xmlns="http://www.mulesoft.org/schema/mule/core"
    xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/s3 http://www.mulesoft.org/schema/mule/s3/current/mule-s3.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
    <s3:config name="Amazon_S3_Configuration" doc:name="Amazon S3 Configuration" doc:id="ea271201-ef56-43ea-9fbf-a43af4abcb99" >
        <s3:connection accessKey="access key value" secretKey="secret key value" region="ap-south-1"/>
    </s3:config>
    <flow name="s3-testFlow" doc:id="8662b6ad-850d-4fc8-b05c-1969bfccf2d7" >
        <scheduler doc:name="Scheduler" doc:id="ab32b21c-7c58-4997-819d-2de287871db6" >
            <scheduling-strategy >
                <fixed-frequency frequency="100" timeUnit="MINUTES"/>
            </scheduling-strategy>
        </scheduler>
        <logger level="INFO" doc:name="Logger" doc:id="7d79bedd-823e-4289-ac65-be09e5e66104" message="************* Before List Object *********"/>
        <s3:list-objects doc:name="List Objects" doc:id="6577acde-c21d-45a3-bf52-0ae2afc5acbc" config-ref="Amazon_S3_Configuration" bucketName="bucketname"/>
        <ee:transform doc:name="Transform Message" doc:id="5bc694e4-689b-44b9-a494-6ee388a03f35" >
            <ee:message >
                <ee:set-payload >
                    <![CDATA[%dw 2.0
output application/json
---
payload.key]]>
                </ee:set-payload>
            </ee:message>
        </ee:transform>
        <logger level="INFO" doc:name="Logger" doc:id="d0bd7823-7b4b-4d70-ab07-f14a95ee1ec2" message="#[payload]"/>
    </flow>
</mule>

One possible cause for the issue could be the prefix value you're using. Please confirm whether there are actually more than 101 files ( keys) that correspond with this prefix.

As your work is just a proof of concept, I would suggest start small and consider setting up a new test bucket. Upload more than 101 files to this new bucket and check if the issue persists. To isolate the problem, refrain from using any prefix on the MuleSoft side and simply use the bucket name.

In addition, you appear to be not using the latest version of the MuleSoft S3 connector. I recommend updating to the latest version for optimal performance. You can check the release notes for the Amazon S3 Connector: Amazon S3 Connector Release Notes - Mule 4

Also sharing the logs as well from the test:

INFO  2023-09-21 12:43:03,513 [[MuleRuntime].uber.10: [s3-test].s3-testFlow.CPU_LITE @4814e4ed] [processor: s3-testFlow/processors/0; event: 4e04e0e0-584e-11ee-843a-e073e72e538f] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: ************* Before List Object *********
INFO  2023-09-21 12:43:04,136 [[MuleRuntime].uber.10: [s3-test].s3-testFlow.CPU_INTENSIVE @3c4fbfd2] [processor: s3-testFlow/processors/3; event: 4e04e0e0-584e-11ee-843a-e073e72e538f] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: [
  "1 - Copy (10).png",
  "1 - Copy (100).png",
  "1 - Copy (101).png",
  "1 - Copy (102).png",
  "1 - Copy (103).png",
  "1 - Copy (104).png",
  "1 - Copy (105).png",
  "1 - Copy (106).png",
  "1 - Copy (107).png",
  "1 - Copy (108).png",
  "1 - Copy (109).png",
  "1 - Copy (11).png",
  "1 - Copy (110).png",
  "1 - Copy (111).png",
  "1 - Copy (112).png",
  "1 - Copy (113).png",
  "1 - Copy (114).png",
  "1 - Copy (115).png",
  "1 - Copy (116).png",
  "1 - Copy (117).png",
  "1 - Copy (118).png",
  "1 - Copy (119).png",
  "1 - Copy (12).png",
  "1 - Copy (120).png",
  "1 - Copy (121).png",
  "1 - Copy (122).png",
  "1 - Copy (123).png",
  "1 - Copy (124).png",
  "1 - Copy (125).png",
  "1 - Copy (126).png",
  "1 - Copy (127).png",
  "1 - Copy (128).png",
  "1 - Copy (129).png",
  "1 - Copy (13).png",
  "1 - Copy (130).png",
  "1 - Copy (131).png",
  "1 - Copy (132).png",
  "1 - Copy (133).png",
  "1 - Copy (134).png",
  "1 - Copy (135).png",
  "1 - Copy (136).png",
  "1 - Copy (137).png",
  "1 - Copy (138).png",
  "1 - Copy (139).png",
  "1 - Copy (14).png",
  "1 - Copy (140).png",
  "1 - Copy (141).png",
  "1 - Copy (142).png",
  "1 - Copy (143).png",
  "1 - Copy (144).png",
  "1 - Copy (145).png",
  "1 - Copy (146).png",
  "1 - Copy (147).png",
  "1 - Copy (148).png",
  "1 - Copy (149).png",
  "1 - Copy (15).png",
  "1 - Copy (150).png",
  "1 - Copy (151).png",
  "1 - Copy (152).png",
  "1 - Copy (153).png",
  "1 - Copy (154).png",
  "1 - Copy (155).png",
  "1 - Copy (156).png",
  "1 - Copy (157).png",
  "1 - Copy (158).png",
  "1 - Copy (159).png",
  "1 - Copy (16).png",
  "1 - Copy (160).png",
  "1 - Copy (161).png",
  "1 - Copy (162).png",
  "1 - Copy (163).png",
  "1 - Copy (164).png",
  "1 - Copy (165).png",
  "1 - Copy (166).png",
  "1 - Copy (167).png",
  "1 - Copy (168).png",
  "1 - Copy (169).png",
  "1 - Copy (17).png",
  "1 - Copy (170).png",
  "1 - Copy (171).png",
  "1 - Copy (172).png",
  "1 - Copy (173).png",
  "1 - Copy (174).png",
  "1 - Copy (175).png",
  "1 - Copy (176).png",
  "1 - Copy (177).png",
  "1 - Copy (178).png",
  "1 - Copy (179).png",
  "1 - Copy (18).png",
  "1 - Copy (180).png",
  "1 - Copy (181).png",
  "1 - Copy (182).png",
  "1 - Copy (183).png",
  "1 - Copy (184).png",
  "1 - Copy (185).png",
  "1 - Copy (186).png",
  "1 - Copy (187).png",
  "1 - Copy (188).png",
  "1 - Copy (189).png",
  "1 - Copy (19).png",
  "1 - Copy (190).png",
  "1 - Copy (191).png",
  "1 - Copy (192).png",
  "1 - Copy (193).png",
  "1 - Copy (194).png",
  "1 - Copy (195).png",
  "1 - Copy (196).png",
  "1 - Copy (197).png",
  "1 - Copy (198).png",
  "1 - Copy (199).png",
  "1 - Copy (2).png",
  "1 - Copy (20).png",
  "1 - Copy (200).png",
  "1 - Copy (201).png",
  "1 - Copy (202).png",
  "1 - Copy (203).png",
  "1 - Copy (204).png",
  "1 - Copy (205).png",
  "1 - Copy (206).png",
  "1 - Copy (207).png",
  "1 - Copy (208).png",
  "1 - Copy (209).png",
  "1 - Copy (21).png",
  "1 - Copy (210).png",
  "1 - Copy (211).png",
  "1 - Copy (212).png",
  "1 - Copy (213).png",
  "1 - Copy (214).png",
  "1 - Copy (215).png",
  "1 - Copy (216).png",
  "1 - Copy (217).png",
  "1 - Copy (218).png",
  "1 - Copy (219).png",
  "1 - Copy (22).png",
  "1 - Copy (220).png",
  "1 - Copy (221).png",
  "1 - Copy (222).png",
  "1 - Copy (223).png",
  "1 - Copy (224).png",
  "1 - Copy (225).png",
  "1 - Copy (226).png",
  "1 - Copy (227).png",
  "1 - Copy (228).png",
  "1 - Copy (229).png",
  "1 - Copy (23).png",
  "1 - Copy (230).png",
  "1 - Copy (231).png",
  "1 - Copy (232).png",
  "1 - Copy (233).png",
  "1 - Copy (234).png",
  "1 - Copy (235).png",
  "1 - Copy (236).png",
  "1 - Copy (237).png",
  "1 - Copy (238).png",
  "1 - Copy (239).png",
  "1 - Copy (24).png",
  "1 - Copy (240).png",
  "1 - Copy (241).png",
  "1 - Copy (242).png",
  "1 - Copy (243).png",
  "1 - Copy (244).png",
  "1 - Copy (245).png",
  "1 - Copy (246).png",
  "1 - Copy (247).png",
  "1 - Copy (248).png",
  "1 - Copy (249).png",
  "1 - Copy (25).png",
  "1 - Copy (250).png",
  "1 - Copy (251).png",
  "1 - Copy (252).png",
  "1 - Copy (253).png",
  "1 - Copy (254).png",
  "1 - Copy (255).png",
  "1 - Copy (256).png",
  "1 - Copy (257).png",
  "1 - Copy (258).png",
  "1 - Copy (259).png",
  "1 - Copy (26).png",
  "1 - Copy (260).png",
  "1 - Copy (261).png",
  "1 - Copy (262).png",
  "1 - Copy (263).png",
  "1 - Copy (264).png",
  "1 - Copy (265).png",
  "1 - Copy (266).png",
  "1 - Copy (267).png",
  "1 - Copy (268).png",
  "1 - Copy (269).png",
  "1 - Copy (27).png",
  "1 - Copy (270).png",
  "1 - Copy (271).png",
  "1 - Copy (272).png",
  "1 - Copy (273).png",
  "1 - Copy (274).png",
  "1 - Copy (275).png",
  "1 - Copy (276).png",
  "1 - Copy (277).png",
  "1 - Copy (278).png",
  "1 - Copy (279).png",
  "1 - Copy (28).png",
  "1 - Copy (280).png",
  "1 - Copy (281).png",
  "1 - Copy (282).png",
  "1 - Copy (283).png",
  "1 - Copy (284).png",
  "1 - Copy (285).png",
  "1 - Copy (286).png",
  "1 - Copy (287).png",
  "1 - Copy (288).png",
  "1 - Copy (289).png",
  "1 - Copy (29).png",
  "1 - Copy (290).png",
  "1 - Copy (291).png",
  "1 - Copy (292).png",
  "1 - Copy (293).png",
  "1 - Copy (294).png",
  "1 - Copy (295).png",
  "1 - Copy (296).png",
  "1 - Copy (297).png",
  "1 - Copy (298).png",
  "1 - Copy (299).png",
  "1 - Copy (3).png",
  "1 - Copy (30).png",
  "1 - Copy (300).png",
  "1 - Copy (301).png",
  "1 - Copy (302).png",
  "1 - Copy (303).png",
  "1 - Copy (304).png",
  "1 - Copy (305).png",
  "1 - Copy (306).png",
  "1 - Copy (307).png",
  "1 - Copy (308).png",
  "1 - Copy (309).png",
  "1 - Copy (31).png",
  "1 - Copy (32).png",
  "1 - Copy (33).png",
  "1 - Copy (34).png",
  "1 - Copy (35).png",
  "1 - Copy (36).png",
  "1 - Copy (37).png",
  "1 - Copy (38).png",
  "1 - Copy (39).png",
  "1 - Copy (4).png",
  "1 - Copy (40).png",
  "1 - Copy (41).png",
  "1 - Copy (42).png",
  "1 - Copy (43).png",
  "1 - Copy (44).png",
  "1 - Copy (45).png",
  "1 - Copy (46).png",
  "1 - Copy (47).png",
  "1 - Copy (48).png",
  "1 - Copy (49).png",
  "1 - Copy (5).png",
  "1 - Copy (50).png",
  "1 - Copy (51).png",
  "1 - Copy (52).png",
  "1 - Copy (53).png",
  "1 - Copy (54).png",
  "1 - Copy (55).png",
  "1 - Copy (56).png",
  "1 - Copy (57).png",
  "1 - Copy (58).png",
  "1 - Copy (59).png",
  "1 - Copy (6).png",
  "1 - Copy (60).png",
  "1 - Copy (61).png",
  "1 - Copy (62).png",
  "1 - Copy (63).png",
  "1 - Copy (64).png",
  "1 - Copy (65).png",
  "1 - Copy (66).png",
  "1 - Copy (67).png",
  "1 - Copy (68).png",
  "1 - Copy (69).png",
  "1 - Copy (7).png",
  "1 - Copy (70).png",
  "1 - Copy (71).png",
  "1 - Copy (72).png",
  "1 - Copy (73).png",
  "1 - Copy (74).png",
  "1 - Copy (75).png",
  "1 - Copy (76).png",
  "1 - Copy (77).png",
  "1 - Copy (78).png",
  "1 - Copy (79).png",
  "1 - Copy (8).png",
  "1 - Copy (80).png",
  "1 - Copy (81).png",
  "1 - Copy (82).png",
  "1 - Copy (83).png",
  "1 - Copy (84).png",
  "1 - Copy (85).png",
  "1 - Copy (86).png",
  "1 - Copy (87).png",
  "1 - Copy (88).png",
  "1 - Copy (89).png",
  "1 - Copy (9).png",
  "1 - Copy (90).png",
  "1 - Copy (91).png",
  "1 - Copy (92).png",
  "1 - Copy (93).png",
  "1 - Copy (94).png",
  "1 - Copy (95).png",
  "1 - Copy (96).png",
  "1 - Copy (97).png",
  "1 - Copy (98).png",
  "1 - Copy (99).png",
  "1 - Copy.png",
  "1.png"
]

Thank you!