RocketChat Retention policy does not free disk space

Question

i'm using RocketChat 6.5.3 (mongo wiredTiger), installed manually on Ubuntu 20.04 LTS, NodeJS 14.21.3, MongoDB 5.0.24.

Retention policy enabled to 180 days and ALL checkboxes are on also – i want to just delete old uploads to save disk space but keep messaging history. If there is a way to fix that 57Gb again to 14Gb then i don't need retention policy at all.

After turning on retention, i see empty messages in the chat history instead of uploads (it's ok) but disk space remains exactly the same even after restart mongo/rocket.

Installation is pretty old (first message was in 2020 with 3.x version) and it took only 14 Gb of space before upgrade. I did upgrade to the latest version (6.5.3) early January this year and now (after a month) it takes whooping 57 Gb with the same users and same activity, which is also too weird for me. I'm using MongoDB only for RocketChat, so there's no other applications that might affect this.

I've also checked collection sizes and got this:

use rocketchat
db.getCollectionNames().forEach((cname) => {
    const csize = db.getCollection(cname).stats(1024 * 1024 * 1024).storageSize;
    if (csize > 1) print(cname + " = " + csize);
});

rocketchat_userDataFiles.chunks = 30.662113189697266
rocketchat_uploads.chunks = 13.659313201904297

But in my disk i have more weird files:

find /var/lib/mongodb/ -type f -size +1G -exec du -h {} +

31G /var/lib/mongodb/collection-766--3268902447285908541.wt
13G /var/lib/mongodb/collection-14-7180290947299615862.wt
14G /var/lib/mongodb/collection-0-5684180545419248738.wt

I've tried to use compact method of MongoDb with no luck:

use rocketchat
db.getCollectionNames().forEach(function (collectionName) {
    print('Compacting: ' + collectionName);
    if ('system.views' == collectionName) return; // This is required because otherwise i'll get an error
    db.runCommand({ compact: collectionName, force: true });
});

Compacting: rocketchat_apps_packages.files
Compacting: rocketchat_banner_dismiss
Compacting: rocketchat_cron
Compacting: rocketchat_export_operations
Compacting: rocketchat_moderation_reports
Compacting: usersSessions
Compacting: rocketchat_video_conference
Compacting: rocketchat_oauth_access_tokens
Compacting: instances
Compacting: rocketchat_import
Compacting: rocketchat_sessions
Compacting: rocketchat_apps_packages.chunks
Compacting: rocketchat_message_reads
Compacting: rocketchat_apps_persistence
Compacting: pbx_events
Compacting: rocketchat_omnichannel_service_level_agreements
Compacting: rocketchat_uploads
Compacting: rocketchat_matrix_bridged_rooms
Compacting: migrations
Compacting: rocketchat_message
Compacting: rocketchat_avatars.files
Compacting: rocketchat_email_inbox
Compacting: rocketchat_livechat_page_visited
Compacting: omnichannel_scheduler
Compacting: rocketchat_room
Compacting: rocketchat_federation_room_events
Compacting: rocketchat_statistics
Compacting: rocketchat_team_member
Compacting: rocketchat_subscription
Compacting: rocketchat_raw_imports
Compacting: rocketchat_user_data_files
Compacting: rocketchat_message_read_receipt
Compacting: rocketchat_credential_tokens
Compacting: users
Compacting: rocketchat_cron_history
Compacting: rocketchat_read_receipts
Compacting: _queue
Compacting: rocketchat_uploads.files
Compacting: rocketchat_webdav_accounts
Compacting: rocketchat_apps_logs
Compacting: rocketchat_nps
Compacting: rocketchat_permissions
Compacting: rocketchat__trash
Compacting: rocketchat_userDataFiles.chunks
Compacting: rocketchat_integrations
Compacting: meteor_accounts_loginServiceConfiguration
Compacting: omnichannel_queue_inactivity_monitor
Compacting: rocketchat_email_message_history
Compacting: ufsTokens
Compacting: rocketchat_livechat_unit_monitors
Compacting: rocketchat_team
Compacting: rocketchat_smarsh_history
Compacting: rocketchat_avatars
Compacting: rocketchat_banner
Compacting: rocketchat_livechat_custom_field
Compacting: omnichannel_auto_close_on_hold_scheduler
Compacting: rocketchat_apps_scheduler
Compacting: rocketchat_livechat_business_hours
Compacting: meteor_oauth_pendingRequestTokens
Compacting: rocketchat_oembed_cache
Compacting: rocketchat_federation_servers
Compacting: rocketchat_custom_sounds
Compacting: system.views
Compacting: rocketchat_livechat_department
Compacting: rocketchat_canned_response
Compacting: meteor_oauth_pendingCredentials
Compacting: rocketchat_oauth_auth_codes
Compacting: rocketchat_oauth_refresh_tokens
Compacting: rocketchat_livechat_tag
Compacting: rocketchat_server_events
Compacting: rocketchat_avatars.chunks
Compacting: rocketchat_settings
Compacting: rocketchat_integration_history
Compacting: rocketchat_custom_user_status
Compacting: rocketchat_uploads.chunks
Compacting: rocketchat_livechat_agent_activity
Compacting: rocketchat_import_data
Compacting: rocketchat_livechat_department_agents
Compacting: rocketchat_nps_vote
Compacting: rocketchat_livechat_visitor
Compacting: rocketchat_notification_queue
Compacting: rocketchat_reports
Compacting: rocketchat_livechat_inquiry
Compacting: rocketchat_livechat_external_message
Compacting: rocketchat_oauth_apps
Compacting: rocketchat_analytics
Compacting: rocketchat_livechat_priority
Compacting: _raix_push_app_tokens
Compacting: rocketchat_federation_keys
Compacting: rocketchat_roles
Compacting: rocketchat_calendar_event
Compacting: rocketchat_userDataFiles.files
Compacting: rocketchat_matrix_bridged_users
Compacting: rocketchat_federation_dns_cache
Compacting: rocketchat_livechat_trigger
Compacting: rocketchat_custom_emoji
Compacting: rocketchat_apps
Compacting: rocketchat_invites

ps: i made an issue in GitHub about it but have no answers so far so i thought maybe i can reach bigger audience in SO: https://github.com/RocketChat/Rocket.Chat/issues/31643

Answer 1

Here is a solution you may probably like:

1. Convert your single mongoDB to replicaSet with 1x member as described here
2. Add hidden member to the replicaSet ( it will take some minutes the member to replicate the data and re-create indexes )
3. Check the size in the new member ( it is expected to be shrinked and pre-allocated space to be released , wiredTiger compress the data on storage , but still pre-allocate some space so it is not released immediately after deletion )
4. Switch PRIMARY to the new member and switch your original member to HIDDEN.
5. Init sync your original member as described here so it shrink to the same size.
6. You can switch back your original member to PRIMARY and remove the new member from the replicaSet or you can keep your new member as a backup member in case something go wrong with your original member again.