I am trying to set a logical replica of Postgres 14 DB. One of the tables from the source DB is about 2.5 TB. This table is (obviously) taking a long time to catch up with the source DB BUT even when I see that the table size on the destination DB has increased to around 850 GB the table is still inaccessible to read. Is this behavior expected? And, how can I help the logical replication to synchronize the data faster?
Thanks for your help in advance.
edit: adding the output of top on the receiving DB.
The original bulk copy needs to finish in one transaction, or it is lost. Starting on v14, the bulk COPY commits when it is done, and then if the system encounters a problem after that only the later steps need to be repeated. Before 14, the COPY would be lost if any later step in the sync process failed.
(You should edit your question to include extra information, with proper formatting, rather than put it in the comments where formatting is not possible.) It looks like your %CPU lines show that those processes are very busy on the CPU, so they are evidently not entirely bottlenecked by IO waits. Usually when large loads into an indexed table start fast but then slow down, it is because the index maintenance starts being bottlenecked by IO once the index is large enough to no longer fit in shared_buffers. Since you aren't bottlenecked, I'm not really sure what that means. You could try dropping the indexes, but alas that would require you interrupt the bulk COPY and repeat it all.
If you drop the replication identity index, then once the COPY is done and it starts trying to replay any UPDATEs or DELETEs which happened since the COPY had started, those will start to fail. Before v14, that was a disaster as the COPY itself was then rolled back. Since v14, you have an opportunity to build the index after the COPY is done and it will then restart the replay.