MarkLogic - XDMP-NEWSTAMP exception

Question

We are using MarkLogic-8(with three nodes - two forests on each) and have been facing the XDMP-NEWSTAMP exception frequently. We have the default merge policy and do not use any point in time queries. But we do use xdmp:eval and xdmp:invoke-fuction(being used a lot in code) to avoid locks on documents only being read(query mode) in update transactions.

There is not very comprehensive information related to XDMP-NEWSTAMP in MarkLogic docs either, other than XDMP-NEWSTAMP and a mention in App developer guide's point in time query section; have gone through both but neither helps.

Please help me understand this exception (and share if any docs have details related to this). Below is a log snippet for reference:

2020-07-16 03:07:31.712 Warning: Forest XXX-YY-003 fast query timestamp (15948687770144645) lags commit timestamp (15948688205890165) by 43574 ms
2020-07-16 03:08:03.583 Warning: Forest XXX-YY-003 fast query timestamp (15948687770144645) lags commit timestamp (15948688803306468) by 103316 ms
2020-07-16 03:08:03.632 Info: Merging 2 MB from G:\Forests\xxxx03\000048af to G:\Forests\xxxx03\000048b1, timestamp=15948682803306468
2020-07-16 03:08:03.933 Debug: OnDiskStand G:\Forests\xxxx03\000048b1, disk=3MB, memory=1MB
2020-07-16 03:08:03.934 Info: Merged 3 MB at 10 MB/sec to G:\Forests\xxxx03\000048b1
2020-07-16 03:08:03.960 Debug: Forest xxxx03 setting minQueryTimestamp to 15948682803306468 due to merge
2020-07-16 03:08:04.936 Debug: ~OnDiskStand G:\Forests\xxxx03\000048af
2020-07-16 03:08:07.166 Info: Deleted 2 MB at 703 MB/sec G:\Forests\xxxx03\000048af
2020-07-16 03:08:17.617 Debug: Forest XXX-YY-003 participant 1232761274262892690 not found in participantBumpMinCommitTimestamp()

2020-07-16 03:22:23.750 Info: XXX-WorkApi: Status 500: XDMP-NEWSTAMP: Timestamp too new for forest XXX-YY-003 (15948695621461959)
2020-07-16 03:22:23.750 Info: XXX-WorkApi: Status 500: XDMP-NEWSTAMP: Timestamp too new for forest XXX-YY-003 (15948695621461959)

Answer 1

This may be due to some long-running queries.

Prior to the XDMP-NEWSTAMP there were "fast query timestamp" messages logged for that forest: https://help.marklogic.com/knowledgebase/article/View/437/0/warning-messages-for-lagging-operations

Every forest has a notion of a "fast query timestamp", also sometimes referred to as a "nonblocking timestamp". This is the maximum timestamp at which a query can run without waiting for the forest's timestamp to advance; it indicates the most current time at which the forest has complete state to answer a query. There are several reasons for forests to have this timestamp.

The first has to do with transaction commits, during which the forest places a finger on the commit timestamp for the duration of the commit. The point of this is to ensure that queries perceive committed transactions to be atomic. There can be multiple (even many) transactions with a finger on various timestamps at any given point in time.

This warning will help flag any problems with overly long transactions that can hold up queries. The warning helps flag the lag issue earlier, rather than later.

Answer 2

Changed the App server config distribute timestamps to strict (earlier it was set to fast) and now the occurrences of XDMP-NEWSTAMP have reduced significantly. Can there be any possible negative effects due to this change; Increased network load is one that I am expecting. can there be any other?