Unable to see Set Test Message log value after test case marked as fail

Question

I have the below code:

*** Settings ***
Library     OperatingSystem
Library     Process
Library     String


*** Variables ***
@{MyList}=   item   items   items2
${LogStr1}   *HTML*

*** Test Cases *** 
#Start Test#
[xxxxx] My tests
    FOR  ${item}  IN   @{MyList}
        General                test.out         testProfile          ${item}
        [Template]    Run Test                                                                                                                                                         
        [Tags]    TestTags
    END

*** Keywords *** 
Run Test
    [Documentation]    Run the test
    [Arguments]       ${type}     ${profile}     ${file}    ${test}
            When suite config is updated 
            And updated the config in directory ${test}
            Then publish test status

suite config is updated
           [Documentation]    Get the variables list
            Log to Console     "Updating get suite config file"


updated the config in directory ${test}
    [Documentation]    Get the variables list 
         Run keyword if      "${test}" == "items"    Stop Test      "This is stop called"

publish test status 
    [Documentation]    Create and check if any issue found
            ${LogStr}=     Catenate     Test Passed : Log created:     <a href="Hello.txt">Hello</a>
            Log to Console       ${LogStr}
            ${LogStr1}=     Catenate         ${LogStr1}              ${LogStr}\n
            Set Test Variable    ${LogStr1}
            Set Test Message     ${LogStr1}

Stop Test  
    [Documentation]    Stop Execution 
    [Arguments]       ${FIALUREMSG}        
    Log To Console    ${FIALUREMSG}
    ${LogStr1}=     Catenate         ${LogStr1}              ${FIALUREMSG} 
      Fail        ${LogStr1}
  

As per the code the test can be pragmatically made to fail at 1st second or third run. So when I have the code like:

Run keyword if      "${test}" == "item"    Stop Test      "This is stop called"

in mentioned keyword, there are 2 test cases that passes for a suite but report states:

Now if I make the second test case to fail I get below test message logs:

Run keyword if      "${test}" == "items"    Stop Test      "This is stop called"

in mentioned keyword, there are 2 test cases that passes for a suite but report states:

Similarly if

 Run keyword if      "${test}" == "items2"    Stop Test      "This is stop called"

And so on - hence it seems that "Set Test Message" log values are ignored in report message when a test case is marked as Fail. Note that below is the log.html content when I ran the code to mark the very first test case as failure:

Run keyword if      "${test}" == "item"    Stop Test      "This is stop called"

In all my question is thus that if I want report.html file to show logs for all fail and passed test cases how can I achieve it?

Answer 1

If you check the documentation of the Set Test Message keyword it says that any failure will override the messages, but you have the option to override the failure message from the teardown.

In test teardown this keyword can alter the possible failure message, but otherwise failures override messages set by this keyword. Notice that in teardown the message is available as a built-in variable ${TEST MESSAGE}.

So what you can do is instead of calling Set Test Message, you could just save the messages into a test variable. Then you should add a teardown in which you call the Set Test Message and concatenate your test variable with the ${TEST MESSAGE}. For example:

*** Test Cases ***
Test
    [template]    Template
    [setup]    Set Test Variable    ${MSG}    ${EMPTY}    # create empty test variable to store messages
    1
    3
    2
    5
    6
    4
    6
    [teardown]    Set Test Message    ${MSG}\n${TEST MESSAGE}    # concatenate failure messages to normal test messages


*** Keywords ***
Template
    [arguments]    ${number}
    No Operation
    Run Keyword If    ${number} == 2    Fail      fail message ${number}
    Run Keyword If    ${number} == 4    Fail      fail message ${number}
    Set Test Variable    ${MSG}    ${MSG}\nMy test message ${number}    # concatenate next test message

This example produces the following report:

With this approach you could have only the template tests in the particular robot suite as all test listed in the Test Cases table will invoke the template.

---

---

Other, completely different solution can be to get rid of the FOR loop as elements in @{MyList} are static. If you move the template into the Settings table and then manually list all iterations, you could separate each one of them into an independent test case. This way failure in one iteration won't affect the test message set in an other iteration. For example:

*** Settings ***
Test Template     Template

*** Test Cases ***
Test1    1
Test2    2
Test3    3
Test4    4
Test5    5
Test6    6


*** Keywords ***
Template
    [arguments]    ${number}
    No Operation
    Run Keyword If    ${number} == 2    Fail      fail message ${number}
    Run Keyword If    ${number} == 4    Fail      fail message ${number}
    Set Test Message    My test message ${number}

This would produce the following report:

Answer 2

You have a third option in addition to my other answer but it is a bit more advanced solution so I have decided to post it as a separate answer.

You can write a small test library in Python that would also act as a listener. In this listener library you would need three functions out of which two would be keyword and one would be a listener function.

1. First function is _log_message which is described in the listener API 2. This function will be called by the framework if any logging happens. So when a test fails this keyword will receive the log entry of the failure which can be saved for later use. This function, as it starts with _ is not available in a robot suite as keyword.

   def _log_message(self, message):
       if message['level'] == 'FAIL':
           self.test_message = f"{self.test_message}\n{message['message']}"    # concatenate failure message
   

2. Second function add_test_message will replace the Set Test Message keyword in your code. Its purpose will be similar, which is to append the messages you want to set as test messages. This can be called from a robot suite as a keyword.

   def add_test_message(self, message):
       self.test_message = f"{self.test_message}\n{message}"    # concatenate normal test message
   

3. The last function set_final_test_message will set the actual test message for you. This keyword has to be called at the end of the test teardown to ensure no other failure will override your test message. It simply calls the Set Test Message keyword internally and sets the string created by the previous two functions. This can be called from a robot suite as a keyword.

   def set_final_test_message(self):
       """
       Call this keyword at the end of the test case teardown.
   
       This keyword can only be used in a test teardown.
       """
       BuiltIn()._get_test_in_teardown('Set Final Test Message')    # Check if we are in the test teardown, fail if not.
       BuiltIn().set_test_message(self.test_message)    # Call Set Test Message internally
   

As the purpose of the library is to set test messages the library scope should be TEST CASE. This means that a new library object will be created before every test case, effectively resetting any messages set by previous tests.

Here is the whole code of the library (TestMessageLibrary.py):

from robot.libraries.BuiltIn import BuiltIn

class TestMessageLibrary(object):
    ROBOT_LIBRARY_SCOPE = 'TEST CASE'    # define library scope
    ROBOT_LISTENER_API_VERSION = 2    # select listener API
    ROBOT_LIBRARY_VERSION = 0.1
    
    def __init__(self):
        self.ROBOT_LIBRARY_LISTENER = self    # tell the framework that it will be a listener library
        self.test_message = ''    # internal variable to build the final test message

    def _log_message(self, message):
        if message['level'] == 'FAIL':
            self.test_message = f"{self.test_message}\n{message['message']}"    # concatenate failure message
    
    def add_test_message(self, message):
        self.test_message = f"{self.test_message}\n{message}"    # concatenate normal test message

    def set_final_test_message(self):
        """
        Call this keyword at the end of the test case teardown.
        
        This keyword can only be used in a test teardown.
        """
        BuiltIn()._get_test_in_teardown('Set Final Test Message')    # Check if we are in the test teardown, fail if not.
        BuiltIn().set_test_message(self.test_message)
        
globals()[__name__] = TestMessageLibrary

This is an example suite with the library:

*** Settings ***
Library    TestMessageLibrary 

*** Test Cases ***
Test
    [template]    Template
    1
    3
    2
    5
    6
    4
    6
    [teardown]    Set Final Test Message
    
Other Test
    Add Test Message    This should not conflict
    [teardown]    Set Final Test Message


*** Keywords ***
Template
    [arguments]    ${number}
    No Operation
    Run Keyword If    ${number} == 2    Fail      fail message ${number}
    Run Keyword If    ${number} == 4    Fail      fail message ${number}
    Add Test Message    My test message ${number}

Example run called with robot --pythonpath ./ SO.robot. As the library is in the same directory as the suite file --pythonpath ./ was need to be able to import the library.

Report file: