[FIXED] single soft assertion error is failing subsequent passed scenarios

Issue

I am using soft assertion in my test class and getting test data from data provider. Below is the sample code:

@Test(dataProvider = "userPages")
public void validatePages(String pageName) {
    try {
        Reporter.log("Logged In User : " + userType, true);
        //Page1 takes extra checks
        if (pageName.equalsIgnoreCase("Page1")) {
            System.out.println("Running in Page1 bloc ");
            softAssert.assertEquals(page.isValidPage1(), true,
                    "Failed to loada : " + pageName);
        } else {
            System.out.println("Running in NON Page1 bloc .. Page Name is :: "+pageName);
            softAssert.assertEquals(page.isValid(), true,
                    "Failed to load : " + pageName);
        }
        softAssert.assertAll();  ---> Appends page1 failures to all the subsequent passed page tests thus marking all of them failed.
    } catch (InterruptedException e) {
        Reporter.log("Test failed for : " + pageName);
    }
    //
}

TestNG dependency is ::

 <dependency>
        <groupId>org.testng</groupId>
        <artifactId>testng</artifactId>
        <version>${testng.version}</version>
  </dependency>

In my case, if a single page fails then all the subsequent pages are marked as failed even if they are passed. I have tried to debug the issue and found that, in SoftAssert class's assertAll() method, the error message of previously failed test-data is getting appended

public void assertAll() {
if (!m_errors.isEmpty()) {
  StringBuilder sb = new StringBuilder("The following asserts failed:");
  boolean first = true;
  for (Map.Entry<AssertionError, IAssert<?>> ae : m_errors.entrySet()) {
    if (first) {
      first = false;
    } else {
      sb.append(",");
    }
    sb.append("\n\t");
    sb.append(ae.getKey().getMessage());
  }
  throw new AssertionError(sb.toString());
}}

This issue is marking false errors in the testNG reports. Please help and advice.

Edit1:

Data Provider

@DataProvider(name = "userPageNames")
public Object[] creatingObjects() {
    Map<String, String> pagesList = null;
    try {
        pagesList = //Populate Map

    } catch (Exception e) {
        Reporter.log("Failed to get Pages List for " + userType, true);
    }

    Object[] object = new Object[pagesList.keySet().size()];
    Integer i = 0;
    for (String pageName : pagesList.keySet()) {
        object[i] = pageName;
        i++;
    }
    return object;
}

Report: Genuine Failure

Logged In User : user1
Running in Page1 bloc ..
Page verified : Page1

java.lang.AssertionError: The following asserts failed:
Failed to loada : Page1 expected [true] but found [false]
Expected :true
Actual   :false



    at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:43)
    at web.dishaPages.ValidatePagesTestNew.validatePages(ValidatePagesTestNew.java:88)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:567)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
    at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:230)
    at org.springframework.test.context.testng.AbstractTestNGSpringContextTests.run(AbstractTestNGSpringContextTests.java:175)
    at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:242)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:576)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at org.testng.TestRunner.privateRun(TestRunner.java:648)
    at org.testng.TestRunner.run(TestRunner.java:505)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
    at org.testng.TestNG.runSuites(TestNG.java:1049)
    at org.testng.TestNG.run(TestNG.java:1017)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

Report: Bogus Failures

Logged In User : user1
Running in NON Page1 bloc .. Page Name is :: ViewDMPage
Page verified : ViewDMPage

java.lang.AssertionError: The following asserts failed:
Failed to loada : Page1 expected [true] but found [false]
Expected :true
Actual   :false



    at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:43)
    at web.dishaPages.ValidatePagesTestNew.validatePages(ValidatePagesTestNew.java:88)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:567)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
    at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:230)
    at org.springframework.test.context.testng.AbstractTestNGSpringContextTests.run(AbstractTestNGSpringContextTests.java:175)
    at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:242)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:576)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at org.testng.TestRunner.privateRun(TestRunner.java:648)
    at org.testng.TestRunner.run(TestRunner.java:505)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
    at org.testng.TestNG.runSuites(TestNG.java:1049)
    at org.testng.TestNG.run(TestNG.java:1017)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

Solution

The problem lies in your test code.

Your dataprovider powered @Test method is basically using one SoftAssert instance and then invoking an assertAll() for all data provider data set iterations using the same instance.

SoftAssert is designed to remember all assertions that have been recorded so far via calls to assertXXX.

To fix this problem you should be instantiating the SoftAssert object inside the @Test method

Answered By - Krishnan Mahadevan