Testng XML not accepted by AWS Device Farm

Question

I'm trying to use the testng.xml... and looks like Device Farm is ignoring the whole file.

My example is simple. I have a factory class that instantiate test classes, and this is how my xml file looks like

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="Factory"/>
        </classes>
    </test>
</suite>

I even tried to exclude methods manually

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="Factory"/>
            <class name="TestSingleUrl">
                    <methods>
                        <exclude name="testCorrectSiteKey"/>
                        <exclude name="testIsTagImplemented"/>
                        <exclude name="testHttpsScheme"/>
                        <exclude name="testCorrectTagName"/>
                        <exclude name="testCorrectBootstrapPath"/>
                    </methods>
            </class>
        </classes>
    </test>
</suite>

But I'm getting a "Can't invoke method: either make it static or create a non-args constructor" . Which means that the Device Farm is trying to run the methods from TestSingleUrl Class..But the one who call these test should be the factory

Does anyone know how can we make the Device Farm to accept our xml file?

Comments

This is my Factory Class:

public class Factory{

    private String targetSiteKey;

    //Data Provider that consumes the first line, which contains the target sitekey of the tag
    @DataProvider(name = "urlProvider")
    public Iterator<Object[]> createData() throws IOException {
        List<String> lines = IOUtils.readLines(new InputStreamReader(this.getClass().getResourceAsStream("Urls.csv")));
        List<Object[]> objectArrays = lines.stream().map(x -> new Object[]{x}).collect(Collectors.toList());

        Iterator<Object[]> itr = objectArrays.iterator();
        targetSiteKey = itr.next()[0].toString();

        return itr;
    }

    @Factory(dataProvider="urlProvider")
    public Object[] creatingTests(String url){
        System.out.println(System.currentTimeMillis());
        String siteKey=null;
        String path = null;
        String tagName = null;
        WebElement browsi;
        try{
            RemoteWebDriver driver = DriverWrapper.getDriver();
            driver.get(url);
            browsi = driver.findElement(By.id("browsi-tag"));

            tagName = browsi.getTagName();
            siteKey = browsi.getAttribute("data-sitekey");
            path = browsi.getAttribute("src");

        }catch(Exception ignored){}
        finally{
            return new Object[] {new TestSingleUrl(targetSiteKey,siteKey,path,tagName,url)};
        }
    }
}

And this is my TestSingleUrl Class:

public class TestSingleUrl {
    private String targetSiteKey,siteKey,path,tagName,url;

    public TestSingleUrl(String targetSiteKey,String siteKey, String path, String tagName,String url){
        this.targetSiteKey = targetSiteKey;
        this.siteKey=siteKey;
        this.path=path;
        this.tagName=tagName;
        this.url=url;
    }

    @Test
    public void testCorrectSiteKey(){
        System.out.println(url);
        Assert.assertEquals(siteKey, targetSiteKey);
    }

    @Test
    public void testIsTagImplemented(){
        System.out.println(url);
        Assert.assertFalse(siteKey.isEmpty());
        Assert.assertFalse(tagName.isEmpty());
        Assert.assertFalse(path.isEmpty());
    }

    @Test
    public void testHttpsScheme(){
        System.out.println(url);
        Assert.assertTrue(path.contains("https"));
    }

    @Test
    public void testCorrectTagName(){
        System.out.println(url);
        Assert.assertEquals(tagName,"script");
    }

    @Test
    public void testCorrectBootstrapPath(){
        System.out.println(url);
        Assert.assertTrue(path.contains("middycdn-a.akamaihd.net/bootstrap/bootstrap.js"));
    }
}

I think that the error messages I'm getting is because the device farm is trying to run every method with a @Test annotation that it finds. If it was reading my xml file it wouldn't happen.

Answer 1

I'm not sure exactly but I believe you should be able to place the testng.xml in a resources directory and reference it from the pom.xml .

Looks like this post shows how to reference the testng.xml from the pom.xml: How to call testng.xml file from pom.xml in Maven

When you build the project then it will be included in the jar.

The awslabs project builds it by using this mvn command

mvn clean package -DskipTests=true

[Update] I downloaded the sample project from awslabs and imported it into eclipse. I then added this plugin to the pom.xml file:

  <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-surefire-plugin</artifactId>
           <version>2.12.4</version>
           <configuration>
              <suiteXmlFiles>
                 <suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
              </suiteXmlFiles>
           </configuration>
        </plugin>     

I then created the testng.xml in ./src/test/resources/

I copied and pasted the xml provided here:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="Factory"/>
        </classes>
    </test>
</suite>

I then built the project using

mvn clean package -DskipTests=true

This produced a zip-with-dependencies.zip file in the target directory.

I uploaded this test package to device farm by creating a new project and selecting Appium Java Testng as the type of tests that I want to run.

I encountered this parse error:

Failed to generate the test suites[TestNG] [ERROR] Cannot find class in classpath: Factory

Which is expected since I don't have that class in my classpath, however, this proves that Device Farm did read the testng.xml.

[update]

I am able to reproduce the problem when I use the exclude tag in the testng.xml. I.E when I use this xml:

 <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="factory.TestSingleUrl">
                <methods>
                    <exclude name="testCorrectSiteKey" />
                    <exclude name="testIsTagImplemented" />
                    <exclude name="testHttpsScheme" />
                    <exclude name="testCorrectTagName" />
                    <exclude name="testCorrectBootstrapPath" />
                </methods>
            </class>
        </classes>
    </test>
</suite>

Going to investigate further on the method tags.

[update]

Looks like this is a known issue in Device Farm: AWS Device farm seems to be ignoring TestNG annotations

Guess we will just need to wait for them to update their environment and the parser.

You might be able to run your tests on a private device. You'll need to contact aws to get one at this email:

aws-devicefarm-support@amazon.com

[Update]

Looks like they've updated the webDriverAgent. See if it works now.

Hope that helps.