Mobile Development: Testing for Multiple Device Configurations

All Android developers should have at least 1 “old” device running OS 2.3.3 and a current “popular” device. Ideally, one should also have a current device that is considered “maxed out” on specs. A company should additionally have the latest “Google” device (currently the Nexus series), and an HTC, Sony, and Samsung device. (These manufacturers are mentioned because of popularity and/or significant differences not found when developing on other devices.) Additionally, OS 4.2, 4.3, and 4.4, though minor OS increments, offer differences that should be considered.

Though development for iPhone/iPad is more forgiving given the fewer configurations, it still offers challenges. For example, if you are developing on a Mac running OS X Mavericks with a version of Xcode above 5.0 for a product that still needs to support iOS 5.x, you will need a physical device because the iOS 5.x simulator isn’t available for that development configuration.

If testing mobile websites, the configurations can be endless.

At Apps World 2014, Perfecto Mobile (http://www.perfectomobile.com) introduced me to mobile cloud device testing. Their product offers access to real devices (not emulators or simulators) connected to actual carriers physically hosted at one of their sites around the world.

The concept of mobile cloud device testing allows the ability to test on a multitude of configurations of devices, locations/timezones, carriers, and operating systems.

Beyond access to multiple devices, Perfecto Mobile offers automation testing across these platforms via scripts written in Java. I wasn’t able to personally delve as far as I wanted into these automation tests, the recording feature, or the object mapper before my trial ran out, but the demo at Apps World gave me the impression it behaves similar to Xcode’s Automation instrument but expanded to all devices.  The scripts enable your team to target certain device configurations and automatically launch, execute the given tests, clean and close the devices, and export the test results to your team.  I wish I could say more because it looked really promising but without actual usage, I can only mention what I viewed during the demo.

It’s impossible to cover every configuration during native Android application development, but after a release, for all platforms, if your product is experiencing issues and a crash report doesn’t reveal enough, mobile cloud device testing offers the a real option for true coverage.

Below is a listing of some features of interest Perfecto Mobile offers:
- Access to real devices connected to actual carriers (not emulators or simulators) physically hosted at one of Perfecto’s sites around the world. Since these are real devices, you can dial numbers, make calls, send text messages, and install apps.
- UI for devices available displays availability, manufacturer, model, os + version, location, network, phone number, device id, firmware, resolution.
- Ability to open multiple devices at the same time.
- Requests for devices and configurations not available are responded to in real-time.
- Ability to take screenshots and record sessions to save and/or share results with others.
- Ability to share the device screen in real-time with your team.
- Ability to collect information for a device such as battery level, CPU, memory, and network activity.
- Export of device logs.
- Beta of MobileCloud for Jenkins plug-in that allows running scripts on actual cloud devices after a build so you can see reports on a single device after a build (multiple devices is not available yet).

Automating your javascript unit tests – Karma

why

  • instant feedback while writing your code and eliminate the need to remember to run tests before checking in your code. thus leading to stable builds.
  • continuos integration with our hudson build system.
  • testing on real and multiple browsers.

what

  • set up karma to watch your source files and run tests on code changes.
  • set up karma to run the tests during hudson build and validate the build.
  • set up karma to track our code coverage.

how

We use mocha as our test framework. This along with chai (expect/should – BDD style) worked out great for us with an effective yet readable tests. I cannot emphasize the importance of readable tests enough. We had a team member who did a feature walk through by running through the tests which i thought was pretty rad. Product and QA could easily see what was the feature set, what was expected of and what was the outcome. I guess we have to do a write up sharing more of our excitement.

Before karma, we were running tests using individual test files. More often, you are working on multiple files and remembering to run tests on all these files manually was becoming cumbersome and error prone. So we started researching on test runners and karma seemed to fit all our necessities: automation, continuos integration, run tests on multiple real browsers and support for mocha.

set up karma to watch your source files and run tests on code changes

This was fairly straight forward. Karma’s setup is driven by a single configuration file where in you provide the location of files you want to watch for changes, browsers that you want to run tests, your testing framework and any preprocessors. Here’s a gist of our configuration file. The only tricky part was preprocessors. We use handlebars along with requirejs-handlebars-plugin for our templating purposes and serve our templates as individual html files. This was causing a problem karma was converting them into js strings because of its default preprocessor: html2js. It needed a bit of reading, but the fix was simple enough. The following additions to the config file fixed the problem.

preprocessors : [{'scripts/**/*.html' : ''}]
files:[...{pattern: 'scripts/**/*.html', served: true, included: false}]

set up karma to run the tests during hudson build and validate the build

We created another karma configuration file for this purpose. We added a junitReporter  so that we could export the tests in a format that could be interpreted by our hudson setup. The key differences are as follows. We are currently using phantomJS for testing on our build systems, but in near future, we want to extend this to real browsers.

reporters: ['progress', 'junit']
junitReporter: {outputFile: "testReports/unit-tests.xml"}
autoWatch: false
browsers: ['PhantomJS']
singleRun: true

set up karma to track our code coverage

Once we were able to configure karma to run in hudson, this was just a natural addition. The only additions to the karma configuration are as follows.

reporters: ['progress', 'junit', 'coverage']
coverageReporter: {
 type : 'cobertura',
 dir : 'coverage/'
}
preprocessors : {
 '**/scripts/**/*.js': 'coverage'
}

As you may have noticed, i may used simple and straight-forward words quite a few times and that is what karmajs is all about.

reads

http://karma-runner.github.io/0.10/index.html

http://googletesting.blogspot.com/2012/11/testacular-spectacular-test-runner-for.html

 

https://www.npmjs.org/package/karma-handlebars-preprocessor

Tips on UI Automation of iOS Applications

Automation support for iOS apps is still a relatively new technology and like all other technologies that are still quite not mature it has a lot of challenges. Below I would like to discuss one of such challenges, identifying and accessing individual custom cells inside a UITableView

Let’s say we create a new phone book application and let’s say that the 3 pieces of information that this application stores are: the last and the first name of the contact along with the type of contact that this person is (family, friend, work). The contact type would be displayed in a non-text form (an icon for instance). Let’s also assume that there would be a screen that lists all the contacts that we have in our phone book and that is the screen that we are going to test.

As of right now every table cell is identified via all the text field values augmented inside of them. For example, a cell for John Smith can be accessed via the following statement:

target.frontMostApp().mainWindow().tableViews()["Empty list"].cells()["John Smith"];

The problem with this identification scheme is that potentially it lacks uniqueness. if we have 2 John Smiths we would not be able to distinguish between the two cells. Moreover, what if one John Smith is classified as family while the other one is a friend, we cannot query how these contacts are classified since these values are non textual

This is when accessibilityLabel property comes to our rescue. In our example, we want to encode 2 non-textual values in the accessibilityLabel. By using the following formula to generate accessibilityLabel for {record unique id},{contact type description} not only do we generate unique identifiers for the table cells but we can now query the contact type of the corresponding record for the cell. In our example the two accessibilityLabel values for the 2 John Smiths (assuming their unique record ids are 1 and 2 respectively) could be accessed via the following statements:

   target.frontMostApp().mainWindow().tableViews()["Empty list"].cells()["1,family"]; 
   target.frontMostApp().mainWindow().tableViews()["Empty list"].cells()["2,friend"];

The native code for creation of cells would look something like this

-(UITableViewCell *) tableView:(UITableView *) tableView cellForRowAtIndexPath:(NSindexPath *) indexPath  
{ 
   UITableViewCell * res = … 
   Contact * c = … 

   //initialize an instance of UITableViewCell 
   //get an instance of Contact that corresponds to the indexPath 
   NSMutableString * accessibilityLabel = [[NSMutableString alloc] init]; 
   [accessibilityLabel appendFormat:@"%d", c.contactId]; 
   if(c.contactType == ContactTypeFriend) 
   { 
      [accessibilityLabel appendString:@",friend"]; 
      res.imageView.image  = [UIImage imageNamed:@"friendsIcon"]; 
   } 
   else if(c.contactType == ContactTypeFamily) 
   { 
     [accessibilityLabel appendString:@",family"]; 
     res.imageView.image  = [UIImage imageNamed:@"familyIcon"]; 
   } 
   else 
   { 
     [accessibilityLabel appendString:@",work"]; 
     res.imageView.image  = [UIImage imageNamed:@"workIcon"]; 
   } 
   res.acessibilityLabel  = accessibilityLabel; 
   [accessibilityLabel release]; 

   return res; 
}

If we want to access a cell for a contact with a known record id but unknown status we would need to write some additional JavaScript code

function findCellByRecordId(recordId)
{
   var cells = target.frontMostApp().mainWindow().tableViews()["Empty list"].cells();
   for(var i=0; i < cells.length; i++) 
   { 
      var cell = cells[i]; 

      //parse the name and check the first part 
      var cellNameParts = cell.name().split(","); 
      if(cellNameParts[0] == ("" + recordId)) 
      { 
         return cell; 
      } 
   } 

   //at this point we did not find anything 
   return null; 
}

Now we have means to locate cells based on contact unique record id, we would want to have an ability to see query what kind of contact type this cell shows

function getRecordTypeFromCell(cell)
{
   return cell.name().parts[1];
}

Now that you can access cells based on the unique record id of the contact that these cells represent as well as the ability to query the status of the contact, you can start writing automated tests for this screen. Happy unit testing!

Note: download sample Phone Book project