Learn the Best Practices in Mobile Game Testing


The Presentation inside:

Slide 0

2 December 2014 Ville-Veikko Helppi Technical Product Manager [email protected] Learn the Best Practices in Mobile Game Testing W E B I N A R


Slide 1

Agenda Different Types of Testing and How Game Testing Differs from Regular App Testing? Frameworks, Tools and Technologies – What is Important in Mobile Game Testing Today? Top Priorities for Mobile Game Testing Infrastructure for Successful Game Testing Hands-on Example (Image Recognition) Testdroid Update Q&A


Slide 2

Let's Define 'Testing'...


Slide 3

Different Types of Testing? Black-box testing vs. White-box testing Methods used in Mobile Game Testing Functional Compatibility Performance Localization Regression Load Smoke Acceptance


Slide 4

Defining Some Ways to Test Feature-based testing Performance testing End-user testing


Slide 5

The Old-Fashion Thinking Deterministic systems Identifiable objects Tool support(s) User-driven actions Simple inputs/outputs Virtualization Random setups/results No handle on objects No standards User reacts to input Complex inputs/outputs Real hardware <> <> <> <> <> <> High-level of automation Low-level of automation X


Slide 6

Test Automation as a Solution Random setups/results -> Easy to generate random events No handle on objects -> Variety of options (e.g. image recognition) No standards -> Variety of frameworks (cross-platform etc.) User reacts to input -> Test scripts can handle random events Complex inputs/outputs -> Test scripts can generate input/content Real hardware -> Availability of real mobile hardware Test Automation brings real value with variety of options to mobile game testing


Slide 7

Challenges that Test Automation Tackles Proliferation of Devices, Brands, Operators, Platforms Coupling the Testing and Development process (Continuous Integration/Deployment/Delivery) Different Types of Testing (as automated) Stress, Load, Performance, Functionality etc. Test script & test asset usability Cons of Manual testing Tedious effort, Expensive human resources, Error-prone (as done by humans) & Not reusable (done once)


Slide 8

OS versions Chipsets CPU + GPU Tens of OEMs Memory Displays (resolutions, physical hw) OEM mods Other hardware (connectivity, calibration) Relation to other software Significant Benefits of Using Automation


Slide 9

Correct behaviour across platforms and browsers Integration with web back-ends Typically need to fully utilize HW (CPU+GPU) Resource (e.g. battery) consumption OpenGL ES 2/3 Functionality and usability Screen orientations, connectivity, user profiles Robustness Robustness and security! Brand Compliances, verification with back-ends and data Different 'Verticals' Require Different Practices


Slide 10

Importance of Mobile Game Testing


Slide 11

Important Aspects in Game Testing User Interface and Functionality UI layouts/elements, Menus, Resolution, Orientation, Overall visuality Graphics Performance Usability and User Experience Responsiveness, Real Occuring Events Multi-player/User Social Integrations Security and Liabilities


Slide 12

User Interface and Functional Compared to regular app testing, mobile game testing may seem to be more challenging e.g. OpenGL ES content not action-identifiable Graphics and UI – Identical look&feel across all devices and variants Usability controls in the context of game Navigation/game-play, Progress, Delays between "game-stages" Animations, Fonts, Other graphics assets Screen Resolution and Orientation


Slide 13

Performance Testing Performance, Load, Stress, Spike, Soak Measure the real user experience Execute tests with all possible devices (SW&HW) Discover the impact of server loads, app performance (CPU/GPU utilization)


Slide 14

Studies: Majority of users leave game due sluggish, general slowness or other bad performance behavior It's very important to understand how game behaves on different devices, under different circumstances Performance Makes +/- UX


Slide 15

Building the User Experience Understanding How quickly gamer assimilates what the game is about, how to fully play with it, what are the functionalities. Usability The main User Interface, menus, design and how usable those are Playing Experience Intuition to follow the game and go forward with its instructions and game-play Monetization Are gamers willing to buy – invest – in game Graphics performance capability is the most important thing for mobile games when forming UX Too many games fail to work properly with low-end/mid-range devices due hardware performance The real understanding of game's performance capability can be only figured out by testing on real devices


Slide 16

End-to-end between tests, AI or more simple implementations Optimize the data communication between server and clients Test simultaneously on several (different) devices Check what happens if game gets interrupted (platform) Location-based features – how to properly mock up those? Hermetic environment – what happens? Multi-Player/User Features


Slide 17

Important part of the after-engagement (how gamers share their results) Many of OEM branded devices provide certain SoMe apps pre-installed (version?) Connectivity, login and data transferring are tested in context of game Hermetic environment – what happens? Social Integrations


Slide 18

Security and Liabilities


Slide 19

4 Top Priorities in Mobile Game Testing


Slide 20

Localization


Slide 21

Device Diversity By OpenSignal, Inc. 2014 - http://opensignal.com/reports/2014/android-fragmentation/


Slide 22

Real Devices vs. Emulators Emulators/simulators won’t help you to test… User Experience and Usability Hardware Software Infrastructure 0 % = the percentage of your app users that use emulator to run your app! Why Real Devices are Must-to-Have?


Slide 23

Manual vs. Automation Smaller coverage, More money burnt & time wasted, Error-prone Large coverage, quickly completed, Less money & time wasted, Exact results. Manual Automation


Slide 24

Infrastructure for Mobile Game Testing


Slide 25

High-Level Infrastructure


Slide 26

Architecture / Infrastructure GAME Game Engine Android iOS ... API & services Server Database App Middleware Platform Connectivity/Network Back-end


Slide 27

Isolated, Hermetic Testing GAME Game Engine Android iOS ... API & services Server Database Hermetic Environment App Middleware Platform Connectivity/Network Back-end


Slide 28

Compatibility & Porting Testing GAME Game Engine Android iOS ... API & services Server Database Real Devices App Middleware Platform Connectivity/Network Back-end


Slide 29

Smoke/Reference Testing GAME Game Engine Android iOS ... API & services Server Database Reference Environment App Middleware Platform Connectivity/Network Back-end


Slide 30

Back-End Performance Testing GAME Game Engine Android iOS ... API & services Server Database Massive no. of users and significant load App Middleware Platform Connectivity/Network Back-end


Slide 31

Continuous Integration


Slide 32

Game Testing Process


Slide 33

Hands-on Example: Integrating Development with Testing Environment


Slide 34

Continuous Integration (e.g. Jenkins) Appium framework (for Android and iOS) No need for devices – we'll use devices on Testdroid Cloud service Test scripts / test cases for game Setup


Slide 35

Continuous Integration


Slide 36

Appium Intro Family Tree: Android Test Automation Frameworks Appium works for both, Android and iOS


Slide 37

© Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 38 Uses Selenium Webdriver (W3C standard) as a scripting framework Supports native Android, native iOS and mobile web: Android via uiautomator (API level >=16) and Selendroid (API level <16) iOS via UI Automation Mobile web as Selenium driver for Android and iOS You can write your Appium scripts on almost any programming language (Haskell/Go/Clojure/Java/Ruby) Appium Basics


Slide 38

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 39 Test Script Test Case Desired Capabilities Desired Capabilities / Android Example


Slide 39

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 40 Test Script Test Case Desired Capabilities Example Desired Capabilities / iOS


Slide 40

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 41 Test Script Test Case Desired Capabilities Appium @ Localhost WebDriver http://localhost_4723/wd/hub Appium Server 4723 Device Example


Slide 41

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Test Script Test Case Desired Capabilities From Localhost to Testdroid Cloud WebDriver http://localhost_4723/wd/hub *Testdroid Caps http://appium.testdroid.com/wd/hub Example


Slide 42

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Test Script Test Case Desired Capabilities From Localhost to Testdroid Cloud WebDriver { “testdroid_username”: “[email protected]”, “testdroid_password”: “p4s$w0rd”, “testdroid_project”: “My First Project”, “testdroid_testrun”: “Test 1”, “testdroid_device”: “iPad Mini 7.0.4 A1432”, “testdroid_app”: “http://domain.com/app_v1.ipa” . . “app”: “com.bitbar.testdroid.BitbarIOSSample” } Example


Slide 43

Projects and Test Runs © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 44 Example


Slide 44

350+ Android & 20+ iOS Devices


Slide 45

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 46 Example


Slide 46

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 47 Example


Slide 47

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 48 Example


Slide 48

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 49 Example


Slide 49

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 50 Example


Slide 50

Running tests on multiple devices © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 51 Example


Slide 51

Example Projects and Test Runs


Slide 52

Appium Broker Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 53 Test Script Behind the Scene Device Cluster Device 1 Device 2 Device 3 Session Map Proxy Appium Device 1 Example


Slide 53

Image Recognition The basic example of Image Recognition in Mobile Game Testing: Template Matching


Slide 54

Code Example


Slide 55

Code Example


Slide 56

Code Example


Slide 57

Code Example


Slide 58

Test Script contains the following 'steps': Identify the used platform (Android / iOS) Search for 'Goldmine' graphical element (using find_image to search for screen) Script runs test to go to shop Script runs test to buy a cannon Script runs test to place the cannon (all these three are defined in .png files) Script runs test to start a battle! Game is brought down and test ends. Example: Clash of Clans


Slide 59

VIDEO Image Recognition with Real Android and iOS Devices in Local Environment


Slide 60

Testdroid Update


Slide 61

New UI in Testdroid Cloud 2.0


Slide 62

Testdroid Cloud's Interactive


Slide 63

Testdroid PrivateCloud - Running user-selected devices at any of our hosting facilities


Slide 64

NEW HOSTING CENTER! Mountain View, CA San Francisco and Mountain View datacenter


Slide 65

Option 1: Public Device Cloud on-demand devices (multitenant) Mobile app testing on over 400+ real Android and iOS devices hosted by Bitbar Option 2: Private Device Cloud reserved devices Hosted by Bitbar in the US and/or Europe Devices chosen by and reserved only for the Customer Option 3: On-premise Device Cloud Automated mobile app testing on real Android and iOS de- vices hosted by the customer, usually 30-500 devices Testdroid Deployment Options


Slide 66


Slide 67

Video recording & slides of this webinar will be soon available at Vimeo and Slideshare! Please visit at www.testdroid.com for more information. THANK YOU! W E B I N A R


Slide 68

We operate the largest global device cloud with over 400 devices instantly available for developers Devices from the US, Europe, China, Japan and Korea – global devices for global customers


×

HTML:





Ссылка: