Solid Step Test

Automatic Testcases   -   Every Single Time

  Greybox Blog             Free Eval             SST Demo

       Discussion Board           Software Services  


Grey Box Testing Center
The Grey Box Software Blog
SST Paint Demo
Howto: Download the Demo
Using the Demo
Saving Testcases
Replaying Testcases
Testcase Scenarios*
Demo FAQ
Solid Step Test
Grey Box Testing Whitepaper
Grey Box Discussion Board
Free Evaluation
Solid Step Newsletter
Contact Solid Step Software




· Open All Folders





Testcase Scenarios in SST Paint

This page includes step-by-step instructions for creating a number of different testcases using the SST Paint demo application, in order to illustrate how SST handles the general case of software processes.

  • Testcase #1 - Quickstart: Save and Replay a Testcase

  • Testcase #2 - Illustrates how SST handles data files in testcases

  • Testcase #3 - Shows how SST handles random elements

  • Testcase #4 - Includes asynchronous process communication

  • Testcase #5 - More complex asynchronous process communication

  • Testcase Summary - Handling the General Case of Software Processes



Testcase #1

This simple testcase consists of the following steps:

  1. Start the SST Paint demo application

  2. Maximize the internal drawing window

  3. Draw a happy face

  4. Sign your initials

  5. Save the drawing as "SstPaint1.SSP" in the "My Documents" folder

  6. Save the testcase (If you want help, see Saving Testcases).


After saving Testcase #1, you can replay it by running SST Paint again and using the Play SST Test Log button (see Replaying Testcases for more information).





Testcase #2

This testcase illustrates how SST handles data files when saving and replaying testcases.

  1. Start SST Paint

  2. Open the drawing file that you created in Testcase #1 (SstPaint1.SSP)

  3. Draw a picture frame around the drawing

  4. Save the drawing

  5. Save the testcase

If you start SST Paint again and open the file SstPaint1.SSP that you just saved, you will see that it is the one with the picture frame around the drawing.

But if you start SST Paint and replay the last saved testcase, you will see that when it opens SstPaint1.SSP while replaying the testcase, it is not the one with the picture frame, it is the original drawing. That is because SST saves the input data file in the testcase itself, so that it can replay the testcase anywhere at anytime. This helps insure complete Reproducibility.

So in this way SST provides a sort of sandbox in which the application runs. When a drawing is opened during testcase playback, SST redirects the application to use the original drawing file that was opened when the testcase was created. And when a drawing is saved during testcase playback, SST redirects the application to save the data file in the testcase sandbox.

For instance, if you delete or rename the file SstPaint1.SSP and replay the last testcase again, you will see that it runs as expected - it already has the input data it needs to run the testcase. Furthermore, it also does not write the file SstPaint1.SPP in the "My Documents" folder. Instead SST redirects the file so it is written in the sandbox folder, which also facilitates easy data verification when it compares the new data with the data saved in the original testcase.


Testcase #3

This testcase illustrates how SST handles random elements in software.

  1. Start SST Paint

  2. Maximize the outer frame window

  3. Maximize the internal drawing window

  4. On the Edit dropdown menu, use "Add Random Points" 6 times

  5. Save the drawing

  6. Save the testcase


The "Add Random Points" function uses the computer's random number generator to add a drawing stroke to the picture starting at the top and going downwards until it hits the edge of the drawing page (not the window - the drawing page may be smaller or larger than the window).

This creates a lightning effect in the drawing:


When you replay this testcase, the exact same lightning strokes will be created in the drawing because SST stores the random numbers in the testcase. In this way SST can enable complete Reproducibility even in applications that are normally indeterminate because of random numbers used in the software.





Testcase #4

Another common thing that causes indeterminacy in software is asynchronous process communication. For instance, on the Query Dropdown menu, the "Asynchronous Query" function shows how SST can handle a software process that sends a message or query to another process, and lets the user continue using the program while waiting for the other process to reply. SST handles this by storing enough information in the testcase to be able to automatically synchronize testcase playback to reproduce the original run of the application.

  1. Start SST Paint

  2. Use the "Asynchonous Query" function on the Query dropdown menu


  3. In the Asynchronous Query window, enter "Wait 20" in the box labeled "Enter any string", and enter "20" in the box labeled "Number of Seconds to Wait for Reply":


  4. Click OK in the Asynchronous Query window

  5. Using about 5 seconds of time, draw a cloud, then wait for the asynchronous reply


  6. Click OK in the Asynchronous Reply window

  7. Draw the sun next to the cloud


  8. Save the testcase


When reproducing a testcase for software that has asynchronous process communication, the asynchronous reply could happen at an earlier or later time during testcase playback than it did during testcase creation. So SST must correctly synchronize the individual playback events to insure that testcase playback happens in the same way as during the original run of the software.

When you replay this testcase, you'll see that after drawing the cloud, SST Paint waits for the asynchronous reply before drawing the sun, just as you did when creating the testcase. This shows the case when the asynchronous reply happens after the last thing that happened before the original reply.

The other case, when the asynchronous reply happens earlier during testcase playback than the last thing that happened before the original reply, is handled by SST in the following manner: When it receives the asynchronous reply, it holds onto the reply until testcase playback gets to the point of the last thing that happened before the reply during testcase creation, and only then does it let the reply through to the application, thus preserving the timeline of the original run of the software.





Testcase #5

This testcase shows how SST handles asynchronous process communication between the AUT (Application Under Test) and multiple external processes, which is a key requirement for handling the general case of software processes.

  1. Start SST Paint

  2. Do a 40 second async query

  3. Quickly draw 3 clouds

  4. Do a 60 second async query

  5. Quickly draw 3 circles

  6. Wait for each async reply

  7. Sign your initials

  8. Save the testcase

  9. Run the testcase.

  10. Edit the second async query to be just 1 second.

    In the directory where you ran SST Paint, look in the subdirectory named "sst_log". The newest directory in the "sst_log" subdirectory will be the last saved testcase, and it will be named something like "SST_Paint_sst_D080703_T150717", where the numbers after the D and T indicate the date and time the testcase was saved.

    In that directory is a file called "sst_log.out". Edit that file (be aware that it includes binary characters). Search for "SstAsyncQueryDlg_2", then 6 lines below the first occurence will be the reply time ("60" followed by the "Control-A" character):

    Change the query time to be 1 second:

  11. Run the new testcase.

When you run the testcase before editing it, it will play back in the same way as when it was originally created. But when you play back the edited testcase, which handles the case when the asynchronous replies happen in a different order during playback than they happened when the testcase was originally created, you see that SST can still synchronize the actions so as to reproduce the original run of the application, i.e. it holds onto the 2nd query's response until after the 1st query's response.

Note: If you want to make a copy of the testcase before editing it, you should be aware that when SST Paint plays back the last saved testcase, it uses the timestamp on the ".sst_recorded" file to decide which testcase to play. It also only looks at testcase directories whose names start with "SST_Paint_sst".




Summary

These testcase scenarios demonstrate how SST automatically creates testcases for any and all runs of the software, even for applications with elements like asynchronous or random processes that can lead to indeterminacy in the software. (Although the software may be indeterminate when it is originally run, SST saves enough information to be able to reproduce each individual run of the program.)

The intent of SST is to enable Complete Reproducibility for any type of software, and these testcase scenarios are meant to show how it can handle the general case of software processes. But there are so many different kinds of software that it's impossible to say for sure how well it will work in practice for all types of software.

Currently grey box testing has only been used in software with a built-in GUI (i.e. a GUI that is not contained in a web browser). This is a fairly broad category that includes applications like office software (word processors, spreadsheets, etc), graphics software (drawing tools, etc), engineering software (3-D modeling programs, chip layout tools, etc), video games, mobile applications, and a whole host of other software with a built-in GUI.

But there are many other kinds of software that don't fall into this category, such as web-based applications, embedded software, command line programs and real-time software. Although there could be practical limitations to grey box testing for some of these applications, we think it may turn out to work remarkably well for most of these other types of software too.

Please Contact Us to relate your experiences using grey box testing, or to get a Free Evaluation of how grey box testing could work in your application.



Previous: Replaying Testcases Next: Demo FAQ

Contact Solid Step Software
© 2014 Solid Step Software