Using VisualStudio webtest for automating API testing

If you look at some of the core guiding principles of modern web applications design, IMO almost all of them have at least following characteristics listed below

  • Mobile first
  • Single Page App (SPA)
  • REST API

In this article I will cover how you can use webtest in VisualStudio to automate API testing. Hopefully you guys who are developing modern applications with these guiding principles in mind will find this useful. I want to emphasize that this article is not about REST API design, but it is all about testing REST APIs.

You might be wondering why automate, why not use POSTMAN or some other REST client to verify?.  investing some time in test automation will ensure quality, additionally these tests can be valuable for API developers. Once integrated into integration build process, they can provide immediate feedback to API developers in the event some change caused regression issues.

Obviously we cannot talk about API testing in the article with out having some API to work with, so for that what I have done is following

  1. Deployed Northwind Sample SQL database into a SQL Azure instance on my Azure subscription
  2. Developed REST APIs against the Northwind sample database that provides full CRUD access to entities listed below. For the purposes of this article we will only cover testing the customer service, rest of the APIs are there if you need to get hands on experience building web performance tests against it. 
    • Customer
    • Order
    • Product
    • Supplier
    • Shipper
  3. I’ve also deployed the REST APIs to my windows azure subscription, for information about API check out the API help page here. Update: 1/13/2015 API deployment has now been taken down to prevent charges from Azure. If you are interested in the entire source drop me a line via comment or email

Currently the Get operation implements following requirements

  1. Returns a list of northwind customers, default 10 records.
  2. Provide a count of total customers, current page size and page number.
  3. Ability to control the number of records returned
  4. Jump to a specific page in result set
  5. Returns URI for getting to the next set and previous set as relevant to the consumer. When the page number is 1 URI to get to previous set will be null, when accessing the last page, URI to get to next set will be null, all other cases, URI for getting to previous and next set is returned to consumer.

Test Cases for Get Operation

  1. Submit a “Get” request to “http://northwindapi.cloudapp.net/api/customers”, this is the REST endpoint for getting list of customers
    • Verify Response Status code of 200 is received from server
    • Verify count of total customers, current page size and page number is received in response
    • Verify default 10 customers exists in response
    • Verify URI to get to next set is returned in response
  2. Submit a “Get” request to “http://northwindapi.cloudapp.net/api/customers”. Set pageNo querystring parameter value to 2
    • Verify URI to get to next set and previous set is returned in response
  3. Submit a “Get” request to “http://northwindapi.cloudapp.net/api/customers”. Set pageSize querystring parameter value to 20
    1. Verify 20 customers exists in response

You can see from the above list of test cases that they cover all the requirements defined for the API. Keeping this in mind let now look at automating our API testing.

Create a new project by selecting “web performance and load test project” template. After the project is created you’ll see a WebTest1.webtest file in the project, rename this to CustomerAPITests.webtest.

I also created an XML file that contains the test data that I want to use and added to “Data” folder in my test project. We’ll add this as a data source for the webtest

In the new test data source wizard, specify a name for test data source and select XML file for data source type and click next, see screenshot below

captureds

Choose the xml file, select table and clicking finish will add the xml data source to webtest

addds

Now let’s Add a web service request to our webtest by right clicking on the test case UI and click on “Add Web Service Request” as shown in screenshot below

addwsrequest

Next we need to parameterize web servers so we don’t have to hard code the web server host names in URL, this is useful when you are sending multiple web service requests in one webtest. Parameterize web server option is available in toolbar within the test case surface. You can also right click on “CustomerAPITests” and click on “parameterize web servers” option

paramws

In “Parameterize web server” dialog click on “Change” as shown in diagram below.

paramws1

In the “change web server” dialog rename the the context parameter name to “Azure” and Web Server to “http://northwindapi.cloudapp.net”

paramws2 

Configure “Expected HTTP Status Code”, “Method” and Url for the web service request by right clicking on the web service request and clicking on “properties”. See screenshot below.

prop1

At this point we have a web service request configured to hit our “api/customers” end point which will return a list of customers from our northwind database. As a part of our first test case we also need to verify that we get a count of total customers, page size, page number, URI for getting next resultset and also verify that the response contains only 10 customers by default, we are going use a custom extraction rule & validation rule to accomplish this.

For more information on using extraction rules and validation rules in webtests see http://msdn.microsoft.com/en-us/library/vstudio/bb385904%28v=vs.110%29.aspx

I created a folder named “Custom Extraction Rules” in my test project, this folder will contain the code files for the custom extraction rules. Code below shows the JSON extraction rule to extract tokens from JSON response

[DisplayName("Extract JSON rule")]

[Description("Custom extraction rule for extracting values from JSON web response generated by web performance test")]

public class JsonExtractionRule : ExtractionRule

{

    public String Token { get; set; }

    public override void Extract(object sender, ExtractionEventArgs e)

    {

        var jsonString = e.Response.BodyString;

        var json = JObject.Parse(jsonString);

        JToken jToken = null;

        if (json == null)

        {

            e.Success = false;

            e.Message = "Response received not in JSON format";

        }

        else

        {

            jToken = json.SelectToken(Token);

            if (jToken == null){

                e.Success = false;

                e.Message = String.Format("{0} : Not found", Token);

            }

            else

            {

                e.Success = true;

                e.WebTest.Context.Add(this.ContextParameterName, jToken);

            }

        }

    }

}

We can now add extraction rules to verify we are getting a count of total customers, page number, page size and URI to get to next set in JSON response.

Right click on web service request and click on “Add extraction Rule” option to add a new extraction rule

extractionrule

In the add extraction rule dialog select “Extract JSON rule” this is the custom extraction rule we created earlier. Specify “paramTotalCustomers” for Context Parameter Name option and for Token under properties specify “totalCustomers”, rule will look for this token in JSON response.

extractionrule1

Do the above steps for “pageNo”, “pageSize”, “nextPageLink”

Now lets add a custom validation rule to verify we are indeed getting the number of items we want (controlled by “pageSize”) in response

Code for custom validation rule to check result count

[DisplayName("Result count validation rule")]

[Description("This rule checks to see the REST API returns number of items requested")]

public class ValidateResultCount : ValidationRule

{

    

    public String PageSizeElementPath { get; set; }

    public String ResultElementPath { get; set; }

    public override void Validate(object sender, ValidationEventArgs e)

    {

        var jsonString = e.Response.BodyString;

        var json = JObject.Parse(jsonString);

        if (json == null)

        {

            e.IsValid = false;

            e.Message = "Response received not in JSON format";

        }

        else

        {

            //extract pageSize from response

            var pageSizeJToken = json.SelectToken(PageSizeElementPath).Value<Int32>();

            var resultsJArray = json.SelectToken(ResultElementPath).Value<JArray>();

            if(resultsJArray.Count != pageSizeJToken)

            {

                e.IsValid = false;

                e.Message = "Results count does not match the page size";

            }

            else

                e.IsValid = true;

        }

    }

}

At this point first test case is full automated. For test case 2 I added another web service request to the webtest and added query string parameter to specify pageNo and added two JSON extraction rules to look for “previousPageLink” and “nextPageLink”

Lastly for test case 3 I added a third web service request to the webtest and added query string parameter to specify pageSize and the custom result count validation rule to verify pageSize and the number of customer records returned match.

Screenshot below shows completed webtest that automates all three test cases.

webtest3

As you can see from the screen shot below all three requests ran successfully, we can be sure that the service is functioning as expected.

webtest4

Additional requests “PUT” and “POST” can be added the the webtest as a part of automating API testing.

Hope this helps. Update: I’ve added the source that includes the NorthWind API and The Test projects to github repo http://github.com/rprakashg/TestingRestAPI


Cheers,

</Ram>

  • Sachin Kumar

    Hi Ram,

    How can I parameterize the value in the web request itself. I am creating a webtest for REST API GET call and my request looks like below.

    http://myserver.domain.com/InventoryReservation/v1/productinventories/10001.

    where 10001 is productID. I can successfully able validate this request to get response but how can I parameterize the productID so I can pass different ProductID for every iteration?

    I tried creating a datasource and adding uerystring parameter to the web service request, but still can’t get it working. Please see the attached screenshot.

    I also tried replacing 10001 by {{IRS_GetSellable_ProductSku#csv.gps_ProductId}} but getting “Request failed: Context parameter ‘IRS_GetSellable_ProductSku#csv.gps_ProductId’ not found in test context”

    Any pointer? I would really appropriate your help. Thank you!

    • Ram Gopinathan

      Hi Sachin,

      It’s because your service is expecting productID as a part of URL to retrieve single Product entity. Unfortunately you can’t databind the URL to datasource, how ever you can define a context parameter and set the value in context parameter and use the context parameter in all requests that way you don’t have it hard coded all over the place. Other option is to convert into coded web test and you can grab the values from datasource and setup the webtestrequest See http://goo.gl/9LgI88