Multi-tenancy refers to a process of sharing resources to run an instance of software to serve multiple clients. The different types of deployment models place heterogeneous roles and requirements in multi-tenancy. It is regarded as an essential attribute of cloud computing and it is important to understand the degree of multi-tenancy. Testing of multi-tenancy during the initial phase is one of the challenges in cloud testing since the same data instance has to behave differently for different clients^{[ 9]}.

Service Level Agreement (SLA) Governance is also a key challenge in cloud testing. The service level agreement should be done between the customer and the cloud service provider before testing. It is also important to follow the initial recommendation that has been setup by the computing standards such as National Institute of Standard and Technology (NIST)^{[ 10]}.

Security is one of the prime challenges of cloud testing. It is one of the major reasons why organizations step back from the adoption of cloud testing. There are few guarantees about the security and service quality in public clouds. Therefore, subscribers should understand the risk posed by public clouds and they should establish the level of trust with cloud providers.

After security, subscribers as well as service providers take data integrity and privacy as the critical challenges during cloud testing. The applications deployed to cloud are not always designed with data integrity and privacy. This problem may lead to the deployment of vulnerable applications into cloud with security issues^{[ 11]}.

License commitment and its access are the integral steps that different organizations have to face during the use of test tools. Different types of license models have to be verified during testing^{[ 12]}. Therefore license management is an important parameter to be observed.

The other basic challenges of cloud testing include, for example, regression testing, maintainability testing, release management, compliance with standard regulatory and integration of on/off premises systems^{[ 13]}.

The organizations can achieve maximum output of their cloud testing by following cloud based test strategy. It is important to initiate and document the test strategy before starting the cloud testing. The document may include testing goals, identifying the various types of testing to be performed, benchmarks, selection of cloud test environment and infrastructure requirements with the available resources. As suggested by Nag^{[ 14]}, Figure 1 provides the steps that can be taken for cloud testing by an organization.

Figure 1

	Figure Option ViewDownloadNew Window
	Figure 1 Cloud Testing Approach[ 14]

(1)Explain Testing Strategy: It is the first step where organizations define their strategy for successful cloud testing. It includes objectives of testing, identifying the various types of testing, for example, system integration testing, user acceptance testing, performance testing, load testing, security testing. During this step, the testing environment is selected along with the timeline and risks like quality risks and inherited risks.

(2)Define Infrastructure Requirements: Infrastructure requirements for cloud testing is the second consideration in testing approach. It is based on an organization’s testing strategy. The infrastructure requirements may include hardware and software requirements, processing power, storage capacity, bandwidth, testing tools etc.

(3)Selection of a Service Provider: The selection of the service provider that can ensure e.g. reliability, quality, flexibility and productivity of services is a critical task. The organization can identify the service provider depending on the infrastructure requirements along with the testing tool licenses and provisioning. It is essential to have detailed insight on the types of services offered by the cloud provider in order to identify the most suitable service provisioning mechanisms.

(4)Start Testing: Organization can start their testing once the above mentioned steps are completed.

(5)Control and Measurement of Testing Goals: Monitoring and measurement is the final step which includes the validation of cloud testing on scheduled or event-driven basis. It helps organization to demonstrate the Return of Investment (ROI) of cloud testing.

Besides different challenges of the cloud testing approach, delivering testing in the cloud depends on the services that organizations are using. However, the main issue centers on the type of cloud used by cloud subscribers. Most of the cloud customers go with public clouds and the primary reason behind it is fast deployment, cheap CPU performance, low initial costs, easy integration with social media, and billing that is flexible and hourly^{[ 15]}.

Testing in public cloud environments presents different levels of responsibilities for security management depending on the services that are subscribed for by an organization. Services may include IaaS, PaaS, SaaS and HuaaS. Trusted third party cloud providers try to provide the best cloud security practices to cope with threats and risks according to the needs of different organizations.

In addition, providers need to support third-party audit and e-discovery initiatives depending on the circumstances of the organization. So, proper visibility should be maintained in the cloud. If data segmentation is not properly implemented in a public cloud, it may compromise the subscribers’ data confidentiality. Therefore the subscriber should understand the associated risks and threats for different cloud-based services.

We propose an approach to secure the data as shown in Figure 2. After having started using the cloud testing approach, organizations could use this approach for better reliability and assurance on the security aspects.

Figure 2

	Figure Option ViewDownloadNew Window
	Figure 2 Structured Security Testing Approach

(1)Defining Strategic Cloud Security Roadmap

There is not any ideal model that could define the fundamentals of security in the cloud as it depends upon the requirements of the organization. A cloud security strategy should include an evaluation of risks in the public cloud model. Third party cloud service providers should have various types of risk assessment methodologies to diminish the risks and threats. They should present their comprehensive frameworks of security to address the requirements of the organization.

①Defining Business and IT Strategy

It is an important step for a cloud provider and a customer organization to employ the security and define the business and IT strategy. In this step, both of them evaluate the following things to define their strategy:

·What is the information that is publicly accessed and needed to be stored or migrated to public clouds?

·What is the type of the organization and what could be the critical and confidential information from the viewpoint of the organization?

·What are the things that would be available and can be accessed by clients after migrating and testing on the cloud?

②Identifying the Risks

Cloud service providers help customer organizations to identify the threats, vulnerabilities and other risks based on the public cloud architecture. Providers help to design initial control mechanisms to diminish risks. They define the testing procedures according to the mechanism that helps to control ongoing risks or possible threats.

③Documenting the Plan

After defining test procedures and mechanisms, it is essential to document the entire plan. Identifying the types of workload and analyzing the requirements for security, legalization and regulations should be included in the plan. Cloud service provider works with the organization and makes a plan to control the risk with the help of identity and access management. The plan may also include ways to balance security controls between the organization and the cloud service provider.

(2)Assessing Cloud Security with the Provider

After developing a strategic cloud security roadmap, cloud service providers perform an assessment of the cloud security. The cloud service providers assess the current scenario of security in the organization with comparison to the organization’s objectives regarding security. During the assessment, they review the public cloud architecture from security viewpoint. It may include security policies and processes for accessing the data and its storage in an organization.

The assessment provides information about the remaining gaps in the current practices. This will disclose the strengths and weaknesses along with the vulnerabilities and possible threats to the organization. Such an assessment shows the possible ways of improvement and security measures that could remove gaps. The service provider can take additional steps by modifying existing security policies and procedures; by using additional security controls on the infrastructures and by implementing new controls for identity and access management.

Trust in a public cloud environment can be defined as the level of customer’s certainty that the cloud provider is capable of providing the subscribed service properly and accurately. Organizations hand over to the cloud provider most of the security aspects which used to be under their direct control. So the subscriber should have a certain level of trust in the cloud provider. Figure 3 shows the level of trust boundaries in public cloud; this is relevant to testing as well. It explains that the trust between organizations and third party cloud providers depend upon the public cloud deployment model services like IaaS, PaaS, SaaS and HuaaS.

Figure 3

	Figure Option ViewDownloadNew Window
	Figure 3 Public Cloud Trust Boundaries[ 16]

As it can be seen from Figure 3, public clouds set the greatest challenge to security because of varying levels of trust boundaries. So, there should be trust in the data ownership and have a proper visibility in services and risk management between the provider and the customer. The concept regarding security measures in trust refers to the situation where all the possible risks are either removed or compressed to a minimum form^{[ 16]}.

In public cloud testing, there is a greater emphasize on the data than on the system. Therefore, it is necessary to govern and operate data and information by both subscribers and providers. While dealing with public clouds, governance refers to the management and control over policies, defining roles and responsibilities, standards for application development and special attention for managing risks^{[ 17]}. In addition, governance also implies design, identification and implementation of organizational structures along with monitoring, control and testing of deployed services in the public cloud^{[ 18]}.

The minimal security processes and procedures set by an organization for acquiring computational resources can be easily bypassed leading to security vulnerabilities. Therefore, standard governance with the policies and procedures for security and privacy should be taken into consideration. So, organizations can assure reasonable security and can achieve proper flow of information with the cloud providers. Special tools and audit mechanism should be used to validate services; to ensure that risk has been managed; to know how the data has been manipulated and to verify the implementation of policies and procedures for security. Special types of programs like a risk management program should be implemented to control the upcoming unknown risks that have to be faced by the organization and the provider.

Compliance is also one of the security issues which are holding back companies from using public cloud services. According to a Gartner report “Compliance is the process of ensuring adherence to policies derived from internal directives, procedures and requirements, or from external laws, regulations, standards and agreements”^{[ 19]}. So, compliance is the way of measuring the effectiveness and adherence of the rules and understanding the followed process.

The following types of questions could probably create an issue for compliance with international regulations^{[ 20]}:

(1)Who will be responsible to assure that compliance has been met during testing in the public cloud - cloud provider or subscriber?

(2)Will the subscribers have information about the physical location of their data?

(3)Will the subscribers have access to log data from the public cloud to correlate it with events in other systems?

(4)Who will be accountable if someone frauds with the data from the subscriber’s public cloud in an attempt to disrupt systems in the subscriber’s internal data center?

(5)Who will be responsible if there is fissure of Personally Identifiable Information (PII)?

(6)Who will be responsible for the penetration test as it is the part of vulnerability management?

Moreover, cloud providers need to provide assurance and proof to the subscribers that they have control over security and the same applies to the subscribers also, as they need to verify with their own auditors. Most of the auditors are unaware about cloud computing and its services, which complicates the issue further. In addition, there are various types of security and privacy laws for handling and testing data within different countries. These kinds of issues make compliance a complicated security issue for public cloud testing.

In the cloud, increasing unauthorized access to the information is becoming a major concern^{[ 18]}. So, Identity and Access Management (IAM) is a type of method which provides protection for the privacy and data sensitivity of information. Accrediting privileges to the users and providing usernames with login passwords are policies that can be managed by IAM. Moreover, IAM is a critical function for the organization and should be properly enforced to make authentication, authorization and auditing for testing of public cloud applications.

The challenges arising in IAM are considered as security issues during testing on public clouds^{[ 21]}:

(1)Managing the identities results from the diversity of organization as they may be customers, staff, partners, employers etc.

(2)Diminishing the duplication of identities, characteristics and authorization.

(3)Manipulating the user’s identities during centralization and decentralization.

(4)Administrating the staff turnover in the organization.

The above mentioned challenges lead organizations towards the centralized and automated identity management system, which guides the identity federation. Therefore, centralized and automated identity management system is one of the solutions for the challenges, which can be achieved with the IAM standard protocols such as Security Assertion Markup Language (SAML), OAuth (Open Authentication), XACML (eXtensible Access Control Markup Language) and the OpenID Standard^{[ 18]}.

Availability concerns mainly the critical applications and data being available. It is a possible security issue in testing of public cloud applications. The effect may be temporary or permanent and the loss to the organization can be in fraction or full. Threats targeting availability can be either denial of service attacks, storage of resources or natural disasters^{[ 15]}.

(1)Interruption of Services

Cloud subscriber or provider has to face the interruption in service even though all the services and architectures are implemented properly for testing. The widely known incidents of cloud disruption include: Gmail had one-day outage in mid-October 2008, Amazon S3 had more than seven hour downtime on July 20, 2008 and similarly Flexi Scale had 18-hour outage on October 31, 2008^{[ 15, 22]}. It is impossible to provide 100% availability and even at the level of 99.95% availability, 4.38 hours of disruption is expected in a year^{[ 15]}. The enterprise users who require higher availability may take precautionary measures on-premises. For critical data, they can locate their backup inside their on-premises storage and for the applications; the users can keep on-premises versions of the applications. Even if the cloud service provider goes down, they can work offline and later on they can update with synchronization.

There may be permanent outages in the availability if the cloud service providers go out of business, incur a facility loss or bankruptcy. In 2009, the Federal Bureau of Investigation (FBI) raided at least two data centers in Texas due to complaints from AT&T and Verizon and after investigation the computing centers were permanently closed and this caused disruption in services to hundreds of businesses^{[ 23]}. So, it is better to select viable providers and vendors with proper Service Level Agreements (SLAs) along with the redundancy links of different cloud providers.

(2)Distributed Denial of Services

Distributed Denial of Services (DDOS) is the attack where servers and networks are brought down by the flood of network traffic and prevent users to access the internet based services. Hackers and snipers are using botnets, which is the collection of compromised computers, to deny services which make it more difficult to troubleshoot this sort of attack. After all, popularity and publicity of the public cloud may also become a problem in the case of availability. The attackers target the place which is popular and the place from where they can get benefits. Similar things are happening to the public clouds. For example, the providers provide a registered mail address with valid passwords to give access to the required resources. If the user forgets the password, then he can recover it with some security questions. The attackers are snooping on the network for such activities^{[ 24]}. This type of snooping may hamper the reliability of the public cloud computing during the testing phase also.

(3)Incompatibility

Incompatibility is one of the challenges during testing of public cloud applications. The storage services provided by one cloud vendor may not be compatible with the applications that need to be tested. So it would be better to choose compatible cloud vendor according to their testing criteria. For example the Simple Storage Service (S3) of Amazon is incompatible with Google, IBM’s Blue Cloud or Dell^{[ 25]}. Therefore, it is better that the subscribers perform testing with the same cloud vendor that is hosting their (subscriber’s) online applications and infrastructures.

Public clouds use multi-tenant environments, so that the data from different users will be stored in the shared environment. Therefore data protection and its confidentiality are also critical issues and key concerns for the vendors which may lead to fraud and financial extortion. Data protection and its security are not only concerned with confidentiality, sometimes they are also concerned with judicial issues. The service providers need to provide assurance to the customer organization that only authorized users have access to the data when in transit, at rest and in use. Access control and Encryption are measures for the data security. For reliable protection of data, the cloud service providers should set the goals on the following points^{[ 26]}:

(1)Assure users that their data has been stored properly and can be retrieved when needed.

(2)Effective search of the malfunctioning server when the data corruption is detected.

(3)Ensure data protection even if the users modify, delete or add.

(4)Minimize the effect on data that has been caused by the system failures or data error.

(5)Allow users to check the status of their data with minimum deprivation.

Therefore in data security, feasible access control of data with confidentiality would be done by encrypting data and providing decryption key only to authorized persons of the organization. This has been considered as a valid method and is widely acknowledged by cloud providers and subscribers for having secure data storage on untrustworthy servers. Data sanitization is another good measure for data security. It is the process of removing sensitive data from a storage device in various situations^{[ 27]}.

Instance isolation is the procedure of ensuring that different instances running on the same physical machine are detached from each other^{[ 27]}. It is possible to get flexible and reliable services in an economical way after having multi-tenancy over a large number of platforms. This could be achieved through the concept of isolation and flexible delivery of the services from the cloud providers. The hypervisor splits physical servers into number of instances, each with their own Operating System (OS), allowing even small customers and guest users to access the public cloud for testing. It allows one virtual machine user to gain access to the data of another virtual machine user, which possibly brings new threats and risks. The suspect can steal user’s information by logging unknowingly and spread malware in the architecture of the public cloud. For example, in 2009, attackers destroyed the data of 100,000 websites. The attackers were able to get access to the servers by exploiting a critical vulnerability in HyperVM (Hyper Virtual Machine) made by a company called LXLabs^{[ 28]}. So when hypervisor is introduced to gain new capabilities, it adds new risks and potential for exploitation. Even though, it is a benefit of the public cloud, it would be difficult for the existing customers as they remain susceptible to attacks.

The alternative solution has been developed to patch up this hypervisor vulnerabilities. NC State and IBM researchers have come out with the prototype security tool which runs outside the hypervisor to determine the hypervisor’s security and the tool is termed as Hyper Sentry software^{[ 29]}.

A public cloud is based on a multi-tenanted environment and the architecture built on it increases complexity. A public cloud enables a decoupling of the layers and it enables one vendor’s SaaS to be hosted within some other vendor’s PaaS, or IaaS service. Therefore, this type of nested hosting arrangement of the architecture has platform and network risks which may bring a lack of transparency between the customers and actual point of operations even during testing^{[ 30]}.

Hypervisor can bring challenging issues in architecture too. It is an additional layer of software between OS and hardware platform. It also helps the public cloud vendor and user in launching and terminating the instances of the Virtual Machine (VM). But such features bring complexity in the architecture of public clouds and it would increase the attack over surface of the cloud. In addition, most of the virtual machines on the same host have the capability to create software based switches and network configurations for efficient and direct communication. However this could bring loss of visibility and protection in the architecture of the public cloud. Client-side protection and server-side protection are also essential to have a secure architecture in the public cloud. For the overall security in the architecture, it is better to have secure client-side before testing the public application and review the existing measures and upgrade if required. Backdoor Trojan, keystroke logger or any types of malware running on the client side could be an issue, so it is better to take precaution on it.