Figure 1: Testing for the "Interoperability Stack"
Keywords: ebXML, OASIS, conformance, interoperability, testing
Biography
Jacques Durand, director of Engineering and Standards at Fujitsu Software, is evaluating new eBusiness standards and advising on their use in the business process management and B2B products at Fujitsu. He is actively involved in the design and implementation of ebXML standards, chairing the OASIS ebXML Implementation, Interoperability and Conformance Committee. He is also actively involved in the Web Services Interoperability organization, of which Fujitsu is a co-founder. He is currently chairing the Testing working group of WS-I, a multi-vendor group which designs and implements tools to verify Web Service interoperability based on a concept of Profiles. Prior to this, Jacques Durand has been V.P. of development at Savvion, a BPM company, for several years. He has more than 20 years of experience in various areas of software development, ranging from R and D to commercial products. He has received an MS and Ph.D. in computer science from Nancy University, France.
Biography
Michael Kass is a Computer Scientist at the National Institute of Standards and Technology (NIST) in Washington DC. A 1984 graduate of the Ohio State University with a B.S. in Geodetic Science, Mr. Kass worked in the Geodesy Department of the Defense Mapping Agency from 1984 to 1989. In 1989 changed to the computer science field as a computer systems and applications programmer for the U.S. Naval Surface Warfare Center in Carderock, Maryland. In 1996, Mr. Kass joined NIST as a software testing developer in the Standards and Conformance Testing Division of the Information Technology Laboratory and is currently working with the OASIS ebXML Implementation, Interoperability and Conformance Committee to develop a general testing framework and test suite for ebXML implementations.
Biography
Pete Wenzel, Senior Architect, SeeBeyond, United States, is a 9-year veteran of Business-to-Business and Enterprise Application Integration solutions at SeeBeyond, Pete Wenzel is currently a member of the Standards and Product Strategy group, and has been active in the standards development community for several years. His experience includes a term as Chief Architect of RosettaNet, during which time he oversaw the technical development of RosettaNet Ready, that consortium's software compliance testing program. Mr. Wenzel is a participant in several OASIS ebXML Technical Committees, including Messaging Services, Collaboration Protocol Profile and Agreement, and Implementation, Interoperability and Conformance, and also represents SeeBeyond in several other Web Services-related standards efforts. A graduate of California Institute of Technology, his specific areas of interest include data communications and security.
Testing tools for Conformance and Interoperability are emerging as a key technology for enabling and maintaining interoperability between e-Business partners. Verifying adherence to a common standard (conformance) is just the first step. In addition, it is critical that the selected options, bindings, and deployment settings of the implementations, are compatible across partners. Also, as a stack of several standards, as in ebXML, are usually involved, interoperability increasingly relies on the bindings across these standards, their version compatibility, and explicit contracts. This paper reviews the challenges and solutions of interoperability testing for the current set of ebXML standards, presents the status of the ebXML Test Framework and test suites developed by the ebXML Implementation, Interoperability and Conformance Technical Committee (IIC) under the Organization for the Advancement of Structured Information Standards (OASIS), and also compares this work with similar efforts on the set of standards more broadly understood as defining Web Services.
Such testing initiatives are taking place under OASIS, and also under Web Services Interoperability organization (WS-I). Although the scope of IIC work spans all ebXML specifications, we focus here on Messaging, as the IIC has recently completed test suites for this specification. The paper also describes the underlying IIC testing architecture: the ebXML Test Framework, currently under development at NIST. WS-I has also chartered a Testing working group, whose goal is to define test assertions and develop testing tools for verifying interoperability between various instances and platforms of Web Services. We also comment on this initiative and its testing approach, and compare with the ebXML IIC work.
The authors have been involved in both organizations and activities for more than a year, working at design and implementation level.
Among the identified challenges in testing eBusiness systems, the following are developed:
Introduction
The Nature of Business-to-Business Testing
The Challenges of Interoperability
The IIC Approach
The Interoperability Stack
Principles of Interoperability Testing
The Test Framework, an Overview
General Features
Components of the Test Framework
The Test Driver
The Test Service
Different Testing Configurations using the Test Framework Components
Testing Conformance of ebXML Messaging
Testing Interoperability of ebXML Messaging, point-to-point
Testing Interoperability of ebXML Messaging, controlled from a third party
Testing Conformance of a BPSS Implementation
Testing Conformance of a Registry Implementation
Testing Interoperability of a BPSS Implementation
Test Scripting
The Testing Methodology
Executing Test Suites
XML Test Scripting
The IIC and Deployment Recommendations
A Notion of a Deployment Template
Future Directions
Implementation Initiatives
How the Testing Practice will Evolve
The Case of Web Services, WS-I Testing
Appendix A. Sample Interoperability Test Case
Bibliography
As supply chains, market places and business transactions increasingly depend on electronic messaging and automated business processes, it is increasingly challenging to ensure and maintain interoperability between business partners. Because communication between business systems involves more than sharing the same networking or messaging middleware, interoperability will depend on diverse factors such as application-dependent configuration, quality of service, and reliance on third party software for additional services (e.g. security). Business-to-business standards are still in their early phase of deployment: different products may vary in the way they interpret and implement the same standards. Once in deployment, these systems will need to maintain their ability to interoperate with their counterparts. The testing functions for support of such requirements call for an appropriate technology that the IIC has investigated and designed.
Interoperability, in B2B systems, can be broken down in four “aspects”, each of them requiring their own verification process. These aspects represent different layers in the testing process, the set of which can be seen as a stack, as these layers must be tested from bottom to top.
Figure 1 shows these layers organized as a stack, as normally a layer should not be tested before all layers below it have been tested. The two bottom layers concern interoperability of the infrastructure itself. It can be tested independently from any application. The two top layers are business-dependent: they involve business-specific guidelines, process and content.
In response to the requirements previously mentioned, the IIC has come up with a set of practical guidelines. These guidelines or principles serve as a foundation to the Test Framework described here, and also to the test suites designed for ebXML standards.
At the basis of the ebXML Test Framework are two principles:
Other major features of the ebXML Test Framework include:
The components of the framework are designed so that they can be combined in different configurations, or Test Harnesses.
We describe here two components that are central to the Test Framework:
These components interface with the ebXML Message Service Handler (MSH), but are not restricted to testing an MSH implementation.
Open Source tools readily exist for building HTTP, SMTP, SOAP and MIME message content. Organizations such as Apache, Sun Microsystems, IBM and other vendors offer toolkits for the construction of an ebXML messages, as well as Web services to send and receive those messages.
However, openly available testing driver tools that will support all of the underlying requirements of [ebTestFramework] do not exist at this time. The requirements for a test driver include:
In particular, requirements 3, 4, and 8 are not readily available in any existing web services testing product.
In particular, the following functions have proved critical for the automation of test case execution:
Persistence of messages for the lifecycle of the test case: As mentioned above, this is necessary for executing test cases (conformance and interoperability) where timing between messages is important. For example, testing "TimeToLive" MSH functionality, as well as reliability through duplication counting requires accessing past received messages. The IIC committee has opted for the use of an XML image of a persistent message store and XPath evaluation of its content.
Conditional expressions on MIME, SOAP and ebXML message envelope and content: This is a fundamental feature of the test suite, and it relies on Xpath notation. XPath is used as a "unifying" query syntax for verification all message content in the message store. MIME headers are mapped into XML syntax for ease of message construction. To the knowledge of the authors, no Web services testing tools that support XPath evaluation of all three content types is openly available today. XML payloads are also accessible to the test driver for examination, using Xpath expressions. Currently, there are no publicly available Web services testing tools that permit evaluation of payload content. This feature must be implemented by the test driver developer.
Perform discreet message and payload validation: Conformance testing requires the ability to validate discreet message content, such as a dateTime, URI or digital signature, using specific test procedures. Current Web services testing framework tools do not support such discreet message content validation, and such a feature must be added to a test driver implementation.
An open application that is a strong candidate as a possible tool for implementing an ebXML test driver is the "Anteater" Web services testing tool available at [SourceForge]. Anteater is a web services testing framework that provides an XML-based scripting syntax to create test cases and test steps, perform XPath and regexp evaluation of any received HTTP message. It also provides SOAP support for sending and receiving both synchronous and asynchronous messages.
However, the underlying object model of Anteater differs from the ebXML Test Framework test driver. There is no "message store" class in Anteater. In addition, ebXML message payloads do not appear to be accessible for XPath query by Anteater. Only the "root" part (the SOAP message) can be queried via XPath. Discreet message content validation is not a standard feature of Anteater. However, it may be possible to extend Anteater to address the unique requirements of the IIC Test Framework.
The Test Service defines a set of Actions that are necessary for Test Case execution. The Test Service represents, or simulates an application layer for a message handler. It receives message content and error notifications from the MSH, and also generates requests to the MSH, which are translated into request or response messages. Some Test Actions are predefined, and are part of the Test Framework (i.e. not user-written). Test Service and Actions will map to the Service and Action header attributes of ebXML messages generated during the testing. The set of pre-defined actions in the Test Service include: reflecting a message, initiating a test case (by sending a new message), comparing payloads, setting and giving configuration data.
Figure 3 shows the Test Service, its interfaces, and how it relates to a Message Service Handler. The functions of the Test Service are:
Test Driver and Test Service components can be combined in different architectures or test harnesses, depending on the implementation under test, or depending on the type of testing (conformance or interoperability).
In this configuration (figure 4), the Test Driver will execute Messaging conformance test suites, simulating another communicating party to the MSH under test. The Test Service component will simulate an application on top of the MSH under test, responding to messages received by the MSH. The test suite is automated by the Test Driver (Host #2 is passive). The testing is "black box", but the Test Driver (in connection mode) has full access to the message envelopes (MIME, SOAP, ebXML), and can also generate these, introducing defects if required by the test cases.
In this configuration (figure 5), there are two MSH instances under test. The interoperability test suite will be controlled from one side, which contains the Test Driver (Host #1). The other side is passive. The Test Driver (here in service mode) will not send and receive messages at transport level, like in the previous configuration, but will initiate test cases and be notified about received messages via the Test Service component. The two identical Test Service components will simulate applications on top of each MSH under test. The testing is non-intrusive, and does not monitor the "wire" at all. The objective of interoperability test suites running on this configuration, is to verify that the entity {MSH 1 + transport + MSH 2} fulfills its contract with the two applications on each side.
In this configuration (figure 6), the same interoperability test suites as for the point-to-point configuration will be run, but will be controlled from a third party called here the "Test Center", which contains the Test Driver (Host #3). The Test Driver will initiate each test case via communication with a single Test Service component, from which it will receive notifications. This communication can be message-based, or use other channels.
In this configuration (figure 7), the component under test is a BPSS engine (or BSI). A standard test process definition is used, featuring samples of business transactions and collaborations, one side of which will be executed on the driver side by a corresponding test suite. Communication is via ebXML messaging. The Test Service component will simulate an application (or a set of application services) on top of the BSI under test, responding to activities controlled by the BSI.
In this configuration (figure 8), the component under test is a Registry server. Standard test scenarios in managing registry objects and querying them will be expressed as test suites with proper message content (payloads). Communication is via ebXML messaging. The Test Driver will verify the content of responses sent by the LifeCycle manager and Query manager.
In this configuration (figure 9), the components under test are two BPSS engines (or BSI), using a Test Center configuration. Communication is via ebXML messaging. Such testing could occur without any intervention from either party under test, e.g. as a "test service" provided by the Test Center, to which they would have subscribed. Test Service components are used for simulating business process activities, so that a standard process definition can be used. However, real applications could be used instead with the BSI instances under test. In such a case, a Test Service component could still be used as a relay to the Test Driver, for initiating test case and notifications. If the Test Center also has routing and monitoring capabilities, the content of messages exchanged can be analyzed and reported to the Test Driver, if required by the test cases.
The testing methodology supported in the ebXML Test Framework is based on "falsification testing". Falsification testing, as defined in [ConfTesting], subjects an ebXML implementation to various combinations of legal and illegal request messages, and compares the resulting response messages to a set of corresponding expected results. In ebXML application testing, the input and output are ebXML messages and their payloads. The ebXML Test Framework applies this testing methodology to all implementations, including ebXML Messaging Services, Business Process Specification Schema, Registry and Collaboration Protocol Profile and Agreement.
Development of Test Cases begins with defining Test Requirements and their associated Test Assertions. In conformance testing, Test Requirements are derived from the specification document through its conformance clause and/or from its normative content. A Test Assertion is a narrowly focused axiomatic assertion of expected behavior, given a set of set of preconditions that exercises that particular Test Requirement. In interoperability testing, Test Requirements are not derived from the specification, but are based upon agreed-upon scenarios and assertions that define interoperability between implementations.
The executable Test Case is the actual implementation of test preconditions and test assertions to verify that an implementation satisfies a Test Requirement. An implementation is said to "pass" a given Test Case if all of its sequential Test Steps area successfully executed and return a boolean value of "true". Within those Test Steps are Test Operations, which consist of message sending, message receiving, and message content verification and validation directives. Verification and validation of received message content may exercise a test precondition, or a test assertion. In all cases, these operations must evaluate to "true" (or pass) in order for the particular Test Step to be considered successful. Otherwise, the Test Step evaluates to "false" (or fail), and the implementation does not pass that Test Step, and consequently fails the Test Case and its corresponding Test Assertion.
In the case of conformance testing, if a Test Case fails, one can correctly deduce that the implementation does not conform to the specification; however, the absence of Test Case failure does not necessarily imply that it is 100% conformant. Falsification testing can only demonstrate non-conformance. Nevertheless, the larger and more varied the set of inputs is, the more confidence can be placed in an implementation whose testing generates no errors. Moreover, conformance does not necessarily indicate that an implementation is interoperable with other implementations.
In the case of interoperability testing, if a Test Case fails, one can correctly deduce that the implementation does not interoperate with the implementation it was tested with. Absence of Test Case failure in an interoperability scenario implies that the implementation is interoperable within the scope of its test requirement only. Also, interoperability does not necessarily mean that an implementation is conformant. Two implementations may interoperate while both being non-conformant with the specification.
Profile-based test case execution is performed by the test driver via selection of test requirement IDs that satisfy a particular set of testing profile requirements. A TestProfile class serves as a container class composed of all TestRequirementRef classes, each uniquely defining a particular test requirement via its id attribute. The associated FunctionalRequirement class provides all of the data necessary to define an interoperability or conformance test requirement, including name, description, specification reference and other key information. This information is used by the test driver in the test report.
By following these references, the Test Driver can select the set of test cases that match a profile. It will then dynamically generate one or more TestCase objects for each TestRequirementRef. The number of TestCase objects created is determined by how many unique TestCase elements in the XML test suite document “point to” a particular TestRequirementRef. Figure 3 shows the relationships between Profile documents, Test Requirements document and test Suite document. Test Suites that are related to each profile, will then be a subset of the master test suite.
The UML Class Diagram in figure 11 illustrates the fundamental classes and their associations defined in an OASIS/ebXML Test Suite. A TestCase object can have one or more TestStep children, based upon the XML describing the TestCase. Actual message generation is performed by the TestStep using its setMessage operation. Message retrieval and selection is performed by the getMessage operation. Asssertions are tested using the testPreCondition and testAssertion operations. Actual message content is verified for content and structure by the verifyContent and validateContent operations.
XML-based test case scripting provides declarative, implementation-neutral execution syntax for the test driver. The XML syntax for a test case is defined in schemas found in [ebXMLTestFramework]. The schemas define a logical series of test steps and possible test operations for a test case. These instructions include directives for message construction, retrieval and examination. The primary elements in the XML test syntax include TestSuite, ConfigurationGroup, TestCase, TestStep, TestPrecondition and TestAssertion.
The TestSuite element (figure 12) is a container for all test cases that may be executed by the test driver. The TestSuite element also contains “bootstrap” configuration information (the ConfigurationGroup element) and may also contain XML message payloads “inlined” into the test suite via the MessagePayload container element.
The ConfigurationGroup element (figure 13) contains default configuration data for generating the content of test messages sent by the Test Driver. In addition, six of the parameters have functions beyond defining message content: The Mode parameter toggles the mode of the Test Driver between connection mode and service mode. StepDelay is a global time value used by the Test Driver to wait between Test Step executions. ResponseURL is the URL endpoint destination of response messages generated by a Test Driver in “connection” mode. NotificationURL is the endpoint destination of notification messages generated by a Test Driver in “connection” mode. PayloadDigests is a container element of MDA-5 digest values pre-computed for test message payloads verified by the Test Driver. Lastly, ConfigurationItem content defines “ad-hoc” parameter names and their values, to be used by the Test Driver for message generation (via declaration) or message result comparison via XPath expression.
The TestCase element (figure 14) is a container element for one or more test steps to be sequentially executed against a particular test requirement. In order for an implementation to pass a particular test case, all child test steps must execute successfully (i.e. evaluate to a boolean value of “true). Documentation for the Test Case is provided through attribute values, and through ID reference back to the actual test requirement using the requirementReferenceId attribute. Configuration of the Test Driver can be altered at the TestCase level through the configurationGroupRef IDREF type attribute.
The TestStep element (figure 15) is a container element for one or more test operations to be performed by the test driver. In order for an implementation to pass a particular test step, all descendent test operations of the test step must complete successfully and evaluate to a boolean value of “true”. Test step operation directives consist of either a primary PutMessage or GetMessage operation directive.
Configuration of the Test Driver can be altered at the TestStep level through the configurationGroupRef IDREF type attribute. In addition, the execution time of the TestStep may be altered by overriding the current Test Driver configuration value using the stepDelay attribute. Lastly, Test Driver configuration can be altered by specifying an integer testStepContext value corresponding to any previously executed Test Step for the current Test Case. In that case, the CPAId, ConversationId and (if available) MessageId element values from that Test Step will be re-used for the current Test Step.
The PutMessage operation directive instructs the test driver to generate an ebXML message using the required MessageDeclaration directive. The PutMessage element may also include a SetPayload and/or a Dsign (digital signature) directive.
The GetMessage operation directive instructs the test driver to retrieve any messages from the persistent message store that match its XPath expression contained in its Filter child element. Additionally, the retrieved message(s) may be further examined through the TestPreCondition or TestAssertion directives
Validation and verification of received message content is accomplished through the TestPrecondition and TestAssertion (figure 9) directive elements. Message content that can be validated (content structure checked) or verified (content value checked) include XML message and payload content, discreet URI, dateTime and digital signature values. For both directives, the evaluation operation must succeed (evaluate to “true”) in order for the test driver to proceed to the next operation directive
Persistence of received messages for the lifecycle of a test case is essential to evaluation of a test case. Messages may arrive synchronously or asynchronously during test case execution, and must be accessible by the test driver to evaluate message content (including MIME, SOAP and ebXML portions of the message). This also includes any message payload content. Aggregation of messages permits comparison between messages as well as “duplicate counting”.
In order to minimize the need for construction of a new query syntax, MIME, SOAP and ebXML message portions are uniformly represented as a single XML document object. Each received “message document” is stored as a “leaf” object on the MessageStore element document tree, and is therefore easily examined via XPath query. The message store XML schema is represented in figure 10 below.
A uniform test reporting capability exists for both conformance and interoperability testing. Metadata describing the test suite is provided via the MetaData element. Individual Test Case result reporting is provided through TestCase attributes, including ID references back to the originating test requirement (via the requirementReferenceID attribute value) and the specification reference (specRef attribute value).
The Test Case result (“pass” or “fail”) is defined in the testCaseResult attribute. Further tracing of test results down to the Test Step level is available through the child TestStep elements.
The IIC has developed a notion of Deployment Template, in order to address the "last mile" of standardization, which can be seen as deployment guidelines for the original standard specification, based on business-specific standard practices. The first Deployment Template has been published for ebMS, and has seen its first instantiation in the Deployment Guidelines for Messaging, published by EAN-UCC. Both documents are available on the OASIS IIC (ebXML "Implement") Web site.
Due to the complex nature of the software involved, and of the layers of generic standards upon which ebXML MS is built, it is unrealistic to expect full interoperability among even a small number of deployments. Since the MS specification describes what is to be seen "on-the-wire", the interoperability concerns may be categorized as either protocol behaviors or data formats. These can be further broken down into values that must be specified and assigned semantics to achieve a common understanding among users, and parameters for which incompatible choices might otherwise be made in different implementations. Most parameters are independent of each other, leading to an inordinately large number of possible combinations, which are impractical to test thoroughly.
The IIC Deployment Template is an attempt at identifying many of the parameters of ebXML MS and its underlying technologies that could or should be specified by groups of users that intend to interoperate. Using this template, an appropriate subset of options that meets the users' business needs can easily be expressed, thus resulting in a Deployment Guide. Such a guide is then taken into account as CPPs and CPAs are formed, and as software is configured. This has the effect of defining standard settings, placing limits and reducing the possibilities to a common subset which can be more successfully tested for interoperability, and finally to be deployed into production.
Several implementation initiatives based on the IIC Test Framework architecture are currently under way:
Other ebXML Test Center initiatives have expressed their commitment in adopting the IIC test suites:
As business-to-business systems become more broadly deployed, we believe the testing technology will evolve in the following directions:
Web Services technology relies on an array of standards less focused than the ebXML standard stack, but of broader scope. This in turn poses a particular integration and interoperability challenge that has been recognized by the Web Services Interoperability (WS-I) consortium (www.ws-i.org). The methodology of WS-I relies on a notion of "profile", which narrows the versions, options, and bindings originally authorized by each individual standard involved. By doing so, interoperability failures are greatly reduced if not fully eliminated. The definition of such profiles is currently under way. This particular interoperability challenge, and also the rich service definition capability explain why the technology is currently being deployed in more homogeneous environments, typically corporate networks, becoming the first Web-based application integration technology.
The notion of WS-I profile will also require a well-rounded testing covering all phases of the life-cycle of a Web service: definition, registration, run-time. Also, conformance to profiles will need to be tested, for both Web services platforms and Web services applications. After deployment, testing tools will still be useful to diagnose possible interoperability failures that might occur due to a discrepancy in the usage of options permitted by a profile. Because of this, current WS-I testing tools are also designed to accommodate diverse testing objectives and to be non-intrusive.
The XML document below is representative of a typical ebXML Messaging Services Test case. This particular example is a simple interoperability test of a candidate MSH’s ability to receive and return an ebXML message containing one payload.
XHTML rendition created by gcapaper Web Publisher v2.0, © 2001-3 Schema Software Inc.
