Subject: (More) Comments on 14766 WD4

From: "D. J. Blackwood" <[email-masked]>
Date: Thu, 27 Jul 2000 11:59:48 -0400

The attached Word 6.0 document contains text that should go on page i of the
document immediately before the table of contents, and on pages iii and iv
immediately following the table of contents.  The table of contents itself
should be renumbered ii.  The page starting with section 1 then becomes page
1.

The following new section should be inserted before the existing section
6.4:

"6.4 Conformance Testing

While a profile in and of itself is not a testable entity, the underlying
infrastructure components are testable and the performance goals of the
profile are measurable and therefore testable. The technical infrastructure
of the profile is testable in both its accuracy of conformance to the
pertinent standards and its ability to successfully address the cultural and
linguistic requirements.

In testing a claim that a particular implementation conforms to a POSIX
National Profile, a systematic approach should be taken, for example,

—	Testing individual claims of conformance to the base standards or
specifications (Some or all of these claims may have been validated in a
previous testing campaign.)

—	Testing the aggregation of all the claims of conformance to the base
standards or specifications

This latter case may require that many interactions have test cases that may
give rise to an unworkable number of tests being required. However, testing
would be restricted to observable behaviour, which may be a subset of
behaviour defined in a base specification.

Conformance testing per se does not guarantee interoperability; it is only a
test of conformance to a set of test assertions based on the standard. One
way to measure conformance is to develop a reference implementation of the
particular standard or standardized profile.

The system then is “exercised” through the use of test scripts. The
behaviour of the system is monitored and compared with the expected outcome
from the reference implementation. The advantage of this approach is that
many vendor products can be tested, thus spawning competition. If a
reference implementation is not available, other conformance test methods
could be used, such as software unit testing, software qualification
testing, and integrated hardware/software testing.

Although conformance testing does not ensure interoperability, such
inter-working would be virtually impossible without conformance to
standards. Interoperability testing is a matter for vendors and users,
rather than standards setting organizations.

Many current standards have registered conformance tests. The JTC1
administers an index of registers of conformance tests that is generally
available. When a product is found on such a registry, it can be assumed
that it has “passed” a battery of test procedures. The availability of test
registries greatly simplifies the hardware/software qualification testing of
a system, thus saving time and money for the developer."

The existing section 6.4 then becomes 6.5.  The existing section 6.6 should
have the following text added immediately before 6.6.1:

"Once a POSIX National Profile has been developed, it is still necessary to
document the implementation conventions associated with each of the selected
standards and/or specifications. This final step is required to prevent
interoperability problems."

The following new section should be inserted before the existing section
6.7:

"6.7 Using PASs

PASs are specifications (e.g., industry profiles, de facto standards) that
have been developed outside the approved process for open standards. For the
purpose of this technical report, PASs are widely implemented and the
documentation is available in the public domain. Examples of widely used
PASs are the RFCs and standards from IETF. The JTC1 has recognized two
procedures for incorporating PASs into standardized profiles. The first is
to reference the PAS directly; the second is to convert the PAS into a
formal standard and then reference that formal standard."

Dave
--
D. J. Blackwood