Compliance testing : exact scope ?
Dear professionals from the industry,
I am looking for information regarding the concept of "compliance testing" (aka 1st party certification) for Sony (TRCs), Microsoft (TCRs, Games for Windows for PC) and Nintendo (Lot-Checks)
As you know, any title has to respect some specific norms so that it can get the approval of Sony/Microsoft/Nintendo to be released on any of their platforms.
These norms are made of... :
A- mainly, some technical-related checks (TRC, TCR, Games For Windows, Lotcheck)
For example: ensuring that the game is compatible with a 7.1 soundsystem, ensuring that the game can be launched with the most basic factory settings of the console, etc
B- and in parallel, some additional terminology-related checks (“xenon terminology list†for Microsoft, “product group naming conventions†for Sony, etc)
For example: checking that the official disconnection message of the Xbox Live Arcade is properly localized for each language available in the game, etc
==============================
Many QA vendors all over the world (Babel, Testronics, etc - just to name a few) officially provide compliance testing services to publishers.
However, I have noticed here and there that the concept of "compliance testing" was often understood and handled in a different way depending on the QA vendor and/or publishers ! Some QA vendors only perform linguistic checks on the title, while some others perform ALL the checks (technical and linguistic-related ones). Some publishers expect their QA vendors to check everything (technical and related checks), while some others only expect them to perform the linguistic-related checks because they have an in-house QA Lab responsible for the technical ones...
It is easy to get lost. My question is : what is the exact scope of services/checks to be covered by "compliance testing" ? To be more precise :
1- Regarding the generic term “compliance testing†: is compliance testing covering the full TCR/TRC/Games for Windows/Lot Checks (technical and linguistic checks) or only the linguistic-related checks from these (terminologies, localized texts, etc) ?
2- Should the QA vendor take care of the master submission process on behalf of the publisher ? If so, in which extent ?
If somebody could provide a clear status on this topic, it would make my life easier.
Thank you !
Email me off-list at xhadoe at gmail dot com, and I'll be glad to answer your questions.
I worked as a QA Compliance Tester at EA for a few years.
@Moik : Thanks for your answer! I must admit I am slightly getting confused with a part of it though 
Following your reasoning, could you please precise what would be the differences between "pre-certification testing" and "compliance testing" ?
@thexhadoe : Thanks a lot for the offer, much appreciated ! I will drop you an email in a short while.
I think there is no real difference between the two terms.
"Pre-Certification Testing" is simply something that appeals to me as being a more accurate description of the testing in question. "Compliance Testing" seems somewhat nebulous and obtuse, but it does refer to the same type of testing. Possibly it could also refer to internal "requirements testing" which would be more about being sure the game does what the designer/producers want it to do.
It's simply an issue of aesthetics from my vantage point, however different people do have different definitions for the same words. if you're basing a business relationship around it, you should "negotiate" the definition with your partner.
As someone from Babel, as they were mentioned in the original post, here's my take on it, I won't comment specifically on platform as that is covered by legal agreements:
1- Regarding the generic term “compliance testing†: is compliance testing covering the full TCR/TRC/Games for Windows/Lot Checks (technical and linguistic checks) or only the linguistic-related checks from these (terminologies, localized texts, etc) ?
Compliance should take into account all aspects of the platform requirements including functional (in some cases), online, platform specific compliance, localization (if required).
There are certain TRC/TCR/LotCheck requirements that can only be checked by the developer as they relate to things that need to comply within the code, calling of libraries etc.
2- Should the QA vendor take care of the master submission process on behalf of the publisher ? If so, in which extent ?
Most platforms do not allow vendors to do this and so this stays with the publisher.
I hope that this helps.
Thanks for your input guys, very much appreciated.
I still have one question though, if I may take a bit more of your time.
Referring to your answers (and especially the one from ben.wibberley) - which made everything crystal clear about the concept of "compliance testing" in general - I have 2 more questions.
1- As for the official terminologies lists ("xenon list" for Microsoft, "product group naming conventions" for Sony, etc)
==> Are these separate supportive documents to conduct compliance testing or are they an addionnal task to be performed in the framework of "compliance testing" ?
2- As for the TCR/TRC/Lot Checks
==> How can the QA vendor easily define which checks are under his responsibility and which are under the developer's ?
Official terminologies are generally provided in a separate document from the requirements list, and is likely considered part of normal compliance testing.
The best thing to do may be to create a client-signed-off document of lot checks being performed and those not being performed by the QA vendor. If you break it down into simple rules like "No glass-box testing will be performed, only tests that are done using retail release build and retail consoles will be performed, etc", then the issue starts getting muddied and leaves room for a misunderstanding/miscommunication that could ruin the vendor/client relationship.
There may even be some tests the client would not want you running if the test would show too much of how the game's technology operates (trade secret/patent sort of things).
Moik is spot on here, most of the information is within the entire platform compliance documentation, so has to be specifically broken out.
Secondly that you simply create lists and next to each test cas state whether or not the developer or the QA vendor is testing, from my experience this list generally comes from the vendor as that's what they're there (and paid!) to do.
My impression of the term usage would be that testing which covers console manufacturer requirements is referred to as "pre-certification testing." "Compliance testing" has a very synonymous appeal. I think the client would expect TRC/TCR/LotCheck to be encompassed when someone offers "Compliance testing." If the vendor does not offer the service, they should probably avoid using the term.
As for an external QA vendor handling the submission docs, that sounds fraught with risks, mainly from the amount of excruciatingly detailed information that would need to be passed back and forth to accurately finish the documents. It's probably best to have the developer prepare the docs rather than the QA vendor or the publisher. This question may be better answered in either the Business & Legal or Production & Management sections.