Validation Case Study: Electronic Protocol Execution vs. Paper-Based Execution

At Ofni Systems, we use FastVal to execute protocols electronically. Testers read the procedure and compare the actual result to the expected result. Users document results with screen shots that are automatically embedded into the executed protocol. If actual results do not match expected results, a deviation is automatically triggered. FastVal is a validated 21 CFR Part 11 compliant system. Testers are required to enter a valid User ID and password before accessing the protocol. All data entry is recorded in compliance with 21 CFR 11, and the audit trail is provided to clients along with other products of validation. The audit trail and security features provide additional information and increased data integrity over paper-based validation documents.

Recently, Ofni Systems was hired to perform an onsite validation of our ExcelSafe product.  The ExcelSafe validation package is a 1200-step protocol which usually takes us four person-days to execute electronically with FastVal. This particular client requested that we perform the validation on paper as a case study to demonstrate the increased value of electronic protocol execution. There were numerous places where paper-based execution meant that the validation took longer or recorded less information than our standard electronic protocol execution method:

• Handwritten Dates have less information than Automatic Date/Time Collection: FastVal automatically attaches the system date/time to the test step after execution. This reduces the error rate in recording dates to zero, and provides more information by including the system time.
• Corrections do not clearly demonstrate when the changes were performed: The standard practice for making a correction on a paper protocol is to line through, initial the change and make the correction with the appropriate date. While changes are marked sequentially on a page, there is no documentation of order of changes with the same change date. With FastVal, all of these changes are automatically documented into the audit trail, making it extremely transparent when changes occurred.
• Screenshots had to be printed, gathered from the printer, then manually sorted and signed/dated: This was particularly difficult issue because our printer was several rooms away from the testers. FastVal embeds screenshots in the IOQ and automatically attaches the username, system date/time, and computer name to the image. With FastVal, the tester only needs to take the screenshot before proceeding with protocol execution. When paper execution required taking a screen shot, the user took the screen shot, walked down the hall to collect the printout, signed, dated and noted the correct test case step on the printout, then ensured that the piece of paper was physically placed in the correct section of the executed paper protocol. Because recording screen shots became so time intensive, paper execution tended to record fewer pictures and less information was recorded into the protocol.
• Deviations had to be completed manually: As always, there were a few minor testing nonconformances (one planned deviation, plus a few tester glitches) during the execution. With the paper execution, deviations had to be filled in by hand. There will also be future costs to ensure that the deviation is mapped to the correct test case and step. In FastVal, once deviations are classified, the default text is populated from a  template, which reduces the amount of time required to complete the documentation of the deviation. This also provides a standard approach to completing all deviations which increases control over the deviation resolution process. Deviations are also linked to their test case and step.
• Typing is faster than writing legibly: By entering information into the keyboard, there was no question about protocol legibility. The same cannot be said for handwritten documents at the end of a long day of completing protocols.
• No automatic generation of validation metrics, reports or summary reports: FastVal automatically generates metrics recording the amount of time required to complete validation testing, including providing accurate reports on estimated time of completion and who executed what parts of the protocol. FastVal also automatically creates the Validation Summary Report; we had to spend a day writing the Summary Report from scratch.

The paper based execution took a total of 10 person-days, two and a half times longer, to execute the protocol and generate the validation summary report (as opposed to 4 person-days when we use FastVal) and recorded less accurate, complete and legible information. The final result was a thick binder of paper that had to be scanned back into a document control system, as opposed to a collection of electronic files and PDF outputs that was already prepared for the client’s document control process. The paper based execution resulted in more time required to complete the project and a significant additional expense to the client.