QSF Audit Agent Guide

Private, Browser-Only Quick Audit

What this tool can and cannot do: The Quick Audit detects structural patterns visible in the QSF. It cannot determine every research-design intention or prove that Qualtrics, a panel platform, custom JavaScript, or an exported dataset will behave correctly. Use Copy Agent Instructions for a deeper audit and complete the generated pre-fielding checklist.

How to Use This Guide

1. For an immediate private structural check, drag the .qsf file into Private, Browser-Only Quick Audit and download the report.
2. For a deeper review, click Copy Agent Instructions and paste the instructions into the agent that will audit the survey.
3. Give the agent both the current .qsf file and the current draft pre-analysis plan. Include any prior QSF or audit report.
4. Specify eligibility, validation, randomization, panel redirect, and export requirements that are not yet documented in the draft plan.
5. Require a plan-to-QSF concordance table, a detailed audit report, and preservation of the original QSF.

Core Workflow

1. Preserve and inventory. Record the source path, file metadata, and SHA-256 checksum; parse the JSON; inventory survey elements; and keep the audit read-only.
2. Reconstruct the active survey. Begin with the Survey Flow, recursively trace referenced blocks and questions, and separate active content from Trash, unused, unreferenced, or unreachable material.
3. Trace routing and termination. Evaluate branch conditions, embedded-data assignments, consent and eligibility failures, attention checks, quotas, End Survey elements, redirects, and conflicting platform settings.
4. Audit validation and randomization. Compare Force Response, Request Response, optional-item policy, custom validation, subset sizes, choice randomization, and treatment-recording fields with the intended design.
5. Audit content and piping. Verify every embedded field and piped value, then inspect wording, choices, grammar, spacing, HTML, and editor residue under all relevant conditions.
6. Protect exported data. Check export tags, recodes, embedded data, treatment indicators, and question structure for duplicate names, ambiguous values, and downstream incompatibilities.
7. Classify evidence. Assign severity separately from status, distinguish confirmed defects from risks and ambiguities, and name the runtime checks that static QSF inspection cannot complete.
8. Report and verify. Produce a detailed audit report, prioritize repairs, and give the researcher a route-by-route, mobile, randomization, and test-export checklist.

Audit Coverage

The agent should inspect the QSF as a connected survey program rather than a list of questions.

Evidence and Classification

Use severity and status as separate dimensions.

Severity

• Critical: Can invalidate consent, eligibility, sample routing, random assignment, termination, completion behavior, or the resulting data.
• High: Can create missing required data, incorrect treatment or piping, broken validation, quota or scoring errors, or substantial export incompatibility.
• Moderate: Can affect interpretation, respondent experience, or professionalism.
• Low: Cleanup or maintainability issue unlikely to affect recorded data.

Finding status

• Confirmed defect: The QSF demonstrates a conflict with an explicit rule or necessary internal condition.
• Risk: Impact depends on Qualtrics runtime behavior or external configuration.
• Ambiguity: Design intent is missing or permits multiple coherent choices.
• Recommendation: Quality improvement rather than a defect.
• Passed check: A named structural check completed without a detected problem.

Do not turn an unknown design choice into a defect. For example, whether an attention check should be forced or optional depends on the researcher’s policy; inconsistency can be reported even when the intended setting is unknown.

Checks That Matter Especially

• Request Response is a soft prompt and is not equivalent to Force Response.
• Embedded-data field names, values, spaces, and capitalization must be used consistently.
• Failure branches must contain or reach the intended End Survey behavior; assigning a flag alone may not terminate a respondent.
• Flow-level redirects can conflict with completion URLs stored in Survey Options.
• Randomizer subset sizes must not exceed the number of eligible children.
• Treatment text, country names, or other piped values must come from the correct experiment and render grammatically.
• Hidden editor bookmarks, pasted comments, redundant nonbreaking spaces, browser-specific tags, and nested markup can survive in QSF text.
• Active export tags should be unique and compatible with downstream analysis, but exact export behavior should be confirmed with synthetic responses.

Verification Beyond the QSF

Static inspection is necessary but not sufficient. Before fielding, preview and test:

1. Every consent, eligibility, attention-check, quota, screen-out, and completion route.
2. Pass, fail, blank, and unexpected-value cases.
3. Every randomized condition and materially different piped-text combination.
4. Desktop and mobile layouts.
5. Panel redirects, messages, recording behavior, and status codes.
6. A synthetic test-response export, including column names, recodes, embedded data, treatment indicators, text entries, and missing values.
7. The exact repaired QSF exported as the fielding candidate.

Plan-to-QSF Concordance

A deep audit should begin with the current draft pre-analysis plan and compare it directly with the QSF. At minimum, the agent should map:

• study population, recruitment, eligibility, and exclusion rules;
• treatment arms, control conditions, randomization levels, probabilities, and stratification;
• primary and secondary outcomes, manipulation checks, attention checks, and covariates;
• required, requested, and optional responses;
• quotas, screen-outs, stopping rules, termination behavior, and completion codes;
• embedded-data fields, treatment indicators, scores, recodes, export tags, and planned analysis variables;
• any QSF content not documented in the plan and any planned design element absent from the QSF.

The report should classify each item as aligned, partly aligned, not aligned, not implemented, not specified in the plan, or not verifiable from the QSF.

Report Structure

Every audit should produce a detailed Markdown report. If the environment supports polished document generation, the agent may also create a PDF from the same source.

The report should include:

• audit date, AI model, reasoning level, workspace, source file, and SHA-256 checksum;
• modification status and relationship to any prior audit;
• draft pre-analysis plan path, version or date, checksum when practical, and a plan-to-QSF concordance table;
• executive summary and fielding-readiness conclusion;
• active element, block, question, randomizer, and excluded-content counts;
• severity definitions and finding statuses;
• detailed findings with identifiers, evidence, impact, repair, and verification needed;
• prior-finding status and newly introduced issues for a re-audit;
• structural checks passed;
• recommended repair order;
• pre-fielding preview and export checklist;
• unresolved design decisions and final conclusion.