Five steps, an evidence standard and ten checks.
The method is published so that a reader who disagrees with a finding can test the procedure as well as the conclusion. The five steps below are how the first two stages of an analysis note, the gap and the work, are carried out. The third stage, the response, is described in the method note.
Last reviewed . Reviewed annually and after any retraction.
The objection this method has to answer
“You have found a difference between a document and a dashboard, and called it a failure.”
If the official responsible could fairly say this of a finding, the finding should not have been published. The steps below are meant to rule that out.
1. The five steps
- 1 Establish intent. Fix what the programme was designed to do, in its own words and with dates, from up to four sources: the scheme document or guideline, with its version and date; the cabinet decision or notification; the budget line, with its head and year; the ministerial statement or parliamentary answer. These are quoted, not paraphrased.
- 2 Establish reality. Fix what the record shows, from independent or administrative evidence, with the same attention to dates: administrative data with its publisher and period; a published evaluation with its design; responses to information requests with reference numbers; tender or contract documents; interviews or observation with the number of sources and the method stated.
- 3 Locate the gap. State the difference in the same unit, over the same period and against the same denominator. Both sides go in one table, dated and sourced, and the difference is computed. Many apparent gaps disappear at this step, which is its purpose.
- 4 Characterise, do not diagnose. Classify the gap by type (design, financing, personnel, data, incentive or measurement artefact) and say what evidence would distinguish between them. A cause is named only where the evidence supports it.
- 5 State what would change the finding. Name the document, dataset or answer that would confirm or refute the reading, and say whether it was asked for. Each analysis note will carry a paragraph on this.
2. What counts as evidence, on each side
The two sides of a gap are weighed differently.
| Establishing intent | Weight | Establishing reality | Weight |
|---|---|---|---|
| Statute or rules | Strongest | Independent evaluation with a stated design | Strongest |
| Cabinet decision, notification, gazetted guideline | Strong | Audit findings: statutory, social or concurrent | Strong |
| Budget line with head, allocation and revised estimate | Strong | Budget execution: actuals against revised estimates | Strong |
| Parliamentary or assembly answer | Strong | Administrative data, published, with period stated | Good, with caveats |
| Ministerial statement or press release | Weak alone | Information-request response with a reference number | Good |
| Brochure, presentation or dashboard headline | Weakest | Structured interviews, with number and method stated | Good for mechanism, weak for magnitude |
| A news report of an announcement | Not used as evidence | A programme’s own impact claim | Not used as evidence |
3. Three common errors this method avoids
A practitioner would spot each of these quickly, so each is excluded.
Targets are usually annual, while dashboards are usually cumulative since inception. Comparing them gives a number that is arithmetically correct and meaningless. Analysis notes do not compare the two.
If the management information system shows 40 per cent and the reality is 70, that is a failure of the data system. It is a real finding, and a different one from non-delivery, and it is reported as such.
Government expenditure and placement figures tend to be concentrated in the final quarter. In-year figures are given with their month, in the sentence.
4. The ten checks, run on each analysis note
- Is intent established from at least two of: statute, notification or guideline, budget line, parliamentary answer? Are they quoted, dated and linked?
- Is reality established from at least two independent sources, at least one of which is not the implementing agency?
- Are both sides expressed in the same unit, over the same period and against the same denominator, with the period in the sentence?
- Have a target and an achievement, an allocation and a release and an expenditure, a registration and a completion, and a cumulative and an annual figure each been kept distinct?
- Are the candidate explanations named, including “this is a measurement artefact”, with a statement of which ones the evidence cannot rule out?
- Has “led to”, “resulted in” or “because of” been removed wherever the design does not support causation?
- Is there a “what would change this finding” paragraph?
- Has the specific critical claim gone to the organisation named, with at least five working days to reply?
- Does the note meet the rules under “What gets published” and “What is not published” in the standards?
- Would the note stand if the official responsible replied in public?
5. Where AI tools are used
AI tools are used for sorting and summarising documents, extracting figures for checking against the source, first-draft translation, transcription, draft structure, checking prose and code for this website. They are not the last step before publication on a factual claim: numbers, names, dates, quotations and citations are read in a primary source after any tool has touched them. The full policy is in the standards, section 2.
A check made before a model rewords a sentence does not verify the sentence that is published.
6. The sourcing plan
This table is filled in before drafting begins. If a row cannot be filled, the note is not ready to be written.
| Step | The question | The document to pull |
|---|---|---|
| Intent | What does it commit to, in whose words, by when? | Cabinet approval; mission or scheme document; demands for grants across three years; annual report chapter; parliamentary answers |
| Reality | What exists, and is it in use? | State dashboards with the access date; parliamentary answers with state-wise figures; tender and empanelment documents; audit reports; an information request for district counts |
| The gap | Target against achieved, same date, same denominator; and records created against records in use | A compiled table, published in the note with both sides dated and sourced |
| Characterisation | Which explanations can the evidence rule out? | Literature on the upstream dependency; the scheme’s own exclusion criteria; practitioner interviews for mechanism |
| What would change it | What single missing thing would settle it? | The information request, with its number and date |
7. What is not written about
Three exclusions apply to what Sushasan Labs publishes.
- Live professional engagements. Nothing is published about a programme on which Sushasan Labs or its people are professionally engaged while that work is live. Afterwards, only material that was already public is used. Disclosure does not replace this rule.
- Material obtained in confidence, in any form, including in generalised or anonymised form.
- Named projects of a client or partner, past or present, without that organisation’s written permission, which is stated in the note.
8. Reuse this method
This page is published under CC BY 4.0. It can be copied, adapted and used to test the findings published here.