Sushasan Labs Policy · Innovation · Impact
Method

Five steps, an evidence standard and ten checks.

The method is published so that a reader who disagrees with a finding can test the procedure as well as the conclusion. The five steps below are how the first two stages of an analysis note, the gap and the work, are carried out. The third stage, the response, is described in the method note.

Last reviewed . Reviewed annually and after any retraction.

The objection this method has to answer

“You have found a difference between a document and a dashboard, and called it a failure.”

If the official responsible could fairly say this of a finding, the finding should not have been published. The steps below are meant to rule that out.

1. The five steps

  1. 1 Establish intent. Fix what the programme was designed to do, in its own words and with dates, from up to four sources: the scheme document or guideline, with its version and date; the cabinet decision or notification; the budget line, with its head and year; the ministerial statement or parliamentary answer. These are quoted, not paraphrased.
  2. 2 Establish reality. Fix what the record shows, from independent or administrative evidence, with the same attention to dates: administrative data with its publisher and period; a published evaluation with its design; responses to information requests with reference numbers; tender or contract documents; interviews or observation with the number of sources and the method stated.
  3. 3 Locate the gap. State the difference in the same unit, over the same period and against the same denominator. Both sides go in one table, dated and sourced, and the difference is computed. Many apparent gaps disappear at this step, which is its purpose.
  4. 4 Characterise, do not diagnose. Classify the gap by type (design, financing, personnel, data, incentive or measurement artefact) and say what evidence would distinguish between them. A cause is named only where the evidence supports it.
  5. 5 State what would change the finding. Name the document, dataset or answer that would confirm or refute the reading, and say whether it was asked for. Each analysis note will carry a paragraph on this.

2. What counts as evidence, on each side

The two sides of a gap are weighed differently.

Table 1. Evidence weighting, by side of the gap
Establishing intentWeight Establishing realityWeight
Statute or rulesStrongest Independent evaluation with a stated designStrongest
Cabinet decision, notification, gazetted guidelineStrong Audit findings: statutory, social or concurrentStrong
Budget line with head, allocation and revised estimateStrong Budget execution: actuals against revised estimatesStrong
Parliamentary or assembly answerStrong Administrative data, published, with period statedGood, with caveats
Ministerial statement or press releaseWeak alone Information-request response with a reference numberGood
Brochure, presentation or dashboard headlineWeakest Structured interviews, with number and method statedGood for mechanism, weak for magnitude
A news report of an announcementNot used as evidence A programme’s own impact claimNot used as evidence

3. Three common errors this method avoids

A practitioner would spot each of these quickly, so each is excluded.

Error 1: a target compared to a cumulative figure

Targets are usually annual, while dashboards are usually cumulative since inception. Comparing them gives a number that is arithmetically correct and meaningless. Analysis notes do not compare the two.

Error 2: a reporting gap reported as a delivery gap

If the management information system shows 40 per cent and the reality is 70, that is a failure of the data system. It is a real finding, and a different one from non-delivery, and it is reported as such.

Error 3: a mid-year figure read as a year-end shortfall

Government expenditure and placement figures tend to be concentrated in the final quarter. In-year figures are given with their month, in the sentence.

4. The ten checks, run on each analysis note

  1. Is intent established from at least two of: statute, notification or guideline, budget line, parliamentary answer? Are they quoted, dated and linked?
  2. Is reality established from at least two independent sources, at least one of which is not the implementing agency?
  3. Are both sides expressed in the same unit, over the same period and against the same denominator, with the period in the sentence?
  4. Have a target and an achievement, an allocation and a release and an expenditure, a registration and a completion, and a cumulative and an annual figure each been kept distinct?
  5. Are the candidate explanations named, including “this is a measurement artefact”, with a statement of which ones the evidence cannot rule out?
  6. Has “led to”, “resulted in” or “because of” been removed wherever the design does not support causation?
  7. Is there a “what would change this finding” paragraph?
  8. Has the specific critical claim gone to the organisation named, with at least five working days to reply?
  9. Does the note meet the rules under “What gets published” and “What is not published” in the standards?
  10. Would the note stand if the official responsible replied in public?

5. Where AI tools are used

AI tools are used for sorting and summarising documents, extracting figures for checking against the source, first-draft translation, transcription, draft structure, checking prose and code for this website. They are not the last step before publication on a factual claim: numbers, names, dates, quotations and citations are read in a primary source after any tool has touched them. The full policy is in the standards, section 2.

Why the order matters

A check made before a model rewords a sentence does not verify the sentence that is published.

6. The sourcing plan

This table is filled in before drafting begins. If a row cannot be filled, the note is not ready to be written.

Table 2. The sourcing plan, completed before drafting
StepThe questionThe document to pull
IntentWhat does it commit to, in whose words, by when? Cabinet approval; mission or scheme document; demands for grants across three years; annual report chapter; parliamentary answers
RealityWhat exists, and is it in use? State dashboards with the access date; parliamentary answers with state-wise figures; tender and empanelment documents; audit reports; an information request for district counts
The gapTarget against achieved, same date, same denominator; and records created against records in use A compiled table, published in the note with both sides dated and sourced
CharacterisationWhich explanations can the evidence rule out? Literature on the upstream dependency; the scheme’s own exclusion criteria; practitioner interviews for mechanism
What would change itWhat single missing thing would settle it? The information request, with its number and date

7. What is not written about

Three exclusions apply to what Sushasan Labs publishes.

  • Live professional engagements. Nothing is published about a programme on which Sushasan Labs or its people are professionally engaged while that work is live. Afterwards, only material that was already public is used. Disclosure does not replace this rule.
  • Material obtained in confidence, in any form, including in generalised or anonymised form.
  • Named projects of a client or partner, past or present, without that organisation’s written permission, which is stated in the note.

8. Reuse this method

This page is published under CC BY 4.0. It can be copied, adapted and used to test the findings published here.