The objective of this study was to assess the cross-culture validity and reliability of a standard psychiatric assessment instrument without the usual "gold standards." Normally criterion validity testing requires comparison with such a standard--usually another instrument or a professional diagnosis. Instead local informants identified persons with and without "agahinda gakabije" (a locally described grief syndrome) who were then asked if they thought they had this syndrome and also interviewed using the depression section of the Hopkins Symptom Checklist (DHSCL). To assess criterion validity, interviews where respondent and informant agreed on the presence or absence of agahinda gakabije were compared with depression diagnosis using the DHSCL. We also assessed construct validity (using factor analysis), internal reliability (Cronbach's alpha), and test-retest reliability using results from a subsequent community-based survey employing the DHSCL. We found a similar relationship between depression and agahinda gakabije as between depression and grief in western countries, which supports criterion validity. Construct validity and internal reliability were good (Cronbach's alpha = 0.87). Test-retest reliability of a DHSCL-based scale was less adequate (0.67). Although not replacing the usual gold standards for testing criterion validity, this approach may prove useful where these standards are unavailable. As this includes much of the developing world, this could result in more accurate mental health assessments among populations for whom this has hitherto not been possible.