During the 2003 outbreak, severe acute respiratory syndrome (SARS) spread to more than 30 countries. Not only did it cause severe health problems but it also imposed a great psychological impact on the public. SARS emerged in Taiwan during April 2003. This study investigates the psychosocial impact and the associated factors of depression of the SARS epidemic in Taiwan when the epidemic had just been controlled. A total of 1552 respondents were recruited in the study by random selection from the telephone book. Demographic data, SARS experience, self-perceived health state, neighborhood relationships, and depression were surveyed by telephone interviewing. Respondents were grouped as 'impacted group' and 'non-impacted group' according to whether they or their friends and family had been quarantined, or suspected of being infected. The psychosocial impact and associated factors were compared between the two groups. The 'impacted group' had higher depressive levels, poorer neighborhood relationships, poorer self-perceived health, and a higher economic impact than the 'non-impacted group'. The poorer self-perceived health and economic impact factors were associated with depression. The neighborhood relationship factor was negatively associated with depression for the 'impacted group', but not for the 'non-impacted group'. The 'impacted group' had experienced greater psychosocial impact possibly due to the SARS impact, the economic downturn, poor self-perceived health conditions, and decreased social support systems. An appropriate mental health intervention to improve the self-perceived health condition, to provide instrumental and psychological support for the 'impacted group', and to decrease the stigmatization and discrimination from the public could have buffered the psychological impact from this epidemic disaster.