Infant social withdrawal is a risk factor for non-optimal child development; thus, it is important to identify risk factors associated with withdrawal. In a large community sample (N = 19,017), we investigate whether symptoms of maternal and partner postpartum depression (PPD; measured with the Edinburgh Postnatal Depression Scale) and prematurity are predictors of infant social withdrawal (measured with the Alarm Distress Baby Scale). Withdrawal was assessed at 2–3, 4–7 and 8–12 months postpartum. Linear regressions showed that prematurity predicted higher infant social withdrawal at all time points, and maternal symptoms of PPD were positively associated with withdrawal at 2–3 months. Logistic regressions showed that odds for elevated social withdrawal were increased with elevated levels of maternal symptoms of PPD at 2–3 and 8–12 months. Partner's symptoms of PPD were not associated with withdrawal. Future studies should investigate how PPD symptoms and prematurity may impact the individual development of social withdrawal.