Adverse conditions in the womb that interfere with a baby's growth may also cause brain differences, the researchers report in the December issue of Biological Psychiatry.
Ian Colman of the University of Alberta and colleagues in Britain studied the records of 4 600 Britons born in 1946 who took part in a 40-year study.
"We found that even people who had just mild or moderate symptoms of depression or anxiety over their life course were smaller babies than those who had better mental health," Colman said in a statement.
"It suggests a dose-response relationship. As birth weight progressively decreases, it's more likely that an individual will suffer from mood disorders later in life."
The researchers simply looked at medical records and did not examine a possible cause. Colman said it is possible that when mothers are stressed, stress hormones are passing through the placenta to the foetus.
"If this theory is correct, you would find that when stressful events occur, the people who were smaller babies would be more likely to become depressed or anxious," he said.
"One of the surprising findings from our research was that people who had worse mental health throughout their lives had also reached developmental milestones, like standing and walking for the first time, later in life than those who had better mental health."
The researchers did not look for any absolute weight but said there was a clear trend.
"Most notably, the group that had absence of symptoms had the highest birth weight, whereas the group with repeated severe symptoms had the lowest birth weight," they wrote.
"As weight at birth increased, the likelihood of symptoms of depression and anxiety across the life course decreased."
Not all small babies are fated to have poor mental health, the researchers said, noting that in 1946 records did not indicate whether the children were born prematurely.
"Being born small isn't necessarily a problem. It is a problem if you were born small because of adverse conditions in the womb - and low birth weight is what we looked at in this study because it is considered a marker of stress in the womb," Colman said.
"When a mother is really stressed, blood flow to the uterus is restricted and the foetus gets fewer nutrients, which tends to lead to lower birth weight."
Other studies have linked low birth weight to a higher risk of diabetes and heart disease in later life.
"I have been asked by many people what the 'take-home message' of this study is, and I would say that, in the simplest terms, it is 'We should take better care of pregnant women,'" Colman said.