You are right that there are certainly countless factors, and it's a messy system. You can clearly see economic troubles suppressing birth numbers, or the introduction of the pill that drove most Western societies to a birth rate below the replacement level. You can also see clear differences caused by politics/ideological systems. But the coupling between birth rate and death rate seems the strongest overall predictor, with the lag between the two driving many population booms. As a consequence, Western nations saw the biggest decline in birth rates in the beginning of the 20th century [1]

1: https://i.imgur.com/jhyvMWX.png