The return from Babylonian exile was not a single event, but rather a process that took place over the course of a century, during which time the temple and walls of Jerusalem were rebuilt. I hope this timeline can help you situate the last part of the Old Testament story in its context.
Watts in the Word 