Should the nativity set have wise men in it?
The Bible is not specific on exactly how old Jesus was when the wise men arrived.
It is presumed that the star that guided them to where Jesus was currently living appeared near the time of His birth. Since they saw the star in the west, the wise men were originally from eastern kingdoms. As they gradually proceeded to the west, the star continued to guide them. But if you look at Numbers 24:14-19, it is clear that the Messiah is also referred to prophetically as a Star. So whether the star the wise men mention was a physical manifestation (a super-nova, a comet or planet alignment), an angelic messenger guiding them, something like the pillar of fire by night that was only visible to them, or whether they were guided by the Holy Spirit as at the time of their departure is not clear. That there was something that prompted them to start their journey seems clear. That it was visible in the sky is also likely. But exactly what it was is hard to say.
Since Herod wanted to kill the child, I would personally think that if there was a physical manifestation still in the sky he would have been able to find the young child himself. But it is also possible that it wasn't as bright an object in the sky at the point the wise men arrived making his job more difficult. Having seen it earlier and still knowing where to look, the wise men may have had an advantage in being able to keep it located as they traveled if it was indeed a physical manifestation. Remember, you didn't just pack up your car and fly off in a private jet and land 30 minutes later in those days. The wise men had probably taken time to prepare for their trip and taken quite a bit more time to travel before ever appearing on the scene.
What is clear is that Herod asked his own wise men and local Jewish religious leaders where the new king was to be born. They told him that He was to be born in the town of Bethlehem. Herod then commanded the wise men to go to Bethlehem and find the child. The Bible only says that they were guided to the house where the young child was. Note clearly the wording for young child and house here.
After Jesus was born, there are few references about what happened next. What there is can be found in Luke 2:21-52. The Bible declares that the baby was circumcised on the eighth day. After this, the next piece of the story says that after Mary's days of purification were ended, Joseph, Mary, and Jesus went to Jerusalem to handle the rest of the religious formalities. The Bible doesn't say whether they remained in Bethlehem during the 41 days of purification or whether they returned to Nazareth, but my belief is that they returned to Nazareth as soon as they were able - probably shortly after the circumcision and that she completed her days of purification there. After all, although they knew she would deliver soon, they probably weren't expecting the child to be born while they were off paying their poll tax and were probably traveling light to make the journey shorter and easier to manage. She would have probably wanted to be near her mother as a new mother herself. Nazareth was quite a ways from Bethlehem, but the distances weren't insurmountable. People routinely went to Jerusalem for feasts, for example.
What is clear is that the Bible declares that after the religious ceremonies they returned to their home in Nazareth. This strongly hints that they returned to Nazareth after the birth and then put a house in order to continue their life. It is very unlikely that the wise men even went to Bethlehem. They are much more likely to have gone directly to Nazareth where Jesus was living. The wise men then gave Jesus - the young child - the gifts they had brought. They then left by a different route rather than returning to see King Herod as he had ordered.
After King Herod determined that the wise men had disobeyed him and not returned to tell where the new King was, he commanded that all children two years old and younger be killed in Bethlehem and the surrounding area (its coasts). The Bible mentions that there was mourning in Rama as prophesied. Rama is 5 miles north of Jerusalem according to my sources, so it appears Herod used at least a 10 mile radius or so around Bethlehem to try to be sure the new King was killed. This would have included Jerusalem, so many children died at his hand. The fact that he picked a two year cutoff indicates that the wise men traveled for a long time before reaching Him. As the Roman governor, his command could have extended throughout the country if he had so chosen.
Fortunately, God warned Joseph to take the family to Egypt and the wise men provided the funding to do this. Jesus was kept safe during this purge.
Thus, all of the nativity sets with wise men at all are in error. For more information about the wise men, see Matthew 2.