First, I have a background in archaeology. One of the questions archaeologists want to answer is WHY people did what they did in the past. And no, it never boils down to, "they were primitive and dumb and didn't know any better." People are complicated. The history of America's second most expensive holiday today is complicated.
Second, this is an adult version of the history. If you wish to pass this along to children, some editing will be required.
It begins in Ireland, some 2500 years ago when the Celts (well, there is currently some dispute on that among British archaeologists but we'll stick with Celts until we know better) discovered this was a great place to live. At this time, the prophets Joel and Malachi were working hard to try to get the people of Israel to change their wicked ways. This is some 500 years before the birth of Jesus among a people the Israelites would have termed "Gentiles" (people Jesus and the evangelist Paul would work with centuries down the line).
The Celts had a problem over there in Ireland. They were concerned at their harvest festival, October 31st, when summer officially ended and the people needed to have enough food stored away to live on until spring, that their harvest might be ruined and their people threatened with madness in the dark and dangerous months ahead. They determined the threat to be coming from ghosts, specifically the ghosts of those among them who had died in the previous year. To counter this threat, on October 31st, during the harvest festival of Samhain, adults would dress up in wild costumes (animals, demons, hobgoblins, and witches, oh my) in an attempt to confuse and spook the spooks away. The idea was that the ghosts of the dead lingered for one year with an eye on ruining crops and possessing living souls (human or animal) for a full year until they could move on to the afterlife.
When the living were appropriately dressed in spooky fashion, they would noisily parade through their homes, wreaking mischief as they went, in order to make the place unappealing to spirits looking to take up residence. The costumes also helped keep you from being identified by old uncle Sean and aunt Annie who had died and never liked you much anyway. Once parading noisily had been accomplished inside the house, the process was repeated outside the house, with neighbors gathering together and parading through town. Outside of town, a roaring bonfire was lit, more mischief made, and here things got particularly dark. In fear of the ghost possessed, at times Celtic townsfolk would identify one who appeared to be possessed already among them (it did not pay to be eccentric back in old Ireland) and sacrifice them. Here we see the use of the admonition against occult practices from Deuteronomy 18:9-12. In fear, we can do terrible things heading down dark paths. The "why" in this case remained trying to protect your people from evil.
Around 2000 years ago, the Romans invaded Ireland, in 43 A.D. They found the celebration of Samhain appealing in the dressing up in costumes, the parading, the bonfires, and the mischief. The human sacrifice part they wanted nothing to do with and replaced that with an Egyptian practice of using effigies in royal tombs (instead of the pharaoh's servants, which was a great relief to the staff when the royal leader died), creating effigies if they felt it necessary to appease restless spirits in some way. So, by the time Jesus had practiced his revolutionary ministry in and around Israel, his disciples were performing their ministries (which would be recorded in Acts), and the apostle Paul was evangelizing and letter writing to the Gentiles in the 40s-60s A.D., the Romans were going to be taking up the lighter side of the Celtic Samhain festival. They also fused it with two festivals of their own, one honoring the beloved departed who had gone before them and were missed and the other honoring a goddess whose symbol was an apple. From this fusion, the Romans created a game: you took a large bucket, filled it with water, dumped in a bunch of those apples, shoved your face into the water, and attempted to retrieve apples with your teeth. Bobbing for apples was a Roman addition to what would become Halloween. However, the Romans had other ideas about how to appease the restless dead (those who had passed violently and remained angry) and in Italy, "bones of the dead" almond cookies are still made (looking like finger bones). Those cookies showed respect for those who had gone before them. For more, see the article referenced by Archaeology Trowels and Tools, and originally appearing in Forbes Science: http://www.forbes.com/sites/kristinakillgrove/2015/10/31/how-the-ancient-romans-gave-us-bones-of-the-dead-cookies-for-halloween/
In 609 A.D., the Catholic Pope Boniface added to this growing tradition. Pope Boniface declared November 1st to be All Martyrs Day, which in time become All Saints Day, the day to remember the beloved and missed departed souls who were residing in heaven (those of more questionable destination will be addressed a little later). Deciding it would be a good thing to bring those Irish Celts into the Catholic fold (a really good idea as Ireland produced some great and very determined evangelists afterwards), it was decided to incorporate aspects of the Celtic holiday into a new holiday, including the bonfires and the costumes and parades and such, while jettisoning the darker, occult heritage. A new holiday needed a new name (one that would stick with variation) and All-hallows Eve was born, coming from the Middle English Alholowmeesse.
After the Irish became Christians, they added another wonderful wrinkle to the holiday, using a folktale to spice things up. This is the story of Stingy Jack, an awful sinner, terrible drunk, and obviously penurious guy (here's a story for the kids but keep it gentle, okay). One day this pretty despicable guy manages to convince the Devil himself (note this is the first mention of the Devil you've seen in this history ... and will be the last until the 1980s) to climb a tree (I don't know how or why precisely). Once the Devil was up in the branches, old Stingy Jack quickly carved a Christian cross in the tree, trapping the Devil where he sat. Stingy Jack kept the Devil there until extracting a promise from Satan that he would never tempt Jack to sin again. Sadly for Jack, the promise came too late. When Jack died, he had sinned too many times for even God's grace to forgive. Heading for Hell, Jack discovered the Devil held a grudge against him for his trick with the tree and refused him access to the fiery underworld. Jack was doomed to trudge the frigid nights of earth until Judgement Day. However, Jack did get one concession from the Devil, a single ember to light his way. Jack placed this one solitary burning coal into a hollowed out turnip (Jack hollowed it with his teeth) and this became Jack's Lantern. Every year since, the Scots and Irish had hollowed out potatoes, turnips, and beets, placed lights in them, and set them out in the night as jack-o-lanterns (is this still done by the way dear readers?) These lights served a singular purpose ... can you guess? That purpose was to ward off evil ghosts with scary faces carved into these lights in the night. Here again, we see protection of the people playing a role in All-hallows Eve. Personally, the image of the lit watchmen in the night protecting people from evil, driving back the darkness with the light, is an image I've always enjoyed.
Moving on to the 800s A.D. in Europe, another innovation is added to this complicated holiday. November 2nd had by this time been declared by the Catholic Church to be All Souls Day, remembering the dead of that more uncertain destination of Purgatory, where souls laden with sins to expunge before entering into heaven waited. On All Souls Day, European villages practiced "souling." Souling involved going from village to town seeking square biscuits laden with currants called "soul cakes." For each soul cake given, the person receiving the gift promised to pray for the soul of a dearly departed loved one suspected to be residing in Purgatory. Every prayer that individual received would shorten the time that soul spent in that limbo world. So, it paid to be generous to those who arrived at your door seeking soul cakes. Note that here the theme has become protecting the beloved departed. This is a recurring answer to the why question for Halloween.
Over in the British colonies in North America and the early United States, Protestants were suspicious of this very European and Catholic holiday they had heard about. Halloween gained no real traction in the U.S. until the 1840s and the Irish potato famine. With an influx of Irish Catholics came the Halloween celebration. The dressing up, the treats, the bonfires, and the mischief arrived on American shores. When Protestant kids saw what their Irish Catholic compatriots were up to, they wanted to join in on the action. The mischief turned toward tipping over outhouses and taking people's front gates off their hinges, having nothing to do with scaring off ghosts anymore. The mischief makers had a lot more to fear from the living if they were caught. Once arriving in New England, the Irish abandoned carving turnips, beets, and potatoes. They turned instead the the impressively large pumpkins. While the Pilgrims gave us uses from the pumpkins interiors for Thanksgiving, the Irish gave us a use for the shells at Halloween ... creating the most enduring symbol of the holiday ever.
The U.S. would continue to have a love/hate relationship with Halloween. In the 1920s and '30s, the mischief took a marked jump as vandals took to that aspect of the holiday with far too much enthusiasm. By the 1950s, communities across the nation had enough of that nonsense. They declared Halloween to be a holiday for small children, in which they could trick or treat in hours set by their communities. Now the holiday had gone from a costumed affair for adults with a serious purpose in mind to a holiday treat fest for young children. In 1982, Pat Robertson and the Christian Coalition declared Halloween to be a gateway drug to the Devil, raising opposition to the holiday among conservative evangelicals. Churches agreeing with Robertson and crew created Hell Houses as a way to scare teen straight into the faith, away from evil, and bolster church attendance numbers.
However, the spookiest aspect of the modern holiday was yet to come. Here was a terror far greater than the ancient Celts, the Romans, the Catholics, or the Protestants could imagine. Modern industry has run with Halloween in a very big way. In 1985, the story goes ... and there may be some folktale to this, but who doesn't like a good folktale ... that the candy industry worked hard on senators (even leaving candy pumpkins on their seats in the Senate Chamber in D.C.) to extend daylight savings time to November. Doing so would keep Halloween evening lit longer, kiddies out collecting candy to later hours, and neighbors buying ever larger supplies of treats to give away. Then there's the abomination of "sexy" anything costumes ... but that is too frightening to contemplate. Despite the misgivings of some, today Halloween is the second most expensive holiday on the calendar, topped only by Christmas.
That's the long and complex history of Halloween. Through it all, the whys comes down to protecting our loved ones from evil in the night and remembering those we have loved and lost. Wishing you and those you love the best in all your All-hallows Eves to come.
Had the Irish of old seen the asteroid of 2015 Halloween day come by, they'd have created a whole bunch of jack-o-lanterns to ward off that evil spirit. See: http://jsbrookspresents.blogspot.com/2015/11/asteroid-ultimate-trick-or-treater-for.html
Panati's Extraordinary Origins of Everyday Things, various articles, and The Oxford Annotated Study Bible NRSV.