The point of the arguments is this: in real life, witchcraft is NOT OK. It is Satanic, and is condemned strongly in the Bible. In both the Old and New Testaments, anyone with supernatural abilities has either received them from God or else from demons; there is no middle ground. But Harry Potter is fiction, and never presents itself as anything but.
There seem to be three criteria that a work needs to meet before critics try to ban it for "promoting witchcraft":
First, magic is used for good, and not just for evil.
Secondly, such "good" (or neutral) magic is used by demonstrably human characters, not just magical or supernatural beings; this exempts classics like The Chronicles of Narnia, Lord of the Rings, and many fairy tales.
Lastly, the magic works in a world that looks and acts like our own.
The last factor may be the most important; it is somewhat rare to see fantastical works that are set in "our" world, rather than a less realistic setting.
A Christian reader of sci-fi/fantasy needs to have a very strong distinction between fiction and reality. Obviously, that's easier to do when the writer does not try to blur the lines between the two. Readers of all ages need to be careful in how they think about stories that are set in the "real world". I find these type of works especially insidious when they are in a pseudo-Christian setting, where characters mix Christian traditions and behaviors with practicing magic. It's easier for me to brush them off as unreal when there is no reference to God or Christians at all.
Christians also need to recognize that most fiction in this realm is written by people who do not have a God-centered worldview, which affects the writers' attitudes towards sex, witchcraft, "gods", and many other thoughts and behaviors which are not desirable for a Christian. A lot of futuristic science fiction, in particular, does not allow for an omniscient, all-powerful God who created and maintains everything in existence. (I find C.S. Lewis's planets trilogy to be fascinating in this aspect: his worldview, like mine, dictates that if there is life on other planets, God created it, and it will worship Him in some fashion.)
Again, I'm not saying that Harry Potter or any other works that meet the three criteria are bad; I quite enjoy them. But I think we need to be careful when reading books that blur the lines between reality and fiction, lest we let too much of the fiction influence our thoughts about the real world.