The evil Bible argument is by far the most effective argument atheists have against Christianity, but it really shouldn't be. Its strength comes primarily from the emotional sensibilities of those who don't bother to look deeper into the issue.
As human beings, it's only natural, or at least it should be natural, to care for children. Thus, atheists are capable of pointing to infanticide, alleged or otherwise, in the Bible, and instantly turn people off it. The irony? These are often the same people who campaign not only for abortion on demand, calling it "healthcare", but also argue for this using disabilities as an excuse. I've even met atheists who are more willing to argue that we should abort babies than that we shouldn't kill them.
This is quite obviously sickening, and is so brazenly hypocritical that I don't understand why people fall for it, but if I was to just leave it at that, it would be the tu quoque fallacy. It wouldn't answer the argument just because it works just as well against atheism, which does indeed justify infanticide. The question is, does the Bible?
First off, it's worth noting that a lot of the verses atheists quote to suggest that it does are taken wildly out of context. Most notably, Psalm 137:9 is an imprecatory Psalm against Babylon. What that means is it is not prescriptive. It is a poem by a Babylonian exile, who merely wishes that the atrocities Babylon committed against Israel would in turn be committed against Babylon. This is evident from the rest of the Psalm, but not many atheists are diligent enough to actually read the context.
Furthermore, the Bible engages in hyperbole, just like we do in English. God sometimes commanded the extermination of entire people groups, yet followed it up with "Oh, and by the way, when you've done that, don't intermarry with them". (e.g. Deuteronomy 7:1-3). Obviously, you can't intermarry with a group you have literally destroyed. Several of these groups even show up later on, showing they weren't completely destroyed.
Furthermore, the entire reason for all of this was, according to God's own words, to prevent a greater evil. In this case, Israel becoming corrupted by pagan groups. Pagan groups who, coincidentally, would engage in child sacrifice. Brutal child sacrifice. These people would set fire to alters, upon which they would place their babies, and would bang drums so loud that the parents couldn't hear the screams as their child burned alive, like an egg on a frying pan.
Now, tell me, was God justified? Obviously, the correct answer is yes, but based purely on your own moral compass, is it ok to destroy a culture that kills babies in order to prevent a culture that doesn't kill babies from becoming another culture that kills babies? God, in His omniscience, knew that if these cultures survived, Israel would learn their ways, and so He commanded Israel to put an end to their ways instead.
Ironically, many atheists complain that God doesn't do this kind of thing often enough. If God had killed baby Hitler, the world would probably have been better off, but it can be assured that atheists would criticise God heavily for it. Yet, they also criticise Him for not doing so. This is the same scenario. God cannot please atheists because they don't want to be pleased. If He stops evil, they call Him evil. If He doesn't stop evil, they call Him evil.
The simple reason for this is that we are evil. As God, He has the right to dictate what is good and what is evil. God gives life to whom He pleases, He takes life from whom He wills, and there is nothing that restricts Him from doing so. Everything is His. This makes it all the more amazing that He defaults towards mercy.
See, God cannot tolerate sin. All evil must eventually be dealt with, be it child sacrifice, or cussing people out on the internet. Unfortunately, the penalty for sin is death. Yet, God takes no pleasure in the death of the wicked. He prefers instead that they would turn from their sin and live. Thus, He gave us the greatest gift He could: The death of His own Son. Jesus Christ came to Earth as an innocent baby boy, and He remained innocent for His entire life. That life was cut short when He was publicly executed on a cross, where He bore the full wrath of God for sin. Because of this, we can be forgiven for our sins, and fully reconciled to Him. All it takes is faith. Confess Jesus as Lord, and believe He rose from the dead. If you do this, you will be saved.