I was recently asked to share my view on story in games for a research project. Here is my reply:
I think the consensus is there all right, but more then saying story is bad, what companies are saying is that good gameplay is much more identifiable then story, so in a sense story becomes a risk factor. Since a story so clearly can be good or bad, putting a story in the game can equally clearly enhance or reduce the game. Building a strong game with no story e.g. Battlefield 1942 / 2142 is in that regard "safer".
The old stigma for the game industry regarding story is the linear nature of a story. Only if you make a linear game can a strong story really add to the game in a big way, but even then you take the risk that the story won't be popular with the gamers.
Another factor why story always takes a beating is because it doesn't really matter... This becomes clear in games with a bad story. Take Gears of War for instance, the story is OK I guess but compared to literature it's very basic and flat like an action comic. Critics and players alike love Gears of War despite this fact which suggest that gameplay/graphics etc is much more appreciated among the audience then story.
I do think that traditional storytelling will continue to face difficulties in games and that the best way ahead is to adopt the art and craft of storytelling into the games medium.
For instance, background story has a wider use then story per se, look at the warcraft universe, they have a very strong back story that they draw from when creating the game, however, they also have a story, that is fairly complex but most players don't know about the specifics of the actual story because it doesn't affect them in any way.
To integrate storytelling in games I think it needs to be merged with aspects of game design and simulation so that the story can grow within the game and also have an impact in the game.
The bottom line is I think, stories are unpopular because they don't matter. Make the story matter within the boundaries of the game.
To mention a few great examples where story have been integrated to some extent within the game:
Beyond Good and Evil
Half Life 2
These games have made an effort to take as much as possible from the story/back story and make it real in the game, this is one of the huge advantages with the game medium, not only create this and that weapon but also having huge tv-monitors on walls talking acting out the story, propaganda leaflets and posters on the walls etc to include you in the story you could otherwise just read about.