I don't like it.
What it comes down to is this, a starter is more important than a closer. Most of the time when a closer saves a game, you'd have won the game with a lesser pitcher. With an excellent offense and an excellent rotation you aren't going to have all that many save opportunities and even a mediocre closer is going to save most of them.
The rotation protects the bullpen. The bullpen on this team is the weak link. Offense among the best, rotation ranging from good to astonishingly good, rotation only okay. Well sure, moving Papelbon to the rotation improves its quality but it also probably puts more of a strain on it and here's why. Papelbon throughout his career as a starter pitched six plus innings fairly regularly. Julian Tavarez isn't going to be able to do that. Kyle Snyder isn't going to be able to do that. I'm not sure any of the fifth starter candidates can be expected to do that. As a result you're putting more innings on the arms of the other guys in the bullpen. You know, the ones you were worried about.
Now there's a lot of guys who can fill that fifth spot and in reality they're all just holding it for Jon Lester so it's not that big a deal for this year.
For the long term future though, I wonder. With Papelbon in the rotation you could let Schilling go and enter 2008-12 with a rotation of Matsuzaka, Beckett, Papelbon, Lester, and Wakefield. With Papelbon in the bullpen sure you have anchored that spot for most of the next ten years and that's good but it means you're not going to have the excellent young rotation. Matsuzaka, Becket, Papelbon, Lester could have been the Red Sox answer to Maddux, Glavine, Smoltz, Avery. Now you're faced with either keeping Schilling which would give you two 40+ pitchers or finding someone else.