I think one important factor is how the library/bookstore is being used. A library is much more likely to be used for concentrated research than a bookstore. So you are more likely to pick a narrow subject (could be for a school paper, or planning a trip, or what have you) and get a bunch of books all on the same subject. There is no extra cost to getting the lot.

In a bookstore, people are more likely to just be looking for the one or two best books on the subject (which they may have researched before coming into the bookstore) or the latest books, on a variety of subjects, that they've recently read good reviews of or have just come out by their favourite authors.

Since their intended actions are different, the optimal system for organizing the books may be different, too. Or so it seems to me.