But from what I can recall Super smash bros melee did sell quite decently, although it was released in the launch-window term, and other games as well such as Animal crossing or Mario tennis fared well considering the userbase.
In all honesty I was not aware that Sunshine sold that poorly in Japan(on the other hand it did well in EU/NA IIRC).
There are a couple of things:
a) Super Smash Bros is a massive seller. It's always been a massive seller. I don't think many other franchises come close in Japan, especially not 3D Mario. Which brings me to my next point...
b) It was a 3D Mario game. Mario games in general weren't selling as well as they used to (remember, the franchise was revitalized with NSMB on the DS in 2006), because 3D Mario games don't go down well with Japanese audiences.
c) Not only was it a 3D Mario game, it was the worst 3D Mario game- it had a lot of negativity post release, and bad word of mouth.
All these factors, coupled with the GCN's miniscule install base, cumulatively contributed to low game sales, I think In spite of all of this, Sunshine managed 800k LTD if I am not wrong, which is nothing to scoff at.