Because that's a very general use of "comic" or "cartoon". It's much the same as looking at a Picasso, a Japanese print by Hiroshige then a cave painting and saying "Well, they're all painting and we don't need specific terms for them". It's what is more properly termed as "bloody nonsense". One of these is Cubism, the other is cave painting, the last is ukiyo-e. The three are very different styles, but also very different philosophies and way of thinking underlying them.

And while there is hardly one uniform style for manga, or for comic books or for bande dessinées (the primarily franco-belgian, and more generally European), the three have clear distinct philosophies and approaches to the publication and creative control of their material that generally set them very apart; the only thing they have in common is that they involve "drawing a story". This difference in perception and approach in the mainstream industry translate directly in a difference in the public perception of the material.

America's comic industries, is, by and large, industry-dominated. That means that, at least in general, (there are indies but they're rarer) it's the companies that own the characters, the settings and the stories, and they hire artists and writers to create stories from them. These artists and writers have greatly limited creative influence, can be fired if what they do with the character is not popular, and often get mandated certain plotline from above as part of "big events" aiming to sell the company's material better. It's perceived in general as niche entertainment catering to a very specific segment of the population.

Manga is a much more creator-oriented industry. Plenty of series, while their publishing rights belong to companies, are under the firm control of specific creators or authors (for the manga) (there are notable exceptions, of course, where the company control the rights to write more stories in a specific setting). In addition, unlike American comics (and Franco-Belgian bande dessinée), which tend to use artist/writer teams, the manga industry tend to have a single creator who is both writer and artist. Cases of a manga's creator being fired and replaced by another artist/writer by the company publishing it are...uncommon, unlike the American comic industry where short runs are often the norm.

Bande-Dessinée, the franco-belgian comics culture, finally, is even more firmly in the creator-controlled camp than Manga; I can think of about one corporate-controlled series, and that one was created by one of the major publishers based on their mascot. Otherwise, nearly every last Franco-Belgian comic series is, even as of now, under the direct control of its creator, or their heirs if they have died. In some cases (Tintin is especially famous) this means that a long-running series dies with its creator. In others, some "mercenary work" get done after the death of the original creator where the continuation of the series is entrusted to another artist by the heirs.

«The approach to publishing the story is also different. In both manga and comics, the printing pace tends toward frantic, and the focus is on putting out a new chapter/booklet each month; these will eventually be collected. Even there you have a significant difference: American collections tend to follow storylines (eg, they feature all the booklets that make up a specific subplot), this is much less common in manga where collected volumes stopping midway beginning midway through one storyline, continuing into a second one, and stopping midway through that second one is hardly uncommon.

Franco-Belgian Bande-Dessinée take the reverse approach and design from a final product (a hardcover book that tells an integrated story, equivalent to the American collected volume although usually shorter) with a set number of pages. Very often, that's all that will be published; at other times, the story will be published, one page or a few pages at a time, in a magazine, along with pages from several other comics. (Manga are similar in that their chapters are usually collected in magazines with other mangas aimed at a similar audience, while comic books tend to be published as stand-alone booklets). The direct result is that bande dessinée procede at a nearly stately pace: one new publication a year, or even one every few years, is not unusual at all, so a character who has been going on for forty, fifty years can still have a lot of story left to tell without retreading old ground. (Bande Dessinée, relatedly, is perceived as an art form on a level with the big ones in many European countries).

Add in to that that while there is of course cross-influence in matters of styles, there are some clear distinctions between the different art styles and you got three categories that have little in common other than being the "same media". Having terms to distinguish these different industries, and the products that results from each, is a good thing, not a bad one. The more specific terms, the better: "Manga" is thus better than "Japanese comic" (because "Japanese comic" imply it's the same thing as "comics", only done by Japanese people).

(As a note, there is arguably another noteworthy approach/philosophy I didn't discuss above: the newspaper strip. It's largely different in that it's more a gag a day philosophy to the other's storytelling approaches, although some go for storytelling)