Yes, you probably should get insurance. travel.state.gov:
While medical care in Japan is good, English-speaking physicians and
medical facilities that cater to U.S. citizens’ expectations are
expensive and not widespread. Japan has a national health insurance
system which is available only to those foreigners with long-term
visas for Japan. ... Medical caregivers in Japan require payment in full at the
time of treatment or concrete proof of ability to pay before they will
treat a foreigner who is not a member of the national health insurance
plan.
So basically, if you're hit by a bus or have a heart attack or something, you're going to be stuck with a large bill and could potentially even be denied care. Japanese hospitals tend to be a bit overloaded, every now and then a case of somebody dying after being denied entry to multiple hospitals makes the news, and not having insurance is certainly not going to help.
That said, simple doctor's consultations and the like are quite inexpensive in Japan even if you foot the bill yourself. So I'd look specifically for basic/catastrophic travel insurance that will cover you in cases of dire need, but has a high deductible/does not cover minor mishaps and thus has a lower premium.