Japan’s next prime minister will face challenges from inflation, volatile financial markets, a worsening security environment and the transition to a new president in the US, Tokyo’s most important ally.