At 7:04 am on a fall Thursday in Tokyo, administrators of the world’s third largest stock market realized they had a problem. A critical data device for the Tokyo Stock Exchange trading system had malfunctioned and the automatic backup failed to kick in. It was less than an hour before the system, called Arrowhead, began processing orders in the $ 6 trillion stock market. Exchange officials saw no solution.
The entire closing day that followed was the longest since the exchange transitioned to a fully electronic trading system in 1
“Trade is a crucial part of the market infrastructure and it is unacceptable that trade opportunities have been denied,” Finance Minister Taro Aso told reporters in Tokyo. “You’re dealing with machines, so it’s always possible for them to break down. They have to build the infrastructure with the possibility of failure in mind.”
TSE’s Arrowhead system was launched with great fanfare in 2010, heralded as a modern solution after a series of outages on a previous system embarrassed the exchange in the 2000s. The “arrow” symbolizes the processing speed of the TSE. order, while the “head” suggests robustness and reliability, according to the exchange. The system of around 350 servers that process buy and sell orders has had a few setbacks, but no major disruptions in its first decade.
That all changed on Thursday, when a piece of hardware called shared disk device No. 1, one of the two square-shaped data storage boxes, has detected a memory error. These devices store management data used on servers and distribute information such as commands and combinations of IDs and passwords to the terminals that monitor operations.
When the failure occurred, the system should have performed the so-called failover – an automatic switch to device # 1. 2. But for reasons that the exchange executives failed to explain, this process also failed. This has had a ripple effect on servers called information distribution gateways which are meant to send market information to traders.
Data that disappears
At 8am, traders getting ready at their desks for the open market an hour later should have seen indicative prices on their terminals as orders were processed. But many saw nothing, while others reported seeing data appear and disappear. They had no idea if the information was accurate.
A minute later, the stock exchange made its first communication, informing the system administrators of the securities firms that there had been a problem. In some brokerage firms, this wasn’t immediately filtered out to confusing trading desks.
At around 8:05 am, Twitter, often used by traders to communicate outside the most official communication channels monitored by compliance, started buzzing with rumors about an issue. Traders described a growing sense of confusion as few responses came from the stock exchange.
“We didn’t know if it was our system or the exchange,” said Masaya Akiba, a broker in the stock trading department of Marusan Securities Co. “We only confirmed it when the exchange released a statement.”
At 8:36 am, the stock exchange finally informed the securities firms that trading would be halted. Three minutes later, he posted a press release on his public website, albeit only in Japanese. A confusingly translated English version would not follow for more than 90 minutes.
It was the first time in nearly fifteen years that the exchange had experienced a complete disruption of trading. The Tokyo Stock Exchange has a policy of not closing even during natural disasters, so for many in the capital’s trading floors this experience was a first.
Some market participants were angry about the closure. Others, with nothing to do, spent their time reading research notes or trading goods.
“I didn’t think about it much at first,” said Kiyoshi Ishigane, chief fund manager of Mitsubishi UFJ Kokusai Asset Management Co. in Tokyo. “Previous outages were resolved quickly, so I figured orders would just be delayed.”
In 2012, after switching to Arrowhead, the exchange quickly resolved limited issues. Many expected the stock exchange to do the same this time too.
But as the hours went by, Hajime Sakai, the chief fund manager of Mito Securities Co., became increasingly uncomfortable.
“I couldn’t really pay attention to much else,” he said. “I wasn’t like, ‘Open the market!’ It was more like, ‘whatever it is, make your call, fast’ “.
The call was daunting. After the failed backup switch, the swap had manually forced a switch to shared disk device # 1. 2. At this point, the administrators had a choice: they could try to restart trading, but that would have resulted in a full system reset, turning off the power and restarting.
Data relating to orders already received from securities firms would have been lost, without having been canceled. This would have led to anarchy, securities firms at the exchange said. After speaking with market participants, the exchange made its decision: trading would be suspended for the whole day.
Many on the market say they have been relieved. A call to resume negotiations would have been chaotic, a worker at a Tokyo-based brokerage firm said, with no way to tell which orders from existing customers remained active, while also trying to process new bids and offers.
At 4:30 pm local time, four TSE executives, including CEO Koichiro Miyahara and Information Director Ryusuke Yokoyama, addressed reporters during the exchange to explain the hiatus. In a briefing that lasted around 100 minutes, they bowed in apology in front of the crowded room before entering into a detailed technical discussion of the fault.
If the stock exchange was criticized for its communications earlier in the day, it received praise for how it handled the press conference. Executives answered media questions with relative ease, discussing areas such as systems architecture in highly technical terms. They also openly accepted responsibility for the incident, rather than trying to divert the blame to the system supplier Fujitsu Ltd. It bore little resemblance to the gaffe-filled briefings of other Japanese companies in the past. On Twitter, the Japanese public expressed their approval.
“The management explained very clearly during last night’s briefing,” said Megumi Takarada, senior analyst at Toyo Securities Co. in Tokyo. “The briefing provided some reassurance that management clearly understands the problem.”
Later in the evening, the announcement came that the stock exchange would restart trading on Friday. Although it passed without problems, many questions remain unanswered. According to local media, the Financial Services Agency has ordered the exchange to publish a report on the outage that could provide more information on some of the issues.
But one of the biggest is whether the same kind of hardware-induced failure could occur in other stock markets. For a strategist, it almost certainly could, but that’s not something to worry about.
“There is nothing uniquely Japanese about this,” said Nicholas Smith of CLSA Ltd. in Tokyo. “I think we just have to put it in the box of ‘things that happen’. These things happen. They shouldn’t, but they do happen.”
(Except for the title, this story was not edited by the NDTV staff and is posted by a syndicated feed.)