I have eagerly awaited this report, both because it is a fascinating study in organizational failure and because I personally know people involved. I have some private thoughts about why it took five years to prepare the report. Suffice it to say that there are political sensitivities involved.
I haven’t had the time to properly read the report yet, and will update in due time, but figured I’d post it right away. Here’s the PDF (in English):
There’s also an official accident investigation film, which provides a quick blow-by-blow and summarizes some key findings overlaid on fairly dramatic footage (English subtitles):
In the 10 seconds immediately following the blackout, a total of approximately 1,000 alarms went off in the IAS. Recovery from a blackout without a standby generator had never been drilled on board. The engineers were therefore faced with a situation they could not readily recognise and were not practised in managing.
It took about 14 minutes from the blackout until refilling of lube oil was started. The engineers
struggled to restart and connect the DGs, and the first DG was connected to the MSB almost 24 minutes after the blackout, and it took another 9 minutes further to restore any propulsive power.
During this time, the vessel drifted to within a ship’s length of running aground despite the attempt to arrest its drift by deploying both anchors. The blackout recovery is analysed in section 2.9 and the evacuation and rescue operation is analysed in section 2.10.
The term alarm or alarms appears 327 times in the report. Basically sounds like the alarm system was poorly designed under normal conditions and in this incident the alarm system turned into a pile of dogshit.
The design and configuration of the alarm interface produced unacceptable
workload which would have had a negative effect on engineers’ processing ability to the extent that it is very likely to have decreased their problem-solving and decision making ability. In situations like a blackout where the alarm burden is extremely high, priority and grouping of alarms are of specific importance.
The NSIA’s observations are also supported by the internal report by the ship management
company where it is stated that the configuration of the alarm system provided an incomplete
understanding of the risks. As they found there were too many alarms and lack of criticality
indicators, they stated that a further study into configuration of the alarm system was
recommended.
I’ve second-hand knowledge of two serious cases of black-outs. In one with the Emergency Diesel Generator (EDG) operating and a second case without the EDG.
In the first case it took 2 or 3 hours to get back on-line. In the second case, without the EDG it took 3 days. Got back on-line just in time to avoid having to be towed.
I think that ship was damn lucky they had engineers that could get back on-line so quickly.
Viking Sky is Diesel-Electric, with 4 x Diesel generators as her “main engines” and 2 x electric driven “Azipods” as main propulsion.
She has SRtP class, so total blackout should not be possible. But apparently it can, when one of the generators are out of order and put off-line.
Let us hope that the lessons learnt from this “near miss” and the recommendations issued by NAIB will be hided. Unless it make it into international and national law and rules it will be another wasted opportunity to improve shipping safety.
Agree with K.C. I can only imagine the chaos when the lights went out. I don’t know what the standard is, but I thought 39 minutes from blackout to getting engines back online to be pretty admirable.
We used to do full blackout recovery testing between wells on the semi sub, 45 seconds and your back on DP.
I guess a ferry only has passengers so of less value than drill pipe and slap ya mum’s chilli powder and cheetos,
so built to a lower standard.
Evidently there was a latent error with the documentation of the lube oil system.
None of the vessels in the fleet of five sister vessels had been provided with instructions on correct lube oil sump tank filling levels or alarm setpoints. In June 2016, the engineers working on board the sister vessel Viking Sea requested information from MAN regarding the recommended oil levels. MAN was unable to give a clear answer as the tanks were designed by the shipyard and not by them. The shore organisation of the ship management company was made aware of the email exchange between Viking Sea and MAN. However, no guidance on correct filling levels or alarm set points was issued by the ship management company until after the accident on Viking Sky.
Also problems with the design:
The shipyard’s design process did not effectively ensure that the lube oil sump tanks complied with all applicable rules, regulations, and recommendations. Likewise, the plan approval process of the classification society was ineffective and did not ensure that the sump tank design, which is critical to safe engine operation, was compliant. Further, no technical guideline or industry standard for application of the SOLAS requirement existed, making design and verification difficult
We used to do the same on drillships but I was on one where the EDG failed during a blackout. There was no script for that. at the time. Kudos to the Viking Sky engineers
This is more common than one would think. I’ve only seen two ships where this information was documented in the drawings. On other ships the operating range for the sump was learned through experience. I used to not give it much thought prior to El Faro, now I run closer to the high level alarm.
In the case of the Viking Sky in rolls of 3-4 degrees with max of about 6 degrees DG4 shut down first followed by DG2 a few seconds later. The vessel slowed to below 6 kts and the stabilizers retracted automatically and the roll increased to 4-6 degrees with max of about 12 degrees.
The engineers got one of the DGs back on-line after 11 minutes but then 3 minutes later DG1 and DG2 tripped out and ship suffered a blackout with loss of steering and propulsion.
That was quite an interesting read for all the various findings of what lead to the event. Well written for an accident report.
I still put some criticism on the Engineers for not addressing/recognizing the low LO level as likely cause more quickly, especially after the initial repeated alarms and the load reduction of the first engine due to oil pressure. After the blackout it was 14 minutes before someone started adding oil.
But holy hell were they set up for failure:
Alarm levels and time delays were changeable by anyone with access to the IAS screen, and were therefor changed several times over the years to lower levels and longer delays, with no apparent record.
More than just a “documentation” error…the sump tanks were built by the shipyard and not the engine OEM and were not actually built to SOLAS or Class or OEM requirements for pitch/roll. They were 30-40% smaller than needed to meet the regulations. Crazy that it wasn’t caught by Class, who blamed it on Builder, who assumed their design was approved by OEM (just because OEM had provided unrelated design feedback).
The alarm system was utterly horrendous…an alarm for refer fridge door open was given same display queuing priority as low engine oil. 1,000 alarms logged 10 seconds after blackout.
The automation integration between various logic systems to the alarm system was equally horrendous…the shutdown registered immediately but the alarm condition that caused it registered several seconds later.
A note about the blackout recovery: The EDG started and was online within 20 seconds, so they were never without lights or emergency circuits. The lack of “standby generator” in the report is referring to the fact that there was no other main generator (1-4) ready to start. 1, 2, or 4 because they had tripped and 3 because it was out of service.
One last nugget that I found particularly interesting, though unrelated to the accident was this one:
Viking Sky had previously experienced two turbocharger failures (DG1 and DG4) in June 2018. During the two months prior to the failure of Viking Sky’s DG3 turbocharger there were five turbocharger failures on three of the sister vessels. Since September 2016 the five sister vessels in the fleet had experienced a total of 33 turbocharger issues
That feels high. Not sure what Wilhemsen was doing with all that data they were collecting, but apparently engine reliability wasn’t one of the concerns.
Good point about the EDG, it wasn’t a blackout but a loss of propulsion and steering.
Given hindsight bias it seems impossible to say for sure precisely what the engineers in this case should have known or done.
Apparently they believed at the time that there was a good possibility there was an issue with the entire control system, which would explain why there was the low lube oil alarm.
They were getting that alarm very often and it was just silenced without investigation, same as with the ECDIS alarms on the deck side.
The remote lube oil sump tank level monitoring system was complex, and the resulting onboard
measurements were inaccurate and unreliable.
The engineering crew on board Viking Sky had gradually lost confidence in the remote monitoring system. Since the level alarms were generated by the remote readings, the crew did not take the level alarms as a true indication of the actual level.
At the rate cruise ships are being built and getting underway, I cannot see anyway well-trained crews can be available to man them. Or for that matter, how long before the cruise ship industry collapses under it’s own weight. Also, there are bound to be “language issues” with crew members speaking many different languages. How do you train or even communicate with that varied a crew?
The Marine crew (Deck & Engine) are recruited from the same pool of seafarers as for other commercial ships, based on STCW’10 qualification requirements.
Special training for tasks that are different from other ships are given per IMO requirements.
I don’t know whether there are any major difference for the Engine crew between a Cruise ship and any other vessels with Diesel/Electric propulsion? (Other than more “hotel services”, that is).
Maybe that the Viking Sky had SRtP class notation?:
The biggest difference between the eclectic system on most MODUs/Drillships and the Viking sky is probably that there are no SCRs involve here:
Been a while since you’ve been on a drill ship eh? Not only propulsion / thrusters use big drives (VFD’s) now but the drilling motors too. So a person from the drilling industry (licensed engineer anyway) would be very familiar with the plant.
Yes it has, but I’m aware of the development, both on rigs and ships.
MOST MODUs in the world are still not there yet, though.(Just look at the fleet lists)
The latest now is back to DC all the way. No need VFDs., or the old SCRs.
Easier to convert what little is required of A/C from D/C I presume.
