This is my first post. Hope it works.
- Ref. to 2.2.1 (“DG2” stall due to closed exhaust damper)
1.1 ) Referring to the exhaust damper of “DG2”, I’d expect its fully open position to be monitored by something like a limit switch in order to automatically shut down the engine as soon as the damper leaves its fully open position.
1.2) Can someone confirm if increasing the exhaust backpressure until the engine stalls is realistic and if it can damage the auxiliary diesel engine? I’d rather expect the engine to be shut down automatically due to an exhaust pressure high alarm or the damper open position monitoring.
Also as soon as the speed decrease below the corresponding frequency limit, the generator trips and that would happen long before the engine stalls though. How the engine is handled after a generator trip depends on how the genset (generator set, an engine driving a generator) controller ist programmed.
1.3) Considering that the “DG2” engine shutdown was not regular, would the restart be automatic as mentioned? Logically the fault should be reset manually and also initiating a restart sequence just by reopening the exhaust damper seems odd to me.
1.4) As HV bus 6.6 kV voltage loss was detected, “DG3” (in stand-by) started automatically. If there’s an automated start sequence including closing the corresponding generator breaker (here “DGR3”, a breaker is some sort of very large switch), why isn’t either “TR1” or “TR2” automatically fed by closing “HR1” or “HR2” and the LV (Low Voltage 440 V 60 Hz) bus fed by automatically closing “LR1” or “LR2”?
Starting automatically another Generator wich is on stand-by when a blackout is detected but not also automatically powering the bus after a short voltage and frequency stabilisation delay is odd. Usually either the whole sequence is automated or the generator is started and brought online both manually.
The report clearly mentions that “HR2” and “LR2” had to be closed manually.
Also it is not clear if “manually” means a manual operator input at the power management system console, the genset controller (from which the generator breaker can or cannot be operated manually depending on how it has been configured), the electrical controls of the breaker or direct mechanical actuation.
Medium voltage (here called high voltage) breakers are typically operated remotely mostly by energizing control solenoids (rarely by de-energizing them), the mechanical energy for the switching process (moving the contacts) is provided by a spring mechanism which is charged by an electric motor (or by hand in case of emergency or if no motor is present), usually the charging is automatically initiated and can be heard and takes only a few seconds.
Some rarely operated breakers can be operated manually. Some are remote-controlled but charged manually.
The spring system always allows at least one opening sequence once the breaker has been closed.
This comment about how a breaker is operated applies to all breakers as the report doesn’t state how very exactly the manually operated breakers have been handled (especially if from the engine control room power management operating panels or locally in front of the breaker, either electrically controlled or operated directly manually).
The control voltage required to remotely control the swichtgear is usually backed up by batteries. Issues with auxiliary control voltage supplies can cause all sorts of problems.
1.5) There are no details about the loss of fuel supply pressure of “DG3”. The trip of “DGR3” was automatically initiated, probably by the genset controller due to generator frequency or engine speed (RPM) fault. Rate of change faults could also have occurred but overall it doesn’t matter which fault tripped the genset as it run out of fuel.
Please not that the discussion above is exclusively related to the incident on 2024-03-25, the day before the allision.
If that fuel supply issue caused problems the following day is unknown. A generator breaker can trip due to problems related the driving engine as well as due to load-sided issues. The report doesn’t mention anything about later breaker trip causes.
- Steering gear, see page 11
2.1) A large Ram Type electro-hydraulic steering gear features typically several (often 4, here 3) main hydraulic pumps as well as typically the same number of small auxiliary hydraulic pumps which provide control oil pressure, the main pumps being variable pumps.
The NTSB mentions 3 Pumps which sort of surprises me, I’d have expected 4 main hydraulic steering gear pumps but 3 are of course possible.
What surprises me more is that when pump number 3 designated as emergency pump is fed from the emergency power (low voltage, 440 V, 60 Hz, 3 phases) its electric motor would rotate at a lower speed. Usually all steering gear pumps, including the small auxiliary pumps, are driven by asynchronous 3-phase motors. Not having all main pump motors running may reduce the angular speed of the rudder stock compared to using all pumps with the same torque load, though running a pump at lower speed if fed by emergency power doesn’t make sense as the required power can be adapted by controlling the variable pump, therefore the speed of the electric motor driving a variable pump doesn’t matter.
Running pump 3 of the steering gear at 2 different speeds (regular or emergency power) doens’t make any sense, it would require either a 2-speed motor, a DC motor or a VSD and all these options just uselessly increase costs and reduce reliability.
The steering gear hydraulic pumps can usually be selected and started remotely from the bridge. It is not known if the hydraulic control is performed by solenoid-controlled valves or a servomotor but it doesn’t seem that there was a failure of the rudder actuation and its controls from the bridge though the angular speed performance was reduced if powered by the emergengy generator (“EG”). The torque is not reduced as the pressure would not be reduced, only the flow would be lower with only one hydraulic pump operating.
Somewhat surprising is, that not all hydraulic pumps can be powered by the emergency generator even if of course only one could be run at the time. If the wrong pump fails when only emergency power is available, the steering is lost. Probabilistic risk assessment…
The emergency generator must anyway be sized to allow some mid-sized motors like a fire pump or a bilge pump to be started.
Is the report accurate when referring to the two different emergency pump speeds?
- Timings vs. CCTV Footage
The timings don’t seem to match the CCTV video (uncut non-manipulated normal speed “raw” version, YouTube from StreamTime Live, duration 07:20) but also unfortunately the NTSB report ist not very formal and neither accurate when it comes to timestamps. I miss a clear chronology. Even if an offset is applied based on the moment of the allision, the report doesn’t seem to match the CCTV recording but I must have a closer look and everyone has probably noticed anyway.
- Smoke
There is no mention about the smoke. Such black smoke is usual when starting an engine, but only for a short time as it’s due to a lack of combustion oxygen in relation to the injected amount of fuel. It also happens during massive load changes when the engine has to catch up with the new load.
Even one of the four auxiliary diesels can generate a lot of smoke but it should not last. If the synchronous HV (6.6 kV, 60 Hz, 3-ph) generator is heavily overloaded the genset trips, typically the frequency drops too low for too long or too quickly (ROCOF protection) and the excitation cannot compensate the voltage drop, so to keep it short, if there an overload the genset can’t handle there won’t be black smoke for long as the generator breaker will be opened (“generator trip”) to protect the generator and the fuel injection reduced to avoid an engine overspeed condition. If there no engine issue, the genset controller will often keep the engine running in order to reclose the generator breaker but it depends on the implementation (hv controls have been set up or programmed).
The NTSB mentions that after the first blackout the main engine (ME) was not restarted (bottom of page 11). I’m wondering if there were attempts to restart it and if such attempts could have lead to the seen smoke, the report doesn’t mention if there were attempts to restart the main engine.
The MAN B&W 9S90-ME-C9.2 is controlled electronically and doesn’t feature a camshaft, also the air starting is handled by individual electrohydraulically controlled individual valves for each cylinder.
4.1) After a blackout, would there have been enough time to try to restart the main engine before the allision?
4.2) Typically after a total blackout there are various subsystems which have to be restarted manually. Can someone confirm if after a blackout some automatic main engine start sequence can still be initiated?
4.3) What lead to the black smoke?
It is unclear if there was an unsuccessful attempt to restart de Main Engine (ME).
4.4) Will propeller drag still turn the main engine at the relevant speeds?
(See the impressive YouTube video called “Container ship engine emergency astern -sea trials-”, the power increase rate is incredible.)
- Power Distribution Diagram (Fig. 5, Page 8)
The diagram is outrageously simplified and IMO it’s even inaccurate as the emergency power switchgear which feeds all emergency-power-fed equipment is located in the emergency diesel generator (“EG”) room or a room adjacent to it (main deck level, often part of the superstructure).
The emergency power distribution Panel is fed with 440 V 60 Hz 3-ph from the main low voltage distribution switchgear (right or left of the “LVR” LV bus-tie breaker). The small “EG” emergency genset (“generator set”, i.e. here a high-speed (1800 RPM for 60 Hz) diesel engine driving a synchronous 3-phase generator), probably around roughly 400 to 600 kVA (only my not-so-educated guess!) does not “send” power back to the main low voltage distribution switchgear as all emergency-power-supplied equipment is fed from the emergency power distribution panel.
During normal operation the emergency busbar is fed from the main low voltage distribution switchgear and during a blackout the emergency busbar is disconnected from the regular supply (to avoid a short-circuit due to phase and voltage mismatch when regular power is back) and fed by the emergency power generator.
It is unknown if once regular power returns the changeover happens without short interruption (retrosynchronizing) or not, it depends on the emergeny genset controller and its setup.
The diagram is sort of misleading as one could believe that EG (emergency genset) can supply the main low voltage busbar (right and left of the “LVR” bus-tie breaker. This is not the case, as detailed above, the “EG” can only supply the equipment which is fed from the emergency power distribution switchgear.
- Missing Breaker Operation or Factual Error in Report?
I’ll try to discuss the electrical issues more in detail in a another post.
First the whole power supply was lost due to the loss of Transformer “TR1”, the cause of the simultaneous (?) or nearly simultaneous (?) trip of “HR1” and “LR1” is totally uknown (possibly a trip due to differential protection but that’s just as example). A trip is a breaker which opens automatically due to a fault, like for example overload, short-circuit or remote-controlled by genset controller).
Interestingly, both Generators “DG3” and “DG4” stayed connected (“DGR3” and “DGR4” remained closed) to the main HV (High Voltage) 6.6 kV Busbar, which means that the excitation regulation and speed regulation worked correctly, if not, the faulty generator would have been disconnected automatically from the corresponding HV busbar segment.
I’m not 100 % sure if the generator power data in the report is correct as generator power is stated in kVA and shaft diesel engine power in kW. Also the data seems to not match well with the ClassNK data where I could only find the total of 20’850 kVA (correctly specified in kVA and not kW), including the emergency power as a total of 5 generators are mentioned.
Possibly the Wikipedia data is more accurate than the NTSB Report data but I’m not sure at all, also sometimes specifications in kW and kVA are mixed.
Generators in the 4000-4400 kVA range like “DG1” to “DG4” (are typically protected by dedicated digital generator protection relays while smaller generators like the “EG” (Emergency Generator) are often protected both by a breaker which trip unit can be more or less advanced (electromechanical (rare by large modern breakers) or electronic trip units, and the genset controller itself.
“DG1” to “DG4” as well the high voltage switchgear (including breakers, digital protection relays, bay controllers and power management system are from Hyundai. Some devices can be identified but their exact model, version and configuration are unknown.
The genset controllers for “DG1” to “DG4” from Hyundai too.
There is no information about the emergency generator “EG” and its switchgear and controllers but it’s rather secondary as it operated as expected.
Once power comes back there is a programmed delay before the emergency load is reverted to the regular power, which explains that the emergency generator (“EG”) “rode through” the second blackout (emergency power fed loads only suffered the automatic startup delay (normally on-line after 45 seconds or less after power loss) after the first blackout).
Referring to page 11, after a sort of suprising trip of both generator breakers “DGR3” and “DGR4” (i.e. simultaneous or nearly simultaneous?, more to be discussed about that), the report does not mention if “TR1” and/or “TR2” are online or not though normal operation seems to be “TR1” or “TR2” with possible short Switchover overlap if some technical design requirements are met.
Referring to the report, as there’s nothing mentioned explicitely I assume that “HR1” and “LR1” weren’t opened as they were both closed to allow the use of “TR1”.
Therefore as soon as “DG2” went online to power the HV busbar, it would have imediately powered the low-voltage busbar as the transformer “TR1” has not been disconnected (as there is no mention about “HR1” and/or "“LR1” having been opened, either under automation authority or manual authority).
Referring to page 12, it is mentioned that at 01h27mn32s local time the crew manually closed “HR2” and “LR2” to restore low-voltage regular (i.e. non-emergency) power, which means that “TR2” is now used to step down 6.6 kV to 440 V.
I know I’m nitpicking but in this odd case of switching events each detail is important.
Implicitly one would expect that “HR1” and/or “LR1” were opened manually or automatically before “DGR2” closed (supposition: “DGR2” automatically closed when voltage and frequency conditions of “DG2” which diesel engine started automatically after voltage loss detection were met; as no other DG than “DG2” was feeding the HV busbar there was no snychronisation required).
- VDR and other Data
The VDR is typically powered by UPS and the discussed model features an internal battery for 2 hours if fully charged and in good state. The VDR itself didn’t stop operating during the whole incident but obviously data links as well as digital and analog inputs became useless as the corresponding source devices lost their supply power.
Various bridge equipment records internally data. Also modern digital control units, including generator controllers, protection relays, bay controllers and of course the power management systems automatically log data. The engine control units of the auxiliary engines powering the 4 generators also record data as does main engine control system.
Overall there should be a large amount of digitally logged data which should allow to find out what happened.
That said there are a few issues.
Ideally, logged data should be retrieved as soon as possible, ideally within 24 hours, or maybe 48 h but not a week.
Some data may be overwritten, some may be lost due to subsequent power off of a device (some logged data may not be stored remanently while other data is not lost).
Devices requiring a backup battery may lose data if the supply power fails and the battery is dead, in other cases the Real-Time Clock (RTC) ist lost, so timestamps can be affected.
Also data can be deleted maliciously, in some cases it can even be forged but that requires specific knowledge (for example SCADA database access if not encrypted for integrity preservation).
For many devices some data can be retrieved by the user but very often the manufactuer can download more data, data which the user cannot access and which is not documented.
Also all setup data should be secured, many devices have various settings, therefore not only the recorded data must be retrieved but also the setup data.
Of course timestamp alignement is required for Real-Time Clocks (RTCs) which are not automatically synchronized. Some devices, especially among the bridge equipment, are synchronized based on GNSS.
RTC mismatch is another reason to secure logged data as soon as possible.
I’m surprised that the critical breakers have not been removed for forensic analysis, or at least some of the digital protection relays and genset controllers. This is a multi-billion USD case. Simply applying a tamperproof sticker won’t help much. Typically in such high-profile I’d have expected the NTSB to install an own recording CCTV system to monitor critical locations like the engine control room (where the LV main witchgear is located) and the MV Switchgear control room as well the area where other critical devices are located.
Also there’s a possible conflict of interest as Hyundai has delivered the complete HV Gensets (“DG1” to “DG4”, 720 RPM medium-speed engines and generators as factory-built unit) as well as the HV switchgear, the protections relays, bay controllers, generator controllers, the power management system and other control equipment.
Therefore basically nearly all equipment which is of major interest is from the same manufacturer.
The main engine has been manufactured by Hyundai under MAN B&W license but the engine control system is from MAN B&W.
- Is there any information about AC UPS and DC batteries, not including those integrated in devices?
Especially some brige equipment and computers and parts of control systems are powered without interruption.
I’ll try to focus on various technical details in a later post as there could be some interconnected causes.
As final note, I regret that the NTSB report doesn’t feature a more serious power distribution diagram, the one provided looks like made by journalists. Also the whole structure of the report lacks formality, there is no clear timeline, time indications don’t include seconds even where known, the generator data is possibly wrong or at least it should be listed correctly as kVA as kW is confusing
.
The transformers for the 440 V 60 Hz 3-ph reefer containers (1400 positions) supply should be shown but there is nothing mentioned about those containers while they can represent a major electrical load. Also the kVA ratings of TR1 and TR2 should have been mentioned.
It can be expected that a lot of technical information will have to be disclosed publicly so why not including more technical data and also the general arrangement (I only found the general arrangement one of sister ship Cézanne in an accident investigation report)?
IMO a more formal and technically accurate preliminary report should have been issued together with a non-binding general public version where diagrams can be simplified and explanations added.
The report as presented is sort of painful to read for anyone with some technical background and many already known factual details could have been added.
More generally, it could possibly help if technical investigation organizations would be led by an engineer or a scientist.
(I apologize for my English, it’s only my 3rd language after German and French.)