Thanks for the baltimore bridge update.
The terminal block is a screwless Wago “CAGE CLAMP” 2.5 mm2 for 35 mm DIN Rail (the most widespread rail type), Wago Art. 280-681. The catalog can be downloaded, 2023/2024 page 314 for the article and page 311 for the contact system design (German catalog 2023/2024 same pages).
Formally speaking it could also be the Ex version (281-681) but it does not matter for this discussion, also some ATEX specified articles are identical excepted the markings but sold at a higher price, for example some increased safety (Ex e) plugs.
For 0.08 to 2.5 mm2 (28 to 12 AWG), In 24 A.
I do not know very well this model, I am more used to the Phoenix Contact. Wago is very widespread though some in the industry do not consider it as top notch. Opinions vary, for some items Wago is better and for some Phoenix Contact or others. Personally I barely use Wago products though their I/O modules are very widespread.
Sure is that I would not recommend such type of terminal for 0.08 mm2.
Problems are mostly related to handling errors when inserting wires, when opening the cage with the screwdriver it is very important to check that the wire is introduced and hold correctly, especially deep enough, it happens that the cage does not open correctly and it also happens that the insulation instead of the wire is pinched.
Another issue is maintenance and troubleshoting if a wire must be temporarily removed, without crimped ferrules single strands often break.
Personally I do not like screwless terminals but they are a good choice for ships due to the vibrations (though there is general design flaw as most cabinets are not mounted with shock absorbers (e.g. wire rope isolators are a good choice)).
Overall the Hyundai electrical design is anyway poor compared to modern electrical engineering.
IMHO using undervoltage trip coils is a fundamental design error for DG and TR breakers, it is more reliable to use two (redundant) shunt trip coils with coil monitoring. A shunt coil trips a breaker when the coil is shortly powered. A undervoltage coils trips the breaker when it is not energized, to close the breaker it must be energized.
Using undervoltage trip coils leads to all sorts of problems and should only be used for dangerous machines (emergency stop function and some cases where breakers are used instead of contactors). Also undervoltage coils cannot be used for “true” undervoltage protection functions as their release voltage range is too wide and delays cannot be programmed either (unvervoltage is handled by the protection relays though).
With undervoltage trip coils the probability of spurious trips is much higher than with the way more commonly used shunt trip coils.
There would be a lot to say about various more or less major electrical design flaws, including large 4-cylinder ram-type steering gears.
Now let’s just hope they did not use massive low cross section conductors.
Buzz click whir click click buzz whir buzz buzz.
Investigative Update Release Date 24 June 2024
The information in this investigative update is preliminary and will be supplemented or corrected during the course of the investigation.
This report provides an update to the NTSB’s preliminary report issued on May 14, 2024, concerning the March 26, 2024, contact of the containership Dali with the Francis Scott Key Bridge in Baltimore, Maryland, and the subsequent collapse of the Key Bridge. Investigative activities have continued since the preliminary report was issued.
NTSB investigators have completed in-person interviews of the vessel’s crew. Onboard examination of engineering systems and testing of electrical systems has been completed. Documentation of the damage to the vessel structure is ongoing.
During the accident voyage, electrical breakers HR1 and LR1 unexpectedly opened when the vessel was three ship lengths from the Key Bridge, causing the first blackout (loss of electrical power) to all shipboard lighting and most equipment. While examining and testing the vessel’s electrical power distribution system and control circuitry, NTSB investigators (in coordination with vessel crew and parties to the investigation) noted an interruption in the control circuit for HR1’s undervoltage release.[1]
Exemplar terminal block identical to model removed from ship. (Source: WAGO)
NTSB investigators subsequently removed an electrical component (a terminal block; see figure) from the control circuit for HR1’s undervoltage release.[2] Two portions of control wiring associated with the terminal block were also removed. We continue to examine the removed components at the NTSB Materials Laboratory. We will continue to evaluate the design and operation of the vessel’s electrical power distribution system, and investigate all aspects of the accident to determine the probable cause and identify potential safety recommendations.
[1] An undervoltage release is a device that opens a breaker when voltage falls below predetermined thresholds.
[2] A terminal block is an insulated block that connects two or more wires together.
I forgot to mention that the terminal block simply interconnects the 3 connections, the round front holes are for the conductors and the square ones for the tool used to open the connection cage, the used tool is typically a screwdriver though preferably one with a parallel blade, not in form of a spade as it can widen the opening and damage the terminal block, a quite common issue.
This type of terminal block, regardless of the manufacturer, can sometimes cause connection problems, it happens that the block (if with one open side) has to be removed to check if the wire is clamped correctly, unfortunately many do not pull on the wire to check if it is retained correctly.
If some parts are deformed, mostly due to incorrect handling, the connection can fail.
Problems mostly occur while wires are connected and when copper strands break, compared to screw terminals the probability to damage accidentally the contact system is higher, it is mostly caused by handling errors though there can also be manufacturing quality issues.
Personally I had far less issues with good screw terminals (like Woertz or Phoenix Contact for example) than with screwless terminals but YMMV.
.
Using undervoltage releases to keep breakers closed requires absolutely rock-solid control circuit power. Even short voltage dips can lead to spurious (i.e. randomly accidental) breaker trips, also there is an inrush current when energizing a breaker control coil and that must not lead to a voltage dip which prevents the coill of the undervoltage relase from holding the breaker closed.
Also as undervoltage coils must remain permanently energized their lifetime is often shorter than the one of shunt coils which are typically only shortly energized to open (shunt trip coils) or close (closing coils) a breaker, otherwise they remain cold and, if monitored, only a very small current is used to check their state (to detect if there is an accidental interruption in the circuit including the coil).
Undervoltage releases do not trip a breaker at a precise untervoltage threshold with a predictable reclosing ability hysteresis, there is a relatively large tolerance (for example 35 to 70 % of the undervoltage release coil supply voltage to trip and 85 to 110 % to hold the breaker closed but the effective values depend on the installed breaker).
Therefore if a short of voltage dip occurs, only some breakers may trip and during the next voltage dip possibly not the same breakers will trip again.
To achieve a deterministic behavior of a breaker, the control voltages must not be too close to the limits as tolerances could lead to unpredictable behavior.
Control voltage stability issues could possibly have caused DGR3 and DGR4 to trip together if using undervoltage trip coils but we don’t know of course.
As the HV BUS remained energized through the totality of the 1st blackout, it is normal that DG2 which was on stand-by (preheated and prelube running) did only start automatically at the beginning of the 2nd blackout, it was also connected automatically to the HV BUS as DGR2 was closed automatically but TR2 did have to be brought online manually which is odd.
Nothing is known about DG1 but it was probably not on stand-by as it did not start automatically as the HV BUS lost power, as two DG’s were running just before DGR3 and DGR4 tripped, the PMS would logically have started the two 2 remaining DG’s, as only DG2 was started it can be supposed that either DG1 was not on stand-by and/or the PMS did not work correctly.
The EBUS was not affected by the 2nd blackout as it was still supplied by the EG.
It is not known when DG3 and DG4 were shut down after DGR3 and DGR4 tripped, typically such medium-speed diesel engines are not run for more than a couple of minutes without minimum load.
In any case the power generation and distribution automation is poorly designed, as is the auxiliary engine automation.
Not monitoring a DG exhaust damper fully open position is just a proof of very questionable engineering.
Various design flaws also affect the electrical part of the steering gear and the list continues, even the ME crankshaft angle acquistion is not well designed. Some ME feature a poorly chosen encoder interface type or even a timing belt drive (!).
A container ship like M/S Dali is a mix of good sound design and awfully bad engineering.
Overall the electrical design is not great, about the only reasonable parts are the ME controls provided by MAN B&W, the integrated navigation system and a couple of stand-alone local controls and devices.
The overall integration (excepted for navigation) is not great and the ME control room design is just awfully outdated and its bad ergonomy influences security quite negatively and reduces the overall situation awareness.
I am very surprised that Maersk, a company with plenty of good engineers, does not design serious electrical systems according to IEC practice internally with European electrical components rather than letting Hyundai design antique curiosities.
Also using COTS PLCs and HMI panels from known manufacturers like for example ABB or Siemens will be way less expensive than relying on proprietary hardware (which sometimes looks great but features consumer-grade low-cost switches).
The whole automation and power management could be performed using freely available well-documented components. Add some Kwant controls and there is no need for very expensive proprietary hardware.
And BTW, the steering gear controls as well as the whole steering control and the ME remote control (non-MAN-B&W part) should be redesigned. Especially also some used sensors are among the least reliable ones and also other design choices are odd.
Not even pilot light and display panel dimming is well designed, logically there should be something like an external 0-10 V general analog control voltage and a local rotary control for each device to balance the luminosity locally.
The dimensions of the larger rectangular panels cutouts are neither standardized, often not even those in the usual DIN (now IEC 61554) cutout size range.
That lack of standardization leads to some sort of ergonomic Babel Tower of controls and displays and in the end the crew covers some pilot lights and displays which cannot be dimmed dark enough with cardboard.
(Now I’m busy so there won’t be any long reply for some time, please feel free to criticize me but please mention where I am wrong. 
)
Can one of you engineer types summarize this so even a simple deck ape can understand?
I’m a deckie too but FWIW it sounds to me like a loose wire in a terminal block caused a drop in voltage and a breaker tripped.
There’s got to be more to it of course, a single loose wire shouldn’t shut down the whole show.
The end of my previous message was of course not directed at Kennebec_Captain. Sorry if there could have been a misunderstanding.
As Kennebec_Captain said, the NTSB suspects a faulty connection which led to an unwanted disconnection (either permanent or spurious, we do not know) but not powering an undervoltage breaker release leads to a breaker trip initiated after a fraction of a second, the delay depends on the breaker (IIRC typically under 0.1 second (100 ms)).
There are also undervoltage protection accessories which allow to ride through very short undervoltage release control voltage interruptions, e.g. adjustable in several steps between 0.5 to 3 seconds, just as example, we do not know if such option has been installed.
Obviously this is only a part of the story. We do not know what caused DGR3 and DGR4 to trip (which caused the 2nd blackout) though it was most probably a common cause.
Also we do not know if first a DGR breaker tripped for any onknown reason leading the remaining one to trip due to overload.
Normally all trip causes (if known) and trip timestamps (even if cause unknown) should have been recorded automatically.
About the reefers I supposee there were not that many as M/S Dali did leave port with only 2 DG’s (DG3 and DG4) running. Even if neither the steering gear necessarily required the full power, it depends on the forces applied on the rudder and the required angular speed of the rudder stock, nor the BT (limiting the propeller pitch as there is no BT propeller RPM control), it would have been wise to run at least 3 DG’s.
I am even surprised that it was not mandatory to run 3 DG’s during departure.
The closed exhaust damper (known human error), the fuel supply issue (probable human error?), both the day before, than the day of the incident TR2 gets disconnected and than DGR3 and DGR4 both trip… Different issues without obvious relationships between these events.
Indeed there are quite some single points of failure.
E.g. I am not sure if there is a redundancy for the the breaker supplying the EBUS. In some cases the EBUS can be powered from either LV BUS segments (left or right of the LV Bus Tie Breaker LVR but I am not sure if the PR diagram page 8 can be relied on as the EBUS is not represented correctly).
My overall impression is that several not necessarily directly related things went wrong though there is a lot of information missing to draw definitive conclusions.
That said, IMHO the PMS (Power Management System) did not work correctly as transformers were not reconnected automatically. For now it is unknown if it was due to design issues and/or human errors and/or technical failures.
Hi Bayrunner
The DGs on the vessel are 6.6kV. This voltage is stepped down to 440V via a 6600/440V transformer for all the engine room ‘essential’ power requirements. These transformers have breakers before and after. The transformer breakers are equipped with an UV coil. The purpose of a UV coil is to arm or allow a breaker to be closed only if the UV coil has power. In this case, this UV coil will prevent the breaker from being closed unless there is voltage on the 6600 board and the UV coil is energized. (Safety feature. Somewhat similar to your motor bike …. You cannot start on the side stand unless it is on neutral).
Voltage dips, loose wires, etc causing a drop in voltage will trip the breaker.
PS: I think there is more to it than a loose connection.
Love the ‘deck ape’, black gang, coal face, flat nose window lookers, etc marine ‘terms of endearment’!
How about “trogs” or “troglodytes” and translated fm my
lingo to english " starfuckers, cravateers , sleepwalkers" 
Forgot to add e/r endearments : smokies and asphalts and e/r is called “the basement” or “cave”
I guess a question I have is why the “normal mode of operation” according to the report, is to have the LV busses tied, and feeding the LV bus with only one 6.6k/440v xfrmr. In the power industry, the tie is not normally closed, and sits there waiting for a loss of one side of the bus feed, then quickly closes to keep the bus hot. With them running with only one xfrmr feeding both LV busses tied, that seems very much non-standard. Its ok, but I would only expect that if the other 6.6 to 440 xfrmr was out of service for maintenance, or was damaged. But for the report to call that standard operating mode feels odd.
Also, FWIW, that has to be the most ridiculous naming convention I have ever seen. HVR, and LVR you would think would be called HVT and LVT (T for tie). And HR1, HR2, LR1, LR2 would be would called HV1, HV2, LV1, LV2 (V for voltage)…
And yes, I agree with prior comments, I cant stand those silly push-in terminal blocks. Give me a screw terminal block every time, and yes every good electrician knows after you make the connection you pull on it to confirm its secure.
(BTW in my message above I wrote incorrectly 50 Hz instead of 60 Hz, I am so used to my local 400/230 V AC 50 Hz 3-phase TN grid. )
.
I fully agree with you.
Indeed there are even large container ships with only a single 6.6 kV/0.44 kV service transformer (!).
Fortunately such transformers are quite reliable but there should still be a second one.
The typical apparent power of service transformers of large vessels is quite high, often around 3’500 kVA to 4’800 kVA.
Usual off-the-shelf Low Voltage (LV) circuit breakers are available up to 6’300 A, which corresponds to 4’800 kW at 440 V.
The common layout is to supply the LV bus (LV BUS, Preliminary Report (PR) Fig. 5, page 8) by only a single service transformer, with the breakers being interlocked in order to prevent accidental parallel operation of two service transformers. This is a very common design.
This also means that switching service transformer (from TR2 to TR1 or TR1 to TR2) can only be done with a short LV BUS blackout.
The HV and LV side breakers of each transformer are respectively tied, if either the HR or LR breaker of a given transformer opens for any reason, the other one is opened automatically.
The LV Bus Tie can be either a breaker (usually only up to 6’300 A) or a set of copper rail parts which are assembled by screws (bus unpowered of course), the idea being that the bus ties (both HV and LV) are only operated to isolate 1/2 of the bus for maintenance purposes.
A detail being that not all designs allow to power the EBUS from both segments of the LV Bus (left side “exclusive or” right side of the tie breaker, with “excluding” interlock (to not accidentally bypass the open bus tie)), the effective design variant is not known as the oversimplified diagram of the PR does not represent the EBUS correctly and the EG breaker is not labeled.
Considering the value of a container vessel with its cargo, one would expect much more modern solutions implemented.
The typical container ship design philosophy is a 1+1 redundancy but without much diagnostic coverage (many things can go wrong fully undetected, especially also zilllions of manually operated vanes which positions (e.g. fully open or fully closed) are not monitored by automation) and there are also often still single points of failure remaining.
Also all DG’s (excepted the EG of course) are often located in the same room, many redundant devices are located next to each other. I do not expext such strict rules as for demanding DP vessels but overall I tend to consider that rules did not follow the recent evolution of container ships. New rules require years to be issued and parts of them are already obsolete when published.
I had a closer look at steering gear control, main engine remote control and bow thruster control. Even for very large container ships the design can be surprisingly outdated with lots of small flaws which reduce reliability.
The designations are the ones uniformly used by Hyundai. Do not ask me about the rationale, I have no idea, not even about Korean shipbuilding electrical engineering in general.
I wonder why there is not a standardized designation system like an adapted KKS, which would not only include the electrical equipment.
Or at least a large company like Maersk could internally require a unified proprietary designation system, which should also include all location designation, parts and documents.
Working with something like a KKS makes life easier for everyone, it must just be done wisely and not lead to useless bureaucracy (like overcomplex document management systems).
Ideally a standardized designation system should be proposed but not made formally mandatory.
.
Lots of small details can impact reliability negatively, even some rules, if followed strictly, lead to a loss of reliabiliy simply because they are outdated or not based on enough lessons learned.
Overall the electrical design of large modern VLCC’s and DPS-3 vessels as well as very large recent cruise ships is quite superior.
Recent large korean-made container ships are still a mix of old-fashioned Hyundai HV and LV switchgear together with a flavor of somewhat less antique (but still not great) Engine Control Room console design. The whole HMI (Human Machine Interface) design is poor and does not enhance situation awareness. IMHO only the integrated part of the navigation system is good.
Even some famous Main Engine 3rd party-supplied remote controls (i.e. not provided by the ME manufacturer), including telegraph systems are not well designed. An example is also a sort of hidden ME emergency stop button, IMHO that should not be allowed, emergency controls should be clearly visible and labeled but provided with a protection to avoid accidental actuation, this is common and there is no need to change established things just because an industrial designer wanted to show off his talent.
IMHO there is no need to reinvent the wheel, with some well-designed redundant fault-tolerant automation the risk of human errors could be massively reduced and especially the whole HMI could be improved. There are even basic design faults like T-shaped consoles in the middle, here referring to large wheelhouses, it should be possible to walk freely along the whole console as well as between the main console and the front windows, etc.
Really? … how does this meet Solas?
I see no reason for the HV breaker to drop out if the LV breaker opens. Both these breakers will be equipped with a UV relay and will need juice upstream to enable the breaker to close. So if the HV opens, the LV will open as the secondary voltage will quickly decay and the UV coil will drop out. Not vice versa. And yes, the 2 LV breakers downstream of the xformers will be interlocked such that only 1 can be in the closed position.
The only reason for the HV breaker to trip along with the LV breaker would be for a transformer fault, such as xfrmr differential.
You lost me on why the 2 LV breakers would be interlocked such that only 1 could be closed. If thats the case, how can you ever do a live bus/no interuption transfer to the other xfrmr?
Example: xfrmr 1 is feeding the LV bus, xfrmr 2 is not. You desire to switch to xfrmr 2. You have the tie closed, then close LV breaker for xfmr2, then after seeing the amps essentially split between xfmf 1 and 2, confirming you have indeed closed in xfmr 2 and its taking load, you then open xfrmr 1 LV breaker. We do this all the time at the power plant. And yes, there is always an alarm that occurs after about 5 seconds if you leave both LV feeds and the tie also closed, to let you know that you have both bus feeds as well as the tie closed so you don’t forget about the configuration and make sure to open the other LV bus feed breaker.
Hi Zino
First of all thanks for the well composed and intelligent post. On the issue of the HV trip if the LV trips, agree if they have differential protection relays installed. Not sure.
The ship generation capacity is tiny compared to a power plant. This vessel has quite a few reefers and thus the generation capacity is about 1.6 MW. Most of the loads are inductive as opposed to a power plant. I am guessing xformers you are referring to are step up transformers to the grid voltage.
On this ‘small captive’ power plant with mostly motors (inductive load), switching on another xformer with different voltage and power factor is generally not done. Perhaps they do have a procedure in port with most of the large loads off to perform the bumpless transfer. But I suspect there is an interlock and an override function to make ensure there is no accidental paralleling.
Curios. What is the generation voltage and the grid voltage? Do you run in isochronous mode and supply constant power (MW) to the grid or is a ‘peaker’ plant? (Only if you are able to share the info)
PS: I have visited only 1 small (slow speed diesel engine driven power plant years ago in Florida and really have no knowledge of hydro/thermal/gas/nuclear power plant operations.
Thanks.
I think this nomenclature is from NTSB. In ‘American’ we abbreviate a circuit breaker to ckt bkr. Maybe ‘R’ is the operative letter. Like some of the Scots from Dundee or Edinburgh would say … breakerrrrrrrr!
I am surprised too about the single service transformer.
As I already mentioned it, the designations of the breakers are the ones used by Hyundai since many years, the NTSB simply kept the original lones.
The whole switchboard design is from Hyundai, not only the components. Also the 4 DG’s including the generators, as well as the two service transformers are from Hyundai (and many other parts).
The power distribution of such ships cannot be compared to Balance of Plant power supply systems or power supply architectures of plants, especially the mission-critical ones.
The system is not intended to switch service transformers without interruption.
It may seem odd but it is simply the usual design, the same applies to the bus tie interconnections which remain closed during normal operation.
It is sort of pointless to discuss some seemingly irrational choices. It is a design decision to always isolate both HV and LV sides of a transformer and that’s it. Technically identical transformers can be run in parallel if some conditions are met. Running different transformers in parallel is trickier and there are additional conditions. The cabling should be identical in length and cable type and number.
Also when connecting shore power there is no synchronization. Technically it would be simple to connect and disconnect without any interruption (short switch over overlap only) but there are some good reasons to not do it. OTOH it is often done with land-based LV mobile gensets to perform switchgear and transformator maintenance without any client-side interruption.
And BTW, the here discussed total generation capacity is around 16 MVA, not 1.6 MW, that would not even cover all auxiliary services.
The main issue is not even that much the failing connection at the terminal block but to use undervoltage releases for transformer breakers, that can be considered as an obvious major basic design mistake. The much more reliable solution is to use two redundant shunt trip releases and to monitor them individually, see above and in the other topic for details.
For now it is unknown if the terminal block was defective or if simply the wire got disconnected.
Usually such terminal blocks are quite reliable IF handled correctly, most problems occur when wires are deconnected before being reconnected as well as handling errors both during the initial connection as during maintenance and troubleshooting. The terminal block can also be damaged by using the wrong tools, not opening the cage correctly, not inserting the wire correctly, preparing the wire as required, etc. I have very often seen one or several broken strands when removing a wire (without ferrule). Also sometimes single strands and bent backward.
I am not sure that all details can still be investigated once the terminal block has been removed as wires and strands are also moved. Let’s hope the NTSB used an endoscopic macro camera to document some connection details before removing parts for further investigation.
It would also be interesting to perform some wire bending cycle tests (to assess vibration resistance).
Senior moment! Yes, 16 MW generating capacity … about 20 MVA.
Absolutely not, please read carefully what has been discussed some time ago. There are some contradictions if we compare various published ratings. I even expect that some data did feature the wrong unit of measure (kW instead of kVA or the vice versa).
The reason I did not mention the two eyact types of Hyundai auxiliary engines is because I have got contradictory power rating data too.
I don’t get your confusion.
NTSB indicates 2x4000 and 2x4400 kW
Wiki indicates 2x3840 and 2x4400 kW
Class NK indicates 20,850 kVA inclusive of the EDG. Guess around 400 kW +/-.
(So far we do have not any info on EDG rating.)
Design pf is 0.8. So, 16 MW is 20 MVA. Yes?