Do you not believe the preliminary report or you think NTSB (after speaking with the switchgear maker) got it wrong? Top of page 6. It was designed to run on one at a time.
KPChief,
Quick question if I may;
As reported by the NTSB, if the step down feed to the LV BUS is indeed designed to run only one transformer at a time, why would you require an LV BUS TIE? Why could it not be an uninterrupted bus feed from either side depending on which transformer was in service?
Hi Sercos
I have a different (not contrarian) opinion. Regulations, Rules and standards are developed based on experience of the respective industry and their primary mission is to ensure safety. No commercial angle at all. Their mission cannot be understated or undermined. In fact in shipping, IMO/Solas in several instances state, these are the rules (actually intent) and if we have a better idea for equivalency they will consider.
Rule writing is also an art. In a few lines you need to get across the intent of what you are trying to get across and that it should not be interpreted in a dozen different ways.
But some clever people/companies (rule beaters) know how to beat the rules and definitely for reasons other than safety.
At this point I’m not saying what is good or bad just noting what the report said. It is often best to split the investigation into two parts. Fact gathering and then later analysis, conclusions and recommendations. But I’ll play along.
Let’s say vital loads were split on different load centers or MCC’s that were fed from different sides of the bus tie. Let’s say a severe enough fault occurred on the far side (away from the in service transformer). The bus tie tripping would keep the plant going and protect the remaining healthy part of the LV switchboard. I mean that’s one point of it.
Another might be for maintenance on the LV bus one side at a time.
Another might be that (through breaker interlocks) it may be possible to run split LV bus (transformers not in parallel) for emergency or special circumstances. I’m just reading the report that “not designed” for it means not the normal plant alignment for this ship. The final report may say something else.
I can think of tens? hundreds? of features that might make this plant better, safer. But I don’t really know what they have now.
So far it’s been the steering gear, reefer container loads, bow thruster, mystery overcurrent or overload faults, bad design and I’m sure I’m leaving a bunch out. The only thing not condemned yet are the machinery foundations.
Thank you for that.
Conversely, if the severe fault occurred on the transformer side of the bus tie then there is nothing preventing the single feed from tripping which indicates that the designers have inadvertently built in a single point of failure.
Others have commented on the lack of redundancy as deduced from this PR allied with fig.5.
As you quite rightly say……all should be correctly revealed in the final report and every page of the PR is claused with …….”subject to change”.
The simplified electrical one line provided in the PR has raised hopes that perhaps the incident could have been avoided given the available redundancy and that it was not utilized. Granted with a few modifications (such as auto change over from either side of the bus to feed the ESB in case one side is down, etc) are implemented, the plant could be operated on a split bus. Of course depending on which side the running lube/cooling water pumps are powered from, no main engine shut down is not guaranteed.
The entire premise of a split bus or rapid change to split bus from closed loop is designed for faults – electrical or otherwise in the power generation system.
We have no indication that there was a fault, or overload, or any other symptom of trouble from the PR. Only ‘unexpected’ opening of the transformer breakers and unexpected opening of the breakers on the running generators.
If these ‘unexpected’ events are related to DG operational stability (voltage, frequency, etc) could a split bus operation averted this disaster? Most likely the same outcome.
Absolutely. Ancient marine engineer proverb…half a loaf is better than none. (Sometimes).
One might quibble with term for “inadvertently”. Are there any rules for this type of vessel that stipulate single points of power distribution failures need to be eliminated?
I speak mostly from US rules now but it seems to me the most pertinent areas to this discussion come from a few sections. Those that govern EDG / loads and automation. Which comes into play when invoking minimally attended or periodically unattended designations.
So while there are many things listed, complete and automatic continuity of power is not one of them.
Automatic power management system is required to:
Not control the EDG
Start and stop generators based on load etc
Fail in a way that this is alarmed and does not result in complete loss of plant power and all manual controls on the switchboard remain operational.
This sort of recitation that can become tedious but what I’m getting at is there may not be anything inadvertent about this design and it may be completely within the rules. This very unsatisfactory given the loss of life and all else.
Given the mass of these ships and the nature of port infrastructure it certainly may be time to reconsider plant power requirements. The DP rules go a long way in that direction. Perhaps some version of those rules could serve as a starting point for what? All ships greater than a certain size, power?
Don’t have time to check a class set of rules or SOLAS just now but there may be some eagle eyes out there that know of power redundancy requirements that have escaped me (other than EDG).
1 Like
Alright I missed that in the report but why was the switchgear designed with two HV busses if the ship could not use them in split buss mode?
Yes, all three generators on-line for mooring/unmooring ops for the BT
Once the tugs are let go the third generator is taken off-line and put in stand-by. So for maneuvering two generators on-line and a third on standby.
When the pilot gets off it’s down to one generator.
We kept three generators on-line till we passed the Key bridge. It was only a few minutes extra, avoids having hands-on in a critical spot.
Nothing can really be made from the report. It obviously wasn’t written for electrical professionals. I’m sure they have a lot more info that will come out with the final report.
"About 0125, the Dali was 0.6 miles—or three ship lengths—from the Key Bridge
when electrical breakers (HR1 and LR1) that fed most of the vessel’s equipment and
lighting unexpectedly opened (tripped) (see figure 7)."
They either opened or they tripped. Those two words don’t carry the same meaning in the world of breakers. If they tripped then there most likely was some trip flag, event log on the relay, something proving a trip. There are possible ways to have no record of a trip though.
Why did both breakers trip together? Massive fault on secondary causing instantaneous on both breakers. There would probably be massive damage in that case. Some type of ZSI or interlocking that caused them to trip together?
Again there’s not enough info to draw any conclusions from in this report.
Could the next step be a ME running on potassium cyanide if one finds out that it will save the world?
Everyone wants cheap goods but not the ships bringing them.
Ammonia is both very toxic and prone to create explosive atmospheres. Even it can be smelled at concentrations way lower than those which become life-threathening, if a pipe or tank fails it can lead to a sudden massive concentration increase.
Ammonia is used in large refrigeration systems but those are operated as closed circuits and don’t require continuous maintenance, also quantities are tiny compared to the use as fuel.
Now there should be ME to be operated by crews facing the same challenges as workers in the Oil & Gas and chemical industry when it comes to explosion risk and toxic gases?
The explosion risk is acceptable (hydrogen and other fuel gases) if everything is handled seriously.
Toxicity is the main issue, I don’t really see how it can be handled considering how a ME has to be run and maintained.
Just my two cents, let’s learn the lesson before people die.
I accidentally replied one of the other topics, I can’t get used to this advanced forum software where the text you write is duplicated when you open the same thread in another tab. I started writing replies to several posts of the same thread and ended messing it up. Maybe I’ll have to ask a moderator to move/merge some of my posts later as I don’t want to spam topics.
To summarize:
The LV BUS tie breaker LVR is not really useful unless maybe for switchboard maintenance (to shut down the left or right busbar segment) but it should be done in port anyaway unless there is a major problem. As TR1 and TR2 are not used simultaneously it is not possible to split the distribution when en route.
The HV BUS tie breaker could be used to supply the bow thuster BT or some reefer container supplying transformers separately.
Though in the PR it is mentioned that both bus tie breakers remain closed.
The pros and cons of segmented single or double busbars can be discussed separately.
As long as everything works as expected, some different distribution architectures may seem comparable, things become interesting when assessing failure scenarios and their respective probabilities (likelihood of occurrence).
Double busbars, which are common in larger or critical substations, would allow more flexibility but also increase costs and require more space.
Double segmented rings with redundant protection relays would be possible but again too expensive. Dynamic positioning with billions of potential losses if something goes seriously wrong is another discussion.
Well, the discussed incident will also cost billions but it was not considered as enough probable to invest in more reliable power generation and distribution.
Very reliable power distributions (including power generation) are not a problem if you can afford them. Possibly also a 5th DG could be required considering that one DG is possibly not available any time due to maintenance.
Where stakes are high even 2 spare DG’s in stand-by may be needed to be available any time (not the case here of course).
It may also depend on the allowed delays until the DG is online, for example for data centers with classical barttery UPS several Emergency Diesel Generator (EDG) start failures are not a problem, the UPS batteries will last at least several minutes, so one has even enough time to replace batteries and start the EDG manually, OTOH dynamic UPS with flywheel won’t possibly allow even a 2nd diesel engine start attempt (usually at least one UPS and one EDG can fail though). Personally I’d never store critical data in a data center without battery UPS but most clients have no clue about such details. I remember a major 20 kV supply design flaw affecting a not that tiny large (> 20 MW) data center which went unnoticed despite the Tier certifications.
@ retdmarineengineer
I fully agree with you.
I am not against standards, rules, regulations, inspections, etc. as long a they are useful. Without standards there would be zillions of screws or many different power outlets in the same country for the same use as it was the case as electrical power became available.
The commercial angle I partially disagree, for example for a very long time in Europe everyone used Pg threads for cable glands. Out of the blue some idiots decided that cable glands suddenly needed metric threads (like if imperial fittings would have to be suddenly replaced by metric ones). Did cost a lot and made manufacturers very happy.
There are other examples where commercial interests prevail, for example building fire protection is ridiculous in many European countries. Due to a lot of lobbying flammable isulation materials are allowed, see the Grenfell Tower in London or the fire in Valencia (Spain).
No one learnt anything, there are many thousands buildings at risk.
My major issue was about excessive bureaucracy, meaningless administrative procedures., expensive approvals.
I’ve no problem with pragmatic viable rules enforced by experienced reasonable and fair specialists.
As side note about electrical components: As I mentioned it, some best-in-class components which would easily get approved by any classification society cannot be used because their manufacturers aren’t interested in the marine market.
Many new DP vessels operate closed bus as opening a breaker is very fast and always tests.
Closing in a panic is asking for trouble.
Its all tested for your DP class approval.
1 Like
Data centers typically store the same data in at less three different centers at widely different locations even different continents. A newly built data center 12 km from me has 5 large diesel backup generators and for all I know stores the details of your bank account.
New Zealand as been seen as attractive for data centers having reliable power, cooler weather and far away from places where strife threatens.
spowiednick,
Many thanks for posting the Chief’s excellent video.
There we have the answer to the LV BUS TIE question. He indicates that it may become regulated that large container vessels manoeuvre on a split LV BUS whilst in US pilotage waters. I was a little disappointed that he did not make mention of the APMS.
This is a ULCV distribution diagram which I came across recently. There are three step down feeds.
2 Likes
but plenty of earthquakes being on the ring of fire?
Here in Singapore its so full of data centres the gov has had to stop them due to the power consumption
Some brainstorming… Parts of this message were written as drafts for various not posted replies. I still haven’t figured out how some details work in this forum, sorry.
When several buses are operated with open tie breakers they woud have to be run permanently with phase angle and voltages both matched tightly enough in order to allow any time an immediate closure of the tie breaker. In real life it just won’t work and especially not under fault conditions.
Synchronizing a generator which doesn’t have to supply any load is basic, synchronizig a loaded busbar segment powered itself by a single or several paralleled generators is much more difficult and it may not even be possible to do it graciously. There can be all sorts of problems.
Also closing immediately the bus tie breaker will typically be required when a fault occurs and that’s exactly when synchronizing conditions are likely to no longer be met due to that fault.
Running separate buses is not a problem per se but do not expect to be able to close a bus tie breaker if a problem occurs!
The only way to achieve truly uninterrupted power is to run several generators in parallel at the the same time, i.e. being connected to a common bus.
It is also to be remembered that breakers are not contactors, any closed breaker can always be immediately opened, an open breaker can be closed immediately only if the spring mechanism is charged. Once the breaker contacts are opened, the spring mechanism has to be charged, which usually automatically by a small electric motor, can often be well heard, and, if required it can always be charged manually turning a crank or pumping using a lever which requires a few seconds in any case. This means that once a breaker has been opened it typically cannot be reclosed immediately.
There is also a anti-pumping function which protects the breaker if open and close commands are issued simultaneouly by the control system (which should not happen, a breaker opening command shall always override any closing command).
Tme power management system or protection relay should monitor the charged state of the spring mechanism as well as the control solenoids, if not, some failures could go unnoticed.
Not all failures can be monitored though, therefore maintenance and testing remain very important.
If power can shortly be interrupted, several buses can be fed separately, each by one or several generators but when bus tie breakers are closed the load management system must close the breakers feeding the loads sequentially if the power reserve available is too low to take over the full load in a single step. Sequences can also be handled manually.
Sequential loading of a generator, i.e. usually supplying the most critical loads first is preferred. If there are a few insignificant loads they can be powered immediately once voltage and frequency are stable if that doesn’t cause a perceptible delay.
When generators are synchronized, i.e. connected to a energized bus, frequency (in Hz), voltage (in V) and phase angle (in electrical degrees sometimes noted like °el) are adjusted by controlling the prime mover (diesel engine) speed (frequency is directly proportional to the RPM’s) and the excitation of the synchronous generator is adjusted to match voltages until the generator breaker is closed.
Running several generators in parallel is not a problem and is performed routinely.
A well designed reliable generator protection must ensure that any detected critical generator fault trips ONLY the generator breaker, e.g. example excessive reverse power which happens when the diesel engine is no longer able to provide enough torque and starts being driven by the generator which becomes a motor, or internal generator faults and other issues.
A generator-side fault shall not become a common cause for other failures.
Extremely important is also the supervision and load sharing, the whole power generation system must run without excessive oscillations and without excessive recirculation of currents which uselessly heat up windings. Resonance effects shall be detected as soon as possible as they can increase and mess up things seriously.
Manual synchronization is the best way to cause damages, considering the price of a 4000 kVA genset even redundant automatic synchronization with additional independant synchrocheck would not make much of a price difference. Mis-synchronization is a lottery, in some cases the generator breaker simply trips and in the worst case it can cause major damages, it also depends a lot on how the generator is protected.
Synchronizing manually is of course possible but it must be done very carefully as an error can potentially lead to the loss of the generator. Fortunately it doesn’t happen often though.
Synchronizing manually under stress is not great as the procedure should never ever be rushed.
Overall, where reasonable, I prefer by far reliable automated processes with, where possible, manual override with some automated protections still being active and, where possible, an ultimate manual control mode
Well designed redundant automation is very reliable.
There general redundancy of the electrical equipment of merchant ships is not very good due to cost contraints. Many systems are available twice but there a lots of single points of failures and the degree of diagnostic coverage may also be an issue (if a part of a redundant system fails, that failure should not go unnoticed, e.g. there are Power Supply Units (PSU’s) run with paralleled outputs but without individual monitorig, one day one PSU fails and that goes unnoticed until the 2nd PSU also fails).
Also the positions of many manually operated valves are not monitored by end switches therefore when looking at diagram displayed on-screen it’s impossible to know if such a valve is open or closed. On a container ship there are of obviously too many manual valves to monitor them individually, only important valves, especially those required to run the ME, the generators including the emergency diesel generator should be monitored.
In industrial process control maybe thousands of manual valves positions have to be monitored to avoid human errors (positions of remotely operated valves should be monitored anyway to make sure that the actuator works correctly).
In configurations with paralleled generators the remaining generators must always be able to provide enough power, the failure of a single generator or, depending on requirements, more than one generator can fail any time without causing any problem.
Where in addition a stand-by generator is available it can be started and synchronized automatically and there is no hurry to load it quickly, its load can be increased progressively.
The risk of Common Cause Failures like e.g. arc flashes must be mitigated by good design and operating procedures (incl. maintenance) being followed.
According to my own experience in the general industry, most incidents are caused by human errors and poor technical design decisions (those being both the result of cost constraints and ignorance/bad engineering).
Finding well trained and reliable responsible staff has also become more challenging but that’s just my very subjective opinion.
Excepted the bow thruster(s), container ship power distributions don’t have to supply any single load which is significant compared to the power provided by any of the diesel generators. After the bow thruster motors, maybe around 2000 to 4000 kW ea., supplied in High Voltage (here 6600 V), all other motors are much smaller, I suppose not exceeding 400 kW and possibly not even 250 kW.
Considering that a typical container ship diesel generator is rated roughly between 3500 and 6000 kVA (read kVA, not kW) only the bow thruster(s) is/are a large load(s), all other single loads do not exceed 10 % of the rated generator power. Transformers are not directly considered as loads themselves as the load they represent depends on the loads they feed.
The ratings of a genset (generator set, a diesel engine with a generator mounted on a single frame) vary depending on its use, if generating 24/7 prime power like on a ship or a power station, the maximum continuously allowed load will be lower than if used only as emergency generator.
Both the engine and the generator can be used up to different power limits but higher powers reduce lifetime.
Generator power is mostly limited by the temperature of the winding insulation, the better a generator is cooled, the more it can be loaded up to a certain limit and if the diesel engine rating allows it.
Well cooling generators and transformers can increase their lifetime significantly though.
As synchronous generators must run at fixed speed tied to the frequency (here 60 Hz) it is not possible to optimize the operating point of the engine for fuel use efficiency, ideally several gensets should be loaded closely to their optimal load but it typically requires maybe 6 or 8 gensets to be able to spread load accordingly.
Also the Power Management System may handle power generation preemptively based on process-dependent power requirements known in advance.
Temporary overload capability also applies to the ME (Main Engine), to avoid a major accident it can be overloaded during a limited time, in most cases it won’t fail carastrophically but some parts of the engine and the hull structure will be highly stressed and the engine lifetime will be reduced.
The ME controls are designed to allow the override of some protections in case of true emergency (and during sea trials).
ME sizing is mainly based on fuel cost optimization as the most economic operating point of any diesel engine is never at maximum allowed speed or power.
I’m wondering if the ECDIS (Electronic Chart Display and Information System) can display stop distances and prospective paths as well as possible path options taking in account effective loading computer data, ballasting, hull trim, wind, water temperature… Even if the simulation ist subject to uncertainities it could still help making urgent decisions.
BTW Did someone perform a simulation of the incident or do conditions make it too complex to simulate enough accurately using a regular simulator?
Performing post-calculations is possible using CFD (Computational Fluid Dynamics) and other tools but will require detailed data and quite some time.
Also as fluid dynamics are not my cup of tea anyway I’ve no idea if a specifically created model basin would allow provide useful data.
(This quite unstructured message is a compilation of several message which were intended to be posted separately.)
@ 
Ausmariner
Thanks a lot fot that diagram, that’s exactly the least I’d have expected from the NTSB.
As mentioned earlier, the NTSB should have issued a formal technically accurate Preliminary Report (PR) as well as simplified non-binding information report for the general public. The issued PR sort of discredits the NTSB because it looks like written by journalists and intended for someone without technical knowledge.
Your single-line diagram also shows the physical 6600 V switchgear cell subdivision and 440 V switchgear cabinet division. Referring to page 5 of the PR, the individuals HV cells and LV cabinets of the switchboards can easily be distinguished. Note that the LV switchboard ist not a true MCC (Motor Control Center), though definitions vary, true modular MCC’s feature withdrawable units which can be replaced any time within seconds without having to shut down power. Even some modifications can be performed without de-energezing the MCC cabinet.
BTW I don’t remember if I’ve posted or only wrote it in a draft, there are even large container ships (similar to M/V Dali, with a >50 MW (MCR) ME) which feature only a SINGLE service transformer. If that one fails the main propulsion is lost as well as all 440 V excepted the reefer container supply and the circuits fed by the EDG (Emergency Diesel Generator) as long as that one can be kept running. MacGyvering the supply of the Main LV 440 V Switchboard from the reefer container supply transformers would not exactly be a safe operating procedure.
Such large service transformers are reliable but can still fail.
Either the provided data is incorrect or the classification society did (in my opinion, I can’t write “IMO” here) a HUGE mistake when allowing a single service transformer.
I don’t care about how reliable it is on the paper.
Would you use a parachute without backup emergency parachute if the manufacturer tells you that some certification company attested a 0.0000000000000000000000001 % failure rate making any redundancy useless? (I’m kidding about the number but you get the idea.)
If I had to design the power distribution I’d propose but not impose a 3rd service transformer, keep the 4 DG’s of sufficient power but include a 2nd more powerful emergency diesel generator sized to be able to operate the steering gear, fire pumps, bilge pumps and other important equipment without any restriction during several consecutive days.
I’d also add a couple of seriously powerful fire pumps for remote controlled piracy countermeasures as Phalanx C-RAM’s are unlikely to get approved. 
Misplaced reply is currently message #60 here:
(Had to edit the link.)