Have you changed the oil? How long has she been sitting with no load and normal operating temperature performance testing? Are the injector sliders free? are all the corresponding linkages lubricated and free? Any springs look funny? actuator rod ok? After you adjust the compensation needle valve keep these things in mind if needed? Remember you can determine the cause of failure with different vessel systems with a good visual inspection.
The most common reasons for hunting (We call it searching) is the Gain abnormality at the speed governor.
The speed of the engine is detected by means of the magnetic pickup, of reading the number of teeth passing through on the flywheel of the engine. (Ii is a teethed ring around the flywheel)
This magnetic pick up counts the number of the teeth at any moment and transfers this data to the electronic governor.
The governor evaluates this data and arranges the speed of the engine accordingly to achieve a constant speed of 1500 or 3000 RPM. (Possibly yours is 1500 RPM or 1800 RPM in the 60 Hz case) At any moment and at any load.
Now the level of the gain is important when evaluating the data from magnetic pickup to the governor.
The speed of the engine and the load applied on the engine is floating. When load increases, speed goes down, when load decreases, speed goes up.
High gain level means quick response of the engine against the load variation. (Speed also varies accordingly). The quick response does not always give good results. High gain level means high feedback and this brings oscillations.
The governor on the engine continuously tries to find the correct speed and infinitely adjusts the speed. But the feedback data from the magnetic pickup is exaggerated by the high gain value, therefore readjusts and readjusts again, and it goes on like a closed loop.
Try to lower the gain setting, or play with the droop setting if existing. But not change the speed setting on the governor. Otherwise your constant speed as well as the frequency changes.
An exception to this was a few of the Woodward governers, we used to take students over to one of the patrol boats (they had 2 x 3/71 Detroit gen sets), and showed them what happened when a governor was set on zero droop - it used to hunt wildly, this was why droop requirements were needed for gen sets (1-1,25%) and main propulsion engines (8-9,5%)
I truly want to see you be sucessful c.captain. You have a huge project there and I commend you for that.