Triggers: support managing columns for data iteration

[labkey-nicka]

Rationale

When query triggers are called within data iterators they are allowed to manipulate the data of the row that is being procured. However, if a trigger modifies the columns (i.e., the keys of the row map) it can result in data being mismanaged due to the columns in the row not aligning with what the data iterator specified.

This introduces the notion of "managed" columns to triggers. Triggers can, at the outset, declare which columns they are going to manage. Any managed columns are then picked up by the data iterator during initialization to ensure they are persisted correctly. To enforce this management more strictly, and to avoid mismanagement of null values, triggers are required to fulfill all managed column values by either clearing the value or setting the value. After the trigger has fired the resulting row will be validated and checked. The result is consistent enforcement and acknowledgement of triggers that manipulate the shape of the data.

Related Pull Requests

• https://github.com/LabKey/limsModules/pull/2087
• https://github.com/LabKey/premiumModules/pull/501
• Triggers: support managing columns for data iteration testAutomation#2938
• Triggers: support managing columns for data iteration targetedms#1191
• Triggers: support managing columns for data iteration nircEHRModules#682
• Triggers: support managing columns for data iteration wnprc-modules#940
• Triggers: support managing columns for data iteration DiscvrLabKeyModules#422
• https://github.com/LabKey/johnsHopkinsEHRModules/pull/627

Changes

• Declare TableInfo.getTriggerManagedColumns() to allow for data iterator to gather all managed columns
• Add difference checks for non-managed columns added or removed by triggers
• Support managed columns in server-side JS trigger scripts
• Add utilities to ensure managed column values are set (setInsertManagedColumns() and setUpdateManagedColumns())
• Add experimental feature flag which when enabled will disable the managed trigger columns feature

Tasks

• Initial review @labkey-matthewb
• Hook up server-side JS triggers
• Only throw errors in development for non-data iterator triggers
• Add experimental feature flag to disable
• Optimize row checks
• Needs automation
• Code review @mbellew
• Manual testing (needs volunteer) 📍
• Verify fix

[labkey-matthewb]

I think this should probably be a failure.

[labkey-nicka]

I think we want to just resolve the known columns and not error on unknown columns as the script may contain "columns" that are not in the underlying table (e.g., "annotations").

[labkey-matthewb]

validate that all managed columns are set for update trigger

[labkey-nicka]

I've switched it to only validate managed columns for update.

[labkey-nicka]

I've re-reverted this to check managed columns for both insert/update.

[labkey-matthewb]

Considerations for additional features (do we want to do now or never)?

• Support different managed columns for insert/update
• Support in JavaScript triggers

[labkey-nicka]

Support different managed columns for insert/update
Support in JavaScript triggers

Implemented both of these.