Skip to content

CLOUDSTACK-10278 idempotent column addition #2449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DaanHoogland merged 7 commits into from
Mar 28, 2018
Merged

CLOUDSTACK-10278 idempotent column addition #2449

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e4b2ca7
CLOUDSTACK-10278 - WIP: need to test this script before create a pull…
ernjvr Feb 9, 2018
ab22ad3
CLOUDSTACK-10278 - added more idempotent stored procs and moved all l…
ernjvr Feb 14, 2018
50c2170
CLOUDSTACK-10278 - changed more sql statements to call idempotent sto…
ernjvr Feb 14, 2018
41c34de
CLOUDSTACK-10278 - WIP: need to test this script before create a pull…
ernjvr Feb 9, 2018
7b1c94d
CLOUDSTACK-10278 - added more idempotent stored procs and moved all l…
ernjvr Feb 14, 2018
72acbf4
CLOUDSTACK-10278 - changed more sql statements to call idempotent sto…
ernjvr Feb 14, 2018
33db67c
Merge branch 'CLOUDSTACK-10278-idempotent-column-add' of https://gith…
ernjvr Mar 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 61 additions & 19 deletions engine/schema/resources/META-INF/db/schema-41000to41100.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,54 @@
-- Schema upgrade from 4.10.0.0 to 4.11.0.0
--;

--;
-- Stored procedure to do idempotent column add;
--;
DROP PROCEDURE IF EXISTS `cloud`.`IDEMPOTENT_ADD_COLUMN`;
Copy link
Member

@yadvr yadvr Feb 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ernjvr /cc @DaanHoogland @rafaelweingartner - is it possible to refactor these methods out to a Java/sql file, and automatically include them in all sql files so that these procedures may be reused (somehow)? Or, perhaps explore migrating to something else that provides these abstractions, perhaps flywaydb etc?

Copy link
Member

@rafaelweingartner rafaelweingartner Feb 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also rather migrate to something like flywaydb too. However, I think it would required quite more work than this one here. What do you think? I have done this analysis before, but I was not having much time, and the discussion died [1]. Maybe we could pick this topic up again?

[1] http://markmail.org/message/2dxtcnqx2euq5tl2

Copy link
Contributor Author

@ernjvr ernjvr Feb 16, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rafaelweingartner yes, we will have to make sure that everyone creating new scripts to add a new column, drop a foreign key or create/drop indexes, call these stored procedures, to ensure their SQL statements are idempotent.
@rhtyd These 'drop procedure' statements are placed before each 'create procedure' statement only. So when the upgrade script runs, the procedures are created but not dropped after the script completes. Therefore the stored procedures will still exist in the database. So they will be available (reusable) to future versions of schema-nn-nn.sql migration scripts. I think these 'drop/create procedure' statements also need to be added to the setup/db/create-schema.sql script to make them available to new cs installs, including the simulator.


CREATE PROCEDURE `cloud`.`IDEMPOTENT_ADD_COLUMN` (
IN in_table_name VARCHAR(200)
, IN in_column_name VARCHAR(200)
, IN in_column_definition VARCHAR(1000)
)
BEGIN

DECLARE CONTINUE HANDLER FOR 1060 BEGIN END; SET @ddl = CONCAT('ALTER TABLE ', in_table_name); SET @ddl = CONCAT(@ddl, ' ', 'ADD COLUMN') ; SET @ddl = CONCAT(@ddl, ' ', in_column_name); SET @ddl = CONCAT(@ddl, ' ', in_column_definition); PREPARE stmt FROM @ddl; EXECUTE stmt; DEALLOCATE PREPARE stmt; END;

DROP PROCEDURE IF EXISTS `cloud`.`IDEMPOTENT_DROP_FOREIGN_KEY`;

CREATE PROCEDURE `cloud`.`IDEMPOTENT_DROP_FOREIGN_KEY` (
IN in_table_name VARCHAR(200)
, IN in_foreign_key_name VARCHAR(200)
)
BEGIN

DECLARE CONTINUE HANDLER FOR 1091 BEGIN END; SET @ddl = CONCAT('ALTER TABLE ', in_table_name); SET @ddl = CONCAT(@ddl, ' ', ' DROP FOREIGN KEY '); SET @ddl = CONCAT(@ddl, ' ', in_foreign_key_name); PREPARE stmt FROM @ddl; EXECUTE stmt; DEALLOCATE PREPARE stmt; END;

DROP PROCEDURE IF EXISTS `cloud`.`IDEMPOTENT_DROP_INDEX`;

CREATE PROCEDURE `cloud`.`IDEMPOTENT_DROP_INDEX` (
IN in_index_name VARCHAR(200)
, IN in_table_name VARCHAR(200)
)
BEGIN

DECLARE CONTINUE HANDLER FOR 1091 BEGIN END; SET @ddl = CONCAT('DROP INDEX ', in_index_name); SET @ddl = CONCAT(@ddl, ' ', ' ON ') ; SET @ddl = CONCAT(@ddl, ' ', in_table_name); PREPARE stmt FROM @ddl; EXECUTE stmt; DEALLOCATE PREPARE stmt; END;

DROP PROCEDURE IF EXISTS `cloud`.`IDEMPOTENT_CREATE_UNIQUE_INDEX`;

CREATE PROCEDURE `cloud`.`IDEMPOTENT_CREATE_UNIQUE_INDEX` (
IN in_index_name VARCHAR(200)
, IN in_table_name VARCHAR(200)
, IN in_index_definition VARCHAR(1000)
)
BEGIN

DECLARE CONTINUE HANDLER FOR 1061 BEGIN END; SET @ddl = CONCAT('CREATE UNIQUE INDEX ', in_index_name); SET @ddl = CONCAT(@ddl, ' ', ' ON ') ; SET @ddl = CONCAT(@ddl, ' ', in_table_name); SET @ddl = CONCAT(@ddl, ' ', in_index_definition); PREPARE stmt FROM @ddl; EXECUTE stmt; DEALLOCATE PREPARE stmt; END;

-- Add For VPC flag
ALTER TABLE cloud.network_offerings ADD COLUMN for_vpc INT(1) NOT NULL DEFAULT 0;
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.network_offerings','for_vpc', 'INT(1) NOT NULL DEFAULT 0');

UPDATE cloud.network_offerings o
SET for_vpc = 1
where
Expand Down Expand Up @@ -88,7 +134,7 @@ CREATE TABLE IF NOT EXISTS `cloud`.`annotations` (
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DROP VIEW IF EXISTS `cloud`.`last_annotation_view`;
CREATE VIEW `last_annotation_view` AS
CREATE VIEW `cloud`.`last_annotation_view` AS
SELECT
`annotations`.`uuid` AS `uuid`,
`annotations`.`annotation` AS `annotation`,
Expand Down Expand Up @@ -405,21 +451,18 @@ UPDATE `cloud`.`monitoring_services` SET pidfile="/var/run/apache2/apache2.pid"
UPDATE `cloud`.`vm_template` SET guest_os_id=99 WHERE id=8;

-- Network External Ids
ALTER TABLE `cloud`.`networks` ADD `external_id` varchar(255);
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.networks','external_id', 'varchar(255)');

-- Separate Subnet for CPVM and SSVM (system vms)
ALTER TABLE `cloud`.`op_dc_ip_address_alloc`
ADD COLUMN `forsystemvms` TINYINT(1) NOT NULL DEFAULT '0' COMMENT 'Indicates if IP is dedicated for CPVM or SSVM';
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.op_dc_ip_address_alloc','forsystemvms', 'TINYINT(1) NOT NULL DEFAULT 0 COMMENT ''Indicates if IP is dedicated for CPVM or SSVM'' ');

ALTER TABLE `cloud`.`op_dc_ip_address_alloc`
ADD COLUMN `vlan` INT(10) UNSIGNED NULL COMMENT 'Vlan the management network range is on';
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.op_dc_ip_address_alloc','vlan', 'INT(10) UNSIGNED NULL COMMENT ''Vlan the management network range is on'' ');

-- CLOUDSTACK-4757: Support multidisk OVA
ALTER TABLE `cloud`.`vm_template` ADD COLUMN `parent_template_id` bigint(20) unsigned DEFAULT NULL COMMENT 'If datadisk template, then id of the root template this template belongs to';
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.vm_template','parent_template_id', 'bigint(20) unsigned DEFAULT NULL COMMENT ''If datadisk template, then id of the root template this template belongs to'' ');

-- CLOUDSTACK-10146: Bypass Secondary Storage for KVM templates
ALTER TABLE `cloud`.`vm_template`
ADD COLUMN `direct_download` TINYINT(1) DEFAULT '0' COMMENT 'Indicates if Secondary Storage is bypassed and template is downloaded to Primary Storage';
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.vm_template','direct_download', 'TINYINT(1) DEFAULT 0 COMMENT ''Indicates if Secondary Storage is bypassed and template is downloaded to Primary Storage'' ');

-- Changes to template_view for both multidisk OVA and bypass secondary storage for KVM templates
DROP VIEW IF EXISTS `cloud`.`template_view`;
Expand Down Expand Up @@ -528,8 +571,7 @@ CREATE VIEW `cloud`.`template_view` AS
OR (`resource_tags`.`resource_type` = 'ISO')))));

-- CLOUDSTACK-10109: Enable dedication of public IPs to SSVM and CPVM
ALTER TABLE `cloud`.`user_ip_address`
ADD COLUMN `forsystemvms` TINYINT(1) NOT NULL DEFAULT '0' COMMENT 'true if IP is set to system vms, false if not';
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.user_ip_address','forsystemvms', 'TINYINT(1) NOT NULL DEFAULT 0 COMMENT ''true if IP is set to system vms, false if not'' ');

-- ldap binding on domain level
CREATE TABLE IF NOT EXISTS `cloud`.`domain_details` (
Expand All @@ -541,11 +583,11 @@ CREATE TABLE IF NOT EXISTS `cloud`.`domain_details` (
CONSTRAINT `fk_domain_details__domain_id` FOREIGN KEY (`domain_id`) REFERENCES `domain`(`id`) ON DELETE CASCADE
)ENGINE=InnoDB DEFAULT CHARSET=utf8;

ALTER TABLE cloud.ldap_configuration ADD COLUMN domain_id BIGINT(20) DEFAULT NULL;
ALTER TABLE cloud.ldap_trust_map ADD COLUMN account_id BIGINT(20) DEFAULT 0;
ALTER TABLE cloud.ldap_trust_map DROP FOREIGN KEY fk_ldap_trust_map__domain_id;
DROP INDEX uk_ldap_trust_map__domain_id ON cloud.ldap_trust_map;
CREATE UNIQUE INDEX uk_ldap_trust_map__bind_location ON ldap_trust_map (domain_id, account_id);
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.ldap_configuration','domain_id', 'BIGINT(20) DEFAULT NULL');
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.ldap_trust_map','account_id', 'BIGINT(20) DEFAULT 0');
CALL `cloud`.`IDEMPOTENT_DROP_FOREIGN_KEY`('cloud.ldap_trust_map','fk_ldap_trust_map__domain_id');
CALL `cloud`.`IDEMPOTENT_DROP_INDEX`('uk_ldap_trust_map__domain_id','cloud.ldap_trust_map');
CALL `cloud`.`IDEMPOTENT_CREATE_UNIQUE_INDEX`('uk_ldap_trust_map__bind_location','cloud.ldap_trust_map', '(domain_id, account_id)');

CREATE TABLE IF NOT EXISTS `cloud`.`netscaler_servicepackages` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
Expand All @@ -565,5 +607,5 @@ CREATE TABLE IF NOT EXISTS `cloud`.`external_netscaler_controlcenter` (
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

ALTER TABLE `cloud`.`sslcerts` ADD COLUMN `name` varchar(255) NULL default NULL COMMENT 'Name of the Certificate';
ALTER TABLE `cloud`.`network_offerings` ADD COLUMN `service_package_id` varchar(255) NULL default NULL COMMENT 'Netscaler ControlCenter Service Package';
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.sslcerts','name', 'varchar(255) NULL default NULL COMMENT ''Name of the Certificate'' ');
CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.network_offerings','service_package_id', 'varchar(255) NULL default NULL COMMENT ''Netscaler ControlCenter Service Package'' ');