Add e2e tests for listing resources by vishesh92 · Pull Request #8281 · apache/cloudstack

vishesh92 · 2023-11-30T05:45:39Z

Description

This PR adds e2e tests for listing resources

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2023-11-30T05:53:00Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (12dcf5c) 28.95% compared to head (26b5aae) 30.83%.
Report is 12 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8281      +/-   ##
============================================
+ Coverage     28.95%   30.83%   +1.88%     
- Complexity    31460    34020    +2560     
============================================
  Files          5329     5341      +12     
  Lines        373523   374759    +1236     
  Branches      54315    54510     +195     
============================================
+ Hits         108163   115568    +7405     
+ Misses       250569   243972    -6597     
- Partials      14791    15219     +428

Flag	Coverage Δ
simulator-marvin-tests	`24.75% <ø> (-0.06%)`	⬇️
uitests	`4.40% <ø> (-0.03%)`	⬇️
unit-tests	`16.44% <ø> (+1.60%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

DaanHoogland · 2023-11-30T08:25:03Z

@blueorangutan package

blueorangutan · 2023-11-30T08:26:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-11-30T09:24:09Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7873

vishesh92 · 2023-12-04T11:12:53Z

@blueorangutan package

blueorangutan · 2023-12-04T11:14:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland

tests look good, just the UI build failed, which is unrelated (annoying though)

vishesh92 · 2023-12-04T15:03:59Z

@blueorangutan package

blueorangutan · 2023-12-04T15:06:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

vishesh92 · 2023-12-05T05:26:50Z

@blueorangutan package

blueorangutan · 2023-12-05T05:28:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-12-05T05:34:11Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7912

vishesh92 · 2023-12-05T07:45:40Z

@blueorangutan test

blueorangutan · 2023-12-18T06:55:17Z

[SF] Trillian test result (tid-8615)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 64636 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8281-t8615-vmware-67u3.zip
Smoke tests completed. 60 look OK, 21 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_deploy_vm_start_failure	`Error`	105.12	test_deploy_vm.py
test_deploy_vm_volume_creation_failure	`Error`	129.25	test_deploy_vm.py
test01_template_download_URL_expire	`Error`	15003.85	test_escalations_templates.py
test_vm_ha	`Error`	99.43	test_vm_ha.py
test_vm_sync	`Error`	179.36	test_vm_sync.py
test_07_project_resources_account_delete	`Error`	5.63	test_projects.py
test_08_cleanup_after_project_delete	`Error`	7.48	test_projects.py
ContextSuite context=TestProjectResources>:teardown	`Error`	9.69	test_projects.py
ContextSuite context=TestProjectSuspendActivate>:teardown	`Error`	96.37	test_projects.py
test_02_list_snapshots_with_removed_data_store	`Error`	15.46	test_snapshots.py
ContextSuite context=TestDeployVmWithAffinityGroup>:teardown	`Error`	159.52	test_affinity_groups_projects.py
test_UpdateConfigParamWithScope	`Error`	0.65	test_global_settings.py
test_02_create_template_with_checksum_sha1	`Error`	5.20	test_templates.py
test_03_create_template_with_checksum_sha256	`Error`	5.19	test_templates.py
test_04_create_template_with_checksum_md5	`Error`	5.19	test_templates.py
test_dedicatePublicIpRange	`Error`	0.04	test_public_ip_range.py
test_create_pvlan_network	`Error`	0.04	test_pvlan.py
test_03_user_role_dont_see_annotations	`Failure`	3.30	test_host_annotations.py
test_03_RVR_Network_check_router_state	`Failure`	145.65	test_routers_network_ops.py
test_02_deploy_vm_root_resize	`Failure`	0.11	test_deploy_vm_root_resize.py
test_02_vpc_privategw_static_routes	`Failure`	225.69	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	208.86	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	331.24	test_privategw_acl.py
runTest	`Error`	0.00	test_network.py
test_03_migrate_detached_volume	`Error`	82.23	test_vm_life_cycle.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	327.76	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	`Failure`	334.51	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	213.45	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Failure`	167.69	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	303.81	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	336.26	test_vpc_redundant.py
test_oobm_background_powerstate_sync	`Failure`	21.73	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	`Error`	31.97	test_outofbandmanagement.py
test_oobm_issue_power_cycle	`Error`	18.75	test_outofbandmanagement.py
test_oobm_issue_power_off	`Error`	19.73	test_outofbandmanagement.py
test_oobm_issue_power_on	`Error`	18.75	test_outofbandmanagement.py
test_oobm_issue_power_reset	`Error`	18.73	test_outofbandmanagement.py
test_oobm_issue_power_soft	`Error`	19.76	test_outofbandmanagement.py
test_oobm_issue_power_status	`Error`	19.74	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	`Failure`	27.88	test_outofbandmanagement.py
test_oobm_zchange_password	`Error`	7.39	test_outofbandmanagement.py
test_createPortablePublicIPAcquire	`Error`	0.03	test_portable_publicip.py
test_createPortablePublicIPRange	`Error`	0.03	test_portable_publicip.py
test_02_cancel_host_maintenace_with_migration_jobs	`Failure`	218.07	test_host_maintenance.py

shwstppr · 2023-12-18T06:57:09Z

@vishesh92 similar failures in VMware test results. Anything related?

vishesh92 · 2023-12-18T07:57:29Z

@vishesh92 similar failures in VMware test results. Anything related?

No. It doesn't have anything related to my changes.

vishesh92 · 2023-12-18T08:13:46Z

@shwstppr I checked a few failures and most of them seem related to the environment itself or some error while cleaning up resources.

vishesh92 · 2023-12-18T08:20:08Z

@blueorangutan package

blueorangutan · 2023-12-18T08:22:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2023-12-18T08:38:53Z

@vishesh92 similar failures in VMware test results. Anything related?

No. It doesn't have anything related to my changes.

Considering the current smoke tests are mostly very stable (based on the results of health check PRs), can we revisit this after 4.19 release ?
@shwstppr @vishesh92

blueorangutan · 2023-12-18T09:22:38Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8076

vishesh92 · 2023-12-18T09:48:15Z

@blueorangutan test matrix

blueorangutan · 2023-12-18T09:50:03Z

@vishesh92 a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

vishesh92 · 2023-12-18T09:51:41Z

@vishesh92 similar failures in VMware test results. Anything related?

No. It doesn't have anything related to my changes.

Considering the current smoke tests are mostly very stable (based on the results of health check PRs), can we revisit this after 4.19 release ? @shwstppr @vishesh92

@weizhouapache Just rebuilt the packages again. Last time there was some issues while packaging. Let's wait for the results of the latest test run.

blueorangutan · 2023-12-18T21:29:27Z

[SF] Trillian test result (tid-8619)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40527 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8281-t8619-xenserver-71.zip
Smoke tests completed. 120 look OK, 8 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_DeployVmAntiAffinityGroup_in_project	`Error`	101.04	test_affinity_groups_projects.py
test_05_list_volumes_isrecursive	`Failure`	0.03	test_list_volumes.py
test_07_list_volumes_listall	`Failure`	0.02	test_list_volumes.py
test_01_add_primary_storage_disabled_host	`Error`	42.16	test_primary_storage.py
test_01_non_strict_host_anti_affinity	`Failure`	132.10	test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity	`Error`	92.42	test_nonstrict_affinity_group.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	35.78	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	50.07	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	57.89	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Error`	57.90	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	`Error`	1.21	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	43.22	test_kubernetes_clusters.py
test_01_sys_vm_start	`Failure`	0.07	test_secondary_storage.py
test_01_deploy_vm_on_specific_host	`Error`	0.07	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	0.10	test_vm_deployment_planner.py
test_08_migrate_vm	`Error`	0.06	test_vm_life_cycle.py

blueorangutan · 2023-12-18T23:49:19Z

[SF] Trillian test result (tid-8621)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 48738 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8281-t8621-kvm-centos7.zip
Smoke tests completed. 126 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_05_list_volumes_isrecursive	`Failure`	0.04	test_list_volumes.py
test_07_list_volumes_listall	`Failure`	0.03	test_list_volumes.py
test_08_migrate_vm	`Error`	0.07	test_vm_life_cycle.py

blueorangutan · 2023-12-19T01:46:11Z

[SF] Trillian test result (tid-8620)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 55870 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8281-t8620-vmware-67u3.zip
Smoke tests completed. 124 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_DeployVmAntiAffinityGroup_in_project	`Error`	103.94	test_affinity_groups_projects.py
test_01_browser_migrate_template	`Failure`	5.36	test_image_store_object_migration.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	747.14	test_kubernetes_clusters.py
test_02_trigger_shutdown	`Failure`	336.79	test_safe_shutdown.py

weizhouapache · 2023-12-19T07:18:51Z

smoke tests look ok. the failures might be caused by the busy trillian envs.

weizhouapache · 2023-12-19T07:18:57Z

@blueorangutan LLtest matrix

blueorangutan · 2023-12-19T07:20:03Z

@weizhouapache a [LL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-12-19T18:55:44Z

[LL]Trillian test result (tid-6901)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40484 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8281-t6901-kvm-centos7.zip
Smoke tests completed. 125 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_05_list_volumes_isrecursive	`Failure`	0.03	test_list_volumes.py
test_07_list_volumes_listall	`Failure`	0.02	test_list_volumes.py
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	`Error`	0.00	test_kubernetes_clusters.py
test_08_migrate_vm	`Error`	0.04	test_vm_life_cycle.py

blueorangutan · 2023-12-19T20:26:33Z

[LL]Trillian test result (tid-6900)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 45993 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8281-t6900-vmware-67u3.zip
Smoke tests completed. 125 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_DeployVmAntiAffinityGroup_in_project	`Error`	86.78	test_affinity_groups_projects.py
test_01_browser_migrate_template	`Failure`	5.25	test_image_store_object_migration.py
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	`Error`	0.00	test_kubernetes_clusters.py

shwstppr · 2023-12-21T12:09:44Z

Based on a discussion with contributors involved in #8392, I'm reverting this for now. We will revisit e2e tests later

This reverts commit 1411da1.

This PR adds e2e tests for listing resources

This reverts commit 1411da1.

boring-cyborg bot added the component:integration label Nov 30, 2023

vishesh92 force-pushed the e2e-test-resource-listing branch from d82e05f to 2294a23 Compare November 30, 2023 06:33

vishesh92 closed this Nov 30, 2023

vishesh92 reopened this Nov 30, 2023

vishesh92 force-pushed the e2e-test-resource-listing branch from 2294a23 to 3723613 Compare December 1, 2023 08:21

DaanHoogland closed this Dec 1, 2023

DaanHoogland reopened this Dec 1, 2023

vishesh92 force-pushed the e2e-test-resource-listing branch from 3723613 to cff8e1a Compare December 1, 2023 16:21

boring-cyborg bot added component:integration-test Python Warning... Python code Ahead! labels Dec 1, 2023

vishesh92 closed this Dec 1, 2023

vishesh92 reopened this Dec 1, 2023

vishesh92 force-pushed the e2e-test-resource-listing branch from 5c32b76 to 1fc6920 Compare December 1, 2023 18:26

vishesh92 closed this Dec 3, 2023

vishesh92 reopened this Dec 3, 2023

vishesh92 marked this pull request as ready for review December 4, 2023 11:14

vishesh92 requested a review from DaanHoogland December 4, 2023 11:15

DaanHoogland approved these changes Dec 4, 2023

View reviewed changes

shwstppr merged commit 1411da1 into apache:main Dec 20, 2023

shwstppr deleted the e2e-test-resource-listing branch December 20, 2023 06:25

shwstppr mentioned this pull request Dec 21, 2023

Fixup test_list_volumes.py failure #8392

Closed

13 tasks

shwstppr added a commit that referenced this pull request Dec 21, 2023

Revert "Add e2e tests for listing resources (#8281)"

85fb0e9

This reverts commit 1411da1.

shwstppr mentioned this pull request Dec 21, 2023

Revert "Add e2e tests for listing resources" #8396

Merged

shwstppr restored the e2e-test-resource-listing branch December 21, 2023 12:10

shwstppr added a commit that referenced this pull request Dec 21, 2023

Revert "Add e2e tests for listing resources (#8281)" (#8396)

ab80899

This reverts commit 1411da1.

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Dec 27, 2023

Add e2e tests for listing resources (apache#8281)

08d1315

This PR adds e2e tests for listing resources

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Dec 27, 2023

Revert "Add e2e tests for listing resources (apache#8281)" (apache#8396)

3a91b43

This reverts commit 1411da1.

vishesh92 mentioned this pull request Dec 28, 2023

Add e2e tests for listing resources #8410

Merged

13 tasks

Conversation

vishesh92 commented Nov 30, 2023 • edited by shwstppr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Nov 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DaanHoogland commented Nov 30, 2023

Uh oh!

blueorangutan commented Nov 30, 2023

Uh oh!

blueorangutan commented Nov 30, 2023

Uh oh!

vishesh92 commented Dec 4, 2023

Uh oh!

blueorangutan commented Dec 4, 2023

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

vishesh92 commented Dec 4, 2023

Uh oh!

blueorangutan commented Dec 4, 2023

Uh oh!

vishesh92 commented Dec 5, 2023

Uh oh!

blueorangutan commented Dec 5, 2023

Uh oh!

blueorangutan commented Dec 5, 2023

Uh oh!

vishesh92 commented Dec 5, 2023

Uh oh!

blueorangutan commented Dec 18, 2023

Uh oh!

shwstppr commented Dec 18, 2023

Uh oh!

vishesh92 commented Dec 18, 2023

Uh oh!

vishesh92 commented Dec 18, 2023

Uh oh!

vishesh92 commented Dec 18, 2023

Uh oh!

blueorangutan commented Dec 18, 2023

Uh oh!

weizhouapache commented Dec 18, 2023

Uh oh!

blueorangutan commented Dec 18, 2023

Uh oh!

vishesh92 commented Dec 18, 2023

Uh oh!

blueorangutan commented Dec 18, 2023

Uh oh!

vishesh92 commented Dec 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueorangutan commented Dec 18, 2023

Uh oh!

blueorangutan commented Dec 18, 2023

Uh oh!

blueorangutan commented Dec 19, 2023

Uh oh!

weizhouapache commented Dec 19, 2023

Uh oh!

weizhouapache commented Dec 19, 2023

Uh oh!

blueorangutan commented Dec 19, 2023

Uh oh!

blueorangutan commented Dec 19, 2023

Uh oh!

blueorangutan commented Dec 19, 2023

Uh oh!

vishesh92 commented Nov 30, 2023 •

edited by shwstppr

Loading

codecov bot commented Nov 30, 2023 •

edited

Loading

vishesh92 commented Dec 18, 2023 •

edited

Loading