Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

reuse ip for non redundant VPC#2712

Merged
yadvr merged 2 commits intoapache:4.11apache/cloudstack:4.11from
shapeblue:privateGatewayRestartVpcFixshapeblue/cloudstack:privateGatewayRestartVpcFixCopy head branch name to clipboard
Jun 21, 2018
Merged

reuse ip for non redundant VPC#2712
yadvr merged 2 commits intoapache:4.11apache/cloudstack:4.11from
shapeblue:privateGatewayRestartVpcFixshapeblue/cloudstack:privateGatewayRestartVpcFixCopy head branch name to clipboard

Conversation

@DaanHoogland
Copy link
Contributor

@DaanHoogland DaanHoogland commented Jun 18, 2018

As rolling restart does not deallocate an IP before configuring it on a new VR,
the code must allow it to be reused on a non-redundant VPCs gateway nic.

Description

removed the condition to only look for reuse of IP on redundant VPCs

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

GitHub Issue/PRs

Screenshots (if appropriate):

How Has This Been Tested?

Checklist:

  • I have read the CONTRIBUTING document.
  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
    Testing
  • I have added tests to cover my changes.
  • All relevant new and existing integration tests have passed.
  • A full integration testsuite with all test that can run on my environment has passed.

@DaanHoogland DaanHoogland added this to the 4.11.1.0 milestone Jun 18, 2018
@DaanHoogland DaanHoogland self-assigned this Jun 18, 2018
@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented Jun 18, 2018

@blueorangutan package

@blueorangutan
Copy link

blueorangutan commented Jun 18, 2018

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

blueorangutan commented Jun 18, 2018

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2127

@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented Jun 18, 2018

@blueorangutan test

@blueorangutan
Copy link

blueorangutan commented Jun 18, 2018

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan
Copy link

blueorangutan commented Jun 18, 2018

Trillian test result (tid-2762)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29911 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2712-t2762-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 62 look OK, 5 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestDeployVirtioSCSIVM>:setup Error 0.00 test_deploy_virtio_scsi_vm.py
test_02_vpc_privategw_static_routes Failure 189.01 test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup Failure 190.38 test_privategw_acl.py
test_02_not_secured_vm_migration Error 5.21 test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration Error 5.17 test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration Error 1.09 test_vm_life_cycle.py
test_08_migrate_vm Error 6.33 test_vm_life_cycle.py
test_01_cancel_host_maintenace_with_no_migration_jobs Failure 0.08 test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs Error 1.26 test_host_maintenance.py
test_hostha_enable_ha_when_host_in_maintenance Error 1.42 test_hostha_kvm.py

@yadvr
Copy link
Member

yadvr commented Jun 19, 2018

@blueorangutan package

@blueorangutan
Copy link

blueorangutan commented Jun 19, 2018

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

blueorangutan commented Jun 19, 2018

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2131

@PaulAngus
Copy link
Member

PaulAngus commented Jun 19, 2018

@DaanHoogland have you looked at the VPC private gateway test failures?

@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented Jun 19, 2018

The failure is a failure to ping. This may have many causes. I think we need to manually test this functionality, to see if either the test or the private gateway (or acl/fw) functionality needs fixing, @PaulAngus

@PaulAngus
Copy link
Member

PaulAngus commented Jun 19, 2018

@blueorangutan test

@blueorangutan
Copy link

blueorangutan commented Jun 19, 2018

@PaulAngus a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@apache apache deleted a comment from blueorangutan Jun 19, 2018
@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented Jun 19, 2018

After fixing the original problem a new bug arises. When restarting a VPC with a tier, the tier seems to come back unconnected and the VPC router and the interface for it has no IP address.

@blueorangutan
Copy link

blueorangutan commented Jun 20, 2018

Trillian test result (tid-2771)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28506 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2712-t2771-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 63 look OK, 4 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestDeployVirtioSCSIVM>:setup Error 0.00 test_deploy_virtio_scsi_vm.py
test_02_vpc_privategw_static_routes Failure 191.27 test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup Failure 1205.21 test_privategw_acl.py
test_04_rvpc_privategw_static_routes Failure 382.48 test_privategw_acl.py
test_01_secured_vm_migration Error 1.09 test_vm_life_cycle.py
test_02_not_secured_vm_migration Error 1.09 test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration Error 1.09 test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration Error 1.10 test_vm_life_cycle.py
test_08_migrate_vm Error 7.27 test_vm_life_cycle.py
test_10_attachAndDetach_iso Error 12.89 test_vm_life_cycle.py
test_01_cancel_host_maintenace_with_no_migration_jobs Failure 0.09 test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs Error 39.94 test_host_maintenance.py

@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented Jun 20, 2018 •
edited
Loading

After some more testing locally I see that sometimes restart with cleanup works and sometimes it doesn't. At one occasion I tried to make a VPC redundant and ...
screenshot from 2018-06-20 14-31-38

I undid the redundancy and after that restarts just worked. The use case in the test is to have two VPCs connected through their private gateways and static routes added towards each other. Then a VPC is restarted with cleanup and it is tested whether the VMs are still pingable.

I am now convinced that the improvement in this PR does not solve everything but neither does it cause the errors in the restart.

@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented Jun 20, 2018

@blueorangutan package

@blueorangutan
Copy link

blueorangutan commented Jun 20, 2018

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

blueorangutan commented Jun 20, 2018

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2140

@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented Jun 20, 2018

@blueorangutan test

@blueorangutan
Copy link

blueorangutan commented Jun 20, 2018

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan
Copy link

blueorangutan commented Jun 21, 2018

Trillian test result (tid-2779)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33440 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2712-t2779-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 4 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestDeployVirtioSCSIVM>:setup Error 0.00 test_deploy_virtio_scsi_vm.py
test_03_vpc_privategw_restart_vpc_cleanup Failure 177.62 test_privategw_acl.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Error 132.50 test_vpc_redundant.py
test_02_redundant_VPC_default_routes Failure 280.05 test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics Failure 294.14 test_vpc_redundant.py
test_02_VPC_default_routes Failure 141.28 test_vpc_router_nics.py

Daan Hoogland and others added 2 commits June 21, 2018 11:11
@yadvr
reuse ip for non redundant VPC
f8fa446 
 As rolling restart does not deallocate an IP before configuring it on a new VR,
 the code must allow it to be reused on a non-redundant VPCs gateway nic.
@yadvr
smoketest: increase pind counts to reduce intermittent failures
24a5baa 
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
@yadvr yadvr force-pushed the privateGatewayRestartVpcFix branch from 8d45f5a to 24a5baa Compare June 21, 2018 06:18
@yadvr
Copy link
Member

yadvr commented Jun 21, 2018 •
edited
Loading

Ran tests manually with increased ping thresholds: (rebased against 4.11, no changes to the original commit by Daan)

test_vpc_router_nics.py:
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 2 tests in 566.266s

OK

test_vpc_redundant.py:

Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | Status : SUCCESS ===
ok
Create a redundant VPC with 1 Tier, 1 VM, 1 ACL, 1 PF and test Network GC Nics ... === TestName: test_04_rvpc_network_garbage_collector_nics | Status : SUCCESS ===
ok
Create a redundant VPC with multiple tiers ... === TestName: test_05_rvpc_multi_tiers | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 5 tests in 2354.352s

OK


test_privategw_acl.py:

test_01_vpc_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_01_vpc_privategw_acl | Status : SUCCESS ===
ok
test_02_vpc_privategw_static_routes (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_02_vpc_privategw_static_routes | Statu
s : SUCCESS ===
ok
test_03_vpc_privategw_restart_vpc_cleanup (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_03_vpc_privategw_restart_vpc_cle
anup | Status : FAILED ===
FAIL
test_04_rvpc_privategw_static_routes (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_04_rvpc_privategw_static_routes | Sta
tus : SUCCESS ===
ok

@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented Jun 21, 2018

thanks @rhtyd so we merge? (cc @PaulAngus )

@yadvr
Copy link
Member

yadvr commented Jun 21, 2018

Still working/testing @DaanHoogland I'll confirm and post once results are here.

@yadvr
Copy link
Member

yadvr commented Jun 21, 2018

The test_03_vpc_privategw_restart_vpc_cleanup | Status : FAILED === test has been consistently failing for me. What I observe is that post restart with cleanup=true of the VPC VR it is unable to connect to the guest VM. I'm investigating this case now.

@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented Jun 21, 2018

I know about test_03_vpc_privategw_restart_vpc_cleanup @rhtyd . The only time I got this functionality to work was on manual testing. I think there is something with the order of applying interface and addresses that makes it work or fail :(

@yadvr
Copy link
Member

yadvr commented Jun 21, 2018

There was no clear solution for fixing the nic/ip sequencing/application issue, I could reproduce the error sometimes and sometimes the test would pass when the sequence applied cleanly. Given a workaround for the non-redundant restart VPC with private gateway enabled exists in case a sequence issue is hit (i.e. user should manually destroy the VPC VR and restart the VPC without cleanup selected), I'll merge this /cc @PaulAngus. I verified that the issue does not arise in case of redundant VPCs.

@yadvr yadvr merged commit 52b02de into apache:4.11 Jun 21, 2018
@yadvr yadvr mentioned this pull request Jun 21, 2018
Closed
@yadvr
Copy link
Member

yadvr commented Jun 21, 2018

Issue logged - #2719

ernjvr pushed a commit to shapeblue/cloudstack that referenced this pull request Jul 3, 2018
@DaanHoogland @ernjvr
vpc: reuse private gateway ip for non redundant VPC (apache#2712)
602596a 
As rolling restart does not deallocate an IP before configuring it on a new VR, the code must allow it to be reused on a non-redundant VPCs gateway nic.
In crease ping counts to reduce intermittent failures in smoketests.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
@snyk-bot snyk-bot mentioned this pull request Jul 16, 2021
Open
@snyk-bot snyk-bot mentioned this pull request Sep 4, 2021
Open
@DaanHoogland DaanHoogland removed their assignment Jun 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yadvr yadvr yadvr approved these changes

@rafaelweingartner rafaelweingartner rafaelweingartner approved these changes

@PaulAngus PaulAngus Awaiting requested review from PaulAngus

@nvazquez nvazquez Awaiting requested review from nvazquez

Assignees

No one assigned

Labels

component:vpc Severity:BLOCKER

Projects

None yet

Milestone

4.11.1.0

Development

Successfully merging this pull request may close these issues.

5 participants

@DaanHoogland @blueorangutan @yadvr @PaulAngus @rafaelweingartner
Morty Proxy This is a proxified and sanitized view of the page, visit original site.