improvement: systemvm slow operations#1604
Closed
romain-dartigues wants to merge 1 commit intoapache:4.9apache/cloudstack:4.9from
romain-dartigues:systemvm-none-gatewayromain-dartigues/cloudstack:systemvm-none-gatewayCopy head branch name to clipboard
Closed
improvement: systemvm slow operations#1604romain-dartigues wants to merge 1 commit intoapache:4.9apache/cloudstack:4.9from romain-dartigues:systemvm-none-gatewayromain-dartigues/cloudstack:systemvm-none-gatewayCopy head branch name to clipboard
romain-dartigues wants to merge 1 commit intoapache:4.9apache/cloudstack:4.9from
romain-dartigues:systemvm-none-gatewayromain-dartigues/cloudstack:systemvm-none-gatewayCopy head branch name to clipboard
Conversation
| dp['gateway'] = self.qFile.data['cmd_line']['localgw'] | ||
| else: | ||
| dp['gateway'] = 'None' | ||
| dp['gateway'] = self.qFile.data['cmd_line'].get('gateway', 'None') |
Contributor
There was a problem hiding this comment.
there could be a possibility of having an ipv6 gateway, in that case, we would be looking at ip6gateway.
Contributor
|
@romain-dartigues do we know why the |
Contributor
Author
Right, but IPv6 does not use ARP but NDP (RFC 4861). It seems that a
Apparently because the file "/etc/cloudstack/ips.json" is generated with a gateway at "None" even when the gateway is present in the input "cmd_line.json" (but my memory could fail me at this time of the day). One month ago we found the "problem" and wanted a fix; trying to understand why it did happen made me start #1575 (now #1603). I though it would be good (at least for me) to get a better understanding of the code; doing hundreds of arping everytime is probably inefficient as each of them is supposed to refresh the ARP table of all equipment on the way... |
Contributor
|
Thanks @romain-dartigues based on your comments the code LGTM |
Contributor
|
we do have to remember that arping is ipv4 only so we need to fix this for ipv6 too |
Member
|
@blueorangutan package |
|
@rhtyd a Trillian-Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✖debian repo: http://packages.shapeblue.com/cloudstack/pr/1604 |
| cmd = "arping -c 1 -I %s -A -U -s %s %s" % ( | ||
| self.dev, self.address['public_ip'], self.address['gateway']) | ||
| CsHelper.execute(cmd) | ||
| CsHelper.execute(cmd, wait=False) |
Contributor
There was a problem hiding this comment.
Flag arguments are an anti-pattern. Please consider introducing and executeAndWait function instead.
Member
|
@romain-dartigues can you fix the merge conflict, squash the changes, open a JIRA issue referencing that in the commit message. |
Contributor
Author
|
@rhtyd < JIRA issue #9568 Rebase done, change merge request target done. |
fce10c5 to
9c47e97
Compare
Member
|
@murali-reddy can you help review this one? |
21f4e6d to
67bfc7c
Compare
Member
|
@romain-dartigues @murali-reddy ping @blueorangutan package |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-208 |
67bfc7c to
ebe6537
Compare
Contributor
Author
|
@rhtyd pong (is it OK like that? please advise) |
Member
|
Thanks @romain-dartigues that looks good. I'll kick a test round soon. |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-227 |
Member
|
@blueorangutan test |
|
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests |
|
Trillian test result (tid-379)
|
Contributor
Author
|
I'll need help to understand what gone wrong. I just peeked at But |
Member
|
@romain-dartigues I'll have a look at the failure, meanwhile can you see my comments against the changes https://github.com/apache/cloudstack/pull/1604/files |
ebe6537 to
a0d0c8d
Compare
Contributor
Author
|
@rhtyd < separate function done. |
Member
|
okay @romain-dartigues I'll kick some tests |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✖centos6 ✖centos7 ✖debian. JID-238 |
Member
|
@blueorangutan package |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-243 |
Member
|
@murali-reddy @abhinandanprateek can you help review this, thanks |
|
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests |
|
Trillian test result (tid-425)
|
Contributor
|
@romain-dartigues we fixed issue where ips.json has wrong gateway due to error in parsing the cmd_line.json. its committed in 4.8 and above [1]. Do you still see (or can you please verify) performance hit due to arpping to wrong gateway on 4.8/4.9/master? [1] 93ac134 |
Contributor
Author
|
@murali-reddy I'm not in position to run tests anymore |
Member
|
Several test failures, unless the PR author can fix, this is on hold. |
Member
|
@romain-dartigues @dpassante any update on this? |
Contributor
Author
|
No comment, since it does not seems required anymore I close this request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On our setup we had issues with vrouters when a large number of VM were created at once.
The vrouters try to arping the gateway (to refresh the ARP cache I suppose), but in some case it take too much time and the vrouters got killed for being unresponsive.
The following patches attempt to solve this by:
This patch has been running for a month on our test environment and seems to solve the initial problem.
An operation (starting 220 VM on XenServer, shared storage) which took 46 mn, now take 4 mn (without errors).