[00:34] <jcw4> wallyworld_: looking at the error messages I was wondering if it was an issue with stale .a files on the test machine... is that possible?
[00:34] <wallyworld_> jcw4: maybe, but i can reproduce locally using compiler=gccgo
[00:34] <wallyworld_> i get lots of segfaults as well
[00:34] <jcw4> wallyworld_: ah.. I misunderstood... I thought you couldn't repro locally.
[00:34] <wallyworld_> but i can get the test failure
[00:35] <jcw4> wallyworld_: is there any way to debug if you don't have a ppc machine?
[00:35] <wallyworld_> i can't reproduce unless i just run one test at a time
[00:35] <wallyworld_> you have to know how gccgo works i think
[00:35] <wallyworld_> i have no idea :-(
[00:35] <jcw4> I see
[00:35] <jcw4> :)
[00:35] <wallyworld_> davecheney: you around?
[00:37] <davecheney> wallyworld_: ack
[00:38] <wallyworld_> davecheney: bug 1365480 is blocking ci. it appears to be a gccgo issue because it fails to run the hooks used to mock out method calls
[00:38] <mup> Bug #1365480: ppc64el unit tests fail in many ways <ci> <ppc64el> <regression> <juju-core:Triaged by wallyworld> <https://launchpad.net/bugs/1365480>
[00:38] <wallyworld_> i have no idea how to fix
[00:39] <wallyworld_> this is failing to work
[00:39] <wallyworld_> 	cleanup := s.srv.Service.Nova.RegisterControlPoint(
[00:39] <wallyworld_> 		"addFloatingIP",
[00:39] <wallyworld_> 		func(sc hook.ServiceControl, args ...interface{}) error {
[00:39] <wallyworld_> 			return fmt.Errorf("failed on purpose")
[00:39] <wallyworld_> 		},
[00:39] <wallyworld_> 	)
[00:39] <wallyworld_> the register func uses the stackframe to figure out what to do
[00:40] <wallyworld_> i guess it's broken again - i think it was broken before at some point?
[00:40] <wallyworld_> there's several tests affected
[00:42] <davecheney> yeah, it breaks a bit
[00:43] <davecheney> has the version of gccgo on the builder machine changed ?
[00:43] <wallyworld_> nfi
[00:43] <wallyworld_> i was running gccgo (Ubuntu 4.9.1-10ubuntu2) 4.9.1
[00:43] <wallyworld_> the build machine had gccgo (Ubuntu 4.9.1-12ubuntu2) 4.9.1
[00:44] <wallyworld_> i updated to -12 locally
[00:44] <davecheney> ok, and it only repro's on ppc ?
[00:44] <wallyworld_> i can repo locally using -compiler=gccgo
[00:44] <wallyworld_> but i get LOTS of segfaults
[00:44] <wallyworld_> i have to specify each test one at a time
[00:44] <wallyworld_> and yes, ci fails when running of ppc
[00:55] <wallyworld_> davecheney: am i able to ask you to look into this a bit? i have no idea where to start with regard to gccgo
[00:59] <davecheney> wallyworld_: can I fix it on monday ?
[01:00] <wallyworld_> davecheney: it's blocking landings sadly, unless we can get the regressionm tag removed
[01:01] <davecheney> i recommend removing the regression tag
[01:01] <davecheney> if this is a compiler fix
[01:01] <davecheney> we can't do that at critical level
[01:01] <wallyworld_> sinzui: bug 1365480 looks like a gccgo issue, is there anyway we can remove the regression tag?
[01:01] <mup> Bug #1365480: ppc64el unit tests fail in many ways <ci> <ppc64el> <regression> <juju-core:Triaged by wallyworld> <https://launchpad.net/bugs/1365480>
[01:01] <wallyworld_> davecheney can do a compiler fix but not till mondau
[01:01] <sinzui> wallyworld_, your made
[01:02] <davecheney> i can look at it on monday
[01:02] <sinzui> mad
[01:02] <davecheney> i can't promise a fix
[01:02] <sinzui> wallyworld_, the old version of juju works, and now it doesn't.
[01:03] <wallyworld_> sinzui: i can prove that code which has not been touched for ages fails because gccgo does not register the monkey patch being applied
[01:03] <sinzui> wallyworld_, we can retest an older revision, maybe the one that passed. If the test fails like the new revision then we know something other than juju changes
[01:04] <wallyworld_> gccgo is can be fragile when it comes to looking at the call stack
[01:04] <wallyworld_> which is how the monkey patching stuff works
[01:04] <wallyworld_> fragile = different to golanggo
[01:05] <sinzui> I will retest the last passing revision, if it fails the same way then you are vindicated
[01:05] <wallyworld_> sinzui: was gccgo updated recently?
[01:05] <wallyworld_> on the test vm?
[01:05] <sinzui> wallyworld_, We would see that in the first test that failed
[01:06] <sinzui> wallyworld_, you loose, http://juju-ci.vapour.ws:8080/job/run-unit-tests-trusty-ppc64el/1213/console clearly states that gcc was already the latest version and that no packages were installed for the test
[01:08] <wallyworld_> and yet the tests that are failing have not changed and the failure is clearly due to gccgo not executing monkey patched code that the tests rely on to pass
[01:08] <wallyworld_> i put a panic in the code and it did not trigger
[01:08] <sinzui> wallyworld_, there is a difference, but it is not see in installs...
[01:09] <sinzui> The passing one has
[01:09] <sinzui> go version xgcc (Ubuntu 4.9.1-10ubuntu3) 4.9.1 linux/ppc64
[01:09] <sinzui> The failing one has
[01:09] <sinzui> go version xgcc (Ubuntu 4.9.1-12ubuntu2) 4.9.1 linux/ppc64
[01:09] <wallyworld_> yes, that's what i used to have here till i upgrade
[01:09] <wallyworld_> i'm on utopic now and it doesn't give me an option to go back to -10
[01:10] <wallyworld_> wait
[01:10] <wallyworld_> yes it does
[01:10] <sinzui> wallyworld_, I can look into this after I avert the disaster that really cannot be averted
[01:10] <wallyworld_> ok
[01:10] <wallyworld_> i'll try testing with -10
[01:11] <davecheney> ok, this is not good
[01:11] <davecheney> -12 must be the new version in proposed which fixes a different bug
[01:12] <sinzui> FU&CKI
[01:13] <sinzui> wallyworld_, even after using s3cmd to sync the tools that are on aws, I still get different filesizes from streams.canonical.com
[01:13] <wallyworld_> i can't seem to get apt to allow me to downgrade to -10 to test
[01:13] <wallyworld_> wot :-(
[01:14] <wallyworld_> sinzui: that is not good :-(
[01:14] <davecheney> wallyworld_: juju bootstrap && juju deploy cs:ubuntu
[01:15] <sinzui> wallyworld_, Am I experiencing this because i finally reported the versioning issue as a bug https://bugs.launchpad.net/juju-core/+bug/1365633
[01:15] <mup> Bug #1365633: cannot rebuild replacement tools for streams <ci> <juju-core:Triaged> <https://launchpad.net/bugs/1365633>
[01:15] <wallyworld_> looking
[01:15] <sinzui> wallyworld_, We have lived with this since Fabruary, I report the bug and now I need the fix
[01:16] <sinzui> wallyworld_, tools that should be identical are not, I cannot given then extra version information to differentiate their origin to avoid confusion or outright malign intent
[01:17] <wallyworld_> sinzui: simplestreams supports versioning using dates
[01:17] <sinzui> wallyworld_, that is not helping the users
[01:17] <wallyworld_> new tools tarballs with different names could be uploaded
[01:18] <wallyworld_> and new metadata with a newer date added
[01:18] <sinzui> wallyworld_, I am going to remake this data, and now I can expect users to complain that tools of the same name dont match
[01:18] <wallyworld_> the tarball name used to matter before simplestreams but it doesn't now
[01:18] <sinzui> wallyworld_, but these tools from two different machines that should be the same have different sums
[01:19] <wallyworld_> the tools tarball could be called juju-1.20.6-release1-precise-amd64 and juju-1.20.6-release2-precise-amd64.
[01:20] <wallyworld_> which one to use comes from the simplestreams metadata
[01:21] <wallyworld_> the latter one would be in the metadata with a later date, so that would be be picked up if juju asks for which tools to use for series/arch/release
[01:21] <wallyworld_> maybe i'm missing something
[01:21] <sinzui> wallyworld_, That would help. when I tested alternate names for tools, the metadata command ignored them :(
[01:22] <wallyworld_> that may be a limitation of that command :-(
[01:22] <wallyworld_> which needs to be fixed
[01:22] <sinzui> I have done evil things to preserve the greater good
[01:23] <wallyworld_> that command i think from memory does use the filename to suck stuff in
[01:23] <wallyworld_> it could be made smarter
[01:23] <sinzui> yeah, the convention is convenient for many people copying tools.
[01:24] <wallyworld_> or made so it can be called from a script, passing in the required tarball and params
[01:24] <sinzui> wallyworld_, maybe...
[01:24] <wallyworld_> sinzui: we are moving to a shared tarball across series
[01:24] <wallyworld_> ie one tarball only for precise/trusty/utopic
[01:25] <wallyworld_> since they are the same
[01:25] <sinzui> I have just reconciled the diffs from what was last in the CPCs and my own machine to make a json that describes what what there and what I am now uploading.
[01:25] <wallyworld_> so the filename will become less relvant
[01:26] <sinzui> wallyworld_, I would like to do that. The number of tools we make and publish do take a lot of time
[01:26] <wallyworld_> yes indeed :-(
[01:26] <wallyworld_> it's sorta happening now as part of moving tools into mongo storage
[01:27] <wallyworld_> and removing the need for cloud storage
[01:27] <sinzui> wallyworld_, I think if this command worked for azure, we might have prevented my misadventure
[01:27] <sinzui> juju metadata validate-tools --juju-version 1.20.7
[01:28] <wallyworld_> oh, azure doesn't currently support custom metadata
[01:29] <wallyworld_> i because there's no central storage we can use from memory
[01:29] <sinzui> wallyworld_, but joyent does. I don't understand? isn't the command getting the json and answering the version question?
[01:29] <wallyworld_> like we have for aws and hp cloud
[01:30] <wallyworld_> it's been ages since i looed at that stuff - from memory it's because there's no support for a custome search path on azure, i can't recallwhy
[01:30] <wallyworld_> i'll have to go digging in the code
[01:30] <wallyworld_> and even if no custom tools location is supported, i would think the metadata command should still work
[01:31] <wallyworld_> don't know why it doesn't :-(
[01:31] <sinzui> wallyworld_, oh yes, now I understand. I faced some of that using their python adk
[01:31] <sinzui> sdk
[01:32] <sinzui> wallyworld_, We add md5 and shasum metadata to each tool we upload to azure and manta because we wrote our own rsync tools to do what real storage systems do
[01:32] <wallyworld_> ok
[01:33] <sinzui> manta still sucks though. there is a 5 minute period where we make 1000+ calls to look up the sums because it doesn't support bulk queries
[01:33] <sinzui> well swift doesn't either, but the web/xml interface does
[01:34] <wallyworld_> 1000+ !!
[01:34] <wallyworld_> sinzui: we will soon not need cloud storage for juju
[01:35] <sinzui> wallyworld_, indeed...part of the tools problem is that each machine is downloading tools from one or more sources and that allows for mismatches
[01:35] <wallyworld_> yeah, so soon all machines will get tools from the state server
[01:35] <wallyworld_> the tools are loaded into the state server on bootstrap
[01:39] <sinzui> wallyworld_, I am 1. starting a rebuild of the last good master rev. I am 2, looking for the old packages to revert one of the machines to
[01:40] <wallyworld_> ty
[01:40] <wallyworld_> i've updated the bug with my thoughts
[01:42] <sinzui> wallyworld_, I might be able to go back to what was in place on Aug 31 http://ports.ubuntu.com/pool/universe/g/gcc-4.9/
[01:43] <wallyworld_> sinzui: that would be great. you may also find that gcc-base and other packages need downgrading also
[01:44] <sinzui> wallyworld_, yeah, that is what makes this hard
[01:44] <wallyworld_> indeed :-(
[02:08] <thumper> wallyworld_: I'm going to see if I can fix this bug: https://bugs.launchpad.net/juju-core/+bug/1348477
[02:08] <mup> Bug #1348477: userAuthenticatorSuite.TearDown failure <ci> <intermittent-failure> <regression> <test-failure> <juju-core:Triaged by cmars> <https://launchpad.net/bugs/1348477>
[02:08] <thumper> wallyworld_: I have a plan
[02:09] <wallyworld_> thumper: awesome, can we catch up in a sec, i'm otp withj axw
[02:27] <sinzui> wallyworld_, you are vindicated by the replay of the passing tarball
[02:28] <sinzui> wallyworld_, I am too tired to install the old packages. Maybe I shouldn't because I am not awake enough to know that this is stupid
[02:28] <wallyworld_> sinzui: \o/ does that mean we can remove the regression tag and unblock landings?
[02:28] <wallyworld_> sinzui: we do need to fix the compiler still
[02:28] <wallyworld_> dave can look at that on monday
[02:28] <sinzui> wallyworld_, I am going to take the tests voting rights away. if it starts passing, then we can assume the code or the compiler are in agreement and reatore the vore
[02:28] <sinzui> vote
[02:29] <wallyworld_> great,sounds good,
[02:29] <sinzui> I can do this now, and then add the real source for the bug
[02:29] <wallyworld_> sinzui: is there an eta then on landings being unblocked?
[02:30] <sinzui> wallyworld_, I will lower the priority of of the bug because obviously we cannot do anything now that it is out of our power...let me fix the vote first
[02:30] <wallyworld_> ok, thank you :-)
[02:31] <sinzui> oh, actually. I cannot go to sleep until this test completes
[02:31] <wallyworld_> :-(
[02:32] <sinzui> wallyworld_, on the other hand the apiserver.metrics might actually have problems. but without a safe compiler, we wont know
[02:32] <wallyworld_> thumper: did you want to talk about your plan?
[02:32] <thumper> wallyworld_: yeah, cause it isn't working
[02:33] <wallyworld_> ok, see you in onyx standup hangout?
[02:33] <thumper> ok
[03:24] <thumper> https://github.com/juju/juju/pull/683 anyone? refactoring work still from this week's mega branch being broken up
[03:24] <thumper> bug fix coming for auth failed
[03:26] <thumper> wallyworld_: https://github.com/juju/juju/pull/685
[03:26] <wallyworld_> looking
[03:35] <katco> wallyworld_: hey thanks for landing all my branches :) take away the fun part why don't ya!
[03:43] <katco> and now i'm off to bed. night all.
[03:51] <thumper> axw: could I get you to cast your eyes over https://github.com/juju/juju/pull/642 again?
[03:51] <axw> looking
[03:51] <thumper> axw: I've updated it based on recent changes and your suggestions
[03:54] <axw> thumper: line 20 can be dropped I think
[03:54] <thumper> sure
[03:54] <thumper> will do
[03:55] <thumper> and pushed
[03:56] <axw> thumper: reviewed, thank you
[03:56] <thumper> nm
[05:43] <axw> wallyworld_: https://github.com/axw/juju/compare/state-tools-take2 if you're interested in seeing the core changes
[05:43] <axw> fixing tests again now
[05:43] <wallyworld_> sure, looking
[05:44] <axw> wallyworld_: apiserver/common/tools.go and apiserver/tools.go are probably of most interest
[05:44] <axw> wallyworld_: also cmd/jujud/bootstrap.go
[05:44] <wallyworld_> kk, just got a phone call, will look soon
[06:14] <wallyworld_> axw: ToolsStorager NOOOOOOOOOOO
[06:15] <axw> heh
[06:15] <wallyworld_> not funny :)
[06:15] <axw> ToolsStorageProvider? it's really a very minor thing, I don't really care
[06:16] <axw> Getter is just as horrible to me
[06:16] <wallyworld_> is there already a "ToolStorage", can't recall
[06:16] <axw> yes, but this is a thing that has a ToolsStorage method
[06:17] <wallyworld_> otherwise ToolsStorageProvider
[06:17] <axw> ok
[06:17] <wallyworld_> sorry, i HATE that particular Go idiom
[07:08] <wallyworld_> fwereade: do you have a moment?
[07:23] <mattyw> morning all
[07:35] <TheMue> morning
[07:36] <axw> wallyworld_: did you find anything obviously wrong, apart from that name?
[07:36] <axw> morning TheMue
[07:36] <wallyworld_> axw: no, looked ok. i got distracted a bit by a bug report, let me just give it one more look
[07:37] <axw> no worries
[07:37] <axw> wallyworld_: doesn't need to be too deep, just wanted a glance over before I get too stuck into fixing tests
[07:37] <axw> which reminds me, tests
[07:38] <wallyworld_> axw: nothing jumped out, but i didn't go over the find in storage logic too closely
[07:39] <axw> ok
[07:39] <axw> thanks
[08:18] <wallyworld_> dimitern: hi there
[08:20] <dimitern> wallyworld_, hey
[08:20] <wallyworld_> dimitern: i backported your fix for allowing maas to disable network config to 1.20. the 1.20 branch is a little different to trunk. could you please review my back port? and type $$merge$$ if you are happy as i have to head to soccer https://github.com/juju/juju/pull/687
[08:20] <dimitern> wallyworld_, sure, looking
[08:21] <wallyworld_> thank you
[08:21]  * wallyworld_ heads out to soccer
[09:59] <TheMue> dimitern: heya, mind another look at https://github.com/juju/juju/pull/626 ?
[10:00] <TheMue> dimitern: it now also covers the simulation and testing of a V0 machiner API.
[10:00] <dimitern> TheMue, cheers, will have a look
[10:01] <TheMue> dimitern: great, thanks
[10:45] <TheMue> dimitern, voidspace: hangout?
[10:46] <voidspace> TheMue: omw
[11:29] <voidspace> dimitern: after changing TIME_WAIT I haven't seen the tests fail...
[11:30] <voidspace> dimitern: not conclusive, but they were failing regularly before
[11:30] <voidspace> dimitern: I'll go to 2MB rate limit (used to fail every time) and see if they now pass
[11:37] <dimitern> voidspace, good news then :)
[11:39] <voidspace> dimitern: ah no, fail :-/
[11:40] <dimitern> voidspace, too bad.. but hey, it's some progress at least
[11:55] <TheMue> so, back from lunch
[11:56] <TheMue> dimitern: thanks for review
[11:56] <TheMue> dimitern: only regarding the test for the providers I don't like to change
[11:57] <TheMue> dimitern: simply so that all providers, also future ones, always follow the same approach
[11:57] <dimitern> TheMue, well, I really don't like passing an opaque array of booleans
[11:57] <TheMue> dimitern: I recognized it as advantage in the moment I added the testing for the V0
[11:58] <TheMue> dimitern: and I don't like to do everything the same way but only ...
[11:58] <TheMue> dimitern: these exceptions always make it more difficult for later maintainers
[11:59] <TheMue> dimitern: but I could change it that I define the standard behavior as a const (ok, it's a var), so the tests read better
[11:59] <dimitern> TheMue, it will be difficult for anyone to see what [16]bool{true,true,true,false,false,...} actually means
[11:59] <dimitern> TheMue, that sounds better, yes
[11:59] <TheMue> var ExpectedStandardBehavior = [16]bool { ... }
[12:00] <dimitern> TheMue, btw why [16]bool and not []bool ?
[12:00] <TheMue> dimitern: OK, that's a compromise for me
[12:00] <TheMue> dimitern: hey, we all love Go for its type safety. so why open a door to pass to few or much values?
[12:01] <TheMue> dimitern: only to safe to chars?
[12:04] <dimitern> TheMue, ok, as long the [16]bool is hidden behind a var, I'm fine for the time being
[12:05] <gsamfira> hello folks. If anyone has some time, can I get a review on: https://github.com/juju/utils/pull/27/ ?
[12:07] <TheMue> dimitern: will hide it
[13:30] <perrito666> natefinch: fetching aurics, brt
[14:04] <perrito666> ericsnow: wwitzel3 do we?
[14:32] <voidspace> so I can confirm that CurrentStatus will report members in PrimaryState/SecondaryState even when primary renegotiation is happening and the replica set is unstable
[14:44] <voidspace> although it looks like it sets Uptime to 0 when that happens
[14:47] <voidspace> who wrote the replicaset code?
[14:47] <voidspace> It's part of juju not mgo
[15:10] <natefinch> voidspace: I wrote the replicaset code.
[15:11] <voidspace> natefinch: ok
[15:11] <voidspace> natefinch: I've butchered the applyRelSetConfig code
[15:11] <voidspace> natefinch: I don't think the loop inside that does quite what it looks like it does
[15:11] <voidspace> natefinch: however I've got rid of it anyway, so my question is now moot
[15:11] <natefinch> voidspace: heh ok
[15:12] <voidspace> natefinch: I have a new WaitForMajorityHealthy function which we can use to tell when the replica set is stable
[15:12] <voidspace> natefinch: so far it's mostly working - except for the times when it doesn't...
[15:12] <sinzui> alexisb, I am going to delay 1.21-alpha1 until Monday. There are too many changes to write up as release notes in a single day. I honestly don't know what features are in this release and how to explain to users who to use them
[15:12] <natefinch> voidspace: That was definitely not the finest code in the world.  I wish there were better ways to do pretty much everything in that code... mostly around querying mongo for "WTF are you doing right now?"
[15:12] <voidspace> natefinch: it's the fact that you change cmd to "Ping"
[15:12] <alexisb> sinzui, understood, no one is pinning for it today
[15:13] <voidspace> natefinch: which is only useful if you re-enter the block "if err == io.EOF"
[15:13] <alexisb> sinzui, you and I and Ian need to sync on release roadmap for 1.21 though
[15:13] <voidspace> natefinch: which almost certainly isn't what Ping returns
[15:13] <voidspace> natefinch: and even if Ping is successful we retry the loop instead of breaking
[15:13] <voidspace> natefinch: as there's no check for err == nil
[15:14] <voidspace> natefinch: if my function is reliable, it will look like this instead
[15:14] <voidspace> natefinch: http://pastebin.ubuntu.com/8260487/
[15:15] <natefinch> voidspace: hmm yeah that's not good. That  code has been tweaked by a lot of people who were trying to make it more reliable... it's quite possible there were some screw ups along the way.  A lot of it was trial and error trying to figure out what mongo will do at any particular time.
[15:15] <sinzui> alexisb, agreed
[15:16] <natefinch> voidspace: can you show me waitformajorityhealthy?  That's the key part that I had difficulty writing myself.
[15:17] <natefinch> voidspace: also, when does session.Run return EOF?  We should comment why that's an ok error to get
[15:17] <voidspace> natefinch: http://pastebin.ubuntu.com/8260518/
[15:17] <voidspace> natefinch: I should add back a comment about that
[15:17] <voidspace> natefinch: it's when changing the config causes primary re-negotiation so existing connections are dropped
[15:18] <voidspace> natefinch: it's fine - we just need to refresh
[15:18] <voidspace> natefinch: which WaitFor... does
[15:18] <voidspace> natefinch: this is currently not stable - I'm sometimes seeing WaitFor... timeout, so I need to add some debugging
[15:18] <voidspace> this is what I'm doing now
[15:18] <voidspace> it *mostly* works
[15:21] <natefinch> voidspace: thanks for putting in time on this, it'll make our code a lot more robust, and hopefully fix a lot of mongo related errors in the tests
[15:21] <voidspace> maybe... :-/
[15:21] <voidspace> it's been dead end after dead end so far
[15:22] <voidspace> this looks really promising, but I'm still seeing timeouts
[15:35] <natefinch> sinzui: is amazon sick today?  one of my PR's failed in a weird way: http://juju-ci.vapour.ws:8080/job/github-merge-juju/546/console
[15:37] <sinzui> natefinch, that indeed looks like aws failed to provide an instance
[15:38] <sinzui> natefinch, I saw messages yesterday that clearly states there weren't any instances of the size requested for the AZ :(
[15:39] <natefinch> sinzui: I suppose AWS could just be busy
[15:55] <perrito666> mattyw: hey, are you around?
[16:00] <mattyw> perrito666, yep
[16:07] <hazmat> sinzui, that's a bug imo, juju should recover and try a different az
[16:08] <hazmat> although that's different then what natefinch build says
[16:09] <perrito666> mattyw: did you see axw's last pr?
[16:10] <mattyw> perrito666, removing the call to setadminmongopassword?
[16:10] <perrito666> yup, I applied that and ran with and without your patch
[16:11] <perrito666> that seems to at least fix half of the erorrs yet the error related to presence is still there
[16:11] <mattyw> perrito666, my patch?
[16:11] <perrito666> http://paste.ubuntu.com/8227111/
[16:11] <perrito666> "patch"
[16:12] <mattyw> perrito666, does axw branch make use of the change that thumper landed overnight?
[16:12] <perrito666> yes
[16:12] <perrito666> https://github.com/juju/juju/pull/688
[16:43] <natefinch> how the hell are you supposed to use juju run?  I can't for the life of me figure out how to get it to do anything but say "unrecognized args <stuff in the command to run>"
[16:43] <wesleymason> natefinch: juju run --service <servicename> 'comand here'
[16:43] <wesleymason> for example
[16:44] <natefinch> in quotes?
[16:44] <wesleymason> yeah, in single quotes so bash/zsh etc. doesn't interpolate first
[16:44] <wesleymason> recommended anyway
[16:45] <natefinch> ahh that was it.  I was trying with -- to keep it from parsing flags.... we really need better help on that command
[16:46] <natefinch> or like ONE example would be nice
[16:46] <wesleymason> +1
[16:47] <natefinch> I'll work on that.  bad help is a pet peeve of mine
[16:56] <voidspace> natefinch: do you know how to debug "no reachable servers" errors?
[16:57] <natefinch> voidspace: when initiating the replicaset?
[16:57] <voidspace> natefinch: no, after applying a config change or during a Dial
[16:57] <voidspace> natefinch: but in both cases I have a replicaset with several members
[16:59] <natefinch> voidspace: either they all still trying to come up, or the addresses are internal to the cloud, not public...
[16:59] <voidspace> natefinch: it's during tests, so not a cloud issue
[16:59] <voidspace> natefinch: and I'd like to know *how* to tell whether or not they're trying to come up
[16:59] <natefinch> voidspace: I wish I knew
[16:59] <voidspace> natefinch: as I've waited five minutes and CurrentStatus is failing
[16:59] <voidspace> because of the connection error
[16:59] <voidspace> hah, right
[16:59] <natefinch> niemeyer: ^^
[17:01] <natefinch> niemeyer: we're trying to make our code more robust with respect to Mongo, especially when initiating a replicaset and when bringing up instances of mongo during testing.  We get what appear to be random failures where sometimes they either never come up or take a really long time, or initiating takes a really long time.   Part of the problem is that we don't really now how to figure out what state mongo is in... all we
[17:01] <natefinch> can do is dial and see if it responds within a timeout.  Is there some better way we can do this?
[17:02] <voidspace> I'm seeing a lot of errors like:
[17:02] <voidspace> [LOG] 6:43.772 DEBUG juju.testing tls.Dial(<IP_ADDRESS>:35846) failed with dial tcp <IP_ADDRESS>:35846: connection refused
[17:02] <voidspace> Even with session.Refresh() and waiting for (up to) five minutes
[17:03] <niemeyer> natefinch: Yes, you can always ask the server for its status
[17:03] <voidspace> niemeyer: how specifically?
[17:03] <voidspace> calling CurrentStatus(session) is failing with connection refused
[17:04] <niemeyer> voidspace: http://docs.mongodb.org/manual/reference/command/replSetGetStatus/
[17:04] <voidspace> niemeyer: that's precisely what CurrentStatus is doing
[17:04] <niemeyer> voidspace: If the connection is refused, you know the status :)
[17:04] <voidspace> niemeyer: any idea *why* sometimes our connections die like that and just don't come back
[17:04] <niemeyer> voidspace: Okay, that's not what Nate said above
[17:04] <niemeyer> voidspace: Hmm
[17:05] <niemeyer> voidspace: Die with connection refused?
[17:05] <voidspace> [LOG] 6:43.772 DEBUG juju.testing tls.Dial(<IP_ADDRESS>:35846) failed with dial tcp <IP_ADDRESS>:35846: connection refused
[17:05] <natefinch> niemeyer: sorry... what I mean is - we tell it to initiate... and then can never get it to respond
[17:05] <niemeyer> voidspace: The TCP port is not open..
[17:06] <niemeyer> natefinch: Look at the logs
[17:06] <niemeyer> natefinch: I've never seen anything similar before
[17:06] <niemeyer> natefinch: the test suite of mgo routinely shoot servers down and bring them back up
[17:06] <natefinch> niemeyer: it's the single most common failure for our tests - mongo going away and never coming back
[17:07] <natefinch> niemeyer: it's quite likely we're just doing something wrong, we just don't know what that is.
[17:07] <niemeyer> natefinch: That makes no sense.. a connection refusal is a TCP port not open, which in general means MongoDB is not even running
[17:07] <niemeyer> natefinch: I'd look at the logs to see why
[17:07] <natefinch> niemeyer: it's not always connection refusal... that's the problem this time, often times the dial will just time out eventually
[17:08] <niemeyer> natefinch: Heh..
[17:08] <voidspace> that particular failure was during a call to instance.MustDialDirect() - *after* waiting for CurrentStatus to report all members up
[17:08] <niemeyer> natefinch: First thing to do is make up your mind about what the symptom is :)
[17:08] <voidspace> well, I just did another test run and got the same symptom
[17:08] <voidspace> [LOG] 6:43.764 DEBUG juju.testing tls.Dial(<IP_ADDRESS>:37222) failed with dial tcp <IP_ADDRESS>:37222: connection refused
[17:09] <niemeyer> Yeah, that's a server down.. the logs will say why
[17:10] <natefinch> voidspace: I think you'll need to hack the code a little to prevent gocheck from cleaning up the mongo directory, so you can look at the logs
[17:10] <voidspace> niemeyer: do you know where the logs should be? I've got a horrible feeling we redirect mongo logging somewhere useless.
[17:10] <voidspace> natefinch: ah, right
[17:10] <voidspace> natefinch: when we start mongo don't we get it to log to standard out so we can parse the logs...
[17:10] <voidspace> natefinch: meaning we get no logs
[17:11] <voidspace> natefinch: or does it log to the directory as well?
[17:11] <niemeyer> voidspace, natefinch: -check.work will prevent it from being removed, and display it as well
[17:11] <voidspace> niemeyer: cool, thanks
[17:11] <natefinch> niemeyer: oh, awesome, thanks
[17:11] <niemeyer> voidspace: But I don't know where the logs are being sent to
[17:11] <natefinch> voidspace: I'm pretty sure mongo's logs are still written to disk, but I honestly don't remember
[17:15] <voidspace> we're still using the launchpad version of gocheck of course
[17:15] <voidspace> wasn't there a thread about that?
[17:16] <voidspace> yeah, looks like we're about to update
[17:18] <natefinch> niemeyer: is that check.work flag available on launchpad's gocheck?  I can't find docs on the flags it takes
[17:18] <niemeyer> natefinch: -gocheck.work, likely
[17:19] <niemeyer> natefinch: -help on the test binary, or just passing a wrong flag, will print the options
[17:19] <voidspace> I don't think it is available
[17:19] <voidspace> we're at the latest revision of launchpad
[17:22] <voidspace> natefinch: copying the gopkg.in one over the top of the launchpad one seems to work though
[17:22] <voidspace> :-p
[17:33] <natefinch> voidspace: heh, we're lucky we always rename the package, otherwise that wouldn't work
[17:35] <voidspace> right
[17:35] <wwitzel3> woo, I have passing tests!
[17:39] <natefinch> anyone know why I'd get "cannot open ports 80-80/tcp on machine 5 due to conflict" when I re-ran my install hook?  Shouldn't open-port be idempotent?
[17:44] <gsamfira> natefinch: there was a discussion on the mailing list about this a while back. Subject was "Port ranges - restricting opening and closing ranges". Not sure of the conclusion on that though
[17:45] <gsamfira> https://lists.ubuntu.com/archives/juju-dev/2014-August/003131.html
[21:57] <perrito666> anyone knows the difference between using net.Listen("tcp", "localhost:0") and net.Listen("tcp", ":0") ?


[00:28] <Sarvatt> bryceh: i think i found your dream video card - http://www.engadget.com/2010/04/30/powercolor-hd5970-eyefinity-12-makes-six-screens-yesterdays-new/
[00:28] <Sarvatt> crossfire 2 of those bad boys :)
[00:36] <RAOF> How many non-DP monitors can it drive?
[00:44] <andrewaclt> RAOF, the issue I'm having is worse in the vanilla kernel, rather than unfreezing after 2-4 minutes, it stays frozen...left it for 30 minutes before rebooting.
[00:44] <RAOF> Ok.
[00:52] <RAOF> andrewaclt: I'd suggest popping up in #intel-gfx.
[00:53] <andrewaclt> I'm more than happy to debug it and try and write a patch for it, but I'm over my head when it comes to xorg/gfx drivers.
[00:53] <andrewaclt> RAOF, thanks
[00:58] <RAOF> ickle's the man to talk to.
[01:02] <andrewaclt> thanks
[01:02] <ScottK> RAOF: I just accepted a kdebase-workpace upload the has some kwin fixes in it.  They are limited to fixes inside the blur effect, so I don't think it will have any impact on the changes you all have been pursuing, but FYI.
[01:02] <RAOF> Ta.
[01:04] <RAOF> ScottK: Want to test the new -intel that doesn't segfault on logout and actually works when you log in the second time?
[01:05] <ScottK> RAOF: Yes, but probably not until my morning.
[01:05] <RAOF> Cool.
[03:05] <nigelb> g32
[07:33] <RAOF> Anyone feel like some mesa & intel DDX sponsorship?
[08:11] <mvo> RAOF: where is the diff?
[08:11] <mvo> RAOF: I can do that
[08:11] <RAOF> mvo: http://cooperteam.net/Packages/
[08:17] <mvo> RAOF: looking now
[08:17] <RAOF> Thanks muchly!
[08:29] <mvo> RAOF: that "e30f0338" is directly applied and not a quilt patch is intentional, right? because you maintain the thing full-source in git?
[08:29] <RAOF> mvo: Right.  That's the pkg-xorg way.
[08:29] <mvo> ok, thanks. its fine, I upload now
[11:06] <tseliot> mvo: are you migrating nvidia-173 and -96 to nouveau on dist-upgrades? Or are you only doing it for -71?
[11:08] <mvo> tseliot: it will only migrate if no /usr/lib/xorg/modules/drivers/nvidia_drv.so is there anymore after the upgrade
[11:13] <tseliot> mvo: that would work, as long as nvidia-173 and nvidia-96 are removed by the upgrade as they won't provide xserver-xorg-video-8. Can you confirm that this is what happens?
[11:15] <mvo> tseliot: I can run a test upgrade with them, sure
[11:16] <tseliot> mvo: that would really help. I just want to make sure that X is not removed
[14:27] <OwaisL> hello X people... what is the state of Intel GM 965 and clutter? Anyone know?
[14:28] <Sarvatt> that should be the most supported chipset with regards to clutter, but it is clutter.. :)
[14:29] <OwaisL> so the bug is in clutter not the drivers..?
[14:29] <Sarvatt> i'm not aware of any issues with that combination at the moment
[14:29] <Sarvatt> what bug?
[14:30] <OwaisL> well, mutter apps don't run on 965
[14:30] <OwaisL> Unity, Shell
[14:30] <OwaisL> they work if CLUTTER_VBLANK=none
[14:30] <Sarvatt> do you have a bug on launchpad?
[14:30] <OwaisL> i think yes
[14:31] <OwaisL> it's almost a year old
[14:31] <OwaisL> there is another issue on Unity
[14:31] <OwaisL> with 965
[14:31] <Sarvatt> what distro? because 965 is well tested on current maverick
[14:31] <Sarvatt> err what release rather
[14:31] <OwaisL> the launcher crashes on a mouseover... but someone is working on it
[14:31] <OwaisL> I'm on maverick.. with lastest updates
[14:32] <Sarvatt> its most likely something odd with your specific setup, can you file a new bug on launchpad with ubuntu-bug xorg so we can see your logs?
[14:34] <Sarvatt> the launcher crashing on mouseover is most likely a unity bug, but needing CLUTTER_VBLANK=none to start shouldn't still be happening at the moment..
[14:34] <OwaisL> I don't think so.. I think lots of people have it.. I'll look for a bug on LP and file it if it is not there
[14:44] <Sarvatt> RAOF: looks like your KDE problem has been around for a long time now.. http://ubuntuforums.org/showthread.php?p=9517771
[14:50] <OwaisL> Sarvatt: while file should i attach? Xorg.log ?
[14:50] <Sarvatt> OwaisL: if you just run ubuntu-bug xorg in a terminal it'll attach everything for you
[14:50] <OwaisL> ok
[14:51] <OwaisL> Sarvatt: Your gdm log files may help developers diagnose the bug, but may contain sensitive information.
[14:51] <OwaisL> what could it contain
[14:51] <OwaisL> my login pass at most
[14:51] <OwaisL> right?
[14:52] <Sarvatt> no, thats not really true anymore, it contains your /var/log/Xorg.0.log as well as any console error output for GDM
[14:52] <Sarvatt> it used to have ~/.xsession-errors that could have like the names of the mp3 files you were playing in it
[14:52] <OwaisL> ahh
[14:53] <OwaisL> not a problem
[14:53] <Sarvatt> those logs are really helpful though, they have a deeper history of previous xorg.0.log's than x stores
[14:59] <OwaisL> Sarvatt: here you go https://bugs.launchpad.net/ubuntu/+source/xorg/+bug/648928
[14:59] <ubot4> Launchpad bug 648928 in xorg (Ubuntu) "Cannot use Unity or Shell with Intel 965GM (affects: 1) (heat: 6)" [Undecided,New]
[15:04] <Sarvatt> thanks OwaisL
[15:10] <Sarvatt> chrisccoulson: you're on gm965 right? are you having any problems starting unity with current maverick?
[15:12] <chrisccoulson> Sarvatt, i've not tried it today
[15:12] <chrisccoulson> one second
[15:14] <Sarvatt> looks like https://bugs.edge.launchpad.net/ubuntu/+source/mutter/+bug/648625 is similar for the right click crash, people want to blame mesa but thats not mesa..
[15:14] <ubot4> Launchpad bug 648625 in unity (Ubuntu) (and 2 other projects) "mutter crashed with SIGSEGV in g_strconcat() (affects: 1) (heat: 12)" [Undecided,New]
[15:18] <chrisccoulson> Sarvatt, yeah, unity just crashes here
[15:24] <Azelphur> Hi, someone told me it's possible to join 2 Twinview sessions together with xrandr to create one big screen without xinerama, is this possible? :)
[15:37] <victorp> RAOF - been told by tseliot that you are the person to ask about a monitor problem I am having with 10.4
[15:39] <tseliot> or Sarvatt (RAOF lives in Australia, therefore I think it's a little late there)
[15:39] <victorp> ah ok!
[15:40] <victorp> no ofense meant Sarvatt :)
[15:41] <victorp> back in a sec
[15:43] <jcristau> might be useful to say what the problem is rather than hope people will guess
[15:45] <tseliot> he has a dell vostro 3300 with intel graphics, running 10.4 - and connected to an external monitor (via VGA) the external monitor is flickering/wavering constantly
[15:45] <tseliot> I thought of a pipe underrun but there's no trace in the logs
[15:46] <tseliot> dmesg: http://pastebin.com/nrNe7rQS
[15:46] <tseliot> Xorg.0.log: http://pastebin.com/vZ3LNVvs
[15:46] <jcristau> i don't think you'd get anything in the log if you don't enable some sort of debug option
[15:47] <jcristau> even if it was pipe underrun
[15:47] <victorp> back now
[15:49] <tseliot> victorp: I have explained what kind of problem you are experiencing
[15:50] <tseliot> jcristau: I thought he would have had something like this in the log: (EE) intel(0): underrun on pipe A!
[15:51] <jcristau> tseliot: not on kms
[15:52] <tseliot> jcristau: aah, right, kms. What level of drm debugging do you suggest?
[15:53] <tseliot> 1 may be a bit excessive
[15:54] <Sarvatt> 4 would have the equivalent
[15:55] <tseliot> victorp: can you type the following command, please?
[15:55] <tseliot> echo 4 | sudo tee /sys/module/drm/parameters/debug
[15:55] <tseliot> as Sarvatt suggested
[15:55] <tseliot> then "echo 0 | sudo tee /sys/module/drm/parameters/debug"
[15:56] <Sarvatt> trying to find any bugs with the same issue
[15:57] <victorp> tseliot - done, and now?
[15:57] <tseliot> victorp: please pastebin your dmesg again
[15:57] <tseliot> victorp: it should be more verbose now
[15:58] <victorp> ok - so restart first then :)
[15:58] <tseliot> victorp: why?
[15:59] <victorp> tseliot - ok
[15:59] <tseliot> victorp: it should be fine as long as you did as I told you while the external monitor was connected and flickering
[15:59] <tseliot> I mean, the current dmesg
[16:00] <tseliot> (without rebooting)
[16:00] <victorp> http://pastebin.com/bQBZeU9b
[16:00] <victorp> flickering all the time
[16:00] <victorp> non-stop wave
[16:05] <tseliot> victorp: maybe disable the external screen, then "echo 4 | sudo tee /sys/module/drm/parameters/debug", enable the external screen and let it flicker and finally "echo 0 | sudo tee /sys/module/drm/parameters/debug" and post your dmesg again
[16:06] <Sarvatt> victorp: can you try with a newer kernel to be sure the problem isn't fixed upstream already and help narrow it down if so? http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.36-rc5-maverick/
[16:07] <Sarvatt> found some upstream bugs - https://bugs.freedesktop.org/show_bug.cgi?id=28306
[16:07] <ubot4> Freedesktop bug 28306 in DRM/Intel "[Arrandale] VGA output buggy (wavering/out-of-phase-ness)" [Major,New]
[16:08] <victorp> Sarvatt - I am running 10.4 at the moment but I will upgrade to maverik today - then I can also try rc5
[16:08] <tseliot> I think you can try that kernel in lucid too
[16:09] <victorp> cant see it in the mainline
[16:09] <victorp> you mean just use the same kernel build in lucid?
[16:09] <Sarvatt> https://bugs.freedesktop.org/show_bug.cgi?id=28858
[16:09] <ubot4> Freedesktop bug 28858 in Driver/intel "[Arrandale] external monitor flicker" [Normal,Resolved: duplicate]
[16:10] <Sarvatt> yeah you can just install the kernel there on lucid, the maverick part means its using the maverick kernel config options
[16:10] <victorp> ok - I will try that now
[16:35] <Sarvatt> victorp: Hmm, can you try running at a 75hz refresh rate resolution to see if it goes away?
[16:35] <Sarvatt> oh he's gone
[16:47] <victorp> sarvatt I tried the kernel you suggested and also Maverick beta from a CD still have the continous flicker
[16:48] <Sarvatt> victorp: do you have any 75hz+ refresh rate resolutions available on the vga output? can you see if they work properly?
[16:50] <victorp> they dont on 2.6.32  - would need to try 2.6.36 would that make a difference?
[16:50] <Sarvatt> nah thats ok, saw someone in another bug saying 75hz+ worked fine on the lucid kernel so its not the same at least
[16:52] <victorp> seem that 28858 was closed as a duplicate of https://bugs.freedesktop.org/show_bug.cgi?id=28306
[16:52] <ubot4> Freedesktop bug 28306 in DRM/Intel "[Arrandale] VGA output buggy (wavering/out-of-phase-ness)" [Major,New]
[16:52] <victorp> which seems still open?
[16:54] <Sarvatt> yeah dupe isn't really closed, its still tracked on the open bug
[16:55] <victorp> any thought? it seems that the bug has plenty of logs already
[16:58] <Sarvatt> i'm at a loss, asking over in #intel-gfx, you may want to join there in case someone responds
[17:13] <bryceh> Sarvatt, good god man
[17:13] <bryceh> Sarvatt, 12 monitors?  drool...
[17:24] <victorp> Sarvatt - any luck?
[17:31] <Sarvatt> no nothing yet, I'm sorry
[17:32] <victorp> no worries, let me know if something comes up
[18:46] <Sarvatt> would anyone be willing to take a look at https://bugs.edge.launchpad.net/ubuntu/+source/compiz/+bug/649137 to see if it is sane?
[18:46] <ubot4> Launchpad bug 649137 in compiz (Ubuntu) "Compiz Sandybridge blacklist is no longer needed. (affects: 1) (heat: 6)" [Undecided,New]
[18:47] <Sarvatt> sorry for all of the compiz uploads, we didn't decide to just disable DRI completely on them until recently in hopes it would be working in 7.9 like we were told
[19:05] <bryceh> Sarvatt, what got blacklisted?
[19:07] <Sarvatt> all sandybridge GPU's
[19:08] <Sarvatt> i just blacklisted them in compiz so the desktop would come up but that only fixed ubuntu-desktop, intel was saying 3D would be working for the Q3 release (mesa 7.9) but it isn't going to happen so we disabled DRI completely on them
[19:13] <Sarvatt> mesa 7.9 rc1 released
[19:14] <Sarvatt> final due october 4th


[21:42] <cjwatson> stgraber: queuebot seems to have gone silent again
[21:42] <cjwatson> should've been some NEW activity
[21:44] <stgraber> that should fix it (had some network problems which probably made the LP plugin hang)
[21:45] <cjwatson> ta