| --- Log opened Wed Feb 18 11:02:11 2009 | ||
| wwoods | viking_ice? adamw? anyone else? | 11:02 |
|---|---|---|
| -!- adamw [n=AdamW@redhat/adamw] has joined #fedora-meeting | 11:02 | |
| -!- warren [n=warren@redhat/wombat/warren] has joined #fedora-meeting | 11:02 | |
| wwoods | so, hi | 11:03 |
| * viking_ice * | 11:03 | |
| adamw | hi | 11:03 |
| wwoods | waiting for evolution to start back up so I can check if there's anything to add to the agenda | 11:03 |
| -!- sdziallas [n=sebastia@p57A2B810.dip.t-dialin.net] has quit ["Ex-Chat"] | 11:04 | |
| -!- MostafaDaneshvar [n=MostafaD@unaffiliated/mostafadaneshvar] has joined #fedora-meeting | 11:04 | |
| -!- nicubunu [n=nicubunu@mail.apsro.com] has left #fedora-meeting ["Fading into sunset"] | 11:04 | |
| -!- sdziallas [n=sebastia@p57A2B810.dip.t-dialin.net] has joined #fedora-meeting | 11:04 | |
| wwoods | ah, okay | 11:06 |
| wwoods | so yes | 11:06 |
| wwoods | first, let's talk about autoqa for a bit! | 11:06 |
| -!- wwoods changed the topic of #fedora-meeting to: Fedora QA Meeting | autoqa | 11:06 | |
| wwoods | The git repo for this is: http://git.fedorahosted.org/git/?p=autoqa.git;a=summary | 11:07 |
| -!- MostafaDaneshvar [n=MostafaD@unaffiliated/mostafadaneshvar] has left #fedora-meeting [] | 11:08 | |
| -!- Matias_Arg [n=root@host169.200-45-216.telecom.net.ar] has joined #fedora-meeting | 11:08 | |
| wwoods | in short, it's a set of a) things that watch for events, b) tests to run when certain things happen, and c) a harness that handles running the tests in response to the event | 11:08 |
| -!- Matias_Arg [n=root@host169.200-45-216.telecom.net.ar] has left #fedora-meeting [] | 11:08 | |
| wwoods | for things like repos updating, new builds in koji, new updates in bodhi, etc. | 11:08 |
| wwoods | last night we got our very first test running | 11:09 |
| viking_ice | yea :) | 11:09 |
| f13 | https://fedoraproject.org/wiki/Automated_QA_Testing_Project is the wiki page covering the project | 11:09 |
| wwoods | f13 wrote code that watches for repo changes | 11:09 |
| wwoods | (post-repo-update/watch-repos.py) | 11:09 |
| wwoods | and I wrote a simple test that runs repoclosure on the given repo(s) | 11:10 |
| wwoods | it's running on a host in the fedora infrastructure and currently just sends mails to the qa-admin group | 11:10 |
| wwoods | currently it tries to send test results and logs to a results server | 11:10 |
| wwoods | we've basically stripped that out and told it to just email us the raw stdout/stderr | 11:11 |
| wwoods | in the near future we want to create an actual results server.. and start sending results to it | 11:11 |
| -!- Gaaruto [n=Gaaruto@fedora/Gaaruto] has quit ["++"] | 11:11 | |
| -!- bo09 [n=bo09@nat/redhat/x-0363b9a26b15aec2] has joined #fedora-meeting | 11:11 | |
| -!- Gaaruto [n=Gaaruto@fedora/Gaaruto] has joined #fedora-meeting | 11:11 | |
| wwoods | we'll also need more watchers, more hooks, and more tests | 11:11 |
| -!- Gaaruto [n=Gaaruto@fedora/Gaaruto] has quit [Client Quit] | 11:11 | |
| wwoods | but first we need to get the harness working properly, and make sure we're happy with how tests are written | 11:12 |
| f13 | with a results server we can do more logical things, like compare results from previous runs, autofile bugs, update websites and send email, etc... | 11:12 |
| -!- Gaaruto [n=Gaaruto@fedora/Gaaruto] has joined #fedora-meeting | 11:12 | |
| f13 | for now, we're just doing low tech fire off an email | 11:12 |
| f13 | which is good, because it's results where none have been before, so it's absolutely an improvement | 11:13 |
| viking_ice | wwoods: regarding more tests you can see if we can use any from the ltp.sf.net ( download for rhel5 and check the tests folder ) | 11:13 |
| adamw | how are the results looking? | 11:13 |
| wwoods | ugh, ltp | 11:13 |
| viking_ice | which could spare us some writing | 11:13 |
| viking_ice | ah ok | 11:13 |
| wwoods | we've packaged up the ltp in the past | 11:13 |
| wwoods | and tried to cram it into the RHTS framework | 11:13 |
| wwoods | it's a horrible pain | 11:13 |
| viking_ice | that bad.. | 11:14 |
| wwoods | and those are generic linux system tests | 11:14 |
| wwoods | we don't care about that yet | 11:14 |
| jlaska | viking_ice: according to tehe project page ... the focus now seems to be on validating the rawhide bits (repos + trees) | 11:14 |
| -!- valente [n=vali@89.121.200.106] has quit ["Leaving."] | 11:14 | |
| wwoods | we care about specifically testing things like package sanity and repoclosure and whether kernel images / metadata are correct | 11:14 |
| viking_ice | jlaska: yes just for da future I usually think a bit ahead | 11:14 |
| -!- nicubunu [n=nicubunu@mail.apsro.com] has joined #fedora-meeting | 11:14 | |
| wwoods | we'll get around to doing functional system testing eventually | 11:14 |
| wwoods | but we're a long way from that | 11:14 |
| wwoods | we *might* have a test that just runs the LTP and reports its results | 11:15 |
| jlaska | viking_ice: right on | 11:15 |
| viking_ice | wwoods: that does not prevent us from keeping ears and eyes open :) | 11:15 |
| jlaska | f13: wwoods: would it capture things like this ... http://kojipkgs.fedoraproject.org/mash/rawhide-20090218/logs/i386.log (see last 2 lines)? | 11:15 |
| wwoods | what would it do if it did? | 11:16 |
| jlaska | note a failure | 11:16 |
| wwoods | where? | 11:16 |
| jlaska | missing images | 11:16 |
| viking_ice | wwoods: if our testing suite can be used between distro we should ( when we are at that point ) perhaps reach out for collaboration between distro's in writing test cases more hands make an hard work an easy task | 11:16 |
| jlaska | due to the failure noted in the logs above | 11:16 |
| wwoods | no, I mean, where would it note the failure | 11:16 |
| jlaska | /tmp/buildinstall.tree.tn9wcm/mk-images: line 313: syntax error near unexpected token `)' | 11:16 |
| jlaska | /tmp/buildinstall.tree.tn9wcm/mk-images: line 313: ` atmel)' | 11:16 |
| wwoods | right now the only thing the system is capable of is sending email | 11:16 |
| f13 | jlaska: no, that's a functional testing error, not a broken dep | 11:16 |
| jlaska | wwoods: where does it note failures now? | 11:16 |
| jlaska | f13: gotcha, thx | 11:17 |
| wwoods | so it sends an email that says "pass" or "fail" near the bottom | 11:17 |
| -!- valente [n=vali@89.121.200.106] has joined #fedora-meeting | 11:17 | |
| jlaska | f13: does it make sense to have a future test for the presence of installation images? | 11:17 |
| f13 | however if I write a hook to look at post-tree-compose, it would notice that the tree was missing images and error the test. | 11:17 |
| wwoods | absolutely | 11:17 |
| f13 | jlaska: it wouldn't make sense in the post-repo-update case, but in post-tree-compose case | 11:17 |
| -!- gregdek [n=gdk@cpe-024-088-244-139.nc.res.rr.com] has quit [Read error: 110 (Connection timed out)] | 11:17 | |
| jlaska | gotcha, great | 11:17 |
| wwoods | but yeah, we need a new watcher for that | 11:17 |
| wwoods | and we can make a test that checks for the *result* of that error | 11:18 |
| wwoods | i.e. no images | 11:18 |
| f13 | which I think we already have in verify-tree | 11:18 |
| f13 | I think our next watcher/test combo will likely be post-tree-compose + verify-tree | 11:18 |
| wwoods | but still, until we have a result server, the only output is email | 11:18 |
| f13 | which is better than nothing (: | 11:19 |
| viking_ice | true | 11:19 |
| jlaska | do you guys have a priority in mind for future tests+watchers? | 11:19 |
| wwoods | (not strictly true - we actually output a directory of logs and test results) | 11:19 |
| f13 | jlaska: not particularly | 11:19 |
| wwoods | (but the only thing that actually gets sent off the machine is output.log) | 11:19 |
| f13 | jlaska: the first priority was $SOMETHING | 11:19 |
| f13 | jlaska: after that... | 11:19 |
| wwoods | highest priority is "whatever is useful and we already have code written for" | 11:20 |
| jlaska | f13: heh, no doubt. Nice work to you guys for getting a ball rolling :) | 11:20 |
| viking_ice | wwoods: what's the workflow after the mail has been sent and it's a failure (: | 11:20 |
| wwoods | just so we can knock the harness and watcher and test design into shape | 11:20 |
| wwoods | viking_ice: I dunno. fix it? | 11:20 |
| wwoods | we'll worry about what to *do* with the results once we actually *have* results | 11:20 |
| wwoods | right now we don't even *test* these things | 11:20 |
| wwoods | so just getting notification is an improvement | 11:20 |
| viking_ice | ah ok | 11:20 |
| f13 | I think initially we need a autoqa-results@lists.fedoraproject.org to dump the results | 11:21 |
| f13 | since they can be numerous | 11:21 |
| wwoods | f13: agreed | 11:21 |
| f13 | as we tweak things to be less annoying | 11:21 |
| wwoods | plus that gives us a public archive of results | 11:21 |
| jlaska | f13: wwoods shouldwe just setup a trac and mailing list for the hosted autoqa project? | 11:21 |
| -!- mcepl [n=mcepl@49-117-207-85.strcechy.adsl-llu.static.bluetone.cz] has left #fedora-meeting [] | 11:21 | |
| wwoods | jlaska: probably? seems like we could just use the existing fedora-qa trac | 11:21 |
| f13 | but it will give us a public view into what's going on and help us poke other people in the eye to fix things | 11:21 |
| wwoods | and just add an autoqa component | 11:21 |
| f13 | seems right to me | 11:22 |
| jlaska | no skin in the game ... but, I'd vote for now to just keep it self-contained | 11:23 |
| jlaska | but whatever folks feel is best, I create compoents etc... | 11:23 |
| viking_ice | fedora-qa sounds good to me | 11:23 |
| wwoods | I don't want to be admin on yet another trac instance | 11:23 |
| wwoods | and since this stuff is almost specifically only useful for fedora-qa | 11:24 |
| wwoods | it just makes sense to use that | 11:24 |
| jlaska | I can see that ... okay | 11:24 |
| -!- Parafeed [n=clive@abbey1.objectsoft-systems.ltd.uk] has joined #fedora-meeting | 11:24 | |
| viking_ice | +1 | 11:24 |
| jlaska | should we move all the code into fedora-qa too? | 11:25 |
| wwoods | not sure about that | 11:25 |
| wwoods | gotta figure out how/if we can migrate it with its history intact | 11:25 |
| wwoods | or if we can just have multiple git repos in one trac instance, or what | 11:25 |
| wwoods | I'll look into it | 11:25 |
| wwoods | we might just use it for ticketing/mailing list stuff | 11:26 |
| f13 | the only thing that ties git repos into trac is for the browser, which gitweb/cgit is better for anyway | 11:26 |
| wwoods | right | 11:26 |
| f13 | so I wouldn't worry about multiple git repos being associated with a single trac instance, you just list them somewhere in the trac wiki | 11:26 |
| f13 | the trac git plugin is flakey anyway | 11:27 |
| wwoods | and mailing lists aren't handled by trac, right? | 11:27 |
| * wwoods adds 'autoqa' component to fedora-qa trac | 11:27 | |
| f13 | wwoods: that is correct | 11:28 |
| jlaska | wwoods: ah, thx ... beat me to it | 11:28 |
| f13 | trac has no concept of mailing lists | 11:28 |
| wwoods | so, yeah, all we're really doing here is adding a component | 11:28 |
| jlaska | I'll get a mailing list request in | 11:28 |
| wwoods | don't need to mess with the repo | 11:28 |
| wwoods | we can use the main fedoraproject wiki | 11:28 |
| viking_ice | hum cant we just resurrect the old one ( fedora-qa ) | 11:28 |
| wwoods | definitely not | 11:28 |
| viking_ice | ok | 11:28 |
| wwoods | that was a user-oriented list | 11:28 |
| viking_ice | ah ok | 11:29 |
| wwoods | people will be really unhappy if I start sending them 20 automated mails a day | 11:29 |
| wwoods | because they subscribed to fedora-qa in 2006 | 11:29 |
| viking_ice | true true | 11:29 |
| jlaska | okay so ... | 11:29 |
| jlaska | * create a autoqa component in fedora-qa trac instance | 11:29 |
| f13 | yeah, right now the post-repo-update sends one mail per repo per arch | 11:29 |
| jlaska | * send out infrastructure mailing list request for fedora-qa | 11:29 |
| jlaska | wwoods: you got the first? | 11:30 |
| poelcat | why can't you just flush the subscribers and repurpose the list? we did that for fedora-triage list | 11:30 |
| wwoods | jlaska: yep, done | 11:30 |
| * jlaska holds | 11:30 | |
| wwoods | poelcat: because "fedora-qa" is not a good name | 11:30 |
| viking_ice | good point | 11:30 |
| f13 | and fedora-qa is likely @redhat.com where we need to be creating lists at @fedoraproject.org or @fedorahosted.org | 11:30 |
| wwoods | we're talking about a list to send at least 3, possibly up to 20+ emails a day | 11:30 |
| viking_ice | fedora-testsuit ? | 11:31 |
| jlaska | fedora-qa would be the name we're going to use on hosted | 11:31 |
| wwoods | which are just logs of automatic test results | 11:31 |
| jlaska | let's just use autoqa ...wwoods you can set me as the track admin if you want | 11:31 |
| jlaska | if nothing else, it keeps it all in one place | 11:31 |
| -!- dash123 [i=d2d43dfb@gateway/web/ajax/mibbit.com/x-c3d3b33c58cdac14] has joined #fedora-meeting | 11:31 | |
| wwoods | jlaska: hm? I already created the trac component and made myself owner | 11:31 |
| jlaska | right, I mean for the mailing list | 11:32 |
| f13 | autoqa-results@ | 11:32 |
| wwoods | yeah, what f13 said | 11:32 |
| jlaska | no objections here | 11:32 |
| f13 | basically the only reason we're creating a list is just to dump results | 11:32 |
| f13 | its not a discussion list | 11:33 |
| jlaska | right | 11:33 |
| f13 | that can continue to happen on fedora-test-list and/or fedora-devel-list | 11:33 |
| -!- balor [n=balor@193.62.183.62] has quit [Read error: 110 (Connection timed out)] | 11:33 | |
| -!- JSchmitt [n=s4504kr@p4FDD1611.dip0.t-ipconnect.de] has joined #fedora-meeting | 11:33 | |
| wwoods | it's really just so people can subscribe to the results and we get a public archive | 11:33 |
| wwoods | a mailing list might not be ideal, but when all we have is hammers, every problem looks like a nail | 11:34 |
| viking_ice | could we hook up somekind of rss subscription option to it | 11:34 |
| wwoods | hm. does mailman do RSS feeds? | 11:34 |
| -!- alexxed [n=alex@dyn-85.186.119.28.tm.upcnet.ro] has joined #fedora-meeting | 11:34 | |
| jlaska | https://fedorahosted.org/fedora-infrastructure/ticket/1193 | 11:35 |
| f13 | wwoods: you can get email2rss feeds hooked up | 11:35 |
| f13 | I don't think mailman does that itself | 11:35 |
| wwoods | hrm. maybe we really just want it to generate results and an RSS feed. | 11:36 |
| wwoods | actually! that'd be a super-simple basic implementation of the results server | 11:36 |
| wwoods | a big dumb RSS feed of the results/logs submitted | 11:36 |
| f13 | as soon as you do that, somebody is going to ask for an rss2email instance of it (: | 11:36 |
| wwoods | ha | 11:36 |
| adamw | that would be me | 11:37 |
| f13 | lets stick with email for now, since it works today | 11:37 |
| * jlaska nods | 11:37 | |
| adamw | rss is good for stuff where I only care about the last day | 11:37 |
| adamw | i can stick it in firefox | 11:37 |
| -!- bristot [n=bristot@galileo.intelbras.com.br] has joined #fedora-meeting | 11:37 | |
| adamw | but it doesn't really fit my (or probably other people's) workflows for longer term stuff. | 11:37 |
| viking_ice | http://taint.org/mmrss/ | 11:38 |
| wwoods | right. we'll stick with email and I'll probably do a webui/rss feed for the first implementation of the results server | 11:38 |
| wwoods | anyway | 11:38 |
| wwoods | so that's autoqa so far | 11:38 |
| wwoods | speaking of webui and results servers. shall we talk about nitrate for a moment? | 11:38 |
| -!- wwoods changed the topic of #fedora-meeting to: Fedora QA Meeting | nitrate | 11:38 | |
| jlaska | hrmm | 11:38 |
| viking_ice | I say not to posting results in wiki I just dont think wiki is a result holder | 11:38 |
| adamw | and who's going to reply to that lovely email we got? :) | 11:39 |
| wwoods | viking_ice: definitely not | 11:39 |
| wwoods | basically, we got an update about nitrate status | 11:39 |
| jlaska | wwoods: can we hold off on a nitrate update until we have some information to share | 11:39 |
| viking_ice | next meeting ? | 11:39 |
| jlaska | we're navigating through some internal issues around resources for the nitrate project ... I expect to have more on that by next meeting | 11:40 |
| viking_ice | I personally see something like a bodhi like for results and instead of karma points just worked did not work | 11:40 |
| -!- spoleeba [n=one@fedora/Jef] has joined #fedora-meeting | 11:40 | |
| wwoods | yeah, really just wanted to mention that nitrate is not going to magically appear anytime soon | 11:41 |
| jlaska | adamw: I've got that mail | 11:41 |
| adamw | OK | 11:41 |
| jlaska | adamw: but may solicit your assistance :) | 11:41 |
| viking_ice | well more like f12 f13 | 11:41 |
| viking_ice | f13: are you prepared for f13 | 11:41 |
| viking_ice | hehe | 11:42 |
| -!- stickster_ [n=npfrield@ip72-205-14-2.dc.dc.cox.net] has joined #fedora-meeting | 11:42 | |
| jlaska | it all depends on available resources, until I get more info ... it's too hard to nail a date | 11:42 |
| f13 | roger. | 11:42 |
| -!- deca [n=deca@host86-164-180-127.range86-164.btcentralplus.com] has joined #fedora-meeting | 11:42 | |
| -!- fab [n=bellet@monkey.creatis.insa-lyon.fr] has quit ["Leaving"] | 11:43 | |
| wwoods | so yes, jlaska's on top of the situation. that's all for now about nitrate. | 11:43 |
| -!- wwoods changed the topic of #fedora-meeting to: Fedora QA Meeting | test day | 11:44 | |
| jlaska | in the meantime, I've been discussing with the laptop.org folks (Mel Chua) their experiences with test case/results mgmt | 11:44 |
| adamw | working in perfect harmony there, guys ;) | 11:44 |
| wwoods | heh! sorry | 11:44 |
| jlaska | I can present moer on that later if folks are interested ... this is mainly intended as a short term solution for fcami's needs (https://www.redhat.com/archives/fedora-test-list/2009-February/msg00756.html) | 11:44 |
| adamw | it does seem like something that would be very welcome, though | 11:45 |
| jlaska | indeed | 11:45 |
| adamw | trying to write instructions as to how to send in test results for the upcoming test day royally sucked | 11:45 |
| -!- warren [n=warren@redhat/wombat/warren] has quit ["Leaving"] | 11:45 | |
| -!- wwoods changed the topic of #fedora-meeting to: Fedora QA Meeting | nitrate / test case/results management | 11:45 | |
| adamw | "just edit the table and do it how i did" isn't great | 11:45 |
| wwoods | yeah, it's been a long-standing problem | 11:45 |
| jlaska | wwoods: sorry, we can postpone the semantic for the end if you like | 11:45 |
| -!- drago01 [n=linux@chello062178124130.3.13.univie.teleweb.at] has quit [Read error: 60 (Operation timed out)] | 11:45 | |
| wwoods | unfortunately there aren't any obviously great solutions | 11:45 |
| wwoods | no, sorry, I thought "hold off on talking about nitrate" meant we should move to the next topic | 11:46 |
| wwoods | but this is important stuff | 11:46 |
| viking_ice | I think we should be working at distro independent solution or atleast host that system for all distro's so we can spread the test writing an test cases and hopefully get more hands on writing them | 11:46 |
| adamw | the process in the laptop.org thing looks workable | 11:46 |
| adamw | that's http://wiki.laptop.org/go/Reporting_test_results | 11:46 |
| -!- warren [n=warren@nat/redhat/x-7d1f93be80bad6ef] has joined #fedora-meeting | 11:47 | |
| jlaska | right ... I summarized their thoughts in my note if anyone wants a quick read ...https://www.redhat.com/archives/fedora-test-list/2009-February/msg00756.html | 11:47 |
| jlaska | mediawiki is great for content ... so it seems to be a good fit for writing plans and cases for both groups | 11:47 |
| viking_ice | every distro and application benefits from a central point of origin regarding testing test cases | 11:48 |
| wwoods | so, are they pretty happy with this solution? | 11:48 |
| -!- stickster_ is now known as stickster | 11:48 | |
| adamw | erk, brb | 11:48 |
| jlaska | wwoods: like anything, pros and cons | 11:48 |
| -!- stickster_afk [n=pfrields@fedora/stickster] has quit [Nick collision from services.] | 11:48 | |
| wwoods | it certainly looks like it would fill our needs and I know we've got some expertise running mediawiki instances... | 11:48 |
| jlaska | the biggest con is that the semantic extension has a performance impact on the entire wiki | 11:48 |
| adamw | viking_ice: i like the idea, but i'm not sure if this would be a case where bringing even more people on board would make the process even messier... | 11:48 |
| viking_ice | adamw: then let's approach it from the other end | 11:49 |
| viking_ice | come up with a solution host it on a neutral ground and offer them to participate | 11:49 |
| viking_ice | I'm sure a lot of distro's would rather want to use a central point then trying to come up with something on their own like we are faced with now | 11:50 |
| jlaska | we're investing in the process side of the house right now with adamw, viking_ice and company driving the wiki cleanup ... our test plan/case organization along with test days. I'm really pleased with how those are maturing. That elephant in the room is those darn'd results | 11:51 |
| viking_ice | I say let's use bodhi like system | 11:51 |
| viking_ice | ui friendly | 11:51 |
| -!- kital [n=Joerg_Si@fedora/kital] has quit [Remote closed the connection] | 11:52 | |
| wwoods | easier said than done, of course | 11:52 |
| viking_ice | if we break things down what does the tester need to report? He just need to report if the test was successful or not and if not a bug number | 11:52 |
| viking_ice | no more no less | 11:52 |
| viking_ice | he can write his life story on a blog | 11:53 |
| -!- kital [n=Joerg_Si@fedora/kital] has joined #fedora-meeting | 11:53 | |
| adamw | viking_ice: well, not always | 11:53 |
| adamw | viking_ice: take the 20 second boot test day for e.g. | 11:53 |
| adamw | viking_ice: the test cases aren't pass/fail | 11:53 |
| -!- kital [n=Joerg_Si@fedora/kital] has quit [Remote closed the connection] | 11:53 | |
| adamw | the test cases produce results in the form of a file, which you have to provide | 11:53 |
| jlaska | adamw: viking_ice: yeah ... the 20second boot test cases could possibly fall into the performance tests category | 11:53 |
| viking_ice | the result for those test cases are meaningless if the kernel team has not been contacted and asked to produce an debug info free kernel | 11:54 |
| jlaska | there is some scrubbing of the results before you can assert pass/fail | 11:54 |
| -!- drago01 [n=linux@chello062178124130.3.13.univie.teleweb.at] has joined #fedora-meeting | 11:54 | |
| viking_ice | mean from the 20second boot | 11:54 |
| adamw | but otoh, the laptop.org system basically does what you say | 11:54 |
| adamw | did you look at the screenshot? | 11:54 |
| viking_ice | did anybody contact the kernel team to come up a with a kernel for this test day | 11:55 |
| jlaska | adamw: right | 11:55 |
| adamw | viking_ice: we're not on that topic yet | 11:55 |
| -!- Pikachu_2014 [n=Pikachu_@81-66-20-24.rev.numericable.fr] has joined #fedora-meeting | 11:55 | |
| viking_ice | ok | 11:55 |
| adamw | well, OK, we're running out of time anyway :\ | 11:56 |
| jlaska | the immediate problem I was investigating with the semantic route was to provide fcami with an interim solution for organizing results for the Xorg test effort | 11:56 |
| adamw | what's the conclusion on this whole topic? do we like the laptop.org system? I do | 11:56 |
| -!- kiilerix [n=mk11@cpe.ge-1-1-0-787.alb2nqu1.customer.tele.dk] has joined #fedora-meeting | 11:56 | |
| adamw | it's simple, does what we need, and we don't have to write any code | 11:56 |
| jlaska | I do ... but there are caveats that I'll need to discuss with infrastructure folks | 11:57 |
| adamw | we can talk to the Wiki guys about whether they'd prefer we attach it to the main wiki or create a special wiki instance | 11:57 |
| jlaska | the "off-the-shelf" aspect is *very* appealing | 11:57 |
| jlaska | I'll kick off a thread on the infrastructure list to see if we can identify if any blockers exist for that solution | 11:57 |
| jlaska | okay ... any other thoughts? Should we move on to tomorrows Test Day? | 11:58 |
| adamw | yes, let's | 11:58 |
| wwoods | sounds good to me | 11:58 |
| adamw | so :) | 11:59 |
| -!- wwoods changed the topic of #fedora-meeting to: Fedora QA Meeting | test day | 11:59 | |
| jlaska | https://fedoraproject.org/wiki/QA/Test_Days/2009-02-19 | 11:59 |
| adamw | viking_ice: to answer your question, afaik, no. i'm not convinced that's a huge problem, though. the impact of debugging stuff is imho rather overstated, and the point of the test day isn't to get absolute benchmarks per se, but to identify bottlenecks so performance can be improved | 11:59 |
| -!- pingou [n=Pingou@fedora/pingou] has quit [Read error: 110 (Connection timed out)] | 11:59 | |
| jlaska | viking_ice: I haven't heard from the devel + qa teams involved on whether that is an issue as well | 12:00 |
| adamw | debugging-enabled packages should make everything slower at roughly the same rate - they won't cause bottlenecks to exist that wouldn't exist in debugging-disabled builds | 12:00 |
| adamw | so the results should still be valid. | 12:00 |
| wwoods | right, the debugging overhead should be the same for all testers | 12:00 |
| f13 | right | 12:01 |
| -!- nicubunu [n=nicubunu@mail.apsro.com] has quit ["Fading into sunset"] | 12:01 | |
| f13 | we're not looking to see if we've hit 20 seconds yet, we're looking to identify the bottlenecks | 12:01 |
| adamw | i'd like to ask, does everyone think the current info on the test day is good enough? | 12:01 |
| * jlaska very pleased with format | 12:01 | |
| adamw | it's one of the topics i have good hopes for 'community' involvement in, as it's the kind of thing people tend to care about | 12:01 |
| wwoods | my only question so far is: why are the test cases still under QA/XXX instead of in the QA: namespace? | 12:01 |
| adamw | so i made an effort to clean it up, and i promoted it via blog and fedoraforums | 12:01 |
| adamw | wwoods: er. didn't know they were supposed to be there :) they can be moved, i don't think i did any external links to the cases. | 12:02 |
| jlaska | adamw: oh good, thanks on the blog post! | 12:02 |
| adamw | all the existing test cases I could find in http://fedoraproject.org/wiki/Category:Test_Cases were under QA/ , so I just followed that. | 12:03 |
| wwoods | testcases, generally, should stay out of the main namespace, since we don't want to clutter search results | 12:03 |
| viking_ice | well if we are going to get some accurate readings we need that kernel to see where we are at this point + I would believe the maintainer(s) should be aware bottleneck where the bottlenecks are and was just asking for the where we are at this point there also have been several threads on devel regarding this so as a say the maintainer(s) should know where to look to cut boot time | 12:03 |
| jlaska | wwoods: adamw: this is where the Forms extension helps ... it hides much of the "where to file" test case stuff | 12:04 |
| jlaska | adamw: yeah I haven't migrated all the old test cases yet :( | 12:04 |
| adamw | viking_ice: if they didn't want more data, they'd not have asked for a test day, presumably :) | 12:04 |
| adamw | so what would be an example of a 'good' location for a testcase now then? | 12:04 |
| adamw | QA:Some_test_case | 12:05 |
| adamw | ? | 12:05 |
| jlaska | https://fedoraproject.org/wiki/QA:TestCases/Assign_ext4_fs_with_mount_point | 12:05 |
| jlaska | one can argue that it shouldn't still have the "TestCases/" | 12:05 |
| adamw | no, that's a bad example, it's using fake directories. :) | 12:05 |
| f13 | blah | 12:05 |
| jlaska | it's part of the name I konw | 12:05 |
| adamw | right. the wiki folks would argue that very loudly =) | 12:05 |
| f13 | I would too | 12:05 |
| -!- nphilipp [n=nils@nat/redhat/x-a806de4627c54100] has quit ["Leaving"] | 12:05 | |
| jlaska | meh ... as long as it's in the QA: namespace ... it doesn't bother me what people call it | 12:06 |
| f13 | QA:Assign_ext4_fs_with_mount_point woudl be fine, and have it exist in Category:TestCase | 12:06 |
| adamw | ok. anyhoo...we're off topic again | 12:06 |
| viking_ice | adamw: perhaps we would like to know where the 20 second feature is at this point hence we need to try it with a kernel and ( other items that might affect that result ) with debug output set to off | 12:06 |
| wwoods | viking_ice: we're ignoring kernel debugging stuff. the readings will still be *accurate*, since there's a known *certain* error | 12:07 |
| jlaska | viking_ice: I've got confirmation from the maintainer that it's not a concern for tomorrow | 12:07 |
| wwoods | we're not concerned with the actual final numeric result being accurate | 12:07 |
| adamw | viking_ice: i think the naming is a bit of a mistake, because putting an arbitrary number on it doesn't really make sense. fedora runs on a huge range of hardware, you can't put a single number on the boot time | 12:07 |
| f13 | viking_ice: we're not to the point yet where features are supposed to be "testable", we're still gathering information to do the work to make them "testable" | 12:07 |
| adamw | viking_ice: the point of the project is just to make it faster. | 12:07 |
| f13 | viking_ice: after beta, then it would be interesting to do timing tests with non-debug kernel to see if we've accomplished the 20second startup. | 12:07 |
| -!- moixs [n=chatzill@81.13.142.103] has joined #fedora-meeting | 12:07 | |
| viking_ice | ok | 12:08 |
| wwoods | we're concerned about what parts we can speed up. turning debugging off is already a known factor | 12:08 |
| wwoods | we're looking for new, unknown factors | 12:08 |
| wwoods | with possible solutions | 12:08 |
| adamw | ...like readahead should be ionice'd...sorry, i'll keep that for tomorrow. :) | 12:08 |
| wwoods | kernel debugging is known and has a known solution | 12:08 |
| jlaska | ;) | 12:08 |
| wwoods | so it's uninteresting | 12:08 |
| jlaska | so for any lurkers ... please do drop by #fedora-qa tomorrow to help gather bootchart data :) | 12:10 |
| viking_ice | based on that the 20second startup should have been more generic like scrape as much seconds as possible during this release cycle of the boot time instead of nailing it to 20 seconds | 12:10 |
| adamw | viking_ice: that's my opinion, yeah. but i think putting a number on it is intended to gather more attention :) | 12:10 |
| jlaska | 6 minute abs | 12:10 |
| -!- dash__ [i=d2d43dfb@gateway/web/ajax/mibbit.com/x-bd304b14deadc302] has joined #fedora-meeting | 12:10 | |
| adamw | "Make boot faster project" doesn't sound as edgy as "20 second boot project". | 12:10 |
| adamw | but, bleh. | 12:10 |
| viking_ice | adamw: well that number can backfire if we dont make it.. | 12:10 |
| adamw | oh, we can always make it. just throw hardware at it. =) | 12:11 |
| wwoods | too late now. don't get hung up on it. | 12:11 |
| adamw | if you stick a couple of SSDs in this system it'd probably be there already. | 12:11 |
| f13 | jlaska: I've got 5 minute abs here. | 12:11 |
| jlaska | f13: that comes in F12 :) | 12:11 |
| adamw | so, uh, yeah, my issue is just - can anyone see anything remaining that would block people from getting involved in this test day? | 12:11 |
| dash__ | asl plzz?? | 12:12 |
| f13 | adamw: time? (: | 12:12 |
| viking_ice | hum where have I heard that before if code is not good enough just throw more hw at it | 12:12 |
| jlaska | f13: we've got all day covered for CEST and EST | 12:12 |
| adamw | apart from the messiness of entering results (which we covered earlier), the other thing that stuck out was the bit where you install rawhide, which is something we talked about... | 12:12 |
| dash__ | neone asl plzz? | 12:12 |
| -!- dash__ [i=d2d43dfb@gateway/web/ajax/mibbit.com/x-bd304b14deadc302] has left #fedora-meeting [] | 12:12 | |
| jlaska | f13: it's listed at the top (12:00 to 20:00 UTC (7am -> 3pm ET)) ... I'll update to include EST coverage | 12:12 |
| adamw | well, that was bizarre. | 12:12 |
| -!- dash123 [i=d2d43dfb@gateway/web/ajax/mibbit.com/x-c3d3b33c58cdac14] has quit ["http://www.mibbit.com ajax IRC Client"] | 12:12 | |
| f13 | I meant time as in I have none to give to the test day | 12:13 |
| wwoods | I can probably run a couple tests | 12:13 |
| wwoods | but yeah, the instructions look good to me | 12:13 |
| viking_ice | Cant we let test day's time a whole day and just say the feature owner/ maintainer will be available during this time | 12:13 |
| jlaska | f13: ah gotcha ... folks are welcome to run through the procedures at any time ... we'll just have "folks on call" during the test day | 12:14 |
| jlaska | viking_ice: that's what we have ... just depends on the TZ of the maintainer and test lead for the feature | 12:14 |
| adamw | that should probably be written in the test day page, actually. i'll update it | 12:14 |
| adamw | f13: but yeah, you can do all the tests today and add your data to the table. ;) | 12:15 |
| adamw | wwoods: it took me about an hour and a half to do the whole set, and I was fiddling with things in the meantime. | 12:15 |
| viking_ice | hum we need to expand the test lead to more people (spread across timezones ) so we can cover a whole day | 12:15 |
| viking_ice | efficiently | 12:15 |
| adamw | viking_ice: the point is the developers, not the qa guys | 12:15 |
| adamw | viking_ice: there's usually only one or two devs associated with a given test day project | 12:15 |
| adamw | viking_ice: and they're not going to want to sit in IRC for a sixteen hour stretch | 12:16 |
| viking_ice | ofcourse but the lead can gather results and feed back and pass it on to the maintainers when they become available | 12:16 |
| adamw | viking_ice: just for the qa group's input, we could probably cover a whole day, but unless it's a big dev group, the devs probably can't. | 12:16 |
| viking_ice | nobody is asking them to | 12:16 |
| viking_ice | think of this as an assignment we collect them from the testers and hand them to the developers when they are available | 12:17 |
| adamw | that might work for future days, we could discuss it when planning those...i guess it depends what the topic is and how the dev wants to structure the day | 12:17 |
| jlaska | I've not yet found 1 way to run a test day that works for every test day. It largely depends on the feature, the maintainer(s) involved, testers involved, tools available etc... | 12:18 |
| jlaska | we've done a great job though of getting a consistent look'n'feel on the Test Day process pages and the individual test day event pages | 12:18 |
| adamw | yeah, i think that's right. for e.g., for something highly technical like iSCSI last week, it'd be difficult to do anything meaningful without the dev there. | 12:18 |
| adamw | but for something broader like this, we could probably cover it for a longer period. | 12:19 |
| jlaska | indeed | 12:19 |
| f13 | also having it be a "test day" drives excitement and interest and participation, rather than "test week" where you get "oh I'll get around to it at some point..." | 12:19 |
| jlaska | any other test day thoughts/concerns around the wiki page content? | 12:19 |
| wwoods | nothing from me | 12:20 |
| jlaska | adamw: you pleased with the feedback on the current state of the event wiki? | 12:20 |
| adamw | jlaska: no-one said it sucked, so yeah. =) | 12:20 |
| jlaska | hah, great ;) | 12:20 |
| jlaska | okay ... next topic then? | 12:20 |
| wwoods | that's all I had | 12:20 |
| -!- wwoods changed the topic of #fedora-meeting to: Fedora QA Meeting | misc. | 12:21 | |
| wwoods | anything else? | 12:21 |
| jlaska | wwoods: I'd like to reserver 10 mins at the next meeting to discuss "QA group activities + goals discussion" - https://www.redhat.com/archives/fedora-test-list/2009-February/msg00707.html | 12:21 |
| jlaska | along with a test case mgmt update | 12:21 |
| adamw | i'm good for now, just please everyone get involved with the test day tomorrow, if we at least have a table with results from all the regulars (especially redhat.com people :>) it'll encourage more people to join in | 12:22 |
| adamw | same reason street buskers put a few quarters in the hat to start with =) | 12:22 |
| viking_ice | i'll probably be offline for a week | 12:22 |
| -!- dkovalsk [n=dkovalsk@r11eu187.net.upc.cz] has quit [Read error: 113 (No route to host)] | 12:22 | |
| adamw | viking_ice: shooting arrows into things? | 12:22 |
| viking_ice | yup | 12:22 |
| viking_ice | well that's just for the weekend | 12:23 |
| adamw | hehe, have fun, don't do a Cheney | 12:23 |
| jlaska | viking_ice: good luck! | 12:23 |
| viking_ice | will be the first of the Iceland so I'm already a champ | 12:23 |
| wwoods | viking_ice: heh! congrats! | 12:23 |
| * viking_ice since he's the only one competing from Iceland | 12:23 | |
| wwoods | jlaska: okay, noted agenda items for next week | 12:24 |
| -!- MostafaDaneshvar [n=MostafaD@unaffiliated/mostafadaneshvar] has joined #fedora-meeting | 12:24 | |
| wwoods | - Test case management update | 12:25 |
| -!- themayor [n=jack@dsl081-200-011.nyc2.dsl.speakeasy.net] has joined #fedora-meeting | 12:25 | |
| wwoods | - qa group activities + goals | 12:25 |
| wwoods | - raise a glass for viking-ice shooting arrows into things | 12:25 |
| wwoods | - feature test plan rundown | 12:26 |
| wwoods | there's a loooot of features | 12:26 |
| wwoods | and we're gonna need to make sure we can test them very soon | 12:26 |
| wwoods | anything else for next week? | 12:26 |
| jlaska | maybe a brief autoqa update? | 12:26 |
| wwoods | oh right | 12:26 |
| wwoods | - autoqa status report | 12:27 |
| jlaska | then the goal for all of us ... to pack that into 1 hour of fun | 12:27 |
| wwoods | heh | 12:27 |
| * jlaska thinks about a 5 lbl bag | 12:28 | |
| wwoods | yeah I suspect we may need to just split up the features and work on them during the rest of the week | 12:28 |
| jlaska | wwoods: perhaps we spend that time to review a game plan? dunno | 12:28 |
| * f13 goes to get food | 12:28 | |
| wwoods | jlaska: yeah, we'll work something out by next week | 12:28 |
| wwoods | anyway, that's all I've got | 12:29 |
| wwoods | if there's nothing else, let's call it done | 12:29 |
| -!- CheekyBoinc [n=CheekyBo@fedora/CheekyBoinc] has joined #fedora-meeting | 12:30 | |
| wwoods | okay - thanks all | 12:30 |
| wwoods | I'll post a log and a summary shortly | 12:30 |
| --- Log closed Wed Feb 18 12:31:02 2009 | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!