--- Log opened Wed May 21 11:02:32 2008
wwoodsoh nice, I finally saved my log settings properly11:02
wwoodsfedora-qa-20080521.log opened. whee11:03
jds2001hehe i have this logged going back probably as far as you need.11:03
wwoodswhat I really wish we had was a meeting-logbot11:03
wwoodsthat I could give it 'start' and 'stop' commands 11:03
wwoodsand it'd irc2html the messages between those two things and put 'em up on some website11:04
f13wwoods: put up a "bounty"11:04
f13Debian has such a bot...11:04
wwoodsa bounty, eh11:04
ianwellerhmm. zodbot can probably be told to do that, there's a ChannelLogger plugin for supybot. just fyi11:04
f13ianweller: good to know!11:04
wwoodshuh. nice.11:05
wwoodsthat'd be a really useful thing for this channel.11:05
* ricky is still waiting on packaging supybot for Fedoar :-)11:05
wwoods/msg zodbot start meeting-id [password].. or I guess it could just require a known, identified user11:06
-!- kital [n=Joerg_Si@fedora/kital] has quit [Remote closed the connection]11:06
wwoodsso, yes, QA meeting go11:06
-!- sahak [n=sahak@pool-71-184-182-102.bstnma.east.verizon.net] has joined #fedora-meeting11:06
wwoodswe're in the part of the cycle where we can actually do planning and development11:06
wwoodsand we are soon to have an intern (yay)11:06
wwoodsso we should talk tools for a bit11:07
jds2001is rawhide actually installable for them to do something?  I haven't tried in the last few days11:07
wwoodswell here's the thing - I'm not so sure I just want to throw her at installing rawhide every day11:07
wwoods(I haven't tried either, to be honest)11:08
f13jds2001: I haven't tried (:11:08
f13I was meaning to today11:08
wwoodsI've been doing a looooot of F8->F9 upgrades11:08
wwoodsfor preupgrade stuff11:09
wwoodsreally want to get that done-ish so I can move on for a while11:09
* jds2001 noticed that preupgrade does not use his local mirror11:09
wwoodsjds2001: it does if you edit releases.list11:09
jds2001so it's not using failovermethod=priority11:09
wwoodsor if you set up mirrormanager 11:09
wwoodsit's using whatever yum's default is11:09
* jds2001 has mirrormanager setup11:10
wwoodswhich should be failovermethod=priority in F811:10
wwoodscheck /etc/yum.conf, 'cuz it works fine here11:10
jds2001hmm, probably OT for here, but I'm seeing otherwise :)11:10
* jds2001 can try again tonight11:10
wwoodsperhaps you have an out-of-date mirrorlist.txt in your preupgrade cache dir11:10
-!- tibbs_ is now known as tibbs|h11:12
wwoodsmy main priority for the F10 cycle is expanding the QA team and making it easier for new people to do useful testing11:12
-!- mether [n=ask@fedora/mether] has joined #fedora-meeting11:12
wwoodsso we need stuff like: better testing docs. stickster has introduced me to one of the docs guys who's looking to help with that11:13
wwoodswe also need stuff like: tools for tracking testing (and testers)11:13
sticksterwwoods++ !!!1!!11:14
wwoodsfurther, I think we need a QA group in FAS11:14
* f13 throws rocks at stickster 11:14
f13wwoods: or to take over the existing one?11:14
wwoodsthat works too11:14
jds2001just remove everybody and turn the reigns over to wwoods?11:15
wwoodsyou can make me an admin for 'qa' and I can boot everyone from the group, I guess?11:15
* viking-ice likes then plan about "hostile" take over11:15
wwoodsthat too11:15
-!- SmootherFrOgZ [n=r@fedora/SmootherFrOgZ] has quit ["Leaving"]11:15
wwoodsbut I want new contributors to have, y'know. a few choices about things to do, docs on how to do it, a list of people to ask when they have questions11:16
wwoodsand I want *us* to have a way to see what people are up to11:16
wwoodsupdates-testing (bodhi) and triage (bugzilla) already have pretty good docs and tools11:16
wwoodsand we can get good metroids (heh, metrics) out of them11:16
jds2001bugzilla metroids are harder - but I think that poelcat had some ideas around that.11:17
wwoodsas for release testing - all we've got is the wiki matrix. which we can hand-count, but yuck11:17
jds2001any aspiring web developers to develop an app for it?11:17
wwoodsjds2001: cool - we should talk in more detail about that at some point, but I totally trust you guys to come up with some good ideas11:18
wwoodswell that's the thing - our intern has some background in web development11:18
wwoodsso getting something like a test case/plan tracker, or bugzilla metroid tracker, or similar web-based tester tools11:18
* viking-ice get's more and more curious about this mysterious intern... 11:18
wwoodsis a distinct possiblity11:18
f13wwoods is now an admin of the qa group.11:18
* wwoods flexes11:18
wwoodsother proposed web tools.. the ever-popular rawhide dashboard11:19
f13Honestly we do have software coming down teh pike for test case management and test result management11:19
* viking-ice throws the dices to see if wwoods gonna smite him..11:19
wwoodsf13: oh?11:19
f13so I would think using the intern for something else would be better.11:19
f13wwoods: that's stage 1 of bill peck's work11:19
wwoodsbut for manual test cases etc?11:19
wwoodsI mean the Testify2 team has been talking about this for a year11:20
f13wwoods: from what I understand it, the way it's supposed tow ork is that you create a test case that /is/ the manual way of doing the test11:20
f13wwoods: then people can do the test and register with the system that they've done said test / steps11:20
* f13 wonders why it is I know more about this than the guy in QA...11:20
wwoodsheh. because you sit near him?11:21
f13anywho, there's supposed to be something working or to the point of being worked on by FUDCon BOS11:22
* jds2001 hears pins drop11:22
wwoodsthat'd be cool11:22
wwoodsso, okay, assuming that bpeck is going to descend from the heavens with a perfect test tracker11:22
f13I would still urge you to seek bpeck out and have a conversation with him bout this and the intern stuff.11:23
wwoodsanyway: what would you suggest that we work on?11:23
wwoodsthat being, y'know, the intern and whoever else has time11:23
wwoodsas it is now, QA gets two real windows for development: summer and winter11:24
wwoodsso. summer dev cycle priorities?11:24
f13test cases would be good11:24
f13or are you talking about infrastructure level stuff?11:24
wwoodsinfrastructure-level stuff11:25
f13I know that I could use a lot of tree validation tests and some form of automated rawhide validation each night11:25
wwoodstest cases are OK, a tool to put test cases in is even better11:25
wwoodsnightly rawhide runs means extending SNAKE11:25
-!- cweyl [n=cweyl@c-24-130-175-238.hsd1.ca.comcast.net] has joined #fedora-meeting11:25
wwoodswhich is something jlaska might be thinking about already11:26
wwoodsIIRC he's been talking to bpeck about using SNAKE to get machines installed for the next-gen testing system11:26
wwoodsso it's an active area of development with a good amount of appeal internally, which means we will probably get support from RHEL QE11:27
wwoodsso is that the worst part of the test cycle?11:28
jlaskaI've heard several suggestions from devleopment mgmt on needs for a QA intern in the fedora space that are visible in both fedora and RHEL11:28
wwoodsmaking sure rawhide works and finding out when it does?11:28
wwoodsI'd sure like to have some stats on how many days we have installable rawhide and how many days it's broken11:29
jds2001i would *love* to see snake work via PXE11:29
jds2001not just from an installed system11:29
wwoodsoh yeah11:29
wwoodsIIRC there were some changes in the livecd tools to make, basically, huge initrd images11:29
wwoodsit'd be perfect to just use a known-good kernel and an initrd that just runs snake11:29
jds2001yeah, i noticed a livecd-iso-to-pxeboot command11:30
wwoodsso if we can figure out how to generate a super-minimal snake livecd11:30
wwoodsthen we are golden11:31
wwoodsin fact.. snake livecd is a good idea by itself11:31
f13wwoods: nightly testing is definitely a sore point right now.  I'd also love to be able to poop out a full split iso tree and turn loose an automated test suite upon it11:31
-!- giallu [n=giallu@81-174-9-190.dynamic.ngi.it] has joined #fedora-meeting11:31
jds2001f13: wow, that'd be painful :)11:31
f13mmmm fibre11:32
wwoodsbetter than pooping out Everything11:32
wwoodsokay, I'm getting the sense that automated installation testing is the Big Dream11:33
-!- poelcat [n=slick@fedora/poelcat] has joined #fedora-meeting11:33
wwoodsit's kind of a prereq for automated functional testing, too, so it's pretty important11:33
wwoodsSo we'll assume bpeck has the testcase / testplan tracking stuff covered11:35
f13hopefully, and maybe he could use some help with it11:35
f13I also think we need to focus on testing more than just the install path11:36
f13working with the SIGs to create test cases to validate their area of responsiblities11:36
poelcatf13: i think that should be part of more specific release criteria11:36
jds2001right now, even though it's on the release criteria, i dont think we test that apps survive a basic click-through.11:36
poelcatand would consider help leading such a session at fudcon11:37
wwoodsthe thing I'm worried about is that.. it leads to enormous growth in test cases11:37
wwoodsand tracking test execution is still kind of a PITA with the wiki11:38
jds2001well, if the SIGs are responsible for having someone in the QA subproject, then we're golden.11:38
wwoodsoh, and I'm going to nominate myself for FESCo so I have a vote on all proposed features11:38
f13Unless the SIG is offering somebody up to do the testing, those tests are not going to be blockers11:38
wwoodsand I'm gonna vote against anything that doesn't have a good test plan11:38
-!- mether [n=ask@fedora/mether] has quit [Read error: 113 (No route to host)]11:38
* jds2001 votes for wwoods 11:38
jds2001not that it looks like there's gonna be much voting at this point :/11:39
* jds2001 was thinking of nominating himself too, but was gonna wait til the last minute.11:39
-!- spoleeba [n=one@fedora/Jef] has joined #fedora-meeting11:39
wwoodsI mean, obviously QA as a team will offer support in writing (and executing) test plans11:40
wwoodsso it shouldn't be too heavy a requirement11:40
wwoodsbut yeah11:40
* viking-ice suggest "hostile" takeover since it worked so well last time ( wwoods is now an admin of the qa group ) 11:40
wwoodsALSO I want test plans to be attached to, y'know, components11:40
wwoodsso when there's a new update for a package, bodhi can show all the test cases for that package11:41
poelcatwwoods: can you provide guidance that I can add to feature policy about what is considered sufficient11:41
poelcati get a lot of push back on "how hard" it is to fill out feature pages :)11:41
poelcatso i'm sure there will be more if we tighten up test plan section11:42
wwoodsthere's a testplan template and plenty of examples11:42
f13basically we'd like a way to verify that the Feature is working as expected, or not working as expected11:42
* jds2001 also thinks that folks that offer that pushback are full of crap.11:42
wwoodserr, test *case* rather. http://fedoraproject.org/wiki/QA/TestCases/TestCaseTemplate11:42
jds2001We're not asking for a doctoral thesis there.11:42
wwoodsWe'll find a good example or two11:43
wwoodsand link people to the testcasetemplate11:43
* jds2001 goes to lunch11:43
jlaskaso the thinking is, that when a feature is proposed, it would be "a good thing" to have some guidance as to steps to validate functionality11:44
wwoodspoelcat: is that sufficient?11:44
jlaskaare we expecting the feature author to write the testcase(s)?11:44
wwoodsright - it's a simple request. Feature owners, please tell QA how to test your feature11:44
poelcatwwoods: can you give me links to the "good example or two" ?11:44
wwoodsjlaska: yes, but we'll certainly provide assistance and guidance11:45
jlaskawwoods: I'm guessing they'll likely want to provide the assistance and guidance, and we focus on what we do well (the test cases) part.11:45
wwoodspoelcat: offhand no - there's probably a couple good 'uns in the installation test plan11:46
wwoodshttp://fedoraproject.org/wiki/QA/TestCases/InstallDualbootMacOSX is nicely verbose11:46
jlaskaI don't know if I'm comfortable asking development to write the test cases ... perhaps propose test cases, or provide input that influences a test case.11:46
wwoodshttp://fedoraproject.org/wiki/QA/TestCases/PartitioningNoSwap is more to the point11:46
wwoodsjlaska: okay, ask that they *propose* a test case, and QA will expand/cleanup until it's acceptable11:47
f13if not actual test case, psuedo test case.11:47
-!- rdieter is now known as rdieter_away11:47
wwoodswell I mean, check the PartitioningNoSwap case11:47
-!- rdieter [n=rdieter@pcp088900pcs.unl.edu] has joined #fedora-meeting11:47
jlaskawwoods: I think that's fair ... they use the same philosophy for patch review for their code.  We should employ the same mechanism11:47
wwoodsthat's a sufficient case, and it doesn't take real dang long to write something like that11:48
jlaskamany features often require fairly extensive test plans ... another case where having tighter involvement between both groups is required to produce a usable outcome11:48
wwoodslarger features will probably be driven by an internal team or a SIG11:49
-!- rdieter [n=rdieter@pcp088900pcs.unl.edu] has quit [Remote closed the connection]11:49
wwoodswhich should already have QA reps11:49
wwoodsso they should have some good guidance on doing the right thing11:49
wwoodsso let's talk about the QA group a bit - any prereqs for membership?11:49
-!- rdieter [n=rdieter@pcp088900pcs.unl.edu] has joined #fedora-meeting11:50
f13FAS and CLA11:50
wwoodsat the moment it doesn't grant you and rights11:50
wwoodserr, any rights11:50
f13the FAS is rather implicit11:50
f13but CLA I'd like so we cna be sure to re-use any submitted test cases in RHEL products11:50
wwoodsis group membership needed for bodhi commentary, or just a FAS account?11:50
viking-iceDirections for turning "debugging" where to look what to strace etc.. and which component ( if not plain obvious ) to file against in bgz, will need to be on the test case page ( new testers will need this stuff )..11:50
f13bodhi commentary can be done anonymously11:51
wwoodsright, but anonymous doesn't count11:51
f13wwoods: I don't know the answer to that.11:51
f13lmacken: ping?11:51
-!- rdieter [n=rdieter@pcp088900pcs.unl.edu] has left #fedora-meeting ["Konversation terminated!"]11:51
wwoods"real QA" should be done by identified users - plus we want to keep metroids on karma given by QA team members11:51
jlaskawwoods: QA group bit ... what's the driver behind that?  Are there problems/concerns in the current process?11:52
jlaskaheh ...  s/Are/What are/11:52
wwoodsyeah, helps identify the team members and keep the team more team-y11:53
wwoodsin the future we'll use it to give access to QA tools11:53
viking-iceSo will this extend to overall QA or just testing..11:53
wwoodslike bpeck's stuff I'd assume11:53
wwoodsoverall QA - writing test plans, f'rinstance11:53
wwoodsor if we ever start doing things like unit testing in the package database11:54
wwoodsso, yeah, mostly the driver is recognizing and rewarding contributors to QA11:55
wwoodsanyway we're very near the end of our hour11:56
wwoodsany last-minute questions / discussion topics?11:56
viking-iceWell it should give the maintainers/developers more time to focus on their work if they can hand over all the testing to QA11:57
wwoodswe don't really have the capacity for that right now though11:57
viking-iceand just post I need testing on this and this and this ..11:57
wwoodswhich is why team-building is important - to expand the team and build (and measure) capacity11:57
* f13 foods.11:59
wwoodsyeah, I think meeting time is over, and food time is now11:59
smoogewwoods, good eating11:59
wwoodsthanks for your time, folks.. further discussion can be moved to #fedora-qa or fedora-test-list12:00
wwoodsoh! one note12:00
wwoodshttp://partner-bugzilla.redhat.com/ has a beta of the Bugzilla 3.2 code we'll be using soon12:00
wwoodsit doesn't send mail or anything, so mess around to your heart's content12:00
wwoods*especially* if you've got any bugzilla-specific code, test it against that instance12:00
wwoodsand make sure everything works OK12:00
-!- stahnma [n=stahnma@fedora/stahnma] has joined #fedora-meeting12:01
wwoodsthanks again!12:01
--- Log closed Wed May 21 12:01:28 2008

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!