--- Log opened Wed Aug 27 11:08:19 2008
-!- wwoods changed the topic of #fedora-meeting to: Fedora QA Meeting | init11:08
wwoodsf13, jlaska, jds2001, poelcat: ping11:08
wwoods(and hello to anyone else coming for the QA Meeting)11:08
* jlaska waves11:09
* kulll wiggles11:09
-!- cebbert [n=cebbert@fedora/cebbert] has joined #fedora-meeting11:09
-!- notting [n=notting@redhat/notting] has joined #fedora-meeting11:09
* jwilliam load f10a one more time :)11:10
wwoodsjwilliam: I swear we're *really* close to having new rawhide trees to test11:10
wwoods(testing the aug. 12 tree is getting a little old)11:10
wwoodsso the big thing we need to talk about is: new package signing keys11:11
-!- thm [n=thomas@1erlei.de] has joined #fedora-meeting11:12
wwoodswarren: ping - so what was the list of rel-eng recommendations sent to FESCo?11:12
warrenwwoods: pretty much exactly what was sent to rel-eng11:12
wwoodsokay, so here's a bit of background, for the record:11:13
bpepplewwoods: you looking for this? http://lists.fedoraproject.org/pipermail/rel-eng/2008-August/001614.html11:13
wwoodsfollowing the intrusion into the fedora infrastructure we've decided it'd be prudent to generate new signing keys11:13
wwoodsbpepple: yes! thank you.11:14
wwoodsnow as I understand it we have two signing keys: one for final releases and official updates, and one for test packages (updates-testing)11:14
wwoodsas that message says, we're currently planning to sign new updates with the new key(s)11:15
-!- mccann [n=jmccann@c-76-118-157-202.hsd1.ma.comcast.net] has joined #fedora-meeting11:15
wwoodswhich means that either a) we figure out a clever way of installing the keys on people's systems11:16
wwoodsor b) everyone has to import the new keys11:16
wwoodsIf you'll recall, the key-import dialog in PackageKit in the original F9 release was.. bad11:16
wwoodsso bad that we hacked anaconda to auto-import keys11:16
wwoodsto avoid ever seeing that dialog11:16
wwoods*current* packagekit is OK11:17
wwoodsand the old pirut dialog works fine.11:17
wwoodsSo let's consider a matrix of what'll need to be tested:11:17
wwoods-F8 GOLD updating to current F811:18
wwoods-F8 + old updates getting new updates11:18
wwoodsrepeat for F911:18
wwoodsF9 GOLD -> new-key-signed updates is going to be the worst one.11:19
wwoodsIn the meantime, we're going to gradually resign all the current updates11:20
wwoodsso that we can then "revoke" the old key11:21
-!- mccann [n=jmccann@c-76-118-157-202.hsd1.ma.comcast.net] has quit ["See ya"]11:21
wwoodsand then we'll resign the existing *install* repos11:22
wwoodsbut not isos11:22
wwoodsthe packages on those will continue to have the old signatures11:22
wwoodswarren: is that right? that's the plan of record?11:22
warrensee my post to rel-eng list for all details11:23
jwbi think that's pretty close wwoods11:23
jwbthere are some minor discussions on-going, but nothing that majorly changes that11:23
wwoodsmostly the thing that confuses me is that we're keeping the old packages on the isos11:24
wwoodsbut I guess that can't be helped11:24
jwbit could11:24
-!- cassmodiah [n=cass@p54AB6BBD.dip.t-dialin.net] has joined #fedora-meeting11:24
jwbit would be a bit odd though11:24
warrenhere's the reason for keeping the ISO's unchanged11:25
warren(I don't neccessarily agree with it)11:25
wwoodsthe problem is that the F9 iso auto-imports the old (less-trusted) key11:25
wwoodsso then we need to revoke it post-install11:25
warrenThe ISO's have known content that are signed as a collective11:25
wwoodsbut then, there's no way to get the current isos *out* of people's hands11:25
warrenI think we *should* redo the ISO's11:25
warrenbut the others in rel-eng disagreed11:25
wwoodsso that's not a problem we can hope to completely solve anyway11:26
warrenThe important point is this:11:26
warrenWe need to make a new key and begin using it to sign new updates immediately.11:26
wwoodswe will still need a way to revoke the bad key post-install until F9 is EOL11:26
warrenEverything beyond that we still need to figure out the details.11:26
nirikif we redo isos, we should just make a 9.0.3 release or something.11:26
wwoodsright. and the QA implications are: we need to make sure this isn't going to screw up people's update feeds.11:26
warrennirik: that would imply having updates in the ISO's, which was not even proposed yet, quite a bit time consuming.11:27
nirikyeah, world of pain.11:27
jwbcould just take the ISOs down and point to FU11:27
warrenperhaps using Fedora Unity resigned wouldn't be bad11:28
nirikwwoods: it's going to. No way around it... we are going to need some nice docs to point the millions at... and even then some people will just not care or get mad and go somewhere else.11:28
warrenthey at least did testing11:28
warrenthat would require FESCO and Board level approval though11:28
wwoodsrespinning the isos doesn't take the already-existing isos out of circulation11:28
wwoodsit doesn't fully solve the problem.11:29
wwoodsand it can't.11:29
wwoodsall those F9 CDs/DVDs we gave away? you can't undo that.11:29
warrenbut do we all agree that we need to generate the new keys, sign updates, THEN figure out the rest?11:29
wwoodsso respinning the ISOs is definitely a Good Idea but it's not critical, since it doesn't actually *solve* the problem11:29
wwoodspretty much, yeah. we're going to need some F8/F9 systems to test this with11:30
wwoodsnow, what happened to the idea of pushing out one last fedora-release update11:30
wwoodsthat contains the new key11:30
jwbi think it failed for some reason11:31
* jwb is extremely helpful on that topic11:31
wwoodsF8 and (updated) F9 already have nice mechanisms for importing needed keys that exist on-disk11:32
nottingthere was an objection to pushing any new content with the old key11:32
wwoodsThis is going to *completely* screw anyone installing from F9 media.11:33
nirikwell, they will get no updates unless they manually update the key... or use updates repo at install time11:34
wwoodsIIRC you can't use the updates repo if you're installing from media.11:34
warrenwwoods: pushing a new fedora-release signed by the old key seems like it would help11:34
warrenwwoods: but the moment you have other updates in the repo signed by the new key, it is no longer automatic upgrade11:34
jeremyit may be a little early still for f13-on-the-west-coast11:35
warrenwwoods: the entire transaction fails and require manual intervention11:35
wwoodsyeah, and with F9's original PK it's a pretty spectacular fail11:35
warrenwwoods: Unless somebody has something else clever, there is no 100% automatic way to migrate to a new key.  Users will need to manually install something.11:35
wwoodsesp. with the number of updates we currently have available for F911:35
nottingjeremy: yes, but he was the one that raised that objection11:36
warrenwwoods: now a later proposed step is to create a future automatic way to migrate a key11:36
wwoodsokay, clumens/alindebe have corrected me - you *can* enable updates during media install11:36
nirikand revoking the old key is also likely going to need to be manual... which is sad, because most people won't bother.11:36
wwoodsbut naturally that requires network access to an updates repo11:36
nirikyeah, again most people won't bother. ;)11:37
nottingthe other option is move to repo signing, which i believe is not ready, according to skvidal?11:37
nottingand has the same migration issues (need a new yum, etc.)11:38
-!- drago01 [n=linux@chello062178124130.3.13.univie.teleweb.at] has joined #fedora-meeting11:38
wwoodsin the long run I feel like repo signing is more sensible, but that's kind of outside my area of expertise11:38
warrenrepo signing actually is better than package signatures?11:38
wwoodsas I said, outside my area of expertise, but we can talk about my opinions on the matter later11:39
wwoodsso. have we generated a new key yet?11:39
wwoodsor should we generate some test packages and a test key11:39
warrenf13 insisited that we cannot do anything without fesco's approval11:40
warrenwhich I disagreed with11:40
nottingyou can test with a test key in any case11:40
wwoodsto see what happens when we try to do updates and there's suddenly a new key11:40
warrenI proposed to FESCO that they look over our recommendations, but vote to give us autonomy to DO THE RIGHT THING so we don't have to wait any longer.11:40
wwoodswarren: rel-eng can't move forward with actually signing actual updates with the actual key. But QA can certainly test close approximations of that process.11:40
mbonnetwwoods: crazy idea: what if we change the repo location?  Push out a new fedora-release to the old repo, signed with the old key, that contains the new key, and also updates /etc/yum.repos.d/fedora-updates.repo to point to a new tree, that would only contain packages signed with the new key?11:40
mbonnetwwoods: could probably be done by appending an extra url var to the mirror manager URL11:41
warrenmbonnet: huge painful mirror churn11:41
mbonnetwarren: why?  It's just one new directory.11:41
warrenmbonnet: and not everyone uses mirrormanager, if people edited their .repo files then they are screwed11:41
* nirik was thinking about that too, but what does it get us?11:41
wwoodsnot really huge churn. the repo just gets a new oldrepo/ subdir with one package (or maybe 5 packages - fedora-release and a new PK) and repodata11:42
mbonnetwarren: but people editing their .repo files are more likely to know how to fix it too.  This would get the majority of people who don't know or care what a .repo file is.11:42
warrenI personally would vote against it.11:42
warrenwwoods: when the old repo gets resigned then what?11:42
mbonnetwarren: the old repo *never* gets resigned11:43
wwoodsold repo gets resigned? huh?11:43
-!- mbacovsk [n=mbacovsk@nat/redhat/x-31ea8cf74bc44f39] has quit [Read error: 110 (Connection timed out)]11:43
warrenif we don't resign the old repo, there is no point in doing a new key.11:43
warrenIt gains us NOTHING to do a new key11:43
warrenThe entire point of a new key is to make the old key no longer acceptable to install packages.11:43
wwoodsno, the point is that we empty the old repo of everything except the new key, a pointer to the new repo, and a PK update that's able to install the new key11:44
warrenwwoods: THAT is what I refer to when I mean huge repo churn11:44
wwoodsso the only update that we have that uses the old key.. updates you to the new key11:44
warrenwwoods: to rsync that's a huge delete and tons of new files.11:44
wwoodsno, it doesn't11:44
warren"empty the old repo of everything except the new key" delete everything at the old URL11:45
wwoodsAFAIK rsync is smart enough to detect that everything's moved to a different dir11:45
nirikhardlink ?11:45
warrennot with the flags that most people use11:45
wwoodsthis is all just "hey whatif" brainstorming11:45
mbonnetor this can be done with a newrepo/ subdir and some mirrormanager trickery I bet11:45
wwoodshonestly, though, you'd rather annoy *all* users than annoy mirrors?11:46
warrenIt would take days to redo all the repos11:46
warrenplus the flood of rawhide11:46
warrenwhich still hasn't happened yet11:46
wwoodsand how long's it going to take to re-sign all the repos?11:47
warrenThat isn't relevant to what you're suggesting11:47
warrenunless you mean leave the old repo with old key11:47
warrenduring the weeks until resigning is done11:47
warrenOK, if this is done in a certain way, it actually need not break people who edited their .repo file.11:48
warrenLeave the existing .repo files intact, and add a new one for the resigned content.11:48
warrenor rather, new key.11:48
wwoodssure - default it to enabled11:49
warrenthen after everything is resigned, move everything11:49
warrenwhich will cause terrible mirror churn11:49
warrenbut least users don't need to manually install the new key11:50
warrenthis might work11:50
warrenalthough you'll need to convince the others to accept it11:50
warrenI'm not fully won over myself.11:50
mbonnetwarren: I don't understand the mirror churn argument.  The newly signed packages are *new packages*, we nave to get them out there somehow.11:50
warrenmbonnet: resigned packages moved to a different directory would be a huge delete and download again operation11:51
jlaskawwoods: I've got to step out ... but didn't want to put out a reminder of Fedora Test Day scheduled for tomorrow (https://www.redhat.com/archives/fedora-test-list/2008-August/msg00326.html)11:51
warrenmost mirrors are NOT using the magic fuzzy switches that would recognize that11:51
warrenit isn't only "mv" but also "mv with slightly different content"11:51
warrencan rsync handle that at all?  I doubt it.11:51
mbonnetwarren: the only thing the oldrepo/ dir would need to contain would be the new fedora-release (and maybe PK).  They wouldn't have to redownload everything.11:51
wwoodsjlaska: right on. Printer Day! Maybe we can get the helpdesk guys to fix CUPS advertising on the RDU network..11:52
warrenmbonnet: I don't think we're on the same page.11:52
warrenmbonnet: you mean oldrepo/ contains fedora-release, new PK and nothing else?11:52
jlaskawwoods: definitly, I plan to bring some in ... and any samba shared printing would be a good focus11:52
mbonnetwarren: right11:52
warrenmbonnet: then that is exactly the mirror churn scenario11:52
-!- chacha_chaudhry [n=rpmbuild@gnu-india/supporter/rakeshpandit] has joined #fedora-meeting11:53
wwoodsin terms of Badness, "mirror churn" < "manual intervention or no updates"11:54
warrenmirror churn would take more than a day to recover from11:54
warrenjust hope you realize11:54
wwoodsrequiring manual user intervention to fix updating is Total Failure11:54
warreni'm surprised we didn't have anything to handle migration and revoking11:54
mbonnetwarren: Ok, remove *everything* from the current repos.  oldrepo/ contains fedora-release and maybe PK (say 5 packages).  Regular repo contains packages resigned with new key.  Isn't this the minimum possible amount of mirror churn?11:54
wwoodsit leaves our users without security updates, makes us look really damn bad11:54
wwoodsit's absolutely not an acceptable choice11:55
mbonnetwwoods: +10011:55
warrenmbonnet: that's almost complete mirror churn11:55
wwoods...assuming we have a choice.11:55
mbonnetwarren: we're replacing *all packages*, of course there's going to be churn.  It's unavoidable.11:55
wwoodsthere might also need to be a new rpm in there, to handle the new key-revocation juju we're theoretically going to add11:56
warrenmbonnet: and there is no point in removing the packages from oldrepo until after they are resigned.11:56
mbonnetwarren: that *increases* churn because mirrors will have to download the oldrepo/ content.11:56
warrenoldrepo we are referring to is the CURRENT location11:56
warrencurrent URL's11:56
warrennewrepo has to be a different URL11:57
mbonnetumm, no necessarily11:57
nottingwwoods: we aren't adding any new juju in the timeframe you're taking about11:57
mbonnetmaybe that's the source of confusion11:57
warrenoldrepo at current URL's, newrepo at new URL, is the only way we will make it fully automatic11:57
wwoodsnotting: gotcha11:57
-!- CheekyBoinc [n=SuL@fedora/CheekyBoinc] has joined #fedora-meeting11:57
warrenoldrepo at current URL's, newrepo at new URL, with old key content in oldrepo until it is resigned11:57
mbonnetwarren: fine, then don't delete content from the oldrepo, it really doesn't matter11:58
wwoodsright. once a package is resigned, it can be moved from current-repo to new-repo11:58
warrenwhat do we call new-repo?11:58
warrenwe need to pick a URL to put it now11:58
mbonnetimplementation detail.  Can we get consensus on the approach first?11:59
wwoodswe can certainly suggest it to FESCo/rel-eng11:59
warrenI suppose this is OK, and the mirror churn happens later (not the same day as rawhide flood)11:59
warrenmbonnet: it is an implementation detail, but we need it IMMEDIATLEY if we are going to sign updates to push12:00
warrenso we might as well decide now12:00
-!- mcepl [n=matej@ppp1053.in.ipex.cz] has left #fedora-meeting ["Ex-Chat"]12:00
wwoodsI don't think it's up to QA to make that decision - pick something sensible and ask f13 what he thinks when he appears12:00
-!- J5 [n=quintice@c-76-24-17-105.hsd1.ma.comcast.net] has joined #fedora-meeting12:00
warrenok, we can pick something to recommend12:00
wwoodsnirik: exactly12:00
wwoodsI'd like input from f13 / mdomsch whenever mirroring juju is involved12:01
warrenI highly recommend we pick something to recommend12:01
wwoodsso pick something12:01
wwoodsI completely don't care12:01
warrencurrent location...12:01
warrenI don't care what it is12:01
warrenbut we should recommend something12:02
wwoodsprobably I'd pick something more like linux/releases/9.newkey/Fedora/...12:02
wwoodsbut, as I've said, not my area of expertise12:02
warrenwwoods: if you do it at that level, then that implies the massive ISO's are also newkey12:03
wwoodsoffer those two suggestions to f13/mdomsch/etc. and see what they say12:03
wwoodswarren: but there won't be any 9.newkey/Fedora/i386/iso dir12:03
warrenwwoods: which will be confusing12:03
wwoodsit's fairly clear - we never produced new isos with the new key12:03
wwoodswe're basically recomposing the tree here. makes sense to apply the change at that level12:04
wwoodswe'll make those two suggestions12:04
warrenok fine.12:04
wwoodsand let mirrormanager/rel-eng experts tell us what makes sense12:04
wwoodsbecause I'm not an expert on either.12:04
warrenNote, it still might be possible logically that this plan of newrepo doesn't work.12:04
warrenI haven't thought of a drawback yet12:04
-!- Frankly3D [n=Frankly3@] has quit [Remote closed the connection]12:04
warrenI'll update the recommendation list after i get back from food12:05
* nirik thinks it's complicated, but a better solution than the other one12:05
warrenseriously need food now12:05
-!- che [n=rkastl@redhat/che] has quit [Remote closed the connection]12:05
wwoodsit's worth discussing, at the very least12:05
wwoodsyeah, it's lunchtime. let's summarize:12:06
wwoods1) bad stuff happened and now we are stuck deciding between a bunch of things that suck.12:06
wwoods2) mirror churn sucks less than forcing users to manually monkey with their keys.12:07
wwoods3) we believe we can make the process fairly painless for the users by pushing out a fedora-release update, signed with the old key, that contains the new key and a new repo.12:08
wwoods3a) the new repo is where we'll put all the packages signed with the new key.12:08
wwoods4) since F9 original PackageKit handles key installation *really* badly, we may need to leave the current PackageKit update in the current repo12:09
-!- CheekyBoinc [n=SuL@fedora/CheekyBoinc] has quit [Remote closed the connection]12:10
-!- nphilipp [n=nils@] has quit ["Leaving"]12:10
wwoodsThings we plan to test:12:11
mbonnetwwoods: sounds good to me.  I'm not sure we gain anything by pulling already-released updates anyway, so might as well leave the current repo as-is (with the exception of pushing the new fedora-release)12:11
-!- hanthana [n=hanthana@] has joined #fedora-meeting12:11
* jwilliam 9.112:11
wwoods1) Install F8; update to current (including packages with new keys)12:12
wwoods2) Install F8 and current (pre-intrusion) updates; update with new-key packages12:12
wwoodsrepeat for F9.12:12
wwoodsExpected results: system updates successfully (although it may ask you to import a new key). 12:13
wwoodssystem continues to notify you of further updates in the new repo(s), signed with the new keys. doesn't ask for the key import again on subsequent updates.12:14
wwoodstesting this is gonna be a PITA.12:14
wwoodswe're gonna need scripts to create custom repos to test these scenarios.12:14
wwoodsand maybe create fake test packages too.12:15
wwoodsoh well. it's stuff we should have tools to test anyway.12:15
-!- giallu [n=giallu@fedora/giallu] has quit [Read error: 110 (Connection timed out)]12:15
wwoodswonder if skvidal has tools like this for testing yum12:16
wwoodsI'll ask him about it.12:16
-!- mccann [n=jmccann@c-65-96-163-181.hsd1.ma.comcast.net] has joined #fedora-meeting12:16
wwoodsokay. does the suggested plan make sense?12:16
mbonnetwwoods: does to me.12:16
-!- themayor [n=jack@dsl081-200-018.nyc2.dsl.speakeasy.net] has joined #fedora-meeting12:16
mbonnetwwoods: it doesn't deal with revoking the old key, but I think that's a different problem.12:16
wwoodsyeah, that's Someone Else's Problem12:16
wwoodswell, no. It's our problem too. We'll worry about that once we've got this part of the plan solidified a bit better.12:17
wwoodsI'll try to summarize this suggestion for FESCo (which meets in ~2hrs)12:18
wwoodswe'll see what comes out of that meeting.12:18
-!- J5 [n=quintice@c-76-24-17-105.hsd1.ma.comcast.net] has quit [Read error: 110 (Connection timed out)]12:19
wwoodsand I'll ping skvidal about yum testing tools so we can make sure that updating will work as expected.12:19
wwoodsokay. the time is late and lunch looms large. anything else we should discuss?12:19
* wwoods taking that as a no12:20
wwoodsthanks for your time, all12:20
--- Log closed Wed Aug 27 12:20:39 2008

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!