| --- Log opened Wed Aug 27 11:08:19 2008 | ||
| -!- wwoods changed the topic of #fedora-meeting to: Fedora QA Meeting | init | 11:08 | |
| wwoods | f13, jlaska, jds2001, poelcat: ping | 11:08 |
|---|---|---|
| wwoods | (and hello to anyone else coming for the QA Meeting) | 11:08 |
| * jlaska waves | 11:09 | |
| * kulll wiggles | 11:09 | |
| -!- cebbert [n=cebbert@fedora/cebbert] has joined #fedora-meeting | 11:09 | |
| -!- notting [n=notting@redhat/notting] has joined #fedora-meeting | 11:09 | |
| * jwilliam load f10a one more time :) | 11:10 | |
| wwoods | jwilliam: I swear we're *really* close to having new rawhide trees to test | 11:10 |
| wwoods | (testing the aug. 12 tree is getting a little old) | 11:10 |
| wwoods | so the big thing we need to talk about is: new package signing keys | 11:11 |
| -!- thm [n=thomas@1erlei.de] has joined #fedora-meeting | 11:12 | |
| wwoods | warren: ping - so what was the list of rel-eng recommendations sent to FESCo? | 11:12 |
| warren | wwoods: pretty much exactly what was sent to rel-eng | 11:12 |
| wwoods | okay, so here's a bit of background, for the record: | 11:13 |
| bpepple | wwoods: you looking for this? http://lists.fedoraproject.org/pipermail/rel-eng/2008-August/001614.html | 11:13 |
| wwoods | following the intrusion into the fedora infrastructure we've decided it'd be prudent to generate new signing keys | 11:13 |
| wwoods | bpepple: yes! thank you. | 11:14 |
| wwoods | now as I understand it we have two signing keys: one for final releases and official updates, and one for test packages (updates-testing) | 11:14 |
| wwoods | as that message says, we're currently planning to sign new updates with the new key(s) | 11:15 |
| -!- mccann [n=jmccann@c-76-118-157-202.hsd1.ma.comcast.net] has joined #fedora-meeting | 11:15 | |
| wwoods | which means that either a) we figure out a clever way of installing the keys on people's systems | 11:16 |
| wwoods | or b) everyone has to import the new keys | 11:16 |
| wwoods | If you'll recall, the key-import dialog in PackageKit in the original F9 release was.. bad | 11:16 |
| wwoods | so bad that we hacked anaconda to auto-import keys | 11:16 |
| wwoods | to avoid ever seeing that dialog | 11:16 |
| wwoods | *current* packagekit is OK | 11:17 |
| wwoods | and the old pirut dialog works fine. | 11:17 |
| wwoods | So let's consider a matrix of what'll need to be tested: | 11:17 |
| wwoods | -F8 GOLD updating to current F8 | 11:18 |
| wwoods | -F8 + old updates getting new updates | 11:18 |
| wwoods | repeat for F9 | 11:18 |
| wwoods | F9 GOLD -> new-key-signed updates is going to be the worst one. | 11:19 |
| wwoods | In the meantime, we're going to gradually resign all the current updates | 11:20 |
| wwoods | re-sign | 11:20 |
| wwoods | so that we can then "revoke" the old key | 11:21 |
| -!- mccann [n=jmccann@c-76-118-157-202.hsd1.ma.comcast.net] has quit ["See ya"] | 11:21 | |
| wwoods | and then we'll resign the existing *install* repos | 11:22 |
| wwoods | but not isos | 11:22 |
| wwoods | the packages on those will continue to have the old signatures | 11:22 |
| wwoods | warren: is that right? that's the plan of record? | 11:22 |
| warren | see my post to rel-eng list for all details | 11:23 |
| jwb | i think that's pretty close wwoods | 11:23 |
| jwb | there are some minor discussions on-going, but nothing that majorly changes that | 11:23 |
| wwoods | mostly the thing that confuses me is that we're keeping the old packages on the isos | 11:24 |
| wwoods | but I guess that can't be helped | 11:24 |
| jwb | it could | 11:24 |
| -!- cassmodiah [n=cass@p54AB6BBD.dip.t-dialin.net] has joined #fedora-meeting | 11:24 | |
| jwb | it would be a bit odd though | 11:24 |
| warren | here's the reason for keeping the ISO's unchanged | 11:25 |
| warren | (I don't neccessarily agree with it) | 11:25 |
| wwoods | the problem is that the F9 iso auto-imports the old (less-trusted) key | 11:25 |
| wwoods | so then we need to revoke it post-install | 11:25 |
| warren | The ISO's have known content that are signed as a collective | 11:25 |
| wwoods | but then, there's no way to get the current isos *out* of people's hands | 11:25 |
| warren | I think we *should* redo the ISO's | 11:25 |
| warren | but the others in rel-eng disagreed | 11:25 |
| wwoods | so that's not a problem we can hope to completely solve anyway | 11:26 |
| warren | The important point is this: | 11:26 |
| warren | We need to make a new key and begin using it to sign new updates immediately. | 11:26 |
| wwoods | we will still need a way to revoke the bad key post-install until F9 is EOL | 11:26 |
| warren | Everything beyond that we still need to figure out the details. | 11:26 |
| nirik | if we redo isos, we should just make a 9.0.3 release or something. | 11:26 |
| wwoods | right. and the QA implications are: we need to make sure this isn't going to screw up people's update feeds. | 11:26 |
| warren | nirik: that would imply having updates in the ISO's, which was not even proposed yet, quite a bit time consuming. | 11:27 |
| nirik | yeah, world of pain. | 11:27 |
| jwb | could just take the ISOs down and point to FU | 11:27 |
| warren | perhaps using Fedora Unity resigned wouldn't be bad | 11:28 |
| nirik | wwoods: it's going to. No way around it... we are going to need some nice docs to point the millions at... and even then some people will just not care or get mad and go somewhere else. | 11:28 |
| warren | they at least did testing | 11:28 |
| warren | that would require FESCO and Board level approval though | 11:28 |
| wwoods | respinning the isos doesn't take the already-existing isos out of circulation | 11:28 |
| wwoods | it doesn't fully solve the problem. | 11:29 |
| wwoods | and it can't. | 11:29 |
| warren | nod | 11:29 |
| wwoods | all those F9 CDs/DVDs we gave away? you can't undo that. | 11:29 |
| warren | but do we all agree that we need to generate the new keys, sign updates, THEN figure out the rest? | 11:29 |
| wwoods | so respinning the ISOs is definitely a Good Idea but it's not critical, since it doesn't actually *solve* the problem | 11:29 |
| wwoods | pretty much, yeah. we're going to need some F8/F9 systems to test this with | 11:30 |
| wwoods | now, what happened to the idea of pushing out one last fedora-release update | 11:30 |
| wwoods | that contains the new key | 11:30 |
| jwb | i think it failed for some reason | 11:31 |
| * jwb is extremely helpful on that topic | 11:31 | |
| wwoods | F8 and (updated) F9 already have nice mechanisms for importing needed keys that exist on-disk | 11:32 |
| notting | there was an objection to pushing any new content with the old key | 11:32 |
| notting | f13? | 11:33 |
| wwoods | This is going to *completely* screw anyone installing from F9 media. | 11:33 |
| nirik | well, they will get no updates unless they manually update the key... or use updates repo at install time | 11:34 |
| wwoods | IIRC you can't use the updates repo if you're installing from media. | 11:34 |
| warren | wwoods: pushing a new fedora-release signed by the old key seems like it would help | 11:34 |
| warren | wwoods: but the moment you have other updates in the repo signed by the new key, it is no longer automatic upgrade | 11:34 |
| jeremy | it may be a little early still for f13-on-the-west-coast | 11:35 |
| warren | wwoods: the entire transaction fails and require manual intervention | 11:35 |
| wwoods | yeah, and with F9's original PK it's a pretty spectacular fail | 11:35 |
| warren | wwoods: Unless somebody has something else clever, there is no 100% automatic way to migrate to a new key. Users will need to manually install something. | 11:35 |
| wwoods | esp. with the number of updates we currently have available for F9 | 11:35 |
| notting | jeremy: yes, but he was the one that raised that objection | 11:36 |
| warren | wwoods: now a later proposed step is to create a future automatic way to migrate a key | 11:36 |
| wwoods | okay, clumens/alindebe have corrected me - you *can* enable updates during media install | 11:36 |
| nirik | and revoking the old key is also likely going to need to be manual... which is sad, because most people won't bother. | 11:36 |
| wwoods | but naturally that requires network access to an updates repo | 11:36 |
| nirik | yeah, again most people won't bother. ;) | 11:37 |
| wwoods | yup. | 11:37 |
| notting | the other option is move to repo signing, which i believe is not ready, according to skvidal? | 11:37 |
| notting | and has the same migration issues (need a new yum, etc.) | 11:38 |
| -!- drago01 [n=linux@chello062178124130.3.13.univie.teleweb.at] has joined #fedora-meeting | 11:38 | |
| wwoods | in the long run I feel like repo signing is more sensible, but that's kind of outside my area of expertise | 11:38 |
| warren | repo signing actually is better than package signatures? | 11:38 |
| wwoods | as I said, outside my area of expertise, but we can talk about my opinions on the matter later | 11:39 |
| wwoods | so. have we generated a new key yet? | 11:39 |
| wwoods | or should we generate some test packages and a test key | 11:39 |
| warren | f13 insisited that we cannot do anything without fesco's approval | 11:40 |
| warren | which I disagreed with | 11:40 |
| notting | you can test with a test key in any case | 11:40 |
| wwoods | to see what happens when we try to do updates and there's suddenly a new key | 11:40 |
| warren | I proposed to FESCO that they look over our recommendations, but vote to give us autonomy to DO THE RIGHT THING so we don't have to wait any longer. | 11:40 |
| wwoods | warren: rel-eng can't move forward with actually signing actual updates with the actual key. But QA can certainly test close approximations of that process. | 11:40 |
| mbonnet | wwoods: crazy idea: what if we change the repo location? Push out a new fedora-release to the old repo, signed with the old key, that contains the new key, and also updates /etc/yum.repos.d/fedora-updates.repo to point to a new tree, that would only contain packages signed with the new key? | 11:40 |
| mbonnet | wwoods: could probably be done by appending an extra url var to the mirror manager URL | 11:41 |
| warren | mbonnet: huge painful mirror churn | 11:41 |
| mbonnet | warren: why? It's just one new directory. | 11:41 |
| warren | mbonnet: and not everyone uses mirrormanager, if people edited their .repo files then they are screwed | 11:41 |
| * nirik was thinking about that too, but what does it get us? | 11:41 | |
| wwoods | not really huge churn. the repo just gets a new oldrepo/ subdir with one package (or maybe 5 packages - fedora-release and a new PK) and repodata | 11:42 |
| mbonnet | warren: but people editing their .repo files are more likely to know how to fix it too. This would get the majority of people who don't know or care what a .repo file is. | 11:42 |
| warren | I personally would vote against it. | 11:42 |
| warren | wwoods: when the old repo gets resigned then what? | 11:42 |
| mbonnet | warren: the old repo *never* gets resigned | 11:43 |
| wwoods | old repo gets resigned? huh? | 11:43 |
| -!- mbacovsk [n=mbacovsk@nat/redhat/x-31ea8cf74bc44f39] has quit [Read error: 110 (Connection timed out)] | 11:43 | |
| warren | if we don't resign the old repo, there is no point in doing a new key. | 11:43 |
| warren | It gains us NOTHING to do a new key | 11:43 |
| warren | The entire point of a new key is to make the old key no longer acceptable to install packages. | 11:43 |
| wwoods | no, the point is that we empty the old repo of everything except the new key, a pointer to the new repo, and a PK update that's able to install the new key | 11:44 |
| warren | wwoods: THAT is what I refer to when I mean huge repo churn | 11:44 |
| wwoods | so the only update that we have that uses the old key.. updates you to the new key | 11:44 |
| warren | wwoods: to rsync that's a huge delete and tons of new files. | 11:44 |
| wwoods | no, it doesn't | 11:44 |
| warren | "empty the old repo of everything except the new key" delete everything at the old URL | 11:45 |
| wwoods | AFAIK rsync is smart enough to detect that everything's moved to a different dir | 11:45 |
| wwoods | anyway | 11:45 |
| nirik | hardlink ? | 11:45 |
| warren | not with the flags that most people use | 11:45 |
| wwoods | this is all just "hey whatif" brainstorming | 11:45 |
| mbonnet | or this can be done with a newrepo/ subdir and some mirrormanager trickery I bet | 11:45 |
| wwoods | honestly, though, you'd rather annoy *all* users than annoy mirrors? | 11:46 |
| warren | It would take days to redo all the repos | 11:46 |
| warren | plus the flood of rawhide | 11:46 |
| warren | which still hasn't happened yet | 11:46 |
| wwoods | and how long's it going to take to re-sign all the repos? | 11:47 |
| warren | That isn't relevant to what you're suggesting | 11:47 |
| warren | oh | 11:47 |
| warren | unless you mean leave the old repo with old key | 11:47 |
| warren | during the weeks until resigning is done | 11:47 |
| warren | OK, if this is done in a certain way, it actually need not break people who edited their .repo file. | 11:48 |
| warren | Leave the existing .repo files intact, and add a new one for the resigned content. | 11:48 |
| warren | or rather, new key. | 11:48 |
| wwoods | sure - default it to enabled | 11:49 |
| warren | then after everything is resigned, move everything | 11:49 |
| warren | which will cause terrible mirror churn | 11:49 |
| warren | but least users don't need to manually install the new key | 11:50 |
| warren | this might work | 11:50 |
| warren | although you'll need to convince the others to accept it | 11:50 |
| warren | I'm not fully won over myself. | 11:50 |
| mbonnet | warren: I don't understand the mirror churn argument. The newly signed packages are *new packages*, we nave to get them out there somehow. | 11:50 |
| warren | mbonnet: resigned packages moved to a different directory would be a huge delete and download again operation | 11:51 |
| jlaska | wwoods: I've got to step out ... but didn't want to put out a reminder of Fedora Test Day scheduled for tomorrow (https://www.redhat.com/archives/fedora-test-list/2008-August/msg00326.html) | 11:51 |
| warren | most mirrors are NOT using the magic fuzzy switches that would recognize that | 11:51 |
| jlaska | s/didn't/did/ | 11:51 |
| warren | it isn't only "mv" but also "mv with slightly different content" | 11:51 |
| warren | can rsync handle that at all? I doubt it. | 11:51 |
| mbonnet | warren: the only thing the oldrepo/ dir would need to contain would be the new fedora-release (and maybe PK). They wouldn't have to redownload everything. | 11:51 |
| wwoods | jlaska: right on. Printer Day! Maybe we can get the helpdesk guys to fix CUPS advertising on the RDU network.. | 11:52 |
| warren | mbonnet: I don't think we're on the same page. | 11:52 |
| warren | mbonnet: you mean oldrepo/ contains fedora-release, new PK and nothing else? | 11:52 |
| jlaska | wwoods: definitly, I plan to bring some in ... and any samba shared printing would be a good focus | 11:52 |
| mbonnet | warren: right | 11:52 |
| warren | mbonnet: then that is exactly the mirror churn scenario | 11:52 |
| -!- chacha_chaudhry [n=rpmbuild@gnu-india/supporter/rakeshpandit] has joined #fedora-meeting | 11:53 | |
| wwoods | in terms of Badness, "mirror churn" < "manual intervention or no updates" | 11:54 |
| warren | mirror churn would take more than a day to recover from | 11:54 |
| warren | just hope you realize | 11:54 |
| wwoods | requiring manual user intervention to fix updating is Total Failure | 11:54 |
| warren | i'm surprised we didn't have anything to handle migration and revoking | 11:54 |
| mbonnet | warren: Ok, remove *everything* from the current repos. oldrepo/ contains fedora-release and maybe PK (say 5 packages). Regular repo contains packages resigned with new key. Isn't this the minimum possible amount of mirror churn? | 11:54 |
| wwoods | it leaves our users without security updates, makes us look really damn bad | 11:54 |
| wwoods | it's absolutely not an acceptable choice | 11:55 |
| mbonnet | wwoods: +100 | 11:55 |
| warren | mbonnet: that's almost complete mirror churn | 11:55 |
| wwoods | ...assuming we have a choice. | 11:55 |
| mbonnet | warren: we're replacing *all packages*, of course there's going to be churn. It's unavoidable. | 11:55 |
| wwoods | there might also need to be a new rpm in there, to handle the new key-revocation juju we're theoretically going to add | 11:56 |
| warren | mbonnet: and there is no point in removing the packages from oldrepo until after they are resigned. | 11:56 |
| mbonnet | warren: that *increases* churn because mirrors will have to download the oldrepo/ content. | 11:56 |
| warren | uh | 11:56 |
| warren | oldrepo we are referring to is the CURRENT location | 11:56 |
| warren | current URL's | 11:56 |
| warren | newrepo has to be a different URL | 11:57 |
| mbonnet | umm, no necessarily | 11:57 |
| mbonnet | not | 11:57 |
| notting | wwoods: we aren't adding any new juju in the timeframe you're taking about | 11:57 |
| mbonnet | maybe that's the source of confusion | 11:57 |
| warren | oldrepo at current URL's, newrepo at new URL, is the only way we will make it fully automatic | 11:57 |
| wwoods | notting: gotcha | 11:57 |
| -!- CheekyBoinc [n=SuL@fedora/CheekyBoinc] has joined #fedora-meeting | 11:57 | |
| warren | oldrepo at current URL's, newrepo at new URL, with old key content in oldrepo until it is resigned | 11:57 |
| mbonnet | warren: fine, then don't delete content from the oldrepo, it really doesn't matter | 11:58 |
| wwoods | right. once a package is resigned, it can be moved from current-repo to new-repo | 11:58 |
| warren | what do we call new-repo? | 11:58 |
| warren | we need to pick a URL to put it now | 11:58 |
| mbonnet | implementation detail. Can we get consensus on the approach first? | 11:59 |
| wwoods | we can certainly suggest it to FESCo/rel-eng | 11:59 |
| warren | I suppose this is OK, and the mirror churn happens later (not the same day as rawhide flood) | 11:59 |
| warren | mbonnet: it is an implementation detail, but we need it IMMEDIATLEY if we are going to sign updates to push | 12:00 |
| warren | so we might as well decide now | 12:00 |
| -!- mcepl [n=matej@ppp1053.in.ipex.cz] has left #fedora-meeting ["Ex-Chat"] | 12:00 | |
| wwoods | I don't think it's up to QA to make that decision - pick something sensible and ask f13 what he thinks when he appears | 12:00 |
| nirik | EBIKESHED | 12:00 |
| -!- J5 [n=quintice@c-76-24-17-105.hsd1.ma.comcast.net] has joined #fedora-meeting | 12:00 | |
| warren | ok, we can pick something to recommend | 12:00 |
| wwoods | nirik: exactly | 12:00 |
| wwoods | I'd like input from f13 / mdomsch whenever mirroring juju is involved | 12:01 |
| warren | I highly recommend we pick something to recommend | 12:01 |
| wwoods | so pick something | 12:01 |
| wwoods | I completely don't care | 12:01 |
| warren | http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Fedora/i386/os/ | 12:01 |
| warren | current location... | 12:01 |
| warren | http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Fedora/i386/os-new-key/ | 12:01 |
| warren | ? | 12:01 |
| warren | I don't care what it is | 12:01 |
| warren | but we should recommend something | 12:02 |
| warren | http://download.fedora.redhat.com/pub/fedora/linux/updates/9/ | 12:02 |
| warren | http://download.fedora.redhat.com/pub/fedora/linux/updates/9-new-key/ | 12:02 |
| wwoods | probably I'd pick something more like linux/releases/9.newkey/Fedora/... | 12:02 |
| wwoods | but, as I've said, not my area of expertise | 12:02 |
| warren | wwoods: if you do it at that level, then that implies the massive ISO's are also newkey | 12:03 |
| wwoods | offer those two suggestions to f13/mdomsch/etc. and see what they say | 12:03 |
| wwoods | warren: but there won't be any 9.newkey/Fedora/i386/iso dir | 12:03 |
| warren | wwoods: which will be confusing | 12:03 |
| wwoods | it's fairly clear - we never produced new isos with the new key | 12:03 |
| wwoods | we're basically recomposing the tree here. makes sense to apply the change at that level | 12:04 |
| wwoods | we'll make those two suggestions | 12:04 |
| warren | ok fine. | 12:04 |
| wwoods | and let mirrormanager/rel-eng experts tell us what makes sense | 12:04 |
| wwoods | because I'm not an expert on either. | 12:04 |
| warren | Note, it still might be possible logically that this plan of newrepo doesn't work. | 12:04 |
| warren | I haven't thought of a drawback yet | 12:04 |
| -!- Frankly3D [n=Frankly3@194.46.164.52] has quit [Remote closed the connection] | 12:04 | |
| wwoods | right | 12:05 |
| warren | I'll update the recommendation list after i get back from food | 12:05 |
| * nirik thinks it's complicated, but a better solution than the other one | 12:05 | |
| warren | seriously need food now | 12:05 |
| -!- che [n=rkastl@redhat/che] has quit [Remote closed the connection] | 12:05 | |
| wwoods | it's worth discussing, at the very least | 12:05 |
| warren | bbl | 12:05 |
| wwoods | yeah, it's lunchtime. let's summarize: | 12:06 |
| wwoods | 1) bad stuff happened and now we are stuck deciding between a bunch of things that suck. | 12:06 |
| wwoods | 2) mirror churn sucks less than forcing users to manually monkey with their keys. | 12:07 |
| wwoods | 3) we believe we can make the process fairly painless for the users by pushing out a fedora-release update, signed with the old key, that contains the new key and a new repo. | 12:08 |
| wwoods | 3a) the new repo is where we'll put all the packages signed with the new key. | 12:08 |
| wwoods | 4) since F9 original PackageKit handles key installation *really* badly, we may need to leave the current PackageKit update in the current repo | 12:09 |
| -!- CheekyBoinc [n=SuL@fedora/CheekyBoinc] has quit [Remote closed the connection] | 12:10 | |
| -!- nphilipp [n=nils@155.56.26.52] has quit ["Leaving"] | 12:10 | |
| wwoods | Things we plan to test: | 12:11 |
| mbonnet | wwoods: sounds good to me. I'm not sure we gain anything by pulling already-released updates anyway, so might as well leave the current repo as-is (with the exception of pushing the new fedora-release) | 12:11 |
| -!- hanthana [n=hanthana@124.43.42.81] has joined #fedora-meeting | 12:11 | |
| * jwilliam 9.1 | 12:11 | |
| wwoods | 1) Install F8; update to current (including packages with new keys) | 12:12 |
| wwoods | 2) Install F8 and current (pre-intrusion) updates; update with new-key packages | 12:12 |
| wwoods | repeat for F9. | 12:12 |
| wwoods | Expected results: system updates successfully (although it may ask you to import a new key). | 12:13 |
| wwoods | system continues to notify you of further updates in the new repo(s), signed with the new keys. doesn't ask for the key import again on subsequent updates. | 12:14 |
| wwoods | testing this is gonna be a PITA. | 12:14 |
| wwoods | we're gonna need scripts to create custom repos to test these scenarios. | 12:14 |
| wwoods | and maybe create fake test packages too. | 12:15 |
| wwoods | oh well. it's stuff we should have tools to test anyway. | 12:15 |
| -!- giallu [n=giallu@fedora/giallu] has quit [Read error: 110 (Connection timed out)] | 12:15 | |
| wwoods | wonder if skvidal has tools like this for testing yum | 12:16 |
| wwoods | I'll ask him about it. | 12:16 |
| -!- mccann [n=jmccann@c-65-96-163-181.hsd1.ma.comcast.net] has joined #fedora-meeting | 12:16 | |
| wwoods | okay. does the suggested plan make sense? | 12:16 |
| mbonnet | wwoods: does to me. | 12:16 |
| -!- themayor [n=jack@dsl081-200-018.nyc2.dsl.speakeasy.net] has joined #fedora-meeting | 12:16 | |
| mbonnet | wwoods: it doesn't deal with revoking the old key, but I think that's a different problem. | 12:16 |
| wwoods | yeah, that's Someone Else's Problem | 12:16 |
| wwoods | well, no. It's our problem too. We'll worry about that once we've got this part of the plan solidified a bit better. | 12:17 |
| wwoods | I'll try to summarize this suggestion for FESCo (which meets in ~2hrs) | 12:18 |
| wwoods | we'll see what comes out of that meeting. | 12:18 |
| -!- J5 [n=quintice@c-76-24-17-105.hsd1.ma.comcast.net] has quit [Read error: 110 (Connection timed out)] | 12:19 | |
| wwoods | and I'll ping skvidal about yum testing tools so we can make sure that updating will work as expected. | 12:19 |
| wwoods | okay. the time is late and lunch looms large. anything else we should discuss? | 12:19 |
| * wwoods taking that as a no | 12:20 | |
| wwoods | thanks for your time, all | 12:20 |
| --- Log closed Wed Aug 27 12:20:39 2008 | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!