11:00:45 [Users #horde-board]
11:00:45 [@bklang] [ duck_] [ mrubinsk   ] [ rlee0001   ] [ Shpoon] 
11:00:45 [ cjh   ] [ liamr] [ protagonist] [ ShadowSpawn] [ wrobel] 
11:00:45 -!- Irssi: #horde-board: Total of 10 nicks [1 ops, 0 halfops, 0 
          voices, 9 normal]
11:00:46 <@bklang> Good morning everyone
11:00:53 < rlee0001> Good morning.
11:00:57 <@bklang> or afternoon, or evening 
11:01:44 < mrubinsk> greetings
11:02:36 <@bklang> We've got what I think will be a pretty full agenda today so 
                   we'll only wait another minute or so to get started
11:03:14 <@bklang> Jan will not be able to join us today due to client 
                   obligations
11:03:24 < cjh> Hi folks
11:04:26 < duck_> hi
11:04:45 <@bklang> Alright let's get started
11:05:19 <@bklang> Welcome everyone, thanks for coming out to the November 
                   Horde Board meeting
11:05:46 <@bklang> The first item on the agenda is to continue our discussions 
                   of using Ohloh.net to host Horde downloads
11:06:21 <@bklang> We have had contact from an Ohloh.net employee who has been 
                   able to address some of our questions directly
11:06:48 <@bklang> Does anyone have any strong feelings one way or the other 
                   about using Ohloh.net, or has anyone's opinion changed since 
                   last month?
11:07:33 < Shpoon> The e-mail sent by the oloh guy did a good job of answering 
                   some of the questions I had
11:07:33 < mrubinsk> personally, I'm feeling more comfortable knowing they use 
                     Amazon on the back end
11:08:04 < Shpoon> and i know that some of the core team is excited for the 
                   download statistics
11:08:19 < cjh> I haven't had a reason to change my opinion from "table it" so 
                far.
11:08:19 < cjh> It's not a pressing issue for us right now, so let's give it a 
                few months
11:08:21 < Shpoon> i don't have any objection
11:08:37 < mrubinsk> that's fine with me as well
11:08:42 <@bklang> so it sounds like something we're interested in pursuing, 
                   just not a priority
11:08:47 < cjh> I don't have any objection either, it's just not a priority for 
                me right now
11:08:58 <@bklang> So unless some volunteer wants to step up and drive it we'll 
                   leave it at that for now
11:09:05 < mrubinsk> yea, I would say lets get the VCS stuff hashed out first
11:09:11 < mrubinsk> one major change at a time :)
11:09:27 < Shpoon> I agree - VCS is much more pressing issue
11:09:49 <@bklang> Alright, well since that segues nicely into another agenda 
                   item I'll skip around just a bit and bring up the next topic
11:09:52 <@bklang> Git migration progress
11:10:14 <@bklang> cjh, I think you're the most qualified to speak to this
11:11:07 < cjh> Jan's latest argument convinced me
11:12:00 < cjh> I think we should be aggressive about moving libraries over, 
                and put up with having some libraries in CVS and some in git
11:12:17 < cjh> so there are a number of libraries that can move over now 
                (Feed, Argv, etc.)
11:12:19 < Shpoon> You mean this argument? 
http://lists.horde.org/archives/dev/Week-of-Mon-20081103/023543.html
11:12:33 < cjh> Shpoon: yes, thanks for the pointer
11:12:39 < Shpoon> np
11:12:59 < cjh> and I think we should try to make clean cuts between cvs and 
                git for specific libraries or applications
11:13:11 < cjh> that'll be a bit more tricky with the mime stuff, but we can do 
                that
11:13:33 < cjh> So, that's my argument
11:13:39 <@bklang> maybe we should create some kind of tool to ease the 
                   transition between CVS and Git.  Some kind of script that 
                   will check out the necessary pieces from each repository and 
                   put them in the right place?
11:13:52 < cjh> I'm hoping I'll have some more time after today, but I'll still 
                have a six month old, so we'll see :)
11:14:10 < cjh> bklang: I don't know about automatic checkouts, but doing some 
                work on the fw-symlinks script makes sense to me
11:14:46 < mrubinsk> I've got to say, I'm excited about the move to git, but 
                     also apprehensive doing it in a mixed way....although I 
                     don't really think there is a clean way around it.
11:14:55 < Shpoon> It also looks like some of the mail commit messages still 
                   need to be tweaked.  The messags keep being sent with a 
                   subject of 'UNNAMED PROJECT'
11:15:16 < cjh> I feel like we should not put a ton of effort into usability 
                for people who are not comfortable using the vc backend 
                directly, for the next few months
11:15:22 < cjh> Shpoon: yeah, definitely
11:15:31 <@bklang> It seems to me the best way to minimize the pain is to put 
                   an official or semi-official emphasis on converting the 
                   Framework libraries 
11:15:35 <@bklang> prioritized in front of application modifications
11:15:39 < Shpoon> cjh: i heartily agree.  IMP is not going to be in a place 
                   for anyone to use for awhile
11:16:26 < cjh> bklang: I think that's right.
11:16:44 < Shpoon> Also need to write up some new git documentation for the 
                   website
11:16:45 < cjh> I think it also makes sense to, while focusing on moving 
                libraries over, to do incremental changes in CVS for the 
                applications
11:17:00 < cjh> so that HEAD apps can continue to work with the latest libraries
11:17:26 < cjh> and then when we do full-scale app conversion, they'll go over 
                to git
11:17:35 < cjh> a bit slower that way perhaps, but maybe smoother?
11:17:53 < cjh> IMP would go over right away, since it's already being ripped 
                into
11:18:10 < cjh> though the final IMP release for Horde 4 might have a different 
                app structure
11:18:54 <@bklang> Are any apps in Git today?
11:18:56 < mrubinsk> cjh: so am I correct in saying that the FW libraries don't 
                     go into git until they pass the points outlined in Jan's 
                     email?
11:19:10 < cjh> bklang: only half-baked apps in the hatchery
11:19:13 < cjh> mrubinsk: that's the proposal, yes
11:19:43 <@bklang> cjh: is there an updated CODING_STANDARDS for use with Horde 
                   4? I'd be happy to convert some libraries but I don't yet 
                   know what changes need to happen
11:19:45 < cjh> The other open question could be the 
                horde/horde-hatchery/horde-support structure, with framework in 
                the main git repo, but no one followed up with any actual 
                arguments on that one.
11:20:00 < cjh> bklang: no, it's not finalized yet
11:20:59 < Shpoon> Are we going to use submodule support?
11:21:22 < cjh> That's a good todo item (coding standard)
11:21:34 < cjh> Shpoon: last I looked, so far as I understood it, it didn't 
                really help us
11:21:52 < cjh> but I could be wrong or it could have changed. I'm also not 
                sure how much up-front work it takes
11:22:02 < mrubinsk> I thought that the submodule support wasn't all that 
                     stable...or maybe I'm thinking of Hg
11:22:05 < Shpoon> http://en.wikibooks.org/wiki/Source_Control_Management_With_Git/Submodules_and_Superprojects
11:22:12 < Shpoon> git submodule support was only added in 1.5.3
11:22:14 < Shpoon> so it is very new
11:22:28 < Shpoon> but that graphic in that link seems to be exactly what we 
                   are looking for
11:23:09 < Shpoon> the only issue is that you have to commit changes twice - 
                   once to the submodule and once to the superproject
11:23:16 <@bklang> Shpoon: is this something that is client-side only?
11:23:34 < cjh> That seems like a pretty big limitation
11:23:48 < cjh> especially since the obvious use of this is apps
11:23:54 < cjh> so we'd need to commit twice per app
11:24:02 < cjh> which adds up quickly especially in early stages of horde 4
11:24:02 < mrubinsk> If it's that new, how do we feel about requiring that new 
                     of a version for people to use our repo?
11:24:19 < Shpoon> I have 0 problem requiring the latest git
11:24:19 < cjh> I don't think requiring a recent version of git is an issue
11:24:25 < cjh> git is moving pretty quickly
11:24:36 < mrubinsk> k
11:24:57 < mrubinsk> ah, I'm on 1.6 anyway ;)
11:25:11 < Shpoon> i guess submodule support doesn't add that much
11:25:13 < Shpoon> for us
11:25:23 <@bklang> what would it gain us?
11:25:49 < Shpoon> since, again, this is only an issue for development purposes.
11:25:49 < cjh> the ability to check out only a few applications instead of the 
                whole repository
11:26:18 < Shpoon> But given the git packing mechanism, it is not that big of a 
                   deal to check out a whole repo anymore
11:26:27 < cjh> I agree
11:26:34 < Shpoon> That's the big limitation with CVS
11:27:12 <@bklang> is submodule support something we could move to later?
11:27:25 <@bklang> At this point I'd rather not have it slow down our progress 
                   if possible
11:27:25 < cjh> the way it's currently done, probably not
11:27:40 < cjh> we'd have to split things out and lose history, as I understand 
                it
11:27:46 < cjh> I don't think it's something we need.
11:27:49 < Shpoon> nope.
11:28:02 <@bklang> and regular users can still install individual apps, or pear 
                   install individual framework libraries
11:28:13 < cjh> or pear install individual apps :)
11:28:16 < Shpoon> the only issue is that people who want to install a 
                   development version for production will have a lot of extra 
                   cruft
11:28:17 <@bklang> even better :)
11:28:28 <@bklang> well that doesn't bother me that much
11:28:29 < cjh> Shpoon: we can still provide snapshots
11:28:47 < Shpoon> sure - but it is nice to run a devel version and be able to 
                   do something like 'git pull' to update
11:29:09 < Shpoon> but again, that's something that can easily be worked around
11:29:12 <@bklang> the only disadvantage being you have all apps, which can be 
                   disabled in the site via permissions and registry
11:29:13 < Shpoon> and is not a showstopper
11:29:47 < cjh> yup. Okay, this one is closed I think.
11:29:59 <@bklang> Great!  Let's move on to the next item
11:30:01 < cjh> Anyone want to advocate for a separate framework repository?
11:30:26 < Shpoon> [silence]
11:30:33 <@bklang> (not I)
11:30:36 < mrubinsk> nope
11:30:42 < cjh> right then.
11:30:49 < Shpoon> and yunosh wouldn't
11:30:53 < cjh> Okay, let's summarize quickly:
11:31:09 < cjh> 1. focus on converting libraries to horde 4, reviewing them, 
                etc.
11:31:20 < cjh> 2. move libraries to git as they're converted
11:31:25 <@bklang> I think CODING_STANDARDS is a prereq to #1
11:31:38 < cjh> 3. remove libraries from cvs when they've been moved
11:31:45 < cjh> bklang: good point, make that #0
11:31:54 < cjh> 4. work on the fw-symlinks script to handle git + cvs
11:32:09 < cjh> 5. make small tweaks to apps for them to work with framework 
                from git before converting over individual apps
11:32:34 < Shpoon> For 5 - I think this is something the autoloader can really 
                   help with
11:32:58 < cjh> well, if we symlink them into the same directory structure, 
                it's not even that big a deal
11:33:05 < Shpoon> good point
11:33:06 < cjh> or we can have two class trees that the autoloader looks at
11:33:11 < cjh> either way is pretty easy, yes
11:34:17 < Shpoon> Well, I might be able to work a bit on 
                   CODING_STANDARDS/website documentation since I would like to 
                   move my libs over to Horde git ASAP
11:34:35 < Shpoon> namely MIME and Horde_Imap_Client
11:35:01 < Shpoon> but for the MIME lib, the old MIME lib needs to continue to 
                   live in CVS for now
11:35:05 < cjh> i think some libs can go over now; we just need the coding 
                standards for large-scale conversion
11:36:04 <@bklang> Since you mentioned Horde_Imap_Client, would you like to say 
                   a bit about the progress and functionality there?
11:36:32 < Shpoon> Sure.
11:36:45 < cjh> nice seque!
11:36:50 < cjh> err, segue
11:37:24 < Shpoon> I think the lib has definitely reached the point where I 
                   would consider it 'beta'.  I think the problem now is 
                   catching the weird fringe cases that break individual IMAP 
                   servers
11:37:58 < Shpoon> I'm working with selsky on an issue with Cyrus/murder that 
                   not only prevents CONDSTORE from being used, but makes the 
                   entire IMAP server unresponsive
11:38:38 < Shpoon> Anyway, all caching is now handled by the client library - 
                   so that means that if other clients use the lib in the 
                   future (i.e. ingo), imp would benefit from the caching
11:39:18 < Shpoon> And, if QRESYNC is supported on the remote IMAP server, we 
                   now support flag caching (which was not possible with 
                   current IMP caching code)
11:40:04 < cjh> Nice. What's the best way for us to help out now? Get it into 
                Git along with the new IMP and use it in devel environments?
11:40:08 < Shpoon> Those are probably the big highlights of the recent 
                   development
11:40:27 < Shpoon> As mentioned previously, while rewriting IMP to handle the 
                   new IMAP lib
11:40:47 < Shpoon> I realized that now would be a good time as any to also 
                   rewrite the MIME Contents-ish rendering code
11:41:05 < Shpoon> since 1.) there's some c-client specific code buried in the 
                   internals
11:41:27 < Shpoon> and 2.) the MIME change would also severely break IMP, so 
                   best to only break once
11:41:50 < Shpoon> But once I get basic rendering down, I will probably add to 
                   git
11:42:20 < Shpoon> The reality is a lot of stuff will probably be broken at 
                   this time - mainly dealing with rendering of all MIME types 
                   other thant what I would consider "basic" messages
11:42:37 < Shpoon> e.g. text/plain, text/html, multipart/alternative, image/*
11:42:53 < cjh> yup
11:42:54 < Shpoon> Stuff like PGP/S/MIME and other more complex MIME messages 
                   will most certainly be broken
11:43:11 < Shpoon> Just because those were the fringe cases that made the MIME 
                   library so unusable in the first case
11:43:36 < cjh> Well, better get them working before we're too happy with the 
                API... :)
11:43:39 < Shpoon> So that's where input from others will be most useful - 
                   because I only have so many test messages
11:44:20 < Shpoon> Well, for the PGP stuff, it mainly has to deal with the fact 
                   that we not only need to keep the MIME headers of a certain 
                   part, but it needs to be *exactly* the same.
11:44:55 < Shpoon> We can't parse them and then rebuild later.  Which is a big 
                   PITA when dealing with large messages (since we are just 
                   chewing up memory and there is no other reason to have them 
                   around)
11:45:30 < Shpoon> But that's not going to change API very much, since it will 
                   be a function that only PGP will use in the first place
11:45:59 < Shpoon> Right now I'm trying to clean up the MIME_Viewer and 
                   IMP_Contents interaction to figure out how best to render 
                   the various data views
11:46:09 < Shpoon> i.e. inline vs. popup window vs. downloading
11:46:19 < Shpoon> that's where some of the major limitations are now.
11:46:40 < Shpoon> so that's my long winding update
11:46:45 < Shpoon> winded update.
11:46:53 < cjh> I'm going to have to take your word for it :)
11:47:11 < Shpoon> i guess so :)
11:47:27 <@bklang> any questions on what Shpoon said?
11:47:53 < Shpoon> all of this probably won't mean much until people start 
                   using the new IMP and everything is broken :)
11:50:00 <@bklang> alright well let's move on then
11:50:07 < duck_> SQL Share performance?:)
11:50:10 < cjh> It'd be great to get some of the institutions involved in early 
                testing if possible
11:50:12 <@bklang> The only agenda item left is SQL Share performance
11:50:13 < cjh> sounds like we've got a ways to go first, though.
11:50:24 < duck_> With shares - the main problem is how to good organize and 
                  index data. Actual bitwise operations does not uses indexes. 
                  But split permission data to use integers is not good as it 
                  seems. As soon as Mysql will predict that is possible that 
                  more than 40% of data matches the criteria, it will scan 
                  table data instead of using indexes. For example in Ansel 
                  browsing: data are mainly public and...
11:50:25 < duck_> ...permission values are 90% the same.. at least in my case.
11:50:33 < duck_> Then we use NULL values for parents. NULLs are not values at 
                  all. So we again full table scan will be used.
11:50:42 <@bklang> I guess Duck was ready to talk about that
11:50:43 <@bklang> :)
11:51:21 < duck_> My test shows that the we get better performance by splitting 
                  permission columns by permission type. Plus remove null vales 
                  with empty data. And then create a combined index on common 
                  processed where conditions (ex. Owner + default_*_show 
                  columns). Here mysql was using the index.
11:51:25 < duck_> But I am really open to suggestions.
11:51:44 < duck_> With MySQL, perhaps, it will be better to split permission 
                  data and attributes into separate tables (one 2 one 
                  relation). Raw data seeks will be much faster - it will 
                  avoiding moving around all not searchable attributes and we 
                  can have fixed row length. But I need to test this.
11:52:43 < duck_> Yes, I spent a second or two thinking about this :)
11:53:16 < cjh> Where along this path is the current patch on #7363?
11:56:28 < duck_> Yes. but I am still not 100% sure, if this is the right 
                  direction. I hoped to receive some feedback and ideas.
11:57:01 < cjh> It's one of those things on my list, but I haven't been able to 
                put time into testing yet.
11:58:16 < mrubinsk> Splitting out the perms from the attributes seems like a 
                     good way to go - but I also haven't really dug into it too 
                     much
11:59:06 < duck_> neither I. I will try to create some test cases and compare 
                  the results and append it on this ticket
11:59:27 < mrubinsk> ...and regarding the NULL values, I believe there are 
                     issues with portability with Oracle - empty strings are 
                     seen as NULL, so by keeping them NULL we avoid some of the 
                     hassle there
11:59:41 < mrubinsk> plus, in some cases, NULL is an appropriate value IMO
11:59:57 < mrubinsk> a top-level share has no parent, it's null
12:00:23 < duck_> as said. NULL are not a value at all :)
12:00:53 < mrubinsk> exactly....there is no value for a top level share's parent
12:01:52 < rlee0001> Unless I'm mistaken, NULL means "Unknown Value", not "No 
                     Value". :)
12:01:58 < duck_> yes but this implements a full table scan.. using empty 
                  value, for the coding view is the same how we mark the top 
                  value. for example only with ":"
12:02:16 < duck_> but for DB internals is completlly a different story
12:03:32 < cjh> It seems to me that this discussion should probably continue 
                either in the ticket and/or on list
12:03:54 <@bklang> Alright, unless anyone else has anything to add...
12:03:55 < mrubinsk> There are also portability issues to consider....yea, good 
                     idea cjh
12:04:26 < duck_> rle0001 not relly for a DB the "NO value" *IS* an value (box 
                  with noting inside). And NULL is a black hole.
12:05:07 <@bklang> Alright well thank you all for coming out today.
12:05:16 <@bklang> I'll post the transcript from today's session to the mailing 
                   list
12:05:23 < cjh> Thanks all
12:05:26 <@bklang> We'll continue the SQL share performance discussion on the 
                   developer lists and ticket system
12:05:40 < cjh> one postscript - I'm not against a mysql-specific driver for 
                shares if it's significantly faster
12:05:44 < mrubinsk> sounds great
12:05:57 < mrubinsk> is the bitwise issue only with mySQL?
12:06:16 <@bklang> See you all on December 2nd for the next Horde Board meeting
12:06:19 < protagonist> err, hi folks!  I'll read the scrollback...
12:07:10 < mrubinsk> heh - Hi protagonist :)
12:07:43 < protagonist> this meeting is right during my commute, so I always 
                        forget it!  I'll have to program it into my cell phone 
                        reminder
12:07:53 < cjh> :)
12:08:38 < cjh> mrubinsk: I'm not sure
12:08:53 < cjh> I'm also pretty sure that mysql is adding bitwise indexes in an 
                upcoming version. not sure if that was 5.1 though.
12:09:54 < mrubinsk> k - thanks
12:10:05 < cjh> Thanks Ben K., thanks again all.
12:13:48 < liamr> hey - can I ask a question about Horde_IMAP_client?
12:14:02 < Shpoon> liamr: go ahead
12:14:14 -!- rlee0001 [n=rlee@rrcs-71-41-149-67.sw.biz.rr.com] has left 
          #horde-board []
12:15:03 < liamr> will it replace c-client (like squirrelmail / roundcube's 
                  PHP-native IMAP stacks)?  or be a wrapper for c-client?
12:15:27 < Shpoon> completely replace.
12:15:48 < Shpoon> c-client will no longer be needed.  Except for POP3 
                   connections since I don't have the desire to write a POP3 
                   client right now
12:16:31 < Shpoon> one less library/dependency to worry about
12:16:33 < liamr> and this is because it's hard to guaranty consistent behavior 
                  across PHP IMAP compiled against different releases of 
                  c-client?
12:16:35 < liamr> k