irc log 28.4.2002

Zapata, 28.04.2002 23:47

Log of the meeting on 28 april 2002

[idefix] agenda?
[rk] hurryhurry
[idefix] - producer
[rk] code design issues
[rk] may be
[idefix] code design or coding design
[idefix] ?
[rk] aeh?
[idefix] discussing about architecture or about the style of the code
[rk] code design, like zapata talked about freemarker factory before, and my issue would be to talk about lib/code splitting
[rk] architecture
[idefix] -logging
[idefix] -multipart/formdata-probs with ie6
[idefix] -multilinugual
[idefix] -producer-daemon
[idefix] this would be my points
[Zapata] I'll briefly talk about my work on localization/producers...
[idefix] jepp
[Zapata] and will announce a couple of small proposals...
[Zapata] I think the details of everything can go over the mailing list...
[Zapata] and multilingual isn't about producers as well?
[zert] waht about loudeye-mir relationship?
[idefix] and perhaps about your plans
[Zapata] yeah
[Zapata] and my timetable...
[idefix] jepp
* idefix feels sorry thet he didnot had time to help zapata
[Zapata] never mind...
[idefix] ok Zapata do you want to start?
[Zapata] ok...
[Zapata] what I've basically done so far is:
[Zapata] - set up a simple localizer structure
[Zapata] - rewrite producers (basically splitting up functionality over more classes and making the actual producers configuratable)
[Zapata] I'm going to continue this... still some producers need to be done...
[Zapata] and for the moment I'm focusing on achieving what imc bolivia wants...
[idefix] cool
[Zapata] when I get a little bit more time I'll write detailed doc etc...
[Zapata] (btw what bolivia wants includes what basque country wants I think...)
[rk] sounds good.
[Zapata] problems I've encountered so far:
[idefix] adnd imc-germany btw
[Zapata] idefix: what does imc germany want?
[Zapata] problems:
[Zapata] - not so great exception handling
[Zapata] - funcionality that is in the wrong places (like the formatting of dates inside entities, stuff like that)
[idefix] imc germany wants to have regional startpages
[Zapata] - templates that are buggy...
[Zapata] - db changes I didn't know of... (mediatypes table had undergone some changes...)
[Zapata] [/end of problems]
[zert] i think so, Zapata . Bolivia's and Basque's wants are similar.
[idefix] germany things can be done by idefix
[Zapata] one of the proposals I want to do is for structuring topics to include "city-topics" btw...
[Zapata] but more on that later...
[Zapata] language specific producers are quite easy... just have multiple xxx-producers in your localizer with different language bundles...
[Zapata] but as I said I will write a mail to the mailinglist detailing my work and how to use it...
* mjordan is afk for a minute
[Zapata] that's my report...
[idefix] exception-handling: a prof of my university wrote a really good exception-handling framework
[Zapata] idefix: I think it's basically a matter of discipline...
[idefix] that propagates all exception through all tiers up to the top
* mjordan is back again.
[idefix] at the top you have really detailed stacktrace
[Zapata] right now when an exception occurs, it's very hard to find out where it was caused and why...
[Zapata] because:
[Zapata] 1. exceptions get raised in "catch" clauses...
[idefix] thats my point
[Zapata] 2. some errors don't raise exceptions (like missing config settings...)
[idefix]  http://www.tfh-berlin.de/~knabe/java/multex/
[Zapata] ah... was about to ask for an url :-)
[Zapata] post it on the list too btw :-)
[idefix] its a really good concept i think
[idefix] jepp
[Zapata] causal chains sounds good :-)...
[Zapata] ok... the key features look great...
* idefix could start refactor the code with this framework
[Zapata] ok...
[Zapata] let's discuss it first a bit over the mailing list I'd say...
[idefix] sure
[Zapata] but at first sight it certainly looks great...
[Zapata] lets move on...
[Zapata] next topic?
[idefix] logging
[idefix] it is really crazy: we use three or four logging apis in our code
[Zapata] yup...
[idefix] i would suggest to start onlly using one api: log4j
[Zapata] so we should pick one and only use that one...
[idefix] it is really the most advanced loggin-api i know
[Zapata] ok... I'll look at log4j then...
[Zapata] got an url?
[idefix] every class has a public final static Logger
[idefix] jakarta.apache.org/log4j
[Zapata] you mean you want to give every mir class a final static Logger?
[idefix] an there is a single point of config in config.properties
[idefix] jepp
[idefix] so these logger are really godd configureable
[Zapata] well... would we differentiate between mircoders/* and mir/* ?
[idefix] you cna set a propety only to log the databse-things with logging-level debug
[Zapata] (since mir/* does not know anything about config I think)
*mumei* does anyone speak in channel here?
[idefix] it knows
[idefix] all the database-stuff and so on
[Zapata] the problem I'd have with this is that "utility classes" should not do their own logging, but defer the logging to the classes that use them...
[idefix] that can be done too
[Zapata] but I'd have to think about this a bit more...
[Zapata] ofcourse I'm all in favour of a good logging framework using only 1 api :-)
[idefix] the Logger-class is instatiatied liek this: Logger = Logger.getLogger(CLass.class)
[idefix] so the Class can be different to Class where logger logs
[Zapata] yeah...
[Zapata] but as I said for utility classes...
[idefix] but pehaps you should have a look on the tutorial on the log4j-website
[Zapata] ok
[Zapata] I'll look at that..
[Zapata] and we'll discuss further on the mailinglist...
[idefix] next topic?
[Zapata] ok
[Zapata] multipart stuff in ie6?
[idefix] rk?
[rk] yes
[idefix] ok just a report
[rk] sorry, i am not really present today
[idefix] we or better the oreilly-package for multipart has a bug
[idefix] with ie6
[idefix] mh tries to fix it.
[Zapata] I also think the oreilly package caused problems for .nl with "accents" (umlauts etc)
[idefix] but i dunno if it works
[idefix] it he has no access, we have to change the code of the oreilly-package
[Zapata] at one time in the future I would like to write our own stuff for the multi-part stuff..
[Zapata] but I think other things are more important?
[idefix] Zapata: my plans are other
[idefix] jepp
[idefix] just a report
[Zapata] idefix: in 1 sentence :-) what are your plans?
[idefix] i want to use the stuts-framework with mir
[idefix] struts
[Zapata] and this includes support for this problem?
[idefix] i think so, but it is much more than this
*** mh has joined #mir
[idefix] ah mh
[Zapata] yeah, I know, you mentioned struts before...
[mh] hi
[Zapata] hi mh
[mh] sorry I forgot was falafel essen
[mh] hi zapata
[Zapata] :-)
* idefix has a meeting now with his mitbewohner...
[Zapata] ok... so far for the multipart problems?
[Zapata] idefix has to go?
[idefix] mh?
[mh] can someone list topics
[mh] ja?
[idefix] jepp, but i am logging
[idefix] mh: multipart-probs
[mh] ah
[idefix] i will be back later this night
[Zapata] ok...
[idefix] sorry
[Zapata] cu then
[mh] ok I tried to use the filter method, put it only takes posted parameters.. it ignores the get stuff
[Zapata] mh: what exactly are the problems?
[mh] and looking at the cos.jar code, I don't think it would make a difference anyway,
[mh] mom. I'll send a link
*** init has joined #mir
[mh]  http://www.servlets.com/soapbox/bugs.html, search for opera or boundary
[Zapata] ok
[mh] exploorer 6.x has the same problem too.
[Zapata] oh, it's the same as the opera problem... I see
[mh] so we need to hack com.oreilly... or find a another solution
[Zapata] yeah...
[Zapata] I see...
[Zapata] ok...
[Zapata] more on your efforts?
[Zapata] or next topic?
[rk] lib/code split?
[Zapata] ok
* rk has to go soon
[mh] I"m going to try to patch the code. struts has a nicer interface.. but that s too much
[mh] is john there?
[Zapata] nope...
[Zapata] I told him about the meeting, but he ain't here...
[mh] ah. I had a question about OutputMir servlet.. why whouldn"t it be a ServletModule?
[rk] ok, i have not prepared anything, but i just wanted to mention, that i want to reconsider the code/lib split
[mh] rk: can you elaborate?
[rk] yes
[Zapata] mh: I think john doesn't really know the mir code too well... send him a mail please explaining how you would have had it done...
[rk] i would really like that Mir is usable for non indymedia purposes
[mh] ok I"ll do that.
[mh] rk: we're currently installing it for PGA. on brazil
[Zapata] my localization effort / producer rewrite effort has that in mind too...
[Zapata] maybe we should one day come to a definition as to what mir is good for (and what not)...
[rk] that means, we should move all general things into the lib layer and so enable others to use mir as a framework for their purposes
[mh] agreed
[rk] the concrete thing i'll be doing is to move the current pre_mir based nadir.org/aktuell to mir
[rk] i have to rewrite some producers, as it is slighly different than mir
[rk] zapata, may be we should join efforts on rewriting the producers
[Zapata] rk: coordinate this with me please :-) as I'm busy with them... better wait till the "new" producer framework is in the main branch...
[Zapata] :-)
[rk] :)
[rk] ok, so for relaunching nadir/aktuell i will use your code-base
[rk] mh, there was one thing unresolved before in HTMLTemplateProcessor with a static name of the servlet, remember?
[Zapata] FYI: as with my producer rewrite, the HTMLTemplateProcessor is not used anymore :-)...
[mh] ja. that really should wait till we get a better config system. but I think it"s not too bad anymore (the static name) or?
[rk] i'll see, i write everything down, that's not "general" enough to run nadir.org with mir :)
[Zapata] nadir is just publising and no open publishing, right?
[rk] closed posting :)
[Zapata] hehe
[rk] but we also need some geographic information, integration of other things on the site, and things like that,
[rk] a link collection database, etc.
[Zapata] rk: write the requirements for nadir...
[Zapata] to the mailing list...
[Zapata] and then we can see how to integrate such functionality...
[rk] ok, i'll do that in a silent moment :)
[Zapata] hehe
[mh] next topic?
[rk] that's about it from me, and i have to go, cu all, will read the protocol.
[Zapata] ok
[Zapata] bye rk!
[mh] cu. rk ripping fo you now
[Zapata] lets discuss what init wants to discuss...
[Zapata] which is multilingual stuff... right?
[Zapata] init?
[Zapata] init is idle for 15 minutes...
[mjordan] init is at runlevel 0 =B)
[init] since we are working on a mir version for the pga-site which is in 6 languages , I just wanted to know, how far the multilingual stuff ios and what the basic idea behind it is.
[Zapata] ok...
[Zapata] well...
[Zapata] I would like to know what PGA's requirements are?
[Zapata] (so I might include them in my work for bolivia etc)
[init] I think what we need for the pga site is the abilety to produce some or all pages in different languages, but only the navigation.
[Zapata] ok
[Zapata] that's basically what I'm making...
[Zapata] having structured translation of "content" is a matter for the future...
[Zapata] but startpages etc. in multiple languages is what I'm now doing...
[init] mir will not have to track stuff like translating articles or versions in different languages of one article, at least for now...
[Zapata] are there any other things that PGA wants? (i.e. specific producers) ?
[mh] (eventually we should be able to handle translated versions of articles. but that's not required for PGA right now)
[Zapata] mh: exactly, but that will have far reaching consequences... i.e. datamodel changes etc...
[init] well, more flexible categorisation, e.g. sub-topix would be nice...
[Zapata] multilingual producers are relatively easy to make...
[Zapata] init: can you give a concrete example of sub-topics?
[mh] the multiling produced pages should of course all come from one i18n'd template but that goes without saying
[Zapata] mh: ofcourse :-)
[mjordan] Zapata: Environmental (Nuclear power, recycling, mobilization)
[init] Zapata: super-topic: pga-capains sub-topics: waterright, farming, education
[Zapata] ok..
[mh] Zapata: think of a hierarchical tree. every topic can itself contain other topics. thing of it as categorization and a generalization
[Zapata] well...
[Zapata] a 2 layer topic structure is not enough?
[Zapata] trees are nice for programmers, but terrible for users in general :-)
[mjordan] Zapata: if every topic can have subtopics all you have to do is make sure there are no cycles.
*** rk has quit IRC (Ping timeout: 240 seconds)
[Zapata] anyway, we can use something similar for city-topics... so it would be nice to think of something that can cover both...
[init] Zapata: 2-3 month ago you sugested somethings like categuries that every mir-site can configure for themseves, I realy liked that. that could also help to have categurisation within one topic, which would be equal to subtopics.
[mjordan] Zapata: I think trees are great for users. Think of directory trees, organisation of the pockets in one's luggage etc.
[Zapata] I was thinking of a 2 layer topic structure: category -] topics...
[Zapata] but anyway... I see we want to have trees...
[Zapata] then trees it will be I guess... :-)
[Zapata] I'll write a detailed proposal etc...
[mh] oh. while we're on the topic (no pun intended). a much asks for feature (twice a day) is a feature/newswire list like the open posting list.
[Zapata] mh: yeah, in .nl too :-)
[Zapata] mh: I'll make them... should be easy with the new producers :-)
[Zapata] mh: and I have a couple of other producers in mind that the .de people will love too :-)
[init] Zapata: 2 layers is great for most purposes, n-layers is great for ALL purposes, mir could also be used to create big multi-media web-archives.
[Zapata] init: yeah, I agree...
[Zapata] anyway... the "topic tree" will probably not be made on the short term... but I think the current topic structure can be used for a while longer...
* zert has to go
[zert] bye ;(
[Zapata] bye zert!
[mh] cu
[Zapata] priority lies on multilingual producers... about every site seems to want that...
[mh] yup.
[Zapata] ETA: wednesday evening...
[Zapata] ok...
[Zapata] next topic?
[init] ok - one more question, we allready have the category 'language', at least in the publishing-form, does this information make it's way into the database allredy ???
[Zapata] yeah
[Zapata] right mh?
[Zapata] (actually indy.nl already has this...)
[mh] don't know. most likely yes
[Zapata] but the admin interface does not support adding of languages...
[Zapata] but that can be done manually...
[Zapata] but I don't think that information is used yet...
[Zapata] but it might be used by templates I think...
[init] so if we start entering trons of articles for pga we can allready specify their language in the database and use this information in new producers?
[mh] nope
[Zapata] (i.e. mentioning the language of an article on the start page)
[Zapata] init: yeah, I guess so...
[mh] init: yes I think so
[Zapata] I'll look into it, and will try to include it in my new producer stuff...
[Zapata] (if it's not already there...)
[init] great, because they want to start feeding the db as soon as possible.
[Zapata] ok...
[Zapata] now next topic?
[Zapata] (producer deamon)
[mh] ya. it's relly bad that the webapp does the producing. the latency is too higg
[mh] ya. it's relly bad that the webapp does the producing. the latency is too high
[Zapata] ok...
[mh] it really should be queued, etc.. and asynchronous
[Zapata] I was working on that...
[mh] like the way the VFS works on an OS.
[mh] was?
[Zapata] am :-)
[mh] :)
[Zapata] I'm thinking of:
[Zapata] 1. making a "producer engine" with a producer queue
[mh] so that the servlet sends the job and that's it. it returns immeadiatly
[Zapata] 2. having the producer queue get serviced by a seperate thread...
[Zapata] but I wasn't thinking of making a seperate java app for this (dunno if you mean this?)
[Zapata] maybe some UI might be needed for the queue...
[Zapata] i.e. looking at what's in the queue
[Zapata] and possibly removing items...
[Zapata] (all this is very easy to implement ofcourse...)
[mh] zapata. I like the separate thread idea.
[Zapata] but do you want a seperate app for this?
[Zapata] (I don't really namely)
[mh] if we need to , we could make a separate app in the future.
[Zapata] ok
[Zapata] cool...
[Zapata] another thing about the queue:
[Zapata] suppose the servlet gets killed...
[mh] having it as a part of the same app makes it much easier to install, etc..
[mh] cool idea
[Zapata] what happens to the queue
[Zapata] ?
[Zapata] (i.e. maybe we ought to make the queue somewhat persistent)...
[Zapata] and another issue:
[mh] Zapata: ya that's an idea.
[Zapata] right now mir doesn't know very well which producer tasks must be done...
[mh] yup
[Zapata] maybe we should have a localizable way to do this as well..
[mh] that is also a problem.
[Zapata] (there's an ide in my head...)
[Zapata] (there's an ideA in my head...)
[Zapata] (but I need to work it out...)
[Zapata] and another issue:
[mh] but back to the queue for a minute, it really should be able to have a status UI, kill a job, stop, start, etc..
[Zapata] yeah, that's what I said about UI :-)
[mh] i don't think that needs to be localizable
[Zapata] mh: it would...
[mh] what do you mean with wich producer tasks?
[Zapata] what doesn't need to be localizable?
[mh] don't know :)
[Zapata] hehe
[Zapata] ok...
[Zapata] let me explain what needs to be localizable...
[Zapata] suppose indy.nl has content html pages as well as, say, content pdf pages... and those in 5 languages...
[Zapata] that amounts to 7 producer task per content entity...
[mh] yes
[Zapata] indy.de might have a completely different need...
[Zapata] and so mir has different things to keep track of for .de and .nl...
[Zapata] and so the tracking of which producer tasks need to be done will have to be done localizable...
[Zapata] (and this has nothing to do with the queue...)
[Zapata] (but more with the is_produced flag of content)
[Zapata] (which I think is not a good way to do this)
[Zapata] I furthermore think...
[Zapata] there should be a cron job putting jobs into the queue
[Zapata] for stuff that is edited in the admin interface...
[Zapata] so the admins don't have to manually force produce stuff...
[mh] or for example the title of an article, changes, the open posting list page needs to be re-produced..
[Zapata] yeah
[Zapata] exactly...
[Zapata] and mir should be so smart as to not to reproduce all the open posting list pages when only 1 article changes...
[Zapata] but I have somewhat of an idea...
[Zapata] but I need to think it over ...
[Zapata] this is a difficult part...
[Zapata] (especially the localizable bit)
[Zapata] the basic idea btw is to have every producer keep track of what it has done...
[Zapata] and then let the producer decide what has changes since...
[Zapata] anyway... I'll think of something and write about it in detail on the mailing list...
[Zapata] the queue will be quite easy in the meantime...
[Zapata] (the first parts of it are already in my localization branch)
[Zapata] ok...
[Zapata] next topic?
[mh] what is it?
[Zapata] well...
[Zapata] I was going to say something about some new proposals...
[Zapata] but I have already mentioned them...
[Zapata] namely topic structure and keeping track of producer tasks...
[Zapata] so
[Zapata] maybe...
[Zapata] we can discuss a little bit of the html filtering thingy...
[Zapata] oh...
[Zapata] wait...
[Zapata] there was this other thing...
[Zapata] I've already discussed this a bit with rk...
[Zapata] I want to seperate entity and templatemodel again...
[Zapata] and...
[Zapata] extend my "adapter factory" in FreemarkerGenerator.java...
[mh] ok. can you elaborate?
[Zapata] and include caching of properties...
[Zapata] pl
[mh] on the entity template thing
[Zapata] whoops
[Zapata] ok
[Zapata] right now, entity supports the interface needed to make it accessible in freemarker...
[Zapata] but...
[Zapata] suppose at one time we want to ditch freemarker...
[Zapata] that would be very ugly...
[Zapata] therefore I want to do what I have done in the producers too..
[Zapata] namely to isolate all the freemarker dependencies...
[Zapata] right now, "my" producers build a vanilla java map...
[Zapata] and the FreemarkerGenerator translates this (in a smart way) into the freemarker map...
[Zapata] by way of an adapter factory...
[mh] ok
[Zapata] this adapter factory knows right now about Lists, Maps and Strings...
[mh] nice.
[Zapata] and I might extend this into Entities...
[mh] i like that too.
[Zapata] and at the same time, I might employ smart caching...
[Zapata] the factory already works on a "at demand" basis: only values that are asked for get "adapted"...
[Zapata] ok...
[Zapata] that's it...
[mh] a little off topic, but what I really want is more transparence, so that a template person will know what data is available. right now other than the DB fields, know one can know what is avail. through the TemplateModule get() method
[Zapata] yeah...
[Zapata] agree...
[Zapata] what I have done right now is:
[Zapata] raise exceptions when a key is not available...
[Zapata] that's very handy:
[Zapata] errors in templates are easily detectable now
[Zapata] freemarker gives you an exception message+stacktrace into html comments
[Zapata] (I did the same thing for config properties btw...)
[Zapata] (VERY handy :-)
[Zapata] but
[Zapata] in general
[Zapata] doc must be written for templates...
[Zapata] and we might generate this doc in part, by checking which keys are available...
[Zapata] ok?
[mh] ok
[Zapata] so...
[Zapata] html filtering?
[mh] sure
[Zapata] ok...
[Zapata] we already fought over this in Berlin ;-)...
[Zapata] but...
[Zapata] I don't see how a regexp filter can:
[Zapata] - specifically ALLOW certain tags
[Zapata] - and allow certain paramters in certain tags
[Zapata] - and automatically close unclosed tags
[Zapata] but maybe you know how to do this?
[mh] regexp will do the first 2. closing will be in the same method
[Zapata] furthermore, I'd like to note that with a localized html filter, we can both be satisfied... I'll use my own html parsing filter for .nl and you can use your regexp for .de :-)
[mh] regexp. is a standard
[mh] sure
[Zapata] I know regexp is... but they are very hard to maintain... I don't consider them usefull in this context... but to each his own, etc...
[Zapata] I'm curious however how your regexp's will look :-)
[mh] ok :)
[mh] I have to go. looking forward to reading what I missed in the logs
[Zapata] ok
[Zapata] that's end of meeting then ...