[Discussion] Designing the new Rendering/Parsing component/API

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Discussion] Designing the new Rendering/Parsing component/API

vmassol
Administrator
Hi,

I've started working on designing the new Rendering/Parsing  
components and API for XWiki. The implementation will be based on  
WikiModel but we need some XWiki wrapping interfaces around it. Note  
that this is a prerequisite for the new WYSIWYG editor based on GWT  
(see http://www.xwiki.org/xwiki/bin/view/Design/ 
NewWysiwygEditorBasedOnGwt).

I've updated http://www.xwiki.org/xwiki/bin/view/Design/ 
WikiModelIntegration with the information below, which I'm pasting  
here so that we can have a discussion about it. I'll consolidate the  
results on that wiki page.

Componentize the Parsing/Rendering APIs
==================================

We need 4 main components:

     * A Scripting component to manage scripting inside XWiki  
documents and to evaluate them.
     * A Rendering component to manage rendering Wiki syntax into  
HTML and other (PDF, RTF, etc)
     * A Wiki Parser component to offer a typed interface to XWiki  
content so that it can be manipulated
     * A HTML Parser component (for the WYSIWYG editor)

Different Syntaxes
===============

Two possible solutions:

    1. Have a WikiSyntax Object (A simple class with one property: a  
combox box with different syntaxes: XWiki Legacy, Creole, MediaWiki,  
Confluence, JSPWiki, etc) that users can attach to pages to tell the  
Renderers what syntax is used. If no such object is attached then  
it'll default to XWiki's default syntax (XWiki Legacy or Creole for  
example).
    2. Have some special syntax, independent of the wiki syntaxes to  
tell the Rendered that such block of content should be rendered with  
that given syntax. Again there would be a default.

XWiki Interfaces
=============

     * ScriptingEngineManager: Manages the different Scripting  
Engines, calling them in turn.
     * ScriptingEngine
           o Method: evaluate(String content)
           o Implementation: VelocityScriptingEngine
           o Implementation: GroovyScriptingEngine
     * RenderingEngineManager: Manages the different Rendering  
Engines, calling them in turn.
     * RenderingEngine
           o Method: render(String content)
           o Implementation: XWikiLegacyRenderingEngine (current  
rendering engine)
           o Implementation: WikiModelRenderingEngine
     * Parser: content parsing
           o HTMLParser: parses HTML syntax
           o WikiParser: parses wiki syntax
           o Implementation: WikiModelHTMLParser
           o Implementation: WikiModelWikiParser

Open Questions:

     * Does WikiModel support a generic syntax for macros?
     * Is the Rendering also in charge of generating PDF, RTF, XML, etc?
           o I think so, need to modify interfaces above to reflect  
this.
     * The WikiParser needs to recognizes scripts since this is  
needed for the WYSIWYG editor.

Use cases
========

     * View page
           o ViewAction -- template -> ScriptingEngineManager.evaluate
() -- wiki syntax -> RenderingEngineManager.render() ---> HTML, XML,  
PDF, RTF, etc
     * Edit page in WYSIWYG editor
           o Uses the WikiParser to create a "DOM" of the page  
content and to render it accordingly. NOTE: This is required since  
rendering in the WYSIWYG editor is different from the final  
rendering. For example, macros need to be shown in a special way to  
make them visible, etc.
           o Changes done by the user are entered in HTML. Note: it  
would be better to capture them so that they are entered in the  
"DOM". Is that possible? If not, then the HTMLParser is used to  
convert from HTML to Wiki Syntax but they're likely be some loss in  
the conversion. The advantage is the ability to take any HTML content  
and generate wiki syntax from it.

This is my very earlier thinking but I wanted to make it visible to  
give everyone the change to 1) know what's happening and 2) suggest  
ideas.

I'll refine this in the coming days and post again on this thread.

Thanks
-Vincent

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Jean-Vincent Drean
> Two possible solutions:
>
>     1. Have a WikiSyntax Object (A simple class with one property: a
> combox box with different syntaxes: XWiki Legacy, Creole, MediaWiki,
> Confluence, JSPWiki, etc) that users can attach to pages to tell the
> Renderers what syntax is used. If no such object is attached then
> it'll default to XWiki's default syntax (XWiki Legacy or Creole for
> example).
>     2. Have some special syntax, independent of the wiki syntaxes to
> tell the Rendered that such block of content should be rendered with
> that given syntax. Again there would be a default.

3) A new "syntax" document meta ? WDYT ?
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

vmassol
Administrator

On Sep 11, 2007, at 6:36 PM, Jean-Vincent Drean wrote:

>> Two possible solutions:
>>
>>     1. Have a WikiSyntax Object (A simple class with one property: a
>> combox box with different syntaxes: XWiki Legacy, Creole, MediaWiki,
>> Confluence, JSPWiki, etc) that users can attach to pages to tell the
>> Renderers what syntax is used. If no such object is attached then
>> it'll default to XWiki's default syntax (XWiki Legacy or Creole for
>> example).
>>     2. Have some special syntax, independent of the wiki syntaxes to
>> tell the Rendered that such block of content should be rendered with
>> that given syntax. Again there would be a default.
>
> 3) A new "syntax" document meta ? WDYT ?

Yes this is like solution 1 in that you cannot mix syntaxes within a  
given document whereas solution 2 allows that.

Not sure which is best.

Thanks
-Vincent
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Jean-Vincent Drean
> Yes this is like solution 1 in that you cannot mix syntaxes within a
> given document whereas solution 2 allows that.

I don't see benefits from being able to mix different syntaxes within
a document so I'd give +1 to solution 1 (with a meta).

In fact I'm not sure to see the advantages of a syntax choice at the
document level (I'd have naturaly put it at the wiki level). Sorry if
it has already been discussed on the list.
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

vmassol
Administrator

On Sep 11, 2007, at 7:22 PM, Jean-Vincent Drean wrote:

>> Yes this is like solution 1 in that you cannot mix syntaxes within a
>> given document whereas solution 2 allows that.
>
> I don't see benefits from being able to mix different syntaxes within
> a document so I'd give +1 to solution 1 (with a meta).
>
> In fact I'm not sure to see the advantages of a syntax choice at the
> document level (I'd have naturaly put it at the wiki level). Sorry if
> it has already been discussed on the list.

It hasn't been discussed AFAIR... so this is the right place and time  
to discuss it.

I can think of one use case:

* I've an existing wiki (says xwiki.org) which is using the old wiki  
syntax (the current one) and I want to migrate it to the new syntax  
(say it's creole for example). It would be nice if I could mix the 2  
syntaxes during this migration.

Now you have a good point, I don't see many more use cases :) and  
thus I'd agree with you if we have another way of doing this migration.

Does anyone know if WikiModel has syntax migration tools?

Thanks
-Vincent
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Fabio Mancinelli-4
In reply to this post by vmassol

On Sep 11, 2007, at 1:46 PM, Vincent Massol wrote:

>
>     1. Have a WikiSyntax Object (A simple class with one property: a
> combox box with different syntaxes: XWiki Legacy, Creole, MediaWiki,
> Confluence, JSPWiki, etc) that users can attach to pages to tell the
> Renderers what syntax is used. If no such object is attached then
> it'll default to XWiki's default syntax (XWiki Legacy or Creole for
> example).
>     2. Have some special syntax, independent of the wiki syntaxes to
> tell the Rendered that such block of content should be rendered with
> that given syntax. Again there would be a default.
>

Regarding multiple syntaxes... Is it really a requirement to support  
them?
I think that adding this feature could lead to confusion.

For example, imagine an XWiki-Pedia as sketched in 1. A contributor  
could potentially need to edit
3 documents written in 3 different syntaxes.

Solution 2. is even worse because it pushes this heterogeneity down  
to the document level.

An alternative solution could be to have a model for the page that is  
"presented" in different syntaxes when the page is edited. In this  
way a document can be edited in whatever syntax the user feels  
comfortable with. However this raises several issues... A model that  
is the LCM of all the supported syntaxes is fine, but what happens  
when an element in the model is requested to be presented in a syntax  
that doesn't support it?

The best option, probably, is to keep it simple by having and  
supporting only a single syntax for editing pages, with some "import"  
plugins that can translate a previously existing content to that  
syntax when a new page is created.

Sorry if this was an already discussed topic or if I missed some  
relevant detail :)

Cheers,
Fabio

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

vmassol
Administrator

On Sep 11, 2007, at 10:43 PM, Fabio Mancinelli wrote:

>
> On Sep 11, 2007, at 1:46 PM, Vincent Massol wrote:
>>
>>     1. Have a WikiSyntax Object (A simple class with one property: a
>> combox box with different syntaxes: XWiki Legacy, Creole, MediaWiki,
>> Confluence, JSPWiki, etc) that users can attach to pages to tell the
>> Renderers what syntax is used. If no such object is attached then
>> it'll default to XWiki's default syntax (XWiki Legacy or Creole for
>> example).
>>     2. Have some special syntax, independent of the wiki syntaxes to
>> tell the Rendered that such block of content should be rendered with
>> that given syntax. Again there would be a default.
>>
>
> Regarding multiple syntaxes... Is it really a requirement to support
> them?
> I think that adding this feature could lead to confusion.
>
> For example, imagine an XWiki-Pedia as sketched in 1. A contributor
> could potentially need to edit
> 3 documents written in 3 different syntaxes.
>
> Solution 2. is even worse because it pushes this heterogeneity down
> to the document level.
>
> An alternative solution could be to have a model for the page that is
> "presented" in different syntaxes when the page is edited. In this
> way a document can be edited in whatever syntax the user feels
> comfortable with. However this raises several issues... A model that
> is the LCM of all the supported syntaxes is fine, but what happens
> when an element in the model is requested to be presented in a syntax
> that doesn't support it?

I agree with all you said.

> The best option, probably, is to keep it simple by having and
> supporting only a single syntax for editing pages, with some "import"
> plugins that can translate a previously existing content to that
> syntax when a new page is created.

I think I prefer the option suggested by JV, i.e. have a  
configuration option at the level of the wiki to decide what syntax a  
wiki is using. Imports are required too but since WikiModel supports  
several syntaxes why not offer this feature to users. That would help  
a lot for people who come from an existing wiki and want to switch to  
xwiki.

Of course, in my initial email there's an open question about  
WikiModel's syntax of macros across syntaxes that we need to resolve.

Thanks
-Vincent

> Sorry if this was an already discussed topic or if I missed some
> relevant detail :)
>
> Cheers,
> Fabio

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Jean-Vincent Drean
> I think I prefer the option suggested by JV, i.e. have a
> configuration option at the level of the wiki to decide what syntax a
> wiki is using. Imports are required too but since WikiModel supports
> several syntaxes why not offer this feature to users. That would help
> a lot for people who come from an existing wiki and want to switch to
> xwiki.

The wiki pages importer will need to know from which syntax he is
importing, so we still have 2 choices :
1) Have the syntax set at the wiki level and exported at the XAR level
(package.xml)
2) Have a default syntax set at the wiki level and a mandatory meta in
the documents (set at the creation, not assignable from the interface)
:
  - we could write a new edit panel which would appear in the case
were currentDocument is not using the default syntax, this panel would
allow to switch the document to the current syntax.
  - it allows to change the default syntax (wiki level) without an
automatic wiki-wide syntax transformation.

WDYT ?
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Stéphane Laurière-6
In reply to this post by vmassol
Hi Vincent, hi everyone,

We discussed the WikiModel integration with Mikhail this afternoon. Here
is below our input.

Vincent Massol wrote:

> Hi,
>
> I've started working on designing the new Rendering/Parsing
> components and API for XWiki. The implementation will be based on
> WikiModel but we need some XWiki wrapping interfaces around it. Note
>  that this is a prerequisite for the new WYSIWYG editor based on GWT
>  (see http://www.xwiki.org/xwiki/bin/view/Design/ 
> NewWysiwygEditorBasedOnGwt).
>
> I've updated http://www.xwiki.org/xwiki/bin/view/Design/ 
> WikiModelIntegration with the information below, which I'm pasting
> here so that we can have a discussion about it. I'll consolidate the
>  results on that wiki page.
>
> Componentize the Parsing/Rendering APIs
> ==================================
>
> We need 4 main components:
>
> * A Scripting component to manage scripting inside XWiki documents
> and to evaluate them.

On the topic of scripting we would like to propose a distinction between
scripts that act on text and scripts that act on the DOM. Typically, the
text rendering processing for flow would be the following, for say "text1":

text1 =TextProcessor=> text2 =Parser=> dom1 =DomProcessor=> dom2 => ...

- the scripts contained in text1 are processed in the context of user1,
this results into a new text: text2
- the parser parses text2 and converts text2 to a DOM tree, dom1
- dom1 is processed by scripts that work directly on the DOM (example:
table of content generator), this results in dom2
- dom2 is made to available as such or is converted to XML, HTML, PDF
etc. depending on the user request

TextProcessor and DomProcessor would have the following interfaces:

TextProcessor
- String execute(String content)

DomProcessor
- DOM execute(DOM content)

That means we should have a syntax to distinguish between scripts that
generate text content, and scripts that manipulate the DOM.

>      * A Rendering component to manage rendering Wiki syntax into  
> HTML and other (PDF, RTF, etc)
>      * A Wiki Parser component to offer a typed interface to XWiki  
> content so that it can be manipulated
>      * A HTML Parser component (for the WYSIWYG editor)
>
> Different Syntaxes ===============
>
> Two possible solutions:
>
>     1. Have a WikiSyntax Object (A simple class with one property: a  
> combox box with different syntaxes: XWiki Legacy, Creole, MediaWiki,  
> Confluence, JSPWiki, etc) that users can attach to pages to tell the  
> Renderers what syntax is used. If no such object is attached then  
> it'll default to XWiki's default syntax (XWiki Legacy or Creole for  
> example).
>     2. Have some special syntax, independent of the wiki syntaxes to  
> tell the Rendered that such block of content should be rendered with  
> that given syntax. Again there would be a default.
>

Here's our view regarding the syntax used in wiki edit mode: document
requested for edition are available from the database in a serialized
format, for instance XHTML. When entering into the edit action, the user
indicates his preferred syntax. If the text of the requested document
contains some blocks that are not handled by the chosen syntax, the user
gets a warning (example: the document contains a table as a list item,
and the user tries to edit the document using JSPWiki syntax). If not,
WikiModel converts the serialized format into a DOM, the user edits the
DOM and the WikiModel serializer serializes it back when the user saves it.

Note that the DOM representation of wiki documents in the latest version
of WikiModel is still pending.

>
> XWiki Interfaces
> =============
>
>      * ScriptingEngineManager: Manages the different Scripting  
> Engines, calling them in turn.
>      * ScriptingEngine
>            o Method: evaluate(String content)
>            o Implementation: VelocityScriptingEngine
>            o Implementation: GroovyScriptingEngine
>      * RenderingEngineManager: Manages the different Rendering  
> Engines, calling them in turn.
>      * RenderingEngine
>            o Method: render(String content)
>            o Implementation: XWikiLegacyRenderingEngine (current  
> rendering engine)
>            o Implementation: WikiModelRenderingEngine
>      * Parser: content parsing
>            o HTMLParser: parses HTML syntax
>            o WikiParser: parses wiki syntax
>            o Implementation: WikiModelHTMLParser
>            o Implementation: WikiModelWikiParser
>
> Open Questions:
>
>      * Does WikiModel support a generic syntax for macros?

WikiModel generates events for blocks that are not to be parsed
(typically because they contain scripts).

For example, in the WikiModel syntax currently called "CommonSyntax",
this looks like the following:
==============
{{{macro:mymacro (String parameters)
dothis
dothat

}}}


$mymacro(parameters)
==============

For each syntax, macro blocks are identified as far as possible (we
still have to check it's the case for all types of macro blocks inde
indeed).


>      * Is the Rendering also in charge of generating PDF, RTF, XML, etc?
>            o I think so, need to modify interfaces above to reflect  
> this.
>      * The WikiParser needs to recognizes scripts since this is  
> needed for the WYSIWYG editor.

the WikiModel parser recognizes scripts indeed.


Mikhail and Stéphane

>
> Use cases
> ========
>
>      * View page
>            o ViewAction -- template -> ScriptingEngineManager.evaluate
> () -- wiki syntax -> RenderingEngineManager.render() ---> HTML, XML,  
> PDF, RTF, etc
>      * Edit page in WYSIWYG editor
>            o Uses the WikiParser to create a "DOM" of the page  
> content and to render it accordingly. NOTE: This is required since  
> rendering in the WYSIWYG editor is different from the final  
> rendering. For example, macros need to be shown in a special way to  
> make them visible, etc.
>            o Changes done by the user are entered in HTML. Note: it  
> would be better to capture them so that they are entered in the  
> "DOM". Is that possible? If not, then the HTMLParser is used to  
> convert from HTML to Wiki Syntax but they're likely be some loss in  
> the conversion. The advantage is the ability to take any HTML content  
> and generate wiki syntax from it.
 >
 >

> This is my very earlier thinking but I wanted to make it visible to  
> give everyone the change to 1) know what's happening and 2) suggest  
> ideas.
>
> I'll refine this in the coming days and post again on this thread.
>
> Thanks
> -Vincent
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>




--
Stéphane Laurière
[hidden email]

XWiki http://www.xwiki.com
http://concerto.xwiki.com
http://nepomuk.semanticdesktop.org

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

vmassol
Administrator
+1 to all that. So let me summarizes and rephrase to see if I have  
understood :)

1) We have 4 types of objects:
* TextProcessors: take text and generate text
* Parsers: take text and generate an internal DOM format (pivot format)
* DomProcessors: take DOM and generate DOM
* Renderers: take DOM and generate anything (text, PDF, RTF, HTML,  
XML, etc)

2) Document contents are stored in the database in textual format in  
the main xwiki syntax (whatever we decide it is - we could  
standardize on creole for example)

3) Use case 1: Viewing a document

a) Get the doc from the DB --> text1 (xwiki text format)
b) Apply TextProcessors --> text2
c) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an  
internal DOM)
d) Apply DomProcessors --> DOM2
e) Call the required Renderer --> PDF, XML, HTML, RTF, text, etc

4) Use case 2: Editing a document, assuming the user wants to use the  
MediaWiki syntax for editing

a) Get the doc from the DB --> text1 (xwiki text format)
b) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an  
internal DOM)
c) Call MediaWikiRenderer --> text2 (text in MediaWiki format)
d) the user edits and hits save
e) MediaWikiParser --> DOM2 (transforms MediaWiki text syntax into  
the internal DOM)
f) Call XWikiRenderer --> text" (transforms DOM into xwiki textual  
format)
g) Save text3 in the database

5) In practice this means the following classes:

* TextProcessorManager: to chain several text processors
* TextProcessor
   - VelocityTextProcessor
   - GroovyTextProcessor
* WikiParser: Takes wiki syntax and generates a DOM in a XWiki-
specific format (independent of the different wiki syntaxes).
   - LegacyXWikiWikiParser
   - XWikiWikiParser (or simply use CreoleWikiParser if we want our  
internal format to be Creole)
   - ConfluenceWikiParser
   - MediaWikiWikiParser
   - JSPWikiWikiParser
   - CreoleWikiParser
   - HTMLParser: I think all parsers above need to support HTML since  
the wiki syntaxes can be mixed with HTML. So this HTMLParser is  
probably a parent of the other parsers in some regard. Anyway we need  
this one for the WYSIWYG editor which may need to transform HTML to  
wiki syntax (so we may need a XWikiDomProcessor too to transform into  
XWiki syntax). The alternative (much better) is to have the WYSIWYG  
editor only use the internal XWiki-specific DOM format for all its  
manipulations.
* DomProcessorManager: to chain several DOM processors
* DomProcessor
   - Don't know yet what we're going to use this for. TOCDomProcessor  
as you say above maybe.
* Renderer
   - XMLRenderer
   - HTMLRenderer
   - PDFRenderer
   - RTFRenderer
   - XWikiRenderer (or simply use CreoleRenderer if we want our  
internal format to be Creole)
   - ConfluenceRenderer
   - MediaWikiRenderer
   - JSPWikiRenderer
   - CreoleRenderer

WDYT? Do I have it right? :)

Thanks
-Vincent

On Sep 13, 2007, at 6:37 PM, Stéphane Laurière wrote:

> Hi Vincent, hi everyone,
>
> We discussed the WikiModel integration with Mikhail this afternoon.  
> Here
> is below our input.
>
> Vincent Massol wrote:
>> Hi,
>>
>> I've started working on designing the new Rendering/Parsing
>> components and API for XWiki. The implementation will be based on
>> WikiModel but we need some XWiki wrapping interfaces around it. Note
>>  that this is a prerequisite for the new WYSIWYG editor based on GWT
>>  (see http://www.xwiki.org/xwiki/bin/view/Design/
>> NewWysiwygEditorBasedOnGwt).
>>
>> I've updated http://www.xwiki.org/xwiki/bin/view/Design/
>> WikiModelIntegration with the information below, which I'm pasting
>> here so that we can have a discussion about it. I'll consolidate the
>>  results on that wiki page.
>>
>> Componentize the Parsing/Rendering APIs
>> ==================================
>>
>> We need 4 main components:
>>
>> * A Scripting component to manage scripting inside XWiki documents
>> and to evaluate them.
>
> On the topic of scripting we would like to propose a distinction  
> between
> scripts that act on text and scripts that act on the DOM.  
> Typically, the
> text rendering processing for flow would be the following, for say  
> "text1":
>
> text1 =TextProcessor=> text2 =Parser=> dom1 =DomProcessor=> dom2  
> => ...
>
> - the scripts contained in text1 are processed in the context of  
> user1,
> this results into a new text: text2
> - the parser parses text2 and converts text2 to a DOM tree, dom1
> - dom1 is processed by scripts that work directly on the DOM (example:
> table of content generator), this results in dom2
> - dom2 is made to available as such or is converted to XML, HTML, PDF
> etc. depending on the user request
>
> TextProcessor and DomProcessor would have the following interfaces:
>
> TextProcessor
> - String execute(String content)
>
> DomProcessor
> - DOM execute(DOM content)
>
> That means we should have a syntax to distinguish between scripts that
> generate text content, and scripts that manipulate the DOM.
>
>>      * A Rendering component to manage rendering Wiki syntax into
>> HTML and other (PDF, RTF, etc)
>>      * A Wiki Parser component to offer a typed interface to XWiki
>> content so that it can be manipulated
>>      * A HTML Parser component (for the WYSIWYG editor)
>>
>> Different Syntaxes ===============
>>
>> Two possible solutions:
>>
>>     1. Have a WikiSyntax Object (A simple class with one property: a
>> combox box with different syntaxes: XWiki Legacy, Creole, MediaWiki,
>> Confluence, JSPWiki, etc) that users can attach to pages to tell the
>> Renderers what syntax is used. If no such object is attached then
>> it'll default to XWiki's default syntax (XWiki Legacy or Creole for
>> example).
>>     2. Have some special syntax, independent of the wiki syntaxes to
>> tell the Rendered that such block of content should be rendered with
>> that given syntax. Again there would be a default.
>>
>
> Here's our view regarding the syntax used in wiki edit mode: document
> requested for edition are available from the database in a serialized
> format, for instance XHTML. When entering into the edit action, the  
> user
> indicates his preferred syntax. If the text of the requested document
> contains some blocks that are not handled by the chosen syntax, the  
> user
> gets a warning (example: the document contains a table as a list item,
> and the user tries to edit the document using JSPWiki syntax). If not,
> WikiModel converts the serialized format into a DOM, the user edits  
> the
> DOM and the WikiModel serializer serializes it back when the user  
> saves it.
>
> Note that the DOM representation of wiki documents in the latest  
> version
> of WikiModel is still pending.
>
>>
>> XWiki Interfaces
>> =============
>>
>>      * ScriptingEngineManager: Manages the different Scripting
>> Engines, calling them in turn.
>>      * ScriptingEngine
>>            o Method: evaluate(String content)
>>            o Implementation: VelocityScriptingEngine
>>            o Implementation: GroovyScriptingEngine
>>      * RenderingEngineManager: Manages the different Rendering
>> Engines, calling them in turn.
>>      * RenderingEngine
>>            o Method: render(String content)
>>            o Implementation: XWikiLegacyRenderingEngine (current
>> rendering engine)
>>            o Implementation: WikiModelRenderingEngine
>>      * Parser: content parsing
>>            o HTMLParser: parses HTML syntax
>>            o WikiParser: parses wiki syntax
>>            o Implementation: WikiModelHTMLParser
>>            o Implementation: WikiModelWikiParser
>>
>> Open Questions:
>>
>>      * Does WikiModel support a generic syntax for macros?
>
> WikiModel generates events for blocks that are not to be parsed
> (typically because they contain scripts).
>
> For example, in the WikiModel syntax currently called "CommonSyntax",
> this looks like the following:
> ==============
> {{{macro:mymacro (String parameters)
> dothis
> dothat
>
> }}}
>
>
> $mymacro(parameters)
> ==============
>
> For each syntax, macro blocks are identified as far as possible (we
> still have to check it's the case for all types of macro blocks inde
> indeed).
>
>
>>      * Is the Rendering also in charge of generating PDF, RTF,  
>> XML, etc?
>>            o I think so, need to modify interfaces above to reflect
>> this.
>>      * The WikiParser needs to recognizes scripts since this is
>> needed for the WYSIWYG editor.
>
> the WikiModel parser recognizes scripts indeed.
>
>
> Mikhail and Stéphane
>
>>
>> Use cases
>> ========
>>
>>      * View page
>>            o ViewAction -- template ->  
>> ScriptingEngineManager.evaluate
>> () -- wiki syntax -> RenderingEngineManager.render() ---> HTML, XML,
>> PDF, RTF, etc
>>      * Edit page in WYSIWYG editor
>>            o Uses the WikiParser to create a "DOM" of the page
>> content and to render it accordingly. NOTE: This is required since
>> rendering in the WYSIWYG editor is different from the final
>> rendering. For example, macros need to be shown in a special way to
>> make them visible, etc.
>>            o Changes done by the user are entered in HTML. Note: it
>> would be better to capture them so that they are entered in the
>> "DOM". Is that possible? If not, then the HTMLParser is used to
>> convert from HTML to Wiki Syntax but they're likely be some loss in
>> the conversion. The advantage is the ability to take any HTML content
>> and generate wiki syntax from it.
>>
>>
>> This is my very earlier thinking but I wanted to make it visible to
>> give everyone the change to 1) know what's happening and 2) suggest
>> ideas.
>>
>> I'll refine this in the coming days and post again on this thread.
>>
>> Thanks
>> -Vincent

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

ebullient
In reply to this post by Jean-Vincent Drean
On 9/11/07, Jean-Vincent Drean <[hidden email]> wrote:
> > Yes this is like solution 1 in that you cannot mix syntaxes within a
> > given document whereas solution 2 allows that.
>
> I don't see benefits from being able to mix different syntaxes within
> a document so I'd give +1 to solution 1 (with a meta).
>
> In fact I'm not sure to see the advantages of a syntax choice at the
> document level (I'd have naturaly put it at the wiki level). Sorry if
> it has already been discussed on the list.

While it would seemingly be confusing, one reason I can think of is
migration... pile of old documents with old syntax, want new documents
to use new syntax... (what a headache that would be.. but if you have
gobs of docs... )

--
'Waste of a good apple' -Samwise Gamgee
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

ebullient
In reply to this post by vmassol
I should never ever reply to documents until I get to the end of the
note chain. My apologies for being redundant:

This is getting complicated. My view (because of what I know our
documents are loaded with:

A) velocity script just intermixed with text has to be supported.
B) The wiki formatting we have now (loosely based on Radeon) has to
work, with {code} and {pre} tags actually working (which is fixed in
1.1, I understand.. I've yet to move up

It's my view that having all this macro stuff (between groovy,
velocity, the macro defs, etc) jammed into the XWikiPreferences doc is
inappropriate-- That doc in general is too big. I think, for example,
that there should be a separate doc for "ad" configuration that you
only use if ads are enabled, etc. That's a separate discussion, I
think. ;)

But anyway, remember consumability with this: if we're mucking around
with the syntax that's supported, the macros that are supported, etc.
those should be specified in a separate prefs doc.

On 9/14/07, Vincent Massol <[hidden email]> wrote:

> +1 to all that. So let me summarizes and rephrase to see if I have
> understood :)
>
> 1) We have 4 types of objects:
> * TextProcessors: take text and generate text
> * Parsers: take text and generate an internal DOM format (pivot format)
> * DomProcessors: take DOM and generate DOM
> * Renderers: take DOM and generate anything (text, PDF, RTF, HTML,
> XML, etc)
>
> 2) Document contents are stored in the database in textual format in
> the main xwiki syntax (whatever we decide it is - we could
> standardize on creole for example)
>
> 3) Use case 1: Viewing a document
>
> a) Get the doc from the DB --> text1 (xwiki text format)
> b) Apply TextProcessors --> text2
> c) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an
> internal DOM)
> d) Apply DomProcessors --> DOM2
> e) Call the required Renderer --> PDF, XML, HTML, RTF, text, etc
>
> 4) Use case 2: Editing a document, assuming the user wants to use the
> MediaWiki syntax for editing
>
> a) Get the doc from the DB --> text1 (xwiki text format)
> b) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an
> internal DOM)
> c) Call MediaWikiRenderer --> text2 (text in MediaWiki format)
> d) the user edits and hits save
> e) MediaWikiParser --> DOM2 (transforms MediaWiki text syntax into
> the internal DOM)
> f) Call XWikiRenderer --> text" (transforms DOM into xwiki textual
> format)
> g) Save text3 in the database
>
> 5) In practice this means the following classes:
>
> * TextProcessorManager: to chain several text processors
> * TextProcessor
>    - VelocityTextProcessor
>    - GroovyTextProcessor
> * WikiParser: Takes wiki syntax and generates a DOM in a XWiki-
> specific format (independent of the different wiki syntaxes).
>    - LegacyXWikiWikiParser
>    - XWikiWikiParser (or simply use CreoleWikiParser if we want our
> internal format to be Creole)
>    - ConfluenceWikiParser
>    - MediaWikiWikiParser
>    - JSPWikiWikiParser
>    - CreoleWikiParser
>    - HTMLParser: I think all parsers above need to support HTML since
> the wiki syntaxes can be mixed with HTML. So this HTMLParser is
> probably a parent of the other parsers in some regard. Anyway we need
> this one for the WYSIWYG editor which may need to transform HTML to
> wiki syntax (so we may need a XWikiDomProcessor too to transform into
> XWiki syntax). The alternative (much better) is to have the WYSIWYG
> editor only use the internal XWiki-specific DOM format for all its
> manipulations.
> * DomProcessorManager: to chain several DOM processors
> * DomProcessor
>    - Don't know yet what we're going to use this for. TOCDomProcessor
> as you say above maybe.
> * Renderer
>    - XMLRenderer
>    - HTMLRenderer
>    - PDFRenderer
>    - RTFRenderer
>    - XWikiRenderer (or simply use CreoleRenderer if we want our
> internal format to be Creole)
>    - ConfluenceRenderer
>    - MediaWikiRenderer
>    - JSPWikiRenderer
>    - CreoleRenderer
>
> WDYT? Do I have it right? :)
>
> Thanks
> -Vincent
>
> On Sep 13, 2007, at 6:37 PM, Stéphane Laurière wrote:
>
> > Hi Vincent, hi everyone,
> >
[snip....]

> >
> > WikiModel generates events for blocks that are not to be parsed
> > (typically because they contain scripts).
> >
> > For example, in the WikiModel syntax currently called "CommonSyntax",
> > this looks like the following:
> > ==============
> > {{{macro:mymacro (String parameters)
> > dothis
> > dothat
> >
> > }}}
> >
> >
> > $mymacro(parameters)
> > ==============
> >
> > For each syntax, macro blocks are identified as far as possible (we
> > still have to check it's the case for all types of macro blocks inde
> > indeed).
> >


--
'Waste of a good apple' -Samwise Gamgee
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Sergiu Dumitriu
Hi,

Sorry for the delay, I've been busy with the Iaşi office setup and
other stuff. I read all the emails and gathered some observations that
I'll just paste here out of context.




"Different Syntaxes"

As a general observation, we should have support for loose document
metadata. We currently define all the properties inside the java
class, like author, creationDate, contentDate, template, format...
Most of these I don't even know what they are for, and if they work or
not. This way of defining metadata is bad, as it is hard to add a new
property, and once a property is added it will be there for all the
people to see, even if it was added for a client project with some
special requirements. We should provide a mechanism that allows adding
new metadata on the fly.

So, instead of adding a new XClass and adding XObjects to the
document, we can define a loose property "wikiSyntax".



"Different Syntaxes => Two possible solutions:"

Why two exclusive options, and not allow both? You can set the default
(wiki-wide) syntax, the document syntax, and the segment syntax. If
we're talking about a farm, then there's also the farm-wide default
syntax that is used when creating a new wiki. I don't think we really
need this, but if it is not hard to implement, then we should do it.





"HTMLParser: parses HTML syntax"

For the WYSIWYG editor, we can use a small trick that will save us a
lot of time and effort.

What if the HTMLRenderer can work in a "debug" mode, in which it
leaves some debug markings that can be used for reverse engineering
the HTML? For example:

----------
1 Example
This is an *important* example of how a complicated thing can be done
in an *easy* manner:
* instead of trying to parse some generated HTML back into wiki
syntax, we ~~explicitely~~ tell it what the wiki source was?
  **  using some kind of debug markers

* this way, we'll have a simple means of obtaining the wiki code.
-----
Other examples: [link>Main.WebHome12] http://www.autodetected.com
<b>not detected</b> {image:a.png}
----------

When rendering for viewing, this will turn into:
----------
<h1 id="HExample">Example</h1><span class="sectionEditMarker">[edit]</span>
This is an <strong>important</strong> example of how a complicated
thing can be done in an <strong>easy</strong> manner:
<ul class="star"><li>instead of trying to parse some generated HTML
back into wiki syntax, we <em>explicitely</em> tell it what the wiki
source was?
<ul class="star"><li>using some kind of debug markers</li></ul
</li>
<li>this way, we'll have a simple means of obtaining the wiki code.</li>
</ul>
<hr/>
Other examples: <span class="createLink"><a
href="/xwiki/bin/view/Main/WebHome12">link</a></span> <a
href="http://www.autodetected.com">http://www.autodetected.com</a>
<b>not detected</b> <img src="..."/>
----------

When rendering for the WYSIWYG editor, this will turn into:
----------
<h1 id="HExample" xw:smarkup="1 ">Example</h1><span
class="sectionEditMarker" xw:skip="true">[edit]</span>
This is an <strong xw:smarkup="*" xw:emarkup="*">important</strong>
example of how a complicated thing can be done in an <strong
xw:smarkup="*" xw:emarkup="*">easy</strong> manner:
<ul class="star" xw:ignore="true"><li xw:smarkup="* "
xw:emarkup="\n">instead of trying to parse some generated HTML back
into wiki syntax, we <em xw:smarkup="~~"
xw:emarkup="~~">explicitely</em> tell it what the wiki source was?
<ul class="star" xw:ignore="true"><li xw:smarkup="  **  "
xw:emarkup="\n\n">using some kind of debug markers</li></ul
</li>
<li xw:smarkup="* " xw:emarkup="\n">this way, we'll have a simple
means of obtaining the wiki code.</li>
</ul>
<hr wk:source="-----"/>
Other examples: <span class="createLink" xw:ignore="true"><a
href="/xwiki/bin/view/Main/WebHome12" xw:smarkup="["
xw:emarkup=">Main.WebHome]">link</a></span> <a
href="http://www.autodetected.com"
xw:smarkup="">http://www.autodetected.com</a>  <b>not detected</b>
<img src="..." wk:source="{image:a.png}"/>
----------

So, whenever we see an xw:ignore, xw:skip, xw:source, xw:smarkup or
xw:emarkup attribute, we know there was a wiki markup in there. The
HTMLParser would just look for these attributes, remove the xw:skip
tags (including the content), remove the xw:ignore tags (but not the
content), replace the start tag with the smarkup value, the end tag
with the emarkup value, or the entire content with the xw:source
attribute. We use the same rendering engine, and we have a simple
back-parser that doesn't need any changes whenever we add or modify
something in the syntax.

We have to make sure that:
- markup that generates more than one html element must add xw:skip or
xw:ignore on the ones that are not needed for back-parsing
- we don't use xw:source on elements that can have other nested
elements (like the {table} macro)
- we properly escape all the quotes and newlines in the attributes
- we write these attributes when entering new stuff in the editor, and
update them as we change the document.

I know that these attribute names or not good, I'll think of better
ones when I'll be less sleepy.




"Note: it would be better to capture them so that they are entered in
the "DOM". Is that possible?"

Yes, you can do almost anything in Javascript, just that the editor
will be much slower, since you'll have to listen to a lot of events,
and do a lot of processing. I think it would be better if we don't do
this.





"Now you have a good point, I don't see many more use cases :) and
thus I'd agree with you if we have another way of doing this
migration."

How about this use case:
A wiki dedicated to wiki engine comparison, like wikiconsultin, in
which fans of all the wiki engines could meet and debate. Now,
wouldn't it be nice to let the mediawiki fans write using the
mediawiki syntax, and the xwiki fans write using the xwiki syntax? And
all in the same wiki?

I'm +1 for the different syntaxes in the same wiki.



Does anyone know if WikiModel has syntax migration tools?

We can make one, if it doesn't have one. I remember that someone
(Stephane? Marius?) said something about wiki-to-wiki conversion,
though.





"HTMLParser: I think all parsers above need to support HTML since
the wiki syntaxes can be mixed with HTML"

I don't understand this. What does manually entered HTML have to do
with wiki parsing?






"One common storage syntax, multiple editing syntaxes"

I like this idea, but is it worth the effort? wiki editing is my
favorite because it is fast. If we'll need to parse and render twice
for an edit, knowing that these are among the most time consuming
step, I think a high-load XWiki will be much slower.





"document requested for edition are available from the database in a
serialized format, for instance XHTML"

I'd say that XHTML is not the best choice. Moreover, I'd say that we
should store the document as it is, and not using a "standard" markup.





"VelocityTextProcessor"

We'd have only this component for velocity processing? Do we need to
process velocity in another way? I can't think of any right now.




Sergiu
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

vmassol
Administrator
Hi Sergiu,

On Sep 18, 2007, at 2:32 AM, Sergiu Dumitriu wrote:

[snip]

> "Different Syntaxes"
>
> As a general observation, we should have support for loose document
> metadata. We currently define all the properties inside the java
> class, like author, creationDate, contentDate, template, format...
> Most of these I don't even know what they are for, and if they work or
> not. This way of defining metadata is bad, as it is hard to add a new
> property, and once a property is added it will be there for all the
> people to see, even if it was added for a client project with some
> special requirements. We should provide a mechanism that allows adding
> new metadata on the fly.
>
> So, instead of adding a new XClass and adding XObjects to the
> document, we can define a loose property "wikiSyntax".

Looks fine to me.

> "Different Syntaxes => Two possible solutions:"
>
> Why two exclusive options, and not allow both? You can set the default
> (wiki-wide) syntax, the document syntax, and the segment syntax. If
> we're talking about a farm, then there's also the farm-wide default
> syntax that is used when creating a new wiki. I don't think we really
> need this, but if it is not hard to implement, then we should do it.

Right. Note that this is not even an issue if we agree about  
presenting the doc content in the user's syntax of choice. See below.

> "HTMLParser: parses HTML syntax"
>
> For the WYSIWYG editor, we can use a small trick that will save us a
> lot of time and effort.
>
> What if the HTMLRenderer can work in a "debug" mode, in which it
> leaves some debug markings that can be used for reverse engineering
> the HTML? For example:

[snip]

That's a good idea. It would still be nice if the WYSIWYG editor was  
using our DOM tree as its source instead of HTML or alternatively to  
enter data typed into both the HTML DOM tree and XWiki's DOM tree.  
But I'm unsure how feasible this is.

> "Note: it would be better to capture them so that they are entered in
> the "DOM". Is that possible?"
>
> Yes, you can do almost anything in Javascript, just that the editor
> will be much slower, since you'll have to listen to a lot of events,
> and do a lot of processing. I think it would be better if we don't do
> this.

Maybe this would needed to be verified in term of speed.

> "Now you have a good point, I don't see many more use cases :) and
> thus I'd agree with you if we have another way of doing this
> migration."
>
> How about this use case:
> A wiki dedicated to wiki engine comparison, like wikiconsultin, in
> which fans of all the wiki engines could meet and debate. Now,
> wouldn't it be nice to let the mediawiki fans write using the
> mediawiki syntax, and the xwiki fans write using the xwiki syntax? And
> all in the same wiki?

This is a very special use case (same as is the migration one).

> I'm +1 for the different syntaxes in the same wiki.

Again this is moot if we agree about presenting the content in the  
user's preferred choice. I'm just worried about the complexity  
inherent to that solution but if WikiModel solves most of it, why not.

> Does anyone know if WikiModel has syntax migration tools?
>
> We can make one, if it doesn't have one. I remember that someone
> (Stephane? Marius?) said something about wiki-to-wiki conversion,
> though.

Mikhkail, Stephane, any input on this?

> "HTMLParser: I think all parsers above need to support HTML since
> the wiki syntaxes can be mixed with HTML"
>
> I don't understand this. What does manually entered HTML have to do
> with wiki parsing?

Because in wiki content user can introduce HTML.

> "One common storage syntax, multiple editing syntaxes"
>
> I like this idea, but is it worth the effort? wiki editing is my
> favorite because it is fast. If we'll need to parse and render twice
> for an edit, knowing that these are among the most time consuming
> step, I think a high-load XWiki will be much slower.

Yes maybe. It would still be nice to get some performance figures  
before deciding.

Mikhail/Stephane could you tell us the current performance of  
WikiModel for:

* wiki syntax to DOM
* DOM to wiki syntax (say for an XWiki renderer for example)

In the following:

> a) Get the doc from the DB --> text1 (xwiki text format)
> b) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an
> internal DOM)
> c) Call MediaWikiRenderer --> text2 (text in MediaWiki format)
> d) the user edits and hits save
> e) MediaWikiParser --> DOM2 (transforms MediaWiki text syntax into
> the internal DOM)
> f) Call XWikiRenderer --> text" (transforms DOM into xwiki textual
> format)
> g) Save text3 in the database

I have the feeling that b), c), e) and f) are going to be  
insignificant compared to a) and g) but I may be wrong. Some figures  
would be nice.

> "document requested for edition are available from the database in a
> serialized format, for instance XHTML"
>
> I'd say that XHTML is not the best choice. Moreover, I'd say that we
> should store the document as it is, and not using a "standard" markup.

Agreed (see my previous email too)

> "VelocityTextProcessor"
>
> We'd have only this component for velocity processing? Do we need to
> process velocity in another way? I can't think of any right now.

Let's see if we need more when we implement it.

To summarize we need to decide on the topic of whether we want to  
display wiki content in the user's preferred syntax or not. To be  
honest I also like that a lot since this is not something you find in  
other wikis but I'm worried about 2 things:

A) feasibility. Aren't there always going to be lots of  
incompatibilities? Macros can be generic and work for all syntaxes so  
that's not an issue but what about links for example. XWiki's link  
syntax is richer than most other wiki's link syntax. For example if  
there's a reference to a another xwiki db in the link  
(otherwiki:SomeDocument) then what's going to happen when viewed  
with, say, a mediawiki syntax?

B) Complexity. Every user is going to be using his favorite syntax  
and thus when users talk together, copy/paste snippets on xwiki.org  
for example, they're all going to be in different syntaxes.

Maybe these 2 points aren't going to be an issue but I'd like to make  
sure they're not since this is an important decision and what we gain  
from implementing it is not so high in my opinion when compared with  
the option of deciding the syntax at the level of the page or the  
whole wiki.

Actually if there are performance issues it might even be possible to  
combine both:
* the page is edited in the default syntax (the one configured at the  
page level or wiki level)
* there's an export option to export in a different syntax, same as  
exporting in PDF, RTF, etc.

Thanks
-Vincent

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Mikhail Kotelnikov-3
In reply to this post by vmassol
Hello!

Just some words about what the wiki model is and what it is not.

The main goal of the WikiModel is the creation of an API giving access and control to the internal structure of individual wiki documents.

Some features of the WikiModel:
- WikiModel itself does not depend on any particular wiki syntax
- The number of possible structural elements and their possible assembling order is strictly fixed (which greatly simplifies the validation and manipulation) but the final result is almost as expressive as XHTML (and even more expressive, taking into account notions of properties and embedded documents which can recursively contain their own embedded documents :-)).
- WikiModel manipulates with a super-set of structural elements available in existing wikis. And it has some features not available in other wikis. For example using embedded documents in WikiModel it is possible to put a table in a list and this table can contains its own headers, paragraphs, and lists... Or using embedded documents with the notion of properties it is possible to define very complex structured objects directly on a wiki page.
- There is at least one wiki syntax ("Common" syntax) giving access to all features of the Wiki Model. This syntax guaranties that all structural elements of the WikiModel can be serialized/de-serialized without loose of information and structure. Using any other syntaxes can lead to the information lost (example: you can not put table in a table in XWiki or in JSPWiki which is possible using the Common Syntax).
- One of the goals of the WikiModel is to give a mean to *import* information from various wiki engines without information lost. The structure of documents can be serialized in various wiki syntaxes as well, but there is no guaranties that some information will not be lost. The information can be lost in the case when a document contains some elements which have no representation in a particular wiki syntax. Example: properties; tables  in lists; parameters of lists, paragraphs, and tables and so on...
- All elements managed by the WikiModel can be serialized/deserialized using XHTML with additional annotations (microformat-like annotations)

Some features of the CommonSyntax:
- It is a native syntax for the WikiModel. It provides full access to all features of the WikiModel.  All structures in the WikiModel can be serizlized/deszerialized in this syntax without any information lost
- It uses markup characters available in most (in ideal situation - in all) keyboard layouts (including Russian :-)). So you don't have to switch keyboard layouts to write text, tables, lists and headers. For example tables can be defined using pipe symbols ("|" - which is not available in many keyboard layouts) or the "::" sequence.
- If there is a choice then the most commonly used markups are used

The current version  of the WikiModel provides just an event-based interface  to  work with the structure of documents (like SAX for XML).
In previous versions of WIkiModel I had Document Object Model in which each structural element had its own object representation. In the current version an Object Model is not implemented (yet). I thought to create just a set of utility classes manipulating with the standard XML DOM. Example: the method WikiTable#setCellContent(int row, int column, String content) should create an XHTML table object, create the required number of cells and columns and put the given string content in this node. The same for all other structural elements (headers, lists, internal documents, properties, styles, macros...)


On 9/14/07, Vincent Massol <[hidden email]> wrote:
+1 to all that. So let me summarizes and rephrase to see if I have
understood :)

1) We have 4 types of objects:
* TextProcessors: take text and generate text
* Parsers: take text and generate an internal DOM format (pivot format)
* DomProcessors: take DOM and generate DOM
* Renderers: take DOM and generate anything (text, PDF, RTF, HTML,
XML, etc)

Yes.

2) Document contents are stored in the database in textual format in
the main xwiki syntax (whatever we decide it is - we could
standardize on creole for example)

It can be the "Common Syntax" for the reasons mentioned above :-). Creole syntax is one of the most restrictive syntaxes. And I tried to uses in the CommonSyntax as much markups of the Creole as possible.

An another possibility is to store directly in XML or in XHTML+microformat enhancements (for additional structural elements).
pro:
- it can be exported/imported directly and used by external applications which knows nothing about wikis; just a standard XML or XHTML
- this content can be transformed with XSLT processors directly without usage of the WikiModel
- it can be faster to parse XML than the CommonWiki syntax (I have no comparisons)
con:
- it is more difficult to work with diffs (but for diffs it is *better* to use WkiModel and to generate a specific wiki syntax; for example "Common syntax");
- it is not a "human readable" format; it is difficult to understand what you loads from the DB

3) Use case 1: Viewing a document
 

a) Get the doc from the DB --> text1 (xwiki text format)
b) Apply TextProcessors --> text2
c) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an
internal DOM)
d) Apply DomProcessors --> DOM2
e) Call the required Renderer --> PDF, XML, HTML, RTF, text, etc

Yes.


4) Use case 2: Editing a document, assuming the user wants to use the
MediaWiki syntax for editing

a) Get the doc from the DB --> text1 (xwiki text format)
b) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an
internal DOM)
c) Call MediaWikiRenderer --> text2 (text in MediaWiki format)
d) the user edits and hits save
e) MediaWikiParser --> DOM2 (transforms MediaWiki text syntax into
the internal DOM)
f) Call XWikiRenderer --> text" (transforms DOM into xwiki textual
format)
g) Save text3 in the database

Yes. (text1 and text3 can be XML, as I said above)


5) In practice this means the following classes:

* TextProcessorManager: to chain several text processors

Yes. But it can be just a composite processor implementing the same ProcessorManager interfaces.

* TextProcessor
   - VelocityTextProcessor
   - GroovyTextProcessor

Yes.

* WikiParser: Takes wiki syntax and generates a DOM in a XWiki-
specific format (independent of the different wiki syntaxes).
   - LegacyXWikiWikiParser
   - XWikiWikiParser (or simply use CreoleWikiParser if we want our
internal format to be Creole)
   - ConfluenceWikiParser
   - MediaWikiWikiParser
   - JSPWikiWikiParser
   - CreoleWikiParser
   - HTMLParser: I think all parsers above need to support HTML since
the wiki syntaxes can be mixed with HTML. So this HTMLParser is
probably a parent of the other parsers in some regard. Anyway we need
this one for the WYSIWYG editor which may need to transform HTML to
wiki syntax (so we may need a XWikiDomProcessor too to transform into
XWiki syntax). The alternative (much better) is to have the WYSIWYG
editor only use the internal XWiki-specific DOM format for all its
manipulations.

If you want, you can put HTML as a non-interpreted block ("verbatim blocks") and interpret it in the client code. But internally the WikiModel does not support "embedded" (X)HTML. The main reason: in this  case I loose control of the document structure. And this control is the main goal of the WikiModel.

* DomProcessorManager: to chain several DOM processors
* DomProcessor
   - Don't know yet what we're going to use this for. TOCDomProcessor
as you say above maybe.

DOMProcessor can be used to transform the original DOM object representing the document in the DB into a new (user and query-specific) DOM object which can contain new elements, generated dynamically. Now all dynamic page elements are interpreted as simple Velocity or Groovy scripts and they generate text documents which should be parsed using Radeox and transformed to the final HTML document. Using the DOM representation it is possible to interpret some nodes of this graph as Groovy scripts. In WikiModel they will correspond to Verbatim blocks which are opaque for WikiModel but they can be interpreted as scripts by the DomProcessor(s). And these "Groovy"-nodes can be executed and they will add new DOM elements to the DOM2. For example this approach can be used to generate search results.

The advantages of this approach:
- You can put your parsed document DOM1 in the cache, which will avoid you to to parse the document for each query. It is a slowest step in the page processing. Even if the current version of WikiModel is faster than before and it should be faster than Radeox processor.
- Your Groovy scripts will manipulate with normal java classes (DOM nodes) and it will produce DOM nodes and not a plain text. It seems especially interesting taking into account Groovy's Builders ( http://groovy.codehaus.org/Builders). It is enough to write a very simple builder (see http://groovy.codehaus.org/BuilderSupport) generating DOM nodes and ... voila! Your Groovy node from a wiki page generates search results as DOM nodes!  These manipulations with DOM objects should be MUCH faster that process plain text for every request. And all following steps are fast as well - to generate an HTML page it is enough to visit all node with an "XHTMLVisitor".


BTW: do you need Velocity at all? Using only Groovy is much cleaner. It can be used as THE language of XWiki. It  can be used as a template *and* programming language at the same time. And if you *really* want it is possible to integrate Jasper (from Tomcat) engine to use it for pure templating. The code from Jetty (th e org.mortbay.jetty.jspc.plugin package) can be used as an example of integration with Jasper (see http://jetty.mortbay.org/xref/index.html).
In this case in templates it will be possible to use:
- JSP tag libraries (including standard ones)
- Multiple scripting languages (like javabeans, javascript, jpython, jruby, groovy,...)


* Renderer
   - XMLRenderer
   - HTMLRenderer
   - PDFRenderer
   - RTFRenderer
   - XWikiRenderer (or simply use CreoleRenderer if we want our
internal format to be Creole)
   - ConfluenceRenderer
   - MediaWikiRenderer
   - JSPWikiRenderer
   - CreoleRenderer

Yes. All these renderers should be written if you want to support all these syntaxes. I think that it should not be very difficult.

WDYT? Do I have it right? :)



Best regards,
Mikhail
 

Thanks
-Vincent

On Sep 13, 2007, at 6:37 PM, Stéphane Laurière wrote:

> Hi Vincent, hi everyone,
>
> We discussed the WikiModel integration with Mikhail this afternoon.
> Here
> is below our input.
>
> Vincent Massol wrote:
>> Hi,
>>

>> I've started working on designing the new Rendering/Parsing
>> components and API for XWiki. The implementation will be based on
>> WikiModel but we need some XWiki wrapping interfaces around it. Note
>>  that this is a prerequisite for the new WYSIWYG editor based on GWT
>>  (see <a href="http://www.xwiki.org/xwiki/bin/view/Design/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> http://www.xwiki.org/xwiki/bin/view/Design/
>> NewWysiwygEditorBasedOnGwt).
>>
>> I've updated <a href="http://www.xwiki.org/xwiki/bin/view/Design/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.xwiki.org/xwiki/bin/view/Design/
>> WikiModelIntegration with the information below, which I'm pasting
>> here so that we can have a discussion about it. I'll consolidate the
>>  results on that wiki page.
>>
>> Componentize the Parsing/Rendering APIs
>> ==================================
>>
>> We need 4 main components:
>>
>> * A Scripting component to manage scripting inside XWiki documents
>> and to evaluate them.
>
> On the topic of scripting we would like to propose a distinction
> between
> scripts that act on text and scripts that act on the DOM.
> Typically, the
> text rendering processing for flow would be the following, for say
> "text1":
>
> text1 =TextProcessor=> text2 =Parser=> dom1 =DomProcessor=> dom2
> => ...
>
> - the scripts contained in text1 are processed in the context of
> user1,
> this results into a new text: text2
> - the parser parses text2 and converts text2 to a DOM tree, dom1
> - dom1 is processed by scripts that work directly on the DOM (example:
> table of content generator), this results in dom2
> - dom2 is made to available as such or is converted to XML, HTML, PDF
> etc. depending on the user request
>
> TextProcessor and DomProcessor would have the following interfaces:
>
> TextProcessor
> - String execute(String content)
>
> DomProcessor
> - DOM execute(DOM content)
>

> That means we should have a syntax to distinguish between scripts that
> generate text content, and scripts that manipulate the DOM.
>
>>      * A Rendering component to manage rendering Wiki syntax into
>> HTML and other (PDF, RTF, etc)
>>      * A Wiki Parser component to offer a typed interface to XWiki
>> content so that it can be manipulated
>>      * A HTML Parser component (for the WYSIWYG editor)
>>
>> Different Syntaxes ===============
>>
>> Two possible solutions:
>>
>>     1. Have a WikiSyntax Object (A simple class with one property: a
>> combox box with different syntaxes: XWiki Legacy, Creole, MediaWiki,
>> Confluence, JSPWiki, etc) that users can attach to pages to tell the
>> Renderers what syntax is used. If no such object is attached then
>> it'll default to XWiki's default syntax (XWiki Legacy or Creole for
>> example).
>>     2. Have some special syntax, independent of the wiki syntaxes to
>> tell the Rendered that such block of content should be rendered with
>> that given syntax. Again there would be a default.
>>
>
> Here's our view regarding the syntax used in wiki edit mode: document
> requested for edition are available from the database in a serialized
> format, for instance XHTML. When entering into the edit action, the
> user
> indicates his preferred syntax. If the text of the requested document
> contains some blocks that are not handled by the chosen syntax, the
> user
> gets a warning (example: the document contains a table as a list item,
> and the user tries to edit the document using JSPWiki syntax). If not,
> WikiModel converts the serialized format into a DOM, the user edits
> the
> DOM and the WikiModel serializer serializes it back when the user
> saves it.
>
> Note that the DOM representation of wiki documents in the latest
> version
> of WikiModel is still pending.
>
>>
>> XWiki Interfaces
>> =============
>>
>>      * ScriptingEngineManager: Manages the different Scripting
>> Engines, calling them in turn.
>>      * ScriptingEngine
>>            o Method: evaluate(String content)
>>            o Implementation: VelocityScriptingEngine
>>            o Implementation: GroovyScriptingEngine
>>      * RenderingEngineManager: Manages the different Rendering
>> Engines, calling them in turn.
>>      * RenderingEngine
>>            o Method: render(String content)
>>            o Implementation: XWikiLegacyRenderingEngine (current
>> rendering engine)
>>            o Implementation: WikiModelRenderingEngine
>>      * Parser: content parsing
>>            o HTMLParser: parses HTML syntax
>>            o WikiParser: parses wiki syntax
>>            o Implementation: WikiModelHTMLParser
>>            o Implementation: WikiModelWikiParser

>>
>> Open Questions:
>>
>>      * Does WikiModel support a generic syntax for macros?
>
> WikiModel generates events for blocks that are not to be parsed
> (typically because they contain scripts).
>
> For example, in the WikiModel syntax currently called "CommonSyntax",
> this looks like the following:
> ==============
> {{{macro:mymacro (String parameters)
> dothis
> dothat

>
> }}}
>
>
> $mymacro(parameters)
> ==============
>
> For each syntax, macro blocks are identified as far as possible (we
> still have to check it's the case for all types of macro blocks inde
> indeed).
>
>
>>      * Is the Rendering also in charge of generating PDF, RTF,
>> XML, etc?
>>            o I think so, need to modify interfaces above to reflect
>> this.
>>      * The WikiParser needs to recognizes scripts since this is
>> needed for the WYSIWYG editor.
>
> the WikiModel parser recognizes scripts indeed.
>
>
> Mikhail and Stéphane
>
>>
>> Use cases
>> ========
>>
>>      * View page
>>            o ViewAction -- template ->
>> ScriptingEngineManager.evaluate
>> () -- wiki syntax -> RenderingEngineManager.render() ---> HTML, XML,
>> PDF, RTF, etc
>>      * Edit page in WYSIWYG editor
>>            o Uses the WikiParser to create a "DOM" of the page
>> content and to render it accordingly. NOTE: This is required since
>> rendering in the WYSIWYG editor is different from the final
>> rendering. For example, macros need to be shown in a special way to
>> make them visible, etc.
>>            o Changes done by the user are entered in HTML. Note: it
>> would be better to capture them so that they are entered in the
>> "DOM". Is that possible? If not, then the HTMLParser is used to
>> convert from HTML to Wiki Syntax but they're likely be some loss in
>> the conversion. The advantage is the ability to take any HTML content
>> and generate wiki syntax from it.
>>
>>
>> This is my very earlier thinking but I wanted to make it visible to
>> give everyone the change to 1) know what's happening and 2) suggest
>> ideas.
>>
>> I'll refine this in the coming days and post again on this thread.
>>
>> Thanks
>> -Vincent



_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Stéphane Laurière-6
In reply to this post by vmassol
Vincent Massol wrote:

> Hi Sergiu,
>
> On Sep 18, 2007, at 2:32 AM, Sergiu Dumitriu wrote:
>
> [snip]
>
>> "Different Syntaxes"
>>
>> As a general observation, we should have support for loose document
>> metadata. We currently define all the properties inside the java
>> class, like author, creationDate, contentDate, template, format...
>> Most of these I don't even know what they are for, and if they work or
>> not. This way of defining metadata is bad, as it is hard to add a new
>> property, and once a property is added it will be there for all the
>> people to see, even if it was added for a client project with some
>> special requirements. We should provide a mechanism that allows adding
>> new metadata on the fly.
>>
>> So, instead of adding a new XClass and adding XObjects to the
>> document, we can define a loose property "wikiSyntax".
>
> Looks fine to me.
>
>> "Different Syntaxes => Two possible solutions:"
>>
>> Why two exclusive options, and not allow both? You can set the default
>> (wiki-wide) syntax, the document syntax, and the segment syntax. If
>> we're talking about a farm, then there's also the farm-wide default
>> syntax that is used when creating a new wiki. I don't think we really
>> need this, but if it is not hard to implement, then we should do it.
>
> Right. Note that this is not even an issue if we agree about  
> presenting the doc content in the user's syntax of choice. See below.
>
>> "HTMLParser: parses HTML syntax"
>>
>> For the WYSIWYG editor, we can use a small trick that will save us a
>> lot of time and effort.
>>
>> What if the HTMLRenderer can work in a "debug" mode, in which it
>> leaves some debug markings that can be used for reverse engineering
>> the HTML? For example:
>
> [snip]
>
> That's a good idea. It would still be nice if the WYSIWYG editor was  
> using our DOM tree as its source instead of HTML or alternatively to  
> enter data typed into both the HTML DOM tree and XWiki's DOM tree.  
> But I'm unsure how feasible this is.
>
>> "Note: it would be better to capture them so that they are entered in
>> the "DOM". Is that possible?"
>>
>> Yes, you can do almost anything in Javascript, just that the editor
>> will be much slower, since you'll have to listen to a lot of events,
>> and do a lot of processing. I think it would be better if we don't do
>> this.
>
> Maybe this would needed to be verified in term of speed.
>
>> "Now you have a good point, I don't see many more use cases :) and
>> thus I'd agree with you if we have another way of doing this
>> migration."
>>
>> How about this use case:
>> A wiki dedicated to wiki engine comparison, like wikiconsultin, in
>> which fans of all the wiki engines could meet and debate. Now,
>> wouldn't it be nice to let the mediawiki fans write using the
>> mediawiki syntax, and the xwiki fans write using the xwiki syntax? And
>> all in the same wiki?
>
> This is a very special use case (same as is the migration one).
>
>> I'm +1 for the different syntaxes in the same wiki.
>
> Again this is moot if we agree about presenting the content in the  
> user's preferred choice. I'm just worried about the complexity  
> inherent to that solution but if WikiModel solves most of it, why not.
>
>> Does anyone know if WikiModel has syntax migration tools?
>>
>> We can make one, if it doesn't have one. I remember that someone
>> (Stephane? Marius?) said something about wiki-to-wiki conversion,
>> though.
>
> Mikhkail, Stephane, any input on this?

currently, WikiModel provides a set of parsers for MediaWiki, JSPWiki,
Creole, XWiki, GWiki and XHTML. For wiki-to-wiki conversion, we now need
a set of serializers (also called renderers previously). I'm writing one
for XWiki. Then the idea would be to add other serializers in order to
show how to convert one syntax to the other.

>
>> "HTMLParser: I think all parsers above need to support HTML since
>> the wiki syntaxes can be mixed with HTML"
>>
>> I don't understand this. What does manually entered HTML have to do
>> with wiki parsing?
>
> Because in wiki content user can introduce HTML.
>
>> "One common storage syntax, multiple editing syntaxes"
>>
>> I like this idea, but is it worth the effort? wiki editing is my
>> favorite because it is fast. If we'll need to parse and render twice
>> for an edit, knowing that these are among the most time consuming
>> step, I think a high-load XWiki will be much slower.
>
> Yes maybe. It would still be nice to get some performance figures  
> before deciding.
>
> Mikhail/Stephane could you tell us the current performance of  
> WikiModel for:
>
> * wiki syntax to DOM
> * DOM to wiki syntax (say for an XWiki renderer for example)

ok, we'll compute some parsing and rendering metrics and see how
WikiModel compares with Radeox. I'll let you know asap about the results.

> In the following:
>> a) Get the doc from the DB --> text1 (xwiki text format)
>> b) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an
>> internal DOM)
>> c) Call MediaWikiRenderer --> text2 (text in MediaWiki format)
>> d) the user edits and hits save
>> e) MediaWikiParser --> DOM2 (transforms MediaWiki text syntax into
>> the internal DOM)
>> f) Call XWikiRenderer --> text" (transforms DOM into xwiki textual
>> format)
>> g) Save text3 in the database
>
> I have the feeling that b), c), e) and f) are going to be  
> insignificant compared to a) and g) but I may be wrong. Some figures  
> would be nice.

ok

>> "document requested for edition are available from the database in a
>> serialized format, for instance XHTML"
>>
>> I'd say that XHTML is not the best choice. Moreover, I'd say that we
>> should store the document as it is, and not using a "standard" markup.
>
> Agreed (see my previous email too)
>
>> "VelocityTextProcessor"
>>
>> We'd have only this component for velocity processing? Do we need to
>> process velocity in another way? I can't think of any right now.
>
> Let's see if we need more when we implement it.
>
> To summarize we need to decide on the topic of whether we want to  
> display wiki content in the user's preferred syntax or not. To be  
> honest I also like that a lot since this is not something you find in  
> other wikis but I'm worried about 2 things:
>
> A) feasibility. Aren't there always going to be lots of  
> incompatibilities? Macros can be generic and work for all syntaxes so  
> that's not an issue but what about links for example. XWiki's link  
> syntax is richer than most other wiki's link syntax. For example if  
> there's a reference to a another xwiki db in the link  
> (otherwiki:SomeDocument) then what's going to happen when viewed  
> with, say, a mediawiki syntax?

I see two options:
- we tell the user he will loose some information in using the mediawiki
syntax in that case
- we complement the supported syntaxes so that they cover exactly the
same semantic capabilities. Mikhail, would this make sense?

>
> B) Complexity. Every user is going to be using his favorite syntax  
> and thus when users talk together, copy/paste snippets on xwiki.org  
> for example, they're all going to be in different syntaxes.

Right, but on xwiki.org we can choose to display snippets in a uniform
syntax whatever syntax was used for entering them, using when possible
the one set in the user preferences, or using the richest one. When
sharing snippets that are not rendered by XWiki, it's true users may use
various syntaxes. We may however consider recommending one preferred
syntax, the richest one, i.e. WikiModel CommonSyntax.

> Maybe these 2 points aren't going to be an issue but I'd like to make  
> sure they're not since this is an important decision and what we gain  
> from implementing it is not so high in my opinion when compared with  
> the option of deciding the syntax at the level of the page or the  
> whole wiki.
>
> Actually if there are performance issues it might even be possible to  
> combine both:
> * the page is edited in the default syntax (the one configured at the  
> page level or wiki level)
> * there's an export option to export in a different syntax, same as  
> exporting in PDF, RTF, etc.
>
> Thanks
> -Vincent
>


Cheers

Stéphane


_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Sergiu Dumitriu
In reply to this post by vmassol
Sorry for the late reply, I didn't see your mail.

>
> > "HTMLParser: I think all parsers above need to support HTML since
> > the wiki syntaxes can be mixed with HTML"
> >
> > I don't understand this. What does manually entered HTML have to do
> > with wiki parsing?
>
> Because in wiki content user can introduce HTML.

Yes, but HTML should be left as-is. I don't see why it should be
parsed. Do you have a use-case for parsing HTML together with the wiki
syntax into a DOM?

>
> To summarize we need to decide on the topic of whether we want to
> display wiki content in the user's preferred syntax or not. To be
> honest I also like that a lot since this is not something you find in
> other wikis but I'm worried about 2 things:
>
> A) feasibility. Aren't there always going to be lots of
> incompatibilities? Macros can be generic and work for all syntaxes so
> that's not an issue but what about links for example. XWiki's link
> syntax is richer than most other wiki's link syntax. For example if
> there's a reference to a another xwiki db in the link
> (otherwiki:SomeDocument) then what's going to happen when viewed
> with, say, a mediawiki syntax?

We either have to extend all the syntaxes, or to restrict the users
only to use the common subset among all syntaxes. Or something in the
middle, extend where possible, but trim all the things that don't have
an equivalent in one of the syntaxes. The best thing would be to try
to extend all syntaxes.

> B) Complexity. Every user is going to be using his favorite syntax
> and thus when users talk together, copy/paste snippets on xwiki.org
> for example, they're all going to be in different syntaxes.
>

C) Included documents. What syntax will they use? #include will copy
the content, and let radeox process it later, along with the includer
document. We can put the included document inside a
{syntax:$idoc.syntax} block.

D) A common practice was to use velocity to generate wiki syntax that
Radeox would process, or to generate (radeox) macro parameters using
velocity, like the {rss:${userobj.feed}}. What happens if we
dynamically change the wiki syntax? Velocity can't know about that.
And with such fragmented code, it will be very hard to dynamically
change the syntax to the current user's preference.

> Maybe these 2 points aren't going to be an issue but I'd like to make
> sure they're not since this is an important decision and what we gain
> from implementing it is not so high in my opinion when compared with
> the option of deciding the syntax at the level of the page or the
> whole wiki.
>
> Actually if there are performance issues it might even be possible to
> combine both:
> * the page is edited in the default syntax (the one configured at the
> page level or wiki level)
> * there's an export option to export in a different syntax, same as
> exporting in PDF, RTF, etc.
>

What happens after the export? The user edits that version in an
offline tool (XEclipse, for example), then he can reimport the changed
document, which will be converted back to the original syntax.

I'd rather have a "convert" button, which will try to convert the
document to another syntax. If there are things that can't be
converted, then warn about this, and offer a Yes/No choice to the
user, allowing him to force the converted syntax, or abandon the
conversion.

Hm, talking about XEclipse, maybe we can leave the "edit using XYZ
syntax" as an XEclipse feature, not present in XWiki platform. This
way we'll remove the stress from the server, as the conversion could
be performed on the client.

> Thanks
> -Vincent
>

And a question on wikimodel, what if there's a feature we need but
doesn't have an equivalent in the WikiDOM?

Sergiu
--
http://purl.org/net/sergiu
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

Mikhail Kotelnikov-3
Hello!

I tried to response to your questions below from the point of view of a WikiModel developer :-)

On 9/20/07, Sergiu Dumitriu <[hidden email]> wrote:
Sorry for the late reply, I didn't see your mail.

>
> > "HTMLParser: I think all parsers above need to support HTML since
> > the wiki syntaxes can be mixed with HTML"
> >
> > I don't understand this. What does manually entered HTML have to do
> > with wiki parsing?
>
> Because in wiki content user can introduce HTML.

Yes, but HTML should be left as-is. I don't see why it should be
parsed. Do you have a use-case for parsing HTML together with the wiki
syntax into a DOM?

If you put your HTML in a wiki page it *can* be left as is without any modifications if it is in a "verbatim" block. In this case it is up to the client code to interpret or not the content of such verbatim blocks. It can be something like: {{{html: <h1>Hello, world</h1>}}}
In this case the content of such a block can be directly inserted in the resulting generated HTML page.
In general verbatim blocks can be used to insert in pages all what you want to interpret yourself. It can be groovy scripts: {{{groovy: print "Hello, world!"}}}.

We need to parse HTML to transform any random web page to editable wiki content. BTW this code already exists and it works.

>

> To summarize we need to decide on the topic of whether we want to
> display wiki content in the user's preferred syntax or not. To be
> honest I also like that a lot since this is not something you find in
> other wikis but I'm worried about 2 things:
>
> A) feasibility. Aren't there always going to be lots of
> incompatibilities? Macros can be generic and work for all syntaxes so
> that's not an issue but what about links for example. XWiki's link
> syntax is richer than most other wiki's link syntax. For example if
> there's a reference to a another xwiki db in the link
> (otherwiki:SomeDocument) then what's going to happen when viewed
> with, say, a mediawiki syntax?

We either have to extend all the syntaxes, or to restrict the users
only to use the common subset among all syntaxes. Or something in the
middle, extend where possible, but trim all the things that don't have
an equivalent in one of the syntaxes. The best thing would be to try
to extend all syntaxes.

The main goal of multiple parsers for multiple wiki syntaxes is a possibility to import without loosing of information or formatting any external wiki pages and transform them to the WikiModel. If you wrote the same sequence of a titles, paragraphs, tables and lists formatted using XWiki, JSPWiki or MediaWiki syntaxes then they will give exactly the same structure in the WIkiModel.
If pages are simple (they don't contain any "advanced" stuff like hierarchical embedded documents or properties) then they *can* be serialized and edited using any particular simple wiki syntax (XWiki, JSPWiki, Creole, ...). WikiModel guaranties that any modifications introduced using these particular syntaxes will not be loosed. If you loose something then it should be considered as a bug.


> B) Complexity. Every user is going to be using his favorite syntax
> and thus when users talk together, copy/paste snippets on xwiki.org
> for example, they're all going to be in different syntaxes.
>
It does not matter what the syntax you use for editing of your document. All these syntaxes will produce the *same* structure. Using this structure it is possible to serialize documents in the CommonSyntax.
The cycle of editing is:
- Create and submit a new text using, for example, the JSPWiki syntax
- Parse the content using the JspWikiParser. This operation will produce a well-formed sequence of events for the listener (like: beginParagraph(..)/endParagraph(...); beginTable(...)/endTable(...)...). This step cleans up all user's errors like non-closed syntactic elements and so on.
- Using the CommonSyntaxSerializer a new wiki document will be generated with exactly the same structure as the original JSPWiki document
- This resulting document should be stored in the DB.

In WikiModel a document written using a particular syntax is just a reflection of the internal structure of this document. Each particular wiki syntax (JSPWiki, XWiki, Creole, ...) reflects only part of possible structural elements of the WikiModel. The CommonSyntax is the "native" syntax and it contains *all* possible elements available in the WikiModel (embedded documents, properties, extensions, ...).  And it was designed taking into account availability and facility of usage of formatting symbols with various keyboard layouts.

Why it is important? Why do we need the CommonSyntax? Just some examples:
Ex1: Using the CommonSyntax it is possible to put 2 paragraphs, a list and a table into an another table. It can be done because there is a notion of "embedded document". AFAIK no other syntaxes give this possibilities. Even MediaWiki which have the most advanced (and most complicated) syntax.
Ex2: In a page containing the information about a person it is possible to define properties like "firstName", "lastName", "birthDate", "address" and so. So the document itself contains well structured semantic information as well as a normal text. In the future it can (and I think - it should) replace the notion of XWiki "objects" attached to documents.
Ex3: The symbol "|" does not exist in the Russian keyboard layout. To enter this symbol you have to switch from Russian to English. Imagine now that you want to create a table with 5 columns and 5 lines. How much times you have to switch? :-) So I use the sequence "::" as table cell delimiters (but "|" is recognized as well). Table cell delimiters are just one example. The same with many other structural elements.

C) Included documents. What syntax will they use? #include will copy
the content, and let radeox process it later, along with the includer
document. We can put the included document inside a
{syntax:$idoc.syntax} block.

As I said above WikiModel has a notion of "embedded documents". In WikiModel each wiki document is constructed from a sequence of block elements (headers, tables, lists, paragraphs,...). And block elements can have "embedded documents" which have exactly the same structural elements as the topmost one.
An example of a page with an embedded document (CommonSyntax):
----------------------------------------------
= Example1 =

The table below contains an embedded document.
Using such embedded documents you can insert table
in a list or a list in a table.
And embedded documents
can contain their own embedded documents!!!


!! Header 1.1 !! Header 1.2
:: Cell 2.1 :: Cell 2.2 with an embedded document: (((
== This is an embedded document! ==
* list item one
* list item two
  * sub-item A
  * sub-item B
* list item three
)))
:: Cell 3.1 :: Cell 3.2

This is a paragraphs after the table...
----------------------------------------------


Please note that these "embedded documents" have nothing to do with external document inclusions. WikiModel works only with the content of one page. Its goal is just to recognize and manipulate with individual structural elements on wiki pages. If you want to make inclusions you can use "extensions" and interpret them in your code as you wish.
It can be something like that:
----------------------------------------------
= Example2 =

The text below will be recognized by the
WikiModel as an "extension" and it can be
interpreted in the user's code for example
to include an external page in this place.

$include(http://www.google.com)

The next paragraph...
----------------------------------------------

D) A common practice was to use velocity to generate wiki syntax that
Radeox would process, or to generate (radeox) macro parameters using
velocity, like the {rss:${userobj.feed}}. What happens if we
dynamically change the wiki syntax? Velocity can't know about that.
And with such fragmented code, it will be very hard to dynamically
change the syntax to the current user's preference.

If I understand well then the decision was to use the WikiModel instead of Radeox. WikiModel does exactly the same as Radeox does. With some differences:
 - WikiModel guaranties that documents are well-formed. It is based on real grammars for JavaCC and not on regular expressions, like Radeox.
-  WIkiModel contains parsers for multiple syntaxes (CommonSyntax, XWiki, JspWiki, Mediawiki, Creole, ...)
- WikiModel does not generate HTML; it just notify listeners about individual structural elements found in a document; And it is up to the implementors of these listeners to do something. For example - there is a listener which generates an HTML. An another can generate a wiki page with another wiki syntax. And so on...

So if you want to include an external document you can extend the HTML Listener, overload the method onExtension(String extensionContent) and make this inclusion operation.
In CommonSyntax extensions are defined as following:
----------------------------------------------
= Example3 =
This is an {{{rss: ${userobj.feed} }}}
----------------------------------------------

About Velocity... Personally I think that it is better NOT to use Velocity at all and to use Groovy templates instead.

> Maybe these 2 points aren't going to be an issue but I'd like to make

> sure they're not since this is an important decision and what we gain
> from implementing it is not so high in my opinion when compared with
> the option of deciding the syntax at the level of the page or the
> whole wiki.
>
> Actually if there are performance issues it might even be possible to
> combine both:
> * the page is edited in the default syntax (the one configured at the
> page level or wiki level)
> * there's an export option to export in a different syntax, same as
> exporting in PDF, RTF, etc.
>

What happens after the export? The user edits that version in an
offline tool (XEclipse, for example), then he can reimport the changed
document, which will be converted back to the original syntax.

I'd rather have a "convert" button, which will try to convert the
document to another syntax. If there are things that can't be
converted, then warn about this, and offer a Yes/No choice to the
user, allowing him to force the converted syntax, or abandon the
conversion.

Hm, talking about XEclipse, maybe we can leave the "edit using XYZ
syntax" as an XEclipse feature, not present in XWiki platform. This
way we'll remove the stress from the server, as the conversion could
be performed on the client.

You can loose some information only when you transform WikiModel-specific structural elements (like embedded documents or properties)  into an external format (XWiki, JSpWiki, ...). When you import from other format to WikiModel you should loose nothing. Otherwise it is considered as a bug in the implementation of WikiModel's parsers .
So if you exported a wiki page to particular syntax without warnings you can be sure that all your modifications will be seamlessly integrated back.


> Thanks
> -Vincent
>

And a question on wikimodel, what if there's a feature we need but
doesn't have an equivalent in the WikiDOM?

Hmm...
- WikiModel works with a super-set of structural elements available in existing wikis (in those wikis which I know :-)) and it contains additional features like embedded documents or properties. So if you found a structural element existing in other wikis and not presented in WikiModel (and which can not be *easly* simulated with existing elements) then you should consider it as a bug. And such a structural element should be added to the WikiModel as soon as possible.
- If you need some additional features and they can not be "externalized" in verbatim blocks then... write me. We will discuss :-)

I think that the WikiModel can give the common infrastructure which works with well-known elements. If you need something specific - just put it in a verbatim block and interpret it yourself in your code.
----------------------------------------------
= Example4 =
This is a verbatim  block:
{{{
This is a verbatim block.
It can be used to insert in
the final page
a <strong>junk
   
and <em>bad-formed</strong>
     html</em>!!!
}}}

And the next block can be interpreted in
your code as a groovy script:

{{{groovy: println "Hello, world!" }}}


----------------------------------------------

WikiModel is written using JavaCC grammars. Modifications of these grammars is not a very complicated task but it is definitely requires more work than just changing of configuration files.

And  about the WikiModel DOM... As I wrote above, the last version of the WikiModel does not contain DOM yet. Just the common infrastructure and a set of parsers for various wiki syntaxes generating well-formed events for structural elements.
 
Best regards,
Mikhail

Sergiu
--
http://purl.org/net/sergiu
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs


_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

vmassol
Administrator
In reply to this post by Sergiu Dumitriu

On Sep 20, 2007, at 5:14 PM, Sergiu Dumitriu wrote:

> Sorry for the late reply, I didn't see your mail.
>
>>
>>> "HTMLParser: I think all parsers above need to support HTML since
>>> the wiki syntaxes can be mixed with HTML"
>>>
>>> I don't understand this. What does manually entered HTML have to do
>>> with wiki parsing?
>>
>> Because in wiki content user can introduce HTML.
>
> Yes, but HTML should be left as-is. I don't see why it should be
> parsed. Do you have a use-case for parsing HTML together with the wiki
> syntax into a DOM?

I wrote this before I understood that they would be inserted in a  
verbatim block. So yes you're right, there's no point to parse them  
for the main view use case.

[snip]

Thanks
-Vincent

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Designing the new Rendering/Parsing component/API

vmassol
Administrator
In reply to this post by Mikhail Kotelnikov-3
Hi Mikhail,

Thanks for sharing this info with us! This makes it more clear for me.

From what I understand below you're recommending to eliminate the need for TextProcessor and instead do the following:

* Store the documents in the database in the DOM format (XML)
* Store scripts as a verbatim block in that DOM
* Only use DOMProcessor to make transformations to the DOM. For example have a VelocityDomProcessor and GroovyDomProcessor to modify the script DOM elements and evaluate them. Note: Velocity or Groovy scripts can generate wiki syntax content and thus these would need to generate new DOM elements. Not sure how easy that would be. This means the VelocityDomProcessor would need internally to use a Parser to parse the result of the evaluation and generate a sub-DOM. Is that correct?

Thus the textual format would only be used when the user enters text or when we want to export the content.

The main advantage would be performance since there'll be no need to go back and forth between textual format and DOM format.

Makes sense to me.

Now you mention removing Velocity. This won't be possible since all current XWiki instances used are using Velocity and we cannot tell our users that they have to rewrite all their pages if they want to move to XWiki v1.3. We'll need to continue supporting Velocity for some time. Personally I currently find that the velocity syntaxes mixes much better with the wiki syntax than groovy. If you look at contributed code snippets you'll see that most are in Velocity which is what most people use.

Now you mention other stuff about Jasper and Jetty but I'm not sure I have understood that part.

Thanks
-Vincent

See below.

On Sep 19, 2007, at 6:54 PM, Mikhail Kotelnikov wrote:

Hello!

Just some words about what the wiki model is and what it is not.

The main goal of the WikiModel is the creation of an API giving access and control to the internal structure of individual wiki documents.

Some features of the WikiModel:
- WikiModel itself does not depend on any particular wiki syntax
- The number of possible structural elements and their possible assembling order is strictly fixed (which greatly simplifies the validation and manipulation) but the final result is almost as expressive as XHTML (and even more expressive, taking into account notions of properties and embedded documents which can recursively contain their own embedded documents :-)).
- WikiModel manipulates with a super-set of structural elements available in existing wikis. And it has some features not available in other wikis. For example using embedded documents in WikiModel it is possible to put a table in a list and this table can contains its own headers, paragraphs, and lists... Or using embedded documents with the notion of properties it is possible to define very complex structured objects directly on a wiki page.
- There is at least one wiki syntax ("Common" syntax) giving access to all features of the Wiki Model. This syntax guaranties that all structural elements of the WikiModel can be serialized/de-serialized without loose of information and structure. Using any other syntaxes can lead to the information lost (example: you can not put table in a table in XWiki or in JSPWiki which is possible using the Common Syntax).
- One of the goals of the WikiModel is to give a mean to *import* information from various wiki engines without information lost. The structure of documents can be serialized in various wiki syntaxes as well, but there is no guaranties that some information will not be lost. The information can be lost in the case when a document contains some elements which have no representation in a particular wiki syntax. Example: properties; tables  in lists; parameters of lists, paragraphs, and tables and so on...
- All elements managed by the WikiModel can be serialized/deserialized using XHTML with additional annotations (microformat-like annotations)

Some features of the CommonSyntax:
- It is a native syntax for the WikiModel. It provides full access to all features of the WikiModel.  All structures in the WikiModel can be serizlized/deszerialized in this syntax without any information lost
- It uses markup characters available in most (in ideal situation - in all) keyboard layouts (including Russian :-)). So you don't have to switch keyboard layouts to write text, tables, lists and headers. For example tables can be defined using pipe symbols ("|" - which is not available in many keyboard layouts) or the "::" sequence.
- If there is a choice then the most commonly used markups are used

The current version  of the WikiModel provides just an event-based interface  to  work with the structure of documents (like SAX for XML).
In previous versions of WIkiModel I had Document Object Model in which each structural element had its own object representation. In the current version an Object Model is not implemented (yet). I thought to create just a set of utility classes manipulating with the standard XML DOM. Example: the method WikiTable#setCellContent(int row, int column, String content) should create an XHTML table object, create the required number of cells and columns and put the given string content in this node. The same for all other structural elements (headers, lists, internal documents, properties, styles, macros...)


On 9/14/07, Vincent Massol <[hidden email]> wrote:
+1 to all that. So let me summarizes and rephrase to see if I have
understood :)

1) We have 4 types of objects:
* TextProcessors: take text and generate text
* Parsers: take text and generate an internal DOM format (pivot format)
* DomProcessors: take DOM and generate DOM
* Renderers: take DOM and generate anything (text, PDF, RTF, HTML,
XML, etc)

Yes.

2) Document contents are stored in the database in textual format in
the main xwiki syntax (whatever we decide it is - we could
standardize on creole for example)

It can be the "Common Syntax" for the reasons mentioned above :-). Creole syntax is one of the most restrictive syntaxes. And I tried to uses in the CommonSyntax as much markups of the Creole as possible.

An another possibility is to store directly in XML or in XHTML+microformat enhancements (for additional structural elements).
pro:
- it can be exported/imported directly and used by external applications which knows nothing about wikis; just a standard XML or XHTML
- this content can be transformed with XSLT processors directly without usage of the WikiModel
- it can be faster to parse XML than the CommonWiki syntax (I have no comparisons)
con:
- it is more difficult to work with diffs (but for diffs it is *better* to use WkiModel and to generate a specific wiki syntax; for example "Common syntax");
- it is not a "human readable" format; it is difficult to understand what you loads from the DB

3) Use case 1: Viewing a document
 

a) Get the doc from the DB --> text1 (xwiki text format)
b) Apply TextProcessors --> text2
c) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an
internal DOM)
d) Apply DomProcessors --> DOM2
e) Call the required Renderer --> PDF, XML, HTML, RTF, text, etc

Yes.


4) Use case 2: Editing a document, assuming the user wants to use the
MediaWiki syntax for editing

a) Get the doc from the DB --> text1 (xwiki text format)
b) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an
internal DOM)
c) Call MediaWikiRenderer --> text2 (text in MediaWiki format)
d) the user edits and hits save
e) MediaWikiParser --> DOM2 (transforms MediaWiki text syntax into
the internal DOM)
f) Call XWikiRenderer --> text" (transforms DOM into xwiki textual
format)
g) Save text3 in the database

Yes. (text1 and text3 can be XML, as I said above)


5) In practice this means the following classes:

* TextProcessorManager: to chain several text processors

Yes. But it can be just a composite processor implementing the same ProcessorManager interfaces.

* TextProcessor
   - VelocityTextProcessor
   - GroovyTextProcessor

Yes.

* WikiParser: Takes wiki syntax and generates a DOM in a XWiki-
specific format (independent of the different wiki syntaxes).
   - LegacyXWikiWikiParser
   - XWikiWikiParser (or simply use CreoleWikiParser if we want our
internal format to be Creole)
   - ConfluenceWikiParser
   - MediaWikiWikiParser
   - JSPWikiWikiParser
   - CreoleWikiParser
   - HTMLParser: I think all parsers above need to support HTML since
the wiki syntaxes can be mixed with HTML. So this HTMLParser is
probably a parent of the other parsers in some regard. Anyway we need
this one for the WYSIWYG editor which may need to transform HTML to
wiki syntax (so we may need a XWikiDomProcessor too to transform into
XWiki syntax). The alternative (much better) is to have the WYSIWYG
editor only use the internal XWiki-specific DOM format for all its
manipulations.

If you want, you can put HTML as a non-interpreted block ("verbatim blocks") and interpret it in the client code. But internally the WikiModel does not support "embedded" (X)HTML. The main reason: in this  case I loose control of the document structure. And this control is the main goal of the WikiModel.

* DomProcessorManager: to chain several DOM processors
* DomProcessor
   - Don't know yet what we're going to use this for. TOCDomProcessor
as you say above maybe.

DOMProcessor can be used to transform the original DOM object representing the document in the DB into a new (user and query-specific) DOM object which can contain new elements, generated dynamically. Now all dynamic page elements are interpreted as simple Velocity or Groovy scripts and they generate text documents which should be parsed using Radeox and transformed to the final HTML document. Using the DOM representation it is possible to interpret some nodes of this graph as Groovy scripts. In WikiModel they will correspond to Verbatim blocks which are opaque for WikiModel but they can be interpreted as scripts by the DomProcessor(s). And these "Groovy"-nodes can be executed and they will add new DOM elements to the DOM2. For example this approach can be used to generate search results.

The advantages of this approach:
- You can put your parsed document DOM1 in the cache, which will avoid you to to parse the document for each query. It is a slowest step in the page processing. Even if the current version of WikiModel is faster than before and it should be faster than Radeox processor.
- Your Groovy scripts will manipulate with normal java classes (DOM nodes) and it will produce DOM nodes and not a plain text. It seems especially interesting taking into account Groovy's Builders ( http://groovy.codehaus.org/Builders). It is enough to write a very simple builder (see http://groovy.codehaus.org/BuilderSupport) generating DOM nodes and ... voila! Your Groovy node from a wiki page generates search results as DOM nodes!  These manipulations with DOM objects should be MUCH faster that process plain text for every request. And all following steps are fast as well - to generate an HTML page it is enough to visit all node with an "XHTMLVisitor".


BTW: do you need Velocity at all? Using only Groovy is much cleaner. It can be used as THE language of XWiki. It  can be used as a template *and* programming language at the same time. And if you *really* want it is possible to integrate Jasper (from Tomcat) engine to use it for pure templating. The code from Jetty (th e org.mortbay.jetty.jspc.plugin package) can be used as an example of integration with Jasper (see http://jetty.mortbay.org/xref/index.html).
In this case in templates it will be possible to use:
- JSP tag libraries (including standard ones)
- Multiple scripting languages (like javabeans, javascript, jpython, jruby, groovy,...)


* Renderer
   - XMLRenderer
   - HTMLRenderer
   - PDFRenderer
   - RTFRenderer
   - XWikiRenderer (or simply use CreoleRenderer if we want our
internal format to be Creole)
   - ConfluenceRenderer
   - MediaWikiRenderer
   - JSPWikiRenderer
   - CreoleRenderer

Yes. All these renderers should be written if you want to support all these syntaxes. I think that it should not be very difficult.

WDYT? Do I have it right? :)



Best regards,
Mikhail
 

Thanks
-Vincent

On Sep 13, 2007, at 6:37 PM, Stéphane Laurière wrote:

> Hi Vincent, hi everyone,
>
> We discussed the WikiModel integration with Mikhail this afternoon.
> Here
> is below our input.
>
> Vincent Massol wrote:
>> Hi,
>>

>> I've started working on designing the new Rendering/Parsing
>> components and API for XWiki. The implementation will be based on
>> WikiModel but we need some XWiki wrapping interfaces around it. Note
>>  that this is a prerequisite for the new WYSIWYG editor based on GWT
>>  (see <A href="http://www.xwiki.org/xwiki/bin/view/Design/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> http://www.xwiki.org/xwiki/bin/view/Design/
>> NewWysiwygEditorBasedOnGwt).
>>
>> I've updated <A href="http://www.xwiki.org/xwiki/bin/view/Design/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.xwiki.org/xwiki/bin/view/Design/
>> WikiModelIntegration with the information below, which I'm pasting
>> here so that we can have a discussion about it. I'll consolidate the
>>  results on that wiki page.
>>
>> Componentize the Parsing/Rendering APIs
>> ==================================
>>
>> We need 4 main components:
>>
>> * A Scripting component to manage scripting inside XWiki documents
>> and to evaluate them.
>
> On the topic of scripting we would like to propose a distinction
> between
> scripts that act on text and scripts that act on the DOM.
> Typically, the
> text rendering processing for flow would be the following, for say
> "text1":
>
> text1 =TextProcessor=> text2 =Parser=> dom1 =DomProcessor=> dom2
> => ...
>
> - the scripts contained in text1 are processed in the context of
> user1,
> this results into a new text: text2
> - the parser parses text2 and converts text2 to a DOM tree, dom1
> - dom1 is processed by scripts that work directly on the DOM (example:
> table of content generator), this results in dom2
> - dom2 is made to available as such or is converted to XML, HTML, PDF
> etc. depending on the user request
>
> TextProcessor and DomProcessor would have the following interfaces:
>
> TextProcessor
> - String execute(String content)
>
> DomProcessor
> - DOM execute(DOM content)
>

> That means we should have a syntax to distinguish between scripts that
> generate text content, and scripts that manipulate the DOM.
>
>>      * A Rendering component to manage rendering Wiki syntax into
>> HTML and other (PDF, RTF, etc)
>>      * A Wiki Parser component to offer a typed interface to XWiki
>> content so that it can be manipulated
>>      * A HTML Parser component (for the WYSIWYG editor)
>>
>> Different Syntaxes ===============
>>
>> Two possible solutions:
>>
>>     1. Have a WikiSyntax Object (A simple class with one property: a
>> combox box with different syntaxes: XWiki Legacy, Creole, MediaWiki,
>> Confluence, JSPWiki, etc) that users can attach to pages to tell the
>> Renderers what syntax is used. If no such object is attached then
>> it'll default to XWiki's default syntax (XWiki Legacy or Creole for
>> example).
>>     2. Have some special syntax, independent of the wiki syntaxes to
>> tell the Rendered that such block of content should be rendered with
>> that given syntax. Again there would be a default.
>>
>
> Here's our view regarding the syntax used in wiki edit mode: document
> requested for edition are available from the database in a serialized
> format, for instance XHTML. When entering into the edit action, the
> user
> indicates his preferred syntax. If the text of the requested document
> contains some blocks that are not handled by the chosen syntax, the
> user
> gets a warning (example: the document contains a table as a list item,
> and the user tries to edit the document using JSPWiki syntax). If not,
> WikiModel converts the serialized format into a DOM, the user edits
> the
> DOM and the WikiModel serializer serializes it back when the user
> saves it.
>
> Note that the DOM representation of wiki documents in the latest
> version
> of WikiModel is still pending.
>
>>
>> XWiki Interfaces
>> =============
>>
>>      * ScriptingEngineManager: Manages the different Scripting
>> Engines, calling them in turn.
>>      * ScriptingEngine
>>            o Method: evaluate(String content)
>>            o Implementation: VelocityScriptingEngine
>>            o Implementation: GroovyScriptingEngine
>>      * RenderingEngineManager: Manages the different Rendering
>> Engines, calling them in turn.
>>      * RenderingEngine
>>            o Method: render(String content)
>>            o Implementation: XWikiLegacyRenderingEngine (current
>> rendering engine)
>>            o Implementation: WikiModelRenderingEngine
>>      * Parser: content parsing
>>            o HTMLParser: parses HTML syntax
>>            o WikiParser: parses wiki syntax
>>            o Implementation: WikiModelHTMLParser
>>            o Implementation: WikiModelWikiParser

>>
>> Open Questions:
>>
>>      * Does WikiModel support a generic syntax for macros?
>
> WikiModel generates events for blocks that are not to be parsed
> (typically because they contain scripts).
>
> For example, in the WikiModel syntax currently called "CommonSyntax",
> this looks like the following:
> ==============
> {{{macro:mymacro (String parameters)
> dothis
> dothat

>
> }}}
>
>
> $mymacro(parameters)
> ==============
>
> For each syntax, macro blocks are identified as far as possible (we
> still have to check it's the case for all types of macro blocks inde
> indeed).
>
>
>>      * Is the Rendering also in charge of generating PDF, RTF,
>> XML, etc?
>>            o I think so, need to modify interfaces above to reflect
>> this.
>>      * The WikiParser needs to recognizes scripts since this is
>> needed for the WYSIWYG editor.
>
> the WikiModel parser recognizes scripts indeed.
>
>
> Mikhail and Stéphane
>
>>
>> Use cases
>> ========
>>
>>      * View page
>>            o ViewAction -- template ->
>> ScriptingEngineManager.evaluate
>> () -- wiki syntax -> RenderingEngineManager.render() ---> HTML, XML,
>> PDF, RTF, etc
>>      * Edit page in WYSIWYG editor
>>            o Uses the WikiParser to create a "DOM" of the page
>> content and to render it accordingly. NOTE: This is required since
>> rendering in the WYSIWYG editor is different from the final
>> rendering. For example, macros need to be shown in a special way to
>> make them visible, etc.
>>            o Changes done by the user are entered in HTML. Note: it
>> would be better to capture them so that they are entered in the
>> "DOM". Is that possible? If not, then the HTMLParser is used to
>> convert from HTML to Wiki Syntax but they're likely be some loss in
>> the conversion. The advantage is the ability to take any HTML content
>> and generate wiki syntax from it.
>>
>>
>> This is my very earlier thinking but I wanted to make it visible to
>> give everyone the change to 1) know what's happening and 2) suggest
>> ideas.
>>
>> I'll refine this in the coming days and post again on this thread.
>>
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
12