non-english languages

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

non-english languages

Andrey V. Karlov
Hello everybody!

  I'm try to using XWiki 0.9.840 in the own projects, but have a lot
  of problem with Russian text. Maybe I do something wrong, so please
  help me if possible.

  I'm using KOI8-R encoding in XWiki. Russian text is edited and
  viewed correctly.

  1. But it's impossible to search on the Russian - search box
  contains "????" after Sumbit. Looks like incorrect encoding is used.

  2. When I try to use Lists Syntax - it's not working at all. Some
  lines are displayed as bold, some lines as regular text.

  3. It's impossible to create new page with Russian name. It's always
  saved as "?????".

  Thank you for your comments!

--
Best regards,
 Andrey                          mailto:[hidden email]




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
Reply | Threaded
Open this post in threaded view
|

Re: non-english languages

nguyen manh toan-2
Hello,

This is our problem. We use Vietnamese and UTF-8 encode.

please help!
----- Original Message -----
From: "Andrey V. Karlov" <[hidden email]>
To: <[hidden email]>
Sent: Monday, October 31, 2005 4:53 PM
Subject: [xwiki-users] non-english languages


> Hello everybody!
>
>   I'm try to using XWiki 0.9.840 in the own projects, but have a lot
>   of problem with Russian text. Maybe I do something wrong, so please
>   help me if possible.
>
>   I'm using KOI8-R encoding in XWiki. Russian text is edited and
>   viewed correctly.
>
>   1. But it's impossible to search on the Russian - search box
>   contains "????" after Sumbit. Looks like incorrect encoding is used.
>
>   2. When I try to use Lists Syntax - it's not working at all. Some
>   lines are displayed as bold, some lines as regular text.
>
>   3. It's impossible to create new page with Russian name. It's always
>   saved as "?????".
>
>   Thank you for your comments!
>
> --
> Best regards,
>  Andrey                          mailto:[hidden email]
>
>
>

----------------------------------------------------------------------------
----


>
> --
> You receive this message as a subscriber of the [hidden email]
mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
>





--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
Reply | Threaded
Open this post in threaded view
|

Re: non-english languages

Youcef Bey
Hi all,

There are no problem if you convert all your data to utf-8 (all work well).
If you want to use both, you can develop an internal converter to utf-8.

Best wishes!

Youcef



----- Original Message -----
From: "Nguyen Manh Toan" <[hidden email]>
To: "Andrey V. Karlov" <[hidden email]>; <[hidden email]>
Sent: Monday, October 31, 2005 6:57 PM
Subject: Re: [xwiki-users] non-english languages


> Hello,
>
> This is our problem. We use Vietnamese and UTF-8 encode.
>
> please help!
> ----- Original Message -----
> From: "Andrey V. Karlov" <[hidden email]>
> To: <[hidden email]>
> Sent: Monday, October 31, 2005 4:53 PM
> Subject: [xwiki-users] non-english languages
>
>
> > Hello everybody!
> >
> >   I'm try to using XWiki 0.9.840 in the own projects, but have a lot
> >   of problem with Russian text. Maybe I do something wrong, so please
> >   help me if possible.
> >
> >   I'm using KOI8-R encoding in XWiki. Russian text is edited and
> >   viewed correctly.
> >
> >   1. But it's impossible to search on the Russian - search box
> >   contains "????" after Sumbit. Looks like incorrect encoding is used.
> >
> >   2. When I try to use Lists Syntax - it's not working at all. Some
> >   lines are displayed as bold, some lines as regular text.
> >
> >   3. It's impossible to create new page with Russian name. It's always
> >   saved as "?????".
> >
> >   Thank you for your comments!
> >
> > --
> > Best regards,
> >  Andrey                          mailto:[hidden email]
> >
> >
> >
>
>
> --------------------------------------------------------------------------
--
> ----
>
>
> >
> > --
> > You receive this message as a subscriber of the
[hidden email]
> mailing list.
> > To unsubscribe: mailto:[hidden email]
> > For general help: mailto:[hidden email]?subject=help
> > ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
> >
>
>
>
>


----------------------------------------------------------------------------
----


>
> --
> You receive this message as a subscriber of the [hidden email]
mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
>




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
Reply | Threaded
Open this post in threaded view
|

Re[2]: non-english languages

Andrey V. Karlov
In reply to this post by Andrey V. Karlov
Youcef,

> Convert your data encoding to utf-8, the procedure to configure utf-8 under
> xwiki it is well explained (I dont remember the link, but try to find it).
> All PB with encoding will be solved for each language!

  thank you for quick response, but can't agree with you. I tried to
  use UTF-8 - the same problems with search and page naming.

  btw, how-to for UTF-8 is not very well in "database" part. For
  instance, text "change the mysql encoding to UTF-8 (check it using
  "show variables")" should be replaced by following instruction
  (based by my experience):

  1. (For MySQL 4.1.x) Create database with default UTF-8 encoding.
  Example: CREATE DATABASE xwiki DEFAULT CHARACTER SET utf8;

  2. Add these lines to WEB-INF\hibernate.cfg.xml (don't know, why
  team suggest to use my.cnf setting for this instead of configure a
  DB pool connection):
  <property name="connection.characterEncoding">utf8</property>
  <property name="connection.useUnicode">true</property>
 


>> Hello everybody!
>>
>>   I'm try to using XWiki 0.9.840 in the own projects, but have a lot
>>   of problem with Russian text. Maybe I do something wrong, so please
>>   help me if possible.
>>
>>   I'm using KOI8-R encoding in XWiki. Russian text is edited and
>>   viewed correctly.
>>
>>   1. But it's impossible to search on the Russian - search box
>>   contains "????" after Sumbit. Looks like incorrect encoding is used.
>>
>>   2. When I try to use Lists Syntax - it's not working at all. Some
>>   lines are displayed as bold, some lines as regular text.
>>
>>   3. It's impossible to create new page with Russian name. It's always
>>   saved as "?????".
>>
>>   Thank you for your comments!





--
Best regards,
 Andrey                            mailto:[hidden email]




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
Reply | Threaded
Open this post in threaded view
|

non-english languages

jeremi joslin
In reply to this post by Andrey V. Karlov
have you read this page :
http://www.xwiki.org/xwiki/bin/view/Dev/CharactersSets

Jérémi

On 10/31/05, Andrey V. Karlov <[hidden email]> wrote:

> Hello everybody!
>
>   I'm try to using XWiki 0.9.840 in the own projects, but have a lot
>   of problem with Russian text. Maybe I do something wrong, so please
>   help me if possible.
>
>   I'm using KOI8-R encoding in XWiki. Russian text is edited and
>   viewed correctly.
>
>   1. But it's impossible to search on the Russian - search box
>   contains "????" after Sumbit. Looks like incorrect encoding is used.
>
>   2. When I try to use Lists Syntax - it's not working at all. Some
>   lines are displayed as bold, some lines as regular text.
>
>   3. It's impossible to create new page with Russian name. It's always
>   saved as "?????".
>
>   Thank you for your comments!


--
http://www.jeremi.info
http://www.xwiki.org
skype: jeremi23


--
http://www.jeremi.info
http://www.xwiki.org
skype: jeremi23



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
Reply | Threaded
Open this post in threaded view
|

Re: non-english languages

Ludovic Dubost
In reply to this post by Andrey V. Karlov

Hi,

You are welcome to modify the instructions concerning UTF-8 on XWiki.org
http://www.xwiki.org/xwiki/bin/view/Dev/CharactersSets

I think all the UTF-8 issues are a mater of setup, since ObjectWeb and
Mandriva are running on UTF-8 without issues.
They might have done some slight modifications in the Java code which
are now in the SVN code.. It's probably a better idea to work with the
SVN code for an UTF-8 setup.

Ludovic

Andrey V. Karlov a écrit :

> Youcef,
>
>  
>> Convert your data encoding to utf-8, the procedure to configure utf-8 under
>> xwiki it is well explained (I dont remember the link, but try to find it).
>> All PB with encoding will be solved for each language!
>>    
>
>   thank you for quick response, but can't agree with you. I tried to
>   use UTF-8 - the same problems with search and page naming.
>
>   btw, how-to for UTF-8 is not very well in "database" part. For
>   instance, text "change the mysql encoding to UTF-8 (check it using
>   "show variables")" should be replaced by following instruction
>   (based by my experience):
>
>   1. (For MySQL 4.1.x) Create database with default UTF-8 encoding.
>   Example: CREATE DATABASE xwiki DEFAULT CHARACTER SET utf8;
>
>   2. Add these lines to WEB-INF\hibernate.cfg.xml (don't know, why
>   team suggest to use my.cnf setting for this instead of configure a
>   DB pool connection):
>   <property name="connection.characterEncoding">utf8</property>
>   <property name="connection.useUnicode">true</property>
>  
>
>
>  
>>> Hello everybody!
>>>
>>>   I'm try to using XWiki 0.9.840 in the own projects, but have a lot
>>>   of problem with Russian text. Maybe I do something wrong, so please
>>>   help me if possible.
>>>
>>>   I'm using KOI8-R encoding in XWiki. Russian text is edited and
>>>   viewed correctly.
>>>
>>>   1. But it's impossible to search on the Russian - search box
>>>   contains "????" after Sumbit. Looks like incorrect encoding is used.
>>>
>>>   2. When I try to use Lists Syntax - it's not working at all. Some
>>>   lines are displayed as bold, some lines as regular text.
>>>
>>>   3. It's impossible to create new page with Russian name. It's always
>>>   saved as "?????".
>>>
>>>   Thank you for your comments!
>>>      
>
>
>
>
>
>
>  
> ------------------------------------------------------------------------
>
>
> --
> You receive this message as a subscriber of the [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
>  

--
Ludovic Dubost
XPertNet: http://www.xpertnet.fr/
Blog: http://www.ludovic.org/blog/
XWiki: http://www.xwiki.com
Skype: ldubost AIM: nvludo Yahoo: ludovic




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
Reply | Threaded
Open this post in threaded view
|

Re[2]: non-english languages

Andrey V. Karlov
In reply to this post by Youcef Bey
Youcef,

> There are no problem if you convert all your data to utf-8 (all work
> well). If you want to use both, you can develop an internal
> converter to utf-8.

  I compiled XWiki from the SVN and installed it with UTF-8. Problem
  with Lists is gone, but I still can't search and create pages with
  non-english characters. What I do wrong?

  Is it possible to provide me link to XWiki with correct UTF-8
  support?

> ----- Original Message -----
> From: "Nguyen Manh Toan" <[hidden email]>
> To: "Andrey V. Karlov" <[hidden email]>; <[hidden email]>
> Sent: Monday, October 31, 2005 6:57 PM
> Subject: Re: [xwiki-users] non-english languages


>> Hello,
>>
>> This is our problem. We use Vietnamese and UTF-8 encode.
>>
>> please help!
>> ----- Original Message -----
>> From: "Andrey V. Karlov" <[hidden email]>
>> To: <[hidden email]>
>> Sent: Monday, October 31, 2005 4:53 PM
>> Subject: [xwiki-users] non-english languages
>>
>>
>> > Hello everybody!
>> >
>> >   I'm try to using XWiki 0.9.840 in the own projects, but have a lot
>> >   of problem with Russian text. Maybe I do something wrong, so please
>> >   help me if possible.
>> >
>> >   I'm using KOI8-R encoding in XWiki. Russian text is edited and
>> >   viewed correctly.
>> >
>> >   1. But it's impossible to search on the Russian - search box
>> >   contains "????" after Sumbit. Looks like incorrect encoding is used.
>> >
>> >   2. When I try to use Lists Syntax - it's not working at all. Some
>> >   lines are displayed as bold, some lines as regular text.
>> >
>> >   3. It's impossible to create new page with Russian name. It's always
>> >   saved as "?????".
>> >
>> >   Thank you for your comments!
>> >
>> > --
>> > Best regards,
>> >  Andrey                          mailto:[hidden email]
>> >
>> >
>> >
>>
>>
>> --------------------------------------------------------------------------
> --
>> ----
>>
>>
>> >
>> > --
>> > You receive this message as a subscriber of the
> [hidden email]
>> mailing list.
>> > To unsubscribe: mailto:[hidden email]
>> > For general help: mailto:[hidden email]?subject=help
>> > ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
>> >
>>
>>
>>
>>

> ----------------------------------------------------------------------------
> ----


>>
>> --
>> You receive this message as a subscriber of the [hidden email]
> mailing list.
>> To unsubscribe: mailto:[hidden email]
>> For general help: mailto:[hidden email]?subject=help
>> ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
>>





--
Best regards,
 Andrey                            mailto:[hidden email]




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
Reply | Threaded
Open this post in threaded view
|

Re: non-english languages

Artem Melentyev
Andrey V. Karlov wrote:

>   I compiled XWiki from the SVN and installed it with UTF-8. Problem
>   with Lists is gone, but I still can't search and create pages with
>   non-english characters. What I do wrong?
>
>   Is it possible to provide me link to XWiki with correct UTF-8
>   support?

I attach my patch. This patch dully recode document name, parent, and
user request parameters from ISO to UTF-8. Tested in Tomcat 5.0.28,
5.5.12. (but it is not work in Jetty 5.1.5rc2, i don't know why)
I use this changes on http://usunet.ru/xwiki/ (temporary unavailable) in
Russian. All works fine, except ru search (i will see it soon).

You may contact me for some support xwiki in _Russian_. But i'm only
"google summer of code" student in xwiki.


The main problem is that Servlet API(tomcat, jetty, etc) recode request
parameters only for POST submit. It does not recode request URI && GET
parameters.


patch created from:
-------- Original Message --------
Subject: [xwiki-dev] Re: UTF-8 characters (again)
Date: Sat, 23 Jul 2005 20:52:57 +0600
From: Artem Melentev <[hidden email]>
Reply-To: [hidden email]
To: [hidden email]
CC: [hidden email]

Xavier MOGHRABI wrote:
 > xWiki works with UTF-8 Characters after some modifications, live
example is on
 > http://zh.objectweb.org
 >
 > There are some modifications to do in the code of xWiki, you need to
configure
 > xWiki to manage UTF-8 and to use a database without non UTF-8 code.

Whether use of non-English _names_ of documents is possible?
I have made all that is written in
http://www.xwiki.org/xwiki/bin/view/Dev/CharactersSets
But Russian names of documents displays incorrectly. On zh.objectweb.org
too.

  From a code I have understood, it of that request.getPathInfo and
getParametr (for parameters in uri) are not decoded in utf.
It so, or I have incorrectly setup xwiki?
(or it is servlet api bug?)

I set SetCharacterEncodingFilter off by add ignore=false and rewrite
XWikiServletRequest.get, getPathInfo, getParameter to decode from
ISO-8859-1 t? UTF-8
Names now working fine, but i found next problem:
when i click on new,unedited link [newlink], open edit form. In this
form field "Parent" fills not decoded.
PrepareEditForm.setParent called 2 times, first call is decoded (from
PrepareEditForm.reset), second - not decoded (from beenutils).

Main questions:
1 That second PrepareEditForm.setParent call for?
2 Why it is not decoded?
3 How i may decode it?

Now i`m decode in setParent. This is bad idea, but works..

Soon i will open http://usunet.ru/wiki/ - Ural State University XWiki.

~Environment~:
Gentoo Linux, LANG="ru_RU.UTF-8"
Sun JDK 1.5.0.04, JAVAOPTS="-Dfile.encoding=UTF-8"
Tomcat 5.0.27-r6
DB: empty unicode db on PostgreSQL v8.0.3

(Sorry my English)
--

--
Artem Melentev, UralSU, CS301. http://usunet.ru/xwiki/
        jabber: [hidden email] | icq: 175349029

Index: /home/mart/workspace/xwiki/src/main/java/com/xpn/xwiki/web/PrepareEditForm.java
===================================================================
--- /home/mart/workspace/xwiki/src/main/java/com/xpn/xwiki/web/PrepareEditForm.java (revision 912)
+++ /home/mart/workspace/xwiki/src/main/java/com/xpn/xwiki/web/PrepareEditForm.java (working copy)
@@ -17,7 +17,7 @@
 
  * Created by
  * User: Ludovic Dubost
- * Date: 30 d?c. 2003
+ * Date: 30 d?c. 2003
  * Time: 09:12:48
  */
 package com.xpn.xwiki.web;
@@ -53,7 +53,7 @@
     }
 
     public void setParent(String parent) {
-        this.parent = parent;
+        this.parent = XWikiServletRequest._decode( parent );
     }
 
     public String getCreator() {
Index: /home/mart/workspace/xwiki/src/main/java/com/xpn/xwiki/web/XWikiServletRequest.java
===================================================================
--- /home/mart/workspace/xwiki/src/main/java/com/xpn/xwiki/web/XWikiServletRequest.java (revision 912)
+++ /home/mart/workspace/xwiki/src/main/java/com/xpn/xwiki/web/XWikiServletRequest.java (working copy)
@@ -46,6 +46,15 @@
 
 public class XWikiServletRequest implements XWikiRequest {
     private HttpServletRequest request;
+
+ public final static String _decode(String s) {
+ if (s==null) return s;
+ try {
+ return new String(s.getBytes("ISO-8859-1"));
+ } catch (UnsupportedEncodingException e) {
+ return s;
+ }
+ }
 
     public XWikiServletRequest(HttpServletRequest request) {
         this.request = request;
@@ -51,9 +60,9 @@
         this.request = request;
     }
 
-    public String get(String name) {
-        return request.getParameter(name);
-    }
+ public String get(String name) {
+ return _decode( request.getParameter(name) );
+ }
 
     public HttpServletRequest getHttpServletRequest() {
         return request;
@@ -75,9 +84,10 @@
         return request.getHeader(s);
     }
 
-    public Enumeration getHeaders(String s) {
-        return request.getHeaders(s);
-    }
+ /** not recoded! */
+ public Enumeration getHeaders(String s) {
+ return request.getHeaders(s);
+ }
 
     public Enumeration getHeaderNames() {
         return request.getHeaderNames();
@@ -91,9 +101,9 @@
         return request.getMethod();
     }
 
-    public String getPathInfo() {
-        return request.getPathInfo();
-    }
+ public String getPathInfo() {
+ return _decode( request.getPathInfo() );
+ }
 
     public String getPathTranslated() {
         return request.getPathTranslated();
@@ -150,7 +160,6 @@
         return request.isRequestedSessionIdValid();
     }
 
-
     public boolean isRequestedSessionIdFromCookie() {
         return request.isRequestedSessionIdFromCookie();
     }
@@ -194,21 +203,24 @@
         return request.getInputStream();
     }
 
-    public String getParameter(String s) {
-        return request.getParameter(s);
-    }
+ public String getParameter(String s) {
+ return _decode( request.getParameter(s) );
+ }
 
-    public Enumeration getParameterNames() {
-        return request.getParameterNames();
-    }
+ /** not recoded! */
+ public Enumeration getParameterNames() {
+ return request.getParameterNames();
+ }
 
-    public String[] getParameterValues(String s) {
-        return request.getParameterValues(s);
-    }
+ /** not recoded! */
+ public String[] getParameterValues(String s) {
+ return request.getParameterValues(s);
+ }
 
-    public Map getParameterMap() {
-        return request.getParameterMap();
-    }
+ /** not recoded! */
+ public Map getParameterMap() {
+ return request.getParameterMap();
+ }
 
     public String getProtocol() {
         return request.getProtocol();
@@ -285,9 +297,9 @@
         return request.getLocalPort();
     }
 
-    /*
-    * Portlet Functions. They do nothing in  the servlet environement
-    */
+ /*
+ * Portlet Functions. They do nothing in the servlet environement
+ */
 
     public boolean isWindowStateAllowed(WindowState windowState) {
         return false;
@@ -345,5 +357,3 @@
         return null;
     }
 }
-
-

Index: /home/mart/workspace/xwiki/src/main/web.xml
===================================================================
--- /home/mart/workspace/xwiki/src/main/web.xml (revision 912)
+++ /home/mart/workspace/xwiki/src/main/web.xml (working copy)
@@ -14,7 +14,11 @@
             <filter-class>com.xpn.xwiki.web.SetCharacterEncodingFilter</filter-class>
             <init-param>
                 <param-name>encoding</param-name>
-                <param-value>ISO-8859-1</param-value>
+                <param-value>UTF-8</param-value>
+            </init-param>
+            <init-param>
+                <param-name>ignore</param-name>
+                <param-value>false</param-value>
             </init-param>
    </filter>
 


--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
Reply | Threaded
Open this post in threaded view
|

Re: non-english languages

Artem Melentyev
Artem Melentev wrote:

> The main problem is that Servlet API(tomcat, jetty, etc) recode request
> parameters only for POST submit. It does not recode request URI && GET
> parameters.

I've checked Jetty 5.1.5rc2. It uses UTF-8 as default for all, and
request recode right .
SVN version works on it fine (in Russian).
(My patch with Jetty works incorrectly because of double recoding. But
it possible to set jetty using ISO)

--
Artem Melentev, UralSU, CS301. http://usunet.ru/xwiki/
        jabber: [hidden email] | icq: 175349029



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws