XWiki Does Not Handle Attachments Efficiently

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

XWiki Does Not Handle Attachments Efficiently

AW-4
Is there anything I can do to make XWiki 1.4 handle large attachments? That
is, attachments from a few megabytes up to a few dozen megabytes.

XWiki does not appear to efficiently handle attachments in general; large
attachments appear to require extraordinary measures and resources.

In testing prior to a deployment I attempted to attach a 56 MB file to an
instance running on a server with 2GB of RAM and using MySQL as the
database.

To attach the file:

1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf to a
larger value, since by default it's only 1MB and XWiki stores attachments
directly in the database, rather than saving them in the file system and
referencing the attachment's path.

2. Next I needed change the maximum attachment size in XWiki, which can be
done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a
NumberProperty field of type long, named upload_maxsize" which I finally
figured out I could accomplish by visiting a page similar to
"http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object"
...although this wasn't immediately obvious.

3. At this point attempting to attach the file resulted in a JDBC error,
which I resolved by manually changing some columns in the database from type
mediumblob to longblob.

4. Attempting to attach the file still didn't work, this time due to Java
heap space limitations. I increased the size of the heap, modifying the
JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap sizes
to 512MB, up from 256). This had no effect on the error, so I increased the
size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the
file.

Next I attempted to delete the attachment along with the page to which it is
attached. My naïve assumption was that deleting an attachment should be
easier than adding one and I didn't anticipate problems. Unfortunately, this
again resulted in out of memory errors. To work around the problem I tried
deleting the attachment by clicking the delete button in the show -->
attachments screen. This still resulted in an out of memory error, so I
upped the heap size to 1.6 GB, or 80% of the RAM in the server. This
resulted in very heavy swapping, yet still failed due to an out of memory
exception. (On the positive side, at least the swapping helped keep the
processor utilization under 100%.)

I've ordered another 2GB of RAM and will try increasing the heap size again
after it's installed. Until then, is there anything I can do to keep
attachments from eating the server alive? If I recall correctly, there has
been talk about saving attachments externally, rather than within the
database. Has anyone built a mechanism to allow this, or is there any news
to report on that front?

Also, does anyone have any advice on how I can excise this page and its 56MB
attachment from the system prior to more memory arriving? (I'd like to do so
as, aside from the attachment problems and dysfunctional WYSIWYG editor, the
system looks deployable.)

As it stands, unless I'm doing something incorrectly, I have to conclude
that it's basically impractical to use attachments larger than a few
megabytes with XWiki, and even small attachments are frightfully ungainly.

- AW


_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: XWiki Does Not Handle Attachments Efficiently

rrodrigueznt
Hi AW,

I think there is a general agreement about the fact that an option to
store attachments, particularly big ones, outside the database. I think
it was of the main reasons to accept/promote WebDAV integration as one
of the projects to be developed during the current Google Summer of Code
(GSoC) mentored by Ludovic Dubost.

http://code.google.com/soc/2008/xwiki/about.html

Perhaps these threads will help you to catch up with this issues and
make some contribution!!!

http://www.nabble.com/About-WebDav-Integration-tp16358565p16358565.html
http://www.nabble.com/Need-help-with-WebDAV-Support.-tp16453419p16453419.html
http://www.nabble.com/-GSOC--WebDAV--Status-...-tp17588475p17588475.html
http://www.nabble.com/-GSOC---WebDAV--Ground-work-done.-tp17674844p17674844.html


Hope this helps,

Ricardo

AW wrote:

> Is there anything I can do to make XWiki 1.4 handle large attachments? That
> is, attachments from a few megabytes up to a few dozen megabytes.
>
> XWiki does not appear to efficiently handle attachments in general; large
> attachments appear to require extraordinary measures and resources.
>
> In testing prior to a deployment I attempted to attach a 56 MB file to an
> instance running on a server with 2GB of RAM and using MySQL as the
> database.
>
> To attach the file:
>
> 1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf to a
> larger value, since by default it's only 1MB and XWiki stores attachments
> directly in the database, rather than saving them in the file system and
> referencing the attachment's path.
>
> 2. Next I needed change the maximum attachment size in XWiki, which can be
> done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a
> NumberProperty field of type long, named upload_maxsize" which I finally
> figured out I could accomplish by visiting a page similar to
> "http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object"
> ...although this wasn't immediately obvious.
>
> 3. At this point attempting to attach the file resulted in a JDBC error,
> which I resolved by manually changing some columns in the database from type
> mediumblob to longblob.
>
> 4. Attempting to attach the file still didn't work, this time due to Java
> heap space limitations. I increased the size of the heap, modifying the
> JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap sizes
> to 512MB, up from 256). This had no effect on the error, so I increased the
> size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the
> file.
>
> Next I attempted to delete the attachment along with the page to which it is
> attached. My naïve assumption was that deleting an attachment should be
> easier than adding one and I didn't anticipate problems. Unfortunately, this
> again resulted in out of memory errors. To work around the problem I tried
> deleting the attachment by clicking the delete button in the show -->
> attachments screen. This still resulted in an out of memory error, so I
> upped the heap size to 1.6 GB, or 80% of the RAM in the server. This
> resulted in very heavy swapping, yet still failed due to an out of memory
> exception. (On the positive side, at least the swapping helped keep the
> processor utilization under 100%.)
>
> I've ordered another 2GB of RAM and will try increasing the heap size again
> after it's installed. Until then, is there anything I can do to keep
> attachments from eating the server alive? If I recall correctly, there has
> been talk about saving attachments externally, rather than within the
> database. Has anyone built a mechanism to allow this, or is there any news
> to report on that front?
>
> Also, does anyone have any advice on how I can excise this page and its 56MB
> attachment from the system prior to more memory arriving? (I'd like to do so
> as, aside from the attachment problems and dysfunctional WYSIWYG editor, the
> system looks deployable.)
>
> As it stands, unless I'm doing something incorrectly, I have to conclude
> that it's basically impractical to use attachments larger than a few
> megabytes with XWiki, and even small attachments are frightfully ungainly.
>
> - AW
>
>
> _______________________________________________
> users mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/users
>
>
>  


--
Ricardo Rodríguez
Your EPEC Network ICT Team

_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: XWiki Does Not Handle Attachments Efficiently

Guillaume Lerouge
Hi AW,
you may want to look at this thread that addresses most of the issues you
encountered AFAIR :

http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query:attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results

Guillaume

On Mon, Jun 9, 2008 at 11:24 PM, [Ricardo Rodriguez] Your EPEC Network ICT
Team <[hidden email]> wrote:

> Hi AW,
>
> I think there is a general agreement about the fact that an option to
> store attachments, particularly big ones, outside the database. I think
> it was of the main reasons to accept/promote WebDAV integration as one
> of the projects to be developed during the current Google Summer of Code
> (GSoC) mentored by Ludovic Dubost.
>
> http://code.google.com/soc/2008/xwiki/about.html
>
> Perhaps these threads will help you to catch up with this issues and
> make some contribution!!!
>
> http://www.nabble.com/About-WebDav-Integration-tp16358565p16358565.html
>
> http://www.nabble.com/Need-help-with-WebDAV-Support.-tp16453419p16453419.html
> http://www.nabble.com/-GSOC--WebDAV--Status-...-tp17588475p17588475.html
>
> http://www.nabble.com/-GSOC---WebDAV--Ground-work-done.-tp17674844p17674844.html
>
>
> Hope this helps,
>
> Ricardo
>
> AW wrote:
> > Is there anything I can do to make XWiki 1.4 handle large attachments?
> That
> > is, attachments from a few megabytes up to a few dozen megabytes.
> >
> > XWiki does not appear to efficiently handle attachments in general; large
> > attachments appear to require extraordinary measures and resources.
> >
> > In testing prior to a deployment I attempted to attach a 56 MB file to an
> > instance running on a server with 2GB of RAM and using MySQL as the
> > database.
> >
> > To attach the file:
> >
> > 1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf
> to a
> > larger value, since by default it's only 1MB and XWiki stores attachments
> > directly in the database, rather than saving them in the file system and
> > referencing the attachment's path.
> >
> > 2. Next I needed change the maximum attachment size in XWiki, which can
> be
> > done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a
> > NumberProperty field of type long, named upload_maxsize" which I finally
> > figured out I could accomplish by visiting a page similar to
> > "http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object"
> > ...although this wasn't immediately obvious.
> >
> > 3. At this point attempting to attach the file resulted in a JDBC error,
> > which I resolved by manually changing some columns in the database from
> type
> > mediumblob to longblob.
> >
> > 4. Attempting to attach the file still didn't work, this time due to Java
> > heap space limitations. I increased the size of the heap, modifying the
> > JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap
> sizes
> > to 512MB, up from 256). This had no effect on the error, so I increased
> the
> > size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the
> > file.
> >
> > Next I attempted to delete the attachment along with the page to which it
> is
> > attached. My naïve assumption was that deleting an attachment should be
> > easier than adding one and I didn't anticipate problems. Unfortunately,
> this
> > again resulted in out of memory errors. To work around the problem I
> tried
> > deleting the attachment by clicking the delete button in the show -->
> > attachments screen. This still resulted in an out of memory error, so I
> > upped the heap size to 1.6 GB, or 80% of the RAM in the server. This
> > resulted in very heavy swapping, yet still failed due to an out of memory
> > exception. (On the positive side, at least the swapping helped keep the
> > processor utilization under 100%.)
> >
> > I've ordered another 2GB of RAM and will try increasing the heap size
> again
> > after it's installed. Until then, is there anything I can do to keep
> > attachments from eating the server alive? If I recall correctly, there
> has
> > been talk about saving attachments externally, rather than within the
> > database. Has anyone built a mechanism to allow this, or is there any
> news
> > to report on that front?
> >
> > Also, does anyone have any advice on how I can excise this page and its
> 56MB
> > attachment from the system prior to more memory arriving? (I'd like to do
> so
> > as, aside from the attachment problems and dysfunctional WYSIWYG editor,
> the
> > system looks deployable.)
> >
> > As it stands, unless I'm doing something incorrectly, I have to conclude
> > that it's basically impractical to use attachments larger than a few
> > megabytes with XWiki, and even small attachments are frightfully
> ungainly.
> >
> > - AW
> >
> >
> > _______________________________________________
> > users mailing list
> > [hidden email]
> > http://lists.xwiki.org/mailman/listinfo/users
> >
> >
> >
>
>
> --
> Ricardo Rodríguez
> Your EPEC Network ICT Team
>
> _______________________________________________
> users mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/users
>



--
http://wikibc.blogspot.com/
_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: XWiki Does Not Handle Attachments Efficiently

goldring, richard
In reply to this post by AW-4
Yes storing attachments directly in the file system and referencing them
would be good.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]On Behalf
Of AW
Sent: 09 June 2008 21:15
To: XWiki Users
Subject: [xwiki-users] XWiki Does Not Handle Attachments Efficiently


Is there anything I can do to make XWiki 1.4 handle large attachments? That
is, attachments from a few megabytes up to a few dozen megabytes.

XWiki does not appear to efficiently handle attachments in general; large
attachments appear to require extraordinary measures and resources.

In testing prior to a deployment I attempted to attach a 56 MB file to an
instance running on a server with 2GB of RAM and using MySQL as the
database.

To attach the file:

1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf to a
larger value, since by default it's only 1MB and XWiki stores attachments
directly in the database, rather than saving them in the file system and
referencing the attachment's path.

2. Next I needed change the maximum attachment size in XWiki, which can be
done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a
NumberProperty field of type long, named upload_maxsize" which I finally
figured out I could accomplish by visiting a page similar to
"http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object"
...although this wasn't immediately obvious.

3. At this point attempting to attach the file resulted in a JDBC error,
which I resolved by manually changing some columns in the database from type
mediumblob to longblob.

4. Attempting to attach the file still didn't work, this time due to Java
heap space limitations. I increased the size of the heap, modifying the
JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap sizes
to 512MB, up from 256). This had no effect on the error, so I increased the
size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the
file.

Next I attempted to delete the attachment along with the page to which it is
attached. My naïve assumption was that deleting an attachment should be
easier than adding one and I didn't anticipate problems. Unfortunately, this
again resulted in out of memory errors. To work around the problem I tried
deleting the attachment by clicking the delete button in the show -->
attachments screen. This still resulted in an out of memory error, so I
upped the heap size to 1.6 GB, or 80% of the RAM in the server. This
resulted in very heavy swapping, yet still failed due to an out of memory
exception. (On the positive side, at least the swapping helped keep the
processor utilization under 100%.)

I've ordered another 2GB of RAM and will try increasing the heap size again
after it's installed. Until then, is there anything I can do to keep
attachments from eating the server alive? If I recall correctly, there has
been talk about saving attachments externally, rather than within the
database. Has anyone built a mechanism to allow this, or is there any news
to report on that front?

Also, does anyone have any advice on how I can excise this page and its 56MB
attachment from the system prior to more memory arriving? (I'd like to do so
as, aside from the attachment problems and dysfunctional WYSIWYG editor, the
system looks deployable.)

As it stands, unless I'm doing something incorrectly, I have to conclude
that it's basically impractical to use attachments larger than a few
megabytes with XWiki, and even small attachments are frightfully ungainly.

- AW


_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users

Thales UK Ltd (Wells) DISCLAIMER: The information contained in this e-mail
is confidential. It may also be legally privileged. It is intended only for
the stated addressee(s) and access to it by any other person is
unauthorised. If you are not an addressee, you must not disclose, copy,
circulate or in any other way use or rely on the information contained in
this e-mail. Such unauthorised use may be unlawful. We may monitor all
e-mail communications through our networks. If you have received this e-mail
in error, please inform us immediately on +44 (0) 1749 672081 and delete it
and all copies from your system. We accept no responsibility for changes to
any e-mail which occur after it has been sent.  Attachments to this e-mail
may contain software viruses which could damage your system.  We therefore
recommend you virus-check all attachments before opening. A business of
Thales UK Ltd. Registered Office: 2 Dashwood Lang Road, The Bourne Business
Park, Addlestone, Weybridge, Surrey KT15 2NX Registered in England No.
868273
_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: XWiki Does Not Handle Attachments Efficiently

Artem Melentyev-3
In reply to this post by AW-4
Hi.

You can set xwiki.store.attachment.versioning=0 in xwiki.cfg
This will turn off attachment versioning and *very* much reduce memory
consumption.
see http://jira.xwiki.org/jira/browse/XWIKI-2275
and
http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Configuration#HOptionalStoreFeatures

AW wrote:
> Is there anything I can do to make XWiki 1.4 handle large attachments? That
> is, attachments from a few megabytes up to a few dozen megabytes.
>
>...

--
  Artem Melentyev
_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: XWiki Does Not Handle Attachments Efficiently

vmassol
Administrator
Hi Artem,

On Jun 12, 2008, at 2:21 PM, Artem Melentyev wrote:

> Hi.
>
> You can set xwiki.store.attachment.versioning=0 in xwiki.cfg
> This will turn off attachment versioning and *very* much reduce memory
> consumption.
> see http://jira.xwiki.org/jira/browse/XWIKI-2275
> and
> http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Configuration#HOptionalStoreFeatures

Why is it using some much memory? Isn't that a bug? (I don't  
understand why we would need to load more than 1 attachment version at  
a time in memory).

Thanks
-Vincent

> AW wrote:
>> Is there anything I can do to make XWiki 1.4 handle large  
>> attachments? That
>> is, attachments from a few megabytes up to a few dozen megabytes.
_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: XWiki Does Not Handle Attachments Efficiently

Artem Melentyev-3
Hi, Vincent.

Vincent Massol wrote:
> Why is it using some much memory? Isn't that a bug? (I don't  
> understand why we would need to load more than 1 attachment version at  
> a time in memory).

Look at
http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query:attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results
> Worse, if this is not the first version of the attachment, then the
> complete attachment history is loaded in memory, so add another
> 24*versionsize*versions of memory needed during upload.

This is a feature of JRCS :)

--
  Artem Melentyev
_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: XWiki Does Not Handle Attachments Efficiently

vmassol
Administrator

On Jun 12, 2008, at 2:45 PM, Artem Melentyev wrote:

> Hi, Vincent.
>
> Vincent Massol wrote:
>> Why is it using some much memory? Isn't that a bug? (I don't
>> understand why we would need to load more than 1 attachment version  
>> at
>> a time in memory).
>
> Look at
> http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query 
> :attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results
>> Worse, if this is not the first version of the attachment, then the
>> complete attachment history is loaded in memory, so add another
>> 24*versionsize*versions of memory needed during upload.
>
> This is a feature of JRCS :)

I see. So I guess the only solution would be drop JRCS and do our own  
stuff... (or find another library that does it fast).

Thanks
-Vincent

_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: XWiki Does Not Handle Attachments Efficiently

Shiva Shankar
Hi
There was another discussion on this topic, where an effort has been made to store the files directly on the filesystem. Not sure how this would reduce memory consumption. Please read Sergiu's comments carefully though, before you try this out.
Link here
Of course, in the meantime, we await the WebDAV implementation.
Shiva

vmassol wrote
On Jun 12, 2008, at 2:45 PM, Artem Melentyev wrote:

> Hi, Vincent.
>
> Vincent Massol wrote:
>> Why is it using some much memory? Isn't that a bug? (I don't
>> understand why we would need to load more than 1 attachment version  
>> at
>> a time in memory).
>
> Look at
> http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query 
> :attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results
>> Worse, if this is not the first version of the attachment, then the
>> complete attachment history is loaded in memory, so add another
>> 24*versionsize*versions of memory needed during upload.
>
> This is a feature of JRCS :)

I see. So I guess the only solution would be drop JRCS and do our own  
stuff... (or find another library that does it fast).

Thanks
-Vincent

_______________________________________________
users mailing list
users@xwiki.org
http://lists.xwiki.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: XWiki Does Not Handle Attachments Efficiently

Sergiu Dumitriu-2
shivshan wrote:

> Hi
> There was another discussion on this topic, where an effort has been made to
> store the files directly on the filesystem. Not sure how this would reduce
> memory consumption. Please read Sergiu's comments carefully though, before
> you try this out.
> Link
> http://www.nabble.com/AttachmentStore---my-filesystem-based-implementation-td16987518.html#a17040326
> here
> Of course, in the meantime, we await the WebDAV implementation.
> Shiva

I'm not sure the webdav implementation will solve the memory problem. If I understand correctly how
it will work, then it will just be an interface over the same core, meaning that all the performance
issues will still be there.

>
> vmassol wrote:
>>
>> On Jun 12, 2008, at 2:45 PM, Artem Melentyev wrote:
>>
>>> Hi, Vincent.
>>>
>>> Vincent Massol wrote:
>>>> Why is it using some much memory? Isn't that a bug? (I don't
>>>> understand why we would need to load more than 1 attachment version  
>>>> at
>>>> a time in memory).
>>> Look at
>>> http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query 
>>> :attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results
>>>> Worse, if this is not the first version of the attachment, then the
>>>> complete attachment history is loaded in memory, so add another
>>>> 24*versionsize*versions of memory needed during upload.
>>> This is a feature of JRCS :)
>> I see. So I guess the only solution would be drop JRCS and do our own  
>> stuff... (or find another library that does it fast).
>>
>> Thanks
>> -Vincent
>>

--
Sergiu Dumitriu
http://purl.org/net/sergiu/
_______________________________________________
users mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/users