[Brainstorm] Notifications Filters Preferences Store.

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Brainstorm] Notifications Filters Preferences Store.

Guillaume Delhumeau
Hi.

[TL;DR]

This thread is about the way we store notification filter preferences for
each user. The constraint is there can be a lot of them (700 is a number a
user has recently reported). So how should we store them?

[Full text]

= Definition =

So what is a filter preference? It's a generic object that can store many
elements, such as a page locations, application names, event types, etc...
They describe a configuration about a given filter for a given user. For
example, a filter preference can say "for the ScopeNotificationFilter and
the user A, include the location Main.WebHome" as it could be "for the
UserNotificationFilter and the user A, exclude the user SPAM". It's generic.

The main usage is for page locations (ScopeNotificationFilter). By default,
we have the "autowatch" mode enabled. It means every time a user modifies a
page, a filter preference for this page and this user is created. So if a
user modifies 700 pages, he gets 700 filter preferences.

= How are they stored =

Currently, we have a simple implementation. There is a generic XClass
called "XWiki.Notifications.Code.NotificationFilterPreferenceClass". For
each preference, we add an XObject on the user page. It's that simple. But
it also means that if a users have 700 filter preferences, she also gets
700 XObjects on her page, and 700 revisions of that page. Which is a pain:
it takes a lot of place in the document's cache, and it's heavy to load
(lot of SQL queries needed). So we have a big problem here.

= Possible solutions =

== A: Minimize the number of xobjects needed for ScopeNotificationFilter ==

Currently, one location is represented by 1 filter preference. But most
filter preferences are very similar. They almost all say "for the
ScopeNotificationFilter, for all event types, for all applications, the
filter preference is enabled". The only different part is the actual
location. But the "location" field is itself a LIST stored with the
"relational storage" option. So we can take advantage of it and store
similar preferences into 1 single object.

1 object with 700 locations instead of 700 objects with 1 location.

However, it's a bit harder than this. Event if the
NotificationFilterPreferences is generic and can contains many locations,
the ScopeNotificationFilter expect it to concern only one location (and
then it perform complex operations to sort the filters preferences
according to a hierarchy). The UI in the user profile makes the same
assumption so it does not handle multiple locations in the same preferences
object. Refactoring this is not simple and cannot be done for 10.6.

=== Variation 1: store only 1 xobject, but make the API return 700
preferences objects anyway ===

This is the variation I am prototyping. Actually it's ok if the filters and
the UI expect only 1 location into the preferences object. All we have to
do is to "smash" the xobject into many NotificationFilterPreferences
objects that we need internally. It would simply be the responsibility of
the Store to detect similarities and to save the minimal amount of XObjects
to store a bunch of preferences.

But it means being very smart when loading, creating, updating and deleting
a preference. Not having one xobject per filter preference introduces
complexity, and complexity can lead to bugs. Again, according to the time
frame, it's hard to implement.

=== Variation 2: use custom mapping ===

Probably the easiest solution that would help making less SQL queries. The
idea is to have a SQL table for notification filter preferences and bind
the XObjects to that table. It would still use a lot of place in the
document's cache but be more efficient on the database level.

=== Other Problem 1: it still creates page revisions ===

As long as we store the filter preferences with xobjects, we create page
revisions. We can get rid of those by using some internal API to not create
a revision when we save an xobject but I wonder if it's what users want. If
a user tries to rollback some changes and don't see all filter preferences
it concerns, I think it's not very transparent.

=== Other Problem 2: Document's cache ===

Sometime we load the a user document to get the avatar of the user, her
name, etc... So we load user documents very frequently, even if the user is
not connected! Having 700 filters in the document and cache them with the
document even if we don't need them is a big waste of memory.

== B: Implement a completely new store with Hibernate ==

A bit like having a custom mapping. We could create a SQL table and
implement an API to handle it. Then, no xobjects would be involved.

Some drawbacks:
* we need to write a custom cache as well.
* the user cannot modify her preferences using the wiki principles
(xobjects all the way).

== C: Refactor the UI and the ScopeNotificationFilter so they do not assume
1 filter preference = 1 location ==

This option is still possible. Probably the best because creating 1 filter
preferences object per location is an obvious waste of memory. A
refactoring of the UI is needed anyway, because we currently have no way to
remove a bunch of filter preferences easily (users have to delete the 700
filters preferences manually) so we can kill 2 birds with the same stone.

But again, it requires some work.

= Conclusion =

That's it. All possible solutions require development effort that is hardly
possible to make before 10.6 (and even 10.7, considering I would probably
be the one implementing it and I'm not fulltime on the subject and I have
holidays soon).

Writing this email helped me to see the problem with perspective. I think
solution C may be the best. But any opinion is good to hear (except if you
propose something even more complex than I do :p).

Thanks,

Guillaume
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorm] Notifications Filters Preferences Store.

Guillaume Delhumeau
Corresponding JIRA issue: https://jira.xwiki.org/browse/XWIKI-15445

2018-07-18 11:07 GMT+02:00 Guillaume Delhumeau <
[hidden email]>:

> Hi.
>
> [TL;DR]
>
> This thread is about the way we store notification filter preferences for
> each user. The constraint is there can be a lot of them (700 is a number a
> user has recently reported). So how should we store them?
>
> [Full text]
>
> = Definition =
>
> So what is a filter preference? It's a generic object that can store many
> elements, such as a page locations, application names, event types, etc...
> They describe a configuration about a given filter for a given user. For
> example, a filter preference can say "for the ScopeNotificationFilter and
> the user A, include the location Main.WebHome" as it could be "for the
> UserNotificationFilter and the user A, exclude the user SPAM". It's generic.
>
> The main usage is for page locations (ScopeNotificationFilter). By
> default, we have the "autowatch" mode enabled. It means every time a user
> modifies a page, a filter preference for this page and this user is
> created. So if a user modifies 700 pages, he gets 700 filter preferences.
>
> = How are they stored =
>
> Currently, we have a simple implementation. There is a generic XClass
> called "XWiki.Notifications.Code.NotificationFilterPreferenceClass". For
> each preference, we add an XObject on the user page. It's that simple. But
> it also means that if a users have 700 filter preferences, she also gets
> 700 XObjects on her page, and 700 revisions of that page. Which is a pain:
> it takes a lot of place in the document's cache, and it's heavy to load
> (lot of SQL queries needed). So we have a big problem here.
>
> = Possible solutions =
>
> == A: Minimize the number of xobjects needed for ScopeNotificationFilter ==
>
> Currently, one location is represented by 1 filter preference. But most
> filter preferences are very similar. They almost all say "for the
> ScopeNotificationFilter, for all event types, for all applications, the
> filter preference is enabled". The only different part is the actual
> location. But the "location" field is itself a LIST stored with the
> "relational storage" option. So we can take advantage of it and store
> similar preferences into 1 single object.
>
> 1 object with 700 locations instead of 700 objects with 1 location.
>
> However, it's a bit harder than this. Event if the
> NotificationFilterPreferences is generic and can contains many locations,
> the ScopeNotificationFilter expect it to concern only one location (and
> then it perform complex operations to sort the filters preferences
> according to a hierarchy). The UI in the user profile makes the same
> assumption so it does not handle multiple locations in the same preferences
> object. Refactoring this is not simple and cannot be done for 10.6.
>
> === Variation 1: store only 1 xobject, but make the API return 700
> preferences objects anyway ===
>
> This is the variation I am prototyping. Actually it's ok if the filters
> and the UI expect only 1 location into the preferences object. All we have
> to do is to "smash" the xobject into many NotificationFilterPreferences
> objects that we need internally. It would simply be the responsibility of
> the Store to detect similarities and to save the minimal amount of XObjects
> to store a bunch of preferences.
>
> But it means being very smart when loading, creating, updating and
> deleting a preference. Not having one xobject per filter preference
> introduces complexity, and complexity can lead to bugs. Again, according to
> the time frame, it's hard to implement.
>
> === Variation 2: use custom mapping ===
>
> Probably the easiest solution that would help making less SQL queries. The
> idea is to have a SQL table for notification filter preferences and bind
> the XObjects to that table. It would still use a lot of place in the
> document's cache but be more efficient on the database level.
>
> === Other Problem 1: it still creates page revisions ===
>
> As long as we store the filter preferences with xobjects, we create page
> revisions. We can get rid of those by using some internal API to not create
> a revision when we save an xobject but I wonder if it's what users want. If
> a user tries to rollback some changes and don't see all filter preferences
> it concerns, I think it's not very transparent.
>
> === Other Problem 2: Document's cache ===
>
> Sometime we load the a user document to get the avatar of the user, her
> name, etc... So we load user documents very frequently, even if the user is
> not connected! Having 700 filters in the document and cache them with the
> document even if we don't need them is a big waste of memory.
>
> == B: Implement a completely new store with Hibernate ==
>
> A bit like having a custom mapping. We could create a SQL table and
> implement an API to handle it. Then, no xobjects would be involved.
>
> Some drawbacks:
> * we need to write a custom cache as well.
> * the user cannot modify her preferences using the wiki principles
> (xobjects all the way).
>
> == C: Refactor the UI and the ScopeNotificationFilter so they do not
> assume 1 filter preference = 1 location ==
>
> This option is still possible. Probably the best because creating 1 filter
> preferences object per location is an obvious waste of memory. A
> refactoring of the UI is needed anyway, because we currently have no way to
> remove a bunch of filter preferences easily (users have to delete the 700
> filters preferences manually) so we can kill 2 birds with the same stone.
>
> But again, it requires some work.
>
> = Conclusion =
>
> That's it. All possible solutions require development effort that is
> hardly possible to make before 10.6 (and even 10.7, considering I would
> probably be the one implementing it and I'm not fulltime on the subject and
> I have holidays soon).
>
> Writing this email helped me to see the problem with perspective. I think
> solution C may be the best. But any opinion is good to hear (except if you
> propose something even more complex than I do :p).
>
> Thanks,
>
> Guillaume
>
>


--
Guillaume Delhumeau ([hidden email])
Research & Development Engineer at XWiki SAS
Committer on the XWiki.org project
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorm] Notifications Filters Preferences Store.

Guillaume Delhumeau
FYI, I'll continue with the solution A, which is very close to solution C,
and I'll try to use the less memory possible.

2018-07-18 11:27 GMT+02:00 Guillaume Delhumeau <
[hidden email]>:

> Corresponding JIRA issue: https://jira.xwiki.org/browse/XWIKI-15445
>
> 2018-07-18 11:07 GMT+02:00 Guillaume Delhumeau <
> [hidden email]>:
>
>> Hi.
>>
>> [TL;DR]
>>
>> This thread is about the way we store notification filter preferences for
>> each user. The constraint is there can be a lot of them (700 is a number a
>> user has recently reported). So how should we store them?
>>
>> [Full text]
>>
>> = Definition =
>>
>> So what is a filter preference? It's a generic object that can store many
>> elements, such as a page locations, application names, event types, etc...
>> They describe a configuration about a given filter for a given user. For
>> example, a filter preference can say "for the ScopeNotificationFilter and
>> the user A, include the location Main.WebHome" as it could be "for the
>> UserNotificationFilter and the user A, exclude the user SPAM". It's generic.
>>
>> The main usage is for page locations (ScopeNotificationFilter). By
>> default, we have the "autowatch" mode enabled. It means every time a user
>> modifies a page, a filter preference for this page and this user is
>> created. So if a user modifies 700 pages, he gets 700 filter preferences.
>>
>> = How are they stored =
>>
>> Currently, we have a simple implementation. There is a generic XClass
>> called "XWiki.Notifications.Code.NotificationFilterPreferenceClass". For
>> each preference, we add an XObject on the user page. It's that simple. But
>> it also means that if a users have 700 filter preferences, she also gets
>> 700 XObjects on her page, and 700 revisions of that page. Which is a pain:
>> it takes a lot of place in the document's cache, and it's heavy to load
>> (lot of SQL queries needed). So we have a big problem here.
>>
>> = Possible solutions =
>>
>> == A: Minimize the number of xobjects needed for ScopeNotificationFilter
>> ==
>>
>> Currently, one location is represented by 1 filter preference. But most
>> filter preferences are very similar. They almost all say "for the
>> ScopeNotificationFilter, for all event types, for all applications, the
>> filter preference is enabled". The only different part is the actual
>> location. But the "location" field is itself a LIST stored with the
>> "relational storage" option. So we can take advantage of it and store
>> similar preferences into 1 single object.
>>
>> 1 object with 700 locations instead of 700 objects with 1 location.
>>
>> However, it's a bit harder than this. Event if the
>> NotificationFilterPreferences is generic and can contains many locations,
>> the ScopeNotificationFilter expect it to concern only one location (and
>> then it perform complex operations to sort the filters preferences
>> according to a hierarchy). The UI in the user profile makes the same
>> assumption so it does not handle multiple locations in the same preferences
>> object. Refactoring this is not simple and cannot be done for 10.6.
>>
>> === Variation 1: store only 1 xobject, but make the API return 700
>> preferences objects anyway ===
>>
>> This is the variation I am prototyping. Actually it's ok if the filters
>> and the UI expect only 1 location into the preferences object. All we have
>> to do is to "smash" the xobject into many NotificationFilterPreferences
>> objects that we need internally. It would simply be the responsibility of
>> the Store to detect similarities and to save the minimal amount of XObjects
>> to store a bunch of preferences.
>>
>> But it means being very smart when loading, creating, updating and
>> deleting a preference. Not having one xobject per filter preference
>> introduces complexity, and complexity can lead to bugs. Again, according to
>> the time frame, it's hard to implement.
>>
>> === Variation 2: use custom mapping ===
>>
>> Probably the easiest solution that would help making less SQL queries.
>> The idea is to have a SQL table for notification filter preferences and
>> bind the XObjects to that table. It would still use a lot of place in the
>> document's cache but be more efficient on the database level.
>>
>> === Other Problem 1: it still creates page revisions ===
>>
>> As long as we store the filter preferences with xobjects, we create page
>> revisions. We can get rid of those by using some internal API to not create
>> a revision when we save an xobject but I wonder if it's what users want. If
>> a user tries to rollback some changes and don't see all filter preferences
>> it concerns, I think it's not very transparent.
>>
>> === Other Problem 2: Document's cache ===
>>
>> Sometime we load the a user document to get the avatar of the user, her
>> name, etc... So we load user documents very frequently, even if the user is
>> not connected! Having 700 filters in the document and cache them with the
>> document even if we don't need them is a big waste of memory.
>>
>> == B: Implement a completely new store with Hibernate ==
>>
>> A bit like having a custom mapping. We could create a SQL table and
>> implement an API to handle it. Then, no xobjects would be involved.
>>
>> Some drawbacks:
>> * we need to write a custom cache as well.
>> * the user cannot modify her preferences using the wiki principles
>> (xobjects all the way).
>>
>> == C: Refactor the UI and the ScopeNotificationFilter so they do not
>> assume 1 filter preference = 1 location ==
>>
>> This option is still possible. Probably the best because creating 1
>> filter preferences object per location is an obvious waste of memory. A
>> refactoring of the UI is needed anyway, because we currently have no way to
>> remove a bunch of filter preferences easily (users have to delete the 700
>> filters preferences manually) so we can kill 2 birds with the same stone.
>>
>> But again, it requires some work.
>>
>> = Conclusion =
>>
>> That's it. All possible solutions require development effort that is
>> hardly possible to make before 10.6 (and even 10.7, considering I would
>> probably be the one implementing it and I'm not fulltime on the subject and
>> I have holidays soon).
>>
>> Writing this email helped me to see the problem with perspective. I think
>> solution C may be the best. But any opinion is good to hear (except if you
>> propose something even more complex than I do :p).
>>
>> Thanks,
>>
>> Guillaume
>>
>>
>
>
> --
> Guillaume Delhumeau ([hidden email])
> Research & Development Engineer at XWiki SAS
> Committer on the XWiki.org project
>



--
Guillaume Delhumeau ([hidden email])
Research & Development Engineer at XWiki SAS
Committer on the XWiki.org project
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorm] Notifications Filters Preferences Store.

Guillaume Delhumeau
XWIKI-15455 </browse/XWIKI-15455> (that I have just created) invalidates
both Solution A and C since it requires to store a Date along with each
location, so we cannot store several locations in the same Notification
Filter Preferences object.

2018-07-19 11:03 GMT+02:00 Guillaume Delhumeau <
[hidden email]>:

> FYI, I'll continue with the solution A, which is very close to solution C,
> and I'll try to use the less memory possible.
>
> 2018-07-18 11:27 GMT+02:00 Guillaume Delhumeau <
> [hidden email]>:
>
>> Corresponding JIRA issue: https://jira.xwiki.org/browse/XWIKI-15445
>>
>> 2018-07-18 11:07 GMT+02:00 Guillaume Delhumeau <
>> [hidden email]>:
>>
>>> Hi.
>>>
>>> [TL;DR]
>>>
>>> This thread is about the way we store notification filter preferences
>>> for each user. The constraint is there can be a lot of them (700 is a
>>> number a user has recently reported). So how should we store them?
>>>
>>> [Full text]
>>>
>>> = Definition =
>>>
>>> So what is a filter preference? It's a generic object that can store
>>> many elements, such as a page locations, application names, event types,
>>> etc... They describe a configuration about a given filter for a given user.
>>> For example, a filter preference can say "for the ScopeNotificationFilter
>>> and the user A, include the location Main.WebHome" as it could be "for the
>>> UserNotificationFilter and the user A, exclude the user SPAM". It's generic.
>>>
>>> The main usage is for page locations (ScopeNotificationFilter). By
>>> default, we have the "autowatch" mode enabled. It means every time a user
>>> modifies a page, a filter preference for this page and this user is
>>> created. So if a user modifies 700 pages, he gets 700 filter preferences.
>>>
>>> = How are they stored =
>>>
>>> Currently, we have a simple implementation. There is a generic XClass
>>> called "XWiki.Notifications.Code.NotificationFilterPreferenceClass".
>>> For each preference, we add an XObject on the user page. It's that simple.
>>> But it also means that if a users have 700 filter preferences, she also
>>> gets 700 XObjects on her page, and 700 revisions of that page. Which is a
>>> pain: it takes a lot of place in the document's cache, and it's heavy to
>>> load (lot of SQL queries needed). So we have a big problem here.
>>>
>>> = Possible solutions =
>>>
>>> == A: Minimize the number of xobjects needed for ScopeNotificationFilter
>>> ==
>>>
>>> Currently, one location is represented by 1 filter preference. But most
>>> filter preferences are very similar. They almost all say "for the
>>> ScopeNotificationFilter, for all event types, for all applications, the
>>> filter preference is enabled". The only different part is the actual
>>> location. But the "location" field is itself a LIST stored with the
>>> "relational storage" option. So we can take advantage of it and store
>>> similar preferences into 1 single object.
>>>
>>> 1 object with 700 locations instead of 700 objects with 1 location.
>>>
>>> However, it's a bit harder than this. Event if the
>>> NotificationFilterPreferences is generic and can contains many locations,
>>> the ScopeNotificationFilter expect it to concern only one location (and
>>> then it perform complex operations to sort the filters preferences
>>> according to a hierarchy). The UI in the user profile makes the same
>>> assumption so it does not handle multiple locations in the same preferences
>>> object. Refactoring this is not simple and cannot be done for 10.6.
>>>
>>> === Variation 1: store only 1 xobject, but make the API return 700
>>> preferences objects anyway ===
>>>
>>> This is the variation I am prototyping. Actually it's ok if the filters
>>> and the UI expect only 1 location into the preferences object. All we have
>>> to do is to "smash" the xobject into many NotificationFilterPreferences
>>> objects that we need internally. It would simply be the responsibility of
>>> the Store to detect similarities and to save the minimal amount of XObjects
>>> to store a bunch of preferences.
>>>
>>> But it means being very smart when loading, creating, updating and
>>> deleting a preference. Not having one xobject per filter preference
>>> introduces complexity, and complexity can lead to bugs. Again, according to
>>> the time frame, it's hard to implement.
>>>
>>> === Variation 2: use custom mapping ===
>>>
>>> Probably the easiest solution that would help making less SQL queries.
>>> The idea is to have a SQL table for notification filter preferences and
>>> bind the XObjects to that table. It would still use a lot of place in the
>>> document's cache but be more efficient on the database level.
>>>
>>> === Other Problem 1: it still creates page revisions ===
>>>
>>> As long as we store the filter preferences with xobjects, we create page
>>> revisions. We can get rid of those by using some internal API to not create
>>> a revision when we save an xobject but I wonder if it's what users want. If
>>> a user tries to rollback some changes and don't see all filter preferences
>>> it concerns, I think it's not very transparent.
>>>
>>> === Other Problem 2: Document's cache ===
>>>
>>> Sometime we load the a user document to get the avatar of the user, her
>>> name, etc... So we load user documents very frequently, even if the user is
>>> not connected! Having 700 filters in the document and cache them with the
>>> document even if we don't need them is a big waste of memory.
>>>
>>> == B: Implement a completely new store with Hibernate ==
>>>
>>> A bit like having a custom mapping. We could create a SQL table and
>>> implement an API to handle it. Then, no xobjects would be involved.
>>>
>>> Some drawbacks:
>>> * we need to write a custom cache as well.
>>> * the user cannot modify her preferences using the wiki principles
>>> (xobjects all the way).
>>>
>>> == C: Refactor the UI and the ScopeNotificationFilter so they do not
>>> assume 1 filter preference = 1 location ==
>>>
>>> This option is still possible. Probably the best because creating 1
>>> filter preferences object per location is an obvious waste of memory. A
>>> refactoring of the UI is needed anyway, because we currently have no way to
>>> remove a bunch of filter preferences easily (users have to delete the 700
>>> filters preferences manually) so we can kill 2 birds with the same stone.
>>>
>>> But again, it requires some work.
>>>
>>> = Conclusion =
>>>
>>> That's it. All possible solutions require development effort that is
>>> hardly possible to make before 10.6 (and even 10.7, considering I would
>>> probably be the one implementing it and I'm not fulltime on the subject and
>>> I have holidays soon).
>>>
>>> Writing this email helped me to see the problem with perspective. I
>>> think solution C may be the best. But any opinion is good to hear (except
>>> if you propose something even more complex than I do :p).
>>>
>>> Thanks,
>>>
>>> Guillaume
>>>
>>>
>>
>>
>> --
>> Guillaume Delhumeau ([hidden email])
>> Research & Development Engineer at XWiki SAS
>> Committer on the XWiki.org project
>>
>
>
>
> --
> Guillaume Delhumeau ([hidden email])
> Research & Development Engineer at XWiki SAS
> Committer on the XWiki.org project
>



--
Guillaume Delhumeau ([hidden email])
Research & Development Engineer at XWiki SAS
Committer on the XWiki.org project
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorm] Notifications Filters Preferences Store.

Guillaume Delhumeau
After more thinking and discussions with Thomas, it appears solution B
(creating a table with Hibernate) is the only valuable option. These
preferences are not real data that would need to be stored on the user
page. They are "temporary" configurations for each user.

In addition, it would permit to optimize the HQL query by linking elements
of the 2 tables instead of introducing 700 "OR location = xxx". This
optimization is actually not a nice-to-have thing, but a real requirement
for scalability.

2018-07-20 21:05 GMT+03:00 Guillaume Delhumeau <
[hidden email]>:

> XWIKI-15455 <http:///browse/XWIKI-15455> (that I have just created)
> invalidates both Solution A and C since it requires to store a Date along
> with each location, so we cannot store several locations in the same
> Notification Filter Preferences object.
>
> 2018-07-19 11:03 GMT+02:00 Guillaume Delhumeau <
> [hidden email]>:
>
>> FYI, I'll continue with the solution A, which is very close to solution
>> C, and I'll try to use the less memory possible.
>>
>> 2018-07-18 11:27 GMT+02:00 Guillaume Delhumeau <
>> [hidden email]>:
>>
>>> Corresponding JIRA issue: https://jira.xwiki.org/browse/XWIKI-15445
>>>
>>> 2018-07-18 11:07 GMT+02:00 Guillaume Delhumeau <
>>> [hidden email]>:
>>>
>>>> Hi.
>>>>
>>>> [TL;DR]
>>>>
>>>> This thread is about the way we store notification filter preferences
>>>> for each user. The constraint is there can be a lot of them (700 is a
>>>> number a user has recently reported). So how should we store them?
>>>>
>>>> [Full text]
>>>>
>>>> = Definition =
>>>>
>>>> So what is a filter preference? It's a generic object that can store
>>>> many elements, such as a page locations, application names, event types,
>>>> etc... They describe a configuration about a given filter for a given user.
>>>> For example, a filter preference can say "for the ScopeNotificationFilter
>>>> and the user A, include the location Main.WebHome" as it could be "for the
>>>> UserNotificationFilter and the user A, exclude the user SPAM". It's generic.
>>>>
>>>> The main usage is for page locations (ScopeNotificationFilter). By
>>>> default, we have the "autowatch" mode enabled. It means every time a user
>>>> modifies a page, a filter preference for this page and this user is
>>>> created. So if a user modifies 700 pages, he gets 700 filter preferences.
>>>>
>>>> = How are they stored =
>>>>
>>>> Currently, we have a simple implementation. There is a generic XClass
>>>> called "XWiki.Notifications.Code.NotificationFilterPreferenceClass".
>>>> For each preference, we add an XObject on the user page. It's that simple.
>>>> But it also means that if a users have 700 filter preferences, she also
>>>> gets 700 XObjects on her page, and 700 revisions of that page. Which is a
>>>> pain: it takes a lot of place in the document's cache, and it's heavy to
>>>> load (lot of SQL queries needed). So we have a big problem here.
>>>>
>>>> = Possible solutions =
>>>>
>>>> == A: Minimize the number of xobjects needed for
>>>> ScopeNotificationFilter ==
>>>>
>>>> Currently, one location is represented by 1 filter preference. But most
>>>> filter preferences are very similar. They almost all say "for the
>>>> ScopeNotificationFilter, for all event types, for all applications, the
>>>> filter preference is enabled". The only different part is the actual
>>>> location. But the "location" field is itself a LIST stored with the
>>>> "relational storage" option. So we can take advantage of it and store
>>>> similar preferences into 1 single object.
>>>>
>>>> 1 object with 700 locations instead of 700 objects with 1 location.
>>>>
>>>> However, it's a bit harder than this. Event if the
>>>> NotificationFilterPreferences is generic and can contains many locations,
>>>> the ScopeNotificationFilter expect it to concern only one location (and
>>>> then it perform complex operations to sort the filters preferences
>>>> according to a hierarchy). The UI in the user profile makes the same
>>>> assumption so it does not handle multiple locations in the same preferences
>>>> object. Refactoring this is not simple and cannot be done for 10.6.
>>>>
>>>> === Variation 1: store only 1 xobject, but make the API return 700
>>>> preferences objects anyway ===
>>>>
>>>> This is the variation I am prototyping. Actually it's ok if the filters
>>>> and the UI expect only 1 location into the preferences object. All we have
>>>> to do is to "smash" the xobject into many NotificationFilterPreferences
>>>> objects that we need internally. It would simply be the responsibility of
>>>> the Store to detect similarities and to save the minimal amount of XObjects
>>>> to store a bunch of preferences.
>>>>
>>>> But it means being very smart when loading, creating, updating and
>>>> deleting a preference. Not having one xobject per filter preference
>>>> introduces complexity, and complexity can lead to bugs. Again, according to
>>>> the time frame, it's hard to implement.
>>>>
>>>> === Variation 2: use custom mapping ===
>>>>
>>>> Probably the easiest solution that would help making less SQL queries.
>>>> The idea is to have a SQL table for notification filter preferences and
>>>> bind the XObjects to that table. It would still use a lot of place in the
>>>> document's cache but be more efficient on the database level.
>>>>
>>>> === Other Problem 1: it still creates page revisions ===
>>>>
>>>> As long as we store the filter preferences with xobjects, we create
>>>> page revisions. We can get rid of those by using some internal API to not
>>>> create a revision when we save an xobject but I wonder if it's what users
>>>> want. If a user tries to rollback some changes and don't see all filter
>>>> preferences it concerns, I think it's not very transparent.
>>>>
>>>> === Other Problem 2: Document's cache ===
>>>>
>>>> Sometime we load the a user document to get the avatar of the user, her
>>>> name, etc... So we load user documents very frequently, even if the user is
>>>> not connected! Having 700 filters in the document and cache them with the
>>>> document even if we don't need them is a big waste of memory.
>>>>
>>>> == B: Implement a completely new store with Hibernate ==
>>>>
>>>> A bit like having a custom mapping. We could create a SQL table and
>>>> implement an API to handle it. Then, no xobjects would be involved.
>>>>
>>>> Some drawbacks:
>>>> * we need to write a custom cache as well.
>>>> * the user cannot modify her preferences using the wiki principles
>>>> (xobjects all the way).
>>>>
>>>> == C: Refactor the UI and the ScopeNotificationFilter so they do not
>>>> assume 1 filter preference = 1 location ==
>>>>
>>>> This option is still possible. Probably the best because creating 1
>>>> filter preferences object per location is an obvious waste of memory. A
>>>> refactoring of the UI is needed anyway, because we currently have no way to
>>>> remove a bunch of filter preferences easily (users have to delete the 700
>>>> filters preferences manually) so we can kill 2 birds with the same stone.
>>>>
>>>> But again, it requires some work.
>>>>
>>>> = Conclusion =
>>>>
>>>> That's it. All possible solutions require development effort that is
>>>> hardly possible to make before 10.6 (and even 10.7, considering I would
>>>> probably be the one implementing it and I'm not fulltime on the subject and
>>>> I have holidays soon).
>>>>
>>>> Writing this email helped me to see the problem with perspective. I
>>>> think solution C may be the best. But any opinion is good to hear (except
>>>> if you propose something even more complex than I do :p).
>>>>
>>>> Thanks,
>>>>
>>>> Guillaume
>>>>
>>>>
>>>
>>>
>>> --
>>> Guillaume Delhumeau ([hidden email])
>>> Research & Development Engineer at XWiki SAS
>>> Committer on the XWiki.org project
>>>
>>
>>
>>
>> --
>> Guillaume Delhumeau ([hidden email])
>> Research & Development Engineer at XWiki SAS
>> Committer on the XWiki.org project
>>
>
>
>
> --
> Guillaume Delhumeau ([hidden email])
> Research & Development Engineer at XWiki SAS
> Committer on the XWiki.org project
>



--
Guillaume Delhumeau ([hidden email])
Research & Development Engineer at XWiki SAS
Committer on the XWiki.org project
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorm] Notifications Filters Preferences Store.

Guillaume Delhumeau
2018-07-25 16:14 GMT+02:00 Guillaume Delhumeau <
[hidden email]>:

> After more thinking and discussions with Thomas, it appears solution B
> (creating a table with Hibernate) is the only valuable option. These
> preferences are not real data that would need to be stored on the user
> page. They are "temporary" configurations for each user.
>
> In addition, it would permit to optimize the HQL query by linking elements
> of the 2 tables instead of introducing 700 "OR location = xxx". This
> optimization is actually not a nice-to-have thing, but a real requirement
> for scalability.
>

As a drawback, it means users' configurations would not be exportable in a
XAR like the rest of the content...


> 2018-07-20 21:05 GMT+03:00 Guillaume Delhumeau <
> [hidden email]>:
>
>> XWIKI-15455 <http:///browse/XWIKI-15455> (that I have just created)
>> invalidates both Solution A and C since it requires to store a Date along
>> with each location, so we cannot store several locations in the same
>> Notification Filter Preferences object.
>>
>> 2018-07-19 11:03 GMT+02:00 Guillaume Delhumeau <
>> [hidden email]>:
>>
>>> FYI, I'll continue with the solution A, which is very close to solution
>>> C, and I'll try to use the less memory possible.
>>>
>>> 2018-07-18 11:27 GMT+02:00 Guillaume Delhumeau <
>>> [hidden email]>:
>>>
>>>> Corresponding JIRA issue: https://jira.xwiki.org/browse/XWIKI-15445
>>>>
>>>> 2018-07-18 11:07 GMT+02:00 Guillaume Delhumeau <
>>>> [hidden email]>:
>>>>
>>>>> Hi.
>>>>>
>>>>> [TL;DR]
>>>>>
>>>>> This thread is about the way we store notification filter preferences
>>>>> for each user. The constraint is there can be a lot of them (700 is a
>>>>> number a user has recently reported). So how should we store them?
>>>>>
>>>>> [Full text]
>>>>>
>>>>> = Definition =
>>>>>
>>>>> So what is a filter preference? It's a generic object that can store
>>>>> many elements, such as a page locations, application names, event types,
>>>>> etc... They describe a configuration about a given filter for a given user.
>>>>> For example, a filter preference can say "for the ScopeNotificationFilter
>>>>> and the user A, include the location Main.WebHome" as it could be "for the
>>>>> UserNotificationFilter and the user A, exclude the user SPAM". It's generic.
>>>>>
>>>>> The main usage is for page locations (ScopeNotificationFilter). By
>>>>> default, we have the "autowatch" mode enabled. It means every time a user
>>>>> modifies a page, a filter preference for this page and this user is
>>>>> created. So if a user modifies 700 pages, he gets 700 filter preferences.
>>>>>
>>>>> = How are they stored =
>>>>>
>>>>> Currently, we have a simple implementation. There is a generic XClass
>>>>> called "XWiki.Notifications.Code.NotificationFilterPreferenceClass".
>>>>> For each preference, we add an XObject on the user page. It's that simple.
>>>>> But it also means that if a users have 700 filter preferences, she also
>>>>> gets 700 XObjects on her page, and 700 revisions of that page. Which is a
>>>>> pain: it takes a lot of place in the document's cache, and it's heavy to
>>>>> load (lot of SQL queries needed). So we have a big problem here.
>>>>>
>>>>> = Possible solutions =
>>>>>
>>>>> == A: Minimize the number of xobjects needed for
>>>>> ScopeNotificationFilter ==
>>>>>
>>>>> Currently, one location is represented by 1 filter preference. But
>>>>> most filter preferences are very similar. They almost all say "for the
>>>>> ScopeNotificationFilter, for all event types, for all applications, the
>>>>> filter preference is enabled". The only different part is the actual
>>>>> location. But the "location" field is itself a LIST stored with the
>>>>> "relational storage" option. So we can take advantage of it and store
>>>>> similar preferences into 1 single object.
>>>>>
>>>>> 1 object with 700 locations instead of 700 objects with 1 location.
>>>>>
>>>>> However, it's a bit harder than this. Event if the
>>>>> NotificationFilterPreferences is generic and can contains many locations,
>>>>> the ScopeNotificationFilter expect it to concern only one location (and
>>>>> then it perform complex operations to sort the filters preferences
>>>>> according to a hierarchy). The UI in the user profile makes the same
>>>>> assumption so it does not handle multiple locations in the same preferences
>>>>> object. Refactoring this is not simple and cannot be done for 10.6.
>>>>>
>>>>> === Variation 1: store only 1 xobject, but make the API return 700
>>>>> preferences objects anyway ===
>>>>>
>>>>> This is the variation I am prototyping. Actually it's ok if the
>>>>> filters and the UI expect only 1 location into the preferences object. All
>>>>> we have to do is to "smash" the xobject into many
>>>>> NotificationFilterPreferences objects that we need internally. It would
>>>>> simply be the responsibility of the Store to detect similarities and to
>>>>> save the minimal amount of XObjects to store a bunch of preferences.
>>>>>
>>>>> But it means being very smart when loading, creating, updating and
>>>>> deleting a preference. Not having one xobject per filter preference
>>>>> introduces complexity, and complexity can lead to bugs. Again, according to
>>>>> the time frame, it's hard to implement.
>>>>>
>>>>> === Variation 2: use custom mapping ===
>>>>>
>>>>> Probably the easiest solution that would help making less SQL queries.
>>>>> The idea is to have a SQL table for notification filter preferences and
>>>>> bind the XObjects to that table. It would still use a lot of place in the
>>>>> document's cache but be more efficient on the database level.
>>>>>
>>>>> === Other Problem 1: it still creates page revisions ===
>>>>>
>>>>> As long as we store the filter preferences with xobjects, we create
>>>>> page revisions. We can get rid of those by using some internal API to not
>>>>> create a revision when we save an xobject but I wonder if it's what users
>>>>> want. If a user tries to rollback some changes and don't see all filter
>>>>> preferences it concerns, I think it's not very transparent.
>>>>>
>>>>> === Other Problem 2: Document's cache ===
>>>>>
>>>>> Sometime we load the a user document to get the avatar of the user,
>>>>> her name, etc... So we load user documents very frequently, even if the
>>>>> user is not connected! Having 700 filters in the document and cache them
>>>>> with the document even if we don't need them is a big waste of memory.
>>>>>
>>>>> == B: Implement a completely new store with Hibernate ==
>>>>>
>>>>> A bit like having a custom mapping. We could create a SQL table and
>>>>> implement an API to handle it. Then, no xobjects would be involved.
>>>>>
>>>>> Some drawbacks:
>>>>> * we need to write a custom cache as well.
>>>>> * the user cannot modify her preferences using the wiki principles
>>>>> (xobjects all the way).
>>>>>
>>>>> == C: Refactor the UI and the ScopeNotificationFilter so they do not
>>>>> assume 1 filter preference = 1 location ==
>>>>>
>>>>> This option is still possible. Probably the best because creating 1
>>>>> filter preferences object per location is an obvious waste of memory. A
>>>>> refactoring of the UI is needed anyway, because we currently have no way to
>>>>> remove a bunch of filter preferences easily (users have to delete the 700
>>>>> filters preferences manually) so we can kill 2 birds with the same stone.
>>>>>
>>>>> But again, it requires some work.
>>>>>
>>>>> = Conclusion =
>>>>>
>>>>> That's it. All possible solutions require development effort that is
>>>>> hardly possible to make before 10.6 (and even 10.7, considering I would
>>>>> probably be the one implementing it and I'm not fulltime on the subject and
>>>>> I have holidays soon).
>>>>>
>>>>> Writing this email helped me to see the problem with perspective. I
>>>>> think solution C may be the best. But any opinion is good to hear (except
>>>>> if you propose something even more complex than I do :p).
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Guillaume
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Guillaume Delhumeau ([hidden email])
>>>> Research & Development Engineer at XWiki SAS
>>>> Committer on the XWiki.org project
>>>>
>>>
>>>
>>>
>>> --
>>> Guillaume Delhumeau ([hidden email])
>>> Research & Development Engineer at XWiki SAS
>>> Committer on the XWiki.org project
>>>
>>
>>
>>
>> --
>> Guillaume Delhumeau ([hidden email])
>> Research & Development Engineer at XWiki SAS
>> Committer on the XWiki.org project
>>
>
>
>
> --
> Guillaume Delhumeau ([hidden email])
> Research & Development Engineer at XWiki SAS
> Committer on the XWiki.org project
>



--
Guillaume Delhumeau ([hidden email])
Research & Development Engineer at XWiki SAS
Committer on the XWiki.org project