james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manuel Carrasco Moñino <man...@apache.org>
Subject Re: NoSQL backend for James
Date Mon, 31 Oct 2011 15:17:23 GMT
Hello Pepijn

On Fri, Oct 28, 2011 at 3:12 PM, Pepijn de Vos <pepijndevos@yahoo.com> wrote:
> Hi,
>
> I decided to stop working on the CouchDB backend. I'm sorry.

Bad news, it was a good idea James supporting couchdb

>
> There is something to be said for persisting and finishing, but I do not enjoy writing
it and I don't need it. I'm going to focus instead on what I set out to do, write a webmail
application. I will persist in that.
>
> I learned that what I wanted to do is not a good design at all. I thought it would be
easy to write my webmail application on top of CouchDB and have James "fish from the same
pool". This would create a tight coupling between my app and James, and leaves me at the mercy
of future changes to the API.
>
> Instead, I'll just write my app on top of IMAP, and in fact, I couldn't care less how
the mail server stores my email.

Yeah, this is the approach  used by HUPA (the webmail of James), take
a look to its architecture in this presentation:
http://www.slideshare.net/dodotis/apache-jameshupa-gwt

IMAP is a good election since you will have your app decoupled from
the email server implementation, but you will need a backend able to
communicate with the imap server and cache session credentials etc, so
you have to think in clustering this tier. Using couchdb rest
services, the clustering should be easier and proly it should perform
better.

>
> Thank you very, very much for your patience, help and support. I learned quite a few
things about Java and IMAP.
>
> If anyone does need a CouchDB backend, my code so far lives at https://github.com/pepijndevos/james-couchdb

Thank you, you did a good job with the implementation, I'm sure it can
be a good base of a couchdb mailbox implementation soon.

Cheers
- Manolo

>
> Pepijn
>
> On Oct 17, 2011, at 1:54 PM, Pepijn de Vos wrote:
>
>> Ah, it's just that I caught the wrong exception in my code. I'll fix that.
>>
>> On Oct 17, 2011, at 1:51 PM, Manuel Carrasco Moñino wrote:
>>
>>> I have added the Stress test and I see the exception, not sure how to fix yet.
>>> https://github.com/manolo/james-couchdb/commit/cf9a41116fca458c440a28ec937b83aeccd7967a
>>>
>>> - Manolo
>>>
>>> Exception in thread "pool-5-thread-61"
>>> org.ektorp.UpdateConflictException: document update conflict: id:
>>> 5e04b2a2c6fd920bc09af5a675000e7d rev:
>>> 3-ce227ff7c45a6905e010684f08011d05
>>>      at org.ektorp.impl.StdCouchDbConnector$6.error(StdCouchDbConnector.java:319)
>>>      at org.ektorp.impl.StdCouchDbConnector$6.error(StdCouchDbConnector.java:308)
>>>      at org.ektorp.http.RestTemplate.handleResponse(RestTemplate.java:98)
>>>      at org.ektorp.http.RestTemplate.put(RestTemplate.java:38)
>>>      at org.ektorp.impl.StdCouchDbConnector.update(StdCouchDbConnector.java:307)
>>>      at org.ektorp.support.CouchDbRepositorySupport.update(CouchDbRepositorySupport.java:155)
>>>      at org.apache.james.mailbox.couchdb.mail.CouchDbMailboxMapper.save(CouchDbMailboxMapper.java:90)
>>>      at org.apache.james.mailbox.couchdb.mail.CouchDbUidProvider.nextUid(CouchDbUidProvider.java:39)
>>>      at org.apache.james.mailbox.store.mail.AbstractMessageMapper.add(AbstractMessageMapper.java:125)
>>>      at org.apache.james.mailbox.store.StoreMessageManager$4.run(StoreMessageManager.java:532)
>>>      at org.apache.james.mailbox.store.StoreMessageManager$4.run(StoreMessageManager.java:1)
>>>      at org.apache.james.mailbox.store.transaction.TransactionalMapper.execute(TransactionalMapper.java:38)
>>>      at org.apache.james.mailbox.store.StoreMessageManager.appendMessageToStore(StoreMessageManager.java:529)
>>>      at org.apache.james.mailbox.store.StoreMessageManager$1.execute(StoreMessageManager.java:326)
>>>      at org.apache.james.mailbox.store.StoreMessageManager$1.execute(StoreMessageManager.java:1)
>>>      at org.apache.james.mailbox.store.AbstractMailboxPathLocker.executeWithLock(AbstractMailboxPathLocker.java:41)
>>>      at org.apache.james.mailbox.store.StoreMessageManager.appendMessage(StoreMessageManager.java:322)
>>>      at org.apache.james.mailbox.AbstractStressTest$2.run(AbstractStressTest.java:92)
>>>      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>      at java.lang.Thread.run(Thread.java:662)
>>>
>>> On Sun, Oct 16, 2011 at 6:24 PM, Pepijn de Vos <pepijndevos@yahoo.com>
wrote:
>>>> Thanks man, really appreciated.
>>>>
>>>> I too think my ModSeq impl looks good, but incrementing the modSeq 4 times
in parallel, I noticed that
>>>>
>>>> the value is incremented only 2-3 times.
>>>> _rev is incremented even less.
>>>> additional documents get created.
>>>>
>>>> What happens is that the same mailbox object gets passed to all invocations
of nextModSeq. This means that incrementing them in one increments it everywhere. Normally
this would only create conflicts in all but one, but because this mailbox does not have an
id yet, it creates a new mailbox for every conflict.
>>>>
>>>> So I assume this is not a problem in practice? Because after your fix, all
created mailboxes actually do have an id. I'll test that later, and then try to figure out
JUnit.
>>>>
>>>> Pepijn
>>>>
>>>> On Oct 16, 2011, at 4:23 PM, Manuel Carrasco Moñino wrote:
>>>>
>>>>> On Sat, Oct 15, 2011 at 5:40 PM, Pepijn de Vos <pepijndevos@yahoo.com>
wrote:
>>>>>> So, the result and questions for today:
>>>>>>
>>>>>> https://github.com/pepijndevos/james-couchdb
>>>>>>
>>>>>> What is serialVersionUID?
>>>>>
>>>>> It is a good practice to define it for serializable classes
>>>>> http://www.javapractices.com/topic/TopicAction.do?Id=45
>>>>>
>>>>>>
>>>>>> Is UidValidity something else than what is provided by CouchDbUidProvider?
>>>>>
>>>>> It's a number that changes when all uids become invalid. The server
>>>>> cannot update this number during a session.
>>>>> http://www.afterlogic.com/mailbee-net/docs/MailBee.ImapMail.Imap.UidValidity.html
>>>>>
>>>>>>
>>>>>> I had a look at the InMemory test, and it seems to fail on remainders
of the InMemory implementation(mentions of ConcurrentHashmap). I'm not sure what it tests,
but I think it's more than just CouchDbMailboxManager.
>>>>>>
>>>>>> Today I tried to implement ModSeq and Uid. They work fine until you
introduce concurrency. I'd be glad if someone would take a look at nextModSeq.
>>>>>
>>>>> Your implementation looks good.
>>>>>
>>>>>>
>>>>>> I'll try to write a test for it tomorrow. The thing is that the test
would need the database, right? How do you handle that?
>>>>>
>>>>> Difficult since it seems there is no way to install couchdb and start
>>>>> it from maven.
>>>>> Right now tests should be ignored when ddbb is not running (see
>>>>> maildir tests which are omitted in windows systems)
>>>>> I've added in the setup a call to empty the 'james' database when the
>>>>> test run, also I have added  initStandardDesignDocument() to
>>>>> initialize database.
>>>>> Be careful before run the test if you do not want the ddbb be re-created.
>>>>>
>>>>> Happy coding.
>>>>> - Manolo
>>>>>
>>>>>>
>>>>>> Pepijn
>>>>>>
>>>>>> On Oct 15, 2011, at 11:41 AM, Pepijn de Vos wrote:
>>>>>>
>>>>>>> Thanks! :)
>>>>>>>
>>>>>>> I have no idea what you did with the generics though. I read
an article about them, and thought I understood them.
>>>>>>>
>>>>>>> Basically they are just statically checked casts, right? So what
does it mean to write X implements Y<Z>?
>>>>>>>
>>>>>>> The tests are just leftovers from InMemory. I'm not doing TDD,
but I rather see tests as a frozen REPL. I write code, test it in the REPL, if I find myself
repeating pieces in the REPL, I make them into a test. I didn't get to the stage where I could
even compile anything.
>>>>>>>
>>>>>>> Pepijn
>>>>>>>
>>>>>>> On Oct 14, 2011, at 8:14 PM, Manuel Carrasco Moñino wrote:
>>>>>>>
>>>>>>>> You are right, you already sent the github link.
>>>>>>>>
>>>>>>>> I've checked out the code and I've made some changes to make
it
>>>>>>>> compile, also I have changed id signature in messages and
mailboxes to
>>>>>>>> String since couchdb uses string. I have hardcoded a user
and password
>>>>>>>> in the Utils class, it should go in the configuration files
though.
>>>>>>>>
>>>>>>>> I just have sent you a pull request
>>>>>>>> https://github.com/manolo/james-couchdb/commit/2b6b0ebce5c78198b14d1e7ec05c178d56482919
>>>>>>>>
>>>>>>>> The unique Test in the tree do no pass, I think you can start
fixing
>>>>>>>> the test so as it should be easier to follow the code taking
a look to
>>>>>>>> tests.
>>>>>>>>
>>>>>>>> - Manolo
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 13, 2011 at 9:54 PM, Pepijn de Vos <pepijndevos@yahoo.com>
wrote:
>>>>>>>>> I added the github link to the beautifully formatted
original email, but I think Google mistreated my message.
>>>>>>>>>
>>>>>>>>> Her it is again: https://github.com/pepijndevos/james-couchdb
>>>>>>>>>
>>>>>>>>> Pepijn
>>>>>>>>>
>>>>>>>>> On Oct 13, 2011, at 9:28 PM, Manuel Carrasco Moñino
wrote:
>>>>>>>>>
>>>>>>>>>> Hi Pepjin, could be possible to share your code anywhere,
so as I
>>>>>>>>>> could checkout it and take a look?
>>>>>>>>>>
>>>>>>>>>> Don't worry about if the code is ok or not, I think
github could be
>>>>>>>>>> ok, but you could send a compressed file via email
or whatever you
>>>>>>>>>> prefer.
>>>>>>>>>>
>>>>>>>>>> - Manolo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Oct 13, 2011 at 8:12 PM, Pepijn de Vos <pepijndevos@yahoo.com>
wrote:
>>>>>>>>>>> Okay, I put up the result for the day.
>>>>>>>>>>> I made a CouchDbMailbox with Ektorp @annotations.
>>>>>>>>>>> I made an attempt to make the MailboxMapper,
but I got stuck at the <type> casting sugar which I don't grok. My IDE keeps complaining
it can't resolve the incompatible types, while both are just Mailboxes.
>>>>>>>>>>> I stuffed the CouchDB connection in a class,
not happy with it.
>>>>>>>>>>> I'm not sure how to implement the findMailboxWithPathLike
and hasChildren methods.
>>>>>>>>>>> Any help appreciated, especially with the...
<> things. list() is the only one that's red wiggly lines, the others are just unchecked
casts. I've been adding random casts and <> left and right.
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/pepijndevos/james-couchdb
>>>>>>>>>>>
>>>>>>>>>>> Pepijn
>>>>>>>>>>>
>>>>>>>>>>> On Oct 13, 2011, at 6:23 PM, Pepijn de Vos wrote:
>>>>>>>>>>>
>>>>>>>>>>>> More questions. I started hacking!
>>>>>>>>>>>>
>>>>>>>>>>>> I'm going with Ektorp. I figured out most
of it, I think. Except that I don't understand the configuration. It does have a Spring module.
Any pointers on how to organize the config and connections?
>>>>>>>>>>>>
>>>>>>>>>>>> What does findMailboxWithPathLike do? The
implementations seem to do weird things with regexes. Preferably I make that into a nice CouchDB
view. CouchDB can't do fulltext search. As far as I can tell, the IMAP RFC doesn't say anything
about it.
>>>>>>>>>>>>
>>>>>>>>>>>> To have custom message and mailbox classes,
do I need to do anything else besides subclassing the corresponding *Manager class to return
one?
>>>>>>>>>>>>
>>>>>>>>>>>> I'm getting there!
>>>>>>>>>>>>
>>>>>>>>>>>> Pepijn
>>>>>>>>>>>>
>>>>>>>>>>>> On Oct 12, 2011, at 9:35 PM, Manuel Carrasco
Moñino wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Actually Ektorp is not a full implementation
of JPA, but it provides a
>>>>>>>>>>>>> JPA like API with support to many of
its annotations etc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyway, based on my experience, Ektorp
simplifies the access from java
>>>>>>>>>>>>> to couchdb and the bootstrap of couchdb,
so as theoretically when
>>>>>>>>>>>>> James starts the first time, the database,
views, design, mapreduce,
>>>>>>>>>>>>> etc should be created.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Manolo
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Oct 12, 2011 at 1:05 PM, Pepijn
de Vos <pepijndevos@yahoo.com> wrote:
>>>>>>>>>>>>>> Ektorp seems nice, but I'm more comfortable
just using something that resembles the HTTP API, since I'm not familiar with JPA. Haven't
decided yet.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pepijn
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Oct 11, 2011, at 5:08 PM, Manuel
Carrasco Moñino wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Pepijn
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Which java library are you considering
to use to connect with couchdb?
>>>>>>>>>>>>>>> I'm using [1] ektorp and makes
really easy to map domain models.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Manolo
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] http://www.ektorp.org/reference_documentation.html#d0e532
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 10, 2011 at 10:44
PM, Pepijn de Vos <pepijndevos@yahoo.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Oct 10, 2011, at 8:24
PM, Ioan Eugen Stan wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Have patience. You will
need it if you wish to complete something.
>>>>>>>>>>>>>>>>> Patience and perseverance
or else you'll be just another one who
>>>>>>>>>>>>>>>>> tried.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I don't expect to have it
finished by the end of the week, but if I'm still completely clueless by then, it's just not
worth the effort.
>>>>>>>>>>>>>>>> I don't have the ambition
to become a James commiter or even a Java dev, I just thought it would be nice to use CouchDB
for my application.
>>>>>>>>>>>>>>>> Somewhere is a point where
pragmatism beats learning. There isn't any technical reason why I can't use JPA.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can I just copy an
existing one and rename stuff? In other words, how are the modules glued into the whole? How
does the server know which class to load? It's not in the pom.xml, afaict.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Not sure what you mean
by that. It uses dependency injection provided
>>>>>>>>>>>>>>>>> by Spring framework (and
soon Guice) to inject object references into
>>>>>>>>>>>>>>>>> other objects at runtime.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ah, dependency injection.
*googles* So just the fact that I implement the interface is enough to @autowire it into James?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The sample config
is gone btw: http://james.apache.org/server/3/config-mailbox.html
>>>>>>>>>>>>>>>>>> Do I inherit tests
as well? I would imagine that a lot of tests are common to all mailbox implementations.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think this is because
the configuration changed and now it's spring
>>>>>>>>>>>>>>>>> based, and more modular.
I see you are very ambitious but I sense you
>>>>>>>>>>>>>>>>> have a lot of catching
up. James is complex so give it time, if you
>>>>>>>>>>>>>>>>> expect too much from
yourself and fail you will probably be too
>>>>>>>>>>>>>>>>> disappointed.
>>>>>>>>>>>>>>>> Yea, I read a Java book long
ago, never did any big projects with it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Make a public repo, commit
something and ask if you get stuck. I will
>>>>>>>>>>>>>>>>> try to help when/if I
can. I suggest you start with simple
>>>>>>>>>>>>>>>>> implementation that passes
some unit tests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So If I take any mailbox
impl, put it in a separate repo, will it work? All sorts of things refer to the parent pom.
I'll put something on github once I figure it out. I think it'll work out once I get to the
point where I can write some code.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> See for example the unit
tests I did for Mailbox interface in HBase
>>>>>>>>>>>>>>>>> implementation:
>>>>>>>>>>>>>>>>> http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/mail/model/HBaseMailboxTest.java
>>>>>>>>>>>>>>>> What I mean with inheriting
tests is that these all look very generic. They look like they could test any mailbox implementation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Pepijn
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Ioan Eugen Stan
>>>>>>>>>>>>>>>>> http://ieugen.blogspot.com/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>>>>>>> For additional commands,
e-mail: server-dev-help@james.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>>>>>> For additional commands,
e-mail: server-dev-help@james.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail:
server-dev-help@james.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail:
server-dev-help@james.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message