camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: imaps component not retrieving specified number of messages
Date Wed, 06 Mar 2019 09:24:36 GMT
Hi

Yeah we can likely improve this to avoid fetching all the mails. You
are welcome to log a JIRA and work on a patch as github PR.
http://camel.apache.org/support.html
https://github.com/apache/camel/blob/master/CONTRIBUTING.md

On Wed, Mar 6, 2019 at 9:54 AM js504 <js504@protonmail.com.invalid> wrote:
>
> Hello,
>
> I am trying to poll a gmail account to process emails using the camel-mail version 2.23.0
imaps component and the Java DSL.
>
> There are thousands of emails I need to process but I am trying to split it up so as
to not overload anything.
>
> I have tried setting maxMessagesPerPoll and fetchSize parameters to different numbers
but no matter what it seems to try to peek every message first ( I think ) and doesn't seem
to retrieve just the number of messages I have specified.  I have tried setting peek to false
as well but this does not seem to help.  I have also tried setting just the maxMessagesPerPoll
without fetchSize and peek as well as setting just fetchSize without maxMessagesPerPoll or
peek set but the same occurs.
>
> below is my from statement using the imaps component:
>
>         from("imaps://{{env:IMAPS_HOSTNAME}}:{{env:IMAPS_PORT}}?" +
>                 "maxMessagesPerPoll=10" +
>                 "&fetchSize=10" +
>                 "&peek=false" +
>                 "&username={{env:EMAIL_ADDRESS}}" +
>                 "&password=" + password +
>                 "&debugMode={{env:IMAP_DEBUG_MODE}}" +
>                 "&consumer.initialDelay=5000")
>
> The following is the output from the connection to the gmail account and the following
'(BODY.PEEK[HEADER.FIELDS (Message-ID)])'  messages that seem to go on and on for every message
before retrieval of the message body has even begun.
> Any explanation as to what is happening and what I could do in order to process say 1000
emails at a time and stop it from doing this pre-processing on all emails instead of the first
1000 per poll would be really appreciated.
>
> 2019-03-05 21:10:01,531  INFO [main] (DefaultCamelContext.java:3197) - Apache Camel 2.23.1
(CamelContext: consume-emails) started in 0.969 seconds
> DEBUG: getProvider() returning javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle]
> DEBUG IMAPS: mail.imap.fetchsize: 16384
> DEBUG IMAPS: mail.imap.ignorebodystructuresize: false
> DEBUG IMAPS: mail.imap.statuscachetimeout: 1000
> DEBUG IMAPS: mail.imap.appendbuffersize: -1
> DEBUG IMAPS: mail.imap.minidletime: 10
> DEBUG IMAPS: closeFoldersOnStoreFailure
> DEBUG IMAPS: trying to connect to host "imap.gmail.com", port 993, isSSL true
> * OK Gimap ready for requests from 45.2.196.68 n3mb457716515jap
> A0 CAPABILITY
> * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY
SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH
> A0 OK Thats all she wrote! n3mb457716515jap
> DEBUG IMAPS: AUTH: XOAUTH2
> DEBUG IMAPS: AUTH: PLAIN
> DEBUG IMAPS: AUTH: PLAIN-CLIENTTOKEN
> DEBUG IMAPS: AUTH: OAUTHBEARER
> DEBUG IMAPS: AUTH: XOAUTH
> DEBUG IMAPS: protocolConnect login, host=imap.gmail.com, user=<removed by me>,
password=<non-null>
> DEBUG IMAPS: AUTHENTICATE PLAIN command trace suppressed
> DEBUG IMAPS: AUTHENTICATE PLAIN command result: A1 OK <removed by me> authenticated
(Success)
> A2 ENABLE UTF8=ACCEPT
> * ENABLED UTF8=ACCEPT
> A2 OK Success [THROTTLED]
> A3 LIST "" INBOX
> * LIST (\HasNoChildren) "/" "INBOX"
> A3 OK Success [THROTTLED]
> DEBUG IMAPS: connection available -- size: 1
> A4 SELECT INBOX
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing
\*)] Flags permitted.
> * OK [UIDVALIDITY 1] UIDs valid.
> * 78900 EXISTS
> * 0 RECENT
> * OK [UIDNEXT 78907] Predicted next UID.
> * OK [HIGHESTMODSEQ 3405665]
> A4 OK [READ-WRITE] INBOX selected. (Success) [THROTTLED]
> A5 SEARCH UNSEEN 1:78900
> * SEARCH 3 4 5 6 10 20 26 39 58 76 78 97 102 105 116 117 118 119 120 121 122 123 124
125 ... <truncated to shorten message> ... 78890 78891 78892 78893 78894 78895 78896
78897 78898 78899 78900
> A5 OK SEARCH completed (Success) [THROTTLED]
> A6 FETCH 3 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 3 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A6 OK Success [THROTTLED]
> A7 FETCH 4 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 4 FETCH (BODY[HEADER.FIELDS (Message-ID)] {65}
> Message-Id: <removed by me>
>
> )
> A7 OK Success [THROTTLED]
> A8 FETCH 5 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 5 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A8 OK Success [THROTTLED]
> A9 FETCH 6 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 6 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A9 OK Success [THROTTLED]
> A10 FETCH 10 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 10 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A10 OK Success [THROTTLED]
> A11 FETCH 20 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 20 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A11 OK Success [THROTTLED]
> A12 FETCH 26 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 26 FETCH (BODY[HEADER.FIELDS (Message-ID)] {65}
> Message-Id: <removed by me>
>
> )
> A12 OK Success [THROTTLED]
> A13 FETCH 39 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 39 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A13 OK Success [THROTTLED]
> A14 FETCH 58 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 58 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A14 OK Success [THROTTLED]
> A15 FETCH 76 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 76 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A15 OK Success [THROTTLED]
> A16 FETCH 78 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 78 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> and so on until for over an hour until I received ..
>
> 2019-03-06 00:07:58,263  WARN [Camel (consume-emails) thread #4 - imaps://imap.gmail.com:993]
(DefaultMailUidGenerator.java:59) - Cannot read headers from mail message. This exception
will be ignored.
> javax.mail.FolderClosedException: * BYE JavaMail Exception: java.io.IOException: Connection
dropped by server?
> at com.sun.mail.imap.IMAPMessage.getHeader(IMAPMessage.java:958) ~[javax.mail-1.6.1.jar:1.6.1]
> at org.apache.camel.component.mail.DefaultMailUidGenerator.generateMessageIdHeader(DefaultMailUidGenerator.java:52)
[camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.component.mail.DefaultMailUidGenerator.generateUuid(DefaultMailUidGenerator.java:36)
[camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.component.mail.MailConsumer.retrieveMessages(MailConsumer.java:273)
[camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:119) [camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
[camel-core-2.23.0.jar:2.23.0]
> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101) [camel-core-2.23.0.jar:2.23.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> 2019-03-06 00:07:58,275  WARN [Camel (consume-emails) thread #4 - imaps://imap.gmail.com:993]
(DefaultMailUidGenerator.java:83) - Cannot read headers from mail message. This exception
will be ignored.
> javax.mail.FolderClosedException: null
> at com.sun.mail.imap.IMAPMessage.getProtocol(IMAPMessage.java:169) ~[javax.mail-1.6.1.jar:1.6.1]
> at com.sun.mail.imap.IMAPMessage.loadHeaders(IMAPMessage.java:1581) ~[javax.mail-1.6.1.jar:1.6.1]
> at com.sun.mail.imap.IMAPMessage.getAllHeaders(IMAPMessage.java:1016) ~[javax.mail-1.6.1.jar:1.6.1]
> at org.apache.camel.component.mail.DefaultMailUidGenerator.generateMessageHash(DefaultMailUidGenerator.java:73)
[camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.component.mail.DefaultMailUidGenerator.generateUuid(DefaultMailUidGenerator.java:38)
[camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.component.mail.MailConsumer.retrieveMessages(MailConsumer.java:273)
[camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:119) [camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
[camel-core-2.23.0.jar:2.23.0]
> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101) [camel-core-2.23.0.jar:2.23.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Mime
View raw message