camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From js504 <js...@protonmail.com.INVALID>
Subject imaps component not retrieving specified number of messages
Date Wed, 06 Mar 2019 01:59:49 GMT
Hello,

I am trying to poll a gmail account to process emails using the camel-mail version 2.23.0
imaps component and the Java DSL.

There are thousands of emails I need to process but I am trying to split it up so as to not
overload anything.

I have tried setting maxMessagesPerPoll and fetchSize parameters to different numbers but
no matter what it seems to try to peek every message first ( I think ) and doesn't seem to
retrieve just the number of messages I have specified.  I have tried setting peek to false
as well but this does not seem to help.  I have also tried setting just the maxMessagesPerPoll
without fetchSize and peek as well as setting just fetchSize without maxMessagesPerPoll or
peek set but the same occurs.

below is my from statement using the imaps component:

        from("imaps://{{env:IMAPS_HOSTNAME}}:{{env:IMAPS_PORT}}?" +
                "maxMessagesPerPoll=10" +
                "&fetchSize=10" +
                "&peek=false" +
                "&username={{env:EMAIL_ADDRESS}}" +
                "&password=" + password +
                "&debugMode={{env:IMAP_DEBUG_MODE}}" +
                "&consumer.initialDelay=5000")

The following is the output from the connection to the gmail account and the following '(BODY.PEEK[HEADER.FIELDS
(Message-ID)])'  messages that seem to go on and on for every message before retrieval of
the message body has even begun.
Any explanation as to what is happening and what I could do in order to process say 1000 emails
at a time and stop it from doing this pre-processing on all emails instead of the first 1000
per poll would be really appreciated.

2019-03-05 21:10:01,531  INFO [main] (DefaultCamelContext.java:3197) - Apache Camel 2.23.1
(CamelContext: consume-emails) started in 0.969 seconds
DEBUG: getProvider() returning javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle]
DEBUG IMAPS: mail.imap.fetchsize: 16384
DEBUG IMAPS: mail.imap.ignorebodystructuresize: false
DEBUG IMAPS: mail.imap.statuscachetimeout: 1000
DEBUG IMAPS: mail.imap.appendbuffersize: -1
DEBUG IMAPS: mail.imap.minidletime: 10
DEBUG IMAPS: closeFoldersOnStoreFailure
DEBUG IMAPS: trying to connect to host "imap.gmail.com", port 993, isSSL true
* OK Gimap ready for requests from 45.2.196.68 n3mb457716515jap
A0 CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR
AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH
A0 OK Thats all she wrote! n3mb457716515jap
DEBUG IMAPS: AUTH: XOAUTH2
DEBUG IMAPS: AUTH: PLAIN
DEBUG IMAPS: AUTH: PLAIN-CLIENTTOKEN
DEBUG IMAPS: AUTH: OAUTHBEARER
DEBUG IMAPS: AUTH: XOAUTH
DEBUG IMAPS: protocolConnect login, host=imap.gmail.com, user=<removed by me>, password=<non-null>
DEBUG IMAPS: AUTHENTICATE PLAIN command trace suppressed
DEBUG IMAPS: AUTHENTICATE PLAIN command result: A1 OK <removed by me> authenticated
(Success)
A2 ENABLE UTF8=ACCEPT
* ENABLED UTF8=ACCEPT
A2 OK Success [THROTTLED]
A3 LIST "" INBOX
* LIST (\HasNoChildren) "/" "INBOX"
A3 OK Success [THROTTLED]
DEBUG IMAPS: connection available -- size: 1
A4 SELECT INBOX
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing \*)]
Flags permitted.
* OK [UIDVALIDITY 1] UIDs valid.
* 78900 EXISTS
* 0 RECENT
* OK [UIDNEXT 78907] Predicted next UID.
* OK [HIGHESTMODSEQ 3405665]
A4 OK [READ-WRITE] INBOX selected. (Success) [THROTTLED]
A5 SEARCH UNSEEN 1:78900
* SEARCH 3 4 5 6 10 20 26 39 58 76 78 97 102 105 116 117 118 119 120 121 122 123 124 125 ...
<truncated to shorten message> ... 78890 78891 78892 78893 78894 78895 78896 78897 78898
78899 78900
A5 OK SEARCH completed (Success) [THROTTLED]
A6 FETCH 3 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 3 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>

)
A6 OK Success [THROTTLED]
A7 FETCH 4 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 4 FETCH (BODY[HEADER.FIELDS (Message-ID)] {65}
Message-Id: <removed by me>

)
A7 OK Success [THROTTLED]
A8 FETCH 5 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 5 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>

)
A8 OK Success [THROTTLED]
A9 FETCH 6 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 6 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>

)
A9 OK Success [THROTTLED]
A10 FETCH 10 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 10 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>

)
A10 OK Success [THROTTLED]
A11 FETCH 20 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 20 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>

)
A11 OK Success [THROTTLED]
A12 FETCH 26 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 26 FETCH (BODY[HEADER.FIELDS (Message-ID)] {65}
Message-Id: <removed by me>

)
A12 OK Success [THROTTLED]
A13 FETCH 39 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 39 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>

)
A13 OK Success [THROTTLED]
A14 FETCH 58 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 58 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>

)
A14 OK Success [THROTTLED]
A15 FETCH 76 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 76 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>

)
A15 OK Success [THROTTLED]
A16 FETCH 78 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 78 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>

and so on until for over an hour until I received ..

2019-03-06 00:07:58,263  WARN [Camel (consume-emails) thread #4 - imaps://imap.gmail.com:993]
(DefaultMailUidGenerator.java:59) - Cannot read headers from mail message. This exception
will be ignored.
javax.mail.FolderClosedException: * BYE JavaMail Exception: java.io.IOException: Connection
dropped by server?
at com.sun.mail.imap.IMAPMessage.getHeader(IMAPMessage.java:958) ~[javax.mail-1.6.1.jar:1.6.1]
at org.apache.camel.component.mail.DefaultMailUidGenerator.generateMessageIdHeader(DefaultMailUidGenerator.java:52)
[camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.component.mail.DefaultMailUidGenerator.generateUuid(DefaultMailUidGenerator.java:36)
[camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.component.mail.MailConsumer.retrieveMessages(MailConsumer.java:273) [camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:119) [camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174) [camel-core-2.23.0.jar:2.23.0]
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101) [camel-core-2.23.0.jar:2.23.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2019-03-06 00:07:58,275  WARN [Camel (consume-emails) thread #4 - imaps://imap.gmail.com:993]
(DefaultMailUidGenerator.java:83) - Cannot read headers from mail message. This exception
will be ignored.
javax.mail.FolderClosedException: null
at com.sun.mail.imap.IMAPMessage.getProtocol(IMAPMessage.java:169) ~[javax.mail-1.6.1.jar:1.6.1]
at com.sun.mail.imap.IMAPMessage.loadHeaders(IMAPMessage.java:1581) ~[javax.mail-1.6.1.jar:1.6.1]
at com.sun.mail.imap.IMAPMessage.getAllHeaders(IMAPMessage.java:1016) ~[javax.mail-1.6.1.jar:1.6.1]
at org.apache.camel.component.mail.DefaultMailUidGenerator.generateMessageHash(DefaultMailUidGenerator.java:73)
[camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.component.mail.DefaultMailUidGenerator.generateUuid(DefaultMailUidGenerator.java:38)
[camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.component.mail.MailConsumer.retrieveMessages(MailConsumer.java:273) [camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:119) [camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174) [camel-core-2.23.0.jar:2.23.0]
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101) [camel-core-2.23.0.jar:2.23.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message