nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple servers.
Date Fri, 01 Jun 2018 19:05:00 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498408#comment-16498408
] 

Hudson commented on NUTCH-1480:
-------------------------------

SUCCESS: Integrated in Jenkins build Nutch-trunk #3527 (See [https://builds.apache.org/job/Nutch-trunk/3527/])
Fixes for NUTCH-1480: Multiple index writer instances with different (roannel.fdez: [https://github.com/apache/nutch/commit/e4a7f871b1b03f901279e24cc1c626e5c1b67643])
* (edit) src/plugin/indexer-dummy/src/java/org/apache/nutch/indexwriter/dummy/DummyIndexWriter.java
* (edit) src/plugin/indexer-solr/src/java/org/apache/nutch/indexwriter/solr/SolrConstants.java
* (add) conf/index-writers.xsd
* (edit) src/java/org/apache/nutch/indexer/IndexWriter.java
* (edit) src/java/org/apache/nutch/indexer/NutchField.java
* (edit) src/java/org/apache/nutch/indexer/NutchDocument.java
* (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
* (edit) src/plugin/indexer-solr/src/java/org/apache/nutch/indexwriter/solr/SolrUtils.java
* (add) src/java/org/apache/nutch/indexer/IndexWriterConfig.java
* (delete) src/plugin/indexer-solr/src/java/org/apache/nutch/indexwriter/solr/SolrMappingReader.java
* (add) conf/index-writers.xml.template
* (edit) src/plugin/indexer-rabbit/src/java/org/apache/nutch/indexwriter/rabbit/RabbitMQConstants.java
* (edit) src/plugin/indexer-rabbit/src/java/org/apache/nutch/indexwriter/rabbit/RabbitIndexWriter.java
* (edit) src/plugin/indexer-solr/src/java/org/apache/nutch/indexwriter/solr/SolrIndexWriter.java
* (edit) src/plugin/indexer-cloudsearch/src/java/org/apache/nutch/indexwriter/cloudsearch/CloudSearchIndexWriter.java
* (edit) src/java/org/apache/nutch/indexer/IndexWriters.java
* (add) src/java/org/apache/nutch/indexer/MappingReader.java
* (edit) src/plugin/indexer-elastic-rest/src/java/org/apache/nutch/indexwriter/elasticrest/ElasticRestIndexWriter.java
Fixes for NUTCH-1480: Some improvements based on reviewers feedback. (roannel.fdez: [https://github.com/apache/nutch/commit/86cd375e267036596f19376e2499e1d1c4ccdcbb])
* (edit) src/java/org/apache/nutch/indexer/MappingReader.java
* (edit) src/plugin/indexer-solr/src/java/org/apache/nutch/indexwriter/solr/SolrIndexWriter.java
* (edit) src/java/org/apache/nutch/indexer/IndexWriters.java
* (edit) src/plugin/indexer-rabbit/src/java/org/apache/nutch/indexwriter/rabbit/RabbitIndexWriter.java
* (edit) conf/index-writers.xml.template
* (edit) src/java/org/apache/nutch/indexer/IndexWriterConfig.java
* (edit) conf/index-writers.xsd
Fixes for NUTCH-1480: Sections for all indexer-* plugins, relaxed (roannel.fdez: [https://github.com/apache/nutch/commit/84246a9e8fb183a28983a70d3d30d7d9a474ce58])
* (edit) src/java/org/apache/nutch/indexer/IndexWriters.java
* (add) src/plugin/indexer-dummy/src/java/org/apache/nutch/indexwriter/dummy/DummyConstants.java
* (edit) src/plugin/indexer-rabbit/src/java/org/apache/nutch/indexwriter/rabbit/RabbitIndexWriter.java
* (edit) src/plugin/indexer-cloudsearch/src/java/org/apache/nutch/indexwriter/cloudsearch/CloudSearchConstants.java
* (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
* (add) src/java/org/apache/nutch/indexer/IndexWriterParams.java
* (edit) src/plugin/indexer-elastic-rest/src/java/org/apache/nutch/indexwriter/elasticrest/ElasticRestConstants.java
* (edit) conf/index-writers.xml.template
* (edit) src/plugin/indexer-dummy/src/java/org/apache/nutch/indexwriter/dummy/DummyIndexWriter.java
* (edit) src/java/org/apache/nutch/indexer/IndexWriter.java
* (edit) src/plugin/indexer-cloudsearch/src/java/org/apache/nutch/indexwriter/cloudsearch/CloudSearchIndexWriter.java
* (edit) src/java/org/apache/nutch/indexer/IndexWriterConfig.java
* (edit) src/plugin/indexer-solr/src/java/org/apache/nutch/indexwriter/solr/SolrIndexWriter.java
* (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticConstants.java
* (edit) src/plugin/indexer-elastic-rest/src/java/org/apache/nutch/indexwriter/elasticrest/ElasticRestIndexWriter.java
Fixes for NUTCH-1480: Changes: - Logs for IndexerOutputFormat class to (roannel.fdez: [https://github.com/apache/nutch/commit/7e9d1df08817c54d50eed3945136033a7fd7af00])
* (edit) conf/log4j.properties
* (edit) src/java/org/apache/nutch/indexer/IndexerOutputFormat.java
* (edit) src/java/org/apache/nutch/indexer/IndexingJob.java
* (edit) src/java/org/apache/nutch/indexer/IndexerMapReduce.java
* (edit) src/java/org/apache/nutch/util/ObjectCache.java
Fixes for NUTCH-1480: Support for NUTCH-2484 and NUTCH-2380. (roannel.fdez: [https://github.com/apache/nutch/commit/d45510c186b3dbee3c3f7882c90ab3d28409a0b8])
* (edit) src/plugin/indexer-elastic-rest/src/java/org/apache/nutch/indexwriter/elasticrest/ElasticRestIndexWriter.java
* (edit) conf/index-writers.xml.template
* (edit) src/plugin/indexer-elastic-rest/src/java/org/apache/nutch/indexwriter/elasticrest/ElasticRestConstants.java
* (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
Fixes for NUTCH-1480: Merge branch 'master' into NUTCH-1480 (roannel.fdez: [https://github.com/apache/nutch/commit/b4e539389437e3b5660d596ebec4c87b1bc6e948])
* (edit) src/plugin/indexer-rabbit/src/java/org/apache/nutch/indexwriter/rabbit/RabbitIndexWriter.java
* (edit) src/java/org/apache/nutch/indexer/IndexingJob.java
* (edit) src/plugin/indexer-solr/src/java/org/apache/nutch/indexwriter/solr/SolrUtils.java
* (edit) src/plugin/indexer-cloudsearch/src/java/org/apache/nutch/indexwriter/cloudsearch/CloudSearchIndexWriter.java
* (edit) src/plugin/indexer-solr/src/java/org/apache/nutch/indexwriter/solr/SolrIndexWriter.java
* (edit) src/java/org/apache/nutch/indexer/IndexWriter.java
* (edit) src/plugin/indexer-elastic-rest/src/java/org/apache/nutch/indexwriter/elasticrest/ElasticRestIndexWriter.java
* (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
* (edit) src/java/org/apache/nutch/indexer/IndexWriters.java
Fixes for NUTCH-1480: Changes: - Internal cache. - Fixed the unit test (roannel.fdez: [https://github.com/apache/nutch/commit/041927ae943da3214ba5dd9d2b00ee52275c9b8b])
* (edit) src/java/org/apache/nutch/indexer/IndexingFiltersChecker.java
* (edit) src/java/org/apache/nutch/indexer/IndexWriterParams.java
* (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
* (edit) src/plugin/indexer-elastic/src/test/org/apache/nutch/indexwriter/elastic/TestElasticIndexWriter.java
* (edit) src/java/org/apache/nutch/indexer/IndexerOutputFormat.java
* (edit) src/java/org/apache/nutch/indexer/CleaningJob.java
* (edit) src/java/org/apache/nutch/indexer/IndexWriters.java
* (edit) src/java/org/apache/nutch/indexer/IndexingJob.java
* (edit) src/java/org/apache/nutch/util/ObjectCache.java


> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
>                 Key: NUTCH-1480
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1480
>             Project: Nutch
>          Issue Type: Improvement
>          Components: indexer
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>            Priority: Minor
>             Fix For: 1.15
>
>         Attachments: NUTCH-1480-1.6.1.patch, adding-support-for-sharding-indexer-for-solr.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited
list of URL's using Configuration.getString(). SolrWriter should be able to handle this list
of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is
available or if you want to send documents to multiple NOCs.
> edit:
> This does not replace NUTCH-1377 but complements it. With NUTCH-1377 this issue allows
you to index to multiple SolrCloud clusters at the same time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message