samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milinda Pathirage" <mili...@apache.org>
Subject Re: Review Request 33142: [SAMZA-561] Review in progress
Date Thu, 16 Apr 2015 18:59:42 GMT


> On April 14, 2015, 10:14 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-sql/src/main/java/org/apache/samza/sql/metadata/RelDataTypeToAvroSchemaConverter.java,
line 28
> > <https://reviews.apache.org/r/33142/diff/1/?file=926292#file926292line28>
> >
> >     One question here: it seems that this converter should be an abstract class
that can be extended to convert RelDataType to Avro or JSON based on implementation classes?

I agree, we should generalize this to handle any type. I am going to remove this from the
new patch because I have not yet implemented operators which require these converters.


> On April 14, 2015, 10:14 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java, line 37
> > <https://reviews.apache.org/r/33142/diff/1/?file=926293#file926293line37>
> >
> >     What's the meaning of this parent?

parent identifies the system. For example 'kafka'. And this is similar to db schema.


> On April 14, 2015, 10:14 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java, line 76
> > <https://reviews.apache.org/r/33142/diff/1/?file=926293#file926293line76>
> >
> >     I am not sure whether I fully understand this interface method: from Calcite
code base, it seems that this should return an iterator for the stream? This seems to be a
polling method to get the rows from the stream. However, Samza tasks are not polling the streams
to get the messages. Instead, Samza SystemConsumers delivers (i.e. pushes) the messages from
the physical streams to Samza tasks, and hence, the physical operators. How is this stream()
API used in Calcite? Can we leave it un-implemented?

AFAIK, this method is only useful in Calcite. We don't have to implement this. May be I should
change this to throw UnsupportedOperationException.


- Milinda


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33142/#review79941
-----------------------------------------------------------


On April 13, 2015, 9:04 p.m., Yi Pan (Data Infrastructure) wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33142/
> -----------------------------------------------------------
> 
> (Updated April 13, 2015, 9:04 p.m.)
> 
> 
> Review request for samza and Milinda Pathirage.
> 
> 
> Bugs: SAMZA-561
>     https://issues.apache.org/jira/browse/SAMZA-561
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> [SAMZA-561] Review in progress
> 
> Post Milinda's patch for SAMZA-561 to ease the comment and discussion.
> 
> 
> Diffs
> -----
> 
>   build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e 
>   gradle/dependency-versions.gradle ee6dfc411b7ab90b187df79f109884127953862e 
>   samza-sql/src/main/java/org/apache/samza/sql/Utils.java PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/data/IntermediateMessageTuple.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/data/serializers/SqlAvroSerdeFactory.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/expressions/Expression.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/expressions/RexToJavaCompiler.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/expressions/RexToJavaUtils.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/metadata/AvroSchemaConverter.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/metadata/RelDataTypeToAvroSchemaConverter.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/factory/TypeAwareOperatorSpec.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/insert/InsertToStreamOp.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/insert/InsertToStreamSpec.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/project/ProjectOp.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/operators/project/ProjectSpec.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/operators/scan/ProjectableFilterableStreamScanOp.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/scan/ProjectableFilterableStreamScanSpec.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/scan/StreamScanSpec.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/planner/ExecutionPlanner.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/planner/QueryPlanner.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/planner/rules/FilterableStreamScanRule.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/planner/rules/ProjectableStreamScanRule.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/planner/rules/RemoveIdentityProjectRule.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/rel/ProjectableFilterableStreamScan.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/rel/StreamScan.java PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/sql/data/serializers/SqlAvroSerdeTest.java
PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/sql/planner/QueryPlannerTest.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/planner/SamzaStreamTableFactory.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/planner/TestExecutionPlanner.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/planner/TestQueryPlanner.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/planner/TestRexToJavaCompiler.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/test/Constants.java PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/sql/test/Utils.java PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/sql/test/metadata/TestAvroSchemaConverter.java
PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION 
>   samza-sql/src/test/resources/orders.avsc PRE-CREATION 
>   samza-sql/src/test/resources/orders.json PRE-CREATION 
>   samza-test/src/main/config/sql-filter.properties PRE-CREATION 
>   samza-test/src/main/java/org/apache/samza/test/integration/sql/OrdersStreamFactory.java
PRE-CREATION 
>   samza-test/src/main/java/org/apache/samza/test/integration/sql/SqlTask.java PRE-CREATION

>   samza-test/src/main/python/integration_tests.py df64e239a2e467c8e4429dbeb7039f1aa9965ecc

>   samza-test/src/main/python/requirements.txt 2ae95908248516b5b26e671f24fa680f7b801675

>   samza-test/src/main/python/samza_job_yarn_deployer.py 38635ca5899c43fb61d6b4042e8543f0508fd41b

>   samza-test/src/main/python/tests/sql_tests.py PRE-CREATION 
>   samza-test/src/main/resources/orders.avsc PRE-CREATION 
>   samza-test/src/main/resources/orders.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33142/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Yi Pan (Data Infrastructure)
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message