beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [beam] chadrik commented on a change in pull request #11038: [BEAM-7746] More typing fixes
Date Mon, 23 Mar 2020 19:07:02 GMT
chadrik commented on a change in pull request #11038: [BEAM-7746] More typing fixes
URL: https://github.com/apache/beam/pull/11038#discussion_r396690657
 
 

 ##########
 File path: sdks/python/apache_beam/transforms/core.py
 ##########
 @@ -1300,12 +1300,13 @@ def to_runner_api_parameter(self, context):
           common_urns.requirements.REQUIRES_STATEFUL_PROCESSING.urn)
     from apache_beam.runners.common import DoFnSignature
     sig = DoFnSignature(self.fn)
-    is_splittable = sig.is_splittable_dofn()
 
 Review comment:
   > Not sure if checking get_restriction_coder() return type instead of is_splittable_dofn()
is future proof.
   
   `get_restriction_coder()` calls `is_splittable_dofn()` and returns `None` if it's not splittable.
 So I interpreted a `None` result from this method to mean "is not splittable". 
   
   ```python
     def get_restriction_coder(self):
       # type: () -> Optional[TupleCoder]
   
       """Get coder for a restriction when processing an SDF. """
       if self.is_splittable_dofn():
         return TupleCoder([
             (self.get_restriction_provider().restriction_coder()),
             (self.get_watermark_estimator_provider().estimator_state_coder())
         ])
       else:
         return None
   ```
   
   > I don't understand the change, from a mypy correctness perspective.
   
   Here's the problem:
   
   ```python
       if is_splittable:
         restriction_coder = sig.get_restriction_coder()  #  returns Optional[TupleCoder]
         restriction_coder_id = context.coders.get_id(restriction_coder)  # does not accept
Optional!
       else:
         restriction_coder_id = None
   ```
   
   With my changes, we naturally drop the optionality before passing the value to `context.coders.get_id()`.
 We also avoid a redundant call to `is_splittable_dofn()`, FWIW.
   
   I see two options:
   
   1) keep my changes and update the documentation of `get_restriction_coder()` to clarify
that `None` result indicates "is not splittable"
   2) revert my changes and add `assert restriction_coder is None` before the call to `context.coders.get_id()`
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message