beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "RK (JIRA)" <>
Subject [jira] [Commented] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt
Date Sat, 27 Jan 2018 02:32:00 GMT


RK commented on BEAM-3106:

This can result in some difficult-to-pin-down errors in the google-cloud-platform python libraries.
For example, on a clean virtualenv:

pip install google-cloud-storage


from import Client
Works as expected, but after installing apache_beam[gcp]


pip install apache_beam[gcp]
# same code as above
from import Client
# File "/Users/karbr001/Documents/tmp_gcs_test/env/lib/python2.7/site-packages/",
line 198, in request
#    uri, method, body=body, headers=request_headers, **kwargs)
# TypeError: request() got an unexpected keyword argument 'data'
[~altay] what's the temporary relief you mentioned? Is it just installing beam with a custom file that points bigquery 0.28?





> Consider not pinning all python dependencies, or moving them to requirements.txt
> --------------------------------------------------------------------------------
>                 Key: BEAM-3106
>                 URL:
>             Project: Beam
>          Issue Type: Wish
>          Components: build-system
>    Affects Versions: 2.1.0
>         Environment: python
>            Reporter: Maximilian Roos
>            Priority: Major
> Currently all python dependencies are [pinned or capped|]
> While there's a good argument for supplying a `requirements.txt` with well tested dependencies,
having them specified in `` forces them to an exact state on each install of Beam.
This makes using Beam in any environment with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have libraries that
won't work with an older version (but Beam _does_ work with an newer version). We have to
do a bunch of gymnastics to get the correct versions installed because of this. Unfortunately,
airflow repeats this practice and conflicts on a number of dependencies, adding further complication
(but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for example
no libraries in numerical python do this. Here's a [discussion on SO|]

This message was sent by Atlassian JIRA

View raw message