nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Weeks <swe...@weeksconsulting.us>
Subject Re: New Distributed Map Cache Impementations
Date Mon, 18 Nov 2019 21:56:53 GMT
Created NIFI-6881 and I've got an almost working implementation. Something I'm trying to work
out is how to store the keys. Most databases don't support byte arrays for primary keys. I
know in virtually every case in NiFi the key's will be Strings but I'm wondering what to do
if they weren't. I could base64 everything but that will make it hard to figure out the column
sizing for keys. I'm trying to keep the SQL generic enough that it should work on Oracle,
MySQL, Postgres, DB2, Derby, H2 and maybe SQL Server if it doesn't do something odd.

Thanks

On 11/14/19, 8:17 AM, "Matt Burgess" <mattyb149@apache.org> wrote:

    Shawn,
    
    There are also Redis and Couchbase distributed map cache clients
    already in NiFi. I don't see any Jiras or PRs related to a DynamoDB or
    JDBC ones. I thought about making ones for JDBC, Hazelcast and/or
    Nitrite [1] (with or without a DataGate server), but never got around
    to it. I think DynamoDB and JDBC implementations would be helpful, the
    latter could support DynamoDB in the meantime using the Simba JDBC
    driver [2].
    
    While thinking about the JDBC one, I figured it might be nice to be
    able to cache the table locally for X amount of time or N number of
    entries, in case you pre-populate the cache and are just reading it
    with the client. Any write (from the NiFi client) would invalidate the
    cache and the table would be re-fetched on the next read operation. I
    did something similar for the DatabaseRecordLookupService, but that's
    a read-only service so I didn't have to worry about writes, I was just
    trying to improve performance where possible.
    
    Regards,
    Matt
    
    [1] https://www.dizitart.org/nitrite-database.html
    [2] https://www.simba.com/drivers/dynamodb-odbc-jdbc/
    
    On Thu, Nov 14, 2019 at 8:31 AM Shawn Weeks <sweeks@weeksconsulting.us> wrote:
    >
    > Has anyone already done some work on adding new services for distribute map cache?
I’m looking at moving to aws and I really don’t want to have to run emr just for hbase.
I’ve been thinking about starting on either a DynamoDB or simple jdbc implementation.
    >
    > Thanks
    > Shawn
    >
    > Sent from my iPhone
    

Mime
View raw message