Hi=C2=A0

I'm struggling with the fo= llowing issue.
I need to build a cube with 6 dimensions for app u= sage
for example:
-------+-------+------+-----+------+-= -----
user =C2=A0| =C2=A0app | =C2=A0 d3 =C2=A0| d4 =C2=A0| d5 = =C2=A0 | =C2=A0d6
-------+-------+------+-----+------+------
<= /div>
=C2=A0 u1 =C2=A0 | =C2=A0a1 =C2=A0 | =C2=A0 x =C2=A0 =C2=A0| =C2= =A0 y =C2=A0 | =C2=A0 z =C2=A0 | =C2=A0 5
-------+-------+--= ----+-----+------+------
=C2=A0 u2 =C2=A0 | =C2=A0= a1 =C2=A0 | =C2=A0 a =C2=A0 =C2=A0| =C2=A0 b =C2=A0| =C2=A0 c =C2=A0 =C2=A0= | =C2=A0 6
-------+-------+------+-----+------+------

the dimensions combinations generate ~100M rows dai= ly.
for each row, I need to calculate the unique monthly active u= sers, weekly active users and daily active users, along with some other dat= a (that can be simply added up)

I can load the dat= a of the last 30 days, each day, and calculate a cube with countDistinct(&#= 39;userId)
but this requires a huge cluster, and is quite expensi= ve.

I tried to use Hyper Log Log, and store the by= te array of the HLL of the previous day, de-serialize it, add the users of = the current day, calc the new distinct, and serialize the byte array for th= e next day.
however, to get 5% error accuracy with HLL, the byte = array has to be 4K long, which makes the 100M rows, be ~ 4000 times bigger.= =C2=A0and i ended up requiring a lot more resources.

<= div>I wonder if one of you can think of a better solution.

Thanks
Tal

=C2=A0

--
Tal Grynbaum=C2=A0/=C2=A0CTO & co-founder
<= p style=3D"margin:0px 0px 10px;font-family:Helvetica,Arial,sans-serif;line-= height:14px">m# +972-54-7875797

=C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0mobile retention done right
--001a11419434a154f9053b263267-- --001a11419434a154fc053b263268 Content-Type: image/png; name="Logo-TH.png" Content-Disposition: inline; filename="Logo-TH.png" Content-Transfer-Encoding: base64 Content-ID: <877A9638-3D63-4272-AE87-CA64A2A4C68B> X-Attachment-Id: 74fb43fdec45dcf3_0.1.1 iVBORw0KGgoAAAANSUhEUgAAAKgAAAAZCAYAAACl3WVkAAAAAXNSR0IArs4c6QAAEuxJREFUeAHt Ww18VcWVn7n3fSSBBBAskgQSkCWQ1+JH/eiKvwps27W2dWv5qIvQX6SY8hXyEk3o0v5WdutPEQIv XwSXIkJtFZt1sW6t/rSrKGItIqy44UskAZIQAQ0GAsl7997Z/7nv3fvu3PcSXpe17a9lILln5pw5 M3PmzJkzZyacfcaprGzJk4yJHMa5ISgxsaUmVP/4Z9zsZfZ/IRLw0DiKiorShgzJuhPqYyBrGNCl E60nX2hsbAxf+jjFl6CcY4kP55wJwXddOs/LHP5aJGAqaEZGxmDG2TOkQJRU/MvKzRoK8BOz4BJ+ CcaVKFeLibkIrMzl72UJ9CsBU0EVJDdVptDJml564uw8mHQzwRRs7woWQeTSmV7m8NciAVNBNa/G vcwnjVnXM/9fFLR6Te0XJMaXM0klcPT+QOG5C+xYoKHpXFICR+EHJWOzfFpYzVt3rNNRnBRsmhHw ZeXqBdyrfM7QtS5MeHP26kOnkxKnUPjaZOYZe9244arhVQ++19Q+ZRvTUqj2fybhwfLFDzHBR8Gy zZG4CFGPfC9+SFFfDoXqfkv4kvKSL3kE+7rgnI48BuPiSM2a+p/PKJuRniNGzEX+ZmzpjaD/T6LH IWkmTkbDAYKPMDRNvFZfX3+AcO5UXFw8LCPDdyt4TwT7YfBXjwo9/Hxt7b99MGPGDF929vDbiYdQ wUuoxonWE6/25ScvXLjwKq9XuQZ7Q6HB+Tmh8d21tbXvUpvzy+bneJjnCi/xwhiECH9SU7PhI3d/ rDz65fUP9H9RNVg2ynp1nTXX1dXts/DJvuivir4J4BIW+qJFi4Z6PEZOTc26vVbd9srASWxa9+Ss 3v+KVdbXt7Wy8GEu+JicVU1390VzrGLs1R7hfxDz8W104ixcuI8444Owk41EficXxursqv1b+6rv Lm99oHAmnLV5XLDJgjNzFwTsB/8dhuBPbFjV9LPlScZq8WmrDLwEvRiG9k09AA/8Z3TGOYnTyUGm s5eyVzftQJ5kZicPOj0bp5c8u8QCOF9sgcIwugGbCuoR4ma4lf9MfmXUZ+X/VVo67xXOMl7nCitA KcOM77TrMlaO8pujec48HvZ9wG4F5aXlJcUQ+gq4wYOJN/ExXWKPvypYvmRluEcLKaryKxMVRbMh Q4ZcCVCyBlCmQekD0lZg/D9A/0xWpv8C01FWXvLWBRGZmc58y8B+ockGFFx4GgAvorwzlZSUZKG/ y+BH/wBkg1nMEfLgGyxbckRwsaJmTd0G1JGESjxyR17VXFa+ZCTBsZV8h9DEu6qHUVvTGFNoctII Twk0XKg8QZmjWPk3zrAKOt0nbdsDhfdCenVCEU8Izbghd82BQxaH9vvHDeNcnQ0jsAGLYrYe+XT2 yFDrBQvv/hI9U7z/jvbyDEOsCHP+vdErmzqIriU4YYTXx+5CW/9aXBFY9H0t8u2RoUNtbh6Uhzm7 BnJcx4S+hykqJsZQcBr3oe7n4PxdD9lubassPG4Io2zkqgOvWzxoi0/wPy2k42sLAwOLznoMiZkx uJL+FAqhnFZyCI8n8Ld5WdSY7MegAMVQmqQJxZX+NPVWN1IfIPvJZDX9fs8u8MkhBU9M/JY05nsH vvARCMaBdvQ3VhoMzs/HCnkJZAVOSqsSxjsGPNYHy0q+0tbaMQvWUrdw5ldgWcYqksgwG2ncw18B z2ui+ATLqkCSCYou8bQzdGZI7DOhWysCRehbSDf0b2Kit9lVYkBse68+VlnwS1V4XlS8g7YuZ613 LE9i/T4sHjNIKN43oV3vhU+ev330ppYeJ7/86v0nkG9oL87eJAYN+bni9bx1pHT0TWNqmhN3IxKC 0N/OrjrwspOHBZPbkpHurVK5+tu2ikAQu8NawilYuBGs3n4PLrSdW4xIHW0YACzVTWh6qrMMBHEl RANOHNZ+HAdEMFjyDQi02EkTg8NQpPZ4Ob8lDkchX48v3i8U+fzqBnQIyiknsmBWCdoagf5OsvL0 he2y8bFy6J7XteicNeIweM3MzR1eES+JQuApjRt0S2zlBAnG5mqT4xApLzg3TzsP3mDuqs/Y0bJx WDSsAcMtH1mVqJx2fQCjVh5s17TIXeDy5XkPBIJOnAWnDU5bD+U8v6N53yy3clo09M1e335+R0vT DIAf+/0Zm504B0y6Js29A8f+pu5wV87K/cUY1DSMoQa7wD8QXqkO1V4dZnqhk5jgzk+60vfs3uvt Ptfja2/teMSBlxQUKjoojhMnIZxXda4cjJfJFpS7rATU3VwpcXrRaTBjDtoeUL2mLqdLdA+Fl1gd x8chXY9PKBT9O1CCb8SxgOBqaEJMPt/dO/DC+fBw8CH3QnIJTHqhSIIrK1tM7sHfWrwwplOYyO9+ euZclhYR8OGMKgsX/SoVc+fOzXSWQciynDif4sQDltqExmGZ9D2BzrqgBe9EWo/X82O0uzNn1b4n nPR9wXmhQ0cwlodhj394vCw33UkHC3sD1sE0nenFMxvhIV4kEY1hsAUYxNeOV4y/zU2OBYkFKMvZ TUP5nJVNz+Oznim8vrkoP808xfMw1MQvk7e0tGjbtm2jHwlBlkHeHk20BmV4sLW149GErQ4Oq8TA sTWVli6YCEHnSXjB760J1dm+5uOhxykWW4bDViasIymYnTRNi0+ywqbbCBMQh06fPjP1ySef7I6V U7hrY0lZyf94OCdn3Bw74bBqbT6TJ0/GGZA/hIm2E+BZoVCt6YOj8Cx+KtCfq9Cf2SYRZ1dkZWUs ALzSzEd/ucYdwwj2LC5CnoYLKR+yhFA8KUwgccHJFLFleYun0zrKZuC88B2QYEippQs9kVUD0n1L FXXgV1DDPNhSTVUoRVgyW2Fpd6XGibGRVft+315Z+KxCdRl7XaoHoTJFS6lfXVxbmsU80z3DMqab QoS1SBAmFNOeNKmhRJ8ShyJWh1P7w4nKCfUzD2sSB5sv556xEgbWDYpgK6cTZxj6T515giORSJwX Y+OdePRpvUM5bVRdqG4nLOIbdgEAxTHZ1177+Zshj6EWHrRNVgTDKot+xXopz3nsIBjD0oS4kzDK Mb7piHo8W13dsF9GY4tPcQKhfsTbHjvxycoT10HU6efDEWlschuJOWytveAHP1P28aESU+HOJfUX E7nES+C4vAgDdlu8xIIwvhR3iPErD56F3F/AKrzdtCIIeWDACToqCcBuJsEiMi3cE/mJhU/ylRjT lXychg+Iw7Qji0TnOkagaazDpzqpGUtPT7d5YfCDyDGzExctNuwGOJNwTh8U7rZ70YzFIY5OrSQk dJ58R2yvbh+TsatdzUjjRrX3QqH6GheNnQVT0Hs2tFcEYKHhjES3cGqHnFmE1cwtHRlYTsHGwZ+V lUcYeO+gnDIVzuaaGgCexzGcqyRqwUYhAtcilaWQEVxvhv1NOAfAUClqijtEtBmBBazcaSpoMgva V19MeTk0ARN2dN26dZ190ZPgnXqDztuKjxk/5dK5cRTaQYyxy83P51MlC0X4cDhs88KkwhXgY6x6 mM7PA37WyktfwSbK9i3uGwmhDnQMD3rAyfkZbtWPuzfyqMAv36KhL3jICsrYL1Hs6K+TmiqQ6nHs EvoHEBlkZig61JEzVQHGzNPOjlbpRq6IlNjJQTVDVNwlTidF/zB4SvyQ9yAQcVHf081VEaqGwdPY SUCY/lgieeiOw7ZV3ueXX0D1dFNBIxi5VyaUOutEAUExDmc67My4YUhUmklYSZt3N+vemcUGUN5k CWXwqqqoQ/5e/Nh0UNorMdJHJUYg6O7utmkginfR0g0oNhNktAQB+Q2PhR6T4nKlpYu/i3ZsOiKG TbT5oHunXbp1Gn3+Hcioj7TYSMpRGIxQ2YRRdo54WYl0i7TJSjgbQvH6TYohIttHrT70Tr9UQMLK TsJ4TZlZtHpEaVe9YuiByoJM2iKt8tS+Sj52tredtOh6B+RixnGd5ReDYflz0bcToIsrJ1WCPMQf dn2eDVG3mwqqGjotVLttTAgPBosGV1dvOmMXxoE4Icqw8KWJiZPZkESPs5zdcToA4bCxBYo1y6KG kfgeAtwBqPFTGNUJ2IsA2lgAmissGuuLQL3NS1fYJlWIYlstQI+A/H+XlpWsQAd26tzIVJjyTYgt IaQF02XzQfB4F/YiZ/IjCnD3+vXr6ZD1ByRS3LiGCk2RFoqbEdZxyockmm3Ut/tMvA7vadpdcGNh 9wCDfxXZ/6CyVFLs9D5JN8TDTnow3wFLNBVlm53lF4OxYiej7nY3Hcpg+RMjD246K48BTgH8nKk8 vYr42ELQ15xkJbO+tHzxNIRcfliGYLSFj7pEVo6I49bHUeoEzTasAqcFpbJwWF8CWZ+08LHvF7FJ rIayPgVr9iNSTtSLuGhYZyciUrFUt6bubQyKrK8zDcOmWIUBvYEXWi9ACRZgbLL6EbUjNgv34kP0 Z4+DSWbGQD+FwqRx0LUsFtcvIJtvOWidoETPOS5I+0t08FFS3gITDhxTonfiT6uK8pNdxcy1Ifbd sOoZtAxy6zi868CbEpUwtsC1uLutsqBAKu8n014aGAWFuAdzBePiTjBNKY7veMWEO8Dn+rDQnjGF iG2wHesRe348YTLvgcXBFZfyCHxFmNtowgRLgkdnpJVs0Vlf042yMvhCUWylouK1a9d+bOgGgrzi uINMAtHEOayD+5yFKIu4owYIKy1DuRUOcpLHYSFoK2uMFwAyzDcHdpHOjB/bGQCQRRGuSd+ndwvB 4OJKxFw3Zwzw7wNiFga/NRhcMs9JT7BtyW1E3LWxiySAgrFxX1hCuTLQZUxNIr/eiPEI+jRqxOBA uatK0iw9UIESViI09aOYgtt0OVUHfo2day/ePNDVbEpJ+Nha8Hs/t2r/CwkVIESBe9eEclfB0QWj huAK7qcgbMivOthsKRv0SDztonVkJWFYdUw82u2/UTj4DkZ4DR3fTq3ympq1b3x6pjsATAiKSvf0 MZ6iB876M5xp16CDLgXmFB+VEoWVqkN1X8V97jwsmyMSkolPoUwrOzvP0tYhuQtwj6RDXu2atb9B X1bI9XkhWXOuKI+SGwIFvJLwpkXmrJwey0j0Lh8RLxr7XcjEKJUJNNuA0GFQEviNrj7QAm/lPvjF /9JaOWGO1B9X5nhZYKxHEc/BZm/Kqdq/xYWmrDB0/R5Y1xvaKwrpYgJg3wk3Pw+CYlLE0G13zUmN Ll/UhWkrL7zOM3AgGZgPL/T0LqX6tvKcFecrMLCXnEwJhhzaERnCthdNUGR3R/tVUBDbbcQ4JKXf uHHj2ZpQbXloTd2E06c6syJhI6f1eMdA3CbdHQqtO4L5g9l3JgpDJE/0JyV0QyaMriG6BqEJY6IW YcPx9G9pZmamD4O6TqppJB5g0Jd/wrKE+4G3rP0kWOwXhXF+SmOoUdqBIDdp3KBLOu4Y66hMUz7l 0q1McsNAygbNnYOj/1rcaT+Pl09/9xqeyFlDaC0fPw6PRB7CgQpuDN+avWrffAvn/tIjE6Hj9RJj /4jXSG+2PhCYulyeT95WUTCpraLwFczyXFwlfZmsnptPNA95JNniyR3BYrkJD0Vq8FZhJ2j3GlrX 31vhMrvjsRubrweDCycggD4OAvXi6HTs/T3v76YbJavRcI9Rr6bxRjy4osfHeANhJFgyi5a+usZn 42SeARC0COYZym4nnmBskYug9lcrLFJPyhgLsNuKQb4whFnqrAehvebMEzwZt0DXXv+FWhwWN8Mq /z52yHvLokM0wI/XRPVY6bYFxTgjFy6EadtPSNXVtXXz589/Jj3dMxftfw1KMRp2Lg0Kjngt/emK 2AKL3ZdL8SsoqQ9McQ9A9tFIduA021wOZujHdl0VFztwmvTYHU4gQtBlZpL8yl21r7G5IrDdy9lS WK5NBTcGRrTfKE5hp8giMw3Z/Qbf23AlmjAXbnY5a/btwUOOCRl+7zJM+C/uqwz4ixl2JwwO//PR 9V6MbWPnOeWuQMO+vvsPGSBstR3WGO3T4kJQjW7z0CfwOomflzUjcos7iuG2hu7+feZ5eueZkzui BQIfgcZo/8cVoDiMLb0ZnbsCCjERQpAtHhM9vT366IaGhg5nB6Hoc+Cd/cwsQ2AcEnyPeIE3BMfz IZg7Aec764BmM6xtkbPsLw2mJ3MRwzNcYdqZ37Ue6kjlbj2ZDJZjsd17f2C8l+l4j8ANjSlteaub aCfDtPWf6H0q1/xYqDqCvIYS1gX3KCpCverpkaEmcrGS8viTKygdMKBUCFCnnsjHTPKXoXh8vWQv BkQB+pQSWaJwr3a9W9FTqnyZ6I8iAclP+qO06GoEbtcH2Cqeo63WhUqSFR04BExLopwM23cmHP53 wKfvbcbJES+d8DD81svK6RTKnx/8J7eglkjotkhV2bfIF8U2nAeDPyqG60InDyD0s6OnO/JrBMz7 VeQ5c+YMGDYsazoCWjdB8fPAIx/8hoLfOewhH8ET2o0YwXPV1fWvWm1f/v75SuB/Af05mOGAguTa AAAAAElFTkSuQmCC --001a11419434a154fc053b263268--