From users-return-41722-apmail-kafka-users-archive=kafka.apache.org@kafka.apache.org Tue Dec 1 08:36:28 2020 Return-Path: X-Original-To: apmail-kafka-users-archive@www.apache.org Delivered-To: apmail-kafka-users-archive@www.apache.org Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by minotaur.apache.org (Postfix) with ESMTP id 738DC1931F for ; Tue, 1 Dec 2020 08:36:28 +0000 (UTC) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 53CC7665ED for ; Tue, 1 Dec 2020 08:36:27 +0000 (UTC) Received: (qmail 65047 invoked by uid 500); 1 Dec 2020 08:36:13 -0000 Delivered-To: apmail-kafka-users-archive@kafka.apache.org Received: (qmail 64951 invoked by uid 500); 1 Dec 2020 08:36:13 -0000 Mailing-List: contact users-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@kafka.apache.org Delivered-To: mailing list users@kafka.apache.org Received: (qmail 64914 invoked by uid 99); 1 Dec 2020 08:36:11 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2020 08:36:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 194CBBFD6E for ; Tue, 1 Dec 2020 08:36:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: 1.002 X-Spam-Level: * X-Spam-Status: No, score=1.002 tagged_above=-999 required=6.31 tests=[DATE_IN_PAST_12_24=0.804, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=adscale-co-nz.20150623.gappssmtp.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id sAnD_dSbkdiy for ; Tue, 1 Dec 2020 08:36:07 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.47; helo=mail-lf1-f47.google.com; envelope-from=liam.clarke@adscale.co.nz; receiver= Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id CE81ABCBC2 for ; Tue, 1 Dec 2020 08:36:06 +0000 (UTC) Received: by mail-lf1-f47.google.com with SMTP id t6so2424538lfl.13 for ; Tue, 01 Dec 2020 00:36:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=adscale-co-nz.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=YfuL0r60E+NTYrTxh0e5I+8uHfzJwb9uDNR7ElQnWGA=; b=ZFhOH1ZRzlZqym08S2tOI9c2oe7u0ucURT92fcuQXWlyZ7ZSDG0TJuXQ5rnqvuPBmp 0R7Au5sZo7vuMygMRzxPzpKqAP2xOhHzWBP1dQ6H1UJbRdOJM02rtdz/LeSLDD6bNI9k feV/G+iQqJdFlPwafvmfRdzAk+FHr635AbO4wNxUtb08pot1+pI6S5/BnCdBxlsnYK+A lKSueaf0niTDKhwzwY6fxTJipdSFWm28ELF0+zdS5vBP09XPWG6XrtSN1bLMnd0ITjzy XdD5D2D/1ABzwxZ3V1hcwov48CTTe5Ns3vWcJPAMLYffy8rF8UvcFGBLFuD17iJMBT2C tUJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=YfuL0r60E+NTYrTxh0e5I+8uHfzJwb9uDNR7ElQnWGA=; b=d2qoEZqpYGA+wZEfGPrrM/Zigp2PvNNJlw9OXq/IFDooFrgN56jB4CUFYNLZg7bHbl k/vcADVO2qlSTC9s1riYzvQVfiuQPToO00ED2f40KncG4T93COvtgpjVykzmpBTaesmq wSyFkv0Bso7kv/kscYB/issSeLei68ArVWE787sD+IrmztNdDxnxAOWjV9RgHZpfxYYF 2+YuUzgayOGVz1V4Am4DNZDFoMYq/qOJ7tbpN4T8S1zGuR8ieJiPk7zyffzieOJWTRty qb1LJ7QQC1fmYkRFLA+DVXUrtPQhWc+Ts4OMrZFyp76YSqB7t9PbXO1P8dmxn4fHB/M8 BETg== X-Gm-Message-State: AOAM531ScDoqs0LZlBeu+VtFn2CHoZBeDqPo39XPQ1TP5BeGrMcu8GKO 2gFU+mN3YJDw33PaxZ03KPoRs1nLQsQPhRXjDx1RCRUGry0Oug== X-Google-Smtp-Source: ABdhPJymBPYctJ3BJnesmOwT6LvTbVcNBV6PaB0ZSwhLf2bI3vsWXQvWR+T11Ojm/ynC5Z8PQH82BVVhf1ylizIzdR8= X-Received: by 2002:a19:f60e:: with SMTP id x14mr670395lfe.199.1606811764662; Tue, 01 Dec 2020 00:36:04 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Liam Clarke-Hutchinson Date: Tue, 1 Dec 2020 08:35:53 +1300 Message-ID: Subject: Re: Large messages max.message.bytes > message.max.bytes To: users@kafka.apache.org Content-Type: multipart/alternative; boundary="000000000000743c1c05b56304a8" --000000000000743c1c05b56304a8 Content-Type: text/plain; charset="UTF-8" Topic limits can't override broker limits. On Tue, Dec 1, 2020 at 6:42 PM Sakke Ferraris wrote: > Hi all! > > I have questions regarding configuration for large messages. I understand > that kafka has settings for broker and topic for message max sizes; > message.max.bytes (broker config) and max.message.bytes (topic config). > > I wonder if max.message.bytes for a topic can be larger that the server > default message.max.bytes? The problem I am facing is that now and then > there are messages that hit the 1MB wall even compressed. It is rare that > the messages get larger, but also there is no theoretical max size for the > messages. > > I am not allowed to tamper with the general setting since it could have > performance implications for all other users, but what could the > performance implications be if the max.message.bytes is considerably higher > on one topic? > > Best Regards, Sakke F > > > --000000000000743c1c05b56304a8--