<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title type="text">Nima Amini - Technical Lead &amp; Systems Architect</title>
  <subtitle type="text">Engineering Team Leader at MotorK in Milan. High-availability automotive retail integrations, AI-first engineering, and resilient data pipelines.</subtitle>
  <link href="https://nima.pro/feed.xml" rel="self" type="application/atom+xml"/>
  <link href="https://nima.pro"/>
  <id>https://nima.pro/</id>
  <updated>2026-09-04T00:00:00.000Z</updated>
  <author>
    <name>Nima Amini</name>
    <email>me@nima.pro</email>
  </author>


  <entry>
    <title type="text">SQS payload size, 64 KiB billing, and the 1 MiB cap</title>
    <id>https://nima.pro/posts/sqs-payload-size/</id>
    <link href="https://nima.pro/posts/sqs-payload-size/"/>
    <updated>2026-09-04T00:00:00.000Z</updated>
    <published>2026-09-04T00:00:00.000Z</published>
    <summary type="html">How Amazon SQS bills by 64 KiB chunk, what raising MaximumMessageSize to 1 MiB costs, how the Extended Client uses S3, and when a pointer to data you already stored is cheaper than all of that.</summary>

    <content type="html"><![CDATA[<p>Amazon SQS looks simple until <code>SendMessage</code> returns <code>Message must be shorter than 262144 bytes</code>. The worker is fine. The producer might think it dispatched. But the queue never saw the job.</p>
<p>I hit that with Symfony Messenger. The default transport serializer is native PHP serialize, then <code>addslashes()</code>, then base64 encoding. The object was fine. The <strong>body SQS received</strong> was non-existent.</p>
<p>There are a few ways around a 256 KiB bounce:</p>
<ol>
<li>JSON encode instead of PHP serialize.</li>
<li>Raise the queue cap to 1 MiB.</li>
<li>Park the payload in S3 (Extended Client).</li>
<li>If the row is already in the database, put a pointer on the queue and hydrate on consume.</li>
</ol>
<p>This post is about the size model, some practical know-how and the bill for each. In the end I chose to JSON encode.</p>
<h2>What SQS counts</h2>
<p>SQS counts the HTTP body you send, including attributes. It does not count your in-memory DTO.</p>
<p>When using Symfony Messenger, the default serializer (<code>messenger.transport.native_php_serializer</code>) inflates that body through three steps:</p>
<ol>
<li><code>serialize($envelope)</code>: class names, visibility markers, null bytes for private fields</li>
<li><code>addslashes()</code></li>
<li><code>base64_encode()</code>: required because <code>serialize()</code> emits null bytes and SQS wants plain text</li>
</ol>
<p>Base64 adds about 33% on its own on top of an already verbose binary dump.</p>
<h2>How SQS bills</h2>
<p>SQS bills <strong>per request</strong>, not per logical message. A request is a <strong>64 KiB chunk of payload</strong>. Ireland standard queues: <strong>$0.40 per million requests</strong> after the first million free. FIFO is $0.50. Same-region data transfer is free.</p>
<table>
<thead>
<tr>
<th>Body size</th>
<th>Units per Send (and again per Receive)</th>
</tr>
</thead>
<tbody>
<tr>
<td>1–64 KB</td>
<td>1</td>
</tr>
<tr>
<td>65–128 KB</td>
<td>2</td>
</tr>
<tr>
<td>129–192 KB</td>
<td>3</td>
</tr>
<tr>
<td>193–256 KB</td>
<td>4</td>
</tr>
<tr>
<td>…</td>
<td>+1 per 64 KB</td>
</tr>
<tr>
<td>1 MiB</td>
<td>16</td>
</tr>
</tbody>
</table>
<p>Delete is usually one unit. A successful trip is send + receive + delete. Empty receives on short polling are extra requests with no payload.</p>
<p><code>SendMessage</code> that fails the size cap never appears in CloudWatch <code>SentMessageSize</code>. That metric only records what landed. A new fattier data source looks like a sudden outage. The error should be logged and looked up in the producer logs. Logging the string length of the body before send can be a useful context in your logs.</p>
<h2>What the alternatives cost</h2>
<p>Figures below are Ireland <strong>standard</strong> SQS at $0.40 per million requests, <strong>per million successful trips</strong>, ignoring empty receives and the free tier. S3 is Standard in the same region: PUT about $0.005 per 1,000, GET about $0.0004 per 1,000, DELETE free. Storage for objects that live minutes is noise <strong>on S3 Standard</strong>, which has no minimum duration. Standard-IA and One Zone-IA still bill 30 days if you delete after consume. Glacier Instant Retrieval and Flexible Retrieval bill 90 days. Deep Archive bills 180 days. So just in case, keep Extended Client payloads on Standard if containing the costs is important and you have no particular reason to keep these objects hanging around (audit, replaying the events, etc...).</p>
<table>
<thead>
<tr>
<th>Approach</th>
<th>SQS body</th>
<th style="text-align:right">SQS requests / trip</th>
<th style="text-align:right">SQS / million trips</th>
<th>Extra</th>
<th style="text-align:right">Total / million</th>
</tr>
</thead>
<tbody>
<tr>
<td>JSON, under 64 KiB</td>
<td>tiny JSON</td>
<td style="text-align:right">3</td>
<td style="text-align:right"><strong>$1.20</strong></td>
<td>none</td>
<td style="text-align:right"><strong>~$1.20</strong></td>
</tr>
<tr>
<td>PHP envelope that almost fills 256 KiB</td>
<td>~251 KiB</td>
<td style="text-align:right">9</td>
<td style="text-align:right"><strong>$3.60</strong></td>
<td>none</td>
<td style="text-align:right"><strong>~$3.60</strong></td>
</tr>
<tr>
<td>Raise cap, body grows toward 1 MiB</td>
<td>1 MiB</td>
<td style="text-align:right">33</td>
<td style="text-align:right"><strong>$13.20</strong></td>
<td>none</td>
<td style="text-align:right"><strong>~$13.20</strong></td>
</tr>
<tr>
<td>Extended Client (S3)</td>
<td>stub, always 1 unit</td>
<td style="text-align:right">3</td>
<td style="text-align:right"><strong>$1.20</strong></td>
<td>1M PUT + 1M GET ≈ <strong>$5.40</strong></td>
<td style="text-align:right"><strong>~$6.60</strong></td>
</tr>
<tr>
<td>Pointer to a row already in MySQL</td>
<td>stub, always 1 unit</td>
<td style="text-align:right">3</td>
<td style="text-align:right"><strong>$1.20</strong></td>
<td>extra <code>SELECT</code> on every consume</td>
<td style="text-align:right"><strong>~$1.20</strong> SQS, more DB load</td>
</tr>
</tbody>
</table>
<p>Raising <code>MaximumMessageSize</code> does not change the bill by itself. A 251 KiB body is still 9 units after the cap moves. The extra spend appears when bodies <strong>use the new headroom</strong>: PHP serialize will, if you let it. A million trips that stay under 64 KiB cost <strong>$1.20</strong>. The same million at ~251 KiB cost <strong>$3.60</strong> (3×). A million that fill 1 MiB cost <strong>$13.20</strong> (11×). One CLI command stops the bounce. It also invites 16 send units and 16 receive units on purpose.</p>
<p>Extended Client is cheaper than a 1 MiB SQS body (<strong>~$6.60</strong> vs <strong>~$13.20</strong> per million) and more expensive than a small SQS body, because S3 PUT dominates. That is a different problem: a blob that belongs in object storage.</p>
<p>Passing a database pointer keeps your SQS costs in the same minimal tier as a small JSON payload. SQS stays cheap, but your database takes on more query traffic. Switch to JSON first. Move to pointers only if you need to shrink the body further and can accept the additional database reads.</p>
<h2>JSON encode, dual decode</h2>
<p>Switching to JSON eliminates class metadata, null bytes, and base64 encoding. The exact same payload fits in a fraction of the space.</p>
<p>However, you cannot switch the encoder to JSON in a single deployment if the queue already holds in-flight PHP-serialized messages. Workers must continue decoding old bodies while new messages arrive as JSON.</p>
<p>The solution is a transport-level <strong>dual-format decoder</strong>:</p>
<ul>
<li><strong>Detect format:</strong> Trim the payload string. JSON objects and arrays start with <code>{</code> or <code>[</code>. PHP-serialized strings start with type markers followed by a colon (<code>O:</code> for objects, <code>a:</code> for arrays, <code>C:</code> for custom classes).</li>
<li><strong>Decode:</strong> If the body starts with <code>{</code> or <code>[</code>, decode it as JSON. Otherwise, fall back to <code>PhpSerializer</code>.</li>
<li><strong>Encode:</strong> Write JSON for all outbound messages immediately.</li>
</ul>
<p>Once legacy PHP envelopes drain completely, drop the <code>PhpSerializer</code> fallback path entirely. Dual-format decoding is a temporary migration tactic, not a permanent architecture.</p>
<h2>Raise the cap to 1 MiB</h2>
<p>In August 2025, AWS raised the SQS <strong>service</strong> maximum from 256 KiB to <strong>1 MiB</strong> for both standard and FIFO queues. Lambda event-source mappings were updated alongside it. There is no account feature flag. The API accepts 1 MiB <strong>if the queue allows it</strong>.</p>
<p>New queues default to 1 MiB, but existing ones do not change automatically. If you previously configured <code>MaximumMessageSize</code>, it stays at that value. Any Infrastructure as Code (IaC) template hardcoded to 262144 will lock your queue at 256 KiB forever.</p>
<p><strong>Set <code>MaximumMessageSize</code></strong>:</p>
<ul>
<li>Check and update the attribute via <strong>CLI</strong>:</li>
</ul>
<pre><code class="language-bash">aws sqs get-queue-attributes \
  --queue-url &quot;$QUEUE_URL&quot; \
  --attribute-names MaximumMessageSize

aws sqs set-queue-attributes \
  --queue-url &quot;$QUEUE_URL&quot; \
  --attributes '{&quot;MaximumMessageSize&quot;:&quot;1048576&quot;}'
</code></pre>
<ul>
<li><strong>Console:</strong> SQS → Queue → Edit → set <strong>Maximum message size</strong> slider to 1024 KiB</li>
<li><strong>CloudFormation:</strong> <code>MaximumMessageSize: 1048576</code></li>
<li><strong>CDK:</strong> <code>maxMessageSize: 1048576</code> on <code>Queue</code> (upgrade <code>aws-cdk</code> if synthesis rejects values over 262144).</li>
</ul>
<p>Always update your IaC code to match. If your template still says <code>262144</code>, the next deployment will overwrite your change and drop the queue back to 256 KiB.</p>
<h2>Extended Client (payload in S3)</h2>
<p>The Extended Client is a client library pattern, not a queue configuration. AWS maintains official SDKs for Java, Python, and .NET, but there is no official PHP library. To use this in PHP, you rely on a community package or write custom wrapper logic around <code>aws/aws-sdk-php</code>.</p>
<p>SQS itself has no idea S3 is involved. Both the producer and consumer must use compatible library logic for it to work.</p>
<p><strong>How it works:</strong></p>
<ol>
<li><strong>Producer:</strong> If the payload exceeds a set threshold (256 KiB by default, or 64 KiB to stay within a single billing chunk), the library uploads the body to S3 via <code>PutObject</code>.</li>
<li><strong>SQS dispatch:</strong> SQS receives a small JSON pointer containing the bucket, key, and size. This stub is only a few hundred bytes, so it always costs exactly one SQS request unit.</li>
<li><strong>Consumer:</strong> The consumer reads the pointer from SQS, fetches the payload from S3 via <code>GetObject</code>, passes the raw bytes to your application logic, and issues a <code>DeleteObject</code> to clean up S3.</li>
</ol>
<p><strong>The catch:</strong></p>
<p>Every worker, Lambda function, and operational replay script in your pipeline must speak the protocol. A consumer without the library receives the raw JSON pointer and treats that stub as the actual message payload.</p>
<p>You also trade SQS size walls for new operational surface area: bucket IAM policies, added S3 GET latency on every read, orphaned objects if a worker crashes before issuing <code>DeleteObject</code>, and bucket lifecycle rules to keep S3 from filling up. Use it for files or batch records that genuinely belong in object storage.</p>
<h2>Pointer on the queue, hydrate on consume</h2>
<p>If the payload is <strong>already</strong> persisted in MySQL (such as an outbox table, buffer record, or primary document), you can shrink the body by queueing record IDs and loading the row during consumption.</p>
<p>This is a secondary optimization, not the primary fix. While SQS payload size stays at a single request unit, every message processed adds a <code>SELECT</code> query. You keep SQS cheap by shifting work to your database.</p>
<p><strong>Key considerations:</strong></p>
<ul>
<li><strong>Commit before dispatch:</strong> Always dispatch the message <em>after</em> the database transaction commits. Sending a pointer to an uncommitted record causes a race condition where the worker attempts to read the row before the insert completes.</li>
<li><strong>Handle missing rows:</strong> Your consumer must gracefully handle missing records caused by early deletions, cleanup jobs, or TTL expiration before the retry window closes.</li>
<li><strong>Support dual decoding during rollout:</strong> Your consumer must handle both full payloads and pointer stubs simultaneously while in-flight messages clear from the queue.</li>
<li><strong>Don't reinvent S3:</strong> If the payload is not already stored in your database, do not save it to MySQL just to queue a pointer. Building a custom claim-check pattern in a database is far worse for performance than using S3.</li>
</ul>
]]></content>

  </entry>


</feed>
