Previously, it would report a final empty token before stopping, which was usually not desired.
      Callers that do want to report a final empty token can do so by returning <code>[]byte{}</code> rather than <code>nil</code>.
    </p>
  </dd>

is ignored except as it separates tokens
that would otherwise combine into a single token. Also, a newline or end of file
may trigger the insertion of a <a href="#Semicolons">semicolon</a>.
While breaking the input into tokens,
the next token is the longest sequence of characters that form a
valid token.
</p>

<h3 id="Semicolons">Semicolons</h3>

<p>

is ignored except as it separates tokens
that would otherwise combine into a single token. Also, a newline or end of file
may trigger the insertion of a <a href="#Semicolons">semicolon</a>.
While breaking the input into tokens,
the next token is the longest sequence of characters that form a
valid token.
</p>

<h3 id="Semicolons">Semicolons</h3>

<p>