|
<dt><code><strong>tokenizer</strong>(<i>string</i> src, <i>array</i> tokens)</code></dt>
|
|
<dt><code><strong>tokenizer</strong>(<i>string</i> src, <i>array</i> tokens)</code></dt>
|
|
<dd>A function that reads string of Markdown text and returns a generated token. The token pattern should be found at the beginning of the <code>src</code> string. Accordingly, if using a Regular Expression to detect a token, it should be anchored to the string start (`^`). The <code>tokens</code> parameter contains the array of tokens that have been generated by the lexer up to that point, and can be used to access the previous token, for instance.
|
|
<dd>A function that reads string of Markdown text and returns a generated token. The token pattern should be found at the beginning of the <code>src</code> string. Accordingly, if using a Regular Expression to detect a token, it should be anchored to the string start (`^`). The <code>tokens</code> parameter contains the array of tokens that have been generated by the lexer up to that point, and can be used to access the previous token, for instance.
|