Skip to main content

Regex.tokenize

tokenizeinput

Group: Conversions

Documentation​

Takes an input string and returns all the matches as a Vector Text. If the pattern contains marked groups, the values are concatenated together; otherwise the whole match is returned. Non-participating groups are omitted.

Arguments​

  • input: The text to tokenize.

Examples​

Split to blocks of 3 characters.​

      Regex.compile '...' . tokenize 'ABCDEF' == ['ABC','DEF']

Split to blocks of 3 characters taking first and third letters.​

      Regex.compile '(.).(.)' . tokenize 'ABCDEF' == ['AC','DF']

Split a text on any white space.​

      Regex.compile '(\S+)(?:\s+|$)' . tokenize 'Hello Big\r\nWide\tWorld\nGoodbye!'
          == ['Hello','Big','Wide','World','Goodbye!']