DB_Table.tokenize_to_columns

tokenize_to_columnscolumnpatterncase_sensitivitycolumn_count on_problems

Group: Conversions

Documentation

Tokenizes a column of text into a set of new columns using a regular expression. If the pattern contains marked groups, the values are concatenated together; otherwise the whole match is returned. The original column will be removed from the table. The new columns will be named with the name of the input column with a incrementing number after.

Arguments

• column: The name or index of the column to tokenize the text of.
• pattern: The pattern used to find within the text.
• case_sensitivity: Specifies if the text values should be compared case sensitively.
• column_count: The number of columns to split to. If Nothing then columns will be added to fit all data.
• on_problems: Specifies the behavior when a problem occurs.

Errors

If the data exceeds the column_count, a Column_Count_Exceeded will be reported according to the on_problems behavior.