XSLT Tokenizes

Definition oF XSLT Tokenizes

XSLT tokenize is defined to break a declared string with one more delimiter character by treating each token as a node having <token> element and a part of XSLT2.0. A tokenizer splits up a string based on a regular expression and returns a node-set of token individual elements and loop over it by then changes the profile of the document.

Syntax:

The XSLT syntax is defined as

<xsl:call-template name="str:tokenize">
<xsl:with-param name="element" select="stringval" />
<xsl:with-param name="entitycharac" select="stringval" />?
</xsl:call-template>

The tokenize function takes two parameters. First, a list of values assigned in the input file, and the last is the delimiter character.

How Tokenize function work in XSLT?

A pure XSLT deals with tokenization. Let’s take a case where we have a comma in a list of values in a single element within the declared XML file, there is no proper way to neglect those in a Map Process, there pays a way for tokenization. The tokenize function checks a sequence of strings from an input file by breaking at any delimiting sub-part. This function can also be used other than a comma (like the semicolon, *). A few Path example to evaluate tokenize in a string is listed here:

tokenize( ‘m n o’, ‘\s’)	(‘m’, ‘n’, ‘o’)
tokenize( ‘m n o’, ‘\s’)	(‘m’, ”, ”, ‘n’, ‘o’)
tokenize( ‘m n o’, ‘\s+’)	(‘m’, ‘n’, ‘o’)
tokenize( ‘ m n’, ‘\s’)	(”, ‘m’, ‘n’)
tokenize( ‘m,n,o’, ‘,’)	(‘m’, ‘n’, ‘o’)
tokenize( ‘m,n,,o’, ‘,’)	(‘m’, ‘n’, ”, ‘o’)
tokenize( ‘2005-10-21T11:14:00’, ‘[\-T:]’)	(‘2006′, ’12’, ’25’, ’12’, ’15’, ’00’)
tokenize( ‘this, dog.’, ‘\W+’)	(‘ this ‘, ‘ dog ‘, ”)
tokenize( (), ‘\s+’)	()
tokenize( ‘mno’, ‘\s’)	mno

When we tokenize on a comma or space then they are combined into one single expression like this:

<xsl:for-each select="tokenize(., ',|\s')">

Tokenizing a number with a string gives the result as

tokenize ("11, 13, 22, 60", ",\s*")
Output is
("11", "13", "22", "60")

The Tokenize() processing of various XML files has made the code run very faster and the output which it produces is a nice mark-up. Let’s see a crisp example now. For example, if we have a source statement like:

tokenize (‘2002-05-02T12:30:23’, ‘-T:’) The following nodeset consists of :
<token>2002</token>
<token>05</token>
<token>02</token>
<token>12</token>
<token>30</token>
<token>23</token>

The first argument within the quotes is tokenized and evaluated into the individual set. The last argument is omitted with the whitespaces.

Examples

A simple XML file we are going to process

Example #1

<?xml-stylesheet type="text/xsl" href="simple.xsl"?>
<eatables>
<types>Vegetables, Fruits, Nuts</types>
</eatables>
Next is the XSL stylesheet needed to do transformation
<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
<xsl:strip-space elements="*" />
<xsl:template match="/">
<Grocelist>
<xsl:apply-templates select="eatables/types"/>
</Grocelist>
</xsl:template>
<xsl:template match="types">
<xsl:call-template name="tokenize">
<xsl:with-param name="aaa" select="." />
</xsl:call-template>
</xsl:template>
<xsl:template name="tokenize">
<xsl:param name="aaa" />
<xsl:variable name="pref1" select="normalize-space(
substring-before( concat( $aaa, ','), ','))" />
<xsl:if test="$pref1">
<first>
<xsl:value-of select="$pref1" />
</first>
<xsl:call-template name="tokenize">
<xsl:with-param name="aaa" select="substring-after($aaa,',')" />
</xsl:call-template>
</xsl:if>
</xsl:template>
</xsl:stylesheet>

Explanation

The stylesheet has a template match that matches the<eatables> element in the XML and it read the content to the variables declared in the stylesheet and then creates a tokenized variable that collects the list of the variable with commas. Finally, it is returned to the Output document as:

Output:

Example #2

Next XML file takes a Stocking database and we are going to process

<?xml version="1.0" encoding="iso-8859-1"?>
<service>
<stock name="europe">AU8702 -e -Ss country/AU8702pfb_AU -pf AU8702_AG</stock>
<stock name="e-exchange">
e231 1112 240 6,703.83.99999 Cango 921 37 -4.8 0.0325 update fiber index
e342 5214 514 5,972.15.99900 Navios 1005 26 -5.1 0.5558 regular internet FT100
e440 214 214 10,950.87.00000 Maritime 718 55 -1.2 0.7886 regular net Euronext100
</stock>
</service>

XSL

<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes" />
<xsl:template match="/">
<volume>
<xsl:apply-templates select="/service/stock" />
</volume>
</xsl:template>
<xsl:template match="stock[@name = 'e-exchange']">
<xsl:variable name="chance" select="." />
<xsl:variable name="tok_l" select="tokenize($chance, '\n')" />
<xsl:for-each select="$tok_l">
<xsl:variable select="tokenize(., '\s{2,}')" name="values" />
<country name="{$values[5]}">
<transit><xsl:value-of select="$values[10]"/></transit>
<connection><xsl:value-of select="$values[11]"/></connection>
</country>
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>

Explanation

Here we have parsed a long string in XML file and the fields are segregated like column 1 to column 11 and we required to print the result with the new elements and therefore we use powerful tokenize() in several ways in a code like using it in variable XSL and taking each-line in a for-each loop and also assigning a tokenized string into a value [no] which is referencing an index.

Output:

This Outputs the XML

Example #3

XML

<?xml version="1.0" encoding="UTF-8" ?>
<Community>
<Boosters>Work Day , programs, events</Boosters>
<Forums> Discussion , Topics,feedback</Forums>
<Usersgroup> inter,extern</Usersgroup>
</Community>

XSL

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="Community">
<Head>
<xsl:variable name="Boosters" select="Boosters" />
<xsl:for-each select="tokenize($Boosters, ',')">
<new1>
<xsl:value-of select="." />
</new1>
</xsl:for-each>
<xsl:variable name="Forums" select="Forums" />
<xsl:for-each select="tokenize($Forums, ',')">
<old>
<xsl:value-of select="." />
</old>
</xsl:for-each>
</Head>
</xsl:template>
</xsl:stylesheet>

Explanation

The code above is structured by splitting commas and delimiters from the source file and produces a code without commas.

Output:

Example #4

XML file for the watch element

<?xml version="1.0" encoding="UTF-8"?>
<Product>
<Watches>
:12:WT67895672333309
Titan:Men
Strap04:525854112,10
</Watches>
</Product>
XSLT file
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="2.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
<html>
<body>
<h3>Tokenizer String</h3>
<xsl:variable name="paragraph" select="'this is a watch description'" />
The Description: '
<xsl:value-of select="$paragraph" />
' follows
<xsl:value-of
select="count(tokenize($paragraph, '\s+'))" />
few words.
<br />
</body>
</html>
</xsl:template></xsl:stylesheet>

Explanation

This stylesheet counts the tokens available in the input file. And it is numbered as ‘5’. The output is shown as:

Output:

Example #5

XML

<Patients>
<field pname="id">01,12,13</field>
<field pname="name">gia,bob,carl</field>
</Patients>

XSL

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output indent="yes" method="xml"/>
<xsl:strip-space elements="*"/>
<xsl:template match="/Patients">
<xsl:copy>
<xsl:apply-templates/>
</xsl:copy>
</xsl:template>
<xsl:template match="field[@pname='id']">
<xsl:variable name="match" select="."/>
<xsl:for-each select="tokenize(.,',')">
<xsl:variable name="xxx" select="position()"/>
<Patients>
<id>
<xsl:value-of select="."/>
</id>
<pname>
<xsl:value-of select="tokenize($match/following-sibling::field[@pname='name'],',')[position() = $xxx]"/>
</pname>
</Patients>
</xsl:for-each>
</xsl:template>
<xsl:template match="text()"/>
</xsl:stylesheet>

Explanation

In the stylesheet code, the expression relies on the context where the value of the attribute is retrieved and each element are tokenized into an individual element.

Output:

Conclusion

Therefore, in this article, we have seen how to create an XSLT stylesheet for the tokenize function and with a simple example, we had gone through how to split a comma-delimited string from the source document into individual elements. This makes things easier when it required comma eliminations.

Recommended Articles

This is a guide to XSLT Tokenizes. Here we discuss definition, syntax, and parameters, How tokenize function work in XSLT? examples with code implementation. You may also have a look at the following articles to learn more –

1. XSLT Transformation
2. XSLT count
3. XSLT substring-after
4. XSLT replace