EDUCBA Logo

EDUCBA

MENUMENU
  • Explore
    • EDUCBA Pro
    • PRO Bundles
    • Featured Skills
    • New & Trending
    • Fresh Entries
    • Finance
    • Data Science
    • Programming and Dev
    • Excel
    • Marketing
    • HR
    • PDP
    • VFX and Design
    • Project Management
    • Exam Prep
    • All Courses
  • Blog
  • Enterprise
  • Free Courses
  • Log in
  • Sign Up
Home Software Development Software Development Tutorials XML Tutorial XSLT Tokenizes
 

XSLT Tokenizes

Updated June 28, 2023

XSLT Tokenizes

 

 

Definition oF XSLT Tokenizes

XSLT tokenize is defined as breaking a declared string with one more delimiter character by treating each token as a node with <token> element and a part of XSLT2.0. A tokenizer splits up a string based on a regular expression. It returns a node set of token individual elements and loops over it by changing the document profile.

Watch our Demo Courses and Videos

Valuation, Hadoop, Excel, Mobile Apps, Web Development & many more.

Syntax:

The XSLT syntax is defined as

<xsl:call-template name="str:tokenize">
<xsl:with-param name="element" select="stringval" />
<xsl:with-param name="entitycharac" select="stringval" />?
</xsl:call-template>

The tokenize function takes two parameters. First, a list of values is assigned in the input file, and the last is the delimiter character.

How does the Tokenize function work in XSLT?

A pure XSLT deals with tokenization. Let’s take a case where we have a comma in a list of values in a single element within the declared XML file; there is no proper way to neglect those in a Map Process; there pays a way for tokenization. The tokenize function checks a sequence of strings from an input file by breaking at any delimiting sub-part. This function can also be used other than a comma (like the semicolon, *). A few Path examples of evaluating tokenize in a string are listed here:

tokenize( ‘m n o’, ‘\s’) (‘m’, ‘n’, ‘o’)
tokenize( ‘m n o’, ‘\s’) (‘m’, ”, ”, ‘n’, ‘o’)
tokenize( ‘m n o’, ‘\s+’) (‘m’, ‘n’, ‘o’)
tokenize( ‘ m n’, ‘\s’) (”, ‘m’, ‘n’)
tokenize( ‘m,n,o’, ‘,’) (‘m’, ‘n’, ‘o’)
tokenize( ‘m,n,,o’, ‘,’) (‘m’, ‘n’, ”, ‘o’)
tokenize( ‘2005-10-21T11:14:00’, ‘[\-T:]’) (‘2006′, ’12’, ’25’, ’12’, ’15’, ’00’)
tokenize( ‘this, dog.’, ‘\W+’) (‘this ‘, ‘dog ‘,”)
tokenize( (), ‘\s+’) ()
tokenize( ‘mno’, ‘\s’) mno

When we tokenize a comma or space, then they are combined into one single expression like this:

<xsl:for-each select="tokenize(., ',|\s')">

Tokenizing a number with a string gives the result as

tokenize ("11, 13, 22, 60", ",\s*")
Output is
("11", "13", "22", "60")

The Tokenize() processing of various XML files has made the code run faster, and its output is a nice mark-up. Let’s see a crisp example now. For example, if we have a source statement like:

tokenize (‘2002-05-02T12:30:23’, ‘-T:’) The following nodeset consists of :
<token>2002</token>
<token>05</token>
<token>02</token>
<token>12</token>
<token>30</token>
<token>23</token>

The first argument within the quotes is tokenized and evaluated into the individual set. The last argument is omitted with the whitespaces.

Examples of XSLT Tokenizes

A simple XML file we are going to process

Example #1

<?xml-stylesheet type="text/xsl" href="simple.xsl"?>
<eatables>
<types>Vegetables, Fruits, Nuts</types>
</eatables>
Next is the XSL stylesheet needed to do transformation
<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
<xsl:strip-space elements="*" />
<xsl:template match="/">
<Grocelist>
<xsl:apply-templates select="eatables/types"/>
</Grocelist>
</xsl:template>
<xsl:template match="types">
<xsl:call-template name="tokenize">
<xsl:with-param name="aaa" select="." />
</xsl:call-template>
</xsl:template>
<xsl:template name="tokenize">
<xsl:param name="aaa" />
<xsl:variable name="pref1" select="normalize-space(
substring-before( concat( $aaa, ','), ','))" />
<xsl:if test="$pref1">
<first>
<xsl:value-of select="$pref1" />
</first>
<xsl:call-template name="tokenize">
<xsl:with-param name="aaa" select="substring-after($aaa,',')" />
</xsl:call-template>
</xsl:if>
</xsl:template>
</xsl:stylesheet>

Explanation

The stylesheet has a template match that matches the<eatables> element in the XML, and it reads the content to the variables declared in the stylesheet and then creates a tokenized variable that collects the list of the variable with commas. Finally, it is returned to the Output document as:

Output:

XSLT Tokenizes 1

Example #2

Next XML file takes a Stocking database, and we going to process

<?xml version="1.0" encoding="iso-8859-1"?>
<service>
<stock name="europe">AU8702 -e -Ss country/AU8702pfb_AU -pf AU8702_AG</stock>
<stock name="e-exchange">
e231 1112 240 6,703.83.99999 Cango 921 37 -4.8 0.0325 update fiber index
e342 5214 514 5,972.15.99900 Navios 1005 26 -5.1 0.5558 regular internet FT100
e440 214 214 10,950.87.00000 Maritime 718 55 -1.2 0.7886 regular net Euronext100
</stock>
</service>

XSL

<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes" />
<xsl:template match="/">
<volume>
<xsl:apply-templates select="/service/stock" />
</volume>
</xsl:template>
<xsl:template match="stock[@name = 'e-exchange']">
<xsl:variable name="chance" select="." />
<xsl:variable name="tok_l" select="tokenize($chance, '\n')" />
<xsl:for-each select="$tok_l">
<xsl:variable select="tokenize(., '\s{2,}')" name="values" />
<country name="{$values[5]}">
<transit><xsl:value-of select="$values[10]"/></transit>
<connection><xsl:value-of select="$values[11]"/></connection>
</country>
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>

Explanation

We have parsed a long string in an XML file, and the fields are segregated like column 1 to column 11. We are required to print the result with the new elements. Therefore we use powerful tokenize() in several ways in a code, like using it in variable XSL, taking each line in a for-each loop, and assigning a tokenized string into a value [no] referencing an index.

Output:

This Outputs the XML

XSLT Tokenizes 2

Example #3

XML

<?xml version="1.0" encoding="UTF-8" ?>
<Community>
<Boosters>Work Day , programs, events</Boosters>
<Forums> Discussion , Topics,feedback</Forums>
<Usersgroup> inter,extern</Usersgroup>
</Community>

XSL

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="Community">
<Head>
<xsl:variable name="Boosters" select="Boosters" />
<xsl:for-each select="tokenize($Boosters, ',')">
<new1>
<xsl:value-of select="." />
</new1>
</xsl:for-each>
<xsl:variable name="Forums" select="Forums" />
<xsl:for-each select="tokenize($Forums, ',')">
<old>
<xsl:value-of select="." />
</old>
</xsl:for-each>
</Head>
</xsl:template>
</xsl:stylesheet>

Explanation

The code above is structured by splitting commas and delimiters from the source file and produces a code without commas.

Output:

XSLT Tokenizes 3

Example #4

XML file for the watch element

<?xml version="1.0" encoding="UTF-8"?>
<Product>
<Watches>
:12:WT67895672333309
Titan:Men
Strap04:525854112,10
</Watches>
</Product>
XSLT file
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="2.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
<html>
<body>
<h3>Tokenizer String</h3>
<xsl:variable name="paragraph" select="'this is a watch description'" />
The Description: '
<xsl:value-of select="$paragraph" />
' follows
<xsl:value-of
select="count(tokenize($paragraph, '\s+'))" />
few words.
<br />
</body>
</html>
</xsl:template></xsl:stylesheet>

Explanation

This stylesheet counts the tokens available in the input file. And it is numbered ‘5’. The output is shown as follows:

Output:

Example 4

Example #5

XML

<Patients>
<field pname="id">01,12,13</field>
<field pname="name">gia,bob,carl</field>
</Patients>

XSL

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output indent="yes" method="xml"/>
<xsl:strip-space elements="*"/>
<xsl:template match="/Patients">
<xsl:copy>
<xsl:apply-templates/>
</xsl:copy>
</xsl:template>
<xsl:template match="field[@pname='id']">
<xsl:variable name="match" select="."/>
<xsl:for-each select="tokenize(.,',')">
<xsl:variable name="xxx" select="position()"/>
<Patients>
<id>
<xsl:value-of select="."/>
</id>
<pname>
<xsl:value-of select="tokenize($match/following-sibling::field[@pname='name'],',')[position() = $xxx]"/>
</pname>
</Patients>
</xsl:for-each>
</xsl:template>
<xsl:template match="text()"/>
</xsl:stylesheet>

Explanation

In the stylesheet code, the expression relies on the context where the attribute’s value is retrieved, and each element is tokenized into an individual element.

Output:

example 5

Conclusion

Therefore, we have seen how to create an XSLT stylesheet for the tokenize function in this article. With a simple example, we have divided a comma-delimited string from the source document into individual elements. This makes things easier when it requires comma eliminations.

Recommended Articles

This is a guide to XSLT Tokenizes. Here we discuss the definition, syntax, and parameters. How does the tokenized function work in XSLT? Examples with code implementation. You may also have a look at the following articles to learn more –

  1. XSLT Transformation
  2. XSLT count
  3. XSLT substring-after
  4. XSLT replace

Primary Sidebar

Footer

Follow us!
  • EDUCBA FacebookEDUCBA TwitterEDUCBA LinkedINEDUCBA Instagram
  • EDUCBA YoutubeEDUCBA CourseraEDUCBA Udemy
APPS
EDUCBA Android AppEDUCBA iOS App
Blog
  • Blog
  • Free Tutorials
  • About us
  • Contact us
  • Log in
Courses
  • Enterprise Solutions
  • Free Courses
  • Explore Programs
  • All Courses
  • All in One Bundles
  • Sign up
Email
  • [email protected]

ISO 10004:2018 & ISO 9001:2015 Certified

© 2025 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA
Free Software Development Course

Web development, programming languages, Software testing & others

By continuing above step, you agree to our Terms of Use and Privacy Policy.
*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

EDUCBA Login

Forgot Password?

🚀 Limited Time Offer! - 🎁 ENROLL NOW