Showing posts with label String. Show all posts
Showing posts with label String. Show all posts

Tuesday, December 30, 2014

String Replacement Methods: StringTemplate

A String Replacement Overview is here.

StringTemplate

StringTemplate saves you the trouble of searching for a String subset within a String to replace or setting up your own marker to flag the StringPosition in the String at which to perform a replacement.

Further, good programming practice dictates that we use selectors and constructors – specialized, dedicated functions to extract a subset of a file or to change a subset of a file – and to always use those rather than ad hoc one liners scattered in our functions and programs.1,2 StringTemplate conveniently formalizes and enforces the use of selectors and constructors.

StringForm is simpler to understand and use than StringTemplate, so I use StringForm when you need to output a message from your function. I don't end the command with a semi-colon so you can see the InputForm of a TemplateObject including its default Options.

stringTemplate1=StringTemplate@"The quick brown `` jumped over the lazy white ``."

TemplateObject[{The quick brown ,TemplateSlot[1], jumped over the lazy white ,TemplateSlot[2],.},InsertionFunction->TextString,CombinerFunction->StringJoin]

You can directly Apply any StringTemplate as a function to a List of its arguments that fits its requirements, or use TemplateApply to do the same thing.

stringTemplate1@@{"mink","peccadillo"}

The quick brown mink jumped over the lazy white peccadillo.

Equivalently, here StringTemplate is used as a function as you would any other function – use it as the Head of an Expression with its arguments.

stringTemplate1["mink","peccadillo"]

The quick brown mink jumped over the lazy white peccadillo.

Equivalently, using TemplateApply:

TemplateApply[stringTemplate1,{"mink","peccadillo"}]

The quick brown mink jumped over the lazy white peccadillo.

1. Maeder, Roman, Computer Science with Mathematica. Cambridge: Cambridge University Press, 2000. Chapter 5.3. Design of Abstract Data Types.

2. Maeder, Roman, M220: Programming in Mathematica  (course given by Wolfram Education Group, which I have taken twice and recommend).

String Replacement Methods: Overview

Here are String replacement methods that I have used in code from one-liners up to programs producing hundreds of thousands of text and html files. In general, use the simplest method or one that you understand clearly. Use StringForm to output messages from your functions and programs. For longer functions or programs, StringTemplate is the new best practice.

There is a function I don't discuss, StringInsert, which inserts a substring at a given StringPosition in a control String. I don't advocate its use since it's very brittle in that if you add or delete even one character before the StringPosition then the insertion point will be wrong.

StringForm

Literal Replacement, Markers, and Delimiters

String Replacement Methods: Literal Replacement, Markers, and Delimiters

A String Replacement Overview is here.

Note that the next three methods all use StringReplace. This is in keeping with my principle that the fastest way to learn Mathematica is to become a power user of its 70 or so core functions. In String processing, for instance, StringInsert is not a function you need to know. Instead learn to use the more powerful and robust function, StringReplace.

Literal Replacement

Literal replacement works by using StringReplace to find a literal substring within a String and substitute another substring for it. Literal replacement is very simple and easy to use.

string1="The quick brown fox jumped over the lazy white dog.";

StringReplace[string1,{"fox"->"mink","dog"->"pecadillo"}]

The quick brown mink jumped over the lazy white pecadillo.

Markers

Using markers to indicate the replacement position can improve code legibility. Use StringReplace to replace just the marked text.

string2="The quick brown <animal1> jumped over the lazy white <animal2>.";

StringReplace[string2,{"<animal1>"->"mink","<animal2>"->"pecadillo"}]

The quick brown mink jumped over the lazy white pecadillo.

Delimiters

Use StringReplace to replace text between the delimiters. This is very useful when you want to replace a lot of text in a document, especially in a long document. However, the new function StringTemplate is a superior method overall.

sitemapTemplate="<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">
<!-- put list of urls here with a line feed after each one -->
</urlset>";

urls="<url><loc>http://www.blah.net/page1.html</loc></url>
<url><loc>http://www.blah.net/page2.html</loc></url>";

Note that you use StringExpression (shorthand "~~") to concatenate quoted Strings with Blanks in the String to be found by StringReplace, but you must use StringJoin (shorthand "<>") if you concatenate different Strings in the replacement String.

sitemapTemplateWithURLs=StringReplace[sitemapTemplate,"<!-- put list"~~urlsList__~~"each one -->"->urls]

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>http://www.blah.net/page1.html</loc></url>
<url><loc>http://www.blah.net/page2.html</loc></url>
</urlset>

String Replacement Methods: StringForm

A String Replacement Overview is here.

StringForm

StringForm is a simple, elegant String template function. Use it in your functions where Print isn't enough since you need to fill in some variables, such as calculations on the fly. In a function use the following form with Print to output it since lines end with an output-suppressing semi-colon.

Print@StringForm["control string", variables];

Here the double backtick marks tell Mathematica where to fill in the blanks with arguments you give in the order in which they are inserted into the String. An argument can be a String or an Expression of unlimited complexity, which will be evaluated before insertion. If you don't want the inserted Expression to be evaluated, though, use HoldForm (example below).

StringForm["Use `` for relatively short and simple String templates, such as output messages in your functions. For example, the cube root of `` is ``.","StringForm",27,27^(1/3)]

Use StringForm for relatively short and simple String templates, such as output messages in your functions. For example, the cube root of 27 is 3.

If you're going to use an argument twice, switch the order, or use a number of arguments and you want to prevent mistakes, use numbered, rather than ordered, backticks. You often want a line break, for which the \n escape character is used within the quotation marks that are Mathematica's String delimiters.

StringForm["Flying or gliding mammals include `1`, `2`,\n`3`, `4`, `5`, `6`, and `7`.\nThe most common species are in the `3` family.","flying possums","greater glider","bats","flying squirrels","flying lemurs","flying monkeys","cats"]

Flying or gliding mammals include flying possums, greater glider,
bats, flying squirrels, flying lemurs, flying monkeys, and cats.
The most common species are in the bats family.

To prevent the inserted Expression from being evaluated, use HoldForm:

StringForm["For example, the sixth term of the Fibonacci series is the sum of the preceding two terms: ``.",HoldForm[1+1+2+3+5=8]]


For example, the sixth term of the Fibonacci series is the sum of the preceding two terms: 1+1+2+3+5=8.

Wednesday, April 16, 2014

String Operations: The Essential Function StringSplit

Split a String and Gather its Words by Their First Letter

Let's process a nice sonnet I just read in Ray Kurzweil's excellent book on what he thinks is the heart of cognitive science and AI, How to Create a Mind: The Secret of Human Thought Revealed.
To start I copy and paste the text from the web between quotation marks.

In[377]:= sonnet73=
"That time of year thou mayst in me behold,
When yellow leaves, or none, or few, do hang
Upon those boughs which shake against the cold,
Bare ruined choirs, where late the sweet birds sang.
In me thou seest the twilight of such day,
As after sunset fadeth in the west,
Which by and by black night doth take away,
Death's second self, that seals up all in rest.
In me thou seest the glowing of such fire,
That on the ashes of his youth doth lie,
As the death-bed whereon it must expire,
Consumed with that which it was nourished by.
This thou perceiv'st, which makes thy love more strong,
To love that well, which thou must leave ere long.";

Can we use ToLowerCase without Map? Yes, since it's Listable.

In[380]:= sonnet73LowerCase=ToLowerCase@%377

Out[380]= that time of year thou mayst in me behold,
when yellow leaves, or none, or few, do hang
upon those boughs which shake against the cold,
bare ruined choirs, where late the sweet birds sang.
in me thou seest the twilight of such day,
as after sunset fadeth in the west,
which by and by black night doth take away,
death's second self, that seals up all in rest.
in me thou seest the glowing of such fire,
that on the ashes of his youth doth lie,
as the death-bed whereon it must expire,
consumed with that which it was nourished by.
this thou perceiv'st, which makes thy love more strong,
to love that well, which thou must leave ere long.

We need StringSplit, not SplitBy, since we are working with a String. StringSplit is a powerful and essential function when importing and transforming large files to the format you need. Examples in the Doc Center like this one show its versatility:

In[1]:= StringSplit["a-b:c-d:e-f-g",{":","-"}]

Out[1]= {a,b,c,d,e,f,g}

Using WordBoundary as the pattern test for where to split works, but leaves much more "noise". Here is the output. Using InputForm is often handy to see what is going in while processing Strings.

In[409]:= StringSplit[sonnet73LowerCase,WordBoundary]//InputForm

Out[409]//InputForm=
{"that", " ", "time", " ", "of", " ", "year", " ", "thou", " ", "mayst", " ", "in",
 " ", "me", " ", "behold", ",\n", "when", " ", "yellow", " ", "leaves", ", ", "or",
 " ", "none", ", ", "or", " ", "few", ", ", "do", " ", "hang", "\n", "upon", " ",
 "those", " ", "boughs", " ", "which", " ", "shake", " ", "against", " ", "the", " ",
 "cold", ",\n", "bare", " ", "ruined", " ", "choirs", ", ", "where", " ", "late", " ",
 "the", " ", "sweet", " ", "birds", " ", "sang", ".\n", "in", " ", "me", " ", "thou",
 " ", "seest", " ", "the", " ", "twilight", " ", "of", " ", "such", " ", "day", ",\n",
 "as", " ", "after", " ", "sunset", " ", "fadeth", " ", "in", " ", "the", " ", "west",
 ",\n", "which", " ", "by", " ", "and", " ", "by", " ", "black", " ", "night", " ",
 "doth", " ", "take", " ", "away", ",\n", "death", "'", "s", " ", "second", " ",
 "self", ", ", "that", " ", "seals", " ", "up", " ", "all", " ", "in", " ", "rest",
 ".\n", "in", " ", "me", " ", "thou", " ", "seest", " ", "the", " ", "glowing", " ",
 "of", " ", "such", " ", "fire", ",\n", "that", " ", "on", " ", "the", " ", "ashes",
 " ", "of", " ", "his", " ", "youth", " ", "doth", " ", "lie", ",\n", "as", " ", "the",
 " ", "death", "-", "bed", " ", "whereon", " ", "it", " ", "must", " ", "expire",
 ",\n", "consumed", " ", "with", " ", "that", " ", "which", " ", "it", " ", "was", " ",
 "nourished", " ", "by", ".\n", "this", " ", "thou", " ", "perceiv", "'", "st", ", ",
 "which", " ", "makes", " ", "thy", " ", "love", " ", "more", " ", "strong", ",\n",
 "to", " ", "love", " ", "that", " ", "well", ", ", "which", " ", "thou", " ", "must",
 " ", "leave", " ", "ere", " ", "long", "."}

What a mess! And using this lengthy construct to DeleteCases of punctuation still left an imperfect result (which I won't show in case you're about to eat).

%//DeleteCases[#,""|" "|","|", "|" ,"|"'"|"s"|"st"|",\n"|",\n"|".\n"|"\n"|"-"|"."|" . "|"  . "]&

Compare the result from proper use of StringSplit.

In[405]:= sonnet73Split=StringSplit[sonnet73LowerCase,{Whitespace,".",","}]//DeleteCases[#,""]&

Out[405]= {that,time,of,year,thou,mayst,in,me,behold,when,yellow,leaves,or,none,or,few,do,hang,upon,those,boughs,which,shake,against,the,cold,bare,ruined,choirs,where,late,the,sweet,birds,sang,in,me,thou,seest,the,twilight,of,such,day,as,after,sunset,fadeth,in,the,west,which,by,and,by,black,night,doth,take,away,death's,second,self,that,seals,up,all,in,rest,in,me,thou,seest,the,glowing,of,such,fire,that,on,the,ashes,of,his,youth,doth,lie,as,the,death-bed,whereon,it,must,expire,consumed,with,that,which,it,was,nourished,by,this,thou,perceiv'st,which,makes,thy,love,more,strong,to,love,that,well,which,thou,must,leave,ere,long}

Here GatherBy groups sub-lists by the first character of each word.

In[411]:= sonnet73SplitGathered=GatherBy[sonnet73Split,First@Characters@#&]

Out[411]= {{that,time,thou,those,the,the,thou,the,twilight,the,take,that,thou,the,that,the,the,that,this,thou,thy,to,that,thou},{of,or,or,of,of,on,of},{year,yellow,youth},{mayst,me,me,me,must,makes,more,must},{in,in,in,in,in,it,it},{behold,boughs,bare,birds,by,by,black,by},{when,which,where,west,which,whereon,with,which,was,which,well,which},{leaves,late,lie,love,love,leave,long},{none,night,nourished},{few,fadeth,fire},{do,day,doth,death's,doth,death-bed},{hang,his},{upon,up},{shake,sweet,sang,seest,such,sunset,second,self,seals,seest,such,strong},{against,as,after,and,away,all,ashes,as},{cold,choirs,consumed},{ruined,rest},{glowing},{expire,ere},{perceiv'st}}

To see the Lists sorted by their initial letter, we should use SortBy, but this didn't work.

In[412]:= SortBy[%,First@Characters@#&]

We need to add another First to apply the sorting function to the First word in each sublist.

In[413]:= SortBy[%,First@Characters@First@#&]

Out[413]= {{against,as,after,and,away,all,ashes,as},{behold,boughs,bare,birds,by,by,black,by},{cold,choirs,consumed},{do,day,doth,death's,doth,death-bed},{expire,ere},{few,fadeth,fire},{glowing},{hang,his},{in,in,in,in,in,it,it},{leaves,late,lie,love,love,leave,long},{mayst,me,me,me,must,makes,more,must},{none,night,nourished},{of,or,or,of,of,on,of},{perceiv'st},{ruined,rest},{shake,sweet,sang,seest,such,sunset,second,self,seals,seest,such,strong},{that,time,thou,those,the,the,thou,the,twilight,the,take,that,thou,the,that,the,the,that,this,thou,thy,to,that,thou},{upon,up},{when,which,where,west,which,whereon,with,which,was,which,well,which},{year,yellow,youth}}

Sunday, March 9, 2014

Example of a File of Functions to Pre-Load at Startup

Here is a portion of my file of functions that I load on startup so they're available in every session. See also Load Functions at Startup Using Initialization Files.

Functions to Pre-Load


String Processing

Cleaners

trimWhiteSpace::usage="trimWhiteSpace is a simple stringtrim function that does a better job than StringTrim since it removes all WhiteSpaceCharacters from beginning, end or middle of Strings. It is Listable and is intended to Map over all fields in all records where all fields are Strings, no matter if they are wrapped in Lists at any level."

Clear@trimWhiteSpace;
SetAttributes[trimWhiteSpace,Listable];
trimWhiteSpace@stringFile_:=StringReplace[stringFile,WhitespaceCharacter->""];

cleanseText::usage="cleanseText is a moderately thorough stringtrim function. The regular expression removes whitespace characters from the beginning and end of the string, including non-breaking spaces. StringReplace removes apostrophes and commas. It is Listable and intended to automatically Map over all fields in all records where all fields are Strings, no matter if they are wrapped in Lists at any level. These rules should be expanded to include more data cleansing functions as needed."

Clear@cleanseText;
SetAttributes[cleanseText,Listable];
cleanseText@text_String:=StringReplace[text,RegularExpression["^(\\s)*|(\\s)*$"]->""]//StringReplace[#,{"'"->"",WhitespaceCharacter->" ","("|")"->""}]&;

Web Page Functions


Make a URL from a String

makeURL::usage="makeURL takes a String to use as the prefix of the URL, cleans it up (removes whitespace and non-breaking spaces (which are often invisible), puts the String in lower case, replaces single spaces with hyphens and finally appends '.html'. Example: 'My Home Page' -> my-home-page.html".

Clear@makeURL;
SetAttributes[makeURL,Listable];
makeURL@urlName_String:=ToLowerCase@cleanseText@urlName//StringReplace[#,Whitespace->"-"]&//StringReplace[#,"---"->"-"]&//FileNameJoin[{#<>".html"}]&;