Showing posts with label StringReplace. Show all posts
Showing posts with label StringReplace. Show all posts

Tuesday, December 30, 2014

String Replacement Methods: StringTemplate

A String Replacement Overview is here.

StringTemplate

StringTemplate saves you the trouble of searching for a String subset within a String to replace or setting up your own marker to flag the StringPosition in the String at which to perform a replacement.

Further, good programming practice dictates that we use selectors and constructors – specialized, dedicated functions to extract a subset of a file or to change a subset of a file – and to always use those rather than ad hoc one liners scattered in our functions and programs.1,2 StringTemplate conveniently formalizes and enforces the use of selectors and constructors.

StringForm is simpler to understand and use than StringTemplate, so I use StringForm when you need to output a message from your function. I don't end the command with a semi-colon so you can see the InputForm of a TemplateObject including its default Options.

stringTemplate1=StringTemplate@"The quick brown `` jumped over the lazy white ``."

TemplateObject[{The quick brown ,TemplateSlot[1], jumped over the lazy white ,TemplateSlot[2],.},InsertionFunction->TextString,CombinerFunction->StringJoin]

You can directly Apply any StringTemplate as a function to a List of its arguments that fits its requirements, or use TemplateApply to do the same thing.

stringTemplate1@@{"mink","peccadillo"}

The quick brown mink jumped over the lazy white peccadillo.

Equivalently, here StringTemplate is used as a function as you would any other function – use it as the Head of an Expression with its arguments.

stringTemplate1["mink","peccadillo"]

The quick brown mink jumped over the lazy white peccadillo.

Equivalently, using TemplateApply:

TemplateApply[stringTemplate1,{"mink","peccadillo"}]

The quick brown mink jumped over the lazy white peccadillo.

1. Maeder, Roman, Computer Science with Mathematica. Cambridge: Cambridge University Press, 2000. Chapter 5.3. Design of Abstract Data Types.

2. Maeder, Roman, M220: Programming in Mathematica  (course given by Wolfram Education Group, which I have taken twice and recommend).

String Replacement Methods: Overview

Here are String replacement methods that I have used in code from one-liners up to programs producing hundreds of thousands of text and html files. In general, use the simplest method or one that you understand clearly. Use StringForm to output messages from your functions and programs. For longer functions or programs, StringTemplate is the new best practice.

There is a function I don't discuss, StringInsert, which inserts a substring at a given StringPosition in a control String. I don't advocate its use since it's very brittle in that if you add or delete even one character before the StringPosition then the insertion point will be wrong.

StringForm

Literal Replacement, Markers, and Delimiters

String Replacement Methods: Literal Replacement, Markers, and Delimiters

A String Replacement Overview is here.

Note that the next three methods all use StringReplace. This is in keeping with my principle that the fastest way to learn Mathematica is to become a power user of its 70 or so core functions. In String processing, for instance, StringInsert is not a function you need to know. Instead learn to use the more powerful and robust function, StringReplace.

Literal Replacement

Literal replacement works by using StringReplace to find a literal substring within a String and substitute another substring for it. Literal replacement is very simple and easy to use.

string1="The quick brown fox jumped over the lazy white dog.";

StringReplace[string1,{"fox"->"mink","dog"->"pecadillo"}]

The quick brown mink jumped over the lazy white pecadillo.

Markers

Using markers to indicate the replacement position can improve code legibility. Use StringReplace to replace just the marked text.

string2="The quick brown <animal1> jumped over the lazy white <animal2>.";

StringReplace[string2,{"<animal1>"->"mink","<animal2>"->"pecadillo"}]

The quick brown mink jumped over the lazy white pecadillo.

Delimiters

Use StringReplace to replace text between the delimiters. This is very useful when you want to replace a lot of text in a document, especially in a long document. However, the new function StringTemplate is a superior method overall.

sitemapTemplate="<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">
<!-- put list of urls here with a line feed after each one -->
</urlset>";

urls="<url><loc>http://www.blah.net/page1.html</loc></url>
<url><loc>http://www.blah.net/page2.html</loc></url>";

Note that you use StringExpression (shorthand "~~") to concatenate quoted Strings with Blanks in the String to be found by StringReplace, but you must use StringJoin (shorthand "<>") if you concatenate different Strings in the replacement String.

sitemapTemplateWithURLs=StringReplace[sitemapTemplate,"<!-- put list"~~urlsList__~~"each one -->"->urls]

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>http://www.blah.net/page1.html</loc></url>
<url><loc>http://www.blah.net/page2.html</loc></url>
</urlset>

Friday, March 21, 2014

String Processing: Compose Last Name + First Initial and Create Email Address

String Processing: Compose First Name + First Initial and Create Email Addresses

I've imported a List of teacher's names. Now I need to create usernames and email addresses for them to upload to an app. Here is the List; note there are nested Lists within it.

In[226]:= teacherNames={{{"Bazan, Dorothy","Bleakney, Tom"},{"Calorio, Marie","Cardoso, Brian","Carpenito, Courtney","Carton, Jacqueline"},{"Coffey, Tara","Conaty, Thomas"}},{{"Davenport, Joseph","DiSabatino, Jennifer"},{"Doran, Robert","Dsida, Terri"},"Eramo, Paula","Needle, Julie",{"O'Connell, Marisa","O'Neil, Christopher"}}};

The nesting is just going to get in the way so let's remove extra braces with Flatten. Incidentally, you need not hesitate to build up deeply nested Lists in a function and then Flatten them all at once, which is very fast. To keep seeing these names as Strings, we Postfix InputForm to display the quotation marks.

In[227]:= teacherNamesFlattened=Flatten@teacherNames;teacherNamesFlattened//InputForm

Out[227]//InputForm=
{"Bazan, Dorothy", "Bleakney, Tom", "Calorio, Marie", "Cardoso, Brian",
 "Carpenito, Courtney", "Carton, Jacqueline", "Coffey, Tara", "Conaty, Thomas",
 "Davenport, Joseph", "DiSabatino, Jennifer", "Doran, Robert", "Dsida, Terri",
 "Eramo, Paula", "Needle, Julie", "O'Connell, Marisa", "O'Neil, Christopher"}

Let's lose the apostrophes. It's as simple as using a replacement Rule sending apostrophe to no character ("" ' "" -> ""). Without InputForm we don't see the quotation marks, but unless we use ToExpression to convert it, a String remains a String forever.

In[228]:= teacherNamesFlattenedCleaned=StringReplace[teacherNamesFlattened,"'"->""]
Out[228]= {Bazan, Dorothy,Bleakney, Tom,Calorio, Marie,Cardoso, Brian,Carpenito, Courtney,Carton, Jacqueline,Coffey, Tara,Conaty, Thomas,Davenport, Joseph,DiSabatino, Jennifer,Doran, Robert,Dsida, Terri,Eramo, Paula,Needle, Julie,OConnell, Marisa,ONeil, Christopher}

To compose usernames, we'll use a standard format, for instance, lastName + firstInitial. We use StringReplace, which takes one or more replacement Rules as arguments. We construct a pattern using deconstructed pieces of the first and last name. Note that named Blank patterns work in this type of String pattern matching.

Since I'm now going to perform several operations on the source List, for clarity, I will use Postfix with Function syntax and avoid deeply nested, hard-to-read code. The rationale behind this syntax is in A New Mathematica Programming Style Part 2..

This code works, but what is going wrong?

In[229]:= userNames=teacherNamesFlattenedCleaned//StringReplace[#,lastName__<>", "<>firstName__:>lastName<>First@Characters@firstName]&

During evaluation of In[229]:= StringJoin::string: String expected at position 1 in lastName__<>, <>firstName__. >>
During evaluation of In[229]:= StringJoin::string: String expected at position 3 in lastName__<>, <>firstName__. >>

Out[229]= {BazanD,BleakneyT,CalorioM,CardosoB,CarpenitoC,CartonJ,CoffeyT,ConatyT,DavenportJ,DiSabatinoJ,DoranR,DsidaT,EramoP,NeedleJ,OConnellM,ONeilC}

The error is due to using StringJoin ("<>") correctly to compose the deconstructed Strings but incorrectly to deconstruct them with patterns. Using the correct operator, StringExpression ("~~") solves the problem. Also let's convert the usernames to all lower case. Since ToLowerCase is Listable, we don't need to Map it over all the usernames.

In[230]:= userNames=teacherNamesFlattenedCleaned//StringReplace[#,lastName__~~", "~~firstName__:>lastName<>First@Characters@firstName]&//ToLowerCase

Out[230]= {bazand,bleakneyt,caloriom,cardosob,carpenitoc,cartonj,coffeyt,conatyt,davenportj,disabatinoj,doranr,dsidat,eramop,needlej,oconnellm,oneilc}

We can just as easily compose usernames with a reversed form, firstInitial + lastName.

In[231]:= userNames2=teacherNamesFlattenedCleaned//StringReplace[#,lastName__~~", "~~firstName__:>First@Characters@firstName<>lastName]&//ToLowerCase

Out[231]= {dbazan,tbleakney,mcalorio,bcardoso,ccarpenito,jcarton,tcoffey,tconaty,jdavenport,jdisabatino,rdoran,tdsida,peramo,jneedle,moconnell,coneil}

To compose email addresses, in a similar fashion, we'll use a template form, firstName.lastName@schoolDomain.org.

In[232]:= emailAddresses=teacherNamesFlattenedCleaned//StringReplace[#,lastName__~~", "~~firstName__:>firstName<>"."<>lastName<>"@hometownschools.org"]&//ToLowerCase

Out[232]= {dorothy.bazan@hometownschools.org,tom.bleakney@hometownschools.org,marie.calorio@hometownschools.org,brian.cardoso@hometownschools.org,courtney.carpenito@hometownschools.org,jacqueline.carton@hometownschools.org,tara.coffey@hometownschools.org,thomas.conaty@hometownschools.org,joseph.davenport@hometownschools.org,jennifer.disabatino@hometownschools.org,robert.doran@hometownschools.org,terri.dsida@hometownschools.org,paula.eramo@hometownschools.org,julie.needle@hometownschools.org,marisa.oconnell@hometownschools.org,christopher.oneil@hometownschools.org}

Now I'd like to make one List with both username and email address. Here is a typical use of Thread, where using the abbreviated operator for List, instead of Thread[List[userNames2,emailAddresses]] makes the code simple to understand.

In[233]:= namesAndEmails=Thread[{userNames2,emailAddresses}]

Out[233]= {{dbazan,dorothy.bazan@hometownschools.org},{tbleakney,tom.bleakney@hometownschools.org},{mcalorio,marie.calorio@hometownschools.org},{bcardoso,brian.cardoso@hometownschools.org},{ccarpenito,courtney.carpenito@hometownschools.org},{jcarton,jacqueline.carton@hometownschools.org},{tcoffey,tara.coffey@hometownschools.org},{tconaty,thomas.conaty@hometownschools.org},{jdavenport,joseph.davenport@hometownschools.org},{jdisabatino,jennifer.disabatino@hometownschools.org},{rdoran,robert.doran@hometownschools.org},{tdsida,terri.dsida@hometownschools.org},{peramo,paula.eramo@hometownschools.org},{jneedle,julie.needle@hometownschools.org},{moconnell,marisa.oconnell@hometownschools.org},{coneil,christopher.oneil@hometownschools.org}}

Finally I'll Export the List to a file in my Documents directory. It is safest to use FileNameJoin to compose a filename, since it will catch syntax errors as well as make sure it's in the correct format for your operating system.

In[234]:= Export[FileNameJoin@{$UserDocumentsDirectory,"HomeTownUserNames.xlsx"}, namesAndEmails]

Out[234]= C:\Users\kwcarlso\Documents\HomeTownUserNames.xlsx

Sunday, March 9, 2014

Example of a File of Functions to Pre-Load at Startup

Here is a portion of my file of functions that I load on startup so they're available in every session. See also Load Functions at Startup Using Initialization Files.

Functions to Pre-Load


String Processing

Cleaners

trimWhiteSpace::usage="trimWhiteSpace is a simple stringtrim function that does a better job than StringTrim since it removes all WhiteSpaceCharacters from beginning, end or middle of Strings. It is Listable and is intended to Map over all fields in all records where all fields are Strings, no matter if they are wrapped in Lists at any level."

Clear@trimWhiteSpace;
SetAttributes[trimWhiteSpace,Listable];
trimWhiteSpace@stringFile_:=StringReplace[stringFile,WhitespaceCharacter->""];

cleanseText::usage="cleanseText is a moderately thorough stringtrim function. The regular expression removes whitespace characters from the beginning and end of the string, including non-breaking spaces. StringReplace removes apostrophes and commas. It is Listable and intended to automatically Map over all fields in all records where all fields are Strings, no matter if they are wrapped in Lists at any level. These rules should be expanded to include more data cleansing functions as needed."

Clear@cleanseText;
SetAttributes[cleanseText,Listable];
cleanseText@text_String:=StringReplace[text,RegularExpression["^(\\s)*|(\\s)*$"]->""]//StringReplace[#,{"'"->"",WhitespaceCharacter->" ","("|")"->""}]&;

Web Page Functions


Make a URL from a String

makeURL::usage="makeURL takes a String to use as the prefix of the URL, cleans it up (removes whitespace and non-breaking spaces (which are often invisible), puts the String in lower case, replaces single spaces with hyphens and finally appends '.html'. Example: 'My Home Page' -> my-home-page.html".

Clear@makeURL;
SetAttributes[makeURL,Listable];
makeURL@urlName_String:=ToLowerCase@cleanseText@urlName//StringReplace[#,Whitespace->"-"]&//StringReplace[#,"---"->"-"]&//FileNameJoin[{#<>".html"}]&;