Rebase.Data.Text

O(n) Group characters in a string by equality.

A space efficient, packed, unboxed Unicode text type.

O(n) all p t determines whether all characters in the Text t satisfy the predicate p.

O(n) any p t determines whether any character in the Text t satisfies the predicate p.

O(n) Concatenate a list of Texts.

O(n) Map a function over a Text that results in a Text, and concatenate the results.

O(n) The elem function takes a character and a Text, and returns True if the element is found in the given Text, or False otherwise.

O(n) foldl, applied to a binary operator, a starting value (typically the left-identity of the operator), and a Text, reduces the Text using the binary operator, from left to right.

O(n) A variant of foldl that has no starting value argument, and thus must be applied to a non-empty Text.

O(n) foldr, applied to a binary operator, a starting value (typically the right-identity of the operator), and a Text, reduces the Text using the binary operator, from right to left.

If the binary operator is strict in its second argument, use foldr' instead.

foldr is lazy like foldr for lists: evaluation actually traverses the Text from left to right, only as far as it needs to.

For example, head can be defined with O(1) complexity using foldr:

head :: Text -> Char
head = foldr const (error "head empty")

Searches from left to right with short-circuiting behavior can also be defined using foldr (e.g., any, all, find, elem).

O(n) A variant of foldr that has no starting value argument, and thus must be applied to a non-empty Text.

O(n) maximum returns the maximum value from a Text, which must be non-empty.

O(n) minimum returns the minimum value from a Text, which must be non-empty.

O(n) The find function takes a predicate and a Text, and returns the first element matching the predicate, or Nothing if there is no such element.

O(n) A strict version of foldl.

O(n) Like a combination of map and foldl'. Applies a function to each element of a Text, passing an accumulating parameter from left to right, and returns a final Text. Performs replacement on invalid scalar values.

The mapAccumR function behaves like a combination of map and a strict foldr; it applies a function to each element of a Text, passing an accumulating parameter from right to left, and returning a final value of this accumulator together with the new Text. Performs replacement on invalid scalar values.

O(1) The empty Text.

O(1) Tests whether a Text is empty or not.

O(n) Returns the number of characters in a Text.

O(n) map f t is the Text obtained by applying f to each element of t.

Example:

>>> let message = pack "I am not angry. Not at all.">>> T.map (\c -> if c == '.' then '!' else c) message"I am not angry! Not at all!"

Performs replacement on invalid scalar values.

O(n) Convert a Text into a String.

O(n) Convert a null-terminated modified UTF-8 (but with a standard UTF-8 representation of characters from supplementary planes) string to a Text. Counterpart to unpackCStringUtf8#. No validation is performed, malformed input can lead to memory access violation.

O(1) Returns all but the last character of a Text, which must be non-empty. This is a partial function, consider using unsnoc instead.

O(n*m) replicate n t is a Text consisting of the input t repeated n times.

O(n) splitAt n t returns a pair whose first element is a prefix of t of length n, and whose second is the remainder of the string. It is equivalent to (take n t, drop n t).

O(n) take n, applied to a Text, returns the prefix of the Text of length n, or the Text itself if n is greater than the length of the Text.

Convert a value to Text.

O(n) takeWhile, applied to a predicate p and a Text, returns the longest prefix (possibly empty) of elements that satisfy p.

O(n) zip takes two Texts and returns a list of corresponding pairs of bytes. If one input Text is short, excess elements of the longer Text are discarded. This is equivalent to a pair of unpack operations.

O(n) filter, applied to a predicate and a Text, returns a Text containing those characters that satisfy the predicate.

O(n) A strict version of foldr.

foldr' evaluates as a right-to-left traversal using constant stack space.

O(n) Text index (subscript) operator, starting from 0.

O(n) zipWith generalises zip by zipping with the function given as the first argument, instead of a tupling function. Performs replacement on invalid scalar values.

O(n) The intercalate function takes a Text and a list of Texts and concatenates the list after interspersing the first argument between each element of the list.

Example:

>>> T.intercalate "NI!" ["We", "seek", "the", "Holy", "Grail"]"WeNI!seekNI!theNI!HolyNI!Grail"

O(n) drop n, applied to a Text, returns the suffix of the Text after the first n characters, or the empty Text if n is greater than the length of the Text.

O(1) Returns the first character of a Text, which must be non-empty. This is a partial function, consider using uncons instead.

O(n) span, applied to a predicate p and text t, returns a pair whose first element is the longest prefix (possibly empty) of t of elements that satisfy p, and whose second is the remainder of the text.

>>> T.span (=='0') "000AB"("000","AB")

Type synonym for the strict flavour of Text.

O(n) Appends one Text to the other by copying both of them into a new Text.

O(n) Convert a String into a Text. Performs replacement on invalid scalar values, so unpack . pack is not id:

>>> Data.Text.unpack (pack "\55555")"\65533"

O(n) The findIndex function takes a predicate and a Text and returns the index of the first element in the Text satisfying the predicate.

O(n) Like unfoldr, unfoldrN builds a Text from a seed value. However, the length of the result should be limited by the first argument to unfoldrN. This function is more efficient than unfoldr when the maximum length of the result is known and correct, otherwise its performance is similar to unfoldr. Performs replacement on invalid scalar values.

O(n) Adds a character to the front of a Text. This function is more costly than its List counterpart because it requires copying a new array. Performs replacement on invalid scalar values.

O(n) A strict version of foldl1.

O(n) A monadic version of foldl'.

O(n) The isPrefixOf function takes two Texts and returns True if and only if the first is a prefix of the second.

O(n) scanl is similar to foldl, but returns a list of successive reduced values from the left. Performs replacement on invalid scalar values.

scanl f z [x1, x2, ...] == [z, z `f` x1, (z `f` x1) `f` x2, ...]

Properties

head (scanl f z xs) = z

last (scanl f z xs) = foldl f z xs

O(n) Adds a character to the end of a Text. This copies the entire array in the process. Performs replacement on invalid scalar values.

O(n), where n is the length of the result. The unfoldr function is analogous to the List unfoldr. unfoldr builds a Text from a seed value. The function takes the element and returns Nothing if it is done producing the Text, otherwise Just (a,b). In this case, a is the next Char in the string, and b is the seed value for further production. Performs replacement on invalid scalar values.

O(n) Test whether Text contains only ASCII code-points (i.e. only U+0000 through U+007F).

This is a more efficient version of all isAscii.

>>> isAscii ""True

>>> isAscii "abc\NUL"True

>>> isAscii "abcd€"False

prop> isAscii t == all (< '\x80') t

O(1) Convert a character into a Text. Performs replacement on invalid scalar values.

O(n) Convert a null-terminated ASCII string to a Text. Counterpart to unpackCString#. No validation is performed, malformed input can lead to memory access violation.

Bidirectional pattern synonym for cons (O(n)) and uncons (O(1)), to be used together with Empty.

Bidirectional pattern synonym for snoc (O(n)) and unsnoc (O(1)) to be used together with Empty.

O(n+m) The count function returns the number of times the query string appears in the given Text. An empty query string is invalid, and will cause an error to be raised.

In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).

O(n) The intersperse function takes a character and places it between the characters of a Text.

Example:

>>> T.intersperse '.' "SHIELD""S.H.I.E.L.D"

Performs replacement on invalid scalar values.

O(m+n) Break a Text into pieces separated by the first Text argument (which cannot be empty), consuming the delimiter. An empty delimiter is invalid, and will cause an error to be raised.

Examples:

>>> splitOn "\r\n" "a\r\nb\r\nd\r\ne"["a","b","d","e"]

>>> splitOn "aaa"  "aaaXaaaXaaaXaaa"["","X","X","X",""]

>>> splitOn "x"    "x"["",""]

and

intercalate s . splitOn s         == id
splitOn (singleton c)             == split (==c)

(Note: the string s to split on above cannot be empty.)

In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).

Bidirectional pattern synonym for empty and null (both O(1)), to be used together with (:<) or (:>).

O(n) break is like span, but the prefix returned is over elements that fail the predicate p.

>>> T.break (=='c') "180cm"("180","cm")

O(n+m) Find the first instance of needle (which must be non-null) in haystack. The first element of the returned tuple is the prefix of haystack before needle is matched. The second is the remainder of haystack, starting with the match.

Examples:

>>> breakOn "::" "a::b::c"("a","::b::c")

>>> breakOn "/" "foobar"("foobar","")

Laws:

append prefix match == haystack
  where (prefix, match) = breakOn needle haystack

If you need to break a string by a substring repeatedly (e.g. you want to break on every instance of a substring), use breakOnAll instead, as it has lower startup overhead.

In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).

O(n+m) Find all non-overlapping instances of needle in haystack. Each element of the returned list consists of a pair:

• The entire string prior to the kth match (i.e. the prefix)

• The kth match, followed by the remainder of the string

Examples:

>>> breakOnAll "::" ""[]

>>> breakOnAll "/" "a/b/c/"[("a","/b/c/"),("a/b","/c/"),("a/b/c","/")]

In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).

The needle parameter may not be empty.

O(n+m) Similar to breakOn, but searches from the end of the string.

The first element of the returned tuple is the prefix of haystack up to and including the last match of needle. The second is the remainder of haystack, following the match.

>>> breakOnEnd "::" "a::b::c"("a::b::","c")

O(n) Center a string to the given length, using the specified fill character on either side. Performs replacement on invalid scalar values.

Examples:

>>> center 8 'x' "HS""xxxHSxxx"

O(n) Splits a Text into components of length k. The last element may be shorter than the other chunks, depending on the length of the input. Examples:

>>> chunksOf 3 "foobarbaz"["foo","bar","baz"]

>>> chunksOf 4 "haskell.org"["hask","ell.","org"]

O(n) Find the longest non-empty common prefix of two strings and return it, along with the suffixes of each string at which they no longer match.

If the strings do not have a common prefix or either one is empty, this function returns Nothing.

Examples:

>>> commonPrefixes "foobar" "fooquux"Just ("foo","bar","quux")

>>> commonPrefixes "veeble" "fetzer"Nothing

>>> commonPrefixes "" "baz"Nothing

O(min(n,c)) Compare the count of characters in a Text to a number.

compareLength t c = compare (length t) c

This function gives the same answer as comparing against the result of length, but can short circuit if the count of characters is greater than the number, and hence be more efficient.

O(n) Make a distinct copy of the given string, sharing no storage with the original string.

As an example, suppose you read a large string, of which you need only a small portion. If you do not use copy, the entire original array will be kept alive in memory by the smaller string. Making a copy "breaks the link" to the original array, allowing it to be garbage collected if there are no other live references to it.

O(n) dropAround p t returns the substring remaining after dropping characters that satisfy the predicate p from both the beginning and end of t.

O(n) dropEnd n t returns the prefix remaining after dropping n characters from the end of t.

Examples:

>>> dropEnd 3 "foobar""foo"

O(n) dropWhile p t returns the suffix remaining after takeWhile p t.

O(n) dropWhileEnd p t returns the prefix remaining after dropping characters that satisfy the predicate p from the end of t.

Examples:

>>> dropWhileEnd (=='.') "foo...""foo"

O(n) Group characters in a string according to a predicate.

O(n) Return all initial segments of the given Text, shortest first.

O(n) Return all initial segments of the given Text, shortest first.

O(n+m) The isInfixOf function takes two Texts and returns True if and only if the first is contained, wholly and intact, anywhere within the second.

In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).

O(n) The isSuffixOf function takes two Texts and returns True if and only if the first is a suffix of the second.

O(n) Left-justify a string to the given length, using the specified fill character on the right. Performs replacement on invalid scalar values.

Examples:

>>> justifyLeft 7 'x' "foo""fooxxxx"

>>> justifyLeft 3 'x' "foobar""foobar"

O(n) Right-justify a string to the given length, using the specified fill character on the left. Performs replacement on invalid scalar values.

Examples:

>>> justifyRight 7 'x' "bar""xxxxbar"

>>> justifyRight 3 'x' "foobar""foobar"

O(1) Returns the last character of a Text, which must be non-empty. This is a partial function, consider using unsnoc instead.

O(n) Breaks a Text up into a list of Texts at newline characters '\n' (LF, line feed). The resulting strings do not contain newlines.

lines does not treat '\r' (CR, carriage return) as a newline character.

O(n) If t is long enough to contain n characters, measureOff n t returns a non-negative number, measuring their size in Word8. Otherwise, if t is shorter, return a non-positive number, which is a negated total count of Char available in t. If t is empty or n = 0, return 0.

This function is used to implement take, drop, splitAt and length and is useful on its own in streaming and parsing libraries.

O(n) The partition function takes a predicate and a Text, and returns the pair of Texts with elements which do and do not satisfy the predicate, respectively; i.e.

partition p t == (filter p t, filter (not . p) t)

O(m+n) Replace every non-overlapping occurrence of needle in haystack with replacement.

This function behaves as though it was defined as follows:

replace needle replacement haystack =
  intercalate replacement (splitOn needle haystack)

As this suggests, each occurrence is replaced exactly once. So if needle occurs in replacement, that occurrence will not itself be replaced recursively:

>>> replace "oo" "foo" "oo""foo"

In cases where several instances of needle overlap, only the first one will be replaced:

>>> replace "ofo" "bar" "ofofo""barfo"

In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).

O(n) scanl1 is a variant of scanl that has no starting value argument. Performs replacement on invalid scalar values.

scanl1 f [x1, x2, ...] == [x1, x1 `f` x2, ...]

O(n) scanr is the right-to-left dual of scanl. Performs replacement on invalid scalar values.

scanr f v == reverse . scanl (flip f) v . reverse

O(n) scanr1 is a variant of scanr that has no starting value argument. Performs replacement on invalid scalar values.

O(length of suffix) spanEndM, applied to a monadic predicate p, a text t, returns a pair (t1, t2) where t2 is the longest suffix of t whose elements satisfy p, and t1 is the remainder of the text.

>>> T.spanEndM (\c -> state $ \i -> (fromEnum c == i, i-1)) "tuvxyz" `runState` 122(("tuv","xyz"),118)

spanEndM p . reverse = fmap (bimap reverse reverse) . spanM p

O(length of prefix) spanM, applied to a monadic predicate p, a text t, returns a pair (t1, t2) where t1 is the longest prefix of t whose elements satisfy p, and t2 is the remainder of the text.

>>> T.spanM (\c -> state $ \i -> (fromEnum c == i, i+1)) "abcefg" `runState` 97(("abc","efg"),101)

span is spanM specialized to Identity:

-- for all p :: Char -> Bool
span p = runIdentity . spanM (pure . p)

O(n) Splits a Text into components delimited by separators, where the predicate returns True for a separator element. The resulting components do not contain the separators. Two adjacent separators result in an empty component in the output. eg.

>>> split (=='a') "aabbaca"["","","bb","c",""]

>>> split (=='a') ""[""]

O(n) Remove leading and trailing white space from a string. Equivalent to:

dropAround isSpace

O(n) Remove trailing white space from a string. Equivalent to:

dropWhileEnd isSpace

O(n) Return the suffix of the second string if its prefix matches the entire first string.

Examples:

>>> stripPrefix "foo" "foobar"Just "bar"

>>> stripPrefix ""    "baz"Just "baz"

>>> stripPrefix "foo" "quux"Nothing

This is particularly useful with the ViewPatterns extension to GHC, as follows:

{-# LANGUAGE ViewPatterns #-}
import Data.Text as T

fnordLength :: Text -> Int
fnordLength (stripPrefix "fnord" -> Just suf) = T.length suf
fnordLength _                                 = -1

O(n) Remove leading white space from a string. Equivalent to:

dropWhile isSpace

O(n) Return the prefix of the second string if its suffix matches the entire first string.

Examples:

>>> stripSuffix "bar" "foobar"Just "foo"

>>> stripSuffix ""    "baz"Just "baz"

>>> stripSuffix "foo" "quux"Nothing

This is particularly useful with the ViewPatterns extension to GHC, as follows:

{-# LANGUAGE ViewPatterns #-}
import Data.Text as T

quuxLength :: Text -> Int
quuxLength (stripSuffix "quux" -> Just pre) = T.length pre
quuxLength _                                = -1

O(1) Returns all characters after the head of a Text, which must be non-empty. This is a partial function, consider using uncons instead.

O(n) Return all final segments of the given Text, longest first.

O(n) Return all final segments of the given Text, longest first.

O(n) takeEnd n t returns the suffix remaining after taking n characters from the end of t.

Examples:

>>> takeEnd 3 "foobar""bar"

O(n) takeWhileEnd, applied to a predicate p and a Text, returns the longest suffix (possibly empty) of elements that satisfy p. Examples:

>>> takeWhileEnd (=='o') "foo""oo"

O(n) Convert a string to folded case.

This function is mainly useful for performing caseless (also known as case insensitive) string comparisons.

A string x is a caseless match for a string y if and only if:

toCaseFold x == toCaseFold y

The result string may be longer than the input string, and may differ from applying toLower to the input string. For instance, the Armenian small ligature "ﬓ" (men now, U+FB13) is case folded to the sequence "մ" (men, U+0574) followed by "ն" (now, U+0576), while the Greek "µ" (micro sign, U+00B5) is case folded to "μ" (small letter mu, U+03BC) instead of itself.

O(n) Convert a string to lower case, using simple case conversion.

The result string may be longer than the input string. For instance, "İ" (Latin capital letter I with dot above, U+0130) maps to the sequence "i" (Latin small letter i, U+0069) followed by " ̇" (combining dot above, U+0307).

O(n) Convert a string to title case, using simple case conversion.

The first letter (as determined by isLetter) of the input is converted to title case, as is every subsequent letter that immediately follows a non-letter. Every letter that immediately follows another letter is converted to lower case.

This function is not idempotent. Consider lower-case letter ʼn (U+0149 LATIN SMALL LETTER N PRECEDED BY APOSTROPHE). Then T.toTitle "ʼn" = "ʼN": the first (and the only) letter of the input is converted to title case, becoming two letters. Now ʼ (U+02BC MODIFIER LETTER APOSTROPHE) is a modifier letter and as such is recognised as a letter by isLetter, so T.toTitle "ʼN" = "'n".

The result string may be longer than the input string. For example, the Latin small ligature fl (U+FB02) is converted to the sequence Latin capital letter F (U+0046) followed by Latin small letter l (U+006C).

Note: this function does not take language or culture specific rules into account. For instance, in English, different style guides disagree on whether the book name "The Hill of the Red Fox" is correctly title cased—but this function will capitalize every word.

O(n) Convert a string to upper case, using simple case conversion.

The result string may be longer than the input string. For instance, the German "ß" (eszett, U+00DF) maps to the two-letter sequence "SS".

O(n) The transpose function transposes the rows and columns of its Text argument. Note that this function uses pack, unpack, and the list version of transpose, and is thus not very efficient.

Examples:

>>> transpose ["green","orange"]["go","rr","ea","en","ng","e"]

>>> transpose ["blue","red"]["br","le","ud","e"]

O(1) Returns the first character and rest of a Text, or Nothing if empty.

O(n) Joins lines, after appending a terminating newline to each.

O(1) Returns all but the last character and the last character of a Text, or Nothing if empty.

O(n) Joins words using single space characters.

O(n) Breaks a Text up into a list of words, delimited by Chars representing white space.

Write a string to a file. The file is truncated to zero length before writing begins.

Write a string to the end of a file.

Read all user input on stdin as a single string.

Read a single line of user input from stdin.

Read the remaining contents of a Handle as a string. The Handle is closed once the contents have been read, or if an exception is thrown.

Internally, this function reads a chunk at a time from the lower-level buffering abstraction, and concatenates the chunks into a single string once the entire file has been read.

As a result, it requires approximately twice as much memory as its result to construct its result. For files more than a half of available RAM in size, this may result in memory exhaustion.

Read a single line from a handle.

Write a string to a handle.

The interact function takes a function of type Text -> Text as its argument. The entire input from the standard input device is passed to this function as its argument, and the resulting string is output on the standard output device.

Write a string to stdout.

The readFile function reads a file and returns the contents of the file as a string. The entire file is read strictly, as with getContents.

Beware that this function (similarly to readFile) is locale-dependent. Unexpected system locale may cause your application to read corrupted data or throw runtime exceptions about "invalid argument (invalid byte sequence)" or "invalid argument (invalid character)". This is also slow, because GHC first converts an entire input to UTF-32, which is afterwards converted to UTF-8.

If your data is UTF-8, using decodeUtf8 . readFile is a much faster and safer alternative.

Write a string to a handle, followed by a newline.

Write a string to stdout, followed by a newline.

Experimental. Read a single chunk of strict text from a Handle. The size of the chunk depends on the amount of input currently buffered.

This function blocks only if there is no data available, and EOF has not yet been reached. Once EOF is reached, this function returns an empty string instead of throwing an exception.

A stream oriented decoding result.

Decode a ByteString containing UTF-8 encoded text.

If the input contains any invalid UTF-8 data, the relevant exception will be returned, otherwise the decoded text.

Encode text to a ByteString Builder using UTF-8 encoding.

Decode a ByteString containing UTF-8 encoded text.

Surrogate code points in replacement character returned by OnDecodeError will be automatically remapped to the replacement char U+FFFD.

Decode a ByteString containing UTF-8 encoded text that is known to be valid.

If the input contains any invalid UTF-8 data, an exception will be thrown that cannot be caught in pure code. For more control over the handling of invalid data, use decodeUtf8' or decodeUtf8With.

This is a partial function: it checks that input is a well-formed UTF-8 sequence and copies buffer or throws an error otherwise.

Encode text using UTF-8 encoding.

Decode a ByteString containing 7-bit ASCII encoded text.

This is a partial function: it checks that input does not contain anything except ASCII and copies buffer or throws an error otherwise.

Decode a ByteString containing Latin-1 (aka ISO-8859-1) encoded text.

decodeLatin1 is semantically equivalent to Data.Text.pack . Data.ByteString.Char8.unpack

This is a total function. However, bear in mind that decoding Latin-1 (non-ASCII) characters to UTf-8 requires actual work and is not just buffer copying.

Encode text using big endian UTF-16 encoding.

Encode text using little endian UTF-16 encoding.

Encode text using big endian UTF-32 encoding.

Encode text using little endian UTF-32 encoding.

Encode text using UTF-8 encoding and escape the ASCII characters using a BoundedPrim.

Use this function is to implement efficient encoders for text-based formats like JSON or HTML.

Initial Utf8State.

Use StrictBuilder to build Text.

A delayed representation of strict Text.

Decode text from big endian UTF-16 encoding.

If the input contains any invalid big endian UTF-16 data, an exception will be thrown. For more control over the handling of invalid data, use decodeUtf16BEWith.

Decode text from big endian UTF-16 encoding.

Decode text from little endian UTF-16 encoding.

If the input contains any invalid little endian UTF-16 data, an exception will be thrown. For more control over the handling of invalid data, use decodeUtf16LEWith.

Decode text from little endian UTF-16 encoding.

Decode text from big endian UTF-32 encoding.

If the input contains any invalid big endian UTF-32 data, an exception will be thrown. For more control over the handling of invalid data, use decodeUtf32BEWith.

Decode text from big endian UTF-32 encoding.

Decode text from little endian UTF-32 encoding.

If the input contains any invalid little endian UTF-32 data, an exception will be thrown. For more control over the handling of invalid data, use decodeUtf32LEWith.

Decode text from little endian UTF-32 encoding.

State of decoding a ByteString in UTF-8. Enables incremental decoding (validateUtf8Chunk, validateUtf8More, decodeUtf8Chunk, decodeUtf8More).

Decode a chunk of UTF-8 text. To be continued with decodeUtf8More.

See decodeUtf8More for details on the result.

Properties

decodeUtf8Chunk = decodeUtf8More startUtf8State

Given:

decodeUtf8Chunk chunk = (builder, rest, ms)

builder is a prefix and rest is a suffix of chunk.

encodeUtf8 (Data.Text.Encoding.strictBuilderToText builder) <> rest = chunk

Decode another chunk in an ongoing UTF-8 stream.

Returns a triple:

1. A StrictBuilder for the decoded chunk of text. You can accumulate chunks with (<>) or output them with toText.

2. The undecoded remainder of the given chunk, for diagnosing errors and resuming (presumably after skipping some bytes).

3. Just the new state, or Nothing if an invalid byte was encountered (it will be within the first 4 bytes of the undecoded remainder).

Properties

Given:

(pre, suf, ms) = decodeUtf8More s chunk

1. If the output pre is nonempty (alternatively, if length chunk > length suf)

   s2b pre `append` suf = p2b s `append` chunk
   

   where

   s2b = encodeUtf8 . toText
   p2b = partUtf8ToByteString
   

2. If the output pre is empty (alternatively, if length chunk = length suf)

   suf = chunk

3. Decoding chunks separately is equivalent to decoding their concatenation.

   Given:

   (pre1, suf1, Just s1) = decodeUtf8More s chunk1
   (pre2, suf2,     ms2) = decodeUtf8More s1 chunk2
   (pre3, suf3,     ms3) = decodeUtf8More s (chunk1 `B.append` chunk2)
   

   we have:

   s2b (pre1 <> pre2) = s2b pre3
   ms2 = ms3
   

Copy Text in a StrictBuilder

Validate a ByteString as UTF-8-encoded text. To be continued using validateUtf8More.

See also validateUtf8More for details on the result of this function.

validateUtf8Chunk = validateUtf8More startUtf8State

Properties

Given:

validateUtf8Chunk chunk = (n, ms)

• The prefix is valid UTF-8. In particular, it should be accepted by this validation:

  validateUtf8Chunk (take n chunk) = (n, Just startUtf8State)
  

Validate another ByteString chunk in an ongoing stream of UTF-8-encoded text.

Returns a pair:

1. The first component n is the end position, relative to the current chunk, of the longest prefix of the accumulated bytestring which is valid UTF-8. n may be negative: that happens when an incomplete code point started in a previous chunk and is not completed by the current chunk (either that code point is still incomplete, or it is broken by an invalid byte).

2. The second component ms indicates the following:

   • if ms = Nothing, the remainder of the chunk contains an invalid byte, within four bytes from position n;

   • if ms = Just s', you can carry on validating another chunk by calling validateUtf8More with the new state s'.

Properties

Given:

validateUtf8More s chunk = (n, ms)

• If the chunk is invalid, it cannot be extended to be valid.

  ms = Nothing
  ==> validateUtf8More s (chunk <> more) = (n, Nothing)
  

• Validating two chunks sequentially is the same as validating them together at once:

  ms = Just s'
  ==> validateUtf8More s (chunk <> more) = first (length chunk +) (validateUtf8More s' more)
  

A delayed representation of strict Text.

Decode a ByteString containing 7-bit ASCII encoded text.

This is a total function which returns either the ByteString converted to a Text containing ASCII text, or Nothing.

Use decodeASCIIPrefix to retain the longest ASCII prefix for an invalid input instead of discarding it.

Decode a ByteString containing ASCII text.

This is a total function which returns a pair of the longest ASCII prefix as Text, and the remaining suffix as ByteString.

Important note: the pair is lazy. This lets you check for errors by testing whether the second component is empty, without forcing the first component (which does a copy). To drop references to the input bytestring, force the prefix (using seq or BangPatterns) and drop references to the suffix.

Properties

• If (prefix, suffix) = decodeAsciiPrefix s, then encodeUtf8 prefix <> suffix = s.

• Either suffix is empty, or head suffix > 127.

Decode a ByteString containing UTF-8 encoded text.

Any invalid input bytes will be replaced with the Unicode replacement character U+FFFD.

Decode, in a stream oriented way, a ByteString containing UTF-8 encoded text that is known to be valid.

If the input contains any invalid UTF-8 data, an exception will be thrown (either by this function or a continuation) that cannot be caught in pure code. For more control over the handling of invalid data, use streamDecodeUtf8With.

Decode, in a stream oriented way, a lazy ByteString containing UTF-8 encoded text.