Streaming.Linear.Internal.Process

The standard way of inspecting the first item in a stream of elements, if the stream is still 'running'. The Right case contains a Haskell pair, where the more general inspect would return a left-strict pair. There is no reason to prefer inspect since, if the Right case is exposed, the first element in the pair will have been evaluated to whnf.

next    :: Control.Monad m => Stream (Of a) m r %1-> m (Either r    (a, Stream (Of a) m r))
inspect :: Control.Monad m => Stream (Of a) m r %1-> m (Either r (Of a (Stream (Of a) m r)))

Inspect the first item in a stream of elements, without a return value.

Split a succession of layers after some number, returning a streaming or effectful pair. This function is the same as the splitsAt exported by the Streaming module, but since this module is imported qualified, it can usurp a Prelude name. It specializes to:

 splitAt :: Control.Monad m => Int -> Stream (Of a) m r %1-> Stream (Of a) m (Stream (Of a) m r)

Split a stream of elements wherever a given element arises. The action is like that of words.

>>> S.stdoutLn $ mapped S.toList $ S.split ' ' $ each' "hello world  "
hello
world

Break during periods where the predicate is not satisfied, grouping the periods when it is.

>>> S.print $ mapped S.toList $ S.breaks not $ S.each' [False,True,True,False,True,True,False]
[True,True]
[True,True]
>>> S.print $ mapped S.toList $ S.breaks id $ S.each' [False,True,True,False,True,True,False]
[False]
[False]
[False]

Break a sequence upon meeting an element that falls under a predicate, keeping it and the rest of the stream as the return value.

>>> rest <- S.print $ S.break even $ each' [1,1,2,3]
1
1
>>> S.print rest
2
3

Yield elements, using a fold to maintain state, until the accumulated value satifies the supplied predicate. The fold will then be short-circuited and the element that breaks it will be put after the break. This function is easiest to use with purely

>>> rest each' [1..10] & L.purely S.breakWhen L.sum (10) & S.print
1
2
3
4
>>> S.print rest
5
6
7
8
9
10

Breaks on the first element to satisfy the predicate

Stream elements until one fails the condition, return the rest.

Group successive equal items together

>>> S.toList $ mapped S.toList $ S.group $ each' "baaaaad"
["b","aaaaa","d"] :> ()

>>> S.toList $ concats $ maps (S.drained . S.splitAt 1) $ S.group $ each' "baaaaaaad"
"bad" :> ()

Group elements of a stream in accordance with the supplied comparison.

>>> S.print $ mapped S.toList $ S.groupBy (>=) $ each' [1,2,3,1,2,3,4,3,2,4,5,6,7,6,5]
[1]
[2]
[3,1,2,3]
[4,3,2,4]
[5]
[6]
[7,6,5]

Swap the order of functors in a sum of functors.

>>> S.toList $ S.print $ separate $ maps S.switch $ maps (S.distinguish (==a)) $ S.each' "banana"
a
a
a
"bnn" :> ()
>>> S.toList $ S.print $ separate $ maps (S.distinguish (==a)) $ S.each' "banana"
b
n
n
"aaa" :> ()

Given a stream on a sum of functors, make it a stream on the left functor, with the streaming on the other functor as the governing monad. This is useful for acting on one or the other functor with a fold, leaving the other material for another treatment. It generalizes partitionEithers, but actually streams properly.

>>> let odd_even = S.maps (S.distinguish even) $ S.each' [1..10::Int]
>>> :t separate odd_even
separate odd_even
 :: Monad m => Stream (Of Int) (Stream (Of Int) m) ()

Now, for example, it is convenient to fold on the left and right values separately:

>>> S.toList $ S.toList $ separate odd_even
[2,4,6,8,10] :> ([1,3,5,7,9] :> ())

Or we can write them to separate files or whatever.

Of course, in the special case of Stream (Of a) m r, we can achieve the above effects more simply by using Streaming.Prelude.copy

>>> S.toList . S.filter even $ S.toList . S.filter odd $ S.copy $ each' [1..10::Int]
[2,4,6,8,10] :> ([1,3,5,7,9] :> ())

But separate and unseparate are functor-general.

Separate left and right values in distinct streams. (separate is a more powerful, functor-general, equivalent using Sum in place of Either).

partitionEithers = separate . maps S.eitherToSum
lefts  = hoist S.effects . partitionEithers
rights = S.effects . partitionEithers
rights = S.concat

The catMaybes function takes a Stream of Maybes and returns a Stream of all of the Just values. concat has the same behavior, but is more general; it works for any foldable container type.

The mapMaybe function is a version of map which can throw out elements. In particular, the functional argument returns something of type Maybe b. If this is Nothing, no element is added on to the result Stream. If it is Just b, then b is included in the result Stream.

Map monadically over a stream, producing a new stream only containing the Just values.

Change the effects of one monad to another with a transformation. This is one of the fundamental transformations on streams. Compare with maps:

maps  :: (Control.Monad m, Control.Functor f) => (forall x. f x %1-> g x) -> Stream f m r %1-> Stream g m r
hoist :: (Control.Monad m, Control.Functor f) => (forall a. m a %1-> n a) -> Stream f m r %1-> Stream f n r

Standard map on the elements of a stream.

>>> S.stdoutLn $ S.map reverse $ each' (words "alpha beta")
ahpla
ateb

Replace each element of a stream with the result of a monadic action

>>> S.print $ S.mapM readIORef $ S.chain (ior -> modifyIORef ior (*100)) $ S.mapM newIORef $ each' [1..6]
100
200
300
400
500
600

See also chain for a variant of this which ignores the return value of the function and just uses the side effects.

Map layers of one functor to another with a transformation. Compare hoist, which has a similar effect on the monadic parameter.

maps id = id
maps f . maps g = maps (f . g)

Map layers of one functor to another with a transformation involving the base monad.

This function is completely functor-general. It is often useful with the more concrete type

mapped :: (forall x. Stream (Of a) IO x -> IO (Of b x)) -> Stream (Stream (Of a) IO) IO r -> Stream (Of b) IO r

to process groups which have been demarcated in an effectful, IO-based stream by grouping functions like Streaming.Prelude.group, Streaming.Prelude.split or Streaming.Prelude.breaks. Summary functions like Streaming.Prelude.fold, Streaming.Prelude.foldM, Streaming.Prelude.mconcat or Streaming.Prelude.toList are often used to define the transformation argument. For example:

>>> S.toList_ $ S.mapped S.toList $ S.split c (S.each' "abcde")
["ab","de"]

Streaming.Prelude.maps and Streaming.Prelude.mapped obey these rules:

maps id              = id
mapped return        = id
maps f . maps g      = maps (f . g)
mapped f . mapped g  = mapped (f <=< g)
maps f . mapped g    = mapped (fmap f . g)
mapped f . maps g    = mapped (f <=< fmap g)

where f and g are Control.Monads

Streaming.Prelude.maps is more fundamental than Streaming.Prelude.mapped, which is best understood as a convenience for effecting this frequent composition:

mapped phi = decompose . maps (Compose . phi)

Map layers of one functor to another with a transformation. Compare hoist, which has a similar effect on the monadic parameter.

mapsPost id = id
mapsPost f . mapsPost g = mapsPost (f . g)
mapsPost f = maps f

mapsPost is essentially the same as maps, but it imposes a Control.Functor constraint on its target functor rather than its source functor. It should be preferred if fmap is cheaper for the target functor than for the source functor.

Map layers of one functor to another with a transformation involving the base monad. mapsMPost is essentially the same as mapsM, but it imposes a Control.Functor constraint on its target functor rather than its source functor. It should be preferred if fmap is cheaper for the target functor than for the source functor.

mapsPost is more fundamental than mapsMPost, which is best understood as a convenience for effecting this frequent composition:

mapsMPost phi = decompose . mapsPost (Compose . phi)

The streaming prelude exports the same function under the better name mappedPost, which overlaps with the lens libraries.

A version of mapped that imposes a Control.Functor constraint on the target functor rather than the source functor. This version should be preferred if fmap on the target functor is cheaper.

for replaces each element of a stream with an associated stream. Note that the associated stream may layer any control functor.

Replace each element in a stream of individual Haskell values (a Stream (Of a) m r) with an associated functorial step.

for str f  = concats (with str f)
with str f = for str (yields . f)
with str f = maps (\(a:>r) -> r <$ f a) str
with = flip subst
subst = flip with

>>> with (each' [1..3]) (yield . Prelude.show) & intercalates (yield "--") & S.stdoutLn
1
--
2
--
3

Replace each element in a stream of individual values with a functorial layer of any sort. subst = flip with and is more convenient in a sequence of compositions that transform a stream.

with = flip subst
for str f = concats $ subst f str
subst f = maps (\(a:>r) -> r <$ f a)
S.concat = concats . subst each

Duplicate the content of a stream, so that it can be acted on twice in different ways, but without breaking streaming. Thus, with each' [1,2] I might do:

>>> S.print $ each' ["one","two"]
"one"
"two"
>>> S.stdoutLn $ each' ["one","two"]
one
two

With copy, I can do these simultaneously:

>>> S.print $ S.stdoutLn $ S.copy $ each' ["one","two"]
"one"
one
"two"
two

copy should be understood together with effects and is subject to the rules

S.effects . S.copy       = id
hoist S.effects . S.copy = id

The similar operations in Data.ByteString.Streaming obey the same rules.

Where the actions you are contemplating are each simple folds over the elements, or a selection of elements, then the coupling of the folds is often more straightforwardly effected with Control.Foldl, e.g.

>>> L.purely S.fold (liftA2 (,) L.sum L.product) $ each' 1..10 :> ()

rather than

>>> S.sum $ S.product . S.copy $ each' [1..10]
55 :> (3628800 :> ())

A Control.Foldl fold can be altered to act on a selection of elements by using handles on an appropriate lens. Some such manipulations are simpler and more Data.List-like, using copy:

>>> L.purely S.fold (liftA2 (,) (L.handles (L.filtered odd) L.sum) (L.handles (L.filtered even) L.product)) $ each' 1..10 :> ()

becomes

>>> S.sum $ S.filter odd $ S.product $ S.filter even $ S.copy' $ each' [1..10]
25 :> (3840 :> ())

or using store

>>> S.sum $ S.filter odd $ S.store (S.product . S.filter even) $ each' [1..10]
25 :> (3840 :> ())

But anything that fold of a Stream (Of a) m r into e.g. an m (Of b r) that has a constraint on m that is carried over into Stream f m - e.g. Control.Monad, Control.Functor, etc. can be used on the stream. Thus, I can fold over different groupings of the original stream:

>>>  (S.toList . mapped S.toList . chunksOf 5) $  (S.toList . mapped S.toList . chunksOf 3) $ S.copy $ each' [1..10]
[[1,2,3,4,5],[6,7,8,9,10]] :> ([[1,2,3],[4,5,6],[7,8,9],[10]] :> ())

The procedure can be iterated as one pleases, as one can see from this (otherwise unadvisable!) example:

>>>  (S.toList . mapped S.toList . chunksOf 4) $ (S.toList . mapped S.toList . chunksOf 3) $ S.copy $ (S.toList . mapped S.toList . chunksOf 2) $ S.copy $ each' [1..12]
[[1,2,3,4],[5,6,7,8],[9,10,11,12]] :> ([[1,2,3],[4,5,6],[7,8,9],[10,11,12]] :> ([[1,2],[3,4],[5,6],[7,8],[9,10],[11,12]] :> ()))

copy can be considered a special case of expand:

 copy = expand $ p (a :> as) -> a :> p (a :> as)

If Of were an instance of Comonad, then one could write

 copy = expand extend

An alias for copy.

Store the result of any suitable fold over a stream, keeping the stream for further manipulation. store f = f . copy :

>>> S.print $ S.store S.product $ each' [1..4]
1
2
3
4
24 :> ()

>>> S.print $ S.store S.sum $ S.store S.product $ each' [1..4]
1
2
3
4
10 :> (24 :> ())

Here the sum (10) and the product (24) have been 'stored' for use when finally we have traversed the stream with print . Needless to say, a second pass is excluded conceptually, so the folds that you apply successively with store are performed simultaneously, and in constant memory -- as they would be if, say, you linked them together with Control.Fold:

>>> L.impurely S.foldM (liftA3 (a b c -> (b, c)) (L.sink Prelude.print) (L.generalize L.sum) (L.generalize L.product)) $ each' [1..4]
1
2
3
4
(10,24) :> ()

Fusing folds after the fashion of Control.Foldl will generally be a bit faster than the corresponding succession of uses of store, but by constant factor that will be completely dwarfed when any IO is at issue.

But store / copy is much more powerful, as you can see by reflecting on uses like this:

>>> S.sum $ S.store (S.sum . mapped S.product . chunksOf 2) $ S.store (S.product . mapped S.sum . chunksOf 2) $ each' [1..6]
21 :> (44 :> (231 :> ()))

It will be clear that this cannot be reproduced with any combination of lenses, Control.Fold folds, or the like. (See also the discussion of copy.)

It would conceivably be clearer to import a series of specializations of store. It is intended to be used at types like this:

storeM ::  (forall s m . Control.Monad m => Stream (Of a) m s %1-> m (Of b s))
        -> (Control.Monad n => Stream (Of a) n r %1-> Stream (Of a) n (Of b r))
storeM = store

It is clear from this type that we are just using the general instance:

instance (Control.Functor f, Control.Monad m)   => Control.Monad (Stream f m)

We thus can't be touching the elements of the stream, or the final return value. It is the same with other constraints that Stream (Of a) inherits from the underlying monad. Thus I can independently filter and write to one file, but nub and write to another, or interact with a database and a logfile and the like:

>>> (S.writeFile "hello2.txt" . S.nubOrd) $ store (S.writeFile "hello.txt" . S.filter (/= "world")) $ each' ["hello", "world", "goodbye", "world"]
>>> :! cat hello.txt
hello
goodbye
>>> :! cat hello2.txt
hello
world
goodbye

Apply an action to all values, re-yielding each. The return value (y) of the function is ignored.

>>> S.product $ S.chain Prelude.print $ S.each' [1..5]
1
2
3
4
5
120 :> ()

See also mapM for a variant of this which uses the return value of the function to transorm the values in the stream.

Like the Data.List.sequence but streaming. The result type is a stream of a's, but is not accumulated; the effects of the elements of the original stream are interleaved in the resulting stream. Compare:

sequence :: Monad m =>         [m a]                 ->  m [a]
sequence :: Control.Monad m => Stream (Of (m a)) m r %1-> Stream (Of a) m r

Remove repeated elements from a Stream. nubOrd of course accumulates a Set of elements that have already been seen and should thus be used with care.

Use nubOrdOn to have a custom ordering function for your elements.

More efficient versions of above when working with Ints that use IntSet.

Skip elements of a stream that fail a predicate

Skip elements of a stream that fail a monadic test

Intersperse given value between each element of the stream.

>>> S.print $ S.intersperse 0 $ each [1,2,3]
1
0
2
0
3

Ignore the first n elements of a stream, but carry out the actions

>>> S.toList $ S.drop 2 $ S.replicateM 5 getLine
aEnter
bEnter
cEnter
dEnter
eEnter
["c","d","e"] :> ()

Because it retains the final return value, drop n is a suitable argument for maps:

>>> S.toList $ concats $ maps (S.drop 4) $ chunksOf 5 $ each [1..20]
[5,10,15,20] :> ()

Ignore elements of a stream until a test succeeds, retaining the rest.

>>> S.print $ S.dropWhile ((< 5) . length) S.stdinLn
oneEnter
twoEnter
threeEnter
"three"
fourEnter
"four"
^CInterrupted.

Strict left scan, streaming, e.g. successive partial results. The seed is yielded first, before any action of finding the next element is performed.

>>> S.print $ S.scan (++) "" id $ each' (words "a b c d")
""
"a"
"ab"
"abc"
"abcd"

scan is fitted for use with Control.Foldl, thus:

>>> S.print $ L.purely S.scan L.list $ each' [3..5]
[]
[3]
[3,4]
[3,4,5]

Strict left scan, accepting a monadic function. It can be used with FoldMs from Control.Foldl using impurely. Here we yield a succession of vectors each recording

>>> let v = L.impurely scanM L.vectorM $ each' [1..4::Int] :: Stream (Of (Vector Int)) IO ()
>>> S.print v
[]
[1]
[1,2]
[1,2,3]
[1,2,3,4]

Label each element in a stream with a value accumulated according to a fold.

>>> S.print $ S.scanned (*) 1 id $ S.each' 100,200,300
(200,20000)
(300,6000000)

>>> S.print $ L.purely S.scanned' L.product $ S.each 100,200,300
(200,20000)
(300,6000000)

Interpolate a delay of n seconds between yields.

Make a stream of strings into a stream of parsed values, skipping bad cases

>>> S.sum_ $ S.read $ S.takeWhile (/= "total") S.stdinLn :: IO Int
1000Enter
2000Enter
totalEnter
3000

The natural cons for a Stream (Of a).

cons a stream = yield a Control.>> stream

Useful for interoperation:

Data.Text.foldr S.cons (return ()) :: Text -> Stream (Of Char) m ()
Lazy.foldrChunks S.cons (return ()) :: Lazy.ByteString -> Stream (Of Strict.ByteString) m ()

and so on.

slidingWindow accumulates the first n elements of a stream, update thereafter to form a sliding window of length n. It follows the behavior of the slidingWindow function in conduit-combinators.

>>> S.print $ S.slidingWindow 4 $ S.each "123456"
fromList "1234"
fromList "2345"
fromList "3456"

Before evaluating the monadic action returning the next step in the Stream, wrapEffect extracts the value in a monadic computation m a and passes it to a computation a -> m y.