Streamly.Data.Fold

Semigroup concat. Append the elements of an input stream to a provided starting value.

Definition:

>>> sconcat = Fold.foldl' (<>)

>>> semigroups = fmap Data.Monoid.Sum $ Stream.enumerateFromTo 1 10>>> Stream.fold (Fold.sconcat 10) semigroupsSum {getSum = 65}

Monoid concat. Fold an input stream consisting of monoidal elements using mappend and mempty.

Definition:

>>> mconcat = Fold.sconcat mempty

>>> monoids = fmap Data.Monoid.Sum $ Stream.enumerateFromTo 1 10>>> Stream.fold Fold.mconcat monoidsSum {getSum = 55}

Definition:

>>> foldMap f = Fold.lmap f Fold.mconcat

Make a fold from a pure function that folds the output of the function using mappend and mempty.

>>> sum = Fold.foldMap Data.Monoid.Sum>>> Stream.fold sum $ Stream.enumerateFromTo 1 10Sum {getSum = 55}

Definition:

>>> foldMapM f = Fold.lmapM f Fold.mconcat

Make a fold from a monadic function that folds the output of the function using mappend and mempty.

>>> sum = Fold.foldMapM (return . Data.Monoid.Sum)>>> Stream.fold sum $ Stream.enumerateFromTo 1 10Sum {getSum = 55}

A fold that drains all its input, running the effects and discarding the results.

>>> drain = Fold.drainMapM (const (return ()))>>> drain = Fold.foldl' (\_ _ -> ()) ()

Definitions:

>>> drainMapM f = Fold.lmapM f Fold.drain>>> drainMapM f = Fold.foldMapM (void . f)

Drain all input after passing it through a monadic function. This is the dual of mapM_ on stream producers.

Determine the length of the input stream.

Definition:

>>> length = Fold.lengthGeneric>>> length = fmap getSum $ Fold.foldMap (Sum . const  1)

Count non-duplicate elements in the stream.

Definition:

>>> countDistinct = fmap Set.size Fold.toSet>>> countDistinct = Fold.postscan Fold.nub $ Fold.catMaybes $ Fold.length

The memory used is proportional to the number of distinct elements in the stream, to guard against using too much memory use it as a scan and terminate if the count reaches more than a threshold.

Space: \mathcal{O}(n)

Pre-release

Like countDistinct but specialized to a stream of Int, for better performance.

Definition:

>>> countDistinctInt = fmap IntSet.size Fold.toIntSet>>> countDistinctInt = Fold.postscan Fold.nubInt $ Fold.catMaybes $ Fold.length

Pre-release

Determine the frequency of each element in the stream.

You can just collect the keys of the resulting map to get the unique elements in the stream.

Definition:

>>> frequency = Fold.toMap id Fold.length

Determine the sum of all elements of a stream of numbers. Returns additive identity (0) when the stream is empty. Note that this is not numerically stable for floating point numbers.

>>> sum = Fold.cumulative Fold.windowSum

Same as following but numerically stable:

>>> sum = Fold.foldl' (+) 0>>> sum = fmap Data.Monoid.getSum $ Fold.foldMap Data.Monoid.Sum

Determine the product of all elements of a stream of numbers. Returns multiplicative identity (1) when the stream is empty. The fold terminates when it encounters (0) in its input.

Same as the following but terminates on multiplication by 0:

>>> product = fmap Data.Monoid.getProduct $ Fold.foldMap Data.Monoid.Product

Compute a numerically stable arithmetic mean of all elements in the input stream.

Compute an Int sized polynomial rolling hash of a stream.

>>> rollingHash = Fold.rollingHashWithSalt Fold.defaultSalt

Compute an Int sized polynomial rolling hash

H = salt * k ^ n + c1 * k ^ (n - 1) + c2 * k ^ (n - 2) + ... + cn * k ^ 0

Where c1, c2, cn are the elements in the input stream and k is a constant.

This hash is often used in Rabin-Karp string search algorithm.

See https://en.wikipedia.org/wiki/Rolling_hash

Folds the input stream to a list.

Warning! working on large lists accumulated as buffers in memory could be very inefficient, consider using Streamly.Data.Array instead.

>>> toList = Fold.foldr' (:) []

Buffers the input stream to a list in the reverse order of the input.

Definition:

>>> toListRev = Fold.foldl' (flip (:)) []

Warning! working on large lists accumulated as buffers in memory could be very inefficient, consider using Streamly.Array instead.

Fold the input to a set.

Definition:

>>> toSet = Fold.foldl' (flip Set.insert) Set.empty

Fold the input to an int set. For integer inputs this performs better than toSet.

Definition:

>>> toIntSet = Fold.foldl' (flip IntSet.insert) IntSet.empty

Get the top n elements using the supplied comparison function.

To get bottom n elements instead:

>>> bottomBy cmp = Fold.topBy (flip cmp)

Example:

>>> stream = Stream.fromList [2::Int,7,9,3,1,5,6,11,17]>>> Stream.fold (Fold.topBy compare 3) stream >>= MutArray.toList[17,11,9]

Pre-release

Returns the latest element of the input stream, if any.

>>> latest = Fold.foldl1' (\_ x -> x)>>> latest = fmap getLast $ Fold.foldMap (Last . Just)

Determine the maximum element in a stream using the supplied comparison function.

Determine the maximum element in a stream.

Definitions:

>>> maximum = Fold.maximumBy compare>>> maximum = Fold.foldl1' max

Same as the following but without a default maximum. The Max Monoid uses the minBound as the default maximum:

>>> maximum = fmap Data.Semigroup.getMax $ Fold.foldMap Data.Semigroup.Max

Computes the minimum element with respect to the given comparison function

Determine the minimum element in a stream using the supplied comparison function.

Definitions:

>>> minimum = Fold.minimumBy compare>>> minimum = Fold.foldl1' min

Same as the following but without a default minimum. The Min Monoid uses the maxBound as the default maximum:

>>> maximum = fmap Data.Semigroup.getMin $ Fold.foldMap Data.Semigroup.Min

Returns the index of the latest element if the element satisfies the given predicate.

Returns the index of the latest element if the element matches the given value.

Definition:

>>> elemIndices a = Fold.findIndices (== a)

Returns the latest element omitting the first occurrence that satisfies the given equality predicate.

Example:

>>> input = Stream.fromList [1,3,3,5]>>> Stream.fold Fold.toList $ Stream.scanMaybe (Fold.deleteBy (==) 3) input[1,3,5]

Return the latest unique element using the supplied comparison function. Returns Nothing if the current element is same as the last element otherwise returns Just.

Example, strip duplicate path separators:

>>> input = Stream.fromList "//a//b">>> f x y = x == '/' && y == '/'>>> Stream.fold Fold.toList $ Stream.scanMaybe (Fold.uniqBy f) input"/a/b"

Space: O(1)

Pre-release

Used as a scan. Returns Just for the first occurrence of an element, returns Nothing for any other occurrences.

Example:

>>> stream = Stream.fromList [1::Int,1,2,3,4,4,5,1,5,7]>>> Stream.fold Fold.toList $ Stream.scanMaybe Fold.nub stream[1,2,3,4,5,7]

Pre-release

Like nub but specialized to a stream of Int, for better performance.

Pre-release

Take one element from the stream and stop.

Definition:

>>> one = Fold.maybe Just

This is similar to the stream Stream.uncons operation.

Consume one element, return True if successful else return False. In other words, test if the input is empty or not.

WARNING! It consumes one element if the stream is not empty. If that is not what you want please use the eof parser instead.

Definition:

>>> null = fmap isJust Fold.one

Return the element at the given index.

Definition:

>>> index = Fold.indexGeneric

Terminates with Nothing as soon as it finds an element different than the previous one, returns the element if the entire input consists of the same element.

Returns the first element that satisfies the given predicate.

Returns the first element that satisfies the given predicate.

Pre-release

In a stream of (key-value) pairs (a, b), return the value b of the first pair where the key equals the given value a.

Definition:

>>> lookup x = fmap snd <$> Fold.find ((== x) . fst)

Returns the first index that satisfies the given predicate.

Returns the first index where a given value is found in the stream.

Definition:

>>> elemIndex a = Fold.findIndex (== a)

Return True if the given element is present in the stream.

Definition:

>>> elem a = Fold.any (== a)

Returns True if the given element is not present in the stream.

Definition:

>>> notElem a = Fold.all (/= a)

Returns True if all elements of the input satisfy the predicate.

Definition:

>>> all p = Fold.lmap p Fold.and

Example:

>>> Stream.fold (Fold.all (== 0)) $ Stream.fromList [1,0,1]False

Returns True if any element of the input satisfies the predicate.

Definition:

>>> any p = Fold.lmap p Fold.or

Example:

>>> Stream.fold (Fold.any (== 0)) $ Stream.fromList [1,0,1]True

Returns True if all elements are True, False otherwise

Definition:

>>> and = Fold.all (== True)

Returns True if any element is True, False otherwise

Definition:

>>> or = Fold.any (== True)

Map a monadic function on the output of a fold.

lmap f fold maps the function f on the input of the fold.

Definition:

>>> lmap = Fold.lmapM return

Example:

>>> sumSquared = Fold.lmap (\x -> x * x) Fold.sum>>> Stream.fold sumSquared (Stream.enumerateFromTo 1 100)338350

lmapM f fold maps the monadic function f on the input of the fold.

Scan the input of a Fold to change it in a stateful manner using another Fold. The scan stops as soon as the fold terminates.

Pre-release

Postscan the input of a Fold to change it in a stateful manner using another Fold.

postscan scanner collector

Pre-release

Use a Maybe returning fold as a filtering scan.

>>> scanMaybe p f = Fold.postscan p (Fold.catMaybes f)

Pre-release

Include only those elements that pass a predicate.

>>> Stream.fold (Fold.filter (> 5) Fold.sum) $ Stream.fromList [1..10]40

>>> filter p = Fold.scanMaybe (Fold.filtering p)>>> filter p = Fold.filterM (return . p)>>> filter p = Fold.mapMaybe (\x -> if p x then Just x else Nothing)

Like filter but with a monadic predicate.

>>> f p x = p x >>= \r -> return $ if r then Just x else Nothing>>> filterM p = Fold.mapMaybeM (f p)

mapMaybe f fold maps a Maybe returning function f on the input of the fold, filters out Nothing elements, and return the values extracted from Just.

>>> mapMaybe f = Fold.lmap f . Fold.catMaybes>>> mapMaybe f = Fold.mapMaybeM (return . f)

>>> f x = if even x then Just x else Nothing>>> fld = Fold.mapMaybe f Fold.toList>>> Stream.fold fld (Stream.enumerateFromTo 1 10)[2,4,6,8,10]

Modify a fold to receive a Maybe input, the Just values are unwrapped and sent to the original fold, Nothing values are discarded.

>>> catMaybes = Fold.mapMaybe id>>> catMaybes = Fold.filter isJust . Fold.lmap fromJust

Discard Rights and unwrap Lefts in an Either stream.

Pre-release

Discard Lefts and unwrap Rights in an Either stream.

Pre-release

Remove the either wrapper and flatten both lefts and as well as rights in the output stream.

Definition:

>>> catEithers = Fold.lmap (either id id)

Pre-release

Take at most n input elements and fold them using the supplied fold. A negative count is treated as 0.

>>> Stream.fold (Fold.take 2 Fold.toList) $ Stream.fromList [1..10][1,2]

Take the input, stop when the predicate succeeds taking the succeeding element as well.

Example:

>>> input = Stream.fromList "hello\nthere\n">>> line = Fold.takeEndBy (== '\n') Fold.toList>>> Stream.fold line input"hello\n"

>>> Stream.fold Fold.toList $ Stream.foldMany line input["hello\n","there\n"]

Like takeEndBy but drops the element on which the predicate succeeds.

Example:

>>> input = Stream.fromList "hello\nthere\n">>> line = Fold.takeEndBy_ (== '\n') Fold.toList>>> Stream.fold line input"hello"

>>> Stream.fold Fold.toList $ Stream.foldMany line input["hello","there"]

Sequential fold application. Apply two folds sequentially to an input stream. The input is provided to the first fold, when it is done - the remaining input is provided to the second fold. When the second fold is done or if the input stream is over, the outputs of the two folds are combined using the supplied function.

Example:

>>> header = Fold.take 8 Fold.toList>>> line = Fold.takeEndBy (== '\n') Fold.toList>>> f = Fold.splitWith (,) header line>>> Stream.fold f $ Stream.fromList "header: hello\n"("header: ","hello\n")

Note: This is dual to appending streams using Data.Stream.append.

Note: this implementation allows for stream fusion but has quadratic time complexity, because each composition adds a new branch that each subsequent fold's input element has to traverse, therefore, it cannot scale to a large number of compositions. After around 100 compositions the performance starts dipping rapidly compared to a CPS style implementation.

For larger number of compositions you can convert the fold to a parser and use ParserK.

Time: O(n^2) where n is the number of compositions.

Collect zero or more applications of a fold. many first second applies the first fold repeatedly on the input stream and accumulates it's results using the second fold.

>>> two = Fold.take 2 Fold.toList>>> twos = Fold.many two Fold.toList>>> Stream.fold twos $ Stream.fromList [1..10][[1,2],[3,4],[5,6],[7,8],[9,10]]

Stops when second fold stops.

See also: Data.Stream.concatMap, Data.Stream.foldMany

groupsOf n split collect repeatedly applies the split fold to chunks of n items in the input stream and supplies the result to the collect fold.

Definition:

>>> groupsOf n split = Fold.many (Fold.take n split)

Example:

>>> twos = Fold.groupsOf 2 Fold.toList Fold.toList>>> Stream.fold twos $ Stream.fromList [1..10][[1,2],[3,4],[5,6],[7,8],[9,10]]

Stops when collect stops.

teeWith k f1 f2 distributes its input to both f1 and f2 until both of them terminate and combines their output using k.

Definition:

>>> teeWith k f1 f2 = fmap (uncurry k) (Fold.tee f1 f2)

Example:

>>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)>>> Stream.fold avg $ Stream.fromList [1.0..100.0]50.5

For applicative composition using this combinator see Streamly.Data.Fold.Tee.

See also: Streamly.Data.Fold.Tee

Note that nested applications of teeWith do not fuse.

Distribute one copy of the stream to each fold and zip the results.

                |-------Fold m a b--------|
---stream m a---|                         |---m (b,c)
                |-------Fold m a c--------|

Definition:

>>> tee = Fold.teeWith (,)

Example:

>>> t = Fold.tee Fold.sum Fold.length>>> Stream.fold t (Stream.enumerateFromTo 1.0 100.0)(5050.0,100)

Distribute one copy of the stream to each fold and collect the results in a container.

                |-------Fold m a b--------|
---stream m a---|                         |---m [b]
                |-------Fold m a b--------|
                |                         |
                           ...

>>> Stream.fold (Fold.distribute [Fold.sum, Fold.length]) (Stream.enumerateFromTo 1 5)[15,5]

>>> distribute = Prelude.foldr (Fold.teeWith (:)) (Fold.fromPure [])

This is the consumer side dual of the producer side sequence operation.

Stops when all the folds stop.

Compose two folds such that the combined fold accepts a stream of Either and routes the Left values to the first fold and Right values to the second fold.

Definition:

>>> partition = Fold.partitionBy id

Split the input stream based on a key field and fold each split using the given fold. Useful for map/reduce, bucketizing the input in different bins or for generating histograms.

Example:

>>> import Data.Map.Strict (Map)>>> :{ let input = Stream.fromList [("ONE",1),("ONE",1.1),("TWO",2), ("TWO",2.2)]     classify = Fold.toMap fst (Fold.lmap snd Fold.toList)  in Stream.fold classify input :: IO (Map String [Double]):}fromList [("ONE",[1.0,1.1]),("TWO",[2.0,2.2])]

Once the classifier fold terminates for a particular key any further inputs in that bucket are ignored.

Space used is proportional to the number of keys seen till now and monotonically increases because it stores whether a key has been seen or not.

See demuxToMap for a more powerful version where you can use a different fold for each key. A simpler version of toMap retaining only the last value for a key can be written as:

>>> toMap = Fold.foldl' (\kv (k, v) -> Map.insert k v kv) Map.empty

Stops: never

Pre-release

Same as toMap but maybe faster because it uses mutable cells as fold accumulators in the Map.

This collects all the results of demux in a Map.

Same as demuxToMap but uses demuxIO for better performance.

Folds the values for each key using the supplied fold. When scanning, as soon as the fold is complete, its result is available in the second component of the tuple. The first component of the tuple is a snapshot of the in-progress folds.

Once the fold for a key is done, any future values of the key are ignored.

Definition:

>>> classify f fld = Fold.demux f (const fld)

Same as classify except that it uses mutable IORef cells in the Map providing better performance. Be aware that if this is used as a scan, the values in the intermediate Maps would be mutable.

Definitions:

>>> classifyIO f fld = Fold.demuxIO f (const fld)

demux getKey getFold: In a key value stream, fold values corresponding to each key using a key specific fold. getFold is invoked to generate a key specific fold when a key is encountered for the first time in the stream.

The first component of the output tuple is a key-value Map of in-progress folds. The fold returns the fold result as the second component of the output tuple whenever a fold terminates.

If a fold terminates, another instance of the fold is started upon receiving an input with that key, getFold is invoked again whenever the key is encountered again.

This can be used to scan a stream and collect the results from the scan output.

Since the fold generator function is monadic we can add folds dynamically. For example, we can maintain a Map of keys to folds in an IORef and lookup the fold from that corresponding to a key. This Map can be changed dynamically, folds for new keys can be added or folds for old keys can be deleted or modified.

Compare with classify, the fold in classify is a static fold.

Pre-release

This is specialized version of demux that uses mutable IO cells as fold accumulators for better performance.

Keep in mind that the values in the returned Map may be changed by the ongoing fold if you are using those concurrently in another thread.

Send the elements of tuples in a stream of tuples through two different folds.

                          |-------Fold m a x--------|
---------stream of (a,b)--|                         |----m (x,y)
                          |-------Fold m b y--------|

Definition:

>>> unzip = Fold.unzipWith id

This is the consumer side dual of the producer side zip operation.

Map a Fold returning function on the result of a Fold and run the returned fold. This operation can be used to express data dependencies between fold operations.

Let's say the first element in the stream is a count of the following elements that we have to add, then:

>>> import Data.Maybe (fromJust)>>> count = fmap fromJust Fold.one>>> total n = Fold.take n Fold.sum>>> Stream.fold (Fold.concatMap total count) $ Stream.fromList [10,9..1]45

This does not fuse completely, see refold for a fusible alternative.

Time: O(n^2) where n is the number of compositions.

See also: foldIterateM, refold