Streamly.Internal.Data.Array

Convert a stream of arrays into a stream of their elements.

>>> concat = Stream.unfoldMany Array.reader

Return element at the specified index without checking the bounds.

O(1) Get the length of the array i.e. the number of elements in the array.

Convert an Array into a stream.

Pre-release

Create two slices of an array without copying the original array. The specified index i is the first index of the second slice.

Like createOf but does not check the array bounds when writing. The fold driver must not call the step function more than n times otherwise it will corrupt the memory and crash. This function exists mainly because any conditional in the step function blocks fusion causing 10x performance slowdown.

createOf n folds a maximum of n elements from the input stream to an Array.

Fold the whole input to a single array.

Caution! Do not use this on infinite streams.

Create an Array from the first N elements of a stream. The array is allocated to size N, if the stream terminates before N elements then the array may hold less than N elements.

>>> fromStreamN n = Stream.fold (Array.writeN n)

Pre-release

Create an Array from a stream. This is useful when we want to create a single array from a stream of unknown size. writeN is at least twice as efficient when the size is already known.

>>> fromStream = Stream.fold Array.write

Note that if the input stream is too large memory allocation for the array may fail. When the stream size is not known, chunksOf followed by processing of indvidual arrays in the resulting stream should be preferred.

Pre-release

Convert a pure stream in Identity monad to an immutable array.

Same as the following but with better performance:

>>> fromPureStream = Array.fromList . runIdentity . Stream.toList

Create an Array from the first N elements of a list. The array is allocated to size N, if the list terminates before N elements then the array may hold less than N elements.

Create an Array from a list. The list must be of finite size.

Unfold an array into a stream.

Convert an Array into a stream in reverse order.

Pre-release

Convert an Array into a list.

chunksOf n stream groups the elements in the input stream into arrays of n elements each.

Same as the following but may be more efficient:

>>> chunksOf n = Stream.foldMany (Array.writeN n)

Pre-release

Same as read

Copy a null terminated immutable Addr# Word8 sequence into an array.

Unsafe: The caller is responsible for safe addressing.

Note that this is completely safe when reading from Haskell string literals because they are guaranteed to be NULL terminated:

>>> Array.toList $ Array.fromByteStr# "\1\2\3\0"#[1,2,3]

Note that this should be evaluated strictly to ensure that we do not hold the reference to the pointer in a lazy thunk.

Like chunksOf but creates pinned arrays.

compactGE n stream coalesces adjacent arrays in the stream until the size becomes greater than or equal to n.

>>> compactGE n = Stream.foldMany (Array.fCompactGE n)

Generates unpinned arrays irrespective of the pinning status of input arrays.

Convert a stream of arrays into a stream of their elements reversing the contents of each array before flattening.

>>> concatRev = Stream.unfoldMany Array.readerRev

Similar to unsafeFreeze but uses rightSize on the mutable array first.

O(1) Get the byte length of the array.

Given a stream of arrays, splice them all together to generate a single array. The stream must be finite.

Return True if the array is allocated in pinned memory.

Return a copy of the Array in pinned memory if unpinned, else return the original array.

Return a copy of the Array in unpinned memory if pinned, else return the original array.

Unfold an array into a stream in reverse order.

Same as readRev

Copy an immutable 'Ptr Word8' sequence into an array.

Unsafe: The caller is responsible for safe addressing.

Note that this should be evaluated strictly to ensure that we do not hold the reference to the pointer in a lazy thunk.

Use an Array a as Ptr a.

See unsafePinnedAsPtr in the Mutable array module for more details.

Unsafe

Pre-release

Like createOf but creates a pinned array.

Like create but creates a pinned array.

Like fromListN but creates a pinned array.

Like fromList but creates a pinned array.

Create an Array from the first N elements of a list in reverse order. The array is allocated to size N, if the list terminates before N elements then the array may hold less than N elements.

Pre-release

Create an Array from a list in reverse order. The list must be of finite size.

Pre-release

Convert an array stream to an array. Note that this requires peak memory that is double the size of the array stream.

Byte compare two arrays. Compare the length of the arrays. If the length is equal, compare the lexicographical ordering of two underlying byte arrays otherwise return the result of length comparison.

Unsafe: Note that the Unbox instance of sum types with constructors of different sizes may leave some memory uninitialized which can make byte comparison unreliable.

Pre-release

Byte equality of two arrays.

>>> byteEq arr1 arr2 = (==) EQ $ Array.byteCmp arr1 arr2

Unsafe: See byteCmp.

Copy two immutable arrays into a new array. If you want to splice more than two arrays then this operation would be highly inefficient because it would make a copy on every splice operation, instead use the fromChunksK operation to combine n immutable arrays.

Fold fCompactGE n coalesces adjacent arrays in the input stream until the size becomes greater than or equal to n.

Generates unpinned arrays irrespective of the pinning status of input arrays.

PInned version of fCompactGE.

Like compactGE but for transforming folds instead of stream.

>>> lCompactGE n = Fold.many (Array.fCompactGE n)

Generates unpinned arrays irrespective of the pinning status of input arrays.

Pinned version of lCompactGE.

pinnedWriteNAligned alignment n folds a maximum of n elements from the input stream to an Array aligned to the given size.

Pre-release

Makes an immutable array using the underlying memory of the mutable array.

Please make sure that there are no other references to the mutable array lying around, so that it is never used after freezing it using unsafeFreeze. If the underlying array is mutated, the immutable promise is lost.

Pre-release

Makes a mutable array using the underlying memory of the immutable array.

Please make sure that there are no other references to the immutable array lying around, so that it is never used after thawing it using unsafeThaw. If the resulting array is mutated, any references to the older immutable array are mutated as well.

Pre-release

Fold "step" has a dependency on "initial", and each step is dependent on the previous invocation of step due to state passing, finally extract depends on the result of step, therefore, as long as the fold is driven in the correct order the operations would be correctly ordered. We need to ensure that we strictly evaluate the previous step completely before the next step.

To not share the same array we need to make sure that the result of "initial" is not shared. Existential type ensures that it does not get shared across different folds. However, if we invoke "initial" multiple times for the same fold, there is a possiblity of sharing the two because the compiler would consider it as a pure value. One such example is the chunksOf combinator, or using an array creation fold with foldMany combinator. Is there a proper way in GHC to tell it to not share a pure expression in a particular case?

For this reason array creation folds have a MonadIO constraint. Pure folds could be unsafe and dangerous. This is dangerous especially when used with foldMany like operations.

>>> unsafePureWrite = Array.unsafeMakePure Array.write

Note that this should be evaluated strictly to ensure that we do not hold the reference to the pointer in a lazy thunk.

Return element at the specified index without checking the bounds.

Unsafe because it does not check the bounds of the array.

Unfold an array into a stream, does not check the end of the array, the user is responsible for terminating the stream within the array bounds. For high performance application where the end condition can be determined by a terminating fold.

Written in the hope that it may be faster than "read", however, in the case for which this was written, "read" proves to be faster even though the core generated with unsafeRead looks simpler.

Pre-release

writeLastN n folds a maximum of n elements from the end of the input stream to an Array.

O(1) Lookup the element at the given index. Index starts from 0.

Like getIndex but indexes the array in reverse from the end.

Pre-release

>>> last arr = Array.getIndexRev arr 0

Pre-release

Given a stream of array indices, read the elements on those indices from the supplied Array. An exception is thrown if an index is out of bounds.

This is the most general operation. We can implement other operations in terms of this:

read =
     let u = lmap (arr -> (0, length arr - 1)) Unfold.enumerateFromTo
      in Unfold.lmap f (indexReader arr)

readRev =
     let i = length arr - 1
      in Unfold.lmap f (indexReaderFromThenTo i (i - 1) 0)

Pre-release

Unfolds (from, then, to, array) generating a finite stream whose first element is the array value from the index from and the successive elements are from the indices in increments of then up to to. Index enumeration can occur downwards or upwards depending on whether then comes before or after from.

getIndicesFromThenTo =
    let f (from, next, to, arr) =
            (Stream.enumerateFromThenTo from next to, arr)
     in Unfold.lmap f getIndices

Unimplemented

>>> null arr = Array.byteLength arr == 0

Pre-release

Given a sorted array, perform a binary search to find the given element. Returns the index of the element if found.

Unimplemented

Unimplemented

Perform a linear search to find all the indices where a given element is present in an array.

Unimplemented

Cast an array having elements of type a into an array having elements of type b. The length of the array should be a multiple of the size of the target element otherwise Nothing is returned.

Cast an Array a into an Array Word8.

Cast an array having elements of type a into an array having elements of type b. The array size must be a multiple of the size of type b otherwise accessing the last element of the array may result into a crash or a random value.

Pre-release

Convert an array of any type into a null terminated CString Ptr. If the array is unpinned it is first converted to a pinned array which requires a copy.

Unsafe

O(n) Time: (creates a copy of the array)

Pre-release

O(1) Slice an array in constant time.

Caution: The bounds of the slice are not checked.

Unsafe

Pre-release

Generate a stream of slices of specified length from an array, starting from the supplied array index. The last slice may be shorter than the requested length.

Pre-release/

Split the array into a stream of slices using a predicate. The element matching the predicate is dropped.

Pre-release

Transform an array into another array using a stream transformation operation.

Pre-release

Fold an array using a stream fold operation.

Pre-release

Fold an array using a Fold.

Pre-release

Insert the given element between arrays and flatten.

>>> interpose x = Stream.interpose x Array.reader

Insert the given element after each array and flatten. This is similar to unlines.

>>> interposeSuffix x = Stream.interposeSuffix x Array.reader

Insert the given array after each array and flatten.

>>> intercalateSuffix = Stream.intercalateSuffix Array.reader

compactLE n coalesces adjacent arrays in the input stream only if the combined size would be less than or equal to n.

Generates unpinned arrays irrespective of the pinning status of input arrays.

Pinned version of compactLE.

Split a stream of arrays on a given separator byte, dropping the separator and coalescing all the arrays between two separators into a single array.

Like compactOnByte considers the separator in suffix position instead of infix position.

Fold a stream of arrays using a Fold. This is equivalent to the following:

>>> foldChunks f = Stream.fold f . Stream.unfoldMany Array.reader

Fold a stream of arrays using a Fold and return the remaining stream.

The following alternative to this function allows composing the fold using the parser Monad:

foldBreakStreamK f s =
      fmap (first (fromRight undefined))
    $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s

We can compare perf and remove this one or define it in terms of that.

Parse an array stream using the supplied Parser. Returns the parse result and the unconsumed stream. Throws ParseError if the parse fails.

The following alternative to this function allows composing the parser using the parser Monad:

>>> parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)

We can compare perf and remove this one or define it in terms of that.

Internal

Properties: 1. Identity: deserialize . serialize == id 2. Encoded equivalence: serialize a == serialize a

Serialize a Haskell type to a pinned byte array. The array is allocated using pinned memory so that it can be used directly in OS APIs for writing to file or sending over the network.

Properties: 1. Identity: deserialize . pinnedSerialize == id 2. Encoded equivalence: pinnedSerialize a == pinnedSerialize a

Decode a Haskell type from a byte array containing its serialized representation.