Recommendation:

• Provide an alternative method/function to access the arrays. The new method would include an optional iterable argument for the requested dimensions, defaulting to all dimensions.
• Or provide an optional list element in the Python filter configuration to identify which dimensions are to be returned in the array. The new option would be valid whether the filter’s “function” option is applied or not. The filter would then be used to a) provide a callback function and/or b) identify which arrays to return.
• Or create another Python filter solely for the purpose of identifying which dimensions to return in the arrays.

Justification:

• Any solution to reduce memory usage could significantly improve the chances that a containerized process, like Docker, would not fail due to exceeding allocated memory.
• A streamable pipeline could be used to help reduce the memory used for the returned arrays. However, when using a Python filter, among other filters, the PDAL pipeline is no longer streamable. This results in the entire dataset returned. That dataset can be extremely large when it contains hundreds of millions of points and has dozens of dimensions.