Split packages by meggart · Pull Request #198 · JuliaDataCubes/EarthDataLab.jl

meggart · Apr 3, 2020

No description provided.

felixcremer · Apr 14, 2020

If I try to use this branch this fails due to problems with reexport

julia> using ESDL
[ Info: Precompiling ESDL [359177bc-a543-11e8-11b7-bb015dba3358]
ERROR: LoadError: syntax: malformed expression
Stacktrace:
 [1] top-level scope at /home/crem_fe/.julia/dev/ESDL/src/ESDL.jl:42
 [2] include(::Module, ::String) at ./Base.jl:377
 [3] top-level scope at none:2
 [4] eval at ./boot.jl:331 [inlined]
 [5] eval(::Expr) at ./client.jl:449
 [6] top-level scope at ./none:3
in expression starting at /home/crem_fe/.julia/dev/ESDL/src/ESDL.jl:42
ERROR: Failed to precompile ESDL [359177bc-a543-11e8-11b7-bb015dba3358] to /home/crem_fe/.julia/compiled/v1.4/ESDL/2Q02O_MQjDs.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
 [3] _require(::Base.PkgId) at ./loading.jl:1029
 [4] require(::Base.PkgId) at ./loading.jl:927
 [5] require(::Module, ::Symbol) at ./loading.jl:922

felixcremer · Apr 14, 2020

Trying to save a cube as a netcdf fails with the following error:

julia> savecube(shpcube, "/home/crem_fe/Daten/kiuic_example_newcube3.nc", backend=:netcdf)
ERROR: NetCDF error code -127:
	NetCDF: Bad chunk sizes.
Stacktrace:
 [1] check at /home/crem_fe/.julia/packages/NetCDF/VkdAm/src/netcdf_helpers.jl:22 [inlined]
 [2] nc_def_var_chunking at /home/crem_fe/.julia/packages/NetCDF/VkdAm/src/netcdf_c.jl:1039 [inlined]
 [3] create_var(::NetCDF.NcFile, ::NetCDF.NcVar{Float32,2,5}, ::UInt16) at /home/crem_fe/.julia/packages/NetCDF/VkdAm/src/NetCDF.jl:1171
 [4] (::NetCDF.var"#65#68"{UInt16,String,String,Array{NetCDF.NcDim,1},NetCDF.NcVar{Float32,2,5}})(::NetCDF.NcFile) at /home/crem_fe/.julia/packages/NetCDF/VkdAm/src/NetCDF.jl:1258
 [5] open(::NetCDF.var"#65#68"{UInt16,String,String,Array{NetCDF.NcDim,1},NetCDF.NcVar{Float32,2,5}}, ::String; kwargs::Base.Iterators.Pairs{Symbol,UInt16,Tuple{Symbol},NamedTuple{(:mode,),Tuple{UInt16}}}) at /home/crem_fe/.julia/packages/NetCDF/VkdAm/src/NetCDF.jl:998
 [6] nccreate(::String, ::String, ::String, ::Vararg{Any,N} where N; atts::Dict{String,Any}, gatts::Dict{Any,Any}, compress::Int64, t::DataType, mode::UInt16, chunksize::Tuple{Int64,Int64}) at /home/crem_fe/.julia/packages/NetCDF/VkdAm/src/NetCDF.jl:1213
 [7] add_var(::YAXArrayBase.NetCDFDataset, ::Type{T} where T, ::String, ::Array{Int64,1}, ::Array{String,1}, ::Dict{String,Any}; chunksize::Tuple{Int64,Int64}, compress::Int64) at /home/crem_fe/.julia/packages/YAXArrayBase/zUe1C/src/datasets/netcdf.jl:28
 [8] (::ESDL.Datasets.var"#60#68"{Union,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},YAXArrayBase.NetCDFDataset,Dict{String,Any},Array{ESDL.Cubes.CleanMe,1}})(::String) at /home/crem_fe/.julia/dev/ESDL/src/DatasetAPI/Datasets.jl:266
 [9] iterate at ./generator.jl:47 [inlined]
 [10] _collect(::Array{String,1}, ::Base.Generator{Array{String,1},ESDL.Datasets.var"#60#68"{Union,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},YAXArrayBase.NetCDFDataset,Dict{String,Any},Array{ESDL.Cubes.CleanMe,1}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:678
 [11] collect_similar(::Array{String,1}, ::Base.Generator{Array{String,1},ESDL.Datasets.var"#60#68"{Union,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},YAXArrayBase.NetCDFDataset,Dict{String,Any},Array{ESDL.Cubes.CleanMe,1}}}) at ./array.jl:607
 [12] map(::Function, ::Array{String,1}) at ./abstractarray.jl:2072
 [13] createdataset(::Type{T} where T, ::Array{CubeAxis,1}; path::String, persist::Bool, T::Type{T} where T, chunksize::Tuple{Int64,Int64}, chunkoffset::Tuple{Int64,Int64}, overwrite::Bool, properties::Dict{String,Any}, datasetaxis::String, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/DatasetAPI/Datasets.jl:258
 [14] generateOutCube(::Type{YAXArrayBase.NetCDFDataset}, ::Type{T} where T, ::ESDL.DAT.OutputCube, ::Array{Int64,1}, ::Tuple{Int64,Int64}; kwargs::Base.Iterators.Pairs{Symbol,String,Tuple{Symbol},NamedTuple{(:path,),Tuple{String}}}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:437
 [15] generateOutCube(::ESDL.DAT.OutputCube, ::Base.RefValue{Bool}, ::Float64, ::Array{Int64,1}, ::Tuple{Int64,Int64}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:458
 [16] (::ESDL.DAT.var"#84#85"{ESDL.DAT.DATConfig{1,1},Tuple{}})(::ESDL.DAT.OutputCube) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:453
 [17] foreach(::ESDL.DAT.var"#84#85"{ESDL.DAT.DATConfig{1,1},Tuple{}}, ::Tuple{ESDL.DAT.OutputCube}) at ./abstractarray.jl:1919
 [18] generateOutCubes(::ESDL.DAT.DATConfig{1,1}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:451
 [19] mapCube(::typeof(copyto!), ::Tuple{ESDL.Cubes.ESDLArray{Union{Missing, Int32},2,Array{Union{Missing, Int32},2},Array{CubeAxis,1}}}; max_cache::Float64, indims::InDims, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, nthreads::Array{Int64,1}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:262
 [20] #mapCube#28 at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:175 [inlined]
 [21] savecube(::ESDL.Cubes.ESDLArray{Union{Missing, Int32},2,Array{Union{Missing, Int32},2},Array{CubeAxis,1}}, ::String; chunksize::Tuple{Int64,Int64}, max_cache::Float64, backend::Symbol, backendargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/Proc/CubeIO.jl:33
 [22] top-level scope at REPL[10]:1

Saving as a zarrcube works.

meggart · Apr 15, 2020

Ah sorry I forgot about all the fixes that I did on the way. You need NetCDF master as well.

meggart · Apr 15, 2020

I also just tagged a new version for NetCDF with the bug fix, so in about an hour updating NetCDF to 0.10.1 should be sufficient.

…tPackages

felixcremer · May 6, 2020

What is the current status of this PR?
Is there anything I can do to move this forward?

meggart · May 7, 2020

I think the last bit would be to actually split YAXArrays and ESDL.jl so that they can live completely separately. Then we should start registering the first packages. I think DiskArrayTools and YAXArrayBase should be good candidates that could be registered now. We only would need a nice Readme and ideally some documentartion for both of them.

meggart · May 7, 2020

Otherwise I am quite happy with the changes so far and would start testing this on my own code.

Ah, and there is still the issue with Reexport, but probably we should just copy-paste the macro into this package, it is only a few lines of code but the package seems unmaintained.

The other thing to make this work is to finish the ArchGDAL PR, the only thing missing there is, again, some documentation, so basically replacing the old raster subsection with examples from the new interface.

If you want to help with any of these, feel free to volunteer.

felixcremer · May 7, 2020

I tested opening an envistack with this and the current diskarray branch at ArchGDAL but it fails with the following:

julia> c = ESDLArray(f)
ERROR: MethodError: no method matching dimvals(::ArchGDAL.RasterDataset{Float32,ArchGDAL.IDataset}, ::Int64)
Closest candidates are:
  dimvals(::ESDL.Cubes.YAXSlice, ::Any) at /home/crem_fe/.julia/dev/ESDL/src/Cubes/Slices.jl:19
Stacktrace:
 [1] (::ESDL.Cubes.var"#6#7"{ArchGDAL.RasterDataset{Float32,ArchGDAL.IDataset}})(::Tuple{Int64,Symbol}) at /home/crem_fe/.julia/dev/ESDL/src/Cubes/Cubes.jl:125
 [2] iterate at ./generator.jl:47 [inlined]
 [3] collect at ./array.jl:665 [inlined]
 [4] map at ./abstractarray.jl:2098 [inlined]
 [5] caxes at /home/crem_fe/.julia/dev/ESDL/src/Cubes/Cubes.jl:123 [inlined]
 [6] ESDLArray(::ArchGDAL.RasterDataset{Float32,ArchGDAL.IDataset}) at /home/crem_fe/.julia/dev/ESDL/src/Cubes/Cubes.jl:114
 [7] top-level scope at REPL[9]:1
 [8] eval(::Module, ::Any) at ./boot.jl:331
 [9] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
 [10] run_backend(::REPL.REPLBackend) at /home/crem_fe/.julia/packages/Revise/C272c/src/Revise.jl:1075
 [11] top-level scope at REPL[1]:0

This should be the same envistack you used for testing this.

meggart · May 8, 2020

Are you sure you are in the latest YAXArrayBase? The method that is missing should be defined here: https://github.com/meggart/YAXArrayBase.jl/blob/master/src/axisarrays/axisarrayimpl.jl#L73-L133

felixcremer · May 8, 2020

The problem was, that the definitions of dimvals, dimname and iscontdim in Cubes/Slices.jl overwrote the definitions in YAXArrayBase.
Should this YAXSlice be a part of YAXArrayBase?

felixcremer · May 9, 2020

Is it correct, that with Zarr behind a require macro the user needs to do using Zarr explicitly?
Should we make Zarr a default backend at least here in ESDL?

meggart · May 9, 2020

The problem was, that the definitions of dimvals, dimname and iscontdim in Cubes/Slices.jl overwrote the definitions in YAXArrayBase.

Ok, this is a bug, I will look at this

Should this YAXSlice be a part of YAXArrayBase?

No, I am not even sure it should be anywhere, it is just me experimenting with a new interface to mapCube. The idea is the following: Image you have a data cube with dims lon x lat x time x var. Then c / ("Time", "Variable") creates a sliced data cube which only has lon and lat dimensions and a Time x Variable matrices as element types. The nice thing here would be that you can describe mapCube as simple broadcast operations over these sliced cubes, so you could even use dot syntax to apply your functions.

felixcremer · May 13, 2020

Saving the data in a new chunking doesn't work.
I am currently working on the GDAL Cube stuff and I tried the example you provided in #193.

But when I am using it like this:

julia> cube
ESDL data cube with the following dimensions
Y                   Axis with 5820 Elements from 561609.15 to 736179.15
X                   Axis with 7500 Elements from 9.48724066e6 to 9.26227066e6
Time                Axis with 200 elements: Band_1 Band_2 .. Band_199 Band_200 
Total size: 40.65 GB
julia> ESDL.Cubes.cubechunks(cube)
(128, 128, 1)
julia> savecube(cube, "./jurua_vv_timechunk2.zarr", chunksize=(30,30,200))
julia> tch2 = Cube("jurua_vv_timechunk2.zarr/")
ESDL data cube with the following dimensions
Y                   Axis with 5820 Elements from 561609.15 to 736179.15
X                   Axis with 7500 Elements from 9.48724066e6 to 9.26227066e6
Time                Axis with 200 elements: Band_1 Band_2 .. Band_199 Band_200 
Total size: 40.65 GB
julia> ESDL.Cubes.cubechunks(tch2)
(30, 4224, 1)

As you can see, I get a totally weird chunking.

felixcremer · May 13, 2020

Does the conversion of a ArchGDAL.RasterDataset in an ESDLArray use the nodata value of the raster data? Or is there an easy way to set a nodata value of a cube later on?

meggart · May 15, 2020

Does the conversion of a ArchGDAL.RasterDataset in an ESDLArray use the nodata value of the raster data? Or is there an easy way to set a nodata value of a cube later on?

Currently I don't know of a very simple way to do that. There would be several possibilities:

Do it directly at the ArchGDAL level, i.e. update the diskarrays PR to automatically apply nodata values, offset and scale_factor. The main problem here is probably that the C library does not like to interact with the Union{T,Missing} from Julia, so the readblock! function would probably have to allocate.
Try to wrap the RasterDataset by either wrapping it into a CFDiskArray from DiskArrayTools or by just using broadcasting like (x->x==-99 ? missing : x).(myarray) The problem with this approach is that this is then a new array type which does not implement the YAXArray interface anymore
Add an option to ESDL InDims to mark values as missing sentinels. This would be possible, but I don't know yet where exactly to apply this best.

I will think a bit about the options and do something next week.

felixcremer · May 20, 2020

Thanks, the chunking works now.
I am wondering, whether we should give the chunksizes as a NamedTuple instead of a dictionary, because this might be faster.

felixcremer · May 20, 2020

The progressbar is not showing up during the computation and is only shown after the computation is finished, or after I have interrupted the computation.

felixcremer · May 20, 2020

Unfortunately, the chunking doesn't work in the parallel use case, there I get the following error:

julia> savecube(cube, "./jurua_vv_timechunk_parallel.zarr", chunksize=(30,30,200))
┌ Warning: Chunksize must be provided as a Dict mapping axis names to chunk size in the future
└ @ ESDL.Proc ~/.julia/dev/ESDL/src/Proc/CubeIO.jl:17
newchunks = Union{Nothing, Int64}[30, 30, 200]
┌ Warning: There are still cache misses
└ @ ESDL.DAT ~/.julia/dev/ESDL/src/DAT/DAT.jl:668
Error during initialization: KeyError(1)
ERROR: KeyError: key 1 not found
Stacktrace:
 [1] runLoop(::ESDL.DAT.DATConfig{1,1}, ::Bool) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:408
 [2] mapCube(::ESDL.Proc.var"#cop#30", ::Tuple{ESDLArray{Float32,3,ArchGDAL.RasterDataset{Float32,ArchGDAL.Dataset},Array{CubeAxis,1}}}; max_cache::Float64, indims::InDims, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, nthreads::Dict{Int64,Int64}, loopchunksize::Dict{Any,Any}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:278
 [3] #mapCube#32 at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:177 [inlined]
 [4] savecube(::ESDLArray{Float32,3,ArchGDAL.RasterDataset{Float32,ArchGDAL.Dataset},Array{CubeAxis,1}}, ::String; chunksize::Tuple{Int64,Int64,Int64}, max_cache::Float64, backend::Symbol, backendargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/Proc/CubeIO.jl:38
 [5] top-level scope at REPL[32]:1

And also the processing is broken with for me undecipherable errors.
I still have to check, whether this also fails for zarrcubes or only for GDAL cubes.

felixcremer · Jun 8, 2020

Trying to save a zarrcube as a netcdf file fails with the following error message:

julia> savecube(c, "jurua_vv2.nc", backend=:netcdf)
newchunks = Union{Nothing, Int64}[nothing, nothing, nothing]
ERROR: ArgumentError: array must be non-empty
Stacktrace:
 [1] popfirst! at ./array.jl:1186 [inlined]
 [2] jl2nc(::DataType) at /home/crem_fe/.julia/packages/NetCDF/dmH61/src/NetCDF.jl:88
 [3] getNCType(::DataType) at /home/crem_fe/.julia/packages/NetCDF/dmH61/src/NetCDF.jl:98
 [4] NcVar(::String, ::Array{NcDim,1}; atts::Dict{String,Any}, t::DataType, compress::Int64, chunksize::Tuple{Int64}) at /home/crem_fe/.julia/packages/NetCDF/dmH61/src/NetCDF.jl:235
 [5] nccreate(::String, ::String, ::String, ::Vararg{Any,N} where N; atts::Dict{String,Any}, gatts::Dict{Any,Any}, compress::Int64, t::DataType, mode::UInt16, chunksize::Tuple{Int64}) at /home/crem_fe/.julia/packages/NetCDF/dmH61/src/NetCDF.jl:1226
 [6] #add_var#65 at /home/crem_fe/.julia/dev/YAXArrayBase/src/datasets/netcdf.jl:30 [inlined]
 [7] add_var at /home/crem_fe/.julia/dev/YAXArrayBase/src/datasets/netcdf.jl:29 [inlined]
 [8] add_var(::YAXArrayBase.NetCDFDataset, ::Array{Zarr.MaxLengthStrings.MaxLengthString{8,UInt8},1}, ::String, ::Tuple{String}, ::Dict{String,Any}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:46
 [9] add_var at /home/crem_fe/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:46 [inlined]
 [10] arrayfromaxis(::YAXArrayBase.NetCDFDataset, ::CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{8,UInt8},:Time,Array{Zarr.MaxLengthStrings.MaxLengthString{8,UInt8},1}}, ::Int64) at /home/crem_fe/.julia/dev/ESDL/src/DatasetAPI/Datasets.jl:324
 [11] (::ESDL.Datasets.var"#66#74"{YAXArrayBase.NetCDFDataset})(::CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{8,UInt8},:Time,Array{Zarr.MaxLengthStrings.MaxLengthString{8,UInt8},1}}, ::Int64) at /home/crem_fe/.julia/dev/ESDL/src/DatasetAPI/Datasets.jl:261
 [12] foreach(::Function, ::Array{CubeAxis,1}, ::Tuple{Int64,Int64,Int64}) at ./abstractarray.jl:1920
 [13] createdataset(::Type{T} where T, ::Array{CubeAxis,1}; path::String, persist::Bool, T::Type{T} where T, chunksize::Tuple{Int64,Int64,Int64}, chunkoffset::Tuple{Int64,Int64,Int64}, overwrite::Bool, properties::Dict{String,Any}, datasetaxis::String, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/DatasetAPI/Datasets.jl:260
 [14] generateOutCube(::Type{YAXArrayBase.NetCDFDataset}, ::Type{T} where T, ::ESDL.DAT.OutputCube, ::Array{Int64,1}, ::Tuple{Int64,Int64,Int64}; kwargs::Base.Iterators.Pairs{Symbol,String,Tuple{Symbol},NamedTuple{(:path,),Tuple{String}}}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:488
 [15] generateOutCube(::ESDL.DAT.OutputCube, ::Base.RefValue{Bool}, ::Float64, ::Array{Int64,1}, ::Tuple{Int64,Int64,Int64}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:509
 [16] (::ESDL.DAT.var"#100#101"{ESDL.DAT.DATConfig{1,1},Tuple{Int64,Int64}})(::ESDL.DAT.OutputCube) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:504
 [17] foreach(::ESDL.DAT.var"#100#101"{ESDL.DAT.DATConfig{1,1},Tuple{Int64,Int64}}, ::Tuple{ESDL.DAT.OutputCube}) at ./abstractarray.jl:1919
 [18] generateOutCubes(::ESDL.DAT.DATConfig{1,1}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:502
 [19] mapCube(::ESDL.Proc.var"#cop#30", ::Tuple{ESDLArray{Union{Missing, Float32},3,ZArray{Union{Missing, Float32},3,Zarr.BloscCompressor,DirectoryStore},Array{CubeAxis,1}}}; max_cache::Float64, indims::InDims, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, nthreads::Array{Int64,1}, loopchunksize::Dict{Any,Any}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:275
 [20] #mapCube#32 at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:177 [inlined]
 [21] savecube(::ESDLArray{Union{Missing, Float32},3,ZArray{Union{Missing, Float32},3,Zarr.BloscCompressor,DirectoryStore},Array{CubeAxis,1}}, ::String; chunksize::Dict{Any,Any}, max_cache::Float64, backend::Symbol, backendargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/Proc/CubeIO.jl:38
 [22] top-level scope at REPL[18]:1

felixcremer · Jun 21, 2020

The netcdf saving brake in 03f9fd2. Unfortunately, I don't know enough of the internals of the NetCDF package or YAXArrayBase to fix it myself.

* Delete almost everything * More updates * Old tests for ESDL pass * More changes * Fix bugs in ESDL * Some fixes * Fix some warnings * tests should pass

* Removed a lot of files * move some files * break more things * Clean up Proc module * Many fixes to get dataset creation running again * savecube to netcdf works * more improvements * Commit many more changes * Separate out API * some test fixes * switch to YAXArrayBase * read variables in a typed way * add ESDLArray export * Fixes to remap * remove NetCDF and Zarr dependency * delete some unused stuff * Do not permute cubes by default * Add concatenations * Fixes to findaxis * Tests pass again * Try to fix chunking * dont depend on reexport * remove reexport dep * dont forgegt using it * update test * Allow NamedTuples * minor changes * Finalsplit (#220) * Delete almost everything * More updates * Old tests for ESDL pass * More changes * Fix bugs in ESDL * Some fixes * Fix some warnings * tests should pass

* Start reworking everything * Loading cubes works again * mapslices running * Simple mapslices works * Access tests pass * Tests pass with new data type * Code improvements and TODOs based on @felixcremer s comments * add CubeAxis import to Shapes.jl * some fixes * small fix for readblock * Show CodeCov badge in README * fix bug when subsetting concatcubes * Implement changes suggested by @felixcremer * Add sync after exporting cube * Require NetCDF 0.10 * make it load again * Fix ambiguity * Fix cartind bug * Add DiskArrays as direct dependency This doesn't solve the test failure because some readblock! method is still missing. * Fix bug in readblock! * Import readblock! from Zarr I also changed the import Zarr to using Zarr: ... and sorted the imports into types and functions which are sorted alphabetically. * Make wrap keyword available in cubefromshape (#199) Wrap is a keyword which is projection specific and which gives the projection value at which the projection wraps into itself. The default values fail for non Lat Lon projections. Therefore, this makes the keyword available in the cubefromshape interface. This is an intermediate solution, because it would be better to have the projection as a metadata information of a cube, so that we could use this to set the wrapping values. * Revert "Make wrap keyword available in cubefromshape (#199)" This reverts commit 6e778a2. Unfortunately, this pull request was not yet ready. I should have indicated that more clearly. I am reverting this for now and then I am going to push a tidied up version of the PR. * Change clone to add in readme install instructions Clone is the keyword from the old package manger. * Update README.md * typos (#224) * Split packages (#198) * Removed a lot of files * move some files * break more things * Clean up Proc module * Many fixes to get dataset creation running again * savecube to netcdf works * more improvements * Commit many more changes * Separate out API * some test fixes * switch to YAXArrayBase * read variables in a typed way * add ESDLArray export * Fixes to remap * remove NetCDF and Zarr dependency * delete some unused stuff * Do not permute cubes by default * Add concatenations * Fixes to findaxis * Tests pass again * Try to fix chunking * dont depend on reexport * remove reexport dep * dont forgegt using it * update test * Allow NamedTuples * minor changes * Finalsplit (#220) * Delete almost everything * More updates * Old tests for ESDL pass * More changes * Fix bugs in ESDL * Some fixes * Fix some warnings * tests should pass * Start reworking everything * Loading cubes works again * mapslices running * Simple mapslices works * Access tests pass * Tests pass with new data type * Code improvements and TODOs based on @felixcremer s comments * add CubeAxis import to Shapes.jl * some fixes * fix bug when subsetting concatcubes * Implement changes suggested by @felixcremer * Add sync after exporting cube * Require NetCDF 0.10 * make it load again * Fix ambiguity * Split packages (#198) * Removed a lot of files * move some files * break more things * Clean up Proc module * Many fixes to get dataset creation running again * savecube to netcdf works * more improvements * Commit many more changes * Separate out API * some test fixes * switch to YAXArrayBase * read variables in a typed way * add ESDLArray export * Fixes to remap * remove NetCDF and Zarr dependency * delete some unused stuff * Do not permute cubes by default * Add concatenations * Fixes to findaxis * Tests pass again * Try to fix chunking * dont depend on reexport * remove reexport dep * dont forgegt using it * update test * Allow NamedTuples * minor changes * Finalsplit (#220) * Delete almost everything * More updates * Old tests for ESDL pass * More changes * Fix bugs in ESDL * Some fixes * Fix some warnings * tests should pass Co-authored-by: Felix Cremer <felix.cremer@uni-jena.de> Co-authored-by: Guido Kraemer <gdkrmr@users.noreply.github.com>

meggart added 11 commits March 25, 2020 10:55

Removed a lot of files

49bbf22

move some files

1b51062

break more things

dd743e4

Clean up Proc module

d8bdef7

Many fixes to get dataset creation running again

11fdc58

savecube to netcdf works

2d60f9b

more improvements

d37a495

Commit many more changes

393e5f1

Separate out API

9dc32d3

some test fixes

f663df1

switch to YAXArrayBase

db424c9

read variables in a typed way

7630266

felixcremer mentioned this pull request Apr 15, 2020

exportcube shifts the netcdf by 360 meters for UTM projection #210

Open

meggart added 8 commits April 17, 2020 14:32

add ESDLArray export

681e58e

Merge branch 'SplitPackages' of github.com:esa-esdl/ESDL.jl into Spli…

a546f97

…tPackages

Fixes to remap

673f34b

remove NetCDF and Zarr dependency

83b43e3

delete some unused stuff

08a4ee0

Do not permute cubes by default

7a7cc00

Add concatenations

f562ab5

Fixes to findaxis

c6e5458

Tests pass again

b2a7fed

meggart added 4 commits May 20, 2020 00:06

Try to fix chunking

03f9fd2

dont depend on reexport

156f116

remove reexport dep

b799dd1

dont forgegt using it

687e277

meggart added 2 commits May 20, 2020 14:43

update test

25e0393

Allow NamedTuples

27d4da8

meggart added 2 commits August 6, 2020 14:29

minor changes

8b91a92

Finalsplit (#220)

ac4620e

* Delete almost everything * More updates * Old tests for ESDL pass * More changes * Fix bugs in ESDL * Some fixes * Fix some warnings * tests should pass

meggart merged commit bd344c0 into NewCubeType Dec 7, 2020

meggart deleted the SplitPackages branch December 7, 2020 18:58

Search code, repositories, users, issues, pull requests...

Conversation

meggart commented Apr 3, 2020

Uh oh!

felixcremer commented Apr 14, 2020

Uh oh!

felixcremer commented Apr 14, 2020

Uh oh!

meggart commented Apr 15, 2020

Uh oh!

meggart commented Apr 15, 2020

Uh oh!

felixcremer commented May 6, 2020

Uh oh!

meggart commented May 7, 2020

Uh oh!

meggart commented May 7, 2020

Uh oh!

felixcremer commented May 7, 2020

Uh oh!

meggart commented May 8, 2020

Uh oh!

felixcremer commented May 8, 2020

Uh oh!

felixcremer commented May 9, 2020

Uh oh!

meggart commented May 9, 2020

Uh oh!

felixcremer commented May 13, 2020

Uh oh!

felixcremer commented May 13, 2020

Uh oh!

meggart commented May 15, 2020

Uh oh!

felixcremer commented May 20, 2020

Uh oh!

felixcremer commented May 20, 2020

Uh oh!

felixcremer commented May 20, 2020

Uh oh!

felixcremer commented Jun 8, 2020

Uh oh!

felixcremer commented Jun 21, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants