WIP: New cube type#181
Conversation
|
Just forgot to mention we need the permute_reshape branch of DiskArrays.jl |
|
I can try this tomorrow. |
|
If I try cubefromshape, I get the following error: julia> oldcube=cubefromshape(oldpath, trend_emd)
ERROR: MethodError: no method matching rasterizepoly!(::Array{Int32,2}, ::Array{Union{Missing, Shapefile.Polygon},1}, ::NamedTuple{(:left, :right, :top, :bottom),NTuple{4,Float64}})
Closest candidates are:
rasterizepoly!(::Any, ::Array{#s86,1} where #s86<:GeoInterface.AbstractMultiPolygon, ::Any) at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:137
rasterizepoly!(::Any, ::Array{T<:GeoInterface.AbstractPoint,1}, ::Any; value, wrap) where T<:GeoInterface.AbstractPoint at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:144
rasterizepoly!(::Any, ::Shapefile.Polygon, ::Any; value) at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:181
Stacktrace:
[1] #rasterize!#27(::NamedTuple{(:left, :right, :top, :bottom),NTuple{4,Float64}}, ::Nothing, ::typeof(ESDL.Proc.rasterize!), ::Array{Int32,2}, ::String) at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:130
[2] (::getfield(ESDL.Proc, Symbol("#kw##rasterize!")))(::NamedTuple{(:bb,),Tuple{NamedTuple{(:left, :right, :top, :bottom),NTuple{4,Float64}}}}, ::typeof(ESDL.Proc.rasterize!), ::Array{Int32,2}, ::String) at ./none:0
[3] #cubefromshape_single#20(::Nothing, ::Type, ::typeof(ESDL.Proc.cubefromshape_single), ::String, ::RangeAxis{Float64,:Lon,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}, ::RangeAxis{Float64,:Lat,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}) at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:85
[4] cubefromshape_single(::String, ::RangeAxis{Float64,:Lon,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}, ::RangeAxis{Float64,:Lat,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}) at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:81
[5] #cubefromshape#22(::Nothing, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(cubefromshape), ::String, ::Vararg{Any,N} where N) at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:106
[6] cubefromshape at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:105 [inlined]
[7] #cubefromshape#21 at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:103 [inlined]
[8] cubefromshape(::String, ::ESDL.Cubes.ConcatCube{Union{Missing, Float32},3}) at /home/crem_fe/.julia/packages/ESDL/lS3UG/src/Proc/Shapes.jl:103
[9] top-level scope at REPL[19]:1I can load a cube and I can subset it. |
|
I just made a few tests and my normal computations work. |
|
I will have a look at what happens in |
|
Exporting the data into netcdf goes through, but the resulting netcdf file is not readable in QGIS, therefore, I can't check, whether the computations are doing what I expect. |
|
@felixcremer the cubefromshape bug should be fixed now. What is the exact reason that you can not read the data in QGIS, which metadata is missing? |
|
Thanks, the cubefrom shape is fixed. Invalid Data Source: /home/crem_fe/Daten/newcube_test.nc is not a valid or recognized data source. and also opening it with gdalinfo doesn't work. I don't know enough about netcdf internals to debug this. |
| import ..Cubes.ESDLZarr: ZArrayCube | ||
| using Base.Iterators: Iterators, product | ||
| using Zarr: ZArray, NoCompressor | ||
| using NetCDF: ncwrite, NcDim, NcVar, ncread, create, nccreate, ncputatt |
There was a problem hiding this comment.
Add NetCDF after the : so that we can use NetCDF.close later on, we need to do it like that, so that it doesn't name clash with Base.close
| cont_loop = Dict(ii=>axname(ax_cont[ii]) for ii in isplit:length(ax_cont)) | ||
| mapCube(writefun,r,length(ax_cont),cont_loop,filename,indims=incubes,include_loopvars=true,ispar=false,max_cache=5e8, | ||
| nthreads=[1]) | ||
| NetCDF.close(file) |
There was a problem hiding this comment.
This line should be restored, because this caused the problems with the exported NetCDF files.
There was a problem hiding this comment.
Depends on your NetCDF branch. I was testing on this meggart/NetCDF.jl#119 , but I agree that we should call sync here.
There was a problem hiding this comment.
I am not sure, whether I tested the right branches, but using the finalizer branch of NetCDF doesn't solve the opening problem of the exported nc files.
There was a problem hiding this comment.
Ok, this is strange. Does a call to NetCDF.sync() help in this case?
There was a problem hiding this comment.
Calling sync(file) doesn't help. the _close function is called six times in my example, but the file is only readable in QGIS when I close julia.
There was a problem hiding this comment.
Ok, I will do some more experiments to find out what is happening. I just realize there is a mixture high-level and low-level access going on here, which might be bad. WIll change this
|
If I try to export cubes with two categorical axes, this fails on this branch but works on master. ERROR: MethodError: no method matching push!(::Array{NetCDF.NcVar,2}, ::NetCDF.NcVar{Float64,1,6})
Closest candidates are:
push!(::Any, ::Any, ::Any) at abstractarray.jl:2159
push!(::Any, ::Any, ::Any, ::Any...) at abstractarray.jl:2160
push!(::Array{Any,1}, ::Any) at array.jl:875
...
Stacktrace:
[1] #create#40(::Dict{Any,Any}, ::UInt16, ::Bool, ::typeof(NetCDF.create), ::String, ::Array{NetCDF.NcVar,2}) at /home/crem_fe/.julia/dev/NetCDF/src/NetCDF.jl:804
[2] create(::String, ::Array{NetCDF.NcVar,2}) at /home/crem_fe/.julia/dev/NetCDF/src/NetCDF.jl:768
[3] #exportcube#14(::Dict{String,Int64}, ::Dict{String,Any}, ::typeof(exportcube), ::ESDL.Cubes.ESDLArray{Union{Missing, Float32},4,ESDL.Cubes.DiskArrayStack{Union{Missing, Float32},4,Zarr.ZArray{Union{Missing, Float32},3,Zarr.NoCompressor,Zarr.DirectoryStore},1},Array{CubeAxis,1}}, ::String) at /home/crem_fe/.julia/dev/ESDL/src/Proc/CubeIO.jl:91
[4] exportcube(::ESDL.Cubes.ESDLArray{Union{Missing, Float32},4,ESDL.Cubes.DiskArrayStack{Union{Missing, Float32},4,Zarr.ZArray{Union{Missing, Float32},3,Zarr.NoCompressor,Zarr.DirectoryStore},1},Array{CubeAxis,1}}, ::String) at /home/crem_fe/.julia/dev/ESDL/src/Proc/CubeIO.jl:81
[5] top-level scope at REPL[5]:1You can find an example cube here: |
felixcremer
left a comment
There was a problem hiding this comment.
Can you have a look at the saving problems with multiple categorical axes?
| common_offset(a::DiskArrays.GridChunks) = a.offset | ||
| function common_offset(a) | ||
| ntuple(ndims(a)) do idim | ||
| otherdims = setdiff(1:ndims(a),idim) |
There was a problem hiding this comment.
Could we combine this with the start of the common_size function into one helper function?
| isempty(newperm) ? subscube : permutedims(subscube,(newperm...,)) | ||
|
|
||
| function Base.map(op, incubes::AbstractCubeData...) | ||
| axlist=copy(caxes(incubes[1])) |
There was a problem hiding this comment.
Why do we need the copy here? This is not going to be changed in the following code?
| cont_loop = Dict(ii=>axname(ax_cont[ii]) for ii in isplit:length(ax_cont)) | ||
| mapCube(writefun,r,length(ax_cont),cont_loop,filename,indims=incubes,include_loopvars=true,ispar=false,max_cache=5e8, | ||
| nthreads=[1]) | ||
| NetCDF.close(file) |
There was a problem hiding this comment.
I am not sure, whether I tested the right branches, but using the finalizer branch of NetCDF doesn't solve the opening problem of the exported nc files.
This doesn't solve the test failure because some readblock! method is still missing.
I also changed the import Zarr to using Zarr: ... and sorted the imports into types and functions which are sorted alphabetically.
Wrap is a keyword which is projection specific and which gives the projection value at which the projection wraps into itself. The default values fail for non Lat Lon projections. Therefore, this makes the keyword available in the cubefromshape interface. This is an intermediate solution, because it would be better to have the projection as a metadata information of a cube, so that we could use this to set the wrapping values.
This reverts commit 6e778a2. Unfortunately, this pull request was not yet ready. I should have indicated that more clearly. I am reverting this for now and then I am going to push a tidied up version of the PR.
Clone is the keyword from the old package manger.
* Removed a lot of files * move some files * break more things * Clean up Proc module * Many fixes to get dataset creation running again * savecube to netcdf works * more improvements * Commit many more changes * Separate out API * some test fixes * switch to YAXArrayBase * read variables in a typed way * add ESDLArray export * Fixes to remap * remove NetCDF and Zarr dependency * delete some unused stuff * Do not permute cubes by default * Add concatenations * Fixes to findaxis * Tests pass again * Try to fix chunking * dont depend on reexport * remove reexport dep * dont forgegt using it * update test * Allow NamedTuples * minor changes * Finalsplit (#220) * Delete almost everything * More updates * Old tests for ESDL pass * More changes * Fix bugs in ESDL * Some fixes * Fix some warnings * tests should pass
* Removed a lot of files * move some files * break more things * Clean up Proc module * Many fixes to get dataset creation running again * savecube to netcdf works * more improvements * Commit many more changes * Separate out API * some test fixes * switch to YAXArrayBase * read variables in a typed way * add ESDLArray export * Fixes to remap * remove NetCDF and Zarr dependency * delete some unused stuff * Do not permute cubes by default * Add concatenations * Fixes to findaxis * Tests pass again * Try to fix chunking * dont depend on reexport * remove reexport dep * dont forgegt using it * update test * Allow NamedTuples * minor changes * Finalsplit (#220) * Delete almost everything * More updates * Old tests for ESDL pass * More changes * Fix bugs in ESDL * Some fixes * Fix some warnings * tests should pass
This is work in progress and prepares ESDL to be used in a wider context and on a variety of input formats. Here, we removed the different data cube types that existed so far and only provide a single
ESDLArrayon which Cube operations can happen. This data type contains the fielddatawhich can be anyAbstractArrayincluding Julia Arrays or DiskArrays. All lazy operations that existed so far, like permutedims, views, concatenation will happen on thedatalevel now and not create an extra Cube type.The current unit tests are passing, but experience says that this package is unfortunately under-tested, so there will still be bugs to find (and unit-tests to be added). @felixcremer maybe you want to give this branch a try and report what works and what doesn't for you?