Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 7766bc6

Browse filesBrowse files
AdamRJensenAdamRJensen
authored
Add retrieval function for BSRN data (#1254)
* Initial commit * Fix typo in doc string * Fix stickler * Update api.rst, __init__.py, and whatsnew * Major refactoring * Coverage for test_bsrn * Coverage for Warnings in case of no files avaiable * Fix stickler * Correct test_get_bsrn_bad_station test * Specify warning category * Update dates used in test_get_bsrn_no_files * Add secret credentials for testing * Documentation updates * Move line_no_dict 7 lines down * Add requires_bsrn_credentials to conftest.py * Add parsing of logical records 0300 and 0500 * Raise os.environ as ValueError for debugging * Export BSRN credentials in conda_linux.yml * Add parse_bsrn * Fix stickler and minor doc changes * Coverage for additional logical records * Refactor warnings in get_bsrn If only some files are missing, give only one warning with a list of missing files * Add Hint section * Add function for empty dataframe and restructure data docs * Add gri to list of variables * Coverage for records not found * Coverage for no logical records found * Formatting of data columns table * Merge read_ and get_bsrn in whatsnew * Changes from review by kandersol-nrel * Add lat/lon ISO 19115 convention to metadata * Add bio.seek(0) to get_bsrn()
1 parent f13d1b1 commit 7766bc6
Copy full SHA for 7766bc6

File tree

Expand file treeCollapse file tree

8 files changed

+535
-100
lines changed
Filter options
Expand file treeCollapse file tree

8 files changed

+535
-100
lines changed

‎ci/azure/conda_linux.yml

Copy file name to clipboardExpand all lines: ci/azure/conda_linux.yml
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ jobs:
3838
- script: |
3939
source activate test_env
4040
export NREL_API_KEY=$(nrelApiKey)
41+
export BSRN_FTP_USERNAME=$(BSRN_FTP_USERNAME)
42+
export BSRN_FTP_PASSWORD=$(BSRN_FTP_PASSWORD)
4143
pytest pvlib --remote-data --junitxml=junit/test-results.xml --cov --cov-report=xml --cov-report=html
4244
displayName: 'pytest'
4345
- task: PublishTestResults@2

‎docs/sphinx/source/api.rst

Copy file name to clipboardExpand all lines: docs/sphinx/source/api.rst
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -491,7 +491,9 @@ of sources and file formats relevant to solar energy modeling.
491491
iotools.read_pvgis_tmy
492492
iotools.get_pvgis_hourly
493493
iotools.read_pvgis_hourly
494+
iotools.get_bsrn
494495
iotools.read_bsrn
496+
iotools.parse_bsrn
495497
iotools.get_cams
496498
iotools.read_cams
497499
iotools.parse_cams

‎docs/sphinx/source/whatsnew/v0.9.0.rst

Copy file name to clipboardExpand all lines: docs/sphinx/source/whatsnew/v0.9.0.rst
+3-2Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -111,8 +111,9 @@ Enhancements
111111
:func:`~pvlib.iotools.get_pvgis_hourly` for reading and retrieving hourly
112112
solar radiation data and PV power output from PVGIS. (:pull:`1186`,
113113
:issue:`849`)
114-
* Add :func:`~pvlib.iotools.read_bsrn` for reading BSRN solar radiation data
115-
files. (:pull:`1145`, :issue:`1015`)
114+
* Add :func:`~pvlib.iotools.get_bsrn` and :func:`~pvlib.iotools.read_bsrn`
115+
for retrieving and reading BSRN solar radiation data files.
116+
(:pull:`1254`, :pull:`1145`, :issue:`1015`)
116117
* Add :func:`~pvlib.iotools.get_cams`,
117118
:func:`~pvlib.iotools.parse_cams`, and
118119
:func:`~pvlib.iotools.read_cams`

‎pvlib/data/variables_style_rules.csv

Copy file name to clipboardExpand all lines: pvlib/data/variables_style_rules.csv
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ dni_extra;direct normal irradiance at top of atmosphere (extraterrestrial)
77
dhi;diffuse horizontal irradiance
88
bhi;beam/direct horizontal irradiance
99
ghi;global horizontal irradiance
10+
gri;ground-reflected irradiance
1011
aoi;angle of incidence between :math:`90\deg` and :math:`90\deg`
1112
aoi_projection;cos(aoi)
1213
airmass;airmass

‎pvlib/iotools/__init__.py

Copy file name to clipboardExpand all lines: pvlib/iotools/__init__.py
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,9 @@
1515
from pvlib.iotools.pvgis import get_pvgis_tmy, read_pvgis_tmy # noqa: F401
1616
from pvlib.iotools.pvgis import read_pvgis_hourly # noqa: F401
1717
from pvlib.iotools.pvgis import get_pvgis_hourly # noqa: F401
18+
from pvlib.iotools.bsrn import get_bsrn # noqa: F401
1819
from pvlib.iotools.bsrn import read_bsrn # noqa: F401
20+
from pvlib.iotools.bsrn import parse_bsrn # noqa: F401
1921
from pvlib.iotools.sodapro import get_cams # noqa: F401
2022
from pvlib.iotools.sodapro import read_cams # noqa: F401
2123
from pvlib.iotools.sodapro import parse_cams # noqa: F401

‎pvlib/iotools/bsrn.py

Copy file name to clipboardExpand all lines: pvlib/iotools/bsrn.py
+405-89Lines changed: 405 additions & 89 deletions
Large diffs are not rendered by default.

‎pvlib/tests/conftest.py

Copy file name to clipboardExpand all lines: pvlib/tests/conftest.py
+13Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import warnings
44

55
import pandas as pd
6+
import os
67
from pkg_resources import parse_version
78
import pytest
89
from functools import wraps
@@ -82,6 +83,18 @@ def assert_frame_equal(left, right, **kwargs):
8283
reason='does not run on windows')
8384

8485

86+
try:
87+
# Attempt to load BSRN credentials used for testing pvlib.iotools.get_bsrn
88+
bsrn_username = os.environ["BSRN_FTP_USERNAME"]
89+
bsrn_password = os.environ["BSRN_FTP_PASSWORD"]
90+
has_bsrn_credentials = True
91+
except KeyError:
92+
has_bsrn_credentials = False
93+
94+
requires_bsrn_credentials = pytest.mark.skipif(
95+
not has_bsrn_credentials, reason='requires bsrn credentials')
96+
97+
8598
try:
8699
import statsmodels # noqa: F401
87100
has_statsmodels = True

‎pvlib/tests/iotools/test_bsrn.py

Copy file name to clipboardExpand all lines: pvlib/tests/iotools/test_bsrn.py
+107-9Lines changed: 107 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,25 +2,123 @@
22
tests for :mod:`pvlib.iotools.bsrn`
33
"""
44

5-
65
import pandas as pd
76
import pytest
7+
import os
8+
from pvlib.iotools import read_bsrn, get_bsrn
9+
from ..conftest import (DATA_DIR, RERUNS, RERUNS_DELAY, assert_index_equal,
10+
requires_bsrn_credentials)
11+
12+
13+
@pytest.fixture(scope="module")
14+
def bsrn_credentials():
15+
"""Supplies the BSRN FTP credentials for testing purposes.
816
9-
from pvlib.iotools import bsrn
10-
from ..conftest import DATA_DIR, assert_index_equal
17+
Users should obtain their own credentials as described in the `read_bsrn`
18+
documentation."""
19+
bsrn_username = os.environ["BSRN_FTP_USERNAME"]
20+
bsrn_password = os.environ["BSRN_FTP_PASSWORD"]
21+
return bsrn_username, bsrn_password
1122

1223

13-
@pytest.mark.parametrize('testfile,expected_index', [
14-
('bsrn-pay0616.dat.gz',
15-
pd.date_range(start='20160601', periods=43200, freq='1min', tz='UTC')),
16-
('bsrn-lr0100-pay0616.dat',
17-
pd.date_range(start='20160601', periods=43200, freq='1min', tz='UTC')),
24+
@pytest.fixture
25+
def expected_index():
26+
return pd.date_range(start='20160601', periods=43200, freq='1min',
27+
tz='UTC')
28+
29+
30+
@pytest.mark.parametrize('testfile', [
31+
('bsrn-pay0616.dat.gz'),
32+
('bsrn-lr0100-pay0616.dat'),
1833
])
1934
def test_read_bsrn(testfile, expected_index):
20-
data = bsrn.read_bsrn(DATA_DIR / testfile)
35+
data, metadata = read_bsrn(DATA_DIR / testfile)
36+
assert_index_equal(expected_index, data.index)
37+
assert 'ghi' in data.columns
38+
assert 'dni_std' in data.columns
39+
assert 'dhi_min' in data.columns
40+
assert 'lwd_max' in data.columns
41+
assert 'relative_humidity' in data.columns
42+
43+
44+
def test_read_bsrn_logical_records(expected_index):
45+
# Test if logical records 0300 and 0500 are correct parsed
46+
# and that 0100 is not passed when not specified
47+
data, metadata = read_bsrn(DATA_DIR / 'bsrn-pay0616.dat.gz',
48+
logical_records=['0300', '0500'])
49+
assert_index_equal(expected_index, data.index)
50+
assert 'lwu' in data.columns
51+
assert 'uva_global' in data.columns
52+
assert 'uvb_reflected_std' in data.columns
53+
assert 'ghi' not in data.columns
54+
55+
56+
def test_read_bsrn_bad_logical_record():
57+
# Test if ValueError is raised if an unsupported logical record is passed
58+
with pytest.raises(ValueError, match='not in'):
59+
read_bsrn(DATA_DIR / 'bsrn-lr0100-pay0616.dat',
60+
logical_records=['dummy'])
61+
62+
63+
def test_read_bsrn_logical_records_not_found():
64+
# Test if an empty dataframe is returned if specified LRs are not present
65+
data, metadata = read_bsrn(DATA_DIR / 'bsrn-lr0100-pay0616.dat',
66+
logical_records=['0300', '0500'])
67+
assert data.empty # assert that the dataframe is empty
68+
assert 'uva_global' in data.columns
69+
assert 'uvb_reflected_std' in data.columns
70+
assert 'uva_global_max' in data.columns
71+
assert 'dni' not in data.columns
72+
assert 'day' not in data.columns
73+
74+
75+
@requires_bsrn_credentials
76+
@pytest.mark.remote_data
77+
@pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
78+
def test_get_bsrn(expected_index, bsrn_credentials):
79+
# Retrieve irradiance data from the BSRN FTP server
80+
# the TAM station is chosen due to its small file sizes
81+
username, password = bsrn_credentials
82+
data, metadata = get_bsrn(
83+
start=pd.Timestamp(2016, 6, 1),
84+
end=pd.Timestamp(2016, 6, 29),
85+
station='tam',
86+
username=username,
87+
password=password,
88+
local_path='')
2189
assert_index_equal(expected_index, data.index)
2290
assert 'ghi' in data.columns
2391
assert 'dni_std' in data.columns
2492
assert 'dhi_min' in data.columns
2593
assert 'lwd_max' in data.columns
2694
assert 'relative_humidity' in data.columns
95+
96+
97+
@requires_bsrn_credentials
98+
@pytest.mark.remote_data
99+
@pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
100+
def test_get_bsrn_bad_station(bsrn_credentials):
101+
# Test if KeyError is raised if a bad station name is passed
102+
username, password = bsrn_credentials
103+
with pytest.raises(KeyError, match='sub-directory does not exist'):
104+
get_bsrn(
105+
start=pd.Timestamp(2016, 6, 1),
106+
end=pd.Timestamp(2016, 6, 29),
107+
station='not_a_station_name',
108+
username=username,
109+
password=password)
110+
111+
112+
@requires_bsrn_credentials
113+
@pytest.mark.remote_data
114+
@pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
115+
def test_get_bsrn_no_files(bsrn_credentials):
116+
username, password = bsrn_credentials
117+
# Test if Warning is given if no files are found for the entire time frame
118+
with pytest.warns(UserWarning, match='No files'):
119+
get_bsrn(
120+
start=pd.Timestamp(1990, 6, 1),
121+
end=pd.Timestamp(1990, 6, 29),
122+
station='tam',
123+
username=username,
124+
password=password)

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.