Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 75d800a

Browse filesBrowse files
AdamRJensenkandersolar
AdamRJensen
and
kandersolar
authored
Add encoding parameter to read_tmy3 (#1737)
* Add encoding to read_tmy3 * Update v0.9.6.rst * Update pvlib/iotools/tmy.py Co-authored-by: Kevin Anderson <kevin.anderso@gmail.com> * Add Andy Lam to contributor list * Remove try/except statement * Inlining _parse_tmy3 * Add DOIs * Use split(':') instead of [:2] to extract hour * Add parsing of minutes * Fix doi rendering * Fix doi rendering in read_tmy2 * Revert "Merge branch 'main' into expose_encoding_tmy3" This reverts commit 13d41af, reversing changes made to c6e8854. * Revert "Merge branch 'expose_encoding_tmy3' of https://github.com/AdamRJensen/pvlib-python into expose_encoding_tmy3" This reverts commit f9c3953, reversing changes made to 8e1f488. * This reverts commit d868bb9. Revert to f638a4d * Revert more stuff manually * Add lines add end of file * Add encoding=None to function def line --------- Co-authored-by: Kevin Anderson <kevin.anderso@gmail.com>
1 parent 5119b42 commit 75d800a
Copy full SHA for 75d800a

File tree

Expand file treeCollapse file tree

3 files changed

+26
-23
lines changed
Filter options
Expand file treeCollapse file tree

3 files changed

+26
-23
lines changed

‎docs/sphinx/source/whatsnew/v0.9.6.rst

Copy file name to clipboardExpand all lines: docs/sphinx/source/whatsnew/v0.9.6.rst
+3Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ Deprecations
2323

2424
Enhancements
2525
~~~~~~~~~~~~
26+
* Add optional encoding parameter to :py:func:`pvlib.iotools.read_tmy3`.
27+
(:issue:`1732`, :pull:`1737`)
2628
* Added function to retrieve horizon data from PVGIS
2729
:py:func:`pvlib.iotools.get_pvgis_horizon`. (:issue:`1290`, :pull:`1395`)
2830
* Added ``map_variables`` argument to the :py:func:`pvlib.iotools.read_tmy3` in
@@ -74,5 +76,6 @@ Contributors
7476
* Siddharth Kaul (:ghuser:`k10blogger`)
7577
* Kshitiz Gupta (:ghuser:`kshitiz305`)
7678
* Stefan de Lange (:ghuser:`langestefan`)
79+
* Andy Lam (:ghuser:`@andylam598`)
7780
* :ghuser:`ooprathamm`
7881
* Kevin Anderson (:ghuser:`kandersolar`)

‎pvlib/iotools/tmy.py

Copy file name to clipboardExpand all lines: pvlib/iotools/tmy.py
+21-22Lines changed: 21 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@
2424
}
2525

2626

27-
def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
27+
def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None,
28+
encoding=None):
2829
"""Read a TMY3 file into a pandas dataframe.
2930
3031
Note that values contained in the metadata dictionary are unchanged
@@ -50,6 +51,11 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
5051
If ``True``, apply standard names to TMY3 columns. Typically this
5152
results in stripping the units from the column name.
5253
Cannot be used in combination with ``map_variables``.
54+
encoding : str, optional
55+
Encoding of the file. For files that contain non-UTF8 characters it may
56+
be necessary to specify an alternative encoding, e.g., for
57+
SolarAnywhere TMY3 files the encoding should be 'iso-8859-1'. Users
58+
may also consider using the 'utf-8-sig' encoding.
5359
5460
Returns
5561
-------
@@ -58,7 +64,7 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
5864
data : DataFrame
5965
A pandas dataframe with the columns described in the table
6066
below. For more detailed descriptions of each component, please
61-
consult the TMY3 User's Manual ([1]_), especially tables 1-1
67+
consult the TMY3 User's Manual [1]_, especially tables 1-1
6268
through 1-6.
6369
6470
metadata : dict
@@ -187,14 +193,12 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
187193
""" # noqa: E501
188194
head = ['USAF', 'Name', 'State', 'TZ', 'latitude', 'longitude', 'altitude']
189195

190-
try:
191-
with open(str(filename), 'r') as fbuf:
192-
firstline, data = _parse_tmy3(fbuf)
193-
# SolarAnywhere files contain non-UTF8 characters and may require
194-
# encoding='iso-8859-1' in order to be parsed
195-
except UnicodeDecodeError:
196-
with open(str(filename), 'r', encoding='iso-8859-1') as fbuf:
197-
firstline, data = _parse_tmy3(fbuf)
196+
with open(str(filename), 'r', encoding=encoding) as fbuf:
197+
# header information on the 1st line (0 indexing)
198+
firstline = fbuf.readline()
199+
# use pandas to read the csv file buffer
200+
# header is actually the second line, but tell pandas to look for
201+
data = pd.read_csv(fbuf, header=0)
198202

199203
meta = dict(zip(head, firstline.rstrip('\n').split(",")))
200204
# convert metadata strings to numeric types
@@ -206,8 +210,10 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
206210

207211
# get the date column as a pd.Series of numpy datetime64
208212
data_ymd = pd.to_datetime(data['Date (MM/DD/YYYY)'], format='%m/%d/%Y')
213+
# extract minutes
214+
minutes = data['Time (HH:MM)'].str.split(':').str[1].astype(int)
209215
# shift the time column so that midnite is 00:00 instead of 24:00
210-
shifted_hour = data['Time (HH:MM)'].str[:2].astype(int) % 24
216+
shifted_hour = data['Time (HH:MM)'].str.split(':').str[0].astype(int) % 24
211217
# shift the dates at midnight (24:00) so they correspond to the next day.
212218
# If midnight is specified as 00:00 do not shift date.
213219
data_ymd[data['Time (HH:MM)'].str[:2] == '24'] += datetime.timedelta(days=1) # noqa: E501
@@ -225,7 +231,8 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
225231
data_ymd.iloc[-1] = data_ymd.iloc[-1].replace(year=coerce_year+1)
226232
# NOTE: as of pvlib-0.6.3, min req is pandas-0.18.1, so pd.to_timedelta
227233
# unit must be in (D,h,m,s,ms,us,ns), but pandas>=0.24 allows unit='hour'
228-
data.index = data_ymd + pd.to_timedelta(shifted_hour, unit='h')
234+
data.index = data_ymd + pd.to_timedelta(shifted_hour, unit='h') \
235+
+ pd.to_timedelta(minutes, unit='min')
229236
# shouldnt' specify both recolumn and map_variables
230237
if recolumn is not None and map_variables is not None:
231238
msg = "`map_variables` and `recolumn` cannot both be specified"
@@ -252,15 +259,6 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
252259
return data, meta
253260

254261

255-
def _parse_tmy3(fbuf):
256-
# header information on the 1st line (0 indexing)
257-
firstline = fbuf.readline()
258-
# use pandas to read the csv file buffer
259-
# header is actually the second line, but tell pandas to look for
260-
data = pd.read_csv(fbuf, header=0)
261-
return firstline, data
262-
263-
264262
def _recolumn(tmy3_dataframe):
265263
"""
266264
Rename the columns of the TMY3 DataFrame.
@@ -328,7 +326,7 @@ def read_tmy2(filename):
328326
data : DataFrame
329327
A dataframe with the columns described in the table below. For a
330328
more detailed descriptions of each component, please consult the
331-
TMY2 User's Manual ([1]_), especially tables 3-1 through 3-6, and
329+
TMY2 User's Manual [1]_, especially tables 3-1 through 3-6, and
332330
Appendix B.
333331
334332
metadata : dict
@@ -430,6 +428,7 @@ def read_tmy2(filename):
430428
----------
431429
.. [1] Marion, W and Urban, K. "Wilcox, S and Marion, W. "User's Manual
432430
for TMY2s". NREL 1995.
431+
:doi:`10.2172/87130`
433432
""" # noqa: E501
434433
# paste in the column info as one long line
435434
string = '%2d%2d%2d%2d%4d%4d%4d%1s%1d%4d%1s%1d%4d%1s%1d%4d%1s%1d%4d%1s%1d%4d%1s%1d%4d%1s%1d%2d%1s%1d%2d%1s%1d%4d%1s%1d%4d%1s%1d%3d%1s%1d%4d%1s%1d%3d%1s%1d%3d%1s%1d%4d%1s%1d%5d%1s%1d%10d%3d%1s%1d%3d%1s%1d%3d%1s%1d%2d%1s%1d' # noqa: E501

‎pvlib/tests/iotools/test_tmy.py

Copy file name to clipboardExpand all lines: pvlib/tests/iotools/test_tmy.py
+2-1Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,8 @@ def test_solaranywhere_tmy3(solaranywhere_index):
121121
# The SolarAnywhere TMY3 format specifies midnight as 00:00 whereas the
122122
# NREL TMY3 format utilizes 24:00. The SolarAnywhere file is therefore
123123
# included to test files with 00:00 timestamps are parsed correctly
124-
data, meta = tmy.read_tmy3(TMY3_SOLARANYWHERE, map_variables=False)
124+
data, meta = tmy.read_tmy3(TMY3_SOLARANYWHERE, encoding='iso-8859-1',
125+
map_variables=False)
125126
pd.testing.assert_index_equal(data.index, solaranywhere_index)
126127
assert meta['USAF'] == 0
127128
assert meta['Name'] == 'Burlington United States'

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.