Thank you @mikofski for the excellent start to an important discussion!

I see two questions here:

1. Is a publication necessary?
2. Is a publication sufficient?

I have been thinking that we could establish a (hopefully) simple points system for proposed features to achieve a more balanced assessment of suitability for inclusion in pvlib as opposed to accept/reject on the basis of this one specific criterion.

Aspects to be rated could be:

• documentation (availability and quality of information)
  • open access vs. paywall
  • included in PR ++
  • complete and sufficient to reproduce or recreate code
• validation
  • level of peer review
  • availability of data and/or code to replicate results
  • evidence of wide-spread use
• usefulness for
  • industry
  • research
  • learning

Of course there is a process question too: who evaluates and who decides?

Full disclosure: I would like to convince other maintainers to accept #1878 for inclusion in pvlib--it is currently blocked.

Hello folks, this is a great discussion and thought that I'd pop back out of the woodwork with some thoughts here. I'm admittedly very out the the loop on a lot of the recent developments on the package, so apologies if any of this is already covered.

When I think about pvlib, I think that the core use-case for it is that it is useful. For most people most of the time, that probably means that core workflows in the package are based on peer-reviewed and archival references, because it is useful to be able to trust their outputs as a part of a larger project, and to know that "someone" has looked at them.

However, for other users it is likely useful to have workflows and tools that haven't yet been reviewed but that have been developed specifically because they are useful. I would also imagine there are a large number of applications that will stand on the border between something that would make sense to peer review, and something that just makes sense in the flow of the model or workflow being implemented.

I would also suspect, as has been pointed out in this thread, that implementation into pvlib and running through the wide range of conditions and test cases for an algorithm is probably a more rigorous test of an algorithm than is required to pass peer review . I think there have been many good examples of the community finding and correcting issues with published models. An aspiration here might be that the pvlib implementation (with associated tests) is the reference implementation of a function (and the output that is incentivized by funding bodies), not the publication.
​
So, perhaps there are two (and maybe three) types of criterion for features:

• Something that affects a core workflow: Core workflows as defined by this group, likely including model chain and other areas that are commonly used as part of larger workflows. Inclusion of updates to these workflows would require peer review, or possible a flag/warning if not based on a peer reviewed document. (e.g. you could run a homebrew backtracking algorithm that is in the package, but the standard implementation of pvlib will throw a warning). At some point in the future, this could also require physical test case development.

• Functions outside core workflows: This does not require peer review, and flagging inside documentation that the function is not published may be sufficient for users of different technical levels to interpret. (call it a Beta or Labs feature, if we want to sound like a billionaire)

• Utility functions? If it is useful, it can be included without reference, as @williamhobbs has said

​​
Possibly this could achieve different streams of review, where items that are likely to impact core workflows receive a higher level of scrutiny - and review through a group as @mikofski has described - but other functions are able to be implemented and "put through their paces" without having to hit this threshold.

Any ideas how to wrap this discussion up?

I propose a policy for references:

• The intent of requiring a reference is to ensure that the pvlib code implements the authors' algorithms as faithfully as possible, and that pvlib maintainers and others can resolve questions about pvlib code.
• References are required for functions that implement a model (e.g., clear-sky irradiance, cell temperature, mismatch loss) or that retrieve or read external data.
• References should be archival documents that explain the model or data that is being read.
• References may include code but should not exclusively comprise code. The concern here is that the algorithm may not be understandable without executing old code, which may be a problem for future maintainers.
• References may include URLs but should not exclusively comprise webpages. Exceptions may be made for popular pvlib content when no other reference material is available.