Summary

Provide an interactive tool to evaluate declarations, statements, and expressions of the Java programming language, together with an API so that other applications can leverage this functionality.

Goals

The JShell API and tool will provide a way to interactively evaluate declarations, statements, and expressions of the Java programming language within the JShell state. The JShell state includes an evolving code and execution state. To facilitate rapid investigation and coding, statements and expressions need not occur within a method, expressions need not have side-effects, variables need not occur within a class, and methods need not occur within a class or interface.

The jshell tool will be a command-line tool with features to ease interaction including: a history with editing, tab-completion, automatic addition of needed terminal semicolons, and configurable predefined imports and definitions.

Non-Goals

A new interactive language is not the goal: All accepted input must match grammar productions in the Java Language Specification (JLS). Further, within an appropriate surrounding context, all accepted input must be valid Java code (JShell will automatically provide that surrounding context -- the "wrapping"). That is, if X is an input that JShell accepts (as opposed to rejects with error) then there is an A and B such that AXB is a valid program in the Java programming language.

Out of scope are graphical interfaces, debugger support, and IDE-like functionality.

Motivation

Immediate feedback is important when learning a programming language. The number one reason schools cite for moving away from Java as a teaching language is that other languages have a "REPL" and have far lower bars to an initial "Hello, world!" program. A Read-Eval-Print Loop (REPL) is an interactive programming tool which loops, continually reading user input, evaluating the input, and printing the value of the input or a description of the state change the input caused. Scala, Ruby, JavaScript, Haskell, Clojure, and Python all have REPLs and all allow small initial programs. JShell adds REPL functionality to the Java platform.

Exploration of coding options is also important for developers prototyping code or investigating a new API. Interactive evaluation is vastly more efficient in this regard than edit/compile/execute and System.out.println.

Without the ceremony of class Foo { public static void main(String[] args) { ... } }, learning and exploration is streamlined.

Description

Functionality

The JShell API will provide all of JShell's evaluation functionality. The code fragments that are input to the API are referred to as "snippets". The jshell tool will also use the JShell completion API to determine when input is incomplete (and the user must be prompted for more) or would be complete if a semicolon were added (in which case the tool will append the semicolon). The tool will have a set of commands for query, saving and restoring work, and configuration. Commands will be distinguished from snippets by a leading slash.

Snippets

A snippet must correspond to one of the following Java Language Specification (JLS) syntax productions:

• ImportDeclaration
• ClassDeclaration
• InterfaceDeclaration
• MethodDeclaration
• FieldDeclaration
• Statement
• Primary

In a top-level declaration, the access modifiers (public, protected, and private), and the modifiers final and static are not allowed and are ignored with a warning; the modifiers default and synchronized are not allowed, and the modifier abstract is allowed only on classes.

The break, continue, and return statements have no appropriate context at the top-level, and are not allowed.

The above modifiers and statements are, of course, allowed within a nested context in which they are valid. For example:

class C { public static final int m() { return 43; } }

Note that a PackageDeclaration is not allowed. All JShell code is placed in the transient jshell package.

State

The JShell state is held in an instance of JShellState. A snippet is evaluated in a JShellState with the eval(...) method, producing an error, declaring code, or executing a statement or expression. In the case of a variable with an initializer, both declaration and execution occur. An instance of JShellState contains previously-defined and modified variables, methods, and classes, previously-defined import declarations, the side-effects of previously-entered statements and expressions (including variable initializers), and external code bases.

Unless explicitly differentiated, for the purpose of this document “class” is meant in the sense used in the Java Virtual Machine Specification (JVMS), which includes JLS classes, interfaces, enums, and annotation interfaces.

Wrapping

The elements of code users would wish to explore include variables, methods, classes, statements, expressions, and imports. Since, of these, only classes and imports can occurs alone (at the top-level of a Java programming language program), an artificial consistent context is needed, as follows:

• Variables, Methods, and Classes
  • As static members of a synthetic class
• Expressions and Statements
  • As expressions and statements within a synthetic method within a synthetic class
• Imports
  • In their normal top-level context

Modification

Since the desired use is exploration, the declarations (variables, methods, and classes) must be able to evolve over time while, at the same time, preserving evaluated data. One choice would be to make a changed declaration a new additional entity in some or all cases, but that is certain to be confusing and does not play well with exploring the interaction between declarations. In JShell, each unique declaration key has exactly one declaration at any given time. For variables and classes the unique declaration key is the name, and, in support of overloading, the unique declaration key for methods is the name and the parameter types (to allow for overloading). As this is Java, variable, methods, and classes each have their own name spaces.

Forward reference

Within the body of a class, references to members which will appear later in the class can appear; this is a forward reference. As code is entered and evaluated sequentially in JShell, these references will be temporarily unresolved. In some cases, for example mutual recursion, forward reference is required. This can also occur in exploratory programming while entering code, for example, realizing that another (so far unwritten) method should be called. JShell supports forward references in method bodies and, within a class, within method bodies or field initializers. Forward references in the signature (variable type and method return or parameter types) is not supported.

Dependencies

The code state is kept up-to-date and consistent; that is, when a snippet is evaluated, any changes to dependent snippets are immediately propagated.

When a snippet is successfully declared, the declaration will be one of three kinds: Added, Modified, or Replaced. A snippet is Added if it is the first declaration with that key. A snippet is Replaced if its key matches a previous snippet, but their signatures differ. A snippet is Modified if its key matches a previous snippet and their signatures match; in this case, no dependent snippets are impacted. In both the Modified and Replaced cases the previous snippet is no longer part of the code state.

When a snippet is Added it may be providing an unresolved reference. When a snippet is Replaced it may update an existing snippet. For example, if a method's return type is declared to be of class C and then class C is Replaced then the method's signature has changed and the method must be Replaced. Note: This can cause previously-valid methods or classes to become invalid.

The desire is that user data persist whenever possible. This is attained except in the case of variable Replace. When a variable is replaced, either directly by the user or indirectly via a dependency update, the variable is set to its default value (null since this can only occur with reference variables).

When a declaration is invalid, either because of a forward-reference or becoming invalid through an update, the declaration is "corralled". A corralled declaration can be used in other declarations and code, however, if an attempt is made to execute it a runtime exception will occur which will explain the unresolved references or other issues.

Implementation

The implementation will make every effort to leverage the accuracy and engineering effort of the existing language support in the JDK. The JShell state is modeled as a JVM instance. Code analysis and the production of executable code will be performed by the Java Compiler (javac) through the Compiler API. Code replacement will use the Java Debug Interface (JDI).

Parsing of raw snippets (i.e., snippets that have not been wrapped) will be done using the Compiler API with a small subclassing of the parser to allow raw snippets. The resulting information will be used to wrap the snippet into a valid compilation unit including a class declaration with imports for previously evaluated code. Further analysis and generation of the class file will be done with unmodified instances of the Java compiler. Wrapped source and the generated class files will be kept in memory and never written to storage. Class files will be sent over a socket to the remote process. A remote agent will handle loading and execution. Replacement will be done via the JDI VirtualMachine.redefineClasses() facility.

Tab-completion analysis will also use the Compiler API. Completion detection will use the javac lexer, custom and table-driven code. The jshell tool will use 'jline2' for console input, editing, and history.

Naming

• Module
  • jdk.jshell
• Tool launcher
  • jshell
• API package
  • jdk.jshell
• API implementation package
  • jdk.internal.jshell.impl
• API implementation remote agent package
  • jdk.internal.jshell.impl.remote
• Tool package
  • jdk.internal.jshell.tool
• OpenJDK Project
  • Kulla

Alternatives

A simpler alternative is just to provide a batch scripting wrapper without interactive/update support.

Another alternative is to maintain the status quo: Use another language or use a third-party REPL such as BeanShell, though that particular REPL has been dormant for many years, is based on JDK 1.3, and makes arbitrary changes to the language.

Many IDEs, for example the NetBeans debugger and BlueJ's CodePad, provide mechanisms to interactively evaluate expressions. Preserved context and code remains class-based, and method granularity is not supported. They use specially crafted parsers/interpreters.

Testing

The API facilitates detailed point testing. A test framework makes writing tests straight-forward.

Because the evaluation and query functionality of the tool is built on the API, most testing is of the API. Command testing and sanity testing of the tool is, however, also needed. The tool is built with hooks for a testing harness, which is used for tool testing.

Tests are comprised of three parts:

1. Tests for the API. These tests cover both positive and negative cases. Each public method must be covered by tests which include adding variables, methods, and class, redefining them, etc.

2. Testing of jshell. These tests check that jshell commands and compilation, and execution of Java code, have correct behavior.

3. Stress testing. To ensure that jshell can compile all allowed Java snippets, correct Java code from the JDK itself will be used. These tests will parse the sources, feed code chunks to the API, and test the behavior of the API.

Risks and Assumptions

Forward reference within class bodies has not yet been implemented and adequate methods of intercepting a corralled class may not be possible.

Some code-breaking changes do not yet allow the complete propagation of updates. For example, if a class which is used in other declarations is defined as extending another class, and then the extending class is changed to be an interface. Mechanisms may not be found to handle this elegantly.

Dependences

The jshell tool uses the jline2 library, which is a Java library for handling console input. jline2 will need to be made accessible to and probably rolled into the JDK.

JEP 159, if implemented, may be of use.