Code Review

The pull request introduces a robust Docker-based sandbox infrastructure for RL training, including two new environments (SandboxEnv and SandboxLMEnv). The implementation is well-structured, leveraging an abstract base class for backends and providing stateful bash execution. I have identified a few areas for improvement, primarily around path safety in shell commands, hardcoded resource limits, and minor path manipulation logic.

        self._container.exec_run(["bash", "-c", f"echo '{encoded_content}' | base64 -d > {shlex.quote(path)}"])

    def __init__(self, image: str = "ubuntu:24.04", timeout: int = 1800, mem_limit: str = "4g"):
        """
        Args:
            image: Docker image to use (default: ubuntu:24.04)
            timeout: Per-command timeout in seconds (default: 1800 / 30 min)
            mem_limit: Memory limit for the container (default: 4g)
        """
        self._image = image
        self._timeout = timeout
        self._mem_limit = mem_limit
        self._container = None
        self._client = None

        self._container = self._client.containers.run(
            self._image, command="sleep infinity", detach=True, remove=True, mem_limit=self._mem_limit, memswap_limit=self._mem_limit
        )

        exit_code, output = self._container.exec_run(["cat", "--", path])

        import posixpath
        parent = posixpath.dirname(path)

Code Review

This pull request introduces a Docker-based sandbox for code execution in RL environments, adding a SandboxBackend abstraction, a DockerBackend implementation, and a GenericSandboxEnv that provides execute_bash and file editing tools. The changes are well-structured and the implementation is solid. I've provided a few suggestions to improve the robustness of the DockerBackend implementation, particularly around file I/O, and a minor correction to a system prompt file.

Code Review

The pull request introduces a Docker sandbox backend and two new environments, SandboxEnv and SandboxLMEnv, for RL training with code execution. It also updates the CHANGELOG.md, open_instruct/environments/__init__.py, open_instruct/environments/tools/tools.py, and pyproject.toml files to integrate these new features and dependencies. A new debug script sandbox_lm_1gpu.sh and its corresponding system prompt file sandbox_lm_system_prompt.txt are added. The changes enhance the project's capability for code-aware RL agents by providing a sandboxed execution environment.

Code Review

This pull request introduces a Docker sandbox backend and two new environments, SandboxEnv and SandboxLMEnv, for RL training with code execution. The changes include new files for backend implementation (backends.py) and the sandbox environments (generic_sandbox.py), updates to __init__.py and tools.py to register these new components, and a new debug script (sandbox_lm_1gpu.sh) along with its system prompt. The pyproject.toml and uv.lock files have also been updated to include the docker dependency. The code generally looks good, with clear abstractions for the sandbox backend and well-defined tool interfaces. I've identified a few areas for improvement regarding error handling, logging, and consistency in the CHANGELOG.md entry.

Code Review

This pull request introduces a robust Docker-based sandbox infrastructure for RL training, featuring a stateful bash environment and a file editor tool. The implementation of state persistence in Docker containers via environment variable and CWD tracking is clever. However, there are a few issues regarding command defaults, potential hangs in file reading, and image-specific dependencies that should be addressed to ensure reliability across different Docker images.

Seems fine -- one question is can we make docker an optional dependency? Not sure how main path you see this to be, especially given all the try-except on importing docker. If its required, we dont need try excepts?