Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Add a Linux Devcontainer for osquery developers #7992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 19 commits into
base: master
Choose a base branch
Loading
from
Open

Add a Linux Devcontainer for osquery developers #7992

wants to merge 19 commits into from

Conversation

NickBorgers
Copy link

@NickBorgers NickBorgers commented Apr 16, 2023 •
edited
Loading

This PR adds a Linux Devcontainer for OSQuery. While this change will only help folks trying to build and test changes for osquery from a Linux development environment, this has been useful to me in making changes to osquery myself. It's possible this doesn't belong in the main repo or something, but I'm less familiar with storing something like this outside the main repository; I know the buildchain is to some degree such a separate codebase.

Devcontainers are a Microsoft spec and are currently (to my knowledge) only supported in IDEs provided by Microsoft (VSCode and GitHub Codespaces), however fully independent CLI clients are possible. I've personally been using VSCode, though I don't see this change as coupling osquery development to VSCode or Microsoft.

There's a YouTube Video demo of this functionality with VSCode here.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Apr 16, 2023 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

NickBorgers
NickBorgers commented Apr 16, 2023
View reviewed changes
.devcontainer/host_setup.sh
@@ -0,0 +1,25 @@
#!/bin/sh
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole script is basically a duplicate of what's published here: https://osquery.readthedocs.io/en/stable/development/building/#linux-ubuntu-18

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes a third place these live.

  1. Docs
  2. CI pipeline
  3. here

Is there a way we can unify these?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reference, this is what @directionless is referring to: https://github.com/osquery/osquery/blob/master/tools/ci/osquery-ubuntu18.04-toolchain.dockerfile

The Docker image we create for the CI though is particularly targeted; also the version of the distribution there is a bit arbitrary (and not updated continuosly just for maintenance reasons), while in a devenv one might want to use the latest.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do we think of this possible change to the docs?
https://github.com/NickBorgers/osquery/blob/30c2cccc65b44cd85af8d70573f40f14e5b50df3/docs/wiki/development/building.md#linux-ubuntu-18

NickBorgers
NickBorgers commented Apr 16, 2023
View reviewed changes
.devcontainer/host_setup.sh
sudo apt update

# Install the prerequisites
sudo apt install -y --no-install-recommends wget sudo xz-utils nano bc git python3 bison flex make cppcheck gdb
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I added here was gdb, as I was trying to debug some behavior I find odd in how osquery applies the DISTINCT constraint.

NickBorgers
NickBorgers commented Apr 16, 2023
View reviewed changes
Makefile
@@ -0,0 +1,33 @@
.DEFAULT_GOAL := help
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a Makefile here is a bit sketchy, but technically avoids stomping on the CMake and generated build/Makefile

I think it's nice to capture some of the test instructions from here, and I leverage this in the setup of the Devcontainer.

@NickBorgers
Copy link
Author

/easycla

NickBorgers
NickBorgers commented Apr 16, 2023
View reviewed changes
.devcontainer/host_setup.sh Outdated Show resolved Hide resolved
@NickBorgers NickBorgers marked this pull request as ready for review April 16, 2023 23:35
@NickBorgers NickBorgers requested review from a team as code owners April 16, 2023 23:35
zwass
zwass reviewed Apr 17, 2023
View reviewed changes
Copy link
Member

@zwass zwass left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks awesome! I use VSCode but haven't tried out the devcontainer stuff. Shall we consider the version change I proposed below and then merge this either way?

NickBorgers and others added 3 commits April 17, 2023 18:52
@NickBorgers @zwass
Adopt suggestion from @zwass
e32b060 
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
@NickBorgers
fix use of Makefile in devcontainer startup
3702482
@NickBorgers
Merge pull request #2 from NickBorgersOnLowSecurityNode/master
415faee 
fix use of Makefile in devcontainer startup
@NickBorgers
Copy link
Author

First, I found a bug in my devcontainer.json file which I didn't catch when testing previously. It took me spinning it up on a totally fresh box; not really sure how that happened.

Second I finally ran cd build; cmake --build . --target test in its entirety and I'm getting two test failures:

vscode ➜ /workspaces/osquery (master) $ cat build/Testing/Temporary/LastTestsFailed.log 
85:tools_tests_testfschangestable
86:tests_integration_tables-test

I'm going to go back to a full VM and try to see what is different.

@NickBorgers
Copy link
Author

NickBorgers commented Apr 20, 2023 •
edited
Loading

On a fresh VM (Ubuntu 22.04) I cloned my fork and ran:

nborgers@osquerybuild:~/osquery$ history
    1  git clone https://github.com/NickBorgers/osquery.git
    2  cd osquery/
    3  make setup
    4  sudo apt install make
    5  make setup
    6  make cmake
    7  make build
    8  make
    9  make test.all
   10  cat build/Testing/Temporary/LastTestsFailed.log
   11  history

And one of these tests is still failing; so I don't think it's related to being in a Devcontainer.

nborgers@osquerybuild:~/osquery$ cat build/Testing/Temporary/LastTestsFailed.log
85:tools_tests_testfschangestable

I am going to try and look at what is different about tests_integration_tables-test, I recall seeing something about a kernel sanity check in the logs in my Devcontainer. Seems reasonable that a container is having some limitations interacting with the the kernel, but I tried running with privileged=True as well.

One thing I'm struggling with a bit is finding the command to run these individual tests by name. Looks like I will need to learn a bit of ctest.

@mike-myers-tob mike-myers-tob changed the title Add Devcontainer Add a Linux Devcontainer for osquery developers May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Smjert Smjert Smjert left review comments

@directionless directionless directionless left review comments

@zwass zwass zwass left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

development Linux

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@NickBorgers @Smjert @directionless @zwass @mike-myers-tob
Morty Proxy This is a proxified and sanitized view of the page, visit original site.