semgrep-network-broker

NOTE: These docs are in-progress. Feel free to direct any questions / feedback / improvements to your private channel on the Semgrep slack!

The Semgrep Network Broker facilitates secure access between Semgrep and a private network.

The broker accomplishes this by establishing a Wireguard VPN tunnel with the Semgrep backend, and then proxying inbound (Semgrep --> customer) HTTP requests through this tunnel. This approach allows Semgrep to interact with on-prem resources without having to expose them to the public internet.

Examples of inbound traffic include:

Pull Request comments
JIRA integrations
Webhooks

Setup

Build

NOTE: The Semgrep Network broker uses Buf for protobuf compilation. If you are building the broker from scratch outside of Docker, make sure you have the Buf CLI installed: https://buf.build/docs/installation

Run make build to build the semgrep-network-broker binary locally
Run make docker to build a docker image
Docker images are also published to ghcr.io/semgrep/semgrep-network-broker

Keypairs

The broker requires a Wireguard keypair in order to establish a secure connection.

semgrep-network-broker genkey generates a random private key in base64 and prints it to stdout
semgrep-network-broker pubkey reads a base64 private key from stdin and prints the corresponding base64 public key to stdout

Example

# generate a private key
> docker run --rm ghcr.io/semgrep/semgrep-network-broker:latest genkey
some_private_key

# derive the matching public key
> echo "some_private_key" | docker run --rm -i ghcr.io/semgrep/semgrep-network-broker:latest pubkey
some_public_key

Your public key is safe to share. Do not share your private key with anyone (including Semgrep).

Configuration

Semgrep will help you create a configuration file tailored to your Semgrep deployment.

Do not alter the wireguard section.

Do not share the value of inbound.wireguard.privateKey. This is your organization's private key. Reach out to Semgrep on Slack if you need to rotate your Wireguard keys.

Example:

inbound:
  wireguard:
    localAddress: ...
    privateKey: ...
    peers:
      - endpoint: ...
  allowlist: [...]

Private key via environment variable

Instead of writing the private key into a config file, you can supply it through the environment. This is useful when your deployment platform injects secrets as environment variables or mounted files.

SEMGREP_NETWORK_BROKER_PRIVATE_KEY holds the base64 private key directly.
SEMGREP_NETWORK_BROKER_PRIVATE_KEY_PATH names a file whose contents are the base64 private key, for example a mounted Kubernetes or Docker secret. A trailing newline is fine.

# directly
export SEMGREP_NETWORK_BROKER_PRIVATE_KEY="$(semgrep-network-broker genkey)"
semgrep-network-broker -c config.yaml

# or from a file
semgrep-network-broker genkey > /run/secrets/broker-private-key
export SEMGREP_NETWORK_BROKER_PRIVATE_KEY_PATH=/run/secrets/broker-private-key
semgrep-network-broker -c config.yaml

The value is the same base64 string you would put in inbound.wireguard.privateKey.
If both variables are set, SEMGREP_NETWORK_BROKER_PRIVATE_KEY wins. If SEMGREP_NETWORK_BROKER_PRIVATE_KEY_PATH points at a missing or empty file, the broker exits with an error.
A key from the environment takes precedence over a private key from any other source (broker token or config file), and the broker logs a warning that it is overriding that key.
The private key must decode to exactly 32 bytes (44 base64 characters). The broker refuses to start with a clear error if a key of the wrong length is supplied, whichever source it came from.

HttpClient

The httpClient configuration section modifies the HTTP client used for proxying requests.

Example:

inbound:
  httpClient:
    additionalCACerts: # Optional. Certificates here will be appended to the Root CA trust of the container. Necessary when the SCM(s) the broker interacts with have self-signed certificates.
      - /path/to/custom/cert.pem
    tlsMinVersion: "1.2" # Optional. Valid values: "1.2", "1.3". Defaults to "1.3" if unset.

An alternative to stipulating additionalCACerts: is setting the $SSL_CERT_DIR environment variable at time of container creation.

Example:

$ docker run \
  ...
  -v /path/containing/your/certs:/certs \ # mount a path from the host machine as a container volume
  -e SSL_CERT_DIR=/certs \ # set the $SSL_CERT_DIR environment variable to the mounted volume
  ...
  -it semgrep-network-broker:latest -c /emt/config.yaml

Refer to the network broker docs on semgrep.dev for more detail on docker setup.

SCMs

inbound.scms lists the source control instances Semgrep should reach. Each entry names its own type, so a deployment can declare several instances of the same provider:

inbound:
  scms:
    - type: github
      baseUrl: https://github.example.com/api/v3
      token: <token>
    - type: github
      baseUrl: https://github-eu.example.com/api/v3
      token: <token>
      allowCodeAccess: true
    - type: gitlab
      baseUrl: https://gitlab.example.com/api/v4
      token: <token>

type is one of github, gitlab, bitbucket or azuredevops. An entry adds the same allowlist endpoints that the matching section below documents, and allowCodeAccess works the same way per entry.

An entry is identified by its type and baseUrl. When the broker is started with several config files, entries sharing both are merged field by field with later files winning, and entries differing in either are kept side by side. Both fields are required.

The github, gitlab, bitbucket and azuredevops sections below remain supported and can be combined with scms, as long as they name different instances. Declaring the same type and baseUrl both ways is rejected at startup, because the two would generate separate allowlists and the more permissive allowCodeAccess would win.

Allowlist

The allowlist configuration section provides fine-grained control over which HTTP requests are allowed to be forwarded out of the broker. By default, the allowlist will automatically be populated and does not need explicit configuration.

Allowlist Behaviour:

When multiple version of an allowlist item exist, the first matching allowlist item is used.
No allowlist match means the request will not be proxied.

Examples:

inbound:
  allowlist:
    # allow GET requests from http://example.com/foo (exact URL match)
    - url: http://example.com/foo
      methods: [GET]
    # allow GET or POST requests from any path on http://example.com
    - url: http://example.com/*
      methods: [GET, POST]
    # allow GET requests from a URL that looks like a GitHub Enterprise review comments URL, and add a bearer token to the request
    - url: http://example.com/api/v3/repos/:owner/:repo/pulls/:number/comments
      methods: [GET]
      setRequestHeaders:
        Authorization: "Bearer ...snip..."

Real-world example

Here's an example of allowing PR comments for a GitHub Enterprise instance hosted on https://git.example.com. Replace <GH TOKEN> with a GitHub PAT.

allowlist:
  - url: https://git.example.com/api/v3/repos/:owner/:repo
    methods: [GET]
    setRequestHeaders:
      Authorization: "Bearer <GH TOKEN>"
  - url: https://git.example.com/api/v3/repos/:owner/:repo/pulls
    methods: [GET]
    setRequestHeaders:
      Authorization: "Bearer <GH TOKEN>"
  - url: https://git.example.com/api/v3/repos/:owner/:repo/pulls/:number/comments
    methods: [POST]
    setRequestHeaders:
      Authorization: "Bearer <GH TOKEN>"
  - url: https://git.example.com/api/v3/repos/:owner/:repo/issues/:number/comments
    methods: [POST]
    setRequestHeaders:
      Authorization: "Bearer <GH TOKEN>"

GitHub

The github configuration section simplifies granting Semgrep access to leave PR comments.

Example:

inbound:
  github:
    baseUrl: https://github.example.com/api/v3
    allowCodeAccess: false # default is false, set to true to allow Semgrep to read file contents

Adding a github field to the config implicitly adds these endpoints to the allowlist:

GET https://github.example.com/api/v3/app
GET https://github.example.com/api/v3/app/hook/config
GET https://github.example.com/api/v3/app/hook/deliveries
GET https://github.example.com/api/v3/app/hook/deliveries/:delivery_id
GET https://github.example.com/api/v3/installation/repositories
GET https://github.example.com/api/v3/organizations
GET https://github.example.com/api/v3/orgs/:org
GET https://github.example.com/api/v3/orgs/:org/hooks
GET https://github.example.com/api/v3/orgs/:org/hooks/:hook_id/deliveries
GET https://github.example.com/api/v3/orgs/:org/hooks/:hook_id/deliveries/:delivery_id
GET https://github.example.com/api/v3/orgs/:org/installation
GET https://github.example.com/api/v3/orgs/:org/members
GET https://github.example.com/api/v3/orgs/:org/repos
GET https://github.example.com/api/v3/orgs/:org/teams
GET https://github.example.com/api/v3/orgs/:org/teams/:team_slug/members
GET https://github.example.com/api/v3/repos/:org/:repo/actions/secrets/public-key
GET https://github.example.com/api/v3/repos/:owner/:repo
GET https://github.example.com/api/v3/repos/:owner/:repo/branches
GET https://github.example.com/api/v3/repos/:owner/:repo/branches/:branch
GET https://github.example.com/api/v3/repos/:owner/:repo/collaborators/:username/permission
GET https://github.example.com/api/v3/repos/:owner/:repo/compare/:basehead
GET https://github.example.com/api/v3/repos/:owner/:repo/contents/.github/workflows/semgrep.yml
GET https://github.example.com/api/v3/repos/:owner/:repo/installation
GET https://github.example.com/api/v3/repos/:owner/:repo/pulls
GET https://github.example.com/api/v3/repos/:owner/:repo/pulls/comments/:comment_id/reactions
GET https://github.example.com/api/v3/repositories/:id(\d+)
GET https://github.example.com/api/v3/repositories/:id(\d+)/actions/secrets/public-key
GET https://github.example.com/api/v3/repositories/:id(\d+)/branches
GET https://github.example.com/api/v3/repositories/:id(\d+)/branches/:branch
GET https://github.example.com/api/v3/repositories/:id(\d+)/collaborators/:username/permission
GET https://github.example.com/api/v3/repositories/:id(\d+)/compare/:basehead
GET https://github.example.com/api/v3/repositories/:id(\d+)/contents/.github/workflows/semgrep.yml
GET https://github.example.com/api/v3/repositories/:id(\d+)/installation
GET https://github.example.com/api/v3/repositories/:id(\d+)/pulls
GET https://github.example.com/api/v3/repositories/:id(\d+)/pulls/comments/:comment_id/reactions
GET https://github.example.com/api/v3/user
GET https://github.example.com/api/v3/user/memberships/orgs/:org
GET https://github.example.com/api/v3/user/repos
GET https://github.example.com/api/v3/users/:user/installation
GET https://github.example.com/api/v3/users/:user/installation/repositories
GET https://github.example.com/api/v3/users/:username
POST https://github.example.com/api/v3/app-manifests/:code/conversions
POST https://github.example.com/api/v3/app/installations/:id/access_tokens
POST https://github.example.com/api/v3/orgs/:org/hooks
POST https://github.example.com/api/v3/repos/:owner/:repo/check-runs
POST https://github.example.com/api/v3/repos/:owner/:repo/git/refs
POST https://github.example.com/api/v3/repos/:owner/:repo/issues/:number/comments
POST https://github.example.com/api/v3/repos/:owner/:repo/pulls/:number/comments
POST https://github.example.com/api/v3/repos/:owner/:repo/pulls/:number/comments/:comment_id/replies
POST https://github.example.com/api/v3/repos/:owner/:repo/statuses/:commit
POST https://github.example.com/api/v3/repositories/:id(\d+)/check-runs
POST https://github.example.com/api/v3/repositories/:id(\d+)/git/refs
POST https://github.example.com/api/v3/repositories/:id(\d+)/issues/:number/comments
POST https://github.example.com/api/v3/repositories/:id(\d+)/pulls/:number/comments
POST https://github.example.com/api/v3/repositories/:id(\d+)/pulls/:number/comments/:comment_id/replies
POST https://github.example.com/api/v3/repositories/:id(\d+)/statuses/:commit
PUT https://github.example.com/api/v3/repos/:org/:repo/actions/secrets/SEMGREP_APP_TOKEN
PUT https://github.example.com/api/v3/repos/:owner/:repo/contents/.github/workflows/semgrep.yml
PUT https://github.example.com/api/v3/repositories/:id(\d+)/actions/secrets/SEMGREP_APP_TOKEN
PUT https://github.example.com/api/v3/repositories/:id(\d+)/contents/.github/workflows/semgrep.yml
PATCH https://github.example.com/api/v3/app/hook/config
PATCH https://github.example.com/api/v3/orgs/:org/hooks/:hook_id
PATCH https://github.example.com/api/v3/repos/:owner/:repo/check-runs/:check_run_id
PATCH https://github.example.com/api/v3/repos/:owner/:repo/issues/comments/:comment_id
PATCH https://github.example.com/api/v3/repos/:owner/:repo/pulls/:number/comments/:comment_id
PATCH https://github.example.com/api/v3/repos/:owner/:repo/pulls/comments/:comment_id
PATCH https://github.example.com/api/v3/repositories/:id(\d+)/check-runs/:check_run_id
PATCH https://github.example.com/api/v3/repositories/:id(\d+)/issues/comments/:comment_id
PATCH https://github.example.com/api/v3/repositories/:id(\d+)/pulls/:number/comments/:comment_id
PATCH https://github.example.com/api/v3/repositories/:id(\d+)/pulls/comments/:comment_id
DELETE https://github.example.com/api/v3/orgs/:org/hooks/:hook_id

And if allowCodeAccess is set, these endpoints are also added to the allowlist:

GET https://github.example.com/:owner/:repo/info/refs
GET https://github.example.com/api/v3/repos/:owner/:repo/commits
GET https://github.example.com/api/v3/repos/:owner/:repo/contents
GET https://github.example.com/api/v3/repos/:owner/:repo/contents/*
GET https://github.example.com/api/v3/repos/:owner/:repo/git/commits/:sha
GET https://github.example.com/api/v3/repos/:owner/:repo/git/ref/*
GET https://github.example.com/api/v3/repositories/:id(\d+)/commits
GET https://github.example.com/api/v3/repositories/:id(\d+)/contents
GET https://github.example.com/api/v3/repositories/:id(\d+)/contents/*
GET https://github.example.com/api/v3/repositories/:id(\d+)/git/commits/:sha
GET https://github.example.com/api/v3/repositories/:id(\d+)/git/ref/*
POST https://github.example.com/:owner/:repo/git-receive-pack
POST https://github.example.com/:owner/:repo/git-upload-pack
POST https://github.example.com/api/v3/repos/:owner/:repo/git/blobs
POST https://github.example.com/api/v3/repos/:owner/:repo/git/commits
POST https://github.example.com/api/v3/repos/:owner/:repo/git/trees
POST https://github.example.com/api/v3/repos/:owner/:repo/pulls
POST https://github.example.com/api/v3/repositories/:id(\d+)/git/blobs
POST https://github.example.com/api/v3/repositories/:id(\d+)/git/commits
POST https://github.example.com/api/v3/repositories/:id(\d+)/git/trees
POST https://github.example.com/api/v3/repositories/:id(\d+)/pulls
PATCH https://github.example.com/api/v3/repos/:owner/:repo/git/refs/*
PATCH https://github.example.com/api/v3/repositories/:id(\d+)/git/refs/*

GitLab

Similarly, the gitlab configuration section grants Semgrep access to leave MR comments.

Example:

inbound:
  gitlab:
    baseUrl: https://gitlab.example.com/api/v4
    allowCodeAccess: false # default is false, set to true to allow Semgrep to read file contents

Adding a gitlab field to the config implicitly adds these endpoints to the allowlist:

GET https://gitlab.example.com/api/v4/:entity_type/:namespace/projects
GET https://gitlab.example.com/api/v4/groups/:namespace/hooks
GET https://gitlab.example.com/api/v4/groups/:namespace/members/all
GET https://gitlab.example.com/api/v4/groups/:namespace/members/all/:user
GET https://gitlab.example.com/api/v4/namespaces/:namespace
GET https://gitlab.example.com/api/v4/personal_access_tokens/self
GET https://gitlab.example.com/api/v4/projects/:project
GET https://gitlab.example.com/api/v4/projects/:project/members/all/:user
GET https://gitlab.example.com/api/v4/projects/:project/merge_requests
GET https://gitlab.example.com/api/v4/projects/:project/merge_requests/:number/discussions
GET https://gitlab.example.com/api/v4/projects/:project/merge_requests/:number/discussions/:discussion/notes/:note/award_emoji
GET https://gitlab.example.com/api/v4/projects/:project/merge_requests/:number/versions
GET https://gitlab.example.com/api/v4/projects/:project/repository/branches
GET https://gitlab.example.com/api/v4/projects/:project/repository/branches/:branch
POST https://gitlab.example.com/api/v4/groups/:namespace/hooks
POST https://gitlab.example.com/api/v4/projects/:project/hooks
POST https://gitlab.example.com/api/v4/projects/:project/merge_requests/:number/discussions
POST https://gitlab.example.com/api/v4/projects/:project/merge_requests/:number/discussions/:discussion/notes
PUT https://gitlab.example.com/api/v4/groups/:namespace/hooks
PUT https://gitlab.example.com/api/v4/projects/:project/merge_requests/:number/discussions/:discussion
PUT https://gitlab.example.com/api/v4/projects/:project/merge_requests/:number/discussions/:discussion/notes/:note
DELETE https://gitlab.example.com/api/v4/groups/:namespace/hooks/:hook
DELETE https://gitlab.example.com/api/v4/projects/:project/hooks/:hook

And if allowCodeAccess is set, these endpoints are also added to the allowlist:

GET https://gitlab.example.com/api/v4/projects/:project/repository/commits
GET https://gitlab.example.com/api/v4/projects/:project/repository/compare
GET https://gitlab.example.com/api/v4/projects/:project/repository/files/*
GET https://gitlab.example.com/api/v4/projects/:project/repository/merge_base
GET https://gitlab.example.com/{:namespace/}+:project/info/refs
POST https://gitlab.example.com/api/v4/projects/:project/merge_requests
POST https://gitlab.example.com/api/v4/projects/:project/repository/branches
POST https://gitlab.example.com/api/v4/projects/:project/repository/commits
POST https://gitlab.example.com/api/v4/projects/:project/statuses/:commit
POST https://gitlab.example.com/{:namespace/}+:project/git-receive-pack
POST https://gitlab.example.com/{:namespace/}+:project/git-upload-pack

Bitbucket

Similarly, the bitbucket configuration section grants Semgrep access to leave MR comments.

inbound:
  bitbucket:
    baseUrl: https://bitbucket.example.com/rest/api/latest
    allowCodeAccess: false # default is false, set to true to allow Semgrep to read file contents (and write them back for Code Autofix)

Adding a bitbucket field to the config implicitly adds these endpoints to the allowlist:

GET https://bitbucket.example.com/rest/api/latest/application-properties
GET https://bitbucket.example.com/rest/api/latest/projects/:project
GET https://bitbucket.example.com/rest/api/latest/projects/:project/repos
GET https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo
GET https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/branches
GET https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/default-branch
GET https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/pull-requests
GET https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/pull-requests/:number/comments/:comment
GET https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/webhooks
GET https://bitbucket.example.com/rest/api/latest/projects/:project/webhooks
POST https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/pull-requests/:number/blocker-comments
POST https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/pull-requests/:number/comments
POST https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/webhooks
POST https://bitbucket.example.com/rest/api/latest/projects/:project/webhooks
PUT https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/pull-requests/:number/comments/:comment
PUT https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/webhooks/:webhook
PUT https://bitbucket.example.com/rest/api/latest/projects/:project/webhooks/:webhook
DELETE https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/webhooks/:webhook
DELETE https://bitbucket.example.com/rest/api/latest/projects/:project/webhooks/:webhook

And if allowCodeAccess is set, these endpoints are also added to the allowlist:

GET https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/browse/*
GET https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/commits
GET https://bitbucket.example.com/scm/:project/:repo/info/refs
POST https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/branches
POST https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/commits/:commit/builds
POST https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/pull-requests
POST https://bitbucket.example.com/scm/:project/:repo/git-upload-pack
PUT https://bitbucket.example.com/rest/api/latest/projects/:project/repos/:repo/browse/*

The POST .../branches, PUT .../browse/*, and POST .../pull-requests entries are what Code Autofix uses to create a branch, commit a fix onto it, and open a pull request. Leave allowCodeAccess unset if you do not want Semgrep writing to your repositories.

Azure DevOps

Similarly, the azuredevops configuration section grants Semgrep access to azure devops.

inbound:
  azuredevops:
    baseUrl: https://example@dev.azure.com/
    allowCodeAccess: false # default is false, set to true to allow Semgrep to read file contents

Adding a azuredevops field to the config implicitly adds these endpoints to the allowlist

GET https://dev.azure.com/:namespace/:project/_apis/git/repositories
GET https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo
GET https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/pullRequests
GET https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/pullRequests/:number/iterations
GET https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/pullRequests/:number/iterations/:iterationId/changes
GET https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/refs
GET https://dev.azure.com/:namespace/_apis/connectionData
GET https://dev.azure.com/:namespace/_apis/hooks/subscriptions
GET https://dev.azure.com/:namespace/_apis/projects/:project
GET https://vsaex.dev.azure.com/:namespace/_apis/groupentitlements
POST https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/pullRequests/:number/threads
POST https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/pullRequests/:number/threads/:threadId/comments
POST https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/refs
POST https://dev.azure.com/:namespace/_apis/hooks/subscriptions
PUT https://dev.azure.com/:namespace/_apis/hooks/subscriptions/:subscriptionId
PATCH https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/pullRequests/:number/threads
PATCH https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/pullRequests/:number/threads/:threadId/comments/:commentId
DELETE https://dev.azure.com/:namespace/_apis/hooks/subscriptions/:subscriptionId

And if allowCodeAccess is set, these endpoints are also added to the allowlist:

GET https://dev.azure.com/:namespace/:project/_apis/git/pullrequests/:number
GET https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/items
GET https://dev.azure.com/:namespace/:project/_git/:repo/info/refs
POST https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/commits/:commit/statuses
POST https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/pullRequests
POST https://dev.azure.com/:namespace/:project/_apis/git/repositories/:repo/pushes
POST https://dev.azure.com/:namespace/:project/_git/:repo/git-upload-pack

Logging

The logging configuration section allows you to set additional logging options for requests that are proxied through the broker.

inbound:
  logging:
    logRequestBody: false # If true, the contents of any proxied HTTP request matching the allowlist will be logged in the request_body field in the proxy.request event
    logResponseBody: false # If true, the contents of any proxied HTTP response will be logged in the response_body field in the proxy.response event

Turn logRequestBody and logResponseBody on only while investigating a problem. Bodies are buffered in memory so they can be logged, so large requests and responses raise the broker's memory use substantially.

Here's an example log output of curl -X POST -H "Content-Type: application/json" "https://httpbin.org/anything" -d '{"foo": "bar"}' being proxied through the network broker:

INFO[0006] request.start                                 client_ip="::1" id=1 method=POST path="/proxy/https://httpbin.org/anything" query= user_agent=curl/8.2.1
INFO[0006] proxy.request                                 allowlist_match="https://httpbin.org/*" client_ip="::1" destinationUrl="https://httpbin.org/anything" id=1 method=POST path="/proxy/https://httpbin.org/anything" query= request_body="{\"foo\": \"bar\"}" user_agent=curl/8.2.1
INFO[0006] proxy.response                                allowlist_match="https://httpbin.org/*" client_ip="::1" destinationUrl="https://httpbin.org/anything" id=1 method=POST path="/proxy/https://httpbin.org/anything" query= response_body="{\n  \"args\": {}, \n  \"data\": \"{\\\"foo\\\": \\\"bar\\\"}\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\": \"gzip\", \n    \"Content-Length\": \"14\", \n    \"Content-Type\": \"application/json\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"curl/8.2.1\", \n    \"X-Amzn-Trace-Id\": \"Root=1-650469a8-0032596526902b563d7e5ebc\"\n  }, \n  \"json\": {\n    \"foo\": \"bar\"\n  }, \n  \"method\": \"POST\", \n  \"origin\": \"::1, ...snip..., ...snip...\", \n  \"url\": \"https://httpbin.org/anything\"\n}\n" user_agent=curl/8.2.1
INFO[0006] request.response                              body_size=511 client_ip="::1" id=1 latency=341.905708ms method=POST path="/proxy/https://httpbin.org/anything" query= status_code=200 user_agent=curl/8.2.1

logRequestBody and logResponseBody can also be set on a per-allowlist basis:

inbound:
  allowlist:
    - url: https://httpbin.org/*
      methods: [GET, POST, DELETE]
      logRequestBody: true
      logResponseBody: true

Reading the logs

The broker logs to stdout, so retrieve them however your deployment exposes container output:

Deployment	Command
Docker	`docker logs <container id>`
Kubernetes	`kubectl logs <pod name>`

Usage

The broker can be run in Kubernetes, as a bare Docker container, or simply as a standalone binary on a machine.

Only one broker instance per Semgrep organization is supported. Instances that share a private key present the same Wireguard public key, so the Semgrep backend sees a single peer and keeps a single endpoint address for it. Whichever instance last completed a handshake owns that endpoint, so inbound requests follow it there, and a request already in flight is dropped when the endpoint moves. Running a second instance makes the broker less available, not more.

Config file(s) are passed to the app with -c:

semgrep-network-broker -c config.yaml

Multiple config files can be overlaid on top of each other by passing multiple -c args (ex. semgrep-network-broker -c config1.yaml -c config2.yaml -c config3.yaml). Note that while maps will be merged together, arrays will be replaced.

Requirements:

internet access to wireguard.semgrep.dev on UDP port 51820
DNS resolution of wireguard.semgrep.dev, whose TXT records supply the peer's public key, allowed IPs and heartbeat URL
outbound HTTPS to semgrep.dev on TCP port 443, if using -d

Loading defaults from Semgrep

-d (--deployment-id) takes your Semgrep organization ID, and the broker fetches the Wireguard settings for that organization at startup. Your own config file then only needs the private key and the SCMs Semgrep should reach:

semgrep-network-broker -c config.yaml -d <organization id>

Fetched settings are merged first and your -c files are layered on top, so anything you set locally wins. Without -d, set inbound.wireguard.localAddress yourself; the peer's public key, allowed IPs and heartbeat URL come from DNS either way.

Putting it together, with the published image and a config file in the working directory:

docker run --rm -it \
  -v "$PWD/config.yaml:/emt/config.yaml" \
  ghcr.io/semgrep/semgrep-network-broker:latest \
  -c /emt/config.yaml -d <organization id>

Other Commands

dump

semgrep-network-broker dump dumps the current config. This is useful to see what the result of multiple configurations overlays would result in

genkey

semgrep-network-broker genkey generates a base64 private key to stdout

pubkey

semgrep-network-broker pubkey generates a base64 public key for a given private key (via stdin)

relay

semgrep-network-broker relay launches an HTTP server that relays request that match a certain rule.

outbound:
  listenPort: 8080
  relay:
    test:
      destinationUrl: https://httpbin.org/anything
      jsonPath: "$.foo"
      equals:
        - bar

would result in requests addressed to http://localhost:8080/relay/test being relayed to https://httpbin.org/anything as long as the result of the jsonpath query $.foo executed on the request body results in the string bar.

Check out an example here for how to use the relay for GitHub PR comments.

You can also define additional relay mappings via the additionalConfigs field:

outbound:
  listenPort: 8080
  relay:
    test:
      destinationUrl: https://httpbin.org/anything
      jsonPath: "$.foo"
      equals:
        - bar
      additionalConfigs:
        - destinationUrl: https://example.com/fallback

The example above would relay traffic to https://httpbin.org/anything if the request body contains {"foo": "bar"}, otherwise, it'd relay traffic to htttps://example.com/fallback.

For other questions or feedback, join us on the Semgrep Community Slack.

Name		Name	Last commit message	Last commit date
.github		.github
build		build
cmd		cmd
examples		examples
it		it
pkg		pkg
protos/broker.v1		protos/broker.v1
tools/readme-url-populator		tools/readme-url-populator
.buf-version		.buf-version
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
CODEOWNERS		CODEOWNERS
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
broker.proto		broker.proto
buf.gen.yaml		buf.gen.yaml
go.mod		go.mod
go.sum		go.sum
kubernetes.yaml		kubernetes.yaml
main.go		main.go

Navigation Menu

Repository files navigation

semgrep-network-broker

Setup

Build

Keypairs

Example

Configuration

Private key via environment variable

HttpClient

SCMs

Allowlist

Real-world example

GitHub

GitLab

Bitbucket

Azure DevOps

Logging

Reading the logs

Usage

Loading defaults from Semgrep

Other Commands

dump

genkey

pubkey

relay

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

semgrep-network-broker

Setup

Build

Keypairs

Example

Configuration

Private key via environment variable

HttpClient

SCMs

Allowlist

Real-world example

GitHub

GitLab

Bitbucket

Azure DevOps

Logging

Reading the logs

Usage

Loading defaults from Semgrep

Other Commands

dump

genkey

pubkey

relay

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages