Development Guide - execd

This comprehensive guide explains how to work on execd as a contributor or maintainer. It covers environment setup, development workflows, testing strategies, architectural patterns, and subsystem-specific implementation details.

Getting Started
Project Structure
Coding Standards
Testing Strategy
Subsystem Guides
Common Development Tasks
Debugging Techniques
Performance Optimization
Contributing Guidelines
Additional Resources

Getting Started

Prerequisites

Required Tools

Go 1.24+ - Match the version declared in go.mod
Git - Version control
Make - Build automation (optional but recommended)

Optional but Recommended

golangci-lint - For comprehensive linting
Docker/Podman - For containerized testing and deployment
Jupyter Server - Required for integration tests with real kernels
VS Code/GoLand - IDE with Go support

Initial Setup

bash

# Clone the repository
git clone https://github.com/alibaba/OpenSandbox.git
cd OpenSandbox/components/execd

# Download dependencies
go mod download

# Verify setup
go build -o bin/execd .

Project Structure

Project Structure Deep Dive

execd/
├── main.go                 # Application entry point
├── go.mod                  # Go module definition
├── Makefile               # Build automation
├── Dockerfile             # Container image definition
│
├── pkg/                   # Public packages
│   ├── flag/              # CLI flag parsing
│   ├── web/               # HTTP layer
│   │   ├── router.go      # Route registration
│   │   ├── controller/    # Request handlers
│   │   └── model/         # API models
│   ├── runtime/           # Execution engine
│   │   ├── ctrl.go        # Main controller
│   │   ├── jupyter.go     # Jupyter execution
│   │   └── command.go     # Shell command execution
│   ├── jupyter/           # Jupyter client
│   │   ├── client.go      # HTTP/WebSocket client
│   │   ├── session/       # Session management
│   │   └── execute/       # Execution protocol
│   └── util/              # Utilities
│
└── tests/                # Integration test scripts

Key Design Patterns

1. Controller Pattern (pkg/web/controller)

Controllers are thin HTTP handlers that parse requests, validate, delegate to runtime, and stream responses via SSE.

2. Runtime Controller Pattern (pkg/runtime)

The runtime controller dispatches requests to appropriate executors (Jupyter, Command, SQL) and manages session lifecycle.

3. Hook Pattern for Streaming

Execution results are streamed via hooks, allowing controllers to transform runtime events into SSE events without tight coupling.

Coding Standards

Go Conventions

Formatting

Always use gofmt before committing:

bash

gofmt -w .
# or
make fmt

Import Organization

Three groups separated by blank lines:

import (
    // Standard library
    "context"
    "fmt"

    // Third-party
    "github.com/beego/beego/v2/core/logs"

    // Internal
    "github.com/alibaba/opensandbox/execd/pkg/runtime"
)

Error Handling

Always handle errors explicitly:

// Good
result, err := someOperation()
if err != nil {
    logs.Error("operation failed: %v", err)
    return fmt.Errorf("failed to do something: %w", err)
}

// Bad - silent failure
result, _ := someOperation()

Logging

Use Beego's structured logger:

logs.Info("starting execution: sessionID=%s", sessionID)
logs.Warning("session busy: sessionID=%s", sessionID)
logs.Error("execution failed: error=%v", err)
logs.Debug("received event: type=%s", eventType)

Concurrency Best Practices

Use safego for goroutines

Always use safego.Go to prevent panics:

import "github.com/alibaba/opensandbox/execd/pkg/util/safego"

safego.Go(func() {
    processInBackground()
})

Context Propagation

Always respect context cancellation:

func (c *Controller) runCommand(ctx context.Context, req *ExecuteCodeRequest) error {
    cmd := exec.CommandContext(ctx, "bash", "-c", req.Code)

    go func() {
        <-ctx.Done()
        if cmd.Process != nil {
            cmd.Process.Kill()
        }
    }()

    return cmd.Run()
}

Testing Strategy

Unit Tests

Located in *_test.go files alongside source code.

Example:

func TestController_Execute_Python(t *testing.T) {
    ctrl := NewController("http://jupyter:8888", "test-token")

    req := &ExecuteCodeRequest{
        Language: Python,
        Code:     "print('hello')",
    }

    err := ctrl.Execute(req)
    assert.NoError(t, err)
}

Running Unit Tests:

bash

go test ./pkg/...
# with coverage
go test -v -cover ./pkg/...

Integration Tests

Located in *_integration_test.go, require real dependencies.

Running Integration Tests:

bash

export JUPYTER_URL=http://localhost:8888
export JUPYTER_TOKEN=your-token
go test -v ./pkg/jupyter/...

Test Coverage

Check coverage:

bash

go test -coverprofile=coverage.out ./pkg/...
go tool cover -html=coverage.out -o coverage.html

Coverage Goals:

Core packages (pkg/runtime, pkg/jupyter): > 80%
Controllers (pkg/web/controller): > 70%
Utilities (pkg/util): > 90%

Subsystem Guides

Working with Jupyter Integration

Architecture

pkg/jupyter/
├── client.go          # Main client
├── transport.go       # Connection handling
├── session/           # Session lifecycle
├── execute/           # Execution protocol
└── auth/              # Authentication

Adding New Kernel Support

Define language in pkg/runtime/language.go:

const Ruby Language = "ruby"

Map to kernel in pkg/runtime/jupyter.go
Test with real kernel:

bash

# Install Ruby kernel
gem install iruby
iruby register --force

# Run test
export JUPYTER_URL=http://localhost:8888
go test -v ./pkg/jupyter/integration_test.go

Debugging Jupyter Communication

Run debug integration test:

bash

go test -v ./pkg/jupyter/debug_integration_test.go

This dumps complete HTTP request/response pairs.

Working with Command Execution

Key Implementation Details

Process Group Management:

cmd.SysProcAttr = &syscall.SysProcAttr{
    Setpgid: true,  // Create new process group
}

This allows signal forwarding to all child processes:

syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)

Signal Forwarding:

signals := make(chan os.Signal, 1)
signal.Notify(signals)

go func() {
    for sig := range signals {
        if sig != syscall.SIGCHLD && sig != syscall.SIGURG {
            syscall.Kill(-cmd.Process.Pid, sig.(syscall.Signal))
        }
    }
}()

Stdout/Stderr Streaming:

Commands write to temporary log files, which are tailed and streamed to hooks.

Common Development Tasks

Adding a New API Endpoint

Define model in pkg/web/model/:

type NewFeatureRequest struct {
    Param1 string `json:"param1" validate:"required"`
    Param2 int    `json:"param2"`
}

Add controller method in pkg/web/controller/:

func (c *MyController) NewFeature() {
    var req model.NewFeatureRequest
    json.Unmarshal(c.Ctx.Input.RequestBody, &req)

    // Business logic
    result := processNewFeature(req)

    c.Data["json"] = result
    c.ServeJSON()
}

Register route in pkg/web/router.go:

myNamespace := web.NewNamespace("/my-feature",
    web.NSRouter("", &controller.MyController{}, "post:NewFeature"),
)
web.AddNamespace(myNamespace)

Adding Configuration Flag

Declare in pkg/flag/flags.go:

var NewFeatureTimeout time.Duration

Parse in pkg/flag/parser.go:

func InitFlags() {
    flag.DurationVar(&NewFeatureTimeout, "new-feature-timeout", 30*time.Second, "Description")

    // Parse environment variable
    if env := os.Getenv("NEW_FEATURE_TIMEOUT"); env != "" {
        if d, err := time.ParseDuration(env); err == nil {
            NewFeatureTimeout = d
        }
    }

    flag.Parse()
}

Update README with new flag documentation

Debugging Techniques

Local Debugging with Delve

bash

# Install delve
go install github.com/go-delve/delve/cmd/dlv@latest

# Start debugging
dlv debug . -- \
  --jupyter-host=http://localhost:8888 \
  --jupyter-token=test

# Set breakpoint
(dlv) break pkg/runtime/ctrl.go:57
(dlv) continue

Debugging SSE Streams

Test with curl:

bash

curl -N -H "x-access-token: dev" \
  -H "Content-Type: application/json" \
  -d '{"language":"python","code":"print(\"test\")"}' \
  http://localhost:44772/code

The -N flag disables buffering for real-time events.

Debug in browser:

javascript

const eventSource = new EventSource('/code');

eventSource.addEventListener('stdout', (e) => {
    console.log('stdout:', e.data);
});

eventSource.addEventListener('error', (e) => {
    console.error('error:', e.data);
});

Performance Profiling

CPU Profile:

bash

# Add to main.go
import _ "net/http/pprof"

go func() {
    http.ListenAndServe("localhost:6060", nil)
}()

# Collect profile
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

Memory Profile:

bash

go tool pprof http://localhost:6060/debug/pprof/heap

Goroutine Inspection:

bash

curl http://localhost:6060/debug/pprof/goroutine?debug=2

Performance Optimization

Optimization Guidelines

Profile before optimizing - Use pprof to identify bottlenecks
Benchmark changes - Measure impact of optimizations
Use sync.Pool for frequently allocated objects
Minimize allocations in hot paths
Buffer channels appropriately

Example: Optimizing SSE Writer

Before:

func writeEvent(w http.ResponseWriter, event, data string) {
    fmt.Fprintf(w, "event: %s\ndata: %s\n\n", event, data)
    w.(http.Flusher).Flush()
}

After:

var bufPool = sync.Pool{
    New: func() interface{} { return new(bytes.Buffer) },
}

func writeEvent(w http.ResponseWriter, event, data string) {
    buf := bufPool.Get().(*bytes.Buffer)
    buf.Reset()
    defer bufPool.Put(buf)

    buf.WriteString("event: ")
    buf.WriteString(event)
    buf.WriteString("\ndata: ")
    buf.WriteString(data)
    buf.WriteString("\n\n")

    w.Write(buf.Bytes())
    w.(http.Flusher).Flush()
}

Benchmark:

func BenchmarkWriteEvent(b *testing.B) {
    w := httptest.NewRecorder()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        writeEvent(w, "test", "data")
    }
}

Contributing Guidelines

Pull Request Process

Fork and clone the repository
Create feature branch from main
Implement changes following coding standards
Add tests for new functionality
Run all tests and ensure they pass
Update documentation as needed
Submit PR with clear description

Code Review Standards

Reviewers check for:

[ ] Correctness and functionality
[ ] Test coverage
[ ] Code style and formatting
[ ] Documentation completeness
[ ] Performance implications
[ ] Security considerations
[ ] Error handling
[ ] Backwards compatibility

Release Checklist

Before releasing:

[ ] All tests pass (unit, integration, e2e)
[ ] Documentation updated (README, DEVELOPMENT, API docs)
[ ] CHANGELOG updated with changes
[ ] Version bumped appropriately (semver)
[ ] Dependencies reviewed and updated
[ ] Security scan passed
[ ] Performance benchmarks run
[ ] Docker image built and tested

Additional Resources

Useful Commands

bash

# Format all Go files
make fmt

# Run linter
make golint

# Run all tests
make test

# Build binary
make build

External Documentation

Getting Help

Issues: Report bugs or request features on GitHub Issues
Discussions: Ask questions in GitHub Discussions
Chat: Join the OpenSandbox community chat
Documentation: Check the wiki for detailed guides

Happy hacking! Feel free to augment this guide with tips you discover along the way. For questions or suggestions, open an issue or discussion on GitHub.

This page is sourced from: components/execd/DEVELOPMENT.md

Development Guide - execd ​

Table of Contents ​

Getting Started ​

Prerequisites ​

Required Tools ​

Optional but Recommended ​

Initial Setup ​

Project Structure ​

Project Structure Deep Dive ​

Key Design Patterns ​

1. Controller Pattern (pkg/web/controller) ​

2. Runtime Controller Pattern (pkg/runtime) ​

3. Hook Pattern for Streaming ​

Coding Standards ​

Go Conventions ​

Formatting ​

Import Organization ​

Error Handling ​

Logging ​

Concurrency Best Practices ​

Use safego for goroutines ​

Context Propagation ​

Testing Strategy ​

Unit Tests ​

Integration Tests ​

Test Coverage ​

Subsystem Guides ​

Working with Jupyter Integration ​

Architecture ​

Adding New Kernel Support ​

Debugging Jupyter Communication ​

Working with Command Execution ​

Key Implementation Details ​

Common Development Tasks ​

Adding a New API Endpoint ​

Adding Configuration Flag ​

Debugging Techniques ​

Local Debugging with Delve ​

Debugging SSE Streams ​

Performance Profiling ​

Performance Optimization ​

Optimization Guidelines ​

Example: Optimizing SSE Writer ​

Contributing Guidelines ​

Pull Request Process ​

Code Review Standards ​

Release Checklist ​

Additional Resources ​

Useful Commands ​

External Documentation ​

Getting Help ​

Development Guide - execd

Table of Contents

Getting Started

Prerequisites

Required Tools

Optional but Recommended

Initial Setup

Project Structure

Project Structure Deep Dive

Key Design Patterns

1. Controller Pattern (pkg/web/controller)

2. Runtime Controller Pattern (pkg/runtime)

3. Hook Pattern for Streaming

Coding Standards

Go Conventions

Formatting

Import Organization

Error Handling

Logging

Concurrency Best Practices

Use safego for goroutines

Context Propagation

Testing Strategy

Unit Tests

Integration Tests

Test Coverage

Subsystem Guides

Working with Jupyter Integration

Architecture

Adding New Kernel Support

Debugging Jupyter Communication

Working with Command Execution

Key Implementation Details

Common Development Tasks

Adding a New API Endpoint

Adding Configuration Flag

Debugging Techniques

Local Debugging with Delve

Debugging SSE Streams

Performance Profiling

Performance Optimization

Optimization Guidelines

Example: Optimizing SSE Writer

Contributing Guidelines

Pull Request Process

Code Review Standards

Release Checklist

Additional Resources

Useful Commands

External Documentation

Getting Help