Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support none root deployment #3000

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Support none root deployment #3000

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions build/images/base/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,10 @@ RUN curl -fsSL "$GOLANG_DOWNLOAD_URL" -o golang.tar.gz \
&& tar -C /usr/local -xzf golang.tar.gz \
&& rm golang.tar.gz \
&& go install github.com/go-delve/delve/cmd/[email protected]

# Create none root user and group
ARG USER_NAME=dragonfly
ARG USER_UID=1000
ARG GROUP_NAME=dragonfly
ARG GROUP_GID=1000
RUN groupadd -g $GROUP_GID $GROUP_NAME && useradd -u $USER_UID -g $GROUP_GID -m -s /bin/bash $USER_NAME
21 changes: 19 additions & 2 deletions build/images/dfdaemon/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,28 @@ RUN if [ "$(uname -m)" = "ppc64le" ]; then \

FROM ${BASE_IMAGE}

ARG USER_NAME=dragonfly
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the default user should be root, please do not to break compatibility with previous versions of dragonfly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code only adds the none root user and group, the default user is still root.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not change the default root user using command.

Copy link
Contributor Author

@TommyLike TommyLike Jan 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not change the default root user using command.

If I understand correctly

  1. the default user when running container is still root and we can use the dragonfly user by adding user argument
  2. the binary would belongs to dragonfly/dragonfly can it can still work within root user.

ARG USER_UID=1000
ARG GROUP_NAME=dragonfly
ARG GROUP_GID=1000

ENV PATH=/opt/dragonfly/bin:$PATH
RUN echo "hosts: files dns" > /etc/nsswitch.conf

COPY --from=builder /opt/dragonfly/bin/dfget /opt/dragonfly/bin/dfget
COPY --from=health /bin/grpc_health_probe /bin/grpc_health_probe
# NOTE: Command works for alpine linux, for other distro please update run statement or create user/group in base image
RUN if [ getent group "$GROUP_NAME" ]; then \
echo "group exists"; \
else \
addgroup -S -g $GROUP_GID $GROUP_NAME; \
fi
RUN if [ getent passwd "$USER_NAME" ]; then \
echo "user exists"; \
else \
adduser -S -u $USER_UID -G $GROUP_NAME -D $USER_NAME; \
fi

COPY --from=builder --chown=$GROUP_NAME:$USER_NAME /opt/dragonfly/bin/dfget /opt/dragonfly/bin/dfget
COPY --from=health --chown=$GROUP_NAME:$USER_NAME /bin/grpc_health_probe /bin/grpc_health_probe

EXPOSE 65001

Expand Down
21 changes: 19 additions & 2 deletions build/images/manager/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,32 @@ RUN if [ "$(uname -m)" = "ppc64le" ]; then \

FROM ${BASE_IMAGE}

ARG USER_NAME=dragonfly
ARG USER_UID=1000
ARG GROUP_NAME=dragonfly
ARG GROUP_GID=1000

WORKDIR /opt/dragonfly

ENV PATH=/opt/dragonfly/bin:$PATH

RUN mkdir -p /opt/dragonfly/bin/manager/console \
&& echo "hosts: files dns" > /etc/nsswitch.conf

COPY --from=server-builder /opt/dragonfly/bin/manager /opt/dragonfly/bin/server
COPY --from=health /bin/grpc_health_probe /bin/grpc_health_probe
# NOTE: Command works for alpine linux, for other distro please update run statement or create user/group in base image
RUN if [ getent group "$GROUP_NAME" ]; then \
echo "group exists"; \
else \
addgroup -S -g $GROUP_GID $GROUP_NAME; \
fi
RUN if [ getent passwd "$USER_NAME" ]; then \
echo "user exists"; \
else \
adduser -S -u $USER_UID -G $GROUP_NAME -D $USER_NAME; \
fi

COPY --from=server-builder --chown=$GROUP_NAME:$USER_NAME /opt/dragonfly/bin/manager /opt/dragonfly/bin/server
COPY --from=health --chown=$GROUP_NAME:$USER_NAME /bin/grpc_health_probe /bin/grpc_health_probe

EXPOSE 8080 65003

Expand Down
21 changes: 19 additions & 2 deletions build/images/scheduler/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,28 @@ RUN if [ "$(uname -m)" = "ppc64le" ]; then \

FROM ${BASE_IMAGE}

ARG USER_NAME=dragonfly
ARG USER_UID=1000
ARG GROUP_NAME=dragonfly
ARG GROUP_GID=1000

ENV PATH=/opt/dragonfly/bin:$PATH
RUN echo "hosts: files dns" > /etc/nsswitch.conf

COPY --from=builder /opt/dragonfly/bin/scheduler /opt/dragonfly/bin/scheduler
COPY --from=health /bin/grpc_health_probe /bin/grpc_health_probe
# NOTE: Command works for alpine linux, for other distro please update run statement or create user/group in base image
RUN if [ getent group "$GROUP_NAME" ]; then \
echo "group exists"; \
else \
addgroup -S -g $GROUP_GID $GROUP_NAME; \
fi
RUN if [ getent passwd "$USER_NAME" ]; then \
echo "user exists"; \
else \
adduser -S -u $USER_UID -G $GROUP_NAME -D $USER_NAME; \
fi

COPY --from=builder --chown=$GROUP_NAME:$USER_NAME /opt/dragonfly/bin/scheduler /opt/dragonfly/bin/scheduler
COPY --from=health --chown=$GROUP_NAME:$USER_NAME /bin/grpc_health_probe /bin/grpc_health_probe

EXPOSE 8002

Expand Down
21 changes: 19 additions & 2 deletions build/images/trainer/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,28 @@ RUN if [ "$(uname -m)" = "ppc64le" ]; then \

FROM ${BASE_IMAGE}

ARG USER_NAME=dragonfly
ARG USER_UID=1000
ARG GROUP_NAME=dragonfly
ARG GROUP_GID=1000

ENV PATH=/opt/dragonfly/bin:$PATH
RUN echo "hosts: files dns" > /etc/nsswitch.conf

COPY --from=builder /opt/dragonfly/bin/trainer /opt/dragonfly/bin/trainer
COPY --from=health /bin/grpc_health_probe /bin/grpc_health_probe
# NOTE: Command works for alpine linux, for other distro please update run statement or create user/group in base image
RUN if [ getent group "$GROUP_NAME" ]; then \
echo "group exists"; \
else \
addgroup -S -g $GROUP_GID $GROUP_NAME; \
fi
RUN if [ getent passwd "$USER_NAME" ]; then \
echo "user exists"; \
else \
adduser -S -u $USER_UID -G $GROUP_NAME -D $USER_NAME; \
fi

COPY --from=builder --chown=$GROUP_NAME:$USER_NAME /opt/dragonfly/bin/trainer /opt/dragonfly/bin/trainer
COPY --from=health --chown=$GROUP_NAME:$USER_NAME /bin/grpc_health_probe /bin/grpc_health_probe

EXPOSE 9090

Expand Down
4 changes: 4 additions & 0 deletions cmd/manager/cmd/root.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,10 @@ func initDfpath(cfg *config.ServerConfig) (dfpath.Dfpath, error) {
options = append(options, dfpath.WithPluginDir(cfg.PluginDir))
}

if cfg.DataDir != "" {
options = append(options, dfpath.WithDataDir(cfg.DataDir))
}

return dfpath.New(options...)
}

Expand Down
9 changes: 9 additions & 0 deletions deploy/docker-compose/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

> Currently, docker compose deploying is tested just in single host, no HA support.

## Build local images for Docker Compose

Try the command below in the dragonfly root directory.

```shell
export D7Y_REGISTRY=dragonflyoss
make docker-build
```

## Deploy with Docker Compose

The `run.sh` script will generate config and deploy all components with `docker-compose`.
Expand Down
12 changes: 12 additions & 0 deletions deploy/docker-compose/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ services:
interval: 1s
timeout: 2s
retries: 30
# Enable when running in none root mode
# user: "1000:1000"
command: ["--verbose", "--console"]
ports:
- 65003:65003
- 8080:8080
Expand All @@ -61,6 +64,9 @@ services:
interval: 1s
timeout: 2s
retries: 30
command: ["--verbose", "--console"]
# Enable when running in none root mode
# user: "1000:1000"
volumes:
- ./log/peer:/var/log/dragonfly/daemon
- ./config/dfget.yaml:/etc/dragonfly/dfget.yaml:ro
Expand All @@ -83,6 +89,9 @@ services:
volumes:
- ./log/scheduler:/var/log/dragonfly/scheduler
- ./config/scheduler.yaml:/etc/dragonfly/scheduler.yaml:ro
# Enable when running in none root mode
# user: "1000:1000"
command: ["--verbose", "--console"]
ports:
- 8002:8002

Expand All @@ -101,6 +110,9 @@ services:
volumes:
- ./log/seed-peer:/var/log/dragonfly/daemon
- ./config/seed-peer.yaml:/etc/dragonfly/dfget.yaml:ro
# Enable when running in none root mode
# user: "1000:1000"
command: ["--verbose", "--console"]
ports:
- 65006:65006
- 65007:65007
Expand Down
15 changes: 10 additions & 5 deletions deploy/docker-compose/template/dfget.template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,27 +6,32 @@ aliveTime: 0s
gcInterval: 1m0s

# WorkHome is working directory.
# In linux, default value is /usr/local/dragonfly.
# In linux, default value is /usr/local/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly
# In macos(just for testing), default value is /Users/$USER/.dragonfly.
workHome: ''

# logDir is the log directory.
# In linux, default value is /var/log/dragonfly.
# In linux, default value is /var/log/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/logs
# In macos(just for testing), default value is /Users/$USER/.dragonfly/logs.
logDir: ''

# cacheDir is dynconfig cache directory.
# In linux, default value is /var/cache/dragonfly.
# In linux, default value is /var/cache/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/cache
# In macos(just for testing), default value is /Users/$USER/.dragonfly/cache.
cacheDir: ''

# pluginDir is the plugin directory.
# In linux, default value is /usr/local/dragonfly/plugins.
# In linux, default value is /usr/local/dragonfly/plugins,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/plugins
# In macos(just for testing), default value is /Users/$USER/.dragonfly/plugins.
pluginDir: ''

# dataDir is the download data directory.
# In linux, default value is /var/lib/dragonfly.
# In linux, default value is /var/lib/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/data
# In macos(just for testing), default value is /Users/$USER/.dragonfly/data.
dataDir: ''

Expand Down
21 changes: 17 additions & 4 deletions deploy/docker-compose/template/manager.template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,35 @@ server:
# REST server address
addr: :8080
# WorkHome is working directory.
# In linux, default value is /usr/local/dragonfly.
# In linux, default value is /usr/local/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly
# In macos(just for testing), default value is /Users/$USER/.dragonfly.
workHome: ''

# logDir is the log directory.
# In linux, default value is /var/log/dragonfly.
# In linux, default value is /var/log/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/logs
# In macos(just for testing), default value is /Users/$USER/.dragonfly/logs.
logDir: ''

# cacheDir is dynconfig cache directory.
# In linux, default value is /var/cache/dragonfly.
# In linux, default value is /var/cache/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/cache
# In macos(just for testing), default value is /Users/$USER/.dragonfly/cache.
cacheDir: ''

# pluginDir is the plugin directory.
# In linux, default value is /usr/local/dragonfly/plugins.
# In linux, default value is /usr/local/dragonfly/plugins,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/plugins
# In macos(just for testing), default value is /Users/$USER/.dragonfly/plugins.
pluginDir: ''

# dataDir is the download data directory.
# In linux, default value is /var/lib/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/data
# In macos(just for testing), default value is /Users/$USER/.dragonfly/data.
dataDir: ''

auth:
jwt:
# Realm name to display to the user, default value is Dragonfly.
Expand Down
17 changes: 11 additions & 6 deletions deploy/docker-compose/template/scheduler.template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,23 +9,28 @@ server:
# # Server host.
# host: localhost
# WorkHome is working directory.
# In linux, default value is /usr/local/dragonfly.
# In linux, default value is /usr/local/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly
# In macos(just for testing), default value is /Users/$USER/.dragonfly.
workHome: ''
# logDir is the log directory.
# In linux, default value is /var/log/dragonfly.
# In linux, default value is /var/log/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/logs
# In macos(just for testing), default value is /Users/$USER/.dragonfly/logs.
logDir: ''
# cacheDir is dynconfig cache directory.
# In linux, default value is /var/cache/dragonfly.
# In linux, default value is /var/cache/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/cache
# In macos(just for testing), default value is /Users/$USER/.dragonfly/cache.
cacheDir: ''
# pluginDir is the plugin directory.
# In linux, default value is /usr/local/dragonfly/plugins.
# In linux, default value is /usr/local/dragonfly/plugins,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/plugins
# In macos(just for testing), default value is /Users/$USER/.dragonfly/plugins.
pluginDir: ''
# dataDir is the directory.
# In linux, default value is /var/lib/dragonfly.
# dataDir is the download data directory.
# In linux, default value is /var/lib/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/data
# In macos(just for testing), default value is /Users/$USER/.dragonfly/data.
dataDir: ''

Expand Down
15 changes: 10 additions & 5 deletions deploy/docker-compose/template/seed-peer.template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,27 +6,32 @@ aliveTime: 0s
gcInterval: 1m0s

# WorkHome is working directory.
# In linux, default value is /usr/local/dragonfly.
# In linux, default value is /usr/local/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly
# In macos(just for testing), default value is /Users/$USER/.dragonfly.
workHome: ''

# logDir is the log directory.
# In linux, default value is /var/log/dragonfly.
# In linux, default value is /var/log/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/logs
# In macos(just for testing), default value is /Users/$USER/.dragonfly/logs.
logDir: ''

# cacheDir is dynconfig cache directory.
# In linux, default value is /var/cache/dragonfly.
# In linux, default value is /var/cache/dragonfly,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/cache
# In macos(just for testing), default value is /Users/$USER/.dragonfly/cache.
cacheDir: ''

# pluginDir is the plugin directory.
# In linux, default value is /usr/local/dragonfly/plugins.
# In linux, default value is /usr/local/dragonfly/plugins,
# And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/plugins
# In macos(just for testing), default value is /Users/$USER/.dragonfly/plugins.
pluginDir: ''

# dataDir is the download data directory.
# In linux, default value is /var/lib/dragonfly.
# In linux, default value is /var/lib/dragonfly,
# # And when running within none root, the path should locate to homedir, for example: /home/dragonfly/.dragonfly/data
# In macos(just for testing), default value is /Users/$USER/.dragonfly/data.
dataDir: ''

Expand Down
3 changes: 3 additions & 0 deletions manager/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ type ServerConfig struct {
// Server plugin directory.
PluginDir string `yaml:"pluginDir" mapstructure:"pluginDir"`

// Server storage data directory.
DataDir string `yaml:"dataDir" mapstructure:"dataDir"`

// GRPC server configuration.
GRPC GRPCConfig `yaml:"grpc" mapstructure:"grpc"`

Expand Down
Loading